././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7759676 pgbackrest-2.59.1/0000755000175100017510000000000015240547776013424 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/LICENSE0000664000175100017510000000222015240547630014414 0ustar00runnerrunnerThe MIT License (MIT) Portions Copyright (c) 2015-2026, The PostgreSQL Global Development Group Portions Copyright (c) 2013-2026, David Steele Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7759676 pgbackrest-2.59.1/README.md0000644000175100017510000000637315240547776014714 0ustar00runnerrunner# pgBackRest Distribution This is a pgBackRest build distribution. Unlike a checkout of the [git repository](https://github.com/pgbackrest/pgbackrest), it ships the generated source and the rendered documentation pre-built, so pgBackRest can be built and installed without the code generation or the documentation tooling. See for complete documentation. ## Contents - `src` -- pgBackRest source, including the pre-generated `*.auto.c.inc` files - `doc/man` -- command reference - `doc/html` -- HTML documentation - `test` -- smoke test to verify the build - `meson.build`, `meson_options.txt` -- build configuration - `LICENSE` ## Build pgBackRest builds with meson and ninja. The required libraries are libpq, OpenSSL (>= 1.1.1), libxml2, lz4, bz2, and zlib; zstd, libssh2 (for SFTP), and libsystemd are optional. For example, on Debian/Ubuntu: ``` apt-get install meson gcc pkg-config libpq-dev libssl-dev libxml2-dev \ liblz4-dev libzstd-dev libbz2-dev libz-dev libssh2-1-dev libsystemd-dev ``` Then configure, build, and install: ``` meson setup build . ninja -C build meson install -C build ``` The `pgbackrest` binary is built at `build/src/pgbackrest`. ## Test A smoke test is included to verify that the build works. It finds each PostgreSQL installation on the system and runs the newly-built `pgbackrest` through a backup and restore cycle against every supported version found: create a cluster, configure archiving, create the stanza, check, back up, restore, and verify that the restored data matches. Versions that pgBackRest does not support are reported and skipped. The test requires nothing beyond the build environment and PostgreSQL -- no additional Python packages. Run it with meson after building: ``` meson test -C build --suite smoke ``` Add `-v` to stream the output of each step as it runs: ``` meson test -C build --suite smoke -v ``` PostgreSQL is searched for in the standard packaging paths: - `/usr/lib/postgresql//bin` (Debian/Ubuntu) - `/usr/pgsql-/bin` (RHEL) - `/usr/libexec/postgresql` (Alpine) - `/usr/bin` Each version is tested in a temporary directory with its own repository and a private unix socket, so a running PostgreSQL cluster is not disturbed. The test fails when there is no PostgreSQL to test against, since a test that silently does nothing does not verify the build. Since PostgreSQL will not run as root, running the test as root drops to the `postgres` user and fails if that user does not exist. The test can also be run directly, which allows the defaults to be overridden: ``` python3 test/smoke.py --pgbackrest build/src/pgbackrest ``` Useful options are `--pg-bin-path` to search an additional path for PostgreSQL, `--version` to test only a specific version, and `--user` to select the user to drop to when running as root. Use `--help` for the complete list. ## Documentation The rendered documentation is in `doc`: the command reference in `doc/man` and the HTML documentation in `doc/html`. Two user guides are provided, Debian/Ubuntu and RHEL/Rocky/Alma. If only one user guide is required, replace user-guide-index.html with the desired user guide. For example, to keep only the Debian/Ubuntu user guide: ``` mv user-guide.html user-guide-index.html rm user-guide-rhel.html ``` ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7759676 pgbackrest-2.59.1/dist0000644000175100017510000000000715240547776014307 0ustar00runnerrunner2.59.1 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7726207 pgbackrest-2.59.1/doc/0000755000175100017510000000000015240547776014171 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7747571 pgbackrest-2.59.1/doc/html/0000755000175100017510000000000015240547776015135 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1786957821.774024 pgbackrest-2.59.1/doc/html/command.html0000640000175100017510000456052715240547776017460 0ustar00runnerrunner pgBackRest - Command Reference
pgBackRest
Command Reference
Home
User Guides
Releases
Configuration
FAQ
Table of Contents
Introduction
Annotate Command (annotate)
Archive Get Command (archive-get)
Archive Push Command (archive-push)
Backup Command (backup)
Check Command (check)
Expire Command (expire)
Help Command (help)
Info Command (info)
Repository Get Command (repo-get)
Repository List Command (repo-ls)
Restore Command (restore)
Server Command (server)
Server Ping Command (server-ping)
Stanza Create Command (stanza-create)
Stanza Delete Command (stanza-delete)
Stanza Upgrade Command (stanza-upgrade)
Start Command (start)
Stop Command (stop)
Verify Command (verify)
Version Command (version)
Introduction
Commands are used to execute the various pgBackRest functions. Here the command options are listed exhaustively, that is, each option applicable to a command is listed with that command even if it applies to one or more other commands. This includes all the options that may also be configured in pgbackrest.conf.
Non-boolean options configured in pgbackrest.conf can be reset to default on the command-line by using the reset- prefix. This feature may be used to restore a backup directly on a repository host. Normally, pgBackRest will error because it can see that the database host is remote and restores cannot be done remotely. By adding --reset-pg1-host on the command-line, pgBackRest will ignore the remote database host and restore locally. It may be necessary to pass a new --pg1-path to force the restore to happen in a specific path, i.e. not the path used on the database host.
The no- prefix may be used to set a boolean option to false on the command-line.
Any option may be set in an environment variable using the PGBACKREST_ prefix and the option name in all caps replacing - with _, e.g. pg1-path becomes PGBACKREST_PG1_PATH and stanza becomes PGBACKREST_STANZA. Boolean options are represented as they would be in a configuration file, e.g. PGBACKREST_COMPRESS="n", and reset-* variants are not allowed. Options that can be specified multiple times on the command-line or in a config file can be represented by separating the values with colons, e.g. PGBACKREST_DB_INCLUDE="db1:db2".
Command-line options override environment options which override config file options.
See Configuration Introduction for information on option types
Annotate Command (annotate)
Annotations included with the backup command can be added, modified, or removed afterwards using the annotate command.
Command Options
Backup Annotation Option (--annotation)
Annotate backup with user-defined key/value pairs.
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output. 
example: --annotation=source="Sunday backup for website database"
Set Option (--set)
Backup set to annotate.
The backup set to annotate. 
example: --set=20150131-153358F_20150131-153401I
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Archive Get Command (archive-get)
This command is used by PostgreSQL to restore a backup, perform PITR, or as an alternative to streaming for keeping a replica up to date. WAL segments are required for PostgreSQL recovery or to maintain a replica.
When multiple repositories are configured, WAL will be fetched from the repositories in priority order (e.g. repo1, repo2, etc.). In general it is better if faster/cheaper storage has higher priority. If a repository is specified with the --repo option then only that repository will be searched.
The archive-get command is configured and generated by pgBackRest during a restore for use by PostgreSQL. See Point-in-Time Recovery for an example.
Command Options
Asynchronous Archiving Option (--archive-async)
Push/get WAL segments asynchronously.
Enables asynchronous operation for the archive-push and archive-get commands.
Asynchronous operation is more efficient because it can reuse connections and take advantage of parallelism. See the spool-path, archive-get-queue-max, and archive-push-queue-max options for more information. 
default: n
example: --archive-async
Maximum Archive Get Queue Size Option (--archive-get-queue-max)
Maximum size of the pgBackRest archive-get queue.
Specifies the maximum size of the archive-get queue when archive-async is enabled. The queue is stored in the spool-path and is used to speed providing WAL to PostgreSQL. 
default: 128MiB
allowed: [0B, 4PiB]
example: --archive-get-queue-max=1GiB
Retry Missing WAL Segment Option (--archive-missing-retry)
Retry missing WAL segment
Retry a WAL segment that was previously reported as missing by the archive-get command when in asynchronous mode. This prevents notifications in the spool path from a prior restore from being used and possibly causing a recovery failure if consistency has not been reached.
Disabling this option allows PostgreSQL to more reliably recognize when the end of the WAL in the archive has been reached, which permits it to switch over to streaming from the primary. With retries enabled, a steady stream of WAL being archived will cause PostgreSQL to continue getting WAL from the archive rather than switch to streaming.
When disabling this option it is important to ensure that the spool path for the stanza is empty. The restore command does this automatically if the spool path is configured at restore time. Otherwise, it is up to the user to ensure the spool path is empty. 
default: y
example: --no-archive-missing-retry
Archive Timeout Option (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: --archive-timeout=30
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
pgBackRest Command Option (--cmd)
pgBackRest command.
pgBackRest may generate a command string, e.g. when the restore command generates the restore_command setting. The command used to run the pgBackRest process will be used in this case unless the cmd option is provided.
CAUTION:
Wrapping the pgBackRest command may cause unpredictable behavior and is not recommended. 
default: [path of executed pgbackrest binary]
example: --cmd=/var/lib/pgsql/bin/pgbackrest_wrapper.sh
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Process Maximum Option (--process-max)
Max processes to use for compress/transfer.
Each process will perform compression and transfer to make the command run faster, but don't set process-max so high that it impacts database performance. 
default: 1
allowed: [1, 999]
example: --process-max=4
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Spool Path Option (--spool-path)
Path where transient data is stored.
This path is used to store data for the asynchronous archive-push and archive-get command.
The asynchronous archive-push command writes acknowledgements into the spool path when it has successfully stored WAL in the archive (and errors on failure) so the foreground process can quickly notify PostgreSQL. Acknowledgement files are very small (zero on success and a few hundred bytes on error).
The asynchronous archive-get command queues WAL in the spool path so it can be provided very quickly when PostgreSQL requests it. Moving files to PostgreSQL is most efficient when the spool path is on the same filesystem as pg_xlog/pg_wal. However, it is not recommended to place the spool path within the pg_xlog/pg_wal directory as this may cause issues for PostgreSQL utilities such as pg_rewind.
The data stored in the spool path is not strictly temporary since it can and should survive a reboot. However, loss of the data in the spool path is not a problem. pgBackRest will simply recheck each WAL segment to ensure it is safely archived for archive-push and rebuild the queue for archive-get.
The spool path is intended to be located on a local Posix-compatible filesystem, not a remote filesystem such as NFS or CIFS. 
default: /var/spool/pgbackrest
example: --spool-path=/backup/db/spool
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Maintainer Options
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: --pg-version-force=15
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Target Time for Repository Option (--repo-target-time)
Target time for repository.
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided. 
example: --repo-target-time=2024-08-08 12:12:12+00
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Stanza Options
PostgreSQL Path Option (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: --pg1-path=/data/db
Deprecated Name: db-path
Archive Push Command (archive-push)
Accepts a WAL segment from PostgreSQL and archives it in each repository defined by the indexed repo-path option (see the Repository section for information on configuring repositories). The WAL segment may be pushed immediately to the archive or stored locally depending on the value of archive-async. With multiple repositories configured, archive-push will attempt to push to as many repositories as possible.
The archive-push is intended to be configured and called by PostgreSQL. See Configure Archiving for an example.
Command Options
Asynchronous Archiving Option (--archive-async)
Push/get WAL segments asynchronously.
Enables asynchronous operation for the archive-push and archive-get commands.
Asynchronous operation is more efficient because it can reuse connections and take advantage of parallelism. See the spool-path, archive-get-queue-max, and archive-push-queue-max options for more information. 
default: n
example: --archive-async
Check Archive Option (--archive-check)
Check that WAL segments are in the archive before backup completes.
Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a good idea to leave this as the default unless you are using another method for archiving.
This option must be enabled if archive-copy is enabled. 
default: y
example: --no-archive-check
Check Archive Mode Option (--archive-mode-check)
Check the PostgreSQL archive_mode setting.
Enabled by default, this option disallows PostgreSQL archive_mode=always.
WAL segments pushed from a standby server might be logically the same as WAL segments pushed from the primary but have different checksums. Disabling archiving from multiple sources is recommended to avoid conflicts.
CAUTION:
If this option is disabled then it is critical to ensure that only one archiver is writing to the repository via the archive-push command. 
default: y
example: --no-archive-mode-check
Archive Push Batch Size Option (--archive-push-batch-size)
Maximum amount of WAL to push per asynchronous run.
In asynchronous mode the archive-push process pushes all the WAL segments that are ready in a single run. Since archive-push-queue-max is only checked at the start of each run, a run that processes a very large number of segments can let the queue grow well beyond the limit before it is rechecked.
This option limits the amount of WAL processed per run so the process exits and is spawned again by the next archive-push, which rechecks the queue. Lower values recheck the queue more often at the cost of spawning the asynchronous process more frequently. The value is rounded down to a whole number of WAL segments but at least one segment is always processed. 
default: 16GiB
allowed: [1MiB, 4PiB]
example: --archive-push-batch-size=1GiB
Maximum Archive Push Queue Size Option (--archive-push-queue-max)
Maximum size of the PostgreSQL archive queue.
After the limit is reached, the following will happen:
  • pgBackRest will notify PostgreSQL that the WAL was successfully archived, then DROP IT.
  • A warning will be output to the PostgreSQL log.
If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.
In asynchronous mode the entire queue will be dropped to prevent spurts of WAL getting through before the queue limit is exceeded again.
In asynchronous mode this limit is only checked at the start of each archive-push run, so the queue can grow beyond it within a single run. Reduce archive-push-batch-size to check the queue more frequently.
The purpose of this feature is to prevent the log volume from filling up at which point PostgreSQL will stop completely. Better to lose the backup than have PostgreSQL go down. 
allowed: [0B, 4PiB]
example: --archive-push-queue-max=1TiB
Deprecated Name: archive-queue-max
Archive Timeout Option (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: --archive-timeout=30
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
pgBackRest Command Option (--cmd)
pgBackRest command.
pgBackRest may generate a command string, e.g. when the restore command generates the restore_command setting. The command used to run the pgBackRest process will be used in this case unless the cmd option is provided.
CAUTION:
Wrapping the pgBackRest command may cause unpredictable behavior and is not recommended. 
default: [path of executed pgbackrest binary]
example: --cmd=/var/lib/pgsql/bin/pgbackrest_wrapper.sh
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Compress Option (--compress)
Use file compression.
Backup files are compatible with command-line compression tools.
This option is now deprecated. The compress-type option should be used instead. 
default: y
example: --no-compress
Compress Level Option (--compress-level)
File compression level.
Sets the level to be used for file compression when compress-type does not equal none or compress=y (deprecated). 
default (depending on compress-type):
    bz2 - 9
    gz - 6
    lz4 - 1
    zst - 3

allow range (depending on compress-type):
    bz2 - [1, 9]
    gz - [-1, 9]
    lz4 - [-5, 12]
    zst - [-7, 22]

example: --compress-level=9
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Compress Type Option (--compress-type)
File compression type.
The following compression types are supported:
  • none - no compression
  • bz2 - bzip2 compression format
  • gz - gzip compression format
  • lz4 - lz4 compression format (not available on all platforms)
  • zst - Zstandard compression format (not available on all platforms) 
default: gz
example: --compress-type=none
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Process Maximum Option (--process-max)
Max processes to use for compress/transfer.
Each process will perform compression and transfer to make the command run faster, but don't set process-max so high that it impacts database performance. 
default: 1
allowed: [1, 999]
example: --process-max=4
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Spool Path Option (--spool-path)
Path where transient data is stored.
This path is used to store data for the asynchronous archive-push and archive-get command.
The asynchronous archive-push command writes acknowledgements into the spool path when it has successfully stored WAL in the archive (and errors on failure) so the foreground process can quickly notify PostgreSQL. Acknowledgement files are very small (zero on success and a few hundred bytes on error).
The asynchronous archive-get command queues WAL in the spool path so it can be provided very quickly when PostgreSQL requests it. Moving files to PostgreSQL is most efficient when the spool path is on the same filesystem as pg_xlog/pg_wal. However, it is not recommended to place the spool path within the pg_xlog/pg_wal directory as this may cause issues for PostgreSQL utilities such as pg_rewind.
The data stored in the spool path is not strictly temporary since it can and should survive a reboot. However, loss of the data in the spool path is not a problem. pgBackRest will simply recheck each WAL segment to ensure it is safely archived for archive-push and rebuild the queue for archive-get.
The spool path is intended to be located on a local Posix-compatible filesystem, not a remote filesystem such as NFS or CIFS. 
default: /var/spool/pgbackrest
example: --spool-path=/backup/db/spool
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Maintainer Options
Check WAL Headers Option (--archive-header-check)
Check PostgreSQL version/id in WAL headers.
Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located.
Therefore, disabling this check is fairly safe but should only be done when needed, e.g. if the WAL is encrypted. 
default: y
example: --no-archive-header-check
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: --pg-version-force=15
Repository Options
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Stanza Options
PostgreSQL Path Option (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: --pg1-path=/data/db
Deprecated Name: db-path
Backup Command (backup)
When multiple repositories are configured, pgBackRest will backup to the highest priority repository (e.g. repo1) unless the --repo option is specified.
pgBackRest does not have a built-in scheduler so it's best to run it from cron or some other scheduling mechanism.
See Perform a Backup for more details and examples.
Command Options
Backup Annotation Option (--annotation)
Annotate backup with user-defined key/value pairs.
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output. 
example: --annotation=source="Sunday backup for website database"
Check Archive Option (--archive-check)
Check that WAL segments are in the archive before backup completes.
Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a good idea to leave this as the default unless you are using another method for archiving.
This option must be enabled if archive-copy is enabled. 
default: y
example: --no-archive-check
Copy Archive Option (--archive-copy)
Copy WAL segments needed for consistency to the backup.
This slightly paranoid option protects against corruption in the WAL segment archive by storing the WAL segments required for consistency directly in the backup. WAL segments are still stored in the archive so this option will use additional space.
It is best if the archive-push and backup commands have the same compress-type (e.g. lz4) when using this option. Otherwise, the WAL segments will need to be recompressed with the compress-type used by the backup, which can be fairly expensive depending on how much WAL was generated during the backup.
On restore, the WAL segments will be present in pg_xlog/pg_wal and PostgreSQL will use them in preference to calling the restore_command.
The archive-check option must be enabled if archive-copy is enabled. 
default: n
example: --archive-copy
Check Archive Mode Option (--archive-mode-check)
Check the PostgreSQL archive_mode setting.
Enabled by default, this option disallows PostgreSQL archive_mode=always.
WAL segments pushed from a standby server might be logically the same as WAL segments pushed from the primary but have different checksums. Disabling archiving from multiple sources is recommended to avoid conflicts.
CAUTION:
If this option is disabled then it is critical to ensure that only one archiver is writing to the repository via the archive-push command. 
default: y
example: --no-archive-mode-check
Archive Timeout Option (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: --archive-timeout=30
Backup from Standby Option (--backup-standby)
Backup from the standby cluster.
Enable backup from standby to reduce load on the primary cluster. This option requires that both the primary and standby hosts be configured.
The following modes are supported:
  • y - Standby is required for backup.
  • prefer - Backup from standby if available otherwise backup from primary.
  • n - Backup from primary only. 
default: n
example: --backup-standby=y
Page Checksums Option (--checksum-page)
Validate data page checksums.
Directs pgBackRest to validate all data page checksums while backing up a cluster. This option is automatically enabled when data page checksums are enabled on the cluster.
Failures in checksum validation will not abort a backup. Rather, warnings will be emitted in the log (and to the console with default settings) and the list of invalid pages will be stored in the backup manifest. 
example: --no-checksum-page
Path/File Exclusions Option (--exclude)
Exclude paths/files from the backup.
All exclusions are relative to $PGDATA. If the exclusion ends with / then only files in the specified directory will be excluded, e.g. --exclude=junk/ will exclude all files in the $PGDATA/junk directory but include the directory itself. If the exclusion does not end with / then the file may match the exclusion exactly or match with / appended to the exclusion, e.g. --exclude=junk will exclude the $PGDATA/junk directory and all the files it contains.
Be careful using this feature -- it is very easy to exclude something critical that will make the backup inconsistent. Be sure to test your restores!
All excluded files will be logged at info level along with the exclusion rule. Be sure to audit the list of excluded files to ensure nothing unexpected is being excluded.
NOTE:
Exclusions are not honored on delta restores. Any files/directories that were excluded by the backup will be removed on delta restore.
This option should not be used to exclude PostgreSQL logs from a backup. Logs can be moved out of the PGDATA directory using the PostgreSQL log_directory setting, which has the benefit of allowing logs to be preserved after a restore.
Multiple exclusions may be specified on the command-line or in a configuration file. 
example: --exclude=junk/
Expire Auto Option (--expire-auto)
Automatically run the expire command after a successful backup.
The setting is enabled by default. Use caution when disabling this option as doing so will result in retaining all backups and archives indefinitely, which could cause your repository to run out of space. The expire command will need to be run regularly to prevent this from happening.
When expire is run automatically after a successful backup it uses the configuration of the backup command, so options set only in an expire command section (e.g. [global:expire]) are not applied. To apply expire-specific configuration, disable this option and run the expire command separately. 
default: y
example: --expire-auto
Force Option (--force)
Force an offline backup.
When used with --no-start-stop a backup will be run even if pgBackRest thinks that PostgreSQL is running. This option should be used with extreme care as it will likely result in a bad backup.
There are some scenarios where a backup might still be desirable under these conditions. For example, if a server crashes and the database cluster volume can only be mounted read-only, it would be a good idea to take a backup even if postmaster.pid is present. In this case it would be better to revert to the prior backup and replay WAL, but possibly there is a very important transaction in a WAL segment that did not get archived. 
default: n
example: --force
Manifest Save Threshold Option (--manifest-save-threshold)
Manifest save threshold during backup.
Defines how often the manifest will be saved during a backup. Saving the manifest is important because it stores the checksums and allows the resume function to work efficiently. The actual threshold used is 1% of the backup size or manifest-save-threshold, whichever is greater. 
default: 1GiB
allowed: [1B, 1TiB]
example: --manifest-save-threshold=8GiB
Online Option (--online)
Perform an online backup.
Specifying --no-online prevents pgBackRest from running the backup start/stop functions on the database cluster. In order for this to work PostgreSQL should be shut down and pgBackRest will generate an error if it is not.
The purpose of this option is to allow offline backups. The pg_xlog/pg_wal directory is copied as-is and archive-check is automatically disabled for the backup. 
default: y
example: --no-online
Resume Option (--resume)
Allow resume of failed backup.
Defines whether the resume feature is enabled. Resume can greatly reduce the amount of time required to run a backup after a previous backup of the same type has failed. It adds complexity, however, so it may be desirable to disable in environments that do not require the feature. 
default: y
example: --no-resume
Start Fast Option (--start-fast)
Force a checkpoint to start backup quickly.
Forces a checkpoint (by passing y to the fast parameter of the backup start function) so the backup begins immediately. Otherwise the backup will start after the next regular checkpoint. 
default: n
example: --start-fast
Type Option (--type)
Backup type.
The following backup types are supported:
  • full - all database cluster files will be copied and there will be no dependencies on previous backups.
  • incr - incremental from the last successful backup.
  • diff - like an incremental backup but always based on the last full backup. 
default: incr
example: --type=full
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
pgBackRest Command Option (--cmd)
pgBackRest command.
pgBackRest may generate a command string, e.g. when the restore command generates the restore_command setting. The command used to run the pgBackRest process will be used in this case unless the cmd option is provided.
CAUTION:
Wrapping the pgBackRest command may cause unpredictable behavior and is not recommended. 
default: [path of executed pgbackrest binary]
example: --cmd=/var/lib/pgsql/bin/pgbackrest_wrapper.sh
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Compress Option (--compress)
Use file compression.
Backup files are compatible with command-line compression tools.
This option is now deprecated. The compress-type option should be used instead. 
default: y
example: --no-compress
Compress Level Option (--compress-level)
File compression level.
Sets the level to be used for file compression when compress-type does not equal none or compress=y (deprecated). 
default (depending on compress-type):
    bz2 - 9
    gz - 6
    lz4 - 1
    zst - 3

allow range (depending on compress-type):
    bz2 - [1, 9]
    gz - [-1, 9]
    lz4 - [-5, 12]
    zst - [-7, 22]

example: --compress-level=9
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Compress Type Option (--compress-type)
File compression type.
The following compression types are supported:
  • none - no compression
  • bz2 - bzip2 compression format
  • gz - gzip compression format
  • lz4 - lz4 compression format (not available on all platforms)
  • zst - Zstandard compression format (not available on all platforms) 
default: gz
example: --compress-type=none
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Database Timeout Option (--db-timeout)
Database query timeout.
Sets the timeout, in seconds, for queries against the database. This includes the backup start/stop functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set start-fast=y and you know that the database cluster will not generate many WAL segments during the backup).
NOTE:
The db-timeout option must be less than the protocol-timeout option. 
default: 30m
allowed: [100ms, 7d]
example: --db-timeout=600
Delta Option (--delta)
Restore or backup using checksums.
During a restore, by default the PostgreSQL data and tablespace directories are expected to be present but empty. This option performs a delta restore using checksums.
During a backup, this option will use checksums instead of the timestamps to determine if files will be copied. 
default: n
example: --delta
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Process Maximum Option (--process-max)
Max processes to use for compress/transfer.
Each process will perform compression and transfer to make the command run faster, but don't set process-max so high that it impacts database performance. 
default: 1
allowed: [1, 999]
example: --process-max=4
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Maintainer Options
Page Header Check Option (--page-header-check)
Check PostgreSQL page headers.
Enabled by default, this option adds page header checks.
Disabling this option should be avoided except when necessary, e.g. if pages are encrypted. 
default: y
example: --no-page-header-check
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: --pg-version-force=15
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Block Incremental Backup Option (--repo-block)
Enable block incremental backup.
Block incremental allows for more granular backups by splitting files into blocks that can be backed up independently. This saves space in the repository and can improve delta restore performance because individual blocks can be fetched without reading the entire file from the repository.
NOTE:
The repo-bundle option must be enabled before repo-block can be enabled.
The block size for a file is determined based on the file size and age. Generally, older/larger files will get larger block sizes. If a file is old enough, it will not be backed up using block incremental.
Block incremental is most efficient when enabled for all backup types, including full. This makes the full a bit larger but subsequent differential and incremental backups can make use of the block maps generated by the full backup to save space. 
default: n
example: --repo1-block
Repository Bundles Option (--repo-bundle)
Bundle files in repository.
Bundle (combine) smaller files to reduce the total number of files written to the repository. Writing fewer files is generally more efficient, especially on object stores such as S3. In addition, zero-length files are not stored (except in the manifest), which saves time and space. 
default: n
example: --repo1-bundle
Repository Bundle Limit Option (--repo-bundle-limit)
Limit for file bundles.
Size limit for files that will be included in bundles. Files larger than this size will be stored separately.
Bundled files cannot be reused when a backup is resumed, so this option controls the files that can be resumed, i.e. higher values result in fewer resumable files. 
default: 2MiB
allowed: [8KiB, 1PiB]
example: --repo1-bundle-limit=10MiB
Repository Bundle Size Option (--repo-bundle-size)
Target size for file bundles.
Defines the target size for files that will be added to a single bundle. The uncompressed bundle size may be as large as repo-bundle-size + repo-bundle-limit, so do not set this option to the maximum size that your file system allows.
In general, it is not a good idea to set this option too high because retries will need to redo the entire bundle. 
default: 20MiB
allowed: [1MiB, 1PiB]
example: --repo1-bundle-size=10MiB
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Hardlink Option (--repo-hardlink)
Hardlink files between backups in the repository.
Enable hard-linking of files in differential and incremental backups to their full backups. This gives the appearance that each backup is a full backup at the file-system level. Be careful, though, because modifying files that are hard-linked can affect all the backups in the set. 
default: n
example: --repo1-hardlink
Deprecated Name: hardlink
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
Archive Retention Option (--repo-retention-archive)
Number of backups worth of continuous WAL to retain.
NOTE:
WAL segments required to make a backup consistent are always retained until the backup is expired regardless of how this option is configured.
If this value is not set and repo-retention-full-type is count (default), then the archive to expire will default to the repo-retention-full (or repo-retention-diff) value corresponding to the repo-retention-archive-type if set to full (or diff). This will ensure that WAL is only expired for backups that are already expired. If repo-retention-full-type is time, then this value will default to removing archives that are earlier than the oldest full backup retained after satisfying the repo-retention-full setting.
This option must be set if repo-retention-archive-type is set to incr. If disk space is at a premium, then this setting, in conjunction with repo-retention-archive-type, can be used to aggressively expire WAL segments. However, doing so negates the ability to perform PITR from the backups with expired WAL and is therefore not recommended. 
allowed: [1, 9999999]
example: --repo1-retention-archive=2
Deprecated Name: retention-archive
Archive Retention Type Option (--repo-retention-archive-type)
Backup type for WAL retention.
If set to full pgBackRest will keep archive logs for the number of full backups defined by repo-retention-archive. If set to diff (differential) pgBackRest will keep archive logs for the number of full and differential backups defined by repo-retention-archive, meaning if the last backup taken was a full backup, it will be counted as a differential for the purpose of repo-retention. If set to incr (incremental) pgBackRest will keep archive logs for the number of full, differential, and incremental backups defined by repo-retention-archive. It is recommended that this setting not be changed from the default which will only expire WAL in conjunction with expiring full backups. 
default: full
example: --repo1-retention-archive-type=diff
Deprecated Name: retention-archive-type
Differential Retention Option (--repo-retention-diff)
Number of differential backups to retain.
When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept until the full backups they depend on expire.
Note that full backups are included in the count of differential backups for the purpose of expiration. This slightly reduces the number of differential backups that need to be retained in most cases. 
allowed: [1, 9999999]
example: --repo1-retention-diff=3
Deprecated Name: retention-diff
Full Retention Option (--repo-retention-full)
Full backup retention count/time.
When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When the option is not defined a warning will be issued. If indefinite retention is desired then set the option to the max value. 
allowed: [1, 9999999]
example: --repo1-retention-full=2
Deprecated Name: retention-full
Full Retention Type Option (--repo-retention-full-type)
Retention type for full backups.
Determines whether the repo-retention-full setting represents a time period (days) or count of full backups to keep.
If set to time then full backups older than repo-retention-full will be removed from the repository if there is at least one other backup that is equal to or greater than the repo-retention-full setting. For example, if repo-retention-full is 30 (days) and there are 2 full backups: one 25 days old and one 35 days old, no full backups will be expired because expiring the 35 day old backup would leave only the 25 day old backup, which would violate the 30 day retention policy of having at least one backup 30 days old before an older one can be expired. Archived WAL older than the oldest full backup remaining will be automatically expired unless repo-retention-archive-type and repo-retention-archive are explicitly set.
If set to count then full backups that exceed repo-retention-full will be expired. For example, if repo-retention-full is 4 and a fifth full backup is completed, then the oldest full backup will be expired to keep the count at 4.
Note that a backup must be successfully completed before it will be considered for retention. For example, if repo-retention-full-type is count and repo-retention-full is 2, then there must be 3 complete full backups before the oldest will be expired. 
default: count
example: --repo1-retention-full-type=time
Backup History Retention Option (--repo-retention-history)
Days of backup history manifests to retain.
A copy of the backup manifest is stored in the backup.history path when a backup completes. By default these files are never expired since they are useful for data mining, e.g. measuring backup and WAL growth over time.
Set repo-retention-history to define the number of days of backup history manifests to retain. Unexpired backups are always kept in the backup history. Specify repo-retention-history=0 to retain the backup history only for unexpired backups.
When a full backup history manifest is expired, all differential and incremental backup history manifests associated with the full backup also expire. 
allowed: [0, 9999999]
example: --repo1-retention-history=365
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Symlink Option (--repo-symlink)
Create symlinks within the repository.
Enable creation of the lastest and tablespace symlinks. These symlinks are most useful when using snapshots to do in-place recovery in the repository, which is an uncommon use case.
While this feature is likely not useful for the vast majority of users it remains on by default for legacy purposes. However, it may be useful to disable symlinks for Posix-like storage that does not support them. 
default: y
example: --no-repo1-symlink
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Stanza Options
PostgreSQL Database Option (--pg-database)
PostgreSQL database.
The database name used when connecting to PostgreSQL. The default is usually best but some installations may not contain this database.
Note that for legacy reasons the setting of the PGDATABASE environment variable will be ignored. 
default: postgres
example: --pg1-database=backupdb
PostgreSQL Host Option (--pg-host)
PostgreSQL host for operating remotely.
Used for backups where the PostgreSQL host is different from the repository host. 
example: --pg1-host=db.domain.com
Deprecated Name: db-host
PostgreSQL Host Certificate Authority File Option (--pg-host-ca-file)
PostgreSQL host certificate authority file.
Use a CA file other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
PostgreSQL Host Certificate Authority Path Option (--pg-host-ca-path)
PostgreSQL host certificate authority path.
Use a CA path other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-path=/etc/pki/tls/certs
PostgreSQL Host Certificate File Option (--pg-host-cert-file)
PostgreSQL host certificate file.
Sent to PostgreSQL host to prove client identity. 
example: --pg1-host-cert-file=/path/to/client.crt
PostgreSQL Host Command Option (--pg-host-cmd)
PostgreSQL host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and PostgreSQL hosts. If not defined, the PostgreSQL host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --pg1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: db-cmd
PostgreSQL Host Configuration Option (--pg-host-config)
pgBackRest database host configuration file.
Sets the location of the configuration file on the PostgreSQL host. This is only required if the PostgreSQL host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --pg1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: db-config
PostgreSQL Host Configuration Include Path Option (--pg-host-config-include-path)
pgBackRest database host configuration include path.
Sets the location of the configuration include path on the PostgreSQL host. This is only required if the PostgreSQL host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --pg1-host-config-include-path=/conf/pgbackrest/conf.d
PostgreSQL Host Configuration Path Option (--pg-host-config-path)
pgBackRest database host configuration path.
Sets the location of the configuration path on the PostgreSQL host. This is only required if the PostgreSQL host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --pg1-host-config-path=/conf/pgbackrest
PostgreSQL Host Key File Option (--pg-host-key-file)
PostgreSQL host key file.
Proves client certificate was sent by owner. 
example: --pg1-host-key-file=/path/to/client.key
PostgreSQL Host Port Option (--pg-host-port)
PostgreSQL host port when pg-host is set.
Use this option to specify a non-default port for the PostgreSQL host protocol.
NOTE:
When pg-host-type=ssh there is no default for pg-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on pg-host-type):
    tls - 8432

allowed: [0, 65535]
example: --pg1-host-port=25
Deprecated Name: db-ssh-port
PostgreSQL Host Protocol Type Option (--pg-host-type)
PostgreSQL host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --pg1-host-type=tls
PostgreSQL Host User Option (--pg-host-user)
PostgreSQL host logon user when pg-host is set.
This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this to work correctly the user should be the PostgreSQL database cluster owner which is generally postgres, the default. 
default: postgres
example: --pg1-host-user=db_owner
Deprecated Name: db-user
PostgreSQL Path Option (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: --pg1-path=/data/db
Deprecated Name: db-path
PostgreSQL Port Option (--pg-port)
PostgreSQL port.
Port that PostgreSQL is running on. This usually does not need to be specified as most PostgreSQL clusters run on the default port. 
default: 5432
allowed: [0, 65535]
example: --pg1-port=6543
Deprecated Name: db-port
PostgreSQL Socket Path Option (--pg-socket-path)
PostgreSQL unix socket path.
The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look in the standard location for your OS so there is usually no need to specify this setting unless the socket directory was explicitly modified with the unix_socket_directories setting in postgresql.conf. 
example: --pg1-socket-path=/var/run/postgresql
Deprecated Name: db-socket-path
PostgreSQL Database User Option (--pg-user)
PostgreSQL database user.
The database user name used when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER. 
example: --pg1-user=backupuser
Check Command (check)
The check command validates that pgBackRest and the archive_command setting are configured correctly for archiving and backups for the specified stanza. It will attempt to check all repositories and databases that are configured for the host on which the command is run. It detects misconfigurations, particularly in archiving, that result in incomplete backups because required WAL segments did not reach the archive. The command can be run on the PostgreSQL or repository host. The command may also be run on the standby host, however, since pg_switch_xlog()/pg_switch_wal() cannot be performed on the standby, the command will only test the repository configuration.
Note that pg_create_restore_point('pgBackRest Archive Check') and pg_switch_xlog()/pg_switch_wal() are called to force PostgreSQL to archive a WAL segment.
Command Options
Check Archive Option (--archive-check)
Check that WAL segments are in the archive before backup completes.
Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a good idea to leave this as the default unless you are using another method for archiving.
This option must be enabled if archive-copy is enabled. 
default: y
example: --no-archive-check
Check Archive Mode Option (--archive-mode-check)
Check the PostgreSQL archive_mode setting.
Enabled by default, this option disallows PostgreSQL archive_mode=always.
WAL segments pushed from a standby server might be logically the same as WAL segments pushed from the primary but have different checksums. Disabling archiving from multiple sources is recommended to avoid conflicts.
CAUTION:
If this option is disabled then it is critical to ensure that only one archiver is writing to the repository via the archive-push command. 
default: y
example: --no-archive-mode-check
Archive Timeout Option (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: --archive-timeout=30
Backup from Standby Option (--backup-standby)
Backup from the standby cluster.
Enable backup from standby to reduce load on the primary cluster. This option requires that both the primary and standby hosts be configured.
The following modes are supported:
  • y - Standby is required for backup.
  • prefer - Backup from standby if available otherwise backup from primary.
  • n - Backup from primary only. 
default: n
example: --backup-standby=y
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Database Timeout Option (--db-timeout)
Database query timeout.
Sets the timeout, in seconds, for queries against the database. This includes the backup start/stop functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set start-fast=y and you know that the database cluster will not generate many WAL segments during the backup).
NOTE:
The db-timeout option must be less than the protocol-timeout option. 
default: 30m
allowed: [100ms, 7d]
example: --db-timeout=600
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Maintainer Options
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: --pg-version-force=15
Repository Options
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Stanza Options
PostgreSQL Database Option (--pg-database)
PostgreSQL database.
The database name used when connecting to PostgreSQL. The default is usually best but some installations may not contain this database.
Note that for legacy reasons the setting of the PGDATABASE environment variable will be ignored. 
default: postgres
example: --pg1-database=backupdb
PostgreSQL Host Option (--pg-host)
PostgreSQL host for operating remotely.
Used for backups where the PostgreSQL host is different from the repository host. 
example: --pg1-host=db.domain.com
Deprecated Name: db-host
PostgreSQL Host Certificate Authority File Option (--pg-host-ca-file)
PostgreSQL host certificate authority file.
Use a CA file other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
PostgreSQL Host Certificate Authority Path Option (--pg-host-ca-path)
PostgreSQL host certificate authority path.
Use a CA path other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-path=/etc/pki/tls/certs
PostgreSQL Host Certificate File Option (--pg-host-cert-file)
PostgreSQL host certificate file.
Sent to PostgreSQL host to prove client identity. 
example: --pg1-host-cert-file=/path/to/client.crt
PostgreSQL Host Command Option (--pg-host-cmd)
PostgreSQL host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and PostgreSQL hosts. If not defined, the PostgreSQL host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --pg1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: db-cmd
PostgreSQL Host Configuration Option (--pg-host-config)
pgBackRest database host configuration file.
Sets the location of the configuration file on the PostgreSQL host. This is only required if the PostgreSQL host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --pg1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: db-config
PostgreSQL Host Configuration Include Path Option (--pg-host-config-include-path)
pgBackRest database host configuration include path.
Sets the location of the configuration include path on the PostgreSQL host. This is only required if the PostgreSQL host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --pg1-host-config-include-path=/conf/pgbackrest/conf.d
PostgreSQL Host Configuration Path Option (--pg-host-config-path)
pgBackRest database host configuration path.
Sets the location of the configuration path on the PostgreSQL host. This is only required if the PostgreSQL host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --pg1-host-config-path=/conf/pgbackrest
PostgreSQL Host Key File Option (--pg-host-key-file)
PostgreSQL host key file.
Proves client certificate was sent by owner. 
example: --pg1-host-key-file=/path/to/client.key
PostgreSQL Host Port Option (--pg-host-port)
PostgreSQL host port when pg-host is set.
Use this option to specify a non-default port for the PostgreSQL host protocol.
NOTE:
When pg-host-type=ssh there is no default for pg-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on pg-host-type):
    tls - 8432

allowed: [0, 65535]
example: --pg1-host-port=25
Deprecated Name: db-ssh-port
PostgreSQL Host Protocol Type Option (--pg-host-type)
PostgreSQL host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --pg1-host-type=tls
PostgreSQL Host User Option (--pg-host-user)
PostgreSQL host logon user when pg-host is set.
This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this to work correctly the user should be the PostgreSQL database cluster owner which is generally postgres, the default. 
default: postgres
example: --pg1-host-user=db_owner
Deprecated Name: db-user
PostgreSQL Path Option (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: --pg1-path=/data/db
Deprecated Name: db-path
PostgreSQL Port Option (--pg-port)
PostgreSQL port.
Port that PostgreSQL is running on. This usually does not need to be specified as most PostgreSQL clusters run on the default port. 
default: 5432
allowed: [0, 65535]
example: --pg1-port=6543
Deprecated Name: db-port
PostgreSQL Socket Path Option (--pg-socket-path)
PostgreSQL unix socket path.
The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look in the standard location for your OS so there is usually no need to specify this setting unless the socket directory was explicitly modified with the unix_socket_directories setting in postgresql.conf. 
example: --pg1-socket-path=/var/run/postgresql
Deprecated Name: db-socket-path
PostgreSQL Database User Option (--pg-user)
PostgreSQL database user.
The database user name used when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER. 
example: --pg1-user=backupuser
Expire Command (expire)
pgBackRest does full backup rotation based on the retention type which can be a count or a time period. When a count is specified, then expiration is not concerned with when the backups were created but with how many must be retained. Differential backups are count-based but will always be expired when the full backup they depend on is expired. Incremental backups are not expired by retention independently — they are always expired with their related full or differential backup. See sections Full Backup Retention and Differential Backup Retention for details and examples.
Archived WAL is retained by default for backups that have not expired, however, although not recommended, this schedule can be modified per repository with the retention-archive options. See section Archive Retention for details and examples.
The expire command is run automatically after each successful backup and can also be run by the user. When run by the user, expiration will occur as defined by the retention settings for each configured repository. If the --repo option is provided, expiration will occur only on the specified repository. Expiration can also be limited by the user to a specific backup set with the --set option and, unless the --repo option is specified, all repositories will be searched and any matching the set criteria will be expired. It should be noted that the archive retention schedule will be checked and performed any time the expire command is run.
Command Options
Expire Archive Before Option (--archive-expire-before)
Remove WAL archive earlier than the specified WAL segment.
Remove WAL segments earlier than the specified segment, but only those that are not required by a retained backup, that is, segments older than the earliest backup set to retain, or any segment when the repository contains no backup. The specified segment and everything after it are kept. This mirrors the value passed by PostgreSQL to archive_cleanup_command as %r.
This option is primarily intended for archive-only repositories where no backups are stored. It can also be used to reclaim WAL archive space before the first backup when full retention has not yet been met to trigger archive expiration. 
example: --archive-expire-before=000000010000000000000010
Oldest Option (--oldest)
Expire the oldest eligible backup set.
Expire the oldest full backup set that can be removed (meaning at least one newer full backup remains). This is equivalent to manually decrementing retention by one, but computed automatically. All backups related to the expired full backup set (differential and incremental) are also expired.
When used, archive retention is also temporarily adjusted so WAL for the expired backups can be removed in the same run.
If time-based full retention is configured (using --repo-retention-full-type=time) then --oldest uses count-based expiration for this execution.
WARNING:
This option cannot be combined with --set. 
default: n
example: --oldest
Set Option (--set)
Backup set to expire.
The specified backup set (i.e. the backup label provided and all of its dependent backups, if any) will be expired regardless of backup retention rules except that at least one full backup must remain in the repository.
WARNING:
Use this option with extreme caution — it will permanently remove all backups and archives not required to make a backup consistent from the pgBackRest repository for the specified backup set. This process may negate the ability to perform PITR. If --repo-retention-full and/or --repo-retention-archive options are configured, then it is recommended that you override these options by setting their values to the maximum while performing ad hoc expiration in order to prevent an unintended expiration of archives. 
example: --set=20150131-153358F_20150131-153401I
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Dry Run Option (--dry-run)
Execute a dry-run for the command.
The --dry-run option is a command-line only option and can be passed when it is desirable to determine what modifications will be made by the command without the command actually making any modifications. 
default: n
example: --dry-run
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
Archive Retention Option (--repo-retention-archive)
Number of backups worth of continuous WAL to retain.
NOTE:
WAL segments required to make a backup consistent are always retained until the backup is expired regardless of how this option is configured.
If this value is not set and repo-retention-full-type is count (default), then the archive to expire will default to the repo-retention-full (or repo-retention-diff) value corresponding to the repo-retention-archive-type if set to full (or diff). This will ensure that WAL is only expired for backups that are already expired. If repo-retention-full-type is time, then this value will default to removing archives that are earlier than the oldest full backup retained after satisfying the repo-retention-full setting.
This option must be set if repo-retention-archive-type is set to incr. If disk space is at a premium, then this setting, in conjunction with repo-retention-archive-type, can be used to aggressively expire WAL segments. However, doing so negates the ability to perform PITR from the backups with expired WAL and is therefore not recommended. 
allowed: [1, 9999999]
example: --repo1-retention-archive=2
Deprecated Name: retention-archive
Archive Retention Type Option (--repo-retention-archive-type)
Backup type for WAL retention.
If set to full pgBackRest will keep archive logs for the number of full backups defined by repo-retention-archive. If set to diff (differential) pgBackRest will keep archive logs for the number of full and differential backups defined by repo-retention-archive, meaning if the last backup taken was a full backup, it will be counted as a differential for the purpose of repo-retention. If set to incr (incremental) pgBackRest will keep archive logs for the number of full, differential, and incremental backups defined by repo-retention-archive. It is recommended that this setting not be changed from the default which will only expire WAL in conjunction with expiring full backups. 
default: full
example: --repo1-retention-archive-type=diff
Deprecated Name: retention-archive-type
Differential Retention Option (--repo-retention-diff)
Number of differential backups to retain.
When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept until the full backups they depend on expire.
Note that full backups are included in the count of differential backups for the purpose of expiration. This slightly reduces the number of differential backups that need to be retained in most cases. 
allowed: [1, 9999999]
example: --repo1-retention-diff=3
Deprecated Name: retention-diff
Full Retention Option (--repo-retention-full)
Full backup retention count/time.
When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When the option is not defined a warning will be issued. If indefinite retention is desired then set the option to the max value. 
allowed: [1, 9999999]
example: --repo1-retention-full=2
Deprecated Name: retention-full
Full Retention Type Option (--repo-retention-full-type)
Retention type for full backups.
Determines whether the repo-retention-full setting represents a time period (days) or count of full backups to keep.
If set to time then full backups older than repo-retention-full will be removed from the repository if there is at least one other backup that is equal to or greater than the repo-retention-full setting. For example, if repo-retention-full is 30 (days) and there are 2 full backups: one 25 days old and one 35 days old, no full backups will be expired because expiring the 35 day old backup would leave only the 25 day old backup, which would violate the 30 day retention policy of having at least one backup 30 days old before an older one can be expired. Archived WAL older than the oldest full backup remaining will be automatically expired unless repo-retention-archive-type and repo-retention-archive are explicitly set.
If set to count then full backups that exceed repo-retention-full will be expired. For example, if repo-retention-full is 4 and a fifth full backup is completed, then the oldest full backup will be expired to keep the count at 4.
Note that a backup must be successfully completed before it will be considered for retention. For example, if repo-retention-full-type is count and repo-retention-full is 2, then there must be 3 complete full backups before the oldest will be expired. 
default: count
example: --repo1-retention-full-type=time
Backup History Retention Option (--repo-retention-history)
Days of backup history manifests to retain.
A copy of the backup manifest is stored in the backup.history path when a backup completes. By default these files are never expired since they are useful for data mining, e.g. measuring backup and WAL growth over time.
Set repo-retention-history to define the number of days of backup history manifests to retain. Unexpired backups are always kept in the backup history. Specify repo-retention-history=0 to retain the backup history only for unexpired backups.
When a full backup history manifest is expired, all differential and incremental backup history manifests associated with the full backup also expire. 
allowed: [0, 9999999]
example: --repo1-retention-history=365
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Symlink Option (--repo-symlink)
Create symlinks within the repository.
Enable creation of the lastest and tablespace symlinks. These symlinks are most useful when using snapshots to do in-place recovery in the repository, which is an uncommon use case.
While this feature is likely not useful for the vast majority of users it remains on by default for legacy purposes. However, it may be useful to disable symlinks for Posix-like storage that does not support them. 
default: y
example: --no-repo1-symlink
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Help Command (help)
Three levels of help are provided. If no command is specified then general help will be displayed. If a command is specified (e.g. pgbackrest help backup) then a full description of the command will be displayed along with a list of valid options. If an option is specified in addition to a command (e.g. pgbackrest help backup type) then a full description of the option as it applies to the command will be displayed.
Command Options
Display Help Option (--help)
Display help.
Displays help even if the help command is not specified and overrides the --version option. 
default: n
example: --help
Display Version Option (--version)
Display version.
Displays version even if the version or help command is not specified. 
default: n
example: --version
Info Command (info)
The info command operates on a single stanza or all stanzas. Text output is the default and gives a human-readable summary of backups for the stanza(s) requested. This format is subject to change with any release.
For machine-readable output use --output=json. The JSON output contains far more information than the text output and is kept stable unless a bug is found.
To speed up execution, limit the output to only progress information by specifying --detail-level=progress. Note that this skips all checks except for availability of the stanza.
Each stanza has a separate section and it is possible to limit output to a single stanza with the --stanza option. The stanza 'status' gives a brief indication of the stanza's health. If this is 'ok' then pgBackRest is functioning normally. If there are multiple repositories, then a status of 'mixed' indicates that the stanza is not in a healthy state on one or more of the repositories; in this case the state of the stanza will be detailed per repository. For cases in which an error on a repository occurred that is not one of the known error codes, then an error code of 'other' will be used and the full error details will be provided. The 'wal archive min/max' shows the minimum and maximum WAL currently stored in the archive and, in the case of multiple repositories, will be reported across all repositories unless the --repo option is set. Note that there may be gaps due to archive retention policies or other reasons.
The 'backup/expire running' and/or 'restore running' messages will appear beside the 'status' information if any of those commands are currently running on the host. Per-repo progress will be also reported in text output and a 'repo' array will be included in JSON output.
The backups are displayed oldest to newest. The oldest backup will always be a full backup (indicated by an F at the end of the label) but the newest backup can be full, differential (ends with D), or incremental (ends with I).
The 'timestamp start/stop' defines the time period when the backup ran. The 'timestamp stop' can be used to determine the backup to use when performing Point-In-Time Recovery. More information about Point-In-Time Recovery can be found in the Point-In-Time Recovery section.
The 'wal start/stop' defines the WAL range that is required to make the database consistent when restoring. The backup command will ensure that this WAL range is in the archive before completing.
The 'database size' is the full uncompressed size of the database while 'database backup size' is the amount of data in the database to actually back up (these will be the same for full backups).
The 'repo' indicates in which repository this backup resides. The 'backup set size' includes all the files from this backup and any referenced backups in the repository that are required to restore the database from this backup while 'backup size' includes only the files in this backup (these will also be the same for full backups). Repository sizes reflect compressed file sizes if compression is enabled in pgBackRest.
The 'backup reference total' summarizes the list of additional backups that are required to restore this backup. Use the --set option to display the complete reference list.
Command Options
Detail level Option (--detail-level)
Output detail level.
The following levels are supported:
  • progress - Output only the current backup/expire progress. This level cannot be used with the --set option.
  • full - Output full info. 
default: full
example: --detail-level=progress
Output Option (--output)
Output format.
The following output types are supported:
  • text - Human-readable summary of backup information.
  • json - Exhaustive machine-readable backup information in JSON format. 
default: text
example: --output=json
Set Option (--set)
Backup set to detail.
Details include a complete list of additional backups that are required to restore this backup, a list of databases (with OIDs) in the backup set (excluding template databases), tablespaces (with OIDs) with the destination where they will be restored by default, and symlinks with the destination where they will be restored when --link-all is specified. 
example: --set=20150131-153358F_20150131-153401I
Type Option (--type)
Filter on backup type.
Filter the output using one of the following backup types:
  • full - Output only full backups.
  • diff - Output only differential backups.
  • incr - Output only incremental backups. 
example: --type=full
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Target Time for Repository Option (--repo-target-time)
Target time for repository.
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided. 
example: --repo-target-time=2024-08-08 12:12:12+00
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Repository Get Command (repo-get)
Similar to the unix cat command but works on any supported repository type. This command requires a fully qualified file name and is primarily for administration, investigation, and testing. It is not a required part of a normal pgBackRest setup.
If the repository is encrypted then repo-get will automatically decrypt the file. Files are not automatically decompressed but the output can be piped through the appropriate decompression command, e.g. gzip -d.
If more than one repository is configured, the command will default to the highest priority repository (e.g. repo1) unless the --repo option is specified.
Command Options
Ignore Missing Option (--ignore-missing)
Ignore missing source file.
Exit with 1 if the source file is missing but don't throw an error. 
default: n
example: --ignore-missing
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Raw Data Option (--raw)
Do not transform data.
Do not transform (i.e, encrypt, decompress, etc.) data for the current command. 
default: n
example: --raw
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Target Time for Repository Option (--repo-target-time)
Target time for repository.
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided. 
example: --repo-target-time=2024-08-08 12:12:12+00
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Repository List Command (repo-ls)
Similar to the unix ls command but works on any supported repository type. This command accepts a path, absolute or relative to the repository path defined by the --repo-path option, and is primarily for administration, investigation, and testing. It is not a required part of a normal pgBackRest setup.
The default text output prints one file name per line. JSON output is available by specifying --output=json.
If more than one repository is configured, the command will default to the highest priority repository (e.g. repo1) unless the --repo option is specified.
Command Options
Filter Output Option (--filter)
Filter output with a regular expression.
The filter is applied against the file/path names before they are output. 
example: --filter="(F|D|I)$"
Output Option (--output)
Output format.
The following output types are supported:
  • text - Simple list with one file/link/path name on each line.
  • json - Detailed file/link/path information in JSON format.
In JSON format the available fields are:
  • name - file/link/path name (and partial path when recursing).
  • type - file, path, or link.
  • size - size in bytes (files only).
  • time - time last modified (files only).
  • destination - link destination (links only). 
default: text
example: --output=json
Recurse Subpaths Option (--recurse)
Include all subpaths in output.
All subpaths and their files will be included in the output. 
default: n
example: --recurse
Sort Output Option (--sort)
Sort output ascending, descending, or none.
The following sort types are supported:
  • asc - sort ascending.
  • desc - sort descending.
  • none - no sorting. 
default: asc
example: --sort=desc
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Target Time for Repository Option (--repo-target-time)
Target time for repository.
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided. 
example: --repo-target-time=2024-08-08 12:12:12+00
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Restore Command (restore)
The restore command automatically defaults to selecting the latest backup from the first repository where backups exist (see Quick Start - Restore a Backup). The order in which the repositories are checked is dictated by the pgbackrest.conf (e.g. repo1 will be checked before repo2). To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.
When PITR of --type=time or --type=lsn is specified, then the target time or target lsn must be specified with the --target option. If a backup is not specified via the --set option, then the configured repositories will be checked, in order, for a backup that contains the requested time or lsn. If no matching backup is found, the latest backup from the first repository containing backups will be used for --type=time while no backup will be selected for --type=lsn. For other types of PITR, e.g. xid, the --set option must be provided if the target is prior to the latest backup. See Point-in-Time Recovery for more details and examples.
Replication slots are not included per recommendation of PostgreSQL. See Backing Up The Data Directory in the PostgreSQL documentation for more information.
Command Options
Archive Mode Option (--archive-mode)
Preserve or disable archiving on restored cluster.
This option allows archiving to be preserved or disabled on a restored cluster. This is useful when the cluster must be promoted to do some work but is not intended to become the new primary. In this case it is not a good idea to push WAL from the cluster into the repository.
The following modes are supported:
  • off - disable archiving by setting archive_mode=off.
  • preserve - preserve current archive_mode setting.
NOTE: This option is not available on PostgreSQL < 12. 
default: preserve
example: --archive-mode=off
Exclude Database Option (--db-exclude)
Restore excluding the specified databases.
Databases excluded will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, those databases will not be accessible but can be removed with the drop database command. The --db-exclude option can be passed multiple times to specify more than one database to exclude.
When used in combination with the --db-include option, --db-exclude will only apply to standard system databases (template0, template1, and postgres). 
example: --db-exclude=db_main
Include Database Option (--db-include)
Restore only specified databases.
This feature allows only selected databases to be restored. Databases not specifically included will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, the databases that were not included will not be accessible but can be removed with the drop database command.
NOTE:
built-in databases (template0, template1, and postgres) are always restored unless specifically excluded.
The --db-include option can be passed multiple times to specify more than one database to include.
See Restore Selected Databases for additional information and caveats. 
example: --db-include=db_main
Force Option (--force)
Force a restore.
By itself this option forces the PostgreSQL data and tablespace paths to be completely overwritten. In combination with --delta a timestamp/size delta will be performed instead of using checksums. 
default: n
example: --force
Link All Option (--link-all)
Restore all symlinks.
By default symlinked directories and files are restored as normal directories and files in $PGDATA. This is because it may not be safe to restore symlinks to their original destinations on a system other than where the original backup was performed. This option restores all the symlinks just as they were on the original system where the backup was performed. 
default: n
example: --link-all
Link Map Option (--link-map)
Modify the destination of a symlink.
Allows the destination file or path of a symlink to be changed on restore. This is useful for restoring to systems that have a different storage layout than the original system where the backup was generated. 
example: --link-map=pg_xlog=/data/xlog
Recovery Option Option (--recovery-option)
Set an option in postgresql.auto.conf or recovery.conf.
See Server Configuration for details on postgresql.auto.conf or recovery.conf options (be sure to select your PostgreSQL version). This option can be used multiple times.
For PostgreSQL >= 12, options will be written into postgresql.auto.conf. For all other versions, options will be written into recovery.conf.
NOTE:
The restore_command option will be automatically generated but can be overridden with this option. Be careful about specifying your own restore_command as pgBackRest is designed to handle this for you. Target Recovery options (recovery_target_name, recovery_target_time, etc.) are generated automatically by pgBackRest and should not be set with this option.
Since pgBackRest does not start PostgreSQL after writing the postgresql.auto.conf or recovery.conf file, it is always possible to edit/check postgresql.auto.conf or recovery.conf before manually restarting. 
example: --recovery-option=primary_conninfo=db.mydomain.com
Set Option (--set)
Backup set to restore.
The backup set to be restored. latest will restore the latest backup, otherwise provide the name of the backup to restore. 
default: latest
example: --set=20150131-153358F_20150131-153401I
Tablespace Map Option (--tablespace-map)
Restore a tablespace into the specified directory.
Moves a tablespace to a new location during the restore. This is useful when tablespace locations are not the same on a replica, or an upgraded system has different mount points.
Tablespace locations are not stored in pg_tablespace so moving tablespaces can be done with impunity. However, moving a tablespace to the data_directory is not recommended and may cause problems. For more information on moving tablespaces http://www.databasesoup.com/2013/11/moving-tablespaces.html is a good resource. 
example: --tablespace-map=ts_01=/db/ts_01
Map All Tablespaces Option (--tablespace-map-all)
Restore all tablespaces into the specified directory.
Tablespaces are restored into their original locations by default. This behavior can be modified for each tablespace with the tablespace-map option, but it is sometimes preferable to remap all tablespaces to a new directory all at once. This is particularly useful for development or staging systems that may not have the same storage layout as the original system where the backup was generated.
The path specified will be the parent path used to create all the tablespaces in the backup.
CAUTION:
Tablespaces created after the backup started will not be mapped. Make a new backup after a tablespace is created if tablespace mapping is required. 
example: --tablespace-map-all=/data/tablespace
Target Option (--target)
Recovery target.
Defines the recovery target when --type is lsn, name, xid, or time. If the target is prior to the latest backup and --type is not time or lsn, then use the --set option to specify the backup set. 
example: --target=2015-01-30 14:15:11 EST
Target Action Option (--target-action)
Action to take when recovery target is reached.
When hot_standby=on, the default since PostgreSQL 10, this option consistently controls what the cluster does when the target is reached or there is no more WAL in the archive.
When hot_standby=off in PostgreSQL >= 12, pause acts like shutdown. When hot_standby=off in PostgreSQL < 12, pause acts like promote.
The following actions are supported:
  • pause - pause when recovery target is reached.
  • promote - promote and switch timeline when recovery target is reached.
  • shutdown - shutdown server when recovery target is reached. (PostgreSQL >= 9.5) 
default: pause
example: --target-action=promote
Target Exclusive Option (--target-exclusive)
Stop just before the recovery target is reached.
Defines whether recovery to the target would be exclusive (the default is inclusive) and is only valid when --type is lsn, time or xid. For example, using --target-exclusive would exclude the contents of transaction 1007 when --type=xid and --target=1007. See the recovery_target_inclusive option in the PostgreSQL docs for more information. 
default: n
example: --no-target-exclusive
Target Timeline Option (--target-timeline)
Recover along a timeline.
See recovery_target_timeline in the PostgreSQL docs for more information. 
example: --target-timeline=3
Type Option (--type)
Recovery type.
The following recovery types are supported:
  • default - recover to the end of the archive stream.
  • immediate - recover only until the database becomes consistent.
  • lsn - recover to the LSN (Log Sequence Number) specified in --target. This option is only supported on PostgreSQL >= 10.
  • name - recover the restore point specified in --target.
  • xid - recover to the transaction id specified in --target.
  • time - recover to the time specified in --target.
  • preserve - preserve the existing postgresql.auto.conf or recovery.conf file.
  • standby - add standby_mode=on to the postgresql.auto.conf or recovery.conf file so cluster will start in standby mode.
  • none - no postgresql.auto.conf or recovery.conf file is written so PostgreSQL will attempt to achieve consistency using WAL segments present in pg_xlog/pg_wal. Provide the required WAL segments or use the archive-copy setting to include them with the backup.
WARNING:
Recovery type=none should be avoided because the timeline will not be incremented at the end of recovery. This can lead to, for example, PostgreSQL attempting to archive duplicate WAL, which will be rejected, and may cause the disk to fill up and result in a PostgreSQL panic. In addition, tools like pg_rewind may not work correctly or may cause corruption.
Note that the default restore type for offline backups is none since Point-in-Time-Recovery is not possible if wal_level=minimal. If type is set explicitly then it will be honored since Point-in-Time-Recovery is possible from offline backups as long as wal_level > minimal. 
default: default
example: --type=xid
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: y
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
pgBackRest Command Option (--cmd)
pgBackRest command.
pgBackRest may generate a command string, e.g. when the restore command generates the restore_command setting. The command used to run the pgBackRest process will be used in this case unless the cmd option is provided.
CAUTION:
Wrapping the pgBackRest command may cause unpredictable behavior and is not recommended. 
default: [path of executed pgbackrest binary]
example: --cmd=/var/lib/pgsql/bin/pgbackrest_wrapper.sh
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Delta Option (--delta)
Restore or backup using checksums.
During a restore, by default the PostgreSQL data and tablespace directories are expected to be present but empty. This option performs a delta restore using checksums.
During a backup, this option will use checksums instead of the timestamps to determine if files will be copied. 
default: n
example: --delta
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Process Maximum Option (--process-max)
Max processes to use for compress/transfer.
Each process will perform compression and transfer to make the command run faster, but don't set process-max so high that it impacts database performance. 
default: 1
allowed: [1, 999]
example: --process-max=4
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Maintainer Options
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: --pg-version-force=15
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Target Time for Repository Option (--repo-target-time)
Target time for repository.
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided. 
example: --repo-target-time=2024-08-08 12:12:12+00
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Stanza Options
PostgreSQL Path Option (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: --pg1-path=/data/db
Deprecated Name: db-path
Server Command (server)
The pgBackRest server allows access to remote hosts without using the SSH protocol.
Command Options
TLS Server Address Option (--tls-server-address)
TLS server address.
IP address the server will listen on for client requests. 
default: localhost
example: --tls-server-address=*
TLS Server Authorized Clients Option (--tls-server-auth)
TLS server authorized clients.
Clients are authorized on the server by verifying their certificate and checking their certificate CN (Common Name) against a list on the server configured with the tls-server-auth option.
A client CN can be authorized for as many stanzas as needed by providing a comma-separated list to the tls-server-auth option or for all stanzas by specifying tls-server-auth=client-cn=*. Wildcards may not be specified for the client CN. 
example: --tls-server-auth=client-cn=stanza1,stanza2
TLS Server Certificate Authorities Option (--tls-server-ca-file)
TLS server certificate authorities.
Checks that client certificates are signed by a trusted certificate authority. 
example: --tls-server-ca-file=/path/to/server.ca
TLS Server Certificate Option (--tls-server-cert-file)
TLS server certificate file.
Sent to the client to show the server identity. 
example: --tls-server-cert-file=/path/to/server.crt
TLS Server Key Option (--tls-server-key-file)
TLS server key file.
Proves server certificate was sent by the owner. 
example: --tls-server-key-file=/path/to/server.key
TLS Server Port Option (--tls-server-port)
TLS server port.
Port the server will listen on for client requests. 
default: 8432
allowed: [1, 65535]
example: --tls-server-port=8000
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Server Ping Command (server-ping)
Ping a pgBackRest TLS server to ensure it is accepting connections. This serves as an aliveness check only since no authentication is attempted.
If no host is specified on the command-line then the tls-server-host option will be used.
Command Options
TLS Server Address Option (--tls-server-address)
TLS server address.
IP address the server will listen on for client requests. 
default: localhost
example: --tls-server-address=*
TLS Server Port Option (--tls-server-port)
TLS server port.
Port the server will listen on for client requests. 
default: 8432
allowed: [1, 65535]
example: --tls-server-port=8000
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Stanza Create Command (stanza-create)
The stanza-create command must be run after the stanza has been configured in pgbackrest.conf. If there is more than one repository configured, the stanza will be created on each. Stanzas that have already been created will be skipped so it is always safe to run stanza-create, even when a new repository has been configured.
See Create the Stanza for more information and an example.
Command Options
Online Option (--online)
Create on an online cluster.
Specifying --no-online prevents pgBackRest from connecting to PostgreSQL when creating the stanza. 
default: y
example: --no-online
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Database Timeout Option (--db-timeout)
Database query timeout.
Sets the timeout, in seconds, for queries against the database. This includes the backup start/stop functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set start-fast=y and you know that the database cluster will not generate many WAL segments during the backup).
NOTE:
The db-timeout option must be less than the protocol-timeout option. 
default: 30m
allowed: [100ms, 7d]
example: --db-timeout=600
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Maintainer Options
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: --pg-version-force=15
Repository Options
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Stanza Options
PostgreSQL Database Option (--pg-database)
PostgreSQL database.
The database name used when connecting to PostgreSQL. The default is usually best but some installations may not contain this database.
Note that for legacy reasons the setting of the PGDATABASE environment variable will be ignored. 
default: postgres
example: --pg1-database=backupdb
PostgreSQL Host Option (--pg-host)
PostgreSQL host for operating remotely.
Used for backups where the PostgreSQL host is different from the repository host. 
example: --pg1-host=db.domain.com
Deprecated Name: db-host
PostgreSQL Host Certificate Authority File Option (--pg-host-ca-file)
PostgreSQL host certificate authority file.
Use a CA file other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
PostgreSQL Host Certificate Authority Path Option (--pg-host-ca-path)
PostgreSQL host certificate authority path.
Use a CA path other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-path=/etc/pki/tls/certs
PostgreSQL Host Certificate File Option (--pg-host-cert-file)
PostgreSQL host certificate file.
Sent to PostgreSQL host to prove client identity. 
example: --pg1-host-cert-file=/path/to/client.crt
PostgreSQL Host Command Option (--pg-host-cmd)
PostgreSQL host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and PostgreSQL hosts. If not defined, the PostgreSQL host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --pg1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: db-cmd
PostgreSQL Host Configuration Option (--pg-host-config)
pgBackRest database host configuration file.
Sets the location of the configuration file on the PostgreSQL host. This is only required if the PostgreSQL host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --pg1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: db-config
PostgreSQL Host Configuration Include Path Option (--pg-host-config-include-path)
pgBackRest database host configuration include path.
Sets the location of the configuration include path on the PostgreSQL host. This is only required if the PostgreSQL host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --pg1-host-config-include-path=/conf/pgbackrest/conf.d
PostgreSQL Host Configuration Path Option (--pg-host-config-path)
pgBackRest database host configuration path.
Sets the location of the configuration path on the PostgreSQL host. This is only required if the PostgreSQL host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --pg1-host-config-path=/conf/pgbackrest
PostgreSQL Host Key File Option (--pg-host-key-file)
PostgreSQL host key file.
Proves client certificate was sent by owner. 
example: --pg1-host-key-file=/path/to/client.key
PostgreSQL Host Port Option (--pg-host-port)
PostgreSQL host port when pg-host is set.
Use this option to specify a non-default port for the PostgreSQL host protocol.
NOTE:
When pg-host-type=ssh there is no default for pg-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on pg-host-type):
    tls - 8432

allowed: [0, 65535]
example: --pg1-host-port=25
Deprecated Name: db-ssh-port
PostgreSQL Host Protocol Type Option (--pg-host-type)
PostgreSQL host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --pg1-host-type=tls
PostgreSQL Host User Option (--pg-host-user)
PostgreSQL host logon user when pg-host is set.
This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this to work correctly the user should be the PostgreSQL database cluster owner which is generally postgres, the default. 
default: postgres
example: --pg1-host-user=db_owner
Deprecated Name: db-user
PostgreSQL Path Option (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: --pg1-path=/data/db
Deprecated Name: db-path
PostgreSQL Port Option (--pg-port)
PostgreSQL port.
Port that PostgreSQL is running on. This usually does not need to be specified as most PostgreSQL clusters run on the default port. 
default: 5432
allowed: [0, 65535]
example: --pg1-port=6543
Deprecated Name: db-port
PostgreSQL Socket Path Option (--pg-socket-path)
PostgreSQL unix socket path.
The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look in the standard location for your OS so there is usually no need to specify this setting unless the socket directory was explicitly modified with the unix_socket_directories setting in postgresql.conf. 
example: --pg1-socket-path=/var/run/postgresql
Deprecated Name: db-socket-path
PostgreSQL Database User Option (--pg-user)
PostgreSQL database user.
The database user name used when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER. 
example: --pg1-user=backupuser
Stanza Delete Command (stanza-delete)
The stanza-delete command removes data in the repository associated with a stanza.
WARNING:
Use this command with caution — it will permanently remove all backups and archives from the pgBackRest repository for the specified stanza.
To delete a stanza:
  • Shut down the PostgreSQL cluster associated with the stanza (or use --force to override).
  • Run the stop command on the host where the stanza-delete command will be run.
  • Run the stanza-delete command.
Once the command successfully completes, it is the responsibility of the user to remove the stanza from all pgBackRest configuration files and/or environment variables.
A stanza may only be deleted from one repository at a time. To delete the stanza from multiple repositories, repeat the stanza-delete command for each repository while specifying the --repo option.
Command Options
Force Option (--force)
Force stanza delete.
If PostgreSQL is still running for the stanza, then this option can be used to force the stanza to be deleted from the repository. 
default: n
example: --no-force
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Database Timeout Option (--db-timeout)
Database query timeout.
Sets the timeout, in seconds, for queries against the database. This includes the backup start/stop functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set start-fast=y and you know that the database cluster will not generate many WAL segments during the backup).
NOTE:
The db-timeout option must be less than the protocol-timeout option. 
default: 30m
allowed: [100ms, 7d]
example: --db-timeout=600
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Maintainer Options
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: --pg-version-force=15
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Stanza Options
PostgreSQL Database Option (--pg-database)
PostgreSQL database.
The database name used when connecting to PostgreSQL. The default is usually best but some installations may not contain this database.
Note that for legacy reasons the setting of the PGDATABASE environment variable will be ignored. 
default: postgres
example: --pg1-database=backupdb
PostgreSQL Host Option (--pg-host)
PostgreSQL host for operating remotely.
Used for backups where the PostgreSQL host is different from the repository host. 
example: --pg1-host=db.domain.com
Deprecated Name: db-host
PostgreSQL Host Certificate Authority File Option (--pg-host-ca-file)
PostgreSQL host certificate authority file.
Use a CA file other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
PostgreSQL Host Certificate Authority Path Option (--pg-host-ca-path)
PostgreSQL host certificate authority path.
Use a CA path other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-path=/etc/pki/tls/certs
PostgreSQL Host Certificate File Option (--pg-host-cert-file)
PostgreSQL host certificate file.
Sent to PostgreSQL host to prove client identity. 
example: --pg1-host-cert-file=/path/to/client.crt
PostgreSQL Host Command Option (--pg-host-cmd)
PostgreSQL host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and PostgreSQL hosts. If not defined, the PostgreSQL host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --pg1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: db-cmd
PostgreSQL Host Configuration Option (--pg-host-config)
pgBackRest database host configuration file.
Sets the location of the configuration file on the PostgreSQL host. This is only required if the PostgreSQL host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --pg1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: db-config
PostgreSQL Host Configuration Include Path Option (--pg-host-config-include-path)
pgBackRest database host configuration include path.
Sets the location of the configuration include path on the PostgreSQL host. This is only required if the PostgreSQL host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --pg1-host-config-include-path=/conf/pgbackrest/conf.d
PostgreSQL Host Configuration Path Option (--pg-host-config-path)
pgBackRest database host configuration path.
Sets the location of the configuration path on the PostgreSQL host. This is only required if the PostgreSQL host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --pg1-host-config-path=/conf/pgbackrest
PostgreSQL Host Key File Option (--pg-host-key-file)
PostgreSQL host key file.
Proves client certificate was sent by owner. 
example: --pg1-host-key-file=/path/to/client.key
PostgreSQL Host Port Option (--pg-host-port)
PostgreSQL host port when pg-host is set.
Use this option to specify a non-default port for the PostgreSQL host protocol.
NOTE:
When pg-host-type=ssh there is no default for pg-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on pg-host-type):
    tls - 8432

allowed: [0, 65535]
example: --pg1-host-port=25
Deprecated Name: db-ssh-port
PostgreSQL Host Protocol Type Option (--pg-host-type)
PostgreSQL host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --pg1-host-type=tls
PostgreSQL Host User Option (--pg-host-user)
PostgreSQL host logon user when pg-host is set.
This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this to work correctly the user should be the PostgreSQL database cluster owner which is generally postgres, the default. 
default: postgres
example: --pg1-host-user=db_owner
Deprecated Name: db-user
PostgreSQL Path Option (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: --pg1-path=/data/db
Deprecated Name: db-path
PostgreSQL Port Option (--pg-port)
PostgreSQL port.
Port that PostgreSQL is running on. This usually does not need to be specified as most PostgreSQL clusters run on the default port. 
default: 5432
allowed: [0, 65535]
example: --pg1-port=6543
Deprecated Name: db-port
PostgreSQL Socket Path Option (--pg-socket-path)
PostgreSQL unix socket path.
The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look in the standard location for your OS so there is usually no need to specify this setting unless the socket directory was explicitly modified with the unix_socket_directories setting in postgresql.conf. 
example: --pg1-socket-path=/var/run/postgresql
Deprecated Name: db-socket-path
PostgreSQL Database User Option (--pg-user)
PostgreSQL database user.
The database user name used when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER. 
example: --pg1-user=backupuser
Stanza Upgrade Command (stanza-upgrade)
Immediately after upgrading PostgreSQL to a newer major version, the pg-path for all pgBackRest configurations must be set to the new database location and the stanza-upgrade command run. If there is more than one repository configured on the host, the stanza will be upgraded on each. If the database is offline use the --no-online option.
Command Options
Online Option (--online)
Update an online cluster.
Specifying --no-online prevents pgBackRest from connecting to PostgreSQL when upgrading the stanza. 
default: y
example: --no-online
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Database Timeout Option (--db-timeout)
Database query timeout.
Sets the timeout, in seconds, for queries against the database. This includes the backup start/stop functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set start-fast=y and you know that the database cluster will not generate many WAL segments during the backup).
NOTE:
The db-timeout option must be less than the protocol-timeout option. 
default: 30m
allowed: [100ms, 7d]
example: --db-timeout=600
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Maintainer Options
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: --pg-version-force=15
Repository Options
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Stanza Options
PostgreSQL Database Option (--pg-database)
PostgreSQL database.
The database name used when connecting to PostgreSQL. The default is usually best but some installations may not contain this database.
Note that for legacy reasons the setting of the PGDATABASE environment variable will be ignored. 
default: postgres
example: --pg1-database=backupdb
PostgreSQL Host Option (--pg-host)
PostgreSQL host for operating remotely.
Used for backups where the PostgreSQL host is different from the repository host. 
example: --pg1-host=db.domain.com
Deprecated Name: db-host
PostgreSQL Host Certificate Authority File Option (--pg-host-ca-file)
PostgreSQL host certificate authority file.
Use a CA file other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
PostgreSQL Host Certificate Authority Path Option (--pg-host-ca-path)
PostgreSQL host certificate authority path.
Use a CA path other than the system default for connecting to the PostgreSQL host. 
example: --pg1-host-ca-path=/etc/pki/tls/certs
PostgreSQL Host Certificate File Option (--pg-host-cert-file)
PostgreSQL host certificate file.
Sent to PostgreSQL host to prove client identity. 
example: --pg1-host-cert-file=/path/to/client.crt
PostgreSQL Host Command Option (--pg-host-cmd)
PostgreSQL host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and PostgreSQL hosts. If not defined, the PostgreSQL host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --pg1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: db-cmd
PostgreSQL Host Configuration Option (--pg-host-config)
pgBackRest database host configuration file.
Sets the location of the configuration file on the PostgreSQL host. This is only required if the PostgreSQL host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --pg1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: db-config
PostgreSQL Host Configuration Include Path Option (--pg-host-config-include-path)
pgBackRest database host configuration include path.
Sets the location of the configuration include path on the PostgreSQL host. This is only required if the PostgreSQL host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --pg1-host-config-include-path=/conf/pgbackrest/conf.d
PostgreSQL Host Configuration Path Option (--pg-host-config-path)
pgBackRest database host configuration path.
Sets the location of the configuration path on the PostgreSQL host. This is only required if the PostgreSQL host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --pg1-host-config-path=/conf/pgbackrest
PostgreSQL Host Key File Option (--pg-host-key-file)
PostgreSQL host key file.
Proves client certificate was sent by owner. 
example: --pg1-host-key-file=/path/to/client.key
PostgreSQL Host Port Option (--pg-host-port)
PostgreSQL host port when pg-host is set.
Use this option to specify a non-default port for the PostgreSQL host protocol.
NOTE:
When pg-host-type=ssh there is no default for pg-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on pg-host-type):
    tls - 8432

allowed: [0, 65535]
example: --pg1-host-port=25
Deprecated Name: db-ssh-port
PostgreSQL Host Protocol Type Option (--pg-host-type)
PostgreSQL host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --pg1-host-type=tls
PostgreSQL Host User Option (--pg-host-user)
PostgreSQL host logon user when pg-host is set.
This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this to work correctly the user should be the PostgreSQL database cluster owner which is generally postgres, the default. 
default: postgres
example: --pg1-host-user=db_owner
Deprecated Name: db-user
PostgreSQL Path Option (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: --pg1-path=/data/db
Deprecated Name: db-path
PostgreSQL Port Option (--pg-port)
PostgreSQL port.
Port that PostgreSQL is running on. This usually does not need to be specified as most PostgreSQL clusters run on the default port. 
default: 5432
allowed: [0, 65535]
example: --pg1-port=6543
Deprecated Name: db-port
PostgreSQL Socket Path Option (--pg-socket-path)
PostgreSQL unix socket path.
The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look in the standard location for your OS so there is usually no need to specify this setting unless the socket directory was explicitly modified with the unix_socket_directories setting in postgresql.conf. 
example: --pg1-socket-path=/var/run/postgresql
Deprecated Name: db-socket-path
PostgreSQL Database User Option (--pg-user)
PostgreSQL database user.
The database user name used when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER. 
example: --pg1-user=backupuser
Start Command (start)
If the pgBackRest processes were previously stopped using the stop command then they can be started again using the start command. Note that this will not immediately start up any pgBackRest processes but they are allowed to run. See Starting and Stopping for more information and examples.
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Stop Command (stop)
Does not allow any new pgBackRest processes to run. By default running processes will be allowed to complete successfully. Use the --force option to terminate running processes.
pgBackRest processes will return an error if they are run after the stop command completes. See Starting and Stopping for more information and examples.
Command Options
Force Option (--force)
Force all pgBackRest processes to stop.
This option will send TERM signals to all running pgBackRest processes to effect a graceful but immediate shutdown. Note that this will also shutdown processes that were initiated on another system but have remotes running on the current system. For instance, if a backup was started on the backup server then running stop --force on the database server will shutdown the backup process on the backup server. 
default: n
example: --force
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: --lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Verify Command (verify)
Verify determines if the backups and archives in a repository are valid.
Command Options
Output Option (--output)
Output type.
The following output types are supported:
  • none - No verify output.
  • text - Output verify information to stdout. 
default: none
example: --output=text
Set Option (--set)
Backup set to verify.
Verify all database and archive files associated with the specified backup set. 
example: --set=20150131-153358F_20150131-153401I
Verbose Option (--verbose)
Verbose output.
Verbose defaults to false, providing a minimal response with important information about errors in the repository. Specifying true provides more information about what was successfully verified. 
default: n
example: --verbose
General Options
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: --allow-root
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: --buffer-size=2MiB
pgBackRest Command Option (--cmd)
pgBackRest command.
pgBackRest may generate a command string, e.g. when the restore command generates the restore_command setting. The command used to run the pgBackRest process will be used in this case unless the cmd option is provided.
CAUTION:
Wrapping the pgBackRest command may cause unpredictable behavior and is not recommended. 
default: [path of executed pgbackrest binary]
example: --cmd=/var/lib/pgsql/bin/pgbackrest_wrapper.sh
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: --cmd-ssh=/usr/bin/ssh
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: --compress-level-network=1
Config Option (--config)
pgBackRest configuration file.
Use this option to specify a different configuration file than the default. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --config=/conf/pgbackrest/pgbackrest.conf
Config Include Path Option (--config-include-path)
Path to additional pgBackRest configuration files.
Configuration files existing in the specified location with extension .conf will be concatenated with the pgBackRest configuration file, resulting in one configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --config-include-path=/conf/pgbackrest/conf.d
Config Path Option (--config-path)
Base path of pgBackRest configuration files.
This setting is used to override the default base path setting for the --config and --config-include-path options unless they are explicitly set on the command-line.
For example, passing only --config-path=/conf/pgbackrest results in the --config default being set to /conf/pgbackrest/pgbackrest.conf and the --config-include-path default being set to /conf/pgbackrest/conf.d. 
default: CFGOPTDEF_CONFIG_PATH
example: --config-path=/conf/pgbackrest
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: --io-timeout=120
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: --no-neutral-umask
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: --priority=19
Process Maximum Option (--process-max)
Max processes to use for compress/transfer.
Each process will perform compression and transfer to make the command run faster, but don't set process-max so high that it impacts database performance. 
default: 1
allowed: [1, 999]
example: --process-max=4
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: --protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: --no-sck-keep-alive
Stanza Option (--stanza)
Defines the stanza.
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod. 
example: --stanza=main
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: --tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: --tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: --tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: --tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: --log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: --log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: --log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: --log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: --log-subprocess
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: --no-log-timestamp
Maintainer Options
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: --pg-version-force=15
Repository Options
Set Repository Option (--repo)
Set repository.
Set the repository for a command to operate on.
For example, this option may be used to perform a restore from a specific repository, rather than letting pgBackRest choose. 
allowed: [1, 256]
example: --repo=1
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: --repo1-azure-container=pg-backup
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: --repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: --repo1-azure-uri-style=path
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: --repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: --repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: --repo1-gcs-endpoint=localhost
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: --repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: --repo1-gcs-user-project=my-project
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: --repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: --repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: --repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: --repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: --repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: --repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: --repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: --repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: --repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: --repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: --repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: --repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: --repo1-path=/backup/db/backrest
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: --repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: --repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: --repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: --repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: --repo1-s3-process-cmd=/usr/local/bin/get-credentials --repo1-s3-process-cmd=--role --repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: --repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: --no-repo1-s3-requester-pays
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: --repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: --repo1-s3-service=s3-outposts
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: --repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: --repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: --repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: --repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: --repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: --repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: --repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: --repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: --repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: --repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: --repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: --repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: --repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: --repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: --repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: --repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: --no-repo1-storage-verify-tls
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Target Time for Repository Option (--repo-target-time)
Target time for repository.
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided. 
example: --repo-target-time=2024-08-08 12:12:12+00
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: --repo1-type=cifs
Version Command (version)
Displays installed pgBackRest version.
Command Options
Output Option (--output)
Output type.
The following output types are supported:
  • text - Display the installed pgBackRest version as text.
  • num - Display the installed pgBackRest version as an integer. 
default: text
example: --output=num
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated August 17, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7744694 pgbackrest-2.59.1/doc/html/configuration.html0000640000175100017510000056465015240547776020706 0ustar00runnerrunner pgBackRest - Configuration Reference
pgBackRest
Configuration Reference
Home
User Guides
Releases
Commands
FAQ
Table of Contents
Introduction
Archive Options
Asynchronous Archiving Option (--archive-async)
Maximum Archive Get Queue Size Option (--archive-get-queue-max)
Retry Missing WAL Segment Option (--archive-missing-retry)
Archive Push Batch Size Option (--archive-push-batch-size)
Maximum Archive Push Queue Size Option (--archive-push-queue-max)
Archive Timeout Option (--archive-timeout)
Backup Options
Backup Annotation Option (--annotation)
Check Archive Option (--archive-check)
Copy Archive Option (--archive-copy)
Check Archive Mode Option (--archive-mode-check)
Backup from Standby Option (--backup-standby)
Page Checksums Option (--checksum-page)
Path/File Exclusions Option (--exclude)
Expire Auto Option (--expire-auto)
Manifest Save Threshold Option (--manifest-save-threshold)
Resume Option (--resume)
Start Fast Option (--start-fast)
General Options
Allow Run as Root Option (--allow-root)
Buffer Size Option (--buffer-size)
pgBackRest Command Option (--cmd)
SSH Client Command Option (--cmd-ssh)
Compress Option (--compress)
Compress Level Option (--compress-level)
Network Compress Level Option (--compress-level-network)
Compress Type Option (--compress-type)
Database Timeout Option (--db-timeout)
Delta Option (--delta)
I/O Timeout Option (--io-timeout)
Lock Path Option (--lock-path)
Neutral Umask Option (--neutral-umask)
Set Process Priority Option (--priority)
Process Maximum Option (--process-max)
Protocol Timeout Option (--protocol-timeout)
Keep Alive Option (--sck-keep-alive)
Spool Path Option (--spool-path)
Keep Alive Count Option (--tcp-keep-alive-count)
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep Alive Interval Option (--tcp-keep-alive-interval)
TLSv1.2 cipher suites Option (--tls-cipher-12)
TLSv1.3 cipher suites Option (--tls-cipher-13)
Log Options
Console Log Level Option (--log-level-console)
File Log Level Option (--log-level-file)
Std Error Log Level Option (--log-level-stderr)
Log Path Option (--log-path)
Log Subprocesses Option (--log-subprocess)
Log Timestamp Option (--log-timestamp)
Maintainer Options
Check WAL Headers Option (--archive-header-check)
Page Header Check Option (--page-header-check)
Force PostgreSQL Version Option (--pg-version-force)
Repository Options
Azure Repository Account Option (--repo-azure-account)
Azure Repository Container Option (--repo-azure-container)
Azure Repository Endpoint Option (--repo-azure-endpoint)
Azure Repository Key Option (--repo-azure-key)
Azure Repository Key Type Option (--repo-azure-key-type)
Azure Repository URI Style Option (--repo-azure-uri-style)
Block Incremental Backup Option (--repo-block)
Repository Bundles Option (--repo-bundle)
Repository Bundle Limit Option (--repo-bundle-limit)
Repository Bundle Size Option (--repo-bundle-size)
Repository Cipher Passphrase Option (--repo-cipher-pass)
Repository Cipher Type Option (--repo-cipher-type)
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS Repository Key Option (--repo-gcs-key)
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS Repository Project ID Option (--repo-gcs-user-project)
Repository Hardlink Option (--repo-hardlink)
Repository Host Option (--repo-host)
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository Host Certificate File Option (--repo-host-cert-file)
Repository Host Command Option (--repo-host-cmd)
Repository Host Configuration Option (--repo-host-config)
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
Repository Host Configuration Path Option (--repo-host-config-path)
Repository Host Key File Option (--repo-host-key-file)
Repository Host Port Option (--repo-host-port)
Repository Host Protocol Type Option (--repo-host-type)
Repository Host User Option (--repo-host-user)
Repository Path Option (--repo-path)
Archive Retention Option (--repo-retention-archive)
Archive Retention Type Option (--repo-retention-archive-type)
Differential Retention Option (--repo-retention-diff)
Full Retention Option (--repo-retention-full)
Full Retention Type Option (--repo-retention-full-type)
Backup History Retention Option (--repo-retention-history)
S3 Repository Bucket Option (--repo-s3-bucket)
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 Repository Access Key Option (--repo-s3-key)
S3 Repository Secret Access Key Option (--repo-s3-key-secret)
S3 Repository Key Type Option (--repo-s3-key-type)
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 Repository Region Option (--repo-s3-region)
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 Repository Role Option (--repo-s3-role)
S3 Repository Service Option (--repo-s3-service)
S3 Repository SSE Customer Key Option (--repo-s3-sse-customer-key)
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 Repository Security Token Option (--repo-s3-token)
S3 Repository URI Style Option (--repo-s3-uri-style)
SFTP Repository Host Option (--repo-sftp-host)
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP Repository Private Key Passphrase Option (--repo-sftp-private-key-passphrase)
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
Repository Storage CA File Option (--repo-storage-ca-file)
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository Storage Host Option (--repo-storage-host)
Repository Storage Port Option (--repo-storage-port)
Repository Storage Tag Option (--repo-storage-tag)
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository Symlink Option (--repo-symlink)
Target Time for Repository Option (--repo-target-time)
Repository Type Option (--repo-type)
Restore Options
Archive Mode Option (--archive-mode)
Exclude Database Option (--db-exclude)
Include Database Option (--db-include)
Link All Option (--link-all)
Link Map Option (--link-map)
Recovery Option Option (--recovery-option)
Tablespace Map Option (--tablespace-map)
Map All Tablespaces Option (--tablespace-map-all)
Server Options
TLS Server Address Option (--tls-server-address)
TLS Server Authorized Clients Option (--tls-server-auth)
TLS Server Certificate Authorities Option (--tls-server-ca-file)
TLS Server Certificate Option (--tls-server-cert-file)
TLS Server Key Option (--tls-server-key-file)
TLS Server Port Option (--tls-server-port)
Stanza Options
PostgreSQL Database Option (--pg-database)
PostgreSQL Host Option (--pg-host)
PostgreSQL Host Certificate Authority File Option (--pg-host-ca-file)
PostgreSQL Host Certificate Authority Path Option (--pg-host-ca-path)
PostgreSQL Host Certificate File Option (--pg-host-cert-file)
PostgreSQL Host Command Option (--pg-host-cmd)
PostgreSQL Host Configuration Option (--pg-host-config)
PostgreSQL Host Configuration Include Path Option (--pg-host-config-include-path)
PostgreSQL Host Configuration Path Option (--pg-host-config-path)
PostgreSQL Host Key File Option (--pg-host-key-file)
PostgreSQL Host Port Option (--pg-host-port)
PostgreSQL Host Protocol Type Option (--pg-host-type)
PostgreSQL Host User Option (--pg-host-user)
PostgreSQL Path Option (--pg-path)
PostgreSQL Port Option (--pg-port)
PostgreSQL Socket Path Option (--pg-socket-path)
PostgreSQL Database User Option (--pg-user)
Introduction
pgBackRest can be used entirely with command-line parameters but a configuration file is more practical for installations that are complex or set a lot of options. The default location for the configuration file is /etc/pgbackrest/pgbackrest.conf. If no file exists in that location then the old default of /etc/pgbackrest.conf will be checked.
The following option types are used:
String: A text string, commonly an identifier, password, etc.
Command line example: --stanza=demo
Configuration file example: repo1-cipher-pass=zWaf6XtpjIVZC5444yXB...
Path: Used to uniquely identify a location in a directory structure. Paths must begin with /, double // is not allowed, and no ending / is expected.
Command line example: --repo1-path=/var/lib/pgbackrest
Configuration file example: repo1-path=/var/lib/pgbackrest
Boolean: Enables or disables the option. Only y/n are valid argument values.
Command line examples: --start-fast, --no-start-fast, --start-fast=y, --start-fast=n
Configuration file examples: start-fast=y, start-fast=n
Integer: Used for ports, retention/retry counts, parallel processes allowed, etc.
Command line example: --compress-level=3
Configuration file example: pg1-port=5432
Size: Used for buffer sizes, disk usage, etc. Size can be specified in bytes (default) or KiB, MiB, GiB, TiB, or PiB where the multiplier is a power of 1024. For example, the case-insensitive value 5GiB (or 5GB, 5g) can be used instead of 5368709120. Fractional values such as 2.5GiB are not allowed, use 2560MiB instead.
Command line example: --archive-get-queue-max=1GiB
Configuration file example: buffer-size=2MiB
Time: Time in seconds.
Command line example: --io-timeout=90
Configuration file example: db-timeout=600
List: Option may be provided multiple times.
Command line example: --db-exclude=db1 --db-exclude=db2 --db-exclude=db5
Configuration file example, each on its own line: db-exclude=db1 db-exclude=db2 db-exclude=db5
Key/Value: Option may be provided multiple times in the form key=value.
Command line example: --tablespace-map=ts_01=/db/ts_01 --tablespace-map=ts_02=/db/ts_02
Configuration file example, each on its own line: tablespace-map=ts_01=/db/ts_01 tablespace-map=ts_02=/db/ts_02
Archive Options
The archive section defines options for the archive-push and archive-get commands.
Asynchronous Archiving Option (--archive-async)
Push/get WAL segments asynchronously.
Enables asynchronous operation for the archive-push and archive-get commands.
Asynchronous operation is more efficient because it can reuse connections and take advantage of parallelism. See the spool-path, archive-get-queue-max, and archive-push-queue-max options for more information. 
default: n
example: archive-async=y
Maximum Archive Get Queue Size Option (--archive-get-queue-max)
Maximum size of the pgBackRest archive-get queue.
Specifies the maximum size of the archive-get queue when archive-async is enabled. The queue is stored in the spool-path and is used to speed providing WAL to PostgreSQL. 
default: 128MiB
allowed: [0B, 4PiB]
example: archive-get-queue-max=1GiB
Retry Missing WAL Segment Option (--archive-missing-retry)
Retry missing WAL segment
Retry a WAL segment that was previously reported as missing by the archive-get command when in asynchronous mode. This prevents notifications in the spool path from a prior restore from being used and possibly causing a recovery failure if consistency has not been reached.
Disabling this option allows PostgreSQL to more reliably recognize when the end of the WAL in the archive has been reached, which permits it to switch over to streaming from the primary. With retries enabled, a steady stream of WAL being archived will cause PostgreSQL to continue getting WAL from the archive rather than switch to streaming.
When disabling this option it is important to ensure that the spool path for the stanza is empty. The restore command does this automatically if the spool path is configured at restore time. Otherwise, it is up to the user to ensure the spool path is empty. 
default: y
example: archive-missing-retry=n
Archive Push Batch Size Option (--archive-push-batch-size)
Maximum amount of WAL to push per asynchronous run.
In asynchronous mode the archive-push process pushes all the WAL segments that are ready in a single run. Since archive-push-queue-max is only checked at the start of each run, a run that processes a very large number of segments can let the queue grow well beyond the limit before it is rechecked.
This option limits the amount of WAL processed per run so the process exits and is spawned again by the next archive-push, which rechecks the queue. Lower values recheck the queue more often at the cost of spawning the asynchronous process more frequently. The value is rounded down to a whole number of WAL segments but at least one segment is always processed. 
default: 16GiB
allowed: [1MiB, 4PiB]
example: archive-push-batch-size=1GiB
Maximum Archive Push Queue Size Option (--archive-push-queue-max)
Maximum size of the PostgreSQL archive queue.
After the limit is reached, the following will happen:
  • pgBackRest will notify PostgreSQL that the WAL was successfully archived, then DROP IT.
  • A warning will be output to the PostgreSQL log.
If this occurs then the archive log stream will be interrupted and PITR will not be possible past that point. A new backup will be required to regain full restore capability.
In asynchronous mode the entire queue will be dropped to prevent spurts of WAL getting through before the queue limit is exceeded again.
In asynchronous mode this limit is only checked at the start of each archive-push run, so the queue can grow beyond it within a single run. Reduce archive-push-batch-size to check the queue more frequently.
The purpose of this feature is to prevent the log volume from filling up at which point PostgreSQL will stop completely. Better to lose the backup than have PostgreSQL go down. 
allowed: [0B, 4PiB]
example: archive-push-queue-max=1TiB
Deprecated Name: archive-queue-max
Archive Timeout Option (--archive-timeout)
Archive timeout.
Set maximum time, in seconds, to wait for each WAL segment to reach the pgBackRest archive repository. The timeout applies to the check and backup commands when waiting for WAL segments required for backup consistency to be archived. 
default: 1m
allowed: [100ms, 1d]
example: archive-timeout=30
Backup Options
The backup section defines settings related to backup.
Backup Annotation Option (--annotation)
Annotate backup with user-defined key/value pairs.
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output. 
example: annotation=source="Sunday backup for website database"
Check Archive Option (--archive-check)
Check that WAL segments are in the archive before backup completes.
Checks that all WAL segments required to make the backup consistent are present in the WAL archive. It's a good idea to leave this as the default unless you are using another method for archiving.
This option must be enabled if archive-copy is enabled. 
default: y
example: archive-check=n
Copy Archive Option (--archive-copy)
Copy WAL segments needed for consistency to the backup.
This slightly paranoid option protects against corruption in the WAL segment archive by storing the WAL segments required for consistency directly in the backup. WAL segments are still stored in the archive so this option will use additional space.
It is best if the archive-push and backup commands have the same compress-type (e.g. lz4) when using this option. Otherwise, the WAL segments will need to be recompressed with the compress-type used by the backup, which can be fairly expensive depending on how much WAL was generated during the backup.
On restore, the WAL segments will be present in pg_xlog/pg_wal and PostgreSQL will use them in preference to calling the restore_command.
The archive-check option must be enabled if archive-copy is enabled. 
default: n
example: archive-copy=y
Check Archive Mode Option (--archive-mode-check)
Check the PostgreSQL archive_mode setting.
Enabled by default, this option disallows PostgreSQL archive_mode=always.
WAL segments pushed from a standby server might be logically the same as WAL segments pushed from the primary but have different checksums. Disabling archiving from multiple sources is recommended to avoid conflicts.
CAUTION:
If this option is disabled then it is critical to ensure that only one archiver is writing to the repository via the archive-push command. 
default: y
example: archive-mode-check=n
Backup from Standby Option (--backup-standby)
Backup from the standby cluster.
Enable backup from standby to reduce load on the primary cluster. This option requires that both the primary and standby hosts be configured.
The following modes are supported:
  • y - Standby is required for backup.
  • prefer - Backup from standby if available otherwise backup from primary.
  • n - Backup from primary only. 
default: n
example: backup-standby=y
Page Checksums Option (--checksum-page)
Validate data page checksums.
Directs pgBackRest to validate all data page checksums while backing up a cluster. This option is automatically enabled when data page checksums are enabled on the cluster.
Failures in checksum validation will not abort a backup. Rather, warnings will be emitted in the log (and to the console with default settings) and the list of invalid pages will be stored in the backup manifest. 
example: checksum-page=n
Path/File Exclusions Option (--exclude)
Exclude paths/files from the backup.
All exclusions are relative to $PGDATA. If the exclusion ends with / then only files in the specified directory will be excluded, e.g. --exclude=junk/ will exclude all files in the $PGDATA/junk directory but include the directory itself. If the exclusion does not end with / then the file may match the exclusion exactly or match with / appended to the exclusion, e.g. --exclude=junk will exclude the $PGDATA/junk directory and all the files it contains.
Be careful using this feature -- it is very easy to exclude something critical that will make the backup inconsistent. Be sure to test your restores!
All excluded files will be logged at info level along with the exclusion rule. Be sure to audit the list of excluded files to ensure nothing unexpected is being excluded.
NOTE:
Exclusions are not honored on delta restores. Any files/directories that were excluded by the backup will be removed on delta restore.
This option should not be used to exclude PostgreSQL logs from a backup. Logs can be moved out of the PGDATA directory using the PostgreSQL log_directory setting, which has the benefit of allowing logs to be preserved after a restore.
Multiple exclusions may be specified on the command-line or in a configuration file. 
example: exclude=junk/
Expire Auto Option (--expire-auto)
Automatically run the expire command after a successful backup.
The setting is enabled by default. Use caution when disabling this option as doing so will result in retaining all backups and archives indefinitely, which could cause your repository to run out of space. The expire command will need to be run regularly to prevent this from happening.
When expire is run automatically after a successful backup it uses the configuration of the backup command, so options set only in an expire command section (e.g. [global:expire]) are not applied. To apply expire-specific configuration, disable this option and run the expire command separately. 
default: y
example: expire-auto=y
Manifest Save Threshold Option (--manifest-save-threshold)
Manifest save threshold during backup.
Defines how often the manifest will be saved during a backup. Saving the manifest is important because it stores the checksums and allows the resume function to work efficiently. The actual threshold used is 1% of the backup size or manifest-save-threshold, whichever is greater. 
default: 1GiB
allowed: [1B, 1TiB]
example: manifest-save-threshold=8GiB
Resume Option (--resume)
Allow resume of failed backup.
Defines whether the resume feature is enabled. Resume can greatly reduce the amount of time required to run a backup after a previous backup of the same type has failed. It adds complexity, however, so it may be desirable to disable in environments that do not require the feature. 
default: y
example: resume=n
Start Fast Option (--start-fast)
Force a checkpoint to start backup quickly.
Forces a checkpoint (by passing y to the fast parameter of the backup start function) so the backup begins immediately. Otherwise the backup will start after the next regular checkpoint. 
default: n
example: start-fast=y
General Options
The general section defines options that are common for many commands.
Allow Run as Root Option (--allow-root)
Allow the command to run as the root user.
By default only the restore command may be run as the root user since it is designed to carefully manage file ownership. Running other commands as root risks creating files (e.g. in the repository) that are owned by root and therefore inaccessible to the PostgreSQL user, causing later commands to fail.
Enable this option to run a command as root anyway. However, it is far better to run pgBackRest as the user that owns the repository and PostgreSQL cluster. 
default: n
example: allow-root=y
Buffer Size Option (--buffer-size)
Buffer size for I/O operations.
Buffer size used for copy, compress, encrypt, and other operations. The number of buffers used depends on options and each operation may use additional memory, e.g. gz compression may use an additional 256KiB of memory.
Allowed values are 16KiB, 32KiB, 64KiB, 128KiB, 256KiB, 512KiB, 1MiB, 2MiB, 4MiB, 8MiB, and 16MiB. 
default: 1MiB
example: buffer-size=2MiB
pgBackRest Command Option (--cmd)
pgBackRest command.
pgBackRest may generate a command string, e.g. when the restore command generates the restore_command setting. The command used to run the pgBackRest process will be used in this case unless the cmd option is provided.
CAUTION:
Wrapping the pgBackRest command may cause unpredictable behavior and is not recommended. 
default: [path of executed pgbackrest binary]
example: cmd=/var/lib/pgsql/bin/pgbackrest_wrapper.sh
SSH Client Command Option (--cmd-ssh)
SSH client command.
Use a specific SSH client command when an alternate is desired or the ssh command is not in $PATH. 
default: ssh
example: cmd-ssh=/usr/bin/ssh
Compress Option (--compress)
Use file compression.
Backup files are compatible with command-line compression tools.
This option is now deprecated. The compress-type option should be used instead. 
default: y
example: compress=n
Compress Level Option (--compress-level)
File compression level.
Sets the level to be used for file compression when compress-type does not equal none or compress=y (deprecated). 
default (depending on compress-type):
    bz2 - 9
    gz - 6
    lz4 - 1
    zst - 3

allow range (depending on compress-type):
    bz2 - [1, 9]
    gz - [-1, 9]
    lz4 - [-5, 12]
    zst - [-7, 22]

example: compress-level=9
Network Compress Level Option (--compress-level-network)
Network compression level.
Sets the network compression level when compress-type=none and the command is not run on the same host as the repository. Compression is used to reduce network traffic. When compress-type does not equal none the compress-level-network setting is ignored and compress-level is used instead so that the file is only compressed once. 
default: 1
allowed: [-5, 12]
example: compress-level-network=1
Compress Type Option (--compress-type)
File compression type.
The following compression types are supported:
  • none - no compression
  • bz2 - bzip2 compression format
  • gz - gzip compression format
  • lz4 - lz4 compression format (not available on all platforms)
  • zst - Zstandard compression format (not available on all platforms) 
default: gz
example: compress-type=none
Database Timeout Option (--db-timeout)
Database query timeout.
Sets the timeout, in seconds, for queries against the database. This includes the backup start/stop functions which can each take a substantial amount of time. Because of this the timeout should be kept high unless you know that these functions will return quickly (i.e. if you have set start-fast=y and you know that the database cluster will not generate many WAL segments during the backup).
NOTE:
The db-timeout option must be less than the protocol-timeout option. 
default: 30m
allowed: [100ms, 7d]
example: db-timeout=600
Delta Option (--delta)
Restore or backup using checksums.
During a restore, by default the PostgreSQL data and tablespace directories are expected to be present but empty. This option performs a delta restore using checksums.
During a backup, this option will use checksums instead of the timestamps to determine if files will be copied. 
default: n
example: delta=y
I/O Timeout Option (--io-timeout)
I/O timeout.
Timeout, in seconds, used for connections and read/write operations.
Note that the entire read/write operation does not need to complete within this timeout but some progress must be made, even if it is only a single byte. 
default: 1m
allowed: [100ms, 1h]
example: io-timeout=120
Lock Path Option (--lock-path)
Path where lock files are stored.
The lock path provides a location for pgBackRest to create lock files to prevent conflicting operations from being run concurrently. 
default: /tmp/pgbackrest
example: lock-path=/backup/db/lock
Neutral Umask Option (--neutral-umask)
Use a neutral umask.
Sets the umask to 0000 so modes in the repository are created in a sensible way. The default directory mode is 0750 and default file mode is 0640.
To use the executing user's umask instead specify neutral-umask=n in the config file or --no-neutral-umask on the command line. 
default: y
example: neutral-umask=n
Set Process Priority Option (--priority)
Set process priority.
Defines how much priority (i.e. niceness) will be given to the process by the kernel scheduler. Positive values decrease priority and negative values increase priority. In most case processes do not have permission to increase their priority. 
allowed: [-20, 19]
example: priority=19
Process Maximum Option (--process-max)
Max processes to use for compress/transfer.
Each process will perform compression and transfer to make the command run faster, but don't set process-max so high that it impacts database performance. 
default: 1
allowed: [1, 999]
example: process-max=4
Protocol Timeout Option (--protocol-timeout)
Protocol timeout.
Sets the timeout, in seconds, that the local or remote process will wait for a new message to be received on the protocol layer. This prevents processes from waiting indefinitely for a message.
NOTE:
The protocol-timeout option must be greater than the db-timeout option. 
default: 31m
allowed: [100ms, 7d]
example: protocol-timeout=630
Keep Alive Option (--sck-keep-alive)
Keep-alive enable.
Enables keep-alive messages on socket connections. 
default: y
example: sck-keep-alive=n
Spool Path Option (--spool-path)
Path where transient data is stored.
This path is used to store data for the asynchronous archive-push and archive-get command.
The asynchronous archive-push command writes acknowledgements into the spool path when it has successfully stored WAL in the archive (and errors on failure) so the foreground process can quickly notify PostgreSQL. Acknowledgement files are very small (zero on success and a few hundred bytes on error).
The asynchronous archive-get command queues WAL in the spool path so it can be provided very quickly when PostgreSQL requests it. Moving files to PostgreSQL is most efficient when the spool path is on the same filesystem as pg_xlog/pg_wal. However, it is not recommended to place the spool path within the pg_xlog/pg_wal directory as this may cause issues for PostgreSQL utilities such as pg_rewind.
The data stored in the spool path is not strictly temporary since it can and should survive a reboot. However, loss of the data in the spool path is not a problem. pgBackRest will simply recheck each WAL segment to ensure it is safely archived for archive-push and rebuild the queue for archive-get.
The spool path is intended to be located on a local Posix-compatible filesystem, not a remote filesystem such as NFS or CIFS. 
default: /var/spool/pgbackrest
example: spool-path=/backup/db/spool
Keep Alive Count Option (--tcp-keep-alive-count)
Keep-alive count.
Specifies the number of TCP keep-alive messages that can be lost before the connection is considered dead.
This option is available on systems that support the TCP_KEEPCNT socket option. 
allowed: [1, 32]
example: tcp-keep-alive-count=3
Keep Alive Idle Option (--tcp-keep-alive-idle)
Keep-alive idle time.
Specifies the amount of time (in seconds) with no network activity after which the operating system should send a TCP keep-alive message.
This option is available on systems that support the TCP_KEEPIDLE socket option. 
allowed: [1, 3600]
example: tcp-keep-alive-idle=60
Keep Alive Interval Option (--tcp-keep-alive-interval)
Keep-alive interval time.
Specifies the amount of time (in seconds) after which a TCP keep-alive message that has not been acknowledged should be retransmitted.
This option is available on systems that support the TCP_KEEPINTVL socket option. 
allowed: [1, 900]
example: tcp-keep-alive-interval=30
TLSv1.2 cipher suites Option (--tls-cipher-12)
Allowed TLSv1.2 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. The example is reasonable choice unless you have specific security requirements. If unset (the default), the default of the underlying OpenSSL library applies. 
example: tls-cipher-12=HIGH:MEDIUM:+3DES:!aNULL
TLSv1.3 cipher suites Option (--tls-cipher-13)
Allowed TLSv1.3 cipher suites.
All TLS connections between the pgBackRest client and server are encrypted. By default, connections to objects stores (e.g. S3) are also encrypted.
NOTE:
The absolute minimum security level for any transport connection is TLSv1.2.
The accepted cipher suites can be adjusted if need arises. If unset (the default), the default of the underlying OpenSSL library applies. 
example: tls-cipher-13=TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
Log Options
The log section defines logging-related settings.
CAUTION:
Trace-level logging may expose secrets such as keys and passwords. Use with caution!
Console Log Level Option (--log-level-console)
Level for console logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: warn
example: log-level-console=error
File Log Level Option (--log-level-file)
Level for file logging.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: info
example: log-level-file=debug
Std Error Log Level Option (--log-level-stderr)
Level for stderr logging.
Specifies which log levels will output to stderr rather than stdout (specified by log-level-console). The timestamp and process will not be output to stderr.
The following log levels are supported:
  • off - No logging at all (not recommended)
  • error - Log only errors
  • warn - Log warnings and errors
  • info - Log info, warnings, and errors
  • detail - Log detail, info, warnings, and errors
  • debug - Log debug, detail, info, warnings, and errors
  • trace - Log trace (very verbose debugging), debug, info, warnings, and errors 
default: off
example: log-level-stderr=error
Log Path Option (--log-path)
Path where log files are stored.
The log path provides a location for pgBackRest to store log files. Note that if log-level-file=off then no log path is required. 
default: /var/log/pgbackrest
example: log-path=/backup/db/log
Log Subprocesses Option (--log-subprocess)
Enable logging in subprocesses.
Enable file logging for any subprocesses created by this process using the log level specified by log-level-file. 
default: n
example: log-subprocess=y
Log Timestamp Option (--log-timestamp)
Enable timestamp in logging.
Enables the timestamp in console and file logging. This option is disabled in special situations such as generating documentation. 
default: y
example: log-timestamp=n
Maintainer Options
Maintainer options are intended to support PostgreSQL forks. The proper settings should be determined by the fork maintainer and then communicated to users of the fork.
WARNING:
Improper use of these options may lead to unexpected behavior or data corruption.
It is the responsibility of the fork maintainer to test pgBackRest with the required options. pgBackRest does not guarantee compatibility with any fork.
Check WAL Headers Option (--archive-header-check)
Check PostgreSQL version/id in WAL headers.
Enabled by default, this option checks the WAL header against the PostgreSQL version and system identifier to ensure that the WAL is being copied to the correct stanza. This is in addition to checking pg_control against the stanza and verifying that WAL is being copied from the same PostgreSQL data directory where pg_control is located.
Therefore, disabling this check is fairly safe but should only be done when needed, e.g. if the WAL is encrypted. 
default: y
example: archive-header-check=n
Page Header Check Option (--page-header-check)
Check PostgreSQL page headers.
Enabled by default, this option adds page header checks.
Disabling this option should be avoided except when necessary, e.g. if pages are encrypted. 
default: y
example: page-header-check=n
Force PostgreSQL Version Option (--pg-version-force)
Force PostgreSQL version.
The specified PostgreSQL version will be used instead of the version automatically detected by reading pg_control or WAL headers. This is mainly useful for PostgreSQL forks or development versions where those values are different from the release version. The version reported by PostgreSQL via `server_version_num` must match the forced version.
WARNING:
Be cautious when using this option because pg_control and WAL headers will still be read with the expected format for the specified version, i.e. the format from the official open-source version of PostgreSQL. If the fork or development version changes the format of the fields that pgBackRest depends on it will lead to unexpected behavior. In general, this option will only work as expected if the fork adds all custom struct members after the standard PostgreSQL members. 
example: pg-version-force=15
Repository Options
The repository section defines options used to configure the repository.
Indexing: All repo- options are indexed to allow for configuring multiple repositories. For example, a single repository is configured with the repo1-path, repo1-host, etc. options. If there is more than one repository configured and the --repo option is not specified for a command, the repositories will be acted upon in highest priority order (e.g. repo1 then repo2).
The repo-retention-* options define how long backups will be retained. Expiration only occurs when the count of complete backups exceeds the allowed retention. In other words, if repo1-retention-full-type is set to count (default) and repo1-retention-full is set to 2, then there must be 3 complete backups before the oldest will be expired. If repo1-retention-full-type is set to time then repo1-retention-full represents days so there must be at least that many days worth of full backups before expiration can occur. Make sure you always have enough space for retention + 1 backups.
Azure Repository Account Option (--repo-azure-account)
Azure repository account.
Azure account used to store the repository. 
example: repo1-azure-account=pg-backup
Azure Repository Container Option (--repo-azure-container)
Azure repository container.
Azure container used to store the repository.
pgBackRest repositories can be stored in the container root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other Azure-generated content can also be stored in the container. 
example: repo1-azure-container=pg-backup
Azure Repository Endpoint Option (--repo-azure-endpoint)
Azure repository endpoint.
Endpoint used to connect to the blob service. The default is generally correct unless using Azure Government.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
default: blob.core.windows.net
example: repo1-azure-endpoint=blob.core.usgovcloudapi.net
Azure Repository Key Option (--repo-azure-key)
Azure repository key.
A shared key or shared access signature depending on the repo-azure-key-type option. 
example: repo1-azure-key=T+9+aov82qNhrcXSNGZCzm9mjd4d75/oxxOr6r1JVpgTLA==
Azure Repository Key Type Option (--repo-azure-key-type)
Azure repository key type.
The following types are supported for authorization:
  • shared - Shared key
  • sas - Shared access signature
  • auto - Automatically authorize using Azure managed identities 
default: shared
example: repo1-azure-key-type=sas
Azure Repository URI Style Option (--repo-azure-uri-style)
Azure URI Style.
The following URI styles are supported:
  • host - Connect to account.endpoint host.
  • path - Connect to endpoint host and prepend account to URIs. 
default: host
example: repo1-azure-uri-style=path
Block Incremental Backup Option (--repo-block)
Enable block incremental backup.
Block incremental allows for more granular backups by splitting files into blocks that can be backed up independently. This saves space in the repository and can improve delta restore performance because individual blocks can be fetched without reading the entire file from the repository.
NOTE:
The repo-bundle option must be enabled before repo-block can be enabled.
The block size for a file is determined based on the file size and age. Generally, older/larger files will get larger block sizes. If a file is old enough, it will not be backed up using block incremental.
Block incremental is most efficient when enabled for all backup types, including full. This makes the full a bit larger but subsequent differential and incremental backups can make use of the block maps generated by the full backup to save space. 
default: n
example: repo1-block=y
Repository Bundles Option (--repo-bundle)
Bundle files in repository.
Bundle (combine) smaller files to reduce the total number of files written to the repository. Writing fewer files is generally more efficient, especially on object stores such as S3. In addition, zero-length files are not stored (except in the manifest), which saves time and space. 
default: n
example: repo1-bundle=y
Repository Bundle Limit Option (--repo-bundle-limit)
Limit for file bundles.
Size limit for files that will be included in bundles. Files larger than this size will be stored separately.
Bundled files cannot be reused when a backup is resumed, so this option controls the files that can be resumed, i.e. higher values result in fewer resumable files. 
default: 2MiB
allowed: [8KiB, 1PiB]
example: repo1-bundle-limit=10MiB
Repository Bundle Size Option (--repo-bundle-size)
Target size for file bundles.
Defines the target size for files that will be added to a single bundle. The uncompressed bundle size may be as large as repo-bundle-size + repo-bundle-limit, so do not set this option to the maximum size that your file system allows.
In general, it is not a good idea to set this option too high because retries will need to redo the entire bundle. 
default: 20MiB
allowed: [1MiB, 1PiB]
example: repo1-bundle-size=10MiB
Repository Cipher Passphrase Option (--repo-cipher-pass)
Repository cipher passphrase.
Passphrase used to encrypt/decrypt files of the repository.
NOTE:
When run without the stanza option the info command reads encryption settings only from the global section. If encryption settings are configured per stanza, run the info command with the stanza option to read an encrypted stanza. 
example: repo1-cipher-pass=zWaf6XtpjIVZC5444yXB+cgFDFl7MxGlgkZSaoPvTGirhPygu4jOKOXf9LO4vjfO
Repository Cipher Type Option (--repo-cipher-type)
Cipher used to encrypt the repository.
The following cipher types are supported:
  • none - The repository is not encrypted
  • aes-256-cbc - Advanced Encryption Standard with 256 bit key length
Note that encryption is always performed client-side even if the repository type (e.g. S3) supports encryption. 
default: none
example: repo1-cipher-type=aes-256-cbc
GCS Repository Bucket Option (--repo-gcs-bucket)
GCS repository bucket.
GCS bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other GCS-generated content can also be stored in the bucket. 
example: repo1-gcs-bucket=/pg-backup
GCS Repository Endpoint Option (--repo-gcs-endpoint)
GCS repository endpoint.
Endpoint used to connect to the storage service. May be updated to use a local GCS server or alternate endpoint. 
default: storage.googleapis.com
example: repo1-gcs-endpoint=localhost
GCS Repository Key Option (--repo-gcs-key)
GCS repository key.
A token or service key file depending on the repo-gcs-key-type option. 
example: repo1-gcs-key=/etc/pgbackrest/gcs-key.json
GCS Repository Key Type Option (--repo-gcs-key-type)
GCS repository key type.
The following types are supported for authorization:
  • auto - Authorize using the instance service account.
  • service - Service account from locally stored key.
  • token - For local testing, e.g. fakegcs.
When repo-gcs-key-type=service the credentials will be reloaded when the authentication token is renewed. 
default: service
example: repo1-gcs-key-type=auto
GCS Repository Project ID Option (--repo-gcs-user-project)
GCS project ID.
GCS project ID used to determine request billing. 
example: repo1-gcs-user-project=my-project
Repository Hardlink Option (--repo-hardlink)
Hardlink files between backups in the repository.
Enable hard-linking of files in differential and incremental backups to their full backups. This gives the appearance that each backup is a full backup at the file-system level. Be careful, though, because modifying files that are hard-linked can affect all the backups in the set. 
default: n
example: repo1-hardlink=y
Deprecated Name: hardlink
Repository Host Option (--repo-host)
Repository host when operating remotely.
When backing up and archiving to a locally mounted filesystem this setting is not required. 
example: repo1-host=repo1.domain.com
Deprecated Name: backup-host
Repository Host Certificate Authority File Option (--repo-host-ca-file)
Repository host certificate authority file.
Use a CA file other than the system default for connecting to the repository host. 
example: repo1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Repository Host Certificate Authority Path Option (--repo-host-ca-path)
Repository host certificate authority path.
Use a CA path other than the system default for connecting to the repository host. 
example: repo1-host-ca-path=/etc/pki/tls/certs
Repository Host Certificate File Option (--repo-host-cert-file)
Repository host certificate file.
Sent to repository host to prove client identity. 
example: repo1-host-cert-file=/path/to/client.crt
Repository Host Command Option (--repo-host-cmd)
Repository host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and repository hosts. If not defined, the repository host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: repo1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: backup-cmd
Repository Host Configuration Option (--repo-host-config)
pgBackRest repository host configuration file.
Sets the location of the configuration file on the repository host. This is only required if the repository host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: repo1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: backup-config
Repository Host Configuration Include Path Option (--repo-host-config-include-path)
pgBackRest repository host configuration include path.
Sets the location of the configuration include path on the repository host. This is only required if the repository host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: repo1-host-config-include-path=/conf/pgbackrest/conf.d
Repository Host Configuration Path Option (--repo-host-config-path)
pgBackRest repository host configuration path.
Sets the location of the configuration path on the repository host. This is only required if the repository host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: repo1-host-config-path=/conf/pgbackrest
Repository Host Key File Option (--repo-host-key-file)
Repository host key file.
Proves client certificate was sent by owner. 
example: repo1-host-key-file=/path/to/client.key
Repository Host Port Option (--repo-host-port)
Repository host port when repo-host is set.
Use this option to specify a non-default port for the repository host protocol.
NOTE:
When repo-host-type=ssh there is no default for repo-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on repo-host-type):
    tls - 8432

allowed: [0, 65535]
example: repo1-host-port=25
Deprecated Name: backup-ssh-port
Repository Host Protocol Type Option (--repo-host-type)
Repository host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: repo1-host-type=tls
Repository Host User Option (--repo-host-user)
Repository host user when repo-host is set.
Defines the user that will be used for operations on the repository host. Preferably this is not the postgres user but rather some other user like pgbackrest. If PostgreSQL runs on the repository host the postgres user can be placed in the pgbackrest group so it has read permissions on the repository without being able to damage the contents accidentally. 
default: pgbackrest
example: repo1-host-user=repo-user
Deprecated Name: backup-user
Repository Path Option (--repo-path)
Path where backups and archive are stored.
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves. 
default: /var/lib/pgbackrest
example: repo1-path=/backup/db/backrest
Archive Retention Option (--repo-retention-archive)
Number of backups worth of continuous WAL to retain.
NOTE:
WAL segments required to make a backup consistent are always retained until the backup is expired regardless of how this option is configured.
If this value is not set and repo-retention-full-type is count (default), then the archive to expire will default to the repo-retention-full (or repo-retention-diff) value corresponding to the repo-retention-archive-type if set to full (or diff). This will ensure that WAL is only expired for backups that are already expired. If repo-retention-full-type is time, then this value will default to removing archives that are earlier than the oldest full backup retained after satisfying the repo-retention-full setting.
This option must be set if repo-retention-archive-type is set to incr. If disk space is at a premium, then this setting, in conjunction with repo-retention-archive-type, can be used to aggressively expire WAL segments. However, doing so negates the ability to perform PITR from the backups with expired WAL and is therefore not recommended. 
allowed: [1, 9999999]
example: repo1-retention-archive=2
Deprecated Name: retention-archive
Archive Retention Type Option (--repo-retention-archive-type)
Backup type for WAL retention.
If set to full pgBackRest will keep archive logs for the number of full backups defined by repo-retention-archive. If set to diff (differential) pgBackRest will keep archive logs for the number of full and differential backups defined by repo-retention-archive, meaning if the last backup taken was a full backup, it will be counted as a differential for the purpose of repo-retention. If set to incr (incremental) pgBackRest will keep archive logs for the number of full, differential, and incremental backups defined by repo-retention-archive. It is recommended that this setting not be changed from the default which will only expire WAL in conjunction with expiring full backups. 
default: full
example: repo1-retention-archive-type=diff
Deprecated Name: retention-archive-type
Differential Retention Option (--repo-retention-diff)
Number of differential backups to retain.
When a differential backup expires, all incremental backups associated with the differential backup will also expire. When not defined all differential backups will be kept until the full backups they depend on expire.
Note that full backups are included in the count of differential backups for the purpose of expiration. This slightly reduces the number of differential backups that need to be retained in most cases. 
allowed: [1, 9999999]
example: repo1-retention-diff=3
Deprecated Name: retention-diff
Full Retention Option (--repo-retention-full)
Full backup retention count/time.
When a full backup expires, all differential and incremental backups associated with the full backup will also expire. When the option is not defined a warning will be issued. If indefinite retention is desired then set the option to the max value. 
allowed: [1, 9999999]
example: repo1-retention-full=2
Deprecated Name: retention-full
Full Retention Type Option (--repo-retention-full-type)
Retention type for full backups.
Determines whether the repo-retention-full setting represents a time period (days) or count of full backups to keep.
If set to time then full backups older than repo-retention-full will be removed from the repository if there is at least one other backup that is equal to or greater than the repo-retention-full setting. For example, if repo-retention-full is 30 (days) and there are 2 full backups: one 25 days old and one 35 days old, no full backups will be expired because expiring the 35 day old backup would leave only the 25 day old backup, which would violate the 30 day retention policy of having at least one backup 30 days old before an older one can be expired. Archived WAL older than the oldest full backup remaining will be automatically expired unless repo-retention-archive-type and repo-retention-archive are explicitly set.
If set to count then full backups that exceed repo-retention-full will be expired. For example, if repo-retention-full is 4 and a fifth full backup is completed, then the oldest full backup will be expired to keep the count at 4.
Note that a backup must be successfully completed before it will be considered for retention. For example, if repo-retention-full-type is count and repo-retention-full is 2, then there must be 3 complete full backups before the oldest will be expired. 
default: count
example: repo1-retention-full-type=time
Backup History Retention Option (--repo-retention-history)
Days of backup history manifests to retain.
A copy of the backup manifest is stored in the backup.history path when a backup completes. By default these files are never expired since they are useful for data mining, e.g. measuring backup and WAL growth over time.
Set repo-retention-history to define the number of days of backup history manifests to retain. Unexpired backups are always kept in the backup history. Specify repo-retention-history=0 to retain the backup history only for unexpired backups.
When a full backup history manifest is expired, all differential and incremental backup history manifests associated with the full backup also expire. 
allowed: [0, 9999999]
example: repo1-retention-history=365
S3 Repository Bucket Option (--repo-s3-bucket)
S3 repository bucket.
S3 bucket used to store the repository.
pgBackRest repositories can be stored in the bucket root by setting repo-path=/ but it is usually best to specify a prefix, such as /repo, so logs and other AWS generated content can also be stored in the bucket. 
example: repo1-s3-bucket=pg-backup
S3 Repository Endpoint Option (--repo-s3-endpoint)
S3 repository endpoint.
The AWS endpoint should be valid for the selected region.
For custom/test configurations the repo-storage-ca-file, repo-storage-ca-path, repo-storage-host, repo-storage-port, and repo-storage-verify-tls options may be useful. 
example: repo1-s3-endpoint=s3.amazonaws.com
S3 Repository Access Key Option (--repo-s3-key)
S3 repository access key.
AWS key used to access this bucket. 
example: repo1-s3-key=AKIAIOSFODNN7EXAMPLE
S3 Repository Secret Access Key Option (--repo-s3-key-secret)
S3 repository secret access key.
AWS secret key used to access this bucket. 
example: repo1-s3-key-secret=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
S3 Repository Key Type Option (--repo-s3-key-type)
S3 repository key type.
The following types are supported:
  • shared - Shared keys
  • auto - Automatically retrieve temporary credentials
  • web-id - Automatically retrieve web identity credentials
  • pod-id - Automatically retrieve EKS pod identity credentials
  • process - Retrieve credentials by executing a process 
default: shared
example: repo1-s3-key-type=auto
S3 Repository KMS Key ID Option (--repo-s3-kms-key-id)
S3 repository KMS key.
Enables S3 server-side encryption using the specified AWS key management service key. 
example: repo1-s3-kms-key-id=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Authentication Process Command Option (--repo-s3-process-cmd)
S3 authentication process command.
Command (and optional arguments) to execute for retrieving temporary S3 credentials. The first list entry is the command and the remaining entries are passed as parameters.
The process must output JSON containing AccessKeyId, SecretAccessKey, SessionToken, and Expiration fields. Credentials will be automatically refreshed before the expiration time. See Process Credential Provider for format details. 
example: repo1-s3-process-cmd=/usr/local/bin/get-credentials
example: repo1-s3-process-cmd=--role
example: repo1-s3-process-cmd=my-role
S3 Repository Region Option (--repo-s3-region)
S3 repository region.
The AWS region where the bucket was created. 
example: repo1-s3-region=us-east-1
S3 Repository Requestor Pays Option (--repo-s3-requester-pays)
S3 repository requester pays.
Enables S3 requester pays. 
default: n
example: repo1-s3-requester-pays=n
S3 Repository Role Option (--repo-s3-role)
S3 repository role.
The AWS role name (not the full ARN) used to retrieve temporary credentials when repo-s3-key-type=auto. 
example: repo1-s3-role=authrole
S3 Repository Service Option (--repo-s3-service)
S3 signing service.
The S3 signing service used in SigV4 authentication. Defaults to s3 for standard S3 endpoints. Set to s3-outposts when using an S3 Outposts endpoint. 
default: s3
example: repo1-s3-service=s3-outposts
S3 Repository SSE Customer Key Option (--repo-s3-sse-customer-key)
S3 repository SSE customer key.
Enables S3 server-side encryption using the specified customer key. 
example: repo1-s3-sse-customer-key=bceb4f13-6939-4be3-910d-df54dee817b7
S3 Repository STS Endpoint Option (--repo-s3-sts-host)
S3 repository STS endpoint.
The STS endpoint used to retrieve temporary credentials when repo-s3-key-type=web-id is configured. Set to a regional endpoint (e.g. sts.us-east-1.amazonaws.com) to use regional STS, which may be required for GovCloud, China regions, or to reduce latency. 
default: sts.amazonaws.com
example: repo1-s3-sts-host=sts.us-east-1.amazonaws.com
S3 Repository Security Token Option (--repo-s3-token)
S3 repository security token.
AWS security token used with temporary credentials. 
example: repo1-s3-token=AQoDYXdzEPT//////////wEXAMPLEtc764bNrC9SAPBSM22 ...
S3 Repository URI Style Option (--repo-s3-uri-style)
S3 URI Style.
The following URI styles are supported:
  • host - Connect to bucket.endpoint host.
  • path - Connect to endpoint host and prepend bucket to URIs. 
default: host
example: repo1-s3-uri-style=path
SFTP Repository Host Option (--repo-sftp-host)
SFTP repository host.
The SFTP host containing the repository. 
example: repo1-sftp-host=sftprepo.domain
SFTP Repository Host Fingerprint Option (--repo-sftp-host-fingerprint)
SFTP repository host fingerprint.
SFTP repository host fingerprint generation should match the repo-sftp-host-key-hash-type. Generate the fingerprint via awk '{print $2}' ssh_host_xxx_key.pub | base64 -d | (md5sum or sha1sum) -b. The ssh host keys are normally found in the /etc/ssh directory. 
example: repo1-sftp-host-fingerprint=f84e172dfead7aeeeae6c1fdfb5aa8cf
SFTP Host Key Check Type Option (--repo-sftp-host-key-check-type)
SFTP host key check type.
The following SFTP host key check types are supported:
  • strict - pgBackRest will never automatically add host keys to the ~/.ssh/known_hosts file, and refuses to connect to hosts whose host key has changed or is not found in the known hosts files. This option forces the user to manually add all new hosts.
  • accept-new - pgBackRest will automatically add new host keys to the user's known hosts file, but will not permit connections to hosts with changed host keys.
  • fingerprint - pgBackRest will check the host key against the fingerprint specified by the repo-sftp-host-fingerprint option.
  • none - no host key checking will be performed. 
default: strict
example: repo1-sftp-host-key-check-type=accept-new
SFTP Repository Host Key Hash Type Option (--repo-sftp-host-key-hash-type)
SFTP repository host key hash type.
SFTP repository host key hash type. Declares the hash type to be used to compute the digest of the remote system's host key on SSH startup. Newer versions of libssh2 support sha256 in addition to md5 and sha1. 
example: repo1-sftp-host-key-hash-type=sha256
SFTP Repository Host Port Option (--repo-sftp-host-port)
SFTP repository host port.
SFTP repository host port. 
default: 22
allowed: [1, 65535]
example: repo1-sftp-host-port=22
SFTP Repository Host User Option (--repo-sftp-host-user)
SFTP repository host user.
User on the host used to store the repository. 
example: repo1-sftp-host-user=pg-backup
SFTP Known Hosts File Option (--repo-sftp-known-host)
SFTP known hosts file.
A known hosts file to search for an SFTP host match during authentication. When unspecified, pgBackRest will default to searching ~/.ssh/known_hosts, ~/.ssh/known_hosts2, /etc/ssh/ssh_known_hosts, and /etc/ssh/ssh_known_hosts2. If configured with one or more file paths, pgBackRest will search those for a match. File paths must be full or leading tilde paths. The repo-sftp-known-host option can be passed multiple times to specify more than one known hosts file to search. To utilize known hosts file checking repo-sftp-host-fingerprint must not be specified. See also repo-sftp-host-check-type option. 
example: repo1-sftp-known-host=/home/postgres/.ssh/known_hosts
SFTP Repository Private Key File Option (--repo-sftp-private-key-file)
SFTP private key file.
SFTP private key file used for authentication. 
example: repo1-sftp-private-key-file=~/.ssh/id_ed25519
SFTP Repository Private Key Passphrase Option (--repo-sftp-private-key-passphrase)
SFTP private key passphrase.
Passphrase used to access the private key. This is an optional feature when creating an SSH public/private key pair. 
example: repo1-sftp-private-key-passphrase=BeSureToGenerateAndUseASecurePassphrase
SFTP Repository Public Key File Option (--repo-sftp-public-key-file)
SFTP public key file.
SFTP public key file used for authentication. Optional if compiled against OpenSSL, required if compiled against a different library. 
example: repo1-sftp-public-key-file=~/.ssh/id_ed25519.pub
Repository Storage CA File Option (--repo-storage-ca-file)
Repository storage CA file.
Use a CA file other than the system default for storage (e.g. S3, Azure) certificates. 
example: repo1-storage-ca-file=/etc/pki/tls/certs/ca-bundle.crt
Deprecated Names: repo-azure-ca-file, repo-s3-ca-file
Repository Storage TLS CA Path Option (--repo-storage-ca-path)
Repository storage CA path.
Use a CA path other than the system default for storage (e.g. S3, Azure) certificates. 
example: repo1-storage-ca-path=/etc/pki/tls/certs
Deprecated Names: repo-azure-ca-path, repo-s3-ca-path
Repository Storage Host Option (--repo-storage-host)
Repository storage host.
Connect to a host other than the storage (e.g. S3, Azure) endpoint. This is typically used for testing. 
example: repo1-storage-host=127.0.0.1
Deprecated Names: repo-azure-host, repo-s3-host
Repository Storage Port Option (--repo-storage-port)
Repository storage port.
Port to use when connecting to the storage (e.g. S3, Azure) endpoint (or host if specified). 
default: 443
allowed: [1, 65535]
example: repo1-storage-port=9000
Deprecated Names: repo-azure-port, repo-s3-port
Repository Storage Tag Option (--repo-storage-tag)
Repository storage tag(s).
Specify tags that will be added to objects when the repository is an object store (e.g. S3). The option can be repeated to add multiple tags.
There is no provision in pgBackRest to modify these tags so be sure to set them correctly before running stanza-create to ensure uniform tags across the entire repository. 
example: repo1-storage-tag=key1=value1
Repository Storage Upload Chunk Size Option (--repo-storage-upload-chunk-size)
Repository storage upload chunk size.
Object stores such as S3 allow files to be uploaded in chunks when the file is too large to be stored in memory. Even if the file can be stored in memory, it is more memory efficient to limit the amount of memory used for uploads.
A larger chunk size will generally lead to better performance because it will minimize upload requests and allow more files to be uploaded in a single request rather than in chunks. The disadvantage is that memory usage will be higher and because the chunk buffer must be allocated per process, larger process-max values will lead to more memory being consumed overall.
Note that valid chunk sizes vary by storage type and by platform. For example, AWS S3 has a minimum chunk size of 5MiB. Terminology for chunk size varies by storage type, so when searching min/max values use part size for AWS S3, chunk size for GCS, and block size for Azure.
If a file is larger than 1GiB (the maximum size PostgreSQL will create by default) then the chunk size will be increased incrementally up to the maximum allowed in order to complete the file upload. 
default (depending on repo-type):
    azure - 4MiB
    gcs - 4MiB
    s3 - 5MiB

allow range (depending on repo-type):
    azure - [4MiB, 1GiB]
    gcs - [4MiB, 1GiB]
    s3 - [5MiB, 1GiB]

example: repo1-storage-upload-chunk-size=16MiB
Repository Storage Certificate Verify Option (--repo-storage-verify-tls)
Repository storage certificate verify.
This option provides the ability to enable/disable verification of the storage (e.g. S3, Azure) server TLS certificate. Disabling should only be used for testing or other scenarios where a certificate has been self-signed. 
default: y
example: repo1-storage-verify-tls=n
Deprecated Names: repo-azure-verify-tls, repo-s3-verify-ssl, repo-s3-verify-tls
Repository Symlink Option (--repo-symlink)
Create symlinks within the repository.
Enable creation of the lastest and tablespace symlinks. These symlinks are most useful when using snapshots to do in-place recovery in the repository, which is an uncommon use case.
While this feature is likely not useful for the vast majority of users it remains on by default for legacy purposes. However, it may be useful to disable symlinks for Posix-like storage that does not support them. 
default: y
example: repo1-symlink=n
Target Time for Repository Option (--repo-target-time)
Target time for repository.
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided. 
example: repo-target-time=2024-08-08 12:12:12+00
Repository Type Option (--repo-type)
Type of storage used for the repository.
The following repository types are supported:
  • azure - Azure Blob Storage Service
  • cifs - Like posix, but disables links and directory fsyncs
  • gcs - Google Cloud Storage
  • posix - Posix-compliant file systems
  • s3 - AWS Simple Storage Service
  • sftp - Secure File Transfer Protocol
When an NFS mount is used as a posix repository, the same rules apply to pgBackRest as described in the PostgreSQL documentation: Creating a Database Cluster - File Systems. 
default: posix
example: repo1-type=cifs
Restore Options
The restore section defines settings used for restoring backups.
Archive Mode Option (--archive-mode)
Preserve or disable archiving on restored cluster.
This option allows archiving to be preserved or disabled on a restored cluster. This is useful when the cluster must be promoted to do some work but is not intended to become the new primary. In this case it is not a good idea to push WAL from the cluster into the repository.
The following modes are supported:
  • off - disable archiving by setting archive_mode=off.
  • preserve - preserve current archive_mode setting.
NOTE: This option is not available on PostgreSQL < 12. 
default: preserve
example: archive-mode=off
Exclude Database Option (--db-exclude)
Restore excluding the specified databases.
Databases excluded will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, those databases will not be accessible but can be removed with the drop database command. The --db-exclude option can be passed multiple times to specify more than one database to exclude.
When used in combination with the --db-include option, --db-exclude will only apply to standard system databases (template0, template1, and postgres). 
example: db-exclude=db_main
Include Database Option (--db-include)
Restore only specified databases.
This feature allows only selected databases to be restored. Databases not specifically included will be restored as sparse, zeroed files to save space but still allow PostgreSQL to perform recovery. After recovery, the databases that were not included will not be accessible but can be removed with the drop database command.
NOTE:
built-in databases (template0, template1, and postgres) are always restored unless specifically excluded.
The --db-include option can be passed multiple times to specify more than one database to include.
See Restore Selected Databases for additional information and caveats. 
example: db-include=db_main
Link All Option (--link-all)
Restore all symlinks.
By default symlinked directories and files are restored as normal directories and files in $PGDATA. This is because it may not be safe to restore symlinks to their original destinations on a system other than where the original backup was performed. This option restores all the symlinks just as they were on the original system where the backup was performed. 
default: n
example: link-all=y
Link Map Option (--link-map)
Modify the destination of a symlink.
Allows the destination file or path of a symlink to be changed on restore. This is useful for restoring to systems that have a different storage layout than the original system where the backup was generated. 
example: link-map=pg_xlog=/data/xlog
Recovery Option Option (--recovery-option)
Set an option in postgresql.auto.conf or recovery.conf.
See Server Configuration for details on postgresql.auto.conf or recovery.conf options (be sure to select your PostgreSQL version). This option can be used multiple times.
For PostgreSQL >= 12, options will be written into postgresql.auto.conf. For all other versions, options will be written into recovery.conf.
NOTE:
The restore_command option will be automatically generated but can be overridden with this option. Be careful about specifying your own restore_command as pgBackRest is designed to handle this for you. Target Recovery options (recovery_target_name, recovery_target_time, etc.) are generated automatically by pgBackRest and should not be set with this option.
Since pgBackRest does not start PostgreSQL after writing the postgresql.auto.conf or recovery.conf file, it is always possible to edit/check postgresql.auto.conf or recovery.conf before manually restarting. 
example: recovery-option=primary_conninfo=db.mydomain.com
Tablespace Map Option (--tablespace-map)
Restore a tablespace into the specified directory.
Moves a tablespace to a new location during the restore. This is useful when tablespace locations are not the same on a replica, or an upgraded system has different mount points.
Tablespace locations are not stored in pg_tablespace so moving tablespaces can be done with impunity. However, moving a tablespace to the data_directory is not recommended and may cause problems. For more information on moving tablespaces http://www.databasesoup.com/2013/11/moving-tablespaces.html is a good resource. 
example: tablespace-map=ts_01=/db/ts_01
Map All Tablespaces Option (--tablespace-map-all)
Restore all tablespaces into the specified directory.
Tablespaces are restored into their original locations by default. This behavior can be modified for each tablespace with the tablespace-map option, but it is sometimes preferable to remap all tablespaces to a new directory all at once. This is particularly useful for development or staging systems that may not have the same storage layout as the original system where the backup was generated.
The path specified will be the parent path used to create all the tablespaces in the backup.
CAUTION:
Tablespaces created after the backup started will not be mapped. Make a new backup after a tablespace is created if tablespace mapping is required. 
example: tablespace-map-all=/data/tablespace
Server Options
The server section defines options used for configuring the TLS server.
TLS Server Address Option (--tls-server-address)
TLS server address.
IP address the server will listen on for client requests. 
default: localhost
example: tls-server-address=*
TLS Server Authorized Clients Option (--tls-server-auth)
TLS server authorized clients.
Clients are authorized on the server by verifying their certificate and checking their certificate CN (Common Name) against a list on the server configured with the tls-server-auth option.
A client CN can be authorized for as many stanzas as needed by providing a comma-separated list to the tls-server-auth option or for all stanzas by specifying tls-server-auth=client-cn=*. Wildcards may not be specified for the client CN. 
example: tls-server-auth=client-cn=stanza1,stanza2
TLS Server Certificate Authorities Option (--tls-server-ca-file)
TLS server certificate authorities.
Checks that client certificates are signed by a trusted certificate authority. 
example: tls-server-ca-file=/path/to/server.ca
TLS Server Certificate Option (--tls-server-cert-file)
TLS server certificate file.
Sent to the client to show the server identity. 
example: tls-server-cert-file=/path/to/server.crt
TLS Server Key Option (--tls-server-key-file)
TLS server key file.
Proves server certificate was sent by the owner. 
example: tls-server-key-file=/path/to/server.key
TLS Server Port Option (--tls-server-port)
TLS server port.
Port the server will listen on for client requests. 
default: 8432
allowed: [1, 65535]
example: tls-server-port=8000
Stanza Options
A stanza defines the backup configuration for a specific PostgreSQL database cluster. The stanza section must define the database cluster path and host/user if the database cluster is remote. Also, any global configuration sections can be overridden to define stanza-specific settings.
Indexing: All pg- options are indexed to allow for configuring multiple PostgreSQL hosts. For example, a single primary is configured with the pg1-path, pg1-port, etc. options. If a standby is configured then index the pg- options on the repository host as pg2- (e.g. pg2-host, pg2-path, etc).
PostgreSQL Database Option (--pg-database)
PostgreSQL database.
The database name used when connecting to PostgreSQL. The default is usually best but some installations may not contain this database.
Note that for legacy reasons the setting of the PGDATABASE environment variable will be ignored. 
default: postgres
example: pg1-database=backupdb
PostgreSQL Host Option (--pg-host)
PostgreSQL host for operating remotely.
Used for backups where the PostgreSQL host is different from the repository host. 
example: pg1-host=db.domain.com
Deprecated Name: db-host
PostgreSQL Host Certificate Authority File Option (--pg-host-ca-file)
PostgreSQL host certificate authority file.
Use a CA file other than the system default for connecting to the PostgreSQL host. 
example: pg1-host-ca-file=/etc/pki/tls/certs/ca-bundle.crt
PostgreSQL Host Certificate Authority Path Option (--pg-host-ca-path)
PostgreSQL host certificate authority path.
Use a CA path other than the system default for connecting to the PostgreSQL host. 
example: pg1-host-ca-path=/etc/pki/tls/certs
PostgreSQL Host Certificate File Option (--pg-host-cert-file)
PostgreSQL host certificate file.
Sent to PostgreSQL host to prove client identity. 
example: pg1-host-cert-file=/path/to/client.crt
PostgreSQL Host Command Option (--pg-host-cmd)
PostgreSQL host pgBackRest command.
Required only if the path to the pgBackRest command is different on the local and PostgreSQL hosts. If not defined, the PostgreSQL host command will be set the same as the local command. 
default: [path of executed pgbackrest binary]
example: pg1-host-cmd=/usr/lib/backrest/bin/pgbackrest
Deprecated Name: db-cmd
PostgreSQL Host Configuration Option (--pg-host-config)
pgBackRest database host configuration file.
Sets the location of the configuration file on the PostgreSQL host. This is only required if the PostgreSQL host configuration file is in a different location than the local configuration file. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE
example: pg1-host-config=/conf/pgbackrest/pgbackrest.conf
Deprecated Name: db-config
PostgreSQL Host Configuration Include Path Option (--pg-host-config-include-path)
pgBackRest database host configuration include path.
Sets the location of the configuration include path on the PostgreSQL host. This is only required if the PostgreSQL host configuration include path is in a different location than the local configuration include path. 
default: CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH
example: pg1-host-config-include-path=/conf/pgbackrest/conf.d
PostgreSQL Host Configuration Path Option (--pg-host-config-path)
pgBackRest database host configuration path.
Sets the location of the configuration path on the PostgreSQL host. This is only required if the PostgreSQL host configuration path is in a different location than the local configuration path. 
default: CFGOPTDEF_CONFIG_PATH
example: pg1-host-config-path=/conf/pgbackrest
PostgreSQL Host Key File Option (--pg-host-key-file)
PostgreSQL host key file.
Proves client certificate was sent by owner. 
example: pg1-host-key-file=/path/to/client.key
PostgreSQL Host Port Option (--pg-host-port)
PostgreSQL host port when pg-host is set.
Use this option to specify a non-default port for the PostgreSQL host protocol.
NOTE:
When pg-host-type=ssh there is no default for pg-host-port. In this case the port will be whatever is configured for the command specified by cmd-ssh. 
default (depending on pg-host-type):
    tls - 8432

allowed: [0, 65535]
example: pg1-host-port=25
Deprecated Name: db-ssh-port
PostgreSQL Host Protocol Type Option (--pg-host-type)
PostgreSQL host protocol type.
The following protocol types are supported:
  • ssh - Secure Shell.
  • tls - pgBackRest TLS server. 
default: ssh
example: pg1-host-type=tls
PostgreSQL Host User Option (--pg-host-user)
PostgreSQL host logon user when pg-host is set.
This user will also own the remote pgBackRest process and will initiate connections to PostgreSQL. For this to work correctly the user should be the PostgreSQL database cluster owner which is generally postgres, the default. 
default: postgres
example: pg1-host-user=db_owner
Deprecated Name: db-user
PostgreSQL Path Option (--pg-path)
PostgreSQL data directory.
This should be the same as the data_directory reported by PostgreSQL. Even though this value can be read from various places, it is prudent to set it in case those resources are not available during a restore or offline backup scenario.
The pg-path option is tested against the value reported by PostgreSQL on every online backup so it should always be current. 
example: pg1-path=/data/db
Deprecated Name: db-path
PostgreSQL Port Option (--pg-port)
PostgreSQL port.
Port that PostgreSQL is running on. This usually does not need to be specified as most PostgreSQL clusters run on the default port. 
default: 5432
allowed: [0, 65535]
example: pg1-port=6543
Deprecated Name: db-port
PostgreSQL Socket Path Option (--pg-socket-path)
PostgreSQL unix socket path.
The unix socket directory that was specified when PostgreSQL was started. pgBackRest will automatically look in the standard location for your OS so there is usually no need to specify this setting unless the socket directory was explicitly modified with the unix_socket_directories setting in postgresql.conf. 
example: pg1-socket-path=/var/run/postgresql
Deprecated Name: db-socket-path
PostgreSQL Database User Option (--pg-user)
PostgreSQL database user.
The database user name used when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER. 
example: pg1-user=backupuser
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated August 17, 2026
././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1786957821.773953 pgbackrest-2.59.1/doc/html/default.css0000644000175100017510000002537715240547776017311 0ustar00runnerrunner/******************************************************************************* Html and body *******************************************************************************/ html { font-family: Avenir, Corbel, sans-serif; font-size: 12pt; margin-top: 8px; margin-left: 1%; margin-right: 1%; width: 98%; } body { background: #f8f8f8; margin: 0px auto; padding: 0px; width: 100%; text-align: justify; font-weight: 400; counter-reset: section1-counter; counter-reset: section1-toc-counter; } a:link, a:visited, a:hover, a:active { text-decoration: underline; color: black; display: inline-block; } @media (min-width: 1000px) { body { width: 1000px; } } @media (prefers-color-scheme: dark) { body { color: #e2e2e2; background: #181818; font-weight: 350; } body a:link, body a:visited, body a:hover, body a:active { text-decoration: underline; color: #e2e2e2; } } /******************************************************************************* Header *******************************************************************************/ .page-header { width:100%; text-align:center; } .page-header-title { font-size: xx-large; font-weight: bolder; } .page-header-subtitle { position: relative; top: -.25em; font-size: large; font-weight: bold; } .page-header-logo { float: left; margin-top: 0.3em; } .page-header-logo > img { width: 69px; height: 69px; } /******************************************************************************* Menu *******************************************************************************/ /*.page-menu*/ .menu-body { text-align: center; } .menu { white-space: nowrap; display: inline; margin-left: .5em; font-size: 13pt; } .menu:first-of-type { margin-left: 0px; } .menu:before { content: "["; } .menu:after { content: "]"; } /*.menu-link*/ a.menu-link:link, a.menu-link:visited, a.menu-link:active { text-decoration: none; } a.menu-link:hover { text-decoration: underline; } /******************************************************************************* Table of Contents *******************************************************************************/ .page-toc-body { margin-top: .25em; margin-bottom: 2em; } .section1-toc:first-of-type { margin-top: 0; } .section1-toc { margin-top: .5em; } .section1-toc-number { display: inline; font-size: 14pt; margin-right: .75em; counter-reset: section2-toc-counter; } .section1-toc-number:before { counter-increment: section1-toc-counter; content: counter(section1-toc-counter); } .section1-toc-title { display: inline; font-size: 14pt; } .section2-toc { margin-left: 1.8em; } .section2-toc-number { display: inline; margin-right: .75em; counter-reset: section3-toc-counter; } .section2-toc-number:before { counter-increment: section2-toc-counter; content: counter(section1-toc-counter) "." counter(section2-toc-counter); } .section2-toc-title { display: inline; } .section3-toc { margin-left: 2.3em; } .section3-toc-number { display: inline; margin-right: .75em; } .section3-toc-number:before { counter-increment: section3-toc-counter; content: counter(section1-toc-counter) "." counter(section2-toc-counter) "." counter(section3-toc-counter); } .section3-toc-title { display: inline; } /******************************************************************************* Section *******************************************************************************/ .section1 { margin-top: 2.5em; counter-reset: section2-counter; counter-increment: section1-counter; } .section1:first-of-type { margin-top: 1em; } .section2, .section3 { margin-top: 1em; } .section2 { counter-reset: section3-counter; counter-increment: section2-counter; } .section3 { counter-increment: section3-counter; } .section1-header, .section2-header, .section3-header, .page-toc-header { margin-top: .5em; font-weight: 500; } .section1-header, .page-toc-header { border-radius: 3px; background-color: #205581; color: #f2f2f2; font-size: 22pt; padding-left: .5em; margin-bottom: .5em; } .section1-number, .section2-number, .section3-number { display: inline; margin-right: .5em; } .section1-title, .section2-title, .section3-title { display: inline; } .section1-subtitle, .section2-subtitle, .section3-subtitle { font-weight: bold; font-size: 16pt; margin-top: .5em; } .section1-subsubtitle, .section2-subsubtitle, .section3-subsubtitle { font-style: italic; font-size: 14pt; margin-top: 0px; } .section1-number:before { content: counter(section1-counter); } .section1-subtitle { font-size: 16pt; } .section1-subsubtitle { font-size: 14pt; } .section2 { margin-top: 1em; } .section2-header { border-bottom: 2px #27689d solid; color: #27689d; font-size: 16pt; } .section2-number:before { content: counter(section1-counter) "." counter(section2-counter); } .section2-subtitle, .section2-subtitle { font-size: 14pt; } .section2-subsubtitle, .section3-subsubtitle { font-size: 12pt; } .section3 { margin-left: 1em; margin-top: 1em; } .section3:first-of-type { border-top: none; } .section3-header { display: inline; font-size: 14pt; color: #27689d; border-bottom: 1px #27689d solid; } .section3-number:before { content: counter(section1-counter) "." counter(section2-counter) "." counter(section3-counter); } .section-intro { margin-top: .75em; } /*.section-body*/ /******************************************************************************* Table Elements *******************************************************************************/ .table { margin-top: .5em; border-spacing: .5em 0; } .table-caption { text-align: left; font-size: 12pt; font-weight: bold; color: #27689d; } /*.table-header-row*/ .table-header-left, .table-header-center, .table-header-right { white-space: nowrap; border-bottom: 1px black solid; } .table-header-left { text-align: left; } .table-header-center { text-align: center; } .table-header-right { text-align: right; } .table-header-fill { width: 100%; } /*.table-row*/ .table-data-left, .table-data-center, .table-data-right { white-space: nowrap; } .table-data-left { padding-top: 0px; text-align: left; } .table-data-center { text-align: center; } .table-data-right { text-align: right; } /******************************************************************************* Config, Execute, Code Block, Release Elements *******************************************************************************/ .section-body-text { margin-top: 1em; } .section-body-text:first-of-type { margin-top: .75em; } .execute, .config, .code-block { /*white-space: nowrap;*/ margin-left: 1em; margin-right: 1em; margin-top: .8em; margin-bottom: 1em; } .execute-title, .config-title { font-size: 11pt; margin-bottom: .125em; } .execute-body, .config-body, .code-block { border-radius: 3px; background-color: #444444; font-family: monospace; unicode-bidi: embed; color: #f8f8f8; /*color: white;*/ font-size: 10pt; } .execute-body-cmd, .execute-body-output, .execute-body-output-highlight, .config-body-title, .code-block { padding-left: .5em; padding-right: .5em; } /* Would rather not use pre at all, but Firefox won't copy code samples correctly without it */ pre { margin: 0px; padding: 0px; white-space: pre-wrap; } .execute-body-cmd:before { content: "$ "; } .execute-body-cmd:first-of-type, .execute-body-output, .config-body-title, .config-body-output, .code-block, .execute-body-output-highlight, .execute-body-output-highlight-error { padding-top: .25em; } .execute-body-cmd, .execute-body-output, .config-body-title, .config-body-output, .code-block, .execute-body-output-highlight, .execute-body-output-highlight-error { padding-bottom: .25em; } .execute-body-output, .config-body-output, .code-block { border-radius: 3px; background-color: #606060; } .execute-body-output-highlight { border-radius: 3px; background-color: #1e7b1e; } .execute-body-output-highlight-error { border-radius: 3px; background-color: #a32929; } .execute-body-output, .execute-body-output-highlight, .execute-body-output-highlight-error, .config-body-output { padding-left: 1.75em; } /*.code-block*/ /*.section-body-execute-output*/ /******************************************************************************* Lists *******************************************************************************/ ul.list-unordered { margin-top: .5em; } li.list-unordered { margin-top: .5em; } /******************************************************************************* Sponsor *******************************************************************************/ .sponsor-list { display: grid; grid-template-columns: repeat(3, max-content); justify-content: center; justify-items: center; align-items: center; gap: .75em 2.5em; margin-top: 1.5em; margin-bottom: 1.5em; } .sponsor { text-align: center; } .sponsor-img-dark { display: none; } @media (prefers-color-scheme: dark) { .sponsor-img-light { display: none; } .sponsor-img-dark { display: inline; } } /******************************************************************************* Admonition *******************************************************************************/ .admonition { padding-left: 20px; font-style: italic; margin: 10px 0px; } .admonition>.note, .admonition>.warning, .admonition>.caution, .admonition>.important, .admonition>.tip { font-weight: bold; float: left; padding-right: 10px; } .admonition>.note-text, .admonition>.warning-text, .admonition>.caution-text, .admonition>.important-text, .admonition>.tip-text { overflow: hidden; } /******************************************************************************* Keywords *******************************************************************************/ .backrest, .cmd, .postgres, .host, .proper { font-weight: 500; } .path, .br-option, .br-setting, .pg-option, .pg-setting, .id, .user, .file, .path { unicode-bidi: embed; font-family: monospace; white-space: nowrap; } /******************************************************************************* Footer *******************************************************************************/ .page-footer { border-top: 2px #cccccc solid; margin-top: 1em; padding-top: .25em; font-size: x-small; white-space: nowrap; text-align: center; } ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7745638 pgbackrest-2.59.1/doc/html/faq.html0000640000175100017510000003760215240547776016576 0ustar00runnerrunner pgBackRest - Frequently Asked Questions
pgBackRest
Frequently Asked Questions
Home
User Guides
Releases
Configuration
Commands
Table of Contents
Introduction
What if I get the could not find WAL segment error?
How do I manually purge a backup set?
How can I configure options independently for each command?
Can I use dots (periods) in my S3 bucket name?
Where can I find packages for older versions of pgBackRest?
Why does a backup attempt fail when backup-standby=y and the standby database is down?
Should I setup my repository on a standby host?
Time-based Point-in-Time Recovery does not appear to work, why?
What does the WAL archive suffix mean?
Does it take longer to restore specific backup types (full, differential, incremental)?
How can I export a backup for use in a network-isolated environment?
Introduction
Frequently Asked Questions are intended to provide details for specific questions that may or may not be covered in the User Guide, Configuration, or Command reference. If you are unable to find details for your specific issue here, remember that the pgBackRest Issues List in GitHub is also a valuable resource.
What if I get the could not find WAL segment error?
The cause of this error can be a result of many different issues, some of which may be:
  • misconfigured archive_command
  • misconfigured pgBackRest configuration files
  • network or permissions issue
  • third party product (e.g. S3, Swift or Minio) configuration issue
  • large amount of WAL queueing to be archived
It is advisable to:
  • check the archive_command in PostgreSQL
  • check the pgBackRest configuration settings on each host (e.g. pg* settings are set on the repository host and repo* settings on the pg host)
  • run the check command with --archive-timeout set to a higher value than in the pgBackRest configuration file (or default) to see if the WAL queue needs more time to clear. If the system is generating a lot of WAL, then consider configuring asynchronous archiving
How do I manually purge a backup set?
A full backup set can be expired using the --set option as explained in Command Reference: Expire.
How can I configure options independently for each command?
pgBackRest has the ability to set options independently in the configuration file for each command. Configure Cluster Stanza details this feature as well as option precedence.
For example, the process-max option can be optimized for each command: 
[global]
# used where not overridden
process-max=2

[global:backup]
# more cores for backup
process-max=4

[global:restore]
# all the cores for restore
process-max=8

[global:archive-push]
# more cores for archive-push
process-max=3

[global:archive-get]
# fewer cores for archive-get
process-max=1
Can I use dots (periods) in my S3 bucket name?
RFC-2818 does not allow wildcards to match on a dot (.) so s3 bucket names must not contain dots. If there are dots in the S3 bucket name then an error such as unable to find hostname 'my.backup.bucket.s3.amazonaws.com' in certificate common name or subject alternative names will occur.
The exception is repo-s3-uri-style=path, which prepends the bucket to the URI instead of the host. Since the bucket is not part of the host name, the certificate is matched against the endpoint alone and dots in the bucket name are not a problem. Note that path-style URIs are not supported by all S3-compatible object stores.
Where can I find packages for older versions of pgBackRest?
The apt.postgresql.org repository maintains an archive of older versions. Debian also maintains snapshots of all test builds.
Why does a backup attempt fail when backup-standby=y and the standby database is down?
Configuring backup from standby is generally intended to reduce load on the primary, so switching backups to the primary when the standby is down often defeats the point. Putting more load on the primary in a situation where there are already failures in the system is not recommended. Backups are not critical as long as you have one that is fairly recent -- the important thing is to keep up with WAL archiving. There is plenty of time to get a backup when the system is stable again.
If you really need a backup, the solution is to have more standbys or remove backup-standby. This can be overridden on the command line with --no-backup-standby, so there is no need to reconfigure for a one-off backup.
Should I setup my repository on a standby host?
No. When primary and standby databases are configured, the pgBackRest configuration files should be symmetric in order to seamlessly handle failovers. If they are not, the configurations will need to be changed on failover or further problems may result.
See the Dedicated Repository Host section of the User Guide for more information.
Time-based Point-in-Time Recovery does not appear to work, why?
The most common mistake when using time-based Point-in-Time Recovery is forgetting to choose a backup set that is before the target time. pgBackRest will attempt to discover a backup to play forward from the time specified by the --target= if the --set option is not specified. If a backup set cannot be found, then restore will default to the latest backup. However, if the latest backup is after the target time, then --target= is not considered valid by PostgreSQL and is therefore ignored, resulting in WAL recovery to the latest time available.
To use the --set option, choose a backup set by running the info command and finding the backup with a timestamp stop that is before the target time. Then when running the restore, specify the option --set=BACKUP_LABEL where BACKUP_LABEL is the chosen backup set.
See the Point-in-Time Recovery section of the User Guide for more information.
What does the WAL archive suffix mean?
The suffix is the SHA1 checksum used to verify file integrity. There is no way to omit it.
Does it take longer to restore specific backup types (full, differential, incremental)?
The various backup types require the same amount of time to restore. Restore retrieves files based on the backup manifest, which may reference files from a previous backup in the case of incremental or differential backups. While there could be differences in time spent making a given backup (depending on backup type), database size determines restore time (disk I/O, network I/O, etc. being equal).
How can I export a backup for use in a network-isolated environment?
pgBackRest uses the repository not only to store backups and WAL archives but also to maintain essential metadata required for features such as compression, encryption, and file bundling. Because of this, simply copying a backup along with a subset of WAL files usually will not work unless very specific and restrictive conditions are met.
However, there is a workaround if your goal is to create a self-contained export of a database that you can transfer (e.g., via USB). You can make a backup with the --archive-copy option enabled to ensure that the necessary WAL segments are stored along with the backup. Then, restore it using --type=none --pg1-path=/your/target/path. This produces a restored PostgreSQL data directory with all required WAL files already placed in pg_wal, similar to what pg_basebackup would create.
You can then copy this directory to another system, and PostgreSQL should be able to recover from it without needing access to the pgBackRest repository.
Please note that recovering this backup will not result in a timeline switch, which means that this cluster should not push WAL to the original repository that it was exported from. If the new cluster is in a network-isolated environment this should not be a problem.
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated August 17, 2026
././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1786957821.774604 pgbackrest-2.59.1/doc/html/index.html0000640000175100017510000004010415240547776017125 0ustar00runnerrunner pgBackRest - Reliable PostgreSQL Backup & Restore
pgBackRest
Reliable PostgreSQL Backup & Restore
User Guides
Releases
Configuration
Commands
FAQ
Introduction
pgBackRest is a reliable backup and restore solution for PostgreSQL that seamlessly scales up to the largest databases and workloads.
pgBackRest v2.59.1 is the current stable release. Release notes are on the Releases page.
Please give us a star on GitHub if you like pgBackRest!
Features
Parallel Backup & Restore
Compression is usually the bottleneck during backup operations so pgBackRest solves this problem with parallel processing and more efficient compression algorithms such as lz4 and zstd.
Local or Remote Operation
A custom protocol allows pgBackRest to backup, restore, and archive locally or remotely via TLS/SSH with minimal configuration. An interface to query PostgreSQL is also provided via the protocol layer so that remote access to PostgreSQL is never required, which enhances security.
Multiple Repositories
Multiple repositories allow, for example, a local repository with minimal retention for fast restores and a remote repository with a longer retention for redundancy and access across the enterprise.
Full, Differential, & Incremental Backups (at File or Block Level)
Full, differential, and incremental backups are supported. pgBackRest is not susceptible to the time resolution issues of rsync, making differential and incremental backups safe without the requirement to checksum each file. Block-level backups save space by only copying the parts of files that have changed.
Backup Rotation & Archive Expiration
Retention policies can be set for full and differential backups to create coverage for any time frame. The WAL archive can be maintained for all backups or strictly for the most recent backups. In the latter case WAL required to make older backups consistent will be maintained in the archive.
Backup Integrity
Checksums are calculated for every file in the backup and rechecked during a restore or verify. After a backup finishes copying files, it waits until every WAL segment required to make the backup consistent reaches the repository.
Backups in the repository may be stored in the same format as a standard PostgreSQL cluster (including tablespaces). If compression is disabled and hard links are enabled it is possible to snapshot a backup in the repository and bring up a PostgreSQL cluster directly on the snapshot. This is advantageous for terabyte-scale databases that are time consuming to restore in the traditional way.
All operations utilize file and directory level fsync to ensure durability.
Page Checksums
If page checksums are enabled pgBackRest will validate the checksums for every file that is copied during a backup. All page checksums are validated during a full backup and checksums in files that have changed are validated during differential and incremental backups.
Validation failures do not stop the backup process, but warnings with details of exactly which pages have failed validation are output to the console and file log.
This feature allows page-level corruption to be detected early, before backups that contain valid copies of the data have expired.
Backup Resume
An interrupted backup can be resumed from the point where it was stopped. Files that were already copied are compared with the checksums in the manifest to ensure integrity. Since this operation can take place entirely on the repository host, it reduces load on the PostgreSQL host and saves time since checksum calculation is faster than compressing and retransmitting data.
Streaming Compression & Checksums
Compression and checksum calculations are performed in stream while files are being copied to the repository, whether the repository is located locally or remotely.
If the repository is on a repository host, compression is performed on the PostgreSQL host and files are transmitted in a compressed format and simply stored on the repository host. When compression is disabled a lower level of compression is utilized to make efficient use of available bandwidth while keeping CPU cost to a minimum.
Delta Restore
The manifest contains checksums for every file in the backup so that during a restore it is possible to use these checksums to speed processing enormously. On a delta restore any files not present in the backup are first removed and then checksums are generated for the remaining files. Files that match the backup are left in place and the rest of the files are restored as usual. Parallel processing can lead to a dramatic reduction in restore times.
Parallel, Asynchronous WAL Push & Get
Dedicated commands are included for pushing WAL to the archive and getting WAL from the archive. Both commands support parallelism to accelerate processing and run asynchronously to provide the fastest possible response time to PostgreSQL.
WAL push automatically detects WAL segments that are pushed multiple times and de-duplicates when the segment is identical, otherwise an error is raised. Asynchronous WAL push allows transfer to be offloaded to another process which compresses WAL segments in parallel for maximum throughput. This can be a critical feature for databases with extremely high write volume.
Asynchronous WAL get maintains a local queue of WAL segments that are decompressed and ready for replay. This reduces the time needed to provide WAL to PostgreSQL which maximizes replay speed. Higher-latency connections and storage (such as S3) benefit the most.
The push and get commands both ensure that the database and repository match by comparing PostgreSQL versions and system identifiers. This virtually eliminates the possibility of misconfiguring the WAL archive location.
Tablespace & Link Support
Tablespaces are fully supported and on restore tablespaces can be remapped to any location. It is also possible to remap all tablespaces to one location with a single command which is useful for development restores.
File and directory links are supported for any file or directory in the PostgreSQL cluster. When restoring it is possible to restore all links to their original locations, remap some or all links, or restore some or all links as normal files or directories within the cluster directory.
S3, Azure, and GCS Compatible Object Store Support
pgBackRest repositories can be located in S3, Azure, and GCS compatible object stores to allow for virtually unlimited capacity and retention.
Encryption
pgBackRest can encrypt the repository to secure backups wherever they are stored.
Ransomware & Malware Protection
When the repository is stored on versioned object storage, pgBackRest can read the repository as it was at a point-in-time. If backups are deleted or corrupted by accident, malware, or ransomware, a target time can be used to recover data from before the damage occurred.
Versioning is supported by S3, Azure, and GCS compatible object stores. Object locking for S3 and soft delete for GCS or Azure can provide additional protection against tampering.
Compatibility with ten versions of PostgreSQL
pgBackRest includes support for ten versions of PostgreSQL, the five supported versions and the last five EOL versions. This allows ample time to upgrade to a supported version.
Getting Started
pgBackRest strives to be easy to configure and operate:
Contributions
Contributions to pgBackRest are always welcome! Please see our Contributing Guidelines for details on how to contribute features, improvements or issues.
Support
pgBackRest is completely free and open source under the MIT license. You may use it for personal or commercial purposes without any restrictions whatsoever. Bug reports are taken very seriously and will be addressed as quickly as possible. Please report bugs here.
Creating a robust disaster recovery policy with proper replication and backup strategies can be a very complex and daunting task. You may find that you need help during the architecture phase and ongoing support to ensure that your enterprise continues running smoothly.
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated August 17, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7739918 pgbackrest-2.59.1/doc/html/logo.svg0000644000175100017510000000200715240547776016615 0ustar00runnerrunner ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7746427 pgbackrest-2.59.1/doc/html/release.html0000640000175100017510000102270315240547776017444 0ustar00runnerrunner pgBackRest Releases
pgBackRest Releases
Home
User Guides
Configuration
Commands
FAQ
Table of Contents
Introduction
Current Stable Release
v2.59.1 Release Notes
Stable Releases
Pre-Stable Releases
Introduction
pgBackRest release numbers consist of two parts, major and minor. A major release may break compatibility with the prior major release, but v2 releases are fully compatible with v1 repositories and will accept all v1 options. Minor releases can include bug fixes and features but do not change the repository format and strive to avoid changing options and naming.

Documentation for the v1 release can be found here.

The notes for a release may also contain Additional Notes but changes in this section are only to documentation or the test suite and have no direct impact on the pgBackRest codebase.
Current Stable Release
v2.59.1 Release Notes
PostgreSQL 19beta3 Support
Released August 17, 2026
Bug Fixes:
  • Fix hang when the chunk buffer is smaller than the input buffer. (Reviewed by Andrew Pogrebnoi, Douglas J Hunley. Reported by crajac66.)
  • Compare a new backup label only against its own full backup set. (Reviewed by Douglas J Hunley. Reported by Anton Glushakov.)
  • Fix source archives generated by GitHub missing files required to build. (Reported by aardvarkzed.)
Features:
  • PostgreSQL 19beta3 support. (Contributed by Lardière Sébastien. Reviewed by David Steele.)
Documentation Improvements:
  • Document exception for dots in S3 bucket names when repo-s3-uri-style=path.
  • Remove explicit hot_standby configuration from user guide.
Stable Releases
v2.59.0 Release Notes
PostgreSQL 19beta2 Support
Released July 20, 2026
NOTE TO PACKAGERS: A new distribution tarball is available which simplifies the build process by providing pregenerated HTML documentation, man page, and code. The tarball is attached to each release as an asset and is named pgbackrest-{version}.tar.gz. See the README.md in the tarball for details. Please use the new distribution tarball to avoid the additional build tooling required to generate code in future releases. See New Distribution Tarball for more information.

NOTE TO PACKAGERS: There is a new optional dependency on libsystemd.

IMPORTANT NOTE: Starting with this release, only the restore command may be run as root by default. Use allow-root to run other commands as root (though this is not recommended).
Bug Fixes:
  • Fix expire removing a backup in progress on another host. (Reviewed by Douglas J Hunley. Reported by Dmitrii.)
  • Fix archive-push-queue-max not enforced when a WAL file errors. (Reviewed by Stefan Fercot. Reported by vut12.)
  • Fix async archive-get failure during recovery across a timeline switch. (Reviewed by Douglas J Hunley. Reported by Jimmy Yih.)
  • Fix potential buffer overrun in error module. (Fixed by Christophe Pettus. Reviewed by David Steele.)
  • Fix use-after-free of parallel protocol client. (Reviewed by Douglas J Hunley. Reported by Georgy Shelkovy.)
  • Fix heap overflow when a pg option is set without pg1. (Reviewed by Douglas J Hunley. Reported by Naveen.)
Features:
  • PostgreSQL 19beta2 support. (Reviewed by Stefan Fercot.)
  • Add archive-expire-before option to clean up WAL archive. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Add batch delete for Azure storage. (Reviewed by Douglas J Hunley, Crispy.)
  • Add support for S3 Outposts. (Contributed by Shiva Kumar Ambigi. Reviewed by David Steele, Roberto Mello.)
  • Add S3 process authentication. (Reviewed by Andrew Charlton. Suggested by Andrew Charlton.)
Improvements:
  • Reconnect SFTP storage after the server drops an idle connection. (Reviewed by Stefan Fercot. Suggested by mustafa0x, tisserat.)
  • Add user/group caching for faster manifest build. (Contributed by Gunnar Lindholm. Reviewed by David Steele.)
  • Add per-repo backup progress to info command output. (Contributed by Will Morland. Reviewed by David Steele, Stefan Fercot.)
  • Add backup.info checks to verify command. (Contributed by Denis Garsh. Reviewed by David Steele, Douglas J Hunley.)
  • Allow the S3 STS endpoint to be configured. (Contributed by Simon Gratton. Reviewed by David Steele.)
  • Add archive-push-batch-size to limit WAL pushed per asynchronous run. (Reviewed by Stefan Fercot.)
  • Exit async archive-push on first error. (Reviewed by Stefan Fercot. Suggested by Lardière Sébastien.)
  • Harden HTTP chunked response parsing. (Contributed by Shubham. Reviewed by David Steele.)
  • Error when running as root unless allow-root is enabled. (Reviewed by Douglas J Hunley.)
  • Fewer binary searches when building manifest. (Contributed by Gunnar Lindholm. Reviewed by David Steele.)
  • Improve seek performance during block incremental delta restore. (Reviewed by David Christensen, Douglas J Hunley.)
  • Bundle backup files in order rather than scanning for a better fit. (Reviewed by Stefan Fercot.)
  • Report the underlying error when a query fails to complete. (Reviewed by Andrew Pogrebnoi. Suggested by Marco Fontana.)
  • Report the actual error when an S3 credential request fails. (Reviewed by Douglas J Hunley. Suggested by Mitchell Grice.)
  • Report archive-push spool path errors to the PostgreSQL log. (Reviewed by Douglas J Hunley. Suggested by Don Seiler.)
  • Hint that pgBackRest may be out of date when a version is unsupported.
  • Add systemd notify integration. (Contributed by Andrew Jackson. Reviewed by David Steele.)
  • Suppress unused parameter errors in meson compiler probes. (Contributed by Jörg Plate. Reviewed by David Steele.)
  • C11 is now the minimum C standard. (Reviewed by Mohammad Ali Nazir Kosar.)
Documentation Improvements:
  • Make stanza option internal for the repo-* commands. (Reviewed by Stefan Fercot.)
  • Document that info reads encryption settings from global section. (Reviewed by Stefan Fercot. Suggested by Ron Johnson.)
  • Document that expire-auto uses the backup command configuration. (Reviewed by Stefan Fercot. Suggested by Lardière Sébastien.)
  • Document adjusting the logrotate su directive for a dedicated user. (Reviewed by Douglas J Hunley. Suggested by lkanbus.)
  • Document that end-of-line comments are not supported in config files. (Reviewed by Douglas J Hunley. Suggested by Alex Richman.)
Test Suite Improvements:
  • Fix Alpine group conflicts in CI containers. (Contributed by Artur Zakirov. Reviewed by David Steele.)
v2.58.0 Release Notes
Object Storage Improvements
Released January 19, 2026
IMPORTANT NOTE: The minimum values for the repo-storage-upload-chunk-size option have increased. They now represent the minimum allowed by the vendors.
Bug Fixes:
  • Fix deadlock due to logging in signal handler. (Fixed by Maxim Michkov. Reviewed by David Steele.)
Features:
  • HTTP support for S3, GCS, and Azure. (Contributed by Will Morland. Reviewed by David Steele.)
  • Allow expiration of oldest full backup regardless of current retention. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Ron Johnson.)
  • Support for Azure managed identities. (Contributed by Moiz Ibrar, Matthew Mols. Reviewed by David Steele.)
  • Experimental support for S3 EKS pod identity. (Contributed by Pierre BOUTELOUP. Reviewed by David Steele.)
  • Allow configuration of TLS cipher suites. (Contributed by Gunnar "Nick" Bluth. Reviewed by David Steele.)
  • Allow process priority to be set. (Reviewed by Douglas J Hunley.)
Improvements:
  • Allow dots in S3 bucket names when using path-style URIs. (Contributed by Joakim Hindersson. Reviewed by David Steele.)
  • Require TLS >= 1.2 unless verification is disabled. (Reviewed by Douglas J Hunley, Gunnar "Nick" Bluth.)
  • Dynamically size S3/GCS/Azure chunks for large uploads. (Reviewed by Douglas J Hunley. Suggested by Timothée Peignier.)
  • Optimize S3/GCS/Azure chunk size for small files. (Reviewed by Douglas J Hunley.)
  • Remove support for PostgreSQL 9.5. (Reviewed by Douglas J Hunley.)
  • Improve logging of default for options with an unresolved dependency. (Reviewed by Stefan Fercot.)
Documentation Improvements:
  • Remove explicit max_wal_senders/wal_level configuration from user guide. (Suggested by Jamie Nguyen.)
  • Clarify that bundling is useful for filesystems with large block sizes. (Suggested by Ron Johnson.)
v2.57.0 Release Notes
Suppress Repository Symlinks
Released October 18, 2025
Bug Fixes:
  • Unnest HTTP/TLS/socket timeouts. (Reviewed by David Christensen.)
  • Fix possible segfault in page checksum error message. (Fixed by Zsolt Parragi. Reviewed by David Steele.)
Features:
  • Add repo-symlink option to suppress creation of repository symlinks. (Reviewed by Douglas J Hunley. Suggested by Ron Johnson.)
Improvements:
  • Add HTTP retries for 408 and 429 errors. (Reviewed by David Christensen.)
v2.56.0 Release Notes
Progress Info Improvements
Released July 21, 2025
Bug Fixes:
  • Fix issue with adhoc expiration when no backups in a repository. (Reviewed by Stefan Fercot. Reported by Anup Gupta.)
Features:
  • Add restore progress to info command output. (Contributed by Denis Garsh, Maxim Michkov. Reviewed by David Steele.)
  • Add progress-only detail level for info command output. (Contributed by Denis Garsh. Reviewed by David Steele, Stefan Fercot.)
Improvements:
  • Retry failed reads on object stores. (Reviewed by David Christensen.)
  • Fix defaults in command-line help. (Reviewed by David Christensen, Chris Bandy.)
Documentation Improvements:
  • Describe discrete option values in a list where appropriate. (Contributed by Anton Kurochkin. Reviewed by David Steele.)
  • Fix less than in help output for archive-mode option. (Contributed by Anton Kurochkin. Reviewed by David Steele.)
v2.55.1 Release Notes
Bug Fixes
Released May 5, 2025
Bug Fixes:
  • Revert calculate content-md5 on S3 only when required. (Reviewed by David Christensen. Reported by Frank Brendel.)
  • Fix lower bounds checking for option keys. (Reviewed by David Christensen, Wolfgang Walther. Reported by Wolfgang Walther.)
v2.55.0 Release Notes
Verification Improvements and PostgreSQL 18 Support
Released April 21, 2025
Bug Fixes:
  • Fix block incremental restore issue on non-default repository. (Reviewed by David Christensen, Aleksander Łukasz. Reported by Aleksander Łukasz.)
  • Do not set recovery_target_timeline=current for PostgreSQL < 12. (Reviewed by Stefan Fercot.)
  • Fix expire archive range logging. (Reviewed by Stefan Fercot. Reported by Aleš Zelený.)
  • Fix error reporting for queries with no results. (Reviewed by Stefan Fercot. Reported by Susantha Bathige.)
Features:
  • Verify recovery target timeline. (Reviewed by Stefan Fercot.)
  • Allow verification of a specified backup. (Contributed by Maxim Michkov. Reviewed by David Steele.)
  • Add support for S3/GCS requester pays. (Contributed by Timothée Peignier. Reviewed by David Steele.)
  • PostgreSQL 18 support. (Reviewed by Stefan Fercot.)
  • Allow connections to PostgreSQL on abstract domain sockets. (Reviewed by Chris Bandy. Suggested by Chris Bandy.)
  • Add numeric output to version command. (Contributed by Stefan Fercot. Reviewed by David Steele.)
Improvements:
  • Allow backup command to operate on remote repositories. (Reviewed by Stefan Fercot.)
  • Use lz4 for protocol compression. (Reviewed by Stefan Fercot.)
  • Calculate content-md5 on S3 only when required. (Reviewed by David Christensen.)
  • Warn when a value for a multi-key option is overwritten. (Reviewed by David Christensen, Stefan Fercot.)
  • Add detail logging for expired archive path. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Remove support for PostgreSQL 9.4. (Reviewed by Stefan Fercot.)
  • Remove autoconf/make build. (Reviewed by David Christensen.)
Documentation Improvements:
  • Fix documentation for specifying multiple stanzas with tls-server-auth. (Reviewed by David Christensen, Stefan Fercot. Suggested by Terry MacAndrew.)
  • Clarify incremental backup expiration. (Reviewed by Stefan Fercot.)
  • Clarify requirement for local/remote pgBackRest versions to match. (Contributed by Greg Clough. Reviewed by David Steele.)
  • Add FAQ about exporting self-contained cluster. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Caveat --tablespace-map-all regarding tablespace creation. (Reviewed by Stefan Fercot, Christophe Courtois. Suggested by Christophe Courtois.)
  • Clarify behavior of --repo-retention-full-type. (Reviewed by Antoine Beaupré. Suggested by Antoine Beaupré.)
  • Change --process-max recommendation for object stores to --repo-bundle. (Reviewed by Stefan Fercot.)
  • Update unix_socket_directory to unix_socket_directories. (Contributed by hyunkyu han. Reviewed by David Steele.)
  • Recommend not placing spool-path within pg_xlog/pg_wal. (Reviewed by Martín Marqués, Don Seiler. Suggested by Martín Marqués.)
v2.54.2 Release Notes
Bug Fix
Released January 20, 2025
Bug Fixes:
  • Fix issue after disabling bundling with block incremental enabled. (Reviewed by David Christensen.)
Documentation Improvements:
  • Clarify behavior of multiple configuration files. (Reviewed by Paul Bierly. Suggested by Paul Bierly.)
v2.54.1 Release Notes
Bug Fix
Released December 16, 2024
Bug Fixes:
  • Fix issue with version/help commands attempting to load pgbackrest.conf. (Reviewed by Stefan Fercot. Reported by Bradford Boyle, Julian.)
Test Suite Improvements:
  • Stabilize async archiving in integration tests. (Contributed by Viktor Kurilko. Reviewed by David Steele.)
v2.54.0 Release Notes
Target Time for Versioned Storage
Released October 21, 2024
NOTE TO PACKAGERS: This is last feature release to support the autoconf/make build. Please migrate to meson if you have not already done so. 2.54.X patch releases (if any) will continue to support autoconf/make.
Bug Fixes:
  • Fix PostgreSQL query performance for large datasets. (Fixed by Thibault Vincent, David Steele. Reviewed by David Christensen, Antoine Millet. Reported by Antoine Millet.)
Features:
  • Allow repositories on versioned storage to be read at a target time. (Reviewed by Stefan Fercot, David Christensen.)
  • Allow requested standby backup to proceed with no standby. (Reviewed by Stefan Fercot.)
Improvements:
  • Summarize backup reference list for info command text output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Refresh web-id token for each S3 authentication. (Contributed by Brent Graveland. Reviewed by David Steele.)
  • Correctly display current values for indexed options in help. (Reviewed by David Christensen.)
  • Save backup.info only when contents have changed. (Reviewed by Stefan Fercot.)
  • Remove limitation on reading files in parallel during restore. (Reviewed by David Christensen.)
  • Improve SFTP error messages. (Contributed by Reid Thompson. Reviewed by David Steele.)
Documentation Features:
  • Add performance tuning section to user guide. (Reviewed by Stefan Fercot.)
Documentation Improvements:
  • Clarify source for data_directory. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Matthias.)
  • Better logic for deciding when a summary should be lower-cased. (Suggested by Daniel Westermann.)
v2.53.1 Release Notes
PostgreSQL 17 Support
Released August 19, 2024
Bug Fixes:
  • Fix permissions when restore run as root user. (Reviewed by Stefan Fercot. Reported by Will M.)
  • Fix segfault on delayed connection errors. (Reviewed by David Christensen. Reported by Anton Glushakov.)
  • Skip local repository duplicate check for SFTP. (Fixed by Reid Thompson. Reviewed by David Steele. Reported by Anton Kurochkin.)
Improvements:
  • PostgreSQL 17 support.
v2.53 Release Notes
Concurrent Backups
Released July 22, 2024
IMPORTANT NOTE: The log-level-stderr option default has been changed from warn to off. This makes it easier to capture errors when only redirecting stdout. To preserve the prior behavior set log-level-stderr=warn.

NOTE TO PACKAGERS: The lz4 library is now required by the meson build.

NOTE TO PACKAGERS: Compiler support for __builtin_clzl() and __builtin_bswap64() is now required by the meson build.
Bug Fixes:
  • Fix SFTP renaming failure when file already exists. (Fixed by Reid Thompson. Reviewed by David Steele. Reported by ahmed112212.)
Features:
  • Allow backups to run concurrently on different repositories. (Reviewed by Reid Thompson, Stefan Fercot.)
  • Support IP-based SANs for TLS certificate validation. (Contributed by David Christensen. Reviewed by David Steele.)
Improvements:
  • Default log-level-stderr option to off. (Reviewed by Greg Sabino Mullane, Stefan Fercot.)
  • Allow alternative WAL segment sizes for PostgreSQL ≤ 10. (Contributed by Viktor Kurilko. Reviewed by David Steele.)
  • Add hint to check SFTP authorization log. (Contributed by Vitalii Zurian. Reviewed by Reid Thompson, David Steele.)
Documentation Improvements:
  • Clarify archive-push multi-repo behavior. (Reviewed by Stefan Fercot.)
v2.52.1 Release Notes
Bug Fix
Released June 25, 2024
Bug Fixes:
  • Fix issue with files larger on the replica than on the primary. (Reviewed by Stefan Fercot. Reported by Nicolas Lassimonne.)
v2.52 Release Notes
PostgreSQL 17beta1 Support
Released May 27, 2024
NOTE TO PACKAGERS: The build system for pgBackRest is now meson. The autoconf/make build will not receive any new features and will be removed after a few releases.
Features:
  • Add GCS batch delete support. (Reviewed by Reid Thompson.)
  • S3 SSE-C encryption support. (Reviewed by Tim Jones. Suggested by Tim Jones.)
  • PostgreSQL 17beta1 support. (Reviewed by Stefan Fercot.)
Improvements:
  • Allow explicit disabling of optional dependencies in meson builds. (Contributed by Michael Schout. Reviewed by David Steele.)
  • Dynamically find python in meson build. (Contributed by Michael Schout. Reviewed by David Steele.)
  • Tag pgbackrest build target in meson as installable. (Contributed by Bradford Boyle. Reviewed by David Steele.)
Documentation Improvements:
  • Update start/stop documentation to reflect actual functionality. (Reviewed by Stefan Fercot.)
v2.51 Release Notes
Meson Build System
Released March 25, 2024
Bug Fixes:
  • Skip zero-length files for block incremental delta restore. (Reviewed by Sebastian Krause, René Højbjerg Larsen. Reported by Sebastian Krause.)
  • Fix performance regression in storage list. (Reviewed by Stephen Frost. Reported by Maksym Boguk.)
  • Fix progress logging when file size changes during backup. (Reviewed by Stephen Frost. Reported by samkingno.)
Improvements:
  • Improved support for dual stack connections. (Reviewed by Stephen Frost. Suggested by Timothée Peignier.)
  • Make meson the primary build system. (Reviewed by Stephen Frost.)
  • Detect files that have not changed during non-delta incremental backup. (Reviewed by Stephen Frost.)
  • Prevent invalid recovery when backup_label removed. (Reviewed by Stephen Frost.)
  • Improve archive-push WAL segment queue handling. (Reviewed by Stephen Frost.)
  • Limit resume functionality to full backups. (Reviewed by Stephen Frost, Stefan Fercot.)
  • Update resume functionality for block incremental. (Reviewed by Stephen Frost.)
  • Allow --version and --help for version and help. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
  • Add detailed backtrace to autoconf/make build. (Reviewed by Stephen Frost.)
Documentation Improvements:
  • Update references to recovery.conf. (Reviewed by Stefan Fercot. Suggested by Stephen Frost.)
v2.50 Release Notes
Performance Improvements and Bug Fixes
Released January 22, 2024
Bug Fixes:
  • Fix short read in block incremental restore. (Reviewed by Stephen Frost, Brent Graveland. Reported by Adol Rodriguez, Brent Graveland.)
  • Fix overflow suppressing backup progress in info output. (Fixed by Robert Donovan. Reviewed by Joe Wildish.)
Improvements:
  • Preserve partial files during block incremental delta restore. (Reviewed by Stephen Frost.)
  • Add support for alternate compile-time page sizes. (Contributed by Viktor Kurilko. Reviewed by David Steele.)
  • Skip files truncated during backup when bundling. (Contributed by Georgy Shelkovy. Reviewed by David Steele.)
  • Improve SFTP storage error messages. (Contributed by Reid Thompson. Reviewed by David Steele.)
v2.49 Release Notes
Remove PostgreSQL 9.3 Support
Released November 27, 2023
Bug Fixes:
  • Fix regression in retries. (Reviewed by Stephen Frost. Reported by Norman Adkins, Tanel Suurhans, Jordan English, Timothée Peignier.)
  • Fix recursive path remove in SFTP storage driver. (Fixed by Reid Thompson. Reviewed by Stephen Frost. Reported by Luc.)
Improvements:
  • Remove support for PostgreSQL 9.3. (Reviewed by Stephen Frost.)
Documentation Features:
  • Document maintainer options. (Reviewed by Stefan Fercot.)
  • Update point-in-time recovery documentation for PostgreSQL >= 13.
Test Suite Improvements:
  • Allow config/load unit test to run without libssh2 installed. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Wu Ning.)
v2.48 Release Notes
Repository Storage Tags
Released September 25, 2023
Bug Fixes:
  • Fix issue restoring block incremental without a block list. (Reviewed by Stephen Frost, Burak Yurdakul. Reported by Burak Yurdakul.)
Features:
  • Add --repo-storage-tag option to create object tags. (Reviewed by Stephen Frost, Stefan Fercot, Timothée Peignier.)
  • Add known hosts checking for SFTP storage driver. (Contributed by Reid Thompson. Reviewed by Stephen Frost, David Steele.)
  • Support for dual stack connections. (Reviewed by Stephen Frost.)
  • Add backup size completed/total to info command JSON output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
Improvements:
  • Multi-stanza check command. (Reviewed by Stephen Frost.)
  • Retry reads of pg_control until checksum is valid. (Reviewed by Stefan Fercot, Stephen Frost.)
  • Optimize WAL segment check after successful backup. (Reviewed by Stephen Frost.)
  • Improve GCS multi-part performance. (Reviewed by Reid Thompson.)
  • Allow archive-get command to run when stanza is stopped. (Reviewed by Tom Swartz, David Christensen, Reid Thompson.)
  • Accept leading tilde in paths for SFTP public/private keys. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Reload GCS credentials before renewing authentication token. (Reviewed by Stephen Frost. Suggested by Daniel Farina.)
Documentation Bug Fixes:
  • Fix configuration reference example for the tls-server-address option. (Fixed by Hartmut Goebel. Reviewed by David Steele.)
  • Fix command reference example for the filter option.
Test Suite Improvements:
  • Allow storage/sftp unit test to run without libssh2 installed. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Wu Ning.)
v2.47 Release Notes
Performance Improvements and Bug Fixes
Released July 24, 2023
Bug Fixes:
  • Preserve block incremental info in manifest during delta backup. (Reviewed by Stephen Frost. Reported by Francisco Miguel Biete Banon.)
  • Fix block incremental file names in verify command. (Reviewed by Reid Thompson. Reported by Francisco Miguel Biete Banon.)
  • Fix spurious automatic delta backup on backup from standby. (Reviewed by Stephen Frost. Reported by krmozejko, Don Seiler.)
  • Skip recovery.signal for PostgreSQL >= 12 when recovery type=none. (Reviewed by Stefan Fercot. Reported by T.Anastacio.)
  • Fix unique label generation for diff/incr backup. (Fixed by Andrey Sokolov. Reviewed by David Steele.)
  • Fix time-based archive expiration when no backups are expired. (Reviewed by Stefan Fercot.)
Improvements:
  • Improve performance of SFTP storage driver. (Contributed by Stephen Frost, Reid Thompson. Reviewed by David Steele.)
  • Add timezone offset to info command date/time output. (Reviewed by Stefan Fercot, Philip Hurst. Suggested by Philip Hurst.)
  • Centralize error handling for unsupported features. (Reviewed by Stefan Fercot.)
Documentation Improvements:
  • Clarify preference to install from packages in the user guide. (Reviewed by Stefan Fercot. Suggested by dr-kd.)
v2.46 Release Notes
Block Incremental Backup and SFTP Storage
Released May 22, 2023
Features:
  • Block incremental backup. (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)
  • SFTP support for repository storage. (Contributed by Reid Thompson. Reviewed by Stephen Frost, David Steele.)
  • PostgreSQL 16 support. (Reviewed by Stefan Fercot.)
Improvements:
  • Allow page header checks to be skipped. (Reviewed by David Christensen. Suggested by David Christensen.)
  • Avoid chown() on recovery files during restore. (Reviewed by Stefan Fercot, Marcelo Henrique Neppel. Suggested by Marcelo Henrique Neppel.)
  • Add error retry detail for HTTP retries.
Documentation Improvements:
  • Add warning about using recovery type=none. (Reviewed by Stefan Fercot.)
  • Add note about running stanza-create on already-created repositories.
v2.45 Release Notes
Block Incremental Backup (BETA)
Released March 20, 2023
Bug Fixes:
  • Skip writing recovery.signal by default for restores of offline backups. (Reviewed by Stefan Fercot. Reported by Marcel Borger.)
Features:
  • Block incremental backup (BETA). (Reviewed by John Morris, Stephen Frost, Stefan Fercot.)
Improvements:
  • Keep only one all-default group index. (Reviewed by Stefan Fercot.)
Documentation Improvements:
  • Add explicit instructions for upgrading between 2.x versions. (Contributed by Christophe Courtois. Reviewed by David Steele.)
  • Remove references to SSH made obsolete when TLS was introduced.
v2.44 Release Notes
Remove PostgreSQL 9.0/9.1/9.2 Support
Released January 30, 2023
Improvements:
  • Remove support for PostgreSQL 9.0/9.1/9.2. (Reviewed by Stefan Fercot.)
  • Restore errors when no backup matches the current version of PostgreSQL. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Soulou.)
  • Add compress-level range checking for each compress-type. (Reviewed by Stefan Fercot. Suggested by gkleen, ViperRu.)
Documentation Improvements:
  • Add warning about enabling hierarchical namespace on Azure storage. (Reviewed by Stefan Fercot. Suggested by Vojtech Galda, Pluggi, asjonos.)
  • Add replacement for linefeeds in monitoring example. (Reviewed by Stefan Fercot. Suggested by rudonx, gmustdie, Ivan Shelestov.)
  • Clarify target-action behavior on various PostgreSQL versions. (Contributed by Chris Bandy. Reviewed by David Steele, Anton Kurochkin, Stefan Fercot. Suggested by Anton Kurochkin, Chris Bandy.)
  • Updates and clarifications to index page. (Reviewed by Stefan Fercot.)
  • Add dark mode to the website. (Suggested by Stephen Frost.)
v2.43 Release Notes
Bug Fix
Released November 28, 2022
Bug Fixes:
  • Fix missing reference in diff/incr backup. (Reviewed by Stefan Fercot. Reported by Marcel Borger, ulfedf, jaymefSO.)
Improvements:
  • Add hint when an option is specified without an index. (Reviewed by Stefan Fercot.)
v2.42 Release Notes
Bug Fixes
Released November 22, 2022
Bug Fixes:
  • Fix memory leak in file bundle backup/restore. (Reviewed by John Morris, Oscar. Reported by Oscar.)
  • Fix protocol error on short read of remote file. (Reviewed by Stephen Frost.)
Improvements:
  • Do not store references for zero-length files when bundling. (Reviewed by Stefan Fercot.)
  • Use more generic descriptions for pg_start_backup()/pg_stop_backup(). (Reviewed by Greg Sabino Mullane, David Christensen. Suggested by Greg Sabino Mullane.)
Test Suite Improvements:
  • Update test.pl --psql-bin option to match command-line help. (Contributed by Koshi Shibagaki. Reviewed by David Steele.)
v2.41 Release Notes
Backup Annotations
Released September 19, 2022
Bug Fixes:
  • Fix incorrect time expiration being used for non-default repositories. (Reviewed by Stefan Fercot. Reported by Adam Brusselback.)
  • Fix issue when listing directories recursively with a filter. (Reviewed by Stephen Frost. Reported by Efremov Egor.)
Features:
  • Backup key/value annotations. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Adam Berlin.)
Improvements:
  • Support --set in JSON output for info command. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Anton Kurochkin.)
  • Allow upload chunk size to be configured for object stores. (Reviewed by Stefan Fercot. Suggested by Anton Glushakov.)
  • Update archive.info timestamps after a successful backup. (Reviewed by Stefan Fercot. Suggested by Alex Richman.)
  • Move standby timeline check after checkpoint. (Reviewed by Stefan Fercot, Keith Fiske. Suggested by Keith Fiske.)
  • Improve warning message on backup resume. (Suggested by Cynthia Shang.)
Documentation Improvements:
  • Add absolute path for kill in pgbackrest.service. (Suggested by Don Seiler.)
v2.40 Release Notes
OpenSSL 3 Support
Released July 18, 2022
NOTE TO PACKAGERS: An experimental meson build has been added but packagers should continue to use the autoconf/make build for the foreseeable future.
Improvements:
  • OpenSSL 3 support. (Reviewed by Stephen Frost.)
  • Create snapshot when listing contents of a path. (Reviewed by John Morris, Stephen Frost.)
  • Force target-timeline=current when restore type=immediate. (Reviewed by Stephen Frost.)
  • Truncate files during delta restore when they are larger than expected. (Reviewed by Stephen Frost.)
  • Disable incremental manifest save when resume=n. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Set backup percent complete to zero before copy start. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Use S3 IsTruncated flag to determine list continuation. (Reviewed by John Morris, Soulou. Suggested by Christian Montagne.)
Documentation Bug Fixes:
  • Skip internal options in the configuration reference. (Reported by Francisco Miguel Biete Banon.)
Documentation Improvements:
  • Add link to PostgreSQL configuration in repository host section. (Reviewed by Stefan Fercot. Suggested by Julien Cigar.)
Test Suite Improvements:
  • Add experimental Meson build. (Reviewed by Eli Schwartz, Sam Bassaly.)
  • Allow any path to be passed to the --test-path option. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
  • Fix compile error when DEBUG_EXEC_TIME is defined without DEBUG. (Contributed by Andrey Sokolov. Reviewed by David Steele.)
v2.39 Release Notes
Verify and File Bundling
Released May 16, 2022
Bug Fixes:
  • Fix error thrown from FINALLY() causing an infinite loop. (Reviewed by Stephen Frost.)
  • Error on all lock failures except another process holding the lock. (Reviewed by Reid Thompson, Geir Råness. Reported by Geir Råness.)
Features:
  • Backup file bundling for improved small file support. (Reviewed by Reid Thompson, Stefan Fercot, Chris Bandy.)
  • Verify command to validate the contents of a repository. (Contributed by Cynthia Shang, Reid Thompson. Reviewed by David Steele, Stefan Fercot.)
  • PostgreSQL 15 support. (Reviewed by Stefan Fercot.)
  • Show backup percent complete in info output. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Auto-select backup for restore command --type=lsn. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele.)
  • Suppress existing WAL warning when archive-mode-check is disabled. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Add AWS IMDSv2 support. (Contributed by Nuno Pires. Reviewed by David Steele.)
Improvements:
  • Allow repo-hardlink option to be changed after full backup. (Reviewed by Reid Thompson.)
  • Increase precision of percent complete logging for backup and restore. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Improve path validation for repo-* commands. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Improve stop command to honor stanza option. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by ragaoua.)
  • Improve error message for invalid repo-azure-key. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Seth Daniel.)
  • Add hint to check the log on archive-get/archive-push async error. (Reviewed by Reid Thompson.)
  • Add ClockError for unexpected clock skew and timezone changes. (Reviewed by Greg Sabino Mullane, Stefan Fercot. Suggested by Greg Sabino Mullane.)
  • Strip extensions from history manifest before showing in error message. (Reviewed by Stefan Fercot.)
  • Add user:group to lock permission error. (Reviewed by Reid Thompson.)
Documentation Bug Fixes:
  • Fix incorrect reference to stanza-update in the user guide. (Fixed by Abubakar Mohammed. Reviewed by David Steele.)
  • Fix example for repo-gcs-key-type option in configuration reference. (Reviewed by Reid Thompson.)
  • Fix tls-server-auth example and add clarifications. (Reviewed by Reid Thompson.)
Documentation Improvements:
  • Simplify messaging around supported versions in the documentation. (Reviewed by Stefan Fercot, Reid Thompson, Greg Sabino Mullane.)
  • Add option type descriptions. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Add FAQ about backup types and restore speed. (Contributed by David Christensen. Reviewed by Reid Thompson.)
  • Document required base branch for pull requests. (Contributed by David Christensen. Reviewed by Reid Thompson.)
v2.38 Release Notes
Minor Bug Fixes and Improvements
Released March 6, 2022
IMPORTANT NOTE: Repository size reported by the info command is now entirely based on what pgBackRest has written to storage. Previously, in certain cases, pgBackRest could detect if additional compression was being applied by the storage but this is no longer supported.
Bug Fixes:
  • Retry errors in S3 batch file delete. (Reviewed by Reid Thompson. Reported by Alex Richman.)
  • Allow case-insensitive matching of HTTP connection header values. (Reviewed by Reid Thompson. Reported by Rémi Vidier.)
Features:
  • Add support for AWS S3 server-side encryption using KMS. (Contributed by Christoph Berg. Reviewed by David Steele, Tharindu Amila.)
  • Add archive-missing-retry option. (Reviewed by Stefan Fercot.)
  • Add backup type filter to info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)
Improvements:
  • Retry on page validation failure during backup. (Reviewed by Stephen Frost, David Christensen.)
  • Handle TLS servers that do not close connections gracefully. (Reviewed by Rémi Vidier, David Christensen, Stephen Frost.)
  • Add backup LSNs to info command output. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Automatically strip trailing slashes for repo-ls paths. (Contributed by David Christensen. Reviewed by David Steele.)
  • Do not retry fatal errors. (Reviewed by Reid Thompson.)
  • Remove support for PostgreSQL 8.3/8.4. (Reviewed by Reid Thompson, Stefan Fercot.)
  • Remove logic that tried to determine additional file system compression. (Reviewed by Reid Thompson, Stefan Fercot.)
Documentation Bug Fixes:
  • Move repo options in TLS documentation to the global section. (Reported by Anton Kurochkin.)
  • Remove unused backup-standby option from stanza commands. (Reported by Stefan Fercot.)
  • Fix typos in help and release notes. (Fixed by Daniel Gustafsson. Reviewed by David Steele.)
Documentation Improvements:
  • Add aliveness check to systemd service configuration. (Suggested by Yogesh Sharma.)
  • Add FAQ explaining WAL archive suffix. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Note that replications slots are not restored. (Contributed by Reid Thompson. Reviewed by David Steele, Stefan Fercot. Suggested by Christophe Courtois.)
v2.37 Release Notes
TLS Server
Released January 3, 2022
IMPORTANT NOTE: If the restore command is unable to find a backup that matches a specified time target then an error will be thrown, whereas before a warning was logged.
Bug Fixes:
  • Fix restore delta link mapping when path/file already exists. (Reviewed by Reid Thompson. Reported by Younes Alhroub.)
  • Fix socket leak on connection retries. (Reviewed by Reid Thompson. Reported by James Coleman.)
Features:
  • Add TLS server. (Reviewed by Stephen Frost, Reid Thompson, Andrew L'Ecuyer.)
  • Add --cmd option. (Contributed by Reid Thompson. Reviewed by Stefan Fercot, David Steele. Suggested by Virgile CREVON.)
Improvements:
  • Check archive immediately after backup start. (Reviewed by Reid Thompson, David Christensen.)
  • Add timeline and checkpoint checks to backup. (Reviewed by Stefan Fercot, Reid Thompson.)
  • Check that clusters are alive and correctly configured during a backup. (Reviewed by Stefan Fercot.)
  • Error when restore is unable to find a backup to match the time target. (Reviewed by Reid Thompson, Douglas J Hunley. Suggested by Douglas J Hunley.)
  • Parse protocol/port in S3/Azure endpoints. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Add warning when checkpoint_timeout exceeds db-timeout. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Add verb to HTTP error output. (Contributed by Christoph Berg. Reviewed by David Steele.)
  • Allow y/n arguments for boolean command-line options. (Contributed by Reid Thompson. Reviewed by David Steele.)
  • Make backup size logging exactly match info command output. (Contributed by Reid Thompson. Reviewed by David Steele. Suggested by Mahomed Hussein.)
Documentation Improvements:
  • Display size option default and allowed values with appropriate units. (Reviewed by Reid Thompson.)
  • Fix typos and improve documentation for the tablespace-map-all option. (Reviewed by Reid Thompson. Suggested by Reid Thompson.)
  • Remove obsolete statement about future multi-repository support. (Suggested by David Christensen.)
v2.36 Release Notes
Minor Bug Fixes and Improvements
Released November 1, 2021
Bug Fixes:
  • Allow global as a stanza prefix. (Reviewed by Stefan Fercot. Reported by Younes Alhroub.)
  • Fix segfault on invalid GCS key file. (Reviewed by Stephen Frost. Reported by Henrik Feldt.)
Improvements:
  • Allow link-map option to create new links. (Reviewed by Don Seiler, Stefan Fercot, Chris Bandy. Suggested by Don Seiler.)
  • Increase max index allowed for pg/repo options to 256. (Reviewed by Cynthia Shang.)
  • Add WebIdentity authentication for AWS S3. (Reviewed by James Callahan, Reid Thompson, Benjamin Blattberg, Andrew L'Ecuyer.)
  • Report backup file validation errors in backup.info. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Add recovery start time to online backup restore log. (Reviewed by Tom Swartz, Stefan Fercot. Suggested by Tom Swartz.)
  • Report original error and retries on local job failure. (Reviewed by Stefan Fercot.)
  • Rename page checksum error to error list in info text output. (Reviewed by Stefan Fercot.)
  • Add hints to standby replay timeout message. (Reviewed by Cynthia Shang, Stefan Fercot. Suggested by Leigh Downs.)
v2.35 Release Notes
Binary Protocol
Released August 23, 2021
IMPORTANT NOTE: The log level for copied files in the backup/restore commands has been changed to detail. This makes the info log level less noisy but if these messages are required then set the log level for the backup/restore commands to detail
Bug Fixes:
  • Detect errors in S3 multi-part upload finalize. (Reviewed by Cynthia Shang, Marco Montagna. Reported by Marco Montagna, Lev Kokotov, Anderson A. Mallmann.)
  • Fix detection of circular symlinks. (Reviewed by Stefan Fercot. Reported by Rohit Raveendran.)
  • Only pass selected repo options to the remote. (Reviewed by David Christensen, Cynthia Shang. Reported by Greg Sabino Mullane, David Christensen.)
Improvements:
  • Binary protocol. (Reviewed by Cynthia Shang.)
  • Automatically create data directory on restore. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Chris Bandy.)
  • Allow restore --type=lsn. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang. Suggested by James Coleman.)
  • Change level of backup/restore copied file logging to detail. (Reviewed by Stefan Fercot. Suggested by Jens Wilke.)
  • Loop while waiting for checkpoint LSN to reach replay LSN. (Contributed by Stefan Fercot. Reviewed by David Steele. Suggested by Fatih Mencutekin.)
  • Log backup file total and restore size/file total. (Reviewed by Cynthia Shang.)
Documentation Bug Fixes:
  • Fix incorrect host names in user guide. (Reviewed by Stefan Fercot. Reported by Greg Sabino Mullane.)
Documentation Improvements:
  • Update contributing documentation and add pull request template. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Rearrange backup documentation in user guide. (Reviewed by Cynthia Shang.)
  • Clarify restore --type behavior in command reference. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Fix documentation and comment typos. (Contributed by Eric Radman. Reviewed by David Steele.)
Test Suite Improvements:
  • Add check for test path inside repo path. (Reviewed by Greg Sabino Mullane. Suggested by Greg Sabino Mullane.)
  • Add CodeQL static code analysis. (Reviewed by Cynthia Shang.)
  • Update tests to use standard patterns. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.34 Release Notes
PostgreSQL 14 Support
Released June 7, 2021
Bug Fixes:
  • Fix issues with leftover spool files from a prior restore. (Reviewed by Cynthia Shang, Stefan Fercot, Floris van Nee. Reported by Floris van Nee.)
  • Fix issue when checking links for large numbers of tablespaces. (Reviewed by Cynthia Shang, Avinash Vallarapu. Reported by Avinash Vallarapu.)
  • Free no longer needed remotes so they do not timeout during restore. (Reviewed by Cynthia Shang. Reported by Francisco Miguel Biete Banon.)
  • Fix help when a valid option is invalid for the specified command. (Reviewed by Stefan Fercot. Reported by Cynthia Shang.)
Features:
  • Add PostgreSQL 14 support. (Reviewed by Cynthia Shang.)
  • Add automatic GCS authentication for GCE instances. (Reviewed by Jan Wieck, Daniel Farina.)
  • Add repo-retention-history option to expire backup history. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)
  • Add db-exclude option. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
Improvements:
  • Change archive expiration logging from detail to info level. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Remove stanza archive spool path on restore. (Reviewed by Cynthia Shang, Stefan Fercot.)
  • Do not write files atomically or sync paths during backup copy. (Reviewed by Stephen Frost, Stefan Fercot, Cynthia Shang.)
Documentation Improvements:
  • Update contributing documentation. (Contributed by Cynthia Shang. Reviewed by David Steele, Stefan Fercot.)
  • Consolidate RHEL/CentOS user guide into a single document. (Reviewed by Cynthia Shang.)
  • Clarify that repo-s3-role is not an ARN. (Contributed by Isaac Yuen. Reviewed by David Steele.)
v2.33 Release Notes
Multi-Repository and GCS Support
Released April 5, 2021
Bug Fixes:
  • Fix option warnings breaking async archive-get/archive-push. (Reviewed by Cynthia Shang. Reported by Lev Kokotov.)
  • Fix memory leak in backup during archive copy. (Reviewed by Cynthia Shang. Reported by Christian ROUX, Efremov Egor.)
  • Fix stack overflow in cipher passphrase generation. (Reviewed by Cynthia Shang. Reported by bsiara.)
  • Fix repo-ls / on S3 repositories. (Reviewed by Cynthia Shang. Reported by Lesovsky Alexey.)
Features:
  • Multiple repository support. (Contributed by Cynthia Shang, David Steele. Reviewed by Stefan Fercot, Stephen Frost.)
  • GCS support for repository storage. (Reviewed by Cynthia Shang, Daniel Farina.)
  • Add archive-header-check option. (Reviewed by Stephen Frost, Cynthia Shang. Suggested by Hans-Jürgen Schönig.)
Improvements:
  • Include recreated system databases during selective restore. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
  • Exclude content-length from S3 signed headers. (Reviewed by Cynthia Shang. Suggested by Brian P Bockelman.)
  • Consolidate less commonly used repository storage options. (Reviewed by Cynthia Shang.)
  • Allow custom config-path default with ./configure --with-configdir. (Contributed by Michael Schout. Reviewed by David Steele.)
  • Log archive copy during backup. (Reviewed by Cynthia Shang, Stefan Fercot.)
Documentation Improvements:
  • Update reference to include links to user guide examples. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Update selective restore documentation with caveats. (Reviewed by Cynthia Shang, Stefan Fercot.)
  • Add compress-type clarification to archive-copy documentation. (Reviewed by Cynthia Shang, Stefan Fercot.)
  • Add compress-level defaults per compress-type value. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Add note about required NFS settings being the same as PostgreSQL. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.32 Release Notes
Repository Commands
Released February 8, 2021
Bug Fixes:
  • Fix resume after partial delete of backup by prior resume. (Reviewed by Cynthia Shang. Reported by Tom Swartz.)
Features:
  • Add repo-ls command. (Reviewed by Cynthia Shang, Stefan Fercot.)
  • Add repo-get command. (Contributed by Stefan Fercot, David Steele. Reviewed by Cynthia Shang.)
  • Add archive-mode-check option. (Contributed by Stefan Fercot. Reviewed by David Steele, Michael Banck.)
Improvements:
  • Improve archive-get performance. (Reviewed by Cynthia Shang.)
Documentation Improvements:
  • Improve expire command documentation. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.31 Release Notes
Minor Bug Fixes and Improvements
Released December 7, 2020
Bug Fixes:
  • Allow [, #, and space as the first character in database names. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Jefferson Alexandre.)
  • Create standby.signal only on PostgreSQL 12 when restore type is standby. (Fixed by Stefan Fercot. Reviewed by David Steele. Reported by Keith Fiske.)
Features:
  • Expire history files. (Contributed by Stefan Fercot. Reviewed by David Steele.)
  • Report page checksum errors in info command text output. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang.)
  • Add repo-azure-endpoint option. (Reviewed by Cynthia Shang, Brian Peterson. Suggested by Brian Peterson.)
  • Add pg-database option. (Reviewed by Cynthia Shang.)
Improvements:
  • Improve info command output when a stanza is specified but missing. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele. Suggested by uspen.)
  • Improve performance of large file lists in backup/restore commands. (Reviewed by Cynthia Shang, Oscar.)
  • Add retries to PostgreSQL sleep when starting a backup. (Reviewed by Cynthia Shang. Suggested by Vitaliy Kukharik.)
Documentation Improvements:
  • Replace RHEL/CentOS 6 documentation with RHEL/CentOS 8.
v2.30 Release Notes
PostgreSQL 13 Support
Released October 5, 2020
Bug Fixes:
  • Error with hints when backup user cannot read pg_settings. (Reviewed by Stefan Fercot, Cynthia Shang. Reported by Mohamed Insaf K.)
Features:
  • PostgreSQL 13 support. (Reviewed by Cynthia Shang.)
Improvements:
  • Improve PostgreSQL version identification. (Reviewed by Cynthia Shang, Stephen Frost.)
  • Improve working directory error message. (Reviewed by Stefan Fercot.)
  • Add hint about starting the stanza when WAL segment not found. (Contributed by David Christensen. Reviewed by David Steele.)
  • Add hint for protocol version mismatch. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)
Documentation Improvements:
  • Add note that pgBackRest versions must match when running remotely. (Reviewed by Cynthia Shang. Suggested by loop-evgeny.)
  • Move info command text to the reference and link to user guide. (Reviewed by Cynthia Shang. Suggested by Christophe Courtois.)
  • Update yum repository path for CentOS/RHEL user guide. (Contributed by Heath Lord. Reviewed by David Steele.)
v2.29 Release Notes
Auto S3 Credentials on AWS
Released August 31, 2020
Bug Fixes:
  • Suppress errors when closing local/remote processes. Since the command has completed it is counterproductive to throw an error but still warn to indicate that something unusual happened. (Reviewed by Cynthia Shang. Reported by argdenis.)
  • Fix issue with = character in file or database names. (Reviewed by Bastian Wegge, Cynthia Shang. Reported by Brad Nicholson, Bastian Wegge.)
Features:
  • Automatically retrieve temporary S3 credentials on AWS instances. (Contributed by David Steele, Stephen Frost. Reviewed by Cynthia Shang, David Youatt, Aleš Zelený, Jeanette Bromage.)
  • Add archive-mode option to disable archiving on restore. (Reviewed by Stephen Frost. Suggested by Stephen Frost.)
Improvements:
  • PostgreSQL 13 beta3 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
  • Asynchronous list/remove for S3/Azure storage. (Reviewed by Cynthia Shang, Stephen Frost.)
  • Improve memory usage of unlogged relation detection in manifest build. (Reviewed by Cynthia Shang, Stephen Frost, Brad Nicholson, Oscar. Suggested by Oscar, Brad Nicholson.)
  • Proactively close file descriptors after forking async process. (Reviewed by Stephen Frost, Cynthia Shang.)
  • Delay backup remote connection close until after archive check. (Contributed by Floris van Nee. Reviewed by David Steele.)
  • Improve detailed error output. (Reviewed by Cynthia Shang.)
  • Improve TLS error reporting. (Reviewed by Cynthia Shang, Stephen Frost.)
Documentation Bug Fixes:
  • Add none to compress-type option reference and fix example. (Reported by Ugo Bellavance, Don Seiler.)
  • Add missing azure type in repo-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)
  • Fix typo in repo-cipher-type option reference. (Fixed by Don Seiler. Reviewed by David Steele.)
Documentation Improvements:
  • Clarify that expire must be run regularly when expire-auto is disabled. (Reviewed by Douglas J Hunley. Suggested by Douglas J Hunley.)
v2.28 Release Notes
Azure Repository Storage
Released July 20, 2020
Bug Fixes:
  • Fix restore --force acting like --force --delta. This caused restore to replace files based on timestamp and size rather than overwriting, which meant some files that should have been updated were left unchanged. Normal restore and restore --delta were not affected by this issue. (Reviewed by Cynthia Shang.)
Features:
  • Azure support for repository storage. (Reviewed by Cynthia Shang, Don Seiler.)
  • Add expire-auto option. This allows automatic expiration after a successful backup to be disabled. (Contributed by Stefan Fercot. Reviewed by Cynthia Shang, David Steele.)
Improvements:
  • Asynchronous S3 multipart upload. (Reviewed by Stephen Frost.)
  • Automatic retry for backup, restore, archive-get, and archive-push. (Reviewed by Cynthia Shang.)
  • Disable query parallelism in PostgreSQL sessions used for backup control. (Reviewed by Stefan Fercot.)
  • PostgreSQL 13 beta2 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
  • Improve handling of invalid HTTP response status. (Reviewed by Cynthia Shang.)
  • Improve error when pg1-path option missing for archive-get command. (Reviewed by Cynthia Shang.)
  • Add hint when checksum delta is enabled after a timeline switch. (Reviewed by Matt Bunter, Cynthia Shang.)
  • Use PostgreSQL instead of postmaster where appropriate. (Reviewed by Cynthia Shang.)
Documentation Bug Fixes:
  • Fix incorrect example for repo-retention-full-type option. (Reported by Höseyin Sönmez.)
  • Remove internal commands from HTML and man command references. (Reported by Cynthia Shang.)
Documentation Improvements:
  • Update PostgreSQL versions used to build user guides. Also add version ranges to indicate that a user guide is accurate for a range of PostgreSQL versions even if it was built for a specific version. (Reviewed by Stephen Frost.)
  • Update FAQ for expiring a specific backup set. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Update FAQ to clarify default PITR behavior. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.27 Release Notes
Expiration Improvements and Compression Drivers
Released May 26, 2020
Bug Fixes:
  • Fix issue checking if file links are contained in path links. (Reviewed by Cynthia Shang. Reported by Christophe Cavallié.)
  • Allow pg-path1 to be optional for synchronous archive-push. (Reviewed by Cynthia Shang. Reported by Jerome Peng.)
  • The expire command now checks if a stop file is present. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Handle missing reason phrase in HTTP response. (Reviewed by Cynthia Shang. Reported by Tenuun.)
  • Increase buffer size for lz4 compression flush. (Reviewed by Cynthia Shang. Reported by Eric Radman.)
  • Ignore pg-host* and repo-host* options for the remote command. (Reviewed by Cynthia Shang. Reported by Pavel Suderevsky.)
  • Fix possibly missing pg1-* options for the remote command. (Reviewed by Cynthia Shang. Reported by Andrew L'Ecuyer.)
Features:
  • Time-based retention for full backups. The --repo-retention-full-type option allows retention of full backups based on a time period, specified in days. (Contributed by Cynthia Shang, Pierre Ducroquet. Reviewed by David Steele.)
  • Ad hoc backup expiration. Allow the user to remove a specified backup regardless of retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Zstandard compression support. Note that setting compress-type=zst will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
  • bzip2 compression support. Note that setting compress-type=bz2 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Contributed by Stephen Frost. Reviewed by David Steele, Cynthia Shang.)
  • Add backup/expire running status to the info command. (Contributed by Stefan Fercot. Reviewed by David Steele.)
Improvements:
  • Expire WAL archive only when repo-retention-archive threshold is met. WAL prior to the first full backup was previously expired after the first full backup. Now it is preserved according to retention settings. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Add local MD5 implementation so S3 works when FIPS is enabled. (Reviewed by Cynthia Shang, Stephen Frost. Suggested by Brian Almeida, John Kelly.)
  • PostgreSQL 13 beta1 support. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace. (Reviewed by Cynthia Shang.)
  • Reduce buffer-size default to 1MiB. (Reviewed by Stephen Frost.)
  • Throw user-friendly error if expire is not run on repository host. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.26 Release Notes
Non-blocking TLS
Released April 20, 2020
Bug Fixes:
  • Remove empty subexpression from manifest regular expression. MacOS was not happy about this though other platforms seemed to work fine. (Fixed by David Raftis. Reviewed by David Steele.)
Improvements:
  • Non-blocking TLS implementation. (Reviewed by Slava Moudry, Cynthia Shang, Stephen Frost.)
  • Only limit backup copy size for WAL-logged files. The prior behavior could possibly lead to postgresql.conf or postgresql.auto.conf being truncated in the backup. (Reviewed by Cynthia Shang.)
  • TCP keep-alive options are configurable. (Suggested by Marc Cousin.)
  • Add io-timeout option. (Reviewed by Cynthia Shang.)
v2.25 Release Notes
LZ4 Compression Support
Released March 26, 2020
Features:
  • Add lz4 compression support. Note that setting compress-type=lz4 will make new backups and archive incompatible (unrestorable) with prior versions of pgBackRest. (Reviewed by Cynthia Shang.)
  • Add --dry-run option to the expire command. Use dry-run to see which backups/archive would be removed by the expire command without actually removing anything. (Contributed by Cynthia Shang, Luca Ferrari. Reviewed by David Steele. Suggested by Marc Cousin.)
Improvements:
  • Improve performance of remote manifest build. (Suggested by Jens Wilke.)
  • Fix detection of keepalive options on Linux. (Contributed by Marc Cousin. Reviewed by David Steele.)
  • Add configure host detection to set standards flags correctly. (Contributed by Marc Cousin. Reviewed by David Steele.)
  • Remove compress/compress-level options from commands where unused. These commands (e.g. restore, archive-get) never used the compress options but allowed them to be passed on the command line. Now they will error when these options are passed on the command line. If these errors occur then remove the unused options. (Reviewed by Cynthia Shang.)
  • Limit backup file copy size to size reported at backup start. If a file grows during the backup it will be reconstructed by WAL replay during recovery so there is no need to copy the additional data. (Reviewed by Cynthia Shang.)
v2.24 Release Notes
Auto-Select Backup Set for Time Target
Released February 25, 2020
Bug Fixes:
  • Prevent defunct processes in asynchronous archive commands. (Reviewed by Stephen Frost. Reported by Adam Brusselback, ejberdecia.)
  • Error when archive-get/archive-push/restore are not run on a PostgreSQL host. (Reviewed by Stephen Frost. Reported by Jesper St John.)
  • Read HTTP content to eof when size/encoding not specified. (Reviewed by Cynthia Shang. Reported by Christian ROUX.)
  • Fix resume when the resumable backup was created by Perl. In this case the resumable backup should be ignored, but the C code was not able to load the partial manifest written by Perl since the format differs slightly. Add validations to catch this case and continue gracefully. (Reported by Kacey Holston.)
Features:
  • Auto-select backup set on restore when time target is specified. Auto-selection is performed only when --set is not specified. If a backup set for the given target time cannot not be found, the latest (default) backup set will be used. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Skip pg_internal.init temp file during backup. (Reviewed by Cynthia Shang. Suggested by Michael Paquier.)
  • Add more validations to the manifest on backup. (Reviewed by Cynthia Shang.)
Documentation Improvements:
  • Prevent lock-bot from adding comments to locked issues. (Suggested by Christoph Berg.)
v2.23 Release Notes
Bug Fix
Released January 27, 2020
Bug Fixes:
  • Fix missing files corrupting the manifest. If a file was removed by PostgreSQL during the backup (or was missing from the standby) then the next file might not be copied and updated in the manifest. If this happened then the backup would error when restored. (Reviewed by Cynthia Shang. Reported by Vitaliy Kukharik.)
Improvements:
  • Use pkg-config instead of xml2-config for libxml2 build options. (Contributed by David Steele, Adrian Vondendriesch.)
  • Validate checksums are set in the manifest on backup/restore. (Reviewed by Cynthia Shang.)
v2.22 Release Notes
Bug Fix
Released January 21, 2020
Bug Fixes:
  • Fix error in timeline conversion. The timeline is required to verify WAL segments in the archive after a backup. The conversion was performed base 10 instead of 16, which led to errors when the timeline was ≥ 0xA. (Reported by Lukas Ertl, Eric Veldhuyzen.)
v2.21 Release Notes
C Migration Complete
Released January 15, 2020
Bug Fixes:
  • Fix options being ignored by asynchronous commands. The asynchronous archive-get/archive-push processes were not loading options configured in command configuration sections, e.g. [global:archive-get]. (Reviewed by Cynthia Shang. Reported by Urs Kramer.)
  • Fix handling of \ in filenames. \ was not being properly escaped when calculating the manifest checksum which prevented the manifest from loading. Since instances of \ in cluster filenames should be rare to nonexistent this does not seem likely to be a serious problem in the field.
Features:
  • pgBackRest is now pure C.
  • Add pg-user option. Specifies the database user name when connecting to PostgreSQL. If not specified pgBackRest will connect with the local OS user or PGUSER, which was the previous behavior. (Contributed by Mike Palmiotto. Reviewed by David Steele.)
  • Allow path-style URIs in S3 driver.
Improvements:
  • The backup command is implemented entirely in C. (Reviewed by Cynthia Shang.)
v2.20 Release Notes
Bug Fixes
Released December 12, 2019
Bug Fixes:
  • Fix archive-push/archive-get when PGDATA is symlinked. These commands tried to use cwd() as PGDATA but this would disagree with the path configured in pgBackRest if PGDATA was symlinked. If cwd() does not match the pgBackRest path then chdir() to the path and make sure the next cwd() matches the result from the first call. (Reported by Stephen Frost, Milosz Suchy.)
  • Fix reference list when backup.info is reconstructed in expire command. Since the backup command is still using the Perl version of reconstruct this issue will not express unless 1) there is a backup missing from backup.info and 2) the expire command is run directly instead of running after backup as usual. This unlikely combination of events means this is probably not a problem in the field.
  • Fix segfault on unexpected EOF in gzip decompression. (Reported by Stephen Frost.)
v2.19 Release Notes
C Migrations and Bug Fixes
Released November 12, 2019
Bug Fixes:
  • Fix remote timeout in delta restore. When performing a delta restore on a largely unchanged cluster the remote could timeout if no files were fetched from the repository within protocol-timeout. Add keep-alives to prevent remote timeout. (Reported by James Sewell, Jens Wilke.)
  • Fix handling of repeated HTTP headers. When HTTP headers are repeated they should be considered equivalent to a single comma-separated header rather than generating an error, which was the prior behavior. (Reported by donicrosby.)
Improvements:
  • JSON output from the info command is no longer pretty-printed. Monitoring systems can more easily ingest the JSON without linefeeds. External tools such as jq can be used to pretty-print if desired. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • The check command is implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Document how to contribute to pgBackRest. (Contributed by Cynthia Shang, David Steele.)
  • Document maximum version for auto-stop option. (Contributed by Brad Nicholson. Reviewed by David Steele.)
Test Suite Improvements:
  • Fix container test path being used when --vm=none. (Suggested by Stephen Frost.)
  • Fix mismatched timezone in expect test. (Suggested by Stephen Frost.)
  • Don't autogenerate embedded libc code by default. (Suggested by Stephen Frost.)
v2.18 Release Notes
PostgreSQL 12 Support
Released October 1, 2019
Features:
  • PostgreSQL 12 support.
  • Add info command set option for detailed text output. The additional details include databases that can be used for selective restore and a list of tablespaces and symlinks with their default destinations. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Stephen Frost, ejberdecia.)
  • Add standby restore type. This restore type automatically adds standby_mode=on to recovery.conf for PostgreSQL < 12 and creates standby.signal for PostgreSQL ≥ 12, creating a common interface between PostgreSQL versions. (Reviewed by Cynthia Shang.)
Improvements:
  • The restore command is implemented entirely in C. (Reviewed by Cynthia Shang.)
Documentation Improvements:
  • Document the relationship between db-timeout and protocol-timeout. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by James Chanco Jr.)
  • Add documentation clarifications regarding standby repositories. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Add FAQ for time-based Point-in-Time Recovery. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v2.17 Release Notes
C Migrations and Bug Fixes
Released September 3, 2019
Bug Fixes:
  • Improve slow manifest build for very large quantities of tables/segments. (Reported by Jens Wilke.)
  • Fix exclusions for special files. (Reported by CluelessTechnologist, Janis Puris, Rachid Broum.)
Improvements:
  • The stanza-create/update/delete commands are implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • The start/stop commands are implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Create log directories/files with 0750/0640 mode. (Suggested by Damiano Albani.)
Documentation Bug Fixes:
  • Fix yum.p.o package being installed when custom package specified. (Reported by Joe Ayers, John Harvey.)
Documentation Improvements:
  • Build pgBackRest as an unprivileged user. (Suggested by Laurenz Albe.)
v2.16 Release Notes
C Migrations and Bug Fixes
Released August 5, 2019
Bug Fixes:
  • Retry S3 RequestTimeTooSkewed errors instead of immediately terminating. (Reported by sean0101n, Tim Garton, Jesper St John, Aleš Zelený.)
  • Fix incorrect handling of transfer-encoding response to HEAD request. (Reported by Pavel Suderevsky.)
  • Fix scoping violations exposed by optimizations in gcc 9. (Reported by Christian Lange, Ned T. Crigler.)
Features:
  • Add repo-s3-port option for setting a non-standard S3 service port.
Improvements:
  • The local command for backup is implemented entirely in C. (Contributed by David Steele, Cynthia Shang.)
  • The check command is implemented partly in C. (Reviewed by Cynthia Shang.)
v2.15 Release Notes
C Implementation of Expire
Released June 25, 2019
Bug Fixes:
  • Fix archive retention expiring too aggressively. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Mohamad El-Rifai.)
Improvements:
  • The expire command is implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • The local command for restore is implemented entirely in C.
  • Remove hard-coded PostgreSQL user so $PGUSER works. (Suggested by Julian Zhang, Janis Puris.)
  • Honor configure --prefix option. (Suggested by Daniel Westermann.)
  • Rename repo-s3-verify-ssl option to repo-s3-verify-tls. The new name is preferred because pgBackRest does not support any SSL protocol versions (they are all considered to be insecure). The old name will continue to be accepted.
Documentation Improvements:
  • Add FAQ to the documentation. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Use wal_level=replica in the documentation for PostgreSQL ≥ 9.6. (Suggested by Patrick McLaughlin.)
v2.14 Release Notes
Bug Fix and Improvements
Released May 20, 2019
Bug Fixes:
  • Fix segfault when process-max > 8 for archive-push/archive-get. (Reported by Jens Wilke.)
Improvements:
  • Bypass database checks when stanza-delete issued with force. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by hatifnatt.)
  • Add configure script for improved multi-platform support.
Documentation Features:
  • Add user guides for CentOS/RHEL 6/7.
v2.13 Release Notes
Bug Fixes
Released April 18, 2019
Bug Fixes:
  • Fix zero-length reads causing problems for IO filters that did not expect them. (Reported by brunre01, Jens Wilke, Tomasz Kontusz, guruguruguru.)
  • Fix reliability of error reporting from local/remote processes.
  • Fix Posix/CIFS error messages reporting the wrong filename on write/sync/close.
v2.12 Release Notes
C Implementation of Archive Push
Released April 11, 2019
IMPORTANT NOTE: The new TLS/SSL implementation forbids dots in S3 bucket names per RFC-2818. This security fix is required for compliant hostname verification.
Bug Fixes:
  • Fix issues when a path option is / terminated. (Reported by Marc Cousin.)
  • Fix issues when log-level-file=off is set for the archive-get command. (Reported by Brad Nicholson.)
  • Fix C code to recognize host:port option format like Perl does. (Reported by Kyle Nevins.)
  • Fix issues with remote/local command logging options.
Improvements:
  • The archive-push command is implemented entirely in C.
  • Increase process-max limit to 999. (Suggested by Rakshitha-BR.)
  • Improve error message when an S3 bucket name contains dots.
Documentation Improvements:
  • Clarify that S3-compatible object stores are supported. (Suggested by Magnus Hagander.)
v2.11 Release Notes
C Implementation of Archive Get
Released March 11, 2019
Bug Fixes:
  • Fix possible truncated WAL segments when an error occurs mid-write. (Reported by blogh.)
  • Fix info command missing WAL min/max when stanza specified. (Fixed by Stefan Fercot. Reviewed by David Steele.)
  • Fix non-compliant JSON for options passed from C to Perl. (Reported by Leo Khomenko.)
Improvements:
  • The archive-get command is implemented entirely in C.
  • Enable socket keep-alive on older Perl versions. (Contributed by Marc Cousin. Reviewed by David Steele.)
  • Error when parameters are passed to a command that does not accept parameters. (Suggested by Jason O'Donnell.)
  • Add hints when unable to find a WAL segment in the archive. (Suggested by Hans-Jürgen Schönig.)
  • Improve error when hostname cannot be found in a certificate. (Suggested by James Badger.)
  • Add additional options to backup.manifest for debugging purposes. (Contributed by blogh. Reviewed by David Steele.)
Documentation Improvements:
  • Update default documentation version to PostgreSQL 10.
v2.10 Release Notes
Bug Fixes
Released February 9, 2019
Bug Fixes:
  • Add unimplemented S3 driver method required for archive-get. (Reported by mibiio.)
  • Fix check for improperly configured pg-path. (Reported by James Chanco Jr.)
v2.09 Release Notes
Minor Improvements and Bug Fixes
Released January 30, 2019
Bug Fixes:
  • Fix issue with multiple async status files causing a hard error. (Reported by Vidhya Gurumoorthi, Joe Ayers, Douglas J Hunley.)
Improvements:
  • The info command is implemented entirely in C. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Simplify info command text message when no stanzas are present. Replace the repository path with the repository.
  • Add _DARWIN_C_SOURCE flag to Makefile for MacOS builds. (Contributed by Douglas J Hunley. Reviewed by David Steele.)
  • Update address lookup in C TLS client to use modern methods. (Suggested by Bruno Friedmann.)
  • Include Posix-compliant header for strcasecmp() and fd_set. (Suggested by ucando.)
Documentation Bug Fixes:
  • Fix hard-coded repository path. (Reported by Heath Lord.)
Documentation Improvements:
  • Clarify that encryption is always performed client-side. (Suggested by Bruce Burdick.)
  • Add examples for building a documentation host.
  • Allow if in manifest variables, lists, and list items.
v2.08 Release Notes
Minor Improvements and Bug Fixes
Released January 2, 2019
Bug Fixes:
  • Remove request for S3 object info directly after putting it. (Reported by Matt Kunkel.)
  • Correct archive-get-queue-max to be size type. (Reported by Ronan Dunklau.)
  • Add error message when current user uid/gid does not map to a name. (Reported by Camilo Aguilar.)
  • Error when --target-action=shutdown specified for PostgreSQL < 9.5.
Improvements:
  • Set TCP keepalives on S3 connections. (Suggested by Ronan Dunklau.)
  • Reorder info command text output so most recent backup is output last. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Ryan Lambert.)
  • Change file ownership only when required.
  • Redact authentication header when throwing S3 errors. (Suggested by Brad Nicholson.)
Documentation Improvements:
  • Clarify when target-action is effective and PostgreSQL version support. (Suggested by Keith Fiske.)
  • Clarify that region/endpoint must be configured correctly for the bucket. (Suggested by Pritam Barhate.)
  • Add documentation for building the documentation.
v2.07 Release Notes
Automatic Backup Checksum Delta
Released November 16, 2018
Bug Fixes:
  • Fix issue with archive-push-queue-max not being honored on connection error. (Reported by Lardière Sébastien.)
  • Fix static WAL segment size used to determine if archive-push-queue-max has been exceeded.
  • Fix error after log file open failure when processing should continue. (Reported by vthriller.)
Features:
  • Automatically enable backup checksum delta when anomalies (e.g. timeline switch) are detected. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Retry all S3 5xx errors rather than just 500 internal errors. (Suggested by Craig A. James.)
v2.06 Release Notes
Checksum Delta Backup and PostgreSQL 11 Support
Released October 15, 2018
Bug Fixes:
  • Fix missing URI encoding in S3 driver. (Reported by Dan Farrell.)
  • Fix incorrect error message for duplicate options in configuration files. (Reported by Jesper St John.)
  • Fix incorrectly reported error return in info logging. A return code of 1 from the archive-get was being logged as an error message at info level but otherwise worked correctly.
Features:
  • Add checksum delta for incremental backups. Checksum delta backups uses checksums rather than timestamps to determine if files have changed. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • PostgreSQL 11 support, including configurable WAL segment size.
Improvements:
  • Ignore all files in a linked tablespace directory except the subdirectory for the current version of PostgreSQL. Previously an error would be generated if other files were present and not owned by the PostgreSQL user.
  • Improve info command to display the stanza cipher type. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Douglas J Hunley.)
  • Improve support for special characters in filenames.
  • Allow delta option to be specified in the pgBackRest configuration file. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Use command in authorized_hosts to improve SSH security. (Suggested by Stephen Frost, Magnus Hagander.)
  • List allowable values for the buffer-size option in the configuration reference. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Stéphane Schildknecht.)
v2.05 Release Notes
Environment Variable Options and Exclude Temporary/Unlogged Relations
Released August 31, 2018
Bug Fixes:
  • Fix issue where relative links in $PGDATA could be stored in the backup with the wrong path. This issue did not affect absolute links and relative tablespace links were caught by other checks. (Reported by Cynthia Shang.)
  • Remove incompletely implemented online option from the check command. Offline operation runs counter to the purpose of this command, which is to check if archiving and backups are working correctly. (Reported by Jason O'Donnell.)
  • Fix issue where errors raised in C were not logged when called from Perl. pgBackRest properly terminated with the correct error code but lacked an error message to aid in debugging. (Reported by Douglas J Hunley.)
  • Fix issue when a boolean option (e.g. delta) was specified more than once. (Reported by Yogesh Sharma.)
Features:
  • Allow any option to be set in an environment variable. This includes options that previously could only be specified on the command line, e.g. stanza, and secret options that could not be specified on the command-line, e.g. repo1-s3-key-secret.
  • Exclude temporary and unlogged relation (table/index) files from backup. Implemented using the same logic as the patches adding this feature to PostgreSQL, 8694cc96 and 920a5e50. Temporary relation exclusion is enabled in PostgreSQL ≥ 9.0. Unlogged relation exclusion is enabled in PostgreSQL ≥ 9.1, where the feature was introduced. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Allow arbitrary directories and/or files to be excluded from a backup. Misuse of this feature can lead to inconsistent backups so read the --exclude documentation carefully before using. (Reviewed by Cynthia Shang.)
  • Add log-subprocess option to allow file logging for local and remote subprocesses.
  • PostgreSQL 11 Beta 3 support.
Improvements:
  • Allow zero-size files in backup manifest to reference a prior manifest regardless of timestamp delta. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Improve asynchronous archive-get/archive-push performance by directly checking status files. (Contributed by Stephen Frost. Reviewed by David Steele.)
  • Improve error message when a command is missing the stanza option. (Suggested by Sarah Conway.)
Documentation Bug Fixes:
  • Fix invalid log level in log-path option reference. (Reported by Camilo Aguilar.)
Documentation Improvements:
  • Stop trying to arrange contributors in release.xml by last/first name. Contributor names have always been presented in the release notes exactly as given, but we tried to assign internal IDs based on last/first name which can be hard to determine and ultimately doesn't make sense. Inspired by Christophe's PostgresOpen 2017 talk, Human Beings Do Not Have a Primary Key. (Suggested by Christophe Pettus.)
Test Suite Improvements:
  • Error if LibC build is performed outside the test environment. LibC is no longer required for production builds.
v2.04 Release Notes
Critical Bug Fix for Backup Resume
Released July 5, 2018
IMPORTANT NOTE: This release fixes a critical bug in the backup resume feature. All resumed backups prior to this release should be considered inconsistent. A backup will be resumed after a prior backup fails, unless resume=n has been specified. A resumed backup can be identified by checking the backup log for the message aborted backup of same type exists, will be cleaned to remove invalid files and resumed. If the message exists, do not use this backup or any backup in the same set for a restore and check the restore logs to see if a resumed backup was restored. If so, there may be inconsistent data in the cluster.
Bug Fixes:
  • Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. See note above for more details. (Reported by David Youatt, Yogesh Sharma, Stephen Frost.)
  • Fix error in selective restore when only one user database exists in the cluster. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Nj Baliyan.)
  • Fix non-compliant ISO-8601 timestamp format in S3 authorization headers. AWS and some gateways were tolerant of space rather than zero-padded hours while others were not. (Fixed by Andrew Schwartz. Reviewed by David Steele.)
Features:
  • PostgreSQL 11 Beta 2 support.
Improvements:
  • Improve the HTTP client to set content-length to 0 when not specified by the server. S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either. (Suggested by Adam K. Sumner.)
  • Set search_path = 'pg_catalog' on PostgreSQL connections. (Suggested by Stephen Frost.)
Documentation Improvements:
  • Create a new section to describe building pgBackRest and build on a separate host.
  • Add sample S3 policy to restrict bucket privileges. (Suggested by Douglas J Hunley, Jason O'Donnell.)
v2.03 Release Notes
Single Executable to Deploy
Released May 22, 2018
Bug Fixes:
  • Fix potential buffer overrun in error message handling. (Reported by Lætitia.)
  • Fix archive write lock being taken for the synchronous archive-get command. (Reported by uspen.)
Improvements:
  • Embed exported C functions and Perl modules directly into the pgBackRest executable.
  • Use time_t instead of __time_t for better portability. (Suggested by Nick Floersch.)
  • Print total runtime in milliseconds at command end.
v2.02 Release Notes
Parallel Asynchronous Archive Get and Configuration Includes
Released May 6, 2018
Bug Fixes:
  • Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
  • Fix archive-copy throwing path not found error for incr/diff backups. (Reported by yummyliu, Vitaliy Kukharik.)
  • Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. (Reported by Vitaliy Kukharik.)
  • Fix delta restore failing when a linked file is missing.
  • Fix rendering of key/value and list options in help. (Reported by Clinton Adams.)
Features:
  • Add asynchronous, parallel archive-get. This feature maintains a queue of WAL segments to help reduce latency when PostgreSQL requests a WAL segment with restore_command.
  • Add support for additional pgBackRest configuration files. The directory is specified by the --config-include-path option. Add --config-path option for overriding the default base path of the --config and --config-include-path option. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Add repo-s3-token option to allow temporary credentials tokens to be configured. pgBackRest currently has no way to request new credentials so the entire command (e.g. backup, restore) must complete before the credentials expire. (Contributed by Yogesh Sharma. Reviewed by David Steele.)
Improvements:
  • Update the archive-push-queue-max, manifest-save-threshold, and buffer-size options to accept values in KB, MB, GB, TB, or PB where the multiplier is a power of 1024. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path.
  • Show command parameters as well as command options in initial info log message.
  • Rename archive-queue-max option to archive-push-queue-max. This is consistent with the new archive-get-queue-max option. The old option name will continue to be accepted.
Documentation Bug Fixes:
  • Update docs with 32-bit support and caveats. 32-bit support was added in v1.26. (Reported by Viorel Tabara.)
Documentation Improvements:
  • Add monitoring examples using PostgreSQL and jq. (Suggested by Stephen Frost, Brian Faherty.)
  • Add example of command section usage to archiving configuration. (Suggested by Christophe Courtois.)
  • Remove documentation describing info --output=json as experimental.
  • Update out-of-date description for the spool-path option.
Test Suite Features:
  • Use lcov for C unit test coverage reporting. Switch from Devel::Cover because it would not report on branch coverage for reports converted from gcov. Incomplete branch coverage for a module now generates an error. Coverage of unit tests is not displayed in the report unless they are incomplete for either statement or branch coverage.
v2.01 Release Notes
Minor Bug Fixes and Improvements
Released March 19, 2018
Bug Fixes:
  • Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate. (Reported by Brad Nicholson.)
  • Immediately error when a secure option (e.g. repo1-s3-key) is passed on the command line. Since pgBackRest would not pass secure options on to sub-processes an obscure error was thrown. The new error is much clearer and provides hints about how to fix the problem. Update command documentation to omit secure options that cannot be specified on the command-line. (Reported by Brad Nicholson.)
  • Fix issue passing --no-config to embedded Perl. (Reported by Ibrahim Edib Kokdemir.)
  • Fix issue where specifying log-level-stderr > warn would cause a local/remote process to error on exit due to output found on stderr when none was expected. The max value for a local/remote process is now error since there is no reason for these processes to emit warnings. (Reported by Clinton Adams.)
  • Fix manifest test in the check command when tablespaces are present. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Thomas Flatley.)
Improvements:
  • Error when multiple arguments are set in the config file for an option that does not accept multiple arguments. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Remove extraneous sudo commands from src/Makefile. (Contributed by Adrian Vondendriesch. Reviewed by David Steele.)
Documentation Improvements:
  • Show index in examples for indexed options, i.e. repo-*, pg-*. (Suggested by Stephen Frost.)
  • Simplify table of contents on command page by only listing commands. (Suggested by Stephen Frost.)
  • Remove references to the C library being optional.
Test Suite Features:
  • Add CentOS/RHEL package builds.
  • Use clang for static code analysis. Nothing found initially except for some functions that should have been marked __noreturn__.
v2.00 Release Notes
Performance Improvements for Archive Push
Released February 23, 2018
Features:
  • The archive-push command is now partially coded in C which allows the PostgreSQL archive_command to run significantly faster when processing status messages from the asynchronous archive process. (Reviewed by Cynthia Shang.)
Improvements:
  • Improve check command to verify that the backup manifest can be built. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Improve performance of HTTPS client. Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
  • Improve S3 delete performance. The constant S3_BATCH_MAX had been replaced with a hard-coded value of 2, probably during testing.
  • Allow any non-command-line option to be reset to default on the command-line. This allows options in pgbackrest.conf to be reset to default which reduces the need to write new configuration files for specific needs.
  • The C library is now required. This eliminates conditional loading and eases development of new library features.
  • The pgbackrest executable is now a C binary instead of Perl. This allows certain time-critical commands (like async archive-push) to run more quickly.
  • Rename db-* options to pg-* and backup-* options to repo-* to improve consistency. repo-* options are now indexed although currently only one is allowed.
Documentation Features:
  • All clusters in the documentation are initialized with checksums.
Documentation Improvements:
  • List deprecated option names in documentation and command-line help.
  • Clarify that S3 buckets must be created by the user. (Suggested by David Youatt.)
v1.29 Release Notes
Critical Bug Fix for Backup Resume
Released July 5, 2018
IMPORTANT NOTE: This release fixes a critical bug in the backup resume feature. All resumed backups prior to this release should be considered inconsistent. A backup will be resumed after a prior backup fails, unless resume=n has been specified. A resumed backup can be identified by checking the backup log for the message aborted backup of same type exists, will be cleaned to remove invalid files and resumed. If the message exists, do not use this backup or any backup in the same set for a restore and check the restore logs to see if a resumed backup was restored. If so, there may be inconsistent data in the cluster.
Bug Fixes:
  • Fix critical bug in resume that resulted in inconsistent backups. A regression in v0.82 removed the timestamp comparison when deciding which files from the aborted backup to keep on resume. See note above for more details. (Reported by David Youatt, Yogesh Sharma, Stephen Frost.)
  • Fix non-compliant ISO-8601 timestamp format in S3 authorization headers. AWS and some gateways were tolerant of space rather than zero-padded hours while others were not. (Fixed by Andrew Schwartz. Reviewed by David Steele.)
  • Fix directory syncs running recursively when only the specified directory should be synced. (Reported by Craig A. James.)
  • Fix --target-action and --recovery-option options being reported as invalid when restoring with --type=immediate. (Reported by Brad Nicholson.)
  • Fix archive-copy throwing path not found error for incr/diff backups. (Reported by yummyliu, Vitaliy Kukharik.)
  • Fix failure in manifest build when two or more files in PGDATA are linked to the same directory. (Reported by Vitaliy Kukharik.)
  • Fix delta restore failing when a linked file was missing.
  • Fix error in selective restore when only one user database exists in the cluster. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Nj Baliyan.)
Improvements:
  • Improve the HTTP client to set content-length to 0 when not specified by the server. S3 (and gateways) always set content-length or transfer-encoding but HTTP 1.1 does not require it and proxies (e.g. HAProxy) may not include either. (Suggested by Adam K. Sumner.)
  • Improve performance of HTTPS client. Buffering now takes the pending bytes on the socket into account (when present) rather than relying entirely on select(). In some instances the final bytes would not be flushed until the connection was closed.
  • Improve S3 delete performance. The constant S3_BATCH_MAX had been replaced with a hard-coded value of 2, probably during testing.
  • Make backup/restore path sync more efficient. Scanning the entire directory can be very expensive if there are a lot of small tables. The backup manifest contains the path list so use it to perform syncs instead of scanning the backup/restore path. Remove recursive path sync functionality since it is no longer used.
Documentation Bug Fixes:
  • Update docs with 32-bit support and caveats. 32-bit support was added in v1.26. (Reported by Viorel Tabara.)
Documentation Improvements:
  • Clarify that S3 buckets must be created by the user. (Suggested by David Youatt.)
  • Update out-of-date description for the spool-path option.
v1.28 Release Notes
Stanza Delete
Released February 1, 2018
Bug Fixes:
  • Fixed inability to restore a single database contained in a tablespace using --db-include. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Chiranjeevi Ravilla.)
  • Ensure latest db-id is selected on when matching archive.info to backup.info. This provides correct matching in the event there are system-id and db-version duplicates (e.g. after reverting a pg_upgrade). (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Adam K. Sumner.)
  • Fixed overly chatty error message when reporting an invalid command. (Reported by Jason O'Donnell.)
Features:
  • Add stanza-delete command to cleanup unused stanzas. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Magnus Hagander.)
Improvements:
  • Improve stanza-create command so that it does not error when the stanza already exists. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Update stanza-create --force documentation to urge caution when using. (Suggested by Jason O'Donnell.)
v1.27 Release Notes
Bug Fixes and Documentation
Released December 19, 2017
Bug Fixes:
  • Fixed an issue that suppressed locality errors for backup and restore. When a backup host is present, backups should only be allowed on the backup host and restores should only be allowed on the database host unless an alternate configuration is created that ignores the remote host. (Reported by Lardière Sébastien.)
  • Fixed an issue where WAL was not expired on PostgreSQL 10. This was caused by a faulty regex that expected all PostgreSQL major versions to be X.X. (Reported by Adam Brusselback.)
  • Fixed an issue where the --no-config option was not passed to child processes. This meant the child processes would still read the local config file and possibly cause unexpected behaviors.
  • Fixed info command to eliminate "db (prior)" output if no backups or archives exist for a prior version of the cluster. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Stephen Frost.)
Documentation Features:
  • Document the relationship between the archive-copy and archive-check options. (Suggested by Markus Nullmeier.)
  • Improve archive-copy reference documentation.
v1.26 Release Notes
Repository Encryption
Released November 21, 2017
Bug Fixes:
  • Fixed an issue that could cause copying large manifests to fail during restore. (Reported by Craig A. James.)
  • Fixed incorrect WAL offset for 32-bit architectures. (Fixed by Javier Wilson. Reviewed by David Steele.)
  • Fixed an issue retrieving WAL for old database versions. After a stanza-upgrade it should still be possible to restore backups from the previous version and perform recovery with archive-get. However, archive-get only checked the most recent db version/id and failed. Also clean up some issues when the same db version/id appears multiple times in the history. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Clinton Adams.)
  • Fixed an issue with invalid backup groups being set correctly on restore. If the backup cannot map a group to a name it stores the group in the manifest as false then uses either the owner of $PGDATA to set the group during restore or failing that the group of the current user. This logic was not working correctly because the selected group was overwriting the user on restore leaving the group undefined and the user incorrectly set to the group. (Reported by Jeff McCormick.)
  • Fixed an issue passing parameters to remotes. When more than one db was specified the path, port, and socket path would for db1 were passed no matter which db was actually being addressed. (Reported by uspen.)
Features:
  • Repository encryption support. (Contributed by Cynthia Shang, David Steele.)
Improvements:
  • Disable gzip filter when --compress-level-network=0. The filter was used with compress level set to 0 which added overhead without any benefit.
  • Inflate performance improvement for gzip filter.
Documentation Features:
  • Add template to improve initial information gathered for issue submissions. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Clarify usage of the archive-timeout option and describe how it is distinct from the PostgreSQL archive_timeout setting. (Contributed by Cynthia Shang. Reviewed by David Steele. Suggested by Keith Fiske.)
Test Suite Features:
  • Automated tests for 32-bit i386/i686 architecture.
v1.25 Release Notes
S3 Performance Improvements
Released October 24, 2017
Bug Fixes:
  • Fix custom settings for compress-level option being ignored. (Reported by Jens Wilke.)
  • Remove error when overlapping timelines are detected. Overlapping timelines are valid in many Point-in-Time-Recovery (PITR) scenarios. (Reported by blogh.)
  • Fix instances where database-id was not rendered as an integer in JSON info output. (Fixed by Cynthia Shang. Reviewed by David Steele. Reported by Jason O'Donnell.)
Features:
  • Improve performance of list requests on S3. Any beginning literal portion of a filter expression is used to generate a search prefix which often helps keep the request small enough to avoid rate limiting. (Suggested by Mihail Shvein.)
Test Suite Features:
  • Add I/O performance tests.
v1.24 Release Notes
New Backup Exclusions
Released September 28, 2017
Bug Fixes:
  • Fixed an issue where warnings were being emitted in place of lower priority log messages during backup from standby initialization. (Reported by uspen.)
  • Fixed an issue where some db-* options (e.g. db-port) were not being passed to remotes. (Reported by uspen.)
Features:
  • Exclude contents of pg_snapshots, pg_serial, pg_notify, and pg_dynshmem from backup since they are rebuilt on startup.
  • Exclude pg_internal.init files from backup since they are rebuilt on startup.
Improvements:
  • Open log file after async process is completely separated from the main process to prevent the main process from also logging to the file. (Suggested by Jens Wilke.)
Documentation Features:
  • Add passwordless SSH configuration.
Documentation Improvements:
  • Rename master to primary in documentation to align with PostgreSQL convention.
v1.23 Release Notes
Multiple Standbys and PostgreSQL 10 Support
Released September 3, 2017
Bug Fixes:
  • Fixed an issue that could cause compression to abort on growing files. (Reported by Jesper St John, Aleksandr Rogozin.)
  • Fixed an issue with keep-alives not being sent to the remote from the local process. (Reported by William Cox.)
Features:
  • Up to seven standbys can be configured for backup from standby. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • PostgreSQL 10 support.
  • Allow content-length (in addition to chunked encoding) when reading XML data to improve compatibility with third-party S3 gateways. (Suggested by Victor Gdalevich.)
Improvements:
  • Increase HTTP timeout for S3.
  • Add HTTP retries to harden against transient S3 network errors.
Documentation Bug Fixes:
  • Fixed document generation to include section summaries on the Configuration page. (Fixed by Cynthia Shang. Reviewed by David Steele.)
v1.22 Release Notes
Fixed S3 Retry
Released August 9, 2017
Bug Fixes:
  • Fixed authentication issue in S3 retry.
v1.21 Release Notes
Improved Info Output and SSH Port Option
Released August 8, 2017
Bug Fixes:
  • The archive_status directory is now recreated on restore to support PostgreSQL 8.3 which does not recreate it automatically like more recent versions do. (Reported by Stephen Frost.)
  • Fixed an issue that could cause the empty archive directory for an old PostgreSQL version to be left behind after a stanza-upgrade. (Fixed by Cynthia Shang. Reviewed by David Steele.)
Features:
  • Modified the info command (both text and JSON output) to display the archive ID and minimum/maximum WAL currently present in the archive for the current and prior, if any, database cluster version. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Added --backup-ssh-port and --db-ssh-port options to support non-default SSH ports. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Retry when S3 returns an internal error (500).
Documentation Bug Fixes:
  • Fix description of --online based on the command context.
Documentation Features:
  • Add creation of /etc/pgbackrest.conf to manual installation instructions.
Documentation Improvements:
  • Move repository options into a separate section in command/command-line help. (Suggested by Stephen Frost.)
v1.20 Release Notes
Critical 8.3/8.4 Bug Fix
Released June 27, 2017
IMPORTANT NOTE: PostgreSQL 8.3 and 8.4 installations utilizing tablespaces should upgrade immediately from any v1 release and run a full backup. A bug prevented tablespaces from being backed up on these versions only. PostgreSQL ≥ 9.0
Bug Fixes:
  • Fixed an issue that prevented tablespaces from being backed up on PostgreSQL ≤ 8.4.
  • Fixed missing flag in C library build that resulted in a mismatched binary on 32-bit systems. (Reported by Adrian Vondendriesch.)
Features:
  • Add s3-repo-ca-path and s3-repo-ca-file options to accommodate systems where CAs are not automatically found by IO::Socket::SSL, i.e. RHEL7, or to load custom CAs. (Suggested by Scott Frazer.)
Test Suite Features:
  • Add documentation builds to CI.
v1.19 Release Notes
S3 Support
Released June 12, 2017
Bug Fixes:
  • Fixed the info command so the WAL archive min/max displayed is for the current database version. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed the backup command so the backup-standby option is reset (and the backup proceeds on the primary) if the standby is not configured and/or reachable. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed config warnings raised from a remote process causing errors in the master process. (Fixed by Cynthia Shang. Reviewed by David Steele.)
Features:
  • Amazon S3 repository support. (Reviewed by Cynthia Shang.)
Documentation Bug Fixes:
  • Changed invalid max-archive-mb option in configuration reference to archive-queue-max.
  • Fixed missing sudo in installation section. (Fixed by Lætitia. Reviewed by David Steele.)
v1.18 Release Notes
Stanza Upgrade, Refactoring, and Locking Improvements
Released April 12, 2017
Bug Fixes:
  • Fixed an issue where read-only operations that used local worker processes (i.e. restore) were creating write locks that could interfere with parallel archive-push. (Reported by Jens Wilke.)
Features:
  • Added the stanza-upgrade command to provide a mechanism for upgrading a stanza after upgrading to a new major version of PostgreSQL. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Added validation of pgbackrest.conf to display warnings if options are not valid or are not in the correct section. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Simplify locking scheme. Now, only the master process will hold write locks (for archive-push and backup commands) and not all local and remote worker processes as before.
  • Do not set timestamps of files in the backup directories to match timestamps in the cluster directory. This was originally done to enable backup resume, but that process is now implemented with checksums.
  • Improved error message when the restore command detects the presence of postmaster.pid. (Suggested by Yogesh Sharma.)
  • Renumber return codes between 25 and 125 to avoid PostgreSQL interpreting some as fatal signal exceptions. (Suggested by Yogesh Sharma.)
v1.17 Release Notes
Page Checksum Bug Fix
Released March 13, 2017
Bug Fixes:
  • Fixed an issue where newly initialized (but unused) pages would cause page checksum warnings. (Reported by Stephen Frost.)
v1.16 Release Notes
Page Checksum Improvements, CI, and Package Testing
Released March 2, 2017
Bug Fixes:
  • Fixed an issue where tables over 1GB would report page checksum warnings after the first segment. (Reported by Stephen Frost.)
  • Fixed an issue where databases created with a non-default tablespace would raise bogus warnings about pg_filenode.map and pg_internal.init not being page aligned. (Reported by blogh.)
Test Suite Features:
  • Continuous integration using travis-ci.
  • Automated builds of Debian packages for all supported distributions.
v1.15 Release Notes
Refactoring and Bug Fixes
Released February 13, 2017
Bug Fixes:
  • Fixed a regression introduced in v1.13 that could cause backups to fail if files were removed (e.g. tables dropped) while the manifest was being built. (Reported by Navid Golpayegani.)
v1.14 Release Notes
Refactoring and Bug Fixes
Released February 13, 2017
Bug Fixes:
  • Fixed an issue where an archive-push error would not be retried and would instead return errors to PostgreSQL indefinitely (unless the .error file was manually deleted). (Reported by Jens Wilke.)
  • Fixed a race condition in parallel archiving where creation of new paths generated an error when multiple processes attempted to do so at the same time. (Reported by Jens Wilke.)
Improvements:
  • Improved performance of wal archive min/max provided by the info command. (Suggested by Jens Wilke.)
Documentation Features:
  • Updated async archiving documentation to more accurately describe how the new method works and how it differs from the old method. (Suggested by Jens Wilke.)
v1.13 Release Notes
Parallel Archiving, Stanza Create, Improved Info and Check
Released February 5, 2017
IMPORTANT NOTE: The new implementation of asynchronous archiving no longer copies WAL to a separate queue. If there is any WAL left over in the old queue after upgrading to 1.13, it will be abandoned and not pushed to the repository.

To prevent this outcome, stop archiving by setting archive_command = false. Next, drain the async queue by running pgbackrest --stanza=[stanza-name] archive-push and wait for the process to complete. Check that the queue in [spool-path]/archive/[stanza-name]/out is empty. Finally, install 1.13 and restore the original archive_command.

IMPORTANT NOTE: The stanza-create command is not longer optional and must be executed before backup or archiving can be performed on a new stanza. Pre-existing stanzas do not require stanza-create to be executed.
Bug Fixes:
  • Fixed const assignment giving compiler warning in C library. (Fixed by Adrian Vondendriesch. Reviewed by David Steele.)
  • Fixed a few directory syncs that were missed for the --repo-sync option.
  • Fixed an issue where a missing user/group on restore could cause an uninitialized value error in File->owner(). (Reported by Leonardo GG Avellar.)
  • Fixed an issue where protocol mismatch errors did not output the expected value.
  • Fixed a spurious archive-get log message that indicated an exit code of 1 was an abnormal termination.
Features:
  • Improved, multi-process implementation of asynchronous archiving.
  • Improved stanza-create command so that it can repair broken repositories in most cases and is robust enough to be made mandatory. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Improved check command to run on a standby, though only basic checks are done because pg_switch_xlog() cannot be executed on a replica. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Added archive and backup WAL ranges to the info command.
  • Added warning to update pg_tablespace.spclocation when remapping tablespaces in PostgreSQL < 9.2. (Contributed by blogh. Reviewed by David Steele.)
  • Remove remote lock requirements for the archive-get, restore, info, and check commands since they are read-only operations. (Suggested by Michael Vitale.)
Improvements:
  • Log file banner is not output until the first log entry is written. (Suggested by Jens Wilke.)
  • Reduced the likelihood of torn pages causing a false positive in page checksums by filtering on start backup LSN.
  • Remove Intel-specific optimization from C library build flags. (Contributed by Adrian Vondendriesch. Reviewed by David Steele.)
  • Remove --lock option. This option was introduced before the lock directory could be located outside the repository and is now obsolete.
  • Added --log-timestamp option to allow timestamps to be suppressed in logging. This is primarily used to avoid filters in the automated documentation.
  • Return proper error code when unable to convert a relative path to an absolute path. (Suggested by Yogesh Sharma.)
Documentation Features:
  • Added documentation to the User Guide for the process-max option. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.12 Release Notes
Page Checksums, Configuration, and Bug Fixes
Released December 12, 2016
IMPORTANT NOTE: In prior releases it was possible to specify options on the command-line that were invalid for the current command without getting an error. An error will now be generated for invalid options so it is important to carefully check command-line options in your environment to prevent disruption.
Bug Fixes:
  • Fixed an issue where options that were invalid for the specified command could be provided on the command-line without generating an error. The options were ignored and did not cause any change in behavior, but it did lead to some confusion. Invalid options will now generate an error. (Reported by Nikhilchandra Kulkarni.)
  • Fixed an issue where internal symlinks were not being created for tablespaces in the repository. This issue was only apparent when trying to bring up clusters in-place manually using filesystem snapshots and did not affect normal backup and restore.
  • Fixed an issue that prevented errors from being output to the console before the logging system was initialized, i.e. while parsing options. Error codes were still being returned accurately so this would not have made a process look like it succeeded when it did not. (Reported by Adrian Vondendriesch.)
  • Fixed an issue where the db-port option specified on the backup server would not be properly passed to the remote unless it was from the first configured database. (Reported by Michael Vitale.)
Features:
  • Added the --checksum-page option to allow pgBackRest to validate page checksums in data files when checksums are enabled on PostgreSQL >= 9.3. Note that this functionality requires a C library which may not initially be available in OS packages. The option will automatically be enabled when the library is present and checksums are enabled on the cluster. (Suggested by Stephen Frost.)
  • Added the --repo-link option to allow internal symlinks to be suppressed when the repository is located on a filesystem that does not support symlinks. This does not affect any pgBackRest functionality, but the convenience link latest will not be created and neither will internal tablespace symlinks, which will affect the ability to bring up clusters in-place manually using filesystem snapshots.
  • Added the --repo-sync option to allow directory syncs in the repository to be disabled for file systems that do not support them, e.g. NTFS.
  • Added a predictable log entry to signal that a command has completed successfully. For example a backup ends successfully with: INFO: backup command end: completed successfully. (Suggested by Jens Wilke.)
Improvements:
  • For simplicity, the pg_control file is now copied with the rest of the files instead of by itself of at the end of the process. The backup command does not require this behavior and the restore copies to a temporary file which is renamed at the end of the restore.
Documentation Bug Fixes:
  • Fixed an issue that suppressed exceptions in PDF builds.
  • Fixed regression in section links introduced in v1.10.
Documentation Features:
  • Added Retention to QuickStart section.
v1.11 Release Notes
Bug Fix for Asynchronous Archiving Efficiency
Released November 17, 2016
Bug Fixes:
  • Fixed an issue where asynchronous archiving was transferring one file per execution instead of transferring files in batches. This regression was introduced in v1.09 and affected efficiency only, all WAL segments were correctly archived in asynchronous mode. (Reported by Stephen Frost.)
v1.10 Release Notes
Stanza Creation and Minor Bug Fixes
Released November 8, 2016
Bug Fixes:
  • Fixed an issue where a backup could error if no changes were made to a database between backups and only pg_control changed.
  • Fixed an issue where tablespace paths with the same prefix would cause an invalid link error. (Reported by Nikhilchandra Kulkarni.)
Features:
  • Added the stanza-create command to formalize creation of stanzas in the repository. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Improvements:
  • Removed extraneous use lib directives from Perl modules. (Suggested by Devrim Gündüz.)
v1.09 Release Notes
9.6 Support, Configurability, and Bug Fixes
Released October 10, 2016
Bug Fixes:
  • Fixed the check command to prevent an error message from being logged if the backup directory does not exist. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed error message to properly display the archive command when an invalid archive command is detected. (Reported by Jason O'Donnell.)
  • Fixed an issue where the async archiver would not be started if archive-push did not have enough space to queue a new WAL segment. This meant that the queue would never be cleared without manual intervention (such as calling archive-push directly). PostgreSQL now receives errors when there is not enough space to store new WAL segments but the async process will still be started so that space is eventually freed. (Reported by Jens Wilke.)
  • Fixed a remote timeout that occurred when a local process generated checksums (during resume or restore) but did not copy files, allowing the remote to go idle. (Reported by Jens Wilke.)
Features:
  • Non-exclusive backups will automatically be used on PostgreSQL 9.6.
  • Added the cmd-ssh option to allow the ssh client to be specified. (Suggested by Jens Wilke.)
  • Added the log-level-stderr option to control whether console log messages are sent to stderr or stdout. By default this is set to warn which represents a change in behavior from previous versions, even though it may be more intuitive. Setting log-level-stderr=off will preserve the old behavior. (Suggested by Sascha Biberhofer.)
  • Set application_name to "pgBackRest [command]" for database connections. (Suggested by Jens Wilke.)
  • Check that archive_mode is enabled when archive-check option enabled.
Improvements:
  • Clarified error message when unable to acquire pgBackRest advisory lock to make it clear that it is not a PostgreSQL backup lock. (Suggested by Jens Wilke.)
  • pgBackRest version number included in command start INFO log output.
  • Process ID logged for local process start/stop INFO log output.
Documentation Features:
  • Added archive-timeout option documentation to the user guide. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.08 Release Notes
Bug Fixes and Log Improvements
Released September 14, 2016
Bug Fixes:
  • Fixed an issue where local processes were not disconnecting when complete and could later timeout. (Reported by Todd Vernick.)
  • Fixed an issue where the protocol layer could timeout while waiting for WAL segments to arrive in the archive. (Reported by Todd Vernick.)
Improvements:
  • Cache file log output until the file is created to create a more complete log.
v1.07 Release Notes
Thread to Process Conversion and Bug Fixes
Released September 7, 2016
Bug Fixes:
  • Fixed an issue where tablespaces were copied from the primary during standby backup.
  • Fixed the check command so backup info is checked remotely and not just locally. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed an issue where retention-archive was not automatically being set when retention-archive-type=diff, resulting in a less aggressive than intended expiration of archive. (Fixed by Cynthia Shang. Reviewed by David Steele.)
Features:
  • Converted Perl threads to processes to improve compatibility and performance.
  • Exclude contents of $PGDATA/pg_replslot directory so that replication slots on the primary do not become part of the backup.
  • The archive-start and archive-stop settings are now filled in backup.manifest even when archive-check=n. (Suggested by Jens Wilke.)
  • Additional warnings when archive retention settings may not have the intended effect or would allow indefinite retention. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 rc1. Changes to the control/catalog/WAL versions in subsequent release candidates may break compatibility but pgBackRest will be updated with each release to keep pace.
Documentation Bug Fixes:
  • Fixed minor documentation reproducibility issues related to binary paths.
Documentation Features:
  • Documentation for archive retention. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.06 Release Notes
Backup from Standby and Bug Fixes
Released August 25, 2016
Bug Fixes:
  • Fixed an issue where a tablespace link that referenced another link would not produce an error, but instead skip the tablespace entirely. (Reported by Michael Vitale.)
  • Fixed an issue where options that should not allow multiple values could be specified multiple times in pgbackrest.conf without an error being raised. (Reported by Michael Vitale.)
  • Fixed an issue where the protocol-timeout option was not automatically increased when the db-timeout option was increased. (Reported by Todd Vernick.)
Features:
  • Backup from a standby cluster. A connection to the primary cluster is still required to start/stop the backup and copy files that are not replicated, but the vast majority of files are copied from the standby in order to reduce load on the primary.
  • More flexible configuration for databases. Master and standby can both be configured on the backup server and pgBackRest will automatically determine which is the primary. This means no configuration changes for backup are required after failing over from a primary to standby when a separate backup server is used.
  • Exclude directories during backup that are cleaned, recreated, or zeroed by PostgreSQL at startup. These include pgsql_tmp and pg_stat_tmp. The postgresql.auto.conf.tmp file is now excluded in addition to files that were already excluded: backup_label.old, postmaster.opts, postmaster.pid, recovery.conf, recovery.done.
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 beta4. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
Improvements:
  • Improve error message for links that reference links in manifest build.
  • Added hints to error message when relative paths are detected in archive-push or archive-get.
  • Improve backup log messages to indicate which host the files are being copied from.
v1.05 Release Notes
Bug Fix for Tablespace Link Checking
Released August 9, 2016
Bug Fixes:
  • Fixed an issue where tablespace paths that had $PGDATA as a substring would be identified as a subdirectories of $PGDATA even when they were not. Also hardened relative path checking a bit. (Reported by Chris Fort.)
Documentation Features:
  • Added documentation for scheduling backups with cron. (Contributed by Cynthia Shang. Reviewed by David Steele.)
Documentation Improvements:
  • Moved the backlog from the pgBackRest website to the GitHub repository wiki. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.04 Release Notes
Various Bug Fixes
Released July 30, 2016
Bug Fixes:
  • Fixed an issue an where an extraneous remote was created causing threaded backup/restore to possibly timeout and/or throw a lock conflict. (Reported by Michael Vitale.)
  • Fixed an issue where db-path was not required for the check command so an assert was raised when it was missing rather than a polite error message. (Reported by Michael Vitale.)
  • Fixed check command to throw an error when database version/id does not match that of the archive. (Fixed by Cynthia Shang. Reviewed by David Steele.)
  • Fixed an issue where a remote could try to start its own remote when the backup-host option was not present in pgbackrest.conf on the database server. (Reported by Lardière Sébastien.)
  • Fixed an issue where the contents of pg_xlog were being backed up if the directory was symlinked. This didn't cause any issues during restore but was a waste of space.
  • Fixed an invalid log() call in lock routines.
Features:
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 beta3. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
Improvements:
  • Suppress banners on SSH protocol connections.
  • Improved remote error messages to identify the host where the error was raised.
  • All remote types now take locks. The exceptions date to when the test harness and pgBackRest were running in the same VM and no longer apply.
Documentation Features:
  • Added clarification on why the default for the backrest-user option is backrest. (Suggested by Michael Vitale.)
  • Updated information about package availability on supported platforms. (Suggested by Michael Vitale.)
v1.03 Release Notes
Check Command and Bug Fixes
Released July 2, 2016
Bug Fixes:
  • Fixed an issue where keep-alives could be starved out by lots of small files during multi-threaded backup. They were also completely absent from single/multi-threaded backup resume and restore checksumming. (Reported by Janice Parkinson, Chris Barber.)
  • Fixed an issue where the expire command would refuse to run when explicitly called from the command line if the db-host option was set. This was not an issue when expire was run automatically after a backup (Reported by Chris Barber.)
  • Fixed an issue where validation was being running on archive_command even when the archive-check option was disabled.
Features:
  • Added check command to validate that pgBackRest is configured correctly for archiving and backups. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Added the protocol-timeout option. Previously protocol-timeout was set as db-timeout + 30 seconds.
  • Failure to shutdown remotes at the end of the backup no longer throws an exception. Instead a warning is generated that recommends a higher protocol-timeout.
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 beta2. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace.
Improvements:
  • Improved handling of users/groups captured during backup that do not exist on the restore host. Also explicitly handle the case where user/group is not mapped to a name.
  • Option handling is now far more strict. Previously it was possible for a command to use an option that was not explicitly assigned to it. This was especially true for the backup-host and db-host options which are used to determine locality.
Documentation Improvements:
  • Allow a static date to be used for documentation to generate reproducible builds. (Suggested by Adrian Vondendriesch.)
  • Added documentation for asynchronous archiving to the user guide. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Recommended install location for pgBackRest modules is now /usr/share/perl5 since /usr/lib/perl5 has been removed from the search path in newer versions of Perl.
  • Added instructions for removing prior versions of pgBackRest.
v1.02 Release Notes
Bug Fix for Perl 5.22
Released June 2, 2016
Bug Fixes:
  • Fix usage of sprintf() due to new constraints in Perl 5.22. Parameters not referenced in the format string are no longer allowed. (Fixed by Adrian Vondendriesch. Reviewed by David Steele.)
Documentation Bug Fixes:
  • Fixed syntax that was not compatible with Perl 5.2X. (Fixed by Christoph Berg, Adrian Vondendriesch. Reviewed by David Steele.)
  • Fixed absolute paths that were used for the PDF logo. (Reported by Adrian Vondendriesch.)
Documentation Features:
  • Release notes are now broken into sections so that bugs, features, and refactors are clearly delineated. An Additional Notes section has been added for changes to documentation and the test suite that do not affect the core code.
  • Added man page generation. (Contributed by Adrian Vondendriesch, David Steele.)
  • The change log was the last piece of documentation to be rendered in Markdown only. Wrote a converter so the document can be output by the standard renderers. The change log will now be located on the website and has been renamed to Releases. (Contributed by Cynthia Shang. Reviewed by David Steele.)
v1.01 Release Notes
Enhanced Info, Selective Restore, and 9.6 Support
Released May 17, 2016
Features:
  • Enhanced text output of info command to include timestamps, sizes, and the reference list for all backups. (Contributed by Cynthia Shang. Reviewed by David Steele.)
  • Allow selective restore of databases from a cluster backup. This feature can result in major space and time savings when only specific databases are restored. Unrestored databases will not be accessible but must be manually dropped before they will be removed from the shared catalogue. (Reviewed by Cynthia Shang, Greg Smith, Stephen Frost. Suggested by Stephen Frost.)
  • Experimental support for non-exclusive backups in PostgreSQL 9.6 beta1. Changes to the control/catalog/WAL versions in subsequent betas may break compatibility but pgBackRest will be updated with each release to keep pace. (Reviewed by Cynthia Shang.)
v1.00 Release Notes
New Repository Format and Configuration Scheme, Link Support
Released April 14, 2016
IMPORTANT NOTE: This flag day release breaks compatibility with older versions of pgBackRest. The manifest format, on-disk structure, configuration scheme, and the exe/path names have all changed. You must create a new repository to hold backups for this version of pgBackRest and keep your older repository for a time in case you need to do a restore. Restores from the prior repository will require the prior version of pgBackRest but because of name changes it is possible to have 1.00 and a prior version of pgBackRest installed at the same time. See the notes below for more detailed information on what has changed.
Features:
  • Implemented a new configuration scheme which should be far simpler to use. See the User Guide and Configuration Reference for details but for a simple configuration all options can now be placed in the stanza section. Options that are shared between stanzas can be placed in the [global] section. More complex configurations can still make use of command sections though this should be a rare use case. (Suggested by Michael Renner.)
  • The repo-path option now always refers to the repository where backups and archive are stored, whether local or remote, so the repo-remote-path option has been removed. The new spool-path option can be used to define a location for queueing WAL segments when archiving asynchronously. A local repository is no longer required.
  • The default configuration filename is now pgbackrest.conf instead of pg_backrest.conf. This was done for consistency with other naming changes but also to prevent old config files from being loaded accidentally when migrating to 1.00. (Suggested by Michael Renner, Stephen Frost.)
  • The default repository name was changed from /var/lib/backup to /var/lib/pgbackrest. (Suggested by Michael Renner, Stephen Frost.)
  • Lock files are now stored in /tmp/pgbackrest by default. These days /run/pgbackrest is the preferred location but that would require init scripts which are not part of this release. The lock-path option can be used to configure the lock directory.
  • Log files are now stored in /var/log/pgbackrest by default and no longer have the date appended so they can be managed with logrotate. The log-path option can be used to configure the log directory. (Suggested by Stephen Frost.)
  • Executable filename changed from pg_backrest to pgbackrest. (Suggested by Michael Renner, Stephen Frost.)
  • All files and directories linked from PGDATA are now included in the backup. By default links will be restored directly into PGDATA as files or directories. The --link-all option can be used to restore all links to their original locations. The --link-map option can be used to remap a link to a new location.
  • Removed --tablespace option and replaced with --tablespace-map-all option which should more clearly indicate its function.
  • Added detail log level which will output more information than info without being as verbose as debug.
Pre-Stable Releases
v0.92 Release Notes
Command-line Repository Path Fix
Released April 6, 2016
Bug Fixes:
  • Fixed an issue where the master process was passing --repo-remote-path instead of --repo-path to the remote and causing the lock files to be created in the default repository directory (/var/lib/backup), generally ending in failure. This was only an issue when --repo-remote-path was defined on the command line rather than in pg_backrest.conf. (Reported by Jan Wieck.)
v0.91 Release Notes
Tablespace Bug Fix and Minor Enhancements
Released March 22, 2016
IMPORTANT BUG FIX FOR TABLESPACES: A change to the repository format was accidentally introduced in 0.90 which means the on-disk backup was no longer a valid PostgreSQL cluster when the backup contained tablespaces. This only affected users who directly copied the backups to restore PostgreSQL clusters rather than using the restore command. However, the fix breaks compatibility with older backups that contain tablespaces no matter how they are being restored (pgBackRest will throw errors and refuse to restore). New full backups should be taken immediately after installing version 0.91 for any clusters that contain tablespaces. If older backups need to be restored then use a version of pgBackRest that matches the backup version.
Bug Fixes:
  • Fixed repository incompatibility introduced in pgBackRest 0.90. (Reported by Evan Benoit.)
Features:
  • Copy global/pg_control last during backups.
  • Write .info and .manifest files to temp before moving them to their final locations and fsync'ing.
  • Rename --no-start-stop option to --no-online.
Test Suite Features:
  • Static source analysis using Perl-Critic, currently passes on gentle.
v0.90 Release Notes
9.5 Support, Various Enhancements, and Minor Bug Fixes
Released February 7, 2016
Bug Fixes:
  • Fixed an issue where specifying --no-archive-check would throw a configuration error. (Reported by Jason O'Donnell.)
  • Fixed an issue where a temp WAL file left over after a well-timed system crash could cause the next archive-push to fail.
  • The retention-archive option can now be be safely set to less than backup retention (retention-full or retention-diff) without also specifying archive-copy=n. The WAL required to make the backups that fall outside of archive retention consistent will be preserved in the archive. However, in this case PITR will not be possible for the backups that fall outside of archive retention.
Features:
  • When backing up and restoring tablespaces pgBackRest only operates on the subdirectory created for the version of PostgreSQL being run against. Since multiple versions can live in a tablespace (especially during a binary upgrade) this prevents too many files from being copied during a backup and other versions possibly being wiped out during a restore. This only applies to PostgreSQL >= 9.0 — prior versions of PostgreSQL could not share a tablespace directory.
  • Generate an error when archive-check=y but archive_command does not execute pg_backrest. (Contributed by Jason O'Donnell. Reviewed by David Steele.)
  • Improved error message when repo-path or repo-remote-path does not exist.
  • Added checks for --delta and --force restore options to ensure that the destination is a valid $PGDATA directory. pgBackRest will check for the presence of PG_VERSION or backup.manifest (left over from an aborted restore). If neither file is found then --delta and --force will be disabled but the restore will proceed unless there are files in the $PGDATA directory (or any tablespace directories) in which case the operation will be aborted.
  • When restore --set=latest (the default) the actual backup restored will be output to the log.
  • Support for PostgreSQL 9.5 partial WAL segments and recovery_target_action setting. The archive_mode = 'always' setting is not yet supported.
  • Support for recovery_target = 'immediate' recovery setting introduced in PostgreSQL 9.4.
  • The following tablespace checks have been added: paths or files in pg_tblspc, relative links in pg_tblspc, tablespaces in $PGDATA. All three will generate errors.
v0.89 Release Notes
Timeout Bug Fix and Restore Read-Only Repositories
Released December 24, 2015
Bug Fixes:
  • Fixed an issue where longer-running backups/restores would timeout when remote and threaded. Keepalives are now used to make sure the remote for the main process does not timeout while the thread remotes do all the work. The error message for timeouts was also improved to make debugging easier. (Reported by Stephen Frost.)
Features:
  • Allow restores to be performed on a read-only repository by using --no-lock and --log-level-file=off. The --no-lock option can only be used with restores.
v0.88 Release Notes
Documentation and Minor Bug Fixes
Released November 22, 2015
Bug Fixes:
  • Fixed an issue where the start/stop commands required the --config option. (Reported by Dmitry Didovicher.)
  • Fixed an issue where log files were being overwritten instead of appended. (Reported by Stephen Frost, Dmitry Didovicher.)
  • Fixed an issue where backup-user was not optional.
Features:
  • Symlinks are no longer created in backup directories in the repository. These symlinks could point virtually anywhere and potentially be dangerous. Symlinks are still recreated during a restore. (Suggested by Stephen Frost.)
  • Added better messaging for backup expiration. Full and differential backup expirations are logged on a single line along with a list of all dependent backups expired.
  • Archive retention is automatically set to full backup retention if not explicitly configured.
Documentation Features:
  • Added documentation in the user guide for delta restores, expiration, dedicated backup hosts, starting and stopping pgBackRest, and replication.
v0.87 Release Notes
Website and User Guide
Released October 28, 2015
Features:
  • The backup_label.old and recovery.done files are now excluded from backups.
Documentation Features:
  • Added a new user guide that covers pgBackRest basics and some advanced topics including PITR. Much more to come, but it's a start. (Contributed by David Steele, Stephen Frost. Reviewed by Michael Renner, Cynthia Shang, Eric Radman, Dmitry Didovicher.)
v0.85 Release Notes
Start/Stop Commands and Minor Bug Fixes
Released October 8, 2015
Bug Fixes:
  • Fixed an issue where an error could be returned after a backup or restore completely successfully.
  • Fixed an issue where a resume would fail if temp files were left in the root backup directory when the backup failed. This scenario was likely if the backup process got terminated during the copy phase.
Features:
  • Added stop and start commands to prevent pgBackRest processes from running on a system where PostgreSQL is shutdown or the system needs to be quiesced for some other reason.
  • Experimental support for PostgreSQL 9.5 beta1. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
v0.82 Release Notes
Refactoring, Command-line Help, and Minor Bug Fixes
Released September 14, 2015
Bug Fixes:
  • Fixed an issue where resumed compressed backups were not preserving existing files.
  • Fixed an issue where resume and incr/diff would not ensure that the prior backup had the same compression and hardlink settings.
  • Fixed an issue where a cold backup using --no-start-stop could be started on a running PostgreSQL cluster without --force specified.
  • Fixed an issue where a thread could be started even when none were requested.
  • Fixed an issue where the pgBackRest version number was not being updated in backup.info and archive.info after an upgrade/downgrade.
  • Fixed an issue where the info command was throwing an exception when the repository contained no stanzas. (Reported by Stephen Frost.)
  • Fixed an issue where the PostgreSQL pg_stop_backup() NOTICEs were being output to stderr. (Reported by Stephen Frost.)
Features:
  • Experimental support for PostgreSQL 9.5 alpha2. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
Improvements:
  • Renamed recovery-setting option and section to recovery-option to be more consistent with pgBackRest naming conventions.
  • Added dynamic module loading to speed up commands, especially asynchronous archiving.
Documentation Features:
  • Command-line help is now extracted from the same XML source that is used for the other documentation and includes much more detail.
v0.80 Release Notes
DBI Support, Stability, and Convenience Features
Released August 9, 2015
Bug Fixes:
  • Fixed an issue that caused the formatted timestamp for both the oldest and newest backups to be reported as the current time by the info command. Only text output was affected -- json output reported the correct epoch values. (Reported by Michael Renner.)
  • Fixed protocol issue that was preventing ssh errors (especially on connection) from being logged.
Features:
  • The repository is now created and updated with consistent directory and file modes. By default umask is set to 0000 but this can be disabled with the neutral-umask setting. (Suggested by Cynthia Shang.)
  • Added the stop-auto option to allow failed backups to automatically be stopped when a new backup starts.
  • Added the db-timeout option to limit the amount of time pgBackRest will wait for pg_start_backup() and pg_stop_backup() to return.
  • Remove pg_control file at the beginning of the restore and copy it back at the very end. This prevents the possibility that a partial restore can be started by PostgreSQL.
  • Added checks to be sure the db-path setting is consistent with db-port by comparing the data_directory as reported by the cluster against the db-path setting and the version as reported by the cluster against the value read from pg_control. The db-socket-path setting is checked to be sure it is an absolute path.
  • Experimental support for PostgreSQL 9.5 alpha1. This may break when the control version or WAL magic changes in future versions but will be updated in each pgBackRest release to keep pace. All regression tests pass except for --target-resume tests (this functionality has changed in 9.5) and there is no testing yet for .partial WAL segments.
Improvements:
  • Now using Perl DBI and DBD::Pg for connections to PostgreSQL rather than psql. The cmd-psql and cmd-psql-option settings have been removed and replaced with db-port and db-socket-path. Follow the instructions in the Installation Guide to install DBD::Pg on your operating system.
Test Suite Features:
  • Added vagrant test configurations for Ubuntu 14.04 and CentOS 7.
v0.78 Release Notes
Remove CPAN Dependencies, Stability Improvements
Released July 13, 2015
Improvements:
  • Removed dependency on CPAN packages for multi-threaded operation. While it might not be a bad idea to update the threads and Thread::Queue packages, it is no longer necessary.
  • Modified wait backoff to use a Fibonacci rather than geometric sequence. This will make wait time grow less aggressively while still giving reasonable values.
Test Suite Features:
  • Added vagrant test configurations for Ubuntu 12.04 and CentOS 6.
v0.77 Release Notes
CentOS/RHEL 6 Support and Protocol Improvements
Released June 30, 2015
Features:
  • Added file and directory syncs to the File object for additional safety during backup/restore and archiving. (Suggested by Andres Freund.)
  • Added support for Perl 5.10.1 and OpenSSH 5.3 which are default for CentOS/RHEL 6. (Suggested by Eric Radman.)
  • Improved error message when backup is run without archive_command set and without --no-archive-check specified. (Suggested by Eric Radman.)
v0.75 Release Notes
New Repository Format, Info Command and Experimental 9.5 Support
Released June 14, 2015
IMPORTANT NOTE: This flag day release breaks compatibility with older versions of pgBackRest. The manifest format, on-disk structure, and the binary names have all changed. You must create a new repository to hold backups for this version of pgBackRest and keep your older repository for a time in case you need to do a restore. The pg_backrest.conf file has not changed but you'll need to change any references to pg_backrest.pl in cron (or elsewhere) to pg_backrest (without the .pl extension).
Features:
  • Added the info command.
  • Logging now uses unbuffered output. This should make log files that are being written by multiple threads less chaotic. (Suggested by Michael Renner.)
  • Experimental support for PostgreSQL 9.5. This may break when the control version or WAL magic changes but will be updated in each release.
Improvements:
  • More efficient file ordering for backup. Files are copied in descending size order so a single thread does not end up copying a large file at the end. This had already been implemented for restore.
v0.70 Release Notes
Stability Improvements for Archiving, Improved Logging and Help
Released June 1, 2015
Bug Fixes:
  • Fixed an issue where archive-copy would fail on an incr/diff backup when hardlink=n. In this case the pg_xlog path does not already exist and must be created. (Reported by Michael Renner.)
  • Fixed an issue in async archiving where archive-push was not properly returning 0 when archive-max-mb was reached and moved the async check after transfer to avoid having to remove the stop file twice. Also added unit tests for this case and improved error messages to make it clearer to the user what went wrong. (Reported by Michael Renner.)
  • Fixed a locking issue that could allow multiple operations of the same type against a single stanza. This appeared to be benign in terms of data integrity but caused spurious errors while archiving and could lead to errors in backup/restore. (Reported by Michael Renner.)
Features:
  • Allow duplicate WAL segments to be archived when the checksum matches. This is necessary for some recovery scenarios.
  • Allow comments/disabling in pg_backrest.conf using the # character. Only # characters in the first character of the line are honored. (Suggested by Michael Renner.)
  • Better logging before pg_start_backup() to make it clear when the backup is waiting on a checkpoint. (Suggested by Michael Renner.)
  • Various command behavior and logging fixes. (Reviewed by Michael Renner. Suggested by Michael Renner.)
Improvements:
  • Replaced JSON module with JSON::PP which ships with core Perl.
Documentation Bug Fixes:
  • Various help fixes. (Reviewed by Michael Renner. Reported by Michael Renner.)
v0.65 Release Notes
Improved Resume and Restore Logging, Compact Restores
Released May 11, 2015
Bug Fixes:
  • Fixed an issue where an absolute path was not written into recovery.conf when the restore was run with a relative path.
Features:
  • Better resume support. Resumed files are checked to be sure they have not been modified and the manifest is saved more often to preserve checksums as the backup progresses. More unit tests to verify each resume case.
  • Resume is now optional. Use the resume setting or --no-resume from the command line to disable.
  • More info messages during restore. Previously, most of the restore messages were debug level so not a lot was output in the log.
  • Added tablespace setting to allow tablespaces to be restored into the pg_tblspc path. This produces compact restores that are convenient for development, staging, etc. Currently these restores cannot be backed up as pgBackRest expects only links in the pg_tblspc path.
v0.61 Release Notes
Bug Fix for Uncompressed Remote Destination
Released April 21, 2015
Bug Fixes:
  • Fixed a buffering error that could occur on large, highly-compressible files when copying to an uncompressed remote destination. The error was detected in the decompression code and resulted in a failed backup rather than corruption so it should not affect successful backups made with previous versions.
v0.60 Release Notes
Better Version Support and WAL Improvements
Released April 19, 2015
Bug Fixes:
  • Pushing duplicate WAL now generates an error. This worked before only if checksums were disabled.
Features:
  • Database System IDs are used to make sure that all WAL in an archive matches up. This should help prevent misconfigurations that send WAL from multiple clusters to the same archive.
Test Suite Features:
  • Regression tests working back to PostgreSQL 8.3.
v0.50 Release Notes
Restore and Much More
Released March 25, 2015
Bug Fixes:
  • Fixed broken checksums and now they work with normal and resumed backups. Finally realized that checksums and checksum deltas should be functionally separated and this simplified a number of things. Issue #28 has been created for checksum deltas.
  • Fixed an issue where a backup could be resumed from an aborted backup that didn't have the same type and prior backup.
Features:
  • Added restore functionality.
  • All options can now be set on the command-line making pg_backrest.conf optional.
  • De/compression is now performed without threads and checksum/size is calculated in stream. That means file checksums are no longer optional.
  • Added option --no-start-stop to allow backups when Postgres is shut down. If postmaster.pid is present then --force is required to make the backup run (though if Postgres is running an inconsistent backup will likely be created). This option was added primarily for the purpose of unit testing, but there may be applications in the real world as well.
  • Checksum for backup.manifest to detect a corrupted/modified manifest.
  • Link latest always points to the last backup. This has been added for convenience and to make restores simpler.
Test Suite Features:
  • More comprehensive unit tests in all areas.
v0.30 Release Notes
Core Restructuring and Unit Tests
Released October 5, 2014
Documentation Features:
  • Added much needed documentation
Test Suite Features:
  • Fairly comprehensive unit tests for all the basic operations. More work to be done here for sure, but then there is always more work to be done on unit tests.
v0.19 Release Notes
Improved Error Reporting/Handling
Released May 13, 2014
Bug Fixes:
  • Found and squashed a nasty bug where file_copy() was defaulted to ignore errors. There was also an issue in file_exists() that was causing the test to fail when the file actually did exist. Together they could have resulted in a corrupt backup with no errors, though it is very unlikely.
v0.18 Release Notes
Return Soft Error When Archive Missing
Released April 13, 2014
Bug Fixes:
  • The archive-get command now returns a 1 when the archive file is missing to differentiate from hard errors (ssh connection failure, file copy error, etc.) This lets PostgreSQL know that the archive stream has terminated normally. However, this does not take into account possible holes in the archive stream. (Reported by Stephen Frost.)
v0.17 Release Notes
Warn When Archive Directories Cannot Be Deleted
Released April 3, 2014
Bug Fixes:
  • If an archive directory which should be empty could not be deleted backrest was throwing an error. There's a good fix for that coming, but for the time being it has been changed to a warning so processing can continue. This was impacting backups as sometimes the final archive file would not get pushed if the first archive file had been in a different directory (plus some bad luck).
v0.16 Release Notes
RequestTTY=yes for SSH Sessions
Released April 1, 2014
Bug Fixes:
  • Added RequestTTY=yes to ssh sessions. Hoping this will prevent random lockups.
v0.15 Release Notes
Added archive-get
Released March 29, 2014
Features:
  • Added archive-get functionality to aid in restores.
  • Added option to force a checkpoint when starting the backup, start-fast=y.
v0.11 Release Notes
Minor Fixes
Released March 26, 2014
Bug Fixes:
  • Removed master_stderr_discard option on database SSH connections. There have been occasional lockups and they could be related to issues originally seen in the file code. (Reported by Stephen Frost.)
  • Changed lock file conflicts on backup and expire commands to ERROR. They were set to DEBUG due to a copy-and-paste from the archive locks.
v0.10 Release Notes
Backup and Archiving are Functional
Released March 5, 2014
Features:
  • No restore functionality, but the backup directories are consistent PostgreSQL data directories. You'll need to either uncompress the files or turn off compression in the backup. Uncompressed backups on a ZFS (or similar) filesystem are a good option because backups can be restored locally via a snapshot to create logical backups or do spot data recovery.
  • Archiving is single-threaded. This has not posed an issue on our multi-terabyte databases with heavy write volume. Recommend a large WAL volume or to use the async option with a large volume nearby.
  • Backups are multi-threaded, but the Net::OpenSSH library does not appear to be 100% thread-safe so it will very occasionally lock up on a thread. There is an overall process timeout that resolves this issue by killing the process. Yes, very ugly.
  • Checksums are lost on any resumed backup. Only the final backup will record checksum on multiple resumes. Checksums from previous backups are correctly recorded and a full backup will reset everything.
  • The backup.manifest is being written as Storable because Config::IniFile does not seem to handle large files well. Would definitely like to save these as human-readable text.
Documentation Features:
  • Absolutely no documentation (outside the code). Well, excepting these release notes.
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated August 17, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7748528 pgbackrest-2.59.1/doc/html/user-guide-index.html0000640000175100017510000000372315240547776021202 0ustar00runnerrunner pgBackRest - User Guides
pgBackRest
User Guides
Home
Releases
Configuration
Commands
FAQ
Available User Guides
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated August 17, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7747571 pgbackrest-2.59.1/doc/html/user-guide-rhel.html0000640000175100017510000057255215240547776021040 0ustar00runnerrunner pgBackRest User Guide - RHEL
pgBackRest User Guide
RHEL
Home
User Guides
Releases
Configuration
Commands
FAQ
Table of Contents
Introduction
Concepts
Backup
Restore
Write Ahead Log (WAL)
Encryption
Upgrading pgBackRest
Upgrading pgBackRest from v2.x to v2.y
Build
Installation
Quick Start
Setup Demo Cluster
Configure Cluster Stanza
Create the Repository
Configure Archiving
Configure Retention
Configure Repository Encryption
Create the Stanza
Check the Configuration
Performance Tuning
Perform a Backup
Schedule a Backup
Backup Information
Restore a Backup
Monitoring
In PostgreSQL
Backup
File Bundling
Block Incremental
Backup Annotations
Retention
Full Backup Retention
Differential Backup Retention
Archive Retention
Restore
File Ownership
Delta Option
Restore Selected Databases
Point-in-Time Recovery
Delete a Stanza
Multiple Repositories
Azure-Compatible Object Store Support
S3-Compatible Object Store Support
SFTP Support
GCS-Compatible Object Store Support
Target Time for Repository
Dedicated Repository Host
Installation
Configuration
Setup TLS Server
Create and Check Stanza
Perform a Backup
Restore a Backup
Parallel Backup / Restore
Starting and Stopping
Replication
Installation
Hot Standby
Streaming Replication
Multiple Stanzas
Installation
Configuration
Setup Demo Cluster
Create the Stanza and Check Configuration
Asynchronous Archiving
Archive Push
Archive Get
Backup from a Standby
Upgrading PostgreSQL
Introduction
This user guide is intended to be followed sequentially from beginning to end — each section depends on the last. For example, the Restore section relies on setup that is performed in the Quick Start section. Once pgBackRest is up and running then skipping around is possible but following the user guide in order is recommended the first time through.
Although the examples in this guide are targeted at RHEL and PostgreSQL 14, it should be fairly easy to apply the examples to any Unix distribution and PostgreSQL version. The only OS-specific commands are those to create, start, stop, and drop PostgreSQL clusters. The pgBackRest commands will be the same on any Unix system though the location of the executable may vary. While pgBackRest strives to operate consistently across versions of PostgreSQL, there are subtle differences between versions of PostgreSQL that may show up in this guide when illustrating certain examples, e.g. PostgreSQL path/file names and settings.
Configuration information and documentation for PostgreSQL can be found in the PostgreSQL Manual.
A somewhat novel approach is taken to documentation in this user guide. Each command is run on a virtual machine when the documentation is built from the XML source. This means you can have a high confidence that the commands work correctly in the order presented. Output is captured and displayed below the command when appropriate. If the output is not included it is because it was deemed not relevant or was considered a distraction from the narrative.
All commands are intended to be run as an unprivileged user that has sudo privileges for both the root and postgres users. It's also possible to run the commands directly as their respective users without modification and in that case the sudo commands can be stripped off.
Concepts
The following concepts are defined as they are relevant to pgBackRest, PostgreSQL, and this user guide.
Backup
A backup is a consistent copy of a database cluster that can be restored to recover from a hardware failure, to perform Point-In-Time Recovery, or to bring up a new standby.
Full Backup: pgBackRest copies the entire contents of the database cluster to the backup. The first backup of the database cluster is always a Full Backup. pgBackRest is always able to restore a full backup directly. The full backup does not depend on any files outside of the full backup for consistency.
Differential Backup: pgBackRest copies only those database cluster files that have changed since the last full backup. pgBackRest restores a differential backup by copying all of the files in the chosen differential backup and the appropriate unchanged files from the previous full backup. The advantage of a differential backup is that it requires less disk space than a full backup, however, the differential backup and the full backup must both be valid to restore the differential backup.
Incremental Backup: pgBackRest copies only those database cluster files that have changed since the last backup (which can be another incremental backup, a differential backup, or a full backup). As an incremental backup only includes those files changed since the prior backup, they are generally much smaller than full or differential backups. As with the differential backup, the incremental backup depends on other backups to be valid to restore the incremental backup. Since the incremental backup includes only those files since the last backup, all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup must all be valid to perform a restore of the incremental backup. If no differential backup exists then all prior incremental backups back to the prior full backup, which must exist, and the full backup itself must be valid to restore the incremental backup.
Restore
A restore is the act of copying a backup to a system where it will be started as a live database cluster. A restore requires the backup files and one or more WAL segments in order to work correctly.
Write Ahead Log (WAL)
WAL is the mechanism that PostgreSQL uses to ensure that no committed changes are lost. Transactions are written sequentially to the WAL and a transaction is considered to be committed when those writes are flushed to disk. Afterwards, a background process writes the changes into the main database cluster files (also known as the heap). In the event of a crash, the WAL is replayed to make the database consistent.
WAL is conceptually infinite but in practice is broken up into individual 16MB files called segments. WAL segments follow the naming convention 0000000100000A1E000000FE where the first 8 hexadecimal digits represent the timeline and the next 16 digits are the logical sequence number (LSN).
Encryption
Encryption is the process of converting data into a format that is unrecognizable unless the appropriate password (also referred to as passphrase) is provided.
pgBackRest will encrypt the repository based on a user-provided password, thereby preventing unauthorized access to data stored within the repository.
Upgrading pgBackRest
Upgrading pgBackRest from v2.x to v2.y
Upgrading from v2.x to v2.y is straight-forward. The repository format has not changed, so for most installations it is simply a matter of installing binaries for the new version. It is also possible to downgrade if you have not used new features that are unsupported by the older version.
IMPORTANT:
The local and remote pgBackRest versions must match exactly so they should be upgraded together. If there is a mismatch, WAL archiving and backups will not function until the versions match. In such a case, the following error will be reported: [ProtocolError] expected value '2.x' for greeting key 'version' but got '2.y'.
Build
Installing pgBackRest from a package is preferable to building from source. See Installation for more information about packages.
When building from source it is best to use a build host rather than building on production. Many of the tools required for the build should generally not be installed in production. pgBackRest consists of a single executable so it is easy to copy to a new host once it is built.
build ⇒ Download version 2.59.1 of pgBackRest to /build path 
mkdir -p /build
curl -fsSL \
       https://github.com/pgbackrest/pgbackrest/releases/download/release%2F2.59.1/pgbackrest-2.59.1.tar.gz | \
       tar zx -C /build
build ⇒ Install build dependencies 
sudo yum install meson gcc postgresql14-devel openssl-devel libxml2-devel \
       lz4-devel libzstd-devel bzip2-devel libssh2-devel systemd-devel
build ⇒ Configure and compile pgBackRest 
meson setup /build/pgbackrest /build/pgbackrest-2.59.1
ninja -C /build/pgbackrest
build ⇒ Optionally run smoke tests to verify pgBackRest was built correctly 
meson test -C /build/pgbackrest --suite smoke
Output suppressed for testing
Installation
A new host named pg-primary is created to contain the demo cluster and run pgBackRest examples.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
RHEL packages for pgBackRest are available at yum.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-primary ⇒ Install dependencies 
sudo yum install postgresql-libs libssh2
pg-primary ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-primary ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
pgBackRest should now be properly installed but it is best to check. If any dependencies were missed then you will get an error when running pgBackRest from the command line.
pg-primary ⇒ Make sure the installation worked 
sudo -u postgres pgbackrest
Output suppressed for testing
Quick Start
The Quick Start section will cover basic configuration of pgBackRest and PostgreSQL and introduce the backup, restore, and info commands.
Setup Demo Cluster
Creating the demo cluster is optional but is strongly recommended, especially for new users, since the example commands in the user guide reference the demo cluster; the examples assume the demo cluster is running on the default port (i.e. 5432). The cluster will not be started until a later section because there is still some configuration to do.
pg-primary ⇒ Create the demo cluster 
sudo -u postgres /usr/pgsql-14/bin/initdb \
       -D /var/lib/pgsql/14/data -k -A peer
By default RHEL includes the day of the week in the log filename. This makes the user guide a bit more complicated so the log_filename is set to a constant.
pg-primary:/var/lib/pgsql/14/data/postgresql.conf ⇒ Set log_filename
Config suppressed for testing
Configure Cluster Stanza
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod.
The name 'demo' describes the purpose of this cluster accurately so that will also make a good stanza name.
pgBackRest needs to know where the base data directory for the PostgreSQL cluster is located. The path can be requested from PostgreSQL directly but in a recovery scenario the PostgreSQL process will not be available. During backups the value supplied to pgBackRest will be compared against the path that PostgreSQL is running on and they must be equal or the backup will return an error. Make sure that pg-path is exactly equal to data_directory as reported by PostgreSQL.
By default RHEL stores clusters in /var/lib/pgsql/[version]/data so it is easy to determine the correct path for the data directory.
When creating the /etc/pgbackrest/pgbackrest.conf file, the database owner (usually postgres) must be granted read privileges.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the PostgreSQL cluster data directory
Config suppressed for testing
pgBackRest configuration files follow a Windows INI-like convention. Sections are denoted by text in brackets and key/value pairs are contained in each section. Lines beginning with # are ignored and can be used as comments, but end-of-line comments following a value on the same line are not supported. Quoting is not supported and whitespace is trimmed from keys and values. Sections will be merged if they appear more than once.
There are multiple ways the pgBackRest configuration files can be loaded:
  • config and config-include-path are default: the default config file will be loaded, if it exists, and *.conf files in the default config include path will be appended, if they exist.
  • config option is specified: only the specified config file will be loaded and is expected to exist.
  • config-include-path is specified: *.conf files in the config include path will be loaded and the path is required to exist. The default config file will be be loaded if it exists. If it is desirable to load only the files in the specified config include path, then the --no-config option can also be passed.
  • config and config-include-path are specified: using the user-specified values, the config file will be loaded and *.conf files in the config include path will be appended. The files are expected to exist.
  • config-path is specified: this setting will override the base path for the default location of the config file and/or the base path of the default config-include-path setting unless the config and/or config-include-path option is explicitly set.
Files are concatenated as if they were one big file and each file must be valid individually. This means sections must be specified in each file where they are needed to store a key/value. Order doesn't matter but there is precedence based on sections. The precedence (highest to lowest) is:
  • [stanza:command]
  • [stanza]
  • [global:command]
  • [global]
NOTE:
--config, --config-include-path and --config-path are command-line only options.
pgBackRest can also be configured using environment variables as described in the command reference.
pg-primary ⇒ Configure log-path using the environment 
sudo -u postgres bash -c ' \
       export PGBACKREST_LOG_PATH=/path/set/by/env && \
       pgbackrest --log-level-console=error help backup log-path'
Output suppressed for testing
Create the Repository
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves.
For this demonstration the repository will be stored on the same host as the PostgreSQL server. This is the simplest configuration and is useful in cases where traditional backup software is employed to backup the database host.
pg-primary ⇒ Create the pgBackRest repository 
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown postgres:postgres /var/lib/pgbackrest
The repository path must be configured so pgBackRest knows where to find it.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the pgBackRest repository path
Config suppressed for testing
Multiple repositories may also be configured. See Multiple Repositories for details.
Configure Archiving
Backing up a running PostgreSQL cluster requires WAL archiving to be enabled. %p is how PostgreSQL specifies the location of the WAL segment to be archived. Note that at least one WAL segment will be created during the backup process even if no explicit writes are made to the cluster.
pg-primary:/var/lib/pgsql/14/data/postgresql.conf ⇒ Configure archive settings
Config suppressed for testing
The PostgreSQL cluster must be restarted after making these changes and before performing a backup.
pg-primary ⇒ Restart the demo cluster 
sudo systemctl restart postgresql-14.service
The hot_standby setting is enabled by default and should be left that way on every cluster. Any cluster may be restored as a standby later, e.g. a primary that is rebuilt as a standby after failover, and a standby will not accept read-only connections without it.
When archiving a WAL segment is expected to take more than 60 seconds (the default) to reach the pgBackRest repository, then the pgBackRest archive-timeout option should be increased. Note that this option is not the same as the PostgreSQL archive_timeout option which is used to force a WAL segment switch; useful for databases where there are long periods of inactivity. For more information on the PostgreSQL archive_timeout option, see PostgreSQL Write Ahead Log.
The archive-push command can be configured with its own options. For example, a lower compression level may be set to speed archiving without affecting the compression used for backups.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Config archive-push to use a lower compression level
Config suppressed for testing
This configuration technique can be used for any command and can even target a specific stanza, e.g. demo:archive-push.
Configure Retention
pgBackRest expires backups based on retention options.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure retention to 2 full backups
Config suppressed for testing
More information about retention can be found in the Retention section.
Configure Repository Encryption
The repository will be configured with a cipher type and key to demonstrate encryption. Encryption is always performed client-side even if the repository type (e.g. S3 or other object store) supports encryption.
It is important to use a long, random passphrase for the cipher key. A good way to generate one is to run: openssl rand -base64 48.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest repository encryption
Config suppressed for testing
NOTE:
Encryption settings are placed in the [global] section, above, so the info command can read all stanzas. Without the stanza option the info command reads encryption settings only from the [global] section, so encryption settings configured per stanza require the stanza option to read an encrypted stanza.
Once the repository has been configured and the stanza created and checked, the repository encryption settings cannot be changed.
Create the Stanza
The stanza-create command must be run to initialize the stanza. It is recommended that the check command be run after stanza-create to ensure archiving and backups are properly configured.
pg-primary ⇒ Create the stanza and check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
Check the Configuration
The check command validates that pgBackRest and the archive_command setting are configured correctly for archiving and backups for the specified stanza. It will attempt to check all repositories and databases that are configured for the host on which the command is run. It detects misconfigurations, particularly in archiving, that result in incomplete backups because required WAL segments did not reach the archive. The command can be run on the PostgreSQL or repository host. The command may also be run on the standby host, however, since pg_switch_xlog()/pg_switch_wal() cannot be performed on the standby, the command will only test the repository configuration.
Note that pg_create_restore_point('pgBackRest Archive Check') and pg_switch_xlog()/pg_switch_wal() are called to force PostgreSQL to archive a WAL segment.
pg-primary ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Performance Tuning
pgBackRest has a number of performance options that are not enabled by default to maintain backward compatibility in the repository. However, when creating a new repository the following options are recommended. They can also be used on an existing repository with the caveat that older versions of pgBackRest will not be able to read the repository. This incompatibility depends on when the feature was introduced, as noted in the list below.
  • compress-type - determines the compression algorithm used by the backup and archive-push commands. The default is gz (Gzip) but zst (Zstandard) is recommended because it is much faster and provides compression similar to gz. zst has been supported by the compress-type option since v2.27. See Compress Type for more details.
  • repo-bundle - combines small files during backup to save space and improve the speed of both the backup and restore commands, especially on object stores such as S3. The repo-bundle option was introduced in v2.39. See File Bundling for more details.
  • repo-block - stores only the portions of files that have changed rather than the entire file during diff/incr backup. This saves space and increases the speed of the backup. The repo-block option was introduced in v2.46 but at least v2.52.1 is recommended. See Block Incremental for more details.
There are other performance options that are not enabled by default because they require additional configuration or because the default is safe (but not optimal). These options are available in all v2 versions of pgBackRest.
  • process-max - determines how many processes will be used for commands. The default is 1, which is almost never the appropriate value. Each command uses process-max differently so refer to each command's documentation for details on usage.
  • archive-async - archives WAL files to the repository in batch which greatly increases archiving speed. It is not enabled by default because it requires a spool path to be created. See Asynchronous Archiving for more details.
  • backup-standby - performs the backup on a standby rather than the primary to reduce load on the primary. It is not enabled by default because it requires additional configuration and the presence of one or more standby hosts. See Backup from a Standby for more details.
Perform a Backup
By default pgBackRest will wait for the next regularly scheduled checkpoint before starting a backup. Depending on the checkpoint_timeout and checkpoint_segments settings in PostgreSQL it may be quite some time before a checkpoint completes and the backup can begin. Generally, it is best to set start-fast=y so that the backup starts immediately. This forces a checkpoint, but since backups are usually run once a day an additional checkpoint should not have a noticeable impact on performance. However, on very busy clusters it may be best to pass --start-fast on the command-line as needed.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure backup fast start
Config suppressed for testing
To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo \
       --log-level-console=info backup
Output suppressed for testing
By default pgBackRest will attempt to perform an incremental backup. However, an incremental backup must be based on a full backup and since no full backup existed pgBackRest ran a full backup instead.
The type option can be used to specify a full or differential backup.
pg-primary ⇒ Differential backup of the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
This time there was no warning because a full backup already existed. While incremental backups can be based on a full or differential backup, differential backups must be based on a full backup. A full backup can be performed by running the backup command with --type=full.
During an online backup pgBackRest waits for WAL segments that are required for backup consistency to be archived. This wait time is governed by the pgBackRest archive-timeout option which defaults to 60 seconds. If archiving an individual segment is known to take longer then this option should be increased.
Schedule a Backup
Backups can be scheduled with utilities such as cron.
In the following example, two cron jobs are configured to run; full backups are scheduled for 6:30 AM every Sunday with differential backups scheduled for 6:30 AM Monday through Saturday. If this crontab is installed for the first time mid-week, then pgBackRest will run a full backup the first time the differential job is executed, followed the next day by a differential backup. 
#m h   dom mon dow   command
30 06  *   *   0     pgbackrest --type=full --stanza=demo backup
30 06  *   *   1-6   pgbackrest --type=diff --stanza=demo backup
Once backups are scheduled it's important to configure retention so backups are expired on a regular schedule, see Retention.
Backup Information
Use the info command to get information about backups.
pg-primary ⇒ Get info for the demo cluster 
sudo -u postgres pgbackrest info
Output suppressed for testing
The info command operates on a single stanza or all stanzas. Text output is the default and gives a human-readable summary of backups for the stanza(s) requested. This format is subject to change with any release.
For machine-readable output use --output=json. The JSON output contains far more information than the text output and is kept stable unless a bug is found.
To speed up execution, limit the output to only progress information by specifying --detail-level=progress. Note that this skips all checks except for availability of the stanza.
Each stanza has a separate section and it is possible to limit output to a single stanza with the --stanza option. The stanza 'status' gives a brief indication of the stanza's health. If this is 'ok' then pgBackRest is functioning normally. If there are multiple repositories, then a status of 'mixed' indicates that the stanza is not in a healthy state on one or more of the repositories; in this case the state of the stanza will be detailed per repository. For cases in which an error on a repository occurred that is not one of the known error codes, then an error code of 'other' will be used and the full error details will be provided. The 'wal archive min/max' shows the minimum and maximum WAL currently stored in the archive and, in the case of multiple repositories, will be reported across all repositories unless the --repo option is set. Note that there may be gaps due to archive retention policies or other reasons.
The 'backup/expire running' and/or 'restore running' messages will appear beside the 'status' information if any of those commands are currently running on the host. Per-repo progress will be also reported in text output and a 'repo' array will be included in JSON output.
The backups are displayed oldest to newest. The oldest backup will always be a full backup (indicated by an F at the end of the label) but the newest backup can be full, differential (ends with D), or incremental (ends with I).
The 'timestamp start/stop' defines the time period when the backup ran. The 'timestamp stop' can be used to determine the backup to use when performing Point-In-Time Recovery. More information about Point-In-Time Recovery can be found in the Point-In-Time Recovery section.
The 'wal start/stop' defines the WAL range that is required to make the database consistent when restoring. The backup command will ensure that this WAL range is in the archive before completing.
The 'database size' is the full uncompressed size of the database while 'database backup size' is the amount of data in the database to actually back up (these will be the same for full backups).
The 'repo' indicates in which repository this backup resides. The 'backup set size' includes all the files from this backup and any referenced backups in the repository that are required to restore the database from this backup while 'backup size' includes only the files in this backup (these will also be the same for full backups). Repository sizes reflect compressed file sizes if compression is enabled in pgBackRest.
The 'backup reference total' summarizes the list of additional backups that are required to restore this backup. Use the --set option to display the complete reference list.
Restore a Backup
Backups can protect you from a number of disaster scenarios, the most common of which are hardware failure and data corruption. The easiest way to simulate data corruption is to remove an important PostgreSQL cluster file.
pg-primary ⇒ Stop the demo cluster and delete the pg_control file 
sudo systemctl stop postgresql-14.service
sudo -u postgres rm /var/lib/pgsql/14/data/global/pg_control
Starting the cluster without this important file will result in an error.
pg-primary ⇒ Attempt to start the corrupted demo cluster 
sudo systemctl start postgresql-14.service
sudo systemctl status postgresql-14.service
Output suppressed for testing
To restore a backup of the PostgreSQL cluster run pgBackRest with the restore command. The cluster needs to be stopped (in this case it is already stopped) and all files must be removed from the PostgreSQL data directory.
pg-primary ⇒ Remove old files from demo cluster 
sudo -u postgres find /var/lib/pgsql/14/data -mindepth 1 -delete
pg-primary ⇒ Restore the demo cluster and start PostgreSQL 
sudo -u postgres pgbackrest --stanza=demo restore
sudo systemctl start postgresql-14.service
This time the cluster started successfully since the restore replaced the missing pg_control file.
More information about the restore command can be found in the Restore section.
Monitoring
Monitoring is an important part of any production system. There are many tools available and pgBackRest can be monitored on any of them with a little work.
pgBackRest can output information about the repository in JSON format which includes a list of all backups for each stanza and WAL archive info.
In PostgreSQL
The PostgreSQL COPY command allows pgBackRest info to be loaded into a table. The following example wraps that logic in a function that can be used to perform real-time queries.
pg-primary ⇒ Load pgBackRest info function for PostgreSQL 
sudo -u postgres cat \
       /var/lib/pgsql/pgbackrest/doc/example/pgsql-pgbackrest-info.sql
Output suppressed for testing
sudo -u postgres psql -f \
       /var/lib/pgsql/pgbackrest/doc/example/pgsql-pgbackrest-info.sql
Now the monitor.pgbackrest_info() function can be used to determine the last successful backup time and archived WAL for a stanza.
pg-primary ⇒ Query last successful backup time and archived WAL 
sudo -u postgres cat \
       /var/lib/pgsql/pgbackrest/doc/example/pgsql-pgbackrest-query.sql
Output suppressed for testing
sudo -u postgres psql -f \
       /var/lib/pgsql/pgbackrest/doc/example/pgsql-pgbackrest-query.sql
Output suppressed for testing
Backup
When multiple repositories are configured, pgBackRest will backup to the highest priority repository (e.g. repo1) unless the --repo option is specified.
pgBackRest does not have a built-in scheduler so it's best to run it from cron or some other scheduling mechanism.
See Perform a Backup for more details and examples.
File Bundling
Bundling files together in the repository saves time during the backup and some space in the repository. This is especially pronounced when the repository is stored on an object store such as S3 or file systems with large block sizes. Per-file creation time on object stores is higher and very small files might cost as much to store as larger files.
The file bundling feature is enabled with the repo-bundle option.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-bundle
Config suppressed for testing
A full backup without file bundling will have 1000+ files in the backup path, but with bundling the total number of files is greatly reduced. An additional benefit is that zero-length files are not stored (except in the manifest), whereas in a normal backup each zero-length file is stored individually.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full backup
pg-primary ⇒ Check file total 
sudo -u postgres find /var/lib/pgbackrest/backup/demo/latest/ -type f | wc -l
Output suppressed for testing
The repo-bundle-size and repo-bundle-limit options can be used for tuning, though the defaults should be optimal in most cases.
While file bundling is generally more efficient, the downside is that it is more difficult to manually retrieve files from the repository. It may not be ideal for deduplicated storage since each full backup will arrange files in the bundles differently. Lastly, file bundles cannot be resumed, so be careful not to set repo-bundle-limit too high.
Block Incremental
Block incremental backups save space by only storing the parts of a file that have changed since the prior backup rather than storing the entire file.
The block incremental feature is enabled with the repo-block option and it works best when enabled for all backup types. File bundling must also be enabled.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-block
Config suppressed for testing
Backup Annotations
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
pg-primary ⇒ Perform a full backup with annotations 
sudo -u postgres pgbackrest --stanza=demo --annotation=source="demo backup" \
       --annotation=key=value --type=full backup
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output.
pg-primary ⇒ Get info for the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] info
Output suppressed for testing
Annotations included with the backup command can be added, modified, or removed afterwards using the annotate command.
pg-primary ⇒ Change backup annotations 
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] \
       --annotation=key= --annotation=new_key=new_value annotate
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] info
Output suppressed for testing
Retention
Generally it is best to retain as many backups as possible to provide a greater window for Point-in-Time Recovery, but practical concerns such as disk space must also be considered. Retention options remove older backups once they are no longer needed.
pgBackRest does full backup rotation based on the retention type which can be a count or a time period. When a count is specified, then expiration is not concerned with when the backups were created but with how many must be retained. Differential backups are count-based but will always be expired when the full backup they depend on is expired. Incremental backups are not expired by retention independently — they are always expired with their related full or differential backup. See sections Full Backup Retention and Differential Backup Retention for details and examples.
Archived WAL is retained by default for backups that have not expired, however, although not recommended, this schedule can be modified per repository with the retention-archive options. See section Archive Retention for details and examples.
The expire command is run automatically after each successful backup and can also be run by the user. When run by the user, expiration will occur as defined by the retention settings for each configured repository. If the --repo option is provided, expiration will occur only on the specified repository. Expiration can also be limited by the user to a specific backup set with the --set option and, unless the --repo option is specified, all repositories will be searched and any matching the set criteria will be expired. It should be noted that the archive retention schedule will be checked and performed any time the expire command is run.
Full Backup Retention
The repo1-retention-full-type determines how the option repo1-retention-full is interpreted; either as the count of full backups to be retained or how many days to retain full backups. New backups must be completed before expiration will occur — that means if repo1-retention-full-type=count and repo1-retention-full=2 then there will be three full backups stored before the oldest one is expired, or if repo1-retention-full-type=time and repo1-retention-full=20 then there must be one full backup that is at least 20 days old before expiration can occur.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-full
Config suppressed for testing
Backup repo1-retention-full=2 but currently there is only one full backup so the next full backup to run will not expire any full backups.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=detail backup
Output suppressed for testing
Archive is expired because WAL segments were generated before the oldest backup. These are not useful for recovery — only WAL segments generated after a backup can be used to recover that backup.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=info backup
Output suppressed for testing
The [Test Variable] full backup is expired and archive retention is based on the [Test Variable] which is now the oldest full backup.
Differential Backup Retention
Set repo1-retention-diff to the number of differential backups required. Differentials only rely on the prior full backup so it is possible to create a rolling set of differentials for the last day or more. This allows quick restores to recent points-in-time but reduces overall space consumption.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-diff
Config suppressed for testing
Backup repo1-retention-diff=1 so two differentials will need to be performed before one is expired. An incremental backup is added to demonstrate incremental expiration, which in this case depends on the differential expiration.
pg-primary ⇒ Perform differential and incremental backups 
sudo -u postgres pgbackrest --stanza=demo --type=diff backup
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
Now performing a differential backup will expire the previous differential and incremental backups leaving only one differential backup.
pg-primary ⇒ Perform a differential backup 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
Archive Retention
Although pgBackRest automatically removes archived WAL segments when expiring backups (the default expires WAL for full backups based on the repo1-retention-full option), it may be useful to expire archive more aggressively to save disk space. Note that full backups are treated as differential backups for the purpose of differential archive retention.
Expiring archive will never remove WAL segments that are required to make a backup consistent. However, since Point-in-Time-Recovery (PITR) only works on a continuous WAL stream, care should be taken when aggressively expiring archive outside of the normal backup expiration process. To determine what will be expired without actually expiring anything, the dry-run option can be provided on the command line with the expire command.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-diff
Config suppressed for testing
pg-primary ⇒ Perform differential backup 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
pg-primary ⇒ Expire archive 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=detail \
       --repo1-retention-archive-type=diff --repo1-retention-archive=1 expire
Output suppressed for testing
The [Test Variable] differential backup has archived WAL segments that must be retained to make the older backups consistent even though they cannot be played any further forward with PITR. WAL segments generated after [Test Variable] but before [Test Variable] are removed. WAL segments generated after the new backup [Test Variable] remain and can be used for PITR.
Since full backups are considered differential backups for the purpose of differential archive retention, if a full backup is now performed with the same settings, only the archive for that full backup is retained for PITR.
Restore
The restore command automatically defaults to selecting the latest backup from the first repository where backups exist (see Quick Start - Restore a Backup). The order in which the repositories are checked is dictated by the pgbackrest.conf (e.g. repo1 will be checked before repo2). To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.
When PITR of --type=time or --type=lsn is specified, then the target time or target lsn must be specified with the --target option. If a backup is not specified via the --set option, then the configured repositories will be checked, in order, for a backup that contains the requested time or lsn. If no matching backup is found, the latest backup from the first repository containing backups will be used for --type=time while no backup will be selected for --type=lsn. For other types of PITR, e.g. xid, the --set option must be provided if the target is prior to the latest backup. See Point-in-Time Recovery for more details and examples.
Replication slots are not included per recommendation of PostgreSQL. See Backing Up The Data Directory in the PostgreSQL documentation for more information.
The following sections introduce additional restore command features.
File Ownership
If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.
If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.
Delta Option
Restore a Backup in Quick Start required the database cluster directory to be cleaned before the restore could be performed. The delta option allows pgBackRest to automatically determine which files in the database cluster directory can be preserved and which ones need to be restored from the backup — it also removes files not present in the backup manifest so it will dispose of divergent changes. This is accomplished by calculating a SHA-1 cryptographic hash for each file in the database cluster directory. If the SHA-1 hash does not match the hash stored in the backup then that file will be restored. This operation is very efficient when combined with the process-max option. Since the PostgreSQL server is shut down during the restore, a larger number of processes can be used than might be desirable during a backup when the PostgreSQL server is running.
pg-primary ⇒ Stop the demo cluster, perform delta restore 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta \
       --log-level-console=detail restore
Output suppressed for testing
pg-primary ⇒ Restart PostgreSQL 
sudo systemctl start postgresql-14.service
Restore Selected Databases
There may be cases where it is desirable to selectively restore specific databases from a cluster backup. This could be done for performance reasons or to move selected databases to a machine that does not have enough space to restore the entire cluster backup.
To demonstrate this feature two databases are created: test1 and test2.
pg-primary ⇒ Create two test databases 
sudo -u postgres psql -c "create database test1;"
Output suppressed for testing
sudo -u postgres psql -c "create database test2;"
Output suppressed for testing
Each test database will be seeded with tables and data to demonstrate that recovery works with selective restore.
pg-primary ⇒ Create a test table in each database 
sudo -u postgres psql -c "create table test1_table (id int); \
       insert into test1_table (id) values (1);" test1
Output suppressed for testing
sudo -u postgres psql -c "create table test2_table (id int); \
       insert into test2_table (id) values (2);" test2
Output suppressed for testing
A fresh backup is run so pgBackRest is aware of the new databases.
pg-primary ⇒ Perform a backup 
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
One of the main reasons to use selective restore is to save space. The size of the test1 database is shown here so it can be compared with the disk utilization after a selective restore.
pg-primary ⇒ Show space used by test1 database 
sudo -u postgres du -sh /var/lib/pgsql/14/data/base/[Test Variable]
Output suppressed for testing
If the database to restore is not known, use the info command set option to discover databases that are part of the backup set.
pg-primary ⇒ Show database list for backup 
sudo -u postgres pgbackrest --stanza=demo \
       --set=[Test Variable] info
Output suppressed for testing
Stop the cluster and restore only the test2 database. Built-in databases (template0, template1, and postgres) are always restored.
WARNING:
Recovery may error unless --type=immediate is specified. This is because after consistency is reached PostgreSQL will flag zeroed pages as errors even for a full-page write. For PostgreSQL ≥ 13 the ignore_invalid_pages setting may be used to ignore invalid pages. In this case it is important to check the logs after recovery to ensure that no invalid pages were reported in the selected databases.
pg-primary ⇒ Restore from last backup including only the test2 database 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta \
       --db-include=test2 --type=immediate --target-action=promote restore
sudo systemctl start postgresql-14.service
Once recovery is complete the test2 database will contain all previously created tables and data.
pg-primary ⇒ Demonstrate that the test2 database was recovered 
sudo -u postgres psql -c "select * from test2_table;" test2
Output suppressed for testing
The test1 database, despite successful recovery, is not accessible. This is because the entire database was restored as sparse, zeroed files. PostgreSQL can successfully apply WAL on the zeroed files but the database as a whole will not be valid because key files contain no data. This is purposeful to prevent the database from being accidentally used when it might contain partial data that was applied during WAL replay.
pg-primary ⇒ Attempting to connect to the test1 database will produce an error 
sudo -u postgres psql -c "select * from test1_table;" test1
Output suppressed for testing
Since the test1 database is restored with sparse, zeroed files it will only require as much space as the amount of WAL that is written during recovery. While the amount of WAL generated during a backup and applied during recovery can be significant it will generally be a small fraction of the total database size, especially for large databases where this feature is most likely to be useful.
It is clear that the test1 database uses far less disk space during the selective restore than it would have if the entire database had been restored.
pg-primary ⇒ Show space used by test1 database after recovery 
sudo -u postgres du -sh /var/lib/pgsql/14/data/base/[Test Variable]
Output suppressed for testing
At this point the only action that can be taken on the invalid test1 database is drop database. pgBackRest does not automatically drop the database since this cannot be done until recovery is complete and the cluster is accessible.
pg-primary ⇒ Drop the test1 database 
sudo -u postgres psql -c "drop database test1;"
Output suppressed for testing
Now that the invalid test1 database has been dropped only the test2 and built-in databases remain.
pg-primary ⇒ List remaining databases 
sudo -u postgres psql -c "select oid, datname from pg_database order by oid;"
Output suppressed for testing
Point-in-Time Recovery
Restore a Backup in Quick Start performed default recovery, which is to play all the way to the end of the WAL stream. In the case of a hardware failure this is usually the best choice but for data corruption scenarios (whether machine or human in origin) Point-in-Time Recovery (PITR) is often more appropriate.
Point-in-Time Recovery (PITR) allows the WAL to be played from a backup to a specified lsn, time, transaction id, or recovery point. For common recovery scenarios time-based recovery is arguably the most useful. A typical recovery scenario is to restore a table that was accidentally dropped or data that was accidentally deleted. Recovering a dropped table is more dramatic so that's the example given here but deleted data would be recovered in exactly the same way.
pg-primary ⇒ Create a table with very important data 
sudo -u postgres psql -c "begin; \
       create table important_table (message text); \
       insert into important_table values ('Important Data'); \
       commit; \
       select * from important_table;"
Output suppressed for testing
It is important to represent the time as reckoned by PostgreSQL and to include timezone offsets. This reduces the possibility of unintended timezone conversions and an unexpected recovery result.
pg-primary ⇒ Get the time from PostgreSQL 
sudo -u postgres psql -Atc "select current_timestamp"
Output suppressed for testing
Now that the time has been recorded the table is dropped. In practice finding the exact time that the table was dropped is a lot harder than in this example. It may not be possible to find the exact time, but some forensic work should be able to get you close.
pg-primary ⇒ Drop the important table 
sudo -u postgres psql -c "begin; \
       drop table important_table; \
       commit; \
       select * from important_table;"
Output suppressed for testing
If the wrong backup is selected for restore then recovery to the required time target will fail. To demonstrate this a new incremental backup is performed where important_table does not exist.
pg-primary ⇒ Perform an incremental backup 
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
sudo -u postgres pgbackrest info
Output suppressed for testing
It will not be possible to recover the lost table from this backup since PostgreSQL can only play forward, not backward.
pg-primary ⇒ Attempt recovery from an incorrect backup 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta \
       --set=[Test Variable] --target-timeline=current \
       --type=time "--target=[Test Variable]" --target-action=promote restore
sudo systemctl start postgresql-14.service
sudo -u postgres cat /var/lib/pgsql/14/data/log/postgresql.log
Output suppressed for testing
A reliable method is to allow pgBackRest to automatically select a backup capable of recovery to the time target, i.e. a backup that ended before the specified time.
NOTE:
pgBackRest cannot automatically select a backup when the restore type is xid or name.
pg-primary ⇒ Restore the demo cluster to [Test Variable] 
sudo -u postgres pgbackrest --stanza=demo --delta \
       --type=time "--target=[Test Variable]" \
       --target-action=promote restore
sudo -u postgres cat /var/lib/pgsql/14/data/postgresql.auto.conf
Output suppressed for testing
pgBackRest has generated the recovery settings in postgresql.auto.conf so PostgreSQL can be started immediately. %f is how PostgreSQL specifies the WAL segment it needs and %p is the location where it should be copied. Once PostgreSQL has finished recovery the table will exist again and can be queried.
pg-primary ⇒ Start PostgreSQL and check that the important table exists 
sudo systemctl start postgresql-14.service
sudo -u postgres psql -c "select * from important_table"
Output suppressed for testing
The PostgreSQL log also contains valuable information. It will indicate the time and transaction where the recovery stopped and also give the time of the last transaction to be applied.
pg-primary ⇒ Examine the PostgreSQL log output 
sudo -u postgres cat /var/lib/pgsql/14/data/log/postgresql.log
Output suppressed for testing
Delete a Stanza
The stanza-delete command removes data in the repository associated with a stanza.
WARNING:
Use this command with caution — it will permanently remove all backups and archives from the pgBackRest repository for the specified stanza.
To delete a stanza:
  • Shut down the PostgreSQL cluster associated with the stanza (or use --force to override).
  • Run the stop command on the host where the stanza-delete command will be run.
  • Run the stanza-delete command.
Once the command successfully completes, it is the responsibility of the user to remove the stanza from all pgBackRest configuration files and/or environment variables.
A stanza may only be deleted from one repository at a time. To delete the stanza from multiple repositories, repeat the stanza-delete command for each repository while specifying the --repo option.
pg-primary ⇒ Stop PostgreSQL cluster to be removed 
sudo systemctl stop postgresql-14.service
pg-primary ⇒ Stop pgBackRest for the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stop
Output suppressed for testing
pg-primary ⇒ Delete the stanza from one repository 
sudo -u postgres pgbackrest --stanza=demo --repo=1 \
       --log-level-console=info stanza-delete
Output suppressed for testing
Multiple Repositories
Multiple repositories may be configured as demonstrated in S3 Support. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.
Some commands, e.g. stanza-create/stanza-upgrade, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.
Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.
The archive-push command will always push WAL to the archive in all configured repositories. When a repository cannot be reached, WAL will still be pushed to other repositories. However, for this to work effectively, archive-async=y must be enabled; otherwise, the other repositories can only get one WAL segment ahead of the unreachable repository. Also, note that if WAL cannot be pushed to any repository, then PostgreSQL will not remove it from the pg_wal directory, which may cause the volume to run out of space.
Backups need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
Azure-Compatible Object Store Support
pgBackRest supports locating repositories in Azure-compatible object stores. The container used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the container root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the container without conflicts.
WARNING:
Do not enable hierarchical namespace as this will cause errors during expire.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure Azure
Config suppressed for testing
Shared access signatures may be used by setting the repo2-azure-key-type option to sas and the repo2-azure-key option to the shared access signature token.
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
File creation time in Azure is relatively slow so backup/restore performance is improved by enabling file bundling.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=2 \
       --log-level-console=info backup
Output suppressed for testing
S3-Compatible Object Store Support
pgBackRest supports locating repositories in S3-compatible object stores. The bucket used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the bucket root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the bucket without conflicts.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure S3
Config suppressed for testing
NOTE:
The region and endpoint will need to be configured to where the bucket is located. The values given here are for the us-east-1 region.
A role should be created to run pgBackRest and the bucket permissions should be set as restrictively as possible. If the role is associated with an instance in AWS then pgBackRest will automatically retrieve temporary credentials when repo3-s3-key-type=auto, which means that keys do not need to be explicitly set in /etc/pgbackrest/pgbackrest.conf.
This sample Amazon S3 policy will restrict all reads and writes to the bucket and repository path. 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket"
            ],
            "Condition": {
                "StringEquals": {
                    "s3:prefix": [
                        "",
                        "demo-repo"
                    ],
                    "s3:delimiter": [
                        "/"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "demo-repo/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket/demo-repo/*"
            ]
        }
    ]
}
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
File creation time in S3 is relatively slow so backup/restore performance is improved by enabling file bundling.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=3 \
       --log-level-console=info backup
Output suppressed for testing
SFTP Support
pgBackRest supports locating repositories on SFTP hosts. SFTP file transfer is relatively slow so commands benefit by increasing process-max to parallelize file transfer.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure SFTP
Config suppressed for testing
When utilizing SFTP, if libssh2 is compiled against OpenSSH then repo4-sftp-public-key-file is optional.
pg-primary ⇒ Generate SSH keypair for SFTP backup 
sudo -u postgres mkdir -m 750 -p /var/lib/pgsql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/pgsql/.ssh/id_rsa_sftp \
       -t rsa -b 4096 -N "" -m PEM
sftp-server ⇒ Copy pg-primary SFTP backup public key to sftp-server 
sudo -u pgbackrest mkdir -m 750 -p /home/pgbackrest/.ssh
(sudo ssh root@pg-primary cat /var/lib/pgsql/.ssh/id_rsa_sftp.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Add sftp-server fingerprint to known_hosts file since repo4-sftp-host-key-check-type defaults to strict 
ssh-keyscan -H sftp-server >> /var/lib/pgsql/.ssh/known_hosts 2>/dev/null
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=4 \
       --log-level-console=info backup
Output suppressed for testing
GCS-Compatible Object Store Support
pgBackRest supports locating repositories in GCS-compatible object stores. The bucket used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the bucket root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the bucket without conflicts.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure GCS
Config suppressed for testing
When running in GCE set repo5-gcs-key-type=auto to automatically authenticate using the instance service account.
Commands are run exactly as if the repository were stored on a local disk.
File creation time in GCS is relatively slow so backup/restore performance is improved by enabling file bundling.
Target Time for Repository
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided.
To demonstrate this feature the demo stanza in the S3 repo is deleted.
pg-primary ⇒ Delete stanza in S3 repository 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo stop
Output suppressed for testing
sudo -u postgres pgbackrest --stanza=demo --repo=3 stanza-delete
Output suppressed for testing
Once the stanza is deleted the info command will show the repository in an error state.
pg-primary ⇒ Error on info 
sudo -u postgres pgbackrest --stanza=demo --repo=3 info
Output suppressed for testing
However, since the storage is versioned, it is possible to look at the repository at a time before the stanza was deleted. Finding the target time can be tricky depending on the situation, but in this case the time when the stanza was deleted can be determined by checking when backup.info was deleted.
pg-primary ⇒ List versions of backup.info in the bucket 
key=demo-repo/backup/demo/backup.info; \
       aws s3api list-object-versions --bucket demo-bucket \
       --prefix $key --output table \
       --query "sort_by([Versions[?Key=='$key'].{Action:'PUT', \
       Modified:LastModified,Object:Key}, \
       DeleteMarkers[?Key=='$key'].{Action:'DELETE', \
       Modified:LastModified,Object:Key}][],&Modified)"
Output suppressed for testing
Now the info command can be run with a target time that will show the repository before it was deleted.
pg-primary ⇒ Info with target time 
sudo -u postgres pgbackrest --stanza=demo --repo=3 \
       --repo-target-time="[Test Variable]" info
Output suppressed for testing
If the required backup is shown by the info command then it can be restored using the same target time.
pg-primary ⇒ Restore with target time 
sudo -u postgres pgbackrest --stanza=demo --repo=3 --delta \
       --repo-target-time="[Test Variable]" --log-level-console=info restore
Output suppressed for testing
sudo systemctl start postgresql-14.service
Dedicated Repository Host
The configuration described in Quickstart is suitable for simple installations but for enterprise configurations it is more typical to have a dedicated repository host where the backups and WAL archive files are stored. This separates the backups and WAL archive from the database server so database host failures have less impact. It is still a good idea to employ traditional backup software to backup the repository host.
On PostgreSQL hosts, pg1-path is required to be the path of the local PostgreSQL cluster and no pg1-host should be configured. When configuring a repository host, the pgbackrest configuration file must have the pg-host option configured to connect to the primary and standby (if any) hosts. The repository host has the only pgbackrest configuration that should be aware of more than one PostgreSQL host. Order does not matter, e.g. pg1-path/pg1-host, pg2-path/pg2-host can be primary or standby.
Installation
A new host named repository is created to store the cluster backups.
NOTE:
The pgBackRest version installed on the repository host must exactly match the version installed on the PostgreSQL host.
The pgbackrest user is created to own the pgBackRest repository. Any user can own the repository but it is best not to use postgres (if it exists) to avoid confusion.
NOTE:
When pgBackRest is installed from a package, a logrotate configuration such as /etc/logrotate.d/pgbackrest may be provided that rotates the logs as a specific user via the su directive (e.g. su postgres postgres). Since the files in /var/log/pgbackrest are owned by the user that runs pgBackRest (here pgbackrest), the su directive must be updated to match this user or logrotate will fail with a permission error.
repository ⇒ Create pgbackrest user 
sudo groupadd pgbackrest
sudo adduser -gpgbackrest -n pgbackrest
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
RHEL packages for pgBackRest are available at yum.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
repository ⇒ Install dependencies 
sudo yum install postgresql-libs libssh2
repository ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
repository ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown pgbackrest:pgbackrest /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown pgbackrest:pgbackrest /etc/pgbackrest/pgbackrest.conf
repository ⇒ Create the pgBackRest repository 
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown pgbackrest:pgbackrest /var/lib/pgbackrest
Configuration
pgBackRest can use TLS with client certificates to enable communication between the hosts. It is also possible to use SSH, see Setup SSH.
pgBackRest expects client/server certificates to be generated in the same way as PostgreSQL. See Secure TCP/IP Connections with TLS for detailed instructions on generating certificates.
The repository host must be configured with the pg-primary host/user and database path. The primary will be configured as pg1 to allow a standby to be added later.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg1-host/pg1-host-user and pg1-path
Config suppressed for testing
The database host must be configured with the repository host/user. The default for the repo1-host-user option is pgbackrest. If the postgres user does restores on the repository host it is best not to also allow the postgres user to perform backups. However, the postgres user can read the repository directly if it is in the same group as the pgbackrest user.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-host/repo1-host-user
Config suppressed for testing
PostgreSQL configuration may be found in the Configure Archiving section.
Commands are run the same as on a single host configuration except that some commands such as backup and expire are run from the repository host instead of the database host.
Setup TLS Server
The pgBackRest TLS server must be configured and started on each host.
repository ⇒ Setup pgBackRest Server 
sudo cat /etc/systemd/system/pgbackrest.service
Output suppressed for testing
sudo systemctl enable pgbackrest
sudo systemctl start pgbackrest
pg-primary ⇒ Setup pgBackRest Server 
sudo cat /etc/systemd/system/pgbackrest.service
Output suppressed for testing
sudo systemctl enable pgbackrest
sudo systemctl start pgbackrest
Create and Check Stanza
Create the stanza in the new repository.
repository ⇒ Create the stanza 
sudo -u pgbackrest pgbackrest --stanza=demo stanza-create
Output suppressed for testing
Check that the configuration is correct on both the database and repository hosts. More information about the check command can be found in Check the Configuration.
pg-primary ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
repository ⇒ Check the configuration 
sudo -u pgbackrest pgbackrest --stanza=demo check
Output suppressed for testing
Perform a Backup
To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command on the repository host.
repository ⇒ Backup the demo cluster 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
Since a new repository was created on the repository host the warning about the incremental backup changing to a full backup was emitted.
Restore a Backup
To perform a restore of the PostgreSQL cluster run pgBackRest with the restore command on the database host.
pg-primary ⇒ Stop the demo cluster, restore, and restart PostgreSQL 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta restore
sudo systemctl start postgresql-14.service
Parallel Backup / Restore
pgBackRest offers parallel processing to improve performance of compression and transfer. The number of processes to be used for this feature is set using the --process-max option.
It is usually best not to use more than 25% of available CPUs for the backup command. Backups don't have to run that fast as long as they are performed regularly and the backup process should not impact database performance, if at all possible.
The restore command can and should use all available CPUs because during a restore the PostgreSQL cluster is shut down and there is generally no other important work being done on the host. If the host contains multiple clusters then that should be considered when setting restore parallelism.
repository ⇒ Perform a backup with single process 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest to use multiple backup processes
Config suppressed for testing
repository ⇒ Perform a backup with multiple processes 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
repository ⇒ Get backup info for the demo cluster 
sudo -u pgbackrest pgbackrest info
Output suppressed for testing
The performance of the last backup should be improved by using multiple processes. For very small backups the difference may not be very apparent, but as the size of the database increases so will time savings.
Starting and Stopping
If a standby is promoted for testing, or a test cluster is restored from a production backup, then it is a good idea to prevent those clusters from writing to pgBackRest repositories. This can be accomplished with the stop command.
The commands that write and are blocked by stop are: archive-push, backup, expire, stanza-create, and stanza-upgrade. Note that stanza-delete is an exception to this rule (see Delete a Stanza for more details).
pg-primary ⇒ Stop pgBackRest write commands 
sudo -u postgres pgbackrest stop
New pgBackRest write commands will no longer run.
repository ⇒ Attempt a backup 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
Specify the --force option to terminate any pgBackRest write commands that are currently running. This includes asynchronous archive-get (though it will run again if PostgreSQL requires it). If pgBackRest is already stopped then stopping again will generate a warning.
pg-primary ⇒ Stop the pgBackRest services again 
sudo -u postgres pgbackrest stop
Output suppressed for testing
Start pgBackRest write commands again with the start command. Write commands that were in progress before the stop will not automatically start again, but they are now allowed to start.
pg-primary ⇒ Start pgBackRest write commands 
sudo -u postgres pgbackrest start
It is also possible to stop pgBackRest for a single stanza.
pg-primary ⇒ Stop pgBackRest write commands for the demo stanza 
sudo -u postgres pgbackrest --stanza=demo stop
New pgBackRest write commands for the specified stanza will no longer run.
repository ⇒ Attempt a backup 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
The stanza must also be specified when starting pgBackRest write commands for a single stanza.
pg-primary ⇒ Start pgBackRest write commands for the demo stanza 
sudo -u postgres pgbackrest --stanza=demo start
Replication
Replication allows multiple copies of a PostgreSQL cluster (called standbys) to be created from a single primary. The standbys are useful for balancing reads and to provide redundancy in case the primary host fails.
Installation
A new host named pg-standby is created to run the standby.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
RHEL packages for pgBackRest are available at yum.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-standby ⇒ Install dependencies 
sudo yum install postgresql-libs libssh2
pg-standby ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-standby ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
Hot Standby
A hot standby performs replication using the WAL archive and allows read-only queries.
pgBackRest configuration is very similar to pg-primary except that the standby recovery type will be used to keep the cluster in recovery mode when the end of the WAL stream has been reached.
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest on the standby
Config suppressed for testing
pg-standby ⇒ Setup pgBackRest Server 
sudo cat /etc/systemd/system/pgbackrest.service
Output suppressed for testing
sudo systemctl enable pgbackrest
sudo systemctl start pgbackrest
Create the path where PostgreSQL will be restored.
pg-standby ⇒ Create PostgreSQL path 
sudo -u postgres mkdir -p -m 700 /var/lib/pgsql/14/data
Now the standby can be created with the restore command.
IMPORTANT:
If the cluster is intended to be promoted without becoming the new primary (e.g. for reporting or testing), use --archive-mode=off or set archive_mode=off in postgresql.conf to disable archiving. If archiving is not disabled then the repository may be polluted with WAL that can make restores more difficult.
pg-standby ⇒ Restore the demo standby cluster 
sudo -u postgres pgbackrest --stanza=demo --type=standby restore
sudo -u postgres cat /var/lib/pgsql/14/data/postgresql.auto.conf
Output suppressed for testing
The configuration is in case the standby is promoted to a primary.
pg-standby:/var/lib/pgsql/14/data/postgresql.conf ⇒ Configure PostgreSQL
Config suppressed for testing
pg-standby ⇒ Start PostgreSQL 
sudo systemctl start postgresql-14.service
The PostgreSQL log gives valuable information about the recovery. Note especially that the cluster has entered standby mode and is ready to accept read-only connections.
pg-standby ⇒ Examine the PostgreSQL log output for log messages indicating success 
sudo -u postgres cat /var/lib/pgsql/14/data/log/postgresql.log
Output suppressed for testing
An easy way to test that replication is properly configured is to create a table on pg-primary.
pg-primary ⇒ Create a new table on the primary 
sudo -u postgres psql -c " \
       begin; \
       create table replicated_table (message text); \
       insert into replicated_table values ('Important Data'); \
       commit; \
       select * from replicated_table";
Output suppressed for testing
And then query the same table on pg-standby.
pg-standby ⇒ Query new table on the standby 
sudo -u postgres psql -c "select * from replicated_table;"
Output suppressed for testing
So, what went wrong? Since PostgreSQL is pulling WAL segments from the archive to perform replication, changes won't be seen on the standby until the WAL segment that contains those changes is pushed from pg-primary.
This can be done manually by calling pg_switch_wal() which pushes the current WAL segment to the archive (a new WAL segment is created to contain further changes).
pg-primary ⇒ Call pg_switch_wal() 
sudo -u postgres psql -c "select *, current_timestamp from pg_switch_wal()";
Output suppressed for testing
Now after a short delay the table will appear on pg-standby.
pg-standby ⇒ Now the new table exists on the standby (may require a few retries) 
sudo -u postgres psql -c " \
       select *, current_timestamp from replicated_table"
Output suppressed for testing
Check the standby configuration for access to the repository.
pg-standby ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Streaming Replication
Instead of relying solely on the WAL archive, streaming replication makes a direct connection to the primary and applies changes as soon as they are made on the primary. This results in much less lag between the primary and standby.
Streaming replication requires a user with the replication privilege.
pg-primary ⇒ Create replication user 
sudo -u postgres psql -c " \
       create user replicator password 'jw8s0F4' replication";
Output suppressed for testing
The pg_hba.conf file must be updated to allow the standby to connect as the replication user. Be sure to replace the IP address below with the actual IP address of your pg-standby. A reload will be required after modifying the pg_hba.conf file.
pg-primary ⇒ Create pg_hba.conf entry for replication user 
sudo -u postgres sh -c 'echo \
       "host    replication     replicator      {[host-pg2-ip]}/32           md5" \
       >> /var/lib/pgsql/14/data/pg_hba.conf'
sudo systemctl reload postgresql-14.service
The standby needs to know how to contact the primary so the primary_conninfo setting will be configured in pgBackRest.
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Set primary_conninfo
Config suppressed for testing
It is possible to configure a password in the primary_conninfo setting but using a .pgpass file is more flexible and secure.
pg-standby ⇒ Configure the replication password in the .pgpass file. 
sudo -u postgres sh -c 'echo \
       "{[host-pg1-ip]}:*:replication:replicator:jw8s0F4" \
       >> /var/lib/pgsql/.pgpass'
sudo -u postgres chmod 600 /var/lib/pgsql/.pgpass
Now the standby can be created with the restore command.
pg-standby ⇒ Stop PostgreSQL and restore the demo standby cluster 
sudo systemctl stop postgresql-14.service
sudo -u postgres pgbackrest --stanza=demo --delta --type=standby restore
sudo -u postgres cat /var/lib/pgsql/14/data/postgresql.auto.conf
Output suppressed for testing
NOTE:
The primary_conninfo setting has been written into the postgresql.auto.conf file because it was configured as a recovery-option in pgbackrest.conf. The --type=preserve option can be used with the restore to leave the existing postgresql.auto.conf file in place if that behavior is preferred.
By default RHEL stores the postgresql.conf file in the PostgreSQL data directory. That means the change made to postgresql.conf was overwritten by the last restore and the hot_standby setting must be enabled again. Other solutions to this problem are to store the postgresql.conf file elsewhere or to enable the hot_standby setting on the pg-primary host where it will be ignored.
pg-standby:/var/lib/pgsql/14/data/postgresql.conf ⇒ Enable hot_standby
Config suppressed for testing
pg-standby ⇒ Start PostgreSQL 
sudo systemctl start postgresql-14.service
The PostgreSQL log will confirm that streaming replication has started.
pg-standby ⇒ Examine the PostgreSQL log output for log messages indicating success 
sudo -u postgres cat /var/lib/pgsql/14/data/log/postgresql.log
Output suppressed for testing
Now when a table is created on pg-primary it will appear on pg-standby quickly and without the need to call pg_switch_wal().
pg-primary ⇒ Create a new table on the primary 
sudo -u postgres psql -c " \
       begin; \
       create table stream_table (message text); \
       insert into stream_table values ('Important Data'); \
       commit; \
       select *, current_timestamp from stream_table";
Output suppressed for testing
pg-standby ⇒ Query table on the standby 
sudo -u postgres psql -c " \
       select *, current_timestamp from stream_table"
Output suppressed for testing
Multiple Stanzas
pgBackRest supports multiple stanzas. The most common usage is sharing a repository host among multiple stanzas.
Installation
A new host named pg-alt is created to run the new primary.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
RHEL packages for pgBackRest are available at yum.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-alt ⇒ Install dependencies 
sudo yum install postgresql-libs libssh2
pg-alt ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-alt ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
Configuration
pgBackRest configuration is nearly identical to pg-primary except that the demo-alt stanza will be used so backups and archive will be stored in a separate location.
pg-alt:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest on the new primary
Config suppressed for testing
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg1-host/pg1-host-user and pg1-path
Config suppressed for testing
pg-alt ⇒ Setup pgBackRest Server 
sudo cat /etc/systemd/system/pgbackrest.service
Output suppressed for testing
sudo systemctl enable pgbackrest
sudo systemctl start pgbackrest
Setup Demo Cluster
pg-alt ⇒ Create the demo cluster 
sudo -u postgres /usr/pgsql-14/bin/initdb \
       -D /var/lib/pgsql/14/data -k -A peer
pg-alt:/var/lib/pgsql/14/data/postgresql.conf ⇒ Configure PostgreSQL settings
Config suppressed for testing
pg-alt ⇒ Start the demo cluster 
sudo systemctl restart postgresql-14.service
Create the Stanza and Check Configuration
The stanza-create command must be run to initialize the stanza. It is recommended that the check command be run after stanza-create to ensure archiving and backups are properly configured.
pg-alt ⇒ Create the stanza and check the configuration 
sudo -u postgres pgbackrest --stanza=demo-alt --log-level-console=info stanza-create
Output suppressed for testing
sudo -u postgres pgbackrest --log-level-console=info check
Output suppressed for testing
If the check command is run from the repository host then all stanzas will be checked.
repository ⇒ Check the configuration for all stanzas 
sudo -u pgbackrest pgbackrest --log-level-console=info check
Output suppressed for testing
Asynchronous Archiving
Asynchronous archiving is enabled with the archive-async option. This option enables asynchronous operation for both the archive-push and archive-get commands.
A spool path is required. The commands will store transient data here but each command works quite a bit differently so spool path usage is described in detail in each section.
pg-primary ⇒ Create the spool directory 
sudo mkdir -p -m 750 /var/spool/pgbackrest
sudo chown postgres:postgres /var/spool/pgbackrest
pg-standby ⇒ Create the spool directory 
sudo mkdir -p -m 750 /var/spool/pgbackrest
sudo chown postgres:postgres /var/spool/pgbackrest
The spool path must be configured and asynchronous archiving enabled. Asynchronous archiving automatically confers some benefit by reducing the number of connections made to remote storage, but setting process-max can drastically improve performance by parallelizing operations. Be sure not to set process-max so high that it affects normal database operations.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the spool path and asynchronous archiving
Config suppressed for testing
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the spool path and asynchronous archiving
Config suppressed for testing
NOTE:
process-max is configured using command sections so that the option is not used by backup and restore. This also allows different values for archive-push and archive-get.
For demonstration purposes streaming replication will be broken to force PostgreSQL to get WAL using the restore_command.
pg-primary ⇒ Break streaming replication by changing the replication password 
sudo -u postgres psql -c "alter user replicator password 'bogus'"
Output suppressed for testing
pg-standby ⇒ Restart standby to break connection 
sudo systemctl restart postgresql-14.service
Archive Push
The asynchronous archive-push command offloads WAL archiving to a separate process (or processes) to improve throughput. It works by looking ahead to see which WAL segments are ready to be archived beyond the request that PostgreSQL is currently making via the archive_command. WAL segments are transferred to the archive directly from the pg_xlog/pg_wal directory and success is only returned by the archive_command when the WAL segment has been safely stored in the archive.
The spool path holds the current status of WAL archiving. Status files written into the spool directory are typically zero length and should consume a minimal amount of space (a few MB at most) and very little IO. All the information in this directory can be recreated so it is not necessary to preserve the spool directory if the cluster is moved to new hardware.
IMPORTANT:
In the original implementation of asynchronous archiving, WAL segments were copied to the spool directory before compression and transfer. The new implementation copies WAL directly from the pg_xlog directory. If asynchronous archiving was utilized in v1.12 or prior, read the v1.13 release notes carefully before upgrading.
The [stanza]-archive-push-async.log file can be used to monitor the activity of the asynchronous process. A good way to test this is to quickly push a number of WAL segments.
pg-primary ⇒ Test parallel asynchronous archiving 
sudo -u postgres psql -c " \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal();"
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Now the log file will contain parallel, asynchronous activity.
pg-primary ⇒ Check results in the log 
sudo -u postgres cat /var/log/pgbackrest/demo-archive-push-async.log
Output suppressed for testing
Archive Get
The asynchronous archive-get command maintains a local queue of WAL to improve throughput. If a WAL segment is not found in the queue it is fetched from the repository along with enough consecutive WAL to fill the queue. The maximum size of the queue is defined by archive-get-queue-max. Whenever the queue is less than half full more WAL will be fetched to fill it.
Asynchronous operation is most useful in environments that generate a lot of WAL or have a high latency connection to the repository storage (i.e., S3 or other object stores). In the case of a high latency connection it may be a good idea to increase process-max.
The [stanza]-archive-get-async.log file can be used to monitor the activity of the asynchronous process.
pg-standby ⇒ Check results in the log 
sudo -u postgres cat /var/log/pgbackrest/demo-archive-get-async.log
Output suppressed for testing
pg-primary ⇒ Fix streaming replication by changing the replication password 
sudo -u postgres psql -c "alter user replicator password 'jw8s0F4'"
Output suppressed for testing
Backup from a Standby
pgBackRest can perform backups on a standby instead of the primary. Standby backups require the pg-standby host to be configured and the backup-standby option enabled. If more than one standby is configured then the first running standby found will be used for the backup.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg2-host/pg2-host-user and pg2-path
Config suppressed for testing
Both the primary and standby databases are required to perform the backup, though the vast majority of the files will be copied from the standby to reduce load on the primary. The database hosts can be configured in any order. pgBackRest will automatically determine which is the primary and which is the standby.
repository ⇒ Backup the demo cluster from pg2 
sudo -u pgbackrest pgbackrest --stanza=demo --log-level-console=detail backup
Output suppressed for testing
This incremental backup shows that most of the files are copied from the pg-standby host and only a few are copied from the pg-primary host.
pgBackRest creates a standby backup that is identical to a backup performed on the primary. It does this by starting/stopping the backup on the pg-primary host, copying only files that are replicated from the pg-standby host, then copying the remaining few files from the pg-primary host. This means that logs and statistics from the primary database will be included in the backup.
Upgrading PostgreSQL
Immediately after upgrading PostgreSQL to a newer major version, the pg-path for all pgBackRest configurations must be set to the new database location and the stanza-upgrade command run. If there is more than one repository configured on the host, the stanza will be upgraded on each. If the database is offline use the --no-online option.
The following instructions are not meant to be a comprehensive guide for upgrading PostgreSQL, rather they outline the general process for upgrading a primary and standby with the intent of demonstrating the steps required to reconfigure pgBackRest. It is recommended that a backup be taken prior to upgrading.
pg-primary ⇒ Stop old cluster 
sudo systemctl stop postgresql-14.service
Stop the old cluster on the standby since it will be restored from the newly upgraded cluster.
pg-standby ⇒ Stop old cluster 
sudo systemctl stop postgresql-14.service
Create the new cluster and perform upgrade.
pg-primary ⇒ Create new cluster and perform the upgrade 
sudo -u postgres /usr/pgsql-15/bin/initdb \
       -D /var/lib/pgsql/15/data -k -A peer
sudo -u postgres sh -c 'cd /var/lib/pgsql && \
       /usr/pgsql-15/bin/pg_upgrade \
       --old-bindir=/usr/pgsql-14/bin \
       --new-bindir=/usr/pgsql-15/bin \
       --old-datadir=/var/lib/pgsql/14/data \
       --new-datadir=/var/lib/pgsql/15/data \
       --old-options=" -c config_file=/var/lib/pgsql/14/data/postgresql.conf" \
       --new-options=" -c config_file=/var/lib/pgsql/15/data/postgresql.conf"'
Output suppressed for testing
Configure the new cluster settings and port.
pg-primary:/var/lib/pgsql/15/data/postgresql.conf ⇒ Configure PostgreSQL
Config suppressed for testing
Update the pgBackRest configuration on all systems to point to the new cluster.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade the pg1-path
Config suppressed for testing
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade the pg-path
Config suppressed for testing
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade pg1-path and pg2-path, disable backup from standby
Config suppressed for testing
pg-primary ⇒ Copy hba configuration 
sudo cp /var/lib/pgsql/14/data/pg_hba.conf \
       /var/lib/pgsql/15/data/pg_hba.conf
Before starting the new cluster, the stanza-upgrade command must be run.
pg-primary ⇒ Upgrade the stanza 
sudo -u postgres pgbackrest --stanza=demo --no-online \
       --log-level-console=info stanza-upgrade
Output suppressed for testing
Start the new cluster and confirm it is successfully installed.
pg-primary ⇒ Start new cluster 
sudo systemctl start postgresql-15.service
Output suppressed for testing
Test configuration using the check command.
pg-primary ⇒ Check configuration 
sudo systemctl status postgresql-15.service
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
Remove the old cluster.
pg-primary ⇒ Remove old cluster 
sudo rm -rf /var/lib/pgsql/14/data
Install the new PostgreSQL binaries on the standby and create the cluster.
pg-standby ⇒ Remove old cluster and create the new cluster 
sudo rm -rf /var/lib/pgsql/14/data
sudo -u postgres mkdir -p -m 700 /usr/pgsql-15/bin
Run the check on the repository host. The warning regarding the standby being down is expected since the standby cluster is down. Running this command demonstrates that the repository server is aware of the standby and is configured properly for the primary server.
repository ⇒ Check configuration 
sudo -u pgbackrest pgbackrest --stanza=demo check
Output suppressed for testing
Run a full backup on the new cluster and then restore the standby from the backup. The backup type will automatically be changed to full if incr or diff is requested.
repository ⇒ Run a full backup 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
pg-standby ⇒ Restore the demo standby cluster 
sudo -u postgres pgbackrest --stanza=demo --type=standby restore
pg-standby ⇒ Start PostgreSQL and check the pgBackRest configuration 
sudo systemctl start postgresql-15.service
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
Backup from standby can be enabled now that the standby is restored.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Reenable backup from standby
Config suppressed for testing
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated August 17, 2026
././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7738323 pgbackrest-2.59.1/doc/html/user-guide.html0000640000175100017510000061263615240547776020106 0ustar00runnerrunner pgBackRest User Guide - Debian & Ubuntu
pgBackRest User Guide
Debian & Ubuntu
Home
User Guides
Releases
Configuration
Commands
FAQ
Table of Contents
Introduction
Concepts
Backup
Restore
Write Ahead Log (WAL)
Encryption
Upgrading pgBackRest
Upgrading pgBackRest from v2.x to v2.y
Build
Installation
Quick Start
Setup Demo Cluster
Configure Cluster Stanza
Create the Repository
Configure Archiving
Configure Retention
Configure Repository Encryption
Create the Stanza
Check the Configuration
Performance Tuning
Perform a Backup
Schedule a Backup
Backup Information
Restore a Backup
Monitoring
In PostgreSQL
Using jq
Backup
File Bundling
Block Incremental
Backup Annotations
Retention
Full Backup Retention
Differential Backup Retention
Archive Retention
Restore
File Ownership
Delta Option
Restore Selected Databases
Point-in-Time Recovery
Delete a Stanza
Multiple Repositories
Azure-Compatible Object Store Support
S3-Compatible Object Store Support
SFTP Support
GCS-Compatible Object Store Support
Target Time for Repository
Dedicated Repository Host
Installation
Setup Passwordless SSH
Configuration
Create and Check Stanza
Perform a Backup
Restore a Backup
Parallel Backup / Restore
Starting and Stopping
Replication
Installation
Setup Passwordless SSH
Hot Standby
Streaming Replication
Multiple Stanzas
Installation
Setup Passwordless SSH
Configuration
Setup Demo Cluster
Create the Stanza and Check Configuration
Asynchronous Archiving
Archive Push
Archive Get
Backup from a Standby
Upgrading PostgreSQL
Introduction
This user guide is intended to be followed sequentially from beginning to end — each section depends on the last. For example, the Restore section relies on setup that is performed in the Quick Start section. Once pgBackRest is up and running then skipping around is possible but following the user guide in order is recommended the first time through.
Although the examples in this guide are targeted at Debian/Ubuntu and PostgreSQL 17, it should be fairly easy to apply the examples to any Unix distribution and PostgreSQL version. The only OS-specific commands are those to create, start, stop, and drop PostgreSQL clusters. The pgBackRest commands will be the same on any Unix system though the location of the executable may vary. While pgBackRest strives to operate consistently across versions of PostgreSQL, there are subtle differences between versions of PostgreSQL that may show up in this guide when illustrating certain examples, e.g. PostgreSQL path/file names and settings.
Configuration information and documentation for PostgreSQL can be found in the PostgreSQL Manual.
A somewhat novel approach is taken to documentation in this user guide. Each command is run on a virtual machine when the documentation is built from the XML source. This means you can have a high confidence that the commands work correctly in the order presented. Output is captured and displayed below the command when appropriate. If the output is not included it is because it was deemed not relevant or was considered a distraction from the narrative.
All commands are intended to be run as an unprivileged user that has sudo privileges for both the root and postgres users. It's also possible to run the commands directly as their respective users without modification and in that case the sudo commands can be stripped off.
Concepts
The following concepts are defined as they are relevant to pgBackRest, PostgreSQL, and this user guide.
Backup
A backup is a consistent copy of a database cluster that can be restored to recover from a hardware failure, to perform Point-In-Time Recovery, or to bring up a new standby.
Full Backup: pgBackRest copies the entire contents of the database cluster to the backup. The first backup of the database cluster is always a Full Backup. pgBackRest is always able to restore a full backup directly. The full backup does not depend on any files outside of the full backup for consistency.
Differential Backup: pgBackRest copies only those database cluster files that have changed since the last full backup. pgBackRest restores a differential backup by copying all of the files in the chosen differential backup and the appropriate unchanged files from the previous full backup. The advantage of a differential backup is that it requires less disk space than a full backup, however, the differential backup and the full backup must both be valid to restore the differential backup.
Incremental Backup: pgBackRest copies only those database cluster files that have changed since the last backup (which can be another incremental backup, a differential backup, or a full backup). As an incremental backup only includes those files changed since the prior backup, they are generally much smaller than full or differential backups. As with the differential backup, the incremental backup depends on other backups to be valid to restore the incremental backup. Since the incremental backup includes only those files since the last backup, all prior incremental backups back to the prior differential, the prior differential backup, and the prior full backup must all be valid to perform a restore of the incremental backup. If no differential backup exists then all prior incremental backups back to the prior full backup, which must exist, and the full backup itself must be valid to restore the incremental backup.
Restore
A restore is the act of copying a backup to a system where it will be started as a live database cluster. A restore requires the backup files and one or more WAL segments in order to work correctly.
Write Ahead Log (WAL)
WAL is the mechanism that PostgreSQL uses to ensure that no committed changes are lost. Transactions are written sequentially to the WAL and a transaction is considered to be committed when those writes are flushed to disk. Afterwards, a background process writes the changes into the main database cluster files (also known as the heap). In the event of a crash, the WAL is replayed to make the database consistent.
WAL is conceptually infinite but in practice is broken up into individual 16MB files called segments. WAL segments follow the naming convention 0000000100000A1E000000FE where the first 8 hexadecimal digits represent the timeline and the next 16 digits are the logical sequence number (LSN).
Encryption
Encryption is the process of converting data into a format that is unrecognizable unless the appropriate password (also referred to as passphrase) is provided.
pgBackRest will encrypt the repository based on a user-provided password, thereby preventing unauthorized access to data stored within the repository.
Upgrading pgBackRest
Upgrading pgBackRest from v2.x to v2.y
Upgrading from v2.x to v2.y is straight-forward. The repository format has not changed, so for most installations it is simply a matter of installing binaries for the new version. It is also possible to downgrade if you have not used new features that are unsupported by the older version.
IMPORTANT:
The local and remote pgBackRest versions must match exactly so they should be upgraded together. If there is a mismatch, WAL archiving and backups will not function until the versions match. In such a case, the following error will be reported: [ProtocolError] expected value '2.x' for greeting key 'version' but got '2.y'.
Build
Installing pgBackRest from a package is preferable to building from source. See Installation for more information about packages.
When building from source it is best to use a build host rather than building on production. Many of the tools required for the build should generally not be installed in production. pgBackRest consists of a single executable so it is easy to copy to a new host once it is built.
build ⇒ Download version 2.59.1 of pgBackRest to /build path 
mkdir -p /build
curl -fsSL \
       https://github.com/pgbackrest/pgbackrest/releases/download/release%2F2.59.1/pgbackrest-2.59.1.tar.gz | \
       tar zx -C /build
build ⇒ Install build dependencies 
sudo apt-get install python3-distutils meson gcc libpq-dev libssl-dev libxml2-dev \
       pkg-config liblz4-dev libzstd-dev libbz2-dev libz-dev libssh2-1-dev libsystemd-dev
build ⇒ Configure and compile pgBackRest 
meson setup /build/pgbackrest /build/pgbackrest-2.59.1
ninja -C /build/pgbackrest
build ⇒ Optionally run smoke tests to verify pgBackRest was built correctly 
meson test -C /build/pgbackrest --suite smoke
Output suppressed for testing
Installation
A new host named pg-primary is created to contain the demo cluster and run pgBackRest examples.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
Debian/Ubuntu packages for pgBackRest are available at apt.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-primary ⇒ Install dependencies 
sudo apt-get install postgresql-client libxml2 libssh2-1
pg-primary ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-primary ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
pgBackRest should now be properly installed but it is best to check. If any dependencies were missed then you will get an error when running pgBackRest from the command line.
pg-primary ⇒ Make sure the installation worked 
sudo -u postgres pgbackrest
Output suppressed for testing
Quick Start
The Quick Start section will cover basic configuration of pgBackRest and PostgreSQL and introduce the backup, restore, and info commands.
Setup Demo Cluster
Creating the demo cluster is optional but is strongly recommended, especially for new users, since the example commands in the user guide reference the demo cluster; the examples assume the demo cluster is running on the default port (i.e. 5432). The cluster will not be started until a later section because there is still some configuration to do.
pg-primary ⇒ Create the demo cluster 
sudo -u postgres /usr/lib/postgresql/17/bin/initdb \
       -D /var/lib/postgresql/17/demo -k -A peer
sudo pg_createcluster 17 demo
Output suppressed for testing
Configure Cluster Stanza
A stanza is the configuration for a PostgreSQL database cluster that defines where it is located, how it will be backed up, archiving options, etc. Most db servers will only have one PostgreSQL database cluster and therefore one stanza, whereas backup servers will have a stanza for every database cluster that needs to be backed up.
It is tempting to name the stanza after the primary cluster but a better name describes the databases contained in the cluster. Because the stanza name will be used for the primary and all replicas it is more appropriate to choose a name that describes the actual function of the cluster, such as app or dw, rather than the local cluster name, such as main or prod.
The name 'demo' describes the purpose of this cluster accurately so that will also make a good stanza name.
pgBackRest needs to know where the base data directory for the PostgreSQL cluster is located. The path can be requested from PostgreSQL directly but in a recovery scenario the PostgreSQL process will not be available. During backups the value supplied to pgBackRest will be compared against the path that PostgreSQL is running on and they must be equal or the backup will return an error. Make sure that pg-path is exactly equal to data_directory as reported by PostgreSQL.
By default Debian/Ubuntu stores clusters in /var/lib/postgresql/[version]/[cluster] so it is easy to determine the correct path for the data directory.
When creating the /etc/pgbackrest/pgbackrest.conf file, the database owner (usually postgres) must be granted read privileges.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the PostgreSQL cluster data directory
Config suppressed for testing
pgBackRest configuration files follow a Windows INI-like convention. Sections are denoted by text in brackets and key/value pairs are contained in each section. Lines beginning with # are ignored and can be used as comments, but end-of-line comments following a value on the same line are not supported. Quoting is not supported and whitespace is trimmed from keys and values. Sections will be merged if they appear more than once.
There are multiple ways the pgBackRest configuration files can be loaded:
  • config and config-include-path are default: the default config file will be loaded, if it exists, and *.conf files in the default config include path will be appended, if they exist.
  • config option is specified: only the specified config file will be loaded and is expected to exist.
  • config-include-path is specified: *.conf files in the config include path will be loaded and the path is required to exist. The default config file will be be loaded if it exists. If it is desirable to load only the files in the specified config include path, then the --no-config option can also be passed.
  • config and config-include-path are specified: using the user-specified values, the config file will be loaded and *.conf files in the config include path will be appended. The files are expected to exist.
  • config-path is specified: this setting will override the base path for the default location of the config file and/or the base path of the default config-include-path setting unless the config and/or config-include-path option is explicitly set.
Files are concatenated as if they were one big file and each file must be valid individually. This means sections must be specified in each file where they are needed to store a key/value. Order doesn't matter but there is precedence based on sections. The precedence (highest to lowest) is:
  • [stanza:command]
  • [stanza]
  • [global:command]
  • [global]
NOTE:
--config, --config-include-path and --config-path are command-line only options.
pgBackRest can also be configured using environment variables as described in the command reference.
pg-primary ⇒ Configure log-path using the environment 
sudo -u postgres bash -c ' \
       export PGBACKREST_LOG_PATH=/path/set/by/env && \
       pgbackrest --log-level-console=error help backup log-path'
Output suppressed for testing
Create the Repository
The repository is where pgBackRest stores backups and archives WAL segments.
It may be difficult to estimate in advance how much space you'll need. The best thing to do is take some backups then record the size of different types of backups (full/incr/diff) and measure the amount of WAL generated per day. This will give you a general idea of how much space you'll need, though of course requirements will likely change over time as your database evolves.
For this demonstration the repository will be stored on the same host as the PostgreSQL server. This is the simplest configuration and is useful in cases where traditional backup software is employed to backup the database host.
pg-primary ⇒ Create the pgBackRest repository 
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown postgres:postgres /var/lib/pgbackrest
The repository path must be configured so pgBackRest knows where to find it.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the pgBackRest repository path
Config suppressed for testing
Multiple repositories may also be configured. See Multiple Repositories for details.
Configure Archiving
Backing up a running PostgreSQL cluster requires WAL archiving to be enabled. %p is how PostgreSQL specifies the location of the WAL segment to be archived. Note that at least one WAL segment will be created during the backup process even if no explicit writes are made to the cluster.
pg-primary:/etc/postgresql/17/demo/postgresql.conf ⇒ Configure archive settings
Config suppressed for testing
The PostgreSQL cluster must be restarted after making these changes and before performing a backup.
pg-primary ⇒ Restart the demo cluster 
sudo pg_ctlcluster 17 demo restart
The hot_standby setting is enabled by default and should be left that way on every cluster. Any cluster may be restored as a standby later, e.g. a primary that is rebuilt as a standby after failover, and a standby will not accept read-only connections without it.
When archiving a WAL segment is expected to take more than 60 seconds (the default) to reach the pgBackRest repository, then the pgBackRest archive-timeout option should be increased. Note that this option is not the same as the PostgreSQL archive_timeout option which is used to force a WAL segment switch; useful for databases where there are long periods of inactivity. For more information on the PostgreSQL archive_timeout option, see PostgreSQL Write Ahead Log.
The archive-push command can be configured with its own options. For example, a lower compression level may be set to speed archiving without affecting the compression used for backups.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Config archive-push to use a lower compression level
Config suppressed for testing
This configuration technique can be used for any command and can even target a specific stanza, e.g. demo:archive-push.
Configure Retention
pgBackRest expires backups based on retention options.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure retention to 2 full backups
Config suppressed for testing
More information about retention can be found in the Retention section.
Configure Repository Encryption
The repository will be configured with a cipher type and key to demonstrate encryption. Encryption is always performed client-side even if the repository type (e.g. S3 or other object store) supports encryption.
It is important to use a long, random passphrase for the cipher key. A good way to generate one is to run: openssl rand -base64 48.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest repository encryption
Config suppressed for testing
NOTE:
Encryption settings are placed in the [global] section, above, so the info command can read all stanzas. Without the stanza option the info command reads encryption settings only from the [global] section, so encryption settings configured per stanza require the stanza option to read an encrypted stanza.
Once the repository has been configured and the stanza created and checked, the repository encryption settings cannot be changed.
Create the Stanza
The stanza-create command must be run to initialize the stanza. It is recommended that the check command be run after stanza-create to ensure archiving and backups are properly configured.
pg-primary ⇒ Create the stanza and check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
Check the Configuration
The check command validates that pgBackRest and the archive_command setting are configured correctly for archiving and backups for the specified stanza. It will attempt to check all repositories and databases that are configured for the host on which the command is run. It detects misconfigurations, particularly in archiving, that result in incomplete backups because required WAL segments did not reach the archive. The command can be run on the PostgreSQL or repository host. The command may also be run on the standby host, however, since pg_switch_xlog()/pg_switch_wal() cannot be performed on the standby, the command will only test the repository configuration.
Note that pg_create_restore_point('pgBackRest Archive Check') and pg_switch_xlog()/pg_switch_wal() are called to force PostgreSQL to archive a WAL segment.
pg-primary ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Performance Tuning
pgBackRest has a number of performance options that are not enabled by default to maintain backward compatibility in the repository. However, when creating a new repository the following options are recommended. They can also be used on an existing repository with the caveat that older versions of pgBackRest will not be able to read the repository. This incompatibility depends on when the feature was introduced, as noted in the list below.
  • compress-type - determines the compression algorithm used by the backup and archive-push commands. The default is gz (Gzip) but zst (Zstandard) is recommended because it is much faster and provides compression similar to gz. zst has been supported by the compress-type option since v2.27. See Compress Type for more details.
  • repo-bundle - combines small files during backup to save space and improve the speed of both the backup and restore commands, especially on object stores such as S3. The repo-bundle option was introduced in v2.39. See File Bundling for more details.
  • repo-block - stores only the portions of files that have changed rather than the entire file during diff/incr backup. This saves space and increases the speed of the backup. The repo-block option was introduced in v2.46 but at least v2.52.1 is recommended. See Block Incremental for more details.
There are other performance options that are not enabled by default because they require additional configuration or because the default is safe (but not optimal). These options are available in all v2 versions of pgBackRest.
  • process-max - determines how many processes will be used for commands. The default is 1, which is almost never the appropriate value. Each command uses process-max differently so refer to each command's documentation for details on usage.
  • archive-async - archives WAL files to the repository in batch which greatly increases archiving speed. It is not enabled by default because it requires a spool path to be created. See Asynchronous Archiving for more details.
  • backup-standby - performs the backup on a standby rather than the primary to reduce load on the primary. It is not enabled by default because it requires additional configuration and the presence of one or more standby hosts. See Backup from a Standby for more details.
Perform a Backup
By default pgBackRest will wait for the next regularly scheduled checkpoint before starting a backup. Depending on the checkpoint_timeout and checkpoint_segments settings in PostgreSQL it may be quite some time before a checkpoint completes and the backup can begin. Generally, it is best to set start-fast=y so that the backup starts immediately. This forces a checkpoint, but since backups are usually run once a day an additional checkpoint should not have a noticeable impact on performance. However, on very busy clusters it may be best to pass --start-fast on the command-line as needed.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure backup fast start
Config suppressed for testing
To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo \
       --log-level-console=info backup
Output suppressed for testing
By default pgBackRest will attempt to perform an incremental backup. However, an incremental backup must be based on a full backup and since no full backup existed pgBackRest ran a full backup instead.
The type option can be used to specify a full or differential backup.
pg-primary ⇒ Differential backup of the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
This time there was no warning because a full backup already existed. While incremental backups can be based on a full or differential backup, differential backups must be based on a full backup. A full backup can be performed by running the backup command with --type=full.
During an online backup pgBackRest waits for WAL segments that are required for backup consistency to be archived. This wait time is governed by the pgBackRest archive-timeout option which defaults to 60 seconds. If archiving an individual segment is known to take longer then this option should be increased.
Schedule a Backup
Backups can be scheduled with utilities such as cron.
In the following example, two cron jobs are configured to run; full backups are scheduled for 6:30 AM every Sunday with differential backups scheduled for 6:30 AM Monday through Saturday. If this crontab is installed for the first time mid-week, then pgBackRest will run a full backup the first time the differential job is executed, followed the next day by a differential backup. 
#m h   dom mon dow   command
30 06  *   *   0     pgbackrest --type=full --stanza=demo backup
30 06  *   *   1-6   pgbackrest --type=diff --stanza=demo backup
Once backups are scheduled it's important to configure retention so backups are expired on a regular schedule, see Retention.
Backup Information
Use the info command to get information about backups.
pg-primary ⇒ Get info for the demo cluster 
sudo -u postgres pgbackrest info
Output suppressed for testing
The info command operates on a single stanza or all stanzas. Text output is the default and gives a human-readable summary of backups for the stanza(s) requested. This format is subject to change with any release.
For machine-readable output use --output=json. The JSON output contains far more information than the text output and is kept stable unless a bug is found.
To speed up execution, limit the output to only progress information by specifying --detail-level=progress. Note that this skips all checks except for availability of the stanza.
Each stanza has a separate section and it is possible to limit output to a single stanza with the --stanza option. The stanza 'status' gives a brief indication of the stanza's health. If this is 'ok' then pgBackRest is functioning normally. If there are multiple repositories, then a status of 'mixed' indicates that the stanza is not in a healthy state on one or more of the repositories; in this case the state of the stanza will be detailed per repository. For cases in which an error on a repository occurred that is not one of the known error codes, then an error code of 'other' will be used and the full error details will be provided. The 'wal archive min/max' shows the minimum and maximum WAL currently stored in the archive and, in the case of multiple repositories, will be reported across all repositories unless the --repo option is set. Note that there may be gaps due to archive retention policies or other reasons.
The 'backup/expire running' and/or 'restore running' messages will appear beside the 'status' information if any of those commands are currently running on the host. Per-repo progress will be also reported in text output and a 'repo' array will be included in JSON output.
The backups are displayed oldest to newest. The oldest backup will always be a full backup (indicated by an F at the end of the label) but the newest backup can be full, differential (ends with D), or incremental (ends with I).
The 'timestamp start/stop' defines the time period when the backup ran. The 'timestamp stop' can be used to determine the backup to use when performing Point-In-Time Recovery. More information about Point-In-Time Recovery can be found in the Point-In-Time Recovery section.
The 'wal start/stop' defines the WAL range that is required to make the database consistent when restoring. The backup command will ensure that this WAL range is in the archive before completing.
The 'database size' is the full uncompressed size of the database while 'database backup size' is the amount of data in the database to actually back up (these will be the same for full backups).
The 'repo' indicates in which repository this backup resides. The 'backup set size' includes all the files from this backup and any referenced backups in the repository that are required to restore the database from this backup while 'backup size' includes only the files in this backup (these will also be the same for full backups). Repository sizes reflect compressed file sizes if compression is enabled in pgBackRest.
The 'backup reference total' summarizes the list of additional backups that are required to restore this backup. Use the --set option to display the complete reference list.
Restore a Backup
Backups can protect you from a number of disaster scenarios, the most common of which are hardware failure and data corruption. The easiest way to simulate data corruption is to remove an important PostgreSQL cluster file.
pg-primary ⇒ Stop the demo cluster and delete the pg_control file 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres rm /var/lib/postgresql/17/demo/global/pg_control
Starting the cluster without this important file will result in an error.
pg-primary ⇒ Attempt to start the corrupted demo cluster 
sudo pg_ctlcluster 17 demo start
Output suppressed for testing
To restore a backup of the PostgreSQL cluster run pgBackRest with the restore command. The cluster needs to be stopped (in this case it is already stopped) and all files must be removed from the PostgreSQL data directory.
pg-primary ⇒ Remove old files from demo cluster 
sudo -u postgres find /var/lib/postgresql/17/demo -mindepth 1 -delete
pg-primary ⇒ Restore the demo cluster and start PostgreSQL 
sudo -u postgres pgbackrest --stanza=demo restore
sudo pg_ctlcluster 17 demo start
This time the cluster started successfully since the restore replaced the missing pg_control file.
More information about the restore command can be found in the Restore section.
Monitoring
Monitoring is an important part of any production system. There are many tools available and pgBackRest can be monitored on any of them with a little work.
pgBackRest can output information about the repository in JSON format which includes a list of all backups for each stanza and WAL archive info.
In PostgreSQL
The PostgreSQL COPY command allows pgBackRest info to be loaded into a table. The following example wraps that logic in a function that can be used to perform real-time queries.
pg-primary ⇒ Load pgBackRest info function for PostgreSQL 
sudo -u postgres cat \
       /var/lib/postgresql/pgbackrest/doc/example/pgsql-pgbackrest-info.sql
Output suppressed for testing
sudo -u postgres psql -f \
       /var/lib/postgresql/pgbackrest/doc/example/pgsql-pgbackrest-info.sql
Now the monitor.pgbackrest_info() function can be used to determine the last successful backup time and archived WAL for a stanza.
pg-primary ⇒ Query last successful backup time and archived WAL 
sudo -u postgres cat \
       /var/lib/postgresql/pgbackrest/doc/example/pgsql-pgbackrest-query.sql
Output suppressed for testing
sudo -u postgres psql -f \
       /var/lib/postgresql/pgbackrest/doc/example/pgsql-pgbackrest-query.sql
Output suppressed for testing
Using jq
jq is a command-line utility that can easily extract data from JSON.
pg-primary ⇒ Install jq utility 
sudo apt-get install jq
Now jq can be used to query the last successful backup time for a stanza.
pg-primary ⇒ Query last successful backup time 
sudo -u postgres pgbackrest --output=json --stanza=demo info | \
       jq '.[0] | .backup[-1] | .timestamp.stop'
Output suppressed for testing
Or the last archived WAL.
pg-primary ⇒ Query last archived WAL 
sudo -u postgres pgbackrest --output=json --stanza=demo info | \
       jq '.[0] | .archive[-1] | .max'
Output suppressed for testing
NOTE:
This syntax requires jq v1.5.
NOTE:
jq may round large numbers such as system identifiers. Test your queries carefully.
Backup
When multiple repositories are configured, pgBackRest will backup to the highest priority repository (e.g. repo1) unless the --repo option is specified.
pgBackRest does not have a built-in scheduler so it's best to run it from cron or some other scheduling mechanism.
See Perform a Backup for more details and examples.
File Bundling
Bundling files together in the repository saves time during the backup and some space in the repository. This is especially pronounced when the repository is stored on an object store such as S3 or file systems with large block sizes. Per-file creation time on object stores is higher and very small files might cost as much to store as larger files.
The file bundling feature is enabled with the repo-bundle option.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-bundle
Config suppressed for testing
A full backup without file bundling will have 1000+ files in the backup path, but with bundling the total number of files is greatly reduced. An additional benefit is that zero-length files are not stored (except in the manifest), whereas in a normal backup each zero-length file is stored individually.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full backup
pg-primary ⇒ Check file total 
sudo -u postgres find /var/lib/pgbackrest/backup/demo/latest/ -type f | wc -l
Output suppressed for testing
The repo-bundle-size and repo-bundle-limit options can be used for tuning, though the defaults should be optimal in most cases.
While file bundling is generally more efficient, the downside is that it is more difficult to manually retrieve files from the repository. It may not be ideal for deduplicated storage since each full backup will arrange files in the bundles differently. Lastly, file bundles cannot be resumed, so be careful not to set repo-bundle-limit too high.
Block Incremental
Block incremental backups save space by only storing the parts of a file that have changed since the prior backup rather than storing the entire file.
The block incremental feature is enabled with the repo-block option and it works best when enabled for all backup types. File bundling must also be enabled.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-block
Config suppressed for testing
Backup Annotations
Users can attach informative key/value pairs to the backup. This option may be used multiple times to attach multiple annotations.
pg-primary ⇒ Perform a full backup with annotations 
sudo -u postgres pgbackrest --stanza=demo --annotation=source="demo backup" \
       --annotation=key=value --type=full backup
Annotations are output by the info command text output when a backup is specified with --set and always appear in the JSON output.
pg-primary ⇒ Get info for the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] info
Output suppressed for testing
Annotations included with the backup command can be added, modified, or removed afterwards using the annotate command.
pg-primary ⇒ Change backup annotations 
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] \
       --annotation=key= --annotation=new_key=new_value annotate
sudo -u postgres pgbackrest --stanza=demo --set=[Test Variable] info
Output suppressed for testing
Retention
Generally it is best to retain as many backups as possible to provide a greater window for Point-in-Time Recovery, but practical concerns such as disk space must also be considered. Retention options remove older backups once they are no longer needed.
pgBackRest does full backup rotation based on the retention type which can be a count or a time period. When a count is specified, then expiration is not concerned with when the backups were created but with how many must be retained. Differential backups are count-based but will always be expired when the full backup they depend on is expired. Incremental backups are not expired by retention independently — they are always expired with their related full or differential backup. See sections Full Backup Retention and Differential Backup Retention for details and examples.
Archived WAL is retained by default for backups that have not expired, however, although not recommended, this schedule can be modified per repository with the retention-archive options. See section Archive Retention for details and examples.
The expire command is run automatically after each successful backup and can also be run by the user. When run by the user, expiration will occur as defined by the retention settings for each configured repository. If the --repo option is provided, expiration will occur only on the specified repository. Expiration can also be limited by the user to a specific backup set with the --set option and, unless the --repo option is specified, all repositories will be searched and any matching the set criteria will be expired. It should be noted that the archive retention schedule will be checked and performed any time the expire command is run.
Full Backup Retention
The repo1-retention-full-type determines how the option repo1-retention-full is interpreted; either as the count of full backups to be retained or how many days to retain full backups. New backups must be completed before expiration will occur — that means if repo1-retention-full-type=count and repo1-retention-full=2 then there will be three full backups stored before the oldest one is expired, or if repo1-retention-full-type=time and repo1-retention-full=20 then there must be one full backup that is at least 20 days old before expiration can occur.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-full
Config suppressed for testing
Backup repo1-retention-full=2 but currently there is only one full backup so the next full backup to run will not expire any full backups.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=detail backup
Output suppressed for testing
Archive is expired because WAL segments were generated before the oldest backup. These are not useful for recovery — only WAL segments generated after a backup can be used to recover that backup.
pg-primary ⇒ Perform a full backup 
sudo -u postgres pgbackrest --stanza=demo --type=full \
       --log-level-console=info backup
Output suppressed for testing
The [Test Variable] full backup is expired and archive retention is based on the [Test Variable] which is now the oldest full backup.
Differential Backup Retention
Set repo1-retention-diff to the number of differential backups required. Differentials only rely on the prior full backup so it is possible to create a rolling set of differentials for the last day or more. This allows quick restores to recent points-in-time but reduces overall space consumption.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-diff
Config suppressed for testing
Backup repo1-retention-diff=1 so two differentials will need to be performed before one is expired. An incremental backup is added to demonstrate incremental expiration, which in this case depends on the differential expiration.
pg-primary ⇒ Perform differential and incremental backups 
sudo -u postgres pgbackrest --stanza=demo --type=diff backup
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
Now performing a differential backup will expire the previous differential and incremental backups leaving only one differential backup.
pg-primary ⇒ Perform a differential backup 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
Archive Retention
Although pgBackRest automatically removes archived WAL segments when expiring backups (the default expires WAL for full backups based on the repo1-retention-full option), it may be useful to expire archive more aggressively to save disk space. Note that full backups are treated as differential backups for the purpose of differential archive retention.
Expiring archive will never remove WAL segments that are required to make a backup consistent. However, since Point-in-Time-Recovery (PITR) only works on a continuous WAL stream, care should be taken when aggressively expiring archive outside of the normal backup expiration process. To determine what will be expired without actually expiring anything, the dry-run option can be provided on the command line with the expire command.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-retention-diff
Config suppressed for testing
pg-primary ⇒ Perform differential backup 
sudo -u postgres pgbackrest --stanza=demo --type=diff \
       --log-level-console=info backup
Output suppressed for testing
pg-primary ⇒ Expire archive 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=detail \
       --repo1-retention-archive-type=diff --repo1-retention-archive=1 expire
Output suppressed for testing
The [Test Variable] differential backup has archived WAL segments that must be retained to make the older backups consistent even though they cannot be played any further forward with PITR. WAL segments generated after [Test Variable] but before [Test Variable] are removed. WAL segments generated after the new backup [Test Variable] remain and can be used for PITR.
Since full backups are considered differential backups for the purpose of differential archive retention, if a full backup is now performed with the same settings, only the archive for that full backup is retained for PITR.
Restore
The restore command automatically defaults to selecting the latest backup from the first repository where backups exist (see Quick Start - Restore a Backup). The order in which the repositories are checked is dictated by the pgbackrest.conf (e.g. repo1 will be checked before repo2). To select from a specific repository, the --repo option can be passed (e.g. --repo=1). The --set option can be passed if a backup other than the latest is desired.
When PITR of --type=time or --type=lsn is specified, then the target time or target lsn must be specified with the --target option. If a backup is not specified via the --set option, then the configured repositories will be checked, in order, for a backup that contains the requested time or lsn. If no matching backup is found, the latest backup from the first repository containing backups will be used for --type=time while no backup will be selected for --type=lsn. For other types of PITR, e.g. xid, the --set option must be provided if the target is prior to the latest backup. See Point-in-Time Recovery for more details and examples.
Replication slots are not included per recommendation of PostgreSQL. See Backing Up The Data Directory in the PostgreSQL documentation for more information.
The following sections introduce additional restore command features.
File Ownership
If a restore is run as a non-root user (the typical scenario) then all files restored will belong to the user/group executing pgBackRest. If existing files are not owned by the executing user/group then an error will result if the ownership cannot be updated to the executing user/group. In that case the file ownership will need to be updated by a privileged user before the restore can be retried.
If a restore is run as the root user then pgBackRest will attempt to recreate the ownership recorded in the manifest when the backup was made. Only user/group names are stored in the manifest so the same names must exist on the restore host for this to work. If the user/group name cannot be found locally then the user/group of the PostgreSQL data directory will be used and finally root if the data directory user/group cannot be mapped to a name.
Delta Option
Restore a Backup in Quick Start required the database cluster directory to be cleaned before the restore could be performed. The delta option allows pgBackRest to automatically determine which files in the database cluster directory can be preserved and which ones need to be restored from the backup — it also removes files not present in the backup manifest so it will dispose of divergent changes. This is accomplished by calculating a SHA-1 cryptographic hash for each file in the database cluster directory. If the SHA-1 hash does not match the hash stored in the backup then that file will be restored. This operation is very efficient when combined with the process-max option. Since the PostgreSQL server is shut down during the restore, a larger number of processes can be used than might be desirable during a backup when the PostgreSQL server is running.
pg-primary ⇒ Stop the demo cluster, perform delta restore 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta \
       --log-level-console=detail restore
Output suppressed for testing
pg-primary ⇒ Restart PostgreSQL 
sudo pg_ctlcluster 17 demo start
Restore Selected Databases
There may be cases where it is desirable to selectively restore specific databases from a cluster backup. This could be done for performance reasons or to move selected databases to a machine that does not have enough space to restore the entire cluster backup.
To demonstrate this feature two databases are created: test1 and test2.
pg-primary ⇒ Create two test databases 
sudo -u postgres psql -c "create database test1;"
Output suppressed for testing
sudo -u postgres psql -c "create database test2;"
Output suppressed for testing
Each test database will be seeded with tables and data to demonstrate that recovery works with selective restore.
pg-primary ⇒ Create a test table in each database 
sudo -u postgres psql -c "create table test1_table (id int); \
       insert into test1_table (id) values (1);" test1
Output suppressed for testing
sudo -u postgres psql -c "create table test2_table (id int); \
       insert into test2_table (id) values (2);" test2
Output suppressed for testing
A fresh backup is run so pgBackRest is aware of the new databases.
pg-primary ⇒ Perform a backup 
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
One of the main reasons to use selective restore is to save space. The size of the test1 database is shown here so it can be compared with the disk utilization after a selective restore.
pg-primary ⇒ Show space used by test1 database 
sudo -u postgres du -sh /var/lib/postgresql/17/demo/base/[Test Variable]
Output suppressed for testing
If the database to restore is not known, use the info command set option to discover databases that are part of the backup set.
pg-primary ⇒ Show database list for backup 
sudo -u postgres pgbackrest --stanza=demo \
       --set=[Test Variable] info
Output suppressed for testing
Stop the cluster and restore only the test2 database. Built-in databases (template0, template1, and postgres) are always restored.
WARNING:
Recovery may error unless --type=immediate is specified. This is because after consistency is reached PostgreSQL will flag zeroed pages as errors even for a full-page write. For PostgreSQL ≥ 13 the ignore_invalid_pages setting may be used to ignore invalid pages. In this case it is important to check the logs after recovery to ensure that no invalid pages were reported in the selected databases.
pg-primary ⇒ Restore from last backup including only the test2 database 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta \
       --db-include=test2 --type=immediate --target-action=promote restore
sudo pg_ctlcluster 17 demo start
Once recovery is complete the test2 database will contain all previously created tables and data.
pg-primary ⇒ Demonstrate that the test2 database was recovered 
sudo -u postgres psql -c "select * from test2_table;" test2
Output suppressed for testing
The test1 database, despite successful recovery, is not accessible. This is because the entire database was restored as sparse, zeroed files. PostgreSQL can successfully apply WAL on the zeroed files but the database as a whole will not be valid because key files contain no data. This is purposeful to prevent the database from being accidentally used when it might contain partial data that was applied during WAL replay.
pg-primary ⇒ Attempting to connect to the test1 database will produce an error 
sudo -u postgres psql -c "select * from test1_table;" test1
Output suppressed for testing
Since the test1 database is restored with sparse, zeroed files it will only require as much space as the amount of WAL that is written during recovery. While the amount of WAL generated during a backup and applied during recovery can be significant it will generally be a small fraction of the total database size, especially for large databases where this feature is most likely to be useful.
It is clear that the test1 database uses far less disk space during the selective restore than it would have if the entire database had been restored.
pg-primary ⇒ Show space used by test1 database after recovery 
sudo -u postgres du -sh /var/lib/postgresql/17/demo/base/[Test Variable]
Output suppressed for testing
At this point the only action that can be taken on the invalid test1 database is drop database. pgBackRest does not automatically drop the database since this cannot be done until recovery is complete and the cluster is accessible.
pg-primary ⇒ Drop the test1 database 
sudo -u postgres psql -c "drop database test1;"
Output suppressed for testing
Now that the invalid test1 database has been dropped only the test2 and built-in databases remain.
pg-primary ⇒ List remaining databases 
sudo -u postgres psql -c "select oid, datname from pg_database order by oid;"
Output suppressed for testing
Point-in-Time Recovery
Restore a Backup in Quick Start performed default recovery, which is to play all the way to the end of the WAL stream. In the case of a hardware failure this is usually the best choice but for data corruption scenarios (whether machine or human in origin) Point-in-Time Recovery (PITR) is often more appropriate.
Point-in-Time Recovery (PITR) allows the WAL to be played from a backup to a specified lsn, time, transaction id, or recovery point. For common recovery scenarios time-based recovery is arguably the most useful. A typical recovery scenario is to restore a table that was accidentally dropped or data that was accidentally deleted. Recovering a dropped table is more dramatic so that's the example given here but deleted data would be recovered in exactly the same way.
pg-primary ⇒ Create a table with very important data 
sudo -u postgres psql -c "begin; \
       create table important_table (message text); \
       insert into important_table values ('Important Data'); \
       commit; \
       select * from important_table;"
Output suppressed for testing
It is important to represent the time as reckoned by PostgreSQL and to include timezone offsets. This reduces the possibility of unintended timezone conversions and an unexpected recovery result.
pg-primary ⇒ Get the time from PostgreSQL 
sudo -u postgres psql -Atc "select current_timestamp"
Output suppressed for testing
Now that the time has been recorded the table is dropped. In practice finding the exact time that the table was dropped is a lot harder than in this example. It may not be possible to find the exact time, but some forensic work should be able to get you close.
pg-primary ⇒ Drop the important table 
sudo -u postgres psql -c "begin; \
       drop table important_table; \
       commit; \
       select * from important_table;"
Output suppressed for testing
If the wrong backup is selected for restore then recovery to the required time target will fail. To demonstrate this a new incremental backup is performed where important_table does not exist.
pg-primary ⇒ Perform an incremental backup 
sudo -u postgres pgbackrest --stanza=demo --type=incr backup
sudo -u postgres pgbackrest info
Output suppressed for testing
It will not be possible to recover the lost table from this backup since PostgreSQL can only play forward, not backward.
pg-primary ⇒ Attempt recovery from an incorrect backup 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta \
       --set=[Test Variable] --target-timeline=current \
       --type=time "--target=[Test Variable]" --target-action=promote restore
sudo pg_ctlcluster 17 demo start
Output suppressed for testing
A reliable method is to allow pgBackRest to automatically select a backup capable of recovery to the time target, i.e. a backup that ended before the specified time.
NOTE:
pgBackRest cannot automatically select a backup when the restore type is xid or name.
pg-primary ⇒ Restore the demo cluster to [Test Variable] 
sudo -u postgres pgbackrest --stanza=demo --delta \
       --type=time "--target=[Test Variable]" \
       --target-action=promote restore
sudo -u postgres cat /var/lib/postgresql/17/demo/postgresql.auto.conf
Output suppressed for testing
pgBackRest has generated the recovery settings in postgresql.auto.conf so PostgreSQL can be started immediately. %f is how PostgreSQL specifies the WAL segment it needs and %p is the location where it should be copied. Once PostgreSQL has finished recovery the table will exist again and can be queried.
pg-primary ⇒ Start PostgreSQL and check that the important table exists 
sudo pg_ctlcluster 17 demo start
sudo -u postgres psql -c "select * from important_table"
Output suppressed for testing
The PostgreSQL log also contains valuable information. It will indicate the time and transaction where the recovery stopped and also give the time of the last transaction to be applied.
pg-primary ⇒ Examine the PostgreSQL log output 
sudo -u postgres cat /var/log/postgresql/postgresql-17-demo.log
Output suppressed for testing
Delete a Stanza
The stanza-delete command removes data in the repository associated with a stanza.
WARNING:
Use this command with caution — it will permanently remove all backups and archives from the pgBackRest repository for the specified stanza.
To delete a stanza:
  • Shut down the PostgreSQL cluster associated with the stanza (or use --force to override).
  • Run the stop command on the host where the stanza-delete command will be run.
  • Run the stanza-delete command.
Once the command successfully completes, it is the responsibility of the user to remove the stanza from all pgBackRest configuration files and/or environment variables.
A stanza may only be deleted from one repository at a time. To delete the stanza from multiple repositories, repeat the stanza-delete command for each repository while specifying the --repo option.
pg-primary ⇒ Stop PostgreSQL cluster to be removed 
sudo pg_ctlcluster 17 demo stop
pg-primary ⇒ Stop pgBackRest for the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stop
Output suppressed for testing
pg-primary ⇒ Delete the stanza from one repository 
sudo -u postgres pgbackrest --stanza=demo --repo=1 \
       --log-level-console=info stanza-delete
Output suppressed for testing
Multiple Repositories
Multiple repositories may be configured as demonstrated in S3 Support. A potential benefit is the ability to have a local repository for fast restores and a remote repository for redundancy.
Some commands, e.g. stanza-create/stanza-upgrade, will automatically work with all configured repositories while others, e.g. stanza-delete, will require a repository to be specified using the repo option. See the command reference for details on which commands require the repository to be specified.
Note that the repo option is not required when only repo1 is configured in order to maintain backward compatibility. However, the repo option is required when a single repo is configured as, e.g. repo2. This is to prevent command breakage if a new repository is added later.
The archive-push command will always push WAL to the archive in all configured repositories. When a repository cannot be reached, WAL will still be pushed to other repositories. However, for this to work effectively, archive-async=y must be enabled; otherwise, the other repositories can only get one WAL segment ahead of the unreachable repository. Also, note that if WAL cannot be pushed to any repository, then PostgreSQL will not remove it from the pg_wal directory, which may cause the volume to run out of space.
Backups need to be scheduled individually for each repository. In many cases this is desirable since backup types and retention will vary by repository. Likewise, restores must specify a repository. It is generally better to specify a repository for restores that has low latency/cost even if that means more recovery time. Only restore testing can determine which repository will be most efficient.
Azure-Compatible Object Store Support
pgBackRest supports locating repositories in Azure-compatible object stores. The container used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the container root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the container without conflicts.
WARNING:
Do not enable hierarchical namespace as this will cause errors during expire.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure Azure
Config suppressed for testing
Shared access signatures may be used by setting the repo2-azure-key-type option to sas and the repo2-azure-key option to the shared access signature token.
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
File creation time in Azure is relatively slow so backup/restore performance is improved by enabling file bundling.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=2 \
       --log-level-console=info backup
Output suppressed for testing
S3-Compatible Object Store Support
pgBackRest supports locating repositories in S3-compatible object stores. The bucket used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the bucket root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the bucket without conflicts.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure S3
Config suppressed for testing
NOTE:
The region and endpoint will need to be configured to where the bucket is located. The values given here are for the us-east-1 region.
A role should be created to run pgBackRest and the bucket permissions should be set as restrictively as possible. If the role is associated with an instance in AWS then pgBackRest will automatically retrieve temporary credentials when repo3-s3-key-type=auto, which means that keys do not need to be explicitly set in /etc/pgbackrest/pgbackrest.conf.
This sample Amazon S3 policy will restrict all reads and writes to the bucket and repository path. 
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket"
            ],
            "Condition": {
                "StringEquals": {
                    "s3:prefix": [
                        "",
                        "demo-repo"
                    ],
                    "s3:delimiter": [
                        "/"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket"
            ],
            "Condition": {
                "StringLike": {
                    "s3:prefix": [
                        "demo-repo/*"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectTagging",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::demo-bucket/demo-repo/*"
            ]
        }
    ]
}
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
File creation time in S3 is relatively slow so backup/restore performance is improved by enabling file bundling.
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=3 \
       --log-level-console=info backup
Output suppressed for testing
SFTP Support
pgBackRest supports locating repositories on SFTP hosts. SFTP file transfer is relatively slow so commands benefit by increasing process-max to parallelize file transfer.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure SFTP
Config suppressed for testing
When utilizing SFTP, if libssh2 is compiled against OpenSSH then repo4-sftp-public-key-file is optional.
pg-primary ⇒ Generate SSH keypair for SFTP backup 
sudo -u postgres mkdir -m 750 -p /var/lib/postgresql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/postgresql/.ssh/id_rsa_sftp \
       -t rsa -b 4096 -N "" -m PEM
sftp-server ⇒ Copy pg-primary SFTP backup public key to sftp-server 
sudo -u pgbackrest mkdir -m 750 -p /home/pgbackrest/.ssh
(sudo ssh root@pg-primary cat /var/lib/postgresql/.ssh/id_rsa_sftp.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
Commands are run exactly as if the repository were stored on a local disk.
pg-primary ⇒ Add sftp-server fingerprint to known_hosts file since repo4-sftp-host-key-check-type defaults to strict 
ssh-keyscan -H sftp-server >> /var/lib/postgresql/.ssh/known_hosts 2>/dev/null
pg-primary ⇒ Create the stanza 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info stanza-create
Output suppressed for testing
pg-primary ⇒ Backup the demo cluster 
sudo -u postgres pgbackrest --stanza=demo --repo=4 \
       --log-level-console=info backup
Output suppressed for testing
GCS-Compatible Object Store Support
pgBackRest supports locating repositories in GCS-compatible object stores. The bucket used to store the repository must be created in advance — pgBackRest will not do it automatically. The repository can be located in the bucket root (/) but it's usually best to place it in a subpath so object store logs or other data can also be stored in the bucket without conflicts.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure GCS
Config suppressed for testing
When running in GCE set repo5-gcs-key-type=auto to automatically authenticate using the instance service account.
Commands are run exactly as if the repository were stored on a local disk.
File creation time in GCS is relatively slow so backup/restore performance is improved by enabling file bundling.
Target Time for Repository
The target time defines the time that commands use to read a repository on versioned storage. This allows the command to read the repository as it was at a point-in-time in order to recover data that has been deleted or corrupted by user accident or malware.
Versioned storage is supported by S3, GCS, and Azure but is generally not enabled by default. In addition to enabling versioning, it may be useful to enable object locking for S3 and soft delete for GCS or Azure.
When the repo-target-time option is specified then the repo option must also be provided. It is likely that not all repository types will support versioning and in general it makes sense to target a single repository for recovery.
Note that comparisons to the storage timestamp are <= the timestamp provided and milliseconds are truncated from the timestamp when provided.
To demonstrate this feature the demo stanza in the S3 repo is deleted.
pg-primary ⇒ Delete stanza in S3 repository 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo stop
Output suppressed for testing
sudo -u postgres pgbackrest --stanza=demo --repo=3 stanza-delete
Output suppressed for testing
Once the stanza is deleted the info command will show the repository in an error state.
pg-primary ⇒ Error on info 
sudo -u postgres pgbackrest --stanza=demo --repo=3 info
Output suppressed for testing
However, since the storage is versioned, it is possible to look at the repository at a time before the stanza was deleted. Finding the target time can be tricky depending on the situation, but in this case the time when the stanza was deleted can be determined by checking when backup.info was deleted.
pg-primary ⇒ List versions of backup.info in the bucket 
key=demo-repo/backup/demo/backup.info; \
       aws s3api list-object-versions --bucket demo-bucket \
       --prefix $key --output table \
       --query "sort_by([Versions[?Key=='$key'].{Action:'PUT', \
       Modified:LastModified,Object:Key}, \
       DeleteMarkers[?Key=='$key'].{Action:'DELETE', \
       Modified:LastModified,Object:Key}][],&Modified)"
Output suppressed for testing
Now the info command can be run with a target time that will show the repository before it was deleted.
pg-primary ⇒ Info with target time 
sudo -u postgres pgbackrest --stanza=demo --repo=3 \
       --repo-target-time="[Test Variable]" info
Output suppressed for testing
If the required backup is shown by the info command then it can be restored using the same target time.
pg-primary ⇒ Restore with target time 
sudo -u postgres pgbackrest --stanza=demo --repo=3 --delta \
       --repo-target-time="[Test Variable]" --log-level-console=info restore
Output suppressed for testing
sudo pg_ctlcluster 17 demo start
Dedicated Repository Host
The configuration described in Quickstart is suitable for simple installations but for enterprise configurations it is more typical to have a dedicated repository host where the backups and WAL archive files are stored. This separates the backups and WAL archive from the database server so database host failures have less impact. It is still a good idea to employ traditional backup software to backup the repository host.
On PostgreSQL hosts, pg1-path is required to be the path of the local PostgreSQL cluster and no pg1-host should be configured. When configuring a repository host, the pgbackrest configuration file must have the pg-host option configured to connect to the primary and standby (if any) hosts. The repository host has the only pgbackrest configuration that should be aware of more than one PostgreSQL host. Order does not matter, e.g. pg1-path/pg1-host, pg2-path/pg2-host can be primary or standby.
Installation
A new host named repository is created to store the cluster backups.
NOTE:
The pgBackRest version installed on the repository host must exactly match the version installed on the PostgreSQL host.
The pgbackrest user is created to own the pgBackRest repository. Any user can own the repository but it is best not to use postgres (if it exists) to avoid confusion.
NOTE:
When pgBackRest is installed from a package, a logrotate configuration such as /etc/logrotate.d/pgbackrest may be provided that rotates the logs as a specific user via the su directive (e.g. su postgres postgres). Since the files in /var/log/pgbackrest are owned by the user that runs pgBackRest (here pgbackrest), the su directive must be updated to match this user or logrotate will fail with a permission error.
repository ⇒ Create pgbackrest user 
sudo adduser --disabled-password --gecos "" pgbackrest
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
Debian/Ubuntu packages for pgBackRest are available at apt.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
repository ⇒ Install dependencies 
sudo apt-get install postgresql-client libxml2 libssh2-1
repository ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
repository ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown pgbackrest:pgbackrest /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown pgbackrest:pgbackrest /etc/pgbackrest/pgbackrest.conf
repository ⇒ Create the pgBackRest repository 
sudo mkdir -p /var/lib/pgbackrest
sudo chmod 750 /var/lib/pgbackrest
sudo chown pgbackrest:pgbackrest /var/lib/pgbackrest
Setup Passwordless SSH
pgBackRest can use passwordless SSH to enable communication between the hosts. It is also possible to use TLS, see Setup TLS.
repository ⇒ Create repository host key pair 
sudo -u pgbackrest mkdir -m 750 /home/pgbackrest/.ssh
sudo -u pgbackrest ssh-keygen -f /home/pgbackrest/.ssh/id_rsa \
       -t rsa -b 4096 -N ""
pg-primary ⇒ Create pg-primary host key pair 
sudo -u postgres mkdir -m 750 -p /var/lib/postgresql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/postgresql/.ssh/id_rsa \
       -t rsa -b 4096 -N ""
Exchange keys between repository and pg-primary.
repository ⇒ Copy pg-primary public key to repository 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@pg-primary cat /var/lib/postgresql/.ssh/id_rsa.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
pg-primary ⇒ Copy repository public key to pg-primary 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@repository cat /home/pgbackrest/.ssh/id_rsa.pub) | \
       sudo -u postgres tee -a /var/lib/postgresql/.ssh/authorized_keys
Test that connections can be made from repository to pg-primary and vice versa.
repository ⇒ Test connection from repository to pg-primary 
sudo -u pgbackrest ssh postgres@pg-primary
pg-primary ⇒ Test connection from pg-primary to repository 
sudo -u postgres ssh pgbackrest@repository
NOTE:
ssh has been configured to only allow pgBackRest to be run via passwordless ssh. This enhances security in the event that one of the service accounts is hijacked.
Configuration
The repository host must be configured with the pg-primary host/user and database path. The primary will be configured as pg1 to allow a standby to be added later.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg1-host/pg1-host-user and pg1-path
Config suppressed for testing
The database host must be configured with the repository host/user. The default for the repo1-host-user option is pgbackrest. If the postgres user does restores on the repository host it is best not to also allow the postgres user to perform backups. However, the postgres user can read the repository directly if it is in the same group as the pgbackrest user.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure repo1-host/repo1-host-user
Config suppressed for testing
PostgreSQL configuration may be found in the Configure Archiving section.
Commands are run the same as on a single host configuration except that some commands such as backup and expire are run from the repository host instead of the database host.
Create and Check Stanza
Create the stanza in the new repository.
repository ⇒ Create the stanza 
sudo -u pgbackrest pgbackrest --stanza=demo stanza-create
Output suppressed for testing
Check that the configuration is correct on both the database and repository hosts. More information about the check command can be found in Check the Configuration.
pg-primary ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
repository ⇒ Check the configuration 
sudo -u pgbackrest pgbackrest --stanza=demo check
Output suppressed for testing
Perform a Backup
To perform a backup of the PostgreSQL cluster run pgBackRest with the backup command on the repository host.
repository ⇒ Backup the demo cluster 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
Since a new repository was created on the repository host the warning about the incremental backup changing to a full backup was emitted.
Restore a Backup
To perform a restore of the PostgreSQL cluster run pgBackRest with the restore command on the database host.
pg-primary ⇒ Stop the demo cluster, restore, and restart PostgreSQL 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta restore
sudo pg_ctlcluster 17 demo start
Parallel Backup / Restore
pgBackRest offers parallel processing to improve performance of compression and transfer. The number of processes to be used for this feature is set using the --process-max option.
It is usually best not to use more than 25% of available CPUs for the backup command. Backups don't have to run that fast as long as they are performed regularly and the backup process should not impact database performance, if at all possible.
The restore command can and should use all available CPUs because during a restore the PostgreSQL cluster is shut down and there is generally no other important work being done on the host. If the host contains multiple clusters then that should be considered when setting restore parallelism.
repository ⇒ Perform a backup with single process 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest to use multiple backup processes
Config suppressed for testing
repository ⇒ Perform a backup with multiple processes 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
repository ⇒ Get backup info for the demo cluster 
sudo -u pgbackrest pgbackrest info
Output suppressed for testing
The performance of the last backup should be improved by using multiple processes. For very small backups the difference may not be very apparent, but as the size of the database increases so will time savings.
Starting and Stopping
If a standby is promoted for testing, or a test cluster is restored from a production backup, then it is a good idea to prevent those clusters from writing to pgBackRest repositories. This can be accomplished with the stop command.
The commands that write and are blocked by stop are: archive-push, backup, expire, stanza-create, and stanza-upgrade. Note that stanza-delete is an exception to this rule (see Delete a Stanza for more details).
pg-primary ⇒ Stop pgBackRest write commands 
sudo -u postgres pgbackrest stop
New pgBackRest write commands will no longer run.
repository ⇒ Attempt a backup 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
Specify the --force option to terminate any pgBackRest write commands that are currently running. This includes asynchronous archive-get (though it will run again if PostgreSQL requires it). If pgBackRest is already stopped then stopping again will generate a warning.
pg-primary ⇒ Stop the pgBackRest services again 
sudo -u postgres pgbackrest stop
Output suppressed for testing
Start pgBackRest write commands again with the start command. Write commands that were in progress before the stop will not automatically start again, but they are now allowed to start.
pg-primary ⇒ Start pgBackRest write commands 
sudo -u postgres pgbackrest start
It is also possible to stop pgBackRest for a single stanza.
pg-primary ⇒ Stop pgBackRest write commands for the demo stanza 
sudo -u postgres pgbackrest --stanza=demo stop
New pgBackRest write commands for the specified stanza will no longer run.
repository ⇒ Attempt a backup 
sudo -u pgbackrest pgbackrest --stanza=demo backup
Output suppressed for testing
The stanza must also be specified when starting pgBackRest write commands for a single stanza.
pg-primary ⇒ Start pgBackRest write commands for the demo stanza 
sudo -u postgres pgbackrest --stanza=demo start
Replication
Replication allows multiple copies of a PostgreSQL cluster (called standbys) to be created from a single primary. The standbys are useful for balancing reads and to provide redundancy in case the primary host fails.
Installation
A new host named pg-standby is created to run the standby.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
Debian/Ubuntu packages for pgBackRest are available at apt.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-standby ⇒ Install dependencies 
sudo apt-get install postgresql-client libxml2 libssh2-1
pg-standby ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-standby ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
Setup Passwordless SSH
pgBackRest can use passwordless SSH to enable communication between the hosts. It is also possible to use TLS, see Setup TLS.
pg-standby ⇒ Create pg-standby host key pair 
sudo -u postgres mkdir -m 750 -p /var/lib/postgresql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/postgresql/.ssh/id_rsa \
       -t rsa -b 4096 -N ""
Exchange keys between repository and pg-standby.
repository ⇒ Copy pg-standby public key to repository 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@pg-standby cat /var/lib/postgresql/.ssh/id_rsa.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
pg-standby ⇒ Copy repository public key to pg-standby 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@repository cat /home/pgbackrest/.ssh/id_rsa.pub) | \
       sudo -u postgres tee -a /var/lib/postgresql/.ssh/authorized_keys
Test that connections can be made from repository to pg-standby and vice versa.
repository ⇒ Test connection from repository to pg-standby 
sudo -u pgbackrest ssh postgres@pg-standby
pg-standby ⇒ Test connection from pg-standby to repository 
sudo -u postgres ssh pgbackrest@repository
Hot Standby
A hot standby performs replication using the WAL archive and allows read-only queries.
pgBackRest configuration is very similar to pg-primary except that the standby recovery type will be used to keep the cluster in recovery mode when the end of the WAL stream has been reached.
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest on the standby
Config suppressed for testing
The demo cluster must be created (even though it will be overwritten on restore) in order to create the PostgreSQL configuration files.
pg-standby ⇒ Create demo cluster 
sudo pg_createcluster 17 demo
Now the standby can be created with the restore command.
IMPORTANT:
If the cluster is intended to be promoted without becoming the new primary (e.g. for reporting or testing), use --archive-mode=off or set archive_mode=off in postgresql.conf to disable archiving. If archiving is not disabled then the repository may be polluted with WAL that can make restores more difficult.
pg-standby ⇒ Restore the demo standby cluster 
sudo -u postgres pgbackrest --stanza=demo --delta --type=standby restore
sudo -u postgres cat /var/lib/postgresql/17/demo/postgresql.auto.conf
Output suppressed for testing
The configuration is in case the standby is promoted to a primary.
pg-standby:/etc/postgresql/17/demo/postgresql.conf ⇒ Configure PostgreSQL
Config suppressed for testing
pg-standby ⇒ Start PostgreSQL 
sudo pg_ctlcluster 17 demo start
The PostgreSQL log gives valuable information about the recovery. Note especially that the cluster has entered standby mode and is ready to accept read-only connections.
pg-standby ⇒ Examine the PostgreSQL log output for log messages indicating success 
sudo -u postgres cat /var/log/postgresql/postgresql-17-demo.log
Output suppressed for testing
An easy way to test that replication is properly configured is to create a table on pg-primary.
pg-primary ⇒ Create a new table on the primary 
sudo -u postgres psql -c " \
       begin; \
       create table replicated_table (message text); \
       insert into replicated_table values ('Important Data'); \
       commit; \
       select * from replicated_table";
Output suppressed for testing
And then query the same table on pg-standby.
pg-standby ⇒ Query new table on the standby 
sudo -u postgres psql -c "select * from replicated_table;"
Output suppressed for testing
So, what went wrong? Since PostgreSQL is pulling WAL segments from the archive to perform replication, changes won't be seen on the standby until the WAL segment that contains those changes is pushed from pg-primary.
This can be done manually by calling pg_switch_wal() which pushes the current WAL segment to the archive (a new WAL segment is created to contain further changes).
pg-primary ⇒ Call pg_switch_wal() 
sudo -u postgres psql -c "select *, current_timestamp from pg_switch_wal()";
Output suppressed for testing
Now after a short delay the table will appear on pg-standby.
pg-standby ⇒ Now the new table exists on the standby (may require a few retries) 
sudo -u postgres psql -c " \
       select *, current_timestamp from replicated_table"
Output suppressed for testing
Check the standby configuration for access to the repository.
pg-standby ⇒ Check the configuration 
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Streaming Replication
Instead of relying solely on the WAL archive, streaming replication makes a direct connection to the primary and applies changes as soon as they are made on the primary. This results in much less lag between the primary and standby.
Streaming replication requires a user with the replication privilege.
pg-primary ⇒ Create replication user 
sudo -u postgres psql -c " \
       create user replicator password 'jw8s0F4' replication";
Output suppressed for testing
The pg_hba.conf file must be updated to allow the standby to connect as the replication user. Be sure to replace the IP address below with the actual IP address of your pg-standby. A reload will be required after modifying the pg_hba.conf file.
pg-primary ⇒ Create pg_hba.conf entry for replication user 
sudo -u postgres sh -c 'echo \
       "host    replication     replicator      {[host-pg2-ip]}/32           md5" \
       >> /etc/postgresql/17/demo/pg_hba.conf'
sudo pg_ctlcluster 17 demo reload
The standby needs to know how to contact the primary so the primary_conninfo setting will be configured in pgBackRest.
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Set primary_conninfo
Config suppressed for testing
It is possible to configure a password in the primary_conninfo setting but using a .pgpass file is more flexible and secure.
pg-standby ⇒ Configure the replication password in the .pgpass file. 
sudo -u postgres sh -c 'echo \
       "{[host-pg1-ip]}:*:replication:replicator:jw8s0F4" \
       >> /var/lib/postgresql/.pgpass'
sudo -u postgres chmod 600 /var/lib/postgresql/.pgpass
Now the standby can be created with the restore command.
pg-standby ⇒ Stop PostgreSQL and restore the demo standby cluster 
sudo pg_ctlcluster 17 demo stop
sudo -u postgres pgbackrest --stanza=demo --delta --type=standby restore
sudo -u postgres cat /var/lib/postgresql/17/demo/postgresql.auto.conf
Output suppressed for testing
NOTE:
The primary_conninfo setting has been written into the postgresql.auto.conf file because it was configured as a recovery-option in pgbackrest.conf. The --type=preserve option can be used with the restore to leave the existing postgresql.auto.conf file in place if that behavior is preferred.
pg-standby ⇒ Start PostgreSQL 
sudo pg_ctlcluster 17 demo start
The PostgreSQL log will confirm that streaming replication has started.
pg-standby ⇒ Examine the PostgreSQL log output for log messages indicating success 
sudo -u postgres cat /var/log/postgresql/postgresql-17-demo.log
Output suppressed for testing
Now when a table is created on pg-primary it will appear on pg-standby quickly and without the need to call pg_switch_wal().
pg-primary ⇒ Create a new table on the primary 
sudo -u postgres psql -c " \
       begin; \
       create table stream_table (message text); \
       insert into stream_table values ('Important Data'); \
       commit; \
       select *, current_timestamp from stream_table";
Output suppressed for testing
pg-standby ⇒ Query table on the standby 
sudo -u postgres psql -c " \
       select *, current_timestamp from stream_table"
Output suppressed for testing
Multiple Stanzas
pgBackRest supports multiple stanzas. The most common usage is sharing a repository host among multiple stanzas.
Installation
A new host named pg-alt is created to run the new primary.
Installing pgBackRest from a package is preferable to building from source. When installing from a package the rest of the instructions in this section are generally not required, but it is possible that a package will skip creating one of the directories or apply incorrect permissions. In that case it may be necessary to manually create directories or update permissions.
Debian/Ubuntu packages for pgBackRest are available at apt.postgresql.org.
If packages are not provided for your distribution/version you can build from source and then install manually as shown here.
pg-alt ⇒ Install dependencies 
sudo apt-get install postgresql-client libxml2 libssh2-1
pg-alt ⇒ Copy pgBackRest binary from build host 
sudo scp build:/build/pgbackrest/src/pgbackrest /usr/bin
sudo chmod 755 /usr/bin/pgbackrest
pgBackRest requires log and configuration directories and a configuration file.
pg-alt ⇒ Create pgBackRest configuration file and directories 
sudo mkdir -p -m 770 /var/log/pgbackrest
sudo chown postgres:postgres /var/log/pgbackrest
sudo mkdir -p /etc/pgbackrest
sudo mkdir -p /etc/pgbackrest/conf.d
sudo touch /etc/pgbackrest/pgbackrest.conf
sudo chmod 640 /etc/pgbackrest/pgbackrest.conf
sudo chown postgres:postgres /etc/pgbackrest/pgbackrest.conf
Setup Passwordless SSH
pgBackRest can use passwordless SSH to enable communication between the hosts. It is also possible to use TLS, see Setup TLS.
pg-alt ⇒ Create pg-alt host key pair 
sudo -u postgres mkdir -m 750 -p /var/lib/postgresql/.ssh
sudo -u postgres ssh-keygen -f /var/lib/postgresql/.ssh/id_rsa \
       -t rsa -b 4096 -N ""
Exchange keys between repository and pg-alt.
repository ⇒ Copy pg-alt public key to repository 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@pg-alt cat /var/lib/postgresql/.ssh/id_rsa.pub) | \
       sudo -u pgbackrest tee -a /home/pgbackrest/.ssh/authorized_keys
pg-alt ⇒ Copy repository public key to pg-alt 
(echo -n 'no-agent-forwarding,no-X11-forwarding,no-port-forwarding,' && \
       echo -n 'command="/usr/bin/pgbackrest ${SSH_ORIGINAL_COMMAND#* }" ' && \
       sudo ssh root@repository cat /home/pgbackrest/.ssh/id_rsa.pub) | \
       sudo -u postgres tee -a /var/lib/postgresql/.ssh/authorized_keys
Test that connections can be made from repository to pg-alt and vice versa.
repository ⇒ Test connection from repository to pg-alt 
sudo -u pgbackrest ssh postgres@pg-alt
pg-alt ⇒ Test connection from pg-alt to repository 
sudo -u postgres ssh pgbackrest@repository
Configuration
pgBackRest configuration is nearly identical to pg-primary except that the demo-alt stanza will be used so backups and archive will be stored in a separate location.
pg-alt:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pgBackRest on the new primary
Config suppressed for testing
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg1-host/pg1-host-user and pg1-path
Config suppressed for testing
Setup Demo Cluster
pg-alt ⇒ Create the demo cluster 
sudo -u postgres /usr/lib/postgresql/17/bin/initdb \
       -D /var/lib/postgresql/17/demo -k -A peer
sudo pg_createcluster 17 demo
Output suppressed for testing
pg-alt:/etc/postgresql/17/demo/postgresql.conf ⇒ Configure PostgreSQL settings
Config suppressed for testing
pg-alt ⇒ Start the demo cluster 
sudo pg_ctlcluster 17 demo restart
Create the Stanza and Check Configuration
The stanza-create command must be run to initialize the stanza. It is recommended that the check command be run after stanza-create to ensure archiving and backups are properly configured.
pg-alt ⇒ Create the stanza and check the configuration 
sudo -u postgres pgbackrest --stanza=demo-alt --log-level-console=info stanza-create
Output suppressed for testing
sudo -u postgres pgbackrest --log-level-console=info check
Output suppressed for testing
If the check command is run from the repository host then all stanzas will be checked.
repository ⇒ Check the configuration for all stanzas 
sudo -u pgbackrest pgbackrest --log-level-console=info check
Output suppressed for testing
Asynchronous Archiving
Asynchronous archiving is enabled with the archive-async option. This option enables asynchronous operation for both the archive-push and archive-get commands.
A spool path is required. The commands will store transient data here but each command works quite a bit differently so spool path usage is described in detail in each section.
pg-primary ⇒ Create the spool directory 
sudo mkdir -p -m 750 /var/spool/pgbackrest
sudo chown postgres:postgres /var/spool/pgbackrest
pg-standby ⇒ Create the spool directory 
sudo mkdir -p -m 750 /var/spool/pgbackrest
sudo chown postgres:postgres /var/spool/pgbackrest
The spool path must be configured and asynchronous archiving enabled. Asynchronous archiving automatically confers some benefit by reducing the number of connections made to remote storage, but setting process-max can drastically improve performance by parallelizing operations. Be sure not to set process-max so high that it affects normal database operations.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the spool path and asynchronous archiving
Config suppressed for testing
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Configure the spool path and asynchronous archiving
Config suppressed for testing
NOTE:
process-max is configured using command sections so that the option is not used by backup and restore. This also allows different values for archive-push and archive-get.
For demonstration purposes streaming replication will be broken to force PostgreSQL to get WAL using the restore_command.
pg-primary ⇒ Break streaming replication by changing the replication password 
sudo -u postgres psql -c "alter user replicator password 'bogus'"
Output suppressed for testing
pg-standby ⇒ Restart standby to break connection 
sudo pg_ctlcluster 17 demo restart
Archive Push
The asynchronous archive-push command offloads WAL archiving to a separate process (or processes) to improve throughput. It works by looking ahead to see which WAL segments are ready to be archived beyond the request that PostgreSQL is currently making via the archive_command. WAL segments are transferred to the archive directly from the pg_xlog/pg_wal directory and success is only returned by the archive_command when the WAL segment has been safely stored in the archive.
The spool path holds the current status of WAL archiving. Status files written into the spool directory are typically zero length and should consume a minimal amount of space (a few MB at most) and very little IO. All the information in this directory can be recreated so it is not necessary to preserve the spool directory if the cluster is moved to new hardware.
IMPORTANT:
In the original implementation of asynchronous archiving, WAL segments were copied to the spool directory before compression and transfer. The new implementation copies WAL directly from the pg_xlog directory. If asynchronous archiving was utilized in v1.12 or prior, read the v1.13 release notes carefully before upgrading.
The [stanza]-archive-push-async.log file can be used to monitor the activity of the asynchronous process. A good way to test this is to quickly push a number of WAL segments.
pg-primary ⇒ Test parallel asynchronous archiving 
sudo -u postgres psql -c " \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal(); \
       select pg_create_restore_point('test async push'); select pg_switch_wal();"
sudo -u postgres pgbackrest --stanza=demo --log-level-console=info check
Output suppressed for testing
Now the log file will contain parallel, asynchronous activity.
pg-primary ⇒ Check results in the log 
sudo -u postgres cat /var/log/pgbackrest/demo-archive-push-async.log
Output suppressed for testing
Archive Get
The asynchronous archive-get command maintains a local queue of WAL to improve throughput. If a WAL segment is not found in the queue it is fetched from the repository along with enough consecutive WAL to fill the queue. The maximum size of the queue is defined by archive-get-queue-max. Whenever the queue is less than half full more WAL will be fetched to fill it.
Asynchronous operation is most useful in environments that generate a lot of WAL or have a high latency connection to the repository storage (i.e., S3 or other object stores). In the case of a high latency connection it may be a good idea to increase process-max.
The [stanza]-archive-get-async.log file can be used to monitor the activity of the asynchronous process.
pg-standby ⇒ Check results in the log 
sudo -u postgres cat /var/log/pgbackrest/demo-archive-get-async.log
Output suppressed for testing
pg-primary ⇒ Fix streaming replication by changing the replication password 
sudo -u postgres psql -c "alter user replicator password 'jw8s0F4'"
Output suppressed for testing
Backup from a Standby
pgBackRest can perform backups on a standby instead of the primary. Standby backups require the pg-standby host to be configured and the backup-standby option enabled. If more than one standby is configured then the first running standby found will be used for the backup.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Configure pg2-host/pg2-host-user and pg2-path
Config suppressed for testing
Both the primary and standby databases are required to perform the backup, though the vast majority of the files will be copied from the standby to reduce load on the primary. The database hosts can be configured in any order. pgBackRest will automatically determine which is the primary and which is the standby.
repository ⇒ Backup the demo cluster from pg2 
sudo -u pgbackrest pgbackrest --stanza=demo --log-level-console=detail backup
Output suppressed for testing
This incremental backup shows that most of the files are copied from the pg-standby host and only a few are copied from the pg-primary host.
pgBackRest creates a standby backup that is identical to a backup performed on the primary. It does this by starting/stopping the backup on the pg-primary host, copying only files that are replicated from the pg-standby host, then copying the remaining few files from the pg-primary host. This means that logs and statistics from the primary database will be included in the backup.
Upgrading PostgreSQL
Immediately after upgrading PostgreSQL to a newer major version, the pg-path for all pgBackRest configurations must be set to the new database location and the stanza-upgrade command run. If there is more than one repository configured on the host, the stanza will be upgraded on each. If the database is offline use the --no-online option.
The following instructions are not meant to be a comprehensive guide for upgrading PostgreSQL, rather they outline the general process for upgrading a primary and standby with the intent of demonstrating the steps required to reconfigure pgBackRest. It is recommended that a backup be taken prior to upgrading.
pg-primary ⇒ Stop old cluster 
sudo pg_ctlcluster 17 demo stop
Stop the old cluster on the standby since it will be restored from the newly upgraded cluster.
pg-standby ⇒ Stop old cluster 
sudo pg_ctlcluster 17 demo stop
Create the new cluster and perform upgrade.
pg-primary ⇒ Create new cluster and perform the upgrade 
sudo -u postgres /usr/lib/postgresql/18/bin/initdb \
       -D /var/lib/postgresql/18/demo -k -A peer
sudo pg_createcluster 18 demo
sudo -u postgres sh -c 'cd /var/lib/postgresql && \
       /usr/lib/postgresql/18/bin/pg_upgrade \
       --old-bindir=/usr/lib/postgresql/17/bin \
       --new-bindir=/usr/lib/postgresql/18/bin \
       --old-datadir=/var/lib/postgresql/17/demo \
       --new-datadir=/var/lib/postgresql/18/demo \
       --old-options=" -c config_file=/etc/postgresql/17/demo/postgresql.conf" \
       --new-options=" -c config_file=/etc/postgresql/18/demo/postgresql.conf"'
Output suppressed for testing
Configure the new cluster settings and port.
pg-primary:/etc/postgresql/18/demo/postgresql.conf ⇒ Configure PostgreSQL
Config suppressed for testing
Update the pgBackRest configuration on all systems to point to the new cluster.
pg-primary:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade the pg1-path
Config suppressed for testing
pg-standby:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade the pg-path
Config suppressed for testing
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Upgrade pg1-path and pg2-path, disable backup from standby
Config suppressed for testing
pg-primary ⇒ Copy hba configuration 
sudo cp /etc/postgresql/17/demo/pg_hba.conf \
       /etc/postgresql/18/demo/pg_hba.conf
Before starting the new cluster, the stanza-upgrade command must be run.
pg-primary ⇒ Upgrade the stanza 
sudo -u postgres pgbackrest --stanza=demo --no-online \
       --log-level-console=info stanza-upgrade
Output suppressed for testing
Start the new cluster and confirm it is successfully installed.
pg-primary ⇒ Start new cluster 
sudo pg_ctlcluster 18 demo start
Output suppressed for testing
Test configuration using the check command.
pg-primary ⇒ Check configuration 
sudo pg_lsclusters
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
Remove the old cluster.
pg-primary ⇒ Remove old cluster 
sudo pg_dropcluster 17 demo
Install the new PostgreSQL binaries on the standby and create the cluster.
pg-standby ⇒ Remove old cluster and create the new cluster 
sudo pg_dropcluster 17 demo
sudo pg_createcluster 18 demo
Run the check on the repository host. The warning regarding the standby being down is expected since the standby cluster is down. Running this command demonstrates that the repository server is aware of the standby and is configured properly for the primary server.
repository ⇒ Check configuration 
sudo -u pgbackrest pgbackrest --stanza=demo check
Output suppressed for testing
Run a full backup on the new cluster and then restore the standby from the backup. The backup type will automatically be changed to full if incr or diff is requested.
repository ⇒ Run a full backup 
sudo -u pgbackrest pgbackrest --stanza=demo --type=full backup
pg-standby ⇒ Restore the demo standby cluster 
sudo -u postgres pgbackrest --stanza=demo --delta --type=standby restore
pg-standby ⇒ Start PostgreSQL and check the pgBackRest configuration 
sudo pg_ctlcluster 18 demo start
sudo -u postgres pgbackrest --stanza=demo check
Output suppressed for testing
Backup from standby can be enabled now that the standby is restored.
repository:/etc/pgbackrest/pgbackrest.conf ⇒ Reenable backup from standby
Config suppressed for testing
Copyright © 2015-2026, The PostgreSQL Global Development Group, MIT License. Updated August 17, 2026
././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1786957821.772618 pgbackrest-2.59.1/doc/man/0000750000175100017510000000000015240547776014737 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003300000000000010211 xustar0027 mtime=1786957821.772618 pgbackrest-2.59.1/doc/man/pgbackrest.1.txt0000640000175100017510000004230715240547776017773 0ustar00runnerrunnerNAME pgBackRest - Reliable PostgreSQL Backup & Restore SYNOPSIS pgbackrest [options] [command] DESCRIPTION pgBackRest is a reliable backup and restore solution for PostgreSQL that seamlessly scales up to the largest databases and workloads. COMMANDS annotate Add or modify backup annotation. archive-get Get a WAL segment from the archive. archive-push Push a WAL segment to the archive. backup Backup a database cluster. check Check the configuration. expire Expire backups that exceed retention. help Get help. info Retrieve information about backups. repo-get Get a file from a repository. repo-ls List files in a repository. restore Restore a database cluster. server pgBackRest server. server-ping Ping pgBackRest server. stanza-create Create the required stanza data. stanza-delete Delete a stanza. stanza-upgrade Upgrade a stanza. start Allow pgBackRest processes to run. stop Stop pgBackRest processes from running. verify Verify contents of a repository. version Get version. OPTIONS Annotate Options: --set Backup set to annotate. Archive Options: --archive-async Push/get WAL segments asynchronously. --archive-get-queue-max Maximum size of the pgBackRest archive-get queue. --archive-missing-retry Retry missing WAL segment --archive-push-batch-size Maximum amount of WAL to push per asynchronous run. --archive-push-queue-max Maximum size of the PostgreSQL archive queue. --archive-timeout Archive timeout. Backup Options: --annotation Annotate backup with user-defined key/value pairs. --archive-check Check that WAL segments are in the archive before backup completes. --archive-copy Copy WAL segments needed for consistency to the backup. --archive-mode-check Check the PostgreSQL archive_mode setting. --backup-standby Backup from the standby cluster. --checksum-page Validate data page checksums. --exclude Exclude paths/files from the backup. --expire-auto Automatically run the expire command after a successful backup. --force Force an offline backup. --manifest-save-threshold Manifest save threshold during backup. --online Perform an online backup. --resume Allow resume of failed backup. --start-fast Force a checkpoint to start backup quickly. --type Backup type. Expire Options: --archive-expire-before Remove WAL archive earlier than the specified WAL segment. --oldest Expire the oldest eligible backup set. --set Backup set to expire. General Options: --allow-root Allow the command to run as the root user. --buffer-size Buffer size for I/O operations. --cmd pgBackRest command. --cmd-ssh SSH client command. --compress Use file compression. --compress-level File compression level. --compress-level-network Network compression level. --compress-type File compression type. --config pgBackRest configuration file. --config-include-path Path to additional pgBackRest configuration files. --config-path Base path of pgBackRest configuration files. --db-timeout Database query timeout. --delta Restore or backup using checksums. --dry-run Execute a dry-run for the command. --io-timeout I/O timeout. --lock-path Path where lock files are stored. --neutral-umask Use a neutral umask. --priority Set process priority. --process-max Max processes to use for compress/transfer. --protocol-timeout Protocol timeout. --raw Do not transform data. --sck-keep-alive Keep-alive enable. --spool-path Path where transient data is stored. --stanza Defines the stanza. --tcp-keep-alive-count Keep-alive count. --tcp-keep-alive-idle Keep-alive idle time. --tcp-keep-alive-interval Keep-alive interval time. --tls-cipher-12 Allowed TLSv1.2 cipher suites. --tls-cipher-13 Allowed TLSv1.3 cipher suites. Help Options: --help Display help. --version Display version. Info Options: --detail-level Output detail level. --output Output format. --set Backup set to detail. --type Filter on backup type. Log Options: --log-level-console Level for console logging. --log-level-file Level for file logging. --log-level-stderr Level for stderr logging. --log-path Path where log files are stored. --log-subprocess Enable logging in subprocesses. --log-timestamp Enable timestamp in logging. Maintainer Options: --archive-header-check Check PostgreSQL version/id in WAL headers. --page-header-check Check PostgreSQL page headers. --pg-version-force Force PostgreSQL version. Repo-get Options: --ignore-missing Ignore missing source file. Repo-ls Options: --filter Filter output with a regular expression. --output Output format. --recurse Include all subpaths in output. --sort Sort output ascending, descending, or none. Repository Options: --repo Set repository. --repo-azure-account Azure repository account. --repo-azure-container Azure repository container. --repo-azure-endpoint Azure repository endpoint. --repo-azure-key Azure repository key. --repo-azure-key-type Azure repository key type. --repo-azure-uri-style Azure URI Style. --repo-block Enable block incremental backup. --repo-bundle Bundle files in repository. --repo-bundle-limit Limit for file bundles. --repo-bundle-size Target size for file bundles. --repo-cipher-pass Repository cipher passphrase. --repo-cipher-type Cipher used to encrypt the repository. --repo-gcs-bucket GCS repository bucket. --repo-gcs-endpoint GCS repository endpoint. --repo-gcs-key GCS repository key. --repo-gcs-key-type GCS repository key type. --repo-gcs-user-project GCS project ID. --repo-hardlink Hardlink files between backups in the repository. --repo-host Repository host when operating remotely. --repo-host-ca-file Repository host certificate authority file. --repo-host-ca-path Repository host certificate authority path. --repo-host-cert-file Repository host certificate file. --repo-host-cmd Repository host pgBackRest command. --repo-host-config pgBackRest repository host configuration file. --repo-host-config-include-path pgBackRest repository host configuration include path. --repo-host-config-path pgBackRest repository host configuration path. --repo-host-key-file Repository host key file. --repo-host-port Repository host port when repo-host is set. --repo-host-type Repository host protocol type. --repo-host-user Repository host user when repo-host is set. --repo-path Path where backups and archive are stored. --repo-retention-archive Number of backups worth of continuous WAL to retain. --repo-retention-archive-type Backup type for WAL retention. --repo-retention-diff Number of differential backups to retain. --repo-retention-full Full backup retention count/time. --repo-retention-full-type Retention type for full backups. --repo-retention-history Days of backup history manifests to retain. --repo-s3-bucket S3 repository bucket. --repo-s3-endpoint S3 repository endpoint. --repo-s3-key S3 repository access key. --repo-s3-key-secret S3 repository secret access key. --repo-s3-key-type S3 repository key type. --repo-s3-kms-key-id S3 repository KMS key. --repo-s3-process-cmd S3 authentication process command. --repo-s3-region S3 repository region. --repo-s3-requester-pays S3 repository requester pays. --repo-s3-role S3 repository role. --repo-s3-service S3 signing service. --repo-s3-sse-customer-key S3 repository SSE customer key. --repo-s3-sts-host S3 repository STS endpoint. --repo-s3-token S3 repository security token. --repo-s3-uri-style S3 URI Style. --repo-sftp-host SFTP repository host. --repo-sftp-host-fingerprint SFTP repository host fingerprint. --repo-sftp-host-key-check-type SFTP host key check type. --repo-sftp-host-key-hash-type SFTP repository host key hash type. --repo-sftp-host-port SFTP repository host port. --repo-sftp-host-user SFTP repository host user. --repo-sftp-known-host SFTP known hosts file. --repo-sftp-private-key-file SFTP private key file. --repo-sftp-private-key-passphrase SFTP private key passphrase. --repo-sftp-public-key-file SFTP public key file. --repo-storage-ca-file Repository storage CA file. --repo-storage-ca-path Repository storage CA path. --repo-storage-host Repository storage host. --repo-storage-port Repository storage port. --repo-storage-tag Repository storage tag(s). --repo-storage-upload-chunk-size Repository storage upload chunk size. --repo-storage-verify-tls Repository storage certificate verify. --repo-symlink Create symlinks within the repository. --repo-target-time Target time for repository. --repo-type Type of storage used for the repository. Restore Options: --archive-mode Preserve or disable archiving on restored cluster. --db-exclude Restore excluding the specified databases. --db-include Restore only specified databases. --force Force a restore. --link-all Restore all symlinks. --link-map Modify the destination of a symlink. --recovery-option Set an option in postgresql.auto.conf or recovery.conf. --set Backup set to restore. --tablespace-map Restore a tablespace into the specified directory. --tablespace-map-all Restore all tablespaces into the specified directory. --target Recovery target. --target-action Action to take when recovery target is reached. --target-exclusive Stop just before the recovery target is reached. --target-timeline Recover along a timeline. --type Recovery type. Server Options: --tls-server-address TLS server address. --tls-server-auth TLS server authorized clients. --tls-server-ca-file TLS server certificate authorities. --tls-server-cert-file TLS server certificate file. --tls-server-key-file TLS server key file. --tls-server-port TLS server port. Stanza Options: --pg-database PostgreSQL database. --pg-host PostgreSQL host for operating remotely. --pg-host-ca-file PostgreSQL host certificate authority file. --pg-host-ca-path PostgreSQL host certificate authority path. --pg-host-cert-file PostgreSQL host certificate file. --pg-host-cmd PostgreSQL host pgBackRest command. --pg-host-config pgBackRest database host configuration file. --pg-host-config-include-path pgBackRest database host configuration include path. --pg-host-config-path pgBackRest database host configuration path. --pg-host-key-file PostgreSQL host key file. --pg-host-port PostgreSQL host port when pg-host is set. --pg-host-type PostgreSQL host protocol type. --pg-host-user PostgreSQL host logon user when pg-host is set. --pg-path PostgreSQL data directory. --pg-port PostgreSQL port. --pg-socket-path PostgreSQL unix socket path. --pg-user PostgreSQL database user. Stanza-create Options: --online Create on an online cluster. Stanza-delete Options: --force Force stanza delete. Stanza-upgrade Options: --online Update an online cluster. Stop Options: --force Force all pgBackRest processes to stop. Verify Options: --output Output type. --set Backup set to verify. --verbose Verbose output. Version Options: --output Output type. FILES /etc/pgbackrest/pgbackrest.conf /var/lib/pgbackrest /var/log/pgbackrest /var/spool/pgbackrest /tmp/pgbackrest EXAMPLES * Create a backup of the PostgreSQL `main` cluster: $ pgbackrest --stanza=main backup The `main` cluster should be configured in `/etc/pgbackrest/pgbackrest.conf` * Show all available backups: $ pgbackrest info * Show all available backups for a specific cluster: $ pgbackrest --stanza=main info * Show backup specific options: $ pgbackrest help backup SEE ALSO /usr/share/doc/pgbackrest-doc/html/index.html https://pgbackrest.org ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/meson.build0000664000175100017510000003004715240547630015561 0ustar00runnerrunner#################################################################################################################################### # pgBackRest Project #################################################################################################################################### project( 'pgbackrest', ['c'], version: '2.59.1', license: 'MIT', meson_version: '>=0.47', default_options: [ # Core options 'buildtype=release', 'warning_level=2', # Base options 'b_ndebug=if-release', # Perform unity builds in a single file 'unity_size=100000', # Compiler options 'c_std=c11', ], ) # Selected C compiler cc = meson.get_compiler('c') #################################################################################################################################### # Check for C11 compiler support (setting c_std has no effect -- works in 1.8.1 so likely a bug in our version) #################################################################################################################################### c11_test = ''' #if !defined __STDC_VERSION__ || __STDC_VERSION__ < 201112L # error "C compiler does not advertise C11 conformance" #endif ''' if not cc.compiles(c11_test) if cc.compiles(c11_test, args: '-std=c11') add_project_arguments('-std=c11', language: 'c') else error('C compiler does not support C11') endif endif #################################################################################################################################### # OS-specific settings #################################################################################################################################### if host_machine.system() == 'linux' add_global_arguments('-D_POSIX_C_SOURCE=200809L', language: 'c') elif host_machine.system() == 'darwin' add_global_arguments('-D_DARWIN_C_SOURCE', language: 'c') endif #################################################################################################################################### # Enable/disable warnings #################################################################################################################################### # Enable as many additional warnings as possible to catch potential errors warning_enable = [ # Warn whenever a pointer is cast so as to remove a type qualifier from the target type '-Wcast-qual', # Warn for implicit conversions that may alter a value '-Wconversion', # Warn about duplicated conditions in an if-else-if chain '-Wduplicated-cond', # Warn when an if-else has identical branches '-Wduplicated-branches', # Warn if floating-point values are used in equality comparisons '-Wfloat-equal', # Enable -Wformat plus -Wnonnull, -Wformat-nonliteral, -Wformat-security, and -Wformat-y2k '-Wformat=2', # Warn about calls to formatted input/output functions such as sprintf and vsprintf that might overflow the destination buffer '-Wformat-overflow=2', # Warn if the format string requires an unsigned argument and the argument is signed and vice versa '-Wformat-signedness', # Warn about uninitialized variables which are initialized with themselves '-Winit-self', # Warn if a global function is defined without a previous prototype declaration '-Wmissing-prototypes', # Warn if a global variable is defined without a previous declaration '-Wmissing-variable-declarations', # Warn about anything that depends on the "size of" a function type or of void '-Wpointer-arith', # Warn if anything is declared more than once in the same scope, even when the extra declaration is valid and changes nothing '-Wredundant-decls', # Warn if a function is declared or defined without specifying the argument types '-Wstrict-prototypes', # Warn if a variable-length array is used '-Wvla', # Give string constants the type const char[length] so that copying the address of one into a non-const char * pointer produces # a warning '-Wwrite-strings', ] # Disable various unhelpful warnings warning_disable = [ # Warn for variables that might be changed by longjmp or vfork. Disable because of constant false positives/negatives. '-Wno-clobbered', # Warn if a structure's initializer has some fields missing. Disable so we can initialize with {0}. '-Wno-missing-field-initializers', ] add_project_arguments(cc.get_supported_arguments(warning_enable, warning_disable), language: 'c') #################################################################################################################################### # Enable additional optimizations for appropriate levels #################################################################################################################################### if get_option('optimization') in ['2', '3'] optimization_enable = [ # Unroll loops whose number of iterations can be determined at compile time or upon entry to the loop '-funroll-loops', # Perform loop vectorization on trees '-ftree-vectorize', ] add_project_arguments(cc.get_supported_arguments(optimization_enable), language: 'c') endif #################################################################################################################################### # Stop after the first error when error on warn enabled. Subsequent errors are often caused by the first error. #################################################################################################################################### if get_option('fatal-errors') add_project_arguments(cc.get_supported_arguments('-Wfatal-errors'), language: 'c') endif #################################################################################################################################### # Generate -fmacro-prefix-map so the relative path to the source directory is not included in the __FILE__ macro. This provides # reproducible builds and minimizes the file path in debug messages, just like an in-tree make build. For test source, prefix with # test/ in case there are any module name collisions. #################################################################################################################################### python = import('python').find_installation() file_prefix = run_command( [ python, '-c', 'import sys, os; print(os.path.relpath(sys.argv[1], sys.argv[2]))', meson.current_source_dir(), meson.current_build_dir(), ], check: true, ).stdout().strip() add_project_arguments(cc.get_supported_arguments('-fmacro-prefix-map=@0@/src/=' . format(file_prefix)), language: 'c') add_project_arguments(cc.get_supported_arguments('-fmacro-prefix-map=@0@/test/src/=test/' . format(file_prefix)), language: 'c') #################################################################################################################################### # Detect a distribution tarball, which is marked by the dist file written into the tarball root by meson dist. A distribution # tarball ships the generated files, so the code generation tooling (build-code, the doc and test tools, libyaml) is not built. A # normal source tree such as a git checkout has no marker and generates the files. #################################################################################################################################### dist = run_command( [ python, '-c', 'import os, sys; sys.exit(0 if os.path.exists(os.path.join(sys.argv[1], "dist")) else 1)', meson.current_source_dir(), ], check: false, ).returncode() == 0 #################################################################################################################################### # Build configuration #################################################################################################################################### configuration = configuration_data() # Ensure the C compiler supports __builtin_clzl() if not cc.links('''int main(void) {__builtin_clzl(1);}''') error('compiler must support __builtin_clzl()') endif # Ensure the C compiler supports __builtin_bswap64() if not cc.links('''int main(void) {__builtin_bswap64(1);}''') error('compiler must support __builtin_bswap64()') endif # Check C compiler support for __builtin_types_compatible_p if cc.compiles('''int main(void) {__builtin_types_compatible_p(char *, const char *);}''') configuration.set('HAVE_BUILTIN_TYPES_COMPATIBLE_P', true, description: 'Is __builtin_types_compatible_p present?') endif # Find optional backtrace library lib_backtrace = cc.find_library('backtrace', required: false) if lib_backtrace.found() configuration.set('HAVE_LIBBACKTRACE', true, description: 'Is libbacktrace present?') endif # Find required bz2 library lib_bz2 = cc.find_library('bz2') # Find required lz4 library lib_lz4 = dependency('liblz4') # Find required openssl library lib_openssl = dependency('openssl', version : '>=1.1.1') # Find required pq library lib_pq = dependency('libpq') # Find required xml library lib_xml = dependency('libxml-2.0') # Find yaml library (only used by the code generation tooling, which is not built from a distribution tarball) if not dist lib_yaml = dependency('yaml-0.1') endif # Find required gz library lib_z = dependency('zlib') configuration.set('ZLIB_CONST', true, description: 'Require zlib const input buffer') # Find optional systemd library lib_systemd = dependency('libsystemd', required: get_option('libsystemd')) if lib_systemd.found() configuration.set('HAVE_LIBSYSTEMD', true, description: 'Is libsystemd present?') endif # Find optional libssh2 library lib_ssh2 = dependency('libssh2', required: get_option('libssh2')) if lib_ssh2.found() configuration.set('HAVE_LIBSSH2', true, description: 'Is libssh2 present?') endif # Find optional zstd library lib_zstd = dependency('libzstd', version: '>=1.0', required: get_option('libzstd')) if lib_zstd.found() configuration.set('HAVE_LIBZST', true, description: 'Is libzstd present?') endif # Suppress -Wsign-conversion for C libraries (e.g. musl) that do no accept int without warning for FD_*() macros if not cc.compiles( '''#include int main(void) {int fd = 1; fd_set s; FD_ZERO(&s); FD_SET(fd, &s); FD_ISSET(fd, &s);}''', args: ['-Wsign-conversion', '-Werror']) message('Disabling -Wsign-conversion because int is not accepted without warning by FD_*() macros') add_project_arguments(cc.get_supported_arguments('-Wno-sign-conversion'), language: 'c') endif # Enable debug code if get_option('debug') configuration.set('DEBUG', true, description: 'Enable debug code') endif # Set configuration path configuration.set_quoted('CFGOPTDEF_CONFIG_PATH', get_option('configdir'), description: 'Configuration path') # Set VR_NON_STRING macro if cc.compiles( '''int main(void) {__attribute__((nonstring)) static char test[3] = "ABC";}''', args: ['-Wattributes', '-Werror']) configuration.set( 'VR_NON_STRING', '__attribute__((nonstring))', description: 'Indicate that the char array is intentionally not NULL terminated') else configuration.set( 'VR_NON_STRING', '', description: 'Indicate that the char array is intentionally not NULL terminated') endif #################################################################################################################################### # Include subdirs #################################################################################################################################### subdir('src') # The smoke test is shipped in the distribution so packagers can verify a build. It is in a subdir because unit test builds comment # out all subdirs, which excludes the smoke test from them. subdir('test') # The doc and test tools generate code and are not built from a distribution tarball if not dist subdir('doc/src') subdir('test/src') # Bake the generated files and the dist marker into the tarball so it builds without the code generation tooling meson.add_dist_script('src/build/dist.sh') endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/meson_options.txt0000664000175100017510000000076415240547630017057 0ustar00runnerrunneroption('configdir', type: 'string', value: '/etc/pgbackrest', description: 'Configuration directory') option('fatal-errors', type: 'boolean', value: false, description: 'Stop compilation on first error') option('libssh2', type: 'feature', value: 'auto', description: 'Enable SFTP storage support') option('libzstd', type: 'feature', value: 'auto', description: 'Enable Zstandard compression support') option('libsystemd', type: 'feature', value: 'auto', description: 'Enable systemd notify support') ././@PaxHeader0000000000000000000000000000003200000000000010210 xustar0026 mtime=1786957821.73262 pgbackrest-2.59.1/src/0000775000175100017510000000000015240547776014215 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/.gitignore0000664000175100017510000000023715240547630016174 0ustar00runnerrunner/.build autom4te.cache /command/help/help.auto.c.inc /config.log /config.status /Makefile /build.auto.h /pgbackrest /postgres/interface.auto.c.inc /build-code ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/build.h0000664000175100017510000000474715240547630015466 0ustar00runnerrunner/*********************************************************************************************************************************** Build Header This header file must be included in all source files. It contains defines that are required in all modules. The build.auto.c header generated by meson is also included here. ***********************************************************************************************************************************/ #ifndef BUILD_H #define BUILD_H /*********************************************************************************************************************************** Ban unsafe functions Poison unsafe functions so any use is a compile error. System headers containing poisoned identifiers must be processed first because build.h is the first include in every file and poisoned identifiers are not exempt inside system headers. #undef prevents a "poisoning existing macro" warning where fortify has defined the function as a macro. ***********************************************************************************************************************************/ #include #include #undef sprintf #undef strcat #undef strcpy #undef strncpy #pragma GCC poison sprintf strcat strcpy strncpy /*********************************************************************************************************************************** Include build.auto.h and other (hopefully small) header files that are commonly used. ***********************************************************************************************************************************/ #include #include /*********************************************************************************************************************************** Function attributes ***********************************************************************************************************************************/ // Indicate that a function should always be inlined #define FN_INLINE_ALWAYS __attribute__((__always_inline__)) static inline // Indicate that a function is formatted like printf (and provide format and args position) #define FN_PRINTF(fmt, args) __attribute__((format(printf, fmt, args))) // Indicate that a function is formatted like strftime (and provide format position) #define FN_STRFTIME(fmt) __attribute__((format(strftime, fmt, 0))) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/0000775000175100017510000000000015240547630015620 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/annotate/0000775000175100017510000000000015240547630017431 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/annotate/annotate.c0000664000175100017510000000744615240547630021421 0ustar00runnerrunner/*********************************************************************************************************************************** Annotate Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/annotate/annotate.h" #include "command/backup/backup.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "config/config.h" #include "info/infoBackup.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdAnnotate(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Initialize the repo index unsigned int repoIdxMin = 0; unsigned int repoIdxMax = cfgOptionGroupIdxTotal(cfgOptGrpRepo) - 1; // If the repo was specified then set index to the array location and max to loop only once if (cfgOptionTest(cfgOptRepo)) { repoIdxMin = cfgOptionGroupIdxDefault(cfgOptGrpRepo); repoIdxMax = repoIdxMin; } // Check the backup label format const String *const backupLabel = cfgOptionStr(cfgOptSet); if (!regExpMatchOne(backupRegExpP(.full = true, .differential = true, .incremental = true), backupLabel)) THROW_FMT(OptionInvalidValueError, "'%s' is not a valid backup label format", strZ(backupLabel)); // Track the number of backup sets to update in the backup info file and any errors that may occur unsigned int backupTotalProcessed = 0; unsigned int errorTotal = 0; for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { TRY_BEGIN() { // Attempt to load the backup info file const CipherType repoCipherType = cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx); InfoBackup *infoBackup = infoBackupLoadFileReconstruct( storageRepoIdx(repoIdx), INFO_BACKUP_PATH_FILE_STR, repoCipherType, cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); if (infoBackupLabelExists(infoBackup, backupLabel)) { // Backup label found in backup.info backupTotalProcessed++; LOG_INFO_FMT( "backup set '%s' to annotate found in %s", strZ(backupLabel), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); // Update annotations infoBackupDataAnnotationSet(infoBackup, backupLabel, cfgOptionKv(cfgOptAnnotation)); // Write the updated backup info infoBackupSaveFile( infoBackup, storageRepoWrite(), INFO_BACKUP_PATH_FILE_STR, repoCipherType, cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); } } CATCH_ANY() { LOG_ERROR_FMT(errorTypeCode(errorType()), "%s: %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorMessage()); errorTotal++; } TRY_END(); } // Error if any errors encountered on one or more repos if (errorTotal > 0) THROW_FMT(CommandError, CFGCMD_ANNOTATE " command encountered %u error(s), check the log file for details", errorTotal); if (backupTotalProcessed == 0) THROW(BackupSetInvalidError, "no backup set to annotate found"); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/annotate/annotate.h0000664000175100017510000000130615240547630021413 0ustar00runnerrunner/*********************************************************************************************************************************** Annotate Command ***********************************************************************************************************************************/ #ifndef COMMAND_ANNOTATE_ANNOTATE_H #define COMMAND_ANNOTATE_ANNOTATE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add or modify backup annotations FN_EXTERN void cmdAnnotate(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/0000775000175100017510000000000015240547630017241 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/common.c0000664000175100017510000004033215240547630020677 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Common ***********************************************************************************************************************************/ #include #include #include #include "command/archive/common.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/type/convert.h" #include "common/wait.h" #include "config/config.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** WAL segment constants ***********************************************************************************************************************************/ STRING_EXTERN(WAL_SEGMENT_REGEXP_STR, WAL_SEGMENT_REGEXP); STRING_EXTERN(WAL_SEGMENT_PARTIAL_REGEXP_STR, WAL_SEGMENT_PARTIAL_REGEXP); STRING_EXTERN(WAL_SEGMENT_DIR_REGEXP_STR, WAL_SEGMENT_DIR_REGEXP); STRING_EXTERN(WAL_SEGMENT_FILE_REGEXP_STR, WAL_SEGMENT_FILE_REGEXP); STRING_EXTERN(WAL_TIMELINE_HISTORY_REGEXP_STR, WAL_TIMELINE_HISTORY_REGEXP); /*********************************************************************************************************************************** Global error file constant ***********************************************************************************************************************************/ #define STATUS_FILE_GLOBAL "global" STRING_STATIC(STATUS_FILE_GLOBAL_STR, STATUS_FILE_GLOBAL); #define STATUS_FILE_GLOBAL_ERROR STATUS_FILE_GLOBAL STATUS_EXT_ERROR STRING_STATIC(STATUS_FILE_GLOBAL_ERROR_STR, STATUS_FILE_GLOBAL_ERROR); /*********************************************************************************************************************************** Get the correct spool queue based on the archive mode ***********************************************************************************************************************************/ static const String * archiveAsyncSpoolQueue(const ArchiveMode archiveMode) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, archiveMode); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST( STRING, archiveMode == archiveModeGet ? STORAGE_SPOOL_ARCHIVE_IN_STR : STORAGE_SPOOL_ARCHIVE_OUT_STR); } /**********************************************************************************************************************************/ FN_EXTERN void archiveAsyncErrorClear(const ArchiveMode archiveMode, const String *const archiveFile) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, archiveMode); FUNCTION_LOG_PARAM(STRING, archiveFile); FUNCTION_LOG_END(); ASSERT(archiveFile != NULL); String *const errorFile = strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s" STATUS_EXT_ERROR, strZ(archiveFile)); storageRemoveP(storageSpoolWrite(), errorFile); storageRemoveP(storageSpoolWrite(), STRDEF(STORAGE_SPOOL_ARCHIVE_OUT "/" STATUS_FILE_GLOBAL_ERROR)); strFree(errorFile); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool archiveAsyncStatus(const ArchiveMode archiveMode, const String *const walSegment, const bool throwOnError, const bool warnOnOk) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, archiveMode); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(BOOL, throwOnError); FUNCTION_LOG_PARAM(BOOL, warnOnOk); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { const String *errorFile = NULL; bool errorFileExists = false; const String *const spoolQueue = archiveAsyncSpoolQueue(archiveMode); String *const okFile = strNewFmt("%s" STATUS_EXT_OK, strZ(walSegment)); const bool okFileExists = storageExistsP(storageSpool(), strNewFmt("%s/%s", strZ(spoolQueue), strZ(okFile))); // If the ok file does not exist then check to see if a file-specific or global error exists if (!okFileExists) { // Check for a file-specific error first errorFile = strNewFmt("%s" STATUS_EXT_ERROR, strZ(walSegment)); errorFileExists = storageExistsP(storageSpool(), strNewFmt("%s/%s", strZ(spoolQueue), strZ(errorFile))); // If that doesn't exist then check for a global error if (!errorFileExists) { errorFile = STATUS_FILE_GLOBAL_ERROR_STR; errorFileExists = storageExistsP(storageSpool(), strNewFmt("%s/%s", strZ(spoolQueue), strZ(errorFile))); } } // If either of them exists then check what happened and report back if (okFileExists || errorFileExists) { // Get the status file content const String *const statusFile = okFileExists ? okFile : errorFile; const String *const content = strNewBuf( storageGetP(storageNewReadP(storageSpool(), strNewFmt("%s/%s", strZ(spoolQueue), strZ(statusFile))))); // Get the code and message if the file has content int code = 0; const String *message = NULL; if (strSize(content) != 0) { // Find the line feed after the error code -- should be the first one const char *const linefeedPtr = strchr(strZ(content), '\n'); // Error if linefeed not found if (linefeedPtr == NULL) THROW_FMT(FormatError, "%s content must have at least two lines", strZ(statusFile)); // Error if message is zero-length if (strlen(linefeedPtr + 1) == 0) THROW_FMT(FormatError, "%s message must be > 0", strZ(statusFile)); // Get contents code = varIntForce(VARSTR(strNewZN(strZ(content), (size_t)(linefeedPtr - strZ(content))))); message = strTrim(strNewZ(linefeedPtr + 1)); } // Process OK files if (okFileExists) { // If there is content in the status file it is a warning if (strSize(content) != 0 && warnOnOk) { // If error code is not success, then this was a renamed error file if (code != 0) { message = strNewFmt( "WAL segment '%s' was not pushed due to error [%d] and was manually skipped: %s", strZ(walSegment), code, strZ(message)); } LOG_WARN(strZ(message)); } result = true; } else if (throwOnError) { // Error status files must have content if (strSize(content) == 0) THROW_FMT(AssertError, "status file '%s' has no content", strZ(statusFile)); // Throw error using the code passed in the file THROW_CODE(code, strZ(message)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void archiveAsyncStatusErrorWrite( const ArchiveMode archiveMode, const String *const walSegment, const int code, const String *const message) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, archiveMode); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(INT, code); FUNCTION_LOG_PARAM(STRING, message); FUNCTION_LOG_END(); ASSERT(code != 0); ASSERT(message != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const errorFile = walSegment == NULL ? STATUS_FILE_GLOBAL_STR : walSegment; storagePutP( storageNewWriteP( storageSpoolWrite(), strNewFmt("%s/%s" STATUS_EXT_ERROR, strZ(archiveAsyncSpoolQueue(archiveMode)), strZ(errorFile))), BUFSTR(strNewFmt("%d\n%s", code, strZ(message)))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void archiveAsyncStatusOkWrite(const ArchiveMode archiveMode, const String *const walSegment, const String *const warning) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, archiveMode); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(STRING, warning); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Write file storagePutP( storageNewWriteP( storageSpoolWrite(), strNewFmt("%s/%s" STATUS_EXT_OK, strZ(archiveAsyncSpoolQueue(archiveMode)), strZ(walSegment))), warning == NULL ? NULL : BUFSTR(strNewFmt("0\n%s", strZ(warning)))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int archiveIdComparator(const void *const archiveId1, const void *const archiveId2) { ASSERT(strstr(strZ(*(const String *const *)archiveId1), "-") != NULL); ASSERT(strstr(strZ(*(const String *const *)archiveId2), "-") != NULL); const int id1 = cvtZToInt(strstr(strZ(*(const String *const *)archiveId1), "-") + 1); const int id2 = cvtZToInt(strstr(strZ(*(const String *const *)archiveId2), "-") + 1); return LST_COMPARATOR_CMP(id1, id2); } /**********************************************************************************************************************************/ FN_EXTERN bool walIsPartial(const String *const walSegment) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); ASSERT(walIsSegment(walSegment)); FUNCTION_LOG_RETURN(BOOL, strEndsWithZ(walSegment, WAL_SEGMENT_PARTIAL_EXT)); } /**********************************************************************************************************************************/ FN_EXTERN String * walPath(const String *walFile, const String *pgPath, const String *command) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walFile); FUNCTION_LOG_PARAM(STRING, pgPath); FUNCTION_LOG_PARAM(STRING, command); FUNCTION_LOG_END(); ASSERT(walFile != NULL); ASSERT(command != NULL); String *result; if (!strBeginsWithZ(walFile, "/")) { // Error if walFile has a relative path and pgPath is not set if (pgPath == NULL) { THROW_FMT( OptionRequiredError, "option '%s' must be specified when relative wal paths are used\n" "HINT: is %%f passed to %s instead of %%p?\n" "HINT: PostgreSQL may pass relative paths even with %%p depending on the environment.", cfgOptionName(cfgOptPgPath), strZ(command)); } // Get the working directory char currentWorkDir[4096]; THROW_ON_SYS_ERROR(getcwd(currentWorkDir, sizeof(currentWorkDir)) == NULL, FormatError, "unable to get cwd"); // Check if the working directory is the same as pgPath if (!strEqZ(pgPath, currentWorkDir)) { // If not we'll change the working directory to pgPath and see if that equals the working directory we got called with THROW_ON_SYS_ERROR_FMT(chdir(strZ(pgPath)) != 0, PathMissingError, "unable to chdir() to '%s'", strZ(pgPath)); // Get the new working directory char newWorkDir[4096]; THROW_ON_SYS_ERROR(getcwd(newWorkDir, sizeof(newWorkDir)) == NULL, FormatError, "unable to get cwd"); // Error if the new working directory is not equal to the original current working directory. This means that PostgreSQL // and pgBackrest have a different idea about where the PostgreSQL data directory is located. if (strcmp(currentWorkDir, newWorkDir) != 0) { THROW_FMT( OptionInvalidValueError, PG_NAME " working directory '%s' is not the same as option %s '%s'\n" "HINT: is the " PG_NAME " data_directory configured the same as the %s option?", currentWorkDir, cfgOptionName(cfgOptPgPath), strZ(pgPath), cfgOptionName(cfgOptPgPath)); } } result = strNewFmt("%s/%s", strZ(pgPath), strZ(walFile)); } else result = strDup(walFile); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN bool walIsSegment(const String *const walSegment) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); // Create the regular expression to identify WAL segments if it does not already exist static RegExp *regExpSegment = NULL; if (regExpSegment == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { regExpSegment = regExpNew(WAL_SEGMENT_PARTIAL_REGEXP_STR); } MEM_CONTEXT_END(); } FUNCTION_LOG_RETURN(BOOL, regExpMatch(regExpSegment, walSegment)); } /**********************************************************************************************************************************/ FN_EXTERN String * walSegmentNext(const String *const walSegment, const size_t walSegmentSize, const unsigned int pgVersion) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(SIZE, walSegmentSize); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); ASSERT(strSize(walSegment) == 24); ASSERT(UINT32_MAX % walSegmentSize == walSegmentSize - 1); // Extract WAL parts uint32_t timeline; uint32_t major; uint32_t minor; MEM_CONTEXT_TEMP_BEGIN() { timeline = pgTimelineFromWalSegment(walSegment); major = (uint32_t)strtol(strZ(strSubN(walSegment, 8, 8)), NULL, 16); minor = (uint32_t)strtol(strZ(strSubN(walSegment, 16, 8)), NULL, 16); // Increment minor and adjust major dir on overflow minor++; if (minor > UINT32_MAX / walSegmentSize) { major++; minor = 0; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, strNewFmt("%08X%08X%08X", timeline, major, minor)); } /**********************************************************************************************************************************/ FN_EXTERN StringList * walSegmentRange( const String *const walSegmentBegin, const size_t walSegmentSize, const unsigned int pgVersion, const unsigned int range) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walSegmentBegin); FUNCTION_LOG_PARAM(SIZE, walSegmentSize); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_END(); ASSERT(range > 0); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { strLstAdd(result, walSegmentBegin); if (range > 1) { String *current = strDup(walSegmentBegin); for (unsigned int rangeIdx = 0; rangeIdx < range - 1; rangeIdx++) { String *const next = walSegmentNext(current, walSegmentSize, pgVersion); strLstAdd(result, next); strFree(current); current = next; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/common.h0000664000175100017510000001267715240547630020717 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Common ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_COMMON_H #define COMMAND_ARCHIVE_COMMON_H #include #include "common/type/stringId.h" /*********************************************************************************************************************************** Archive mode enum Used for functions that are common to both archive-push and archive-get so they can tailor their behavior to the command being run. ***********************************************************************************************************************************/ typedef enum { archiveModeGet = STRID5("get", 0x50a70), archiveModePush = STRID5("push", 0x44eb00), } ArchiveMode; #include "common/compress/helper.h" #include "common/type/stringList.h" #include "storage/storage.h" /*********************************************************************************************************************************** Status file extension constants ***********************************************************************************************************************************/ #define STATUS_EXT_ERROR ".error" #define STATUS_EXT_ERROR_SIZE (sizeof(STATUS_EXT_ERROR) - 1) #define STATUS_EXT_OK ".ok" #define STATUS_EXT_OK_SIZE (sizeof(STATUS_EXT_OK) - 1) /*********************************************************************************************************************************** WAL segment constants ***********************************************************************************************************************************/ // Extension for partial segments #define WAL_SEGMENT_PARTIAL_EXT ".partial" // Match when the first 24 characters match a WAL segment #define WAL_SEGMENT_PREFIX_REGEXP "^[0-F]{24}" // Match on a WAL segment without checksum appended #define WAL_SEGMENT_REGEXP WAL_SEGMENT_PREFIX_REGEXP "$" STRING_DECLARE(WAL_SEGMENT_REGEXP_STR); // Match on a WAL segment with partial allowed #define WAL_SEGMENT_PARTIAL_REGEXP WAL_SEGMENT_PREFIX_REGEXP "(\\.partial){0,1}$" STRING_DECLARE(WAL_SEGMENT_PARTIAL_REGEXP_STR); // Defines the size of standard WAL segment name -- hopefully this won't change #define WAL_SEGMENT_NAME_SIZE ((unsigned int)24) // WAL segment directory/file #define WAL_SEGMENT_DIR_REGEXP "^[0-F]{16}$" STRING_DECLARE(WAL_SEGMENT_DIR_REGEXP_STR); #define WAL_SEGMENT_FILE_REGEXP "^[0-F]{24}-[0-f]{40}" COMPRESS_TYPE_REGEXP "{0,1}$" STRING_DECLARE(WAL_SEGMENT_FILE_REGEXP_STR); // Timeline history file #define WAL_TIMELINE_HISTORY_REGEXP "^[0-F]{8}.history$" STRING_DECLARE(WAL_TIMELINE_HISTORY_REGEXP_STR); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Remove errors for an archive file. This should be done before forking the async process to prevent a race condition where an old // error may be reported rather than waiting for the async process to succeed or fail. FN_EXTERN void archiveAsyncErrorClear(ArchiveMode archiveMode, const String *archiveFile); // Check for ok/error status files in the spool in/out directory. throwOnError determines whether an error will be thrown when an // error file is found. warnOnOk determines whether a warning will be output when found in an ok file. FN_EXTERN bool archiveAsyncStatus(ArchiveMode archiveMode, const String *walSegment, bool throwOnError, bool warnOnOk); // Write an ok status file FN_EXTERN void archiveAsyncStatusOkWrite(ArchiveMode archiveMode, const String *walSegment, const String *warning); // Write an error status file FN_EXTERN void archiveAsyncStatusErrorWrite(ArchiveMode archiveMode, const String *walSegment, int code, const String *message); // Comparator function for sorting archive ids by the database history id (the number after the dash) e.g. 9.6-1, 10-2 FN_EXTERN int archiveIdComparator(const void *item1, const void *item2); // Is the segment partial? FN_EXTERN bool walIsPartial(const String *walSegment); // Is the file a segment or some other file (e.g. .history, .backup, etc) FN_EXTERN bool walIsSegment(const String *walSegment); // Generates the location of the wal directory using a relative wal path and the supplied pg path FN_EXTERN String *walPath(const String *walFile, const String *pgPath, const String *command); // Get the next WAL segment given a WAL segment and WAL segment size FN_EXTERN String *walSegmentNext(const String *walSegment, size_t walSegmentSize, unsigned int pgVersion); // Build a list of WAL segments based on a beginning WAL and number of WAL in the range (inclusive) FN_EXTERN StringList *walSegmentRange( const String *walSegmentBegin, size_t walSegmentSize, unsigned int pgVersion, unsigned int range); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/find.c0000664000175100017510000002231415240547630020327 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Segment Find ***********************************************************************************************************************************/ #include #include "command/archive/common.h" #include "command/archive/find.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/wait.h" #include "storage/helper.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct WalSegmentFind { const Storage *storage; // Storage to find WAL in const String *archiveId; // Archive id to find segments in bool single; // Optimize for a single segment? TimeMSec timeout; // Timeout for each segment String *prefix; // Current list prefix StringList *list; // List of found segments }; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_WAL_SEGMENT_FIND_TYPE \ WalSegmentFind * #define FUNCTION_LOG_WAL_SEGMENT_FIND_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "WalSegmentFind", buffer, bufferSize) /**********************************************************************************************************************************/ FN_EXTERN WalSegmentFind * walSegmentFindNew(const Storage *const storage, const String *const archiveId, const bool single, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(BOOL, single); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(archiveId != NULL); OBJ_NEW_BEGIN(WalSegmentFind, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (WalSegmentFind) { .storage = storage, .archiveId = strDup(archiveId), .single = single, .timeout = timeout, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(WAL_SEGMENT_FIND, this); } /**********************************************************************************************************************************/ FN_EXTERN String * walSegmentFind(WalSegmentFind *const this, const String *const walSegment) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(WAL_SEGMENT_FIND, this); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); ASSERT(walIsSegment(walSegment)); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { Wait *const wait = waitNew(this->timeout); const String *const prefix = strSubN(walSegment, 0, 16); const String *const path = strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(this->archiveId), strZ(prefix)); const String *const expression = strNewFmt( "^%s%s-[0-f]{40}" COMPRESS_TYPE_REGEXP "{0,1}$", strZ(strSubN(walSegment, 0, 24)), walIsPartial(walSegment) ? WAL_SEGMENT_PARTIAL_EXT : ""); RegExp *regExp = NULL; do { // Get a list of all WAL segments that match the directory (and prefix when finding a single WAL) if (this->list == NULL || !strEq(prefix, this->prefix)) { MEM_CONTEXT_OBJ_BEGIN(this) { // Free and store prefix if (!strEq(prefix, this->prefix)) { strFree(this->prefix); this->prefix = strDup(prefix); } // Free list strLstFree(this->list); // Get list this->list = strLstSort( storageListP(this->storage, path, .expression = this->single ? expression : NULL), sortOrderAsc); } MEM_CONTEXT_OBJ_END(); } // If there are results if (!strLstEmpty(this->list)) { // By default the match size is the list size since filtering happened above. When not finding a single WAL then // non-matching entries before the matching WAL will need to be removed and then the matching WAL counted. unsigned int match = strLstSize(this->list); if (!this->single) { // Build regexp if not yet built if (regExp == NULL) regExp = regExpNew(expression); // Remove list items that do not match. This prevents us from having check them again on the next find. while (!strLstEmpty(this->list) && !regExpMatch(regExp, strLstGet(this->list, 0))) strLstRemoveIdx(this->list, 0); // Find matches at the beginning of the remaining list match = 0; while (match < strLstSize(this->list) && regExpMatch(regExp, strLstGet(this->list, match))) match++; } // Error if there is more than one match if (match > 1) { // Build list of duplicate WAL StringList *const matchList = strLstNew(); for (unsigned int matchIdx = 0; matchIdx < match; matchIdx++) strLstAdd(matchList, strLstGet(this->list, matchIdx)); // Clear list for next find strLstFree(this->list); this->list = NULL; THROW_FMT( ArchiveDuplicateError, "duplicates found in archive for WAL segment %s: %s\n" "HINT: are multiple primaries archiving to this stanza?", strZ(walSegment), strZ(strLstJoin(matchList, ", "))); } // On match copy file name of WAL segment found into the prior context if (match == 1) { MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(strLstGet(this->list, 0)); } MEM_CONTEXT_PRIOR_END(); } // Remove matching entries so list will be reloaded when empty if (!this->single) { ASSERT(regExp != NULL); while (!strLstEmpty(this->list) && regExpMatch(regExp, strLstGet(this->list, 0))) strLstRemoveIdx(this->list, 0); } } // Clear list for next find if (this->single || strLstEmpty(this->list)) { strLstFree(this->list); this->list = NULL; } } while (result == NULL && waitMore(wait)); } MEM_CONTEXT_TEMP_END(); // Error if segment not found before timeout if (result == NULL && this->timeout != 0) { THROW_FMT( ArchiveTimeoutError, "WAL segment %s was not archived before the %" PRIu64 "ms timeout\n" "HINT: check the archive_command to ensure that all options are correct (especially --stanza).\n" "HINT: check the PostgreSQL server log for errors.\n" "HINT: run the 'start' command if the stanza was previously stopped.", strZ(walSegment), this->timeout); } FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * walSegmentFindOne( const Storage *const storage, const String *const archiveId, const String *const walSegment, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); String *result; MEM_CONTEXT_TEMP_BEGIN() { WalSegmentFind *const find = walSegmentFindNew(storage, archiveId, true, timeout); MEM_CONTEXT_PRIOR_BEGIN() { result = walSegmentFind(find, walSegment); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/find.h0000664000175100017510000000452015240547630020333 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Segment Find Find a WAL segment (or segments) in a repository. The code paths for finding single or multiple WAL segments are both optimized. ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_FIND_H #define COMMAND_ARCHIVE_FIND_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct WalSegmentFind WalSegmentFind; #include "common/type/string.h" #include "storage/storage.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN WalSegmentFind *walSegmentFindNew(const Storage *storage, const String *archiveId, bool single, TimeMSec timeout); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Find a WAL segment in the repository. The file name can have several things appended such as a hash, compression extension, and // partial extension so it is possible to have multiple files that match the segment, though more than one match is not a good // thing. FN_EXTERN String *walSegmentFind(WalSegmentFind *this, const String *walSegment); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Find a single WAL segment (see walSegmentFind() for details) FN_EXTERN String *walSegmentFindOne(const Storage *storage, const String *archiveId, const String *walSegment, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/get/0000775000175100017510000000000015240547630020020 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/get/file.c0000664000175100017510000001000615240547630021100 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get File ***********************************************************************************************************************************/ #include #include "command/archive/common.h" #include "command/archive/get/file.h" #include "command/control/common.h" #include "common/compress/helper.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/io/filter/group.h" #include "common/log.h" #include "config/config.h" #include "info/infoArchive.h" #include "postgres/interface.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN ArchiveGetFileResult archiveGetFile( const Storage *const storage, const String *const request, const List *const actualList, const String *const walDestination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, request); FUNCTION_LOG_PARAM(LIST, actualList); FUNCTION_LOG_PARAM(STRING, walDestination); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(request != NULL); ASSERT(actualList != NULL && !lstEmpty(actualList)); ASSERT(walDestination != NULL); ArchiveGetFileResult result = {.warnList = strLstNew()}; // Check all files in the actual list and return as soon as one is copied bool copied = false; for (unsigned int actualIdx = 0; actualIdx < lstSize(actualList); actualIdx++) { const ArchiveGetFile *const actual = lstGet(actualList, actualIdx); // Is the file compressible during the copy? bool compressible = true; TRY_BEGIN() { MEM_CONTEXT_TEMP_BEGIN() { StorageWrite *const destination = storageNewWriteP( storage, walDestination, .noCreatePath = true, .noSyncFile = true, .noSyncPath = true, .noAtomic = true); // If there is a cipher then add the decrypt filter if (actual->cipherType != cipherTypeNone) { ioFilterGroupAdd( ioWriteFilterGroup(storageWriteIo(destination)), cipherBlockNewP(cipherModeDecrypt, actual->cipherType, BUFSTR(actual->cipherPassArchive))); compressible = false; } // If file is compressed then add the decompression filter CompressType compressType = compressTypeFromName(actual->file); if (compressType != compressTypeNone) { ioFilterGroupAdd(ioWriteFilterGroup(storageWriteIo(destination)), decompressFilterP(compressType)); compressible = false; } // Copy the file storageCopyP( storageNewReadP( storageRepoIdx(actual->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(actual->file)), .compressible = compressible), destination); } MEM_CONTEXT_TEMP_END(); // File was successfully copied result.actualIdx = actualIdx; copied = true; } // Log errors as warnings and continue CATCH_ANY() { strLstAddFmt( result.warnList, "%s: %s [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, actual->repoIdx), strZ(actual->file), errorTypeName(errorType()), errorMessage()); } TRY_END(); // Stop on success if (copied) break; } // If no file was successfully copied then error if (!copied) { ASSERT(!strLstEmpty(result.warnList)); THROW_FMT(FileReadError, "unable to get %s:\n%s", strZ(request), strZ(strLstJoin(result.warnList, "\n"))); } FUNCTION_LOG_RETURN_STRUCT(result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/get/file.h0000664000175100017510000000334715240547630021117 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get File ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_GET_FILE_H #define COMMAND_ARCHIVE_GET_FILE_H #include "common/crypto/common.h" #include "common/type/string.h" #include "storage/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Copy a file from the archive to the specified destination typedef struct ArchiveGetFile { const String *file; // File in the repo (with path, checksum, ext, etc.) unsigned int repoIdx; // Repo idx const String *archiveId; // Repo archive id CipherType cipherType; // Repo cipher type const String *cipherPassArchive; // Repo archive cipher pass } ArchiveGetFile; typedef struct ArchiveGetFileResult { unsigned int actualIdx; // Index of the file from actual list that was retrieved StringList *warnList; // Warnings from a successful operation } ArchiveGetFileResult; FN_EXTERN ArchiveGetFileResult archiveGetFile( const Storage *storage, const String *request, const List *actualList, const String *walDestination); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/get/get.c0000664000175100017510000015136115240547630020752 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get Command ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "command/archive/common.h" #include "command/archive/get/file.h" #include "command/archive/get/get.h" #include "command/archive/get/protocol.h" #include "command/command.h" #include "command/lock.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/wait.h" #include "config/config.h" #include "config/exec.h" #include "config/load.h" #include "info/infoArchive.h" #include "postgres/interface.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" #include "storage/write.h" /*********************************************************************************************************************************** Constants for log messages that are used multiple times to keep them consistent ***********************************************************************************************************************************/ #define FOUND_IN_ARCHIVE_MSG "found %s in the archive" #define FOUND_IN_REPO_ARCHIVE_MSG "found %s in the %s: %s archive" #define UNABLE_TO_FIND_IN_ARCHIVE_MSG "unable to find %s in the archive" #define UNABLE_TO_FIND_VALID_REPO_MSG "unable to find a valid repository" #define REPO_INVALID_OR_ERR_MSG "some repositories were invalid or encountered errors" /*********************************************************************************************************************************** Check for a list of archive files in the repository ***********************************************************************************************************************************/ typedef struct ArchiveFileMap { const String *request; // Archive file requested by archive_command List *actualList; // Actual files in various repos/archiveIds StringList *warnList; // Warnings that need to be reported by the async process } ArchiveFileMap; typedef struct ArchiveGetCheckResult { List *archiveFileMapList; // List of mapped archive files, i.e. found in the repo // Global error that affects all repos const ErrorType *errorType; // Error type if there was an error const String *errorFile; // Error file if there was an error const String *errorMessage; // Error message if there was an error const StringList *warnList; // Warnings that need to be reported by the async process } ArchiveGetCheckResult; // Helper to add an error to an error list and warn if the error is not already in the list static void archiveGetErrorAdd( StringList *const warnList, const bool log, const unsigned int repoIdx, const ErrorType *const type, const String *const message) { const String *const warn = strNewFmt( "%s: [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(type), strZ(message)); if (!strLstExists(warnList, warn)) { if (log) LOG_WARN(strZ(warn)); strLstAdd(warnList, warn); } } // Helper to find a single archive file in the repository using a cache to speed up the process and minimize storageListP() calls typedef struct ArchiveGetFindCachePath { const String *path; // Cached path in the archiveId const StringList *fileList; // List of files in the cache path } ArchiveGetFindCachePath; typedef struct ArchiveGetFindCacheArchive { const String *archiveId; // ArchiveId in the repo List *pathList; // List of paths cached for archiveId } ArchiveGetFindCacheArchive; typedef struct ArchiveGetFindCacheRepo { unsigned int repoIdx; CipherType cipherType; // Repo cipher type const String *cipherPassArchive; // Repo archive cipher pass List *archiveList; // Cached list of archiveIds and associated paths StringList *warnList; // Track repo warnings so each is only reported once } ArchiveGetFindCacheRepo; static bool archiveGetFind( const String *const archiveFileRequest, ArchiveGetCheckResult *const getCheckResult, List *const cacheRepoList, const StringList *const warnList, const bool single) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, archiveFileRequest); FUNCTION_LOG_PARAM_P(VOID, getCheckResult); FUNCTION_LOG_PARAM(LIST, cacheRepoList); FUNCTION_LOG_PARAM(STRING_LIST, warnList); FUNCTION_LOG_PARAM(BOOL, single); FUNCTION_LOG_END(); ASSERT(archiveFileRequest != NULL); ASSERT(getCheckResult != NULL); ASSERT(cacheRepoList != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // Is the archive file a WAL segment? const bool isSegment = walIsSegment(archiveFileRequest); // Get the WAL segment path const String *const path = isSegment ? strSubN(archiveFileRequest, 0, 16) : NULL; // List to hold matches for the requested file List *const matchList = lstNewP(sizeof(ArchiveGetFile), .comparator = lstComparatorStr); // List of file level warnings StringList *const fileWarnList = strLstDup(warnList); // Errored repo total to track if all repos errored unsigned int repoErrorTotal = 0; // Check each repo for (unsigned int repoCacheIdx = 0; repoCacheIdx < lstSize(cacheRepoList); repoCacheIdx++) { const ArchiveGetFindCacheRepo *const cacheRepo = lstGet(cacheRepoList, repoCacheIdx); TRY_BEGIN() { // Check each archiveId for (unsigned int archiveCacheIdx = 0; archiveCacheIdx < lstSize(cacheRepo->archiveList); archiveCacheIdx++) { const ArchiveGetFindCacheArchive *const cacheArchive = lstGet(cacheRepo->archiveList, archiveCacheIdx); // If a WAL segment then search among the possible file names if (isSegment) { StringList *segmentList; // If a single file is requested then optimize by adding a restrictive expression to reduce bandwidth if (single) { segmentList = storageListP( storageRepoIdx(cacheRepo->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(cacheArchive->archiveId), strZ(path)), .expression = strNewFmt( "^%s%s-[0-f]{40}" COMPRESS_TYPE_REGEXP "{0,1}$", strZ(strSubN(archiveFileRequest, 0, 24)), walIsPartial(archiveFileRequest) ? WAL_SEGMENT_PARTIAL_EXT : "")); } // Else multiple files will be requested so cache list results else { // Partial files cannot be in a list with multiple requests ASSERT(!walIsPartial(archiveFileRequest)); // If the path does not exist in the cache then fetch it const ArchiveGetFindCachePath *cachePath = lstFind(cacheArchive->pathList, &path); if (cachePath == NULL) { MEM_CONTEXT_BEGIN(lstMemContext(cacheArchive->pathList)) { const ArchiveGetFindCachePath archiveGetFindCachePath = { .path = strDup(path), .fileList = storageListP( storageRepoIdx(cacheRepo->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(cacheArchive->archiveId), strZ(path)), .expression = strNewFmt( "^%s[0-F]{8}-[0-f]{40}" COMPRESS_TYPE_REGEXP "{0,1}$", strZ(path))), }; cachePath = lstAdd(cacheArchive->pathList, &archiveGetFindCachePath); } MEM_CONTEXT_END(); } // Get a list of all WAL segments that match segmentList = strLstNew(); for (unsigned int fileIdx = 0; fileIdx < strLstSize(cachePath->fileList); fileIdx++) { if (strBeginsWith(strLstGet(cachePath->fileList, fileIdx), archiveFileRequest)) strLstAdd(segmentList, strLstGet(cachePath->fileList, fileIdx)); } } // Add segments to match list for (unsigned int segmentIdx = 0; segmentIdx < strLstSize(segmentList); segmentIdx++) { MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { const ArchiveGetFile archiveGetFile = { .file = strNewFmt( "%s/%s/%s", strZ(cacheArchive->archiveId), strZ(path), strZ(strLstGet(segmentList, segmentIdx))), .repoIdx = cacheRepo->repoIdx, .archiveId = cacheArchive->archiveId, .cipherType = cacheRepo->cipherType, .cipherPassArchive = cacheRepo->cipherPassArchive, }; lstAdd(matchList, &archiveGetFile); } MEM_CONTEXT_END(); } } // Else if not a WAL segment, see if it exists in the archiveId path else if ( storageExistsP( storageRepoIdx(cacheRepo->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(cacheArchive->archiveId), strZ(archiveFileRequest)))) { MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { const ArchiveGetFile archiveGetFile = { .file = strNewFmt("%s/%s", strZ(cacheArchive->archiveId), strZ(archiveFileRequest)), .repoIdx = cacheRepo->repoIdx, .archiveId = cacheArchive->archiveId, .cipherType = cacheRepo->cipherType, .cipherPassArchive = cacheRepo->cipherPassArchive, }; lstAdd(matchList, &archiveGetFile); } MEM_CONTEXT_END(); } } } // Log errors as warnings and continue CATCH_ANY() { repoErrorTotal++; archiveGetErrorAdd(cacheRepo->warnList, true, cacheRepo->repoIdx, errorType(), STR(errorMessage())); archiveGetErrorAdd(fileWarnList, false, cacheRepo->repoIdx, errorType(), STR(errorMessage())); } TRY_END(); } // If all repos errored out then set the global error since processing cannot continue past this segment ASSERT(repoErrorTotal <= lstSize(cacheRepoList)); if (repoErrorTotal == lstSize(cacheRepoList)) { ASSERT(!strLstEmpty(fileWarnList)); MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { getCheckResult->errorType = &RepoInvalidError; getCheckResult->errorFile = strDup(archiveFileRequest); getCheckResult->errorMessage = strNewZ(UNABLE_TO_FIND_VALID_REPO_MSG); getCheckResult->warnList = strLstMove(fileWarnList, memContextCurrent()); } MEM_CONTEXT_END(); } // Else if a file was found else if (!lstEmpty(matchList)) { bool error = false; // If a segment match list is > 1 then check for duplicates if (isSegment && lstSize(matchList) > 1) { // Count the number of unique hashes StringList *const hashList = strLstNew(); for (unsigned int matchIdx = 0; matchIdx < lstSize(matchList); matchIdx++) strLstAddIfMissing(hashList, strSubN(((ArchiveGetFile *)lstGet(matchList, matchIdx))->file, 25, 40)); // If there is more than one unique hash then there are duplicates if (strLstSize(hashList) > 1) { // Build list of duplicates unsigned int repoKeyLast = 0; String *const message = strNew(); bool first = true; // Sort the matches so they are logged in a consistent order lstSort(matchList, sortOrderAsc); for (unsigned int matchIdx = 0; matchIdx < lstSize(matchList); matchIdx++) { const ArchiveGetFile *const file = lstGet(matchList, matchIdx); const unsigned int repoKey = cfgOptionGroupIdxToKey(cfgOptGrpRepo, file->repoIdx); if (repoKey != repoKeyLast) { strCatFmt(message, "\n%s:", cfgOptionGroupName(cfgOptGrpRepo, file->repoIdx)); repoKeyLast = repoKey; first = true; } if (first) first = false; else strCatChr(message, ','); strCatFmt(message, " %s", strZ(file->file)); } // Set as global error since processing cannot continue past this segment MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { getCheckResult->errorType = &ArchiveDuplicateError; getCheckResult->errorFile = strDup(archiveFileRequest); getCheckResult->errorMessage = strNewFmt( "duplicates found for WAL segment %s:%s\n" "HINT: are multiple primaries archiving to this stanza?", strZ(archiveFileRequest), strZ(message)); getCheckResult->warnList = strLstMove(fileWarnList, memContextCurrent()); } MEM_CONTEXT_END(); error = true; } } // Files are valid so add them to the map if (!error) { MEM_CONTEXT_BEGIN(lstMemContext(getCheckResult->archiveFileMapList)) { const ArchiveFileMap map = { .request = strDup(archiveFileRequest), .actualList = lstNewP(sizeof(ArchiveGetFile)), .warnList = strLstMove(fileWarnList, memContextCurrent()), }; for (unsigned int matchIdx = 0; matchIdx < lstSize(matchList); matchIdx++) lstAdd(map.actualList, lstGet(matchList, matchIdx)); lstAdd(getCheckResult->archiveFileMapList, &map); } MEM_CONTEXT_END(); result = true; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } static ArchiveGetCheckResult archiveGetCheck(const StringList *const archiveRequestList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_LIST, archiveRequestList); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(archiveRequestList != NULL); ASSERT(!strLstEmpty(archiveRequestList)); ArchiveGetCheckResult result = {.archiveFileMapList = lstNewP(sizeof(ArchiveFileMap), .comparator = lstComparatorStr)}; MEM_CONTEXT_TEMP_BEGIN() { // List of warnings StringList *const warnList = strLstNew(); // Get pg_control info. Error if the invalid checkpoint written by restore is detected and backup_label is not present. const PgControl controlInfo = pgControlFromFile(storagePg(), cfgOptionStrNull(cfgOptPgVersionForce)); if (controlInfo.checkpoint == PG_CONTROL_CHECKPOINT_INVALID && !storageExistsP(storagePg(), STRDEF(PG_FILE_BACKUPLABEL))) { THROW( FormatError, "pg_control from backup is not valid without backup_label\n" "HINT: was the backup_label file removed?"); } // Build list of repos/archiveIds where WAL may be found List *const cacheRepoList = lstNewP(sizeof(ArchiveGetFindCacheRepo)); for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { // If a repo was specified then skip all other repos if (cfgOptionTest(cfgOptRepo) && cfgOptionUInt(cfgOptRepo) != cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoIdx)) continue; TRY_BEGIN() { // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepoIdx(repoIdx); ArchiveGetFindCacheRepo cacheRepo = { .repoIdx = repoIdx, .cipherType = cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), .archiveList = lstNewP(sizeof(ArchiveGetFindCacheArchive)), .warnList = strLstNew(), }; // Attempt to load the archive info file const InfoArchive *const info = infoArchiveLoadFile( storageRepoIdx(repoIdx), INFO_ARCHIVE_PATH_FILE_STR, cacheRepo.cipherType, cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); // Copy cipher pass into the result list context once rather than making a copy per candidate file later MEM_CONTEXT_BEGIN(lstMemContext(result.archiveFileMapList)) { cacheRepo.cipherPassArchive = strDup(infoArchiveCipherPass(info)); } MEM_CONTEXT_END(); // Loop through pg history and determine which archiveIds to use const StringList *archivePathList = NULL; for (unsigned int pgIdx = 0; pgIdx < infoPgDataTotal(infoArchivePg(info)); pgIdx++) { InfoPgData pgData = infoPgData(infoArchivePg(info), pgIdx); // Only use the archive id if it matches the current cluster if (pgData.systemId == controlInfo.systemId && pgData.version == controlInfo.version) { const String *const archiveId = infoPgArchiveId(infoArchivePg(info), pgIdx); bool found = true; // If the archiveId is in the past make sure the path exists if (pgIdx != 0) { // Get list of archiveId paths in the archive path if (archivePathList == NULL) archivePathList = storageListP(storageRepoIdx(repoIdx), STORAGE_REPO_ARCHIVE_STR); if (!strLstExists(archivePathList, archiveId)) found = false; } // If the archiveId is most recent or has files then add it if (found) { ArchiveGetFindCacheArchive cacheArchive = { .pathList = lstNewP(sizeof(ArchiveGetFindCachePath), .comparator = lstComparatorStr), }; // Copy archiveId into the result list context once rather than making a copy per candidate file later MEM_CONTEXT_BEGIN(lstMemContext(result.archiveFileMapList)) { cacheArchive.archiveId = strDup(archiveId); } MEM_CONTEXT_END(); lstAdd(cacheRepo.archiveList, &cacheArchive); } } } // Error if no archive id was found -- this indicates a mismatch with the current cluster if (lstEmpty(cacheRepo.archiveList)) { archiveGetErrorAdd( warnList, true, repoIdx, &ArchiveMismatchError, strNewFmt( "unable to retrieve the archive id for database version '%s' and system-id '%" PRIu64 "'", strZ(pgVersionToStr(controlInfo.version)), controlInfo.systemId)); } // Else add repo to list else lstAdd(cacheRepoList, &cacheRepo); } // Log errors as warnings and continue CATCH_ANY() { archiveGetErrorAdd(warnList, true, repoIdx, errorType(), STR(errorMessage())); } TRY_END(); } // Error if there are no repos to check if (lstEmpty(cacheRepoList)) { ASSERT(!strLstEmpty(warnList)); // Set as global error since processing cannot continue past this segment MEM_CONTEXT_BEGIN(lstMemContext(result.archiveFileMapList)) { result.errorType = &RepoInvalidError; result.errorMessage = strNewZ(UNABLE_TO_FIND_VALID_REPO_MSG); result.warnList = strLstMove(warnList, memContextCurrent()); } MEM_CONTEXT_END(); } else { // Any remaining errors will be reported as warnings since at least one repo is valid MEM_CONTEXT_BEGIN(lstMemContext(result.archiveFileMapList)) { result.warnList = strLstMove(warnList, memContextCurrent()); } MEM_CONTEXT_END(); // Find files in the list for (unsigned int archiveRequestIdx = 0; archiveRequestIdx < strLstSize(archiveRequestList); archiveRequestIdx++) { if (!archiveGetFind( strLstGet(archiveRequestList, archiveRequestIdx), &result, cacheRepoList, warnList, strLstSize(archiveRequestList) == 1)) { break; } } // Sort the list to make searching for files faster lstSort(result.archiveFileMapList, sortOrderAsc); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /*********************************************************************************************************************************** Clean the queue and prepare a list of WAL segments that the async process should get ***********************************************************************************************************************************/ static StringList * queueNeed( const String *const walSegment, const bool found, const uint64_t queueSize, const size_t walSegmentSize, const unsigned int pgVersion) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(BOOL, found); FUNCTION_LOG_PARAM(UINT64, queueSize); FUNCTION_LOG_PARAM(SIZE, walSegmentSize); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Determine the first WAL segment for the async process to get. If the WAL segment requested by PostgreSQL was not found // then use that. If the segment was found but the queue is not full then start with the next segment. const String *const walSegmentFirst = found ? walSegmentNext(walSegment, walSegmentSize, pgVersion) : walSegment; // Determine how many WAL segments should be in the queue. The queue total must be at least 2 or it doesn't make sense to // have async turned on at all. unsigned int walSegmentQueueTotal = (unsigned int)(queueSize / walSegmentSize); if (walSegmentQueueTotal < 2) walSegmentQueueTotal = 2; // Build the ideal queue -- the WAL segments we want in the queue after the async process has run const StringList *const idealQueue = strLstSort( walSegmentRange(walSegmentFirst, walSegmentSize, pgVersion, walSegmentQueueTotal), sortOrderAsc); // Get the list of files actually in the queue const StringList *const actualQueue = strLstSort( storageListP(storageSpool(), STORAGE_SPOOL_ARCHIVE_IN_STR, .errorOnMissing = true), sortOrderAsc); // Build a list of WAL segments that are being kept so we can later make a list of what is needed StringList *const keepQueue = strLstNew(); for (unsigned int actualQueueIdx = 0; actualQueueIdx < strLstSize(actualQueue); actualQueueIdx++) { // Get file from actual queue const String *const file = strLstGet(actualQueue, actualQueueIdx); // Does this match a file we want to preserve? if (strLstExists(idealQueue, file)) { strLstAdd(keepQueue, file); } // Else delete if it does not match an ok file for a WAL segment that has already been preserved. If an ok file exists // in addition to the segment then it contains warnings which need to be preserved. else if ( !strEndsWithZ(file, STATUS_EXT_OK) || !strLstExists(actualQueue, strSubN(file, 0, strSize(file) - STATUS_EXT_OK_SIZE))) { storageRemoveP(storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strZ(file)), .errorOnMissing = true); } } // Generate a list of the WAL that are needed by removing kept WAL from the ideal queue strLstSort(keepQueue, sortOrderAsc); for (unsigned int idealQueueIdx = 0; idealQueueIdx < strLstSize(idealQueue); idealQueueIdx++) { if (!strLstExists(keepQueue, strLstGet(idealQueue, idealQueueIdx))) strLstAdd(result, strLstGet(idealQueue, idealQueueIdx)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Get a single WAL segment synchronously, i.e. copy it directly from the repository to the destination rather than using the async spool queue. Returns 0 if the segment was found and copied, else 1. Throws on error. ***********************************************************************************************************************************/ static int archiveGetSingle(const String *const walSegment, const String *const walDestination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walSegment); FUNCTION_LOG_PARAM(STRING, walDestination); FUNCTION_LOG_END(); ASSERT(walSegment != NULL); ASSERT(walDestination != NULL); int result = 1; MEM_CONTEXT_TEMP_BEGIN() { // Check for the archive file StringList *const archiveRequestList = strLstNew(); strLstAdd(archiveRequestList, walSegment); const ArchiveGetCheckResult checkResult = archiveGetCheck(archiveRequestList); // If there was an error then throw it if (checkResult.errorType != NULL) THROW_CODE(errorTypeCode(checkResult.errorType), strZ(checkResult.errorMessage)); // Get the archive file if (!lstEmpty(checkResult.archiveFileMapList)) { // There can only be one file mapping since only one file was requested ASSERT(lstSize(checkResult.archiveFileMapList) == 1); const ArchiveFileMap *const fileMap = lstGet(checkResult.archiveFileMapList, 0); // Get the file const ArchiveGetFileResult fileResult = archiveGetFile( storageLocalWrite(), fileMap->request, fileMap->actualList, walDestination); // Output file warnings for (unsigned int warnIdx = 0; warnIdx < strLstSize(fileResult.warnList); warnIdx++) LOG_WARN(strZ(strLstGet(fileResult.warnList, warnIdx))); // If there was no error then the file existed const ArchiveGetFile *const file = lstGet(fileMap->actualList, fileResult.actualIdx); ASSERT(file != NULL); LOG_INFO_FMT( FOUND_IN_REPO_ARCHIVE_MSG, strZ(walSegment), cfgOptionGroupName(cfgOptGrpRepo, file->repoIdx), strZ(file->archiveId)); result = 0; } // Else log that the file was not found else LOG_INFO_FMT(UNABLE_TO_FIND_IN_ARCHIVE_MSG, strZ(walSegment)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN int cmdArchiveGet(void) { FUNCTION_LOG_VOID(logLevelDebug); // PostgreSQL must be local pgIsLocalVerify(); // Set the result assuming the archive file will not be found int result = 1; MEM_CONTEXT_TEMP_BEGIN() { // Check the parameters const StringList *const commandParam = cfgCommandParam(); if (strLstSize(commandParam) != 2) { if (strLstEmpty(commandParam)) THROW(ParamRequiredError, "WAL segment to get required"); if (strLstSize(commandParam) == 1) THROW(ParamRequiredError, "path to copy WAL segment required"); THROW(ParamInvalidError, "extra parameters found"); } // Get the segment name const String *const walSegment = strBase(strLstGet(commandParam, 0)); // Destination is wherever we were told to move the WAL segment const String *const walDestination = walPath(strLstGet(commandParam, 1), cfgOptionStr(cfgOptPgPath), STR(cfgCommandName())); // Async get can only be performed on WAL segments, history or other files must use synchronous mode if (cfgOptionBool(cfgOptArchiveAsync) && walIsSegment(walSegment)) { bool first = true; // Is this the first time the loop has run? bool found = false; // Has the WAL segment been found yet? bool foundOk = false; // Was an OK file found which confirms the file was missing? bool queueFull = false; // Is the queue half or more full? bool forked = false; // Has the async process been forked yet? bool syncGet = false; // Was the segment resolved by a synchronous get? // Loop and wait for the WAL segment to be pushed Wait *const wait = waitNew(cfgOptionUInt64(cfgOptArchiveTimeout)); do { // Check if the WAL segment is already in the queue found = storageExistsP(storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strZ(walSegment))); // Determine whether a missing WAL segment will be retried. Retrying is safer, but not retrying lets PostgreSQL // know that there are probably no more WAL segments in the archive which means it can switch to streaming. const bool missingRetry = first && cfgOptionBool(cfgOptArchiveMissingRetry); // Check for errors or missing files. For archive-get ok indicates that the process succeeded but there is no WAL // file to download, or that there was a warning. Do not error on the first run so the async process can be spawned // to correct any errors from a previous run. Do not warn on the first run if the segment was not found so the async // process can be spawned to check for the file again. if (archiveAsyncStatus(archiveModeGet, walSegment, !first, found || !missingRetry)) { storageRemoveP( storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s" STATUS_EXT_OK, strZ(walSegment)), .errorOnMissing = true); // Break if an ok file was found but no segment exists, which means the segment was missing. However, don't // break if this is the first time through the loop since this means the ok file was written by an async process // spawned by a prior archive-get execution, which means we should spawn the async process again to see if the // file exists now. This also prevents spool files from a previous recovery interfering with the current // recovery. if (!found && !missingRetry) { foundOk = true; break; } } // If found then move the WAL segment to the destination directory if (found) { // Source is the WAL segment in the spool queue StorageRead *const source = storageNewReadP( storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strZ(walSegment))); // A move will be attempted but if the spool queue and the WAL path are on different file systems then a copy // will be performed instead. // // It looks scary that we are disabling syncs and atomicity (in case we need to copy instead of move) but this // is safe because if the system crashes Postgres will not try to reuse a restored WAL segment but will instead // request it again using the restore_command. In the case of a move this hardly matters since path syncs are // cheap but if a copy is required we could save a lot of writes. StorageWrite *const destination = storageNewWriteP( storageLocalWrite(), walDestination, .noCreatePath = true, .noSyncFile = true, .noSyncPath = true, .noAtomic = true); // Move (or copy if required) the file storageMoveP(storageSpoolWrite(), source, destination); // Return success LOG_INFO_FMT(FOUND_IN_ARCHIVE_MSG " asynchronously", strZ(walSegment)); result = 0; // Get a list of WAL segments left in the queue const StringList *const queue = storageListP( storageSpool(), STORAGE_SPOOL_ARCHIVE_IN_STR, .expression = WAL_SEGMENT_REGEXP_STR, .errorOnMissing = true); if (!strLstEmpty(queue)) { // Get size of the WAL segment const uint64_t walSegmentSize = storageInfoP(storageLocal(), walDestination).size; // Use WAL segment size to estimate queue size and determine if the async process should be launched queueFull = strLstSize(queue) * walSegmentSize > cfgOptionUInt64(cfgOptArchiveGetQueueMax) / 2; } } // If the WAL segment has not already been found then start the async process to get it. There's no point in forking // the async process off more than once so track that as well. Use an archive lock to prevent forking if the async // process was launched by another process. if (!forked && (!found || !queueFull) && cmdLockAcquireP(.returnOnNoLock = true)) { // Get control info const PgControl pgControl = pgControlFromFile(storagePg(), cfgOptionStrNull(cfgOptPgVersionForce)); // Create the queue storagePathCreateP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_IN_STR); // The async process should not output on the console at all KeyValue *const optionReplace = kvNew(); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARSTRDEF("off")); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARSTRDEF("off")); // Generate command options StringList *const commandExec = cfgExecParam(cfgCmdArchiveGet, cfgCmdRoleAsync, optionReplace, true, false); strLstInsert(commandExec, 0, cfgBin()); // Clean the current queue using the list of WAL that we ideally want in the queue. queueNeed() will return the // list of WAL needed to fill the queue and this will be passed to the async process. const StringList *const queue = queueNeed( walSegment, found, cfgOptionUInt64(cfgOptArchiveGetQueueMax), pgControl.walSegmentSize, pgControl.version); for (unsigned int queueIdx = 0; queueIdx < strLstSize(queue); queueIdx++) strLstAdd(commandExec, strLstGet(queue, queueIdx)); // Clear errors for the current wal segment archiveAsyncErrorClear(archiveModeGet, walSegment); // Release the lock so the child process can acquire it cmdLockReleaseP(); // Execute the async process cmdAsyncExec(CFGCMD_ARCHIVE_GET, commandExec); // Mark the async process as forked so it doesn't get forked again. A single run of the async process should be // enough to do the job, running it again won't help anything. forked = true; } // If the segment is not in the queue and our own async process is not holding the lock then the segment will not be // fetched by us: either a foreign async process holds the lock (spawned by a different main process, e.g. filling // the queue for the old timeline after a timeline switch) and will not fetch our segment, or no async process is // running because ours failed. Rather than wait for it and risk a timeout on a segment that actually exists -- // which can end recovery prematurely -- fetch the segment synchronously. Skip the first run so an async process // spawned above (or by a prior invocation) gets a chance to take the lock first. if (!found && !first && !cmdLockOwn()) { result = archiveGetSingle(walSegment, walDestination); found = result == 0; syncGet = true; break; } // Exit loop if WAL was found if (found) break; // No longer the first run, so errors will be thrown and missing files will be reported first = false; } while (waitMore(wait)); // If the WAL segment was not found and it was not resolved by a synchronous get (which does its own logging) if (!found && !syncGet) { // If no ok file was found then something may be wrong with the async process. It's better to throw an error here // than report not found for debugging purposes. Either way PostgreSQL will halt if it has not reached consistency. if (!foundOk) { THROW_FMT( ArchiveTimeoutError, "unable to get WAL file '%s' from the archive asynchronously after %s second(s)\n" "HINT: check '%s' for errors.", strZ(walSegment), strZ( strNewDivP((uint64_t)cfgOptionInt64(cfgOptArchiveTimeout), MSEC_PER_SEC, .precision = 3, .trim = true)), strZ(cfgLoadLogFileName(cfgCmdRoleAsync))); } // Else report that the WAL segment could not be found else LOG_INFO_FMT(UNABLE_TO_FIND_IN_ARCHIVE_MSG " asynchronously", strZ(walSegment)); } } // Else perform synchronous get else result = archiveGetSingle(walSegment, walDestination); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INT, result); } /**********************************************************************************************************************************/ typedef struct ArchiveGetAsyncData { const List *const archiveFileMapList; // List of wal segments to process unsigned int archiveFileIdx; // Current index in the list to be processed } ArchiveGetAsyncData; static ProtocolParallelJob * archiveGetAsyncCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_END(); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // No special logic based on the client, we'll just get the next job (void)clientIdx; // Get a new job if there are any left ArchiveGetAsyncData *const jobData = data; if (jobData->archiveFileIdx < lstSize(jobData->archiveFileMapList)) { const ArchiveFileMap *const archiveFileMap = lstGet(jobData->archiveFileMapList, jobData->archiveFileIdx); jobData->archiveFileIdx++; PackWrite *const param = protocolPackNew(); pckWriteStrP(param, archiveFileMap->request); // Add actual files to get for (unsigned int actualIdx = 0; actualIdx < lstSize(archiveFileMap->actualList); actualIdx++) { const ArchiveGetFile *const actual = lstGet(archiveFileMap->actualList, actualIdx); pckWriteStrP(param, actual->file); pckWriteU32P(param, actual->repoIdx); pckWriteStrP(param, actual->archiveId); pckWriteU64P(param, actual->cipherType); pckWriteStrP(param, actual->cipherPassArchive); } MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew(VARSTR(archiveFileMap->request), PROTOCOL_COMMAND_ARCHIVE_GET_FILE, param); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } FN_EXTERN void cmdArchiveGetAsync(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { // PostgreSQL must be local pgIsLocalVerify(); // Check the parameters if (strLstSize(cfgCommandParam()) < 1) THROW(ParamInvalidError, "at least one wal segment is required"); LOG_INFO_FMT( "get %u WAL file(s) from archive: %s%s", strLstSize(cfgCommandParam()), strZ(strLstGet(cfgCommandParam(), 0)), strLstSize(cfgCommandParam()) == 1 ? "" : zNewFmt("...%s", strZ(strLstGet(cfgCommandParam(), strLstSize(cfgCommandParam()) - 1)))); // Check for archive files const ArchiveGetCheckResult checkResult = archiveGetCheck(cfgCommandParam()); // If any files are missing get the first one (used to construct the "unable to find" warning) const String *archiveFileMissing = NULL; if (lstSize(checkResult.archiveFileMapList) < strLstSize(cfgCommandParam())) archiveFileMissing = strLstGet(cfgCommandParam(), lstSize(checkResult.archiveFileMapList)); // Get archive files that were found if (!lstEmpty(checkResult.archiveFileMapList)) { // Create the parallel executor ArchiveGetAsyncData jobData = {.archiveFileMapList = checkResult.archiveFileMapList}; ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, archiveGetAsyncCallback, &jobData); for (unsigned int processIdx = 1; processIdx <= cfgOptionUInt(cfgOptProcessMax); processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypeRepo, 0, processIdx)); // Process jobs MEM_CONTEXT_TEMP_RESET_BEGIN() { do { const unsigned int completed = protocolParallelProcess(parallelExec); for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { // Get the job ProtocolParallelJob *const job = protocolParallelResult(parallelExec); const unsigned int processId = protocolParallelJobProcessId(job); // Get wal segment name and archive file map const String *const walSegment = varStr(protocolParallelJobKey(job)); const ArchiveFileMap *const fileMap = lstFind(checkResult.archiveFileMapList, &walSegment); ASSERT(fileMap != NULL); // Build warnings for status file String *const warning = strNew(); if (!strLstEmpty(fileMap->warnList)) strCatFmt(warning, "%s", strZ(strLstJoin(fileMap->warnList, "\n"))); // The job was successful if (protocolParallelJobErrorCode(job) == 0) { // Get the actual file retrieved PackRead *const fileResult = protocolParallelJobResult(job); const ArchiveGetFile *const file = lstGet(fileMap->actualList, pckReadU32P(fileResult)); ASSERT(file != NULL); // Output file warnings const StringList *const fileWarnList = pckReadStrLstP(fileResult); for (unsigned int warnIdx = 0; warnIdx < strLstSize(fileWarnList); warnIdx++) LOG_WARN_PID(processId, strZ(strLstGet(fileWarnList, warnIdx))); // Build file warnings for status file if (!strLstEmpty(fileWarnList)) { strCatFmt( warning, "%s%s", strSize(warning) == 0 ? "" : "\n", strZ(strLstJoin(fileWarnList, "\n"))); } if (strSize(warning) != 0) archiveAsyncStatusOkWrite(archiveModeGet, walSegment, warning); LOG_DETAIL_PID_FMT( processId, FOUND_IN_REPO_ARCHIVE_MSG, strZ(walSegment), cfgOptionGroupName(cfgOptGrpRepo, file->repoIdx), strZ(file->archiveId)); // Rename temp WAL segment to actual name. This is done after the ok file is written so the ok file // is guaranteed to exist before the foreground process finds the WAL segment. storageMoveP( storageSpoolWrite(), storageNewReadP( storageSpool(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s." STORAGE_FILE_TEMP_EXT, strZ(walSegment))), storageNewWriteP( storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s", strZ(walSegment)))); } // Else the job errored else { LOG_WARN_PID_FMT( processId, "[%s] %s", errorTypeName(errorTypeFromCode(protocolParallelJobErrorCode(job))), strZ(protocolParallelJobErrorMessage(job))); archiveAsyncStatusErrorWrite( archiveModeGet, walSegment, protocolParallelJobErrorCode(job), strNewFmt( "%s%s", strZ(protocolParallelJobErrorMessage(job)), strSize(warning) == 0 ? "" : zNewFmt("\n%s", strZ(warning)))); } protocolParallelJobFree(job); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); } // Log an error from archiveGetCheck() after any existing files have been fetched. This ordering is important because we // need to fetch as many valid files as possible before throwing an error. if (checkResult.errorType != NULL) { LOG_WARN_FMT("[%s] %s", errorTypeName(checkResult.errorType), strZ(checkResult.errorMessage)); String *const message = strCat(strNew(), checkResult.errorMessage); if (!strLstEmpty(checkResult.warnList)) strCatFmt(message, "\n%s", strZ(strLstJoin(checkResult.warnList, "\n"))); archiveAsyncStatusErrorWrite( archiveModeGet, checkResult.errorFile, errorTypeCode(checkResult.errorType), message); } // If any files were missing write an ok file for the first missing file and add any warnings. It is important that this // happen right before the async process exits so the main process can immediately respawn the async process to retry // missing files. else if (archiveFileMissing != NULL) { LOG_DETAIL_FMT(UNABLE_TO_FIND_IN_ARCHIVE_MSG, strZ(archiveFileMissing)); const String *message = NULL; if (!strLstEmpty(checkResult.warnList)) message = strLstJoin(checkResult.warnList, "\n"); archiveAsyncStatusOkWrite(archiveModeGet, archiveFileMissing, message); } } // On any global error write a single error file to cover all unprocessed files CATCH_ANY() { archiveAsyncStatusErrorWrite(archiveModeGet, NULL, errorCode(), STR(errorMessage())); RETHROW(); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/get/get.h0000664000175100017510000000153715240547630020756 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get Command ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_GET_GET_H #define COMMAND_ARCHIVE_GET_GET_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get an archive file from the repository (WAL segment, history file, etc.) FN_EXTERN int cmdArchiveGet(void); // Async version of archive get that runs in parallel for performance FN_EXTERN void cmdArchiveGetAsync(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/get/protocol.c0000664000175100017510000000411115240547630022022 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get Protocol Handler ***********************************************************************************************************************************/ #include #include "command/archive/get/file.h" #include "command/archive/get/protocol.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" #include "storage/write.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * archiveGetFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Get request const String *const request = pckReadStrP(param); // Build the actual list List *const actualList = lstNewP(sizeof(ArchiveGetFile)); while (!pckReadNullP(param)) { ArchiveGetFile actual = {.file = pckReadStrP(param)}; actual.repoIdx = pckReadU32P(param); actual.archiveId = pckReadStrP(param); actual.cipherType = pckReadU64P(param); actual.cipherPassArchive = pckReadStrP(param); lstAdd(actualList, &actual); } // Get file const ArchiveGetFileResult fileResult = archiveGetFile( storageSpoolWrite(), request, actualList, strNewFmt(STORAGE_SPOOL_ARCHIVE_IN "/%s." STORAGE_FILE_TEMP_EXT, strZ(request))); // Return result PackWrite *const data = protocolServerResultData(result); pckWriteU32P(data, fileResult.actualIdx); pckWriteStrLstP(data, fileResult.warnList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/get/protocol.h0000664000175100017510000000271615240547630022040 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Get Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_GET_PROTOCOL_H #define COMMAND_ARCHIVE_GET_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *archiveGetFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_ARCHIVE_GET_FILE STRID5("ag-f", 0x36ce10) #define PROTOCOL_SERVER_HANDLER_ARCHIVE_GET_LIST \ {.command = PROTOCOL_COMMAND_ARCHIVE_GET_FILE, .process = archiveGetFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/push/0000775000175100017510000000000015240547630020220 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/push/file.c0000664000175100017510000003377015240547630021315 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push File ***********************************************************************************************************************************/ #include #include "command/archive/common.h" #include "command/archive/find.h" #include "command/archive/push/file.h" #include "command/control/common.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/filter/group.h" #include "common/io/io.h" #include "common/log.h" #include "config/config.h" #include "postgres/interface.h" #include "storage/helper.h" /*********************************************************************************************************************************** Catch write errors during processing We want to continue when there are write errors during processing so add them to a list to be reported later and return false so the caller knows to stop writing on the affected repo. ***********************************************************************************************************************************/ typedef enum { archivePushFileIoTypeOpen = STRID5("open", 0x7160f0), archivePushFileIoTypeWrite = STRID5("write", 0x5a26570), archivePushFileIoTypeClose = STRID5("close", 0x59bd830), } ArchivePushFileIoType; // Helper to add errors to the list static void archivePushErrorAdd(StringList *const errorList, const unsigned int repoIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, errorList); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_END(); strLstAddFmt(errorList, "%s: [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(errorType()), errorMessage()); FUNCTION_TEST_RETURN_VOID(); } static bool archivePushFileIo( const ArchivePushFileIoType type, IoWrite *const write, const Buffer *const buffer, const unsigned int repoIdx, StringList *const errorList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(IO_WRITE, write); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(STRING_LIST, errorList); FUNCTION_TEST_END(); ASSERT(write != NULL); ASSERT(errorList != NULL); bool result = true; // Process write operation TRY_BEGIN() { switch (type) { case archivePushFileIoTypeOpen: ioWriteOpen(write); break; case archivePushFileIoTypeWrite: ASSERT(buffer != NULL); ioWrite(write, buffer); break; case archivePushFileIoTypeClose: ioWriteClose(write); break; } } // Handle errors CATCH_ANY() { archivePushErrorAdd(errorList, repoIdx); result = false; } TRY_END(); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN ArchivePushFileResult archivePushFile( const String *const walSource, const bool headerCheck, const bool modeCheck, const unsigned int pgVersion, const uint64_t pgSystemId, const String *const archiveFile, const CompressType compressType, const int compressLevel, const List *const repoList, const StringList *const priorErrorList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walSource); FUNCTION_LOG_PARAM(BOOL, headerCheck); FUNCTION_LOG_PARAM(BOOL, modeCheck); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(STRING, archiveFile); FUNCTION_LOG_PARAM(ENUM, compressType); FUNCTION_LOG_PARAM(INT, compressLevel); FUNCTION_LOG_PARAM_P(VOID, repoList); FUNCTION_LOG_PARAM(STRING_LIST, priorErrorList); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(walSource != NULL); ASSERT(archiveFile != NULL); ASSERT(repoList != NULL); ASSERT(priorErrorList != NULL); ASSERT(lstSize(repoList) > 0); ArchivePushFileResult result = {.warnList = strLstNew()}; MEM_CONTEXT_TEMP_BEGIN() { StringList *const errorList = strLstDup(priorErrorList); // Is this a WAL segment? const bool isSegment = walIsSegment(archiveFile); // If this is a segment compare archive version and systemId to the WAL header if (headerCheck && isSegment) { const PgWal walInfo = pgWalFromFile(walSource, storageLocal(), cfgOptionStrNull(cfgOptPgVersionForce)); if (walInfo.version != pgVersion || walInfo.systemId != pgSystemId) { THROW_FMT( ArchiveMismatchError, "WAL file '%s' version %s, system-id %" PRIu64 " do not match stanza version %s, system-id %" PRIu64, strZ(walSource), strZ(pgVersionToStr(walInfo.version)), walInfo.systemId, strZ(pgVersionToStr(pgVersion)), pgSystemId); } } // Set archive destination initially to the archive file, this will be updated later for wal segments String *const archiveDestination = strCat(strNew(), archiveFile); // Assume that all repos need a copy of the archive file bool destinationCopyAny = true; bool *const destinationCopy = memNew(sizeof(bool) * lstSize(repoList)); for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) destinationCopy[repoListIdx] = true; // Get wal segment checksum and compare it to what exists in the repo, if any if (isSegment) { // Assume that no repos need a copy of the WAL segment and update when a repo needing a copy is found destinationCopyAny = false; // Generate a sha1 checksum for the wal segment IoRead *const read = storageReadIo(storageNewReadP(storageLocal(), walSource)); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioReadDrain(read); const String *const walSegmentChecksum = strNewEncode( encodingHex, pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE))); // Check each repo for the WAL segment for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const ArchivePushFileRepoData *const repoData = lstGet(repoList, repoListIdx); // Check if the WAL segment already exists in the repo const String *walSegmentFile; TRY_BEGIN() { walSegmentFile = walSegmentFindOne(storageRepoIdx(repoData->repoIdx), repoData->archiveId, archiveFile, 0); } CATCH_ANY() { archivePushErrorAdd(errorList, repoData->repoIdx); destinationCopy[repoListIdx] = false; } TRY_END(); // If there was an error try the next repo if (!destinationCopy[repoListIdx]) continue; // If the WAL segment was found validate the checksum if (walSegmentFile != NULL) { const String *const walSegmentRepoChecksum = strSubN( walSegmentFile, strSize(archiveFile) + 1, HASH_TYPE_SHA1_SIZE_HEX); // If the checksums are the same then succeed but warn if archive-mode-check is enabled in case this is a // symptom of some other issue if (strEq(walSegmentChecksum, walSegmentRepoChecksum)) { if (modeCheck) { // Add warning to the result that will be returned to the main process strLstAddFmt( result.warnList, "WAL file '%s' already exists in the %s archive with the same checksum" "\nHINT: this is valid in some recovery scenarios but may also indicate a problem.", strZ(archiveFile), cfgOptionGroupName(cfgOptGrpRepo, repoData->repoIdx)); } // No need to copy to this repo destinationCopy[repoListIdx] = false; } // Else error so we don't overwrite the existing segment. Do not continue processing after this error since it // indicates corruption, split brain, or some other unrecoverable error. else { THROW_FMT( ArchiveDuplicateError, "WAL file '%s' already exists in the %s archive with a different checksum", strZ(archiveFile), cfgOptionGroupName(cfgOptGrpRepo, repoData->repoIdx)); } } // Else the repo needs a copy else destinationCopyAny = true; } // Append the checksum to the archive destination strCatFmt(archiveDestination, "-%s", strZ(walSegmentChecksum)); } // Copy the file if one or more repos require it if (destinationCopyAny) { // Source file is read once and copied to all repos StorageRead *const source = storageNewReadP(storageLocal(), walSource); // Is the file compressible during the copy? bool compressible = true; // If the file will be compressed then add compression filter if (isSegment && compressType != compressTypeNone) { compressExtCat(archiveDestination, compressType); ioFilterGroupAdd(ioReadFilterGroup(storageReadIo(source)), compressFilterP(compressType, compressLevel)); compressible = false; } // Initialize per-repo destination files StorageWrite **const destination = memNew(sizeof(StorageWrite *) * lstSize(repoList)); for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const ArchivePushFileRepoData *const repoData = lstGet(repoList, repoListIdx); // Does this repo need a copy? if (destinationCopy[repoListIdx]) { // Create destination file destination[repoListIdx] = storageNewWriteP( storageRepoIdxWrite(repoData->repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(repoData->archiveId), strZ(archiveDestination)), .compressible = compressible); // If there is a cipher then add the encrypt filter if (repoData->cipherType != cipherTypeNone) { ioFilterGroupAdd( ioWriteFilterGroup(storageWriteIo(destination[repoListIdx])), cipherBlockNewP(cipherModeEncrypt, repoData->cipherType, BUFSTR(repoData->cipherPass))); } } } // Open source file ioReadOpen(storageReadIo(source)); // Open the destination files now that we know the source file exists and is readable for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const unsigned int repoIdx = ((ArchivePushFileRepoData *)lstGet(repoList, repoListIdx))->repoIdx; if (destinationCopy[repoListIdx]) { destinationCopy[repoListIdx] = archivePushFileIo( archivePushFileIoTypeOpen, storageWriteIo(destination[repoListIdx]), NULL, repoIdx, errorList); } } // Copy data from source to destination Buffer *const read = bufNew(ioBufferSize()); do { // Read from source ioRead(storageReadIo(source), read); // Write to each destination for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const unsigned int repoIdx = ((ArchivePushFileRepoData *)lstGet(repoList, repoListIdx))->repoIdx; if (destinationCopy[repoListIdx]) { destinationCopy[repoListIdx] = archivePushFileIo( archivePushFileIoTypeWrite, storageWriteIo(destination[repoListIdx]), read, repoIdx, errorList); } } // Clear buffer bufUsedZero(read); } while (!ioReadEof(storageReadIo(source))); // Close the source and destination files ioReadClose(storageReadIo(source)); for (unsigned int repoListIdx = 0; repoListIdx < lstSize(repoList); repoListIdx++) { const unsigned int repoIdx = ((ArchivePushFileRepoData *)lstGet(repoList, repoListIdx))->repoIdx; if (destinationCopy[repoListIdx]) { destinationCopy[repoListIdx] = archivePushFileIo( archivePushFileIoTypeClose, storageWriteIo(destination[repoListIdx]), NULL, repoIdx, errorList); } } } // Throw any errors, even if some pushes were successful. It is important that PostgreSQL receives an error so it does not // remove the file. if (strLstSize(errorList) > 0) THROW_FMT(CommandError, CFGCMD_ARCHIVE_PUSH " command encountered error(s):\n%s", strZ(strLstJoin(errorList, "\n"))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/push/file.h0000664000175100017510000000361415240547630021314 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push File ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_PUSH_FILE_H #define COMMAND_ARCHIVE_PUSH_FILE_H #include "common/compress/helper.h" #include "common/crypto/common.h" #include "common/type/string.h" #include "storage/storage.h" /*********************************************************************************************************************************** Structure to hold information for each repository the archive file will be pushed to. An array of these must be passed to archivePushFile() with size equal to cfgOptionGroupIdxTotal(cfgOptGrpRepo). ***********************************************************************************************************************************/ typedef struct ArchivePushFileRepoData { unsigned int repoIdx; const String *archiveId; CipherType cipherType; const String *cipherPass; } ArchivePushFileRepoData; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ typedef struct ArchivePushFileResult { StringList *warnList; // Warnings from a successful operation } ArchivePushFileResult; // Copy a file from the source to the archive FN_EXTERN ArchivePushFileResult archivePushFile( const String *walSource, bool headerCheck, bool modeCheck, unsigned int pgVersion, uint64_t pgSystemId, const String *archiveFile, CompressType compressType, int compressLevel, const List *repoList, const StringList *priorErrorList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/push/protocol.c0000664000175100017510000000502415240547630022226 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push Protocol Handler ***********************************************************************************************************************************/ #include #include "command/archive/push/file.h" #include "command/archive/push/protocol.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * archivePushFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Read parameters const String *const walSource = pckReadStrP(param); const bool headerCheck = pckReadBoolP(param); const bool modeCheck = pckReadBoolP(param); const unsigned int pgVersion = pckReadU32P(param); const uint64_t pgSystemId = pckReadU64P(param); const String *const archiveFile = pckReadStrP(param); const CompressType compressType = pckReadU32P(param); const int compressLevel = pckReadI32P(param); const StringList *const priorErrorList = pckReadStrLstP(param); // Read repo data List *const repoList = lstNewP(sizeof(ArchivePushFileRepoData)); pckReadArrayBeginP(param); while (!pckReadNullP(param)) { pckReadObjBeginP(param); ArchivePushFileRepoData repo = {.repoIdx = pckReadU32P(param)}; repo.archiveId = pckReadStrP(param); repo.cipherType = pckReadU64P(param); repo.cipherPass = pckReadStrP(param); pckReadObjEndP(param); lstAdd(repoList, &repo); } pckReadArrayEndP(param); // Push file const ArchivePushFileResult fileResult = archivePushFile( walSource, headerCheck, modeCheck, pgVersion, pgSystemId, archiveFile, compressType, compressLevel, repoList, priorErrorList); // Return result pckWriteStrLstP(protocolServerResultData(result), fileResult.warnList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/push/protocol.h0000664000175100017510000000272415240547630022237 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_PUSH_PROTOCOL_H #define COMMAND_ARCHIVE_PUSH_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *archivePushFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_ARCHIVE_PUSH_FILE STRID5("ap-f", 0x36e010) #define PROTOCOL_SERVER_HANDLER_ARCHIVE_PUSH_LIST \ {.command = PROTOCOL_COMMAND_ARCHIVE_PUSH_FILE, .process = archivePushFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/push/push.c0000664000175100017510000007643015240547630021355 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push Command ***********************************************************************************************************************************/ #include #include #include #include "command/archive/common.h" #include "command/archive/push/file.h" #include "command/archive/push/protocol.h" #include "command/archive/push/push.h" #include "command/command.h" #include "command/control/common.h" #include "command/lock.h" #include "common/compress/helper.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/wait.h" #include "config/config.h" #include "config/exec.h" #include "config/load.h" #include "info/infoArchive.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants for log messages that are used multiple times to keep them consistent ***********************************************************************************************************************************/ #define UNABLE_TO_FIND_VALID_REPO_MSG "unable to find a valid repository" /*********************************************************************************************************************************** Ready file extension constants ***********************************************************************************************************************************/ #define STATUS_EXT_READY ".ready" #define STATUS_EXT_READY_SIZE (sizeof(STATUS_EXT_READY) - 1) /*********************************************************************************************************************************** Format the warning when a file is dropped ***********************************************************************************************************************************/ static String * archivePushDropWarning(const String *const walFile, const uint64_t queueMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, walFile); FUNCTION_TEST_PARAM(UINT64, queueMax); FUNCTION_TEST_END(); String *const size = strSizeFormat(queueMax); String *const result = strNewFmt("dropped WAL file '%s' because archive queue exceeded %s", strZ(walFile), strZ(size)); strFree(size); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Determine if the WAL process list has become large enough to drop ***********************************************************************************************************************************/ static bool archivePushDrop(const String *const walPath, const StringList *const processList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walPath); FUNCTION_LOG_PARAM(STRING_LIST, processList); FUNCTION_LOG_END(); ASSERT(walPath != NULL); ASSERT(processList != NULL); const uint64_t queueMax = cfgOptionUInt64(cfgOptArchivePushQueueMax); uint64_t queueSize = 0; MEM_CONTEXT_TEMP_BEGIN() { if (!strLstEmpty(processList)) { queueSize = storageInfoP( storagePg(), strNewFmt("%s/%s", strZ(walPath), strZ(strLstGet(processList, 0)))).size * strLstSize(processList); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, queueSize > queueMax); } /*********************************************************************************************************************************** Get the list of WAL files ready to be pushed according to PostgreSQL ***********************************************************************************************************************************/ static StringList * archivePushReadyList(const String *const walPath) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walPath); FUNCTION_LOG_END(); ASSERT(walPath != NULL); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Read the ready files from the archive_status directory const StringList *const readyListRaw = strLstSort( storageListP( storagePg(), strNewFmt("%s/" PG_PATH_ARCHIVE_STATUS, strZ(walPath)), .expression = STRDEF("\\" STATUS_EXT_READY "$"), .errorOnMissing = true), sortOrderAsc); for (unsigned int readyIdx = 0; readyIdx < strLstSize(readyListRaw); readyIdx++) { const String *const ready = strLstGet(readyListRaw, readyIdx); strLstAddSub(result, ready, strSize(ready) - STATUS_EXT_READY_SIZE); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Determine which WAL files need to be pushed to the archive when in async mode This is the heart of the "look ahead" functionality in async archiving. Any files in the out directory that do not end in ok are removed and any ok files that do not have a corresponding ready file in archive_status (meaning it has been acknowledged by PostgreSQL) are removed. Then all ready files that do not have a corresponding ok file (meaning it has already been processed) are returned for processing. ***********************************************************************************************************************************/ static StringList * archivePushProcessList(const String *const walPath) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, walPath); FUNCTION_LOG_END(); ASSERT(walPath != NULL); StringList *result; MEM_CONTEXT_TEMP_BEGIN() { // Read the status files from the spool directory, then remove any files that do not end in ok and create a list of the // ok files for further processing. The spool out path is created by the foreground process before the async process is // forked. const StringList *const statusList = strLstSort( storageListP(storageSpool(), STORAGE_SPOOL_ARCHIVE_OUT_STR, .errorOnMissing = true), sortOrderAsc); StringList *const okList = strLstNew(); for (unsigned int statusIdx = 0; statusIdx < strLstSize(statusList); statusIdx++) { const String *const statusFile = strLstGet(statusList, statusIdx); if (strEndsWithZ(statusFile, STATUS_EXT_OK)) strLstAddSub(okList, statusFile, strSize(statusFile) - STATUS_EXT_OK_SIZE); else { storageRemoveP( storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s", strZ(statusFile)), .errorOnMissing = true); } } // Read the ready files from the archive_status directory const StringList *const readyList = archivePushReadyList(walPath); // Remove ok files that are not in the ready list const StringList *const okRemoveList = strLstMergeAnti(okList, readyList); for (unsigned int okRemoveIdx = 0; okRemoveIdx < strLstSize(okRemoveList); okRemoveIdx++) { storageRemoveP( storageSpoolWrite(), strNewFmt(STORAGE_SPOOL_ARCHIVE_OUT "/%s" STATUS_EXT_OK, strZ(strLstGet(okRemoveList, okRemoveIdx))), .errorOnMissing = true); } // Return all ready files that are not in the ok list result = strLstMove(strLstMergeAnti(readyList, okList), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Check that pg_control and archive.info match and get the archive id and archive cipher passphrase (if present) As much information as possible is collected here so that async archiving has as little work as possible to do for each file. Sync archiving does not benefit but it makes sense to use the same function. ***********************************************************************************************************************************/ typedef struct ArchivePushCheckResult { unsigned int pgVersion; // PostgreSQL version uint64_t pgSystemId; // PostgreSQL system id List *repoList; // Data for each repo StringList *errorList; // Errors while checking repos } ArchivePushCheckResult; static ArchivePushCheckResult archivePushCheck(const bool pgPathSet) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, pgPathSet); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ArchivePushCheckResult result = {.repoList = lstNewP(sizeof(ArchivePushFileRepoData)), .errorList = strLstNew()}; MEM_CONTEXT_TEMP_BEGIN() { // If we have access to pg_control then load it to get the pg version and system id. If we can't load pg_control then we'll // still compare the pg info stored in the repo to the WAL segment and also all the repos against each other. if (pgPathSet) { // Get info from pg_control PgControl pgControl = pgControlFromFile(storagePg(), cfgOptionStrNull(cfgOptPgVersionForce)); result.pgVersion = pgControl.version; result.pgSystemId = pgControl.systemId; } for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { TRY_BEGIN() { // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepoIdx(repoIdx); // Get cipher type const CipherType repoCipherType = cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx); // Attempt to load the archive info file const InfoArchive *const info = infoArchiveLoadFile( storageRepoIdx(repoIdx), INFO_ARCHIVE_PATH_FILE_STR, repoCipherType, cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); // Get archive id for the most recent version -- archive-push will only operate against the most recent version const String *const archiveId = infoPgArchiveId(infoArchivePg(info), infoPgDataCurrentId(infoArchivePg(info))); const InfoPgData archiveInfo = infoPgData(infoArchivePg(info), infoPgDataCurrentId(infoArchivePg(info))); // Ensure that stanza version and system identifier match pg_control when available or the other repos when // pg_control is not available if (pgPathSet || repoIdx > 0) { if (result.pgVersion != archiveInfo.version || result.pgSystemId != archiveInfo.systemId) { THROW_FMT( ArchiveMismatchError, "%s version %s, system-id %" PRIu64 " do not match %s stanza version %s, system-id %" PRIu64 "\nHINT: are you archiving to the correct stanza?", pgPathSet ? PG_NAME : zNewFmt("%s stanza", cfgOptionGroupName(cfgOptGrpRepo, 0)), strZ(pgVersionToStr(result.pgVersion)), result.pgSystemId, cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(pgVersionToStr(archiveInfo.version)), archiveInfo.systemId); } } MEM_CONTEXT_PRIOR_BEGIN() { result.pgVersion = archiveInfo.version; result.pgSystemId = archiveInfo.systemId; const ArchivePushFileRepoData archivePushFileRepoData = { .repoIdx = repoIdx, .archiveId = strDup(archiveId), .cipherType = repoCipherType, .cipherPass = strDup(infoArchiveCipherPass(info)), }; lstAdd(result.repoList, &archivePushFileRepoData); } MEM_CONTEXT_PRIOR_END(); } CATCH_ANY() { strLstAddFmt( result.errorList, "%s: [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(errorType()), errorMessage()); } TRY_END(); } } MEM_CONTEXT_TEMP_END(); // If no valid repos were found then error if (lstEmpty(result.repoList)) { ASSERT(strLstSize(result.errorList) > 0); THROW_FMT(RepoInvalidError, UNABLE_TO_FIND_VALID_REPO_MSG ":\n%s", strZ(strLstJoin(result.errorList, "\n"))); } FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdArchivePush(void) { FUNCTION_LOG_VOID(logLevelDebug); ASSERT(cfgCommand() == cfgCmdArchivePush); // PostgreSQL must be local pgIsLocalVerify(); MEM_CONTEXT_TEMP_BEGIN() { // Make sure there is a parameter to retrieve the WAL segment from const StringList *const commandParam = cfgCommandParam(); if (strLstSize(commandParam) != 1) THROW(ParamRequiredError, "WAL segment to push required"); // Test for stop file lockStopTest(); // Get the segment name const String *const walFile = walPath(strLstGet(commandParam, 0), cfgOptionStrNull(cfgOptPgPath), STR(cfgCommandName())); const String *const archiveFile = strBase(walFile); if (cfgOptionBool(cfgOptArchiveAsync)) { bool pushed = false; // Has the WAL segment been pushed yet? bool forked = false; // Has the async process been forked yet? bool throwOnError = false; // Should we throw errors? // pg1-path is not optional for async mode if (!cfgOptionTest(cfgOptPgPath)) { THROW_FMT( OptionRequiredError, "'" CFGCMD_ARCHIVE_PUSH "' command in async mode requires option '%s'", cfgOptionName(cfgOptPgPath)); } // Loop and wait for the WAL segment to be pushed Wait *const wait = waitNew(cfgOptionUInt64(cfgOptArchiveTimeout)); do { // Check if the WAL segment has been pushed. Errors will not be thrown on the first try to allow the async process // a chance to fix them. pushed = archiveAsyncStatus(archiveModePush, archiveFile, throwOnError, true); // If the WAL segment has not already been pushed then start the async process to push it. There's no point in // forking the async process off more than once so track that as well. Use an archive lock to prevent more than one // async process being launched. if (!pushed && !forked && cmdLockAcquireP(.returnOnNoLock = true)) { // The async process should not output on the console at all KeyValue *const optionReplace = kvNew(); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARSTRDEF("off")); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARSTRDEF("off")); // Generate command options StringList *const commandExec = cfgExecParam(cfgCmdArchivePush, cfgCmdRoleAsync, optionReplace, true, false); strLstInsert(commandExec, 0, cfgBin()); strLstAdd(commandExec, strPath(walFile)); // Create the spool out path in the foreground. If this fails (e.g. permission denied) the error is thrown here // and reaches the PostgreSQL log immediately, rather than only in the async process, where it cannot be written // to the spool as a status file and only appears as an archive timeout. storagePathCreateP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_OUT_STR); // Clear errors for the current archive file archiveAsyncErrorClear(archiveModePush, archiveFile); // Release the lock so the child process can acquire it cmdLockReleaseP(); // Execute the async process cmdAsyncExec(CFGCMD_ARCHIVE_PUSH, commandExec); // Mark the async process as forked so it doesn't get forked again. A single run of the async process should be // enough to do the job, running it again won't help anything. forked = true; } // Now that the async process has been launched, throw any errors that are found throwOnError = true; } while (!pushed && waitMore(wait)); // If the WAL segment was not pushed then error if (!pushed) { THROW_FMT( ArchiveTimeoutError, "unable to push WAL file '%s' to the archive asynchronously after %s second(s)\n" "HINT: check '%s' for errors.", strZ(archiveFile), strZ(cfgOptionDisplay(cfgOptArchiveTimeout)), strZ(cfgLoadLogFileName(cfgCmdRoleAsync))); } // Log success LOG_INFO_FMT("pushed WAL file '%s' to the archive asynchronously", strZ(archiveFile)); } else { // Check if the push queue has been exceeded if (cfgOptionTest(cfgOptArchivePushQueueMax) && archivePushDrop(strPath(walFile), archivePushReadyList(strPath(walFile)))) { LOG_WARN(strZ(archivePushDropWarning(archiveFile, cfgOptionUInt64(cfgOptArchivePushQueueMax)))); } // Else push the file else { // Check archive info for each repo const ArchivePushCheckResult archiveInfo = archivePushCheck(cfgOptionTest(cfgOptPgPath)); // Push the file to the archive const ArchivePushFileResult fileResult = archivePushFile( walFile, cfgOptionBool(cfgOptArchiveHeaderCheck), cfgOptionBool(cfgOptArchiveModeCheck), archiveInfo.pgVersion, archiveInfo.pgSystemId, archiveFile, compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), cfgOptionInt(cfgOptCompressLevel), archiveInfo.repoList, archiveInfo.errorList); // If a warning was returned then log it for (unsigned int warnIdx = 0; warnIdx < strLstSize(fileResult.warnList); warnIdx++) LOG_WARN(strZ(strLstGet(fileResult.warnList, warnIdx))); // Log success LOG_INFO_FMT("pushed WAL file '%s' to the archive", strZ(archiveFile)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ typedef struct ArchivePushAsyncData { const String *walPath; // Path to pg_wal/pg_xlog StringList *walFileList; // List of wal files to process unsigned int walFileIdx; // Current index in the list to be processed CompressType compressType; // Type of compression for WAL segments int compressLevel; // Compression level for wal files ArchivePushCheckResult archiveInfo; // Archive info bool errorFound; // Has a job errored? If so, stop scheduling new jobs } ArchivePushAsyncData; static ProtocolParallelJob * archivePushAsyncCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_END(); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // No special logic based on the client, we'll just get the next job (void)clientIdx; // Get a new job if there are any left. Stop scheduling new jobs as soon as one errors so the async process exits and the // next run rechecks the queue. Continuing past an error does not relieve disk pressure since PostgreSQL cannot recycle any // WAL past the oldest unarchived segment, so the sooner the queue is rechecked the sooner WAL can be dropped if needed. ArchivePushAsyncData *const jobData = data; if (!jobData->errorFound && jobData->walFileIdx < strLstSize(jobData->walFileList)) { const String *const walFile = strLstGet(jobData->walFileList, jobData->walFileIdx); jobData->walFileIdx++; PackWrite *const param = protocolPackNew(); pckWriteStrP(param, strNewFmt("%s/%s", strZ(jobData->walPath), strZ(walFile))); pckWriteBoolP(param, cfgOptionBool(cfgOptArchiveHeaderCheck)); pckWriteBoolP(param, cfgOptionBool(cfgOptArchiveModeCheck)); pckWriteU32P(param, jobData->archiveInfo.pgVersion); pckWriteU64P(param, jobData->archiveInfo.pgSystemId); pckWriteStrP(param, walFile); pckWriteU32P(param, jobData->compressType); pckWriteI32P(param, jobData->compressLevel); pckWriteStrLstP(param, jobData->archiveInfo.errorList); // Add data for each repo to push to pckWriteArrayBeginP(param); for (unsigned int repoListIdx = 0; repoListIdx < lstSize(jobData->archiveInfo.repoList); repoListIdx++) { ArchivePushFileRepoData *data = lstGet(jobData->archiveInfo.repoList, repoListIdx); pckWriteObjBeginP(param); pckWriteU32P(param, data->repoIdx); pckWriteStrP(param, data->archiveId); pckWriteU64P(param, data->cipherType); pckWriteStrP(param, data->cipherPass); pckWriteObjEndP(param); } pckWriteArrayEndP(param); MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew(VARSTR(walFile), PROTOCOL_COMMAND_ARCHIVE_PUSH_FILE, param); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } FN_EXTERN void cmdArchivePushAsync(void) { FUNCTION_LOG_VOID(logLevelDebug); ASSERT(cfgCommand() == cfgCmdArchivePush && cfgCommandRole() == cfgCmdRoleAsync); // PostgreSQL must be local pgIsLocalVerify(); MEM_CONTEXT_TEMP_BEGIN() { // Make sure there is a parameter with the wal path const StringList *const commandParam = cfgCommandParam(); if (strLstSize(commandParam) != 1) THROW(ParamRequiredError, "WAL path to push required"); ArchivePushAsyncData jobData = { .walPath = strLstGet(commandParam, 0), .compressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), .compressLevel = cfgOptionInt(cfgOptCompressLevel), }; TRY_BEGIN() { // Test for stop file lockStopTest(); // Get a list of WAL files that are ready for processing jobData.walFileList = archivePushProcessList(jobData.walPath); // The archive-push:async command should not have been called unless there are WAL files to process if (strLstEmpty(jobData.walFileList)) THROW(AssertError, "no WAL files to process"); // Determine if the queue has grown large enough that all WAL should be dropped. The queue is measured against the ready // list (all WAL that PostgreSQL is waiting to archive) rather than the process list (WAL not yet pushed). If an earlier // WAL file errors repeatedly then PostgreSQL keeps retrying it and never acknowledges the later files that look-ahead // has already pushed, so those files drop out of the process list even though their WAL remains in pg_wal. Using the // process list here would let the queue grow without bound in that case, exactly when dropping is most needed. const bool drop = cfgOptionTest(cfgOptArchivePushQueueMax) && archivePushDrop(jobData.walPath, archivePushReadyList(jobData.walPath)); // If not dropping then limit the number of WAL files pushed in a single run to archive-push-batch-size. The queue is // only rechecked at the start of each run so this bounds how large it can grow before the next run rechecks it. Convert // the size to a segment count using the size of the first ready segment and always push at least one segment. A ready // WAL segment should never be missing or zero size, so error immediately if it is. if (!drop) { const uint64_t segmentSize = storageInfoP( storagePg(), strNewFmt("%s/%s", strZ(jobData.walPath), strZ(strLstGet(jobData.walFileList, 0)))).size; CHECK_FMT(FormatError, segmentSize != 0, "size of WAL segment '%s' is 0", strZ(strLstGet(jobData.walFileList, 0))); const uint64_t batchMax = cfgOptionUInt64(cfgOptArchivePushBatchSize) / segmentSize; while (strLstSize(jobData.walFileList) > 1 && strLstSize(jobData.walFileList) > batchMax) strLstRemoveIdx(jobData.walFileList, strLstSize(jobData.walFileList) - 1); } LOG_INFO_FMT( "push %u WAL file(s) to archive: %s%s", strLstSize(jobData.walFileList), strZ(strLstGet(jobData.walFileList, 0)), strLstSize(jobData.walFileList) == 1 ? "" : zNewFmt("...%s", strZ(strLstGet(jobData.walFileList, strLstSize(jobData.walFileList) - 1)))); // Drop files if queue max has been exceeded if (drop) { for (unsigned int walFileIdx = 0; walFileIdx < strLstSize(jobData.walFileList); walFileIdx++) { const String *const walFile = strLstGet(jobData.walFileList, walFileIdx); const String *const warning = archivePushDropWarning(walFile, cfgOptionUInt64(cfgOptArchivePushQueueMax)); archiveAsyncStatusOkWrite(archiveModePush, walFile, warning); LOG_WARN(strZ(warning)); } } // Else continue processing else { // Check archive info for each repo jobData.archiveInfo = archivePushCheck(true); // Create the parallel executor ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, archivePushAsyncCallback, &jobData); for (unsigned int processIdx = 1; processIdx <= cfgOptionUInt(cfgOptProcessMax); processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypeRepo, 0, processIdx)); // Process jobs MEM_CONTEXT_TEMP_RESET_BEGIN() { do { const unsigned int completed = protocolParallelProcess(parallelExec); for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { protocolKeepAlive(); // Get the job and job key ProtocolParallelJob *const job = protocolParallelResult(parallelExec); const unsigned int processId = protocolParallelJobProcessId(job); const String *const walFile = varStr(protocolParallelJobKey(job)); // The job was successful if (protocolParallelJobErrorCode(job) == 0) { // Output file warnings const StringList *const fileWarnList = pckReadStrLstP(protocolParallelJobResult(job)); for (unsigned int warnIdx = 0; warnIdx < strLstSize(fileWarnList); warnIdx++) LOG_WARN_PID(processId, strZ(strLstGet(fileWarnList, warnIdx))); // Log success LOG_DETAIL_PID_FMT(processId, "pushed WAL file '%s' to the archive", strZ(walFile)); // Write the status file archiveAsyncStatusOkWrite( archiveModePush, walFile, strLstEmpty(fileWarnList) ? NULL : strLstJoin(fileWarnList, "\n")); } // Else the job errored else { jobData.errorFound = true; LOG_WARN_PID_FMT( processId, "could not push WAL file '%s' to the archive (will be retried): [%d] %s", strZ(walFile), protocolParallelJobErrorCode(job), strZ(protocolParallelJobErrorMessage(job))); archiveAsyncStatusErrorWrite( archiveModePush, walFile, protocolParallelJobErrorCode(job), protocolParallelJobErrorMessage(job)); } protocolParallelJobFree(job); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); // If processing was stopped early because a job errored then log a warning. The remaining WAL is left for the next // run, which will recheck the queue and drop WAL if it now exceeds queue-max. if (jobData.errorFound) LOG_WARN("stopped archive-push after an error, remaining WAL will be processed on the next run"); } } // On any global error write a single error file to cover all unprocessed files CATCH_FATAL() { archiveAsyncStatusErrorWrite(archiveModePush, NULL, errorCode(), STR(errorMessage())); RETHROW(); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/archive/push/push.h0000664000175100017510000000150315240547630021347 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Push Command ***********************************************************************************************************************************/ #ifndef COMMAND_ARCHIVE_PUSH_PUSH_H #define COMMAND_ARCHIVE_PUSH_PUSH_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Push a WAL segment to the repository FN_EXTERN void cmdArchivePush(void); // Async version of archive push that runs in parallel for performance FN_EXTERN void cmdArchivePushAsync(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/0000775000175100017510000000000015240547630017065 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/backup.c0000664000175100017510000035633115240547630020511 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/archive/find.h" #include "command/backup/backup.h" #include "command/backup/common.h" #include "command/backup/file.h" #include "command/backup/protocol.h" #include "command/check/common.h" #include "command/control/common.h" #include "command/lock.h" #include "command/stanza/common.h" #include "common/compress/helper.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/io/filter/size.h" #include "common/log.h" #include "common/regExp.h" #include "common/time.h" #include "common/type/convert.h" #include "common/type/json.h" #include "config/common.h" #include "config/config.h" #include "config/parse.h" #include "db/helper.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" #include "version.h" /********************************************************************************************************************************** Generate a unique backup label that does not contain a timestamp from a previous backup ***********************************************************************************************************************************/ static String * backupLabelCreate(const BackupType type, const String *const backupLabelPrior, const time_t timestamp) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, backupLabelPrior); FUNCTION_LOG_PARAM(TIME, timestamp); FUNCTION_LOG_END(); ASSERT((type == backupTypeFull && backupLabelPrior == NULL) || (type != backupTypeFull && backupLabelPrior != NULL)); ASSERT(timestamp > 0); String *result; MEM_CONTEXT_TEMP_BEGIN() { const String *backupLabelLatest = NULL; // For full backup compare against all backups. For other backup types compare only against backups in the same full backup // set, i.e. those whose name begins with the full backup part of backupLabelPrior. Since the new label is generated from // backupLabelPrior it can never conflict with a label in another full backup set. This prevents label generation from // failing when the last full backup is removed and then a diff/incr is generated from the last remaining full backup, or // when a removed full backup is still returned by the repository listing, which can happen on object stores that retain // empty prefixes. const String *const labelRegExp = type == backupTypeFull ? backupRegExpP(.full = true, .differential = true, .incremental = true, .noAnchorEnd = true) : strNewFmt("^%.*sF\\_" DATE_TIME_REGEX "(D|I)", DATE_TIME_LEN, strZ(backupLabelPrior)); // Get the newest backup const StringList *const backupList = strLstSort( storageListP(storageRepo(), STRDEF(STORAGE_REPO_BACKUP), .expression = strNewFmt("%s$", strZ(labelRegExp))), sortOrderDesc); if (!strLstEmpty(backupList)) backupLabelLatest = strLstGet(backupList, 0); // Get the newest history const StringList *const historyYearList = strLstSort( storageListP(storageRepo(), STRDEF(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY), .expression = STRDEF("^2[0-9]{3}$")), sortOrderDesc); if (!strLstEmpty(historyYearList)) { const StringList *const historyList = strLstSort( storageListP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s", strZ(strLstGet(historyYearList, 0))), .expression = strNewFmt("%s\\.manifest\\.%s$", strZ(labelRegExp), strZ(compressTypeStr(compressTypeGz)))), sortOrderDesc); if (!strLstEmpty(historyList)) { const String *const historyLabelLatest = strLstGet(historyList, 0); if (backupLabelLatest == NULL || strCmp(historyLabelLatest, backupLabelLatest) > 0) { // Strip off the compression and manifest extensions in case this ends up in an error message backupLabelLatest = compressExtStrip(historyLabelLatest, compressTypeFromName(historyLabelLatest)); backupLabelLatest = strSubN(backupLabelLatest, 0, strSize(backupLabelLatest) - sizeof(BACKUP_MANIFEST_EXT) + 1); } } } // Now that we have the latest label check if the provided timestamp will give us an even later label result = backupLabelFormat(type, backupLabelPrior, timestamp); if (backupLabelLatest != NULL && strCmp(result, backupLabelLatest) <= 0) { // If that didn't give us a later label then add one second. It's possible that two backups (they would need to be // offline or halted online) have run very close together. result = backupLabelFormat(type, backupLabelPrior, timestamp + 1); // If the label is still not latest then error. There is probably a timezone change or massive clock skew. if (strCmp(result, backupLabelLatest) <= 0) { THROW_FMT( ClockError, "new backup label '%s' is not later than latest backup label '%s'\n" "HINT: has the timezone changed?\n" "HINT: is there clock skew?", strZ(result), strZ(backupLabelLatest)); } // If adding a second worked then sleep the remainder of the current second so we don't start early sleepMSec(MSEC_PER_SEC - (timeMSec() % MSEC_PER_SEC)); } MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(result); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /*********************************************************************************************************************************** Get the postgres database and storage objects ***********************************************************************************************************************************/ #define FUNCTION_LOG_BACKUP_DATA_TYPE \ BackupData * #define FUNCTION_LOG_BACKUP_DATA_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "BackupData", buffer, bufferSize) typedef struct BackupData { unsigned int pgIdxPrimary; // cfgOptGrpPg index of the primary Db *dbPrimary; // Database connection to the primary const Storage *storagePrimary; // Storage object for the primary const String *hostPrimary; // Host name of the primary unsigned int pgIdxStandby; // cfgOptGrpPg index of the standby Db *dbStandby; // Database connection to the standby const Storage *storageStandby; // Storage object for the standby const String *hostStandby; // Host name of the standby const InfoArchive *archiveInfo; // Archive info const String *archiveId; // Archive where backup WAL will be stored unsigned int timeline; // Primary timeline unsigned int version; // PostgreSQL version unsigned int walSegmentSize; // PostgreSQL wal segment size PgPageSize pageSize; // PostgreSQL page size } BackupData; static BackupData * backupInit(const InfoBackup *const infoBackup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(infoBackup != NULL); // Initialize for offline backup BackupData *const result = memNew(sizeof(BackupData)); *result = (BackupData){0}; // Don't allow backup from standby when offline unsigned int backupStandby = cfgOptionSeq(cfgOptBackupStandby); const InfoPgData infoPg = infoPgDataCurrent(infoBackupPg(infoBackup)); if (!cfgOptionBool(cfgOptOnline) && backupStandby != CFGOPTVAL_BACKUP_STANDBY_N) { LOG_WARN( "option " CFGOPT_BACKUP_STANDBY " is enabled but backup is offline - backups will be performed from the primary"); backupStandby = CFGOPTVAL_BACKUP_STANDBY_N; } // Get database info when online PgControl pgControl = {0}; if (cfgOptionBool(cfgOptOnline)) { const DbGetResult dbInfo = dbGet(backupStandby == CFGOPTVAL_BACKUP_STANDBY_N, true, backupStandby); // If no standby was found but using the primary is allowed then warn and proceed if (backupStandby == CFGOPTVAL_BACKUP_STANDBY_PREFER && dbInfo.standby == NULL) LOG_WARN("unable to find a standby to perform the backup, using primary instead"); result->pgIdxPrimary = dbInfo.primaryIdx; result->dbPrimary = dbInfo.primary; if (dbInfo.standby != NULL) { ASSERT(dbInfo.standby != NULL); result->pgIdxStandby = dbInfo.standbyIdx; result->dbStandby = dbInfo.standby; result->storageStandby = storagePgIdx(result->pgIdxStandby); result->hostStandby = cfgOptionIdxStrNull(cfgOptPgHost, result->pgIdxStandby); } // Get pg_control info from the primary pgControl = dbPgControl(result->dbPrimary); } // Else get pg_control info directly from the file else pgControl = pgControlFromFile(storagePgIdx(result->pgIdxPrimary), cfgOptionStrNull(cfgOptPgVersionForce)); // Add primary info result->storagePrimary = storagePgIdx(result->pgIdxPrimary); result->hostPrimary = cfgOptionIdxStrNull(cfgOptPgHost, result->pgIdxPrimary); result->timeline = pgControl.timeline; result->version = pgControl.version; result->walSegmentSize = pgControl.walSegmentSize; result->pageSize = pgControl.pageSize; // Validate pg_control info against the stanza if (result->version != infoPg.version || pgControl.systemId != infoPg.systemId) { THROW_FMT( BackupMismatchError, PG_NAME " version %s, system-id %" PRIu64 " do not match stanza version %s, system-id %" PRIu64 "\n" "HINT: is this the correct stanza?", strZ(pgVersionToStr(pgControl.version)), pgControl.systemId, strZ(pgVersionToStr(infoPg.version)), infoPg.systemId); } // If checksum page is not explicitly set then automatically enable it when checksums are available if (!cfgOptionTest(cfgOptChecksumPage)) { // If online then use the value in pg_control to set checksum-page if (cfgOptionBool(cfgOptOnline)) { cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, VARBOOL(pgControl.pageChecksumVersion != 0)); } // Else set to false. An offline cluster is likely to have false positives so better if the user enables manually. else cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, BOOL_FALSE_VAR); } // Else if checksums have been explicitly enabled but are not available then warn and reset. ??? We should be able to make this // determination when offline as well, but the integration tests don't write pg_control accurately enough to support it. else if (cfgOptionBool(cfgOptOnline) && pgControl.pageChecksumVersion == 0 && cfgOptionBool(cfgOptChecksumPage)) { LOG_WARN(CFGOPT_CHECKSUM_PAGE " option set to true but checksums are not enabled on the cluster, resetting to false"); cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, BOOL_FALSE_VAR); } // Get archive info if (cfgOptionBool(cfgOptArchiveCheck)) { result->archiveInfo = infoArchiveLoadFile( storageRepo(), INFO_ARCHIVE_PATH_FILE_STR, cfgOptionStrId(cfgOptRepoCipherType), cfgOptionStrNull(cfgOptRepoCipherPass)); result->archiveId = infoArchiveId(result->archiveInfo); } FUNCTION_LOG_RETURN(BACKUP_DATA, result); } /********************************************************************************************************************************** Build block incremental maps ***********************************************************************************************************************************/ // Size map. Block size is increased when the block map would be larger than a single block. The break can be calculated with this // formula: [block size in KiB] / (1024 / [block size in KiB] * [checksum size]) * 1073741824. static const ManifestBlockIncrSizeMap manifestBlockIncrSizeMapDefault[] = { {.fileSize = 914 * 1024 * 1024, .blockSize = 88 * 1024}, {.fileSize = 740 * 1024 * 1024, .blockSize = 80 * 1024}, {.fileSize = 585 * 1024 * 1024, .blockSize = 72 * 1024}, {.fileSize = 448 * 1024 * 1024, .blockSize = 64 * 1024}, {.fileSize = 329 * 1024 * 1024, .blockSize = 56 * 1024}, {.fileSize = 228 * 1024 * 1024, .blockSize = 48 * 1024}, {.fileSize = 146 * 1024 * 1024, .blockSize = 40 * 1024}, {.fileSize = 96 * 1024 * 1024, .blockSize = 32 * 1024}, {.fileSize = 43 * 1024 * 1024, .blockSize = 24 * 1024}, {.fileSize = 11 * 1024 * 1024, .blockSize = 16 * 1024}, {.fileSize = 16 * 1024, .blockSize = 8 * 1024}, }; // Age map static const ManifestBlockIncrAgeMap manifestBlockIncrAgeMapDefault[] = { {.fileAge = 4 * 7 * SEC_PER_DAY, .blockMultiplier = 0}, {.fileAge = 2 * 7 * SEC_PER_DAY, .blockMultiplier = 4}, {.fileAge = 7 * SEC_PER_DAY, .blockMultiplier = 2}, }; // Checksum size map static const ManifestBlockIncrChecksumSizeMap manifestBlockIncrChecksumSizeMapDefault[] = { {.blockSize = 4 * 1024 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 6}, {.blockSize = 2 * 1024 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 5}, {.blockSize = 1024 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 4}, {.blockSize = 512 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 3}, {.blockSize = 128 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 2}, {.blockSize = 32 * 1024, .checksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN + 1}, }; // All maps static const ManifestBlockIncrMap manifestBlockIncrMap = { .sizeMap = LSTDEF(manifestBlockIncrSizeMapDefault), .ageMap = LSTDEF(manifestBlockIncrAgeMapDefault), .checksumSizeMap = LSTDEF(manifestBlockIncrChecksumSizeMapDefault), }; // Convert map size static unsigned int backupBlockIncrMapSize(const ConfigOption optionId, const unsigned int optionKeyIdx, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionKeyIdx); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); int64_t result; TRY_BEGIN() { result = cfgParseSize(value); // Error if value is out of range (no need for an error message since that will be generated in the catch block) if (result <= 0 || result > UINT_MAX) THROW(OptionInvalidValueError, ""); } CATCH_ANY() { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for '%s' option", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } TRY_END(); ASSERT(result > 0 && result < UINT_MAX); FUNCTION_TEST_RETURN(UINT, (unsigned int)result); } // Convert map checksum size static unsigned int backupBlockIncrMapChecksumSize(const ConfigOption optionId, const unsigned int optionKeyIdx, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionKeyIdx); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); unsigned int result = 0; TRY_BEGIN() { result = varUIntForce(value); } CATCH_ANY() { } TRY_END(); if (result < BLOCK_INCR_CHECKSUM_SIZE_MIN) { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for '%s' option", strZ(varStr(value)), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } FUNCTION_TEST_RETURN(UINT, result); } static ManifestBlockIncrMap backupBlockIncrMap(void) { FUNCTION_TEST_VOID(); FUNCTION_AUDIT_HELPER(); ManifestBlockIncrMap result = manifestBlockIncrMap; if (cfgOptionBool(cfgOptRepoBlock)) { // Build size map const KeyValue *const manifestBlockIncrSizeKv = cfgOptionKvNull(cfgOptRepoBlockSizeMap); if (manifestBlockIncrSizeKv != NULL) { List *const map = lstNewP(sizeof(ManifestBlockIncrSizeMap), .comparator = lstComparatorUInt); const VariantList *const mapKeyList = kvKeyList(manifestBlockIncrSizeKv); for (unsigned int mapKeyIdx = 0; mapKeyIdx < varLstSize(mapKeyList); mapKeyIdx++) { const Variant *mapKey = varLstGet(mapKeyList, mapKeyIdx); ManifestBlockIncrSizeMap manifestBuildBlockIncrSizeMap = { .fileSize = backupBlockIncrMapSize( cfgOptRepoBlockSizeMap, cfgOptionIdxDefault(cfgOptRepoBlockSizeMap), varStr(mapKey)), .blockSize = backupBlockIncrMapSize( cfgOptRepoBlockSizeMap, cfgOptionIdxDefault(cfgOptRepoBlockSizeMap), varStr(kvGet(manifestBlockIncrSizeKv, mapKey))), }; lstAdd(map, &manifestBuildBlockIncrSizeMap); } lstSort(map, sortOrderDesc); result.sizeMap = map; } // Build age map const KeyValue *const manifestBlockIncrAgeKv = cfgOptionKvNull(cfgOptRepoBlockAgeMap); if (manifestBlockIncrAgeKv != NULL) { List *const map = lstNewP(sizeof(ManifestBlockIncrAgeMap), .comparator = lstComparatorUInt); const VariantList *const mapKeyList = kvKeyList(manifestBlockIncrAgeKv); for (unsigned int mapKeyIdx = 0; mapKeyIdx < varLstSize(mapKeyList); mapKeyIdx++) { const Variant *mapKey = varLstGet(mapKeyList, mapKeyIdx); ManifestBlockIncrAgeMap manifestBuildBlockIncrAgeMap = { .fileAge = (unsigned int)(varUIntForce(mapKey) * SEC_PER_DAY), .blockMultiplier = varUIntForce(kvGet(manifestBlockIncrAgeKv, mapKey)), }; lstAdd(map, &manifestBuildBlockIncrAgeMap); } lstSort(map, sortOrderDesc); result.ageMap = map; } // Build checksum size map const KeyValue *const manifestBlockIncrChecksumSizeKv = cfgOptionKvNull(cfgOptRepoBlockChecksumSizeMap); if (manifestBlockIncrChecksumSizeKv != NULL) { List *const map = lstNewP(sizeof(ManifestBlockIncrChecksumSizeMap), .comparator = lstComparatorUInt); const VariantList *const mapKeyList = kvKeyList(manifestBlockIncrChecksumSizeKv); for (unsigned int mapKeyIdx = 0; mapKeyIdx < varLstSize(mapKeyList); mapKeyIdx++) { const Variant *const mapKey = varLstGet(mapKeyList, mapKeyIdx); ManifestBlockIncrChecksumSizeMap manifestBuildBlockIncrChecksumSizeMap = { .blockSize = backupBlockIncrMapSize( cfgOptRepoBlockSizeMap, cfgOptionIdxDefault(cfgOptRepoBlockChecksumSizeMap), varStr(mapKey)), .checksumSize = backupBlockIncrMapChecksumSize( cfgOptRepoBlockSizeMap, cfgOptionIdxDefault(cfgOptRepoBlockChecksumSizeMap), kvGet(manifestBlockIncrChecksumSizeKv, mapKey)), }; lstAdd(map, &manifestBuildBlockIncrChecksumSizeMap); } lstSort(map, sortOrderDesc); result.checksumSizeMap = map; } } FUNCTION_TEST_RETURN_TYPE(ManifestBlockIncrMap, result); } /********************************************************************************************************************************** Get time from the database or locally depending on online ***********************************************************************************************************************************/ static time_t backupTime(const BackupData *const backupData, const bool waitRemainder) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(BOOL, waitRemainder); FUNCTION_LOG_END(); // Offline backups will just grab the time from the local system since the value of copyStart is not important in this context. // No worries about causing a delta backup since switching online will do that anyway. time_t result = time(NULL); // When online get the time from the database server if (cfgOptionBool(cfgOptOnline)) { // Get time from the database TimeMSec timeMSec = dbTimeMSec(backupData->dbPrimary); result = (time_t)(timeMSec / MSEC_PER_SEC); // Sleep the remainder of the second when requested (this is so copyStart is not subject to one second resolution issues) if (waitRemainder) { unsigned int retry = 0; // Just to be safe we'll loop until PostgreSQL reports that we have slept long enough do { // Error if the clock has not advanced after several attempts if (retry == 3) THROW_FMT(KernelError, PG_NAME " clock has not advanced to the next second after %u tries", retry); // Sleep remainder of current second sleepMSec(((TimeMSec)(result + 1) * MSEC_PER_SEC) - timeMSec); // Check time again to be sure we slept long enough timeMSec = dbTimeMSec(backupData->dbPrimary); // Increment retry to prevent an infinite loop retry++; } while ((time_t)(timeMSec / MSEC_PER_SEC) <= result); } } FUNCTION_LOG_RETURN(TIME, result); } /*********************************************************************************************************************************** Create an incremental backup if type is not full and a compatible prior backup exists ***********************************************************************************************************************************/ // Helper to find a compatible prior backup static Manifest * backupBuildIncrPrior(const InfoBackup *const infoBackup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); Manifest *result = NULL; // No incremental if backup type is full const BackupType type = (BackupType)cfgOptionStrId(cfgOptType); if (type != backupTypeFull) { MEM_CONTEXT_TEMP_BEGIN() { const InfoPgData infoPg = infoPgDataCurrent(infoBackupPg(infoBackup)); const String *backupLabelPrior = NULL; const unsigned int backupTotal = infoBackupDataTotal(infoBackup); for (unsigned int backupIdx = backupTotal - 1; backupIdx < backupTotal; backupIdx--) { const InfoBackupData backupPrior = infoBackupData(infoBackup, backupIdx); // The prior backup for a diff must be full if (type == backupTypeDiff && backupPrior.backupType != backupTypeFull) continue; // The backups must come from the same cluster ??? This should enable delta instead if (infoPg.id != backupPrior.backupPgId) continue; // This backup is a candidate for prior backupLabelPrior = strDup(backupPrior.backupLabel); break; } // If there is a prior backup then check that options for the new backup are compatible if (backupLabelPrior != NULL) { result = manifestLoadFile( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabelPrior)), cfgOptionStrId(cfgOptRepoCipherType), infoPgCipherPass(infoBackupPg(infoBackup))); const ManifestData *const manifestPriorData = manifestData(result); LOG_INFO_FMT( "last backup label = %s, version = %s", strZ(manifestData(result)->backupLabel), strZ(manifestData(result)->backrestVersion)); // Warn if compress-type option changed if (compressTypeEnum(cfgOptionStrId(cfgOptCompressType)) != manifestPriorData->backupOptionCompressType) { LOG_WARN_FMT( "%s backup cannot alter " CFGOPT_COMPRESS_TYPE " option to '%s', reset to value in %s", strZ(cfgOptionDisplay(cfgOptType)), strZ(cfgOptionDisplay(cfgOptCompressType)), strZ(backupLabelPrior)); // Set the compression type back to whatever was in the prior backup. This is not strictly needed since we could // store compression type on a per file basis, but it seems simplest and safest for now. cfgOptionSet( cfgOptCompressType, cfgSourceParam, VARSTR(compressTypeStr(manifestPriorData->backupOptionCompressType))); // There's a small chance that the prior manifest is old enough that backupOptionCompressLevel was not recorded. // There's an even smaller chance that the user will also alter compression-type in this scenario right after // upgrading to a newer version. Because we judge this combination of events to be nearly impossible just check // here so no test coverage is needed. CHECK( FormatError, manifestPriorData->backupOptionCompressLevel != NULL, "compress-level missing in prior manifest"); // Set the compression level back to whatever was in the prior backup cfgOptionSet( cfgOptCompressLevel, cfgSourceParam, VARINT64(varUInt(manifestPriorData->backupOptionCompressLevel))); } // If not defined this backup was done in a version prior to page checksums being introduced. Just set checksum-page // to false and move on without a warning. Page checksums will start on the next full backup. if (manifestData(result)->backupOptionChecksumPage == NULL) { cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, BOOL_FALSE_VAR); } // Don't allow the checksum-page option to change in a diff or incr backup. This could be confusing as only certain // files would be checksummed and the list could be incomplete during reporting. else { const bool checksumPagePrior = varBool(manifestData(result)->backupOptionChecksumPage); // Warn if an incompatible setting was explicitly requested if (checksumPagePrior != cfgOptionBool(cfgOptChecksumPage)) { LOG_WARN_FMT( "%s backup cannot alter '" CFGOPT_CHECKSUM_PAGE "' option to '%s', reset to '%s' from %s", strZ(cfgOptionDisplay(cfgOptType)), strZ(cfgOptionDisplay(cfgOptChecksumPage)), cvtBoolToConstZ(checksumPagePrior), strZ(manifestData(result)->backupLabel)); } cfgOptionSet(cfgOptChecksumPage, cfgSourceParam, VARBOOL(checksumPagePrior)); } manifestMove(result, memContextPrior()); } else { LOG_WARN_FMT("no prior backup exists, %s backup has been changed to full", strZ(cfgOptionDisplay(cfgOptType))); cfgOptionSet(cfgOptType, cfgSourceParam, VARUINT64(backupTypeFull)); } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(MANIFEST, result); } static bool backupBuildIncr( const InfoBackup *const infoBackup, Manifest *const manifest, Manifest *const manifestPrior, const String *const archiveStart) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(MANIFEST, manifestPrior); FUNCTION_LOG_PARAM(STRING, archiveStart); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); ASSERT(manifest != NULL); bool result = false; // No incremental if no prior manifest if (manifestPrior != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // Move the manifest to this context so it will be freed when we are done manifestMove(manifestPrior, MEM_CONTEXT_TEMP()); // Build incremental manifest manifestBuildIncr(manifest, manifestPrior, (BackupType)cfgOptionStrId(cfgOptType), archiveStart); // Set the cipher subpass from prior manifest since we want a single subpass for the entire backup set manifestCipherSubPassSet(manifest, manifestCipherSubPass(manifestPrior)); // Incremental was built result = true; } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Check for a backup that can be resumed and merge into the manifest if found ***********************************************************************************************************************************/ // Helper to clean invalid paths/files/links out of the resumable backup path static void backupResumeClean( StorageIterator *const storageItr, Manifest *const manifest, const Manifest *const manifestResume, const CompressType compressType, const bool delta, const String *const backupParentPath, const String *const manifestParentName) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_ITERATOR, storageItr); // Storage info FUNCTION_LOG_PARAM(MANIFEST, manifest); // New manifest FUNCTION_LOG_PARAM(MANIFEST, manifestResume); // Resumed manifest FUNCTION_LOG_PARAM(ENUM, compressType); // Backup compression type FUNCTION_LOG_PARAM(BOOL, delta); // Is this a delta backup? FUNCTION_LOG_PARAM(STRING, backupParentPath); // Path to the current level of the backup being cleaned FUNCTION_LOG_PARAM(STRING, manifestParentName); // Parent manifest name used to construct manifest name FUNCTION_LOG_END(); ASSERT(storageItr != NULL); ASSERT(manifest != NULL); ASSERT(manifestResume != NULL); ASSERT(backupParentPath != NULL); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); // Skip backup.manifest.copy -- it must be preserved to allow resume again if this process throws an error before // writing the manifest for the first time if (manifestParentName == NULL && strEqZ(info.name, BACKUP_MANIFEST_FILE INFO_COPY_EXT)) continue; // Build the name used to lookup files in the manifest const String *manifestName = manifestParentName != NULL ? strNewFmt("%s/%s", strZ(manifestParentName), strZ(info.name)) : info.name; // Build the backup path used to remove files/links/paths that are invalid const String *const backupPath = strNewFmt("%s/%s", strZ(backupParentPath), strZ(info.name)); // Process file types switch (info.type) { // Check paths // ----------------------------------------------------------------------------------------------------------------- case storageTypePath: { // If the path was not found in the new manifest then remove it if (manifestPathFindDefault(manifest, manifestName, NULL) == NULL) { LOG_DETAIL_FMT("remove path '%s' from resumed backup", strZ(storagePathP(storageRepo(), backupPath))); storagePathRemoveP(storageRepoWrite(), backupPath, .recurse = true); } // Else recurse into the path else { backupResumeClean( storageNewItrP(storageRepo(), backupPath, .sortOrder = sortOrderAsc), manifest, manifestResume, compressType, delta, backupPath, manifestName); } break; } // Check files // ----------------------------------------------------------------------------------------------------------------- case storageTypeFile: { // If the file is compressed then strip off the extension before doing the lookup const CompressType fileCompressType = compressTypeFromName(manifestName); if (fileCompressType != compressTypeNone) manifestName = compressExtStrip(manifestName, fileCompressType); // If the file is block incremental then strip off the extension before doing the lookup const bool blockIncr = strEndsWithZ(manifestName, BACKUP_BLOCK_INCR_EXT); if (blockIncr) manifestName = strSubN(manifestName, 0, strSize(manifestName) - (sizeof(BACKUP_BLOCK_INCR_EXT) - 1)); // Check if the file can be resumed or must be removed const char *removeReason = NULL; if (fileCompressType != compressType && !blockIncr) removeReason = "mismatched compression type"; else if (!manifestFileExists(manifest, manifestName)) removeReason = "missing in manifest"; else { ManifestFile file = manifestFileFind(manifest, manifestName); ASSERT(file.reference == NULL); if (!manifestFileExists(manifestResume, manifestName)) removeReason = "missing in resumed manifest"; else { const ManifestFile fileResume = manifestFileFind(manifestResume, manifestName); ASSERT(fileResume.reference == NULL); if (fileResume.checksumSha1 == NULL) removeReason = "no checksum in resumed manifest"; else if (file.size != fileResume.size) removeReason = "mismatched size"; else if (!delta && file.timestamp != fileResume.timestamp) removeReason = "mismatched timestamp"; else if (file.size == 0) // ??? don't resume zero size files because Perl wouldn't -- can be removed after the migration) removeReason = "zero size"; else { ASSERT(file.copy); ASSERT(file.bundleId == 0); ASSERT(file.blockIncrMapSize == 0); file.sizeRepo = fileResume.sizeRepo; file.checksumSha1 = fileResume.checksumSha1; file.checksumRepoSha1 = fileResume.checksumRepoSha1; file.blockIncrSize = fileResume.blockIncrSize; file.blockIncrChecksumSize = fileResume.blockIncrChecksumSize; file.blockIncrMapSize = fileResume.blockIncrMapSize; file.checksumPage = fileResume.checksumPage; file.checksumPageError = fileResume.checksumPageError; file.checksumPageErrorList = fileResume.checksumPageErrorList; file.resume = true; file.delta = delta; manifestFileUpdate(manifest, &file); } } } // Remove the file if it could not be resumed if (removeReason != NULL) { LOG_DETAIL_FMT( "remove file '%s' from resumed backup (%s)", strZ(storagePathP(storageRepo(), backupPath)), removeReason); storageRemoveP(storageRepoWrite(), backupPath); } break; } // Remove links. We could check that the link has not changed and preserve it but it doesn't seem worth the extra // testing. The link will be recreated during the backup if needed. // ----------------------------------------------------------------------------------------------------------------- case storageTypeLink: storageRemoveP(storageRepoWrite(), backupPath); break; // Remove special files // ----------------------------------------------------------------------------------------------------------------- case storageTypeSpecial: LOG_WARN_FMT("remove special file '%s' from resumed backup", strZ(storagePathP(storageRepo(), backupPath))); storageRemoveP(storageRepoWrite(), backupPath); break; } } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } // Helper to find a resumable backup static const Manifest * backupResumeFind(const Manifest *const manifest, const String *const cipherPassBackup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM(STRING, cipherPassBackup); FUNCTION_LOG_END(); ASSERT(manifest != NULL); Manifest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Only the last backup can be resumed const StringList *const backupList = strLstSort( storageListP( storageRepo(), STRDEF(STORAGE_REPO_BACKUP), .expression = backupRegExpP(.full = true, .differential = true, .incremental = true)), sortOrderDesc); if (!strLstEmpty(backupList)) { const String *const backupLabel = strLstGet(backupList, 0); const String *const manifestFile = strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)); // Resumable backups do not have backup.manifest if (!storageExistsP(storageRepo(), manifestFile)) { const bool resume = cfgOptionBool(cfgOptResume) && cfgOptionStrId(cfgOptType) == backupTypeFull; bool usable = false; const char *reason = "resume only valid for full backup"; Manifest *manifestResume = NULL; if (cfgOptionStrId(cfgOptType) == backupTypeFull) { reason = "partially deleted by prior resume or invalid"; // Resumable backups must have backup.manifest.copy if (storageExistsP(storageRepo(), strNewFmt("%s" INFO_COPY_EXT, strZ(manifestFile)))) { reason = "resume is disabled"; // Attempt to read the manifest file in the resumable backup to see if it can be used. If any error at all // occurs then the backup will be considered unusable and a resume will not be attempted. if (resume) { TRY_BEGIN() { manifestResume = manifestLoadFile( storageRepo(), manifestFile, cfgOptionStrId(cfgOptRepoCipherType), cipherPassBackup); } CATCH_ANY() { reason = zNewFmt("unable to read %s" INFO_COPY_EXT, strZ(manifestFile)); } TRY_END(); if (manifestResume != NULL) { const ManifestData *manifestResumeData = manifestData(manifestResume); // Check pgBackRest version. This allows the resume implementation to be changed with each version // of pgBackRest at the expense of users losing a resumable back after an upgrade, which seems worth // the cost. if (!strEq(manifestResumeData->backrestVersion, manifestData(manifest)->backrestVersion)) { reason = zNewFmt( "new " PROJECT_NAME " version '%s' does not match resumable " PROJECT_NAME " version '%s'", strZ(manifestData(manifest)->backrestVersion), strZ(manifestResumeData->backrestVersion)); } // Check backup type because new backup label must be the same type as resume backup label else if (manifestResumeData->backupType != cfgOptionStrId(cfgOptType)) { reason = zNewFmt( "new backup type '%s' does not match resumable backup type '%s'", strZ(cfgOptionDisplay(cfgOptType)), zNewStrId(manifestResumeData->backupType)); } // Check prior backup label ??? Do we really care about the prior backup label? else if (!strEq(manifestResumeData->backupLabelPrior, manifestData(manifest)->backupLabelPrior)) { reason = zNewFmt( "new prior backup label '%s' does not match resumable prior backup label '%s'", manifestResumeData->backupLabelPrior ? strZ(manifestResumeData->backupLabelPrior) : "", manifestData(manifest)->backupLabelPrior ? strZ(manifestData(manifest)->backupLabelPrior) : ""); } // Check compression. Compression can't be changed between backups so resume won't work either. else if ( manifestResumeData->backupOptionCompressType != compressTypeEnum(cfgOptionStrId(cfgOptCompressType))) { reason = zNewFmt( "new compression '%s' does not match resumable compression '%s'", strZ(cfgOptionDisplay(cfgOptCompressType)), strZ(compressTypeStr(manifestResumeData->backupOptionCompressType))); } else usable = true; } } } } // If the backup is usable then return the manifest if (usable) { result = manifestMove(manifestResume, memContextPrior()); } // Else warn and remove the unusable backup else { LOG_FMT( resume ? logLevelWarn : logLevelInfo, 0, "backup '%s' cannot be resumed: %s", strZ(backupLabel), reason); storagePathRemoveP( storageRepoWrite(), strNewFmt(STORAGE_REPO_BACKUP "/%s", strZ(backupLabel)), .recurse = true); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(MANIFEST, result); } static bool backupResume(Manifest *const manifest, const String *const cipherPassBackup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM(STRING, cipherPassBackup); FUNCTION_LOG_END(); ASSERT(manifest != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { const Manifest *const manifestResume = backupResumeFind(manifest, cipherPassBackup); // If a resumable backup was found set the label and cipher subpass if (manifestResume) { // Resuming result = true; // Set the backup label to the resumed backup manifestBackupLabelSet(manifest, manifestData(manifestResume)->backupLabel); LOG_WARN_FMT( "resumable backup %s of same type exists -- invalid files will be removed then the backup will resume", strZ(manifestData(manifest)->backupLabel)); // Copy cipher subpass since it was used to encrypt the resumable files manifestCipherSubPassSet(manifest, manifestCipherSubPass(manifestResume)); // Clean resumed backup const String *const backupPath = strNewFmt(STORAGE_REPO_BACKUP "/%s", strZ(manifestData(manifest)->backupLabel)); backupResumeClean( storageNewItrP(storageRepo(), backupPath, .sortOrder = sortOrderAsc), manifest, manifestResume, compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), cfgOptionBool(cfgOptDelta), backupPath, NULL); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Start the backup ***********************************************************************************************************************************/ typedef struct BackupStartResult { String *lsn; String *walSegmentName; Pack *dbList; Pack *tablespaceList; } BackupStartResult; static BackupStartResult backupStart(const BackupData *const backupData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); BackupStartResult result = {.lsn = NULL}; MEM_CONTEXT_TEMP_BEGIN() { // If this is an offline backup if (!cfgOptionBool(cfgOptOnline)) { // Check if Postgres is running and if so only continue when forced if (storageExistsP(backupData->storagePrimary, PG_FILE_POSTMTRPID_STR)) { if (cfgOptionBool(cfgOptForce)) { LOG_WARN( "--no-" CFGOPT_ONLINE " passed and " PG_FILE_POSTMTRPID " exists but --" CFGOPT_FORCE " was passed so" " backup will continue though it looks like " PG_NAME " is running and the backup will probably not be" " consistent"); } else { THROW( PgRunningError, "--no-" CFGOPT_ONLINE " passed but " PG_FILE_POSTMTRPID " exists - looks like " PG_NAME " is running. Shut" " down " PG_NAME " and try again, or use --force."); } } } // Else start the backup normally else { // Check database configuration checkDbConfig(backupData->version, backupData->pgIdxPrimary, backupData->dbPrimary, false); // Start backup LOG_INFO_FMT( "execute backup start: backup begins after the %s checkpoint completes", cfgOptionBool(cfgOptStartFast) ? "requested immediate" : "next regular"); const DbBackupStartResult dbBackupStartResult = dbBackupStart( backupData->dbPrimary, cfgOptionBool(cfgOptStartFast), cfgOptionBool(cfgOptArchiveCheck)); MEM_CONTEXT_PRIOR_BEGIN() { result.lsn = strDup(dbBackupStartResult.lsn); result.walSegmentName = strDup(dbBackupStartResult.walSegmentName); result.dbList = dbList(backupData->dbPrimary); result.tablespaceList = dbTablespaceList(backupData->dbPrimary); } MEM_CONTEXT_PRIOR_END(); LOG_INFO_FMT("backup start archive = %s, lsn = %s", strZ(result.walSegmentName), strZ(result.lsn)); // Wait for replay on the standby to catch up if (backupData->dbStandby != NULL) { LOG_INFO_FMT("wait for replay on the standby to reach %s", strZ(result.lsn)); dbReplayWait(backupData->dbStandby, result.lsn, backupData->timeline, cfgOptionUInt64(cfgOptArchiveTimeout)); LOG_INFO_FMT("replay on the standby reached %s", strZ(result.lsn)); } // Check that WAL segments are being archived. If archiving is not working then the backup will eventually fail so // better to catch it as early as possible. A segment to check may not be available on older versions of PostgreSQL or // if archive-check is false. if (dbBackupStartResult.walSegmentCheck != NULL) { LOG_INFO_FMT( "check archive for %ssegment %s", strEq(result.walSegmentName, dbBackupStartResult.walSegmentCheck) ? "" : "prior ", strZ(dbBackupStartResult.walSegmentCheck)); walSegmentFindOne( storageRepo(), backupData->archiveId, dbBackupStartResult.walSegmentCheck, cfgOptionUInt64(cfgOptArchiveTimeout)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /*********************************************************************************************************************************** Stop the backup ***********************************************************************************************************************************/ // Helper to write a file from a string to the repository and update the manifest static void backupFilePut( const BackupData *const backupData, Manifest *const manifest, const String *const name, const time_t timestamp, const String *const content) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(TIME, timestamp); FUNCTION_LOG_PARAM(STRING, content); FUNCTION_LOG_END(); // Skip files with no content if (content != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // Create file bool repoChecksum = false; const String *const manifestName = strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(name)); const CompressType compressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType)); StorageWrite *const write = storageNewWriteP( storageRepoWrite(), backupFileRepoPathP( manifestData(manifest)->backupLabel, .manifestName = manifestName, .compressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType))), .compressible = true); IoFilterGroup *const filterGroup = ioWriteFilterGroup(storageWriteIo(write)); // Add SHA1 filter ioFilterGroupAdd(filterGroup, cryptoHashNew(hashTypeSha1)); // Add compression if (compressType != compressTypeNone) { ioFilterGroupAdd( ioWriteFilterGroup(storageWriteIo(write)), compressFilterP(compressType, cfgOptionInt(cfgOptCompressLevel))); repoChecksum = true; } // Add encryption filter if required if (manifestCipherSubPass(manifest) != NULL) { ioFilterGroupAdd( ioWriteFilterGroup(storageWriteIo(write)), cipherBlockNewP( cipherModeEncrypt, cfgOptionStrId(cfgOptRepoCipherType), BUFSTR(manifestCipherSubPass(manifest)))); repoChecksum = true; } // Capture checksum of file stored in the repo if filters that modify the output have been applied if (repoChecksum) ioFilterGroupAdd(filterGroup, cryptoHashNew(hashTypeSha1)); // Add size filter last to calculate repo size ioFilterGroupAdd(filterGroup, ioSizeNew()); // Write file storagePutP(write, BUFSTR(content)); // Use base path to set ownership and mode const ManifestPath *const basePath = manifestPathFind(manifest, MANIFEST_TARGET_PGDATA_STR); // Add to manifest ManifestFile file = { .name = manifestName, .mode = basePath->mode & (S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH), .user = basePath->user, .group = basePath->group, .size = strSize(content), .sizeOriginal = strSize(content), .sizeRepo = pckReadU64P(ioFilterGroupResultP(filterGroup, SIZE_FILTER_TYPE)), .timestamp = timestamp, .checksumSha1 = bufPtr(pckReadBinP(ioFilterGroupResultP(filterGroup, CRYPTO_HASH_FILTER_TYPE, .idx = 0))), }; if (repoChecksum) file.checksumRepoSha1 = bufPtr(pckReadBinP(ioFilterGroupResultP(filterGroup, CRYPTO_HASH_FILTER_TYPE, .idx = 1))); manifestFileAdd(manifest, &file); LOG_DETAIL_FMT("wrote '%s' file returned from backup stop function", strZ(name)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /*--------------------------------------------------------------------------------------------------------------------------------*/ typedef struct BackupStopResult { String *lsn; String *walSegmentName; time_t timestamp; } BackupStopResult; static BackupStopResult backupStop(BackupData *const backupData, Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); BackupStopResult result = {.lsn = NULL}; if (cfgOptionBool(cfgOptOnline)) { MEM_CONTEXT_TEMP_BEGIN() { // Stop the backup LOG_INFO_FMT( "execute backup stop and wait for all WAL segments to archive"); const DbBackupStopResult dbBackupStopResult = dbBackupStop(backupData->dbPrimary); MEM_CONTEXT_PRIOR_BEGIN() { result.timestamp = backupTime(backupData, false); result.lsn = strDup(dbBackupStopResult.lsn); result.walSegmentName = strDup(dbBackupStopResult.walSegmentName); } MEM_CONTEXT_PRIOR_END(); LOG_INFO_FMT("backup stop archive = %s, lsn = %s", strZ(result.walSegmentName), strZ(result.lsn)); // Save files returned by stop backup backupFilePut(backupData, manifest, STRDEF(PG_FILE_BACKUPLABEL), result.timestamp, dbBackupStopResult.backupLabel); backupFilePut(backupData, manifest, STRDEF(PG_FILE_TABLESPACEMAP), result.timestamp, dbBackupStopResult.tablespaceMap); } MEM_CONTEXT_TEMP_END(); } else result.timestamp = backupTime(backupData, false); FUNCTION_LOG_RETURN_STRUCT(result); } /*********************************************************************************************************************************** Convert page checksum error pack to a VariantList ***********************************************************************************************************************************/ // Helper to output pages and page ranges static void backupJobResultPageChecksumOut(VariantList *const result, const unsigned int pageBegin, const unsigned int pageEnd) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, result); FUNCTION_TEST_PARAM(UINT, pageBegin); FUNCTION_TEST_PARAM(UINT, pageEnd); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); // Output a single page if (pageBegin == pageEnd) { varLstAdd(result, varNewUInt64(pageBegin)); } // Else output a page range else { VariantList *errorListSub = varLstNew(); varLstAdd(errorListSub, varNewUInt64(pageBegin)); varLstAdd(errorListSub, varNewUInt64(pageEnd)); varLstAdd(result, varNewVarLst(errorListSub)); } FUNCTION_TEST_RETURN_VOID(); } static VariantList * backupJobResultPageChecksum(PackRead *const checksumPageResult) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, checksumPageResult); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); VariantList *result = NULL; // If there is an error result array if (!pckReadNullP(checksumPageResult)) { result = varLstNew(); pckReadArrayBeginP(checksumPageResult); bool first = false; unsigned int pageBegin = 0; unsigned int pageEnd = 0; // Combine results into a more compact form while (pckReadNext(checksumPageResult)) { const unsigned int pageId = pckReadId(checksumPageResult) - 1; pckReadObjBeginP(checksumPageResult, .id = pageId + 1); // If first error then just store page if (!first) { pageBegin = pageId; pageEnd = pageId; first = true; } // Expand list when the page is in sequence else if (pageId == pageEnd + 1) { pageEnd = pageId; } // Else output the page or page range else { backupJobResultPageChecksumOut(result, pageBegin, pageEnd); // Start again with a single page range pageBegin = pageId; pageEnd = pageId; } pckReadObjEndP(checksumPageResult); } // Check that the array was not empty CHECK(FormatError, first, "page checksum result array is empty"); // Output last page or page range backupJobResultPageChecksumOut(result, pageBegin, pageEnd); pckReadArrayEndP(checksumPageResult); } FUNCTION_LOG_RETURN(VARIANT_LIST, result); } /*********************************************************************************************************************************** Log the results of a job and throw errors ***********************************************************************************************************************************/ static void backupJobResult( Manifest *const manifest, const String *const host, const Storage *const storagePg, StringList *const fileRemove, ProtocolParallelJob *const job, const bool bundle, const PgPageSize pageSize, const uint64_t sizeTotal, uint64_t *const sizeProgress, unsigned int *const currentPercentComplete) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(STORAGE, storagePg); FUNCTION_LOG_PARAM(STRING_LIST, fileRemove); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, job); FUNCTION_LOG_PARAM(BOOL, bundle); FUNCTION_LOG_PARAM(ENUM, pageSize); FUNCTION_LOG_PARAM(UINT64, sizeTotal); FUNCTION_LOG_PARAM_P(UINT64, sizeProgress); FUNCTION_LOG_PARAM_P(UINT, currentPercentComplete); FUNCTION_LOG_END(); ASSERT(manifest != NULL); ASSERT(storagePg != NULL); ASSERT(fileRemove != NULL); ASSERT(job != NULL); // The job was successful if (protocolParallelJobErrorCode(job) == 0) { MEM_CONTEXT_TEMP_BEGIN() { const unsigned int processId = protocolParallelJobProcessId(job); const uint64_t bundleId = varType(protocolParallelJobKey(job)) == varTypeUInt64 ? varUInt64(protocolParallelJobKey(job)) : 0; PackRead *const jobResult = protocolParallelJobResult(job); unsigned int percentComplete = 0; while (!pckReadNullP(jobResult)) { ManifestFile file = manifestFileFind(manifest, pckReadStrP(jobResult)); const BackupCopyResult copyResult = (BackupCopyResult)pckReadU32P(jobResult); const bool repoInvalid = pckReadBoolP(jobResult); const uint64_t copySize = pckReadU64P(jobResult); const uint64_t bundleOffset = pckReadU64P(jobResult); const uint64_t blockIncrMapSize = pckReadU64P(jobResult); const uint64_t repoSize = pckReadU64P(jobResult); const Buffer *const copyChecksum = pckReadBinP(jobResult); const Buffer *const repoChecksum = pckReadBinP(jobResult); PackRead *const checksumPageResult = pckReadPackReadP(jobResult); // Increment backup copy progress. Use the original size since the size may have changed during the copy but for the // purpose of reporting progress we need to increment by the original size used to generate the total size. *sizeProgress += file.sizeOriginal; // Create log file name const String *const fileName = storagePathP(storagePg, manifestPathPg(file.name)); const String *const fileLog = host == NULL ? fileName : strNewFmt("%s:%s", strZ(host), strZ(fileName)); // Format log progress String *const logProgress = strNew(); if (bundleId != 0 && copyResult != backupCopyResultNoOp && copyResult != backupCopyResultTruncate) strCatFmt(logProgress, "bundle %" PRIu64 "/%" PRIu64 ", ", bundleId, bundleOffset); // Log original manifest size if copy size differs if (copySize != file.sizeOriginal) strCatFmt(logProgress, "%s->", strZ(strSizeFormat(file.sizeOriginal))); // Store percentComplete as an integer (used to update progress in the lock file) percentComplete = cvtPctToUInt(*sizeProgress, sizeTotal); strCatFmt(logProgress, "%s, %s", strZ(strSizeFormat(copySize)), strZ(strNewPct(*sizeProgress, sizeTotal))); // Format log checksum const String *const logChecksum = copySize != 0 ? strNewFmt(" checksum %s", strZ(strNewEncode(encodingHex, copyChecksum))) : EMPTY_STR; // If the file is in a prior backup and nothing changed, just log it if (copyResult == backupCopyResultNoOp) { LOG_DETAIL_PID_FMT( processId, "match file from prior backup %s (%s)%s", strZ(fileLog), strZ(logProgress), strZ(logChecksum)); } // Else if the repo matched the expect checksum, just log it else if (copyResult == backupCopyResultChecksum) { LOG_DETAIL_PID_FMT( processId, "checksum resumed file %s (%s)%s", strZ(fileLog), strZ(logProgress), strZ(logChecksum)); } // Else if the file was removed during backup add it to the list of files to be removed from the manifest when the // backup is complete. It can't be removed right now because that will invalidate the pointers that are being used // for processing. else if (copyResult == backupCopyResultSkip) { LOG_DETAIL_PID_FMT(processId, "skip file removed by database %s", strZ(fileLog)); strLstAdd(fileRemove, file.name); } // Else file was copied so update manifest else { LOG_DETAIL_PID_FMT( processId, "%s file %s (%s)%s", copyResult != backupCopyResultTruncate ? "backup" : "store truncated", strZ(fileLog), strZ(logProgress), strZ(logChecksum)); // If the repo file was invalid warn that there might be corruption in the repository if (repoInvalid) { ASSERT(copyResult == backupCopyResultCopy); LOG_WARN_FMT( "resumed backup file %s did not have expected checksum %s. The file was recopied and backup will" " continue but this may be an issue unless the resumed backup path in the repository is known to be" " corrupted.\n" "NOTE: this does not indicate a problem with the PostgreSQL page checksums.", strZ(file.name), strZ(strNewEncode(encodingHex, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE)))); } // If the file had page checksums calculated during the copy ASSERT((!file.checksumPage && checksumPageResult == NULL) || (file.checksumPage && checksumPageResult != NULL)); bool checksumPageError = false; const VariantList *checksumPageErrorList = NULL; if (checksumPageResult != NULL) { checksumPageErrorList = backupJobResultPageChecksum(checksumPageResult); // If the checksum was valid if (!pckReadBoolP(checksumPageResult)) { checksumPageError = true; if (!pckReadBoolP(checksumPageResult)) { checksumPageErrorList = NULL; // ??? Update formatting after migration LOG_WARN_FMT( "page misalignment in file %s: file size %" PRIu64 " is not divisible by page size %u", strZ(fileLog), copySize, pageSize); } else { // Format the page checksum errors CHECK(FormatError, checksumPageErrorList != NULL, "page checksum error list is missing"); CHECK(FormatError, !varLstEmpty(checksumPageErrorList), "page checksum error list is empty"); String *const error = strNew(); unsigned int errorTotalMin = 0; for (unsigned int errorIdx = 0; errorIdx < varLstSize(checksumPageErrorList); errorIdx++) { const Variant *const errorItem = varLstGet(checksumPageErrorList, errorIdx); // Add a comma if this is not the first item if (errorIdx != 0) strCatZ(error, ", "); // If an error range if (varType(errorItem) == varTypeVariantList) { const VariantList *const errorItemList = varVarLst(errorItem); ASSERT(varLstSize(errorItemList) == 2); strCatFmt( error, "%" PRIu64 "-%" PRIu64, varUInt64(varLstGet(errorItemList, 0)), varUInt64(varLstGet(errorItemList, 1))); errorTotalMin += 2; } // Else a single error else { ASSERT(varType(errorItem) == varTypeUInt64); strCatFmt(error, "%" PRIu64, varUInt64(errorItem)); errorTotalMin++; } } // Make message plural when appropriate const char *const plural = errorTotalMin > 1 ? "s" : ""; // ??? Update formatting after migration LOG_WARN_FMT( "invalid page checksum%s found in file %s at page%s %s", plural, strZ(fileLog), plural, strZ(error)); } } } // Update file info and remove any reference to the file's existence in a prior backup file.size = copySize; file.sizeRepo = repoSize; file.checksumSha1 = bufPtrConst(copyChecksum); file.checksumRepoSha1 = repoChecksum != NULL ? bufPtrConst(repoChecksum) : NULL; file.reference = NULL; file.checksumPageError = checksumPageError; file.checksumPageErrorList = checksumPageErrorList != NULL ? jsonFromVar(varNewVarLst(checksumPageErrorList)) : NULL; // Truncated file is not put in bundle file.bundleId = copyResult != backupCopyResultTruncate ? bundleId : 0; file.bundleOffset = bundleOffset; file.blockIncrMapSize = blockIncrMapSize; manifestFileUpdate(manifest, &file); } } // Update currentPercentComplete and lock file when the change is significant enough if (percentComplete - *currentPercentComplete > 10) { *currentPercentComplete = percentComplete; cmdLockWriteP( .percentComplete = VARUINT(*currentPercentComplete), .sizeComplete = VARUINT64(*sizeProgress), .size = VARUINT64(sizeTotal)); } } MEM_CONTEXT_TEMP_END(); // Free the job protocolParallelJobFree(job); } // Else the job errored else THROW_CODE(protocolParallelJobErrorCode(job), strZ(protocolParallelJobErrorMessage(job))); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Save a copy of the backup manifest during processing to preserve checksums for a possible resume. Only save the final copy when resume is disabled since an incremental copy will not be used in a future backup unless resume is enabled beforehand. ***********************************************************************************************************************************/ static void backupManifestSaveCopy(Manifest *const manifest, const String *const cipherPassBackup, const bool final) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM(STRING, cipherPassBackup); FUNCTION_LOG_PARAM(BOOL, final); FUNCTION_LOG_END(); ASSERT(manifest != NULL); if (final || cfgOptionBool(cfgOptResume)) { MEM_CONTEXT_TEMP_BEGIN() { // Open file for write IoWrite *const write = storageWriteIo( storageNewWriteP( storageRepoWrite(), strNewFmt( STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(manifestData(manifest)->backupLabel)))); // Add encryption filter if required cipherBlockFilterGroupAdd( ioWriteFilterGroup(write), cfgOptionStrId(cfgOptRepoCipherType), cipherModeEncrypt, cipherPassBackup); // Save file manifestSave(manifest, write); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Check that the clusters are alive and correctly configured during the backup ***********************************************************************************************************************************/ static void backupDbPing(const BackupData *const backupData, const bool force) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(BOOL, force); FUNCTION_LOG_END(); ASSERT(backupData != NULL); if (cfgOptionBool(cfgOptOnline)) { dbPing(backupData->dbPrimary, force); if (backupData->dbStandby != NULL) dbPing(backupData->dbStandby, force); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Process the backup manifest ***********************************************************************************************************************************/ typedef struct BackupJobData { const Manifest *const manifest; // Backup manifest const String *const backupLabel; // Backup label (defines the backup path) const bool backupStandby; // Backup from standby RegExp *standbyExp; // Identify files that may be copied from the standby const CipherType cipherType; // Cipher type const String *const cipherSubPass; // Passphrase used to encrypt files in the backup const PgPageSize pageSize; // Page size const CompressType compressType; // Backup compression type const int compressLevel; // Compress level if backup is compressed const bool delta; // Is this a checksum delta backup? const bool bundle; // Bundle files? uint64_t bundleSize; // Target bundle size uint64_t bundleLimit; // Limit on files to bundle uint64_t bundleId; // Bundle id const bool blockIncr; // Block incremental? size_t blockIncrSizeSuper; // Super block size List *queueList; // List of processing queues } BackupJobData; // Identify files that must be copied from the primary static bool backupProcessFilePrimary(RegExp *const standbyExp, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(REGEXP, standbyExp); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(standbyExp != NULL); ASSERT(name != NULL); FUNCTION_TEST_RETURN( BOOL, strEqZ(name, MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL) || !regExpMatch(standbyExp, name)); } // Comparator to order ManifestFile objects by size, date, and name static const Manifest *backupProcessQueueComparatorManifest = NULL; static bool backupProcessQueueComparatorBundle; static uint64_t backupProcessQueueComparatorBundleLimit; static int backupProcessQueueComparator(const void *const item1, const void *const item2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, item1); FUNCTION_TEST_PARAM_P(VOID, item2); FUNCTION_TEST_END(); ASSERT(item1 != NULL); ASSERT(item2 != NULL); // Unpack files const ManifestFile file1 = manifestFileUnpack(backupProcessQueueComparatorManifest, *(const ManifestFilePack *const *)item1); const ManifestFile file2 = manifestFileUnpack(backupProcessQueueComparatorManifest, *(const ManifestFilePack *const *)item2); // If the size differs then that's enough to determine order if (!backupProcessQueueComparatorBundle || file1.size > backupProcessQueueComparatorBundleLimit || file2.size > backupProcessQueueComparatorBundleLimit) { if (file1.size < file2.size) FUNCTION_TEST_RETURN(INT, 1); else if (file1.size > file2.size) FUNCTION_TEST_RETURN(INT, -1); } // If bundling order by time desc so that older files are bundled with older files and newer with newer if (backupProcessQueueComparatorBundle) { if (file1.timestamp > file2.timestamp) FUNCTION_TEST_RETURN(INT, 1); else if (file1.timestamp < file2.timestamp) FUNCTION_TEST_RETURN(INT, -1); } // If size/time is the same then use name to generate a deterministic ordering (names must be unique) FUNCTION_TEST_RETURN(INT, strCmp(file2.name, file1.name)); } // Helper to generate the backup queues static uint64_t backupProcessQueue(const BackupData *const backupData, Manifest *const manifest, BackupJobData *const jobData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM_P(VOID, jobData); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(manifest != NULL); uint64_t result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Create list of process queues (use void * instead of List * to avoid Coverity false positive) jobData->queueList = lstNewP(sizeof(void *)); // Generate the list of targets StringList *const targetList = strLstNew(); strLstAddZ(targetList, MANIFEST_TARGET_PGDATA "/"); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); if (target->tablespaceId != 0) strLstAddFmt(targetList, "%s/", strZ(target->name)); } // Generate the processing queues (there is always at least one) const unsigned int queueOffset = jobData->backupStandby ? 1 : 0; MEM_CONTEXT_BEGIN(lstMemContext(jobData->queueList)) { for (unsigned int queueIdx = 0; queueIdx < strLstSize(targetList) + queueOffset; queueIdx++) { List *const queue = lstNewP(sizeof(ManifestFile *), .comparator = backupProcessQueueComparator); lstAdd(jobData->queueList, &queue); } } MEM_CONTEXT_END(); // Now put all files into the processing queues uint64_t fileTotal = 0; bool pgControlFound = false; for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFilePack *const filePack = manifestFilePackGet(manifest, fileIdx); const ManifestFile file = manifestFileUnpack(manifest, filePack); // Only process files that need to be copied if (!file.copy) { // If bundling log zero-length files as stored since they will never be copied if (file.size == 0 && jobData->bundle) { LOG_DETAIL_FMT( "store zero-length file %s", strZ(storagePathP(backupData->storagePrimary, manifestPathPg(file.name)))); } continue; } // Is pg_control in the backup? if (strEq(file.name, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL))) pgControlFound = true; // Files that must be copied from the primary are always put in queue 0 when backup from standby if (jobData->backupStandby && backupProcessFilePrimary(jobData->standbyExp, file.name)) { lstAdd(*(List **)lstGet(jobData->queueList, 0), &filePack); } // Else find the correct queue by matching the file to a target else { // Find the target that contains this file unsigned int targetIdx = 0; do { CHECK(AssertError, targetIdx < strLstSize(targetList), "backup target not found"); if (strBeginsWith(file.name, strLstGet(targetList, targetIdx))) break; targetIdx++; } while (1); // Add file to queue lstAdd(*(List **)lstGet(jobData->queueList, targetIdx + queueOffset), &filePack); } // Add size to total result += file.sizeOriginal; // Increment total files fileTotal++; } // pg_control should always be in an online backup if (!pgControlFound && cfgOptionBool(cfgOptOnline)) { THROW( FileMissingError, PG_FILE_PGCONTROL " must be present in all online backups\n" "HINT: is something wrong with the clock or filesystem timestamps?"); } // If there are no files to backup then we'll exit with an error. This could happen if the database is down and backup is // called with --no-online twice in a row. if (fileTotal == 0) THROW(FileMissingError, "no files have changed since the last backup - this seems unlikely"); // Sort the queues backupProcessQueueComparatorManifest = manifest; backupProcessQueueComparatorBundle = jobData->bundle; backupProcessQueueComparatorBundleLimit = jobData->bundleLimit; for (unsigned int queueIdx = 0; queueIdx < lstSize(jobData->queueList); queueIdx++) lstSort(*(List **)lstGet(jobData->queueList, queueIdx), sortOrderAsc); // Move process queues to prior context lstMove(jobData->queueList, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT64, result); } // Helper to calculate the next queue to scan based on the client index static int backupJobQueueNext(const unsigned int clientIdx, int queueIdx, const unsigned int queueTotal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_PARAM(INT, queueIdx); FUNCTION_TEST_PARAM(UINT, queueTotal); FUNCTION_TEST_END(); // Move (forward or back) to the next queue queueIdx += clientIdx % 2 ? -1 : 1; // Deal with wrapping on either end if (queueIdx < 0) FUNCTION_TEST_RETURN(INT, (int)queueTotal - 1); else if (queueIdx == (int)queueTotal) FUNCTION_TEST_RETURN(INT, 0); FUNCTION_TEST_RETURN(INT, queueIdx); } // Callback to fetch backup jobs for the parallel executor static ProtocolParallelJob * backupJobCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_END(); ASSERT(data != NULL); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get a new job if there are any left BackupJobData *const jobData = data; // Determine where to begin scanning the queue (we'll stop when we get back here). When copying from the primary during // backup from standby only queue 0 will be used. const unsigned int queueOffset = jobData->backupStandby && clientIdx > 0 ? 1 : 0; int queueIdx = jobData->backupStandby && clientIdx == 0 ? 0 : (int)(clientIdx % (lstSize(jobData->queueList) - queueOffset)); const int queueEnd = queueIdx; // Create backup job PackWrite *param = NULL; uint64_t fileTotal = 0; uint64_t fileSize = 0; do { List *const queue = *(List **)lstGet(jobData->queueList, (unsigned int)queueIdx + queueOffset); unsigned int fileIdx = 0; bool bundle = jobData->bundle; const String *fileName = NULL; while (fileIdx < lstSize(queue)) { const ManifestFile file = manifestFileUnpack(jobData->manifest, *(ManifestFilePack **)lstGet(queue, fileIdx)); // Is this file a block incremental? const bool blockIncr = jobData->blockIncr && file.blockIncrSize > 0; // Add common parameters before first file if (param == NULL) { param = protocolPackNew(); if (bundle && file.size <= jobData->bundleLimit) { pckWriteStrP(param, backupFileRepoPathP(jobData->backupLabel, .bundleId = jobData->bundleId)); pckWriteU64P(param, jobData->bundleId); pckWriteBoolP(param, manifestData(jobData->manifest)->bundleRaw); } else { CHECK(AssertError, fileTotal == 0, "cannot bundle file"); pckWriteStrP( param, backupFileRepoPathP( jobData->backupLabel, .manifestName = file.name, .compressType = jobData->compressType, .blockIncr = blockIncr)); pckWriteU64P(param, 0); fileName = file.name; bundle = false; } // Provide the backup reference pckWriteU64P(param, strLstSize(manifestReferenceList(jobData->manifest)) - 1); pckWriteU32P(param, jobData->compressType); pckWriteI32P(param, jobData->compressLevel); pckWriteU64P(param, jobData->cipherSubPass == NULL ? cipherTypeNone : cipherTypeAes256Cbc); pckWriteStrP(param, jobData->cipherSubPass); pckWriteU32P(param, jobData->pageSize); pckWriteStrP(param, cfgOptionStrNull(cfgOptPgVersionForce)); } pckWriteStrP(param, manifestPathPg(file.name)); pckWriteBoolP(param, file.delta); pckWriteBoolP(param, !strEq(file.name, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL))); pckWriteU64P(param, file.size); pckWriteU64P(param, file.sizeOriginal); pckWriteBoolP(param, !backupProcessFilePrimary(jobData->standbyExp, file.name)); pckWriteBinP(param, file.checksumSha1 != NULL ? BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE) : NULL); pckWriteBoolP(param, file.checksumPage); pckWriteBoolP(param, cfgOptionBool(cfgOptPageHeaderCheck)); // If block incremental then provide the location of the prior map when available if (blockIncr) { pckWriteU64P(param, file.blockIncrSize); pckWriteU64P(param, file.blockIncrChecksumSize); pckWriteU64P(param, jobData->blockIncrSizeSuper); if (file.blockIncrMapSize != 0 && !file.resume) { pckWriteStrP( param, backupFileRepoPathP( file.reference, .manifestName = file.name, .bundleId = file.bundleId, .blockIncr = true)); pckWriteU64P(param, file.bundleOffset + file.sizeRepo - file.blockIncrMapSize); pckWriteU64P(param, file.blockIncrMapSize); } else pckWriteNullP(param); } else pckWriteU64P(param, 0); pckWriteStrP(param, file.name); pckWriteBinP(param, file.checksumRepoSha1 != NULL ? BUF(file.checksumRepoSha1, HASH_TYPE_SHA1_SIZE) : NULL); pckWriteU64P(param, file.sizeRepo); pckWriteBoolP(param, file.resume); pckWriteBoolP(param, file.reference != NULL); fileTotal++; fileSize += file.sizeOriginal; // Remove job from the queue lstRemoveIdx(queue, fileIdx); // Break if not bundling or bundle size has been reached if (!bundle || fileSize >= jobData->bundleSize) break; } if (fileTotal > 0) { // Assign job to result MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew( bundle ? VARUINT64(jobData->bundleId) : VARSTR(fileName), PROTOCOL_COMMAND_BACKUP_FILE, param); if (bundle) jobData->bundleId++; } MEM_CONTEXT_PRIOR_END(); break; } // Don't get next queue when copying from primary during backup from standby since the primary only has one queue if (!jobData->backupStandby || clientIdx > 0) queueIdx = backupJobQueueNext(clientIdx, queueIdx, lstSize(jobData->queueList) - queueOffset); } while (queueIdx != queueEnd); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } static void backupProcess(const BackupData *const backupData, Manifest *const manifest, const String *const cipherPassBackup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM(STRING, cipherPassBackup); FUNCTION_LOG_END(); ASSERT(manifest != NULL); uint64_t sizeTotal = 0; MEM_CONTEXT_TEMP_BEGIN() { // Get backup info const BackupType backupType = manifestData(manifest)->backupType; const String *const backupLabel = manifestData(manifest)->backupLabel; const String *const backupPathExp = strNewFmt(STORAGE_REPO_BACKUP "/%s", strZ(backupLabel)); const bool hardLink = cfgOptionBool(cfgOptRepoHardlink) && storageFeature(storageRepoWrite(), storageFeatureHardLink); const bool backupStandby = backupData->dbStandby != NULL; BackupJobData jobData = { .manifest = manifest, .backupLabel = backupLabel, .backupStandby = backupData->dbStandby != NULL, .compressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), .compressLevel = cfgOptionInt(cfgOptCompressLevel), .cipherType = cfgOptionStrId(cfgOptRepoCipherType), .cipherSubPass = manifestCipherSubPass(manifest), .pageSize = backupData->pageSize, .delta = cfgOptionBool(cfgOptDelta), .bundle = cfgOptionBool(cfgOptRepoBundle), .bundleId = 1, .blockIncr = cfgOptionBool(cfgOptRepoBlock), // Build expression to identify files that can be copied from the standby when standby backup is supported .standbyExp = regExpNew( strNewFmt( "^((" MANIFEST_TARGET_PGDATA "/(" PG_PATH_BASE "|" PG_PATH_GLOBAL "|%s|" PG_PATH_PGMULTIXACT "))" "|" MANIFEST_TARGET_PGTBLSPC ")/", strZ(pgXactPath(backupData->version)))), }; if (jobData.bundle) { jobData.bundleSize = cfgOptionUInt64(cfgOptRepoBundleSize); jobData.bundleLimit = cfgOptionUInt64(cfgOptRepoBundleLimit); } if (jobData.blockIncr) { // Set super block size based on the backup type jobData.blockIncrSizeSuper = backupType == backupTypeFull ? (size_t)cfgOptionUInt64(cfgOptRepoBlockSizeSuperFull) : (size_t)cfgOptionUInt64(cfgOptRepoBlockSizeSuper); } // If this is a full backup or hard-linked and paths are supported then create all paths explicitly so that empty paths will // exist in the repo. Also create tablespace symlinks when symlinks are available. This makes it possible for the user to // make a copy of the backup path and get a valid cluster. if ((backupType == backupTypeFull && !jobData.bundle) || hardLink) { // Create paths when available if (storageFeature(storageRepoWrite(), storageFeaturePath)) { for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(manifest); pathIdx++) { storagePathCreateP( storageRepoWrite(), strNewFmt("%s/%s", strZ(backupPathExp), strZ(manifestPath(manifest, pathIdx)->name))); } } // Create tablespace symlinks when available if (storageFeature(storageRepoWrite(), storageFeatureSymLink)) { for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); if (target->tablespaceId != 0) { const String *const link = storagePathP( storageRepo(), strNewFmt("%s/" MANIFEST_TARGET_PGDATA "/%s", strZ(backupPathExp), strZ(target->name))); const String *const linkDestination = strNewFmt( "../../" MANIFEST_TARGET_PGTBLSPC "/%u", target->tablespaceId); storageLinkCreateP(storageRepoWrite(), linkDestination, link); } } } } // Generate processing queues sizeTotal = backupProcessQueue(backupData, manifest, &jobData); // Create the parallel executor ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, backupJobCallback, &jobData); // First client is always on the primary protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypePg, backupData->pgIdxPrimary, 1)); // Create the rest of the clients on the primary or standby depending on the value of backup-standby. Note that standby // backups don't count the primary client in process-max. const unsigned int processMax = cfgOptionUInt(cfgOptProcessMax) + (jobData.backupStandby ? 1 : 0); const unsigned int pgIdx = jobData.backupStandby ? backupData->pgIdxStandby : backupData->pgIdxPrimary; for (unsigned int processIdx = 2; processIdx <= processMax; processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypePg, pgIdx, processIdx)); // Maintain a list of files that need to be removed from the manifest when the backup is complete StringList *const fileRemove = strLstNew(); // Determine how often the manifest will be saved (every one percent or threshold size, whichever is greater) uint64_t manifestSaveLast = 0; uint64_t manifestSaveSize = sizeTotal / 100; if (manifestSaveSize < cfgOptionUInt64(cfgOptManifestSaveThreshold)) manifestSaveSize = cfgOptionUInt64(cfgOptManifestSaveThreshold); // Process jobs uint64_t sizeProgress = 0; // Initialize percent complete and bytes completed/total unsigned int currentPercentComplete = 0; cmdLockWriteP( .percentComplete = VARUINT(currentPercentComplete), .sizeComplete = VARUINT64(sizeProgress), .size = VARUINT64(sizeTotal)); MEM_CONTEXT_TEMP_RESET_BEGIN() { do { const unsigned int completed = protocolParallelProcess(parallelExec); for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { ProtocolParallelJob *const job = protocolParallelResult(parallelExec); backupJobResult( manifest, backupStandby && protocolParallelJobProcessId(job) > 1 ? backupData->hostStandby : backupData->hostPrimary, protocolParallelJobProcessId(job) > 1 ? storagePgIdx(pgIdx) : backupData->storagePrimary, fileRemove, job, jobData.bundle, jobData.pageSize, sizeTotal, &sizeProgress, ¤tPercentComplete); } // A keep-alive is required here for the remote holding open the backup connection protocolKeepAlive(); // Check that the clusters are alive and correctly configured during the backup backupDbPing(backupData, false); // Save the manifest periodically to preserve checksums for resume if (sizeProgress - manifestSaveLast >= manifestSaveSize) { backupManifestSaveCopy(manifest, cipherPassBackup, false); manifestSaveLast = sizeProgress; } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); #ifdef DEBUG // Ensure that all processing queues are empty for (unsigned int queueIdx = 0; queueIdx < lstSize(jobData.queueList); queueIdx++) ASSERT(lstEmpty(*(List **)lstGet(jobData.queueList, queueIdx))); #endif // Remove files from the manifest that were removed during the backup. This must happen after processing to avoid // invalidating pointers by deleting items from the list. for (unsigned int fileRemoveIdx = 0; fileRemoveIdx < strLstSize(fileRemove); fileRemoveIdx++) manifestFileRemove(manifest, strLstGet(fileRemove, fileRemoveIdx)); // Log references or create hardlinks for all files const char *const compressExt = strZ(compressExtStr(jobData.compressType)); for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); // If the file has a reference, then it was not copied since it can be retrieved from the referenced backup. However, // if hardlinking is enabled the link will need to be created. if (file.reference != NULL) { // If hardlinking is enabled then create a hardlink for files that have not changed since the last backup if (hardLink) { LOG_DETAIL_FMT("hardlink %s to %s", strZ(file.name), strZ(file.reference)); const String *const linkName = storagePathP( storageRepo(), strNewFmt("%s/%s%s", strZ(backupPathExp), strZ(file.name), compressExt)); const String *const linkDestination = storagePathP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/%s%s", strZ(file.reference), strZ(file.name), compressExt)); storageLinkCreateP(storageRepoWrite(), linkDestination, linkName, .linkType = storageLinkHard); } // Else log the reference. With delta, it is possible that references may have been removed if a file needed to be // recopied. else LOG_DETAIL_FMT("reference %s to %s", strZ(file.name), strZ(file.reference)); } } // Sync backup paths if required if (storageFeature(storageRepoWrite(), storageFeaturePathSync)) { for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(manifest); pathIdx++) { const String *const path = strNewFmt("%s/%s", strZ(backupPathExp), strZ(manifestPath(manifest, pathIdx)->name)); // Always sync the path if it exists or if the backup is full (without bundling) or hardlinked. In the latter cases // the directory should always exist so we want to error if it does not. if ((backupType == backupTypeFull && !jobData.bundle) || hardLink || storagePathExistsP(storageRepo(), path)) storagePathSyncP(storageRepoWrite(), path); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Check and copy WAL segments required to make the backup consistent ***********************************************************************************************************************************/ static void backupArchiveCheckCopy(const BackupData *const backupData, Manifest *const manifest, const String *const cipherPassBackup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BACKUP_DATA, backupData); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM(STRING, cipherPassBackup); FUNCTION_LOG_END(); ASSERT(manifest != NULL); // If archive logs are required to complete the backup, then check them. This is the default, but can be overridden if the // archive logs are going to a different server. Be careful of disabling this option because there is no way to verify that the // backup will be consistent - at least not here. if (cfgOptionBool(cfgOptArchiveCheck)) { MEM_CONTEXT_TEMP_BEGIN() { const uint64_t lsnStart = pgLsnFromStr(manifestData(manifest)->lsnStart); const uint64_t lsnStop = pgLsnFromStr(manifestData(manifest)->lsnStop); LOG_INFO_FMT( "check archive for segment(s) %s:%s", strZ(pgLsnToWalSegment(backupData->timeline, lsnStart, backupData->walSegmentSize)), strZ(pgLsnToWalSegment(backupData->timeline, lsnStop, backupData->walSegmentSize))); // Save the backup manifest before getting archive logs in case of failure backupManifestSaveCopy(manifest, cipherPassBackup, false); // Use base path to set ownership and mode const ManifestPath *const basePath = manifestPathFind(manifest, MANIFEST_TARGET_PGDATA_STR); // Loop through all the segments in the lsn range const StringList *const walSegmentList = pgLsnRangeToWalSegmentList( backupData->timeline, lsnStart, lsnStop, backupData->walSegmentSize); WalSegmentFind *const find = walSegmentFindNew( storageRepo(), backupData->archiveId, strLstSize(walSegmentList) == 1, cfgOptionUInt64(cfgOptArchiveTimeout)); for (unsigned int walSegmentIdx = 0; walSegmentIdx < strLstSize(walSegmentList); walSegmentIdx++) { MEM_CONTEXT_TEMP_BEGIN() { // Find the actual wal segment file (including checksum compression extension) in the archive const String *const walSegment = strLstGet(walSegmentList, walSegmentIdx); const String *const archiveFile = walSegmentFind(find, walSegment); if (cfgOptionBool(cfgOptArchiveCopy)) { // Copy can be a pretty expensive operation so log it LOG_DETAIL_FMT("copy segment %s to backup", strZ(walSegment)); // Get compression type of the WAL segment and backup const CompressType archiveCompressType = compressTypeFromName(archiveFile); const CompressType backupCompressType = compressTypeEnum(cfgOptionStrId(cfgOptCompressType)); // Open the archive file StorageRead *const read = storageNewReadP( storageRepo(), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(backupData->archiveId), strZ(archiveFile))); IoFilterGroup *const filterGroup = ioReadFilterGroup(storageReadIo(read)); // Decrypt with archive key if encrypted cipherBlockFilterGroupAdd( filterGroup, cfgOptionStrId(cfgOptRepoCipherType), cipherModeDecrypt, infoArchiveCipherPass(backupData->archiveInfo)); // Compress/decompress if archive and backup do not have the same compression settings if (archiveCompressType != backupCompressType) { if (archiveCompressType != compressTypeNone) ioFilterGroupAdd(filterGroup, decompressFilterP(archiveCompressType)); if (backupCompressType != compressTypeNone) { ioFilterGroupAdd( filterGroup, compressFilterP(backupCompressType, cfgOptionInt(cfgOptCompressLevel))); } } // Encrypt with backup key if encrypted cipherBlockFilterGroupAdd( filterGroup, cfgOptionStrId(cfgOptRepoCipherType), cipherModeEncrypt, manifestCipherSubPass(manifest)); // Add size filter last to calculate repo size ioFilterGroupAdd(filterGroup, ioSizeNew()); // Copy the file const String *const manifestName = strNewFmt( MANIFEST_TARGET_PGDATA "/%s/%s", strZ(pgWalPath(manifestData(manifest)->pgVersion)), strZ(walSegment)); storageCopyP( read, storageNewWriteP( storageRepoWrite(), backupFileRepoPathP( manifestData(manifest)->backupLabel, manifestName, 0, compressTypeEnum(cfgOptionStrId(cfgOptCompressType)), false))); // Add to manifest ManifestFile file = { .name = manifestName, .mode = basePath->mode & (S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH), .user = basePath->user, .group = basePath->group, .size = backupData->walSegmentSize, .sizeOriginal = backupData->walSegmentSize, .sizeRepo = pckReadU64P(ioFilterGroupResultP(filterGroup, SIZE_FILTER_TYPE)), .timestamp = manifestData(manifest)->backupTimestampStop, .checksumSha1 = bufPtr(bufNewDecode(encodingHex, strSubN(archiveFile, 25, 40))), }; manifestFileAdd(manifest, &file); } } MEM_CONTEXT_TEMP_END(); // A keep-alive is required here for the remote holding the backup lock protocolKeepAlive(); } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Save and update all files required to complete the backup ***********************************************************************************************************************************/ static void backupComplete(InfoBackup *const infoBackup, Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const backupLabel = manifestData(manifest)->backupLabel; // Validation and final save of the backup manifest. Validate in strict mode to catch as many potential issues as possible. // ------------------------------------------------------------------------------------------------------------------------- manifestValidate(manifest, true); backupManifestSaveCopy(manifest, infoPgCipherPass(infoBackupPg(infoBackup)), true); storageCopy( storageNewReadP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(backupLabel))), storageNewWriteP( storageRepoWrite(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)))); // Copy a compressed version of the manifest to history. If the repo is encrypted then the passphrase to open the manifest // is required. We can't just do a straight copy since the destination needs to be compressed and that must happen before // encryption in order to be efficient. Compression will always be gz for compatibility and since it is always available. // ------------------------------------------------------------------------------------------------------------------------- StorageRead *const manifestRead = storageNewReadP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel))); cipherBlockFilterGroupAdd( ioReadFilterGroup(storageReadIo(manifestRead)), cfgOptionStrId(cfgOptRepoCipherType), cipherModeDecrypt, infoPgCipherPass(infoBackupPg(infoBackup))); StorageWrite *const manifestWrite = storageNewWriteP( storageRepoWrite(), strNewFmt( STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s/%s.manifest%s", strZ(strSubN(backupLabel, 0, 4)), strZ(backupLabel), strZ(compressExtStr(compressTypeGz)))); ioFilterGroupAdd(ioWriteFilterGroup(storageWriteIo(manifestWrite)), compressFilterP(compressTypeGz, 9)); cipherBlockFilterGroupAdd( ioWriteFilterGroup(storageWriteIo(manifestWrite)), cfgOptionStrId(cfgOptRepoCipherType), cipherModeEncrypt, infoPgCipherPass(infoBackupPg(infoBackup))); storageCopyP(manifestRead, manifestWrite); // Sync history path if required if (storageFeature(storageRepoWrite(), storageFeaturePathSync)) storagePathSyncP(storageRepoWrite(), STRDEF(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY)); // Create a symlink to the most recent backup if supported. This link is purely informational for the user and is never used // by us since symlinks are not supported on all storage types. // ------------------------------------------------------------------------------------------------------------------------- backupLinkLatest(backupLabel, cfgOptionGroupIdxDefault(cfgOptGrpRepo)); // Add manifest and save backup.info (infoBackupSaveFile() is responsible for proper syncing) // ------------------------------------------------------------------------------------------------------------------------- infoBackupDataAdd(infoBackup, manifest); infoBackupSaveFile( infoBackup, storageRepoWrite(), INFO_BACKUP_PATH_FILE_STR, cfgOptionStrId(cfgOptRepoCipherType), cfgOptionStrNull(cfgOptRepoCipherPass)); // Save archive.info/copy so the timestamps will be updated to prevent lifecycle settings from removing the files early // ------------------------------------------------------------------------------------------------------------------------- infoArchiveSaveFile( infoArchiveLoadFile( storageRepo(), INFO_ARCHIVE_PATH_FILE_STR, cfgOptionStrId(cfgOptRepoCipherType), cfgOptionStrNull(cfgOptRepoCipherPass)), storageRepoWrite(), INFO_ARCHIVE_PATH_FILE_STR, cfgOptionStrId(cfgOptRepoCipherType), cfgOptionStrNull(cfgOptRepoCipherPass)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdBackup(void) { FUNCTION_LOG_VOID(logLevelDebug); // Test for stop file lockStopTest(); MEM_CONTEXT_TEMP_BEGIN() { // If the repo option was not provided and more than one repo is configured, then log the default repo chosen if (!cfgOptionTest(cfgOptRepo) && cfgOptionGroupIdxTotal(cfgOptGrpRepo) > 1) { LOG_INFO_FMT( "repo option not specified, defaulting to %s", cfgOptionGroupName(cfgOptGrpRepo, cfgOptionGroupIdxDefault(cfgOptGrpRepo))); } // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepo(); // Load backup.info InfoBackup *const infoBackup = infoBackupLoadFileReconstruct( storageRepo(), INFO_BACKUP_PATH_FILE_STR, cfgOptionStrId(cfgOptRepoCipherType), cfgOptionStrNull(cfgOptRepoCipherPass)); const InfoPgData infoPg = infoPgDataCurrent(infoBackupPg(infoBackup)); const String *const cipherPassBackup = infoPgCipherPass(infoBackupPg(infoBackup)); // Get pg storage and database objects BackupData *const backupData = backupInit(infoBackup); // Get the start timestamp which will later be written into the manifest to track total backup time const time_t timestampStart = backupTime(backupData, false); // Check if there is a prior manifest when backup type is diff/incr Manifest *const manifestPrior = backupBuildIncrPrior(infoBackup); // Start the backup const BackupStartResult backupStartResult = backupStart(backupData); // Build the manifest const ManifestBlockIncrMap blockIncrMap = backupBlockIncrMap(); Manifest *const manifest = manifestNewBuild( backupData->storagePrimary, infoPg.version, infoPg.catalogVersion, timestampStart, cfgOptionBool(cfgOptOnline), cfgOptionBool(cfgOptChecksumPage), cfgOptionBool(cfgOptRepoBundle), cfgOptionBool(cfgOptRepoBlock), &blockIncrMap, strLstNewVarLst(cfgOptionLst(cfgOptExclude)), backupStartResult.tablespaceList); // Validate the manifest using the copy start time manifestBuildValidate( manifest, cfgOptionBool(cfgOptDelta), backupTime(backupData, true), compressTypeEnum(cfgOptionStrId(cfgOptCompressType))); // Build an incremental backup if type is not full (manifestPrior will be freed in this call) if (!backupBuildIncr(infoBackup, manifest, manifestPrior, backupStartResult.walSegmentName)) manifestCipherSubPassSet(manifest, cipherPassGen(cfgOptionStrId(cfgOptRepoCipherType))); // Set delta if it is not already set and the manifest requires it if (!cfgOptionBool(cfgOptDelta) && varBool(manifestData(manifest)->backupOptionDelta)) cfgOptionSet(cfgOptDelta, cfgSourceParam, BOOL_TRUE_VAR); // Resume a backup when possible if (!backupResume(manifest, cipherPassBackup)) { manifestBackupLabelSet( manifest, backupLabelCreate( (BackupType)cfgOptionStrId(cfgOptType), manifestData(manifest)->backupLabelPrior, timestampStart)); } // Save the manifest before processing starts backupManifestSaveCopy(manifest, cipherPassBackup, false); // Process the backup manifest backupProcess(backupData, manifest, cipherPassBackup); // Check that the clusters are alive and correctly configured after the backup backupDbPing(backupData, true); // The standby db object and protocol won't be used anymore so free them if (backupData->dbStandby != NULL) { dbFree(backupData->dbStandby); protocolHelperFree(protocolRemoteGet(protocolStorageTypePg, backupData->pgIdxStandby, false)); } // Stop the backup const BackupStopResult backupStopResult = backupStop(backupData, manifest); // Complete manifest manifestBuildComplete( manifest, backupStartResult.lsn, backupStartResult.walSegmentName, backupStopResult.timestamp, backupStopResult.lsn, backupStopResult.walSegmentName, infoPg.id, infoPg.systemId, backupStartResult.dbList, cfgOptionBool(cfgOptArchiveCheck), cfgOptionBool(cfgOptArchiveCopy), cfgOptionUInt(cfgOptBufferSize), cfgOptionUInt(cfgOptCompressLevel), cfgOptionUInt(cfgOptCompressLevelNetwork), cfgOptionBool(cfgOptRepoHardlink), cfgOptionUInt(cfgOptProcessMax), backupData->dbStandby != NULL, cfgOptionTest(cfgOptAnnotation) ? cfgOptionKv(cfgOptAnnotation) : NULL); // The primary db object won't be used anymore so free it dbFree(backupData->dbPrimary); // Check and copy WAL segments required to make the backup consistent backupArchiveCheckCopy(backupData, manifest, cipherPassBackup); // The primary protocol connection won't be used anymore so free it. This needs to happen after backupArchiveCheckCopy() so // the backup lock is held on the remote which allows conditional archiving based on the backup lock. Any further access to // the primary storage object may result in an error (likely eof). protocolHelperFree(protocolRemoteGet(protocolStorageTypePg, backupData->pgIdxPrimary, false)); // Complete the backup LOG_INFO_FMT("new backup label = %s", strZ(manifestData(manifest)->backupLabel)); backupComplete(infoBackup, manifest); // Backup info LOG_INFO_FMT( "%s backup size = %s, file total = %u", zNewStrId(manifestData(manifest)->backupType), strZ(strSizeFormat(infoBackupDataByLabel(infoBackup, manifestData(manifest)->backupLabel)->backupInfoSizeDelta)), manifestFileTotal(manifest)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/backup.h0000664000175100017510000000124715240547630020507 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Command ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_BACKUP_H #define COMMAND_BACKUP_BACKUP_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Make a backup FN_EXTERN void cmdBackup(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/blockIncr.c0000664000175100017510000004725215240547630021151 0ustar00runnerrunner/*********************************************************************************************************************************** Block Incremental Filter ***********************************************************************************************************************************/ #include #include "command/backup/blockIncr.h" #include "command/backup/blockMap.h" #include "common/compress/helper.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/xxhash.h" #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/bufferWrite.h" #include "common/io/filter/buffer.h" #include "common/io/filter/size.h" #include "common/io/io.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockIncr { unsigned int reference; // Current backup reference uint64_t bundleId; // Bundle id StringId compressType; // Compress filter type const Pack *compressParam; // Compress filter parameters const Pack *encryptParam; // Encrypt filter parameters unsigned int blockNo; // Block number uint64_t superBlockNo; // Block no in super block uint64_t blockOffset; // Block offset uint64_t superBlockSize; // Super block size_t blockSize; // Block size size_t checksumSize; // Checksum size Buffer *block; // Block buffer Buffer *blockOut; // Block output buffer IoWrite *blockOutWrite; // Write to the block block buffer List *blockOutList; // List of block map items that need an updated size size_t blockOutSize; // Amount written to block output (excluding block no) size_t blockOutOffset; // Block output offset (already copied to output buffer) const BlockMap *blockMapPrior; // Prior block map BlockMap *blockMapOut; // Output block map uint64_t blockMapOutSize; // Output block map size (if any) bool blockMapWrite; // Write block map (at least one new/changed block) size_t inputOffset; // Input offset bool inputSame; // Input the same data bool done; // Is the filter done? } BlockIncr; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void blockIncrToLog(const BlockIncr *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{superBlockSize, %" PRIu64 ", blockSize: %zu}", this->superBlockSize, this->blockSize); } #define FUNCTION_LOG_BLOCK_INCR_TYPE \ BlockIncr * #define FUNCTION_LOG_BLOCK_INCR_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, blockIncrToLog, buffer, bufferSize) /*********************************************************************************************************************************** Generate block incremental ***********************************************************************************************************************************/ static void blockIncrProcess(THIS_VOID, const Buffer *const input, Buffer *const output) { THIS(BlockIncr); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_INCR, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_PARAM(BUFFER, output); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(output != NULL); // Is the input done? this->done = input == NULL; // Loop until the input is consumed or there is output do { // If not done and not flushing out then get more block data if (!this->done && this->blockOutOffset == 0) { // If all input can be copied if (bufUsed(input) - this->inputOffset <= bufRemains(this->block)) { bufCatSub(this->block, input, this->inputOffset, bufUsed(input) - this->inputOffset); this->inputOffset = 0; // Same input no longer required this->inputSame = false; } // Else only part of the input can be copied else { const size_t copySize = bufRemains(this->block); bufCatSub(this->block, input, this->inputOffset, copySize); this->inputOffset += copySize; // The same input will be needed again to copy the rest this->inputSame = true; } } // If done with a partial block or block is full if ((this->done && bufUsed(this->block) > 0) || bufUsed(this->block) == this->blockSize) { MEM_CONTEXT_TEMP_BEGIN() { // Get block checksum const Buffer *const checksum = xxHashOne(this->checksumSize, this->block); // Does the block exist in the input map? const BlockMapItem *const blockMapItemIn = this->blockMapPrior != NULL && this->blockNo < blockMapSize(this->blockMapPrior) ? blockMapGet(this->blockMapPrior, this->blockNo) : NULL; // If the block is new or has changed then write it if (blockMapItemIn == NULL || memcmp(blockMapItemIn->checksum, bufPtrConst(checksum), this->checksumSize) != 0) { // Begin the super block if (this->blockOutWrite == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->blockOutWrite = ioBufferWriteNew(this->blockOut); this->blockOutList = lstNewP(sizeof(unsigned int)); } MEM_CONTEXT_OBJ_END(); // Add compress filter if (this->compressParam != NULL) { ioFilterGroupAdd( ioWriteFilterGroup(this->blockOutWrite), compressFilterPack(this->compressType, this->compressParam)); } // Add encrypt filter if (this->encryptParam != NULL) ioFilterGroupAdd(ioWriteFilterGroup(this->blockOutWrite), cipherBlockNewPack(this->encryptParam)); // Add size filter ioFilterGroupAdd(ioWriteFilterGroup(this->blockOutWrite), ioSizeNew()); ioWriteOpen(this->blockOutWrite); } // Copy block data through the filters ioCopyP(ioBufferReadNewOpen(this->block), this->blockOutWrite); this->blockOutSize += bufUsed(this->block); bufUsedZero(this->block); // Write to block map BlockMapItem blockMapItem = { .reference = this->reference, .superBlockSize = this->superBlockSize, .bundleId = this->bundleId, .offset = this->blockOffset, .block = this->superBlockNo, }; memcpy(blockMapItem.checksum, bufPtrConst(checksum), bufUsed(checksum)); const unsigned int blockMapItemIdx = blockMapSize(this->blockMapOut); blockMapAdd(this->blockMapOut, &blockMapItem); lstAdd(this->blockOutList, &blockMapItemIdx); // Increment super block no this->superBlockNo++; } // Else write a reference to the block in the prior backup else { blockMapAdd(this->blockMapOut, blockMapItemIn); bufUsedZero(this->block); } this->blockNo++; } MEM_CONTEXT_TEMP_END(); } // Write the super block if (this->blockOutWrite != NULL && (this->done || this->blockOutSize >= this->superBlockSize)) { // Close write ioWriteClose(this->blockOutWrite); // Update size and super block size for items already added to the block map PackRead *const filter = ioFilterGroupResultP(ioWriteFilterGroup(this->blockOutWrite), SIZE_FILTER_TYPE); const uint64_t blockOutSize = pckReadU64P(filter); for (unsigned int blockMapIdx = 0; blockMapIdx < lstSize(this->blockOutList); blockMapIdx++) { BlockMapItem *const blockMapItem = blockMapGet( this->blockMapOut, *(unsigned int *)lstGet(this->blockOutList, blockMapIdx)); blockMapItem->size = blockOutSize; blockMapItem->superBlockSize = this->blockOutSize; } pckReadFree(filter); lstFree(this->blockOutList); // Set to NULL so the super block can be flushed ioWriteFree(this->blockOutWrite); this->blockOutWrite = NULL; // Increment block offset this->blockOffset += blockOutSize; // Reset block out size and super block no this->blockOutSize = 0; this->superBlockNo = 0; // Block map must be written since there are new/changed blocks this->blockMapWrite = true; } // Write the block map if done processing and there are new/changed blocks or block list has been truncated if (this->done && this->blockOutOffset == 0 && (this->blockMapWrite || (this->blockMapPrior != NULL && blockMapSize(this->blockMapOut) < blockMapSize(this->blockMapPrior)))) { MEM_CONTEXT_TEMP_BEGIN() { // Size of block output before starting to write the map const size_t blockOutBegin = bufUsed(this->blockOut); // Write the map IoWrite *const write = ioBufferWriteNew(this->blockOut); if (this->encryptParam != NULL) ioFilterGroupAdd(ioWriteFilterGroup(write), cipherBlockNewPack(this->encryptParam)); // Write the map ioWriteOpen(write); blockMapWrite(this->blockMapOut, write, this->blockSize, this->checksumSize); ioWriteClose(write); // Get total bytes written for the map this->blockMapOutSize = bufUsed(this->blockOut) - blockOutBegin; } MEM_CONTEXT_TEMP_END(); } // Copy to output buffer if output has been completely written if (this->blockOutWrite == NULL) { const size_t blockOutSize = bufUsed(this->blockOut) - this->blockOutOffset; if (blockOutSize > 0) { // Output the rest of the block if it will fit if (bufRemains(output) >= blockOutSize) { bufCatSub(output, this->blockOut, this->blockOutOffset, blockOutSize); bufUsedZero(this->blockOut); this->blockOutOffset = 0; this->inputSame = this->inputOffset != 0; } // Else output as much of the block as possible else { const size_t blockOutSize = bufRemains(output); bufCatSub(output, this->blockOut, this->blockOutOffset, blockOutSize); this->blockOutOffset += blockOutSize; this->inputSame = true; } } } } while (this->inputSame && bufEmpty(output)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** The result is the size of the block map ***********************************************************************************************************************************/ static Pack * blockIncrResult(THIS_VOID) { THIS(BlockIncr); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_INCR, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, this->blockMapOutSize); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /*********************************************************************************************************************************** Is filter done? ***********************************************************************************************************************************/ static bool blockIncrDone(const THIS_VOID) { THIS(const BlockIncr); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_INCR, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done && !this->inputSame); } /*********************************************************************************************************************************** Should the same input be provided again? ***********************************************************************************************************************************/ static bool blockIncrInputSame(const THIS_VOID) { THIS(const BlockIncr); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_INCR, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * blockIncrNew( const uint64_t superBlockSize, const size_t blockSize, const size_t checksumSize, const unsigned int reference, const uint64_t bundleId, const uint64_t bundleOffset, const Buffer *const blockMapPrior, const IoFilter *const compress, const IoFilter *const encrypt) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(UINT64, superBlockSize); FUNCTION_LOG_PARAM(SIZE, blockSize); FUNCTION_LOG_PARAM(SIZE, checksumSize); FUNCTION_LOG_PARAM(UINT, reference); FUNCTION_LOG_PARAM(UINT64, bundleId); FUNCTION_LOG_PARAM(UINT64, bundleOffset); FUNCTION_LOG_PARAM(BUFFER, blockMapPrior); FUNCTION_LOG_PARAM(IO_FILTER, compress); FUNCTION_LOG_PARAM(IO_FILTER, encrypt); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(BlockIncr, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (BlockIncr) { .superBlockSize = (superBlockSize / blockSize + (superBlockSize % blockSize == 0 ? 0 : 1)) * blockSize, .blockSize = blockSize, .checksumSize = checksumSize, .reference = reference, .bundleId = bundleId, .blockOffset = bundleOffset, .block = bufNew(blockSize), .blockOut = bufNew(0), .blockMapOut = blockMapNew(), }; // Duplicate compress filter if (compress != NULL) { this->compressType = ioFilterType(compress); this->compressParam = pckDup(ioFilterParamList(compress)); } // Duplicate encrypt filter if (encrypt != NULL) this->encryptParam = pckDup(ioFilterParamList(encrypt)); // Load prior block map if (blockMapPrior) { MEM_CONTEXT_TEMP_BEGIN() { IoRead *const read = ioBufferReadNewOpen(blockMapPrior); MEM_CONTEXT_PRIOR_BEGIN() { this->blockMapPrior = blockMapNewRead(read, blockSize, checksumSize); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, this->superBlockSize); pckWriteU64P(packWrite, blockSize); pckWriteU64P(packWrite, checksumSize); pckWriteU32P(packWrite, reference); pckWriteU64P(packWrite, bundleId); pckWriteU64P(packWrite, bundleOffset); pckWriteBinP(packWrite, blockMapPrior); pckWritePackP(packWrite, this->compressParam); if (this->compressParam != NULL) pckWriteStrIdP(packWrite, this->compressType); pckWritePackP(packWrite, this->encryptParam); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( BLOCK_INCR_FILTER_TYPE, this, paramList, .done = blockIncrDone, .inOut = blockIncrProcess, .inputSame = blockIncrInputSame, .result = blockIncrResult)); } FN_EXTERN IoFilter * blockIncrNewPack(const Pack *const paramList) { IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const paramListPack = pckReadNew(paramList); const uint64_t superBlockSize = pckReadU64P(paramListPack); const size_t blockSize = (size_t)pckReadU64P(paramListPack); const size_t checksumSize = (size_t)pckReadU64P(paramListPack); const unsigned int reference = pckReadU32P(paramListPack); const uint64_t bundleId = pckReadU64P(paramListPack); const uint64_t bundleOffset = pckReadU64P(paramListPack); const Buffer *blockMapPrior = pckReadBinP(paramListPack); // Create compress filter const Pack *const compressParam = pckReadPackP(paramListPack); const IoFilter *compress = NULL; if (compressParam != NULL) compress = compressFilterPack(pckReadStrIdP(paramListPack), compressParam); // Create encrypt filter const Pack *const encryptParam = pckReadPackP(paramListPack); const IoFilter *encrypt = NULL; if (encryptParam != NULL) encrypt = cipherBlockNewPack(encryptParam); result = ioFilterMove( blockIncrNew( superBlockSize, blockSize, checksumSize, reference, bundleId, bundleOffset, blockMapPrior, compress, encrypt), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/blockIncr.h0000664000175100017510000000701015240547630021142 0ustar00runnerrunner/*********************************************************************************************************************************** Block Incremental Filter The block incremental filter builds a super block list and map (see BlockMap object) either from a file (the first time) or from a file and a prior map (each subsequent time). The super block list is stored first and the block map second. The filter returns the size of the block map so it can be extracted separately. Super blocks contain a list of blocks that may or may not be contiguous. The advantage is that multiple blocks can be compressed together which leads to better compression. The disadvantage is that the super block must be read sequentially to get at any block in the super block. For large block sizes the block size will usually equal the super block size to negate the read penalty. The super block list is followed by the block map, which is encrypted separately when required but not compressed. The return value of the filter is the stored block map size. Combined with the repo size this allows the block map to be read separately. The map is duplicated in each backup where the file has changed so the block map only needs to be retrieved from the most recent backup. However, the block map may reference super block lists (or parts thereof) in any prior (or the current) backup. The block incremental should be read using BlockDelta since reconstructing the delta is quite involved. The xxHash algorithm is used to determine which blocks have changed. A 128-bit xxHash is generated and then checksumSize bytes are used from the hash depending on the size of the block. xxHash claims to have excellent dispersion characteristics, which has been verified by testing with SMHasher and a custom test suite. xxHash-32 is used for up to 4MiB content blocks in lz4 and the lower 32-bits of xxHash-64 is used for content blocks in zstd. In general 32-bit checksums are pretty common defaults across filesystems such as BtrFS and ZFS. We use at least 5 bytes even for the smallest blocks since we are looking for changes and not just corruption. Ultimately if there is a collision and a block change is not detected it will almost certainly be caught by the overall SHA1 file checksum. This will fail the backup, which is not ideal, but better than restoring corrupted data. ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_BLOCK_INCR_H #define COMMAND_BACKUP_BLOCK_INCR_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BLOCK_INCR_FILTER_TYPE STRID5("blk-incr", 0x90dc9dad820) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *blockIncrNew( uint64_t superBlockSize, size_t blockSize, size_t checksumSize, unsigned int reference, uint64_t bundleId, uint64_t bundleOffset, const Buffer *blockMapPrior, const IoFilter *compress, const IoFilter *encrypt); FN_EXTERN IoFilter *blockIncrNewPack(const Pack *paramList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/blockMap.c0000664000175100017510000005241415240547630020767 0ustar00runnerrunner/*********************************************************************************************************************************** Block Incremental Map The block map is stored as a flag and a series of reference, super block, and block info: - Varint-128 flag that contains the version and info about the map (e.g. are the super blocks and blocks equal size). - List of references: - Varint-128 encoded reference (which is an index into the reference list maintained in the manifest). If this is the first time the reference appears it will be followed by a bundle id and an offset if they are not 0. If the reference has appeared before it might update the offset or be a continuation of a prior super block. Continuations happen when a super block is split by a newer super block. The continuation allows the prior super block values for the reference to be used without encoding them again. - List of super blocks: - Varint-128 encoded super block size. The very first size in the map will be encoded directly and subsequent sizes will be encoded as the delta from the last size. - List of blocks: - Varint-128 encoded block number if super block size does not equal block size. If they are equal then there is one block per super block so no reason to encode the block number. - Checksum. References, super blocks, and blocks are encoded with a bit that indicates when the last one has been reached. ***********************************************************************************************************************************/ #include #include "command/backup/blockMap.h" #include "common/debug.h" #include "common/log.h" /**********************************************************************************************************************************/ #define BLOCK_MAP_FLAG_LAST 1 // Last reference, super block, etc. #define BLOCK_MAP_FLAG_OFFSET 4 // Reference has an offset #define BLOCK_MAP_FLAG_BUNDLE_ID 2 // Reference has a bundle id #define BLOCK_MAP_FLAG_CONTINUE 2 // Reference continues a prior super block #define BLOCK_MAP_FLAG_CONTINUE_LAST 4 // Continued super block is last for the reference #define BLOCK_MAP_REFERENCE_SHIFT 3 // Shift bits for reference #define BLOCK_MAP_FLAG_SUPER_BLOCK_SIZE_REMAINDER 1 // Remainder for super block size #define BLOCK_MAP_SUPER_BLOCK_SIZE_SHIFT 1 // Shift bits for super block size #define BLOCK_MAP_FLAG_SUPER_BLOCK_CHANGE 2 // The super block size has changed #define BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET 4 // Block total/offset when not complete #define BLOCK_MAP_SUPER_BLOCK_SHIFT 3 // Shift bits for super block #define BLOCK_MAP_FLAG_BLOCK_TOTAL_OFFSET 1 // Block total has an offset #define BLOCK_MAP_BLOCK_TOTAL_SHIFT 1 // Shift bits for block total typedef enum { blockMapFlagVersion = 0, // Version (currently always 0) } BlockMapFlag; // Stores current information about a reference to avoid needed to encode it again typedef struct BlockMapReference { unsigned int reference; // Reference uint64_t superBlockSize; // Super block size uint64_t bundleId; // Bundle id uint64_t offset; // Offset uint64_t size; // Stored super block size (with compression, etc.) uint64_t block; // Block no } BlockMapReference; // Reference comparator static int lstComparatorBlockMapReference(const void *const blockMapRef1, const void *const blockMapRef2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, blockMapRef1); FUNCTION_TEST_PARAM_P(VOID, blockMapRef2); FUNCTION_TEST_END(); ASSERT(blockMapRef1 != NULL); ASSERT(blockMapRef2 != NULL); const unsigned int reference1 = ((const BlockMapReference *)blockMapRef1)->reference; const unsigned int reference2 = ((const BlockMapReference *)blockMapRef2)->reference; FUNCTION_TEST_RETURN(INT, LST_COMPARATOR_CMP(reference1, reference2)); } FN_EXTERN BlockMap * blockMapNewRead(IoRead *const map, const size_t blockSize, const size_t checksumSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, map); FUNCTION_LOG_END(); // Read flags. Currently the version flag must always be zero. This may be used in the future to indicate if the map version // has changed. CHECK(FormatError, (ioReadVarIntU64(map) & (1 << blockMapFlagVersion)) == 0, "block map version must be zero"); // Read all references in packed format BlockMap *const this = blockMapNew(); List *const refList = lstNewP(sizeof(BlockMapReference), .comparator = lstComparatorBlockMapReference); Buffer *const checksum = bufNew(checksumSize); int64_t sizeLast = 0; bool referenceContinue = false; do { // Read reference const uint64_t referenceEncoded = ioReadVarIntU64(map); BlockMapItem blockMapItem = {.reference = (unsigned int)(referenceEncoded >> BLOCK_MAP_REFERENCE_SHIFT)}; BlockMapReference *referenceData = lstFind(refList, &(BlockMapReference){.reference = blockMapItem.reference}); // If this is the first time this reference has been read if (referenceData == NULL) { // Read bundle id if (referenceEncoded & BLOCK_MAP_FLAG_BUNDLE_ID) blockMapItem.bundleId = ioReadVarIntU64(map); // Read offset if (referenceEncoded & BLOCK_MAP_FLAG_OFFSET) blockMapItem.offset = ioReadVarIntU64(map); // Default super block size blockMapItem.superBlockSize = blockSize; // Add reference to list BlockMapReference referenceDataAdd = { .reference = blockMapItem.reference, .superBlockSize = blockMapItem.superBlockSize, .bundleId = blockMapItem.bundleId, .offset = blockMapItem.offset, }; referenceData = lstAdd(refList, &referenceDataAdd); } // Else this reference has been read before else { blockMapItem.superBlockSize = referenceData->superBlockSize; blockMapItem.bundleId = referenceData->bundleId; // If the reference is continued use the prior offset and size values if (referenceEncoded & BLOCK_MAP_FLAG_CONTINUE) { blockMapItem.offset = referenceData->offset; blockMapItem.size = referenceData->size; referenceContinue = true; } // Else this is a new reference and super block with a possible offset update else { blockMapItem.offset = referenceData->offset + referenceData->size; if (referenceEncoded & BLOCK_MAP_FLAG_OFFSET) blockMapItem.offset += ioReadVarIntU64(map); referenceData->offset = blockMapItem.offset; } } // Read all super blocks in the current reference in packed format bool superBlockFirst = true; do { uint64_t superBlockEncoded = 0; // If the reference was continued check if this is the last super block in the reference if (referenceContinue) { if (referenceEncoded & BLOCK_MAP_FLAG_CONTINUE_LAST) superBlockEncoded = BLOCK_MAP_FLAG_LAST; superBlockEncoded |= BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET; referenceContinue = false; } // Else read the super block size for the reference else { superBlockEncoded = ioReadVarIntU64(map); // If this is the first size read then just read the size. Otherwise read the difference from the prior size and // add sizeLast. blockMapItem.size = superBlockEncoded >> BLOCK_MAP_SUPER_BLOCK_SHIFT; if (sizeLast != 0) blockMapItem.size = (uint64_t)(cvtInt64FromZigZag(blockMapItem.size) + sizeLast); // If the super block size has changed then read it if (superBlockEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_CHANGE) { const uint64_t superBlockSizeEncoded = ioReadVarIntU64(map); blockMapItem.superBlockSize = (superBlockSizeEncoded >> BLOCK_MAP_SUPER_BLOCK_SIZE_SHIFT) * blockSize; if (superBlockSizeEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_SIZE_REMAINDER) blockMapItem.superBlockSize += ioReadVarIntU64(map); referenceData->superBlockSize = blockMapItem.superBlockSize; } // Set offset, size, and block for the super block if (superBlockFirst) referenceData->offset = blockMapItem.offset; else referenceData->offset += (uint64_t)sizeLast; referenceData->size = blockMapItem.size; referenceData->block = 0; } // Update sizeLast with the current size and clear superBlockFirst sizeLast = (int64_t)blockMapItem.size; superBlockFirst = false; // Read or calculate block total uint64_t blockTotal; if (superBlockEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET) { const uint64_t blockTotalEncoded = ioReadVarIntU64(map); blockTotal = (blockTotalEncoded >> BLOCK_MAP_BLOCK_TOTAL_SHIFT) + 1; // Offset block no from expected if (blockTotalEncoded & BLOCK_MAP_FLAG_BLOCK_TOTAL_OFFSET) referenceData->block += ioReadVarIntU64(map); } else blockTotal = blockMapItem.superBlockSize / blockSize + (blockMapItem.superBlockSize % blockSize == 0 ? 0 : 1); // Read checksums for (uint64_t blockIdx = 0; blockIdx < blockTotal; blockIdx++) { // Set block no blockMapItem.block = referenceData->block + blockIdx; // Read checksum bufUsedZero(checksum); ioRead(map, checksum); memcpy(blockMapItem.checksum, bufPtr(checksum), bufUsed(checksum)); // Add to block list lstAdd((List *)this, &blockMapItem); } // Update block in reference with all blocks read referenceData->block += blockTotal; // Update offset with the super block size blockMapItem.offset += blockMapItem.size; // Break when this is the last super block in the reference if (superBlockEncoded & BLOCK_MAP_FLAG_LAST) break; } while (true); // Break when this is the last reference if (referenceEncoded & BLOCK_MAP_FLAG_LAST) break; } while (true); lstFree(refList); bufFree(checksum); FUNCTION_LOG_RETURN(BLOCK_MAP, this); } /**********************************************************************************************************************************/ FN_EXTERN void blockMapWrite(const BlockMap *const this, IoWrite *const output, const size_t blockSize, const size_t checksumSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_MAP, this); FUNCTION_LOG_PARAM(IO_WRITE, output); FUNCTION_LOG_PARAM(SIZE, blockSize); FUNCTION_LOG_PARAM(SIZE, checksumSize); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(blockMapSize(this) > 0); ASSERT(blockSize > 0); ASSERT(output != NULL); // Write flags ioWriteVarIntU64(output, 0); // Write all references in packed format List *const refList = lstNewP(sizeof(BlockMapReference), .comparator = lstComparatorBlockMapReference); unsigned int referenceIdx = 0; int64_t sizeLast = 0; bool referenceContinue = false; while (referenceIdx < blockMapSize(this)) { const BlockMapItem *const reference = blockMapGet(this, referenceIdx); unsigned int superBlockIdx = referenceIdx; unsigned int blockIdx = referenceIdx; // Determine if this is the last reference uint64_t referenceEncoded = BLOCK_MAP_FLAG_LAST; for (referenceIdx++; referenceIdx < blockMapSize(this); referenceIdx++) { if (reference->reference != blockMapGet(this, referenceIdx)->reference) { referenceEncoded = 0; break; } ASSERT(reference->offset <= blockMapGet(this, referenceIdx)->offset); } // If this is the first time this reference has been written BlockMapReference *referenceData = lstFind(refList, &(BlockMapReference){.reference = reference->reference}); if (referenceData == NULL) { // Add bundle id and offset flags if (reference->bundleId > 0) referenceEncoded |= BLOCK_MAP_FLAG_BUNDLE_ID; if (reference->offset > 0) referenceEncoded |= BLOCK_MAP_FLAG_OFFSET; // Write the references ioWriteVarIntU64(output, referenceEncoded | reference->reference << BLOCK_MAP_REFERENCE_SHIFT); // Write bundle id and offset if (referenceEncoded & BLOCK_MAP_FLAG_BUNDLE_ID) ioWriteVarIntU64(output, reference->bundleId); if (referenceEncoded & BLOCK_MAP_FLAG_OFFSET) ioWriteVarIntU64(output, reference->offset); // Add reference to list const BlockMapReference referenceAdd = { .reference = reference->reference, .superBlockSize = blockSize, .bundleId = reference->bundleId, .offset = reference->offset, }; referenceData = lstAdd(refList, &referenceAdd); } // Else this reference has been written before else { ASSERT(reference->reference == referenceData->reference); ASSERT(reference->bundleId == referenceData->bundleId); ASSERT(reference->offset >= referenceData->offset); // If the offset is identical then reference is continuing an already started super block. The super block size and // block no should be reused. Note that writing the reference is deferred until we know if the continued super block is // the last one for the reference. if (reference->offset == referenceData->offset) { ASSERT(reference->superBlockSize == referenceData->superBlockSize); referenceEncoded |= BLOCK_MAP_FLAG_CONTINUE; referenceContinue = true; } // Else the reference starts a new super block which means the offset may need to be stored if there is a gap from the // prior super block else { if (reference->offset > referenceData->offset + referenceData->size) referenceEncoded |= BLOCK_MAP_FLAG_OFFSET; ioWriteVarIntU64(output, referenceEncoded | reference->reference << BLOCK_MAP_REFERENCE_SHIFT); if (referenceEncoded & BLOCK_MAP_FLAG_OFFSET) ioWriteVarIntU64(output, reference->offset - (referenceData->offset + referenceData->size)); referenceData->offset = reference->offset; referenceData->size = reference->size; } } // Write all super blocks in the current reference in packed format while (superBlockIdx < referenceIdx) { const BlockMapItem *const superBlock = blockMapGet(this, superBlockIdx); // Determine if this is the last super block in the reference uint64_t superBlockEncoded = BLOCK_MAP_FLAG_LAST; for (superBlockIdx++; superBlockIdx < referenceIdx; superBlockIdx++) { if (superBlock->offset != blockMapGet(this, superBlockIdx)->offset) { superBlockEncoded = 0; break; } } // If block offset or total need to be stored then add the flag const unsigned int blockTotal = superBlockIdx - blockIdx; ASSERT(blockTotal > 0); if (referenceContinue || superBlock->block != 0 || blockTotal != superBlock->superBlockSize / blockSize + (superBlock->superBlockSize % blockSize == 0 ? 0 : 1)) { superBlockEncoded |= BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET; } // Write the continued reference now that we know if this will be the last super block in the reference if (referenceContinue) { ASSERT(superBlock->superBlockSize == referenceData->superBlockSize); if (superBlockEncoded & BLOCK_MAP_FLAG_LAST) referenceEncoded |= BLOCK_MAP_FLAG_CONTINUE_LAST; ioWriteVarIntU64(output, referenceEncoded | reference->reference << BLOCK_MAP_REFERENCE_SHIFT); referenceContinue = false; } // Else write the super block size for the reference else { // Set offset, size, and block for the super block referenceData->offset = superBlock->offset; referenceData->size = superBlock->size; referenceData->block = 0; // If the super block size has changed then add the flag ASSERT(reference->superBlockSize > 0); if (superBlock->superBlockSize != referenceData->superBlockSize) superBlockEncoded |= BLOCK_MAP_FLAG_SUPER_BLOCK_CHANGE; // If this is the first size written then just write the size. Otherwise write the difference from the prior size. // This depends on the expectation that the compressed size of equal-sized blocks will be similar in order to be // most efficient. ioWriteVarIntU64( output, superBlockEncoded | (sizeLast == 0 ? superBlock->size : cvtInt64ToZigZag((int64_t)superBlock->size - sizeLast)) << BLOCK_MAP_SUPER_BLOCK_SHIFT); // If the super block size has changed then write it if (superBlockEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_CHANGE) { const uint64_t superBlockSizeEncoded = (superBlock->superBlockSize / blockSize) << BLOCK_MAP_SUPER_BLOCK_SIZE_SHIFT | (superBlock->superBlockSize % blockSize == 0 ? 0 : BLOCK_MAP_FLAG_SUPER_BLOCK_SIZE_REMAINDER); ioWriteVarIntU64(output, superBlockSizeEncoded); if (superBlockSizeEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_SIZE_REMAINDER) ioWriteVarIntU64(output, superBlock->superBlockSize % blockSize); referenceData->superBlockSize = superBlock->superBlockSize; } } sizeLast = (int64_t)superBlock->size; // Write block total if the super block does not include all blocks with no block offset if (superBlockEncoded & BLOCK_MAP_FLAG_SUPER_BLOCK_TOTAL_OFFSET) { // Write total blocks in the super block const uint64_t blockTotalEncoded = (blockTotal - 1) << BLOCK_MAP_BLOCK_TOTAL_SHIFT | (superBlock->block - referenceData->block > 0 ? BLOCK_MAP_FLAG_BLOCK_TOTAL_OFFSET : 0); ioWriteVarIntU64(output, blockTotalEncoded); // If there is a gap in block no from the prior super block. This can happen when a super block is continued or // had blocks at the beginning overridden by a newer super block. if (blockTotalEncoded & BLOCK_MAP_FLAG_BLOCK_TOTAL_OFFSET) { ioWriteVarIntU64(output, superBlock->block - referenceData->block); referenceData->block = superBlock->block; } } ASSERT(superBlock->block >= referenceData->block); // Increment reference block by number of blocks written referenceData->block += blockTotal; // Write checksums for (; blockIdx < superBlockIdx; blockIdx++) { ASSERT( superBlock == blockMapGet(this, blockIdx) || (blockIdx > 0 && (blockMapGet(this, blockIdx)->block == blockMapGet(this, blockIdx - 1)->block + 1))); ioWrite(output, BUF(blockMapGet(this, blockIdx)->checksum, checksumSize)); } } } lstFree(refList); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/blockMap.h0000664000175100017510000001120615240547630020766 0ustar00runnerrunner/*********************************************************************************************************************************** Block Incremental Map The block incremental map stores the location of blocks of data that have been backed up incrementally. When a file changes, instead of copying the entire file, just the blocks that have been changed can be stored. This map does not store the blocks themselves, just the location where they can be found. It must be combined with a super block list to be useful (see BlockIncr filter). ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_BLOCKMAP_H #define COMMAND_BACKUP_BLOCKMAP_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockMap BlockMap; #include "common/crypto/xxhash.h" #include "common/type/list.h" #include "common/type/object.h" typedef struct BlockMapItem { unsigned int reference; // Reference to backup where the block is stored uint64_t superBlockSize; // Super block size uint64_t bundleId; // Bundle where the block is stored (0 if not bundled) uint64_t offset; // Offset of super block into the bundle uint64_t size; // Stored super block size (with compression, etc.) uint64_t block; // Block no inside of super block uint8_t checksum[XX_HASH_SIZE_MAX]; // Checksum of the block } BlockMapItem; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Create empty block map FN_INLINE_ALWAYS BlockMap * blockMapNew(void) { return (BlockMap *)OBJ_NAME(lstNewP(sizeof(BlockMapItem)), BlockMap::List); } // New block map from IO FN_EXTERN BlockMap *blockMapNewRead(IoRead *map, size_t blockSize, size_t checksumSize); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a block map item FN_INLINE_ALWAYS BlockMapItem * blockMapAdd(BlockMap *const this, const BlockMapItem *const item) { ASSERT_INLINE(item != NULL); return (BlockMapItem *)lstAdd((List *const)this, item); } // Write map to IO FN_EXTERN void blockMapWrite(const BlockMap *this, IoWrite *output, size_t blockSize, size_t checksumSize); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Get a block map item FN_INLINE_ALWAYS BlockMapItem * blockMapGet(const BlockMap *const this, const unsigned int mapIdx) { return (BlockMapItem *)lstGet((const List *const)this, mapIdx); } // Block map size FN_INLINE_ALWAYS unsigned int blockMapSize(const BlockMap *const this) { return lstSize((const List *const)this); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void blockMapFree(BlockMap *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_BLOCK_MAP_TYPE \ BlockMap * #define FUNCTION_LOG_BLOCK_MAP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "BlockMap", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/common.c0000664000175100017510000001441115240547630020522 0ustar00runnerrunner/*********************************************************************************************************************************** Common Functions and Definitions for Backup and Expire Commands ***********************************************************************************************************************************/ #include #include #include "command/backup/common.h" #include "common/debug.h" #include "common/log.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define BACKUP_LINK_LATEST "latest" /**********************************************************************************************************************************/ FN_EXTERN String * backupFileRepoPath(const String *const backupLabel, const BackupFileRepoPathParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(STRING, param.manifestName); FUNCTION_TEST_PARAM(UINT64, param.bundleId); FUNCTION_TEST_PARAM(ENUM, param.compressType); FUNCTION_TEST_PARAM(BOOL, param.blockIncr); FUNCTION_TEST_END(); ASSERT(backupLabel != NULL); ASSERT(param.bundleId != 0 || param.manifestName != NULL); String *const result = strCatFmt(strNew(), STORAGE_REPO_BACKUP "/%s/", strZ(backupLabel)); if (param.bundleId != 0) strCatFmt(result, MANIFEST_PATH_BUNDLE "/%" PRIu64, param.bundleId); else { strCatFmt( result, "%s%s", strZ(param.manifestName), param.blockIncr ? BACKUP_BLOCK_INCR_EXT : strZ(compressExtStr(param.compressType))); } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * backupLabelFormat(const BackupType type, const String *const backupLabelPrior, const time_t timestamp) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, backupLabelPrior); FUNCTION_LOG_PARAM(TIME, timestamp); FUNCTION_LOG_END(); ASSERT((type == backupTypeFull && backupLabelPrior == NULL) || (type != backupTypeFull && backupLabelPrior != NULL)); ASSERT(timestamp > 0); // Format the timestamp char buffer[DATE_TIME_LEN + 1]; cvtTimeToZP("%Y%m%d-%H%M%S", timestamp, buffer, sizeof(buffer)); // If full label String *result; if (type == backupTypeFull) { result = strNewFmt("%sF", buffer); } // Else diff or incr label else { // Get the full backup portion of the prior backup label and append the diff/incr timestamp result = strNewFmt("%.*s_%s%s", DATE_TIME_LEN + 1, strZ(backupLabelPrior), buffer, type == backupTypeDiff ? "D" : "I"); } FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * backupRegExp(const BackupRegExpParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BOOL, param.full); FUNCTION_LOG_PARAM(BOOL, param.differential); FUNCTION_LOG_PARAM(BOOL, param.incremental); FUNCTION_LOG_PARAM(BOOL, param.noAnchorEnd); FUNCTION_LOG_END(); ASSERT(param.full || param.differential || param.incremental); // Start the expression with the anchor, date/time regexp and full backup indicator String *const result = strCatZ(strNew(), "^" DATE_TIME_REGEX "F"); // Add the diff and/or incr expressions if requested if (param.differential || param.incremental) { // If full requested then diff/incr is optional if (param.full) { strCatZ(result, "(\\_"); } // Else diff/incr is required else { strCatZ(result, "\\_"); } // Append date/time regexp for diff/incr strCatZ(result, DATE_TIME_REGEX); // Filter on both diff/incr if (param.differential && param.incremental) { strCatZ(result, "(D|I)"); } // Else just diff else if (param.differential) { strCatChr(result, 'D'); } // Else just incr else { strCatChr(result, 'I'); } // If full requested then diff/incr is optional if (param.full) { strCatZ(result, "){0,1}"); } } // Append the end anchor if (!param.noAnchorEnd) strCatZ(result, "$"); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void backupLinkLatest(const String *const backupLabel, const unsigned int repoIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // Create a symlink to the most recent backup if supported. This link is purely informational for the user and is never used // by us since symlinks are not supported on all storage types. // ------------------------------------------------------------------------------------------------------------------------- const String *const latestLink = storagePathP(storageRepoIdx(repoIdx), STRDEF(STORAGE_REPO_BACKUP "/" BACKUP_LINK_LATEST)); // Remove an existing latest link/file in case symlink capabilities have changed storageRemoveP(storageRepoIdxWrite(repoIdx), latestLink); if (storageFeature(storageRepoIdxWrite(repoIdx), storageFeatureSymLink)) storageLinkCreateP(storageRepoIdxWrite(repoIdx), backupLabel, latestLink); // Sync backup path if required if (storageFeature(storageRepoIdxWrite(repoIdx), storageFeaturePathSync)) storagePathSyncP(storageRepoIdxWrite(repoIdx), STORAGE_REPO_BACKUP_STR); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/common.h0000664000175100017510000000615615240547630020536 0ustar00runnerrunner/*********************************************************************************************************************************** Common Functions and Definitions for Backup and Expire Commands ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_COMMON_H #define COMMAND_BACKUP_COMMON_H #include #include #include "common/compress/helper.h" #include "common/type/string.h" #include "info/infoBackup.h" /*********************************************************************************************************************************** Backup constants ***********************************************************************************************************************************/ #define BACKUP_PATH_HISTORY "backup.history" #define BACKUP_BLOCK_INCR_EXT ".pgbi" // Date and time must be in %Y%m%d-%H%M%S format, for example 20220901-193409 #define DATE_TIME_REGEX "[0-9]{8}\\-[0-9]{6}" #define DATE_TIME_LEN (8 + 1 + 6) /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Determine the path/file where the file is backed up in the repo typedef struct BackupFileRepoPathParam { const String *manifestName; // File name in manifest uint64_t bundleId; // Is the file bundled? CompressType compressType; // Is the file compressed? bool blockIncr; // Is the file a block incremental? } BackupFileRepoPathParam; #define backupFileRepoPathP(backupLabel, ...) \ backupFileRepoPath(backupLabel, (BackupFileRepoPathParam){__VA_ARGS__}) FN_EXTERN String *backupFileRepoPath(const String *backupLabel, BackupFileRepoPathParam param); // Format a backup label from a type and timestamp with an optional prior label FN_EXTERN String *backupLabelFormat(BackupType type, const String *backupLabelPrior, time_t timestamp); // Returns an anchored regex string for filtering backups based on the type (at least one type is required to be true) typedef struct BackupRegExpParam { bool full; bool differential; bool incremental; bool noAnchorEnd; } BackupRegExpParam; #define backupRegExpP(...) \ backupRegExp((BackupRegExpParam){__VA_ARGS__}) FN_EXTERN String *backupRegExp(BackupRegExpParam param); // Create a symlink to the specified backup (if symlinks are supported) FN_EXTERN void backupLinkLatest(const String *backupLabel, unsigned int repoIdx); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/file.c0000664000175100017510000005646015240547630020163 0ustar00runnerrunner/*********************************************************************************************************************************** Backup File ***********************************************************************************************************************************/ #include #include #include "command/backup/blockIncr.h" #include "command/backup/file.h" #include "command/backup/pageChecksum.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/filter/group.h" #include "common/io/filter/size.h" #include "common/io/io.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/convert.h" #include "common/type/json.h" #include "storage/helper.h" /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ static unsigned int segmentNumber(const String *const pgFile) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, pgFile); FUNCTION_TEST_END(); // Determine which segment number this is by checking for a numeric extension. No extension means segment 0. FUNCTION_TEST_RETURN(UINT, regExpMatchOne(STRDEF("\\.[0-9]+$"), pgFile) ? cvtZToUInt(strrchr(strZ(pgFile), '.') + 1) : 0); } /**********************************************************************************************************************************/ FN_EXTERN List * backupFile( const String *const repoFile, const uint64_t bundleId, const bool bundleRaw, const unsigned int blockIncrReference, const CompressType repoFileCompressType, const int repoFileCompressLevel, const CipherType cipherType, const String *const cipherPass, const String *const pgVersionForce, const PgPageSize pageSize, const List *const fileList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, repoFile); // Repo file FUNCTION_LOG_PARAM(UINT64, bundleId); // Bundle id (0 if none) FUNCTION_LOG_PARAM(BOOL, bundleRaw); // Raw compress/encrypt format in bundles? FUNCTION_LOG_PARAM(UINT, blockIncrReference); // Block incremental reference to use in map FUNCTION_LOG_PARAM(ENUM, repoFileCompressType); // Compress type for repo file FUNCTION_LOG_PARAM(INT, repoFileCompressLevel); // Compression level for repo file FUNCTION_LOG_PARAM(STRING_ID, cipherType); // Encryption type FUNCTION_TEST_PARAM(STRING, cipherPass); // Password to access the repo file if encrypted FUNCTION_LOG_PARAM(ENUM, pageSize); // Page size FUNCTION_LOG_PARAM(STRING, pgVersionForce); // Force pg version FUNCTION_LOG_PARAM(LIST, fileList); // List of files to backup FUNCTION_LOG_END(); ASSERT(repoFile != NULL); ASSERT((cipherType == cipherTypeNone && cipherPass == NULL) || (cipherType != cipherTypeNone && cipherPass != NULL)); ASSERT(fileList != NULL && !lstEmpty(fileList)); ASSERT(pgPageSizeValid(pageSize)); // Backup file results List *const result = lstNewP(sizeof(BackupFileResult)); MEM_CONTEXT_TEMP_BEGIN() { // Check files to determine which ones need to be copied for (unsigned int fileIdx = 0; fileIdx < lstSize(fileList); fileIdx++) { // Use a per-file mem context to reduce memory usage MEM_CONTEXT_TEMP_BEGIN() { const BackupFile *const file = lstGet(fileList, fileIdx); ASSERT(file->pgFile != NULL); ASSERT(file->manifestFile != NULL); ASSERT((!file->pgFileDelta && !file->manifestFileResume) || file->pgFileChecksum != NULL); BackupFileResult *const fileResult = lstAdd( result, &(BackupFileResult){.manifestFile = file->manifestFile, .backupCopyResult = backupCopyResultCopy}); // Does the file in pg match the checksum and size passed? bool pgFileMatch = false; // If delta then check the pg checksum if (file->pgFileDelta) { // Generate checksum/size for the pg file. Only read as many bytes as passed in pgFileSize. If the file has // grown since the manifest was built we don't need to consider the extra bytes since they will be replayed from // WAL during recovery. IoRead *const read = storageReadIo( storageNewReadP( storagePg(), file->pgFile, .ignoreMissing = file->pgFileIgnoreMissing, .limit = file->pgFileCopyExactSize ? VARUINT64(file->pgFileSizeOriginal) : NULL)); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioFilterGroupAdd(ioReadFilterGroup(read), ioSizeNew()); // If the pg file exists check the checksum/size if (ioReadDrain(read)) { const Buffer *const pgTestChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE)); const uint64_t pgTestSize = pckReadU64P(ioFilterGroupResultP(ioReadFilterGroup(read), SIZE_FILTER_TYPE)); // Does the pg file match? if (file->pgFileSize == pgTestSize && bufEq(file->pgFileChecksum, pgTestChecksum)) { pgFileMatch = true; // If it matches and is a reference to a previous backup then no need to copy the file if (file->manifestFileHasReference) { MEM_CONTEXT_BEGIN(lstMemContext(result)) { fileResult->backupCopyResult = backupCopyResultNoOp; fileResult->copySize = file->pgFileSize; fileResult->copyChecksum = file->pgFileChecksum; } MEM_CONTEXT_END(); } } } // Else the source file is missing from the database so skip this file else fileResult->backupCopyResult = backupCopyResultSkip; } // On resume check the manifest file if (file->manifestFileResume) { // Resumed files should never have a reference to a prior backup ASSERT(!file->manifestFileHasReference); // If the file is missing from pg, then remove it from the repo (backupJobResult() will remove it from the // manifest) if (fileResult->backupCopyResult == backupCopyResultSkip) { storageRemoveP(storageRepoWrite(), repoFile); } // Else if the pg file matches or is unknown because delta was not performed then check the repo file else if (!file->pgFileDelta || pgFileMatch) { // Generate checksum/size for the repo file IoRead *const read = storageReadIo(storageNewReadP(storageRepo(), repoFile)); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioFilterGroupAdd(ioReadFilterGroup(read), ioSizeNew()); ioReadDrain(read); // Test checksum/size const Buffer *const pgTestChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE)); const uint64_t pgTestSize = pckReadU64P(ioFilterGroupResultP(ioReadFilterGroup(read), SIZE_FILTER_TYPE)); // No need to recopy if checksum/size match. When the repo checksum is missing still compare to repo size // since the repo checksum should only be missing when the repo file was not compressed/encrypted, i.e. the // repo size should match the original size. There is no need to worry about old manifests here since resume // does not work across versions. if (file->repoFileSize == pgTestSize && bufEq(file->repoFileChecksum != NULL ? file->repoFileChecksum : file->pgFileChecksum, pgTestChecksum)) { MEM_CONTEXT_BEGIN(lstMemContext(result)) { fileResult->backupCopyResult = backupCopyResultChecksum; fileResult->copySize = file->pgFileSize; fileResult->copyChecksum = file->pgFileChecksum; } MEM_CONTEXT_END(); } // Else copy when repo file is invalid else fileResult->repoInvalid = true; } } } MEM_CONTEXT_TEMP_END(); } // Are the files compressible during the copy? const bool compressible = repoFileCompressType == compressTypeNone && cipherType == cipherTypeNone; // Copy files that need to be copied StorageWrite *write = NULL; uint64_t bundleOffset = 0; for (unsigned int fileIdx = 0; fileIdx < lstSize(fileList); fileIdx++) { // Use a per-file mem context to reduce memory usage MEM_CONTEXT_TEMP_BEGIN() { const BackupFile *const file = lstGet(fileList, fileIdx); BackupFileResult *const fileResult = lstGet(result, fileIdx); if (fileResult->backupCopyResult == backupCopyResultCopy) { // Setup pg file for read. Only read as many bytes as passed in pgFileSize. If the file is growing it does no // good to copy data past the end of the size recorded in the manifest since those blocks will need to be // replayed from WAL during recovery. pg_control requires special handling since it needs to be retried on crc // validation failure. bool repoChecksum = false; IoRead *readIo; if (strEqZ(file->pgFile, PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)) readIo = ioBufferReadNew(pgControlBufferFromFile(storagePg(), pgVersionForce)); else { readIo = storageReadIo( storageNewReadP( storagePg(), file->pgFile, .ignoreMissing = file->pgFileIgnoreMissing, .compressible = compressible, .limit = file->pgFileCopyExactSize ? VARUINT64(file->pgFileSizeOriginal) : NULL)); } ioFilterGroupAdd(ioReadFilterGroup(readIo), cryptoHashNew(hashTypeSha1)); ioFilterGroupAdd(ioReadFilterGroup(readIo), ioSizeNew()); // Add page checksum filter if (file->pgFileChecksumPage) { ioFilterGroupAdd( ioReadFilterGroup(readIo), pageChecksumNew( segmentNumber(file->pgFile), PG_SEGMENT_SIZE_DEFAULT / pageSize, pageSize, file->pgFilePageHeaderCheck, storagePathP(storagePg(), file->pgFile))); } // Compress filter IoFilter *const compress = repoFileCompressType != compressTypeNone ? compressFilterP( repoFileCompressType, repoFileCompressLevel, .raw = bundleRaw || file->blockIncrSize != 0) : NULL; // Encrypt filter IoFilter *const encrypt = cipherType != cipherTypeNone ? cipherBlockNewP( cipherModeEncrypt, cipherType, BUFSTR(cipherPass), .raw = bundleRaw || file->blockIncrSize != 0) : NULL; // If block incremental then add the filter and pass compress/encrypt filters to it since each block is // compressed/encrypted separately if (file->blockIncrSize != 0) { // Read prior block map const Buffer *blockMap = NULL; if (file->blockIncrMapPriorFile != NULL) { StorageRead *const blockMapRead = storageNewReadP( storageRepo(), file->blockIncrMapPriorFile, .offset = file->blockIncrMapPriorOffset, .limit = VARUINT64(file->blockIncrMapPriorSize)); if (cipherType != cipherTypeNone) { ioFilterGroupAdd( ioReadFilterGroup(storageReadIo(blockMapRead)), cipherBlockNewP(cipherModeDecrypt, cipherType, BUFSTR(cipherPass), .raw = true)); } blockMap = storageGetP(blockMapRead); } // Add block incremental filter ioFilterGroupAdd( ioReadFilterGroup(readIo), blockIncrNew( file->blockIncrSuperSize, file->blockIncrSize, file->blockIncrChecksumSize, blockIncrReference, bundleId, bundleOffset, blockMap, compress, encrypt)); repoChecksum = true; } // Else apply compress/encrypt filters to the entire file else { // Add compress filter if (compress != NULL) { ioFilterGroupAdd(ioReadFilterGroup(readIo), compress); repoChecksum = true; } // Add encrypt filter if (encrypt != NULL) { ioFilterGroupAdd(ioReadFilterGroup(readIo), encrypt); repoChecksum = true; } } // Capture checksum of file stored in the repo if filters that modify the output have been applied if (repoChecksum) ioFilterGroupAdd(ioReadFilterGroup(readIo), cryptoHashNew(hashTypeSha1)); // Add size filter last to calculate repo size ioFilterGroupAdd(ioReadFilterGroup(readIo), ioSizeNew()); // Open the source if (ioReadOpen(readIo)) { Buffer *const buffer = bufNew(ioBufferSize()); bool readEof = false; // Read the first buffer to determine if the file was truncated or was not changed. Detecting truncation // matters only when bundling is enabled as otherwise the file will be stored anyway. ioRead(readIo, buffer); if (ioReadEof(readIo)) { // Close the source and set eof ioReadClose(readIo); readEof = true; // Get file size fileResult->copySize = pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), SIZE_FILTER_TYPE, .idx = 0)); // If file is zero-length then it was truncated during the backup. When bundling we can simply mark it // as truncated since no file needs to be stored. if (bundleId != 0 && fileResult->copySize == 0) { fileResult->backupCopyResult = backupCopyResultTruncate; fileResult->copyChecksum = HASH_TYPE_SHA1_ZERO_BUF; ASSERT( bufEq( fileResult->copyChecksum, pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(readIo), CRYPTO_HASH_FILTER_TYPE, .idx = 0)))); } // Else check if size is equal to prior size else if (file->manifestFileHasReference && fileResult->copySize == file->pgFileSize) { const Buffer *const copyChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(readIo), CRYPTO_HASH_FILTER_TYPE)); // If checksum is also equal then no need to copy the file if (bufEq(file->pgFileChecksum, copyChecksum)) { // If block incremental make sure no map was returned but a prior map was provided ASSERT( file->blockIncrSize == 0 || (pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), BLOCK_INCR_FILTER_TYPE)) == 0 && file->blockIncrMapPriorFile != NULL)); fileResult->backupCopyResult = backupCopyResultNoOp; fileResult->copyChecksum = file->pgFileChecksum; } } } // Copy the file if (fileResult->backupCopyResult == backupCopyResultCopy) { // Setup the repo file for write. There is no need to write the file atomically (e.g. via a temp file on // Posix) because checksums are tested on resume after a failed backup. The path does not need to be // synced for each file because all paths are synced at the end of the backup. It needs to be created in // the prior context because it will live longer than a single loop when more than one file is being // written. if (write == NULL) { MEM_CONTEXT_PRIOR_BEGIN() { write = storageNewWriteP( storageRepoWrite(), repoFile, .compressible = compressible, .noAtomic = true, .noSyncPath = true); ioWriteOpen(storageWriteIo(write)); } MEM_CONTEXT_PRIOR_END(); } // Write the first buffer ioWrite(storageWriteIo(write), buffer); bufFree(buffer); // Copy remainder of the file if not eof if (!readEof) { ioCopyP(readIo, storageWriteIo(write)); // Close the source ioReadClose(readIo); } // Get copy results MEM_CONTEXT_BEGIN(lstMemContext(result)) { // Get size and checksum fileResult->copySize = pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), SIZE_FILTER_TYPE, .idx = 0)); fileResult->copyChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(readIo), CRYPTO_HASH_FILTER_TYPE, .idx = 0)); // Get bundle offset fileResult->bundleOffset = bundleOffset; // Get repo size fileResult->repoSize = pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), SIZE_FILTER_TYPE, .idx = 1)); // Get results of page checksum validation if (file->pgFileChecksumPage) { fileResult->pageChecksumResult = pckDup( ioFilterGroupResultPackP(ioReadFilterGroup(readIo), PAGE_CHECKSUM_FILTER_TYPE)); } // Get results of block incremental if (file->blockIncrSize != 0) { fileResult->blockIncrMapSize = pckReadU64P( ioFilterGroupResultP(ioReadFilterGroup(readIo), BLOCK_INCR_FILTER_TYPE)); // There must be a map because the file should have changed or shrunk ASSERT(fileResult->blockIncrMapSize > 0); } // Get repo checksum if (repoChecksum) { fileResult->repoChecksum = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(readIo), CRYPTO_HASH_FILTER_TYPE, .idx = 1)); } } MEM_CONTEXT_END(); bundleOffset += fileResult->repoSize; } } // Else if source file is missing and the read setup indicated ignore a missing file, the database removed it so // skip it else fileResult->backupCopyResult = backupCopyResultSkip; } } MEM_CONTEXT_TEMP_END(); } // Close the repository file if it was opened if (write != NULL) ioWriteClose(storageWriteIo(write)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/file.h0000664000175100017510000001053315240547630020157 0ustar00runnerrunner/*********************************************************************************************************************************** Backup File ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_FILE_H #define COMMAND_BACKUP_FILE_H #include "common/compress/helper.h" #include "common/crypto/common.h" #include "common/type/keyValue.h" #include "postgres/interface.h" /*********************************************************************************************************************************** Backup file types ***********************************************************************************************************************************/ typedef enum { backupCopyResultChecksum, backupCopyResultCopy, backupCopyResultSkip, backupCopyResultNoOp, backupCopyResultTruncate, } BackupCopyResult; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Copy a file from the PostgreSQL data directory to the repository typedef struct BackupFile { const String *pgFile; // Pg file to backup bool pgFileDelta; // Checksum pg file before copying bool pgFileIgnoreMissing; // Ignore missing pg file uint64_t pgFileSize; // Expected pg file size uint64_t pgFileSizeOriginal; // Original size (from manifest build) bool pgFileCopyExactSize; // Copy only pg expected size const Buffer *pgFileChecksum; // Expected pg file checksum bool pgFileChecksumPage; // Validate page checksums? bool pgFilePageHeaderCheck; // Validate page headers? size_t blockIncrSize; // Perform block incremental on this file? size_t blockIncrChecksumSize; // Block checksum size uint64_t blockIncrSuperSize; // Size of the super block const String *blockIncrMapPriorFile; // File containing prior block incremental map (NULL if none) uint64_t blockIncrMapPriorOffset; // Offset of prior block incremental map uint64_t blockIncrMapPriorSize; // Size of prior block incremental map const String *manifestFile; // Repo file const Buffer *repoFileChecksum; // Expected repo file checksum uint64_t repoFileSize; // Expected repo file size bool manifestFileResume; // Checksum repo file before copying bool manifestFileHasReference; // Reference to prior backup, if any } BackupFile; typedef struct BackupFileResult { const String *manifestFile; // Manifest file BackupCopyResult backupCopyResult; bool repoInvalid; // File was recopied because repo file was invalid uint64_t copySize; const Buffer *copyChecksum; // Checksum of pg file const Buffer *repoChecksum; // Checksum of repo file (including compression, etc.) uint64_t bundleOffset; // Offset in bundle if any uint64_t repoSize; uint64_t blockIncrMapSize; // Size of block incremental map (0 if no map) Pack *pageChecksumResult; } BackupFileResult; FN_EXTERN List *backupFile( const String *repoFile, uint64_t bundleId, bool bundleRaw, unsigned int blockIncrReference, CompressType repoFileCompressType, int repoFileCompressLevel, CipherType cipherType, const String *cipherPass, const String *pgVersionForce, PgPageSize pageSize, const List *fileList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/pageChecksum.c0000664000175100017510000002744515240547630021644 0ustar00runnerrunner/*********************************************************************************************************************************** Page Checksum Filter ***********************************************************************************************************************************/ #include #include "command/backup/pageChecksum.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/json.h" #include "common/type/object.h" #include "postgres/interface/static.vendor.h" #include "storage/posix/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct PageChecksum { unsigned int segmentPageTotal; // Total pages in a segment PgPageSize pageSize; // Page size unsigned int pageNoOffset; // Page number offset for subsequent segments bool headerCheck; // Perform additional header checks? const String *fileName; // Used to load the file to retry pages uint8_t *pageBuffer; // Buffer to hold a page while verifying the checksum bool valid; // Is the relation structure valid? bool align; // Is the relation alignment valid? PackWrite *error; // List of checksum errors } PageChecksum; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void pageChecksumToLog(const PageChecksum *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{valid: %s, align: %s}", cvtBoolToConstZ(this->valid), cvtBoolToConstZ(this->align)); } #define FUNCTION_LOG_PAGE_CHECKSUM_TYPE \ PageChecksum * #define FUNCTION_LOG_PAGE_CHECKSUM_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, pageChecksumToLog, buffer, bufferSize) /*********************************************************************************************************************************** Verify page checksums ***********************************************************************************************************************************/ static void pageChecksumProcess(THIS_VOID, const Buffer *const input) { THIS(PageChecksum); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PAGE_CHECKSUM, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); // Calculate total pages in the buffer unsigned int pageTotal = (unsigned int)(bufUsed(input) / this->pageSize); // If there is a partial page make sure there is enough of it to validate the checksum const unsigned int pageRemainder = (unsigned int)(bufUsed(input) % this->pageSize); if (pageRemainder != 0) { // Misaligned blocks, if any, should only be at the end of the file if (!this->align) THROW(AssertError, "should not be possible to see two misaligned pages in a row"); // Mark this buffer as misaligned in case we see another one this->align = false; // If there at least 512 bytes then we'll treat this as a partial write (modern file systems will have at least 4096) if (pageRemainder >= 512) { pageTotal++; } // Else this appears to be a corrupted file and we'll stop doing page checksums else this->valid = false; } // Verify the checksums of complete pages in the buffer if (this->valid) { for (unsigned int pageIdx = 0; pageIdx < pageTotal; pageIdx++) { // Get a pointer to the page header const PageHeaderData *const pageHeader = (const PageHeaderData *)(bufPtrConst(input) + pageIdx * this->pageSize); // Block number relative to all segments in the relation const unsigned int blockNo = this->pageNoOffset + pageIdx; // Only validate page checksum if the page is complete if (this->align || pageIdx < pageTotal - 1) { bool pageValid = true; // Skip new all-zero pages. To speed this up first check pd_upper when header check is enabled or pd_checksum when // header check is disabled. The latter is required when the page is encrypted. if ((this->headerCheck && pageHeader->pd_upper == 0) || (!this->headerCheck && pageHeader->pd_checksum == 0)) { // Check that the entire page is zero for (unsigned int pageIdx = 0; pageIdx < this->pageSize / sizeof(size_t); pageIdx++) { if (((const size_t *)pageHeader)[pageIdx] != 0) { pageValid = false; break; } } // If the entire page is zero it is valid if (pageValid) continue; } // Only validate the checksum if the page is valid if (pageValid) { // Make a copy of the page since it will be modified by the page checksum function memcpy(this->pageBuffer, pageHeader, this->pageSize); // Continue if the checksum matches if (pageHeader->pd_checksum == pgPageChecksum(this->pageBuffer, blockNo, this->pageSize)) continue; } // On error retry the page bool changed = false; MEM_CONTEXT_TEMP_BEGIN() { // Reload the page const Buffer *const pageRetry = storageGetP( storageNewReadP( storagePosixNewP(FSLASH_STR), this->fileName, .offset = (uint64_t)(blockNo % this->segmentPageTotal) * this->pageSize, .limit = VARUINT64(this->pageSize))); // Check if the page has changed since it was last read changed = !bufEq(pageRetry, BUF(pageHeader, this->pageSize)); } MEM_CONTEXT_TEMP_END(); // If the page has changed then PostgreSQL must be updating it so we won't consider it to be invalid. We are not // concerned about whether this new page has a valid checksum or not, since the page will be replayed from WAL on // recovery. This prevents (theoretically) limitless retries for a hot page. if (changed) continue; } // Create the error list if it does not exist yet if (this->error == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->error = pckWriteNewP(); pckWriteArrayBeginP(this->error); } MEM_CONTEXT_OBJ_END(); } // Add page number and lsn to the error list pckWriteObjBeginP(this->error, .id = blockNo + 1); pckWriteObjEndP(this->error); } this->pageNoOffset += pageTotal; } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Return filter result ***********************************************************************************************************************************/ static Pack * pageChecksumResult(THIS_VOID) { THIS(PageChecksum); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PAGE_CHECKSUM, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result; MEM_CONTEXT_OBJ_BEGIN(this) { // End the error array if (this->error != NULL) { pckWriteArrayEndP(this->error); this->valid = false; } // Else create a pack to hold the flags else { this->error = pckWriteNewP(); pckWriteNullP(this->error); } // Valid and align flags pckWriteBoolP(this->error, this->valid, .defaultWrite = true); pckWriteBoolP(this->error, this->align, .defaultWrite = true); // End pack pckWriteEndP(this->error); result = pckMove(pckWriteResult(this->error), memContextPrior()); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * pageChecksumNew( const unsigned int segmentNo, const unsigned int segmentPageTotal, const PgPageSize pageSize, const bool headerCheck, const String *const fileName) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(UINT, segmentNo); FUNCTION_LOG_PARAM(UINT, segmentPageTotal); FUNCTION_LOG_PARAM(ENUM, pageSize); FUNCTION_LOG_PARAM(BOOL, headerCheck); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_END(); ASSERT(pgPageSizeValid(pageSize)); ASSERT(segmentPageTotal > 0 && segmentPageTotal % pageSize == 0); OBJ_NEW_BEGIN(PageChecksum, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (PageChecksum) { .segmentPageTotal = segmentPageTotal, .pageSize = pageSize, .pageNoOffset = segmentNo * segmentPageTotal, .headerCheck = headerCheck, .fileName = strDup(fileName), .pageBuffer = bufPtr(bufNew(pageSize)), .valid = true, .align = true, }; } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU32P(packWrite, segmentNo); pckWriteU32P(packWrite, segmentPageTotal); pckWriteU32P(packWrite, pageSize); pckWriteBoolP(packWrite, headerCheck); pckWriteStrP(packWrite, fileName); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP(PAGE_CHECKSUM_FILTER_TYPE, this, paramList, .in = pageChecksumProcess, .result = pageChecksumResult)); } FN_EXTERN IoFilter * pageChecksumNewPack(const Pack *const paramList) { IoFilter *result; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const paramListPack = pckReadNew(paramList); const unsigned int segmentNo = pckReadU32P(paramListPack); const unsigned int segmentPageTotal = pckReadU32P(paramListPack); const PgPageSize pageSize = (PgPageSize)pckReadU32P(paramListPack); const bool headerCheck = pckReadBoolP(paramListPack); const String *const fileName = pckReadStrP(paramListPack); result = ioFilterMove(pageChecksumNew(segmentNo, segmentPageTotal, pageSize, headerCheck, fileName), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/pageChecksum.h0000664000175100017510000000261015240547630021634 0ustar00runnerrunner/*********************************************************************************************************************************** Page Checksum Filter Check all pages in a PostgreSQL relation to ensure the checksums are valid. ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_PAGE_CHECKSUM_H #define COMMAND_BACKUP_PAGE_CHECKSUM_H #include "common/io/filter/filter.h" #include "postgres/interface.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define PAGE_CHECKSUM_FILTER_TYPE STRID5("pg-chksum", 0xdacd681ecf00) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *pageChecksumNew( unsigned int segmentNo, unsigned int segmentPageTotal, PgPageSize pageSize, bool headerCheck, const String *fileName); FN_EXTERN IoFilter *pageChecksumNewPack(const Pack *paramList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/protocol.c0000664000175100017510000001121715240547630021074 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Protocol Handler ***********************************************************************************************************************************/ #include #include "command/backup/file.h" #include "command/backup/protocol.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/json.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * backupFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Backup options that apply to all files const String *const repoFile = pckReadStrP(param); const uint64_t bundleId = pckReadU64P(param); const bool bundleRaw = bundleId != 0 ? pckReadBoolP(param) : false; const unsigned int blockIncrReference = (unsigned int)pckReadU64P(param); const CompressType repoFileCompressType = (CompressType)pckReadU32P(param); const int repoFileCompressLevel = pckReadI32P(param); const CipherType cipherType = (CipherType)pckReadU64P(param); const String *const cipherPass = pckReadStrP(param); const PgPageSize pageSize = pckReadU32P(param); const String *const pgVersionForce = pckReadStrP(param); // Build the file list List *const fileList = lstNewP(sizeof(BackupFile)); while (!pckReadNullP(param)) { BackupFile file = {.pgFile = pckReadStrP(param)}; file.pgFileDelta = pckReadBoolP(param); file.pgFileIgnoreMissing = pckReadBoolP(param); file.pgFileSize = pckReadU64P(param); file.pgFileSizeOriginal = pckReadU64P(param); file.pgFileCopyExactSize = pckReadBoolP(param); file.pgFileChecksum = pckReadBinP(param); file.pgFileChecksumPage = pckReadBoolP(param); file.pgFilePageHeaderCheck = pckReadBoolP(param); file.blockIncrSize = (size_t)pckReadU64P(param); if (file.blockIncrSize > 0) { file.blockIncrChecksumSize = (size_t)pckReadU64P(param); file.blockIncrSuperSize = pckReadU64P(param); file.blockIncrMapPriorFile = pckReadStrP(param); if (file.blockIncrMapPriorFile != NULL) { file.blockIncrMapPriorOffset = pckReadU64P(param); file.blockIncrMapPriorSize = pckReadU64P(param); } } file.manifestFile = pckReadStrP(param); file.repoFileChecksum = pckReadBinP(param); file.repoFileSize = pckReadU64P(param); file.manifestFileResume = pckReadBoolP(param); file.manifestFileHasReference = pckReadBoolP(param); lstAdd(fileList, &file); } // Backup file const List *const resultList = backupFile( repoFile, bundleId, bundleRaw, blockIncrReference, repoFileCompressType, repoFileCompressLevel, cipherType, cipherPass, pgVersionForce, pageSize, fileList); // Return result PackWrite *const data = protocolServerResultData(result); for (unsigned int resultIdx = 0; resultIdx < lstSize(resultList); resultIdx++) { const BackupFileResult *const fileResult = lstGet(resultList, resultIdx); ASSERT( fileResult->backupCopyResult == backupCopyResultSkip || fileResult->copySize != 0 || bufEq(fileResult->copyChecksum, HASH_TYPE_SHA1_ZERO_BUF)); pckWriteStrP(data, fileResult->manifestFile); pckWriteU32P(data, fileResult->backupCopyResult); pckWriteBoolP(data, fileResult->repoInvalid); pckWriteU64P(data, fileResult->copySize); pckWriteU64P(data, fileResult->bundleOffset); pckWriteU64P(data, fileResult->blockIncrMapSize); pckWriteU64P(data, fileResult->repoSize); pckWriteBinP(data, fileResult->copyChecksum); pckWriteBinP(data, fileResult->repoChecksum); pckWritePackP(data, fileResult->pageChecksumResult); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/backup/protocol.h0000664000175100017510000000266215240547630021105 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_PROTOCOL_H #define COMMAND_BACKUP_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *backupFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_BACKUP_FILE STRID5("bp-f", 0x36e020) #define PROTOCOL_SERVER_HANDLER_BACKUP_LIST \ {.command = PROTOCOL_COMMAND_BACKUP_FILE, .process = backupFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/check/0000775000175100017510000000000015240547630016675 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/check/check.c0000664000175100017510000002263615240547630020127 0ustar00runnerrunner/*********************************************************************************************************************************** Check Command ***********************************************************************************************************************************/ #include #include #include "command/archive/find.h" #include "command/check/check.h" #include "command/check/common.h" #include "command/check/report.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "config/load.h" #include "config/parse.h" #include "db/helper.h" #include "info/infoArchive.h" #include "postgres/interface.h" #include "protocol/helper.h" #include "storage/helper.h" /*********************************************************************************************************************************** Helper functions (to assist with testing) ***********************************************************************************************************************************/ static unsigned int checkManifest(void) { FUNCTION_LOG_VOID(logLevelDebug); // Return the actual number of pg* defined unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Loop through all defined databases and attempt to build a manifest for (unsigned int pgIdx = 0; pgIdx < cfgOptionGroupIdxTotal(cfgOptGrpPg); pgIdx++) { result++; // ??? Placeholder for manifest build storageListP(storagePgIdx(pgIdx), cfgOptionIdxStr(cfgOptPgPath, pgIdx)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } static void checkStandby(const DbGetResult dbGroup, const unsigned int pgPathDefinedTotal) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB_GET_RESULT, dbGroup); FUNCTION_LOG_PARAM(UINT, pgPathDefinedTotal); FUNCTION_LOG_END(); // If a standby is defined, check the configuration if (dbGroup.standby != NULL) { // If primary was not found (only have 1 pg configured locally, and we want to still run because this is a standby) if (dbGroup.primary == NULL) { // If any repo is local or more than one pg-path is found then a primary should have been found so error bool error = pgPathDefinedTotal > 1; unsigned int repoIdx = 0; while (!error && repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo)) { if (repoIsLocal(repoIdx)) error = true; repoIdx++; } if (error) THROW(ConfigError, "primary database not found\nHINT: check indexed pg-path/pg-host configurations"); } // Check the user configured path and version against the database checkDbConfig(dbPgControl(dbGroup.standby).version, dbGroup.standbyIdx, dbGroup.standby, true); // Check each repository configured for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT(CFGCMD_CHECK " %s (standby)", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); // Get the repo storage in case it is remote and encryption settings need to be pulled down (performed here for testing) const Storage *const storageRepo = storageRepoIdx(repoIdx); // Check that the backup and archive info files exist and are valid for the current database of the stanza checkStanzaInfoPg( storageRepo, dbPgControl(dbGroup.standby).version, dbPgControl(dbGroup.standby).systemId, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); } LOG_INFO("switch wal not performed because this is a standby"); // Free the standby connection dbFree(dbGroup.standby); } // If backup from standby is true then warn when a standby not found else if (cfgOptionSeq(cfgOptBackupStandby) != CFGOPTVAL_BACKUP_STANDBY_N) { LOG_WARN("option '" CFGOPT_BACKUP_STANDBY "' is enabled but standby is not properly configured"); } FUNCTION_LOG_RETURN_VOID(); } static void checkPrimary(const DbGetResult dbGroup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB_GET_RESULT, dbGroup); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); // If a primary is defined, check the configuration and perform a WAL switch and make sure the WAL is archived if (dbGroup.primary != NULL) { // Check the user configured path and version against the database checkDbConfig(dbPgControl(dbGroup.primary).version, dbGroup.primaryIdx, dbGroup.primary, false); // Check configuration of each repo const String **const repoArchiveId = memNew(sizeof(String *) * cfgOptionGroupIdxTotal(cfgOptGrpRepo)); for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT(CFGCMD_CHECK " %s configuration (primary)", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); // Get the repo storage in case it is remote and encryption settings need to be pulled down (performed here for testing) const Storage *const storageRepo = storageRepoIdx(repoIdx); // Check that the backup and archive info files exist and are valid for the current database of the stanza checkStanzaInfoPg( storageRepo, dbPgControl(dbGroup.primary).version, dbPgControl(dbGroup.primary).systemId, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); // Attempt to load the archive info file and retrieve the archiveId const InfoArchive *const archiveInfo = infoArchiveLoadFile( storageRepo, INFO_ARCHIVE_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); repoArchiveId[repoIdx] = infoArchiveId(archiveInfo); } // Perform a WAL switch const String *const walSegment = dbWalSwitch(dbGroup.primary); // Wait for the WAL to appear in each repo for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT(CFGCMD_CHECK " %s archive for WAL (primary)", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); const String *const walSegmentFile = walSegmentFindOne( storageRepoIdx(repoIdx), repoArchiveId[repoIdx], walSegment, cfgOptionUInt64(cfgOptArchiveTimeout)); LOG_INFO_FMT( "WAL segment %s successfully archived to '%s' on %s", strZ(walSegment), strZ( storagePathP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(repoArchiveId[repoIdx]), strZ(walSegmentFile)))), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); } dbFree(dbGroup.primary); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdCheck(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { if (cfgOptionBool(cfgOptReport)) ioFdWriteOneStr(STDOUT_FILENO, checkReport()); else { // Build stanza list based on whether a stanza was specified or not StringList *stanzaList; bool stanzaSpecified = cfgOptionTest(cfgOptStanza); if (stanzaSpecified) { stanzaList = strLstNew(); strLstAdd(stanzaList, cfgOptionStr(cfgOptStanza)); } else { stanzaList = cfgParseStanzaList(); if (strLstSize(stanzaList) == 0) { LOG_WARN( "no stanzas found to check\n" "HINT: are there non-empty stanza sections in the configuration?"); } } // Iterate stanzas for (unsigned int stanzaIdx = 0; stanzaIdx < strLstSize(stanzaList); stanzaIdx++) { // Switch stanza if required if (!stanzaSpecified) { const String *const stanza = strLstGet(stanzaList, stanzaIdx); LOG_INFO_FMT("check stanza '%s'", strZ(stanza)); // Free storage and protocol cache storageHelperFree(); protocolFree(); // Reload config with new stanza cfgLoadStanza(stanza); } // Get the primary/standby connections (standby is only required if backup from standby is enabled) DbGetResult dbGroup = dbGet(false, false, CFGOPTVAL_BACKUP_STANDBY_N); if (dbGroup.standby == NULL && dbGroup.primary == NULL) THROW(ConfigError, "no database found\nHINT: check indexed pg-path/pg-host configurations"); const unsigned int pgPathDefinedTotal = checkManifest(); checkStandby(dbGroup, pgPathDefinedTotal); checkPrimary(dbGroup); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/check/check.h0000664000175100017510000000125315240547630020124 0ustar00runnerrunner/*********************************************************************************************************************************** Check Command ***********************************************************************************************************************************/ #ifndef COMMAND_CHECK_CHECK_H #define COMMAND_CHECK_CHECK_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Perform standard checks FN_EXTERN void cmdCheck(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/check/common.c0000664000175100017510000001562615240547630020343 0ustar00runnerrunner/*********************************************************************************************************************************** Check Common Handler ***********************************************************************************************************************************/ #include #include #include "command/check/common.h" #include "common/debug.h" #include "config/config.h" #include "db/helper.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "postgres/interface.h" #include "storage/helper.h" #include "version.h" /*********************************************************************************************************************************** Helper function ***********************************************************************************************************************************/ static bool checkArchiveCommand(const String *const archiveCommand) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, archiveCommand); FUNCTION_TEST_END(); bool result = archiveCommand != NULL; if (result && strstr(strZ(archiveCommand), PROJECT_BIN) == NULL) result = false; if (!result) { THROW_FMT( ArchiveCommandInvalidError, "archive_command '%s' must contain %s", archiveCommand != NULL ? strZ(archiveCommand) : "[" NULL_Z "]", PROJECT_BIN); } FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void checkDbConfig(const unsigned int pgVersion, const unsigned int pgIdx, const Db *const dbObject, const bool isStandby) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_PARAM(UINT, pgIdx); FUNCTION_TEST_PARAM(DB, dbObject); FUNCTION_TEST_PARAM(BOOL, isStandby); FUNCTION_TEST_END(); ASSERT(dbObject != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Error if the version from the control file and the configured pg-path do not match the values obtained from the database const unsigned int dbVersion = dbPgVersion(dbObject); const String *const dbPath = dbPgDataPath(dbObject); if (pgVersion != dbVersion || strCmp(cfgOptionIdxStr(cfgOptPgPath, pgIdx), dbPath) != 0) { THROW_FMT( DbMismatchError, "version '%s' and path '%s' queried from cluster do not match version '%s' and '%s' read from '%s/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "'\nHINT: the %s and %s settings likely reference different clusters.", strZ(pgVersionToStr(dbVersion)), strZ(dbPath), strZ(pgVersionToStr(pgVersion)), strZ(cfgOptionIdxDisplay(cfgOptPgPath, pgIdx)), strZ(cfgOptionIdxDisplay(cfgOptPgPath, pgIdx)), cfgOptionIdxName(cfgOptPgPath, pgIdx), cfgOptionIdxName(cfgOptPgPort, pgIdx)); } // Check archive configuration if option is valid for the command and set if (!isStandby && cfgOptionValid(cfgOptArchiveCheck) && cfgOptionBool(cfgOptArchiveCheck)) { // Error if archive_mode = off since backup start will fail if (strCmpZ(dbArchiveMode(dbObject), "off") == 0) { THROW(ArchiveDisabledError, "archive_mode must be enabled"); } // Error if archive_mode = always unless check is disabled (support has not been added yet) if (cfgOptionBool(cfgOptArchiveModeCheck) && strCmpZ(dbArchiveMode(dbObject), "always") == 0) { THROW(FeatureNotSupportedError, "archive_mode=always not supported"); } // Check if archive_command is set and is valid checkArchiveCommand(dbArchiveCommand(dbObject)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void checkStanzaInfo(const InfoPgData *const archiveInfo, const InfoPgData *const backupInfo) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(INFO_PG_DATA, archiveInfo); FUNCTION_TEST_PARAM_P(INFO_PG_DATA, backupInfo); FUNCTION_TEST_END(); ASSERT(archiveInfo != NULL); ASSERT(backupInfo != NULL); // Error if there is a mismatch between the archive and backup info files if (archiveInfo->id != backupInfo->id || archiveInfo->systemId != backupInfo->systemId || archiveInfo->version != backupInfo->version) { THROW_FMT( FileInvalidError, "backup info file and archive info file do not match\n" "archive: id = %u, version = %s, system-id = %" PRIu64 "\n" "backup : id = %u, version = %s, system-id = %" PRIu64 "\n" "HINT: this may be a symptom of repository corruption!", archiveInfo->id, strZ(pgVersionToStr(archiveInfo->version)), archiveInfo->systemId, backupInfo->id, strZ(pgVersionToStr(backupInfo->version)), backupInfo->systemId); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void checkStanzaInfoPg( const Storage *const storage, const unsigned int pgVersion, const uint64_t pgSystemId, const CipherType cipherType, const String *const cipherPass) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE, storage); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_PARAM(UINT64, pgSystemId); FUNCTION_TEST_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_TEST_END(); ASSERT(storage != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Check that the backup and archive info files exist const InfoArchive *const infoArchive = infoArchiveLoadFile(storage, INFO_ARCHIVE_PATH_FILE_STR, cipherType, cipherPass); const InfoPgData archiveInfoPg = infoPgData(infoArchivePg(infoArchive), infoPgDataCurrentId(infoArchivePg(infoArchive))); const InfoBackup *const infoBackup = infoBackupLoadFile(storage, INFO_BACKUP_PATH_FILE_STR, cipherType, cipherPass); const InfoPgData backupInfoPg = infoPgData(infoBackupPg(infoBackup), infoPgDataCurrentId(infoBackupPg(infoBackup))); // Check that the info files pg data match each other checkStanzaInfo(&archiveInfoPg, &backupInfoPg); // Check that the version and system id match the current database if (pgVersion != archiveInfoPg.version || pgSystemId != archiveInfoPg.systemId) { THROW( FileInvalidError, "backup and archive info files exist but do not match the database\n" "HINT: is this the correct stanza?\n" "HINT: did an error occur during stanza-upgrade?"); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/check/common.h0000664000175100017510000000244015240547630020336 0ustar00runnerrunner/*********************************************************************************************************************************** Check Command Common ***********************************************************************************************************************************/ #ifndef COMMAND_CHECK_COMMON_H #define COMMAND_CHECK_COMMON_H #include "common/type/string.h" #include "db/db.h" #include "info/infoPg.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Check the database path and version are configured correctly FN_EXTERN void checkDbConfig(const unsigned int pgVersion, const unsigned int pgIdx, const Db *dbObject, bool isStandby); // Validate the archive and backup info files FN_EXTERN void checkStanzaInfo(const InfoPgData *archiveInfo, const InfoPgData *backupInfo); // Load and validate the database data of the info files against each other and the current database FN_EXTERN void checkStanzaInfoPg( const Storage *storage, const unsigned int pgVersion, const uint64_t pgSystemId, CipherType cipherType, const String *cipherPass); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/check/report.c0000664000175100017510000002022215240547630020352 0ustar00runnerrunner/*********************************************************************************************************************************** Support Command ***********************************************************************************************************************************/ #include #include "command/check/report.h" #include "common/debug.h" #include "common/log.h" #include "common/type/json.h" #include "config/config.h" #include "config/parse.h" /*********************************************************************************************************************************** Render config ***********************************************************************************************************************************/ // Helper to render an individual config value static void checkReportConfigVal(JsonWrite *const json, const String *const optionName, const StringList *valueList, const bool env) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, json); FUNCTION_TEST_PARAM(STRING, optionName); FUNCTION_TEST_PARAM(STRING_LIST, valueList); FUNCTION_TEST_PARAM(BOOL, env); FUNCTION_TEST_END(); ASSERT(json != NULL); ASSERT(optionName != NULL); ASSERT(valueList != NULL); MEM_CONTEXT_TEMP_BEGIN() { jsonWriteObjectBegin(json); const CfgParseOptionResult option = cfgParseOptionP(optionName); // Render value if (option.found) { jsonWriteKeyStrId(json, STRID5("val", 0x30360)); // Multiple values if (option.multi) { ASSERT(strLstSize(valueList) >= 1); ASSERT(!cfgParseOptionSecure(option.id)); // Split environment values if (env) { ASSERT(strLstSize(valueList) == 1); valueList = strLstNewSplitZ(strLstGet(valueList, 0), ":"); } // Render each value jsonWriteArrayBegin(json); for (unsigned int valueIdx = 0; valueIdx < strLstSize(valueList); valueIdx++) jsonWriteStr(json, strLstGet(valueList, valueIdx)); jsonWriteArrayEnd(json); } // Single value else { ASSERT(strLstSize(valueList) == 1); if (cfgParseOptionSecure(option.id)) jsonWriteStr(json, strNewFmt("<%s>", strZ(optionName))); else jsonWriteStr(json, strLstGet(valueList, 0)); } } // Warn on invalid options or option modifiers if (!option.found || option.negate || option.reset) { jsonWriteKeyStrId(json, STRID5("warn", 0x748370)); // Warn if the option not found if (!option.found) { jsonWriteZ(json, "invalid option"); } // Warn if negate option else if (option.negate) { jsonWriteZ(json, "invalid negate option"); } // Warn if reset option else { ASSERT(option.reset); jsonWriteZ(json, "invalid reset option"); } } jsonWriteObjectEnd(json); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } // Helper to render the environment static void checkReportConfigEnv(JsonWrite *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, json); FUNCTION_TEST_END(); ASSERT(json != NULL); MEM_CONTEXT_TEMP_BEGIN() { jsonWriteKeyStrId(json, STRID5("env", 0x59c50)); jsonWriteObjectBegin(json); // Loop through all environment variables and look for our env vars by matching the prefix unsigned int environIdx = 0; KeyValue *const keyValue = kvNew(); while (environ[environIdx] != NULL) { const char *const environKeyValue = environ[environIdx]; environIdx++; if (strstr(environKeyValue, PGBACKREST_ENV) == environKeyValue) { // Find the first = char const char *const equalPtr = strchr(environKeyValue, '='); ASSERT(equalPtr != NULL); // Get key and value const String *const key = strNewZN(environKeyValue, (size_t)(equalPtr - environKeyValue)); const String *const value = STR(equalPtr + 1); kvPut(keyValue, VARSTR(key), VARSTR(value)); } } // Render environment variables const StringList *const keyList = strLstSort(strLstNewVarLst(kvKeyList(keyValue)), sortOrderAsc); for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { const String *const key = strLstGet(keyList, keyIdx); StringList *const valueList = strLstNew(); strLstAdd(valueList, varStr(kvGet(keyValue, VARSTR(key)))); jsonWriteKey(json, key); checkReportConfigVal( json, strReplaceChr(strLower(strNewZ(strZ(key) + PGBACKREST_ENV_SIZE)), '_', '-'), valueList, true); } jsonWriteObjectEnd(json); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } // Helper to render the config file static void checkReportConfigFile(JsonWrite *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, json); FUNCTION_TEST_END(); ASSERT(json != NULL); MEM_CONTEXT_TEMP_BEGIN() { const Ini *const ini = cfgParseIni(); jsonWriteKeyStrId(json, STRID5("file", 0x2b1260)); // If no config if (ini == NULL) { jsonWriteNull(json); } // Else config exists else { // Iterate config sections const StringList *const sectionList = strLstSort(iniSectionList(ini), sortOrderAsc); jsonWriteObjectBegin(json); for (unsigned int sectionIdx = 0; sectionIdx < strLstSize(sectionList); sectionIdx++) { const String *const section = strLstGet(sectionList, sectionIdx); jsonWriteKey(json, section); jsonWriteObjectBegin(json); // Iterate config keys const StringList *const keyList = strLstSort(iniSectionKeyList(ini, section), sortOrderAsc); for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { const String *const key = strLstGet(keyList, keyIdx); // Render value StringList *valueList; if (iniSectionKeyIsList(ini, section, key)) valueList = iniGetList(ini, section, key); else { valueList = strLstNew(); strLstAdd(valueList, iniGet(ini, section, key)); } jsonWriteKey(json, key); checkReportConfigVal(json, key, valueList, false); } jsonWriteObjectEnd(json); } jsonWriteObjectEnd(json); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } static void checkReportConfig(JsonWrite *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, json); FUNCTION_TEST_END(); ASSERT(json != NULL); jsonWriteKeyStrId(json, STRID5("cfg", 0x1cc30)); jsonWriteObjectBegin(json); checkReportConfigEnv(json); checkReportConfigFile(json); jsonWriteObjectEnd(json); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * checkReport(void) { FUNCTION_TEST_VOID(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { JsonWrite *const json = jsonWriteNewP(.json = result); // Render report jsonWriteObjectBegin(json); checkReportConfig(json); jsonWriteObjectEnd(json); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/check/report.h0000664000175100017510000000132015240547630020355 0ustar00runnerrunner/*********************************************************************************************************************************** Check Command Report ***********************************************************************************************************************************/ #ifndef COMMAND_CHECK_REPORT_H #define COMMAND_CHECK_REPORT_H #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Check report FN_EXTERN String *checkReport(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/command.c0000664000175100017510000003050015240547630017400 0ustar00runnerrunner/*********************************************************************************************************************************** Common Command Routines ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "command/command.h" #include "common/debug.h" #include "common/fork.h" #include "common/log.h" #include "common/memContext.h" #include "common/stat.h" #include "common/time.h" #include "common/type/json.h" #include "config/config.intern.h" #include "config/parse.h" #include "version.h" /*********************************************************************************************************************************** Track time command started ***********************************************************************************************************************************/ static TimeMSec timeBegin; static String *cmdOptionStr; /**********************************************************************************************************************************/ FN_EXTERN void cmdInit(void) { FUNCTION_LOG_VOID(logLevelTrace); timeBegin = timeMSec(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const String * cmdOption(void) { FUNCTION_TEST_VOID(); if (cmdOptionStr == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { cmdOptionStr = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // Add command parameters if they exist const StringList *const commandParamList = cfgCommandParam(); if (!strLstEmpty(commandParamList)) { strCatZ(cmdOptionStr, " ["); for (unsigned int commandParamIdx = 0; commandParamIdx < strLstSize(commandParamList); commandParamIdx++) { const String *commandParam = strLstGet(commandParamList, commandParamIdx); if (commandParamIdx != 0) strCatZ(cmdOptionStr, ", "); if (strchr(strZ(commandParam), ' ') != NULL) commandParam = strNewFmt("\"%s\"", strZ(commandParam)); strCat(cmdOptionStr, commandParam); } strCatZ(cmdOptionStr, "]"); } // Loop though options and add the ones that are interesting for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { // Skip the option if not valid for this command. Generally only one command runs at a time, but sometimes // commands are chained together (e.g. backup and expire) and the second command may not use all the options of // the first command. Displaying them is harmless but might cause confusion. if (!cfgOptionValid(optionId) || !cfgParseOptionValid(cfgCommand(), cfgCommandRole(), optionId)) continue; // Loop through option indexes const unsigned int optionIdxTotal = cfgOptionGroup(optionId) ? cfgOptionGroupIdxTotal(cfgOptionGroupId(optionId)) : 1; for (unsigned int optionIdx = 0; optionIdx < optionIdxTotal; optionIdx++) { // If option was negated if (cfgOptionIdxNegate(optionId, optionIdx)) strCatFmt(cmdOptionStr, " --no-%s", cfgOptionIdxName(optionId, optionIdx)); // If option was reset else if (cfgOptionIdxReset(optionId, optionIdx)) strCatFmt(cmdOptionStr, " --reset-%s", cfgOptionIdxName(optionId, optionIdx)); // Else not default else if (cfgOptionIdxSource(optionId, optionIdx) != cfgSourceDefault) { // Don't show redacted options if (cfgParseOptionSecure(optionId)) strCatFmt(cmdOptionStr, " --%s=", cfgOptionIdxName(optionId, optionIdx)); // Output boolean option else if (cfgParseOptionType(optionId) == cfgOptTypeBoolean) strCatFmt(cmdOptionStr, " --%s", cfgOptionIdxName(optionId, optionIdx)); // Output other options else { StringList *valueList = NULL; // Generate the values of hash options if (cfgParseOptionType(optionId) == cfgOptTypeHash) { valueList = strLstNew(); const KeyValue *const optionKv = cfgOptionIdxKv(optionId, optionIdx); const VariantList *const keyList = kvKeyList(optionKv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { strLstAddFmt( valueList, "%s=%s", strZ(varStr(varLstGet(keyList, keyIdx))), strZ(varStrForce(kvGet(optionKv, varLstGet(keyList, keyIdx))))); } } // Generate values for list options else if (cfgParseOptionType(optionId) == cfgOptTypeList) { valueList = strLstNewVarLst(cfgOptionIdxLst(optionId, optionIdx)); } // Else only one value else { valueList = strLstNew(); strLstAdd(valueList, cfgOptionIdxDisplay(optionId, optionIdx)); } // Output options and values for (unsigned int valueListIdx = 0; valueListIdx < strLstSize(valueList); valueListIdx++) { const String *value = strLstGet(valueList, valueListIdx); strCatFmt(cmdOptionStr, " --%s", cfgOptionIdxName(optionId, optionIdx)); if (strchr(strZ(value), ' ') != NULL) value = strNewFmt("\"%s\"", strZ(value)); strCatFmt(cmdOptionStr, "=%s", strZ(value)); } } } } } } MEM_CONTEXT_TEMP_END(); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN(STRING, cmdOptionStr); } /**********************************************************************************************************************************/ FN_EXTERN void cmdBegin(void) { FUNCTION_LOG_VOID(logLevelTrace); ASSERT(cfgInited()); // This is fairly expensive log message to generate so skip it if it won't be output if (logAny(cfgLogLevelDefault())) { MEM_CONTEXT_TEMP_BEGIN() { // Basic info on command start String *const info = strCatFmt(strNew(), "%s command begin", strZ(cfgCommandRoleName())); // Free the old option string if it exists. This is needed when more than one command is run in a row so an option // string gets created for the new command. strFree(cmdOptionStr); cmdOptionStr = NULL; // Add version and options strCatFmt(info, " %s:%s", PROJECT_VERSION, strZ(cmdOption())); LOG(cfgLogLevelDefault(), 0, strZ(info)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdEnd(const int code, const String *const errorMessage) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, code); FUNCTION_LOG_PARAM(STRING, errorMessage); FUNCTION_LOG_END(); ASSERT(cfgInited()); // Skip this log message if it won't be output. It's not too expensive but since we skipped cmdBegin(), may as well. if (logAny(cfgLogLevelDefault())) { MEM_CONTEXT_TEMP_BEGIN() { // Output statistics if there are any const String *const statJson = statToJson(); if (statJson != NULL) LOG_DETAIL_FMT("statistics: %s", strZ(statJson)); // Basic info on command end String *const info = strCatFmt(strNew(), "%s command end: ", strZ(cfgCommandRoleName())); if (errorMessage == NULL) { strCatZ(info, "completed successfully"); if (cfgOptionBool(cfgOptLogTimestamp)) strCatFmt(info, " (%" PRIu64 "ms)", timeMSec() - timeBegin); } else strCat(info, errorMessage); LOG(cfgLogLevelDefault(), 0, strZ(info)); } MEM_CONTEXT_TEMP_END(); } // Reset timeBegin in case there is another command following this one timeBegin = timeMSec(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdAsyncExec(const char *const command, const StringList *const commandExec) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRINGZ, command); FUNCTION_LOG_PARAM(STRING_LIST, commandExec); FUNCTION_LOG_END(); ASSERT(commandExec != NULL); // Fork off the async process const pid_t pid = forkSafe(); if (pid == 0) { // Disable logging and close log file logClose(); // Detach from parent process forkDetach(); // Close any open file descriptors above the standard three (stdin, stdout, stderr). Don't check the return value since we // don't know which file descriptors are actually open (might be none). It's possible that there are open files >= 1024 but // there is no easy way to detect that and this should give us enough descriptors to do our work. for (int fd = 3; fd < 1024; fd++) close(fd); // Execute the binary. This statement will not return if it is successful. execvp() requires non-const parameters because it // modifies them after the fork. #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" THROW_ON_SYS_ERROR_FMT( execvp(strZ(strLstGet(commandExec, 0)), UNCONSTIFY(char **, strLstPtr(commandExec))) == -1, ExecuteError, "unable to execute asynchronous '%s'", command); #pragma GCC diagnostic pop } #ifdef DEBUG_EXEC_TIME // Get the time to measure how long it takes for the forked process to exit const TimeMSec timeBegin = timeMSec(); #endif // The process that was just forked should return immediately int processStatus; THROW_ON_SYS_ERROR(waitpid(pid, &processStatus, 0) == -1, ExecuteError, "unable to wait for forked process"); // The first fork should exit with success. If not, something went wrong during the second fork. CHECK(ExecuteError, WIFEXITED(processStatus) && WEXITSTATUS(processStatus) == 0, "error on first fork"); #ifdef DEBUG_EXEC_TIME // If the process does not exit immediately then something probably went wrong with the double fork. It's possible that this // test will fail on very slow systems so it may need to be tuned. The idea is to make sure that the waitpid() above is not // waiting on the async process. CHECK(AssertError, timeMSec() - timeBegin < 10, "the process does not exit immediately"); #endif FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/command.h0000664000175100017510000000245215240547630017412 0ustar00runnerrunner/*********************************************************************************************************************************** Common Command Routines ***********************************************************************************************************************************/ #ifndef COMMAND_COMMAND_H #define COMMAND_COMMAND_H #include "common/type/stringList.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Capture time at the very start of main so total time is more accurate FN_EXTERN void cmdInit(void); // Begin the command FN_EXTERN void cmdBegin(void); // Get the command options as a string. This is output in cmdBegin() if the log level is high enough but can also be used for // debugging or error reporting later on. FN_EXTERN const String *cmdOption(void); // End the command FN_EXTERN void cmdEnd(int code, const String *errorMessage); // Execute an async process. This function will only return in the calling process and the implementation is platform dependent. FN_EXTERN void cmdAsyncExec(const char *command, const StringList *commandExec); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/control/0000775000175100017510000000000015240547630017300 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/control/common.c0000664000175100017510000000321615240547630020736 0ustar00runnerrunner/*********************************************************************************************************************************** Common Handler for Control Commands ***********************************************************************************************************************************/ #include #include "command/control/common.h" #include "common/debug.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN String * lockStopFileName(const String *const stanza) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, stanza); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( STRING, strNewFmt("%s/%s" STOP_FILE_EXT, strZ(cfgOptionStr(cfgOptLockPath)), stanza != NULL ? strZ(stanza) : "all")); } /**********************************************************************************************************************************/ FN_EXTERN void lockStopTest(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Check the current stanza (if any) if (cfgOptionTest(cfgOptStanza)) { if (storageExistsP(storageLocal(), lockStopFileName(cfgOptionStr(cfgOptStanza)))) THROW_FMT(StopError, "stop file exists for stanza %s", strZ(cfgOptionDisplay(cfgOptStanza))); } // Check all stanzas if (storageExistsP(storageLocal(), lockStopFileName(NULL))) THROW(StopError, "stop file exists for all stanzas"); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/control/common.h0000664000175100017510000000226215240547630020743 0ustar00runnerrunner/*********************************************************************************************************************************** Common Handler for Control Commands ***********************************************************************************************************************************/ #ifndef COMMAND_CONTROL_COMMON_H #define COMMAND_CONTROL_COMMON_H #include "common/type/string.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define STOP_FILE_EXT ".stop" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Create the stop filename FN_EXTERN String *lockStopFileName(const String *stanza); // Test for the existence of a stop file FN_EXTERN void lockStopTest(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/control/start.c0000664000175100017510000000265215240547630020606 0ustar00runnerrunner/*********************************************************************************************************************************** Start Command ***********************************************************************************************************************************/ #include #include "command/control/common.h" #include "command/control/start.h" #include "common/debug.h" #include "config/config.h" #include "storage/helper.h" #include "storage/storage.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStart(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Remove the stop file so processes can run const String *const stopFile = lockStopFileName(cfgOptionStrNull(cfgOptStanza)); // If the stop file exists, then remove it if (storageExistsP(storageLocal(), stopFile)) { // If the file cannot be removed, storageRemove() will throw an error if the error is not ENOENT storageRemoveP(storageLocalWrite(), stopFile); } else { LOG_WARN_FMT( "stop file does not exist%s", (cfgOptionTest(cfgOptStanza) ? zNewFmt(" for stanza %s", strZ(cfgOptionDisplay(cfgOptStanza))) : "")); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/control/start.h0000664000175100017510000000122415240547630020605 0ustar00runnerrunner/*********************************************************************************************************************************** Start Command ***********************************************************************************************************************************/ #ifndef COMMAND_CONTROL_START_H #define COMMAND_CONTROL_START_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void cmdStart(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/control/stop.c0000664000175100017510000001062115240547630020431 0ustar00runnerrunner/*********************************************************************************************************************************** Stop Command ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include "command/control/common.h" #include "command/control/stop.h" #include "common/debug.h" #include "common/lock.h" #include "common/type/convert.h" #include "config/config.h" #include "storage/helper.h" #include "storage/storage.h" #include "storage/storage.intern.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStop(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { const String *const stopFile = lockStopFileName(cfgOptionStrNull(cfgOptStanza)); // If the stop file does not already exist, then create it if (!storageExistsP(storageLocal(), stopFile)) { // Create the lock path (ignore if already created) storagePathCreateP(storageLocalWrite(), strPath(stopFile), .mode = 0770); // Create the stop file with without truncating an existing file IoWrite *const stopWrite = storageWriteIo( storageNewWriteP(storageLocalWrite(), stopFile, .noAtomic = true, .noTruncate = true, .modePath = 0770)); ioWriteOpen(stopWrite); ioWriteClose(stopWrite); // If --force was specified then send term signals to running processes if (cfgOptionBool(cfgOptForce)) { const String *const lockPath = cfgOptionStr(cfgOptLockPath); const String *const stanzaPrefix = cfgOptionTest(cfgOptStanza) ? strNewFmt("%s-", strZ(cfgOptionStr(cfgOptStanza))) : NULL; const StringList *const lockPathFileList = strLstSort( storageListP(storageLocal(), lockPath, .errorOnMissing = true), sortOrderAsc); List *const pidList = lstNewP(sizeof(int), .comparator = lstComparatorInt); // Find each lock file and send term signals to the processes for (unsigned int lockPathFileIdx = 0; lockPathFileIdx < strLstSize(lockPathFileList); lockPathFileIdx++) { const String *const lockFile = strLstGet(lockPathFileList, lockPathFileIdx); // Skip any file that is not a lock file. Skip lock files for other stanzas if a stanza is provided. if (!strEndsWithZ(lockFile, LOCK_FILE_EXT) || (stanzaPrefix != NULL && !strBeginsWith(lockFile, stanzaPrefix))) continue; // Read the lock file const LockReadResult lockResult = lockReadP(lockFile, .remove = true); // If we cannot read the lock file for any reason then warn and continue to next file if (lockResult.status != lockReadStatusValid) { LOG_WARN_FMT("unable to read lock file %s/%s", strZ(lockPath), strZ(lockFile)); continue; } // If the process has already been killed then skip it. This is possible because a single process may be holding // multiple lock files. if (lstFind(pidList, &lockResult.data.processId) != NULL) continue; lstAdd(pidList, &lockResult.data.processId); // The lock file is valid so that means there is a running process -- send a term signal to the process if (kill(lockResult.data.processId, SIGTERM) != 0) LOG_WARN_FMT("unable to send term signal to process %d", lockResult.data.processId); else LOG_INFO_FMT("sent term signal to process %d", lockResult.data.processId); } } } else { LOG_WARN_FMT( "stop file already exists for %s", cfgOptionTest(cfgOptStanza) ? zNewFmt("stanza %s", strZ(cfgOptionDisplay(cfgOptStanza))) : "all stanzas"); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/control/stop.h0000664000175100017510000000122015240547630020431 0ustar00runnerrunner/*********************************************************************************************************************************** Stop Command ***********************************************************************************************************************************/ #ifndef COMMAND_CONTROL_STOP_H #define COMMAND_CONTROL_STOP_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void cmdStop(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/exit.c0000664000175100017510000001355515240547630016746 0ustar00runnerrunner/*********************************************************************************************************************************** Exit Routines ***********************************************************************************************************************************/ #include #include #include #include #include "command/command.h" #include "command/exit.h" #include "command/lock.h" #include "common/debug.h" #include "common/log.h" #include "config/config.intern.h" #include "protocol/helper.h" #include "version.h" /*********************************************************************************************************************************** Indicates that exit() is in progress so a signal cannot call exit() recursively, which is undefined behavior and crashes on musl libc >= 1.2.6 ***********************************************************************************************************************************/ static volatile sig_atomic_t exitInProgress = false; /*********************************************************************************************************************************** Return signal names ***********************************************************************************************************************************/ static const char * exitSignalName(const SignalType signalType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, signalType); FUNCTION_TEST_END(); const char *name = NULL; switch (signalType) { case signalTypeHup: name = "SIGHUP"; break; case signalTypeInt: name = "SIGINT"; break; case signalTypeTerm: name = "SIGTERM"; break; case signalTypeNone: THROW(AssertError, "no name for signal none"); } FUNCTION_TEST_RETURN_CONST(STRINGZ, name); } /*********************************************************************************************************************************** Catch signals ***********************************************************************************************************************************/ static void exitOnSignal(const int signalType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, signalType); FUNCTION_TEST_END(); // Ignore the signal when exit() is already in progress and allow it to complete if (!exitInProgress) { exitInProgress = true; logSignal( cfgLogLevelDefault(), signalType == signalTypeNone ? "from child process" : exitSignalName((SignalType)signalType)); exit(errorTypeCode(&TermError)); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void exitInit(void) { FUNCTION_LOG_VOID(logLevelTrace); // Reset exit in progress in case this process was forked by a process that was exiting exitInProgress = false; signal(SIGHUP, exitOnSignal); signal(SIGINT, exitOnSignal); signal(SIGTERM, exitOnSignal); // Ignore SIGPIPE and check for EPIPE errors on write() instead signal(SIGPIPE, SIG_IGN); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to provide details for error logging static String * exitErrorDetail(void) { FUNCTION_TEST_VOID(); String *const role = cfgCommandRoleName(); String *const result = strNewFmt( "--------------------------------------------------------------------\n" "If SUBMITTING AN ISSUE please provide the following information:\n" "\n" "version: " PROJECT_VERSION "\n" "command: %s\n" "options:%s\n" "\n" "stack trace:\n%s\n" "--------------------------------------------------------------------", strZ(role), strZ(cmdOption()), errorStackTrace()); strFree(role); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN int exitSafe(int result, const bool error, const SignalType signalType) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INT, result); FUNCTION_LOG_PARAM(BOOL, error); FUNCTION_LOG_PARAM(ENUM, signalType); FUNCTION_LOG_END(); // Report error if one was thrown if (error) { LogLevel logLevel = errorCode() == errorTypeCode(&AssertError) ? logLevelAssert : logLevelError; // Output stack track on assert or when an output (e.g. console) is at least debug level if (logLevel == logLevelAssert || logAny(logLevelDebug)) { String *const detail = exitErrorDetail(); LOG_INTERNAL_FMT( logLevel, logLevel == logLevelAssert ? LOG_LEVEL_MIN : logLevelDebug, LOG_LEVEL_MAX, 0, errorCode(), "%s\n%s", errorMessage(), strZ(detail)); strFree(detail); } // Log just the error to non-debug levels if (logLevel != logLevelAssert) LOG_INTERNAL(logLevel, LOG_LEVEL_MIN, logLevelDetail, 0, errorCode(), errorMessage()); result = errorCode(); } // On error generate an error message for command end when config is initialized if (cfgInited()) { String *errorMessage = NULL; if (result != 0 && error) errorMessage = strNewFmt("aborted with exception [%03d]", result); cmdEnd(result, errorMessage); strFree(errorMessage); } // Release any locks but ignore errors TRY_BEGIN() { cmdLockReleaseP(.returnOnNoLock = true); } TRY_END(); // Set exit in progress so a signal arriving while exit() runs will not call exit() recursively exitInProgress = true; // Return result - caller should immediate pass this result to exit() FUNCTION_LOG_RETURN(INT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/exit.h0000664000175100017510000000232415240547630016743 0ustar00runnerrunner/*********************************************************************************************************************************** Exit Routines ***********************************************************************************************************************************/ #ifndef COMMAND_EXIT_H #define COMMAND_EXIT_H #include #include /*********************************************************************************************************************************** Signal type ***********************************************************************************************************************************/ typedef enum { signalTypeNone = 0, signalTypeHup = SIGHUP, signalTypeInt = SIGINT, signalTypeTerm = SIGTERM, } SignalType; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Setup signal handlers FN_EXTERN void exitInit(void); // Do cleanup and return result code FN_EXTERN int exitSafe(int result, bool error, SignalType signalType); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/expire/0000775000175100017510000000000015240547630017114 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/expire/expire.c0000664000175100017510000017506515240547630020572 0ustar00runnerrunner/*********************************************************************************************************************************** Expire Command ***********************************************************************************************************************************/ #include #include "command/archive/common.h" #include "command/backup/common.h" #include "command/control/common.h" #include "command/expire/expire.h" #include "common/debug.h" #include "common/regExp.h" #include "common/time.h" #include "common/type/list.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "protocol/helper.h" #include "storage/helper.h" #include /*********************************************************************************************************************************** Helper functions and structures ***********************************************************************************************************************************/ typedef struct ArchiveExpired { uint64_t total; String *start; String *stop; } ArchiveExpired; typedef struct ArchiveRange { const String *start; const String *stop; } ArchiveRange; /*********************************************************************************************************************************** Given a backup label, expire a backup and all its dependents (if any). ***********************************************************************************************************************************/ static StringList * expireBackup(InfoBackup *const infoBackup, const String *const backupLabel, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); ASSERT(backupLabel != NULL); // Return a list of all backups being expired StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Get the backup and all its dependents sorted from newest to oldest - that way if the process is aborted before all // dependencies have been dealt with, the backups remaining should still be usable. const StringList *const backupList = strLstSort(infoBackupDataDependentList(infoBackup, backupLabel), sortOrderDesc); // Expire each backup in the list for (unsigned int backupIdx = 0; backupIdx < strLstSize(backupList); backupIdx++) { const String *const removeBackupLabel = strLstGet(backupList, backupIdx); // Execute the real expiration and deletion only if the dry-run option is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { // Remove the manifest files to invalidate the backup storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(removeBackupLabel))); storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE INFO_COPY_EXT, strZ(removeBackupLabel))); } // Remove the backup from the info object infoBackupDataDelete(infoBackup, removeBackupLabel); strLstAdd(result, removeBackupLabel); } // Sort from oldest to newest and return the list of expired backups strLstSort(result, sortOrderAsc); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Function to expire a selected backup (and all its dependents) regardless of retention rules. ***********************************************************************************************************************************/ static unsigned int expireAdhocBackup(InfoBackup *const infoBackup, const String *const backupLabel, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(STRING, backupLabel); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); ASSERT(backupLabel != NULL); unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Get a list of all full backups with most recent in position 0 const StringList *const fullList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true)), sortOrderDesc); // If the requested backup to expire is the latest full backup if (strCmp(strLstGet(fullList, 0), backupLabel) == 0) { // If the latest full backup requested is the only backup or the prior full backup is not for the same db-id // then the backup requested cannot be expired if (strLstSize(fullList) == 1 || infoBackupDataByLabel(infoBackup, backupLabel)->backupPgId != infoBackupDataByLabel(infoBackup, strLstGet(fullList, 1))->backupPgId) { THROW_FMT( BackupSetInvalidError, "full backup %s cannot be expired until another full backup has been created on this repo", strZ(backupLabel)); } } // Save off what is currently the latest backup (it may be removed if it is the adhoc backup or is a dependent of the // adhoc backup const String *const latestBackup = infoBackupData(infoBackup, infoBackupDataTotal(infoBackup) - 1).backupLabel; // Expire the requested backup and any dependents const StringList *const backupExpired = expireBackup(infoBackup, backupLabel, repoIdx); // If the latest backup was removed, then update the latest link if not a dry-run if (!infoBackupLabelExists(infoBackup, latestBackup)) { // If retention settings have been configured, then there may be holes in the archives. For example, if the archive // for db-id=1 has 01,02,03,04,05 and F1 backup has archive start-stop 02-03 and retention-full=1 // (hence retention-archive=1 and retention-archive-type=full), then when F2 backup is created and assuming its // archive start-stop=05-06 then archives 01 and 04 will be removed resulting in F1 not being able to play through // PITR, which is expected. Now adhoc expire is attempted on F2 - it will be allowed but now there will be no // backups that can be recovered through PITR until the next full backup is created. Same problem for differential // backups with retention-diff. LOG_WARN_FMT( "%s: expiring latest backup %s - the ability to perform point-in-time-recovery (PITR) may be affected\n" "HINT: non-default settings for '%s'/'%s' (even in prior expires) can cause gaps in the WAL.", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(latestBackup), cfgOptionIdxName(cfgOptRepoRetentionArchive, repoIdx), cfgOptionIdxName(cfgOptRepoRetentionArchiveType, repoIdx)); // Adhoc expire is never performed through backup command so only check to determine if dry-run has been set or not if (!cfgOptionBool(cfgOptDryRun)) backupLinkLatest(infoBackupData(infoBackup, infoBackupDataTotal(infoBackup) - 1).backupLabel, repoIdx); } result = strLstSize(backupExpired); // Log the expired backup list (prepend "set:" if there were any dependents that were also expired) LOG_INFO_FMT( "%s: expire adhoc backup %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), (result > 1 ? "set " : ""), strZ(strLstJoin(backupExpired, ", "))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /*********************************************************************************************************************************** Expire differential backups ***********************************************************************************************************************************/ static unsigned int expireDiffBackup(InfoBackup *const infoBackup, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); // Total of all backups being expired unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { const unsigned int differentialRetention = cfgOptionIdxTest(cfgOptRepoRetentionDiff, repoIdx) ? cfgOptionIdxUInt(cfgOptRepoRetentionDiff, repoIdx) : 0; // Find all the expired differential backups if (differentialRetention > 0) { // Get a list of full and differential backups. Full are considered differential for the purpose of retention. // Example: F1, D1, D2, F2, repo-retention-diff=2, then F1,D2,F2 will be retained, not D2 and D1 as might be expected. const StringList *const currentBackupList = infoBackupDataLabelList( infoBackup, backupRegExpP(.full = true, .differential = true)); // If there are more backups than the number to retain, then expire the oldest ones if (strLstSize(currentBackupList) > differentialRetention) { for (unsigned int diffIdx = 0; diffIdx < strLstSize(currentBackupList) - differentialRetention; diffIdx++) { // Skip if this is a full backup. Full backups only count as differential when deciding which differential // backups to expire. if (regExpMatchOne(backupRegExpP(.full = true), strLstGet(currentBackupList, diffIdx))) continue; // Expire the differential and any dependent backups const StringList *const backupExpired = expireBackup( infoBackup, strLstGet(currentBackupList, diffIdx), repoIdx); result += strLstSize(backupExpired); // Log the expired backups. If there is more than one backup, then prepend "set:" LOG_INFO_FMT( "%s: expire diff backup %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), (strLstSize(backupExpired) > 1 ? "set " : ""), strZ(strLstJoin(backupExpired, ", "))); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /*********************************************************************************************************************************** Expire full backups ***********************************************************************************************************************************/ static unsigned int expireFullBackup(InfoBackup *const infoBackup, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); // Total of all backups being expired unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { const unsigned int fullRetention = cfgOptionIdxTest(cfgOptRepoRetentionFull, repoIdx) ? cfgOptionIdxUInt(cfgOptRepoRetentionFull, repoIdx) : 0; // Find all the expired full backups if (fullRetention > 0) { // Get list of current full backups (default order is oldest to newest) const StringList *const currentBackupList = infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true)); // If there are more full backups then the number to retain, then expire the oldest ones if (strLstSize(currentBackupList) > fullRetention) { // Expire all backups that depend on the full backup for (unsigned int fullIdx = 0; fullIdx < strLstSize(currentBackupList) - fullRetention; fullIdx++) { // Expire the full backup and all its dependents const StringList *const backupExpired = expireBackup( infoBackup, strLstGet(currentBackupList, fullIdx), repoIdx); result += strLstSize(backupExpired); // Log the expired backups. If there is more than one backup, then prepend "set:" LOG_INFO_FMT( "%s: expire full backup %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), (strLstSize(backupExpired) > 1 ? "set " : ""), strZ(strLstJoin(backupExpired, ", "))); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /*********************************************************************************************************************************** Expire backups based on time ***********************************************************************************************************************************/ static unsigned int expireTimeBasedBackup(InfoBackup *const infoBackup, const time_t minTimestamp, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(TIME, minTimestamp); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); ASSERT(minTimestamp > 0); unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Get the list of full backups const StringList *const currentBackupList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true)), sortOrderAsc); unsigned int backupIdx = strLstSize(currentBackupList); // Find out the point where we will have to stop purging backups. Starting with the newest backup (the end of the list), // find the first backup that is older than the expire time period by checking the backup stop time. This way, if the // backups are F1 D1a D1b D1c F2 D2a D2b F3 D3a D3b and the expiration time period is at D2b, then purge only F1, D1a, D1b // and D1c, and keep the next full backups (F2 and F3) and all intermediate non-full backups. if (backupIdx > 0) { const String *retentionMetBackupLabel = NULL; do { backupIdx--; InfoBackupData *info = infoBackupDataByLabel(infoBackup, strLstGet(currentBackupList, backupIdx)); // We can start deleting before this backup. This way, we keep one full backup and its dependents. if (info->backupTimestampStop < minTimestamp) { retentionMetBackupLabel = info->backupLabel; break; } } while (backupIdx != 0); // If retention has not been met there is nothing to expire if (retentionMetBackupLabel != NULL) { // Count number of full backups being expired unsigned int numFullExpired = 0; // Since expireBackup() will remove the requested entry from the backup list, we keep checking the first entry which // is always the oldest so if it is not the backup to retain then we can remove it while (!strEq(infoBackupData(infoBackup, 0).backupLabel, retentionMetBackupLabel)) { const StringList *const backupExpired = expireBackup( infoBackup, infoBackupData(infoBackup, 0).backupLabel, repoIdx); result += strLstSize(backupExpired); numFullExpired++; // Log the expired backups. If there is more than one backup, then prepend "set:" LOG_INFO_FMT( "%s: expire time-based backup %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), (strLstSize(backupExpired) > 1 ? "set " : ""), strZ(strLstJoin(backupExpired, ", "))); } // Set archive retention (if it is not already set) based on the number of full backups retained if (cfgOptionIdxStrId(cfgOptRepoRetentionArchiveType, repoIdx) == backupTypeFull && !cfgOptionIdxTest(cfgOptRepoRetentionArchive, repoIdx)) { cfgOptionIdxSet( cfgOptRepoRetentionArchive, repoIdx, cfgSourceDefault, VARINT64(strLstSize(currentBackupList) - numFullExpired)); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /*********************************************************************************************************************************** Log detailed information about archive logs removed ***********************************************************************************************************************************/ static void logExpire(ArchiveExpired *const archiveExpire, const String *const archiveId, const unsigned int repoIdx) { if (archiveExpire->start != NULL) { // Force out any remaining message LOG_INFO_FMT( "%s: %s remove archive, start = %s, stop = %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId), strZ(archiveExpire->start), strZ(archiveExpire->stop)); archiveExpire->start = NULL; } } /*********************************************************************************************************************************** Expire WAL for a single archive id, removing everything not covered by the supplied retention ranges ***********************************************************************************************************************************/ static void expireArchiveId( const String *const archiveId, const List *const archiveRangeList, const String *const archiveExpireMax, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(LIST, archiveRangeList); FUNCTION_LOG_PARAM(STRING, archiveExpireMax); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(archiveId != NULL); ASSERT(archiveRangeList != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Initialize the expired archive information for this archive id ArchiveExpired archiveExpire = {.total = 0, .start = NULL, .stop = NULL}; // Get all major archive paths (timeline and first 32 bits of LSN) const StringList *const walPathList = strLstSort( storageListP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId)), .expression = STRDEF(WAL_SEGMENT_DIR_REGEXP)), sortOrderAsc); for (unsigned int walIdx = 0; walIdx < strLstSize(walPathList); walIdx++) { const String *const walPath = strLstGet(walPathList, walIdx); bool removeArchive = true; // Keep the path if it falls in the range of any backup in retention for (unsigned int rangeIdx = 0; rangeIdx < lstSize(archiveRangeList); rangeIdx++) { const ArchiveRange *const archiveRange = lstGet(archiveRangeList, rangeIdx); if (strCmp(walPath, strSubN(archiveRange->start, 0, 16)) >= 0 && (archiveRange->stop == NULL || strCmp(walPath, strSubN(archiveRange->stop, 0, 16)) <= 0)) { removeArchive = false; break; } } // Remove the entire directory if all archive is expired if (removeArchive) { // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storagePathRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(archiveId), strZ(walPath)), .recurse = true); } LOG_DETAIL_FMT( "%s: %s remove archive path %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId), strZ(walPath)); archiveExpire.total++; archiveExpire.stop = strDup(walPath); // Update start for the first WAL path removed if (archiveExpire.start == NULL) archiveExpire.start = strDup(walPath); } // Else delete individual files instead if the major path is at or below the scan boundary. This optimization prevents // scanning through major paths that could not possibly have anything to expire. When there is no retention backup // (archiveExpireMax is NULL, e.g. archive-expire-before with no backup) the optimization does not apply, so scan // whenever the path was not removed in bulk. else if (archiveExpireMax == NULL || strCmp(walPath, strSubN(archiveExpireMax, 0, 16)) <= 0) { // Look for files in the archive directory const StringList *const walSubPathList = strLstSort( storageListP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(archiveId), strZ(walPath)), .expression = STRDEF("^[0-F]{24}.*$")), sortOrderAsc); for (unsigned int subIdx = 0; subIdx < strLstSize(walSubPathList); subIdx++) { removeArchive = true; const String *const walSubPath = strLstGet(walSubPathList, subIdx); // Determine if the individual archive log is used in a backup for (unsigned int rangeIdx = 0; rangeIdx < lstSize(archiveRangeList); rangeIdx++) { const ArchiveRange *const archiveRange = lstGet(archiveRangeList, rangeIdx); if (strCmp(strSubN(walSubPath, 0, 24), archiveRange->start) >= 0 && (archiveRange->stop == NULL || strCmp(strSubN(walSubPath, 0, 24), archiveRange->stop) <= 0)) { removeArchive = false; break; } } // Remove archive log if it is not used in a backup if (removeArchive) { // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s/%s", strZ(archiveId), strZ(walPath), strZ(walSubPath))); } // Track that this archive was removed archiveExpire.total++; archiveExpire.stop = strDup(strSubN(walSubPath, 0, 24)); if (archiveExpire.start == NULL) archiveExpire.start = strDup(strSubN(walSubPath, 0, 24)); } else logExpire(&archiveExpire, archiveId, repoIdx); } } } // Log if no archive was expired if (archiveExpire.total == 0) LOG_INFO_FMT("%s: %s no archive to remove", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId)); // Log if there is more to log else logExpire(&archiveExpire, archiveId, repoIdx); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Process archive retention ***********************************************************************************************************************************/ static void removeExpiredArchive(const InfoBackup *const infoBackup, const bool timeBasedFullRetention, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(BOOL, timeBasedFullRetention); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Get the retention options. repo-archive-retention-type always has a value as it defaults to "full" const BackupType archiveRetentionType = (BackupType)cfgOptionIdxStrId(cfgOptRepoRetentionArchiveType, repoIdx); const unsigned int archiveRetention = cfgOptionIdxTest( cfgOptRepoRetentionArchive, repoIdx) ? cfgOptionIdxUInt(cfgOptRepoRetentionArchive, repoIdx) : 0; // If archive retention is undefined, then ignore archiving. The user does not have to set this - it will be defaulted in // cfgLoadUpdateOption based on certain rules. if (archiveRetention == 0) { const String *const msg = strNewZ("- archive logs will not be expired"); // Only notify user if not time-based retention if (!timeBasedFullRetention) LOG_INFO_FMT("option '%s' is not set %s", cfgOptionIdxName(cfgOptRepoRetentionArchive, repoIdx), strZ(msg)); else { LOG_INFO_FMT( "%s: time-based archive retention not met %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(msg)); } } else { // Determine which backup type to use for archive retention (full, differential, incremental) and get a list of the // remaining non-expired backups, from newest to oldest, based on the type. const StringList *globalBackupRetentionList = NULL; switch (archiveRetentionType) { case backupTypeFull: globalBackupRetentionList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true)), sortOrderDesc); break; case backupTypeDiff: globalBackupRetentionList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true, .differential = true)), sortOrderDesc); break; case backupTypeIncr: // Incrementals can depend on Full or Diff so get a list of all incrementals globalBackupRetentionList = strLstSort( infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true, .differential = true, .incremental = true)), sortOrderDesc); break; } // Cap removal at the requested segment when --archive-expire-before is set: only WAL strictly before it may be removed const String *const walExpireBefore = cfgOptionTest(cfgOptArchiveExpireBefore) ? cfgOptionStr(cfgOptArchiveExpireBefore) : NULL; // Expire archives. If no backups were found or the number of backups found is not enough to satisfy archive retention // then preserve current archive logs - too soon to expire them. if (walExpireBefore != NULL || (!strLstEmpty(globalBackupRetentionList) && archiveRetention <= strLstSize(globalBackupRetentionList))) { // Attempt to load the archive info file const InfoArchive *const infoArchive = infoArchiveLoadFile( storageRepoIdx(repoIdx), INFO_ARCHIVE_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); const InfoPg *const infoArchivePgData = infoArchivePg(infoArchive); // Get a list of archive directories (e.g. 9.6-1, 10-2, etc) sorted by the db-id (number after the dash). const StringList *const listArchiveDisk = strLstSort( strLstComparatorSet( storageListP( storageRepoIdx(repoIdx), STORAGE_REPO_ARCHIVE_STR, .expression = STRDEF(REGEX_ARCHIVE_DIR_DB_VERSION)), archiveIdComparator), sortOrderAsc); StringList *const globalBackupArchiveRetentionList = strLstNew(); // globalBackupRetentionList is ordered newest to oldest backup, so create globalBackupArchiveRetentionList of the // newest backups whose archives will be retained const unsigned int retentionSize = archiveRetention < strLstSize(globalBackupRetentionList) ? archiveRetention : strLstSize(globalBackupRetentionList); for (unsigned int retentionIdx = 0; retentionIdx < retentionSize; retentionIdx++) strLstAdd(globalBackupArchiveRetentionList, strLstGet(globalBackupRetentionList, retentionIdx)); // From newest to oldest, confirm the pgVersion and pgSystemId from the archive.info history id match that of the // same history id of the backup.info and if not, there is a mismatch between the info files so do not continue. // NOTE: If the archive.info file was reconstructed and contains history 4: version 10, sys-id 456 and history 3: // version 9.6, sys-id 123, but backup.info contains history 4 and 3 (identical to archive.info) but also 2 and 1 // then an error will not be thrown since only archive 4 and 3 would have existed and the others expired so // nothing really to do. for (unsigned int infoPgIdx = 0; infoPgIdx < infoPgDataTotal(infoArchivePgData); infoPgIdx++) { const InfoPgData archiveInfoPgHistory = infoPgData(infoArchivePgData, infoPgIdx); const InfoPgData backupInfoPgHistory = infoPgData(infoBackupPg(infoBackup), infoPgIdx); if (archiveInfoPgHistory.id != backupInfoPgHistory.id || archiveInfoPgHistory.systemId != backupInfoPgHistory.systemId || archiveInfoPgHistory.version != backupInfoPgHistory.version) { THROW( FormatError, "archive expiration cannot continue - archive and backup history lists do not match"); } } // Loop through the archive.info history from oldest to newest and if there is a corresponding directory on disk // then remove WAL that are not part of retention as long as the db:history id version/system-id matches backup.info for (unsigned int pgIdx = infoPgDataTotal(infoArchivePgData) - 1; (int)pgIdx >= 0; pgIdx--) { const String *const archiveId = infoPgArchiveId(infoArchivePgData, pgIdx); StringList *const localBackupRetentionList = strLstNew(); for (unsigned int archiveIdx = 0; archiveIdx < strLstSize(listArchiveDisk); archiveIdx++) { // Is there an archive directory for this archiveId? If not, move on to the next. if (strCmp(archiveId, strLstGet(listArchiveDisk, archiveIdx)) != 0) continue; const unsigned int archivePgId = cvtZToUInt(strrchr(strZ(archiveId), '-') + 1); // From the global list of backups to retain, create a list of backups, oldest to newest, associated with // this archiveId (e.g. 9.6-1), e.g. If globalBackupRetention has 4F, 3F, 2F, 1F then // localBackupRetentionList will have 1F, 2F, 3F, 4F (assuming they all have same history id) for (unsigned int retentionIdx = strLstSize(globalBackupRetentionList) - 1; (int)retentionIdx >= 0; retentionIdx--) { for (unsigned int backupIdx = 0; backupIdx < infoBackupDataTotal(infoBackup); backupIdx++) { InfoBackupData backupData = infoBackupData(infoBackup, backupIdx); if (strCmp(backupData.backupLabel, strLstGet(globalBackupRetentionList, retentionIdx)) == 0 && backupData.backupPgId == archivePgId) { strLstAdd(localBackupRetentionList, backupData.backupLabel); } } } // If no backup to retain was found if (strLstEmpty(localBackupRetentionList)) { const InfoPgData currentPg = infoPgDataCurrent(infoArchivePgData); // If archive-expire-before is set, remove only WAL earlier than the requested segment, keeping the // segment and everything after it (still required to restart recovery). Model the boundary as a single // open range [walExpireBefore, NULL] so the shared expiration logic keeps all WAL >= the segment. if (walExpireBefore != NULL && currentPg.id == archivePgId) { List *const archiveRangeList = lstNewP(sizeof(ArchiveRange)); ArchiveRange archiveRange = {.start = strDup(walExpireBefore), .stop = NULL}; lstAdd(archiveRangeList, &archiveRange); expireArchiveId(archiveId, archiveRangeList, NULL, repoIdx); } // Otherwise, if this is not the current database, delete the whole archive directory. The current DB // archive directory is the live destination for archive-push and must not be deleted. else if (currentPg.id != archivePgId) { String *fullPath = storagePathP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId))); LOG_INFO_FMT( "%s: remove archive path %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(fullPath)); // Execute the real expiration and deletion only if the dry-run option is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) storagePathRemoveP(storageRepoIdxWrite(repoIdx), fullPath, .recurse = true); } // Continue to next directory break; } // If we get here, then a local backup was found for retention StringList *const localBackupArchiveRetentionList = strLstNew(); // From the full list of backups in archive retention, find the intersection of local backups to retain // from oldest to newest for (unsigned int globalIdx = strLstSize(globalBackupArchiveRetentionList) - 1; (int)globalIdx >= 0; globalIdx--) { for (unsigned int localIdx = 0; localIdx < strLstSize(localBackupRetentionList); localIdx++) { if (strCmp( strLstGet(globalBackupArchiveRetentionList, globalIdx), strLstGet(localBackupRetentionList, localIdx)) == 0) { strLstAdd(localBackupArchiveRetentionList, strLstGet(localBackupRetentionList, localIdx)); } } } // If no local backups were found as part of retention then set the backup archive retention to the newest // backup so that the database is fully recoverable (can be recovered from the last backup through pitr) if (strLstEmpty(localBackupArchiveRetentionList)) { strLstAdd( localBackupArchiveRetentionList, strLstGet(localBackupRetentionList, strLstSize(localBackupRetentionList) - 1)); } // Get the data for the backup selected for retention and all backups associated with this archive id List *const archiveIdBackupList = lstNewP(sizeof(InfoBackupData)); InfoBackupData archiveRetentionBackup = {0}; for (unsigned int infoBackupIdx = 0; infoBackupIdx < infoBackupDataTotal(infoBackup); infoBackupIdx++) { const InfoBackupData archiveIdBackup = infoBackupData(infoBackup, infoBackupIdx); // If this is the backup selected for retention, store its data if (strCmp(archiveIdBackup.backupLabel, strLstGet(localBackupArchiveRetentionList, 0)) == 0) archiveRetentionBackup = infoBackupData(infoBackup, infoBackupIdx); // If this is a backup associated with this archive Id, then add it to the list to check if (archiveIdBackup.backupPgId == archivePgId) lstAdd(archiveIdBackupList, &archiveIdBackup); } // Only expire if the selected backup has archive data - backups performed with --no-online will // not have archive data and cannot be used for expiration. if (archiveRetentionBackup.backupArchiveStart != NULL) { // Get archive ranges to preserve. Because archive retention can be less than total retention it is // important to preserve archive that is required to make the older backups consistent even though they // cannot be played any further forward with PITR. const String *archiveExpireMax = NULL; List *const archiveRangeList = lstNewP(sizeof(ArchiveRange)); // From the full list of backups, loop through those associated with this archiveId for (unsigned int backupListIdx = 0; backupListIdx < lstSize(archiveIdBackupList); backupListIdx++) { const InfoBackupData *const backupData = lstGet(archiveIdBackupList, backupListIdx); // If the backup is earlier than or the same as the retention backup and the backup has an // archive start if (strCmp(backupData->backupLabel, archiveRetentionBackup.backupLabel) <= 0 && backupData->backupArchiveStart != NULL) { ArchiveRange archiveRange = { .start = strDup(backupData->backupArchiveStart), .stop = NULL, }; // If this is not the retention backup, then set the stop, otherwise set the expire max to // the archive start of the archive to retain if (strCmp(backupData->backupLabel, archiveRetentionBackup.backupLabel) != 0) archiveRange.stop = strDup(backupData->backupArchiveStop); else archiveExpireMax = strDup(archiveRange.start); LOG_DETAIL_FMT( "%s: %s archive retention on backup %s, start = %s%s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId), strZ(backupData->backupLabel), strZ(archiveRange.start), archiveRange.stop != NULL ? zNewFmt(", stop = %s", strZ(archiveRange.stop)) : ""); // Add the archive range to the list lstAdd(archiveRangeList, &archiveRange); } } // If --archive-expire-before is set, preserve everything from the requested segment up to the retention // backup start, so that only WAL strictly before the segment is eligible for removal. The condition // walExpireBefore < archiveExpireMax avoids an inverted range and skips the case where the segment is // newer than the retention backup start (nothing extra to preserve there). archiveExpireMax is always // set here: the retention backup is in archiveIdBackupList and has an archive start (checked above). if (walExpireBefore != NULL && strCmp(walExpireBefore, archiveExpireMax) < 0) { ArchiveRange archiveRange = { .start = strDup(walExpireBefore), .stop = strDup(archiveExpireMax), }; lstAdd(archiveRangeList, &archiveRange); } // Expire WAL for this archive id according to the retention ranges computed above expireArchiveId(archiveId, archiveRangeList, archiveExpireMax, repoIdx); // Look for history files to expire based on the timeline of backupArchiveStart const String *const backupArchiveStartTimeline = strSubN( archiveRetentionBackup.backupArchiveStart, 0, 8); const StringList *const historyFilesList = strLstSort( storageListP( storageRepoIdx(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId)), .expression = WAL_TIMELINE_HISTORY_REGEXP_STR), sortOrderAsc); for (unsigned int historyFileIdx = 0; historyFileIdx < strLstSize(historyFilesList); historyFileIdx++) { const String *const historyFile = strLstGet(historyFilesList, historyFileIdx); // Expire history files older than the oldest retained timeline if (strCmp(strSubN(historyFile, 0, 8), backupArchiveStartTimeline) < 0) { // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%s", strZ(archiveId), strZ(historyFile))); } LOG_INFO_FMT( "%s: %s remove history file %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(archiveId), strZ(historyFile)); } } } } } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Remove expired backups from repo ***********************************************************************************************************************************/ static void removeExpiredBackup(const InfoBackup *const infoBackup, const String *const adhocBackupLabel, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(STRING, adhocBackupLabel); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Get all the current backups in backup.info - these will not be expired const StringList *const currentBackupList = strLstSort(infoBackupDataLabelList(infoBackup, NULL), sortOrderDesc); // Get all the backups on disk const StringList *const backupList = strLstSort( storageListP( storageRepoIdx(repoIdx), STORAGE_REPO_BACKUP_STR, .expression = backupRegExpP(.full = true, .differential = true, .incremental = true)), sortOrderDesc); // Initialize the index to the latest backup on disk unsigned int backupIdx = 0; // The latest backup on disk may be resumable or in-progress (has a backup.manifest.copy but no backup.manifest). A backup // in this state must not be removed by default: it may be a crashed backup that the next backup will resume or clean, or it // may be actively running on another host (e.g. backup runs on the primary while expire runs on a standby). Removing it in // the latter case corrupts the in-progress backup. Only the latest backup can be in this state. if (!strLstEmpty(backupList)) { const String *const manifestFileName = strNewFmt( STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(strLstGet(backupList, 0))); const String *const manifestCopyFileName = strNewFmt("%s" INFO_COPY_EXT, strZ(manifestFileName)); // If the latest backup is resumable (has a backup.manifest.copy but no backup.manifest) if (!storageExistsP(storageRepoIdx(repoIdx), manifestFileName) && storageExistsP(storageRepoIdx(repoIdx), manifestCopyFileName)) { // Protect the resumable/in-progress backup by default backupIdx = 1; // During adhoc expiration the resumable backup may be a dependent of the label being expired, in which case it // should be removed along with it -- but only when its ancestor no longer exists in backup.info if (adhocBackupLabel != NULL && strBeginsWith(strLstGet(backupList, 0), strSubN(adhocBackupLabel, 0, 16))) { const Manifest *const manifestResume = manifestLoadFile( storageRepoIdx(repoIdx), manifestFileName, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), infoPgCipherPass(infoBackupPg(infoBackup))); // If the ancestor of the resumable backup no longer exists in backup.info then it can be removed if (!infoBackupLabelExists(infoBackup, manifestData(manifestResume)->backupLabelPrior)) backupIdx = 0; } } } // Remove non-current backups from disk for (; backupIdx < strLstSize(backupList); backupIdx++) { if (!strLstExists(currentBackupList, strLstGet(backupList, backupIdx))) { LOG_INFO_FMT( "%s: remove expired backup %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(strLstGet(backupList, backupIdx))); // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storagePathRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/%s", strZ(strLstGet(backupList, backupIdx))), .recurse = true); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Remove expired backup history manifests from repo ***********************************************************************************************************************************/ static void removeExpiredHistory(const InfoBackup *const infoBackup, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); MEM_CONTEXT_TEMP_BEGIN() { if (cfgOptionIdxTest(cfgOptRepoRetentionHistory, repoIdx)) { // Get current backups in backup.info - these will not be expired const StringList *const currentBackupList = strLstSort(infoBackupDataLabelList(infoBackup, NULL), sortOrderDesc); // If the full backup history manifests are expired, we should expire diff and incr too. So, format the oldest full // backup label to retain. const time_t minTimestamp = time(NULL) - (time_t)(cfgOptionIdxUInt(cfgOptRepoRetentionHistory, repoIdx) * SEC_PER_DAY); const String *const minBackupLabel = backupLabelFormat(backupTypeFull, NULL, minTimestamp); // Get all history years const StringList *const historyYearList = strLstSort( storageListP( storageRepo(), STRDEF(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY), .expression = STRDEF("^2[0-9]{3}$")), sortOrderAsc); for (unsigned int historyYearIdx = 0; historyYearIdx < strLstSize(historyYearList); historyYearIdx++) { // Get all the backup history manifests const String *const historyYear = strLstGet(historyYearList, historyYearIdx); // If the entire year is less than the year of the minimum backup to retain, then remove completely the directory if (strCmp(historyYear, strSubN(minBackupLabel, 0, 4)) < 0) { LOG_INFO_FMT( "%s: remove expired backup history path %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(historyYear)); // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storagePathRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s", strZ(historyYear)), .recurse = true); } } // Else find and remove individual files else if (strEq(historyYear, strSubN(minBackupLabel, 0, 4))) { const StringList *const historyList = strLstSort( storageListP( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s", strZ(historyYear)), .expression = strNewFmt( "%s\\.manifest\\.%s$", strZ(backupRegExpP(.full = true, .differential = true, .incremental = true, .noAnchorEnd = true)), strZ(compressTypeStr(compressTypeGz)))), sortOrderDesc); for (unsigned int historyIdx = 0; historyIdx < strLstSize(historyList); historyIdx++) { const String *const historyBackupFile = strLstGet(historyList, historyIdx); const String *const historyBackupLabel = strLstGet(strLstNewSplitZ(historyBackupFile, "."), 0); // Keep backup history manifests for unexpired backups and backups newer than the oldest backup to retain if (!strLstExists(currentBackupList, historyBackupLabel) && strCmp(historyBackupLabel, minBackupLabel) < 0) { LOG_INFO_FMT( "%s: remove expired backup history manifest %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(historyBackupFile)); // Execute the real expiration and deletion only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { storageRemoveP( storageRepoIdxWrite(repoIdx), strNewFmt( STORAGE_REPO_BACKUP "/" BACKUP_PATH_HISTORY "/%s/%s", strZ(historyYear), strZ(historyBackupFile))); } } } } else break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdExpire(void) { FUNCTION_LOG_VOID(logLevelDebug); // Test for stop file lockStopTest(); MEM_CONTEXT_TEMP_BEGIN() { // Initialize the repo index unsigned int repoIdxMin = 0; unsigned int repoIdxMax = cfgOptionGroupIdxTotal(cfgOptGrpRepo) - 1; // If the repo was specified then set index to the array location and max to loop only once if (cfgOptionTest(cfgOptRepo)) { repoIdxMin = cfgOptionGroupIdxDefault(cfgOptGrpRepo); repoIdxMax = repoIdxMin; } // Get the backup label if specified const String *adhocBackupLabel = NULL; bool adhocBackupFound = false; // Check if this run should expire the --oldest backup set const bool expireOldest = cfgOptionValid(cfgOptOldest) ? cfgOptionBool(cfgOptOldest) : false; // If the --set option is valid (i.e. expire is called on its own) then check the label format if (cfgOptionTest(cfgOptSet)) { adhocBackupLabel = cfgOptionStr(cfgOptSet); // Conflict check with the --oldest option if (expireOldest) THROW(OptionInvalidError, "--oldest and --set cannot be used together"); // If the label format is invalid, then error if (!regExpMatchOne(backupRegExpP(.full = true, .differential = true, .incremental = true), adhocBackupLabel)) THROW_FMT(OptionInvalidValueError, "'%s' is not a valid backup label format", strZ(adhocBackupLabel)); } // If --archive-expire-before is set, the value must be an exact WAL segment name. Match the strict segment expression // rather than walIsSegment(), which also accepts a .partial suffix that would shift the removal boundary by a segment. if (cfgOptionTest(cfgOptArchiveExpireBefore) && !regExpMatchOne(WAL_SEGMENT_REGEXP_STR, cfgOptionStr(cfgOptArchiveExpireBefore))) { THROW_FMT( OptionInvalidValueError, "'%s' is not a valid WAL segment name for option '--archive-expire-before'", strZ(cfgOptionStr(cfgOptArchiveExpireBefore))); } // Track any errors that may occur unsigned int errorTotal = 0; for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { // Get the repo storage in case it is remote and encryption settings need to be pulled down const Storage *const storageRepo = storageRepoIdx(repoIdx); InfoBackup *infoBackup = NULL; // Is full retention time-based? bool timeBasedFullRetention = cfgOptionIdxSeq(cfgOptRepoRetentionFullType, repoIdx) == CFGOPTVAL_REPO_RETENTION_FULL_TYPE_TIME; TRY_BEGIN() { // Load backup.info for this repo infoBackup = infoBackupLoadFileReconstruct( storageRepo, INFO_BACKUP_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); // When expire oldest requested, expire the oldest full backup by lowering retention for this execution if (expireOldest) { // Count full backups in this repo (we want to expire the oldest one) const unsigned int fullLstSize = strLstSize(infoBackupDataLabelList(infoBackup, backupRegExpP(.full = true))); // If there is more than one full, temporarily keep one fewer if (fullLstSize > 1) { // If time-based retention is configured, force count-based behaviour for this run if (timeBasedFullRetention) { LOG_INFO_FMT( "repo%u: expire oldest overrides configured time-based retention with count-based retention for" " this execution", repoIdx + 1); timeBasedFullRetention = false; } cfgOptionIdxSet(cfgOptRepoRetentionFull, repoIdx, cfgSourceParam, VARINT64(fullLstSize - 1)); // Also lower archive retention so WAL for the expired backups will be removed cfgOptionIdxSet(cfgOptRepoRetentionArchiveType, repoIdx, cfgSourceParam, VARSTRDEF("full")); cfgOptionIdxSet(cfgOptRepoRetentionArchive, repoIdx, cfgSourceParam, VARINT64(fullLstSize - 1)); LOG_DETAIL_FMT( "repo%u: expire oldest enforcing %s=" CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT_Z ", %s=%u, %s=" CFGOPTVAL_DETAIL_LEVEL_FULL_Z ", and %s=%u for this execution", repoIdx + 1, cfgOptionIdxName(cfgOptRepoRetentionFullType, repoIdx), cfgOptionIdxName(cfgOptRepoRetentionFull, repoIdx), fullLstSize - 1, cfgOptionIdxName(cfgOptRepoRetentionArchiveType, repoIdx), cfgOptionIdxName(cfgOptRepoRetentionArchive, repoIdx), fullLstSize - 1); } else LOG_WARN_FMT("repo%u: expire oldest requested but no eligible full backup to expire", repoIdx + 1); } // If a backupLabel was set, then attempt to expire the requested backup if (adhocBackupLabel != NULL) { if (infoBackupLabelExists(infoBackup, adhocBackupLabel)) { adhocBackupFound = true; expireAdhocBackup(infoBackup, adhocBackupLabel, repoIdx); } // If the adhoc backup was not found and this was the last repo to check, then log a warning but continue to // process the expiration based on retention if (!adhocBackupFound && repoIdx == repoIdxMax) { LOG_WARN_FMT( "backup %s does not exist\nHINT: run the info command and confirm the backup is listed", strZ(adhocBackupLabel)); } } else { // If time-based retention for full backups is set, then expire based on time period if (timeBasedFullRetention) { // If a time period was provided then run time-based expiration otherwise do nothing (the user has already // been warned by the config system that retention-full was not set) if (cfgOptionIdxTest(cfgOptRepoRetentionFull, repoIdx)) { expireTimeBasedBackup( infoBackup, time(NULL) - (time_t)(cfgOptionIdxUInt(cfgOptRepoRetentionFull, repoIdx) * SEC_PER_DAY), repoIdx); } } else expireFullBackup(infoBackup, repoIdx); expireDiffBackup(infoBackup, repoIdx); } // Store the new backup info only if the dry-run mode is disabled if (!cfgOptionValid(cfgOptDryRun) || !cfgOptionBool(cfgOptDryRun)) { infoBackupSaveFile( infoBackup, storageRepoIdxWrite(repoIdx), INFO_BACKUP_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); } // Remove all files on disk that are now expired removeExpiredBackup(infoBackup, adhocBackupLabel, repoIdx); removeExpiredArchive(infoBackup, timeBasedFullRetention, repoIdx); removeExpiredHistory(infoBackup, repoIdx); } CATCH_ANY() { LOG_ERROR_FMT(errorTypeCode(errorType()), "%s: %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorMessage()); errorTotal++; } TRY_END(); } // Error if any errors encountered on one or more repos if (errorTotal > 0) THROW_FMT(CommandError, CFGCMD_EXPIRE " command encountered %u error(s), check the log file for details", errorTotal); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/expire/expire.h0000664000175100017510000000126515240547630020565 0ustar00runnerrunner/*********************************************************************************************************************************** Expire Command ***********************************************************************************************************************************/ #ifndef COMMAND_EXPIRE_EXPIRE_H #define COMMAND_EXPIRE_EXPIRE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Expire backups and archives FN_EXTERN void cmdExpire(void); #endif ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7626204 pgbackrest-2.59.1/src/command/help/0000775000175100017510000000000015240547776016563 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7606204 pgbackrest-2.59.1/src/command/help/help.auto.c.inc0000640000175100017510000016734115240547776021404 0ustar00runnerrunner/*********************************************************************************************************************************** Help Data Automatically generated by 'build-code help' -- do not modify directly. ***********************************************************************************************************************************/ VR_NON_STRING static const char helpData[21523] = "BZh91AY&SY\365\004\371\?\000\001\336\177\377\377\377\377\377\377\377\377\377\277\377\377\376\377\377\377\377\?.\177\361\236;\213\3\ 74\377\276\347-\333\3769c\340s\335\354{\\\035\351E\355\253\345\240P\223\356\335\003\204\333\r\266\355\251\261\245\002\201Y\006@J\24\ 5J\000\022Z\326\231Kj\r!\335\316\231\000\000\032\002\254\315\025\323w\262\271Y\335\335\260\242\215\001\203{j\213\352l*\216\276\235w\ \233\330:\337N\340{\266n\355;\336\323\332\236\256\272f\332\000\326\330eM-m\334\337o\272m\265\252\000-[\350\000\326\337m\270\000N\23\ 5\231\336\367\227)\236\273\201n\367+\027k\256\354P\r\016\200\024\av\210\313m\247\337\a8z\rZ\322(>UcN\301j\347\247smx\356\200\320]\a\ J\247@\321\312\330\f\272t)U\025u5\244\240\350\356\316\rn\234\253[b\217\276\247}\265V\254\016\237s\337\005U(\003\267\213\330JE\003yu\ \313\\^^\366\366\244=\314\332\333cE\033\261v\353\336V\357\237z|\313g\276\203M\b\023 $\301&\bi\246Si\241M\350SL\201\352b44\r\000\03\ 2\000\000\r\000\000\000\000\000d\r\032\017\n2\r\205\r\350\232OM\245\?J\r2\004\020D\"i\2414j\233HOOQ\241\036\215C\021\210i\211\2200\\ 0010\230\206\001\030LL\201\246A\202b\031\006\200\323!\246&C&\004\321\247\244\323 `\220H\211\032%O\322mLD\323G\251\243C@\320\320\000\ \000\000\000\000\000\320\310\003 \320\000\000\000\000\000\032\000\000\000\000\000\bR\"\022b\004\320\t=\004\322o%\?S'\2514\031\001\2\ 46\207\250\r\000\001\241\246\214\215\a\250\000\000\000\000\000\000\000\000\000\000\000\000\002$\220\231\004\323L\223!\250\237\241\2\ 51\211\246\243L\023\t\212zi\251\2432&54zhL\230\215\250\332\236P\320\006\200\000\000\320\000\r\000\000\000\000\r\f\206\020$\324D\004\ F\202\236\204h\217T\375B\236Q\352z\200\036\220\001\240\000\000\000\000\000\000\r\000\000\000\000\000\000\000\000\000\000\001\247\32\ 3L'\301\?\005I_\234\266\217\323\337\2338\r\240Q\244>\314\344\374\237\331\376\202\353\2178\237I\364\177\003\320\276\374\262\316|\027\ \353\317\241\372/}c\341}\257\363\375\357\357|\237\320\377\317\340\374\257\306\373\317\207\373_*\337\v\3625\377'\306\374LMp\377\217\\ 215\237\226<\360A\364=\377\225\361>\037ko\257[\370\275\367\276\307\374\?\305\177\313\336\373\336\326^\217\253\333\363{\235n\257[\36\ 6\220\257\3568a\345u~\a\233\326\365|\255=OO\375y\376\217\301\361\037\263\360~\027\247\321\347\366=.\357\344!\341\367\370|.\317@z\03\ 0z\036\277\255\377>-\272\033\3767\201\347s}\247\235\361~'O\324\267\304\036\313\377}\217\307\342\355z=\240\235\270r\366\365z\336\247\ w\333\360<\217m\334\355\362u\375\267\221\335\373~\367\255\333\356z\375\325\273\236\277\027\254S\346\234a\030\277\362\331F0\2561\373\ ;\212\274\b@\215&\306\337\246\252\261j\003\033\274\313\374\277ow\353\212\372\017\342\304W\321\202\001\350\304\017\203=\375\036\223\\ 347g\373Z\256\210\231\364i\224\277=\n\252a]m0`\306]i\315fys{0\212\n\223m\005\234\312\177\372P\003\307\323\326\322G\255\006\341}D9\2\ 30\344d\261\025NE\031fe\024\323c\370\376L\217\251\303\"\bif\372\031\236@\035\030kF8\301\314\270\301\374Z\261\227\202\204\263\266d4Q\ s\214\270aCE\a\317\314))b(((]\220\247O\344\317M_\034\260\326\242l\023\357\234\374V\367\212\250\?\003\341\214\232\363\357){~\024\032\ \306\315\021L\334\245\342\004\333\016\243\0378\244p\?\263b2\t\307\377{\375\206hi\254\243\351\267\262\204\024y\224\023t\200\232\377o\ \251\264D\227\272\372\216xl\231C\232\260:\322\311rE(\222MH\260R\362\236\242\270\330M\257\020\324\352\264\367\365\200\344\263_\375\3\ 20AQ\334\250d\020U\263\257PX\?+\347#\277U\361\036\210g\265-P\2343Gou,\307(E\201\242\f\304\313C\245\225^\344\330#}\331\243\221\003\3\ 47\371\223>a3\245 \315\"'\271\3039\253(ml2\"\242\3023T\032\272\r\035>\004\227\025\032\347\262*\005\371\254\\$dh\202\211\v\207$\207\\ n\320u\bH\220\214\212P\310\330\032\021\214`\324\000\341M#\257(\361\342[)9\016#\251\201\231\t\355\352\0302-\235\366z0%S\206q\304k\?\\ 322\364]1\026\332\003\033Uk\231(\\\267i\225`\203vs>\235\206x\201\3130\223\336g\313\\\3670\325t\252l@k\355W\252)2\243\214u(\276\n\20\ 3Y\001Zf\364\360\230R\221\342\032%}\2661\305\305x\264x\372\301\337\aI\\g\206\213J\001\305B\205\031\306\213\233)I\235\205\211:/06\26\ 7\340\020\303:\337\217j\220K\246\004\026\025\021\353\230\372a\301\177\027\357\351\220\r8,\361\0013\216\261\270\226\016\245\374\350\\ 356\237\371\257\366\017\254\365T\317\227y\351/\310}:\001\350\2242,\"\257sH\020X\241\r\376\027\314\365\230\2442zo\247\302\367\326\33\ 3|X4\321\355\360]\247~\020.*\"\217\240$`\0327\3208\216l\255\353S\327\242h20\374G\303K`(\t(\t\320\341~qD\266\031Z7\344\367\302l\nb\2\ 41`\323:\236\352\305my\330\364\"\330\234\\^gV\240\033/\306\2444\030w\344e\3062&\254\022\b\377\370\320\204\242\337\327\271\252\252\0\ 35\350t\364\322\"\227#\200(^T2\274\247\234\234ju\004I\321\345,\320\024\210\325\316\270\\g\211\324\200\366t\313\202\321\225\241\221\\ 016\372\331Z\177\034)\034\363\206\203u\226\360`p>T5\241xCM\0228J\b\2129\230\225\277\204\001Y\253\254\t>\326\322\315\330\333\327\205\ \364G\034bF\244\apz>\335QK\317p3\351\347\002\376\243\364\330g\236K\316\226yw\363\251\002\261+\032\002\372\265\213\235TKa{\203\351IV\ \323\023\0038\300\314\270\300\?\003\302v\220\367\363\263\250\306.\224\310\255\036\313A\200\207\t\020s\211\034\3474\215\213\000\202F\ \213\a\326\004\326\246zwf\271\277f+\215\2\ 05\3078\310\202\321\360z\204F\035\310\320pF|E|\362&\300\323\323`\bP\256D&\223.]pyM|\374\324\026\302)=\2148)\305\205\345\202\021\201\ Q\212-\333\034\366\351\371\026\f(\025\025\005\223|\234\300\370\353n\303{\303\327\3065\315\?\031X4\233\221\225I\021!\t\br\034\003\35\ 4\357\252gvAk.c\006D\221\202\354\351JB~\332\034\220 \260\203z\207 \372\365\2703!\315^\240#\035O\207\033\332\305\342\350\312\204\002\ Y\206\227\305YmbC\350\265\362\256Q\024Q\a\005\002F\241\000\345Q\342\004\t,\222D\002\220\323\323\214\233#&X\264\035\257\2705KD\247vz\ \005\222\033\361\016\322\024\274\303\247d\221\334C\335tm3p\3262\335\265\273\023l\307\273Y\246\330FG6I\251\t^\236kR\244\343\204\325d\ ,\272u\321\221\266\306\335a[+\3153WZ\204\306\022A\232\226\351\2557p\204\033\226j\2675 \306\343Vh\307\247nf\260,\a\323\016\270O\363f\ <\330\353w\212\033(\022j\003\256\276d*\024\326a\?\223.c\337p\337\b\006\273\231r\233\324/^\246\241\247\323\?\370\340]\237%\311\225\0\ 04\365\244\316ab\026\257&\030o\3514\244\027\002xu\323\240n\210\332F\030\322\0167\0059\345~\275\026\223\306\224\345]\273\273\324\353\ {FO'\\=\261\032h;\355\250\306\331\234\035\265\a}S4\205\025B\210DTRp\020#2\357\363}T\363\354\250\303\f||C\217\265*1O\317\212\346\245\ \364\303JL\231\"L\334\313O\177]jE\323#\211\362\376\262\303\267\n\223Q~#\357\227\313\r\036x\303F\033\000$\207\276\020\177\225t3\325d\ \363]L4\rs\030\022Adxy\034V\022\267\266\203\356\032H\313\257\222Q\320\261\267\226\316:\234V\370wXd\017\375\006\027\203@{\252\250\03\ 1\224b3 CH\262\026X\016\234\036\022\354\355\"\217\306\275o\224\333Qx\362\215o\256\352\333\326WpcIywo\215\004\304\020 \212\245y\225\\ 210l\021\216\201Dd\262\024\202\254\313)\270\277(\325V\202asM\216E\237\201\330b\b6\026\2352\216\312\363\356\307H\237\245G\2518\351Ou\ \024\"y\2365-\212\"YT\"Q\353\355\214OTA \334!\356\347\325\350\220%*I_hp\333\365\0162\001\241\207\?0-\260\034\215^7\353VT/@\355)A\20\ 6\a\221\\kU\031$\030\345j\273\322\340\276l\363\237\f\304|k\276\214\331^\255/\\w\203\336Q\210Z:U)\347%\024\202\314K\020gcMB\023\217\250R\242}\316:)\00\ 4\366\r\273=\r\373u\334Y\276C\262\005\276Zp!4\370km\327\261$\211\340\030\003O\f\240\306\004WFS1c\354\343;\031\356\247\035\341\257\2\ 12\315\207\251~\373s\n6>'2\t\323\237~b\374\260.h\371\v\n\256MB\004T\032MXnME\247 \262e\030q\236\f\vb'N\334J\211]<\034\347,\303\312\\ 347\255=YPBm\f\236y\020\357\200\264\225#S\251r&m\3024\022 H\3305P\342YF\225\201 6|\021\034anE\214\270\256\304M(P\221\024\027\327\21\ 3)\260\270nF\024,4\003r\324\200\361\301a\?8\345>\325B.\300&p\3266\313\032%+\363\022\375p\203A\031\213\314*\\a\221$\276\276\266Tr\01\ 7\024\023I\a.\220\036\3207(k\244\345\304S\210\2048\026\234\212\027\rhf\261\234\?\334Z\017\024\341\313v\275\354A\361N\v\363\235\207\\ 234:\334\",\024\t\032=\322\351\302\241\263\362\263\267\023\214b\366/a\n\371f]\004e\221,\220\321B\216n\207\332\343\305s>q_/R\242\335\ G]\307\247k0\251\323\262_65(\034\224F\036\374TR\004H\341\306\243u\352\tAR\002$G\226\024\252\334\270\225\263\233n\275\301\272(\v\363\ \263\230:\030\241\0046\v\022\264\"\000\206\222\352\030\306d{\367\027\024\356v\200\3065-\257X\365\342\203\2138\306\314\0164\363\305/\ \213\300\201\201\227\316\244\005\207\372\"\330\364\263\210\033\242&;[\275\346\363\273\237P\350\026\332\002\352(\375\2008l\272\200\2\ 02W\304\b\276j!\212O\314O\2214\204\361\021\367\225\006\201\337v.0\350\221\223f]+\357\331\244{\371E\272Z\362s\245\000\032!_\200\226\\ 2607\312o\263\221y\322\306\203\367\003z\300\210\307\035\207n\365d\t\026[\245\213\210\332\362\234\310c\021)AW\272\214Z\027\004T\315\\ 353B\361\267 \343\322\376\031\r\367\214\354\341`\203\277\v\223\357\231\023\003$\a#\355\v^\377\030 \230\334H\2373\234\317@\324\370\2\ 50|\004\3623~E\025\327uUZi\246\232~\222_G\234\344\253\316\023\237\017o&\233k\332\366X\252\242\333\0259*\222\342\025EO\022lFfe\276}\\ 256=0\224\262\336\\W\247@c\330\362\377I\363\340K\344,\247\035\351sX&\361#\331\303\345\251H\022\256\252W\307A\331\377\232\356\353Z\3\ 26S{\2661\005/S\216\365\0234\0041\365\004C\235\310dD\f\025\366\267\002\345\352\236\017\273\307\3115f\214&\277)M\330x\026\n\257i\023\ @R\225\000\360o\033{\215\274\333\271\225\366LZ\b\201\243\202\337!\b\336P5'\327\026\342M\021\301\267\331\347\?\232\3120\327F\270\001\ {\207\001la\310\016,\262\322{AsK\257\022\262\357W){\362\020\030\241\226\355U\272\374\004\204\262\023\\P\201\f\222\344!1p\344)\025i\\ 351|\024\201\277\221\\ 300\340\201]yzc\314\265\220\027\203N\363F\315\036;\016\340K\033\265s3\001\002\0022Gp\251 \021\027\204\320\b\004\300\300A\301\303\25\ 6\241\204\020\324)\234U\t\373r\250w\357\346\v\226\v\211v\245\003\004\032\024(*H\026C\247g3^\fmC\375\326\006|,\216(9\332\220x\317\37\ 6\230\201\271\000\025\202.\002s\036\365\0309\273\020\245\223\260\265\017\344\t\3166\276\\\312Y\004\n#\350\367Pm\220h\313\341\3348% \ \346\334A\017i@\341\335\371\305\303b\255\317\264\201\340f\250\374H\361\210\272\234\221&\204\240\342\203\340(\006\325i\336\336-\324~\ \b\304\306m\261\230s\222\356\274 \313\030\016\201\025q\354N\244\303]4\274\371\267\251\260\206\316I\352\231\272uzz\202-Il\254fq\201c\ \236&\017\336D\266\216\315\035;\202]\200\312)\233D9is\020\203\\\260O\303\333\0316@\222\336\373\244\340(\bK\246\017C\314e\340\300\22\ 3)\311\300(g-\365v\202\236m\004\356\3302\254\211\034\203=+\213M\"%\321\322\372B\032e\211zB*\302\3071\005\016i\212\317DE\024\216\350\ m\271k!Q(\032\303\322 \037\000\2651\341b\004t\376\256H;\227\344\361\254\340\b\037p\023*#l\b|!<`;\235\214\257\037l^\360\362\213\203\\ 237f\361\347\3237b\"\352f\220d\354\000W\240%\271\201\206`0\310(\031\316P\020'\232\233\"\362)j2bt\263l\356j*\311\323C\345t<\273\364I\ \343\242uLh\347\261B<\"R \275i\001#\001\315\306\303\031\300ZY2\a\006\303B\315,\000\223\270}\323\342\030\220\036\\\a\317Y:\360\334\3\ 67\345:\?.\214z\315g\236\001\361\247\336\335\350\021\tH\020\2472\370\310s-\256:\354.\004\006\004f\273\232Qz\272\250\0303\005\026\33\ 5\206\016\335\v|\371\342\371\016\r\214.O\"\315\241\212\216\232\325\260a\360\236\225\334\3562\210\321\247\024\0321\207A\030H\337\025\ \302z5h\025cW\243M\343\262\320-\256n\030\253\325\200\342:\311\2540\302\232\035\241\365\260\251\270\\\024(\353\3146:(\304\024\236a/N\ \276\\\337\260\336\000\251i\016\\/&\v7$\023\235\n\340{\n\034g\t\363\241\220X \362\356\264N\245|\"\204\347\332\244\304\216\362\367\2\ 31b\315z\257@\327=\360\001D\364)\270oP\2240\310\311\300\274\202\331\345\a\304\264H\332ZX\f\216\037}i\361\001cHd\355\027\3201G \330\\ 205\0046\037\370>t\\ 030b\nm\026\361i\310\224\200\321o\243\3079\370\345\250S\325>\rI\300c\031J\205QX\324\351\354k\036\016|\356\303s\275(\301\004h@\303\2\ 40\245\335\335\231$4Y\\\025\234\3263\265\234\021\3565\260\334p\360O\036'|J!:\003]t\376\277\005c\220o\261\246\342\252=}\330\035\306J\ \261\322\313\234\204\305\303\277g\243\022\225\247\a\v\306\257\3225\340`<\351\334\244{8+r1\361BW\247\321\221\232\242K3GR \346x\203\3\ 41\35268|#\"\021f\327\236\2448\245\320\371*k\266\242\031\3353{\020<\220\315\2614%\367\274s\241C&\033\255\243\005r\"\261-R37:n\017A\\ 000\275Jl\226\271mh\254\221\342\317\201\303\304\311\341\n\203b[\311$\265\241N\202#\n\034\222\327\035\277\274\320\370\246R\336\2448.\ \342\227\003\000\226\035\307n\341}/U\354\034_\272\367BL\n\345\275\364\357k{r\342\343\322\260]\341\326\226\352j\310\360\303\204\001<\ 2\272{j\016%!(\261aC\274\263\314\214M\b1\020e\212\244\\\336\301\244\232\337j\3545hG.>\v\345\0305\225\277\023\326\032\260\005xN[E\235\324\240}z\206\336HV(\331p\001\a\261TR\006\271\t\273\215\305\003\266\3318\005\225\020\302I\3\ 32\313\333\0262\260\355\357A\313:\032{\244\250W\312!\347/\215[\3234k\360\357\332\261\206\vi\325\237 \310n\317-\250\r59\273-\353\340\ d\363^\216\204\314\b1\341\030\3731X\343#\276\024a\267\254\362Y\370\275\215\211d\310\362\371tH-\351\021L\306%\222\303\350\263mF`\a$P\ \020\347k\r\346\027a\320\354\213\352R\271O\005\324\254c}T/\022Y\202\222r\003d\023c^\033\257P\032\324\016\232\220\001\276B\a\036\247\ \307c\022\004\021\243\207A\210\237ih\243\210\303]\205\356p\343\322\374\330Z\a\320\364wy\355\375\346Z\230v\231\023\361\337\002\224\2\ 66\2539\246\266\350\362\036\250v\337\303\032\351\334\231\200\301f\204M3D-*\336\231\3667\a\312\272\246\230\240\244\243}\253\030*\"x\\ 264\016\f\363}\273o\254\265;xg\3275\211i\371\265\366\\zR\242L\031 g\022\314\265\351T6a\?05\356\3134zl)\241\345\332\263So\326B>\025u\ \214@\033\266~a\262\332K\023\0054b\334\247\236\262*E\222B!J\342\303g\335o\243\216\037\203\324Ns\236\361\305n\257Y\220z\370\304\006\\ 345#\236\267\210 Q\225\031\270\322\034\254n j^Rm\026O\021\031\236.,\213\203\316\3439\006|\343^\030u\206\302\306b\306H$yy\312\b.=b3\\ 240\370\274\367\326\253\"\320/\\\302\227\017\n;P\373\351\255\210\213;G\304\275\350X\251\b\331<\300yk\226t\334\350\345\340p/\215\351\ \307\234`\315\357o\276\252igL\?\213\305\351\225\322=\351\03\ 7\207\035\307G\243\221q\320\"w\2769\t\361k\035\343\207a\275\216\270S\021\001{\252\0008\271\244/_\331oM\372\216\361\313\320a\341\f\3\ 27}\277\005\235\346 C\207\005u\203\r\353\301M\215\264\312c\016D\035\234\275\235\"\233\223\277Z\000\365n\306\027\300Of\251`r\272m\23\ 6=\213e6\234\207<\266\223\326\240\217}zX^JSf\275Z~Y\rw&\364\252\302e%\357\201\300\035}v\271cg\0018\230K~\321z\243\200\3656XR\321\22\ 7V\b\3708\373\373\037\224\016F.\327\023\326\\\227\037T^R>\250f\317\367\022\342\262\265bKN\245\a*6\032Z\200\254\342U\bU\343\305\326*\ \362\235}z\346\376\332\273\361\276t/\223x\2703\207z\370\n\304Ld5\352|c\234N\244\254sU\326\220\\\374\021\226\225\316\202@\200\205\21\ 5L\004`\360`\0212\207\247^\331\234\016\v\373\271\317\005\357\202\352\272\360\003\247,\265\316v[\255\263nk\364\351\325\n\355\367'AD;\ \350\240;o\272\231\201\231r\003\263@\330\332\355\034\261.\337\2719,\310\370P\277\272\321ZK[y8\260x\235&\003\270\356\300Yc\333r\213\\ 344\314\334h\304\304\223\270\364\344\nOL\033\345\206\351\023\f8E\004@\214Im\264\0366\b\212\"3c\033)\245\032(s\325*\240\362B\002A\23\ 3\3253 \"\265\003u\351\213\336I\t\340;B\361~\355\034\f\210\001ES{\235M\220s\250\360\020\f3\357\005\342\\\230\027\366\331g\342\276\3\ 16\272f0\265\252B\t\f\2341@\324\206w\0230!\211 \270\222*\265\240\024r\364!`8\267Hx\317^F\232\273\310\362\260z\202t\\\274\237\v\303\\ 300\220z\302\373\247\357\256n\022\371\2432(\201\312\247\032\216\037w\242A\316\t\2049x\215+\273I\351\340!\271\257[\317\021\3008\037}\ X\0267[\267\?Z\205G'|\210\277\244\021p\254\214\254O|P\363/Tn)\240\253[\021\333\375/Nd<\272\016J\321<\\/y\252\324\276*(\302\372Y\236\ \220I\350\266'|X[AUh\224\022\204|\037\034\247\3368\305\002O\316\205\270\315\240[_^\032\247\034b\317\257\234=c\001\341\335\341;\306O\ /\030\352s\034gF!\365\220\367\302\341\376\245y\231>\376fA\201\247\324_]|.\246\246\271\310\343\350\371\257\327\255\023M\323b\227\240\ \034\257\202\036\362\347\034f\024\ni#\031,.7\315\3556\262\000rCJ\312*\376&\307xF\353\306}~\214\204\246:F\372\215v\357\021\332\300r*\ \302\243=\375:z\302z|AD\342\024{\356=\215\206\242G\253\371\212x\3529\324n_1O|\f\276-\374\315\305i\033\366\250\321_u\0268a$\023d\"\2\ 75\2230pF\240Sr\236\350l\374\325\332z\273\372\203\375\316\241X\201\226|L\303\322^j\302\327[\264\353|\201\270\336\0042\317\244\341\3\ 13\301n.\343\336\023\271A\341Q\037rZ&\251\017B\362d\245(\374`8\371F^\314G\017\020'%\334\034yq\242+_#\325\211w\322\316\240\243\250k\\ 306\2461\213\367\351\344g\027\020\211\306\214B\225)Q\026j\rf\221y\327\016\371\241\221\361+Y\345^\035\f\216*\212\251:\031\034p\267g\\ 234q:\340F\216\0166(9\203\270C\212w\253\205p8\0276o\327\020\346\331=m!\252x\030/0\342\342b>\261\212\372\270\212\207\233\a\360=^\245\ S\302w#\026\204a\377\275\2251\350\\\034\274\005!}\237\252\337K\343\342\337\203\275\030\266K\233\233\237\236\342\310\372e\367\032\24\ 2Bg;T\006\240\256\331R\221\343\352\241\034\037\326\236\317\274\313.\300\346\214\252+P\255\205\356\234a\206yl\351\006I\315 6\354Mh\3\ 43\2306\240\345\304\216\216I\222\245\342\000l\a=\202\361\316\357\267\370\204n\314j\352\230\214\224s\325\201\346YyIAJ\005\330}p=\207\ ui\244\362T\031^\242\362A\"^7c\202\033\360\332oG\210v\263\2450P\365\acv\034v\251\340\204\254T1@\210P\0036\t\024\307\302oU\202\260.8\ m\314\006\276\024]\001\214v\210\236f\350\265\344\311\330\343{[r\352Cds\300-$C\307\250\tw\?\264\353[\242\350t\?0\236N}\a>s\313\3572\\ 020;|\37464\354\003\256\272\213\252\366\277S\355\237\352\022\372\235\340k\366!\317\367\021\200N\332\000\200/\261\354\242|\253\004\3\ 70\244|\227G\034\334u\b\036N\236Kg\346ru\265\371\277=\351dLo\353\327{\241\376\333Q\"|h[\332V\374\377o\211\361\017\365\270\017$_#\30\ 5\021K7\353\240;R\035\243\247\237\017tt(\0058\275\237\317\303.|\275=a\341\315\203\2203 \016\375\b\035.\230O\373\a\203\241\351\271\1\ 77\347\034\376]|cz\272p\255\325\204;H\036\257\342\343d\3662o\254\247\227=\033\344]\260\374e\003\242\353\275RD(\364TC\326\277\303\37\ 2\276\225\330\213\227\346G\320\361\240A\333K\376Y \372Nq\277\?]\372\021\3378\216\363\377\271\323\360#f\362\024t\317\202D\210\373\f\\ 274V\357\207v86\204\003T\a;\205@U\360\373\275#\020A@ABS\302\350\363\375\337}u\324\237\334*\373\217R2\0358\001\340\000\r\274:\357~\?\ \213\342xZpx\223\261\321\272\340=\320C\bj\356N\225{\314o\260\327\223n\030\362E\202\244\257\237d\237K\230\225\023\367\035kN\360\302\\ 227\353\037\235\235\017)\201|\030@s\374\246\273\313|\177\221\352\372\276Q\355}\307{\362lW\236\247\255$# H\215\020RJ\220\021\023\357\ \020d\223\025\b\027\327\331-R\025M6\254\023\353\276@>x>\257\342\361\336{\277\026\201=\237\272g\211\263\207\224\366=\3077\274\231\33\ 3\301\301\364\274\317\032C\331\036\177{\306zh\006\305\347\304E\021=R\201\373/\306P<\201#\002k\367\266\366FTiCE\036\"%\337\037\"\340\ <\?\222\322w\204\331\257\266\r\214\027\320\016\037M\230\240\205\020o}\247K\223\331x\330c\016\3077b\313\\\337\343a;\023\257\021\345\\ 216\272\b\360\303\277\366\207\273\261\353\275w\221\312\035\r\035\302\250\366\376\267,\023\ \231\221\346\362\206z\237\v\020\315\317\217~An\242\247\331{~\353W[_8s\255\315nS\320\346\336zB\360\a\aq\017r\252:\215\221\367\214{\310\276\230\214\352\ S\343\304\362\217\022\207\333\354\327\217\205k\3176\207\207\312\205\311\363\320\357\313\212q\276\216_Q\210f\363o\303\332\375/\277\2\ 65\300x\376\260\245\313\2204\367\233K\375\207\037_\243\006\341\356\221%G\265\361y\272\276'\252\301{\261\r\003\326t\352\344\3633\035\ \317\261\303\033\0066\374\314~\357\3624\276\346\342\320\332\036\375\035\257\320@\232 F\241<\035uv\230_\351\032\205>\326\317[\350\21\ 7g^\3735\363\300\243\?{\034\347\025T\204\f}\325\027\364|;\361\204\323\005NS\021&SK\251!\242\025\037\221\245\024tF\245\331RbcXG\034<\ \225\272\354a\t\337N\251\313\317\355n\037y9\204l\215\344\rs#\006\3043&l\336\223\326\000zW\357i\021A\260\031\356>}x\236C\f\251\023(\232\254S\003\25\ 7$}\004\320q\370\211\304r\324\363\256\265\354\215t\301x\331*\rg\030j\212X+BR\277\251R\300E\320B\244W\201\001\320\267\020\"`\305\200\ \345z\304\276\022j\027\275Uo\235\aU\260\336\361M\032\233\331\250\037aM\317\246\a\210n\200RM\230C\216-\215d+\001<\2205\2700r\323\004\ fYGd\223\253\265\207\216\327,\3117\3028\016\366/\005I\034\267\325\b\336\205\372\035\312<>\347\305\334\0361%\005D\305\?[\a'/Tp\225\330\357s\017]f\354\325\336\237`\350\247\001y\026\030\023\302MHD&\247#\022\\\314\022\250r\310\2\ 10\244\343\fV\200\320\375\217\200% \310\254`\024\233\321\350\342ko\312\252\313\272\342_v\232\341\3001\255\364\3079EA\270X_\aPr\001\\ 265\245`@\2462\204:\001\341\304\304Tk2\335M\a,\321\266\006\227\233h\241\304\203.&\024pEH\304ET\245K\332\3120E1 \214C\300imN\234-\206ym0\233\312\246!qQ\002=\\ 316\202\374\270\363\367U\t\004S\f4L\025\vU\202\276bh\347\250\?Hh\365[%\354@\005!\022\241J\230\017\\:N@\031j\023Y\206f\024\211\001\3\ 10'\246\236\003s\000\236\230\0049\203R\247@\227\n\216\220\a\232H\211\203\000\037\016SZ\307\016(\347\0054\243\260\326\026\032\026b\0\ 20\371:\f\355\200<\020\n\020\213!\243\021y\363\301\205:\214\202'&\267\002\343&bOG\274\330\017\240\215\313P\305Ml\210\362L\207\0224h\ c\210\215\266\303\316\252\223\206\342$CZ\241\025T\202\372T)\367b\265\300\033\203T\024\214R\304\205\021&V\361\324\350\002\313 \312\2\ 02\230\221\245(\022\244\313-oC\243\246\264\232\202\252\2328\263\tS22\n\244z\310\344\216\340\346\002\2358t\327\001R\247\020\361!\020\ \005,&!\311\020F\222\210\334\207\016\324\033\250\212SE\016\026\220A\211\2621L!3\243\203\241\266Pc\260#I\356\006\tT\305\004QMA\230u\\ 265\004t\260h\223 jM\340\241\246s`\206\r\r\225\353\"\30444\202\312\225\220R\224%9Y\200o0\252\f\271\215M\035,\"fX\246\3472S\2312I\36\ 3`\367}\357\000\332E\335\335-\244\224\300[\336\341\271\277&\223\f8IB+Pn2H\236a:\302j\n\035C\204T\305jr\002\255\347\247X\352q\303\24\ 1\240\034\037\r\216\334D\367`\"\2044\327\334\335\025\243`z\t\361\375\313\251\365\364O\311{\264/rO\036\360\365!\325\244\036!\261\221\ y\205R4\340\313$r\021\3112*\332b\\6 |O\235\356\224iv\320\335\336\252\212\340\365 \246/\001\246\343|oz\263\032\344\a\304\221<\360\01\ 6C\266\300\206\205\272ko\204\206\"\232\0064\270~g\241\250u\r\300\224\247\235Z\v\031jj\241*\261\000\314\204@\323\0211M\002A\224\332:\ \252\200\235\022\nm\b\363\317\023\236\002\351D\250d=\274Q)\021$\244!\236{\21145\2146@\356\v\r`h\2018\212+\031\244\203\202\207t\247{\ \234NdW9\246\232F\236t\320\035\335\"/T\002X\220\212@\227Wmxx\217\037L\t\022Q%\306\323\234\320)t\204\323\247\025O\005\327J\0165<\314\ \033;r\006\212\201(u\367\321WQ\360z\033\361\327z\027\026\023rD\022\023 \333\322aE\203{\031\302\v\234\f0\205h\f\336\244-;H\235\325#m\ \021\204}g`\263\344\325#\316\327\327\2775\224[\\E\303\275\231\2746\b\004a\312\300\200\231\002'\031#\031\303\254\210\260\204\0204\35\ 0\r3j\265\231~g\267\343\240\345\373\347tFHf\032\005\207\025\223:#\265g\276\303P]\311\214\262\322k\265\206\036qQI\204\224\355\210[`c\ +z\207e\326e\234\322\255\034v\0239gG\032\030,\016\b\004\375\251\027\250\307\205z_\003\3020~\345\270\323\227\353\241,E\034P\027>\327r@\366+\314\006|\33\ 1j)\250j\",%\226}z\217\254;\036~\005\352\316\350\260\246\316\271\242\242\346\214@\310#\030\246\n\270\022\367&\364\266\370\337n\225\\ 323m5w\025n./-CDT:Iu\300\247\217\273S\353\206B\276\216\002\367}aX\223.\344/\212\213\005\270\022\334)\035\227;\251\276\037F%\303\344\ \240D\033\302\221\017\t\203\210\243-%\236\241\314\244\323\204\334|\360\035\215\"\025\323\266;\245&\002\320\177\250\031\021\204f/4(S\ e`\207\"\335\331\332x=C\214\305\0305@\330\210b\304\2661\306\000{\244\344=\016|\345\301\277\203\262!\244\213\304/\034\310\320\240vC\ \233\360\355\222\346\245\207\244\016\016ko\331\234\fQ\262\232Y;k1\004n{\ w\230\341\336\214\023*(\026\023&\274w\206\232\254\227|\274\263\257\035EA\2158\032Kr\260\257\273qFJw\252@\242\373x\035\235\270\305\0\ 05z\304\304\233\006O\247\321\207\030(\351\362\361\306d\363T\350\020\206\310.:\334u\204.u\2370\236E\200t%)\231\275k\222]\320\245\324\ \234k\242&WShl7\367\024\205B\242qk\210Cpz\300\203\273\201\025\344\232P\330\005#bF*'\266\303\263LJOw\202\344\321\036\222x{\274\275\r\ \216\000$\vX]\3611\273r\027cMA,\272\230bl\211\253R\224\334on\r{7\213\3644\204\034G_&\312\272T\2220\033\342\022l\311H\265Z\223q=\202\ \022)\313\306\000\250\037\000\210\310l!\303\017[\267`O^\rY14\320>E\264o\023d\006\211\324:%\n\324\215\030B\354\326&\341\265\335\236\\ 275\365A\212\247\vw\036\201\315z\205\311\272p\030\227\004&\262\3421\226\203\000\310.\016R\tfr\234[A\371\230\243$\204\212\357\343\27\ 5\264\313!\310\345\3340\350c\244\016(\036\343\000C&\202Ij^-\246D\nhvM\256$\302qHC\311\231\333.\362\335\300\274\310\332r\276aA-\020\\ 242\222\002C\312D/!\t\314.N\267\247\260N\274:^\247\0359\354\247\206\2141\251]\\0\276\244yN\201\325\035bz\360\242\024\0012\024DPU P\\ 005@H\000X\364\364\025QI\264\357\333!&\242Hb\315}]{\035\315\325\260\225$Lz\270^\250B\r\006)\345\026\257X\272\220M\240\217\302\225\0\ 33\3643P\344 \t\364\032\315gU\274\v>\366\fYji\324\ab\311g\204\023\202\216>\n\274\032\234\034\354\343A\v\025\036(\326\250z\332\\\230\ l\300,\026\312eDC\273\324FT\202\342XPF\264\266I\212\242\244\247\b\221\0168\210tp\231\236SA\376\344\266\344h\362[\347\264i\005\276\\\ 97l\314\3033\311\234\033\034\225\017\031\032%\244\307\356\a\202\371YX\212R\216\204\035\001N_\032\204;{'c\263)\267\251\v\300\365\024\ \024\361q\341\312\033W\026\311\316U\315\303\300V\273\314-\033\370.\024\361\227\274\023\227\252\373P)\025\001\236\204\340:TZ\305<\36\ 4\"\222\v\017Y\351\215f\3674\322^\205t\0355\365q\235<18\312qt\213\312q\262\212\312\036\241\210-\242\000+\212C\177\005\314`e4XUBLmM\\ "\351\313\301_c\261}\251\243\2620Eu\313\266\271\3431d\224[w\256\031\247(t/C\2163\001\372\347\203~\356(\362\024s\031\235\2503\242.\0\ 046\372\355W\255\215\306\363;\342/\24267\262\a\005\374s\017\326y\200A\311\b\021\f\234v\222\221\270\316\250\261\346\026X\027Xx\036 d\ <&\215\363\020|\037\263\336Gdh``\273\366S\345\337Z{\201E\302D\264\003\223\267\300\003\320\001\206\003\305\260\035\323RC\322\2419-\0\ 17\004--)\241\212\"Y\2064d\030\3534Q\022\024G\212\330\024@\004\004\335+\242\034S~$\350C^\2527k\301[\020\v\252\232\f\233\261sR1\307\\ 342\360\242\b\345\201\306\342\r\361\0279B\225\211\322 \3024Vx\272\212u \347`\362\017\r\344\304\233\232 \301b!\t\246\312%@\266\036\0\ 01\335\343\016\363\343\034\370\340\277o\262\240\226\216\213@\206\341\272\202h-\244)\n\034\344EL\313\020\233=Q\263W%\211IPA\026\252\\ 241D\255j\203\346y\035\202{\2301\301\036\000\353\274\006\360\005\354]\230\020\202H\bo\\8\325\341\316i\253\323\002$d\206\366\372\232\ ]P\336\270\034\321*(`\\\361\236\f8P.)\341\003\024\362\330.\373\aj\340&\a\021\263\236\205\327\271PV\200y\244Q\351F\r\375\224\362\024\ R\210\246\240Oz\037=\215\022\344&E/\020s\323\n\004\366I~\312\325B\276T\250u!R\317D\330\261\214\024\217Z\224'\tI\263\250L`\367va!z\3\ 04\340VIf\360\324T\271c\253\020\310\2474\0224\030g\237\2701\265A\302\v@J1\260\260\031\033\206Vc\215E%)\224XVXQ\026F9P\214*\252\016n\ \221\272;\332\026\n1\227\22737IH\350\027d1U{`\2450*\335\v0\240)P\304B\364\207f\251\233|\373n\305\v7\265\336\f<\3022\022\br\310]\220\ \023\266\000\261*P\320\276\231h}D\253\250:\240\306\023\2142pt\031\023H\200\354\251\332$\r\034\345\220\313m7\000fA\240lYp\202-\363C\\ 244\017\230E\t\276\303\314\022\275\335(h\004\230f\" \273\366\357F2\345\300\211\250z\370\275\352\344\213\251P\307'\027,\232\004\230\\ 324\230\025 \314\204J\005\"\004\300\301\003\035\2614\3010|\260\340\341\3034j\023\337d\220\202\001\227\023j)\360\034bjP\356:\322=\33\ 0\034\005\2615\336k\205<\262\035\241\313$\247$\311_{\f\217p\313\240\?\n\214\207R^z/I\353\213\326\212;\267\353]:\246\001\221\2044U\t\ \207\177\022yY:kAE\022KI\3040\201\300\032C<\001_q\300\020\326\006}T\031\264\235\344\217R\250\301\v\241\355\323a\335\334\360\324\301\ a\230.F\336\376\310!\302\353\220\352>\222F\367\302\307\fr\260\036\342p\321X`fb\323f\006\021A\220\353A\211\221K\222\226\246\260\251\\ 254\214\"\202\3246aCE1\020\320Q\253Id\"\260D\2526\250\325\025i\221\214\024\022I\300\234E7\244 \374\306\317\227U\244O\?*/\312\201)B\\ 201\217\\\330\apl\370\351A1P\005\003$\200EJP\020T)T\2245IB\304\243IM!J\323CB\320-!E\0051RR1#IIE(D\321\r\"H\005 4R\245%R\024P\224\32\ 05@\224S\023H\320\024\253\020\001\022R\001\022U\003@R\024\224!T\200R\323BRP45\240;\277\"JL\224\001\022\asP\006\304\3568C\023\2657\3\ 30bA\206RG\033`7##\212\021\262\021H2\004\221\264I\f\2023*&\250\253+\023\02731\304\301< \364\271\203\201 Ud\350\216\3541r\306!jrN\30\ 4\031;M\275\264\216\000\347U0z\354;\270r\242+\255\326\002\212Y \252*(h\032\251\246\251$\252H\212\032e\241\202\252)\242 \210\"\272'\\ 267\343\227\"\213\207\031\313\333f\215\200jxv\253\004!\022$Q@J\037O\005\fLf<\354$I)uZ\202j\034\367\271\003\021\324\03237\t\233\0308\ \034\227\306vz\201\366\245D>\264\036\271\3576\br\250\244+\335X\246\260K\205\316{\2230\024\362\037)\211\367l\r\213\223C\307\243\r\35\ 6\200:\320E\325U\260\251\300\037\022\033|\340&\000\320\236\350t\334\272w3^\207\264\256_\250\307]\312\316O7X\303\216\300\367\211\230\ Q\240\201\340Y\n\310\314\264\313zCW\225d\005\002UT\003F\233B\331\255\202\303\244\031LQ\251\213b\032/\vi\267\f\227Z2\314\362\313\331\ Q\363\bx\210\3740\024'`:\002sh\325\002u3\006\222\251\002\212\r\313\266\036n\3320m\347\030\340P(\224\261*\252F\3408\257\v\357\316\24\ 7\2452l:P\322\031 \205\317q\020\352\006v\002b\034<\267f\320x\305T\032 \r@*d\024\320\211\350:\216&\345JJ\025\343\340\031\371\000\002\ \220\005\002\021@[A\2764\003\203\032\250\\\357#\345(\\\223\225 \214\"\244\200\233C^\374\023\r\213q\301K\306\334\020\326 \367A\350\2\ 20\311\017k\336\300\325KB\032\323\2026\310L0\024\002\253\322\2569\nj\036&q\355%\367\023\314zN\250Q\317b\022!~D\277\032T\360\223\330\ !2\0178>Ic\327\303#\277\305\304\337\226\270\261b\270\303\000\215HW\"\247\315Ax\324B\210id\357\264\210u\334\024u\331\370`\234\257\32\ 2\245\332_H\244\231\254\363\206\215\275\036\3663\260\236\264(\201!\362\326\267\274O\330\226\361\252u\306:\021\231\271\347\304\320\3\ 01\r\203\327Z w\332\030\256J\325\303\207c\2012^\230%\342\356 K[\260\236!\307\212\004n\374\nR+\256\006\317\201\273\004\330B^\315v\36\ 7u{\357\"\005\241\326\036\240\202AS\b\r\023h`\211 O\r/\221\236\235m\242\267\214H\345\243Y\215\244\3749,O\210\200\354\322n\bI\016=\0\ 31\262\000\206\314\025\t\264\017\"0\314\3625 ta\235\364\252`\031\t\307\214U\004V\v\b\260\260\275\330\0060p\\,:\364\301\024B\252\210\ \205\252\n \200\b$(\026\205*`\240(B\250Z(F\225\242\"\272\316J\322\021%\n\025\231\202\024\331\031E\000S\026M.5!M!AM!Cf#b\351\022Aa*\\ 216\350\3236\365\272k0\020\334bjxD\327\251\310^\220\3217\310\204\365\224\371R\266h\202\240\252C\2101\262\311\t2\302J\222\016O+\351\\ 303\324>\224=\022R\037\032\311(!\251 \bB\224i%\212Z\006\231B)\244&\000i\002\001j\b\226!Y\226RD(\242$h*bI\344'\"\242\206\276\270z\27\ 4\016\376\232\027\341\232\301\266X\2036&\347\312\327\345\240\372\001\321\340\207\220<\335\036\263F\030b4\2454'\272C\341Ws5\321\262M\ \220\246\373I\336i\224\243\3101\t=\350(H\\\\oN\t\250\206J\353\343H\252\256\255Z\210C\004\312\372\3333\220.\022\fMk\233\f)\257u\303\\ 300\335\233\325\023j\242\230\232\320%\006h$tRg\267i\024\244=\2103\\h\302\032_\030\206\231\344P|\243+\247%\222o\356\242L8\034\260\n\\ 210)\355>\bug\201\3443\352M\217qE\235\246==:\"#m\002)R\252\313\311)\337\005\033\202\223t\324Uj[-i\326\363\"\215\342X\"\235\303\360\\ rC\201\262^\302kc\310dhp\330\016(\373\203\037*W[\216\222\365 \3312\363\341\322,Bh:4<\024\264DQb\020 x\361\301D\3462\273a\203\a\263\\ 234Z-\342\343\250\310b$7eU\2228A\230X\3454\321\021Q\025\025@\003f\364\350IZ`\226\265\246*\326N\206b\203\321\306\265\3068\3529\203U!\ \0224\205\25420L\2602 \202\336&IA\226C9\201\222\3231\026\352(\303y\262\252\017\021^E\n\025\250hP\314U\3504\032\f\245Y\201\037\306\2\ 40\260f\265\263z\320\317\001\264\217V\016\316\002\204\301x\030\266\311\307h:\272S\036\332\f\023L\361\303\000T\310u\314\2101-(\274o+\ ujx\212\323je(\334\342\225\203o\031\266j\246CQYsdd\026.\033\223\n\235\217!\233j\272`B\270\362\016\242\001F\r3l\331\206\f\033`\231+\\ f\313m\262,2\375@\315#\227)7\207N\n\223\327@b\031:$\255h\n\271K\ng\244\324\305\312\364\355\200\3565\322%2u\022\322\331%+\001@\036\2\ 63\213\025j%\222\036\\d\243!v\031\336s$\225S\001\236\274\035\363\004\031\2304\241FL'[\262\303\f\033\222\214:\246~\246fU\300\\\2622\31\ 4\263\224\372c\037^>8<\343K\314:\214\2254@\207\222\\\351\275\003\271\023qKM.\347$\240bE\230u5\326tO%=\201\204:\365L\025)Mk\275\v)\3\ 27\355\201\340\034_\f\354\227(7\005\312\246\201\262\201\325\241(\f\201\326\2321\330\032e\253*\346:\220Y\004\357!s\216\350&\360\b\03\ 7D\022\370\307\217\316X\231&f\230\a\303\311\235\206w\003\275\334\360\303C\276\257v\260\034L\375\276\320\347\315W\262^\245$\"n\326\0\ 35QT'\275\3605\305\025\311\354\t;q\201\354h\va'D\222t1\006\r\ra\230_\f\252\265\260\224\034\224\360\035\260z\376\016K$D@\314\224R\26\ 4\304\023\025Q\002D!H\205\000\320\024\004J\025G2\230M\021#\024\022\300\224\003(z\2407\356\341\271(\003\326!\016 \017\002(\a}pB\234\\ 220\f\221\370\273\266G\276y\361\016\203\266\220\355\006JEa\231\320\332\251\3414\003J\276\302i\303\201\365\202\020)R\274\257)r\216\3\ 42\300\340\210j\201\"2\t\240\274C~\"\3546'}\253\004n3\231\300\356l4D5+\222fn\270!p\250\032\257\037\207n\362\366\002 \257\266I\222\0\ 249\030M\345\260\f\314\030\343\tb\320\361\240c\243\n\351\353\314 m\351\264\266D\262H\343\223\rq\230QY\205&\024\231\0066\243\rX\321)\ o3\211\304\202$\002\202\215\343\243F\355|\213\016L\360\020;\244\306\320Db\234k&O\004\230\246\240\264\n)\021\237\037\004\303V\036\25\ 4\302\320Vc\216!\231\024\311CJ\352Q\252Bs\020N\006|\375\304w\346\022\020\036\371\314\264\020\275\026\021{\204\360D\360\017G\253\032\ 3\344\347\307\232\267\202/l\303\300\023\313)J\001\242D0\224\211(\005\220\204\352|\235\307\000\232\205Q\350\343\034$\000\006t\313p2\\ 235\247xb'\243\f\021\361\2012\024:*\217*\345-)\346G\261\002\314*P\244M\217y\240\356\203\240\0372H\335\320t\362\205\260P\236\344\240\ \310N\355\3762\024^0RS\350\210\224\362\341\213z\317\021l|\211\022T\244g\247\262\303\277\241\274\363\022(\307\344\357\371\373W[\256\\ 362\202\333\305\312\271\224>V\200\002|\236)\342\022\236\001\303\005\270p\004\366x8\221#'=\0057Z:\n\n%\r\027D\371\227\365DK\204\231\\ 235FG\206-1\205I`a5\360\236\257_\225\317\243t\037A\223\271\017)[\266P\341\024\315\211@\246\250P\33300,I\221\tH\027\361U\211\271@\33\ 4\271Pt\2252\016`8dv\004\366H\224\204\331!\235J\321\264\266Ms\257\320\306}d\003N\351'\020\252l\240\372\310C\330b\"\217\225\0030\342\ H\025\0200{M\005\241l\n\f\bo\347:\002\256\307A\202\br\2220d\b\314\315x\246\374\320h\364g\316z\006J\\\371;\226\003\317\320\\U\320)\0\ 27\322D\0176y\260<\317BW\253\357\350\017c\020\240\207\213\350\312\363\342\241\320#\022\001!$8\366'\022\034C\263\036\004\347\234\\A\\ 243\211\025\350T\314\031\235\302_\021\030D Xr\353\r\262\2544\311+D\335\314\0028\236\aP\250\222\251\360,U\240\324\206@\030\256\220\3\ 15b\343!\203e\021\271\34016\024\213\277\222\232UW\205\037\220|`\336\324\320B\003\274\250C\003\2330\265\a@!\277(\\\304\027\340\350ZD\ \341\202\aN\v \367\321w\310\216$\004\325\275\020|`\245w\000\344\336\277\210\306\3340\210\331\206\032\2070\360\324#\273%C\222\034!\b\ \205\324\216C\362a\016^^,\232\211\272nY\t\022\006\341R\34388\256\3232\0319\215.\006\004$\207`\200\305O\202\004\351dc\2048M\024u\226\ \314\241$\252\202\222$\252\n\030\262\302\b\232\232\006\207\333\265=\006\017\32\ 7\aFq\313\032\022\243T\321\234`h\223yB\250\240*\244\244*\234\3403+U3\343'p\350\206\"#\242\037VH\264\b\304)@\344\313\016\2621\203'\2\ 34\256*5\033\214\bU\t\f31\330\230\332\221\212\274e\254\210c\v\"\310\224c\307\033H\305F'\2167c\302\222\270\325#l\a XV\024\031\a&F2\2\ 54E\240X\340\r\210LEF6\021c2\022\264\262F1\270\023-\264\254\267 \260m\2128\334yaI\b^\344\222\024\250\350\204o\214\016\311\r\003\226\ \220\263\003\300\b\246MRU\216y\025\037\002\036\000\352\306\220\247^\376\363+!k`\352#\363\260\275pN+@\234\205\340\250\315-\021\206\0\ 35$p\264UP\222e\026\252\024\307\247B,\347\200<(\373-\322\fy\a\220Hn\221<\247P^\343\246\305:\r\336d\310\365\262\241H4l\223\316\205\?\ \t\357\?,\373\367\027\327b<\220R\024\245:\?\033\205\301\2027@A;\n\241;\362\000;\rd8HT\337\267\031h\\J\266\2647G\036\027 \340\206 \3\ 61L\004\205P-u\304\271\035\320S\337\225\371\340@\036\313\b\376\a\322e\366P\376\002\240\rR\207\220J\006'\000\035\020\036\024RB\343;\\ 310lK\372S2\3353@\242,\222\003@\026\016\207\240\276\270\036\305o\033\213hq\313\202\030\356'/\332KSO}\311A\270\300c\r`\311\242\000\2\ 07\n\235\215/h\003\t\001\200\231'O&\223\004'\220\366\241SE\212\002\2031\016x\213`\323\235\347\203\200\034\366\363|\000\300\305\360\\ 254\r\017f\033\fEH(\030\271\326\035\bu\356%\204t\200w\301\202\021UkZ\032sfd\213\241\332\354\252\251 &\207Q\222\304\204\222P\224U \3\ 16\002FV\360\317\036\267p\a\246Z\001\311_\000\316\252\001\263\000\324\233&\020\244c\001[\252\022]\005\332\202\244\nU\243\020\312\26\ 2\\\232r4\332\263\fhh\313%\226h\246*\t\224\311\312\274I2\ra\211\231\215\231\210A\271Ju:\017\021K\025C\344\374\316\303\304\214;\325\\ b\262\303\253@1U\002z\273\321)\202\223\b\322m\200\315\f\002\225\\\261V6g\230E\307\320\351\236\201\t:8w\366\355\337\277M\270\030f\31\ 3\026z\335\301A|\271\026\251\210\016&Y\034.\321r\327U[4\311o\252\212\221W\305\231\261a\026\355\366C\324\272bR\016RL\033\2660\223\00\ 1\202Lr\237\230\263\001L\000\334\224Z\037\"H\212\246\006\235\207\222\264\020\271\210\321\270\216\374\246\347\271v\343\213\327\030&\\ 315\331G\b[\334\303\f\210\3069\336\257j\255\246p\340\b\343Fqu\243A\246.)\027\030MN8\316\346A\235Z9K\233\033>=E\303\r\266\000N\2501f\ \214\200\206\370\357\231V\302\260B!\210S\305\016\353\251\307f\227Fv\254\226t\347\307\006\r\352\024\233 R\035\020:\224\017\230\305ds\ o'\214\004\324`\206\222\t\004p\312:y\2125\350\024k\232\230M\036\374\333\266\232m\235Ih\030heEb0XK\022K#\004b\034la\364\261*\025\320\ \364\374\315\355\302~\240i\307\031\0166a\331'\213\243\305\035K\002M\314c^;\264;\2121\0256\306\031w\252\253K\253\333\336\242fD\264=\ \260}\362\346\253Mop\232*{;\353wD\321\361K\3264vd\244\234\bv\346\261\241YaL\203\030`\342\303n\205\325\357;\242\342\033\270\306\363\\ 016\234e\273\227\275\342,\3329f#$\f\341\001\223/\340\274\024\236\313\\ b2\252\352\312\272\253|p\360C\261\3048\343j\0248t\021\220\2711\035\?Gs~e\016\263u\251\3567\316\261\357\221\203\242T\\diQm\220\316eB\ w\343g\361pH\340j\307\312t\022F\3752\367\263\316P\322\3072\\s\242Z\312\324\\`m\306\232\246\3425\021\260\307\350D\374\324\".\231\n\3\ 40,\325\256\242L\314\270\253V\306\r\247\216:\334c\311Uczp\360\344\246\213\274iZT\310\252\230\205\005qT\206Z\216\206,\323\263\261\24\ 4\232\246\361\334\211\344\350\316\354s1\356\220\326\261\3111\2601\300\002\202\213\3320\036\025O-A\3372^\023\352\341{H\225\302\355S\\ 200\227\364Q\370!\223\342\203\275\357.\270a\023\026V.Ux\370\232\323\303\347;\260\taZ(T\304\2616'\021\320h\210\"\345\234#0\310q\236\\ 027\311\260\337\233O\247\201^\335\020Y\274\315\213\200\242\350&\320\312\b\301E\b\353`V\232\310-\326X\035\241\320k\023\f\227\b\244\3\ 06\310\350<\004\340\000\003]\vd\323\004\3333\350\260\2050\212\023\223\205J2y\350\350\323\327\322\006\004\264D2\262@\3600\035>Vd0\257\200\272\023D>\022\322\321x\"\256\203\221\2\ 00\342\241\340\334\233\356\203\314Q \340\221\3004\024$\222\204\313\000S5\r\004%\034\241\333X\020\232:!2\321\231\216\035\006\n\355 \\ \\a\237\351\037:\311\2240\300/\023=\002\227\201\246PJ\240\357\017e\334\202C\354l\371\367\302Cd\241\005\223\322W%\n:\r4\022<8x\356B\\ 037\\\v*~\337\234D\b\030sjZp\354\354\201\211\217It.(8iI\037\275\\\310\021E\356\360\222\313\300x!4\264n mVQi!\020&\213\035L\314a\205\ \001D4\210\315L\323\022\305\202\373\332\006*\210\210\246u+t\337&\372\025\276\216\267\2325\225T\327^vh\222\016l\r.\256,9\211\327\a\3\ 33J\231\230\365\307M\224C\000lc\002f\261/6\200\240'\275Fc&\r\357\"`\233b\200H\262\310r\344;\204\321\252\261\177\\\210P\301\a*\346\0\ 34\r\b\313f\270\261\207F\213&\n\256\344\314\212\016\205\020\221\201\255=\206\250\221Sg@\240\205\203\225\221T/-9q\023\221%m4\222-\"u\ )\334\311a\206j\213\221\266\320\361\327\260o\207swF\270\201\276\033\3326\203\035`\345\304oL]\263ooL\264\347p\316&\370fh\235\006Wm\2\ 00\205:.\022&\321\225\241\002\210} \204\226\301\026$\3006i\220\2123l(E\316\035\344gG\316k\216\231\203\f\350Q\263:\020[op\024,\307 \\ 034^u\004nrl//J\350yA/\360.\320@\3265\306\025dQq\201 \200\227<\304\361C\273\252\034!\303\324[8\335\231\336\36789\267\246\310\202\33\ 0\"\bl!\231\214VLMT\242\255\b\304I\262\312c\021\223T\024P@r\206\vA;V\264\354\202Xe4t\252\262\230m\233woEv\225=\023\266^\036\203}\26\ 38\201\tNE\211,\036\25732\3260\260\211\004\346F\344(\b\325\270\247\246jF1v\035\225f\271\214\021#,\212@\214\004\314\341\312\"\222\00\ 5\212\nvcr\020w\2515\212\323\215\000w\260\360C\250}\262\024]\2140'v\342\0040W(\022\362\356-\000\226A\230x\203\270\236\240;\312\016\\ 367\230qs\017#U L\204\030\352\004D\020\v\323\272\301t\223\242\347LT\224R!\020\020\220\351\332R\201XX4\005U\210Z\224\332\211\2168B\3\ 42B\201\026\004\2351\330\236L\207\217\274\256\241\232\r\3055R\024@R\340\302\035\321I\260\211\003\356\262P\035\341!\222)J\034\t\326H\ k\260o`\221MDN\035\224e\315\344\347\263\021\210QE\223\274\231\270S\201EiF\333M\004\225\3324\243(\303D0\313\212:\220\260f.\205\"\226\ \237\vl4\222\214>\033\367h\351\216-\301\276\272\331 \020\210\321\274\020\232\016\0251\261\002\201s\a\267NX\231\"\242\005^\212\251\2\ 04\322@\016m\351\234a\005p\356J\0007yv\326\020D\"pm$4\367\f\016 \334\236\226P\322ipw\207\225\225\020\334\020\234Ohk^bY'\022\331^\36\ 1Eqh\244X\302b\221\210\240\030y\337Y5\233f\352k}\265\331\022\272Q5D\300\346\341\361\023\335HD$\351\025g\351c\220\n\n\2113\036\332q\\ 032k\313&\021\350\212\017\265F\211\221<\2208\024\322\2360\a\035\302\">yb\001B\220@qyC\252eAP(B\260\001G\323\225\r\006\023`\357\326\\ 364@\022\261F\207(\340\220\342\216u.\330\t\252\003\201q\b\251Kb\314\004\312\314&\370G6E\226\220\312Nj\006\034t\246\332\244\3002uH|#\ 5\223\340\356\343\327]\306'\256\3737O\203\265\324\232\3359(<(\225\001\300\264P\020_6\206O\005\335 \b\003\"Eg%\004\200\326H\325\221\\ 246\354 (\340\221\aX\250\355\200\335L\v\262\351\264)\272\251h\260\202\200\205\325H\240\240:3\003\n\203*\226.\306D9\274l\315\221\306\ \214&\a\216H\342\200\332,\321\244\222\230\236Hi\301\217%\226\2142\263(\200\251\343\"$\336\024Di\306\302bc\215@\216B8\245\245\254s]\\ 202\r\243LO\221\243\207\307wQ\214\036\312\331\251\0177\267V\017\bDq\234tM\t\0061\002\322\005\n\001S\000R\006!\350\a\023\207J\221\20\ 1\001\355\260\314\261*\001\323\25141!\214&T\024\024\001J\205%4R`f\031\025\222R\025\220\345U@\231\006JENJ\241\263\214w\263\0006X\300\ \333\035\220\314\017@\206\233\224\272\326YM\202\314\360\224s\236l\352\226\251\214T\361\300\0367\213\314\263.Y-\003j(\322\3445\316\2\ 42iF\221\024i\241DW\3015(F\f:\tJ\276~S]\266\027\bx\301\242`:o\257*\276BJ\363\234\036\217\032&\bhj(\245`\240%O\021;~\017^H\347\254\3\ 01R\211UY\344`\205\350v(H\363#\253#l\204)\223\263\331yz\026P!\227\214`\264\0264\036\266\203\315\016\225\016@\201\ \270!6\217\"\302\003\226 S\035\004\234!\214C\226/6g\005\005\331R\374\017{\037o\330/\325\247:\317V\376Y;\234\211\325\224\271\\ 243\302\021\020\272\210\214, \306\035\370K!\030\357\2033\036\2554\360v\001\211\261\357(\350\233M\252\314~,\252\266\032\342K;)5\333;\ t\240\316\2334&ri\213\2348\004r0@\314\\\211\317\212\257aN7@\221%\034\242\373^G\2174\252\037D)\347-\357<\270\000\2752\b@\027\233[K\2\ 11\233M\323\262\201\t\002\032\216&D\026D$E;\023\255\f\017C\220\323gx\020O\377\027rE8P\220\365\004\371\?"; ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/help/help.c0000664000175100017510000007115315240547630017653 0ustar00runnerrunner/*********************************************************************************************************************************** Help Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/help/help.h" #include "common/compress/bz2/decompress.h" #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/memContext.h" #include "common/type/pack.h" #include "config/config.intern.h" #include "config/parse.h" #include "version.h" /*********************************************************************************************************************************** Define the console width - use a fixed width of 80 since this should be safe on virtually all consoles ***********************************************************************************************************************************/ #define CONSOLE_WIDTH 80 /*********************************************************************************************************************************** Helper function to split a string into a string list based on a delimiter and max size per item. In other words each item in the list will be no longer than size even if multiple delimiters are skipped. This is useful for breaking up text on spaces, for example. ***********************************************************************************************************************************/ static StringList * helpRenderSplitSize(const String *const string, const char *const delimiter, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_PARAM(STRINGZ, delimiter); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(string != NULL); ASSERT(delimiter != NULL); ASSERT(size > 0); // Create the list StringList *const this = strLstNew(); // Base points to the beginning of the string that is being searched const char *stringBase = strZ(string); // Match points to the next delimiter match that has been found const char *stringMatchLast = NULL; const char *stringMatch = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { do { // Find a delimiter match stringMatch = strstr(stringMatchLast == NULL ? stringBase : stringMatchLast, delimiter); // If a match was found then add the string if (stringMatch != NULL) { if ((size_t)(stringMatch - stringBase) >= size) { if (stringMatchLast != NULL) stringMatch = stringMatchLast - strlen(delimiter); strLstAddZSub(this, stringBase, (size_t)(stringMatch - stringBase)); stringBase = stringMatch + strlen(delimiter); stringMatchLast = NULL; } else stringMatchLast = stringMatch + strlen(delimiter); } // Else make whatever is left the last string else { if (stringMatchLast != NULL && strlen(stringBase) - strlen(delimiter) >= size) { strLstAddZSub(this, stringBase, (size_t)((stringMatchLast - strlen(delimiter)) - stringBase)); stringBase = stringMatchLast; } strLstAddZ(this, stringBase); } } while (stringMatch != NULL); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN String * helpRenderText( const String *const text, const bool internal, const bool beta, const size_t indent, const bool indentFirst, const size_t length) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, text); FUNCTION_TEST_PARAM(BOOL, internal); FUNCTION_TEST_PARAM(BOOL, beta); FUNCTION_TEST_PARAM(SIZE, indent); FUNCTION_TEST_PARAM(BOOL, indentFirst); FUNCTION_TEST_PARAM(SIZE, length); FUNCTION_TEST_END(); ASSERT(text != NULL); ASSERT(length > 0); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // Split the text into paragraphs const StringList *const lineList = strLstNewSplitZ( strNewFmt( "%s%s", strZ(text), beta ? "\n\nFOR BETA TESTING ONLY. DO NOT USE IN PRODUCTION." : (internal ? "\n\nFOR INTERNAL USE ONLY. DO NOT USE IN PRODUCTION." : "")), "\n"); // Iterate through each paragraph and split the lines according to the line length for (unsigned int lineIdx = 0; lineIdx < strLstSize(lineList); lineIdx++) { // Add LF if there is already content if (strSize(result) != 0) strCat(result, LF_STR); // Split the paragraph into lines that don't exceed the line length const StringList *const partList = helpRenderSplitSize(strLstGet(lineList, lineIdx), " ", length - indent); for (unsigned int partIdx = 0; partIdx < strLstSize(partList); partIdx++) { // Indent when required if (partIdx != 0 || indentFirst) { if (partIdx != 0) strCat(result, LF_STR); if (strSize(strLstGet(partList, partIdx))) strCatFmt(result, "%*s", (int)indent, ""); } // Add the line strCat(result, strLstGet(partList, partIdx)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Helper functions for helpRender() to output values as strings ***********************************************************************************************************************************/ static String * helpRenderValueIdx(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); String *result = NULL; if (cfgOptionIdxSource(optionId, optionIdx) != cfgSourceDefault) { result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { const Variant *const value = cfgOptionIdxVar(optionId, optionIdx); ASSERT(value != NULL); switch (varType(value)) { case varTypeKeyValue: { const KeyValue *const optionKv = varKv(value); const VariantList *const keyList = kvKeyList(optionKv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { if (keyIdx != 0) strCatZ(result, ", "); strCatFmt( result, "%s=%s", strZ(varStr(varLstGet(keyList, keyIdx))), strZ(varStrForce(kvGet(optionKv, varLstGet(keyList, keyIdx))))); } break; } case varTypeVariantList: { const VariantList *const list = varVarLst(value); for (unsigned int listIdx = 0; listIdx < varLstSize(list); listIdx++) { if (listIdx != 0) strCatZ(result, ", "); strCatFmt(result, "%s", strZ(varStr(varLstGet(list, listIdx)))); } break; } default: strCat(result, cfgOptionIdxDisplay(optionId, optionIdx)); break; } } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } static String * helpRenderValue(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); String *result = helpRenderValueIdx(optionId, 0); if (cfgOptionGroup(optionId)) { MEM_CONTEXT_TEMP_BEGIN() { for (unsigned int optionIdx = 1; optionIdx < cfgOptionGroupIdxTotal(cfgOptionGroupId(optionId)); optionIdx++) { const String *const value = helpRenderValueIdx(optionId, optionIdx); if (!strEq(result, value)) { MEM_CONTEXT_PRIOR_BEGIN() { strFree(result); result = strNewZ(""); } MEM_CONTEXT_PRIOR_END(); break; } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Helper function for helpRender() to output defaults ***********************************************************************************************************************************/ static String * helpRenderDefault(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); String *result = strDup(cfgOptionIdxDefaultValue(optionId, 0)); if (cfgOptionGroup(optionId)) { MEM_CONTEXT_TEMP_BEGIN() { for (unsigned int optionIdx = 1; optionIdx < cfgOptionGroupIdxTotal(cfgOptionGroupId(optionId)); optionIdx++) { const String *const defaultValue = cfgOptionIdxDefaultValue(optionId, optionIdx); if (!strEq(result, defaultValue)) { MEM_CONTEXT_PRIOR_BEGIN() { strFree(result); result = strNewZ(""); } MEM_CONTEXT_PRIOR_END(); break; } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Determine if the first character of a summary should be lower-case ***********************************************************************************************************************************/ static String * helpRenderSummary(const String *const summary) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, summary); FUNCTION_TEST_END(); ASSERT(summary != NULL); // Strip final period off summary String *const result = strCatN(strNew(), summary, strSize(summary) - 1); // Lower-case first letter if first word does not appear to be an acronym or proper name unsigned int totalLetter = 0; unsigned int totalCapital = 0; for (unsigned int resultIdx = 0; resultIdx < strSize(result); resultIdx++) { const char resultChar = strZ(result)[resultIdx]; if (resultChar == ' ') break; if (isalpha(resultChar)) totalLetter++; if (isupper(resultChar)) totalCapital++; } if (totalCapital == 1 && totalCapital != totalLetter) strFirstLower(result); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Render help to a string ***********************************************************************************************************************************/ // Stored unpacked command data typedef struct HelpCommandData { bool internal; // Is the command internal? const String *summary; // Short summary of the command const String *description; // Full description of the command } HelpCommandData; // Stored unpacked option data typedef struct HelpOptionData { bool internal; // Is the option internal? const String *section; // eg. general, command const String *summary; // Short summary of the option const String *description; // Full description of the option StringList *deprecatedNames; // Deprecated names for the option } HelpOptionData; static String * helpRender(const Buffer *const helpData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, helpData); FUNCTION_TEST_END(); String *const result = strCatZ(strNew(), PROJECT_NAME " " PROJECT_VERSION); // Display version only if (!cfgCommandHelp() && ((cfgCommand() == cfgCmdHelp && cfgOptionBool(cfgOptVersion) && !cfgOptionBool(cfgOptHelp)) || cfgCommand() == cfgCmdVersion)) { if (cfgCommand() == cfgCmdVersion && cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_VERSION_OUTPUT_NUM) FUNCTION_TEST_RETURN(STRING, strCatFmt(strTrunc(result), "%d", PROJECT_VERSION_NUM)); else { strCatChr(result, '\n'); FUNCTION_TEST_RETURN(STRING, result); } } MEM_CONTEXT_TEMP_BEGIN() { // Set a small buffer size to minimize memory usage ioBufferSizeSet(8192); // Read pack from compressed buffer IoRead *const helpRead = ioBufferReadNew(helpData); ioFilterGroupAdd(ioReadFilterGroup(helpRead), bz2DecompressNew(false)); ioReadOpen(helpRead); PackRead *const pckHelp = pckReadNewIo(helpRead); // Unpack command data HelpCommandData *const commandData = memNew(sizeof(HelpCommandData) * CFG_COMMAND_TOTAL); pckReadArrayBeginP(pckHelp); for (ConfigCommand commandId = 0; commandId < CFG_COMMAND_TOTAL; commandId++) { commandData[commandId] = (HelpCommandData) { .internal = pckReadBoolP(pckHelp), .summary = pckReadStrP(pckHelp), .description = pckReadStrP(pckHelp), }; } pckReadArrayEndP(pckHelp); // Message for more help when it is available const String *more = NULL; // Display general help if (!cfgCommandHelp()) { strCatZ( result, " - General help\n" "\n" "Usage:\n" " " PROJECT_BIN " [options] [command]\n" "\n" "Commands:\n"); // Find size of longest command name size_t commandSizeMax = 0; for (ConfigCommand commandId = 0; commandId < CFG_COMMAND_TOTAL; commandId++) { if (commandData[commandId].internal) continue; if (strlen(cfgParseCommandName(commandId)) > commandSizeMax) commandSizeMax = strlen(cfgParseCommandName(commandId)); } // Output help for each command for (ConfigCommand commandId = 0; commandId < CFG_COMMAND_TOTAL; commandId++) { if (commandData[commandId].internal) continue; strCatFmt( result, " %s%*s%s\n", cfgParseCommandName(commandId), (int)(commandSizeMax - strlen(cfgParseCommandName(commandId)) + 2), "", strZ( helpRenderText( helpRenderSummary(commandData[commandId].summary), false, false, commandSizeMax + 6, false, CONSOLE_WIDTH))); } // Construct message for more help more = strNewZ("[command]"); } else { const ConfigCommand commandId = cfgCommand(); const char *const commandName = cfgParseCommandName(commandId); // Unpack option data HelpOptionData *const optionData = memNew(sizeof(HelpOptionData) * CFG_OPTION_TOTAL); pckReadArrayBeginP(pckHelp); for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { optionData[optionId] = (HelpOptionData) { .internal = pckReadBoolP(pckHelp), .section = pckReadStrP(pckHelp, .defaultValue = STRDEF("general")), .summary = pckReadStrP(pckHelp), .description = pckReadStrP(pckHelp), }; // Unpack deprecated names if (!pckReadNullP(pckHelp)) { optionData[optionId].deprecatedNames = strLstNew(); pckReadArrayBeginP(pckHelp); while (pckReadNext(pckHelp)) strLstAdd(optionData[optionId].deprecatedNames, pckReadStrP(pckHelp)); pckReadArrayEndP(pckHelp); } // Unpack command overrides if (!pckReadNullP(pckHelp)) { pckReadArrayBeginP(pckHelp); while (pckReadNext(pckHelp)) { // Get command override id const ConfigCommand commandIdArray = pckReadId(pckHelp) - 1; // Unpack override data pckReadObjBeginP(pckHelp, .id = commandIdArray + 1); const bool internal = pckReadBoolP(pckHelp, .defaultValue = optionData[optionId].internal); const String *const summary = pckReadStrP(pckHelp, .defaultValue = optionData[optionId].summary); const String *const description = pckReadStrP(pckHelp, .defaultValue = optionData[optionId].description); pckReadObjEndP(pckHelp); // Only use overrides for the current command if (commandId == commandIdArray) { optionData[optionId].internal = internal; optionData[optionId].section = NULL; optionData[optionId].summary = summary; optionData[optionId].description = description; } } pckReadArrayEndP(pckHelp); } } pckReadArrayEndP(pckHelp); // Output command part of title strCatFmt(result, " - '%s' command", commandName); // If no additional params then this is command help if (strLstEmpty(cfgCommandParam())) { // Output command summary and description. Add a warning for internal commands. CHECK( AssertError, commandData[commandId].summary != NULL && commandData[commandId].description != NULL, "command help missing"); strCatFmt( result, " help\n" "\n" "%s\n" "\n" "%s\n", strZ(helpRenderText(commandData[commandId].summary, false, false, 0, true, CONSOLE_WIDTH)), strZ( helpRenderText( commandData[commandId].description, commandData[commandId].internal, false, 0, true, CONSOLE_WIDTH))); // Construct key/value of sections and options KeyValue *const optionKv = kvNew(); size_t optionSizeMax = 0; for (unsigned int optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { if (cfgParseOptionValid(commandId, cfgCmdRoleMain, optionId) && !optionData[optionId].internal) { const String *section = optionData[optionId].section; if (section == NULL || (!strEqZ(section, "general") && !strEqZ(section, "log") && !strEqZ(section, "maintainer") && !strEqZ(section, "repository") && !strEqZ(section, "stanza"))) { section = strNewZ("command"); } kvAdd(optionKv, VARSTR(section), VARUINT(optionId)); if (strlen(cfgParseOptionName(optionId)) > optionSizeMax) optionSizeMax = strlen(cfgParseOptionName(optionId)); } } // Output sections const StringList *const sectionList = strLstSort(strLstNewVarLst(kvKeyList(optionKv)), sortOrderAsc); for (unsigned int sectionIdx = 0; sectionIdx < strLstSize(sectionList); sectionIdx++) { const String *const section = strLstGet(sectionList, sectionIdx); strCatFmt(result, "\n%s Options:\n\n", strZ(strFirstUpper(strDup(section)))); // Output options const VariantList *const optionList = kvGetList(optionKv, VARSTR(section)); for (unsigned int optionIdx = 0; optionIdx < varLstSize(optionList); optionIdx++) { const ConfigOption optionId = varUInt(varLstGet(optionList, optionIdx)); String *const summary = helpRenderSummary(optionData[optionId].summary); // Output current and default values if they exist const String *const defaultValue = helpRenderDefault(optionId); const String *const value = helpRenderValue(optionId); if (value != NULL || defaultValue != NULL) { strCatZ(summary, " ["); if (value != NULL) strCatFmt(summary, "current=%s", cfgParseOptionSecure(optionId) ? "" : strZ(value)); if (defaultValue != NULL) { if (value != NULL) strCatZ(summary, ", "); strCatFmt(summary, "default=%s", strZ(defaultValue)); } strCatZ(summary, "]"); } // Output option help strCatFmt( result, " --%s%*s%s\n", cfgParseOptionName(optionId), (int)(optionSizeMax - strlen(cfgParseOptionName(optionId)) + 2), "", strZ(helpRenderText(summary, false, false, optionSizeMax + 6, false, CONSOLE_WIDTH))); } } // Construct message for more help ASSERT(optionSizeMax > 0); more = strNewFmt("%s [option]", commandName); } // Else option help for the specified command else { // Make sure only one option was specified if (strLstSize(cfgCommandParam()) > 1) THROW(ParamInvalidError, "only one option allowed for option help"); // Ensure the option is valid const String *const optionName = strLstGet(cfgCommandParam(), 0); const CfgParseOptionResult option = cfgParseOptionP(optionName, .ignoreMissingIndex = true); // Error when option is not found or is invalid for the current command if (!option.found || !cfgParseOptionValid(cfgCommand(), cfgCmdRoleMain, option.id)) THROW_FMT(OptionInvalidError, "option '%s' is not valid for command '%s'", strZ(optionName), commandName); // Output option summary and description. Add a warning for internal and beta options. CHECK( AssertError, optionData[option.id].summary != NULL && optionData[option.id].description != NULL, "option help missing"); strCatFmt( result, " - '%s' option help\n" "\n" "%s\n" "\n" "%s\n", cfgParseOptionName(option.id), strZ(helpRenderText(optionData[option.id].summary, false, false, 0, true, CONSOLE_WIDTH)), strZ( helpRenderText( optionData[option.id].description, optionData[option.id].internal, option.beta, 0, true, CONSOLE_WIDTH))); // Output current and default values if they exist const String *const defaultValue = helpRenderDefault(option.id); const String *const value = helpRenderValue(option.id); if (value != NULL || defaultValue != NULL) { strCat(result, LF_STR); if (value != NULL) { strCatZ(result, "current:"); if (cfgParseOptionSecure(option.id)) strCatZ(result, " \n"); else if (!strEqZ(value, "")) strCatFmt(result, " %s\n", strZ(value)); else { const unsigned int groupId = cfgOptionGroupId(option.id); strCatChr(result, '\n'); for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(groupId); optionIdx++) { const String *const value = helpRenderValueIdx(option.id, optionIdx); if (value != NULL) strCatFmt(result, " %s: %s\n", cfgOptionGroupName(groupId, optionIdx), strZ(value)); } } } if (defaultValue != NULL) { strCatZ(result, "default:"); if (!strEqZ(defaultValue, "")) strCatFmt(result, " %s\n", strZ(defaultValue)); else { const unsigned int groupId = cfgOptionGroupId(option.id); strCatChr(result, '\n'); for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(groupId); optionIdx++) { const String *const defaultValue = cfgOptionIdxDefaultValue(option.id, optionIdx); if (defaultValue != NULL) strCatFmt(result, " %s: %s\n", cfgOptionGroupName(groupId, optionIdx), strZ(defaultValue)); } } } } // Output alternate name (call it deprecated so the user will know not to use it) if (optionData[option.id].deprecatedNames != NULL) { strCatFmt( result, "\ndeprecated name%s: %s\n", strLstSize(optionData[option.id].deprecatedNames) > 1 ? "s" : "", strZ(strLstJoin(optionData[option.id].deprecatedNames, ", "))); } } } // If there is more help available output a message to let the user know if (more != NULL) strCatFmt(result, "\nUse '" PROJECT_BIN " help %s' for more information.\n", strZ(more)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdHelp(const Buffer *const helpData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BUFFER, helpData); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { ioFdWriteOneStr(STDOUT_FILENO, helpRender(helpData)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/help/help.h0000664000175100017510000000157415240547630017660 0ustar00runnerrunner/*********************************************************************************************************************************** Help Command ***********************************************************************************************************************************/ #ifndef COMMAND_HELP_HELP_H #define COMMAND_HELP_HELP_H #include "common/type/buffer.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Render help and output to stdout FN_EXTERN void cmdHelp(const Buffer *const helpData); // Render help text FN_EXTERN String *helpRenderText(const String *text, bool internal, bool beta, size_t indent, bool indentFirst, size_t length); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/help/meson.build0000664000175100017510000000155415240547630020717 0ustar00runnerrunner#################################################################################################################################### # Generate help #################################################################################################################################### # Generate help.auto.c.inc, unless this is a distribution tarball that ships the file if not dist help_auto_c_inc = custom_target( 'help.auto.c.inc', output: 'help.auto.c.inc', depend_files: [ '../../build/config/config.yaml', '../../build/help/help.xml', ], command: [ build_code, 'help', '@CURRENT_SOURCE_DIR@/../../..', ], ) else # The shipped file is included by main.c via the implicit source include path, so it is not a compiled source help_auto_c_inc = [] endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/info/0000775000175100017510000000000015240547630016553 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/info/info.c0000664000175100017510000027253615240547630017671 0ustar00runnerrunner/*********************************************************************************************************************************** Info Command ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/archive/common.h" #include "command/info/info.h" #include "command/lock.h" #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/type/json.h" #include "config/config.h" #include "info/info.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/infoPg.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ // Naming convention: _KEY__VAR. If the key exists in multiple sections, then _ is omitted. VARIANT_STRDEF_STATIC(ARCHIVE_KEY_MIN_VAR, "min"); VARIANT_STRDEF_STATIC(ARCHIVE_KEY_MAX_VAR, "max"); VARIANT_STRDEF_STATIC(BACKREST_KEY_FORMAT_VAR, "format"); VARIANT_STRDEF_STATIC(BACKREST_KEY_VERSION_VAR, "version"); VARIANT_STRDEF_STATIC(BACKUP_KEY_ANNOTATION_VAR, "annotation"); VARIANT_STRDEF_STATIC(BACKUP_KEY_BACKREST_VAR, "backrest"); VARIANT_STRDEF_STATIC(BACKUP_KEY_ERROR_VAR, "error"); VARIANT_STRDEF_STATIC(BACKUP_KEY_ERROR_LIST_VAR, "error-list"); VARIANT_STRDEF_STATIC(BACKUP_KEY_DATABASE_REF_VAR, "database-ref"); VARIANT_STRDEF_STATIC(BACKUP_KEY_INFO_VAR, "info"); VARIANT_STRDEF_STATIC(BACKUP_KEY_LABEL_VAR, "label"); VARIANT_STRDEF_STATIC(BACKUP_KEY_LINK_VAR, "link"); VARIANT_STRDEF_STATIC(BACKUP_KEY_LSN_VAR, "lsn"); VARIANT_STRDEF_STATIC(BACKUP_KEY_PRIOR_VAR, "prior"); VARIANT_STRDEF_STATIC(BACKUP_KEY_REFERENCE_VAR, "reference"); VARIANT_STRDEF_STATIC(BACKUP_KEY_REFERENCE_FULL_VAR, "reference-full"); VARIANT_STRDEF_STATIC(BACKUP_KEY_REFERENCE_DIFF_VAR, "reference-diff"); VARIANT_STRDEF_STATIC(BACKUP_KEY_REFERENCE_INCR_VAR, "reference-incr"); VARIANT_STRDEF_STATIC(BACKUP_KEY_TABLESPACE_VAR, "tablespace"); VARIANT_STRDEF_STATIC(BACKUP_KEY_TIMESTAMP_VAR, "timestamp"); VARIANT_STRDEF_STATIC(BACKUP_KEY_TYPE_VAR, "type"); VARIANT_STRDEF_STATIC(DB_KEY_ID_VAR, "id"); VARIANT_STRDEF_STATIC(DB_KEY_SYSTEM_ID_VAR, "system-id"); VARIANT_STRDEF_STATIC(DB_KEY_VERSION_VAR, "version"); VARIANT_STRDEF_STATIC(INFO_KEY_REPOSITORY_VAR, "repository"); VARIANT_STRDEF_STATIC(KEY_ARCHIVE_VAR, "archive"); VARIANT_STRDEF_STATIC(KEY_CIPHER_VAR, "cipher"); VARIANT_STRDEF_STATIC(KEY_DATABASE_VAR, "database"); VARIANT_STRDEF_STATIC(KEY_DELTA_VAR, "delta"); VARIANT_STRDEF_STATIC(KEY_DELTA_MAP_VAR, "delta-map"); VARIANT_STRDEF_STATIC(KEY_DESTINATION_VAR, "destination"); VARIANT_STRDEF_STATIC(KEY_NAME_VAR, "name"); VARIANT_STRDEF_STATIC(KEY_OID_VAR, "oid"); VARIANT_STRDEF_STATIC(KEY_REPO_KEY_VAR, "repo-key"); VARIANT_STRDEF_STATIC(KEY_SIZE_VAR, "size"); VARIANT_STRDEF_STATIC(KEY_SIZE_MAP_VAR, "size-map"); VARIANT_STRDEF_STATIC(KEY_START_VAR, "start"); VARIANT_STRDEF_STATIC(KEY_STOP_VAR, "stop"); VARIANT_STRDEF_STATIC(REPO_KEY_KEY_VAR, "key"); VARIANT_STRDEF_STATIC(STANZA_KEY_BACKUP_VAR, "backup"); VARIANT_STRDEF_STATIC(STANZA_KEY_REPO_VAR, "repo"); VARIANT_STRDEF_STATIC(STANZA_KEY_STATUS_VAR, "status"); VARIANT_STRDEF_STATIC(STANZA_KEY_DB_VAR, "db"); VARIANT_STRDEF_STATIC(STATUS_KEY_CODE_VAR, "code"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_VAR, "lock"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_BACKUP_VAR, "backup"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_HELD_VAR, "held"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR, "pct-cplt"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_RESTORE_VAR, "restore"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_SIZE_COMPLETE_VAR, "size-cplt"); VARIANT_STRDEF_STATIC(STATUS_KEY_LOCK_SIZE_VAR, "size"); VARIANT_STRDEF_STATIC(STATUS_KEY_MESSAGE_VAR, "message"); #define INFO_STANZA_STATUS_OK "ok" #define INFO_STANZA_STATUS_ERROR "error" #define INFO_STANZA_MIXED "mixed" #define INFO_STANZA_STATUS_CODE_OK 0 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_OK_STR, "ok"); #define INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH 1 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_PATH_STR, "missing stanza path"); #define INFO_STANZA_STATUS_CODE_NO_BACKUP 2 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_NO_BACKUP_STR, "no valid backups"); #define INFO_STANZA_STATUS_CODE_MISSING_STANZA_DATA 3 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_DATA_STR, "missing stanza data"); #define INFO_STANZA_STATUS_CODE_MIXED 4 // If the cipher or status of the stanza is different across repos, then the overall cipher or status message is mixed STRING_STATIC(INFO_STANZA_MESSAGE_MIXED_STR, "different across repos"); #define INFO_STANZA_STATUS_CODE_PG_MISMATCH 5 STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_PG_MISMATCH_STR, "database mismatch across repos"); #define INFO_STANZA_STATUS_CODE_BACKUP_MISSING 6 STRING_STATIC(INFO_STANZA_STATUS_CODE_BACKUP_MISSING_STR, "requested backup not found"); #define INFO_STANZA_STATUS_CODE_OTHER 99 #define INFO_STANZA_STATUS_MESSAGE_OTHER "other" STRING_STATIC(INFO_STANZA_STATUS_MESSAGE_OTHER_STR, INFO_STANZA_STATUS_MESSAGE_OTHER); STRING_STATIC(INFO_STANZA_INVALID_STR, "[invalid]"); #define INFO_STANZA_STATUS_MESSAGE_LOCK_BACKUP "backup/expire running" #define INFO_STANZA_STATUS_MESSAGE_LOCK_RESTORE "restore running" /*********************************************************************************************************************************** Data types and structures ***********************************************************************************************************************************/ // Information for a lockfile of a stanza typedef struct InfoStanzaLock { bool held; // Is lock held on the system where info command is run? uint64_t sizeComplete; // Completed size of the backup/restore in bytes uint64_t size; // Total size of the backup/restore in bytes } InfoStanzaLock; #define FUNCTION_LOG_INFO_STANZA_LOCK_TYPE \ InfoStanzaLock * #define FUNCTION_LOG_INFO_STANZA_LOCK_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoStanzaLock", buffer, bufferSize) // Repository information for a stanza typedef struct InfoRepoData { unsigned int key; // User-defined repo key CipherType cipher; // Encryption type const String *cipherPass; // Passphrase if the repo is encrypted (else NULL) int stanzaStatus; // Status code of the stanza on this repo unsigned int backupIdx; // Index of the next backup that may be a candidate for sorting InfoBackup *backupInfo; // Contents of the backup.info file of the stanza on this repo InfoArchive *archiveInfo; // Contents of the archive.info file of the stanza on this repo Manifest *manifest; // Contents of manifest if backup requested and is on this repo String *error; // Formatted error InfoStanzaLock backupLock; // Info for backup lock on this repo } InfoRepoData; #define FUNCTION_LOG_INFO_REPO_DATA_TYPE \ InfoRepoData * #define FUNCTION_LOG_INFO_REPO_DATA_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoRepoData", buffer, bufferSize) // Stanza with repository list of information for each repository typedef struct InfoStanzaRepo { const String *name; // Name of the stanza uint64_t currentPgSystemId; // Current postgres system id for the stanza unsigned int currentPgVersion; // Current postgres version for the stanza InfoStanzaLock restoreLock; // Info for restore lock InfoRepoData *repoList; // List of configured repositories } InfoStanzaRepo; #define FUNCTION_LOG_INFO_STANZA_REPO_TYPE \ InfoStanzaRepo * #define FUNCTION_LOG_INFO_STANZA_REPO_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoStanzaRepo", buffer, bufferSize) // Group all databases with the same system-id and version together regardless of db-id or repo typedef struct DbGroup { uint64_t systemId; // Postgres database system id const String *version; // Postgres database version bool current; // Is this the current postgres database? String *archiveMin; // Minimum WAL found for this database over all repositories String *archiveMax; // Maximum WAL found for this database over all repositories VariantList *backupList; // List of backups found for this database over all repositories } DbGroup; #define FUNCTION_LOG_DB_GROUP_TYPE \ DbGroup * #define FUNCTION_LOG_DB_GROUP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "DbGroup", buffer, bufferSize) /*********************************************************************************************************************************** Helper function for reporting errors ***********************************************************************************************************************************/ static void infoStanzaErrorAdd(InfoRepoData *const repoList, const ErrorType *const type, const String *const message) { repoList->stanzaStatus = INFO_STANZA_STATUS_CODE_OTHER; repoList->error = strNewFmt("[%s] %s", errorTypeName(type), strZ(message)); // Free the info objects for this stanza since we cannot process it infoBackupFree(repoList->backupInfo); infoArchiveFree(repoList->archiveInfo); manifestFree(repoList->manifest); repoList->backupInfo = NULL; repoList->archiveInfo = NULL; repoList->manifest = NULL; } /*********************************************************************************************************************************** Add lock information to the target key-value ***********************************************************************************************************************************/ static void stanzaStatusLockAdd(KeyValue *targetKv, const Variant *const lockKey, const InfoStanzaLock *lock) { KeyValue *const lockKv = kvPutKv(targetKv, lockKey); kvPut(lockKv, STATUS_KEY_LOCK_HELD_VAR, VARBOOL(lock->held)); if (lock->size != 0) { kvPut(lockKv, STATUS_KEY_LOCK_SIZE_COMPLETE_VAR, VARUINT64(lock->sizeComplete)); kvPut(lockKv, STATUS_KEY_LOCK_SIZE_VAR, VARUINT64(lock->size)); if (cfgOptionSeq(cfgOptOutput) != CFGOPTVAL_OUTPUT_JSON) kvPut(lockKv, STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR, VARUINT(cvtPctToUInt(lock->sizeComplete, lock->size))); } } /*********************************************************************************************************************************** Add per-repo backup lock information to the target key-value ***********************************************************************************************************************************/ static void stanzaStatusBackupLockAdd( KeyValue *const targetKv, const InfoRepoData *const repoList, const unsigned int repoIdxMin, const unsigned int repoIdxMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, targetKv); FUNCTION_TEST_PARAM(INFO_REPO_DATA, repoList); FUNCTION_TEST_PARAM(UINT, repoIdxMin); FUNCTION_TEST_PARAM(UINT, repoIdxMax); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(targetKv != NULL); ASSERT(repoList != NULL); ASSERT(repoIdxMin <= repoIdxMax); KeyValue *const lockKv = kvPutKv(targetKv, STATUS_KEY_LOCK_BACKUP_VAR); bool held = false; MEM_CONTEXT_TEMP_BEGIN() { // Report backup progress per repo VariantList *const repoLockList = varLstNew(); uint64_t sizeTotal = 0; uint64_t sizeCompleteTotal = 0; for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { const InfoStanzaLock *const lock = &repoList[repoIdx].backupLock; if (lock->held) { held = true; if (lock->size != 0) { Variant *const repoLockInfo = varNewKv(kvNew()); kvPut(varKv(repoLockInfo), REPO_KEY_KEY_VAR, VARUINT(repoList[repoIdx].key)); kvPut(varKv(repoLockInfo), STATUS_KEY_LOCK_SIZE_VAR, VARUINT64(lock->size)); kvPut(varKv(repoLockInfo), STATUS_KEY_LOCK_SIZE_COMPLETE_VAR, VARUINT64(lock->sizeComplete)); varLstAdd(repoLockList, repoLockInfo); sizeTotal += lock->size; sizeCompleteTotal += lock->sizeComplete; } } } // Report backup progress aggregated across all repos kvPut(lockKv, STATUS_KEY_LOCK_HELD_VAR, VARBOOL(held)); if (!varLstEmpty(repoLockList)) { // Add summed size/size-cplt for backwards compatibility kvPut(lockKv, STATUS_KEY_LOCK_SIZE_COMPLETE_VAR, VARUINT64(sizeCompleteTotal)); kvPut(lockKv, STATUS_KEY_LOCK_SIZE_VAR, VARUINT64(sizeTotal)); // Add pct-cplt for text output only if (cfgOptionSeq(cfgOptOutput) != CFGOPTVAL_OUTPUT_JSON) kvPut(lockKv, STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR, VARUINT(cvtPctToUInt(sizeCompleteTotal, sizeTotal))); // Add per-repo detail kvPut(lockKv, STANZA_KEY_REPO_VAR, varNewVarLst(repoLockList)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set the overall error status code and message for the stanza to the code and message passed ***********************************************************************************************************************************/ static void stanzaStatus( const int code, const InfoStanzaRepo *const stanzaData, const Variant *const stanzaInfo, const unsigned int repoIdxMin, const unsigned int repoIdxMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_PARAM(INFO_STANZA_REPO, stanzaData); FUNCTION_TEST_PARAM(VARIANT, stanzaInfo); FUNCTION_TEST_PARAM(UINT, repoIdxMin); FUNCTION_TEST_PARAM(UINT, repoIdxMax); FUNCTION_TEST_END(); ASSERT((code >= 0 && code <= 6) || code == 99); ASSERT(stanzaData != NULL); ASSERT(stanzaInfo != NULL); KeyValue *const statusKv = kvPutKv(varKv(stanzaInfo), STANZA_KEY_STATUS_VAR); kvPut(statusKv, STATUS_KEY_CODE_VAR, VARINT(code)); switch (code) { case INFO_STANZA_STATUS_CODE_OK: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_OK_STR)); break; case INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_PATH_STR)); break; case INFO_STANZA_STATUS_CODE_MISSING_STANZA_DATA: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_DATA_STR)); break; case INFO_STANZA_STATUS_CODE_NO_BACKUP: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_NO_BACKUP_STR)); break; case INFO_STANZA_STATUS_CODE_MIXED: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_MESSAGE_MIXED_STR)); break; case INFO_STANZA_STATUS_CODE_PG_MISMATCH: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_PG_MISMATCH_STR)); break; case INFO_STANZA_STATUS_CODE_BACKUP_MISSING: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_CODE_BACKUP_MISSING_STR)); break; case INFO_STANZA_STATUS_CODE_OTHER: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_OTHER_STR)); break; } // Construct a specific lock part KeyValue *const lockKv = kvPutKv(statusKv, STATUS_KEY_LOCK_VAR); stanzaStatusBackupLockAdd(lockKv, stanzaData->repoList, repoIdxMin, repoIdxMax); stanzaStatusLockAdd(lockKv, STATUS_KEY_LOCK_RESTORE_VAR, &stanzaData->restoreLock); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set the error status code and message for the stanza on the repo to the code and message passed ***********************************************************************************************************************************/ static void repoStanzaStatus(const int code, const Variant *const repoStanzaInfo, const InfoRepoData *const repoData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_PARAM(VARIANT, repoStanzaInfo); FUNCTION_TEST_PARAM(INFO_REPO_DATA, repoData); FUNCTION_TEST_END(); ASSERT((code >= 0 && code <= 3) || code == 99 || code == 6); ASSERT(repoStanzaInfo != NULL); KeyValue *const statusKv = kvPutKv(varKv(repoStanzaInfo), STANZA_KEY_STATUS_VAR); kvPut(statusKv, STATUS_KEY_CODE_VAR, VARINT(code)); switch (code) { case INFO_STANZA_STATUS_CODE_OK: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_OK_STR)); break; case INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_PATH_STR)); break; case INFO_STANZA_STATUS_CODE_MISSING_STANZA_DATA: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_MISSING_STANZA_DATA_STR)); break; case INFO_STANZA_STATUS_CODE_NO_BACKUP: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_MESSAGE_NO_BACKUP_STR)); break; case INFO_STANZA_STATUS_CODE_BACKUP_MISSING: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(INFO_STANZA_STATUS_CODE_BACKUP_MISSING_STR)); break; case INFO_STANZA_STATUS_CODE_OTHER: kvPut(statusKv, STATUS_KEY_MESSAGE_VAR, VARSTR(repoData->error)); break; } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set the data for the archive section of the stanza for the database info from the backup.info file ***********************************************************************************************************************************/ static void archiveDbList( const String *const stanza, const InfoPgData *const pgData, VariantList *const archiveSection, const InfoArchive *const info, const bool currentDb, const unsigned int repoIdx, const unsigned int repoKey) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, stanza); FUNCTION_TEST_PARAM_P(INFO_PG_DATA, pgData); FUNCTION_TEST_PARAM(VARIANT_LIST, archiveSection); FUNCTION_TEST_PARAM(BOOL, currentDb); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(UINT, repoKey); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(stanza != NULL); ASSERT(pgData != NULL); ASSERT(archiveSection != NULL); // With multiple DB versions, the backup.info history-id may not be the same as archive.info history-id, so the archive path // must be built by retrieving the archive id given the db version and system id of the backup.info file. If there is no match, // an error will be thrown. const String *const archiveId = infoArchiveIdHistoryMatch(info, pgData->id, pgData->version, pgData->systemId); String *const archivePath = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/%s", strZ(stanza), strZ(archiveId)); const String *archiveStart = NULL; const String *archiveStop = NULL; Variant *const archiveInfo = varNewKv(kvNew()); const Storage *const storageRepo = storageRepoIdx(repoIdx); // Get a list of WAL directories in the archive repo from oldest to newest, if any exist const StringList *const walDir = strLstSort( storageListP(storageRepo, archivePath, .expression = WAL_SEGMENT_DIR_REGEXP_STR), sortOrderAsc); if (!strLstEmpty(walDir)) { // Not every WAL dir has WAL files so check each for (unsigned int idx = 0; idx < strLstSize(walDir); idx++) { // Get a list of all WAL in this WAL dir and sort the list from oldest to newest to get the oldest starting WAL archived // for this db const StringList *const list = strLstSort( storageListP( storageRepo, strNewFmt("%s/%s", strZ(archivePath), strZ(strLstGet(walDir, idx))), .expression = WAL_SEGMENT_FILE_REGEXP_STR), sortOrderAsc); // If wal segments are found, get the oldest one as the archive start if (!strLstEmpty(list)) { archiveStart = strSubN(strLstGet(list, 0), 0, 24); break; } } // Iterate through the directory list in reverse processing newest first. Cast comparison to an int for readability. for (unsigned int idx = strLstSize(walDir) - 1; (int)idx >= 0; idx--) { // Get a list of all WAL in this WAL dir and sort the list from newest to oldest to get the newest ending WAL archived // for this db const StringList *const list = strLstSort( storageListP( storageRepo, strNewFmt("%s/%s", strZ(archivePath), strZ(strLstGet(walDir, idx))), .expression = WAL_SEGMENT_FILE_REGEXP_STR), sortOrderDesc); // If wal segments are found, get the newest one as the archive stop if (!strLstEmpty(list)) { archiveStop = strSubN(strLstGet(list, 0), 0, 24); break; } } } // If there is an archive or the database is the current database then store it if (currentDb || archiveStart != NULL) { // Add empty database section to archiveInfo and then fill in database id from the backup.info KeyValue *const databaseInfo = kvPutKv(varKv(archiveInfo), KEY_DATABASE_VAR); kvPut(databaseInfo, DB_KEY_ID_VAR, VARUINT(pgData->id)); kvPut(databaseInfo, KEY_REPO_KEY_VAR, VARUINT(repoKey)); kvPut(varKv(archiveInfo), DB_KEY_ID_VAR, VARSTR(archiveId)); kvPut(varKv(archiveInfo), ARCHIVE_KEY_MIN_VAR, (archiveStart != NULL ? VARSTR(archiveStart) : (Variant *)NULL)); kvPut(varKv(archiveInfo), ARCHIVE_KEY_MAX_VAR, (archiveStop != NULL ? VARSTR(archiveStop) : (Variant *)NULL)); varLstAdd(archiveSection, archiveInfo); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Add the backup data to the backup section ***********************************************************************************************************************************/ static void backupListAdd( VariantList *backupSection, InfoBackupData *backupData, const String *backupLabel, InfoRepoData *repoData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, backupSection); // The section to add the backup data to FUNCTION_TEST_PARAM_P(INFO_BACKUP_DATA, backupData); // The data for the backup FUNCTION_TEST_PARAM(STRING, backupLabel); // Backup label to filter if requested by the user FUNCTION_TEST_PARAM(INFO_REPO_DATA, repoData); // The repo data where this backup is located FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(backupSection != NULL); ASSERT(backupData != NULL); ASSERT(repoData != NULL); Variant *const backupInfo = varNewKv(kvNew()); // Flags used to decide what data to add const bool outputJson = cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_JSON; // main keys kvPut(varKv(backupInfo), BACKUP_KEY_LABEL_VAR, VARSTR(backupData->backupLabel)); kvPut(varKv(backupInfo), BACKUP_KEY_TYPE_VAR, VARSTR(strNewStrId(backupData->backupType))); kvPut( varKv(backupInfo), BACKUP_KEY_PRIOR_VAR, (backupData->backupPrior != NULL ? VARSTR(backupData->backupPrior) : NULL)); kvPut( varKv(backupInfo), BACKUP_KEY_REFERENCE_VAR, (backupData->backupReference != NULL ? varNewVarLst(varLstNewStrLst(backupData->backupReference)) : NULL)); // Display complete reference list only for json output or --set text. Otherwise keep track of full, diff and incremental to // display a summary. if (backupData->backupReference != NULL && backupLabel == NULL && !outputJson) { StringList *const fullList = strLstNew(); StringList *const diffList = strLstNew(); StringList *const incrList = strLstNew(); for (unsigned int backupIdx = 0; backupIdx < strLstSize(backupData->backupReference); backupIdx++) { const String *const backupReferenceLabel = strLstGet(backupData->backupReference, backupIdx); if (regExpMatchOne(backupRegExpP(.full = true), backupReferenceLabel)) strLstAdd(fullList, backupReferenceLabel); if (regExpMatchOne(backupRegExpP(.differential = true), backupReferenceLabel)) strLstAdd(diffList, backupReferenceLabel); if (regExpMatchOne(backupRegExpP(.incremental = true), backupReferenceLabel)) strLstAdd(incrList, backupReferenceLabel); } // The reference list will always contain at least 1 full kvPut(varKv(backupInfo), BACKUP_KEY_REFERENCE_FULL_VAR, varNewVarLst(varLstNewStrLst(fullList))); kvPut( varKv(backupInfo), BACKUP_KEY_REFERENCE_DIFF_VAR, strLstSize(diffList) > 0 ? varNewVarLst(varLstNewStrLst(diffList)) : NULL); kvPut( varKv(backupInfo), BACKUP_KEY_REFERENCE_INCR_VAR, strLstSize(incrList) > 0 ? varNewVarLst(varLstNewStrLst(incrList)) : NULL); } // archive section KeyValue *const archiveInfo = kvPutKv(varKv(backupInfo), KEY_ARCHIVE_VAR); kvPut( archiveInfo, KEY_START_VAR, (backupData->backupArchiveStart != NULL ? VARSTR(backupData->backupArchiveStart) : NULL)); kvPut( archiveInfo, KEY_STOP_VAR, (backupData->backupArchiveStop != NULL ? VARSTR(backupData->backupArchiveStop) : NULL)); // backrest section KeyValue *const backrestInfo = kvPutKv(varKv(backupInfo), BACKUP_KEY_BACKREST_VAR); kvPut(backrestInfo, BACKREST_KEY_FORMAT_VAR, VARUINT(backupData->backrestFormat)); kvPut(backrestInfo, BACKREST_KEY_VERSION_VAR, VARSTR(backupData->backrestVersion)); // database section KeyValue *const dbInfo = kvPutKv(varKv(backupInfo), KEY_DATABASE_VAR); kvPut(dbInfo, DB_KEY_ID_VAR, VARUINT(backupData->backupPgId)); kvPut(dbInfo, KEY_REPO_KEY_VAR, VARUINT(repoData->key)); // info section KeyValue *const infoInfo = kvPutKv(varKv(backupInfo), BACKUP_KEY_INFO_VAR); kvPut(infoInfo, KEY_SIZE_VAR, VARUINT64(backupData->backupInfoSize)); kvPut(infoInfo, KEY_DELTA_VAR, VARUINT64(backupData->backupInfoSizeDelta)); // info:repository section KeyValue *const repoInfo = kvPutKv(infoInfo, INFO_KEY_REPOSITORY_VAR); if (backupData->backupInfoRepoSizeMap == NULL) kvPut(repoInfo, KEY_SIZE_VAR, VARUINT64(backupData->backupInfoRepoSize)); kvPut(repoInfo, KEY_DELTA_VAR, VARUINT64(backupData->backupInfoRepoSizeDelta)); if (outputJson && backupData->backupInfoRepoSizeMap != NULL) { kvPut(repoInfo, KEY_SIZE_MAP_VAR, backupData->backupInfoRepoSizeMap); kvPut(repoInfo, KEY_DELTA_MAP_VAR, backupData->backupInfoRepoSizeMapDelta); } // timestamp section KeyValue *const timeInfo = kvPutKv(varKv(backupInfo), BACKUP_KEY_TIMESTAMP_VAR); // time_t is generally a signed int so cast it to uint64 since it can never be negative (before 1970) in our system kvPut(timeInfo, KEY_START_VAR, VARUINT64((uint64_t)backupData->backupTimestampStart)); kvPut(timeInfo, KEY_STOP_VAR, VARUINT64((uint64_t)backupData->backupTimestampStop)); // Report errors only if the error status is known if (backupData->backupError != NULL) kvPut(varKv(backupInfo), BACKUP_KEY_ERROR_VAR, backupData->backupError); // Add start/stop backup lsn info to json output or --set text if ((outputJson || backupLabel != NULL) && backupData->backupLsnStart != NULL && backupData->backupLsnStop != NULL) { KeyValue *const lsnInfo = kvPutKv(varKv(backupInfo), BACKUP_KEY_LSN_VAR); kvPut(lsnInfo, KEY_START_VAR, VARSTR(backupData->backupLsnStart)); kvPut(lsnInfo, KEY_STOP_VAR, VARSTR(backupData->backupLsnStop)); } // Add annotations to json output or --set text if ((outputJson || backupLabel != NULL) && backupData->backupAnnotation != NULL) kvPut(varKv(backupInfo), BACKUP_KEY_ANNOTATION_VAR, backupData->backupAnnotation); // If a backup label was specified and this is that label, then get the data from the loaded manifest if (backupLabel != NULL) { // Get the list of databases in this backup VariantList *const databaseSection = varLstNew(); for (unsigned int dbIdx = 0; dbIdx < manifestDbTotal(repoData->manifest); dbIdx++) { const ManifestDb *const db = manifestDb(repoData->manifest, dbIdx); // Do not display template databases if (!pgDbIsTemplate(db->name)) { Variant *const database = varNewKv(kvNew()); kvPut(varKv(database), KEY_NAME_VAR, VARSTR(db->name)); kvPut(varKv(database), KEY_OID_VAR, VARUINT64(db->id)); varLstAdd(databaseSection, database); } } // Add the database section even if none found kvPut(varKv(backupInfo), BACKUP_KEY_DATABASE_REF_VAR, varNewVarLst(databaseSection)); // Get symlinks and tablespaces VariantList *const linkSection = varLstNew(); VariantList *const tablespaceSection = varLstNew(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(repoData->manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(repoData->manifest, targetIdx); Variant *const link = varNewKv(kvNew()); Variant *const tablespace = varNewKv(kvNew()); if (target->type == manifestTargetTypeLink) { if (target->tablespaceName != NULL) { kvPut(varKv(tablespace), KEY_NAME_VAR, VARSTR(target->tablespaceName)); kvPut(varKv(tablespace), KEY_DESTINATION_VAR, VARSTR(target->path)); kvPut(varKv(tablespace), KEY_OID_VAR, VARUINT64(target->tablespaceId)); varLstAdd(tablespaceSection, tablespace); } else if (target->file != NULL) { kvPut(varKv(link), KEY_NAME_VAR, varNewStr(target->file)); kvPut( varKv(link), KEY_DESTINATION_VAR, varNewStr(strNewFmt("%s/%s", strZ(target->path), strZ(target->file)))); varLstAdd(linkSection, link); } else { kvPut(varKv(link), KEY_NAME_VAR, VARSTR(manifestPathPg(target->name))); kvPut(varKv(link), KEY_DESTINATION_VAR, VARSTR(target->path)); varLstAdd(linkSection, link); } } } kvPut(varKv(backupInfo), BACKUP_KEY_LINK_VAR, (!varLstEmpty(linkSection) ? varNewVarLst(linkSection) : NULL)); kvPut( varKv(backupInfo), BACKUP_KEY_TABLESPACE_VAR, (!varLstEmpty(tablespaceSection) ? varNewVarLst(tablespaceSection) : NULL)); // Get the list of files with an error VariantList *const checksumPageErrorList = varLstNew(); for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(repoData->manifest); fileIdx++) { const ManifestFile file = manifestFile(repoData->manifest, fileIdx); if (file.checksumPageError) varLstAdd(checksumPageErrorList, varNewStr(manifestPathPg(file.name))); } if (!varLstEmpty(checksumPageErrorList)) { kvPut(varKv(backupInfo), BACKUP_KEY_ERROR_LIST_VAR, varNewVarLst(checksumPageErrorList)); // It is possible that backup-error is not set in backup.info but there are errors in manifest because backup-error was // added in a later version than manifest errors. However, it should not be possible for backup-error to be present but // false if there are errors in the manifest. In production this condition will be ignored and error set to true. ASSERT( kvGet(varKv(backupInfo), BACKUP_KEY_ERROR_VAR) == NULL || varBool(kvGet(varKv(backupInfo), BACKUP_KEY_ERROR_VAR))); kvPut(varKv(backupInfo), BACKUP_KEY_ERROR_VAR, BOOL_TRUE_VAR); } manifestFree(repoData->manifest); repoData->manifest = NULL; } varLstAdd(backupSection, backupInfo); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** For each current backup in the backup.info file of the stanza, set the data for the backup section ***********************************************************************************************************************************/ static void backupList( VariantList *const backupSection, InfoStanzaRepo *const stanzaData, const String *const backupLabel, const unsigned int repoIdxMin, const unsigned int repoIdxMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, backupSection); // The section to add the backup data to FUNCTION_TEST_PARAM(INFO_STANZA_REPO, stanzaData); // The data for the stanza FUNCTION_TEST_PARAM(STRING, backupLabel); // Backup label to filter if requested by the user FUNCTION_TEST_PARAM(UINT, repoIdxMin); // The start index of the repo array to begin checking FUNCTION_TEST_PARAM(UINT, repoIdxMax); // The index of the last repo to check FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(backupSection != NULL); ASSERT(stanzaData != NULL); unsigned int backupNextRepoIdx = 0; unsigned int backupTotal = 0; unsigned int backupTotalProcessed = 0; // Get the number of backups to be processed for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { const InfoRepoData *const repoData = &stanzaData->repoList[repoIdx]; if (repoData->backupInfo != NULL && infoBackupDataTotal(repoData->backupInfo) > 0) backupTotal += infoBackupDataTotal(repoData->backupInfo); } // Process any backups while (backupTotalProcessed < backupTotal) { time_t backupNextTime = 0; // Backups are sorted for each repo, so iterate over the lists to create a single list ordered by backup-timestamp-stop for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { const InfoRepoData *const repoData = &stanzaData->repoList[repoIdx]; // If there are current backups on this repo for this stanza and the end of this backup list has not been reached // determine the next backup for display if (repoData->backupInfo != NULL && infoBackupDataTotal(repoData->backupInfo) > 0 && repoData->backupIdx < infoBackupDataTotal(repoData->backupInfo)) { InfoBackupData backupData = infoBackupData(repoData->backupInfo, repoData->backupIdx); // See if this backup should be next in the list, ordering from oldest to newest if (backupNextTime == 0 || backupData.backupTimestampStop < backupNextTime) { backupNextTime = backupData.backupTimestampStop; backupNextRepoIdx = repoIdx; } } } InfoRepoData *const repoData = &stanzaData->repoList[backupNextRepoIdx]; InfoBackupData backupData = infoBackupData(repoData->backupInfo, repoData->backupIdx); repoData->backupIdx++; backupTotalProcessed++; // Don't add the backup data to the backup section if a backup label was specified but this is not it if (backupLabel != NULL && !strEq(backupData.backupLabel, backupLabel)) continue; // Add the backup data to the backup section if (!cfgOptionTest(cfgOptType) || cfgOptionStrId(cfgOptType) == backupData.backupType) backupListAdd(backupSection, &backupData, backupLabel, repoData); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set the stanza data for each stanza found in the repo ***********************************************************************************************************************************/ static VariantList * stanzaInfoList( List *const stanzaRepoList, const String *const backupLabel, const unsigned int repoIdxMin, const unsigned int repoIdxMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, stanzaRepoList); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(UINT, repoIdxMin); FUNCTION_TEST_PARAM(UINT, repoIdxMax); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(stanzaRepoList != NULL); // Is full output requested? const bool outputFull = cfgOptionSeq(cfgOptDetailLevel) == CFGOPTVAL_DETAIL_LEVEL_FULL; VariantList *const result = varLstNew(); // Sort the list of stanzas lstSort(stanzaRepoList, sortOrderAsc); // Process each stanza requested for (unsigned int idx = 0; idx < lstSize(stanzaRepoList); idx++) { InfoStanzaRepo *const stanzaData = lstGet(stanzaRepoList, idx); // Create the stanzaInfo and section variables Variant *const stanzaInfo = varNewKv(kvNew()); VariantList *const dbSection = varLstNew(); VariantList *const backupSection = varLstNew(); VariantList *const archiveSection = varLstNew(); VariantList *const repoSection = varLstNew(); int stanzaStatusCode = -1; uint64_t stanzaCipherType = cipherTypeNone; // Set the stanza name and initialize the overall stanza variables kvPut(varKv(stanzaInfo), KEY_NAME_VAR, VARSTR(stanzaData->name)); // Get the stanza for each requested repo for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { InfoRepoData *const repoData = &stanzaData->repoList[repoIdx]; // When full output is not requested (progress mode), skip collecting detailed information and only update status code if (!outputFull) { if (repoIdx == repoIdxMin) stanzaStatusCode = repoData->stanzaStatus; else { stanzaStatusCode = stanzaStatusCode != repoData->stanzaStatus ? INFO_STANZA_STATUS_CODE_MIXED : repoData->stanzaStatus; } continue; } Variant *const repoInfo = varNewKv(kvNew()); kvPut(varKv(repoInfo), REPO_KEY_KEY_VAR, VARUINT(repoData->key)); kvPut(varKv(repoInfo), KEY_CIPHER_VAR, VARSTR(strNewStrId(repoData->cipher))); // If the stanza on this repo has the default status of ok but the backupInfo was not read, then the stanza exists on // other repos but not this one if (repoData->stanzaStatus == INFO_STANZA_STATUS_CODE_OK && repoData->backupInfo == NULL) repoData->stanzaStatus = INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH; TRY_BEGIN() { // If the backup.info file has been read, then get the backup and archive information on this repo if (repoData->backupInfo != NULL) { // If the backup.info file exists, get the database history information (oldest to newest) and corresponding // archive for (unsigned int pgIdx = infoPgDataTotal(infoBackupPg(repoData->backupInfo)) - 1; (int)pgIdx >= 0; pgIdx--) { const InfoPgData pgData = infoPgData(infoBackupPg(repoData->backupInfo), pgIdx); Variant *const pgInfo = varNewKv(kvNew()); kvPut(varKv(pgInfo), DB_KEY_ID_VAR, VARUINT(pgData.id)); kvPut(varKv(pgInfo), DB_KEY_SYSTEM_ID_VAR, VARUINT64(pgData.systemId)); kvPut(varKv(pgInfo), DB_KEY_VERSION_VAR, VARSTR(pgVersionToStr(pgData.version))); kvPut(varKv(pgInfo), KEY_REPO_KEY_VAR, VARUINT(repoData->key)); varLstAdd(dbSection, pgInfo); // Get the archive info for the DB from the archive.info file archiveDbList( stanzaData->name, &pgData, archiveSection, repoData->archiveInfo, (pgIdx == 0 ? true : false), repoIdx, repoData->key); } // Set stanza status if the current db sections do not match across repos const InfoPgData backupInfoCurrentPg = infoPgData( infoBackupPg(repoData->backupInfo), infoPgDataCurrentId(infoBackupPg(repoData->backupInfo))); // The current PG system and version must match across repos for the stanza, if not, a failure may have occurred // during an upgrade or the repo may have been disabled during the stanza upgrade to protect from error // propagation if (stanzaData->currentPgVersion != backupInfoCurrentPg.version || stanzaData->currentPgSystemId != backupInfoCurrentPg.systemId) { stanzaStatusCode = INFO_STANZA_STATUS_CODE_PG_MISMATCH; } } } CATCH_ANY() { infoStanzaErrorAdd(repoData, errorType(), STR(errorMessage())); } TRY_END(); // If there are no current backups on this repo then set status to no backup if (repoData->stanzaStatus == INFO_STANZA_STATUS_CODE_OK && infoBackupDataTotal(repoData->backupInfo) == 0) repoData->stanzaStatus = INFO_STANZA_STATUS_CODE_NO_BACKUP; // Track the status over all repos if the status for the stanza has not already been determined if (stanzaStatusCode != INFO_STANZA_STATUS_CODE_PG_MISMATCH) { if (repoIdx == repoIdxMin) stanzaStatusCode = repoData->stanzaStatus; else { stanzaStatusCode = stanzaStatusCode != repoData->stanzaStatus ? INFO_STANZA_STATUS_CODE_MIXED : repoData->stanzaStatus; } } // Track cipher type over all repos if (repoIdx == repoIdxMin) stanzaCipherType = repoData->cipher; else stanzaCipherType = stanzaCipherType != repoData->cipher ? INFO_STANZA_STATUS_CODE_MIXED : repoData->cipher; // Add the status of the stanza on the repo to the repo section, and the repo to the repo array repoStanzaStatus(repoData->stanzaStatus, repoInfo, repoData); varLstAdd(repoSection, repoInfo); // Add the database history, backup, archive and repo arrays to the stanza info kvPut(varKv(stanzaInfo), STANZA_KEY_DB_VAR, varNewVarLst(dbSection)); kvPut(varKv(stanzaInfo), KEY_ARCHIVE_VAR, varNewVarLst(archiveSection)); kvPut(varKv(stanzaInfo), STANZA_KEY_REPO_VAR, varNewVarLst(repoSection)); } // Collect backup and cipher data if full output is requested if (outputFull) { // Get a sorted list of the data for all existing backups for this stanza over all repos backupList(backupSection, stanzaData, backupLabel, repoIdxMin, repoIdxMax); kvPut(varKv(stanzaInfo), STANZA_KEY_BACKUP_VAR, varNewVarLst(backupSection)); // Set the overall cipher type if (stanzaCipherType != INFO_STANZA_STATUS_CODE_MIXED) kvPut(varKv(stanzaInfo), KEY_CIPHER_VAR, VARSTR(strNewStrId(stanzaCipherType))); else kvPut(varKv(stanzaInfo), KEY_CIPHER_VAR, VARSTRDEF(INFO_STANZA_MIXED)); } // Set the overall stanza status and gather progress information stanzaStatus(stanzaStatusCode, stanzaData, stanzaInfo, repoIdxMin, repoIdxMax); varLstAdd(result, stanzaInfo); } FUNCTION_TEST_RETURN(VARIANT_LIST, result); } /*********************************************************************************************************************************** Format the text output for archive and backups for a database group of a stanza ***********************************************************************************************************************************/ // Helper to format date/time with timezone offset static String * formatTextBackupDateTime(const time_t epoch) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TIME, epoch); FUNCTION_TEST_END(); // Construct date/time String *const result = strNew(); struct tm timePart; char timeBuffer[20]; strftime(timeBuffer, sizeof(timeBuffer), "%Y-%m-%d %H:%M:%S", localtime_r(&epoch, &timePart)); strCatZ(result, timeBuffer); // Add timezone offset. Since this is not directly available in Posix-compliant APIs, use the difference between gmtime_r() and // localtime_r to determine the offset. Handle minute offsets when present. struct tm timePartGm; gmtime_r(&epoch, &timePartGm); timePart.tm_isdst = -1; timePartGm.tm_isdst = -1; time_t offset = mktime(&timePart) - mktime(&timePartGm); if (offset >= 0) strCatChr(result, '+'); else { offset *= -1; strCatChr(result, '-'); } const unsigned int minute = (unsigned int)(offset / 60); const unsigned int hour = minute / 60; strCatFmt(result, "%02u", hour); if (minute % 60 != 0) strCatFmt(result, ":%02u", minute - (hour * 60)); FUNCTION_TEST_RETURN(STRING, result); } static void formatTextBackup(const DbGroup *const dbGroup, String *const resultStr) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(DB_GROUP, dbGroup); FUNCTION_TEST_PARAM(STRING, resultStr); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(dbGroup != NULL); strCatFmt(resultStr, "\n wal archive min/max (%s): ", strZ(dbGroup->version)); // Get the archive min/max if there are any archives for the database if (dbGroup->archiveMin != NULL) strCatFmt(resultStr, "%s/%s\n", strZ(dbGroup->archiveMin), strZ(dbGroup->archiveMax)); else strCatZ(resultStr, "none present\n"); for (unsigned int backupIdx = 0; backupIdx < varLstSize(dbGroup->backupList); backupIdx++) { const KeyValue *const backupInfo = varKv(varLstGet(dbGroup->backupList, backupIdx)); strCatFmt( resultStr, "\n %s backup: %s\n", strZ(varStr(kvGet(backupInfo, BACKUP_KEY_TYPE_VAR))), strZ(varStr(kvGet(backupInfo, BACKUP_KEY_LABEL_VAR)))); // Get and format backup start/stop time const KeyValue *const timestampInfo = varKv(kvGet(backupInfo, BACKUP_KEY_TIMESTAMP_VAR)); strCatFmt( resultStr, " timestamp start/stop: %s / %s\n", strZ(formatTextBackupDateTime((time_t)varUInt64(kvGet(timestampInfo, KEY_START_VAR)))), strZ(formatTextBackupDateTime((time_t)varUInt64(kvGet(timestampInfo, KEY_STOP_VAR))))); strCatZ(resultStr, " wal start/stop: "); // Get and format WAL start/stop const KeyValue *const archiveBackupInfo = varKv(kvGet(backupInfo, KEY_ARCHIVE_VAR)); if (kvGet(archiveBackupInfo, KEY_START_VAR) != NULL && kvGet(archiveBackupInfo, KEY_STOP_VAR) != NULL) { strCatFmt( resultStr, "%s / %s\n", strZ(varStr(kvGet(archiveBackupInfo, KEY_START_VAR))), strZ(varStr(kvGet(archiveBackupInfo, KEY_STOP_VAR)))); } else strCatZ(resultStr, "n/a\n"); const KeyValue *const lsnInfo = varKv(kvGet(backupInfo, BACKUP_KEY_LSN_VAR)); if (lsnInfo != NULL) { strCatFmt( resultStr, " lsn start/stop: %s / %s\n", strZ(varStr(kvGet(lsnInfo, KEY_START_VAR))), strZ(varStr(kvGet(lsnInfo, KEY_STOP_VAR)))); } const KeyValue *const info = varKv(kvGet(backupInfo, BACKUP_KEY_INFO_VAR)); strCatFmt( resultStr, " database size: %s, database backup size: %s\n", strZ(strSizeFormat(varUInt64Force(kvGet(info, KEY_SIZE_VAR)))), strZ(strSizeFormat(varUInt64Force(kvGet(info, KEY_DELTA_VAR))))); const KeyValue *const repoInfo = varKv(kvGet(info, INFO_KEY_REPOSITORY_VAR)); const Variant *const repoSizeInfo = kvGet(repoInfo, KEY_SIZE_VAR); strCatFmt(resultStr, " repo%u: ", varUInt(kvGet(varKv(kvGet(backupInfo, KEY_DATABASE_VAR)), KEY_REPO_KEY_VAR))); if (repoSizeInfo != NULL) strCatFmt(resultStr, "backup set size: %s, ", strZ(strSizeFormat(varUInt64Force(kvGet(repoInfo, KEY_SIZE_VAR))))); strCatFmt(resultStr, "backup size: %s\n", strZ(strSizeFormat(varUInt64Force(kvGet(repoInfo, KEY_DELTA_VAR))))); if (kvGet(backupInfo, BACKUP_KEY_REFERENCE_VAR) != NULL) { if (kvGet(backupInfo, BACKUP_KEY_REFERENCE_FULL_VAR) != NULL) { const String *const diffListSizeText = kvGet(backupInfo, BACKUP_KEY_REFERENCE_DIFF_VAR) != NULL ? strNewFmt(", %u diff", varLstSize(varVarLst(kvGet(backupInfo, BACKUP_KEY_REFERENCE_DIFF_VAR)))) : EMPTY_STR; const String *const incrListSizeText = kvGet(backupInfo, BACKUP_KEY_REFERENCE_INCR_VAR) != NULL ? strNewFmt(", %u incr", varLstSize(varVarLst(kvGet(backupInfo, BACKUP_KEY_REFERENCE_INCR_VAR)))) : EMPTY_STR; strCatFmt( resultStr, " backup reference total: %u full%s%s\n", varLstSize(varVarLst(kvGet(backupInfo, BACKUP_KEY_REFERENCE_FULL_VAR))), strZ(diffListSizeText), strZ(incrListSizeText)); } else { const StringList *const referenceList = strLstNewVarLst(varVarLst(kvGet(backupInfo, BACKUP_KEY_REFERENCE_VAR))); strCatFmt(resultStr, " backup reference list: %s\n", strZ(strLstJoin(referenceList, ", "))); } } if (kvGet(backupInfo, BACKUP_KEY_DATABASE_REF_VAR) != NULL) { const VariantList *const dbSection = kvGetList(backupInfo, BACKUP_KEY_DATABASE_REF_VAR); strCatZ(resultStr, " database list:"); if (varLstEmpty(dbSection)) strCatZ(resultStr, " none\n"); else { for (unsigned int dbIdx = 0; dbIdx < varLstSize(dbSection); dbIdx++) { const KeyValue *const db = varKv(varLstGet(dbSection, dbIdx)); strCatFmt( resultStr, " %s (%s)", strZ(varStr(kvGet(db, KEY_NAME_VAR))), strZ(varStrForce(kvGet(db, KEY_OID_VAR)))); if (dbIdx != varLstSize(dbSection) - 1) strCatZ(resultStr, ","); } strCat(resultStr, LF_STR); } } if (kvGet(backupInfo, BACKUP_KEY_LINK_VAR) != NULL) { const VariantList *const linkSection = kvGetList(backupInfo, BACKUP_KEY_LINK_VAR); strCatZ(resultStr, " symlinks:\n"); for (unsigned int linkIdx = 0; linkIdx < varLstSize(linkSection); linkIdx++) { const KeyValue *const link = varKv(varLstGet(linkSection, linkIdx)); strCatFmt( resultStr, " %s => %s", strZ(varStr(kvGet(link, KEY_NAME_VAR))), strZ(varStr(kvGet(link, KEY_DESTINATION_VAR)))); if (linkIdx != varLstSize(linkSection) - 1) strCat(resultStr, LF_STR); } strCat(resultStr, LF_STR); } if (kvGet(backupInfo, BACKUP_KEY_TABLESPACE_VAR) != NULL) { const VariantList *const tablespaceSection = kvGetList(backupInfo, BACKUP_KEY_TABLESPACE_VAR); strCatZ(resultStr, " tablespaces:\n"); for (unsigned int tblIdx = 0; tblIdx < varLstSize(tablespaceSection); tblIdx++) { const KeyValue *const tablespace = varKv(varLstGet(tablespaceSection, tblIdx)); strCatFmt( resultStr, " %s (%s) => %s", strZ(varStr(kvGet(tablespace, KEY_NAME_VAR))), strZ(varStrForce(kvGet(tablespace, KEY_OID_VAR))), strZ(varStr(kvGet(tablespace, KEY_DESTINATION_VAR)))); if (tblIdx != varLstSize(tablespaceSection) - 1) strCat(resultStr, LF_STR); } strCat(resultStr, LF_STR); } // If errors were detected during the backup if (kvGet(backupInfo, BACKUP_KEY_ERROR_VAR) != NULL && varBool(kvGet(backupInfo, BACKUP_KEY_ERROR_VAR))) { // Output error list if present if (kvGet(backupInfo, BACKUP_KEY_ERROR_LIST_VAR) != NULL) { const StringList *const checksumPageErrorList = strLstNewVarLst( varVarLst(kvGet(backupInfo, BACKUP_KEY_ERROR_LIST_VAR))); strCatFmt(resultStr, " error list: %s\n", strZ(strLstJoin(checksumPageErrorList, ", "))); } // Else output a general message else strCatZ(resultStr, " error(s) detected during backup\n"); } // Annotations metadata if (kvGet(backupInfo, BACKUP_KEY_ANNOTATION_VAR) != NULL) { const KeyValue *const annotationKv = varKv(kvGet(backupInfo, BACKUP_KEY_ANNOTATION_VAR)); const StringList *const annotationKeyList = strLstNewVarLst(kvKeyList(annotationKv)); String *const annotationStr = strNew(); for (unsigned int keyIdx = 0; keyIdx < strLstSize(annotationKeyList); keyIdx++) { const String *const key = strLstGet(annotationKeyList, keyIdx); const String *const value = varStr(kvGet(annotationKv, VARSTR(key))); ASSERT(value != NULL); strCatFmt(annotationStr, " %s: %s\n", strZ(key), strZ(value)); } strCatFmt(resultStr, " annotation(s)\n%s", strZ(annotationStr)); } } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Format the text output for each database of the stanza ***********************************************************************************************************************************/ static void formatTextDb( const KeyValue *const stanzaInfo, String *const resultStr, const String *const currentPgVersion, const uint64_t currentPgSystemId, const String *const backupLabel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, stanzaInfo); FUNCTION_TEST_PARAM(STRING, resultStr); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(STRING, currentPgVersion); FUNCTION_TEST_PARAM(UINT64, currentPgSystemId); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(stanzaInfo != NULL); ASSERT(currentPgVersion != NULL); const VariantList *const dbSection = kvGetList(stanzaInfo, STANZA_KEY_DB_VAR); const VariantList *const archiveSection = kvGetList(stanzaInfo, KEY_ARCHIVE_VAR); const VariantList *const backupSection = kvGetList(stanzaInfo, STANZA_KEY_BACKUP_VAR); List *const dbGroupList = lstNewP(sizeof(DbGroup)); // For each database update the corresponding archive info for (unsigned int dbIdx = 0; dbIdx < varLstSize(dbSection); dbIdx++) { const KeyValue *const pgInfo = varKv(varLstGet(dbSection, dbIdx)); const uint64_t dbSysId = varUInt64(kvGet(pgInfo, DB_KEY_SYSTEM_ID_VAR)); const String *const dbVersion = varStr(kvGet(pgInfo, DB_KEY_VERSION_VAR)); const unsigned int dbId = varUInt(kvGet(pgInfo, DB_KEY_ID_VAR)); const unsigned int dbRepoKey = varUInt(kvGet(pgInfo, KEY_REPO_KEY_VAR)); DbGroup *dbGroup = NULL; for (unsigned int dbGrpIdx = 0; dbGrpIdx < lstSize(dbGroupList); dbGrpIdx++) { DbGroup *const dbGroupInfo = lstGet(dbGroupList, dbGrpIdx); if (dbGroupInfo->systemId == dbSysId && strEq(dbGroupInfo->version, dbVersion)) { dbGroup = dbGroupInfo; break; } } // If the group was not found, then add it if (dbGroup == NULL) { const DbGroup dbGroupInfo = { .systemId = dbSysId, .version = dbVersion, .current = (currentPgSystemId == dbSysId && strEq(currentPgVersion, dbVersion)), .archiveMin = NULL, .archiveMax = NULL, .backupList = varLstNew(), }; lstAdd(dbGroupList, &dbGroupInfo); dbGroup = lstGetLast(dbGroupList); } // For each archive of this stanza, update the archive min/max for this database group if necessary for (unsigned int archiveIdx = 0; archiveIdx < varLstSize(archiveSection); archiveIdx++) { const KeyValue *const archiveInfo = varKv(varLstGet(archiveSection, archiveIdx)); const KeyValue *const archiveDbInfo = varKv(kvGet(archiveInfo, KEY_DATABASE_VAR)); const unsigned int archiveDbId = varUInt(kvGet(archiveDbInfo, DB_KEY_ID_VAR)); const unsigned int archiveRepoKey = varUInt(kvGet(archiveDbInfo, KEY_REPO_KEY_VAR)); // If there are archives and the min is less than that for this database group, then update the group if (archiveDbId == dbId && archiveRepoKey == dbRepoKey && varStr(kvGet(archiveInfo, ARCHIVE_KEY_MIN_VAR)) != NULL) { // Although archives should continue to increment over system-id/version with different db-ids, there may be cases // where an archived WAL may exist on both, and if the archive id on a later db is less than a prior instance of // the same PG, then ensure it is updated as the min. Any need to error should not be handled in the info command. if (dbGroup->archiveMin == NULL || strCmp(dbGroup->archiveMin, varStr(kvGet(archiveInfo, ARCHIVE_KEY_MIN_VAR))) > 0) dbGroup->archiveMin = varStrForce(kvGet(archiveInfo, ARCHIVE_KEY_MIN_VAR)); if (dbGroup->archiveMax == NULL || strCmp(dbGroup->archiveMax, varStr(kvGet(archiveInfo, ARCHIVE_KEY_MAX_VAR))) < 0) dbGroup->archiveMax = varStrForce(kvGet(archiveInfo, ARCHIVE_KEY_MAX_VAR)); } } } unsigned int backupDbGrpIdxMin = 0; unsigned int backupDbGrpIdxMax = lstSize(dbGroupList); // For every backup (oldest to newest) for the stanza, add it to the database group based on system-id and version for (unsigned int backupIdx = 0; backupIdx < varLstSize(backupSection); backupIdx++) { const KeyValue *const backupInfo = varKv(varLstGet(backupSection, backupIdx)); const KeyValue *const backupDbInfo = varKv(kvGet(backupInfo, KEY_DATABASE_VAR)); const unsigned int backupDbId = varUInt(kvGet(backupDbInfo, DB_KEY_ID_VAR)); const unsigned int backupRepoKey = varUInt(kvGet(backupDbInfo, KEY_REPO_KEY_VAR)); // Find the database group this backup belongs to and add it for (unsigned int dbIdx = 0; dbIdx < varLstSize(dbSection); dbIdx++) { const KeyValue *const pgInfo = varKv(varLstGet(dbSection, dbIdx)); const unsigned int dbId = varUInt(kvGet(pgInfo, DB_KEY_ID_VAR)); const unsigned int dbRepoKey = varUInt(kvGet(pgInfo, KEY_REPO_KEY_VAR)); if (backupDbId == dbId && backupRepoKey == dbRepoKey) { for (unsigned int dbGrpIdx = 0; dbGrpIdx < lstSize(dbGroupList); dbGrpIdx++) { const DbGroup *const dbGroupInfo = lstGet(dbGroupList, dbGrpIdx); if (dbGroupInfo->systemId == varUInt64(kvGet(pgInfo, DB_KEY_SYSTEM_ID_VAR)) && strEq(dbGroupInfo->version, varStr(kvGet(pgInfo, DB_KEY_VERSION_VAR)))) { varLstAdd(dbGroupInfo->backupList, varLstGet(backupSection, backupIdx)); // If we're only looking for one backup, then narrow the db group iterators if (backupLabel != NULL) { backupDbGrpIdxMin = dbGrpIdx; backupDbGrpIdxMax = dbGrpIdx + 1; } dbGrpIdx = lstSize(dbGroupList); } } dbIdx = varLstSize(dbSection); } } } String *const resultCurrent = strCatZ(strNew(), "\n db (current)"); bool displayCurrent = false; for (unsigned int dbGrpIdx = backupDbGrpIdxMin; dbGrpIdx < backupDbGrpIdxMax; dbGrpIdx++) { const DbGroup *const dbGroupInfo = lstGet(dbGroupList, dbGrpIdx); // Sort the results based on current or prior and only show the prior if it has archives or backups if (dbGroupInfo->current) { formatTextBackup(dbGroupInfo, resultCurrent); displayCurrent = true; } else if (dbGroupInfo->archiveMin != NULL || !varLstEmpty(dbGroupInfo->backupList)) { strCatZ(resultStr, "\n db (prior)"); formatTextBackup(dbGroupInfo, resultStr); } } // Add the current results to the end if necessary (e.g. current not displayed if a specified backup label is only in prior) if (displayCurrent == true) strCat(resultStr, resultCurrent); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get the lock info of the specified lock type for the stanza ***********************************************************************************************************************************/ static void infoUpdateStanzaLock( InfoStanzaLock *const stanzaLock, const String *const stanzaName, const unsigned int repoIdx, const LockType lockType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_STANZA_LOCK, stanzaLock); FUNCTION_TEST_PARAM(STRING, stanzaName); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(ENUM, lockType); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(stanzaLock != NULL); ASSERT(stanzaName != NULL); // If there is a valid lock for this stanza then backup/expire/restore must be running const LockReadResult lockResult = cmdLockRead(lockType, stanzaName, repoIdx); if (lockResult.status == lockReadStatusValid) { stanzaLock->held = true; if (lockResult.data.size != NULL) { ASSERT(lockResult.data.size != NULL); stanzaLock->sizeComplete += varUInt64(lockResult.data.sizeComplete); stanzaLock->size += varUInt64(lockResult.data.size); } } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get the backup and archive info files on the specified repo for the stanza ***********************************************************************************************************************************/ static void infoUpdateStanza( const Storage *const storage, InfoStanzaRepo *const stanzaRepo, const unsigned int repoIdx, const bool stanzaExists, const String *const backupLabel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE, storage); FUNCTION_TEST_PARAM(INFO_STANZA_REPO, stanzaRepo); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(BOOL, stanzaExists); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(storage != NULL); ASSERT(stanzaRepo != NULL); // If the stanza exists, attempt to get the info files if (stanzaExists) { volatile int stanzaStatus = INFO_STANZA_STATUS_CODE_OK; TRY_BEGIN() { // If full output is requested read info and manifest files const bool outputFull = cfgOptionSeq(cfgOptDetailLevel) == CFGOPTVAL_DETAIL_LEVEL_FULL; if (outputFull) { // Catch certain errors TRY_BEGIN() { // Attempt to load the backup info file stanzaRepo->repoList[repoIdx].backupInfo = infoBackupLoadFile( storage, strNewFmt(STORAGE_PATH_BACKUP "/%s/%s", strZ(stanzaRepo->name), INFO_BACKUP_FILE), stanzaRepo->repoList[repoIdx].cipher, stanzaRepo->repoList[repoIdx].cipherPass); } CATCH(FileMissingError) { // If there is no backup.info then set the status to indicate missing stanzaStatus = INFO_STANZA_STATUS_CODE_MISSING_STANZA_DATA; } CATCH(CryptoError) { // If a reason for the error is due to a an encryption error, add a hint THROW_FMT( CryptoError, "%s\n" "HINT: use option --stanza if encryption settings are different for the stanza than the global settings.", errorMessage()); } TRY_END(); // If backup.info was found, then get the archive.info file, which must exist if backup.info exists, else the failed // load will throw an error which will be trapped and recorded if (stanzaRepo->repoList[repoIdx].backupInfo != NULL) { stanzaRepo->repoList[repoIdx].archiveInfo = infoArchiveLoadFile( storage, strNewFmt(STORAGE_PATH_ARCHIVE "/%s/%s", strZ(stanzaRepo->name), INFO_ARCHIVE_FILE), stanzaRepo->repoList[repoIdx].cipher, stanzaRepo->repoList[repoIdx].cipherPass); // If a specific backup exists on this repo then attempt to load the manifest if (backupLabel != NULL) { stanzaRepo->repoList[repoIdx].manifest = manifestLoadFile( storage, strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)), stanzaRepo->repoList[repoIdx].cipher, infoPgCipherPass(infoBackupPg(stanzaRepo->repoList[repoIdx].backupInfo))); } } } // Read the lock files if backup.info is present. Exception: when only progress is requested, backup.info is skipped for // performance, so the lock files are read unconditionally -- though they may be outdated in this case. if (stanzaRepo->repoList[repoIdx].backupInfo != NULL || !outputFull) { infoUpdateStanzaLock(&stanzaRepo->repoList[repoIdx].backupLock, stanzaRepo->name, repoIdx, lockTypeBackup); infoUpdateStanzaLock(&stanzaRepo->restoreLock, stanzaRepo->name, repoIdx, lockTypeRestore); } stanzaRepo->repoList[repoIdx].stanzaStatus = stanzaStatus; } CATCH_ANY() { infoStanzaErrorAdd(&stanzaRepo->repoList[repoIdx], errorType(), STR(errorMessage())); } TRY_END(); } else stanzaRepo->repoList[repoIdx].stanzaStatus = INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH; // If the backup.info and therefore archive.info exist, and the currentPg has not been set for the stanza, then set it if (stanzaRepo->currentPgVersion == 0 && stanzaRepo->repoList[repoIdx].backupInfo != NULL) { InfoPgData backupInfoCurrentPg = infoPgData( infoBackupPg(stanzaRepo->repoList[repoIdx].backupInfo), infoPgDataCurrentId(infoBackupPg(stanzaRepo->repoList[repoIdx].backupInfo))); stanzaRepo->currentPgVersion = backupInfoCurrentPg.version; stanzaRepo->currentPgSystemId = backupInfoCurrentPg.systemId; } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Render the information for the stanza based on the command parameters ***********************************************************************************************************************************/ static String * infoRender(void) { FUNCTION_LOG_VOID(logLevelDebug); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get stanza if specified const String *const stanza = cfgOptionStrNull(cfgOptStanza); // Initialize the list of stanzas on all repos List *const stanzaRepoList = lstNewP(sizeof(InfoStanzaRepo), .sortOrder = sortOrderAsc, .comparator = lstComparatorStr); // Get the backup label if specified const String *const backupLabel = cfgOptionStrNull(cfgOptSet); bool backupFound = false; // If only progress info is requested then details about a specific backup may not be requested if (backupLabel != NULL && cfgOptionSeq(cfgOptDetailLevel) == CFGOPTVAL_DETAIL_LEVEL_PROGRESS) { THROW_FMT(OptionInvalidError, "option '%s' cannot be used with option '%s' = '%s'", cfgOptionName(cfgOptSet), cfgOptionName(cfgOptDetailLevel), CFGOPTVAL_DETAIL_LEVEL_PROGRESS_Z); } // Initialize the repo index unsigned int repoIdxMin = 0; const unsigned int repoTotal = cfgOptionGroupIdxTotal(cfgOptGrpRepo); unsigned int repoIdxMax = repoTotal - 1; // If the repo was specified then set index to the array location and max to loop only once if (cfgOptionTest(cfgOptRepo)) { repoIdxMin = cfgOptionGroupIdxDefault(cfgOptGrpRepo); repoIdxMax = repoIdxMin; } // Initialize error reporting at the repo level InfoRepoData *const repoErrorList = memNew(repoTotal * sizeof(InfoRepoData)); bool repoError = false; for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { // Initialize the error list on this repo repoErrorList[repoIdx] = (InfoRepoData) { .key = cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoIdx), .cipher = cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), .cipherPass = cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx), .error = NULL, }; // Initialize backup label indicator const String *backupExistsOnRepo = NULL; // Catch any repo errors TRY_BEGIN() { // Get the repo storage in case it is remote and encryption settings need to be pulled down const Storage *const storageRepo = storageRepoIdx(repoIdx); // If a backup set was specified, see if the manifest exists if (backupLabel != NULL) { // If the backup exists on this repo, set the global indicator that we found it on at least one repo and // set the exists label for later loading of the manifest if (storageExistsP(storageRepo, strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)))) { backupFound = true; backupExistsOnRepo = backupLabel; } } // Get a list of stanzas in the backup directory StringList *stanzaNameList = strLstSort( storageListP(storageRepo, STORAGE_PATH_BACKUP_STR), sortOrderAsc); // All stanzas will be "found" if they are in the storage list bool stanzaExists = true; if (stanza != NULL) { // If a specific stanza was requested and it is not on this repo, then stanzaExists flag will be reset to false if (strLstEmpty(stanzaNameList) || !strLstExists(stanzaNameList, stanza)) stanzaExists = false; // Narrow the list to only the requested stanza strLstFree(stanzaNameList); stanzaNameList = strLstNew(); strLstAdd(stanzaNameList, stanza); } // Process each stanza for (unsigned int stanzaIdx = 0; stanzaIdx < strLstSize(stanzaNameList); stanzaIdx++) { const String *const stanzaName = strLstGet(stanzaNameList, stanzaIdx); // Get the stanza if it is already in the list InfoStanzaRepo *const stanzaRepo = lstFind(stanzaRepoList, &stanzaName); // If the stanza was already added to the array, then update this repo for the stanza, else the stanza has not // yet been added to the list, so add it if (stanzaRepo != NULL) infoUpdateStanza(storageRepo, stanzaRepo, repoIdx, stanzaExists, backupExistsOnRepo); else { InfoStanzaRepo stanzaRepo = { .name = stanzaName, .currentPgVersion = 0, .currentPgSystemId = 0, .repoList = memNew(repoTotal * sizeof(InfoRepoData)), }; // Initialize all the repos for (unsigned int repoListIdx = 0; repoListIdx < repoTotal; repoListIdx++) { stanzaRepo.repoList[repoListIdx] = (InfoRepoData) { .key = cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoListIdx), .cipher = cfgOptionIdxStrId(cfgOptRepoCipherType, repoListIdx), .cipherPass = cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoListIdx), .error = NULL, }; } // Update the info for this repo infoUpdateStanza(storageRepo, &stanzaRepo, repoIdx, stanzaExists, backupExistsOnRepo); lstAdd(stanzaRepoList, &stanzaRepo); } } } CATCH_ANY() { // At this point, stanza-level errors were caught and stored in the stanza structure, any errors caught here are due // to higher level problems (e.g. repo inaccessible, invalid permissions on the repo, etc) and will be reported // later if there are valid stanzas. If there are no valid stanzas after the loop exits, then these errors will be // reported with a stanza named "[invalid]". infoStanzaErrorAdd(&repoErrorList[repoIdx], errorType(), STR(errorMessage())); repoError = true; } TRY_END(); } VariantList *infoList = varLstNew(); String *resultStr = strNew(); // Record any repository-level errors with each stanza -- if there are no stanzas and one was not requested, then create an // "[invalid]" one for reporting if (repoError) { if (lstEmpty(stanzaRepoList)) { const InfoStanzaRepo stanzaRepo = { .name = stanza != NULL ? stanza : INFO_STANZA_INVALID_STR, .currentPgVersion = 0, .currentPgSystemId = 0, .repoList = repoErrorList, }; lstAdd(stanzaRepoList, &stanzaRepo); } else { // For each stanza report the repos that were in error for (unsigned int idx = 0; idx < lstSize(stanzaRepoList); idx++) { const InfoStanzaRepo *const stanzaData = lstGet(stanzaRepoList, idx); for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { if (repoErrorList[repoIdx].error != NULL) stanzaData->repoList[repoIdx] = repoErrorList[repoIdx]; } } } } // If a backup label was requested but it was not found on any repo, report the error here rather than individually to avoid // listing each repo as "requested backup not found" if (backupLabel != NULL && !backupFound) { // Get the stanza record and update each repo to indicate backup not found where there is not already an error status // so that errors on other repositories will be displayed and not overwritten const InfoStanzaRepo *const stanzaRepo = lstFind(stanzaRepoList, &stanza); for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { if (stanzaRepo->repoList[repoIdx].stanzaStatus == INFO_STANZA_STATUS_CODE_OK) { stanzaRepo->repoList[repoIdx].stanzaStatus = INFO_STANZA_STATUS_CODE_BACKUP_MISSING; infoBackupFree(stanzaRepo->repoList[repoIdx].backupInfo); stanzaRepo->repoList[repoIdx].backupInfo = NULL; } } } // If the backup storage exists, then search for and process any stanzas if (!lstEmpty(stanzaRepoList)) infoList = stanzaInfoList(stanzaRepoList, backupLabel, repoIdxMin, repoIdxMax); // Format text output if (cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_TEXT) { // Process any stanza directories if (!varLstEmpty(infoList)) { // Is full output requested? const bool outputFull = cfgOptionSeq(cfgOptDetailLevel) == CFGOPTVAL_DETAIL_LEVEL_FULL; for (unsigned int stanzaIdx = 0; stanzaIdx < varLstSize(infoList); stanzaIdx++) { const KeyValue *const stanzaInfo = varKv(varLstGet(infoList, stanzaIdx)); const String *const stanzaName = varStr(kvGet(stanzaInfo, KEY_NAME_VAR)); // Add a carriage return between stanzas if (stanzaIdx > 0) strCatZ(resultStr, "\n"); // Stanza name and status strCatFmt(resultStr, "stanza: %s\n status: ", strZ(stanzaName)); // If an error has occurred, provide the information that is available and move onto next stanza const KeyValue *const stanzaStatus = varKv(kvGet(stanzaInfo, STANZA_KEY_STATUS_VAR)); const int statusCode = varInt(kvGet(stanzaStatus, STATUS_KEY_CODE_VAR)); // Get the backup lock info const KeyValue *const lockKv = varKv(kvGet(stanzaStatus, STATUS_KEY_LOCK_VAR)); const KeyValue *const backupLockKv = varKv(kvGet(lockKv, STATUS_KEY_LOCK_BACKUP_VAR)); const bool backupLockHeld = varBool(kvGet(backupLockKv, STATUS_KEY_LOCK_HELD_VAR)); const Variant *const backupPercentComplete = kvGet(backupLockKv, STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR); const String *const backupPercentCompleteStr = backupPercentComplete != NULL ? strNewFmt(" - %s complete", strZ(strNewPct(varUInt(backupPercentComplete), 10000))) : EMPTY_STR; // Get the restore lock info const KeyValue *const restoreLockKv = varKv(kvGet(lockKv, STATUS_KEY_LOCK_RESTORE_VAR)); const bool restoreLockHeld = varBool(kvGet(restoreLockKv, STATUS_KEY_LOCK_HELD_VAR)); const Variant *const restorePercentComplete = kvGet(restoreLockKv, STATUS_KEY_LOCK_PERCENT_COMPLETE_VAR); const String *const restorePercentCompleteStr = restorePercentComplete != NULL ? strNewFmt(" - %s complete", strZ(strNewPct(varUInt(restorePercentComplete), 10000))) : EMPTY_STR; // Build stanza status const bool errorStatus = statusCode != INFO_STANZA_STATUS_CODE_OK && (outputFull == false || statusCode != INFO_STANZA_STATUS_CODE_MIXED); const bool progressStatus = backupLockHeld == true || restoreLockHeld == true; const String *const statusLabelStr = statusCode == INFO_STANZA_STATUS_CODE_OK ? strNewZ(INFO_STANZA_STATUS_OK) : errorStatus == true ? strNewZ(INFO_STANZA_STATUS_ERROR) : strNewZ(INFO_STANZA_MIXED); const String *const statusErrorStr = errorStatus ? varStr(kvGet(stanzaStatus, STATUS_KEY_MESSAGE_VAR)) : EMPTY_STR; const String *const progressStr = backupLockHeld == true && restoreLockHeld == true ? strNewFmt( INFO_STANZA_STATUS_MESSAGE_LOCK_BACKUP "%s, " INFO_STANZA_STATUS_MESSAGE_LOCK_RESTORE "%s", strZ(backupPercentCompleteStr), strZ(restorePercentCompleteStr)) : backupLockHeld == true ? strNewFmt(INFO_STANZA_STATUS_MESSAGE_LOCK_BACKUP "%s", strZ(backupPercentCompleteStr)) : restoreLockHeld == true ? strNewFmt(INFO_STANZA_STATUS_MESSAGE_LOCK_RESTORE "%s", strZ(restorePercentCompleteStr)) : EMPTY_STR; if (progressStatus) { // Status: error (message, progress) if (errorStatus) { strCatFmt(resultStr, "%s (%s, %s)\n", strZ(statusLabelStr), strZ(statusErrorStr), strZ(progressStr)); } // Status: ok/mixed (progress) else strCatFmt(resultStr, "%s (%s)\n", strZ(statusLabelStr), strZ(progressStr)); } else { // Status: error (message) if (errorStatus) { strCatFmt(resultStr, "%s (%s)\n", strZ(statusLabelStr), strZ(statusErrorStr)); } // Status: ok/mixed else strCatFmt(resultStr, "%s\n", strZ(statusLabelStr)); } // Output per-repo backup progress when multiple backups are running const Variant *const backupRepoVar = kvGet(backupLockKv, STANZA_KEY_REPO_VAR); const VariantList *const backupRepoList = backupRepoVar != NULL ? varVarLst(backupRepoVar) : NULL; if (backupRepoList != NULL && varLstSize(backupRepoList) > 1) { for (unsigned int repoLockIdx = 0; repoLockIdx < varLstSize(backupRepoList); repoLockIdx++) { const KeyValue *const repoLockKv = varKv(varLstGet(backupRepoList, repoLockIdx)); strCatFmt( resultStr, " repo%u backup: %s complete\n", varUInt(kvGet(repoLockKv, REPO_KEY_KEY_VAR)), strZ(strNewPct( cvtPctToUInt( varUInt64(kvGet(repoLockKv, STATUS_KEY_LOCK_SIZE_COMPLETE_VAR)), varUInt64(kvGet(repoLockKv, STATUS_KEY_LOCK_SIZE_VAR))), 10000))); } } // Output the status per repo if (outputFull && (statusCode == INFO_STANZA_STATUS_CODE_MIXED || statusCode == INFO_STANZA_STATUS_CODE_PG_MISMATCH || statusCode == INFO_STANZA_STATUS_CODE_OTHER)) { const VariantList *const repoSection = kvGetList(stanzaInfo, STANZA_KEY_REPO_VAR); const bool multiRepo = varLstSize(repoSection) > 1; const char *const formatSpacer = multiRepo ? " " : " "; for (unsigned int repoIdx = 0; repoIdx < varLstSize(repoSection); repoIdx++) { const KeyValue *const repoInfo = varKv(varLstGet(repoSection, repoIdx)); const KeyValue *const repoStatus = varKv(kvGet(repoInfo, STANZA_KEY_STATUS_VAR)); // If more than one repo configured, then add the repo status per repo if (multiRepo) strCatFmt(resultStr, " repo%u: ", varUInt(kvGet(repoInfo, REPO_KEY_KEY_VAR))); if (varInt(kvGet(repoStatus, STATUS_KEY_CODE_VAR)) == INFO_STANZA_STATUS_CODE_OK) strCatZ(resultStr, INFO_STANZA_STATUS_OK "\n"); else { if (varInt(kvGet(repoStatus, STATUS_KEY_CODE_VAR)) == INFO_STANZA_STATUS_CODE_OTHER) { const StringList *const repoError = strLstNewSplit( varStr(kvGet(repoStatus, STATUS_KEY_MESSAGE_VAR)), STRDEF("\n")); strCatFmt( resultStr, "%s%s%s\n", multiRepo ? INFO_STANZA_STATUS_ERROR " (" INFO_STANZA_STATUS_MESSAGE_OTHER ")\n" : "", formatSpacer, strZ(strLstJoin(repoError, zNewFmt("\n%s", formatSpacer)))); } else { strCatFmt( resultStr, INFO_STANZA_STATUS_ERROR " (%s)\n", strZ(varStr(kvGet(repoStatus, STATUS_KEY_MESSAGE_VAR)))); } } } } // Add cipher type if the stanza is found on at least one repo if (outputFull && statusCode != INFO_STANZA_STATUS_CODE_MISSING_STANZA_PATH) { strCatFmt(resultStr, " cipher: %s\n", strZ(varStr(kvGet(stanzaInfo, KEY_CIPHER_VAR)))); // If the cipher is mixed across repos for this stanza then display the per-repo cipher type if (strEq(varStr(kvGet(stanzaInfo, KEY_CIPHER_VAR)), STRDEF(INFO_STANZA_MIXED))) { const VariantList *const repoSection = kvGetList(stanzaInfo, STANZA_KEY_REPO_VAR); for (unsigned int repoIdx = 0; repoIdx < varLstSize(repoSection); repoIdx++) { const KeyValue *const repoInfo = varKv(varLstGet(repoSection, repoIdx)); strCatFmt( resultStr, " repo%u: %s\n", varUInt(kvGet(repoInfo, REPO_KEY_KEY_VAR)), strZ(varStr(kvGet(repoInfo, KEY_CIPHER_VAR)))); } } } // Get the current database for this stanza if (outputFull && !varLstEmpty(kvGetList(stanzaInfo, STANZA_KEY_DB_VAR))) { const InfoStanzaRepo *const stanzaRepo = lstFind(stanzaRepoList, &stanzaName); formatTextDb( stanzaInfo, resultStr, pgVersionToStr(stanzaRepo->currentPgVersion), stanzaRepo->currentPgSystemId, backupLabel); } } } else resultStr = strNewZ("No stanzas exist in the repository.\n"); } // Format json output else { ASSERT(cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_JSON); resultStr = jsonFromVar(varNewVarLst(infoList)); } MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(resultStr); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdInfo(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { ioFdWriteOneStr(STDOUT_FILENO, infoRender()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/info/info.h0000664000175100017510000000125615240547630017663 0ustar00runnerrunner/*********************************************************************************************************************************** Info Command ***********************************************************************************************************************************/ #ifndef COMMAND_INFO_INFO_H #define COMMAND_INFO_INFO_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Render info and output to stdout FN_EXTERN void cmdInfo(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/local/0000775000175100017510000000000015240547630016712 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/local/local.c0000664000175100017510000000334115240547630020151 0ustar00runnerrunner/*********************************************************************************************************************************** Local Command ***********************************************************************************************************************************/ #include #include "command/archive/get/protocol.h" #include "command/archive/push/protocol.h" #include "command/backup/protocol.h" #include "command/local/local.h" #include "command/restore/protocol.h" #include "command/verify/protocol.h" #include "common/debug.h" #include "common/log.h" #include "config/config.intern.h" #include "config/protocol.h" #include "protocol/helper.h" #include "protocol/server.h" /*********************************************************************************************************************************** Command handlers ***********************************************************************************************************************************/ static const ProtocolServerHandler commandLocalHandler[] = { PROTOCOL_SERVER_HANDLER_ARCHIVE_GET_LIST PROTOCOL_SERVER_HANDLER_ARCHIVE_PUSH_LIST PROTOCOL_SERVER_HANDLER_BACKUP_LIST PROTOCOL_SERVER_HANDLER_RESTORE_LIST PROTOCOL_SERVER_HANDLER_VERIFY_LIST }; static const List *const commandLocalHandlerList = LSTDEF(commandLocalHandler); /**********************************************************************************************************************************/ FN_EXTERN void cmdLocal(ProtocolServer *const server) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { protocolServerProcess(server, cfgCommandJobRetry(), commandLocalHandlerList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/local/local.h0000664000175100017510000000132115240547630020152 0ustar00runnerrunner/*********************************************************************************************************************************** Local Command ***********************************************************************************************************************************/ #ifndef COMMAND_LOCAL_LOCAL_H #define COMMAND_LOCAL_LOCAL_H #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Local command FN_EXTERN void cmdLocal(ProtocolServer *server); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/lock.c0000664000175100017510000001740115240547630016717 0ustar00runnerrunner/*********************************************************************************************************************************** Command Lock Handler ***********************************************************************************************************************************/ #include #include "command/lock.h" #include "common/debug.h" #include "common/log.h" #include "config/config.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct CmdLockLocal { bool held; // Is the lock being held? } cmdLockLocal; /*********************************************************************************************************************************** Lock type names ***********************************************************************************************************************************/ static const char *const lockTypeName[] = { "archive", // lockTypeArchive "backup", // lockTypeBackup "restore", // lockTypeRestore }; /*********************************************************************************************************************************** Generate lock file name ***********************************************************************************************************************************/ static String * cmdLockFileName(const String *const stanza, const LockType lockType, const unsigned int repoKey) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, stanza); FUNCTION_TEST_PARAM(ENUM, lockType); FUNCTION_TEST_PARAM(UINT, repoKey); FUNCTION_TEST_END(); ASSERT(stanza != NULL); ASSERT(lockType < lockTypeAll); ASSERT(repoKey > 0); FUNCTION_TEST_RETURN(STRING, strNewFmt("%s-%s-%u" LOCK_FILE_EXT, strZ(stanza), lockTypeName[lockType], repoKey)); } /**********************************************************************************************************************************/ FN_EXTERN StringList * cmdLockList(void) { FUNCTION_TEST_VOID(); ASSERT(cfgLockType() != lockTypeNone); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { const LockType lockType = cfgLockType(); const LockType lockMin = lockType == lockTypeAll ? lockTypeArchive : lockType; const LockType lockMax = lockType == lockTypeAll ? (lockTypeAll - 1) : lockType; const bool repoSet = cfgOptionTest(cfgOptRepo); const unsigned int repoMin = repoSet ? cfgOptionGroupIdxDefault(cfgOptGrpRepo) : 0; const unsigned int repoMax = repoSet ? repoMin : cfgOptionGroupIdxTotal(cfgOptGrpRepo) - 1; for (LockType lockIdx = lockMin; lockIdx <= lockMax; lockIdx++) { for (unsigned int repoIdx = repoMin; repoIdx <= repoMax; repoIdx++) { strLstAdd( result, cmdLockFileName(cfgOptionStr(cfgOptStanza), lockIdx, cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoIdx))); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN bool cmdLockAcquire(const LockAcquireParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, param.returnOnNoLock); FUNCTION_LOG_END(); bool result = true; // Don't allow return on no lock when locking more than one file. This makes cleanup difficult and there are no known use cases. ASSERT(!param.returnOnNoLock || cfgLockType() != lockTypeAll); // Don't allow another lock if one is already held if (cmdLockLocal.held) THROW(AssertError, "lock is already held by this process"); // Lock files MEM_CONTEXT_TEMP_BEGIN() { const StringList *const lockList = cfgOptionTest(cfgOptLock) ? strLstNewVarLst(cfgOptionLst(cfgOptLock)) : cmdLockList(); for (unsigned int lockListIdx = 0; lockListIdx < strLstSize(lockList); lockListIdx++) result = lockAcquire(strLstGet(lockList, lockListIdx), param); } MEM_CONTEXT_TEMP_END(); // Set lock held flag cmdLockLocal.held = result; FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN bool cmdLockOwn(void) { FUNCTION_LOG_VOID(logLevelDebug); ASSERT(cfgLockType() != lockTypeNone); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { const StringList *const lockList = cmdLockList(); // A valid lock held by our own exec id means our own async process is running and holding the lock for (unsigned int lockListIdx = 0; lockListIdx < strLstSize(lockList); lockListIdx++) { const LockReadResult lockResult = lockReadP(strLstGet(lockList, lockListIdx)); if (lockResult.status == lockReadStatusValid && strEq(lockResult.data.execId, lockExecId())) { result = true; break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdLockWrite(const LockWriteParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(VARIANT, param.percentComplete); FUNCTION_LOG_PARAM(VARIANT, param.sizeComplete); FUNCTION_LOG_PARAM(VARIANT, param.size); FUNCTION_LOG_END(); String *const lockFileName = cmdLockFileName( cfgOptionStr(cfgOptStanza), cfgLockType(), cfgOptionGroupIdxToKey(cfgOptGrpRepo, cfgOptionGroupIdxDefault(cfgOptGrpRepo))); lockWrite(lockFileName, param); strFree(lockFileName); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN LockReadResult cmdLockRead(const LockType lockType, const String *const stanza, const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(ENUM, lockType); FUNCTION_LOG_PARAM(STRING, stanza); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); ASSERT(lockType == lockTypeBackup || lockType == lockTypeRestore); ASSERT(stanza != NULL); FUNCTION_AUDIT_STRUCT(); LockReadResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { const String *const lockFileName = cmdLockFileName(stanza, lockType, cfgOptionGroupIdxToKey(cfgOptGrpRepo, repoIdx)); MEM_CONTEXT_PRIOR_BEGIN() { result = lockReadP(lockFileName); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN bool cmdLockRelease(const LockReleaseParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BOOL, param.returnOnNoLock); FUNCTION_LOG_END(); bool result = true; // Release lock(s) if (cmdLockLocal.held) { result = lockRelease(param); cmdLockLocal.held = false; } // Else error when requested else if (!param.returnOnNoLock) THROW(AssertError, "no lock is held by this process"); FUNCTION_LOG_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/lock.h0000664000175100017510000000440215240547630016721 0ustar00runnerrunner/*********************************************************************************************************************************** Command Lock Handler ***********************************************************************************************************************************/ #ifndef COMMAND_LOCK_H #define COMMAND_LOCK_H #include "common/lock.h" #include "config/config.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get list of required locks FN_EXTERN StringList *cmdLockList(void); // Acquire a command lock. This will involve locking one or more files on disk depending on the lock type. Most operations only // acquire a single lock type (archive or backup), but the stanza commands all need to lock both. #define cmdLockAcquireP(...) \ cmdLockAcquire((LockAcquireParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool cmdLockAcquire(LockAcquireParam param); // Write data to command lock file #define cmdLockWriteP(...) \ cmdLockWrite((LockWriteParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void cmdLockWrite(LockWriteParam param); // Read a command lock file held by another process to get information about what the process is doing FN_EXTERN LockReadResult cmdLockRead(LockType lockType, const String *stanza, unsigned int repoKey); // Is a command lock held by this process's own exec id, i.e. is our own async process running? Async archive-get uses this to // decide whether to wait for its own async process or fetch the segment synchronously (when the lock is free or held by a foreign // process). FN_EXTERN bool cmdLockOwn(void); // Release command lock(s) #define cmdLockReleaseP(...) \ cmdLockRelease((LockReleaseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool cmdLockRelease(LockReleaseParam param); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/manifest/0000775000175100017510000000000015240547630017426 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/manifest/manifest.c0000664000175100017510000005730015240547630021405 0ustar00runnerrunner/*********************************************************************************************************************************** Manifest Command ***********************************************************************************************************************************/ #include #include #include "command/backup/blockMap.h" #include "command/manifest/manifest.h" #include "command/restore/blockChecksum.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/log.h" #include "common/type/json.h" #include "config/config.h" #include "info/manifest/manifest.h" #include "storage/helper.h" /*********************************************************************************************************************************** Block map render ***********************************************************************************************************************************/ typedef struct ManifestBlockDeltaReference { unsigned int reference; // Reference List *blockList; // List of blocks in the block map for the reference } ManifestBlockDeltaReference; // Reads that must be performed in order to extract blocks typedef struct ManifestBlockDeltaRead { unsigned int reference; // Reference to read from uint64_t bundleId; // Bundle to read from uint64_t offset; // Offset to begin read from uint64_t size; // Size of the read List *superBlockList; // Super block list } ManifestBlockDeltaRead; // Super blocks to be extracted from the read typedef struct ManifestBlockDeltaSuperBlock { uint64_t superBlockSize; // Super block size uint64_t size; // Stored super block size (with compression, etc.) List *blockList; // Block list } ManifestBlockDeltaSuperBlock; // Blocks to be extracted from the super block typedef struct ManifestBlockDeltaBlock { uint64_t no; // Block number in the super block uint64_t offset; // Offset into original file uint8_t checksum[XX_HASH_SIZE_MAX]; // Checksum of the block } ManifestBlockDeltaBlock; static List * cmdManifestBlockDelta( const BlockMap *const blockMap, const size_t blockSize, const size_t checksumSize, const Buffer *const blockChecksum) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_MAP, blockMap); FUNCTION_TEST_PARAM(SIZE, blockSize); FUNCTION_TEST_PARAM(SIZE, checksumSize); FUNCTION_TEST_PARAM(BUFFER, blockChecksum); FUNCTION_TEST_END(); ASSERT(blockMap != NULL); ASSERT(blockSize > 0); List *const result = lstNewP(sizeof(ManifestBlockDeltaRead)); MEM_CONTEXT_TEMP_BEGIN() { // Build list of references and for each reference the list of blocks for that reference const unsigned int blockChecksumSize = blockChecksum == NULL ? 0 : (unsigned int)(bufUsed(blockChecksum) / checksumSize); List *const referenceList = lstNewP(sizeof(ManifestBlockDeltaReference), .comparator = lstComparatorUInt); for (unsigned int blockMapIdx = 0; blockMapIdx < blockMapSize(blockMap); blockMapIdx++) { const BlockMapItem *const blockMapItem = blockMapGet(blockMap, blockMapIdx); // The block must be updated if it is beyond the blocks that exist in the block checksum list or when the checksum // stored in the repository is different from the block checksum list if (blockMapIdx >= blockChecksumSize || !bufEq( BUF(blockMapItem->checksum, checksumSize), BUF(bufPtrConst(blockChecksum) + blockMapIdx * checksumSize, checksumSize))) { const unsigned int reference = blockMapItem->reference; ManifestBlockDeltaReference *const referenceData = lstFind(referenceList, &reference); // If the reference has not been added if (referenceData == NULL) { ManifestBlockDeltaReference *referenceData = lstAdd( referenceList, &(ManifestBlockDeltaReference){.reference = reference, .blockList = lstNewP(sizeof(unsigned int))}); lstAdd(referenceData->blockList, &blockMapIdx); } // Else add the new block else lstAdd(referenceData->blockList, &blockMapIdx); } } // Sort the reference list ascending. This is an arbitrary choice as the order does not matter. lstSort(referenceList, sortOrderAsc); // Build delta for (unsigned int referenceIdx = 0; referenceIdx < lstSize(referenceList); referenceIdx++) { const ManifestBlockDeltaReference *const referenceData = (const ManifestBlockDeltaReference *)lstGet( referenceList, referenceIdx); ManifestBlockDeltaRead *blockDeltaRead = NULL; ManifestBlockDeltaSuperBlock *blockDeltaSuperBlock = NULL; const BlockMapItem *blockMapItemPrior = NULL; for (unsigned int blockIdx = 0; blockIdx < lstSize(referenceData->blockList); blockIdx++) { const unsigned int blockMapIdx = *(unsigned int *)lstGet(referenceData->blockList, blockIdx); const BlockMapItem *const blockMapItem = blockMapGet(blockMap, blockMapIdx); // Add read when it has changed if (blockMapItemPrior == NULL || (blockMapItemPrior->offset != blockMapItem->offset && blockMapItemPrior->offset + blockMapItemPrior->size != blockMapItem->offset)) { MEM_CONTEXT_OBJ_BEGIN(result) { ManifestBlockDeltaRead blockDeltaReadNew = { .reference = blockMapItem->reference, .bundleId = blockMapItem->bundleId, .offset = blockMapItem->offset, .superBlockList = lstNewP(sizeof(ManifestBlockDeltaSuperBlock)), }; blockDeltaRead = lstAdd(result, &blockDeltaReadNew); } MEM_CONTEXT_OBJ_END(); } // Add super block when it has changed if (blockMapItemPrior == NULL || blockMapItemPrior->offset != blockMapItem->offset) { MEM_CONTEXT_OBJ_BEGIN(blockDeltaRead->superBlockList) { ManifestBlockDeltaSuperBlock blockDeltaSuperBlockNew = { .superBlockSize = blockMapItem->superBlockSize, .size = blockMapItem->size, .blockList = lstNewP(sizeof(ManifestBlockDeltaBlock)), }; blockDeltaSuperBlock = lstAdd(blockDeltaRead->superBlockList, &blockDeltaSuperBlockNew); blockDeltaRead->size += blockMapItem->size; } MEM_CONTEXT_OBJ_END(); } // Add block ManifestBlockDeltaBlock blockDeltaBlockNew = { .no = blockMapItem->block, .offset = blockMapIdx * blockSize, }; memcpy( blockDeltaBlockNew.checksum, blockMapItem->checksum, SIZE_OF_STRUCT_MEMBER(ManifestBlockDeltaBlock, checksum)); lstAdd(blockDeltaSuperBlock->blockList, &blockDeltaBlockNew); // Set prior item for comparison on the next loop blockMapItemPrior = blockMapItem; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(LIST, result); } /*********************************************************************************************************************************** Block map render ***********************************************************************************************************************************/ static String * cmdManifestBlockDeltaRender(const Manifest *const manifest, const ManifestFile *const file, const bool json) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(MANIFEST_FILE, file); FUNCTION_LOG_PARAM(BOOL, json); FUNCTION_LOG_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // Load block checksums for the file if the pg option is explicitly set const Buffer *blockChecksum = NULL; const Buffer *checksum = NULL; if (cfgOptionSource(cfgOptPg) != cfgSourceDefault) { IoRead *const read = storageReadIo(storageNewReadP(storagePg(), manifestPathPg(file->name))); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioFilterGroupAdd(ioReadFilterGroup(read), blockChecksumNew(file->blockIncrSize, file->blockIncrChecksumSize)); ioReadDrain(read); checksum = pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE)); blockChecksum = pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), BLOCK_CHECKSUM_FILTER_TYPE)); } // If the file is up-to-date if (checksum != NULL && bufEq(checksum, BUF(file->checksumSha1, HASH_TYPE_SHA1_SIZE))) { if (json) strCatZ(result, "null"); else strCatZ(result, " file is up-to-date\n"); } // Else render the block delta else { StorageRead *const read = storageNewReadP( storageRepo(), backupFileRepoPathP( file->reference != NULL ? file->reference : manifestData(manifest)->backupLabel, .manifestName = file->name, .bundleId = file->bundleId, .compressType = manifestData(manifest)->backupOptionCompressType, .blockIncr = true), .offset = file->bundleOffset + file->sizeRepo - file->blockIncrMapSize, .limit = VARUINT64(file->blockIncrMapSize)); if (manifestCipherSubPass(manifest) != NULL) { ioFilterGroupAdd( ioReadFilterGroup(storageReadIo(read)), cipherBlockNewP( cipherModeDecrypt, cipherTypeAes256Cbc, BUFSTR(manifestCipherSubPass(manifest)), .raw = true)); } ioReadOpen(storageReadIo(read)); List *const blockDelta = cmdManifestBlockDelta( blockMapNewRead(storageReadIo(read), file->blockIncrSize, file->blockIncrChecksumSize), file->blockIncrSize, file->blockIncrChecksumSize, blockChecksum); unsigned int referenceRead = 0; uint64_t referenceReadSize = 0; unsigned int referenceSuperBlock = 0; uint64_t referenceSuperBlockSize = 0; unsigned int referenceBlock = 0; unsigned int totalRead = 0; uint64_t totalReadSize = 0; unsigned int totalSuperBlock = 0; uint64_t totalSuperBlockSize = 0; unsigned int totalBlock = 0; bool first = true; if (json) strCatChr(result, '['); else strCatChr(result, '\n'); for (unsigned int readIdx = 0; readIdx < lstSize(blockDelta); readIdx++) { const ManifestBlockDeltaRead *const read = lstGet(blockDelta, readIdx); referenceRead++; referenceReadSize += read->size; referenceSuperBlock += lstSize(read->superBlockList); for (unsigned int superBlockIdx = 0; superBlockIdx < lstSize(read->superBlockList); superBlockIdx++) { const ManifestBlockDeltaSuperBlock *const superBlock = lstGet(read->superBlockList, superBlockIdx); referenceSuperBlockSize += superBlock->superBlockSize; referenceBlock += lstSize(superBlock->blockList); } if (readIdx == lstSize(blockDelta) - 1 || read->reference != ((const ManifestBlockDeltaRead *)lstGet(blockDelta, readIdx + 1))->reference) { if (json) { if (first) first = false; else strCatChr(result, ','); strCatFmt( result, "{\"reference\":%u,\"read\":{\"total\":%u,\"size\":%" PRIu64 "},\"superBlock\":{\"total\":%u" ",\"size\":%" PRIu64 "},\"block\":{\"total\":%u}}", read->reference, referenceRead, referenceReadSize, referenceSuperBlock, referenceSuperBlockSize, referenceBlock); } else { const String *const reference = strSub( backupFileRepoPathP( strLstGet(manifestReferenceList(manifest), read->reference), .manifestName = file->name, .bundleId = read->bundleId, .compressType = manifestData(manifest)->backupOptionCompressType, .blockIncr = true), sizeof(STORAGE_REPO_BACKUP)); strCatFmt( result, " reference: %s, read: %u/%s, superBlock: %u/%s, block: %u/%s\n", strZ(reference), referenceRead, strZ(strSizeFormat(referenceReadSize)), referenceSuperBlock, strZ(strSizeFormat(referenceSuperBlockSize)), referenceBlock, strZ(strSizeFormat(referenceBlock * file->blockIncrSize))); } totalRead += referenceRead; totalReadSize += referenceReadSize; totalSuperBlock += referenceSuperBlock; totalSuperBlockSize += referenceSuperBlockSize; totalBlock += referenceBlock; referenceRead = 0; referenceReadSize = 0; referenceSuperBlock = 0; referenceSuperBlockSize = 0; referenceBlock = 0; } } if (json) strCatChr(result, ']'); else { strCatFmt( result, " total read: %u/%s, superBlock: %u/%s, block: %u/%s\n", totalRead, strZ(strSizeFormat(totalReadSize)), totalSuperBlock, strZ(strSizeFormat(totalSuperBlockSize)), totalBlock, strZ(strSizeFormat(totalBlock * file->blockIncrSize))); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /*********************************************************************************************************************************** Manifest file render ***********************************************************************************************************************************/ static String * cmdManifestFileRender(const Manifest *const manifest, const ManifestFile *const file, const bool blockDelta, const bool json) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(MANIFEST_FILE, file); FUNCTION_LOG_PARAM(BOOL, blockDelta); FUNCTION_LOG_PARAM(BOOL, json); FUNCTION_LOG_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { if (json) { strCatFmt(result, "{\"name\":%s", strZ(jsonFromVar(VARSTR(file->name)))); if (file->reference != NULL) strCatFmt(result, ",\"reference\":%s", strZ(jsonFromVar(VARSTR(file->reference)))); strCatFmt(result, ",\"size\":%" PRIu64, file->size); strCatFmt( result, ",\"checksum\":\"%s\"", strZ(strNewEncode(encodingHex, BUF(file->checksumSha1, HASH_TYPE_SHA1_SIZE)))); strCatFmt(result, ",\"repo\":{\"size\":%" PRIu64 "}", file->sizeRepo); if (file->bundleId != 0) strCatFmt(result, ",\"bundle\":{\"id\":%" PRIu64 ",\"offset\":%" PRIu64 "}", file->bundleId, file->bundleOffset); if (file->blockIncrSize != 0) { strCatFmt( result, ",\"block\":{\"size\":%zu,\"map\":{\"size\":%" PRIu64, file->blockIncrSize, file->blockIncrMapSize); if (blockDelta) strCatFmt(result, ",\"delta\":%s", strZ(cmdManifestBlockDeltaRender(manifest, file, json))); strCatFmt(result, "},\"checksum\":{\"size\":%zu}}", file->blockIncrChecksumSize); } strCatChr(result, '}'); } else { strCatFmt(result, " - %s\n", strZ(file->name)); if (file->reference != NULL) strCatFmt(result, " reference: %s\n", strZ(file->reference)); strCatFmt( result, " size: %s, repo %s\n", strZ(strSizeFormat(file->size)), strZ(strSizeFormat(file->sizeRepo))); strCatFmt(result, " checksum: %s\n", strZ(strNewEncode(encodingHex, BUF(file->checksumSha1, HASH_TYPE_SHA1_SIZE)))); if (file->bundleId != 0) strCatFmt(result, " bundle: %" PRIu64 "\n", file->bundleId); if (file->blockIncrSize != 0) { strCatFmt( result, " block: size %s, map size %s, checksum size %s\n", strZ(strSizeFormat(file->blockIncrSize)), strZ(strSizeFormat(file->blockIncrMapSize)), strZ(strSizeFormat(file->blockIncrChecksumSize))); if (blockDelta) { strCatZ(result, " block delta:"); strCat(result, cmdManifestBlockDeltaRender(manifest, file, json)); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /*********************************************************************************************************************************** Manifest render ***********************************************************************************************************************************/ static String * cmdManifestRender(void) { FUNCTION_LOG_VOID(logLevelDebug); String *const result = strNew(); // Set dry run to make sure this command never writes anything storageHelperDryRunInit(true); MEM_CONTEXT_TEMP_BEGIN() { // Load backup.info and cipher const InfoBackup *const infoBackup = infoBackupLoadFile( storageRepo(), INFO_BACKUP_PATH_FILE_STR, cfgOptionStrId(cfgOptRepoCipherType), cfgOptionStrNull(cfgOptRepoCipherPass)); const String *const cipherPass = infoPgCipherPass(infoBackupPg(infoBackup)); const CipherType cipherType = cipherPass == NULL ? cipherTypeNone : cipherTypeAes256Cbc; // Load manifest const Manifest *const manifest = manifestLoadFile( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(cfgOptionStr(cfgOptSet))), cipherType, cipherPass); // Manifest info const ManifestData *const data = manifestData(manifest); const bool json = cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_JSON; if (json) { strCatFmt(result, "{\"label\":\"%s\"", strZ(data->backupLabel)); strCatFmt(result, ",\"reference\":[%s]", strZ(strLstJoinQuote(manifestReferenceList(manifest), ",", "\""))); strCatFmt(result, ",\"type\":\"%s\"", zNewStrId(data->backupType)); strCatFmt( result, ",\"time\":{\"start\":%" PRId64 ",\"copy\":%" PRId64 ",\"stop\":%" PRId64 "}", (int64_t)data->backupTimestampStart, (int64_t)data->backupTimestampCopyStart, (int64_t)data->backupTimestampStop); strCatFmt( result, ",\"bundle\":{\"bundle\":%s,\"raw\":%s}", cvtBoolToConstZ(data->bundle), cvtBoolToConstZ(data->bundleRaw)); strCatFmt(result, ",\"block\":{\"block\":%s}", cvtBoolToConstZ(data->blockIncr)); } else { strCatFmt(result, "label: %s\n", strZ(data->backupLabel)); strCatFmt(result, "reference: %s\n", strZ(strLstJoin(manifestReferenceList(manifest), ", "))); strCatFmt(result, "type: %s\n", zNewStrId(data->backupType)); int64_t duration = (int64_t)(data->backupTimestampStop - data->backupTimestampStart); strCatFmt( result, "time: start: %s, stop: %s, duration: %" PRId64 ":%02" PRId64 ":%02" PRId64 "\n", strZ(strNewTimeP("%Y-%m-%d %H:%M:%S", data->backupTimestampStart)), strZ(strNewTimeP("%Y-%m-%d %H:%M:%S", data->backupTimestampStop)), duration / 3600, duration % 3600 / 60, duration % 60); strCatFmt(result, "bundle: %s\n", cvtBoolToConstZ(data->bundle)); strCatFmt(result, "block: %s\n", cvtBoolToConstZ(data->blockIncr)); strCatChr(result, '\n'); } // File list if (json) strCatZ(result, ",\"fileList\":["); else strCatZ(result, "file list:\n"); const String *const filter = cfgOptionStrNull(cfgOptFilter); // Single file if (filter != NULL) { const ManifestFile file = manifestFileFind(manifest, filter); strCat(result, cmdManifestFileRender(manifest, &file, true, json)); } // Multiple files else { bool first = true; for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); // Filter on reference if (cfgOptionTest(cfgOptReference)) { if (strEqZ(cfgOptionStr(cfgOptReference), "latest")) { if (file.reference != NULL) continue; } else if (!strEq(cfgOptionStr(cfgOptReference), file.reference)) continue; } if (!first) { if (json) strCatChr(result, ','); else strCatChr(result, '\n'); } else first = false; strCat(result, cmdManifestFileRender(manifest, &file, false, json)); } } if (json) strCatZ(result, "]}"); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdManifest(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { ioFdWriteOneStr(STDOUT_FILENO, cmdManifestRender()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/manifest/manifest.h0000664000175100017510000000126315240547630021407 0ustar00runnerrunner/*********************************************************************************************************************************** Manifest Command ***********************************************************************************************************************************/ #ifndef COMMAND_MANIFEST_MANIFEST_H #define COMMAND_MANIFEST_MANIFEST_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Manifest info FN_EXTERN void cmdManifest(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/remote/0000775000175100017510000000000015240547630017113 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/remote/remote.c0000664000175100017510000001036415240547630020556 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Command ***********************************************************************************************************************************/ #include #include #include "command/backup/blockIncr.h" #include "command/backup/pageChecksum.h" #include "command/control/common.h" #include "command/lock.h" #include "command/remote/remote.h" #include "command/restore/blockChecksum.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/filter/sink.h" #include "common/io/filter/size.h" #include "common/log.h" #include "config/config.h" #include "config/protocol.h" #include "db/protocol.h" #include "protocol/helper.h" #include "protocol/server.h" #include "storage/remote/protocol.h" /*********************************************************************************************************************************** Command handlers ***********************************************************************************************************************************/ static const ProtocolServerHandler commandRemoteHandler[] = { PROTOCOL_SERVER_HANDLER_DB_LIST PROTOCOL_SERVER_HANDLER_OPTION_LIST PROTOCOL_SERVER_HANDLER_STORAGE_REMOTE_LIST }; static const List *const commandRemoteHandlerList = LSTDEF(commandRemoteHandler); /*********************************************************************************************************************************** Filter handlers ***********************************************************************************************************************************/ static const StorageRemoteFilterHandler storageRemoteFilterHandler[] = { {.type = BLOCK_CHECKSUM_FILTER_TYPE, .handlerParam = blockChecksumNewPack}, {.type = BLOCK_INCR_FILTER_TYPE, .handlerParam = blockIncrNewPack}, {.type = CIPHER_BLOCK_FILTER_TYPE, .handlerParam = cipherBlockNewPack}, {.type = CRYPTO_HASH_FILTER_TYPE, .handlerParam = cryptoHashNewPack}, {.type = PAGE_CHECKSUM_FILTER_TYPE, .handlerParam = pageChecksumNewPack}, {.type = SINK_FILTER_TYPE, .handlerNoParam = ioSinkNew}, {.type = SIZE_FILTER_TYPE, .handlerNoParam = ioSizeNew}, }; static const List *const storageRemoteFilterHandlerList = LSTDEF(storageRemoteFilterHandler); /**********************************************************************************************************************************/ FN_EXTERN void cmdRemote(ProtocolServer *const server) { FUNCTION_LOG_VOID(logLevelDebug); // Set filter handlers storageRemoteFilterHandlerSet(storageRemoteFilterHandlerList); MEM_CONTEXT_TEMP_BEGIN() { // Acquire a lock if this command needs one. We'll use the noop that is always sent from the client right after the // handshake to return an error. We can't take a lock earlier than this because we want the error to go back through the // protocol layer. volatile bool success = false; TRY_BEGIN() { // Get the command. No need to check parameters since we know this is the first noop. CHECK(FormatError, protocolServerRequest(server).id == PROTOCOL_COMMAND_NOOP, "noop expected"); // Only try the lock if this is process 0, i.e. the remote started from the main process if (cfgOptionUInt(cfgOptProcess) == 0) { // Acquire a lock if this command requires a lock if (cfgLockRemoteRequired()) { // Make sure the local host is not stopped lockStopTest(); // Acquire the lock cmdLockAcquireP(); } } // Notify the client of success protocolServerResponseP(server); success = true; } CATCH_ANY() { protocolServerError(server, errorCode(), STR(errorMessage()), STR(errorStackTrace())); } TRY_END(); // If not successful we'll just exit if (success) protocolServerProcess(server, NULL, commandRemoteHandlerList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/remote/remote.h0000664000175100017510000000133015240547630020554 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Command ***********************************************************************************************************************************/ #ifndef COMMAND_REMOTE_REMOTE_H #define COMMAND_REMOTE_REMOTE_H #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Remote command FN_EXTERN void cmdRemote(ProtocolServer *server); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/0000775000175100017510000000000015240547630016565 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/common.c0000664000175100017510000000525415240547630020227 0ustar00runnerrunner/*********************************************************************************************************************************** Common Functions and Definitions for Repo Commands ***********************************************************************************************************************************/ #include #include "command/repo/common.h" #include "common/debug.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN String * repoPathIsValid(const String *path) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_END(); ASSERT(path != NULL); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Make sure there are no occurrences of // if (strstr(strZ(path), "//") != NULL) THROW_FMT(ParamInvalidError, "path '%s' cannot contain //", strZ(path)); // Make sure the path does not end with a slash if (strEndsWith(path, FSLASH_STR)) path = strPath(path); // Validate absolute paths if (strBeginsWith(path, FSLASH_STR)) { // Get the repo path using repo storage in case it is remotely configured const String *const repoPath = storagePathP(storageRepo(), NULL); // If the path is exactly equal to the repo path then the relative path is empty if (strEq(path, repoPath)) { MEM_CONTEXT_PRIOR_BEGIN() { result = strNew(); } MEM_CONTEXT_PRIOR_END(); } // Else check that the file path begins with the repo path else { if (!strEq(repoPath, FSLASH_STR) && !strBeginsWith(path, strNewFmt("%s/", strZ(repoPath)))) { THROW_FMT( ParamInvalidError, "absolute path '%s' is not in base path '%s'", strZ(path), strZ(cfgOptionDisplay(cfgOptRepoPath))); } MEM_CONTEXT_PRIOR_BEGIN() { // Get the relative part of the path/file result = strSub(path, strEq(repoPath, FSLASH_STR) ? 1 : strSize(repoPath) + 1); } MEM_CONTEXT_PRIOR_END(); } } else { MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(path); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/common.h0000664000175100017510000000154415240547630020232 0ustar00runnerrunner/*********************************************************************************************************************************** Repo Command Common ***********************************************************************************************************************************/ #ifndef COMMAND_REPO_COMMON_H #define COMMAND_REPO_COMMON_H #include "common/type/string.h" /*********************************************************************************************************************************** Functions and Definitions ***********************************************************************************************************************************/ // Path cannot contain //. Strip trailing /. Absolute path must fall under repo path. Throw error or return validated relative path. FN_EXTERN String *repoPathIsValid(const String *path); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/get.c0000664000175100017510000002044215240547630017512 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Get Command ***********************************************************************************************************************************/ #include #include #include "command/repo/common.h" #include "command/repo/get.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" #include "info/infoArchive.h" #include "info/infoBackup.h" /*********************************************************************************************************************************** Write source file to destination IO ***********************************************************************************************************************************/ static int storageGetProcess(IoWrite *const destination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, destination); FUNCTION_LOG_END(); // Get source file if (strLstSize(cfgCommandParam()) != 1) THROW(ParamRequiredError, "source file required"); const String *file = strLstGet(cfgCommandParam(), 0); // Assume the file is missing int result = 1; MEM_CONTEXT_TEMP_BEGIN() { // Is path valid for repo? file = repoPathIsValid(file); // Create new file read IoRead *const source = storageReadIo( storageNewReadP(storageRepo(), file, .ignoreMissing = cfgOptionBool(cfgOptIgnoreMissing))); // Add decryption if needed if (!cfgOptionBool(cfgOptRaw)) { const CipherType repoCipherType = cfgOptionStrId(cfgOptRepoCipherType); if (repoCipherType != cipherTypeNone) { // Check for a passphrase parameter const String *cipherPass = cfgOptionStrNull(cfgOptCipherPass); // If not passed as a parameter then determine the passphrase using the following pattern: // // REPO / (repo passphrase) // / archive / (repo passphrase) // / archive / stanza / (archive passphrase) // / backup / (repo passphrase) // / backup / stanza / (backup passphrase) // / backup / stanza / set / (manifest passphrase) // / backup / stanza / backup.history / (backup passphrase) // // Nothing should be stored at the top level of the repo except the backup/archive paths. The backup/archive paths // should contain only stanza paths. // ----------------------------------------------------------------------------------------------------------------- if (cipherPass == NULL) { const StringList *const filePathSplitLst = strLstNewSplit(file, FSLASH_STR); // At a minimum the path must contain archive/backup, a stanza, and a file if (strLstSize(filePathSplitLst) > 2) { const String *const stanza = strLstGet(filePathSplitLst, 1); // If stanza option is specified then it must match the given file path if (cfgOptionStrNull(cfgOptStanza) != NULL && !strEq(stanza, cfgOptionStr(cfgOptStanza))) { THROW_FMT( OptionInvalidValueError, "stanza name '%s' given in option doesn't match the given path", strZ(cfgOptionDisplay(cfgOptStanza))); } // Archive path if (strEq(strLstGet(filePathSplitLst, 0), STORAGE_PATH_ARCHIVE_STR)) { cipherPass = cfgOptionStr(cfgOptRepoCipherPass); // Find the archive passphrase if (!strEndsWithZ(file, INFO_ARCHIVE_FILE) && !strEndsWithZ(file, INFO_ARCHIVE_FILE INFO_COPY_EXT)) { const InfoArchive *const info = infoArchiveLoadFile( storageRepo(), strNewFmt(STORAGE_PATH_ARCHIVE "/%s/%s", strZ(stanza), INFO_ARCHIVE_FILE), repoCipherType, cipherPass); cipherPass = infoArchiveCipherPass(info); } } // Backup path if (strEq(strLstGet(filePathSplitLst, 0), STORAGE_PATH_BACKUP_STR)) { cipherPass = cfgOptionStr(cfgOptRepoCipherPass); if (!strEndsWithZ(file, INFO_BACKUP_FILE) && !strEndsWithZ(file, INFO_BACKUP_FILE INFO_COPY_EXT)) { // Find the backup passphrase const InfoBackup *const info = infoBackupLoadFile( storageRepo(), strNewFmt(STORAGE_PATH_BACKUP "/%s/%s", strZ(stanza), INFO_BACKUP_FILE), repoCipherType, cipherPass); cipherPass = infoBackupCipherPass(info); // Find the manifest passphrase if (!strEq(strLstGet(filePathSplitLst, 2), STRDEF(BACKUP_PATH_HISTORY)) && !strEndsWithZ(file, BACKUP_MANIFEST_FILE) && !strEndsWithZ(file, BACKUP_MANIFEST_FILE INFO_COPY_EXT)) { const Manifest *const manifest = manifestLoadFile( storageRepo(), strNewFmt( STORAGE_PATH_BACKUP "/%s/%s/%s", strZ(stanza), strZ(strLstGet(filePathSplitLst, 2)), BACKUP_MANIFEST_FILE), repoCipherType, cipherPass); cipherPass = manifestCipherSubPass(manifest); } } } } } // Error when unable to determine cipher passphrase if (cipherPass == NULL) THROW_FMT(OptionInvalidValueError, "unable to determine cipher passphrase for '%s'", strZ(file)); // Add encryption filter cipherBlockFilterGroupAdd(ioReadFilterGroup(source), repoCipherType, cipherModeDecrypt, cipherPass); } } // Open source if (ioReadOpen(source)) { // Open the destination file now that we know the source exists and is readable ioWriteOpen(destination); // Copy data from source to destination ioCopyP(source, destination); // Close the source and destination ioReadClose(source); ioWriteClose(destination); // Source file exists result = 0; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN int cmdStorageGet(void) { FUNCTION_LOG_VOID(logLevelDebug); // Assume the file is missing int result = 1; MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { result = storageGetProcess(ioFdWriteNew(STRDEF("stdout"), STDOUT_FILENO, cfgOptionUInt64(cfgOptIoTimeout))); } // Ignore write errors because it's possible (even likely) that this output is being piped to something like head which will // exit when it gets what it needs and leave us writing to a broken pipe. It would be better to just ignore the broken pipe // error but currently we don't store system error codes. CATCH(FileWriteError) { } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/get.h0000664000175100017510000000127715240547630017524 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Get Command ***********************************************************************************************************************************/ #ifndef COMMAND_STORAGE_GET_H #define COMMAND_STORAGE_GET_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get a file from the repository FN_EXTERN int cmdStorageGet(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/ls.c0000664000175100017510000001352415240547630017354 0ustar00runnerrunner/*********************************************************************************************************************************** Repository List Command ***********************************************************************************************************************************/ #include #include #include "command/repo/common.h" #include "command/repo/ls.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/type/json.h" #include "common/type/string.h" #include "config/config.h" #include "storage/helper.h" /*********************************************************************************************************************************** Render storage list ***********************************************************************************************************************************/ static void storageListRenderInfo(const StorageInfo *const info, IoWrite *const write, const bool json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_PARAM(IO_WRITE, write); FUNCTION_TEST_PARAM(BOOL, json); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(info != NULL); ASSERT(write != NULL); // Render in json if (json) { ioWriteStr(write, jsonFromVar(VARSTR(info->name))); ioWrite(write, BUFSTRDEF(":{\"type\":\"")); switch (info->type) { case storageTypeFile: ioWrite(write, BUFSTRDEF("file\"")); break; case storageTypeLink: ioWrite(write, BUFSTRDEF("link\"")); break; case storageTypePath: ioWrite(write, BUFSTRDEF("path\"")); break; case storageTypeSpecial: ioWrite(write, BUFSTRDEF("special\"")); break; } if (info->type == storageTypeFile) { ioWriteStr(write, strNewFmt(",\"size\":%" PRIu64, info->size)); ioWriteStr(write, strNewFmt(",\"time\":%" PRId64, (int64_t)info->timeModified)); if (info->versionId != NULL) ioWriteStr(write, strNewFmt(",\"version\":%s", strZ(jsonFromVar(VARSTR(info->versionId))))); } if (info->type == storageTypeLink) ioWriteStr(write, strNewFmt(",\"destination\":%s", strZ(jsonFromVar(VARSTR(info->linkDestination))))); ioWrite(write, BRACER_BUF); } // Render in text else { ioWrite(write, BUFSTR(info->name)); ioWrite(write, LF_BUF); } FUNCTION_TEST_RETURN_VOID(); } static void storageListRender(IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); // Get path const String *path = NULL; // Get and validate if path is valid for repo if (strLstSize(cfgCommandParam()) == 1) path = repoPathIsValid(strLstGet(cfgCommandParam(), 0)); else if (strLstSize(cfgCommandParam()) > 1) THROW(ParamInvalidError, "only one path may be specified"); // Get options const bool json = cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_OUTPUT_JSON; const String *const expression = cfgOptionStrNull(cfgOptFilter); RegExp *const regExp = expression == NULL ? NULL : regExpNew(expression); ioWriteOpen(write); if (json) ioWrite(write, BRACEL_BUF); // Check if this is a file StorageInfo info = storageInfoP(storageRepo(), path, .ignoreMissing = true); if (info.exists && info.type == storageTypeFile) { if (regExp == NULL || regExpMatch(regExp, storagePathP(storageRepo(), path))) { info.name = DOT_STR; storageListRenderInfo(&info, write, json); } } // Else try to list the path else { // The path will always be reported as existing so we don't get different results from storage that does not support paths bool first = true; if (json && (regExp == NULL || regExpMatch(regExp, DOT_STR))) { storageListRenderInfo(&(StorageInfo){.type = storageTypePath, .name = DOT_STR}, write, json); first = false; } // List content of the path StorageIterator *const storageItr = storageNewItrP( storageRepo(), path, .sortOrder = cfgOptionSeq(cfgOptSort), .expression = expression, .recurse = cfgOptionBool(cfgOptRecurse)); while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); // Add separator character if (!first && json) ioWrite(write, COMMA_BUF); else first = false; storageListRenderInfo(&info, write, json); } } if (json) { ioWrite(write, BRACER_BUF); ioWrite(write, LF_BUF); } ioWriteClose(write); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdStorageList(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { storageListRender(ioFdWriteNew(STRDEF("stdout"), STDOUT_FILENO, cfgOptionUInt64(cfgOptIoTimeout))); } // Ignore write errors because it's possible (even likely) that this output is being piped to something like head which will // exit when it gets what it needs and leave us writing to a broken pipe. It would be better to just ignore the broken pipe // error but currently we don't store system error codes. CATCH(FileWriteError) { } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/ls.h0000664000175100017510000000130215240547630017350 0ustar00runnerrunner/*********************************************************************************************************************************** Repository List Command ***********************************************************************************************************************************/ #ifndef COMMAND_STORAGE_LIST_H #define COMMAND_STORAGE_LIST_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // List files in the repository FN_EXTERN void cmdStorageList(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/put.c0000664000175100017510000000602615240547630017545 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Put Command ***********************************************************************************************************************************/ #include #include #include "command/repo/common.h" #include "command/repo/put.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/io/fdRead.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" /*********************************************************************************************************************************** Write source IO to destination file ***********************************************************************************************************************************/ static void storagePutProcess(IoRead *source) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, source); FUNCTION_LOG_END(); // Get destination file const String *file = NULL; if (strLstSize(cfgCommandParam()) == 1) file = strLstGet(cfgCommandParam(), 0); else THROW(ParamRequiredError, "destination file required"); MEM_CONTEXT_TEMP_BEGIN() { // Is path valid for repo? file = repoPathIsValid(file); StorageWrite *const destination = storageNewWriteP(storageRepoWrite(), file); // Add encryption if needed if (!cfgOptionBool(cfgOptRaw)) { const CipherType repoCipherType = cfgOptionStrId(cfgOptRepoCipherType); if (repoCipherType != cipherTypeNone) { // Check for a passphrase parameter const String *cipherPass = cfgOptionStrNull(cfgOptCipherPass); // If not passed as a parameter use the repo passphrase if (cipherPass == NULL) cipherPass = cfgOptionStr(cfgOptRepoCipherPass); // Add encryption filter cipherBlockFilterGroupAdd( ioWriteFilterGroup(storageWriteIo(destination)), repoCipherType, cipherModeEncrypt, cipherPass); } } // Open source and destination ioReadOpen(source); ioWriteOpen(storageWriteIo(destination)); // Copy data from source to destination ioCopyP(source, storageWriteIo(destination)); // Close the source and destination ioReadClose(source); ioWriteClose(storageWriteIo(destination)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cmdStoragePut(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { storagePutProcess(ioFdReadNew(STRDEF("stdin"), STDIN_FILENO, ioTimeoutMs())); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/put.h0000664000175100017510000000130015240547630017540 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Put Command ***********************************************************************************************************************************/ #ifndef COMMAND_STORAGE_PUT_H #define COMMAND_STORAGE_PUT_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Put a file into the repository FN_EXTERN void cmdStoragePut(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/rm.c0000664000175100017510000000346315240547630017355 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Remove Command ***********************************************************************************************************************************/ #include #include "command/repo/common.h" #include "command/repo/rm.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStorageRemove(void) { FUNCTION_LOG_VOID(logLevelDebug); // Get path const String *path = NULL; if (strLstSize(cfgCommandParam()) == 1) path = strLstGet(cfgCommandParam(), 0); else if (strLstSize(cfgCommandParam()) > 1) THROW(ParamInvalidError, "only one path may be specified"); MEM_CONTEXT_TEMP_BEGIN() { // Is path valid for repo? if (path != NULL) path = repoPathIsValid(path); // Check if this is a file const StorageInfo info = storageInfoP(storageRepo(), path, .ignoreMissing = true); if (info.exists && info.type == storageTypeFile) { storageRemoveP(storageRepoWrite(), path); } // Else try to remove a path else { const bool recurse = cfgOptionBool(cfgOptRecurse); if (!recurse && !strLstEmpty(storageListP(storageRepo(), path))) THROW(OptionInvalidError, CFGOPT_RECURSE " option must be used to delete non-empty path"); storagePathRemoveP(storageRepoWrite(), path, .recurse = recurse); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/repo/rm.h0000664000175100017510000000132415240547630017354 0ustar00runnerrunner/*********************************************************************************************************************************** Repository Remove Command ***********************************************************************************************************************************/ #ifndef COMMAND_STORAGE_REMOVE_H #define COMMAND_STORAGE_REMOVE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Remove paths/files from the repository FN_EXTERN void cmdStorageRemove(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/0000775000175100017510000000000015240547630017303 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/blockChecksum.c0000664000175100017510000001555015240547630022232 0ustar00runnerrunner/*********************************************************************************************************************************** Block Hash List ***********************************************************************************************************************************/ #include #include "command/restore/blockChecksum.h" #include "common/crypto/common.h" #include "common/crypto/xxhash.h" #include "common/debug.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockChecksum { size_t blockSize; // Block size for checksums size_t checksumSize; // Checksum size size_t blockCurrent; // Size of current block IoFilter *checksum; // Checksum of current block List *list; // List of checksums } BlockChecksum; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_BLOCK_CHECKSUM_TYPE \ BlockChecksum * #define FUNCTION_LOG_BLOCK_CHECKSUM_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "BlockChecksum", buffer, bufferSize) /*********************************************************************************************************************************** Generate block checksum list ***********************************************************************************************************************************/ static void blockChecksumProcess(THIS_VOID, const Buffer *const input) { THIS(BlockChecksum); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_CHECKSUM, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); size_t inputOffset = 0; // Loop until input is consumed while (inputOffset != bufUsed(input)) { // Create checksum object if needed if (this->checksum == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->checksum = xxHashNew(this->checksumSize); this->blockCurrent = 0; } MEM_CONTEXT_OBJ_END(); } // Calculate how much data to checksum and perform checksum const size_t blockRemains = this->blockSize - this->blockCurrent; const size_t inputRemains = bufUsed(input) - inputOffset; const size_t blockChecksumSize = blockRemains < inputRemains ? blockRemains : inputRemains; ioFilterProcessIn(this->checksum, BUF(bufPtrConst(input) + inputOffset, blockChecksumSize)); // Update amount of data checksummed inputOffset += blockChecksumSize; this->blockCurrent += blockChecksumSize; // If the block size has been reached then output the checksum if (this->blockCurrent == this->blockSize) { MEM_CONTEXT_TEMP_BEGIN() { lstAdd(this->list, bufPtrConst(pckReadBinP(pckReadNew(ioFilterResult(this->checksum))))); ioFilterFree(this->checksum); this->checksum = NULL; } MEM_CONTEXT_TEMP_END(); } } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get a binary representation of the checksum list ***********************************************************************************************************************************/ static Pack * blockChecksumResult(THIS_VOID) { THIS(BlockChecksum); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BLOCK_CHECKSUM, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); // If there is a remainder in the checksum if (this->checksum) lstAdd(this->list, bufPtrConst(pckReadBinP(pckReadNew(ioFilterResult(this->checksum))))); pckWriteBinP(packWrite, BUF(lstGet(this->list, 0), lstSize(this->list) * this->checksumSize)); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * blockChecksumNew(const size_t blockSize, const size_t checksumSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SIZE, blockSize); FUNCTION_LOG_PARAM(SIZE, checksumSize); FUNCTION_LOG_END(); ASSERT(blockSize != 0); ASSERT(checksumSize != 0); // Allocate memory to hold process state OBJ_NEW_BEGIN(BlockChecksum, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (BlockChecksum) { .blockSize = blockSize, .checksumSize = checksumSize, .list = lstNewP(checksumSize), }; } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, blockSize); pckWriteU64P(packWrite, checksumSize); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP(BLOCK_CHECKSUM_FILTER_TYPE, this, paramList, .in = blockChecksumProcess, .result = blockChecksumResult)); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * blockChecksumNewPack(const Pack *const paramList) { IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const paramListPack = pckReadNew(paramList); const size_t blockSize = (size_t)pckReadU64P(paramListPack); const size_t checksumSize = (size_t)pckReadU64P(paramListPack); result = ioFilterMove(blockChecksumNew(blockSize, checksumSize), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/blockChecksum.h0000664000175100017510000000257015240547630022235 0ustar00runnerrunner/*********************************************************************************************************************************** Block Hash List Build a list of hashes based on a block size. This is used to compare the contents of a file to block map to determine what needs to be updated. ***********************************************************************************************************************************/ #ifndef COMMAND_RESTORE_BLOCK_CHECKSUM_H #define COMMAND_RESTORE_BLOCK_CHECKSUM_H #include "common/io/filter/filter.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BLOCK_CHECKSUM_FILTER_TYPE STRID5("blk-chk", 0x2d03dad820) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *blockChecksumNew(size_t blockSize, size_t checksumSize); FN_EXTERN IoFilter *blockChecksumNewPack(const Pack *paramList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/blockDelta.c0000664000175100017510000003372715240547630021527 0ustar00runnerrunner/*********************************************************************************************************************************** Block Restore ***********************************************************************************************************************************/ #include #include "command/backup/blockIncr.h" #include "command/restore/blockDelta.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/io/limitRead.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockDeltaSuperBlock { uint64_t superBlockSize; // Super block size uint64_t size; // Stored size of superblock (with compression, etc.) List *blockList; // Block list } BlockDeltaSuperBlock; typedef struct BlockDeltaBlock { uint64_t no; // Block number in the super block uint64_t offset; // Offset into original file uint8_t checksum[XX_HASH_SIZE_MAX]; // Checksum of the block } BlockDeltaBlock; struct BlockDelta { BlockDeltaPub pub; // Publicly accessible variables size_t blockSize; // Block size size_t checksumSize; // Checksum size CipherType cipherType; // Cipher type String *cipherPass; // Cipher passphrase CompressType compressType; // Compress type const BlockDeltaSuperBlock *superBlockData; // Current super block data unsigned int superBlockIdx; // Current super block index IoRead *limitRead; // Limit read for current super block const BlockDeltaBlock *blockData; // Current block data unsigned int blockIdx; // Current block index unsigned int blockTotal; // Block total for super block unsigned int blockFindIdx; // Index of the block to find in the super block BlockDeltaWrite write; // Block/offset to be returned for write }; /**********************************************************************************************************************************/ typedef struct BlockDeltaReference { unsigned int reference; // Reference List *blockList; // List of blocks in the block map for the reference } BlockDeltaReference; FN_EXTERN BlockDelta * blockDeltaNew( const BlockMap *const blockMap, const size_t blockSize, const size_t checksumSize, const Buffer *const blockChecksum, const CipherType cipherType, const String *const cipherPass, const CompressType compressType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_MAP, blockMap); FUNCTION_TEST_PARAM(SIZE, blockSize); FUNCTION_TEST_PARAM(SIZE, checksumSize); FUNCTION_TEST_PARAM(BUFFER, blockChecksum); FUNCTION_TEST_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_TEST_PARAM(ENUM, compressType); FUNCTION_TEST_END(); ASSERT(blockMap != NULL); ASSERT(blockSize > 0); ASSERT(cipherType == cipherTypeNone || cipherPass != NULL); OBJ_NEW_BEGIN(BlockDelta, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (BlockDelta) { .pub = { .readList = lstNewP(sizeof(BlockDeltaRead)), }, .blockSize = blockSize, .checksumSize = checksumSize, .cipherType = cipherType, .cipherPass = strDup(cipherPass), .compressType = compressType, .write = { .block = bufNew(blockSize), } }; MEM_CONTEXT_TEMP_BEGIN() { // Build list of references and for each reference the list of blocks for that reference const unsigned int blockChecksumSize = blockChecksum == NULL ? 0 : (unsigned int)(bufUsed(blockChecksum) / this->checksumSize); List *const referenceList = lstNewP(sizeof(BlockDeltaReference), .comparator = lstComparatorUInt); for (unsigned int blockMapIdx = 0; blockMapIdx < blockMapSize(blockMap); blockMapIdx++) { const BlockMapItem *const blockMapItem = blockMapGet(blockMap, blockMapIdx); // The block must be updated if it is beyond the blocks that exist in the block checksum list or when the checksum // stored in the repository is different from the block checksum list if (blockMapIdx >= blockChecksumSize || !bufEq( BUF(blockMapItem->checksum, this->checksumSize), BUF(bufPtrConst(blockChecksum) + blockMapIdx * this->checksumSize, this->checksumSize))) { const unsigned int reference = blockMapItem->reference; BlockDeltaReference *const referenceData = lstFind(referenceList, &reference); // If the reference has not been added if (referenceData == NULL) { const BlockDeltaReference *const referenceData = lstAdd( referenceList, &(BlockDeltaReference){.reference = reference, .blockList = lstNewP(sizeof(unsigned int))}); lstAdd(referenceData->blockList, &blockMapIdx); } // Else add the new block else lstAdd(referenceData->blockList, &blockMapIdx); } } // Sort the reference list descending. This is an arbitrary choice as the order does not matter. lstSort(referenceList, sortOrderDesc); // Build delta for (unsigned int referenceIdx = 0; referenceIdx < lstSize(referenceList); referenceIdx++) { const BlockDeltaReference *const referenceData = (const BlockDeltaReference *)lstGet(referenceList, referenceIdx); BlockDeltaRead *blockDeltaRead = NULL; const BlockDeltaSuperBlock *blockDeltaSuperBlock = NULL; const BlockMapItem *blockMapItemPrior = NULL; for (unsigned int blockIdx = 0; blockIdx < lstSize(referenceData->blockList); blockIdx++) { const unsigned int blockMapIdx = *(unsigned int *)lstGet(referenceData->blockList, blockIdx); const BlockMapItem *const blockMapItem = blockMapGet(blockMap, blockMapIdx); // Add read when it has changed if (blockMapItemPrior == NULL || (blockMapItemPrior->offset != blockMapItem->offset && blockMapItemPrior->offset + blockMapItemPrior->size != blockMapItem->offset)) { MEM_CONTEXT_OBJ_BEGIN(this->pub.readList) { const BlockDeltaRead blockDeltaReadNew = { .reference = blockMapItem->reference, .bundleId = blockMapItem->bundleId, .offset = blockMapItem->offset, .superBlockList = lstNewP(sizeof(BlockDeltaSuperBlock)), }; blockDeltaRead = lstAdd(this->pub.readList, &blockDeltaReadNew); } MEM_CONTEXT_OBJ_END(); } // Add super block when it has changed if (blockMapItemPrior == NULL || blockMapItemPrior->offset != blockMapItem->offset) { MEM_CONTEXT_OBJ_BEGIN(blockDeltaRead->superBlockList) { const BlockDeltaSuperBlock blockDeltaSuperBlockNew = { .superBlockSize = blockMapItem->superBlockSize, .size = blockMapItem->size, .blockList = lstNewP(sizeof(BlockDeltaBlock)), }; blockDeltaSuperBlock = lstAdd(blockDeltaRead->superBlockList, &blockDeltaSuperBlockNew); blockDeltaRead->size += blockMapItem->size; } MEM_CONTEXT_OBJ_END(); } // Add block BlockDeltaBlock blockDeltaBlockNew = { .no = blockMapItem->block, .offset = blockMapIdx * blockSize, }; memcpy(blockDeltaBlockNew.checksum, blockMapItem->checksum, SIZE_OF_STRUCT_MEMBER(BlockDeltaBlock, checksum)); lstAdd(blockDeltaSuperBlock->blockList, &blockDeltaBlockNew); // Set prior item for comparison on the next loop blockMapItemPrior = blockMapItem; } } } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(BLOCK_DELTA, this); } /**********************************************************************************************************************************/ FN_EXTERN const BlockDeltaWrite * blockDeltaNext(BlockDelta *const this, const BlockDeltaRead *const readDelta, IoRead *const readIo) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOCK_DELTA, this); FUNCTION_TEST_PARAM_P(VOID, readDelta); FUNCTION_TEST_PARAM(IO_READ, readIo); FUNCTION_TEST_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(readDelta != NULL); ASSERT(readIo != NULL); BlockDeltaWrite *result = NULL; // Iterate super blocks while (this->superBlockIdx < lstSize(readDelta->superBlockList)) { // If the super block read has not begun yet if (this->superBlockData == NULL) { // Free prior limit read and create limit read for current super block ioReadFree(this->limitRead); this->superBlockData = lstGet(readDelta->superBlockList, this->superBlockIdx); MEM_CONTEXT_OBJ_BEGIN(this) { this->limitRead = ioLimitReadNew(readIo, this->superBlockData->size); } MEM_CONTEXT_OBJ_END(); if (this->cipherType != cipherTypeNone) { ioFilterGroupAdd( ioReadFilterGroup(this->limitRead), cipherBlockNewP(cipherModeDecrypt, this->cipherType, BUFSTR(this->cipherPass), .raw = true)); } if (this->compressType != compressTypeNone) ioFilterGroupAdd(ioReadFilterGroup(this->limitRead), decompressFilterP(this->compressType, .raw = true)); ioReadOpen(this->limitRead); // Set block info this->blockIdx = 0; this->blockFindIdx = 0; this->blockTotal = (unsigned int)(this->superBlockData->superBlockSize / this->blockSize) + (this->superBlockData->superBlockSize % this->blockSize == 0 ? 0 : 1); this->blockData = lstGet(this->superBlockData->blockList, this->blockFindIdx); } // Find required blocks in the super block while (this->blockIdx < this->blockTotal) { // Clear buffer and read block bufUsedZero(this->write.block); bufLimitClear(this->write.block); ioRead(this->limitRead, this->write.block); // If the block matches the block we are expecting if (this->blockIdx == this->blockData->no) { ASSERT(result == NULL); this->write.offset = this->blockData->offset; result = &this->write; this->blockFindIdx++; // Get the next block if there are any more to read if (this->blockFindIdx < lstSize(this->superBlockData->blockList)) this->blockData = lstGet(this->superBlockData->blockList, this->blockFindIdx); } // Increment the block to read in the super block this->blockIdx++; // Break if there is a result if (result != NULL) break; } // Break if there is a result if (result != NULL) break; // Check that no bytes remain to be written. It is possible that some bytes remain in the super block, however, since we may // have gotten all the bytes we needed but just missed reading something important, e.g. an end of file marker. If we do not // read the remaining bytes then the next read will start too early. ioReadFlushP(this->limitRead, .errorOnBytes = true); this->superBlockData = NULL; this->superBlockIdx++; } // If no result then the super blocks have been read. Reset for the next read. if (result == NULL) { ASSERT(this->superBlockIdx == lstSize(readDelta->superBlockList)); ASSERT(this->blockIdx == this->blockTotal); this->superBlockData = NULL; this->superBlockIdx = 0; } FUNCTION_TEST_RETURN_TYPE_P(BlockDeltaWrite, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/blockDelta.h0000664000175100017510000001073415240547630021525 0ustar00runnerrunner/*********************************************************************************************************************************** Block Restore Calculate and return the blocks required to restore a file using an optional block checksum list. The block checksum list is optional because the file to restore may not exist so all the blocks will need to be restored. ***********************************************************************************************************************************/ #ifndef COMMAND_BACKUP_BLOCKDELTA_H #define COMMAND_BACKUP_BLOCKDELTA_H #include "command/backup/blockMap.h" #include "common/compress/helper.h" #include "common/crypto/common.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct BlockDelta BlockDelta; // Reads that must be performed in order to extract blocks typedef struct BlockDeltaRead { unsigned int reference; // Reference to read from uint64_t bundleId; // Bundle to read from uint64_t offset; // Offset to begin read from uint64_t size; // Size of the read List *superBlockList; // Super block list } BlockDeltaRead; // Writes that need to be performed to restore the file typedef struct BlockDeltaWrite { uint64_t offset; // Offset for the write Buffer *block; // Block to write } BlockDeltaWrite; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN BlockDelta *blockDeltaNew( const BlockMap *blockMap, size_t blockSize, size_t checksumSize, const Buffer *blockChecksum, CipherType cipherType, const String *cipherPass, const CompressType compressType); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get the next write for the restore FN_EXTERN const BlockDeltaWrite *blockDeltaNext(BlockDelta *this, const BlockDeltaRead *readDelta, IoRead *readIo); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct BlockDeltaPub { List *readList; // Read list } BlockDeltaPub; // Get read info FN_INLINE_ALWAYS const BlockDeltaRead * blockDeltaReadGet(const BlockDelta *const this, const unsigned int readIdx) { return (BlockDeltaRead *)lstGet(THIS_PUB(BlockDelta)->readList, readIdx); } // Read list size FN_INLINE_ALWAYS unsigned int blockDeltaReadSize(const BlockDelta *const this) { return lstSize(THIS_PUB(BlockDelta)->readList); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void blockDeltaFree(BlockMap *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_BLOCK_DELTA_TYPE \ BlockDelta * #define FUNCTION_LOG_BLOCK_DELTA_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "BlockDelta", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/clean.c.inc0000664000175100017510000010106015240547630021277 0ustar00runnerrunner/*********************************************************************************************************************************** Check ownership of items in the manifest ***********************************************************************************************************************************/ // Helper to determine what the user/group of a path/file/link should be static const String * restoreManifestOwnerReplace(const String *const owner, const String *const ownerDefaultRoot) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, owner); FUNCTION_TEST_PARAM(STRING, ownerDefaultRoot); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, userRoot() ? (owner == NULL ? ownerDefaultRoot : owner) : NULL); } // Helper to get list of owners from a file/link/path list #define RESTORE_MANIFEST_OWNER_GET(type, deref) \ for (unsigned int itemIdx = 0; itemIdx < manifest##type##Total(manifest); itemIdx++) \ { \ const Manifest##type item = deref manifest##type(manifest, itemIdx); \ \ if (item.user == NULL) \ userNull = true; \ else \ strLstAddIfMissing(userList, item.user); \ \ if (item.group == NULL) \ groupNull = true; \ else \ strLstAddIfMissing(groupList, item.group); \ } // Helper to warn when an owner is missing and must be remapped #define RESTORE_MANIFEST_OWNER_WARN(type) \ do \ { \ if (type##Null) \ LOG_WARN("unknown " #type " in backup manifest mapped to current " #type); \ \ for (unsigned int ownerIdx = 0; ownerIdx < strLstSize(type##List); ownerIdx++) \ { \ const String *const owner = strLstGet(type##List, ownerIdx); \ \ if (type##Name() == NULL || !strEq(type##Name(), owner)) \ LOG_WARN_FMT("unknown " #type " '%s' in backup manifest mapped to current " #type, strZ(owner)); \ } \ } \ while (0) static void restoreManifestOwner(const Manifest *const manifest, const String **const rootReplaceUser, const String **const rootReplaceGroup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM_P(VOID, rootReplaceUser); FUNCTION_LOG_PARAM_P(VOID, rootReplaceGroup); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build a list of users and groups in the manifest // ------------------------------------------------------------------------------------------------------------------------- bool userNull = false; StringList *const userList = strLstNew(); bool groupNull = false; StringList *const groupList = strLstNew(); RESTORE_MANIFEST_OWNER_GET(File, ); RESTORE_MANIFEST_OWNER_GET(Link, *(const ManifestLink *)); RESTORE_MANIFEST_OWNER_GET(Path, *(const ManifestPath *)); // Update users and groups in the manifest (this can only be done as root) // ------------------------------------------------------------------------------------------------------------------------- if (userRoot()) { // Get user/group info from data directory to use for invalid user/groups StorageInfo pathInfo = storageInfoP(storagePg(), manifestTargetBase(manifest)->path, .ignoreMissing = true); // If user/group is null then set it to root if (pathInfo.user == NULL) // {vm_covered} pathInfo.user = userName(); // {vm_covered} if (pathInfo.group == NULL) // {vm_covered} pathInfo.group = groupName(); // {vm_covered} if (userNull || groupNull) { if (userNull) LOG_WARN_FMT("unknown user in backup manifest mapped to '%s'", strZ(pathInfo.user)); if (groupNull) LOG_WARN_FMT("unknown group in backup manifest mapped to '%s'", strZ(pathInfo.group)); MEM_CONTEXT_PRIOR_BEGIN() { *rootReplaceUser = strDup(pathInfo.user); *rootReplaceGroup = strDup(pathInfo.group); } MEM_CONTEXT_PRIOR_END(); } } // Else set owners to NULL. This means we won't make any attempt to update ownership and will just leave it as written by // the current user/group. If there are existing files that are not owned by the current user/group then we will attempt to // update them, which will generally cause an error, though some systems allow updates to the group ownership. // ------------------------------------------------------------------------------------------------------------------------- else { RESTORE_MANIFEST_OWNER_WARN(user); RESTORE_MANIFEST_OWNER_WARN(group); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Clean the data directory of any paths/files/links that are not in the manifest and create missing links/paths ***********************************************************************************************************************************/ typedef struct RestoreCleanCallbackData { const Manifest *manifest; // Manifest to compare against const ManifestTarget *target; // Current target being compared const String *targetName; // Name to use when finding files/paths/links const String *targetPath; // Path of target currently being compared const String *subPath; // Subpath in target currently being compared bool basePath; // Is this the base path? bool exists; // Does the target path exist? bool delta; // Is this a delta restore? StringList *fileIgnore; // Files to ignore during clean const String *rootReplaceUser; // User to replace invalid users when root const String *rootReplaceGroup; // Group to replace invalid group when root } RestoreCleanCallbackData; // Helper to update ownership on a file/link/path static void restoreCleanOwnership( const String *const pgPath, const String *manifestUserName, const String *const rootReplaceUser, const String *manifestGroupName, const String *const rootReplaceGroup, const uid_t actualUserId, const gid_t actualGroupId, const bool new) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, pgPath); FUNCTION_TEST_PARAM(STRING, manifestUserName); FUNCTION_TEST_PARAM(STRING, manifestGroupName); FUNCTION_TEST_PARAM(UINT, actualUserId); FUNCTION_TEST_PARAM(UINT, actualGroupId); FUNCTION_TEST_PARAM(BOOL, new); FUNCTION_TEST_END(); ASSERT(pgPath != NULL); // Get the expected user id uid_t expectedUserId = userId(); manifestUserName = restoreManifestOwnerReplace(manifestUserName, rootReplaceUser); if (manifestUserName != NULL) { const uid_t manifestUserId = userIdFromName(manifestUserName); if (manifestUserId != (uid_t)-1) expectedUserId = manifestUserId; } // Get the expected group id gid_t expectedGroupId = groupId(); manifestGroupName = restoreManifestOwnerReplace(manifestGroupName, rootReplaceGroup); if (manifestGroupName != NULL) { const uid_t manifestGroupId = groupIdFromName(manifestGroupName); if (manifestGroupId != (uid_t)-1) expectedGroupId = manifestGroupId; } // Update ownership if not as expected if (actualUserId != expectedUserId || actualGroupId != expectedGroupId) { // If this is a newly created file/link/path then there's no need to log updated permissions if (!new) LOG_DETAIL_FMT("update ownership for '%s'", strZ(pgPath)); THROW_ON_SYS_ERROR_FMT( lchown(strZ(pgPath), expectedUserId, expectedGroupId) == -1, FileOwnerError, "unable to set ownership for '%s'", strZ(pgPath)); } FUNCTION_TEST_RETURN_VOID(); } // Helper to update mode on a file/path static void restoreCleanMode(const String *const pgPath, const mode_t manifestMode, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, pgPath); FUNCTION_TEST_PARAM(MODE, manifestMode); FUNCTION_TEST_PARAM(INFO, info); FUNCTION_TEST_END(); ASSERT(pgPath != NULL); ASSERT(info != NULL); // Update mode if not as expected if (manifestMode != info->mode) { LOG_DETAIL_FMT("update mode for '%s' to %04o", strZ(pgPath), manifestMode); THROW_ON_SYS_ERROR_FMT( chmod(strZ(pgPath), manifestMode) == -1, FileModeError, "unable to set mode for '%s'", strZ(pgPath)); } FUNCTION_TEST_RETURN_VOID(); } // Recurse paths static void restoreCleanBuildRecurse(StorageIterator *const storageItr, const RestoreCleanCallbackData *const cleanData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_ITERATOR, storageItr); FUNCTION_TEST_PARAM_P(VOID, cleanData); FUNCTION_TEST_END(); ASSERT(storageItr != NULL); ASSERT(cleanData != NULL); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); // Don't include backup.manifest or recovery.conf (when preserved) in the comparison or empty directory check if (cleanData->basePath && info.type == storageTypeFile && strLstExists(cleanData->fileIgnore, info.name)) continue; // If this is not a delta then error because the directory is expected to be empty. Ignore the . path. if (!cleanData->delta) { THROW_FMT( PathNotEmptyError, "unable to restore to path '%s' because it contains files\n" "HINT: try using --delta if this is what you intended.", strZ(cleanData->targetPath)); } // Construct the name used to find this file/link/path in the manifest const String *const manifestName = strNewFmt("%s/%s", strZ(cleanData->targetName), strZ(info.name)); // Construct the path of this file/link/path in the PostgreSQL data directory const String *const pgPath = strNewFmt("%s/%s", strZ(cleanData->targetPath), strZ(info.name)); switch (info.type) { case storageTypeFile: { if (manifestFileExists(cleanData->manifest, manifestName) && manifestLinkFindDefault(cleanData->manifest, manifestName, NULL) == NULL) { const ManifestFile manifestFile = manifestFileFind(cleanData->manifest, manifestName); restoreCleanOwnership( pgPath, manifestFile.user, cleanData->rootReplaceUser, manifestFile.group, cleanData->rootReplaceGroup, info.userId, info.groupId, false); restoreCleanMode(pgPath, manifestFile.mode, &info); } else { LOG_DETAIL_FMT("remove invalid file '%s'", strZ(pgPath)); storageRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true); } break; } case storageTypeLink: { const ManifestLink *const manifestLink = manifestLinkFindDefault(cleanData->manifest, manifestName, NULL); if (manifestLink != NULL) { if (!strEq(manifestLink->destination, info.linkDestination)) { LOG_DETAIL_FMT("remove link '%s' because destination changed", strZ(pgPath)); storageRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true); } else { restoreCleanOwnership( pgPath, manifestLink->user, cleanData->rootReplaceUser, manifestLink->group, cleanData->rootReplaceGroup, info.userId, info.groupId, false); } } else { LOG_DETAIL_FMT("remove invalid link '%s'", strZ(pgPath)); storageRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true); } break; } case storageTypePath: { const ManifestPath *const manifestPath = manifestPathFindDefault(cleanData->manifest, manifestName, NULL); if (manifestPath != NULL && manifestLinkFindDefault(cleanData->manifest, manifestName, NULL) == NULL) { // Check ownership/permissions restoreCleanOwnership( pgPath, manifestPath->user, cleanData->rootReplaceUser, manifestPath->group, cleanData->rootReplaceGroup, info.userId, info.groupId, false); restoreCleanMode(pgPath, manifestPath->mode, &info); // Recurse into the path RestoreCleanCallbackData cleanDataSub = *cleanData; cleanDataSub.targetName = strNewFmt("%s/%s", strZ(cleanData->targetName), strZ(info.name)); cleanDataSub.targetPath = strNewFmt("%s/%s", strZ(cleanData->targetPath), strZ(info.name)); cleanDataSub.basePath = false; restoreCleanBuildRecurse( storageNewItrP( storageLocalWrite(), cleanDataSub.targetPath, .errorOnMissing = true, .sortOrder = sortOrderAsc), &cleanDataSub); } else { LOG_DETAIL_FMT("remove invalid path '%s'", strZ(pgPath)); storagePathRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true, .recurse = true); } break; } // Special file types cannot exist in the manifest so just delete them case storageTypeSpecial: LOG_DETAIL_FMT("remove special file '%s'", strZ(pgPath)); storageRemoveP(storageLocalWrite(), pgPath, .errorOnMissing = true); break; } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } static void restoreCleanBuild(const Manifest *const manifest, const String *const rootReplaceUser, const String *const rootReplaceGroup) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STRING, rootReplaceUser); FUNCTION_LOG_PARAM(STRING, rootReplaceGroup); FUNCTION_LOG_END(); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Is this a delta restore? const bool delta = cfgOptionBool(cfgOptDelta) || cfgOptionBool(cfgOptForce); // Allocate data for each target RestoreCleanCallbackData *const cleanDataList = memNew(sizeof(RestoreCleanCallbackData) * manifestTargetTotal(manifest)); // Step 1: Check permissions and validity (is the directory empty without delta?) if the target directory exists // ------------------------------------------------------------------------------------------------------------------------- StringList *const pathChecked = strLstNew(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { RestoreCleanCallbackData *const cleanData = &cleanDataList[targetIdx]; *cleanData = (RestoreCleanCallbackData) { .manifest = manifest, .target = manifestTarget(manifest, targetIdx), .delta = delta, .fileIgnore = strLstNew(), .rootReplaceUser = rootReplaceUser, .rootReplaceGroup = rootReplaceGroup, }; cleanData->targetName = cleanData->target->name; cleanData->targetPath = manifestTargetPath(manifest, cleanData->target); cleanData->basePath = strEq(cleanData->targetName, MANIFEST_TARGET_PGDATA_STR); // Ignore backup.manifest while cleaning since it may exist from an prior incomplete restore strLstAdd(cleanData->fileIgnore, BACKUP_MANIFEST_FILE_STR); // Also ignore recovery files when recovery type = preserve if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_PRESERVE) { // If recovery GUCs then three files must be preserved if (manifestData(manifest)->pgVersion >= PG_VERSION_RECOVERY_GUC) { strLstAdd(cleanData->fileIgnore, PG_FILE_POSTGRESQLAUTOCONF_STR); strLstAdd(cleanData->fileIgnore, PG_FILE_RECOVERYSIGNAL_STR); strLstAdd(cleanData->fileIgnore, PG_FILE_STANDBYSIGNAL_STR); } // Else just recovery.conf else strLstAdd(cleanData->fileIgnore, PG_FILE_RECOVERYCONF_STR); } // If this is a tablespace append the tablespace identifier if (cleanData->target->type == manifestTargetTypeLink && cleanData->target->tablespaceId != 0) { const String *const tablespaceId = pgTablespaceId( manifestData(manifest)->pgVersion, manifestData(manifest)->pgCatalogVersion); cleanData->targetName = strNewFmt("%s/%s", strZ(cleanData->targetName), strZ(tablespaceId)); cleanData->targetPath = strNewFmt("%s/%s", strZ(cleanData->targetPath), strZ(tablespaceId)); } strLstSort(cleanData->fileIgnore, sortOrderAsc); // Check that the path exists. If not, there's no need to do any cleaning and we'll attempt to create it later. Don't // log check for the same path twice. There can be multiple links to files in the same path, but logging it more than // once makes the logs noisy and looks like a bug. if (!strLstExists(pathChecked, cleanData->targetPath)) LOG_DETAIL_FMT("check '%s' exists", strZ(cleanData->targetPath)); const StorageInfo info = storageInfoP(storageLocal(), cleanData->targetPath, .ignoreMissing = true, .followLink = true); strLstAdd(pathChecked, cleanData->targetPath); if (info.exists) { // Make sure our uid will be able to write to this directory if (!userRoot() && userId() != info.userId) { THROW_FMT( PathOpenError, "unable to restore to path '%s' not owned by current user", strZ(cleanData->targetPath)); } if ((info.mode & 0700) != 0700) { THROW_FMT( PathOpenError, "unable to restore to path '%s' without rwx permissions", strZ(cleanData->targetPath)); } // If not a delta restore then check that the directories are empty, or if a file link, that the file doesn't exist if (!cleanData->delta) { if (cleanData->target->file == NULL) { restoreCleanBuildRecurse( storageNewItrP(storageLocal(), cleanData->targetPath, .errorOnMissing = true), cleanData); } else { const String *const file = strNewFmt("%s/%s", strZ(cleanData->targetPath), strZ(cleanData->target->file)); if (storageExistsP(storageLocal(), file)) { THROW_FMT( FileExistsError, "unable to restore file '%s' because it already exists\n" "HINT: try using --delta if this is what you intended.", strZ(file)); } } // Now that we know there are no files in this target enable delta for processing in step 2 cleanData->delta = true; } // The target directory exists and is valid and will need to be cleaned cleanData->exists = true; } } // Skip the tablespace_map file when present so PostgreSQL does not rewrite links in pg_tblspc. The tablespace links will be // created after paths are cleaned. if (manifestFileExists(manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_FILE_TABLESPACEMAP))) { LOG_DETAIL_FMT("skip '" PG_FILE_TABLESPACEMAP "' -- tablespace links will be created based on mappings"); manifestFileRemove(manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_FILE_TABLESPACEMAP)); } // Skip postgresql.auto.conf if preserve is set and the PostgreSQL version supports recovery GUCs if (manifestData(manifest)->pgVersion >= PG_VERSION_RECOVERY_GUC && cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_PRESERVE && manifestFileExists(manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_FILE_POSTGRESQLAUTOCONF))) { LOG_DETAIL_FMT("skip '" PG_FILE_POSTGRESQLAUTOCONF "' -- recovery type is preserve"); manifestFileRemove(manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_FILE_POSTGRESQLAUTOCONF)); } // Step 2: Clean target directories // ------------------------------------------------------------------------------------------------------------------------- // Delete the pg_control file (if it exists) so the cluster cannot be started if restore does not complete. Sync the path so // the file does not return, zombie-like, in the case of a host crash. if (storageExistsP(storagePg(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL))) { LOG_DETAIL_FMT( "remove '" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "' so cluster will not start if restore does not complete"); storageRemoveP(storagePgWrite(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)); storagePathSyncP(storagePgWrite(), PG_PATH_GLOBAL_STR); } for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const RestoreCleanCallbackData *const cleanData = &cleanDataList[targetIdx]; // Only clean if the target exists if (cleanData->exists) { // Don't clean file links. It doesn't matter whether the file exists or not since we know it is in the manifest. if (cleanData->target->file == NULL) { // Only log when doing a delta restore because otherwise the targets should be empty. We'll still run the clean // to fix permissions/ownership on the target paths. if (delta) LOG_INFO_FMT("remove invalid files/links/paths from '%s'", strZ(cleanData->targetPath)); // Check target ownership/permissions const ManifestPath *const manifestPath = manifestPathFind(cleanData->manifest, cleanData->targetName); const StorageInfo info = storageInfoP(storageLocal(), cleanData->targetPath, .followLink = true); restoreCleanOwnership( cleanData->targetPath, manifestPath->user, rootReplaceUser, manifestPath->group, rootReplaceGroup, info.userId, info.groupId, false); restoreCleanMode(cleanData->targetPath, manifestPath->mode, &info); // Clean the target restoreCleanBuildRecurse( storageNewItrP( storageLocalWrite(), cleanData->targetPath, .errorOnMissing = true, .sortOrder = sortOrderAsc), cleanData); } } // If the target does not exist we'll attempt to create it else { const ManifestPath *path = NULL; // There is no path information for a file link so we'll need to use the data directory if (cleanData->target->file != NULL) { path = manifestPathFind(manifest, MANIFEST_TARGET_PGDATA_STR); } // Else grab the info for the path that matches the link name else path = manifestPathFind(manifest, cleanData->target->name); storagePathCreateP(storageLocalWrite(), cleanData->targetPath, .mode = path->mode); restoreCleanOwnership( cleanData->targetPath, path->user, rootReplaceUser, path->group, rootReplaceGroup, userId(), groupId(), true); } } // Step 3: Create missing paths and path links // ------------------------------------------------------------------------------------------------------------------------- for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(manifest); pathIdx++) { const ManifestPath *const path = manifestPath(manifest, pathIdx); // Skip the pg_tblspc path because it only maps to the manifest. We should remove this in a future release but not much // can be done about it for now. if (strEq(path->name, MANIFEST_TARGET_PGTBLSPC_STR)) continue; // If this path has been mapped as a link then create a link. The path has already been created as part of target // creation (or it might have already existed). const ManifestLink *const link = manifestLinkFindDefault( manifest, strBeginsWith(path->name, MANIFEST_TARGET_PGTBLSPC_STR) ? strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(path->name)) : path->name, NULL); if (link != NULL) { const String *const pgPath = storagePathP(storagePg(), manifestPathPg(link->name)); const StorageInfo linkInfo = storageInfoP(storagePg(), pgPath, .ignoreMissing = true); // Create the link if it is missing. If it exists it should already have the correct ownership and destination. if (!linkInfo.exists) { LOG_DETAIL_FMT("create symlink '%s' to '%s'", strZ(pgPath), strZ(link->destination)); storageLinkCreateP(storagePgWrite(), link->destination, pgPath); restoreCleanOwnership( pgPath, link->user, rootReplaceUser, link->group, rootReplaceGroup, userId(), groupId(), true); } } // Create the path normally else { const String *const pgPath = storagePathP(storagePg(), manifestPathPg(path->name)); const StorageInfo pathInfo = storageInfoP(storagePg(), pgPath, .ignoreMissing = true); // Create the path if it is missing. If it exists it should already have the correct ownership and mode. if (!pathInfo.exists) { LOG_DETAIL_FMT("create path '%s'", strZ(pgPath)); storagePathCreateP(storagePgWrite(), pgPath, .mode = path->mode, .noParentCreate = true, .errorOnExists = true); restoreCleanOwnership( storagePathP(storagePg(), pgPath), path->user, rootReplaceUser, path->group, rootReplaceGroup, userId(), groupId(), true); } } } // Step 4: Create file links. These don't get created during path creation because they do not have a matching path entry. // ------------------------------------------------------------------------------------------------------------------------- for (unsigned int linkIdx = 0; linkIdx < manifestLinkTotal(manifest); linkIdx++) { const ManifestLink *const link = manifestLink(manifest, linkIdx); const String *const pgPath = storagePathP(storagePg(), manifestPathPg(link->name)); const StorageInfo linkInfo = storageInfoP(storagePg(), pgPath, .ignoreMissing = true); // Create the link if it is missing. If it exists it should already have the correct ownership and destination. if (!linkInfo.exists) { LOG_DETAIL_FMT("create symlink '%s' to '%s'", strZ(pgPath), strZ(link->destination)); storageLinkCreateP(storagePgWrite(), link->destination, pgPath); restoreCleanOwnership( pgPath, link->user, rootReplaceUser, link->group, rootReplaceGroup, userId(), groupId(), true); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/config.c.inc0000664000175100017510000004525215240547630021474 0ustar00runnerrunner/*********************************************************************************************************************************** Recovery constants ***********************************************************************************************************************************/ #define RESTORE_COMMAND "restore_command" STRING_STATIC(RESTORE_COMMAND_STR, RESTORE_COMMAND); #define RECOVERY_TARGET "recovery_target" #define RECOVERY_TARGET_LSN "recovery_target_lsn" #define RECOVERY_TARGET_NAME "recovery_target_name" #define RECOVERY_TARGET_TIME "recovery_target_time" #define RECOVERY_TARGET_XID "recovery_target_xid" #define RECOVERY_TARGET_ACTION "recovery_target_action" #define RECOVERY_TARGET_INCLUSIVE "recovery_target_inclusive" #define RECOVERY_TARGET_TIMELINE "recovery_target_timeline" #define RECOVERY_TARGET_TIMELINE_CURRENT "current" #define STANDBY_MODE "standby_mode" STRING_STATIC(STANDBY_MODE_STR, STANDBY_MODE); #define ARCHIVE_MODE "archive_mode" /*********************************************************************************************************************************** Generate the recovery file ***********************************************************************************************************************************/ // Helper to generate recovery options static KeyValue * restoreRecoveryOption(const unsigned int pgVersion) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_END(); KeyValue *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { result = kvNew(); StringList *recoveryOptionKey = strLstNew(); if (cfgOptionTest(cfgOptRecoveryOption)) { const KeyValue *const recoveryOption = cfgOptionKv(cfgOptRecoveryOption); recoveryOptionKey = strLstSort(strLstNewVarLst(kvKeyList(recoveryOption)), sortOrderAsc); for (unsigned int keyIdx = 0; keyIdx < strLstSize(recoveryOptionKey); keyIdx++) { // Get the key and value String *const key = strLstGet(recoveryOptionKey, keyIdx); const String *const value = varStr(kvGet(recoveryOption, VARSTR(key))); // Replace - in key with _. Since we use - users naturally will as well. strReplaceChr(key, '-', '_'); kvPut(result, VARSTR(key), VARSTR(value)); } strLstSort(recoveryOptionKey, sortOrderAsc); } // If archive-mode is not preserve if (cfgOptionSeq(cfgOptArchiveMode) != CFGOPTVAL_ARCHIVE_MODE_PRESERVE) { if (pgVersion < PG_VERSION_12) { THROW_FMT( OptionInvalidError, "option '" CFGOPT_ARCHIVE_MODE "' is not supported on " PG_NAME " < " PG_VERSION_12_Z "\n" "HINT: 'archive_mode' should be manually set to 'off' in postgresql.conf."); } // The only other valid option is off ASSERT(cfgOptionSeq(cfgOptArchiveMode) == CFGOPTVAL_ARCHIVE_MODE_OFF); // If archive-mode=off then set archive_mode=off kvPut(result, VARSTRDEF(ARCHIVE_MODE), VARSTRDEF(CFGOPTVAL_ARCHIVE_MODE_OFF_Z)); } // Write restore_command if (!strLstExists(recoveryOptionKey, RESTORE_COMMAND_STR)) { // Null out options that it does not make sense to pass from the restore command to archive-get. All of these have // reasonable defaults so there is no danger of an error -- they just might not be optimal. In any case, it seems better // than, for example, passing --process-max=32 to archive-get because it was specified for restore. KeyValue *const optionReplace = kvNew(); kvPut(optionReplace, VARSTRDEF(CFGOPT_EXEC_ID), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_JOB_RETRY), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_JOB_RETRY_INTERVAL), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_FILE), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_SUBPROCESS), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_TIMESTAMP), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_PROCESS_MAX), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_CMD), NULL); kvPut( result, VARSTRDEF(RESTORE_COMMAND), VARSTR( strNewFmt( "%s %s %%f \"%%p\"", strZ(cfgOptionStr(cfgOptCmd)), strZ(strLstJoin(cfgExecParam(cfgCmdArchiveGet, cfgCmdRoleMain, optionReplace, true, true), " "))))); } // If recovery type is immediate if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_IMMEDIATE) { kvPut(result, VARSTRDEF(RECOVERY_TARGET), VARSTRDEF(CFGOPTVAL_RESTORE_TYPE_IMMEDIATE_Z)); } // Else recovery type is standby else if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_STANDBY) { // Write standby_mode for PostgreSQL versions that support it if (pgVersion < PG_VERSION_RECOVERY_GUC) kvPut(result, VARSTR(STANDBY_MODE_STR), VARSTRDEF("on")); } // Else recovery type is not default so write target options else if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_DEFAULT) { // Write the recovery target kvPut( result, VARSTR(strNewFmt(RECOVERY_TARGET "_%s", strZ(cfgOptionDisplay(cfgOptType)))), VARSTR(cfgOptionStr(cfgOptTarget))); // Write recovery_target_inclusive if (cfgOptionTest(cfgOptTargetExclusive) && cfgOptionBool(cfgOptTargetExclusive)) kvPut(result, VARSTRDEF(RECOVERY_TARGET_INCLUSIVE), VARSTR(FALSE_STR)); } // Write recovery_target_action if (cfgOptionTest(cfgOptTargetAction)) { const StringId targetAction = cfgOptionStrId(cfgOptTargetAction); if (targetAction != CFGOPTVAL_TARGET_ACTION_PAUSE) { kvPut(result, VARSTRDEF(RECOVERY_TARGET_ACTION), VARSTR(strNewStrId(targetAction))); } } // Write recovery_target_timeline if set if (cfgOptionTest(cfgOptTargetTimeline)) { // Do not set current when PostgreSQL < 12 since this is the default and if current is explicitly set it acts as latest if (pgVersion >= PG_VERSION_12 || !strEqZ(cfgOptionStr(cfgOptTargetTimeline), RECOVERY_TARGET_TIMELINE_CURRENT)) kvPut(result, VARSTRDEF(RECOVERY_TARGET_TIMELINE), VARSTR(cfgOptionStr(cfgOptTargetTimeline))); } // Else explicitly set target timeline to "current" when type=immediate and PostgreSQL >= 12. We do this because // type=immediate means there won't be any actual attempt to change timelines, but if we leave the target timeline as the // default of "latest" then PostgreSQL might fail to restore because it can't reach the "latest" timeline in the repository // from this backup. // // This is really a PostgreSQL bug and will hopefully be addressed there, but we'll handle it here for older versions, at // least until they aren't really seen in the wild any longer. // // PostgreSQL < 12 defaults to "current" (but does not accept "current" as a parameter) so no need set it explicitly. else if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_IMMEDIATE && pgVersion >= PG_VERSION_12) kvPut(result, VARSTRDEF(RECOVERY_TARGET_TIMELINE), VARSTRDEF(RECOVERY_TARGET_TIMELINE_CURRENT)); // Move to prior context kvMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(KEY_VALUE, result); } // Helper to convert recovery options to text format static String * restoreRecoveryConf(const unsigned int pgVersion, const String *const restoreLabel) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(STRING, restoreLabel); FUNCTION_LOG_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { strCatFmt(result, "# Recovery settings generated by " PROJECT_NAME " restore on %s\n", strZ(restoreLabel)); // Output all recovery options const KeyValue *const optionKv = restoreRecoveryOption(pgVersion); const VariantList *const optionKeyList = kvKeyList(optionKv); for (unsigned int optionKeyIdx = 0; optionKeyIdx < varLstSize(optionKeyList); optionKeyIdx++) { const Variant *const optionKey = varLstGet(optionKeyList, optionKeyIdx); strCatFmt(result, "%s = '%s'\n", strZ(varStr(optionKey)), strZ(varStr(kvGet(optionKv, optionKey)))); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } // Helper to write recovery options into recovery.conf static void restoreRecoveryWriteConf( const Manifest *const manifest, const StorageInfo *const fileInfo, const unsigned int pgVersion, const String *const restoreLabel) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STORAGE_INFO, fileInfo); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(STRING, restoreLabel); FUNCTION_LOG_END(); // Only write recovery.conf if recovery type != none if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_NONE) { MEM_CONTEXT_TEMP_BEGIN() { LOG_INFO_FMT("write %s", strZ(storagePathP(storagePg(), PG_FILE_RECOVERYCONF_STR))); // Write recovery.conf storagePutP( storageNewWriteP( storagePgWrite(), PG_FILE_RECOVERYCONF_STR, .noCreatePath = true, .modeFile = fileInfo->mode, .noAtomic = true, .noSyncPath = true, .user = fileInfo->user, .group = fileInfo->group), BUFSTR(restoreRecoveryConf(pgVersion, restoreLabel))); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } // Helper to write recovery options into postgresql.auto.conf static void restoreRecoveryWriteAutoConf( const Manifest *const manifest, const StorageInfo *const fileInfo, const unsigned int pgVersion, const String *const restoreLabel) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STORAGE_INFO, fileInfo); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(STRING, restoreLabel); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { String *const content = strNew(); // Load postgresql.auto.conf so we can preserve the existing contents const Buffer *const autoConf = storageGetP( storageNewReadP(storagePg(), PG_FILE_POSTGRESQLAUTOCONF_STR, .ignoreMissing = true)); // It is unusual for the file not to exist, but we'll continue processing by creating a blank file if (autoConf == NULL) { LOG_WARN(PG_FILE_POSTGRESQLAUTOCONF " does not exist -- creating to contain recovery settings"); } // Else the file does exist so comment out old recovery options that could interfere with the current recovery. Don't // comment out *all* recovery options because some should only be commented out if there is a new option to replace it, e.g. // primary_conninfo. If the option shouldn't be commented out all the time then it won't ever be commented out -- this may // not be ideal but it is what was decided. PostgreSQL will use the last value set so this is safe as long as the option // does not have dependencies on other options. else { // Generate a regexp that will match on all current recovery_target settings RegExp *const recoveryExp = regExpNew( STRDEF( "^[\t ]*(" RECOVERY_TARGET "|" RECOVERY_TARGET_ACTION "|" RECOVERY_TARGET_INCLUSIVE "|" RECOVERY_TARGET_LSN "|" RECOVERY_TARGET_NAME "|" RECOVERY_TARGET_TIME "|" RECOVERY_TARGET_TIMELINE "|" RECOVERY_TARGET_XID ")[\t ]*=")); // Check each line for recovery settings const StringList *const contentList = strLstNewSplit(strNewBuf(autoConf), LF_STR); for (unsigned int contentIdx = 0; contentIdx < strLstSize(contentList); contentIdx++) { if (contentIdx != 0) strCat(content, LF_STR); const String *const line = strLstGet(contentList, contentIdx); if (regExpMatch(recoveryExp, line)) strCatFmt(content, "# Removed by " PROJECT_NAME " restore on %s # ", strZ(restoreLabel)); strCat(content, line); } // If settings will be appended then format the file so a blank line will be between old and new settings if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_NONE) { strTrim(content); strCatZ(content, "\n\n"); } } // If recovery was requested then write the recovery options if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_NONE) { // If the user specified standby_mode as a recovery option then error. It's tempting to just set type=standby in this // case but since config parsing has already happened the target options could be in an invalid state. if (cfgOptionTest(cfgOptRecoveryOption)) { const KeyValue *const recoveryOption = cfgOptionKv(cfgOptRecoveryOption); const StringList *const recoveryOptionKey = strLstNewVarLst(kvKeyList(recoveryOption)); for (unsigned int keyIdx = 0; keyIdx < strLstSize(recoveryOptionKey); keyIdx++) { // Get the key and value String *const key = strLstGet(recoveryOptionKey, keyIdx); // Replace - in key with _. Since we use - users naturally will as well. strReplaceChr(key, '-', '_'); if (strEq(key, STANDBY_MODE_STR)) { THROW_FMT( OptionInvalidError, "'" STANDBY_MODE "' setting is not valid for " PG_NAME " >= %s\n" "HINT: use --" CFGOPT_TYPE "=" CFGOPTVAL_RESTORE_TYPE_STANDBY_Z " instead of --" CFGOPT_RECOVERY_OPTION "=" STANDBY_MODE "=on.", strZ(pgVersionToStr(PG_VERSION_RECOVERY_GUC))); } } } strCat(content, restoreRecoveryConf(pgVersion, restoreLabel)); } LOG_INFO_FMT( "write %s%s", autoConf == NULL ? "" : "updated ", strZ(storagePathP(storagePg(), PG_FILE_POSTGRESQLAUTOCONF_STR))); // Write postgresql.auto.conf storagePutP( storageNewWriteP( storagePgWrite(), PG_FILE_POSTGRESQLAUTOCONF_STR, .noCreatePath = true, .modeFile = fileInfo->mode, .noAtomic = true, .noSyncPath = true, .user = fileInfo->user, .group = fileInfo->group), BUFSTR(content)); // The standby.signal file is required for standby mode if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_STANDBY) { storagePutP( storageNewWriteP( storagePgWrite(), PG_FILE_STANDBYSIGNAL_STR, .noCreatePath = true, .modeFile = fileInfo->mode, .noAtomic = true, .noSyncPath = true, .user = fileInfo->user, .group = fileInfo->group), NULL); } // Else the recovery.signal file is required for targeted recovery. Skip writing this file if the backup was offline and // recovery type is none since PostgreSQL will error in this case when wal_level=minimal. else if (cfgOptionSeq(cfgOptType) != CFGOPTVAL_RESTORE_TYPE_NONE) { storagePutP( storageNewWriteP( storagePgWrite(), PG_FILE_RECOVERYSIGNAL_STR, .noCreatePath = true, .modeFile = fileInfo->mode, .noAtomic = true, .noSyncPath = true, .user = fileInfo->user, .group = fileInfo->group), NULL); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } static void restoreRecoveryWrite(const Manifest *const manifest, const StorageInfo *const fileInfo) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STORAGE_INFO, fileInfo); FUNCTION_LOG_END(); // Get PostgreSQL version to write recovery for const unsigned int pgVersion = manifestData(manifest)->pgVersion; MEM_CONTEXT_TEMP_BEGIN() { // If recovery type is preserve then leave recovery file as it is if (cfgOptionSeq(cfgOptType) == CFGOPTVAL_RESTORE_TYPE_PRESERVE) { // Determine which file recovery settings will be written to const String *const recoveryFile = pgVersion >= PG_VERSION_RECOVERY_GUC ? PG_FILE_POSTGRESQLAUTOCONF_STR : PG_FILE_RECOVERYCONF_STR; if (!storageExistsP(storagePg(), recoveryFile)) { LOG_WARN_FMT( "recovery type is " CFGOPTVAL_RESTORE_TYPE_PRESERVE_Z " but recovery file does not exist at '%s'", strZ(storagePathP(storagePg(), recoveryFile))); } } // Else write recovery file else { // Generate a label used to identify this restore in the recovery file const String *const restoreLabel = strNewTimeP("%Y-%m-%d %H:%M:%S", time(NULL)); // Write recovery file based on PostgreSQL version if (pgVersion >= PG_VERSION_RECOVERY_GUC) restoreRecoveryWriteAutoConf(manifest, fileInfo, pgVersion, restoreLabel); else restoreRecoveryWriteConf(manifest, fileInfo, pgVersion, restoreLabel); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/file.c0000664000175100017510000005367215240547630020403 0ustar00runnerrunner/*********************************************************************************************************************************** Restore File ***********************************************************************************************************************************/ #include #include #include #include #include "command/backup/blockIncr.h" #include "command/backup/blockMap.h" #include "command/restore/blockChecksum.h" #include "command/restore/blockDelta.h" #include "command/restore/file.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/io/filter/group.h" #include "common/io/filter/size.h" #include "common/io/io.h" #include "common/io/limitRead.h" #include "common/log.h" #include "config/config.h" #include "info/manifest/manifest.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN List * restoreFile( const String *const repoFile, const unsigned int repoIdx, const CompressType repoFileCompressType, const time_t copyTimeBegin, const bool delta, const bool deltaForce, const bool bundleRaw, const String *const cipherPass, const StringList *const referenceList, List *const fileList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, repoFile); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(ENUM, repoFileCompressType); FUNCTION_LOG_PARAM(TIME, copyTimeBegin); FUNCTION_LOG_PARAM(BOOL, delta); FUNCTION_LOG_PARAM(BOOL, deltaForce); FUNCTION_LOG_PARAM(BOOL, bundleRaw); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_LOG_PARAM(STRING_LIST, referenceList); // List of references (for block incremental) FUNCTION_LOG_PARAM(LIST, fileList); // List of files to restore FUNCTION_LOG_END(); ASSERT(repoFile != NULL); // Restore file results List *const result = lstNewP(sizeof(RestoreFileResult)); MEM_CONTEXT_TEMP_BEGIN() { // Check files to determine which ones need to be restored for (unsigned int fileIdx = 0; fileIdx < lstSize(fileList); fileIdx++) { // Use a per-file mem context to reduce memory usage MEM_CONTEXT_TEMP_BEGIN() { RestoreFile *const file = lstGet(fileList, fileIdx); ASSERT(file->name != NULL); ASSERT(file->limit == NULL || varType(file->limit) == varTypeUInt64); RestoreFileResult *const fileResult = lstAdd( result, &(RestoreFileResult){.manifestFile = file->manifestFile, .result = restoreResultCopy}); // Perform delta if requested. Delta zero-length files to avoid overwriting the file if the timestamp is correct. if (delta && !file->zero) { // Perform delta if the file exists StorageInfo info = storageInfoP(storagePg(), file->name, .ignoreMissing = true, .followLink = true); if (info.exists) { // If force then use size/timestamp delta if (deltaForce) { // Make sure that timestamp/size are equal and the timestamp is before the copy start time of the backup if (info.size == file->size && info.timeModified == file->timeModified && info.timeModified < copyTimeBegin) { fileResult->result = restoreResultPreserve; } } // Else use size and checksum else { // Only continue delta if the file size is as expected or larger (for normal files) or if block // incremental and the file to delta is not zero-length. Block incremental can potentially use almost // any portion of an existing file, but of course zero-length files do not have anything to reuse. if (info.size >= file->size || (file->blockIncrMapSize != 0 && info.size != 0)) { const char *const fileName = strZ(storagePathP(storagePg(), file->name)); // If the file was extended since the backup, then truncate it to the size it was during the backup // as it might have only been appended to with the earlier portion being unchanged (we will verify // this using the checksum below) if (info.size > file->size) { // Open file for write IoWrite *const pgWriteTruncate = storageWriteIo( storageNewWriteP( storagePgWrite(), file->name, .noAtomic = true, .noCreatePath = true, .noSyncPath = true, .noTruncate = true)); ioWriteOpen(pgWriteTruncate); // Truncate to original size THROW_ON_SYS_ERROR_FMT( ftruncate(ioWriteFd(pgWriteTruncate), (off_t)file->size) == -1, FileWriteError, "unable to truncate file '%s'", fileName); // Close file ioWriteClose(pgWriteTruncate); // Update info info = storageInfoP(storagePg(), file->name, .followLink = true); // For block incremental it is very important that the file be exactly the expected size, so // make sure the truncate worked as expected CHECK_FMT( FileWriteError, info.size == file->size, "unable to truncate '%s' to %" PRIu64 " bytes", strZ(file->name), file->size); } // Generate checksum for the file if size is not zero IoRead *read = NULL; if (file->size != 0) { read = storageReadIo(storageNewReadP(storagePg(), file->name)); // Calculate checksum only when size matches if (info.size == file->size) ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); // Generate block checksum list if block incremental if (file->blockIncrMapSize != 0) { ioFilterGroupAdd( ioReadFilterGroup(read), blockChecksumNew(file->blockIncrSize, file->blockIncrChecksumSize)); } ioReadDrain(read); } // If size/checksum is the same (or file is zero size) then no need to copy the file if (file->size == 0 || (info.size == file->size && bufEq( file->checksum, pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE))))) { // If the checksum/size are now the same but the time is not, then set the time back to the // backup time. This helps with unit testing, but also presents a pristine version of the // database after restore. if (info.timeModified != file->timeModified) { const struct utimbuf uTimeBuf = { .actime = file->timeModified, .modtime = file->timeModified, }; THROW_ON_SYS_ERROR_FMT( utime(fileName, &uTimeBuf) == -1, FileInfoError, "unable to set time for '%s'", fileName); } fileResult->result = restoreResultPreserve; } // If block incremental and not preserving the file, store the block checksum list for later use in // reconstructing the pg file if (file->blockIncrMapSize != 0 && fileResult->result != restoreResultPreserve) { PackRead *const blockChecksumResult = ioFilterGroupResultP( ioReadFilterGroup(read), BLOCK_CHECKSUM_FILTER_TYPE); MEM_CONTEXT_OBJ_BEGIN(fileList) { file->blockChecksum = pckReadBinP(blockChecksumResult); } MEM_CONTEXT_OBJ_END(); } } } } } // Create zeroed and zero-length files if (fileResult->result == restoreResultCopy && (file->size == 0 || file->zero)) { // Create destination file StorageWrite *const pgFileWrite = storageNewWriteP( storagePgWrite(), file->name, .modeFile = file->mode, .user = file->user, .group = file->group, .timeModified = file->timeModified, .noAtomic = true, .noCreatePath = true, .noSyncPath = true); ioWriteOpen(storageWriteIo(pgFileWrite)); // Truncate the file to specified length (note in this case the file will grow, not shrink) if (file->zero) { THROW_ON_SYS_ERROR_FMT( ftruncate(ioWriteFd(storageWriteIo(pgFileWrite)), (off_t)file->size) == -1, FileWriteError, "unable to truncate '%s'", strZ(file->name)); } ioWriteClose(storageWriteIo(pgFileWrite)); // Report the file as zeroed or zero-length fileResult->result = restoreResultZero; } } MEM_CONTEXT_TEMP_END(); } // Copy files from repository to database StorageRead *repoFileRead = NULL; uint64_t repoFileLimit = 0; for (unsigned int fileIdx = 0; fileIdx < lstSize(fileList); fileIdx++) { // Copy file from repository to database RestoreFileResult *const fileResult = lstGet(result, fileIdx); if (fileResult->result == restoreResultCopy) { // Use a per-file mem context to reduce memory usage MEM_CONTEXT_TEMP_BEGIN() { const RestoreFile *const file = lstGet(fileList, fileIdx); // If no repo file is currently open if (repoFileLimit == 0) { // If a limit is specified then we need to use it, even if there is only one pg file to copy, because we // might be reading from the middle of a repo file containing many pg files if (file->limit != NULL) { ASSERT(varUInt64(file->limit) != 0); repoFileLimit = varUInt64(file->limit); // Determine how many files can be copied with one read for (unsigned int fileNextIdx = fileIdx + 1; fileNextIdx < lstSize(fileList); fileNextIdx++) { // Only files that are being copied are considered if (((const RestoreFileResult *)lstGet(result, fileNextIdx))->result == restoreResultCopy) { const RestoreFile *const fileNext = lstGet(fileList, fileNextIdx); ASSERT(fileNext->limit != NULL && varUInt64(fileNext->limit) != 0); // Break if the offset is not the first file's offset + limit of all additional files so far if (fileNext->offset != file->offset + repoFileLimit) break; repoFileLimit += varUInt64(fileNext->limit); } // Else if the file was not copied then there is a gap so break else break; } } // Create and open the repo file. It needs to be created in the prior context because it will live longer // than a single loop when more than one file is being read. MEM_CONTEXT_PRIOR_BEGIN() { repoFileRead = storageNewReadP( storageRepoIdx(repoIdx), repoFile, .compressible = repoFileCompressType == compressTypeNone && cipherPass == NULL, .offset = file->offset, .limit = repoFileLimit != 0 ? VARUINT64(repoFileLimit) : NULL); ioReadOpen(storageReadIo(repoFileRead)); } MEM_CONTEXT_PRIOR_END(); } // Create pg file StorageWrite *const pgFileWrite = storageNewWriteP( storagePgWrite(), file->name, .modeFile = file->mode, .user = file->user, .group = file->group, .timeModified = file->timeModified, .noAtomic = true, .noCreatePath = true, .noSyncPath = true, .noTruncate = file->blockChecksum != NULL); // If block incremental file const Buffer *checksum = NULL; if (file->blockIncrMapSize != 0) { ASSERT(referenceList != NULL); // Read block map. This will be compared to the block checksum list already created to determine which // blocks need to be fetched from the repository. If we got here there must be at least one block to fetch. IoRead *const blockMapRead = ioLimitReadNew(storageReadIo(repoFileRead), varUInt64(file->limit)); if (cipherPass != NULL) { ioFilterGroupAdd( ioReadFilterGroup(blockMapRead), cipherBlockNewP(cipherModeDecrypt, cipherTypeAes256Cbc, BUFSTR(cipherPass), .raw = true)); } ioReadOpen(blockMapRead); const BlockMap *const blockMap = blockMapNewRead( blockMapRead, file->blockIncrSize, file->blockIncrChecksumSize); ioReadFree(blockMapRead); // Open file to write ioWriteOpen(storageWriteIo(pgFileWrite)); // Apply delta to file BlockDelta *const blockDelta = blockDeltaNew( blockMap, file->blockIncrSize, file->blockIncrChecksumSize, file->blockChecksum, cipherPass == NULL ? cipherTypeNone : cipherTypeAes256Cbc, cipherPass, repoFileCompressType); for (unsigned int readIdx = 0; readIdx < blockDeltaReadSize(blockDelta); readIdx++) { const BlockDeltaRead *const read = blockDeltaReadGet(blockDelta, readIdx); // Open the super block list for read. Using one read for all super blocks is cheaper than reading from // the file multiple times, which is especially noticeable on object stores. StorageRead *const superBlockRead = storageNewReadP( storageRepoIdx(repoIdx), backupFileRepoPathP( strLstGet(referenceList, read->reference), .manifestName = file->manifestFile, .bundleId = read->bundleId, .blockIncr = true), .offset = read->offset, .limit = VARUINT64(read->size)); ioReadOpen(storageReadIo(superBlockRead)); // Write updated blocks to the file const BlockDeltaWrite *deltaWrite = blockDeltaNext(blockDelta, read, storageReadIo(superBlockRead)); while (deltaWrite != NULL) { // Seek to the block offset. It is possible we are already at the correct position but it is easier // and safer to let ioWriteSeek() figure it out. ioWriteSeek(storageWriteIo(pgFileWrite), deltaWrite->offset); // Write block ioWrite(storageWriteIo(pgFileWrite), deltaWrite->block); fileResult->blockIncrDeltaSize += bufUsed(deltaWrite->block); deltaWrite = blockDeltaNext(blockDelta, read, storageReadIo(superBlockRead)); } storageReadFree(superBlockRead); } // Close the file to complete the update ioWriteClose(storageWriteIo(pgFileWrite)); // Calculate checksum. In theory this is not needed because the file should always be reconstructed // correctly. However, it seems better to check and the pages should still be buffered making the operation // very fast. IoRead *const read = storageReadIo(storageNewReadP(storagePg(), file->name)); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); ioReadDrain(read); checksum = pckReadBinP(ioFilterGroupResultP(ioReadFilterGroup(read), CRYPTO_HASH_FILTER_TYPE)); } // Else normal file else { IoFilterGroup *const filterGroup = ioWriteFilterGroup(storageWriteIo(pgFileWrite)); // Add decryption filter if (cipherPass != NULL) { ioFilterGroupAdd( filterGroup, cipherBlockNewP(cipherModeDecrypt, cipherTypeAes256Cbc, BUFSTR(cipherPass), .raw = bundleRaw)); } // Add decompression filter if (repoFileCompressType != compressTypeNone) ioFilterGroupAdd(filterGroup, decompressFilterP(repoFileCompressType, .raw = bundleRaw)); // Add sha1 filter ioFilterGroupAdd(filterGroup, cryptoHashNew(hashTypeSha1)); // Add size filter ioFilterGroupAdd(filterGroup, ioSizeNew()); // Copy file ioWriteOpen(storageWriteIo(pgFileWrite)); ioCopyP(storageReadIo(repoFileRead), storageWriteIo(pgFileWrite), .limit = file->limit); ioWriteClose(storageWriteIo(pgFileWrite)); // Get checksum result checksum = pckReadBinP(ioFilterGroupResultP(filterGroup, CRYPTO_HASH_FILTER_TYPE)); } // If more than one file is being copied from a single read then decrement the limit if (repoFileLimit != 0) repoFileLimit -= varUInt64(file->limit); // Free the repo file when there are no more files to copy from it if (repoFileLimit == 0) storageReadFree(repoFileRead); // Validate checksum if (!bufEq(file->checksum, checksum)) { THROW_FMT( ChecksumError, "error restoring '%s': actual checksum '%s' does not match expected checksum '%s'", strZ(file->name), strZ(strNewEncode(encodingHex, checksum)), strZ(strNewEncode(encodingHex, file->checksum))); } } MEM_CONTEXT_TEMP_END(); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/file.h0000664000175100017510000000635315240547630020402 0ustar00runnerrunner/*********************************************************************************************************************************** Restore File ***********************************************************************************************************************************/ #ifndef COMMAND_RESTORE_FILE_H #define COMMAND_RESTORE_FILE_H #include "common/compress/helper.h" #include "common/type/variant.h" /*********************************************************************************************************************************** Restore file types ***********************************************************************************************************************************/ typedef enum { restoreResultPreserve, restoreResultZero, restoreResultCopy, } RestoreResult; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Copy a file from the backup to the specified destination typedef struct RestoreFile { const String *name; // File to restore const Buffer *checksum; // Expected checksum uint64_t size; // Expected size time_t timeModified; // Original modification time mode_t mode; // Original mode bool zero; // Should the file be zeroed? const String *user; // Original user const String *group; // Original group uint64_t offset; // Offset into repo file where pg file is located const Variant *limit; // Limit for read in the repo file uint64_t blockIncrMapSize; // Block incremental map size (0 if not incremental) size_t blockIncrSize; // Block incremental size (when map size > 0) size_t blockIncrChecksumSize; // Checksum size (when map size > 0) const String *manifestFile; // Manifest file const Buffer *blockChecksum; // Checksums for block incremental restore, set in restoreFile() } RestoreFile; typedef struct RestoreFileResult { const String *manifestFile; // Manifest file RestoreResult result; // Restore result (e.g. preserve, copy) uint64_t blockIncrDeltaSize; // Size restored by block incremental delta } RestoreFileResult; FN_EXTERN List *restoreFile( const String *repoFile, unsigned int repoIdx, CompressType repoFileCompressType, time_t copyTimeBegin, bool delta, bool deltaForce, bool bundleRaw, const String *cipherPass, const StringList *referenceList, List *fileList); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/process.c.inc0000664000175100017510000004673315240547630021712 0ustar00runnerrunner/*********************************************************************************************************************************** Generate a list of queues that determine the order of file processing ***********************************************************************************************************************************/ // Comparator to order ManifestFile objects by size then name static const Manifest *restoreProcessQueueComparatorManifest = NULL; static int restoreProcessQueueComparator(const void *const item1, const void *const item2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, item1); FUNCTION_TEST_PARAM_P(VOID, item2); FUNCTION_TEST_END(); ASSERT(item1 != NULL); ASSERT(item2 != NULL); // Unpack files const ManifestFile file1 = manifestFileUnpack(restoreProcessQueueComparatorManifest, *(const ManifestFilePack *const *)item1); const ManifestFile file2 = manifestFileUnpack(restoreProcessQueueComparatorManifest, *(const ManifestFilePack *const *)item2); // Zero length files should be ordered at the end if (file1.size == 0) { if (file2.size != 0) FUNCTION_TEST_RETURN(INT, 1); } else if (file2.size == 0) FUNCTION_TEST_RETURN(INT, -1); // If the bundle id differs that is enough to determine order if (file1.bundleId < file2.bundleId) FUNCTION_TEST_RETURN(INT, -1); else if (file1.bundleId > file2.bundleId) FUNCTION_TEST_RETURN(INT, 1); // If the bundle ids are 0 if (file1.bundleId == 0) { // If the size differs then that's enough to determine order if (file1.size < file2.size) FUNCTION_TEST_RETURN(INT, 1); else if (file1.size > file2.size) FUNCTION_TEST_RETURN(INT, -1); // If size is the same then use name to generate a deterministic ordering (names must be unique) ASSERT(!strEq(file1.name, file2.name)); FUNCTION_TEST_RETURN(INT, strCmp(file2.name, file1.name)); } // If the reference differs that is enough to determine order if (file1.reference == NULL) { if (file2.reference != NULL) FUNCTION_TEST_RETURN(INT, 1); } else if (file2.reference == NULL) FUNCTION_TEST_RETURN(INT, -1); else { const int backupLabelCmp = strCmp(file2.reference, file1.reference) * -1; if (backupLabelCmp != 0) FUNCTION_TEST_RETURN(INT, backupLabelCmp); } // Finally order by bundle offset ASSERT(file1.bundleOffset != file2.bundleOffset); if (file1.bundleOffset < file2.bundleOffset) FUNCTION_TEST_RETURN(INT, -1); FUNCTION_TEST_RETURN(INT, 1); } static uint64_t restoreProcessQueue(const Manifest *const manifest, List **const queueList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM_P(LIST, queueList); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(manifest != NULL); uint64_t result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Create list of process queues (use void * instead of List * to avoid Coverity false positive) *queueList = lstNewP(sizeof(void *)); // Generate the list of processing queues (there is always at least one) StringList *const targetList = strLstNew(); strLstAddZ(targetList, MANIFEST_TARGET_PGDATA "/"); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); if (target->tablespaceId != 0) strLstAddFmt(targetList, "%s/", strZ(target->name)); } // Generate the processing queues MEM_CONTEXT_BEGIN(lstMemContext(*queueList)) { for (unsigned int targetIdx = 0; targetIdx < strLstSize(targetList); targetIdx++) { List *const queue = lstNewP(sizeof(ManifestFile *), .comparator = restoreProcessQueueComparator); lstAdd(*queueList, &queue); } } MEM_CONTEXT_END(); // Now put all files into the processing queues for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFilePack *const filePack = manifestFilePackGet(manifest, fileIdx); const ManifestFile file = manifestFileUnpack(manifest, filePack); // Find the target that contains this file unsigned int targetIdx = 0; do { // A target should always be found CHECK(FormatError, targetIdx < strLstSize(targetList), "backup target not found"); if (strBeginsWith(file.name, strLstGet(targetList, targetIdx))) break; targetIdx++; } while (1); // Add file to queue lstAdd(*(List **)lstGet(*queueList, targetIdx), &filePack); // Add size to total result += file.size; } // Sort the queues restoreProcessQueueComparatorManifest = manifest; for (unsigned int targetIdx = 0; targetIdx < strLstSize(targetList); targetIdx++) lstSort(*(List **)lstGet(*queueList, targetIdx), sortOrderAsc); // Move process queues to prior context lstMove(*queueList, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT64, result); } /*********************************************************************************************************************************** Log the results of a job and throw errors ***********************************************************************************************************************************/ // Helper function to determine if a file should be zeroed static bool restoreFileZeroed(const String *const manifestName, RegExp *const zeroExp) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, manifestName); FUNCTION_TEST_PARAM(REGEXP, zeroExp); FUNCTION_TEST_END(); ASSERT(manifestName != NULL); FUNCTION_TEST_RETURN( BOOL, zeroExp == NULL ? false : regExpMatch(zeroExp, manifestName) && !strEndsWith(manifestName, STRDEF("/" PG_FILE_PGVERSION))); } // Helper function to construct the absolute pg path for any file. Add a temp extension to pg_control so a partially restored // cluster cannot be started. static String * restoreFilePgPath(const Manifest *const manifest, const String *const manifestName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM(STRING, manifestName); FUNCTION_TEST_END(); ASSERT(manifest != NULL); ASSERT(manifestName != NULL); String *const pathPg = manifestPathPg(manifestName); String *const result = strNewFmt( "%s/%s%s", strZ(manifestTargetBase(manifest)->path), strZ(pathPg), strEqZ(manifestName, MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL) ? "." STORAGE_FILE_TEMP_EXT : ""); strFree(pathPg); FUNCTION_TEST_RETURN(STRING, result); } static uint64_t restoreJobResult( const Manifest *const manifest, ProtocolParallelJob *const job, RegExp *const zeroExp, const uint64_t sizeTotal, uint64_t sizeRestored, unsigned int *const currentPercentComplete) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, job); FUNCTION_LOG_PARAM(REGEXP, zeroExp); FUNCTION_LOG_PARAM(UINT64, sizeTotal); FUNCTION_LOG_PARAM(UINT64, sizeRestored); FUNCTION_LOG_PARAM_P(UINT, currentPercentComplete); FUNCTION_LOG_END(); ASSERT(manifest != NULL); // The job was successful if (protocolParallelJobErrorCode(job) == 0) { MEM_CONTEXT_TEMP_BEGIN() { PackRead *const jobResult = protocolParallelJobResult(job); unsigned int percentComplete = 0; while (!pckReadNullP(jobResult)) { const ManifestFile file = manifestFileFind(manifest, pckReadStrP(jobResult)); const bool zeroed = restoreFileZeroed(file.name, zeroExp); const RestoreResult result = (RestoreResult)pckReadU32P(jobResult); const uint64_t blockIncrDeltaSize = pckReadU64P(jobResult); String *const log = strCatZ(strNew(), "restore"); // Note if file was zeroed (i.e. selective restore) if (zeroed) strCatZ(log, " zeroed"); // Add filename strCatFmt(log, " file %s", strZ(restoreFilePgPath(manifest, file.name))); // If preserved add details to explain why it was not copied or zeroed if (result == restoreResultPreserve) { strCatZ(log, " - "); // On force we match on size and modification time if (cfgOptionBool(cfgOptForce)) { strCatFmt( log, "exists and matches size %" PRIu64 " and modification time %" PRIu64, file.size, (uint64_t)file.timestamp); } // Else a checksum delta or file is zero-length else { strCatZ(log, "exists and "); // No need to copy zero-length files if (file.size == 0) { strCatZ(log, "is zero size"); } // The file matched the manifest checksum so did not need to be copied else strCatZ(log, "matches backup"); } } // Add bundle info strCatZ(log, " ("); if (file.bundleId != 0) { ASSERT(varUInt64(protocolParallelJobKey(job)) == file.bundleId); strCatZ(log, "bundle "); if (file.reference != NULL) strCatFmt(log, "%s/", strZ(file.reference)); strCatFmt(log, "%" PRIu64 "/%" PRIu64 ", ", file.bundleId, file.bundleOffset); } // Add block incremental delta size, i.e. amount of the file that block incremental updated if (file.blockIncrMapSize != 0 && result != restoreResultPreserve) { strCatZ(log, "bi "); if (blockIncrDeltaSize != file.size) strCatFmt(log, "%s/", strZ(strSizeFormat(blockIncrDeltaSize))); } // Add size and percent complete sizeRestored += file.size; // Store percentComplete as an integer (used to update progress in the lock file) percentComplete = cvtPctToUInt(sizeRestored, sizeTotal); strCatFmt(log, "%s, %s)", strZ(strSizeFormat(file.size)), strZ(strNewPct(sizeRestored, sizeTotal))); // If not zero-length add the checksum if (file.size != 0 && !zeroed) strCatFmt(log, " checksum %s", strZ(strNewEncode(encodingHex, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE)))); LOG_DETAIL_PID(protocolParallelJobProcessId(job), strZ(log)); } // Update currentPercentComplete and lock file when the change is significant enough if (percentComplete - *currentPercentComplete > 10) { *currentPercentComplete = percentComplete; cmdLockWriteP( .percentComplete = VARUINT(*currentPercentComplete), .sizeComplete = VARUINT64(sizeRestored), .size = VARUINT64(sizeTotal)); } } MEM_CONTEXT_TEMP_END(); // Free the job protocolParallelJobFree(job); } // Else the job errored else THROW_CODE(protocolParallelJobErrorCode(job), strZ(protocolParallelJobErrorMessage(job))); FUNCTION_LOG_RETURN(UINT64, sizeRestored); } /*********************************************************************************************************************************** Return new restore jobs as requested ***********************************************************************************************************************************/ typedef struct RestoreJobData { unsigned int repoIdx; // Internal repo idx Manifest *manifest; // Backup manifest List *queueList; // List of processing queues RegExp *zeroExp; // Identify files that should be sparse zeroed const String *cipherSubPass; // Passphrase used to decrypt files in the backup const String *rootReplaceUser; // User to replace invalid users when root const String *rootReplaceGroup; // Group to replace invalid group when root } RestoreJobData; // Helper to calculate the next queue to scan based on the client index static int restoreJobQueueNext(const unsigned int clientIdx, int queueIdx, const unsigned int queueTotal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_PARAM(INT, queueIdx); FUNCTION_TEST_PARAM(UINT, queueTotal); FUNCTION_TEST_END(); // Move (forward or back) to the next queue queueIdx += clientIdx % 2 ? -1 : 1; // Deal with wrapping on either end if (queueIdx < 0) FUNCTION_TEST_RETURN(INT, (int)queueTotal - 1); else if (queueIdx == (int)queueTotal) FUNCTION_TEST_RETURN(INT, 0); FUNCTION_TEST_RETURN(INT, queueIdx); } // Callback to fetch restore jobs for the parallel executor static ProtocolParallelJob * restoreJobCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(UINT, clientIdx); FUNCTION_TEST_END(); ASSERT(data != NULL); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get a new job if there are any left RestoreJobData *const jobData = data; // Determine where to begin scanning the queue (we'll stop when we get back here) PackWrite *param = NULL; int queueIdx = (int)(clientIdx % lstSize(jobData->queueList)); const int queueEnd = queueIdx; // Create restore job do { List *const queue = *(List **)lstGet(jobData->queueList, (unsigned int)queueIdx); bool fileAdded = false; const String *fileName = NULL; uint64_t bundleId = 0; const String *reference = NULL; while (!lstEmpty(queue)) { const ManifestFile file = manifestFileUnpack(jobData->manifest, *(ManifestFilePack **)lstGet(queue, 0)); // Break if bundled files have already been added and 1) the bundleId has changed or 2) the reference has changed if (fileAdded && (bundleId != file.bundleId || !strEq(reference, file.reference))) break; // Add common parameters before first file if (param == NULL) { param = protocolPackNew(); if (file.bundleId != 0) { bundleId = file.bundleId; reference = file.reference; } else fileName = file.name; pckWriteStrP( param, backupFileRepoPathP( file.reference != NULL ? file.reference : manifestData(jobData->manifest)->backupLabel, .manifestName = file.name, .bundleId = file.bundleId, .compressType = manifestData(jobData->manifest)->backupOptionCompressType, .blockIncr = file.blockIncrMapSize != 0)); pckWriteU32P(param, jobData->repoIdx); pckWriteU32P(param, manifestData(jobData->manifest)->backupOptionCompressType); pckWriteTimeP(param, manifestData(jobData->manifest)->backupTimestampCopyStart); pckWriteBoolP(param, cfgOptionBool(cfgOptDelta)); pckWriteBoolP(param, cfgOptionBool(cfgOptDelta) && cfgOptionBool(cfgOptForce)); pckWriteBoolP(param, file.bundleId != 0 && manifestData(jobData->manifest)->bundleRaw); pckWriteStrP(param, jobData->cipherSubPass); pckWriteStrLstP(param, manifestReferenceList(jobData->manifest)); fileAdded = true; } pckWriteStrP(param, restoreFilePgPath(jobData->manifest, file.name)); pckWriteBinP(param, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE)); pckWriteU64P(param, file.size); pckWriteTimeP(param, file.timestamp); pckWriteModeP(param, file.mode); pckWriteBoolP(param, restoreFileZeroed(file.name, jobData->zeroExp)); pckWriteStrP(param, restoreManifestOwnerReplace(file.user, jobData->rootReplaceUser)); pckWriteStrP(param, restoreManifestOwnerReplace(file.group, jobData->rootReplaceGroup)); // If block incremental then modify offset and size to where the map is stored since we need to read that first. if (file.blockIncrMapSize != 0) { pckWriteBoolP(param, true); pckWriteU64P(param, file.bundleOffset + file.sizeRepo - file.blockIncrMapSize); pckWriteU64P(param, file.blockIncrMapSize); } // Else write bundle offset/size else if (file.bundleId != 0) { pckWriteBoolP(param, true); pckWriteU64P(param, file.bundleOffset); pckWriteU64P(param, file.sizeRepo); } // Else restore as a whole file else pckWriteBoolP(param, false); // Block incremental pckWriteU64P(param, file.blockIncrMapSize); if (file.blockIncrMapSize != 0) { pckWriteU64P(param, file.blockIncrSize); pckWriteU64P(param, file.blockIncrChecksumSize); } pckWriteStrP(param, file.name); // Remove job from the queue lstRemoveIdx(queue, 0); // Break if the file is not bundled if (bundleId == 0) break; } if (fileAdded) { // Assign job to result MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew( bundleId != 0 ? VARUINT64(bundleId) : VARSTR(fileName), PROTOCOL_COMMAND_RESTORE_FILE, param); } MEM_CONTEXT_PRIOR_END(); break; } queueIdx = restoreJobQueueNext(clientIdx, queueIdx, lstSize(jobData->queueList)); } while (queueIdx != queueEnd); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/protocol.c0000664000175100017510000000660015240547630021312 0ustar00runnerrunner/*********************************************************************************************************************************** Restore Protocol Handler ***********************************************************************************************************************************/ #include #include "command/restore/file.h" #include "command/restore/protocol.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * restoreFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Restore file const String *const repoFile = pckReadStrP(param); const unsigned int repoIdx = pckReadU32P(param); const CompressType repoFileCompressType = (CompressType)pckReadU32P(param); const time_t copyTimeBegin = pckReadTimeP(param); const bool delta = pckReadBoolP(param); const bool deltaForce = pckReadBoolP(param); const bool bundleRaw = pckReadBoolP(param); const String *const cipherPass = pckReadStrP(param); const StringList *const referenceList = pckReadStrLstP(param); // Build the file list List *const fileList = lstNewP(sizeof(RestoreFile)); while (!pckReadNullP(param)) { RestoreFile file = {.name = pckReadStrP(param)}; file.checksum = pckReadBinP(param); file.size = pckReadU64P(param); file.timeModified = pckReadTimeP(param); file.mode = pckReadModeP(param); file.zero = pckReadBoolP(param); file.user = pckReadStrP(param); file.group = pckReadStrP(param); if (pckReadBoolP(param)) { file.offset = pckReadU64P(param); file.limit = varNewUInt64(pckReadU64P(param)); } // Block incremental file.blockIncrMapSize = pckReadU64P(param); if (file.blockIncrMapSize != 0) { file.blockIncrSize = (size_t)pckReadU64P(param); file.blockIncrChecksumSize = (size_t)pckReadU64P(param); } file.manifestFile = pckReadStrP(param); lstAdd(fileList, &file); } // Restore files const List *const resultList = restoreFile( repoFile, repoIdx, repoFileCompressType, copyTimeBegin, delta, deltaForce, bundleRaw, cipherPass, referenceList, fileList); // Return result PackWrite *const data = protocolServerResultData(result); for (unsigned int resultIdx = 0; resultIdx < lstSize(resultList); resultIdx++) { const RestoreFileResult *const fileResult = lstGet(resultList, resultIdx); pckWriteStrP(data, fileResult->manifestFile); pckWriteU32P(data, fileResult->result); pckWriteU64P(data, fileResult->blockIncrDeltaSize); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/protocol.h0000664000175100017510000000267015240547630021322 0ustar00runnerrunner/*********************************************************************************************************************************** Restore Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_RESTORE_PROTOCOL_H #define COMMAND_RESTORE_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *restoreFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_RESTORE_FILE STRID5("rs-f", 0x36e720) #define PROTOCOL_SERVER_HANDLER_RESTORE_LIST \ {.command = PROTOCOL_COMMAND_RESTORE_FILE, .process = restoreFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/remap.c.inc0000664000175100017510000002750715240547630021336 0ustar00runnerrunner/*********************************************************************************************************************************** Remap the manifest based on mappings provided by the user ***********************************************************************************************************************************/ static void restoreManifestMap(Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Remap the data directory // ------------------------------------------------------------------------------------------------------------------------- const String *const pgPath = cfgOptionStr(cfgOptPgPath); const ManifestTarget *const targetBase = manifestTargetBase(manifest); if (!strEq(targetBase->path, pgPath)) { LOG_INFO_FMT("remap data directory to '%s'", strZ(pgPath)); manifestTargetUpdate(manifest, targetBase->name, pgPath, NULL); } // Remap tablespaces // ------------------------------------------------------------------------------------------------------------------------- const KeyValue *const tablespaceMap = cfgOptionKvNull(cfgOptTablespaceMap); const String *const tablespaceMapAllPath = cfgOptionStrNull(cfgOptTablespaceMapAll); if (tablespaceMap != NULL || tablespaceMapAllPath != NULL) { StringList *const tablespaceRemapped = strLstNew(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); // Is this a tablespace? if (target->tablespaceId != 0) { const String *tablespacePath = NULL; // Check for an individual mapping for this tablespace if (tablespaceMap != NULL) { // Attempt to get the tablespace by name const String *const tablespacePathByName = varStr(kvGet(tablespaceMap, VARSTR(target->tablespaceName))); if (tablespacePathByName != NULL) strLstAdd(tablespaceRemapped, target->tablespaceName); // Attempt to get the tablespace by id const String *const tablespacePathById = varStr( kvGet(tablespaceMap, VARSTR(varStrForce(VARUINT(target->tablespaceId))))); if (tablespacePathById != NULL) strLstAdd(tablespaceRemapped, varStrForce(VARUINT(target->tablespaceId))); // Error when both are set but the paths are different if (tablespacePathByName != NULL && tablespacePathById != NULL && ! strEq(tablespacePathByName, tablespacePathById)) { THROW_FMT( TablespaceMapError, "tablespace remapped by name '%s' and id %u with different paths", strZ(target->tablespaceName), target->tablespaceId); } // Else set the path by name else if (tablespacePathByName != NULL) { tablespacePath = tablespacePathByName; } // Else set the path by id else if (tablespacePathById != NULL) tablespacePath = tablespacePathById; } // If not individual mapping check if all tablespaces are being remapped if (tablespacePath == NULL && tablespaceMapAllPath != NULL) tablespacePath = strNewFmt("%s/%s", strZ(tablespaceMapAllPath), strZ(target->tablespaceName)); // Remap tablespace if a mapping was found if (tablespacePath != NULL) { LOG_INFO_FMT("map tablespace '%s' to '%s'", strZ(target->name), strZ(tablespacePath)); manifestTargetUpdate(manifest, target->name, tablespacePath, NULL); manifestLinkUpdate(manifest, strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(target->name)), tablespacePath); } } } // Error on invalid tablespaces if (tablespaceMap != NULL) { const VariantList *const tablespaceMapList = kvKeyList(tablespaceMap); strLstSort(tablespaceRemapped, sortOrderAsc); for (unsigned int tablespaceMapIdx = 0; tablespaceMapIdx < varLstSize(tablespaceMapList); tablespaceMapIdx++) { const String *const tablespace = varStr(varLstGet(tablespaceMapList, tablespaceMapIdx)); if (!strLstExists(tablespaceRemapped, tablespace)) THROW_FMT(TablespaceMapError, "unable to remap invalid tablespace '%s'", strZ(tablespace)); } } } // Remap links // ------------------------------------------------------------------------------------------------------------------------- const KeyValue *const linkMap = cfgOptionKvNull(cfgOptLinkMap); if (linkMap != NULL) { const StringList *const linkMapList = strLstSort(strLstNewVarLst(kvKeyList(linkMap)), sortOrderAsc); for (unsigned int linkMapIdx = 0; linkMapIdx < strLstSize(linkMapList); linkMapIdx++) { const String *const link = strLstGet(linkMapList, linkMapIdx); const String *const linkPath = varStr(kvGet(linkMap, VARSTR(link))); const String *const manifestName = strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(link)); // Attempt to find the link target ManifestTarget target = {0}; if (manifestTargetFindDefault(manifest, manifestName, NULL) != NULL) target = *manifestTargetFind(manifest, manifestName); // If the target was not found then check if the link is a valid file or path bool create = false; if (target.name == NULL) { // Is the specified link a file or a path? Error if they both match. const bool pathExists = manifestPathFindDefault(manifest, manifestName, NULL) != NULL; const bool fileExists = manifestFileExists(manifest, manifestName); CHECK(FormatError, !pathExists || !fileExists, "link may not be both file and path"); target = (ManifestTarget){.name = manifestName, .path = linkPath, .type = manifestTargetTypeLink}; // If a file if (fileExists) { // File needs to be set so the file/path is updated later but set it to something invalid just in case it // it does not get updated due to a regression target.file = DOT_STR; } // Else error if not a path else if (!pathExists) { THROW_FMT( LinkMapError, "unable to map link '%s'\n" "HINT: Does the link reference a valid backup path or file?", strZ(link)); } // Add the link. Copy user/group from the base data directory. const ManifestPath *const pathBase = manifestPathFind(manifest, MANIFEST_TARGET_PGDATA_STR); const ManifestLink manifestLink = { .name = manifestName, .destination = linkPath, .group = pathBase->group, .user = pathBase->user, }; manifestLinkAdd(manifest, &manifestLink); create = true; } // Else update target to new path else target.path = linkPath; // The target must be a link since pg_data/ was prepended and pgdata is the only allowed path CHECK(FormatError, target.type == manifestTargetTypeLink, "target must be a link"); // Error if the target is a tablespace if (target.tablespaceId != 0) { THROW_FMT( LinkMapError, "unable to remap tablespace '%s'\n" "HINT: use '" CFGOPT_TABLESPACE_MAP "' option to remap tablespaces.", strZ(link)); } LOG_INFO_FMT("%slink '%s' to '%s'", create ? "" : "map ", strZ(link), strZ(target.path)); // If the link was not created update to the new destination if (!create) manifestLinkUpdate(manifest, target.name, target.path); // If the link is a file separate the file name from the path if (target.file != NULL) { // The link destination must have at least one path component in addition to the file part. So '..' would // not be a valid destination but '../file' or '/file' is. if (strSize(strPath(target.path)) == 0) { THROW_FMT( LinkMapError, "'%s' is not long enough to be the destination for file link '%s'", strZ(target.path), strZ(link)); } target.file = strBase(target.path); target.path = strPath(target.path); } // Create a new target or update the existing target file/path if (create) manifestTargetAdd(manifest, &target); else manifestTargetUpdate(manifest, target.name, target.path, target.file); } } // If all links are not being restored then check for links that were not remapped and remove them if (!cfgOptionBool(cfgOptLinkAll)) { unsigned int targetIdx = 0; while (targetIdx < manifestTargetTotal(manifest)) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); // Is this a non-tablespace link? if (target->type == manifestTargetTypeLink && target->tablespaceId == 0) { const String *const link = strSub(target->name, strSize(MANIFEST_TARGET_PGDATA_STR) + 1); // If the link was not remapped then remove it if (linkMap == NULL || kvGet(linkMap, VARSTR(link)) == NULL) { if (target->file != NULL) LOG_WARN_FMT("file link '%s' will be restored as a file at the same location", strZ(link)); else { LOG_WARN_FMT( "contents of directory link '%s' will be restored in a directory at the same location", strZ(link)); } manifestLinkRemove(manifest, target->name); manifestTargetRemove(manifest, target->name); continue; } } targetIdx++; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/restore.c0000664000175100017510000003130015240547630021127 0ustar00runnerrunner/*********************************************************************************************************************************** Restore Command ***********************************************************************************************************************************/ #include #include #include "command/lock.h" #include "command/restore/file.h" #include "command/restore/protocol.h" #include "command/restore/restore.h" #include "command/restore/timeline.h" #include "common/regExp.h" #include "common/user.h" #include "config/exec.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" #include "clean.c.inc" #include "config.c.inc" #include "process.c.inc" #include "remap.c.inc" #include "select.c.inc" #include "validate.c.inc" /**********************************************************************************************************************************/ FN_EXTERN void cmdRestore(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Get information for the current user userInit(); // PostgreSQL must be local pgIsLocalVerify(); // Validate restore path restorePathValidate(); // Remove stanza archive spool path so existing files do not interfere with the new cluster. For instance, old archive-push // acknowledgements could cause a new cluster to skip archiving. This should not happen if a new timeline is selected but // better to be safe. Missing stanza spool paths are ignored. storagePathRemoveP(storageSpoolWrite(), STORAGE_SPOOL_ARCHIVE_STR, .recurse = true); // Get the backup set const RestoreBackupData backupData = restoreBackupSet(); // Load manifest RestoreJobData jobData = {.repoIdx = backupData.repoIdx}; jobData.manifest = manifestLoadFile( storageRepoIdx(backupData.repoIdx), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupData.backupSet)), backupData.repoCipherType, backupData.backupCipherPass); // Verify that the selected timeline is valid for the backup -- including current and latest timelines if (manifestData(jobData.manifest)->backupOptionOnline) { const ManifestData *const data = manifestData(jobData.manifest); const InfoArchive *const archiveInfo = infoArchiveLoadFile( storageRepoIdx(backupData.repoIdx), INFO_ARCHIVE_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, backupData.repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, backupData.repoIdx)); timelineVerify( storageRepoIdx(backupData.repoIdx), strNewFmt("%s-%u", strZ(pgVersionToStr(data->pgVersion)), data->pgId), data->pgVersion, cvtZToUIntBase(strZ(strSubN(data->archiveStart, 0, 8)), 16), pgLsnFromStr(data->lsnStart), cfgOptionStrNull(cfgOptTargetTimeline), cfgOptionSeq(cfgOptType), cfgOptionIdxStrId(cfgOptRepoCipherType, backupData.repoIdx), infoArchiveCipherPass(archiveInfo)); } // Remotes (if any) are no longer needed since the rest of the repository reads will be done by the local processes protocolFree(); // If the backup was made offline and no restore type was specified then set to none. This prevents recovery.signal from // being generated by default for PostgreSQL >= 12. Offline backups created with wal_level=minimal will error in this case // so this should be a good default. However, if the user explicitly sets type and it does not equal none then they will get // an error if wal_level=minimal. if (!manifestData(jobData.manifest)->backupOptionOnline && cfgOptionSource(cfgOptType) == cfgSourceDefault) cfgOptionSet(cfgOptType, cfgSourceParam, VARUINT64(CFGOPTVAL_RESTORE_TYPE_NONE_STRID)); // Validate manifest. Don't use strict mode because we'd rather ignore problems that won't affect a restore. manifestValidate(jobData.manifest, false); // Get the cipher subpass used to decrypt files in the backup jobData.cipherSubPass = manifestCipherSubPass(jobData.manifest); // Validate the manifest restoreManifestValidate(jobData.manifest, backupData.backupSet); // Log the backup set to restore. If the backup was online then append the time recovery will start from. String *const message = strCatFmt( strNew(), "%s: restore backup set %s", cfgOptionGroupName(cfgOptGrpRepo, backupData.repoIdx), strZ(backupData.backupSet)); if (manifestData(jobData.manifest)->backupOptionOnline) { strCatZ(message, ", recovery will start at "); strCatTimeP(message, "%Y-%m-%d %H:%M:%S", manifestData(jobData.manifest)->backupTimestampStart); } LOG_INFO(strZ(message)); // Map manifest restoreManifestMap(jobData.manifest); // Check that links are sane manifestLinkCheck(jobData.manifest); // Update ownership restoreManifestOwner(jobData.manifest, &jobData.rootReplaceUser, &jobData.rootReplaceGroup); // Generate the selective restore expression const String *const expression = restoreSelectiveExpression(jobData.manifest); jobData.zeroExp = expression == NULL ? NULL : regExpNew(expression); // Clean the data directory and build path/link structure restoreCleanBuild(jobData.manifest, jobData.rootReplaceUser, jobData.rootReplaceGroup); // Generate processing queues const uint64_t sizeTotal = restoreProcessQueue(jobData.manifest, &jobData.queueList); // Save manifest to the data directory so we can restart a delta restore even if the PG_VERSION file is missing manifestSave(jobData.manifest, storageWriteIo(storageNewWriteP(storagePgWrite(), BACKUP_MANIFEST_FILE_STR))); // Create the parallel executor ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, restoreJobCallback, &jobData); for (unsigned int processIdx = 1; processIdx <= cfgOptionUInt(cfgOptProcessMax); processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypeRepo, 0, processIdx)); // Process jobs uint64_t sizeRestored = 0; // Initialize percent complete and bytes completed/total unsigned int currentPercentComplete = 0; cmdLockWriteP( .percentComplete = VARUINT(currentPercentComplete), .sizeComplete = VARUINT64(sizeRestored), .size = VARUINT64(sizeTotal)); MEM_CONTEXT_TEMP_RESET_BEGIN() { do { const unsigned int completed = protocolParallelProcess(parallelExec); for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { sizeRestored = restoreJobResult( jobData.manifest, protocolParallelResult(parallelExec), jobData.zeroExp, sizeTotal, sizeRestored, ¤tPercentComplete); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); // Write recovery settings. Use the data directory to set permissions and ownership for recovery files. StorageInfo fileInfo = storageInfoP(storagePg(), NULL); fileInfo.user = restoreManifestOwnerReplace(fileInfo.user, jobData.rootReplaceUser); fileInfo.group = restoreManifestOwnerReplace(fileInfo.group, jobData.rootReplaceGroup); fileInfo.mode = fileInfo.mode & (S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); restoreRecoveryWrite(jobData.manifest, &fileInfo); // Remove backup.manifest storageRemoveP(storagePgWrite(), BACKUP_MANIFEST_FILE_STR); // Sync file link paths. These need to be synced separately because they are not linked from the data directory. StringList *const pathSynced = strLstNew(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(jobData.manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(jobData.manifest, targetIdx); if (target->type == manifestTargetTypeLink && target->file != NULL) { const String *const pgPath = manifestTargetPath(jobData.manifest, target); // Don't sync the same path twice. There can be multiple links to files in the same path, but syncing it more than // once makes the logs noisy and looks like a bug even though it doesn't hurt anything or realistically affect // performance. if (strLstExists(pathSynced, pgPath)) continue; else strLstAdd(pathSynced, pgPath); // Sync the path LOG_DETAIL_FMT("sync path '%s'", strZ(pgPath)); storagePathSyncP(storageLocalWrite(), pgPath); } } // Sync paths in the data directory for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(jobData.manifest); pathIdx++) { const String *const manifestName = manifestPath(jobData.manifest, pathIdx)->name; // Skip the pg_tblspc path because it only maps to the manifest. We should remove this in a future release but not much // can be done about it for now. if (strEqZ(manifestName, MANIFEST_TARGET_PGTBLSPC)) continue; // We'll sync global after pg_control is written if (strEq(manifestName, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL))) continue; const String *const pgPath = storagePathP(storagePg(), manifestPathPg(manifestName)); LOG_DETAIL_FMT("sync path '%s'", strZ(pgPath)); storagePathSyncP(storagePgWrite(), pgPath); } // Rename pg_control to remove the temp extension. This is done last to prevent a partially restored (or unsynced) cluster // from being started. if (storageExistsP(storagePg(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "." STORAGE_FILE_TEMP_EXT))) { // Invalidate the checkpoint in pg_control so the cluster cannot be started without backup_label if (manifestData(jobData.manifest)->backupOptionOnline) { Buffer *const pgControlBuffer = storageGetP( storageNewReadP(storagePg(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "." STORAGE_FILE_TEMP_EXT))); const ManifestFile pgControlFile = manifestFileFind( jobData.manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)); pgControlCheckpointInvalidate(pgControlBuffer, cfgOptionStrNull(cfgOptPgVersionForce)); storagePutP( storageNewWriteP( storagePgWrite(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "." STORAGE_FILE_TEMP_EXT), .timeModified = pgControlFile.timestamp, .noAtomic = true, .noCreatePath = true, .noSyncPath = true), pgControlBuffer); } LOG_INFO( "restore " PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL " (performed last to ensure aborted restores cannot be started)"); storageMoveP( storagePgWrite(), storageNewReadP(storagePg(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "." STORAGE_FILE_TEMP_EXT)), storageNewWriteP(storagePgWrite(), STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL), .noSyncPath = true)); } else LOG_WARN("backup does not contain '" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL "' -- cluster will not start"); // Sync global path to persist pg_control rename. If this fails or if the system crashes before it can complete, then // pg_control should retain the temp extension, which will prevent the cluster from being started. LOG_DETAIL_FMT("sync path '%s'", strZ(storagePathP(storagePg(), PG_PATH_GLOBAL_STR))); storagePathSyncP(storagePgWrite(), PG_PATH_GLOBAL_STR); // Restore info LOG_INFO_FMT( "restore size = %s, file total = %u", strZ(strSizeFormat(sizeRestored)), manifestFileTotal(jobData.manifest)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/restore.h0000664000175100017510000000126015240547630021136 0ustar00runnerrunner/*********************************************************************************************************************************** Restore Command ***********************************************************************************************************************************/ #ifndef COMMAND_RESTORE_RESTORE_H #define COMMAND_RESTORE_RESTORE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Restore a backup FN_EXTERN void cmdRestore(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/select.c.inc0000664000175100017510000004571015240547630021505 0ustar00runnerrunner/*********************************************************************************************************************************** Get the backup set to restore ***********************************************************************************************************************************/ typedef struct RestoreBackupData { unsigned int repoIdx; // Internal repo index CipherType repoCipherType; // Repo encryption type (0 = none) const String *backupCipherPass; // Passphrase of backup files if repo is encrypted (else NULL) const String *backupSet; // Backup set to restore } RestoreBackupData; #define FUNCTION_LOG_RESTORE_BACKUP_DATA_TYPE \ RestoreBackupData #define FUNCTION_LOG_RESTORE_BACKUP_DATA_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "RestoreBackupData", buffer, bufferSize) // Helper function for restoreBackupSet static RestoreBackupData restoreBackupData(const String *const backupLabel, const unsigned int repoIdx, const String *const backupCipherPass) { ASSERT(backupLabel != NULL); RestoreBackupData restoreBackup = {0}; MEM_CONTEXT_PRIOR_BEGIN() { restoreBackup.backupSet = strDup(backupLabel); restoreBackup.repoIdx = repoIdx; restoreBackup.repoCipherType = cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx); restoreBackup.backupCipherPass = strDup(backupCipherPass); } MEM_CONTEXT_PRIOR_END(); return restoreBackup; } static RestoreBackupData restoreBackupSet(void) { FUNCTION_LOG_VOID(logLevelDebug); FUNCTION_AUDIT_STRUCT(); RestoreBackupData result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Initialize the repo index unsigned int repoIdxMin = 0; unsigned int repoIdxMax = cfgOptionGroupIdxTotal(cfgOptGrpRepo) - 1; // If the repo was specified then set index to the array location and max to loop only once if (cfgOptionTest(cfgOptRepo)) { repoIdxMin = cfgOptionGroupIdxDefault(cfgOptGrpRepo); repoIdxMax = repoIdxMin; } // If the set option was not provided by the user but a target was set, then we will need to search for a backup set that // satisfies the target condition, else we will use the backup provided const String *backupSetRequested = NULL; const unsigned int targetType = cfgOptionSeq(cfgOptType); union { time_t time; uint64_t lsn; } target = {0}; if (cfgOptionSource(cfgOptSet) == cfgSourceDefault) { if (targetType == CFGOPTVAL_RESTORE_TYPE_TIME) { TRY_BEGIN() { target.time = cvtZToTime(strZ(cfgOptionStr(cfgOptTarget))); } CATCH_ANY() { THROW_FMT( FormatError, "automatic backup set selection cannot be performed with provided time '%s'\n" "HINT: time format must be YYYY-MM-DD HH:MM:SS with optional msec and optional timezone (+/- HH or HHMM or" " HH:MM) - if timezone is omitted, local time is assumed (for UTC use +00)", strZ(cfgOptionStr(cfgOptTarget))); } TRY_END(); } else if (targetType == CFGOPTVAL_RESTORE_TYPE_LSN) target.lsn = pgLsnFromStr(cfgOptionStr(cfgOptTarget)); } else backupSetRequested = cfgOptionStr(cfgOptSet); // Search through the repo list for a backup set to use for recovery for (unsigned int repoIdx = repoIdxMin; repoIdx <= repoIdxMax; repoIdx++) { // Get the repo storage in case it is remote and encryption settings need to be pulled down storageRepoIdx(repoIdx); const InfoBackup *infoBackup = NULL; // Attempt to load backup.info TRY_BEGIN() { infoBackup = infoBackupLoadFile( storageRepoIdx(repoIdx), INFO_BACKUP_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); } CATCH_ANY() { LOG_WARN_FMT("%s: [%s] %s", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(errorType()), errorMessage()); } TRY_END(); // If unable to load the backup info file, then move on to next repo if (infoBackup == NULL) continue; if (infoBackupDataTotal(infoBackup) == 0) { LOG_WARN_FMT( "%s: [%s] no backup sets to restore", cfgOptionGroupName(cfgOptGrpRepo, repoIdx), errorTypeName(&BackupSetInvalidError)); continue; } // If a backup set was not specified, then see if a target was requested if (backupSetRequested == NULL) { // Get the latest backup const InfoBackupData latestBackup = infoBackupData(infoBackup, infoBackupDataTotal(infoBackup) - 1); // If a target was requested, attempt to determine the backup set if (targetType == CFGOPTVAL_RESTORE_TYPE_TIME || targetType == CFGOPTVAL_RESTORE_TYPE_LSN) { bool found = false; // Search current backups from newest to oldest for (unsigned int keyIdx = infoBackupDataTotal(infoBackup) - 1; (int)keyIdx >= 0; keyIdx--) { // Get the backup data const InfoBackupData backupData = infoBackupData(infoBackup, keyIdx); // If target is lsn and no backupLsnStop exists, exit this repo and log that backup may be manually selected if (targetType == CFGOPTVAL_RESTORE_TYPE_LSN && !backupData.backupLsnStop) { LOG_WARN_FMT( "%s reached backup from prior version missing required LSN info before finding a match -- backup" " auto-select has been disabled for this repo\n" "HINT: you may specify a backup to restore using the --set option.", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); break; } // If the end of the backup is valid for the target, then select this backup if ((targetType == CFGOPTVAL_RESTORE_TYPE_TIME && backupData.backupTimestampStop < target.time) || (targetType == CFGOPTVAL_RESTORE_TYPE_LSN && pgLsnFromStr(backupData.backupLsnStop) <= target.lsn)) { found = true; result = restoreBackupData(backupData.backupLabel, repoIdx, infoPgCipherPass(infoBackupPg(infoBackup))); break; } } // If a backup was found on this repo matching the criteria for time then exit if (found) break; } // Else use backup set found else { // Is this backup part of the latest pg history? const InfoPgData backupInfoPg = infoPgData( infoBackupPg(infoBackup), infoPgDataCurrentId(infoBackupPg(infoBackup))); if (latestBackup.backupPgId < backupInfoPg.id) { THROW_FMT( BackupSetInvalidError, "the latest backup set found '%s' is from a prior version of " PG_NAME "\n" "HINT: was a backup created after the stanza-upgrade?\n" "HINT: specify --" CFGOPT_SET " or --" CFGOPT_TYPE "=time/lsn to restore from a prior version of" " " PG_NAME ".", strZ(latestBackup.backupLabel)); } result = restoreBackupData(latestBackup.backupLabel, repoIdx, infoPgCipherPass(infoBackupPg(infoBackup))); break; } } // Otherwise check to see if the specified backup set is on this repo else { for (unsigned int backupIdx = 0; backupIdx < infoBackupDataTotal(infoBackup); backupIdx++) { if (strEq(infoBackupData(infoBackup, backupIdx).backupLabel, backupSetRequested)) { result = restoreBackupData(backupSetRequested, repoIdx, infoPgCipherPass(infoBackupPg(infoBackup))); break; } } // If the backup set is found, then exit, else continue to next repo if (result.backupSet != NULL) break; } } // Still no backup set to use after checking all the repos required to be checked? if (result.backupSet == NULL) { if (backupSetRequested != NULL) THROW_FMT(BackupSetInvalidError, "backup set %s is not valid", strZ(backupSetRequested)); else if (targetType == CFGOPTVAL_RESTORE_TYPE_TIME || targetType == CFGOPTVAL_RESTORE_TYPE_LSN) { THROW_FMT( BackupSetInvalidError, "unable to find backup set with %s '%s'", targetType == CFGOPTVAL_RESTORE_TYPE_LSN ? "lsn less than or equal to" : "stop time less than", strZ(cfgOptionDisplay(cfgOptTarget))); } else THROW(BackupSetInvalidError, "no backup set found to restore"); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /*********************************************************************************************************************************** Generate the expression to zero files that are not needed for selective restore ***********************************************************************************************************************************/ static String * restoreSelectiveExpression(const Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); ASSERT(manifest != NULL); String *result = NULL; // Continue if databases to include or exclude have been specified if (cfgOptionTest(cfgOptDbExclude) || cfgOptionTest(cfgOptDbInclude)) { MEM_CONTEXT_TEMP_BEGIN() { // Generate base expression RegExp *const baseRegExp = regExpNew(STRDEF("^" MANIFEST_TARGET_PGDATA "/" PG_PATH_BASE "/[0-9]+/" PG_FILE_PGVERSION)); // Generate tablespace expression const String *const tablespaceId = pgTablespaceId( manifestData(manifest)->pgVersion, manifestData(manifest)->pgCatalogVersion); RegExp *const tablespaceRegExp = regExpNew( strNewFmt("^" MANIFEST_TARGET_PGTBLSPC "/[0-9]+/%s/[0-9]+/" PG_FILE_PGVERSION, strZ(tablespaceId))); // Generate a list of databases in base or in a tablespace and get all standard system databases, even in cases where // users have recreated them StringList *systemDbIdList = strLstNew(); StringList *dbList = strLstNew(); for (unsigned int systemDbIdx = 0; systemDbIdx < manifestDbTotal(manifest); systemDbIdx++) { const ManifestDb *const systemDb = manifestDb(manifest, systemDbIdx); if (pgDbIsSystem(systemDb->name) || pgDbIsSystemId(systemDb->id)) { // Build the system id list and add to the dbList for logging and checking const String *const systemDbId = varStrForce(VARUINT(systemDb->id)); strLstAdd(systemDbIdList, systemDbId); strLstAdd(dbList, systemDbId); } } for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const String *const fileName = manifestFileNameGet(manifest, fileIdx); if (regExpMatch(baseRegExp, fileName) || regExpMatch(tablespaceRegExp, fileName)) { const String *const dbId = strBase(strPath(fileName)); // In the highly unlikely event that a system database was somehow added after the backup began, it will only be // found in the file list and not the manifest db section, so add it to the system database list if (pgDbIsSystemId(cvtZToUInt(strZ(dbId)))) strLstAddIfMissing(systemDbIdList, dbId); strLstAddIfMissing(dbList, dbId); } } strLstSort(dbList, sortOrderAsc); // If no databases were found then this backup is not a valid cluster if (strLstEmpty(dbList)) THROW(FormatError, "no databases found for selective restore\nHINT: is this a valid cluster?"); // Log databases found LOG_DETAIL_FMT("databases found for selective restore (%s)", strZ(strLstJoin(dbList, ", "))); // Generate list with ids of databases to exclude StringList *const excludeDbIdList = strLstNew(); const StringList *const excludeList = strLstNewVarLst(cfgOptionLst(cfgOptDbExclude)); for (unsigned int excludeIdx = 0; excludeIdx < strLstSize(excludeList); excludeIdx++) { const String *excludeDb = strLstGet(excludeList, excludeIdx); // If the db to exclude is not in the list as an id then search by name if (!strLstExists(dbList, excludeDb)) { const ManifestDb *const db = manifestDbFindDefault(manifest, excludeDb, NULL); if (db == NULL || !strLstExists(dbList, varStrForce(VARUINT(db->id)))) THROW_FMT(DbMissingError, "database to exclude '%s' does not exist", strZ(excludeDb)); // Set the exclude db to the id if the name mapping was successful excludeDb = varStrForce(VARUINT(db->id)); } // Add to exclude list strLstAdd(excludeDbIdList, excludeDb); } // Remove included databases from the list const StringList *const includeList = strLstNewVarLst(cfgOptionLst(cfgOptDbInclude)); for (unsigned int includeIdx = 0; includeIdx < strLstSize(includeList); includeIdx++) { const String *includeDb = strLstGet(includeList, includeIdx); // If the db to include is not in the list as an id then search by name if (!strLstExists(dbList, includeDb)) { const ManifestDb *const db = manifestDbFindDefault(manifest, includeDb, NULL); if (db == NULL || !strLstExists(dbList, varStrForce(VARUINT(db->id)))) THROW_FMT(DbMissingError, "database to include '%s' does not exist", strZ(includeDb)); // Set the include db to the id if the name mapping was successful includeDb = varStrForce(VARUINT(db->id)); } // Error if the db is a system db if (strLstExists(systemDbIdList, includeDb)) THROW(DbInvalidError, "system databases (template0, postgres, etc.) are included by default"); // Error if the db id is in the exclude list if (strLstExists(excludeDbIdList, includeDb)) THROW_FMT(DbInvalidError, "database to include '%s' is in the exclude list", strZ(includeDb)); // Remove from list of DBs to zero strLstRemove(dbList, includeDb); } // Only exclude specified db in case no db to include has been provided if (strLstEmpty(includeList)) { dbList = strLstDup(excludeDbIdList); } // Else, remove the system databases from list of DBs to zero unless they are excluded explicitly else { strLstSort(systemDbIdList, sortOrderAsc); strLstSort(excludeDbIdList, sortOrderAsc); systemDbIdList = strLstMergeAnti(systemDbIdList, excludeDbIdList); dbList = strLstMergeAnti(dbList, systemDbIdList); } // Build regular expression to identify files that will be zeroed String *expression = NULL; if (!strLstEmpty(dbList)) { LOG_DETAIL_FMT("databases excluded (zeroed) from selective restore (%s)", strZ(strLstJoin(dbList, ", "))); // Generate the expression from the list of databases to be zeroed. Only user created databases can be zeroed, never // system databases. for (unsigned int dbIdx = 0; dbIdx < strLstSize(dbList); dbIdx++) { const String *const db = strLstGet(dbList, dbIdx); // Create expression string or append | if (expression == NULL) expression = strNew(); else strCatZ(expression, "|"); // Filter files in base directory strCatFmt(expression, "(^" MANIFEST_TARGET_PGDATA "/" PG_PATH_BASE "/%s/)", strZ(db)); // Filter files in tablespace directories for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); if (target->tablespaceId != 0) strCatFmt(expression, "|(^%s/%s/%s/)", strZ(target->name), strZ(tablespaceId), strZ(db)); } } } // If all user databases have been selected then nothing to do if (expression == NULL) { LOG_INFO_FMT("nothing to filter - all user databases have been selected"); } // Else return the expression else { MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(expression); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/timeline.c0000664000175100017510000003133515240547630021262 0ustar00runnerrunner/*********************************************************************************************************************************** Timeline Management ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/restore/timeline.h" #include "common/crypto/cipherBlock.h" #include "common/log.h" #include "config/config.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** Parse history file ***********************************************************************************************************************************/ typedef struct HistoryItem { unsigned int timeline; // Parent timeline uint64_t lsn; // Boundary lsn } HistoryItem; static List * historyParse(const String *const history) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, history); FUNCTION_LOG_END(); List *const result = lstNewP(sizeof(HistoryItem)); MEM_CONTEXT_TEMP_BEGIN() { const StringList *const historyList = strLstNewSplitZ(history, "\n"); for (unsigned int historyIdx = 0; historyIdx < strLstSize(historyList); historyIdx++) { const String *const line = strTrim(strLstGet(historyList, historyIdx)); // Skip empty lines if (strSize(line) == 0) continue; // Skip comments. PostgreSQL does not write comments into history files but it does allow for them. if (strBeginsWithZ(line, "#")) continue; // Split line const StringList *const split = strLstNewSplitZ(line, "\t"); if (strLstSize(split) < 2) THROW_FMT(FormatError, "invalid history line format '%s'", strZ(strLstGet(historyList, historyIdx))); const HistoryItem historyItem = { .timeline = cvtZToUInt(strZ(strLstGet(split, 0))), .lsn = pgLsnFromStr(strLstGet(split, 1)), }; ASSERT(historyItem.lsn > 0); ASSERT(historyItem.timeline > 0); lstAdd(result, &historyItem); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(LIST, result); } /*********************************************************************************************************************************** Load history file ***********************************************************************************************************************************/ static List * historyLoad( const Storage *const storageRepo, const String *const archiveId, const unsigned int timeline, const CipherType cipherType, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storageRepo); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(UINT, timeline); FUNCTION_LOG_PARAM(STRING_ID, cipherType); FUNCTION_LOG_PARAM(STRING, cipherPass); FUNCTION_LOG_END(); List *result; MEM_CONTEXT_TEMP_BEGIN() { const String *const historyFile = strNewFmt(STORAGE_REPO_ARCHIVE "/%s/%08X.history", strZ(archiveId), timeline); StorageRead *const storageRead = storageNewReadP(storageRepo, historyFile); cipherBlockFilterGroupAdd(ioReadFilterGroup(storageReadIo(storageRead)), cipherType, cipherModeDecrypt, cipherPass); const Buffer *const history = storageGetP(storageRead); TRY_BEGIN() { result = lstMove(historyParse(strNewBuf(history)), memContextPrior()); } CATCH_ANY() { THROW_FMT( FormatError, "invalid timeline '%X' at '%s': %s", timeline, strZ(storagePathP(storageRepo, historyFile)), errorMessage()); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(LIST, result); } /*********************************************************************************************************************************** Find latest timeline ***********************************************************************************************************************************/ static unsigned int timelineLatest(const Storage *const storageRepo, const String *const archiveId, const unsigned int timelineCurrent) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storageRepo); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(UINT, timelineCurrent); FUNCTION_LOG_END(); ASSERT(storageRepo != NULL); unsigned int result = timelineCurrent; MEM_CONTEXT_TEMP_BEGIN() { const StringList *const historyList = strLstSort( storageListP( storageRepo, strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId)), .expression = STRDEF("[0-F]{8}\\.history")), sortOrderDesc); if (!strLstEmpty(historyList)) { const unsigned int timelineLatest = cvtZToUIntBase(strZ(strSubN(strLstGet(historyList, 0), 0, 8)), 16); if (timelineLatest > timelineCurrent) result = timelineLatest; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /**********************************************************************************************************************************/ STRING_STATIC(TIMELINE_CURRENT_STR, "current"); STRING_STATIC(TIMELINE_LATEST_STR, "latest"); FN_EXTERN void timelineVerify( const Storage *const storageRepo, const String *const archiveId, const unsigned int pgVersion, const unsigned int timelineBackup, const uint64_t lsnBackup, const String *timelineTargetStr, const unsigned int recoveryType, const CipherType cipherType, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storageRepo); FUNCTION_LOG_PARAM(STRING, archiveId); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT, timelineBackup); FUNCTION_LOG_PARAM(UINT64, lsnBackup); FUNCTION_LOG_PARAM(STRING, timelineTargetStr); FUNCTION_LOG_PARAM(UINT, recoveryType); FUNCTION_LOG_PARAM(STRING_ID, cipherType); FUNCTION_LOG_PARAM(STRING, cipherPass); FUNCTION_LOG_END(); ASSERT(storageRepo != NULL); ASSERT(archiveId != NULL); ASSERT(pgVersion != 0); ASSERT(timelineBackup > 0); ASSERT(lsnBackup > 0); // If timeline target is not specified then set based on the default by PostgreSQL version. Force timeline to current when the // recovery type is immediate since no timeline switch will be needed but PostgreSQL will error if there is a problem with the // latest timeline. if (timelineTargetStr == NULL) { if (pgVersion <= PG_VERSION_11 || recoveryType == CFGOPTVAL_RESTORE_TYPE_IMMEDIATE) timelineTargetStr = TIMELINE_CURRENT_STR; else timelineTargetStr = TIMELINE_LATEST_STR; } // If target timeline is not current then verify that the timeline is valid based on backup timeline and lsn. Following the // current/backup timeline is guaranteed as long as the WAL is available -- no history file required. if (!strEq(timelineTargetStr, TIMELINE_CURRENT_STR)) { MEM_CONTEXT_TEMP_BEGIN() { // Determine timeline target in order to load the history file. If target timeline is latest scan the history files to // find the most recent. If a target timeline is a number then parse it. unsigned int timelineTarget; if (strEq(timelineTargetStr, TIMELINE_LATEST_STR)) timelineTarget = timelineLatest(storageRepo, archiveId, timelineBackup); else { TRY_BEGIN() { if (strBeginsWithZ(timelineTargetStr, "0x")) timelineTarget = cvtZToUIntBase(strZ(timelineTargetStr) + 2, 16); else timelineTarget = cvtZToUInt(strZ(timelineTargetStr)); } CATCH_ANY() { THROW_FMT(DbMismatchError, "invalid target timeline '%s'", strZ(timelineTargetStr)); } TRY_END(); } // Only proceed if target timeline is not the backup timeline if (timelineTarget != timelineBackup) { // Search through the history for the target timeline to make sure it includes the backup timeline. This follows // the logic in PostgreSQL's readTimeLineHistory() and tliOfPointInHistory() but is a bit simplified for our usage. const List *const historyList = historyLoad(storageRepo, archiveId, timelineTarget, cipherType, cipherPass); unsigned int timelineFound = 0; for (unsigned int historyIdx = 0; historyIdx < lstSize(historyList); historyIdx++) { const HistoryItem *const historyItem = lstGet(historyList, historyIdx); // If backup timeline exists in the target timeline's history before the fork LSN then restore can proceed if (historyItem->timeline == timelineBackup && lsnBackup < historyItem->lsn) { timelineFound = timelineBackup; break; } } // Error when the timeline was not found or not found in the expected range if (timelineFound == 0) { uint64_t lsnFound = 0; // Check if timeline exists but did not fork off when expected for (unsigned int historyIdx = 0; historyIdx < lstSize(historyList); historyIdx++) { const HistoryItem *const historyItem = lstGet(historyList, historyIdx); if (historyItem->timeline == timelineBackup) { timelineFound = timelineBackup; lsnFound = historyItem->lsn; } } // The timeline was found but it forked off earlier than the backup LSN. This covers the common case where a // standby is promoted by accident, which creates a new timeline, and then the user tries to restore a backup // from the original primary that was made after the new timeline was created. Since PostgreSQL >= 12 will // always try to follow the latest timeline this restore will fail since there is no path to the target // timeline. if (timelineFound != 0) { ASSERT(lsnFound != 0); THROW_FMT( DbMismatchError, "target timeline %X forked from backup timeline %X at %s which is before backup lsn of %s\n" "HINT: was the target timeline created by accidentally promoting a standby?\n" "HINT: was the target timeline created by testing a restore without --archive-mode=off?\n" "HINT: was the backup made after the target timeline was created?", timelineTarget, timelineBackup, strZ(pgLsnToStr(lsnFound)), strZ(pgLsnToStr(lsnBackup))); } // Else the backup timeline was not found at all. This less common case occurs when, for example, a restore from // timeline 4 results in timeline 7 being created on promotion because 5 and 6 have already been created by // prior promotions and then the user tries to restore a backup from timeline 5 with a target of timeline 7. // There is no relationship between timeline 5 and 7 in this case at any LSN range. else { THROW_FMT( DbMismatchError, "backup timeline %X, lsn %s is not in the history of target timeline %X\n" "HINT: was the target timeline created by promoting from a timeline < latest?", timelineBackup, strZ(pgLsnToStr(lsnBackup)), timelineTarget); } } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/timeline.h0000664000175100017510000000201515240547630021260 0ustar00runnerrunner/*********************************************************************************************************************************** Timeline Management ***********************************************************************************************************************************/ #ifndef COMMAND_RESTORE_TIMELINE_H #define COMMAND_RESTORE_TIMELINE_H #include "common/crypto/common.h" #include "storage/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Verify that target timeline is valid for a backup FN_EXTERN void timelineVerify( const Storage *storageRepo, const String *archiveId, unsigned int pgVersion, unsigned int timelineBackup, uint64_t lsnBackup, const String *timelineTargetStr, unsigned int recoveryType, CipherType cipherType, const String *cipherPass); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/restore/validate.c.inc0000664000175100017510000000646015240547630022016 0ustar00runnerrunner/*********************************************************************************************************************************** Validate restore path ***********************************************************************************************************************************/ static void restorePathValidate(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // PostgreSQL must not be running if (storageExistsP(storagePg(), PG_FILE_POSTMTRPID_STR)) { THROW_FMT( PgRunningError, "unable to restore while PostgreSQL is running\n" "HINT: presence of '" PG_FILE_POSTMTRPID "' in '%s' indicates PostgreSQL is running.\n" "HINT: remove '" PG_FILE_POSTMTRPID "' only if PostgreSQL is not running.", strZ(cfgOptionDisplay(cfgOptPgPath))); } // If the restore will be destructive attempt to verify that PGDATA looks like a valid PostgreSQL directory if ((cfgOptionBool(cfgOptDelta) || cfgOptionBool(cfgOptForce)) && !storageExistsP(storagePg(), PG_FILE_PGVERSION_STR) && !storageExistsP(storagePg(), BACKUP_MANIFEST_FILE_STR)) { LOG_WARN_FMT( "--delta or --force specified but unable to find '" PG_FILE_PGVERSION "' or '" BACKUP_MANIFEST_FILE "' in '%s' to" " confirm that this is a valid $PGDATA directory. --delta and --force have been disabled and if any files exist" " in the destination directories the restore will be aborted.", strZ(cfgOptionDisplay(cfgOptPgPath))); // Disable delta and force so restore will fail if the directories are not empty cfgOptionSet(cfgOptDelta, cfgSourceDefault, BOOL_FALSE_VAR); cfgOptionSet(cfgOptForce, cfgSourceDefault, BOOL_FALSE_VAR); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Validate the manifest ***********************************************************************************************************************************/ static void restoreManifestValidate(const Manifest *const manifest, const String *const backupSet) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_PARAM(STRING, backupSet); FUNCTION_LOG_END(); ASSERT(manifest != NULL); ASSERT(backupSet != NULL); MEM_CONTEXT_TEMP_BEGIN() { // If there are no files in the manifest then something has gone horribly wrong CHECK(FormatError, manifestFileTotal(manifest) > 0, "manifest missing files"); // Sanity check to ensure the manifest has not been moved to a new directory const ManifestData *const data = manifestData(manifest); if (!strEq(data->backupLabel, backupSet)) { THROW_FMT( FormatError, "requested backup '%s' and manifest label '%s' do not match\n" "HINT: this indicates some sort of corruption (at the very least paths have been renamed).", strZ(backupSet), strZ(data->backupLabel)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/server/0000775000175100017510000000000015240547630017126 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/server/ping.c0000664000175100017510000000346415240547630020236 0ustar00runnerrunner/*********************************************************************************************************************************** Server Ping Command ***********************************************************************************************************************************/ #include #include "command/server/ping.h" #include "common/debug.h" #include "common/io/socket/client.h" #include "common/io/tls/client.h" #include "config/config.h" #include "protocol/client.h" #include "protocol/helper.h" FN_EXTERN void cmdServerPing(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { // Check for user-specified host const String *host = cfgOptionStr(cfgOptTlsServerAddress); const StringList *const commandParam = cfgCommandParam(); if (strLstSize(commandParam) == 1) host = strLstGet(commandParam, 0); else if (strLstSize(commandParam) > 1) THROW(ParamInvalidError, "extra parameters found"); // Connect to server without any verification const TimeMSec timeout = cfgOptionUInt64(cfgOptIoTimeout); IoClient *const tlsClient = tlsClientNewP( sckClientNew(host, cfgOptionUInt(cfgOptTlsServerPort), timeout, timeout), host, timeout, timeout, false); IoSession *const tlsSession = ioClientOpen(tlsClient); // Send ping ProtocolClient *const protocolClient = protocolClientNew( strNewFmt(PROTOCOL_SERVICE_REMOTE " socket protocol on '%s'", strZ(host)), PROTOCOL_SERVICE_REMOTE_STR, ioSessionIoReadP(tlsSession), ioSessionIoWrite(tlsSession)); protocolClientNoExit(protocolClient); protocolClientNoOp(protocolClient); protocolClientFree(protocolClient); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/server/ping.h0000664000175100017510000000123315240547630020233 0ustar00runnerrunner/*********************************************************************************************************************************** Server Ping Command ***********************************************************************************************************************************/ #ifndef COMMAND_SERVER_PING_H #define COMMAND_SERVER_PING_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void cmdServerPing(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/server/server.c0000664000175100017510000002023215240547630020577 0ustar00runnerrunner/*********************************************************************************************************************************** Server Command ***********************************************************************************************************************************/ #include #include #ifdef HAVE_LIBSYSTEMD #include #endif #include "command/exit.h" #include "command/remote/remote.h" #include "command/server/server.h" #include "common/debug.h" #include "common/fork.h" #include "common/io/socket/server.h" #include "common/io/tls/server.h" #include "config/config.h" #include "config/load.h" #include "protocol/helper.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct ServerLocal { MemContext *memContext; // Mem context for server unsigned int argListSize; // Argument list size const char **argList; // Argument list List *processList; // List of child processes bool sigHup; // SIGHUP was caught bool sigTerm; // SIGTERM was caught IoServer *socketServer; // Socket server IoServer *tlsServer; // TLS server } serverLocal; /*********************************************************************************************************************************** Initialization can be redone when options change ***********************************************************************************************************************************/ static void cmdServerInit(void) { // Initialize mem context if (serverLocal.memContext == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(Server, .childQty = MEM_CONTEXT_QTY_MAX) { serverLocal.memContext = MEM_CONTEXT_NEW(); serverLocal.processList = lstNewP(sizeof(pid_t)); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } MEM_CONTEXT_BEGIN(serverLocal.memContext) { // Free old servers ioServerFree(serverLocal.socketServer); ioServerFree(serverLocal.tlsServer); // Create new servers serverLocal.socketServer = sckServerNew( cfgOptionStr(cfgOptTlsServerAddress), cfgOptionUInt(cfgOptTlsServerPort), cfgOptionUInt64(cfgOptProtocolTimeout)); serverLocal.tlsServer = tlsServerNew( cfgOptionStr(cfgOptTlsServerAddress), cfgOptionStr(cfgOptTlsServerCaFile), cfgOptionStr(cfgOptTlsServerKeyFile), cfgOptionStr(cfgOptTlsServerCertFile), cfgOptionUInt64(cfgOptProtocolTimeout)); } MEM_CONTEXT_END(); } /*********************************************************************************************************************************** Handlers to set flags on signals ***********************************************************************************************************************************/ static void cmdServerSigHup(const int signalType) { (void)signalType; serverLocal.sigHup = true; } static void cmdServerSigTerm(const int signalType) { (void)signalType; serverLocal.sigTerm = true; } /*********************************************************************************************************************************** Handler to reap child processes ***********************************************************************************************************************************/ static void cmdServerSigChild(const int signalType, siginfo_t *const signalInfo, void *const context) { (void)signalType; (void)context; // Find the process and remove it for (unsigned int processIdx = 0; processIdx < lstSize(serverLocal.processList); processIdx++) { if (*(int *)lstGet(serverLocal.processList, processIdx) == signalInfo->si_pid) { ASSERT(signalInfo->si_code == CLD_EXITED); lstRemoveIdx(serverLocal.processList, processIdx); } } } /**********************************************************************************************************************************/ FN_EXTERN void cmdServer(const unsigned int argListSize, const char *argList[]) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, argListSize); FUNCTION_LOG_PARAM(CHARPY, argList); FUNCTION_LOG_END(); // Initialize server cmdServerInit(); // Set arguments used for reload serverLocal.argListSize = argListSize; serverLocal.argList = argList; MEM_CONTEXT_TEMP_BEGIN() { // Set signal handlers sigaction(SIGHUP, &(struct sigaction){.sa_handler = cmdServerSigHup}, NULL); sigaction(SIGTERM, &(struct sigaction){.sa_handler = cmdServerSigTerm}, NULL); sigaction( SIGCHLD, &(struct sigaction){.sa_sigaction = cmdServerSigChild, .sa_flags = SA_NOCLDSTOP | SA_NOCLDWAIT | SA_SIGINFO}, NULL); // Notify systemd that we are ready to accept connections #ifdef HAVE_LIBSYSTEMD sd_notify(0, "READY=1"); #endif // Accept connections indefinitely. The only way to exit this loop is for the process to receive a signal. do { // Accept a new connection IoSession *const socketSession = ioServerAccept(serverLocal.socketServer, NULL); if (socketSession != NULL) { // Fork off the child process pid_t pid = forkSafe(); if (pid == 0) { // Reset SIGCHLD to default sigaction(SIGCHLD, &(struct sigaction){.sa_handler = SIG_DFL}, NULL); // Set standard signal handlers exitInit(); // Close the server socket so we don't hold the port open if the parent exits first ioServerFree(serverLocal.socketServer); // Disable logging and close log file logClose(); // Start standard remote processing if a server is returned ProtocolServer *server = protocolServer(serverLocal.tlsServer, socketSession); if (server != NULL) cmdRemote(server); break; } // Add process to list else lstAdd(serverLocal.processList, &pid); // Free the socket since the child is now using it ioSessionFree(socketSession); } // Reload configuration if (serverLocal.sigHup) { LOG_DETAIL("configuration reload begin"); // Reload configuration cfgLoad(serverLocal.argListSize, serverLocal.argList); // Reinitialize server cmdServerInit(); LOG_DETAIL("configuration reload end"); // Reset flag serverLocal.sigHup = false; } } while (!serverLocal.sigTerm); } MEM_CONTEXT_TEMP_END(); // Notify systemd that we are shutting down #ifdef HAVE_LIBSYSTEMD sd_notify(0, "STOPPING=1"); #endif // Terminate any remaining children on SIGTERM. Disable the callback so it does not fire in the middle of the loop. if (serverLocal.sigTerm) { sigaction(SIGCHLD, &(struct sigaction){.sa_flags = SA_NOCLDSTOP | SA_NOCLDWAIT}, NULL); for (unsigned int processIdx = 0; processIdx < lstSize(serverLocal.processList); processIdx++) { pid_t pid = *(int *)lstGet(serverLocal.processList, processIdx); LOG_WARN_FMT("terminate child process %d", pid); kill(pid, SIGTERM); } } FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/server/server.h0000664000175100017510000000133015240547630020602 0ustar00runnerrunner/*********************************************************************************************************************************** Server Command ***********************************************************************************************************************************/ #ifndef COMMAND_SERVER_SERVER_H #define COMMAND_SERVER_SERVER_H #include /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void cmdServer(unsigned int argListSize, const char *argList[]); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/stanza/0000775000175100017510000000000015240547630017120 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/stanza/common.c0000664000175100017510000000453315240547630020561 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Commands Handler ***********************************************************************************************************************************/ #include #include "command/check/common.h" #include "command/stanza/common.h" #include "common/debug.h" #include "common/log.h" #include "config/config.h" #include "db/helper.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN String * cipherPassGen(const CipherType cipherType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, cipherType); FUNCTION_TEST_END(); String *result = NULL; if (cipherType != cipherTypeNone) { uint8_t buffer[48]; // 48 is the amount of entropy needed to get a 64 base key cryptoRandomBytes(buffer, sizeof(buffer)); result = strNewEncode(encodingBase64, BUF(buffer, sizeof(buffer))); } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN PgControl pgValidate(void) { FUNCTION_TEST_VOID(); PgControl result = {0}; MEM_CONTEXT_TEMP_BEGIN() { if (cfgOptionBool(cfgOptOnline)) { // Check the primary connections (and standby, if any) and return the primary database object. const DbGetResult dbObject = dbGet(false, true, CFGOPTVAL_BACKUP_STANDBY_N); // Get the pgControl information from the pg*-path deemed to be the primary result = dbPgControl(dbObject.primary); // Check the user configured path and version against the database checkDbConfig(dbPgControl(dbObject.primary).version, dbObject.primaryIdx, dbObject.primary, false); } // If the database is not online, assume that pg1 is the primary else result = pgControlFromFile(storagePg(), cfgOptionStrNull(cfgOptPgVersionForce)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_TYPE(PgControl, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/stanza/common.h0000664000175100017510000000153115240547630020561 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Commands Handler ***********************************************************************************************************************************/ #ifndef COMMAND_STANZA_COMMON_H #define COMMAND_STANZA_COMMON_H #include "info/infoPg.h" #include "postgres/interface.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Generate a cipher FN_EXTERN String *cipherPassGen(CipherType cipherType); // Validate and return database information FN_EXTERN PgControl pgValidate(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/stanza/create.c0000664000175100017510000002017515240547630020534 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Create Command ***********************************************************************************************************************************/ #include #include #include #include #include "command/check/common.h" #include "command/control/common.h" #include "command/stanza/common.h" #include "command/stanza/create.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStanzaCreate(void) { FUNCTION_LOG_VOID(logLevelDebug); // Verify that a stop was not issued before proceeding lockStopTest(); MEM_CONTEXT_TEMP_BEGIN() { if (cfgOptionBool(cfgOptForce)) LOG_WARN("option --" CFGOPT_FORCE " is no longer supported"); // Get the version and system information - validating it if the database is online const PgControl pgControl = pgValidate(); // For each repository configured, create the stanza for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT( CFGCMD_STANZA_CREATE " for stanza '%s' on %s", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); const Storage *const storageRepoReadStanza = storageRepoIdx(repoIdx); const Storage *const storageRepoWriteStanza = storageRepoIdxWrite(repoIdx); InfoArchive *infoArchive = NULL; InfoBackup *infoBackup = NULL; const bool archiveInfoFileExists = storageExistsP(storageRepoReadStanza, INFO_ARCHIVE_PATH_FILE_STR); const bool archiveInfoFileCopyExists = storageExistsP(storageRepoReadStanza, INFO_ARCHIVE_PATH_FILE_COPY_STR); const bool backupInfoFileExists = storageExistsP(storageRepoReadStanza, INFO_BACKUP_PATH_FILE_STR); const bool backupInfoFileCopyExists = storageExistsP(storageRepoReadStanza, INFO_BACKUP_PATH_FILE_COPY_STR); // If neither archive info nor backup info files exist and nothing else exists in the stanza directory // then create the stanza if (!archiveInfoFileExists && !archiveInfoFileCopyExists && !backupInfoFileExists && !backupInfoFileCopyExists) { const bool archiveNotEmpty = strLstSize( storageListP(storageRepoReadStanza, STORAGE_REPO_ARCHIVE_STR)) > 0 ? true : false; const bool backupNotEmpty = strLstSize( storageListP(storageRepoReadStanza, STORAGE_REPO_BACKUP_STR)) > 0 ? true : false; // If something else exists in the backup or archive directories for this stanza, then error if (archiveNotEmpty || backupNotEmpty) { THROW_FMT( PathNotEmptyError, "%s%s%snot empty", (backupNotEmpty ? "backup directory " : ""), (backupNotEmpty && archiveNotEmpty ? "and/or " : ""), (archiveNotEmpty ? "archive directory " : "")); } // If the repo is encrypted, generate a cipher passphrase for encrypting subsequent archive files const String *cipherPassSub = cipherPassGen(cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx)); // Create and save archive info infoArchive = infoArchiveNew(pgControl.version, pgControl.systemId, cipherPassSub); infoArchiveSaveFile( infoArchive, storageRepoWriteStanza, INFO_ARCHIVE_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); // If the repo is encrypted, generate a cipher passphrase for encrypting subsequent backup files cipherPassSub = cipherPassGen(cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx)); // Create and save backup info infoBackup = infoBackupNew(pgControl.version, pgControl.systemId, pgControl.catalogVersion, cipherPassSub); infoBackupSaveFile( infoBackup, storageRepoWriteStanza, INFO_BACKUP_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); } // Else if at least one archive and one backup info file exists, then ensure both are valid else if ((archiveInfoFileExists || archiveInfoFileCopyExists) && (backupInfoFileExists || backupInfoFileCopyExists)) { // Error if there is a mismatch between the archive and backup info files or the database version/system Id matches // current database checkStanzaInfoPg( storageRepoReadStanza, pgControl.version, pgControl.systemId, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); // The files are valid - upgrade const String *sourceFile = NULL; const String *destinationFile = NULL; // If the existing files are valid, then, if a file is missing, copy the existing one to the missing one to ensure // there is both a .info and .info.copy if (!archiveInfoFileExists || !archiveInfoFileCopyExists) { sourceFile = archiveInfoFileExists ? INFO_ARCHIVE_PATH_FILE_STR : INFO_ARCHIVE_PATH_FILE_COPY_STR; destinationFile = !archiveInfoFileExists ? INFO_ARCHIVE_PATH_FILE_STR : INFO_ARCHIVE_PATH_FILE_COPY_STR; // Using get and put instead of copy in case the storage is remote storagePutP( storageNewWriteP(storageRepoWriteStanza, destinationFile), storageGetP(storageNewReadP(storageRepoReadStanza, sourceFile))); } if (!backupInfoFileExists || !backupInfoFileCopyExists) { sourceFile = backupInfoFileExists ? INFO_BACKUP_PATH_FILE_STR : INFO_BACKUP_PATH_FILE_COPY_STR; destinationFile = !backupInfoFileExists ? INFO_BACKUP_PATH_FILE_STR : INFO_BACKUP_PATH_FILE_COPY_STR; // Using get and put instead of copy in case the storage is remote storagePutP( storageNewWriteP(storageRepoWriteStanza, destinationFile), storageGetP(storageNewReadP(storageRepoReadStanza, sourceFile))); } // If no files copied, then the stanza was already valid if (sourceFile == NULL) { LOG_INFO_FMT( "stanza '%s' already exists on %s and is valid", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); } } // Else if both .info and corresponding .copy file are missing for one but not the other, then error - the user will // have to make a conscious effort to determine if deleting the stanza on this repo is appropriate or other action is else { THROW_FMT( FileMissingError, "%s on %s\n" "HINT: this may be a symptom of repository corruption!", (archiveInfoFileExists || archiveInfoFileCopyExists) ? "archive.info exists but backup.info is missing" : "backup.info exists but archive.info is missing", cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/stanza/create.h0000664000175100017510000000127415240547630020540 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Create Command ***********************************************************************************************************************************/ #ifndef COMMAND_STANZA_CREATE_H #define COMMAND_STANZA_CREATE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process stanza-create FN_EXTERN void cmdStanzaCreate(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/stanza/delete.c0000664000175100017510000001066315240547630020534 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Delete Command ***********************************************************************************************************************************/ #include #include "command/backup/common.h" #include "command/control/common.h" #include "command/stanza/delete.h" #include "common/debug.h" #include "common/memContext.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStanzaDelete(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { const StringList *const archiveList = storageListP(storageRepo(), STORAGE_REPO_ARCHIVE_STR, .nullOnMissing = true); const StringList *const backupList = storageListP(storageRepo(), STORAGE_REPO_BACKUP_STR, .nullOnMissing = true); // For most drivers, NULL indicates the directory does not exist at all. For those that do not support paths (e.g. S3) an // empty StringList will be returned; in such a case, the directory will attempt to be deleted (this is OK). if (archiveList != NULL || backupList != NULL) { const bool archiveNotEmpty = (archiveList != NULL && !strLstEmpty(archiveList)) ? true : false; const bool backupNotEmpty = (backupList != NULL && !strLstEmpty(backupList)) ? true : false; // If something exists in either directory, then remove if (archiveNotEmpty || backupNotEmpty) { // If the stop file does not exist, then error. This check is required even when --force is issued. if (!storageExistsP(storageLocal(), lockStopFileName(cfgOptionStr(cfgOptStanza)))) { THROW_FMT( FileMissingError, "stop file does not exist for stanza '%s'\n" "HINT: has the pgbackrest stop command been run on this server for this stanza?", strZ(cfgOptionDisplay(cfgOptStanza))); } // If a force has not been issued and Postgres is running, then error if (!cfgOptionBool(cfgOptForce) && storageExistsP(storagePg(), STRDEF(PG_FILE_POSTMTRPID))) { THROW_FMT( PgRunningError, PG_FILE_POSTMTRPID " exists - looks like " PG_NAME " is running. " "To delete stanza '%s' on %s, shut down " PG_NAME " for stanza '%s' and try again, or use --force.", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, cfgOptionGroupIdxDefault(cfgOptGrpRepo)), strZ(cfgOptionDisplay(cfgOptStanza))); } // Delete the archive info files if (archiveNotEmpty) { storageRemoveP(storageRepoWrite(), INFO_ARCHIVE_PATH_FILE_STR); storageRemoveP(storageRepoWrite(), INFO_ARCHIVE_PATH_FILE_COPY_STR); } // Delete the backup info files if (backupNotEmpty) { storageRemoveP(storageRepoWrite(), INFO_BACKUP_PATH_FILE_STR); storageRemoveP(storageRepoWrite(), INFO_BACKUP_PATH_FILE_COPY_STR); } } // Recursively remove the entire stanza repo if exists. S3 will attempt to remove even if not. if (archiveList != NULL) storagePathRemoveP(storageRepoWrite(), STORAGE_REPO_ARCHIVE_STR, .recurse = true); if (backupList != NULL) storagePathRemoveP(storageRepoWrite(), STORAGE_REPO_BACKUP_STR, .recurse = true); // Remove the stop file - this will not error if the stop file does not exist. If the stanza directories existed but // nothing was in them, then no pgbackrest commands can be in progress without the info files so a stop is technically // not necessary storageRemoveP(storageLocalWrite(), lockStopFileName(cfgOptionStr(cfgOptStanza))); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/stanza/delete.h0000664000175100017510000000127415240547630020537 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Delete Command ***********************************************************************************************************************************/ #ifndef COMMAND_STANZA_DELETE_H #define COMMAND_STANZA_DELETE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process stanza-delete FN_EXTERN void cmdStanzaDelete(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/stanza/upgrade.c0000664000175100017510000001150415240547630020714 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Update Command ***********************************************************************************************************************************/ #include #include #include #include #include "command/check/common.h" #include "command/control/common.h" #include "command/stanza/common.h" #include "command/stanza/upgrade.h" #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN void cmdStanzaUpgrade(void) { FUNCTION_LOG_VOID(logLevelDebug); // Verify that a stop was not issued before proceeding lockStopTest(); MEM_CONTEXT_TEMP_BEGIN() { // Get the version and system information - validating it if the database is online PgControl pgControl = pgValidate(); // For each repository configured, upgrade the stanza for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { LOG_INFO_FMT( CFGCMD_STANZA_UPGRADE " for stanza '%s' on %s", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); const Storage *const storageRepoReadStanza = storageRepoIdx(repoIdx); const Storage *const storageRepoWriteStanza = storageRepoIdxWrite(repoIdx); bool infoArchiveUpgrade = false; bool infoBackupUpgrade = false; // Load the info files (errors if missing) InfoArchive *const infoArchive = infoArchiveLoadFile( storageRepoReadStanza, INFO_ARCHIVE_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); InfoPgData archiveInfo = infoPgData(infoArchivePg(infoArchive), infoPgDataCurrentId(infoArchivePg(infoArchive))); InfoBackup *const infoBackup = infoBackupLoadFile( storageRepoReadStanza, INFO_BACKUP_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); InfoPgData backupInfo = infoPgData(infoBackupPg(infoBackup), infoPgDataCurrentId(infoBackupPg(infoBackup))); // Since the file save of archive.info and backup.info are not atomic, then check and update each separately. // Update archive if (pgControl.version != archiveInfo.version || pgControl.systemId != archiveInfo.systemId) { infoArchivePgSet(infoArchive, pgControl.version, pgControl.systemId); infoArchiveUpgrade = true; } // Update backup if (pgControl.version != backupInfo.version || pgControl.systemId != backupInfo.systemId) { infoBackupPgSet(infoBackup, pgControl.version, pgControl.systemId, pgControl.catalogVersion); infoBackupUpgrade = true; } // Get the backup and archive info pg data and throw an error if the ids do not match before saving (even if only one // needed to be updated) backupInfo = infoPgData(infoBackupPg(infoBackup), infoPgDataCurrentId(infoBackupPg(infoBackup))); archiveInfo = infoPgData(infoArchivePg(infoArchive), infoPgDataCurrentId(infoArchivePg(infoArchive))); checkStanzaInfo(&archiveInfo, &backupInfo); // Save archive info if (infoArchiveUpgrade) { infoArchiveSaveFile( infoArchive, storageRepoWriteStanza, INFO_ARCHIVE_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); } // Save backup info if (infoBackupUpgrade) { infoBackupSaveFile( infoBackup, storageRepoWriteStanza, INFO_BACKUP_PATH_FILE_STR, cfgOptionIdxStrId(cfgOptRepoCipherType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoCipherPass, repoIdx)); } if (!(infoArchiveUpgrade || infoBackupUpgrade)) { LOG_INFO_FMT( "stanza '%s' on %s is already up to date", strZ(cfgOptionDisplay(cfgOptStanza)), cfgOptionGroupName(cfgOptGrpRepo, repoIdx)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/stanza/upgrade.h0000664000175100017510000000130115240547630020713 0ustar00runnerrunner/*********************************************************************************************************************************** Stanza Upgrade Command ***********************************************************************************************************************************/ #ifndef COMMAND_STANZA_UPGRADE_H #define COMMAND_STANZA_UPGRADE_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process stanza-upgrade FN_EXTERN void cmdStanzaUpgrade(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/verify/0000775000175100017510000000000015240547630017124 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/verify/file.c0000664000175100017510000000672215240547630020216 0ustar00runnerrunner/*********************************************************************************************************************************** Verify File ***********************************************************************************************************************************/ #include #include "command/verify/file.h" #include "common/crypto/cipherBlock.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/filter/group.h" #include "common/io/filter/sink.h" #include "common/io/filter/size.h" #include "common/io/io.h" #include "common/log.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN VerifyResult verifyFile( const String *const filePathName, const uint64_t offset, const Variant *const limit, const CompressType compressType, const Buffer *const fileChecksum, const uint64_t fileSize, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, filePathName); // Fully qualified file name FUNCTION_LOG_PARAM(UINT64, offset); // Offset to read in file FUNCTION_LOG_PARAM(VARIANT, limit); // Limit to read from file FUNCTION_LOG_PARAM(ENUM, compressType); // Compression type FUNCTION_LOG_PARAM(BUFFER, fileChecksum); // Checksum for the file FUNCTION_LOG_PARAM(UINT64, fileSize); // Size of file FUNCTION_TEST_PARAM(STRING, cipherPass); // Password to access the repo file if encrypted FUNCTION_LOG_END(); ASSERT(filePathName != NULL); ASSERT(fileChecksum != NULL); ASSERT(limit == NULL || varType(limit) == varTypeUInt64); // Is the file valid? VerifyResult result = verifyOk; MEM_CONTEXT_TEMP_BEGIN() { // Prepare the file for reading IoRead *const read = storageReadIo( storageNewReadP(storageRepo(), filePathName, .ignoreMissing = true, .offset = offset, .limit = limit)); IoFilterGroup *const filterGroup = ioReadFilterGroup(read); // Add decryption filter if (cipherPass != NULL) ioFilterGroupAdd(filterGroup, cipherBlockNewP(cipherModeDecrypt, cipherTypeAes256Cbc, BUFSTR(cipherPass))); // Add decompression filter if (compressType != compressTypeNone) ioFilterGroupAdd(filterGroup, decompressFilterP(compressType)); // Add sha1 filter ioFilterGroupAdd(filterGroup, cryptoHashNew(hashTypeSha1)); // Add size filter ioFilterGroupAdd(filterGroup, ioSizeNew()); // Add IoSink so the file data is not transmitted from the remote ioFilterGroupAdd(filterGroup, ioSinkNew()); // If the file exists check the checksum/size if (ioReadDrain(read)) { // Validate checksum if (!bufEq(fileChecksum, pckReadBinP(ioFilterGroupResultP(filterGroup, CRYPTO_HASH_FILTER_TYPE)))) { result = verifyChecksumMismatch; } // If the size can be checked, do so else if (fileSize != pckReadU64P(ioFilterGroupResultP(ioReadFilterGroup(read), SIZE_FILTER_TYPE))) result = verifySizeInvalid; } else result = verifyFileMissing; } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/verify/file.h0000664000175100017510000000265515240547630020224 0ustar00runnerrunner/*********************************************************************************************************************************** Verify File ***********************************************************************************************************************************/ #ifndef COMMAND_VERIFY_FILE_H #define COMMAND_VERIFY_FILE_H #include "common/compress/helper.h" #include "common/crypto/common.h" /*********************************************************************************************************************************** File result ***********************************************************************************************************************************/ typedef enum { verifyOk, // Default result - file OK verifyFileMissing, verifyChecksumMismatch, verifySizeInvalid, verifyOtherError, } VerifyResult; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Verify a file in the pgBackRest repository FN_EXTERN VerifyResult verifyFile( const String *filePathName, uint64_t offset, const Variant *limit, CompressType compressType, const Buffer *fileChecksum, uint64_t fileSize, const String *cipherPass); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/verify/protocol.c0000664000175100017510000000346515240547630021141 0ustar00runnerrunner/*********************************************************************************************************************************** Verify Protocol Handler ***********************************************************************************************************************************/ #include #include "command/verify/file.h" #include "command/verify/protocol.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "config/config.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * verifyFileProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Verify file const String *const filePathName = pckReadStrP(param); uint64_t offset = 0; const Variant *limit = NULL; if (pckReadBoolP(param)) { offset = pckReadU64P(param); limit = varNewUInt64(pckReadU64P(param)); } const CompressType compressType = (CompressType)pckReadU32P(param); const Buffer *const fileChecksum = pckReadBinP(param); const uint64_t fileSize = pckReadU64P(param); const String *const cipherPass = pckReadStrP(param); // Return result pckWriteU32P( protocolServerResultData(result), verifyFile(filePathName, offset, limit, compressType, fileChecksum, fileSize, cipherPass)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/verify/protocol.h0000664000175100017510000000266215240547630021144 0ustar00runnerrunner/*********************************************************************************************************************************** Verify Protocol Handler ***********************************************************************************************************************************/ #ifndef COMMAND_VERIFY_PROTOCOL_H #define COMMAND_VERIFY_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process protocol requests FN_EXTERN ProtocolServerResult *verifyFileProtocol(PackRead *param); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_VERIFY_FILE STRID5("vf-f", 0x36cd60) #define PROTOCOL_SERVER_HANDLER_VERIFY_LIST \ {.command = PROTOCOL_COMMAND_VERIFY_FILE, .process = verifyFileProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/verify/verify.c0000664000175100017510000026500215240547630020601 0ustar00runnerrunner/*********************************************************************************************************************************** Verify Command Verify contents of the repository. ***********************************************************************************************************************************/ #include #include #include #include #include "command/archive/common.h" #include "command/check/common.h" #include "command/verify/file.h" #include "command/verify/protocol.h" #include "command/verify/verify.h" #include "common/compress/helper.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/log.h" #include "common/regExp.h" #include "config/config.h" #include "info/infoArchive.h" #include "info/infoBackup.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "protocol/parallel.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define VERIFY_STATUS_OK "ok" #define VERIFY_STATUS_ERROR "error" /*********************************************************************************************************************************** Data Types and Structures ***********************************************************************************************************************************/ #define FUNCTION_LOG_VERIFY_ARCHIVE_RESULT_TYPE \ VerifyArchiveResult #define FUNCTION_LOG_VERIFY_ARCHIVE_RESULT_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "VerifyArchiveResult", buffer, bufferSize) #define FUNCTION_LOG_VERIFY_BACKUP_RESULT_TYPE \ VerifyBackupResult #define FUNCTION_LOG_VERIFY_BACKUP_RESULT_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "VerifyBackupResult", buffer, bufferSize) // Structure for verifying repository info files typedef struct VerifyInfoFile { InfoBackup *backup; // Backup.info file contents InfoArchive *archive; // Archive.info file contents Manifest *manifest; // Manifest file contents const String *checksum; // File checksum int errorCode; // Error code else 0 for no error } VerifyInfoFile; // Job data results structures for archive and backup typedef struct VerifyArchiveResult { String *archiveId; // Archive Id (e.g. 9.6-1, 10-2) unsigned int totalWalFile; // Total number of WAL files listed in directory on first read unsigned int totalValidWal; // Total number of WAL that were verified and valid PgWal pgWalInfo; // PG version, WAL size, system id List *walRangeList; // List of WAL file ranges - new item is when WAL is missing } VerifyArchiveResult; // WAL range includes the start/stop of sequential WAL and start/stop includes the timeline (e.g. 000000020000000100000005) typedef struct VerifyWalRange { String *stop; // Last WAL segment in this sequential range String *start; // First WAL segment in this sequential range List *invalidFileList; // After all jobs complete, list of VerifyInvalidFile } VerifyWalRange; // Invalid file information (not missing but files failing verification) - for archive and backup typedef struct VerifyInvalidFile { String *fileName; // Name of the file (includes path within the stanza) VerifyResult reason; // Reason file is invalid (e.g. incorrect checksum) } VerifyInvalidFile; // Status result of a backup typedef enum { backupValid, // Default: All files in backup label repo passed verification backupInvalid, // One of more files in backup label repo failed verification backupMissingManifest, // Backup manifest missing (backup may have expired) backupInProgress, // Backup appeared to be in progress (so was skipped) } VerifyBackupResultStatus; typedef struct VerifyBackupResult { String *backupLabel; // Label assigned to the backup VerifyBackupResultStatus status; // Final status of the backup bool fileVerifyComplete; // Have all the files of the backup completed verification? unsigned int totalFileManifest; // Total number of backup files in the manifest unsigned int totalFileVerify; // Total number of backup files being verified unsigned int totalFileValid; // Total number of backup files that were verified and valid String *backupPrior; // Prior backup that this backup depends on, if any unsigned int pgId; // PG id will be used to find WAL for the backup in the repo unsigned int pgVersion; // PG version will be used with PG id to find WAL in the repo String *archiveStart; // First WAL segment in the backup String *archiveStop; // Last WAL segment in the backup List *invalidFileList; // List of invalid files found in the backup } VerifyBackupResult; // Job data structure for processing and results collection typedef struct VerifyJobData { MemContext *memContext; // Context for memory allocations in this struct StringList *archiveIdList; // List of archive ids to verify StringList *walPathList; // WAL path list for a single archive id StringList *walFileList; // WAL file list for a single WAL path StringList *backupList; // List of backups to verify Manifest *manifest; // Manifest contents with list of files to verify unsigned int manifestFileIdx; // Index of the file within the manifest file list to process String *currentBackup; // In progress backup, if any const InfoPg *pgHistory; // Database history list bool backupProcessing; // Are we processing WAL or are we processing backups const String *manifestCipherPass; // Cipher pass for reading backup manifests const String *walCipherPass; // Cipher pass for reading WAL files const String *backupCipherPass; // Cipher pass for reading backup files referenced in a manifest unsigned int jobErrorTotal; // Total errors that occurred during the job execution List *archiveIdResultList; // Archive results List *backupResultList; // Backup results bool enableArchiveFilter; // Only check archives in the specified range const String *archiveStart; // Start of the WAL range to be verified const String *archiveStop; // End of the WAL range to be verified } VerifyJobData; /*********************************************************************************************************************************** Helper function to add a file to an invalid file list ***********************************************************************************************************************************/ static void verifyInvalidFileAdd(List *const invalidFileList, const VerifyResult reason, const String *const fileName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, invalidFileList); // Invalid file list to add the filename to FUNCTION_TEST_PARAM(ENUM, reason); // Reason for invalid file FUNCTION_TEST_PARAM(STRING, fileName); // Name of invalid file FUNCTION_TEST_END(); ASSERT(invalidFileList != NULL); ASSERT(fileName != NULL); MEM_CONTEXT_BEGIN(lstMemContext(invalidFileList)) { const VerifyInvalidFile invalidFile = { .fileName = strDup(fileName), .reason = reason, }; lstAdd(invalidFileList, &invalidFile); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Load a file into memory ***********************************************************************************************************************************/ static StorageRead * verifyFileLoad(const String *const pathFileName, const String *const cipherPass) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, pathFileName); // Fully qualified path/file name FUNCTION_TEST_PARAM(STRING, cipherPass); // Password to open file if encrypted FUNCTION_TEST_END(); ASSERT(pathFileName != NULL); // Read the file and error if missing StorageRead *const result = storageNewReadP(storageRepo(), pathFileName); // *read points to a location within result so update result with contents based on necessary filters IoRead *const read = storageReadIo(result); cipherBlockFilterGroupAdd( ioReadFilterGroup(read), cfgOptionStrId(cfgOptRepoCipherType), cipherModeDecrypt, cipherPass); ioFilterGroupAdd(ioReadFilterGroup(read), cryptoHashNew(hashTypeSha1)); // If the file is compressed, add a decompression filter if (compressTypeFromName(pathFileName) != compressTypeNone) ioFilterGroupAdd(ioReadFilterGroup(read), decompressFilterP(compressTypeFromName(pathFileName))); FUNCTION_TEST_RETURN(STORAGE_READ, result); } /*********************************************************************************************************************************** Get status of info files in the repository ***********************************************************************************************************************************/ static VerifyInfoFile verifyInfoFile(const String *const pathFileName, const bool keepFile, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, pathFileName); // Fully qualified path/file name FUNCTION_LOG_PARAM(BOOL, keepFile); // Should the file be kept in memory? FUNCTION_TEST_PARAM(STRING, cipherPass); // Password to open file if encrypted FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(pathFileName != NULL); VerifyInfoFile result = {.errorCode = 0}; MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { IoRead *const infoRead = storageReadIo(verifyFileLoad(pathFileName, cipherPass)); // If directed to keep the loaded file in memory, then move the file into the result, else drain the io and close it if (keepFile) { if (strBeginsWith(pathFileName, INFO_BACKUP_PATH_FILE_STR)) result.backup = infoBackupMove(infoBackupNewLoad(infoRead), memContextPrior()); else if (strBeginsWith(pathFileName, INFO_ARCHIVE_PATH_FILE_STR)) result.archive = infoArchiveMove(infoArchiveNewLoad(infoRead), memContextPrior()); else result.manifest = manifestMove(manifestNewLoad(infoRead), memContextPrior()); } else ioReadDrain(infoRead); const Buffer *const filterResult = pckReadBinP( ioFilterGroupResultP(ioReadFilterGroup(infoRead), CRYPTO_HASH_FILTER_TYPE)); MEM_CONTEXT_PRIOR_BEGIN() { result.checksum = strNewEncode(encodingHex, filterResult); } MEM_CONTEXT_PRIOR_END(); } CATCH_ANY() { result.errorCode = errorCode(); String *const errorMsg = strCatZ(strNew(), errorMessage()); if (result.errorCode == errorTypeCode(&ChecksumError)) strCat(errorMsg, strNewFmt(" %s", strZ(pathFileName))); LOG_DETAIL(strZ(errorMsg)); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /*********************************************************************************************************************************** Get the archive.info file ***********************************************************************************************************************************/ static InfoArchive * verifyArchiveInfoFile(void) { FUNCTION_LOG_VOID(logLevelDebug); InfoArchive *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get the main info file const VerifyInfoFile verifyArchiveInfo = verifyInfoFile( INFO_ARCHIVE_PATH_FILE_STR, true, cfgOptionStrNull(cfgOptRepoCipherPass)); // If the main file did not error, then report on the copy's status and check checksums if (verifyArchiveInfo.errorCode == 0) { result = verifyArchiveInfo.archive; infoArchiveMove(result, memContextPrior()); // Attempt to load the copy and report on it's status but don't keep it in memory const VerifyInfoFile verifyArchiveInfoCopy = verifyInfoFile( INFO_ARCHIVE_PATH_FILE_COPY_STR, false, cfgOptionStrNull(cfgOptRepoCipherPass)); // If the copy loaded successfully, then check the checksums if (verifyArchiveInfoCopy.errorCode == 0) { // If the info and info.copy checksums don't match each other than one (or both) of the files could be corrupt so // log a warning but must trust main if (!strEq(verifyArchiveInfo.checksum, verifyArchiveInfoCopy.checksum)) LOG_DETAIL("archive.info.copy does not match archive.info"); } } else { // Attempt to load the copy const VerifyInfoFile verifyArchiveInfoCopy = verifyInfoFile( INFO_ARCHIVE_PATH_FILE_COPY_STR, true, cfgOptionStrNull(cfgOptRepoCipherPass)); // If loaded successfully, then return the copy as usable if (verifyArchiveInfoCopy.errorCode == 0) { result = verifyArchiveInfoCopy.archive; infoArchiveMove(result, memContextPrior()); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_ARCHIVE, result); } /*********************************************************************************************************************************** Get the backup.info file ***********************************************************************************************************************************/ static InfoBackup * verifyBackupInfoFile(void) { FUNCTION_LOG_VOID(logLevelDebug); InfoBackup *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get the main info file const VerifyInfoFile verifyBackupInfo = verifyInfoFile( INFO_BACKUP_PATH_FILE_STR, true, cfgOptionStrNull(cfgOptRepoCipherPass)); // If the main file did not error, then report on the copy's status and check checksums if (verifyBackupInfo.errorCode == 0) { result = verifyBackupInfo.backup; infoBackupMove(result, memContextPrior()); // Attempt to load the copy and report on it's status but don't keep it in memory const VerifyInfoFile verifyBackupInfoCopy = verifyInfoFile( INFO_BACKUP_PATH_FILE_COPY_STR, false, cfgOptionStrNull(cfgOptRepoCipherPass)); // If the copy loaded successfully, then check the checksums if (verifyBackupInfoCopy.errorCode == 0) { // If the info and info.copy checksums don't match each other than one (or both) of the files could be corrupt so // log a warning but must trust main if (!strEq(verifyBackupInfo.checksum, verifyBackupInfoCopy.checksum)) LOG_DETAIL("backup.info.copy does not match backup.info"); } } else { // Attempt to load the copy const VerifyInfoFile verifyBackupInfoCopy = verifyInfoFile( INFO_BACKUP_PATH_FILE_COPY_STR, true, cfgOptionStrNull(cfgOptRepoCipherPass)); // If loaded successfully, then return the copy as usable if (verifyBackupInfoCopy.errorCode == 0) { result = verifyBackupInfoCopy.backup; infoBackupMove(result, memContextPrior()); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, result); } /*********************************************************************************************************************************** Get the manifest file ***********************************************************************************************************************************/ static Manifest * verifyManifestFile( VerifyBackupResult *const backupResult, const String *const cipherPass, bool currentBackup, const InfoPg *const pgHistory, unsigned int *const jobErrorTotal) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM_P(VERIFY_BACKUP_RESULT, backupResult); // The result set for the backup being processed FUNCTION_TEST_PARAM(STRING, cipherPass); // Passphrase to access the manifest file FUNCTION_LOG_PARAM(BOOL, currentBackup); // Is this possibly a backup currently in progress? FUNCTION_LOG_PARAM(INFO_PG, pgHistory); // Database history FUNCTION_LOG_PARAM_P(UINT, jobErrorTotal); // Pointer to the overall job error total FUNCTION_LOG_END(); Manifest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { const String *const fileName = strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupResult->backupLabel)); // Get the main manifest file const VerifyInfoFile verifyManifestInfo = verifyInfoFile(fileName, true, cipherPass); // If the main file did not error, then report on the copy's status and check checksums if (verifyManifestInfo.errorCode == 0) { result = verifyManifestInfo.manifest; // The current in-progress backup is only notional until the main file is checked because the backup may have // completed by the time the main manifest is checked here. So having a main manifest file means this backup is not // (or is no longer) the currentBackup. currentBackup = false; // Attempt to load the copy and report on it's status but don't keep it in memory const VerifyInfoFile verifyManifestInfoCopy = verifyInfoFile( strNewFmt("%s%s", strZ(fileName), INFO_COPY_EXT), false, cipherPass); // If the copy loaded successfully, then check the checksums if (verifyManifestInfoCopy.errorCode == 0) { // If the manifest and manifest.copy checksums don't match each other than one (or both) of the files could be // corrupt so log a warning but trust main if (!strEq(verifyManifestInfo.checksum, verifyManifestInfoCopy.checksum)) LOG_DETAIL_FMT("backup '%s' manifest.copy does not match manifest", strZ(backupResult->backupLabel)); } } else { // If this might be an in-progress backup and the main manifest is simply missing, it is assumed the backup is an // actual in-progress backup and verification is skipped, otherwise, if the main is not simply missing, or this is not // an in-progress backup then attempt to load the copy. if (!(currentBackup && verifyManifestInfo.errorCode == errorTypeCode(&FileMissingError))) { currentBackup = false; const VerifyInfoFile verifyManifestInfoCopy = verifyInfoFile( strNewFmt("%s%s", strZ(fileName), INFO_COPY_EXT), true, cipherPass); // If loaded successfully, then return the copy as usable if (verifyManifestInfoCopy.errorCode == 0) { LOG_DETAIL_FMT("%s/backup.manifest is missing or unusable, using copy", strZ(backupResult->backupLabel)); result = verifyManifestInfoCopy.manifest; } else if (verifyManifestInfo.errorCode == errorTypeCode(&FileMissingError) && verifyManifestInfoCopy.errorCode == errorTypeCode(&FileMissingError)) { backupResult->status = backupMissingManifest; LOG_DETAIL_FMT("manifest missing for '%s' - backup may have expired", strZ(backupResult->backupLabel)); } } else { backupResult->status = backupInProgress; LOG_INFO_FMT("backup '%s' appears to be in progress, skipping", strZ(backupResult->backupLabel)); } } // If found a usable manifest then check that the database it was based on is in the history if (result != NULL) { bool found = false; const ManifestData *const manData = manifestData(result); // Confirm the PG database information from the manifest is in the history list for (unsigned int infoPgIdx = 0; infoPgIdx < infoPgDataTotal(pgHistory); infoPgIdx++) { const InfoPgData pgHistoryData = infoPgData(pgHistory, infoPgIdx); if (pgHistoryData.id == manData->pgId && pgHistoryData.systemId == manData->pgSystemId && pgHistoryData.version == manData->pgVersion) { found = true; break; } } // If the PG data is not found in the backup.info history, then error and reset the result if (!found) { LOG_INFO_FMT( "'%s' may not be recoverable - PG data (id %u, version %s, system-id %" PRIu64 ") is not in the backup.info" " history, skipping", strZ(backupResult->backupLabel), manData->pgId, strZ(pgVersionToStr(manData->pgVersion)), manData->pgSystemId); manifestFree(result); result = NULL; } else manifestMove(result, memContextPrior()); } // If the result is NULL and the backup status has not yet been set, then the backup is unusable (invalid) if (result == NULL && backupResult->status == backupValid) { backupResult->status = backupInvalid; (*jobErrorTotal)++; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(MANIFEST, result); } /*********************************************************************************************************************************** Check the history in the info files ***********************************************************************************************************************************/ static void verifyPgHistory(const InfoPg *const archiveInfoPg, const InfoPg *const backupInfoPg) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_PG, archiveInfoPg); // Postgres information from the archive.info file FUNCTION_TEST_PARAM(INFO_PG, backupInfoPg); // Postgres information from the backup.info file FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // Check archive.info and backup.info current PG data matches. If there is a mismatch, verify cannot continue since // the database is not considered accessible during the verify command so no way to tell which would be valid. const InfoPgData archiveInfoPgData = infoPgData(archiveInfoPg, infoPgDataCurrentId(archiveInfoPg)); const InfoPgData backupInfoPgData = infoPgData(backupInfoPg, infoPgDataCurrentId(backupInfoPg)); checkStanzaInfo(&archiveInfoPgData, &backupInfoPgData); const unsigned int archiveInfoHistoryTotal = infoPgDataTotal(archiveInfoPg); const unsigned int backupInfoHistoryTotal = infoPgDataTotal(backupInfoPg); String *const errMsg = strNewZ("archive and backup history lists do not match"); if (archiveInfoHistoryTotal != backupInfoHistoryTotal) THROW(FormatError, strZ(errMsg)); // Confirm the lists are the same for (unsigned int infoPgIdx = 0; infoPgIdx < archiveInfoHistoryTotal; infoPgIdx++) { const InfoPgData archiveInfoPgHistory = infoPgData(archiveInfoPg, infoPgIdx); const InfoPgData backupInfoPgHistory = infoPgData(backupInfoPg, infoPgIdx); if (archiveInfoPgHistory.id != backupInfoPgHistory.id || archiveInfoPgHistory.systemId != backupInfoPgHistory.systemId || archiveInfoPgHistory.version != backupInfoPgHistory.version) { THROW(FormatError, strZ(errMsg)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Populate the WAL ranges from the provided, sorted, WAL files list for a given archiveId ***********************************************************************************************************************************/ static void verifyCreateArchiveIdRange( const VerifyArchiveResult *const archiveIdResult, StringList *const walFileList, unsigned int *const jobErrorTotal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VERIFY_ARCHIVE_RESULT, archiveIdResult); // The result set for the archive Id being processed FUNCTION_TEST_PARAM(STRING_LIST, walFileList); // Sorted (ascending) list of WAL files in a timeline FUNCTION_TEST_PARAM_P(UINT, jobErrorTotal); // Pointer to the overall job error total FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(archiveIdResult != NULL); ASSERT(walFileList != NULL); unsigned int walFileIdx = 0; // Initialize the WAL range VerifyWalRange *walRange = NULL; // If there is a WAL range for this archiveID, get the last one. If there is no timeline change then continue updating the last // WAL range. if (!lstEmpty(archiveIdResult->walRangeList) && strEq( strSubN(((VerifyWalRange *)lstGetLast(archiveIdResult->walRangeList))->stop, 0, 8), strSubN(strSubN(strLstGet(walFileList, walFileIdx), 0, WAL_SEGMENT_NAME_SIZE), 0, 8))) { walRange = lstGetLast(archiveIdResult->walRangeList); } do { const String *const walSegment = strSubN(strLstGet(walFileList, walFileIdx), 0, WAL_SEGMENT_NAME_SIZE); // The lists are sorted so look ahead to see if this is a duplicate of the next one in the list if (walFileIdx + 1 < strLstSize(walFileList)) { if (strEq(walSegment, strSubN(strLstGet(walFileList, walFileIdx + 1), 0, WAL_SEGMENT_NAME_SIZE))) { LOG_INFO_FMT("duplicate WAL '%s' for '%s' exists, skipping", strZ(walSegment), strZ(archiveIdResult->archiveId)); (*jobErrorTotal)++; bool foundDup = true; // Remove all duplicates of this WAL, including this WAL, from the list while (walFileIdx < strLstSize(walFileList) && foundDup) { if (strEq(walSegment, strSubN(strLstGet(walFileList, walFileIdx), 0, WAL_SEGMENT_NAME_SIZE))) strLstRemoveIdx(walFileList, walFileIdx); else foundDup = false; } continue; } } // Initialize the range if it has not yet been initialized and continue to next if (walRange == NULL || !strEq( walSegmentNext(walRange->stop, (size_t)archiveIdResult->pgWalInfo.size, archiveIdResult->pgWalInfo.version), walSegment)) { // Add the initialized wal range to the range list MEM_CONTEXT_BEGIN(lstMemContext(archiveIdResult->walRangeList)) { const VerifyWalRange walRangeNew = { .start = strDup(walSegment), .stop = strDup(walSegment), .invalidFileList = lstNewP(sizeof(VerifyInvalidFile), .comparator = lstComparatorStr), }; lstAdd(archiveIdResult->walRangeList, &walRangeNew); } MEM_CONTEXT_END(); // Set the current wal range being processed to what was just added walRange = lstGetLast(archiveIdResult->walRangeList); } // If the next WAL is the appropriate distance away, then there is no gap else { MEM_CONTEXT_BEGIN(lstMemContext(archiveIdResult->walRangeList)) { strFree(walRange->stop); walRange->stop = strDup(walSegment); } MEM_CONTEXT_END(); } walFileIdx++; } while (walFileIdx < strLstSize(walFileList)); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Check if backup is block incremental and populate backup list from its references and set the WAL range to verify based on WAL required to make the backup consistent. ***********************************************************************************************************************************/ static void verifyBackupSet(VerifyJobData *const jobData, const String *const backupLabel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, jobData); // Pointer to the job data FUNCTION_TEST_PARAM(STRING, backupLabel); // Label of backup to check FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { const Manifest *const manifest = manifestLoadFile( storageRepo(), strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)), cfgOptionStrId(cfgOptRepoCipherType), jobData->manifestCipherPass); // Check files for block incremental bool hasBlockIncr = false; for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); if (file.blockIncrMapSize != 0) { hasBlockIncr = true; break; } } // Block incremental backups can depend on any referenced backups through block maps which means we have to verify all // referenced backups as well. ??? Make this more efficient by verifying only required blocks. if (hasBlockIncr) { const StringList *const referenceList = manifestReferenceList(manifest); MEM_CONTEXT_BEGIN(jobData->memContext) { for (unsigned int referenceIdx = 0; referenceIdx < strLstSize(referenceList); referenceIdx++) strLstAddIfMissing(jobData->backupList, strLstGet(referenceList, referenceIdx)); strLstSort(jobData->backupList, sortOrderAsc); } MEM_CONTEXT_END(); } // Save WAL range required to make the backup consistent MEM_CONTEXT_BEGIN(jobData->memContext) { const ManifestData *const manData = manifestData(manifest); jobData->archiveStart = strDup(manData->archiveStart); jobData->archiveStop = strDup(manData->archiveStop); } MEM_CONTEXT_END(); } CATCH_ANY() { jobData->archiveStart = NULL; jobData->archiveStop = NULL; } TRY_END(); jobData->enableArchiveFilter = true; } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Return verify jobs for the archive ***********************************************************************************************************************************/ static ProtocolParallelJob * verifyArchive(VerifyJobData *const jobData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, jobData); FUNCTION_TEST_END(); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Process archive files, if any while (!strLstEmpty(jobData->archiveIdList)) { // Add archiveId to the result list if the list is empty or the last processed is not equal to the current archiveId if (lstEmpty(jobData->archiveIdResultList) || !strEq( ((VerifyArchiveResult *)lstGetLast(jobData->archiveIdResultList))->archiveId, strLstGet(jobData->archiveIdList, 0))) { const String *const archiveId = strLstGet(jobData->archiveIdList, 0); MEM_CONTEXT_BEGIN(lstMemContext(jobData->archiveIdResultList)) { const VerifyArchiveResult archiveIdResult = { .archiveId = strDup(archiveId), .walRangeList = lstNewP(sizeof(VerifyWalRange), .comparator = lstComparatorStr), }; lstAdd(jobData->archiveIdResultList, &archiveIdResult); } MEM_CONTEXT_END(); // Free the old WAL path list strLstFree(jobData->walPathList); // Get the WAL paths for the archive Id const String *const archiveIdPath = strNewFmt(STORAGE_REPO_ARCHIVE "/%s", strZ(archiveId)); MEM_CONTEXT_BEGIN(jobData->memContext) { jobData->walPathList = strLstSort( storageListP(storageRepo(), archiveIdPath, .expression = WAL_SEGMENT_DIR_REGEXP_STR), sortOrderAsc); } MEM_CONTEXT_END(); } // If there are WAL paths then get the file lists if (!strLstEmpty(jobData->walPathList)) { // Get the archive id info for the current (last) archive id being processed VerifyArchiveResult *const archiveResult = lstGetLast(jobData->archiveIdResultList); do { const String *const walPath = strLstGet(jobData->walPathList, 0); // Get the WAL files for the first item in the WAL paths list and initialize WAL info and ranges if (strLstEmpty(jobData->walFileList)) { // Free the old WAL file list strLstFree(jobData->walFileList); // Get WAL file list const String *const walFilePath = strNewFmt( STORAGE_REPO_ARCHIVE "/%s/%s", strZ(archiveResult->archiveId), strZ(walPath)); MEM_CONTEXT_BEGIN(jobData->memContext) { jobData->walFileList = strLstSort( storageListP(storageRepo(), walFilePath, .expression = WAL_SEGMENT_FILE_REGEXP_STR), sortOrderAsc); } MEM_CONTEXT_END(); // Filter WAL files if needed if (jobData->enableArchiveFilter) { // If backup manifest is broken, skip all archives to report the error immediately if (!jobData->archiveStart) jobData->walFileList = strLstNew(); // Skip WAL files that come before range while (strLstSize(jobData->walFileList) > 0) { const String *const item = strLstGet(jobData->walFileList, 0); if (strCmp(strSubN(item, 0, WAL_SEGMENT_NAME_SIZE), jobData->archiveStart) < 0) jobData->walFileList = strLstRemoveIdx(jobData->walFileList, 0); else break; } // Skip WAL files that come after range while (strLstSize(jobData->walFileList) > 0) { unsigned int lastIdx = strLstSize(jobData->walFileList) - 1; const String *const item = strLstGet(jobData->walFileList, lastIdx); if (strCmp(strSubN(item, 0, WAL_SEGMENT_NAME_SIZE), jobData->archiveStop) > 0) jobData->walFileList = strLstRemoveIdx(jobData->walFileList, lastIdx); else break; } } if (!strLstEmpty(jobData->walFileList)) { if (archiveResult->pgWalInfo.size == 0) { // Initialize the WAL segment size from the first WAL StorageRead *const walRead = verifyFileLoad( strNewFmt( STORAGE_REPO_ARCHIVE "/%s/%s/%s", strZ(archiveResult->archiveId), strZ(walPath), strZ(strLstGet(jobData->walFileList, 0))), jobData->walCipherPass); const PgWal walInfo = pgWalFromBuffer( storageGetP(walRead, .exactSize = PG_WAL_HEADER_SIZE), cfgOptionStrNull(cfgOptPgVersionForce)); archiveResult->pgWalInfo.size = walInfo.size; archiveResult->pgWalInfo.version = walInfo.version; } // Add total number of WAL files in the directory to the total WAL - this number will include // duplicates, if any, that will be filtered out and not checked but will be reported as errors in the // log archiveResult->totalWalFile += strLstSize(jobData->walFileList); verifyCreateArchiveIdRange(archiveResult, jobData->walFileList, &jobData->jobErrorTotal); } } // If there are WAL files, then verify them if (!strLstEmpty(jobData->walFileList)) { // Get the fully qualified file name and checksum const String *const fileName = strLstGet(jobData->walFileList, 0); const String *const filePathName = strNewFmt( STORAGE_REPO_ARCHIVE "/%s/%s/%s", strZ(archiveResult->archiveId), strZ(walPath), strZ(fileName)); const Buffer *const checksum = bufNewDecode( encodingHex, strSubN(fileName, WAL_SEGMENT_NAME_SIZE + 1, HASH_TYPE_SHA1_SIZE_HEX)); // Set up the job PackWrite *const param = protocolPackNew(); pckWriteStrP(param, filePathName); pckWriteBoolP(param, false); pckWriteU32P(param, compressTypeFromName(filePathName)); pckWriteBinP(param, checksum); pckWriteU64P(param, archiveResult->pgWalInfo.size); pckWriteStrP(param, jobData->walCipherPass); // Assign job to result, prepending the archiveId to the key for consistency with backup processing const String *const jobKey = strNewFmt("%s/%s", strZ(archiveResult->archiveId), strZ(filePathName)); MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew(VARSTR(jobKey), PROTOCOL_COMMAND_VERIFY_FILE, param); } MEM_CONTEXT_PRIOR_END(); // Remove the file to process from the list strLstRemoveIdx(jobData->walFileList, 0); // If this is the last file to process for this timeline, then remove the path if (strLstEmpty(jobData->walFileList)) strLstRemoveIdx(jobData->walPathList, 0); } else { // No valid WAL to process (may be only duplicates or nothing in WAL path) - remove WAL path from the list LOG_DETAIL_FMT( "path '%s/%s' does not contain any valid WAL to be processed", strZ(archiveResult->archiveId), strZ(walPath)); strLstRemoveIdx(jobData->walPathList, 0); } // If a job was found to be processed then break out to process it if (result != NULL) break; } while (!strLstEmpty(jobData->walPathList)); // If this is the last timeline to process for this archiveId, then remove the archiveId if (strLstEmpty(jobData->walPathList)) strLstRemoveIdx(jobData->archiveIdList, 0); // If a file was sent to be processed then break so we can process it if (result != NULL) break; } else { // Log that no WAL paths exist in the archive Id dir - remove the archive Id from the list (nothing to process) LOG_DETAIL_FMT("archive path '%s' is empty", strZ(strLstGet(jobData->archiveIdList, 0))); strLstRemoveIdx(jobData->archiveIdList, 0); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } /*********************************************************************************************************************************** Verify the job data backups ***********************************************************************************************************************************/ static ProtocolParallelJob * verifyBackup(VerifyJobData *const jobData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, jobData); FUNCTION_TEST_END(); ProtocolParallelJob *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Process backup files, if any while (!strLstEmpty(jobData->backupList)) { // If result list is empty or the last processed is not equal to the backup being processed, then initialize the backup // data and results if (lstEmpty(jobData->backupResultList) || !strEq( ((VerifyBackupResult *)lstGetLast(jobData->backupResultList))->backupLabel, strLstGet(jobData->backupList, 0))) { MEM_CONTEXT_BEGIN(lstMemContext(jobData->backupResultList)) { const VerifyBackupResult backupResultNew = { .backupLabel = strDup(strLstGet(jobData->backupList, 0)), .invalidFileList = lstNewP(sizeof(VerifyInvalidFile), .comparator = lstComparatorStr), }; // Add the backup to the result list lstAdd(jobData->backupResultList, &backupResultNew); } MEM_CONTEXT_END(); // Get the result just added so it can be updated directly VerifyBackupResult *const backupResult = lstGetLast(jobData->backupResultList); // If currentBackup is set (meaning the newest backup label on disk was not in the db:current section when the // backup.info file was read) and this is the same label, then set inProgressBackup to true, else false. // inProgressBackup may be changed in verifyManifestFile if a main backup.manifest exists since that would indicate // the backup completed during the verify process. const bool inProgressBackup = strEq(jobData->currentBackup, backupResult->backupLabel); // Get a usable backup manifest file Manifest *const manifest = verifyManifestFile( backupResult, jobData->manifestCipherPass, inProgressBackup, jobData->pgHistory, &jobData->jobErrorTotal); // If a usable backup.manifest file is not found if (manifest == NULL) { // Remove this backup from the processing list strLstRemoveIdx(jobData->backupList, 0); // No files to process so continue to the next backup in the list continue; } // Initialize the backup results and manifest for processing else { // Move the manifest to the jobData for processing jobData->manifest = manifestMove(manifest, jobData->memContext); // Initialize the jobData MEM_CONTEXT_BEGIN(jobData->memContext) { // Get the cipher subpass used to decrypt files in the backup and initialize the file list index jobData->backupCipherPass = strDup(manifestCipherSubPass(jobData->manifest)); jobData->manifestFileIdx = 0; } MEM_CONTEXT_END(); const ManifestData *const manData = manifestData(jobData->manifest); MEM_CONTEXT_BEGIN(lstMemContext(jobData->backupResultList)) { backupResult->totalFileManifest = manifestFileTotal(jobData->manifest); backupResult->backupPrior = strDup(manData->backupLabelPrior); backupResult->pgId = manData->pgId; backupResult->pgVersion = manData->pgVersion; backupResult->archiveStart = strDup(manData->archiveStart); backupResult->archiveStop = strDup(manData->archiveStop); } MEM_CONTEXT_END(); } } VerifyBackupResult *const backupResult = lstGetLast(jobData->backupResultList); // Process any files in the manifest if (jobData->manifestFileIdx < manifestFileTotal(jobData->manifest)) { do { const ManifestFile fileData = manifestFile(jobData->manifest, jobData->manifestFileIdx); // Track the files verified in order to determine when the processing of the backup is complete backupResult->totalFileVerify++; // Check the file if it is not zero-length or not bundled if (fileData.size != 0 || !manifestData(jobData->manifest)->bundle) { // Check if the file is referenced in a prior backup const String *fileBackupLabel = NULL; if (fileData.reference != NULL) { // If the prior backup is not in the result list, then that backup was never processed (likely due to // the --set option) so verify the file const unsigned int backupPriorIdx = lstFindIdx(jobData->backupResultList, &fileData.reference); if (backupPriorIdx == LIST_NOT_FOUND) { fileBackupLabel = fileData.reference; } // Else the backup this file references has a result so check the processing state for the referenced // backup else { const VerifyBackupResult *const backupResultPrior = lstGet( jobData->backupResultList, backupPriorIdx); // If the verify-state of the backup is not complete then verify the file if (!backupResultPrior->fileVerifyComplete) { fileBackupLabel = fileData.reference; } // Else skip verification else { const String *const priorFile = strNewFmt( "%s/%s%s", strZ(fileData.reference), strZ(fileData.name), strZ(compressExtStr((manifestData(jobData->manifest))->backupOptionCompressType))); const unsigned int backupPriorInvalidIdx = lstFindIdx( backupResultPrior->invalidFileList, &priorFile); // If the file is in the invalid file list of the prior backup where it is referenced then add // the file as invalid to this backup result and set the backup result status; since already // logged an error on this file, don't log again if (backupPriorInvalidIdx != LIST_NOT_FOUND) { VerifyInvalidFile *invalidFile = lstGet( backupResultPrior->invalidFileList, backupPriorInvalidIdx); verifyInvalidFileAdd( backupResult->invalidFileList, invalidFile->reason, invalidFile->fileName); backupResult->status = backupInvalid; } // Else file in prior backup was valid so increment total valid files for this backup else { backupResult->totalFileValid++; } } } } // Else file is not referenced in a prior backup else fileBackupLabel = backupResult->backupLabel; // If backup label is not null then send it off for processing if (fileBackupLabel != NULL) { // Set up the job PackWrite *const param = protocolPackNew(); const String *const filePathName = backupFileRepoPathP( fileBackupLabel, .manifestName = fileData.name, .bundleId = fileData.bundleId, .compressType = manifestData(jobData->manifest)->backupOptionCompressType, .blockIncr = fileData.blockIncrMapSize != 0); pckWriteStrP(param, filePathName); if (fileData.bundleId != 0) { pckWriteBoolP(param, true); pckWriteU64P(param, fileData.bundleOffset); pckWriteU64P(param, fileData.sizeRepo); } else pckWriteBoolP(param, false); // Use the repo checksum when present if (fileData.checksumRepoSha1 != NULL) { pckWriteU32P(param, compressTypeNone); pckWriteBinP(param, BUF(fileData.checksumRepoSha1, HASH_TYPE_SHA1_SIZE)); pckWriteU64P(param, fileData.sizeRepo); pckWriteStrP(param, NULL); } // Else use the file checksum, which may require additional filters, e.g. decompression else { pckWriteU32P(param, manifestData(jobData->manifest)->backupOptionCompressType); pckWriteBinP(param, BUF(fileData.checksumSha1, HASH_TYPE_SHA1_SIZE)); pckWriteU64P(param, fileData.size); pckWriteStrP(param, jobData->backupCipherPass); } // Assign job to result (prepend backup label being processed to the key since some files are in a prior // backup) const String *const jobKey = strNewFmt("%s/%s", strZ(backupResult->backupLabel), strZ(filePathName)); MEM_CONTEXT_PRIOR_BEGIN() { result = protocolParallelJobNew(VARSTR(jobKey), PROTOCOL_COMMAND_VERIFY_FILE, param); } MEM_CONTEXT_PRIOR_END(); } } // Else mark the zero-length file as valid else backupResult->totalFileValid++; // Increment the index to point to the next file jobData->manifestFileIdx++; // If this was the last file to process for this backup, then free the manifest and remove this backup from the // processing list if (jobData->manifestFileIdx == backupResult->totalFileManifest) { manifestFree(jobData->manifest); jobData->manifest = NULL; strLstRemoveIdx(jobData->backupList, 0); } // If a job was found to be processed then break out to process it if (result != NULL) break; } while (jobData->manifestFileIdx < backupResult->totalFileManifest); } else { // Nothing to process so report an error, free the manifest, set the status, and remove the backup from processing // list LOG_INFO_FMT("backup '%s' manifest does not contain any target files to verify", strZ(backupResult->backupLabel)); jobData->jobErrorTotal++; manifestFree(jobData->manifest); jobData->manifest = NULL; backupResult->status = backupInvalid; strLstRemoveIdx(jobData->backupList, 0); } // If a job was found to be processed then break out to process it if (result != NULL) break; } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } /*********************************************************************************************************************************** Process the job data ***********************************************************************************************************************************/ static ProtocolParallelJob * verifyJobCallback(void *const data, const unsigned int clientIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); // Pointer to the job data (void)clientIdx; // Client index (not used for this process) FUNCTION_TEST_END(); ASSERT(data != NULL); // Initialize the result ProtocolParallelJob *result = NULL; VerifyJobData *const jobData = data; if (!jobData->backupProcessing) { result = verifyArchive(jobData); // Set the backupProcessing flag if the archive processing is finished so backup processing can begin immediately after jobData->backupProcessing = strLstEmpty(jobData->archiveIdList); } if (jobData->backupProcessing) { // Only begin backup verification if the last archive result was processed if (result == NULL) result = verifyBackup(jobData); } FUNCTION_TEST_RETURN(PROTOCOL_PARALLEL_JOB, result); } /*********************************************************************************************************************************** Helper function for returning a string corresponding to the result code ***********************************************************************************************************************************/ static const char * verifyErrorMsg(const VerifyResult verifyResult) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, verifyResult); // Result code from the verifyFile() function FUNCTION_TEST_END(); ASSERT(verifyResult != verifyOk); switch (verifyResult) { case verifyFileMissing: FUNCTION_TEST_RETURN_CONST(STRINGZ, "file missing"); case verifyChecksumMismatch: FUNCTION_TEST_RETURN_CONST(STRINGZ, "invalid checksum"); case verifySizeInvalid: FUNCTION_TEST_RETURN_CONST(STRINGZ, "invalid size"); default: ASSERT(verifyResult == verifyOtherError); FUNCTION_TEST_RETURN_CONST(STRINGZ, "invalid result"); } } /*********************************************************************************************************************************** Helper function to output a log message based on job result that is not verifyOk and return an error count ***********************************************************************************************************************************/ static unsigned int verifyLogInvalidResult( const String *const fileType, const VerifyResult verifyResult, const unsigned int processId, const String *const filePathName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, fileType); // Indicates archive or backup file FUNCTION_TEST_PARAM(ENUM, verifyResult); // Result code from the verifyFile() function FUNCTION_TEST_PARAM(UINT, processId); // Process Id reporting the result FUNCTION_TEST_PARAM(STRING, filePathName); // File for which results are being reported FUNCTION_TEST_END(); ASSERT(fileType != NULL); ASSERT(filePathName != NULL); // Log a warning because the WAL may have gone missing if expire came through and removed it // legitimately so it is not necessarily an error so the jobErrorTotal should not be incremented if (strEq(fileType, STORAGE_REPO_ARCHIVE_STR) && verifyResult == verifyFileMissing) { LOG_WARN_PID_FMT(processId, "%s '%s'", verifyErrorMsg(verifyResult), strZ(filePathName)); FUNCTION_TEST_RETURN(UINT, 0); } LOG_INFO_PID_FMT(processId, "%s '%s'", verifyErrorMsg(verifyResult), strZ(filePathName)); FUNCTION_TEST_RETURN(UINT, 1); } /*********************************************************************************************************************************** Helper function to set the currently processing backup label, if any, and check that the archiveIds are in the db history ***********************************************************************************************************************************/ static String * verifySetBackupCheckArchive( const StringList *const backupList, const InfoBackup *const backupInfo, const StringList *const archiveIdList, const InfoPg *const pgHistory, unsigned int *const jobErrorTotal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, backupList); // List of backup labels in the backup directory FUNCTION_TEST_PARAM(INFO_BACKUP, backupInfo); // Contents of the backup.info file FUNCTION_TEST_PARAM(STRING_LIST, archiveIdList); // List of archiveIds in the archive directory FUNCTION_TEST_PARAM(INFO_PG, pgHistory); // Pointer to InfoPg of archive.info for accessing PG history FUNCTION_TEST_PARAM_P(UINT, jobErrorTotal); // Pointer to overall job error total FUNCTION_TEST_END(); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // If there are backups, set the last backup as current if it is not in backup.info - if it is, then it is complete, else // it will be checked later if (!strLstEmpty(backupList)) { // Get the last backup as current if it is not in backup.info current list const String *const backupLabel = strLstGet(backupList, strLstSize(backupList) - 1); if (!infoBackupLabelExists(backupInfo, backupLabel)) { // Duplicate the string into the prior context MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(backupLabel); } MEM_CONTEXT_PRIOR_END(); } } // If there are archive directories on disk, make sure they are in the database history list if (!strLstEmpty(archiveIdList)) { StringList *const archiveIdHistoryList = strLstNew(); for (unsigned int histIdx = 0; histIdx < infoPgDataTotal(pgHistory); histIdx++) strLstAdd(archiveIdHistoryList, infoPgArchiveId(pgHistory, histIdx)); // Sort the history list strLstSort(strLstComparatorSet(archiveIdHistoryList, archiveIdComparator), sortOrderAsc); String *const missingFromHistory = strNew(); // Check if the archiveId on disk exists in the archive.info history list and report it if not for (unsigned int archiveIdx = 0; archiveIdx < strLstSize(archiveIdList); archiveIdx++) { const String *const archiveId = strLstGet(archiveIdList, archiveIdx); if (!strLstExists(archiveIdHistoryList, archiveId)) strCat(missingFromHistory, (strEmpty(missingFromHistory) ? archiveId : strNewFmt(", %s", strZ(archiveId)))); } if (!strEmpty(missingFromHistory)) { LOG_INFO_FMT("archiveIds '%s' are not in the archive.info history list", strZ(missingFromHistory)); (*jobErrorTotal)++; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Add the file to the invalid file list for the range in which it exists ***********************************************************************************************************************************/ static void verifyAddInvalidWalFile( const List *const walRangeList, const VerifyResult fileResult, const String *const fileName, const String *const walSegment) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, walRangeList); // List of WAL ranges for an archive Id FUNCTION_TEST_PARAM(UINT, fileResult); // Result of verifyFile() FUNCTION_TEST_PARAM(STRING, fileName); // File name (without the REPO prefix) FUNCTION_TEST_PARAM(STRING, walSegment); // WAL segment, i.e. 000000010000000000000005 FUNCTION_TEST_END(); ASSERT(walRangeList != NULL); ASSERT(fileName != NULL); ASSERT(walSegment != NULL); MEM_CONTEXT_TEMP_BEGIN() { for (unsigned int walIdx = 0; walIdx < lstSize(walRangeList); walIdx++) { const VerifyWalRange *const walRange = lstGet(walRangeList, walIdx); // If the WAL segment is less/equal to the stop file then it falls in this range since ranges are sorted by stop file in // ascending order, therefore first one found is the range if (strCmp(walRange->stop, walSegment) >= 0) { // Add the file to the range where it was found and exit the loop verifyInvalidFileAdd(walRange->invalidFileList, fileResult, fileName); break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Create file errors string ***********************************************************************************************************************************/ static String * verifyCreateFileErrorsStr( const unsigned int errMissing, const unsigned int errChecksum, const unsigned int errSize, const unsigned int errOther, const bool verboseText) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, errMissing); // Number of files missing FUNCTION_TEST_PARAM(UINT, errChecksum); // Number of files with checksum errors FUNCTION_TEST_PARAM(UINT, errSize); // Number of files with invalid size FUNCTION_TEST_PARAM(UINT, errOther); // Number of files with other errors FUNCTION_TEST_PARAM(BOOL, verboseText); // Is verbose output requested FUNCTION_TEST_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // List all if verbose text, otherwise only list if type has errors strCatFmt( result, "\n %s%s%s%s", verboseText || errMissing ? zNewFmt("missing: %u, ", errMissing) : "", verboseText || errChecksum ? zNewFmt("checksum invalid: %u, ", errChecksum) : "", verboseText || errSize ? zNewFmt("size invalid: %u, ", errSize) : "", verboseText || errOther ? zNewFmt("other: %u", errOther) : ""); // Clean up trailing comma when necessary if (strEndsWithZ(result, ", ")) strTruncIdx(result, (int)strSize(result) - 2); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Render the results of the verify command ***********************************************************************************************************************************/ static String * verifyRender(const List *const archiveIdResultList, const List *const backupResultList, const bool verboseText) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, archiveIdResultList); // Result list for all archive Ids in the repo FUNCTION_TEST_PARAM(LIST, backupResultList); // Result list for all backups in the repo FUNCTION_TEST_PARAM(BOOL, verboseText); // Is verbose output requested? FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(archiveIdResultList != NULL); ASSERT(backupResultList != NULL); String *const result = strNew(); // Render archive results if (verboseText && lstEmpty(archiveIdResultList)) strCatZ(result, "\n archiveId: none found"); else { for (unsigned int archiveIdx = 0; archiveIdx < lstSize(archiveIdResultList); archiveIdx++) { const VerifyArchiveResult *const archiveIdResult = lstGet(archiveIdResultList, archiveIdx); if (verboseText || archiveIdResult->totalWalFile - archiveIdResult->totalValidWal != 0) { strCatFmt( result, "\n archiveId: %s, total WAL checked: %u, total valid WAL: %u", strZ(archiveIdResult->archiveId), archiveIdResult->totalWalFile, archiveIdResult->totalValidWal); } if (archiveIdResult->totalWalFile > 0) { unsigned int errMissing = 0; unsigned int errChecksum = 0; unsigned int errSize = 0; unsigned int errOther = 0; for (unsigned int walIdx = 0; walIdx < lstSize(archiveIdResult->walRangeList); walIdx++) { const VerifyWalRange *const walRange = lstGet(archiveIdResult->walRangeList, walIdx); LOG_DETAIL_FMT( "archiveId: %s, wal start: %s, wal stop: %s", strZ(archiveIdResult->archiveId), strZ(walRange->start), strZ(walRange->stop)); unsigned int invalidIdx = 0; while (invalidIdx < lstSize(walRange->invalidFileList)) { VerifyInvalidFile *invalidFile = lstGet(walRange->invalidFileList, invalidIdx); if (invalidFile->reason == verifyFileMissing) errMissing++; else if (invalidFile->reason == verifyChecksumMismatch) errChecksum++; else if (invalidFile->reason == verifySizeInvalid) errSize++; else errOther++; invalidIdx++; } } // Create/append file errors string if (verboseText || errMissing + errChecksum + errSize + errOther > 0) strCat(result, verifyCreateFileErrorsStr(errMissing, errChecksum, errSize, errOther, verboseText)); } } } // Render backup results if (verboseText && lstEmpty(backupResultList)) strCatZ(result, "\n backup: none found"); else { for (unsigned int backupIdx = 0; backupIdx < lstSize(backupResultList); backupIdx++) { const VerifyBackupResult *const backupResult = lstGet(backupResultList, backupIdx); const char *status; switch (backupResult->status) { case backupInvalid: status = "invalid"; break; case backupMissingManifest: status = "manifest missing"; break; case backupInProgress: status = "in-progress"; break; default: { ASSERT(backupResult->status == backupValid); status = "valid"; break; } } if (verboseText || (strcmp(status, "valid") != 0 && strcmp(status, "in-progress") != 0)) { strCatFmt( result, "\n backup: %s, status: %s, total files checked: %u, total valid files: %u", strZ(backupResult->backupLabel), status, backupResult->totalFileVerify, backupResult->totalFileValid); } if (backupResult->totalFileVerify > 0) { unsigned int errMissing = 0; unsigned int errChecksum = 0; unsigned int errSize = 0; unsigned int errOther = 0; for (unsigned int invalidIdx = 0; invalidIdx < lstSize(backupResult->invalidFileList); invalidIdx++) { const VerifyInvalidFile *const invalidFile = lstGet(backupResult->invalidFileList, invalidIdx); if (invalidFile->reason == verifyFileMissing) errMissing++; else if (invalidFile->reason == verifyChecksumMismatch) errChecksum++; else if (invalidFile->reason == verifySizeInvalid) errSize++; else errOther++; } // Create/append file errors string if (verboseText || errMissing + errChecksum + errSize + errOther > 0) strCat(result, verifyCreateFileErrorsStr(errMissing, errChecksum, errSize, errOther, verboseText)); } } } FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Process the verify command ***********************************************************************************************************************************/ static String * verifyProcess(const bool verboseText) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, verboseText); // Is verbose output requested? FUNCTION_LOG_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { unsigned int errorTotal = 0; String *resultStr = strNew(); // Get the repo storage in case it is remote and encryption settings need to be pulled down const Storage *const storage = storageRepo(); // Get a usable backup info file const InfoBackup *const backupInfo = verifyBackupInfoFile(); // If a usable backup.info file is not found, then report an error in the log if (backupInfo == NULL) { strCatZ(resultStr, "\n No usable backup.info file"); errorTotal++; } // Get a usable archive info file const InfoArchive *const archiveInfo = verifyArchiveInfoFile(); // If a usable archive.info file is not found, then report an error in the log if (archiveInfo == NULL) { strCatZ(resultStr, "\n No usable archive.info file"); errorTotal++; } // If both a usable archive info and backup info file were found, then proceed with verification if (archiveInfo != NULL && backupInfo != NULL) { TRY_BEGIN() { // Verify that the archive.info and backup.info current database info and history lists are the same verifyPgHistory(infoArchivePg(archiveInfo), infoBackupPg(backupInfo)); } CATCH_ANY() { strCatFmt(resultStr, "\n%s", errorMessage()); errorTotal++; } TRY_END(); } // If valid info files, then begin process of checking backups and archives in the repo if (errorTotal == 0) { // Initialize the job data VerifyJobData jobData = { .memContext = memContextCurrent(), .walPathList = NULL, .walFileList = strLstNew(), .pgHistory = infoArchivePg(archiveInfo), .manifestCipherPass = infoPgCipherPass(infoBackupPg(backupInfo)), .walCipherPass = infoPgCipherPass(infoArchivePg(archiveInfo)), .archiveIdResultList = lstNewP(sizeof(VerifyArchiveResult), .comparator = archiveIdComparator), .backupResultList = lstNewP(sizeof(VerifyBackupResult), .comparator = lstComparatorStr), }; // Use backup label if specified via --set const String *const backupLabel = cfgOptionStrNull(cfgOptSet); const String *backupRegExpStr = backupRegExpP(.full = true, .differential = true, .incremental = true); bool backupLabelInvalid = false; if (backupLabel != NULL) { if (!regExpMatchOne(backupRegExpStr, backupLabel)) { strCatFmt(resultStr, "\n '%s' is not a valid backup label format", strZ(backupLabel)); backupLabelInvalid = true; errorTotal++; } else backupRegExpStr = strNewFmt("^%s$", strZ(backupLabel)); } // Get a list of backups in the repo sorted ascending if (!backupLabelInvalid) { jobData.backupList = strLstSort( storageListP(storage, STORAGE_REPO_BACKUP_STR, .expression = backupRegExpStr), sortOrderAsc); // Warn if backups in the repository are not described in backup.info. Do not add them for processing since it is // possible for backups to exist in the repository but not in backup.info if an error occurs while backups are being // expired. for (unsigned int repoIdx = 0; repoIdx < strLstSize(jobData.backupList); repoIdx++) { const String *const label = strLstGet(jobData.backupList, repoIdx); if (!infoBackupLabelExists(backupInfo, label)) LOG_WARN_FMT("backup '%s' found in the repository but not in " INFO_BACKUP_FILE, strZ(label)); } } if (!backupLabelInvalid && backupLabel != NULL && strLstEmpty(jobData.backupList)) { strCatFmt(resultStr, "\n backup set %s is not valid", strZ(backupLabel)); backupLabelInvalid = true; errorTotal++; } // Get a list of archive ids in the repo (e.g. 9.4-1, 10-2, etc) sorted ascending by the db-id (number after the dash) if (!backupLabelInvalid) { jobData.archiveIdList = strLstSort( strLstComparatorSet( storageListP(storage, STORAGE_REPO_ARCHIVE_STR, .expression = STRDEF(REGEX_ARCHIVE_DIR_DB_VERSION)), archiveIdComparator), sortOrderAsc); } // Check for block map dependencies if --set option is specified if (!backupLabelInvalid && backupLabel != NULL) verifyBackupSet(&jobData, backupLabel); // Check if backup.info contains backups not found in the repository and add them for processing if (backupLabel == NULL) { const StringList *const infoBackupLabelList = infoBackupDataLabelList(backupInfo, NULL); for (unsigned int infoIdx = 0; infoIdx < strLstSize(infoBackupLabelList); infoIdx++) { const String *const infoLabel = strLstGet(infoBackupLabelList, infoIdx); if (strLstFindIdxP(jobData.backupList, infoLabel) == LIST_NOT_FOUND) { LOG_WARN_FMT("backup '%s' found in " INFO_BACKUP_FILE " but not in the repository", strZ(infoLabel)); strLstAdd(jobData.backupList, infoLabel); } } jobData.backupList = strLstSort(jobData.backupList, sortOrderAsc); } // Only begin processing if there are some archives or backups in the repo if (!backupLabelInvalid && (!strLstEmpty(jobData.archiveIdList) || !strLstEmpty(jobData.backupList))) { // Warn if there are no archives or there are no backups in the repo so that the callback need not try to // distinguish between having processed all of the list or if the list was missing in the first place if (strLstEmpty(jobData.archiveIdList) || strLstEmpty(jobData.backupList)) LOG_DETAIL_FMT("no %s exist in the repo", strLstEmpty(jobData.archiveIdList) ? "archives" : "backups"); // If there are no archives to process, then set the processing flag to skip to processing the backups if (strLstEmpty(jobData.archiveIdList)) jobData.backupProcessing = true; // Set current backup if there is one and verify the archive history on disk is in the database history jobData.currentBackup = verifySetBackupCheckArchive( jobData.backupList, backupInfo, jobData.archiveIdList, jobData.pgHistory, &jobData.jobErrorTotal); // Create the parallel executor ProtocolParallel *const parallelExec = protocolParallelNew( cfgOptionUInt64(cfgOptProtocolTimeout) / 2, verifyJobCallback, &jobData); for (unsigned int processIdx = 1; processIdx <= cfgOptionUInt(cfgOptProcessMax); processIdx++) protocolParallelClientAdd(parallelExec, protocolLocalGet(protocolStorageTypeRepo, 0, processIdx)); // Process jobs MEM_CONTEXT_TEMP_RESET_BEGIN() { do { unsigned int completed = protocolParallelProcess(parallelExec); // Process completed jobs for (unsigned int jobIdx = 0; jobIdx < completed; jobIdx++) { // Get the job and job key ProtocolParallelJob *const job = protocolParallelResult(parallelExec); const unsigned int processId = protocolParallelJobProcessId(job); StringList *const filePathLst = strLstNewSplit(varStr(protocolParallelJobKey(job)), FSLASH_STR); // Remove the result and file type identifier and recreate the path file name const String *const resultId = strLstGet(filePathLst, 0); strLstRemoveIdx(filePathLst, 0); const String *const fileType = strLstGet(filePathLst, 0); strLstRemoveIdx(filePathLst, 0); const String *const filePathName = strLstJoin(filePathLst, "/"); // Initialize the result sets VerifyArchiveResult *archiveIdResult = NULL; VerifyBackupResult *backupResult = NULL; // Get archiveId result data if (strEq(fileType, STORAGE_REPO_ARCHIVE_STR)) { // Find the archiveId in the list - assert if not found since this should never happen const unsigned int index = lstFindIdx(jobData.archiveIdResultList, &resultId); ASSERT(index != LIST_NOT_FOUND); archiveIdResult = lstGet(jobData.archiveIdResultList, index); } // Else get the backup result data else { const unsigned int index = lstFindIdx(jobData.backupResultList, &resultId); ASSERT(index != LIST_NOT_FOUND); backupResult = lstGet(jobData.backupResultList, index); } // The job was successful if (protocolParallelJobErrorCode(job) == 0) { const VerifyResult verifyResult = (VerifyResult)pckReadU32P(protocolParallelJobResult(job)); // Update the result set for the type of file being processed if (strEq(fileType, STORAGE_REPO_ARCHIVE_STR)) { if (verifyResult == verifyOk) archiveIdResult->totalValidWal++; else { jobData.jobErrorTotal += verifyLogInvalidResult( fileType, verifyResult, processId, filePathName); // Add invalid file to the WAL range verifyAddInvalidWalFile( archiveIdResult->walRangeList, verifyResult, filePathName, strSubN( strLstGet(filePathLst, strLstSize(filePathLst) - 1), 0, WAL_SEGMENT_NAME_SIZE)); } } else { if (verifyResult == verifyOk) backupResult->totalFileValid++; else { jobData.jobErrorTotal += verifyLogInvalidResult( fileType, verifyResult, processId, filePathName); backupResult->status = backupInvalid; verifyInvalidFileAdd(backupResult->invalidFileList, verifyResult, filePathName); } } } // Else the job errored else { // Log a protocol error and increment the jobErrorTotal LOG_INFO_PID_FMT( processId, "%s %s: [%d] %s", verifyErrorMsg(verifyOtherError), strZ(filePathName), protocolParallelJobErrorCode(job), strZ(protocolParallelJobErrorMessage(job))); jobData.jobErrorTotal++; // Add invalid file with "OtherError" reason to invalid file list if (strEq(fileType, STORAGE_REPO_ARCHIVE_STR)) { // Add invalid file to the WAL range verifyAddInvalidWalFile( archiveIdResult->walRangeList, verifyOtherError, filePathName, strSubN(strLstGet(filePathLst, strLstSize(filePathLst) - 1), 0, WAL_SEGMENT_NAME_SIZE)); } else { backupResult->status = backupInvalid; verifyInvalidFileAdd(backupResult->invalidFileList, verifyOtherError, filePathName); } } // Set backup verification complete for a backup if all files have run through verification if (strEq(fileType, STORAGE_REPO_BACKUP_STR) && backupResult->totalFileVerify == backupResult->totalFileManifest) { backupResult->fileVerifyComplete = true; } // Free the job protocolParallelJobFree(job); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } while (!protocolParallelDone(parallelExec)); } MEM_CONTEXT_TEMP_END(); // ??? Need to do the final reconciliation - checking backup required WAL against, valid WAL // Report results resultStr = verifyRender(jobData.archiveIdResultList, jobData.backupResultList, verboseText); } else if (!backupLabelInvalid) strCatZ(resultStr, "\n no archives or backups exist in the repo"); errorTotal += jobData.jobErrorTotal; } // If verbose output or errors then output results if (verboseText || errorTotal > 0) { strCatFmt( result, "stanza: %s\nstatus: %s%s", strZ(cfgOptionStr(cfgOptStanza)), errorTotal > 0 ? VERIFY_STATUS_ERROR : VERIFY_STATUS_OK, strZ(resultStr)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cmdVerify(void) { FUNCTION_LOG_VOID(logLevelDebug); MEM_CONTEXT_TEMP_BEGIN() { const String *const result = verifyProcess(cfgOptionBool(cfgOptVerbose)); // Output results if any if (!strEmpty(result)) { // Log results LOG_INFO_FMT("%s", strZ(result)); // Output to console when requested if (cfgOptionSeq(cfgOptOutput) == CFGOPTVAL_VERIFY_OUTPUT_TEXT) { ioFdWriteOneStr(STDOUT_FILENO, result); ioFdWriteOneStr(STDOUT_FILENO, LF_STR); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/command/verify/verify.h0000664000175100017510000000127715240547630020610 0ustar00runnerrunner/*********************************************************************************************************************************** Verify Command ***********************************************************************************************************************************/ #ifndef COMMAND_VERIFY_VERIFY_H #define COMMAND_VERIFY_VERIFY_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Verify the contents of the repository FN_EXTERN void cmdVerify(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/0000775000175100017510000000000015240547630015472 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/assert.h0000664000175100017510000001272615240547630017154 0ustar00runnerrunner/*********************************************************************************************************************************** Assert Routines ***********************************************************************************************************************************/ #ifndef COMMON_ASSERT_H #define COMMON_ASSERT_H #include "common/error/error.h" /*********************************************************************************************************************************** Asserts are used in test code to ensure that certain conditions are true. They are omitted from production builds. ***********************************************************************************************************************************/ #ifdef DEBUG #define ASSERT(condition) \ do \ { \ if (!(condition)) \ THROW_FMT(AssertError, "assertion '%s' failed", #condition); \ } \ while (0) // Skip inline asserts when coverage testing because they will not have branch coverage. Generally speaking inline assertions // should be of the "this != NULL" variety which is also caught effectively by Valgrind. #ifndef DEBUG_COVERAGE #define ASSERT_INLINE(condition) \ do \ { \ if (!(condition)) \ THROW_FMT(AssertError, "assertion '%s' failed", #condition); \ } \ while (0) #else #define ASSERT_INLINE(condition) #endif // Used when execution reaches an invalid location rather than an invalid condition #define ASSERT_MSG(message) \ THROW_FMT(AssertError, message); // Declare variables that will be used by later assertions with the goal of making them easier to read and maintain #define ASSERT_DECLARE(declaration) \ declaration // Add a parameter to a function that is only used by assertions #define ASSERT_PARAM(param) \ , param #else #define ASSERT(condition) #define ASSERT_INLINE(condition) #define ASSERT_MSG(message) #define ASSERT_DECLARE(declaration) #define ASSERT_PARAM(param) #endif /*********************************************************************************************************************************** Checks are used in production builds to test very important conditions. Be sure to limit use to the most critical cases. ***********************************************************************************************************************************/ #define CHECK(type, condition, message) \ do \ { \ if (!(condition)) \ THROW(type, message); \ } \ while (0) #define CHECK_FMT(type, condition, message, ...) \ do \ { \ if (!(condition)) \ THROW_FMT(type, message, __VA_ARGS__); \ } \ while (0) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/0000775000175100017510000000000015240547630017325 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/bz2/0000775000175100017510000000000015240547630020022 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/bz2/common.c0000664000175100017510000000403615240547630021461 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Common ***********************************************************************************************************************************/ #include #include #include "common/compress/bz2/common.h" #include "common/debug.h" #include "common/memContext.h" /**********************************************************************************************************************************/ FN_EXTERN int bz2Error(const int error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, error); FUNCTION_TEST_END(); if (error < 0) { const char *errorMsg; const ErrorType *errorType = &AssertError; switch (error) { case BZ_SEQUENCE_ERROR: errorMsg = "sequence error"; break; case BZ_PARAM_ERROR: errorMsg = "parameter error"; break; case BZ_MEM_ERROR: errorMsg = "memory error"; errorType = &MemoryError; break; case BZ_DATA_ERROR: errorMsg = "data error"; errorType = &FormatError; break; case BZ_DATA_ERROR_MAGIC: errorMsg = "data error magic"; errorType = &FormatError; break; case BZ_IO_ERROR: errorMsg = "io error"; break; case BZ_UNEXPECTED_EOF: errorMsg = "unexpected eof"; break; case BZ_OUTBUFF_FULL: errorMsg = "outbuff full"; break; case BZ_CONFIG_ERROR: errorMsg = "config error"; break; default: errorMsg = "unknown error"; break; } THROWP_FMT(errorType, "bz2 error: [%d] %s", error, errorMsg); } FUNCTION_TEST_RETURN(INT, error); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/bz2/common.h0000664000175100017510000000217115240547630021464 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Common Developed using the documentation in https://www.sourceware.org/bzip2/manual/manual.html#libprog ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_BZ2_COMMON_H #define COMMON_COMPRESS_BZ2_COMMON_H #include /*********************************************************************************************************************************** BZ2 extension ***********************************************************************************************************************************/ #define BZ2_EXT "bz2" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN int bz2Error(int error); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/bz2/compress.c0000664000175100017510000001632115240547630022024 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Compress ***********************************************************************************************************************************/ #include #include #include #include "common/compress/bz2/common.h" #include "common/compress/bz2/compress.h" #include "common/compress/common.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Bz2Compress { bz_stream stream; // Compression stream bool inputSame; // Is the same input required on the next process call? bool flushing; // Is input complete and flushing in progress? bool done; // Is compression done? } Bz2Compress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void bz2CompressToLog(const Bz2Compress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s, flushing: %s, avail_in: %u}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done), cvtBoolToConstZ(this->flushing), this->stream.avail_in); } #define FUNCTION_LOG_BZ2_COMPRESS_TYPE \ Bz2Compress * #define FUNCTION_LOG_BZ2_COMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, bz2CompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free compression stream ***********************************************************************************************************************************/ static void bz2CompressFreeResource(THIS_VOID) { THIS(Bz2Compress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BZ2_COMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); BZ2_bzCompressEnd(&this->stream); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Compress data ***********************************************************************************************************************************/ static void bz2CompressProcess(THIS_VOID, const Buffer *const uncompressed, Buffer *const compressed) { THIS(Bz2Compress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BZ2_COMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->done); ASSERT(compressed != NULL); ASSERT(!this->flushing || uncompressed == NULL); ASSERT(this->flushing || (!this->inputSame || this->stream.avail_in != 0)); // If input is NULL then start flushing if (uncompressed == NULL) { this->stream.avail_in = 0; this->flushing = true; } // Else still have input data else { // Is new input allowed? if (!this->inputSame) { this->stream.avail_in = (unsigned int)bufUsed(uncompressed); // bzip2 does not accept const input buffers #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" this->stream.next_in = (char *)UNCONSTIFY(uint8_t *, bufPtrConst(uncompressed)); #pragma GCC diagnostic pop } } // Initialize compressed output buffer this->stream.avail_out = (unsigned int)bufRemains(compressed); this->stream.next_out = (char *)bufPtr(compressed) + bufUsed(compressed); // Perform compression, check for error const int result = bz2Error(BZ2_bzCompress(&this->stream, this->flushing ? BZ_FINISH : BZ_RUN)); // Set buffer used space bufUsedSet(compressed, bufSize(compressed) - (size_t)this->stream.avail_out); // Is compression done? if (this->flushing && result == BZ_STREAM_END) this->done = true; // Can more input be provided on the next call? this->inputSame = this->flushing ? !this->done : this->stream.avail_in != 0; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is compress done? ***********************************************************************************************************************************/ static bool bz2CompressDone(const THIS_VOID) { THIS(const Bz2Compress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BZ2_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool bz2CompressInputSame(const THIS_VOID) { THIS(const Bz2Compress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BZ2_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * bz2CompressNew(const int level, const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, level); (void)raw; // Raw unsupported FUNCTION_LOG_END(); ASSERT(level >= BZ2_COMPRESS_LEVEL_MIN && level <= BZ2_COMPRESS_LEVEL_MAX); OBJ_NEW_BEGIN(Bz2Compress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Bz2Compress) { .stream = {.bzalloc = NULL}, }; // Initialize context bz2Error(BZ2_bzCompressInit(&this->stream, level, 0, 0)); // Set callback to ensure bz2 stream is freed memContextCallbackSet(objMemContext(this), bz2CompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( BZ2_COMPRESS_FILTER_TYPE, this, compressParamList(level, raw), .done = bz2CompressDone, .inOut = bz2CompressProcess, .inputSame = bz2CompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/bz2/compress.h0000664000175100017510000000305715240547630022033 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Compress Compress IO to the bz2 format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_BZ2_COMPRESS_H #define COMMON_COMPRESS_BZ2_COMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BZ2_COMPRESS_FILTER_TYPE STRID5("bz2-cmp", 0x41a3df3420) /*********************************************************************************************************************************** Level constants ***********************************************************************************************************************************/ #define BZ2_COMPRESS_LEVEL_MIN 1 #define BZ2_COMPRESS_LEVEL_MAX 9 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *bz2CompressNew(int level, bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/bz2/decompress.c0000664000175100017510000001551115240547630022335 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Decompress ***********************************************************************************************************************************/ #include #include #include #include "common/compress/bz2/common.h" #include "common/compress/bz2/decompress.h" #include "common/compress/common.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Bz2Decompress { bz_stream stream; // Decompression stream state int result; // Result of last operation bool inputSame; // Is the same input required on the next process call? bool done; // Is decompression done? } Bz2Decompress; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void bz2DecompressToLog(const Bz2Decompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s, avail_in: %u}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done), this->stream.avail_in); } #define FUNCTION_LOG_BZ2_DECOMPRESS_TYPE \ Bz2Decompress * #define FUNCTION_LOG_BZ2_DECOMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, bz2DecompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free inflate stream ***********************************************************************************************************************************/ static void bz2DecompressFreeResource(THIS_VOID) { THIS(Bz2Decompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BZ2_DECOMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); BZ2_bzDecompressEnd(&this->stream); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Decompress data ***********************************************************************************************************************************/ static void bz2DecompressProcess(THIS_VOID, const Buffer *const compressed, Buffer *const uncompressed) { THIS(Bz2Decompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BZ2_DECOMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(uncompressed != NULL); // There should never be a flush because in a valid compressed stream the end of data can be determined and done will be set. // If a flush is received it means the compressed stream terminated early, e.g. a zero-length or truncated file. if (compressed == NULL) THROW(FormatError, "unexpected eof in compressed data"); if (!this->inputSame) { this->stream.avail_in = (unsigned int)bufUsed(compressed); // bzip2 does not accept const input buffers #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" this->stream.next_in = (char *)UNCONSTIFY(uint8_t *, bufPtrConst(compressed)); #pragma GCC diagnostic pop } this->stream.avail_out = (unsigned int)bufRemains(uncompressed); this->stream.next_out = (char *)bufPtr(uncompressed) + bufUsed(uncompressed); this->result = bz2Error(BZ2_bzDecompress(&this->stream)); // Set buffer used space bufUsedSet(uncompressed, bufSize(uncompressed) - (size_t)this->stream.avail_out); // Is decompression done? this->done = this->result == BZ_STREAM_END; // Is the same input expected on the next call? this->inputSame = this->done ? false : this->stream.avail_in != 0; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is decompress done? ***********************************************************************************************************************************/ static bool bz2DecompressDone(const THIS_VOID) { THIS(const Bz2Decompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BZ2_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool bz2DecompressInputSame(const THIS_VOID) { THIS(const Bz2Decompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BZ2_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * bz2DecompressNew(const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); (void)raw; // Raw unsupported FUNCTION_LOG_END(); OBJ_NEW_BEGIN(Bz2Decompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Bz2Decompress) { .stream = {.bzalloc = NULL}, }; // Create bz2 stream bz2Error(this->result = BZ2_bzDecompressInit(&this->stream, 0, 0)); // Set free callback to ensure bz2 context is freed memContextCallbackSet(objMemContext(this), bz2DecompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( BZ2_DECOMPRESS_FILTER_TYPE, this, decompressParamList(raw), .done = bz2DecompressDone, .inOut = bz2DecompressProcess, .inputSame = bz2DecompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/bz2/decompress.h0000664000175100017510000000221415240547630022336 0ustar00runnerrunner/*********************************************************************************************************************************** BZ2 Decompress Decompress IO from the bz2 format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_BZ2_DECOMPRESS_H #define COMMON_COMPRESS_BZ2_DECOMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BZ2_DECOMPRESS_FILTER_TYPE STRID5("bz2-dcmp", 0x83464df3420) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *bz2DecompressNew(bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/common.c0000664000175100017510000000322615240547630020764 0ustar00runnerrunner/*********************************************************************************************************************************** Compression Common ***********************************************************************************************************************************/ #include #include "common/compress/common.h" #include "common/debug.h" /**********************************************************************************************************************************/ FN_EXTERN Pack * compressParamList(const int level, const bool raw) { FUNCTION_TEST_BEGIN(); FUNCTION_LOG_PARAM(INT, level); FUNCTION_TEST_PARAM(BOOL, raw); FUNCTION_TEST_END(); Pack *result; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteI32P(packWrite, level); pckWriteBoolP(packWrite, raw); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN Pack * decompressParamList(const bool raw) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, raw); FUNCTION_TEST_END(); Pack *result; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteBoolP(packWrite, raw); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(PACK, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/common.h0000664000175100017510000000147715240547630020777 0ustar00runnerrunner/*********************************************************************************************************************************** Compression Common ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_COMMON_H #define COMMON_COMPRESS_COMMON_H #include "common/type/pack.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Build compress param list FN_EXTERN Pack *compressParamList(int level, bool raw); // Build decompress param list FN_EXTERN Pack *decompressParamList(bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/gz/0000775000175100017510000000000015240547630017745 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/gz/common.c0000664000175100017510000000412515240547630021403 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Common ***********************************************************************************************************************************/ #include #include #include "common/compress/gz/common.h" #include "common/debug.h" #include "common/memContext.h" /**********************************************************************************************************************************/ FN_EXTERN int gzError(const int error) { if (error != Z_OK && error != Z_STREAM_END) { const char *errorMsg; const ErrorType *errorType = &AssertError; switch (error) { // Not exactly an error, but since we are not using custom dictionaries it shouldn't be possible to get this result case Z_NEED_DICT: errorMsg = "need dictionary"; break; // We should not get this error -- included for completeness case Z_ERRNO: errorMsg = "file error"; break; case Z_STREAM_ERROR: errorMsg = "stream error"; errorType = &FormatError; break; case Z_DATA_ERROR: errorMsg = "data error"; errorType = &FormatError; break; case Z_MEM_ERROR: errorMsg = "insufficient memory"; errorType = &MemoryError; break; // This error indicates an error in the code -- there should always be space in the buffer case Z_BUF_ERROR: errorMsg = "no space in buffer"; break; case Z_VERSION_ERROR: errorMsg = "incompatible version"; errorType = &FormatError; break; default: errorMsg = "unknown error"; break; } THROWP_FMT(errorType, "zlib threw error: [%d] %s", error, errorMsg); } return error; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/gz/common.h0000664000175100017510000000266415240547630021416 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Common ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_GZ_COMMON_H #define COMMON_COMPRESS_GZ_COMMON_H /*********************************************************************************************************************************** Gz extension ***********************************************************************************************************************************/ #define GZ_EXT "gz" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define WINDOW_BITS 15 #define WANT_GZ 16 /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process gz errors FN_EXTERN int gzError(int error); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/gz/compress.c0000664000175100017510000001671415240547630021755 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Compress Based on the documentation at https://github.com/madler/zlib/blob/master/zlib.h ***********************************************************************************************************************************/ #include #include #include #include "common/compress/common.h" #include "common/compress/gz/common.h" #include "common/compress/gz/compress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct GzCompress { z_stream stream; // Compression stream state bool inputSame; // Is the same input required on the next process call? bool flushing; // Is input complete and flushing in progress? bool done; // Is compression done? } GzCompress; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void gzCompressToLog(const GzCompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s, flushing: %s, availIn: %u}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done), cvtBoolToConstZ(this->flushing), this->stream.avail_in); } #define FUNCTION_LOG_GZ_COMPRESS_TYPE \ GzCompress * #define FUNCTION_LOG_GZ_COMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, gzCompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Compression constants ***********************************************************************************************************************************/ #define MEM_LEVEL 9 /*********************************************************************************************************************************** Free deflate stream ***********************************************************************************************************************************/ static void gzCompressFreeResource(THIS_VOID) { THIS(GzCompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(GZ_COMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); deflateEnd(&this->stream); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Compress data ***********************************************************************************************************************************/ static void gzCompressProcess(THIS_VOID, const Buffer *const uncompressed, Buffer *const compressed) { THIS(GzCompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(GZ_COMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->done); ASSERT(compressed != NULL); ASSERT(!this->flushing || uncompressed == NULL); ASSERT(this->flushing || (!this->inputSame || this->stream.avail_in != 0)); // Flushing if (uncompressed == NULL) { this->stream.avail_in = 0; this->flushing = true; } // More input else { // Is new input allowed? if (!this->inputSame) { this->stream.avail_in = (unsigned int)bufUsed(uncompressed); // Not all versions of zlib (and none by default) will accept const input buffers this->stream.next_in = bufPtrConst(uncompressed); } } // Initialize compressed output buffer this->stream.avail_out = (unsigned int)bufRemains(compressed); this->stream.next_out = bufPtr(compressed) + bufUsed(compressed); // Perform compression const int result = gzError(deflate(&this->stream, this->flushing ? Z_FINISH : Z_NO_FLUSH)); // Set buffer used space bufUsedSet(compressed, bufSize(compressed) - (size_t)this->stream.avail_out); // Is compression done? if (this->flushing && result == Z_STREAM_END) this->done = true; // Can more input be provided on the next call? this->inputSame = this->flushing ? !this->done : this->stream.avail_in != 0; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is compress done? ***********************************************************************************************************************************/ static bool gzCompressDone(const THIS_VOID) { THIS(const GzCompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(GZ_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool gzCompressInputSame(const THIS_VOID) { THIS(const GzCompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(GZ_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * gzCompressNew(const int level, const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, level); FUNCTION_LOG_PARAM(BOOL, raw); FUNCTION_LOG_END(); ASSERT(level >= GZ_COMPRESS_LEVEL_MIN && level <= GZ_COMPRESS_LEVEL_MAX); OBJ_NEW_BEGIN(GzCompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (GzCompress) { .stream = {.zalloc = NULL}, }; // Create gz stream gzError(deflateInit2(&this->stream, level, Z_DEFLATED, (raw ? 0 : WANT_GZ) | WINDOW_BITS, MEM_LEVEL, Z_DEFAULT_STRATEGY)); // Set free callback to ensure gz context is freed memContextCallbackSet(objMemContext(this), gzCompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( GZ_COMPRESS_FILTER_TYPE, this, compressParamList(level, raw), .done = gzCompressDone, .inOut = gzCompressProcess, .inputSame = gzCompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/gz/compress.h0000664000175100017510000000305415240547630021753 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Compress Compress IO using the gz format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_GZ_COMPRESS_H #define COMMON_COMPRESS_GZ_COMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define GZ_COMPRESS_FILTER_TYPE STRID5("gz-cmp", 0x20d1ef470) /*********************************************************************************************************************************** Level constants ***********************************************************************************************************************************/ #define GZ_COMPRESS_LEVEL_MIN -1 #define GZ_COMPRESS_LEVEL_MAX 9 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *gzCompressNew(int level, bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/gz/decompress.c0000664000175100017510000001535415240547630022265 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Decompress Based on the documentation at https://github.com/madler/zlib/blob/master/zlib.h ***********************************************************************************************************************************/ #include #include #include #include "common/compress/common.h" #include "common/compress/gz/common.h" #include "common/compress/gz/decompress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/macro.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct GzDecompress { z_stream stream; // Decompression stream state int result; // Result of last operation bool inputSame; // Is the same input required on the next process call? bool done; // Is decompression done? } GzDecompress; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void gzDecompressToLog(const GzDecompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s, availIn: %u}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done), this->stream.avail_in); } #define FUNCTION_LOG_GZ_DECOMPRESS_TYPE \ GzDecompress * #define FUNCTION_LOG_GZ_DECOMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, gzDecompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free inflate stream ***********************************************************************************************************************************/ static void gzDecompressFreeResource(THIS_VOID) { THIS(GzDecompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(GZ_DECOMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); inflateEnd(&this->stream); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Decompress data ***********************************************************************************************************************************/ static void gzDecompressProcess(THIS_VOID, const Buffer *const compressed, Buffer *const uncompressed) { THIS(GzDecompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(GZ_DECOMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(uncompressed != NULL); // There should never be a flush because in a valid compressed stream the end of data can be determined and done will be set. // If a flush is received it means the compressed stream terminated early, e.g. a zero-length or truncated file. if (compressed == NULL) THROW(FormatError, "unexpected eof in compressed data"); if (!this->inputSame) { this->stream.avail_in = (unsigned int)bufUsed(compressed); // Not all versions of zlib (and none by default) will accept const input buffers this->stream.next_in = bufPtrConst(compressed); } this->stream.avail_out = (unsigned int)bufRemains(uncompressed); this->stream.next_out = bufPtr(uncompressed) + bufUsed(uncompressed); this->result = gzError(inflate(&this->stream, Z_NO_FLUSH)); // Set buffer used space bufUsedSet(uncompressed, bufSize(uncompressed) - (size_t)this->stream.avail_out); // Is decompression done? this->done = this->result == Z_STREAM_END; // Is the same input expected on the next call? this->inputSame = this->done ? false : this->stream.avail_in != 0; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is decompress done? ***********************************************************************************************************************************/ static bool gzDecompressDone(const THIS_VOID) { THIS(const GzDecompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(GZ_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool gzDecompressInputSame(const THIS_VOID) { THIS(const GzDecompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(GZ_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * gzDecompressNew(const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BOOL, raw); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(GzDecompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (GzDecompress) { .stream = {.zalloc = NULL}, }; // Create gz stream gzError(this->result = inflateInit2(&this->stream, (raw ? 0 : WANT_GZ) | WINDOW_BITS)); // Set free callback to ensure gz context is freed memContextCallbackSet(objMemContext(this), gzDecompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( GZ_DECOMPRESS_FILTER_TYPE, this, decompressParamList(raw), .done = gzDecompressDone, .inOut = gzDecompressProcess, .inputSame = gzDecompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/gz/decompress.h0000664000175100017510000000220415240547630022260 0ustar00runnerrunner/*********************************************************************************************************************************** Gz Decompress Decompress IO from the gz format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_GZ_DECOMPRESS_H #define COMMON_COMPRESS_GZ_DECOMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define GZ_DECOMPRESS_FILTER_TYPE STRID5("gz-dcmp", 0x41a326f470) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *gzDecompressNew(bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/helper.c0000664000175100017510000002505615240547630020760 0ustar00runnerrunner/*********************************************************************************************************************************** Compression Helper ***********************************************************************************************************************************/ #include #include #include "common/compress/bz2/common.h" #include "common/compress/bz2/compress.h" #include "common/compress/bz2/decompress.h" #include "common/compress/gz/common.h" #include "common/compress/gz/compress.h" #include "common/compress/gz/decompress.h" #include "common/compress/helper.h" #include "common/compress/lz4/common.h" #include "common/compress/lz4/compress.h" #include "common/compress/lz4/decompress.h" #include "common/compress/zst/common.h" #include "common/compress/zst/compress.h" #include "common/compress/zst/decompress.h" #include "common/debug.h" #include "common/log.h" #include "version.h" /*********************************************************************************************************************************** Compression type constants ***********************************************************************************************************************************/ #define COMPRESS_TYPE_NONE "none" // Constants for currently unsupported compression types #define XZ_EXT "xz" /*********************************************************************************************************************************** Configuration for supported and future compression types ***********************************************************************************************************************************/ static const struct CompressHelperLocal { StringId typeId; // Compress type id const String *const type; // Compress type -- must be extension without period prefixed const String *const ext; // File extension with period prefixed StringId compressType; // Type of the compression filter IoFilter *(*compressNew)(int, bool); // Function to create new compression filter StringId decompressType; // Type of the decompression filter IoFilter *(*decompressNew)(bool); // Function to create new decompression filter } compressHelperLocal[] = { [compressTypeNone] = { .typeId = STRID5("none", 0x2b9ee0), .type = STRDEF(COMPRESS_TYPE_NONE), .ext = STRDEF(""), }, [compressTypeBz2] = { .typeId = STRID5("bz2", 0x73420), .type = STRDEF(BZ2_EXT), .ext = STRDEF("." BZ2_EXT), .compressType = BZ2_COMPRESS_FILTER_TYPE, .compressNew = bz2CompressNew, .decompressType = BZ2_DECOMPRESS_FILTER_TYPE, .decompressNew = bz2DecompressNew, }, [compressTypeGz] = { .typeId = STRID5("gz", 0x3470), .type = STRDEF(GZ_EXT), .ext = STRDEF("." GZ_EXT), .compressType = GZ_COMPRESS_FILTER_TYPE, .compressNew = gzCompressNew, .decompressType = GZ_DECOMPRESS_FILTER_TYPE, .decompressNew = gzDecompressNew, }, [compressTypeLz4] = { .typeId = STRID6("lz4", 0x2068c1), .type = STRDEF(LZ4_EXT), .ext = STRDEF("." LZ4_EXT), .compressType = LZ4_COMPRESS_FILTER_TYPE, .compressNew = lz4CompressNew, .decompressType = LZ4_DECOMPRESS_FILTER_TYPE, .decompressNew = lz4DecompressNew, }, [compressTypeZst] = { .typeId = STRID5("zst", 0x527a0), .type = STRDEF(ZST_EXT), .ext = STRDEF("." ZST_EXT), #ifdef HAVE_LIBZST .compressType = ZST_COMPRESS_FILTER_TYPE, .compressNew = zstCompressNew, .decompressType = ZST_DECOMPRESS_FILTER_TYPE, .decompressNew = zstDecompressNew, #endif }, [compressTypeXz] = { .typeId = STRID5("xz", 0x3580), .type = STRDEF(XZ_EXT), .ext = STRDEF("." XZ_EXT), }, }; /**********************************************************************************************************************************/ FN_EXTERN CompressType compressTypeEnum(const StringId type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); ASSERT(type != 0); CompressType result = compressTypeNone; for (; result < LENGTH_OF(compressHelperLocal); result++) { if (type == compressHelperLocal[result].typeId) break; } if (result == LENGTH_OF(compressHelperLocal)) THROW_FMT(AssertError, "invalid compression type '%s'", zNewStrId(type)); FUNCTION_TEST_RETURN(ENUM, result); } /**********************************************************************************************************************************/ static void compressTypePresent(const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); if (type != compressTypeNone && compressHelperLocal[type].compressNew == NULL) THROW_FMT(OptionInvalidValueError, PROJECT_NAME " not built with %s support", strZ(compressHelperLocal[type].type)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const String * compressTypeStr(const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); FUNCTION_TEST_RETURN_CONST(STRING, compressHelperLocal[type].type); } /**********************************************************************************************************************************/ FN_EXTERN CompressType compressTypeFromName(const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); CompressType result = compressTypeNone + 1; for (; result < LENGTH_OF(compressHelperLocal); result++) { if (strEndsWith(name, compressHelperLocal[result].ext)) break; } if (result == LENGTH_OF(compressHelperLocal)) result = compressTypeNone; FUNCTION_TEST_RETURN(ENUM, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * compressFilter(const CompressType type, const int level, const CompressFilterParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(INT, level); FUNCTION_TEST_PARAM(BOOL, param.raw); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); ASSERT(type != compressTypeNone); compressTypePresent(type); FUNCTION_TEST_RETURN(IO_FILTER, compressHelperLocal[type].compressNew(level, param.raw)); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * compressFilterPack(const StringId filterType, const Pack *const filterParam) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, filterType); FUNCTION_LOG_PARAM(VARIANT_LIST, filterParam); FUNCTION_LOG_END(); ASSERT(filterType != 0); IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { for (CompressType compressIdx = compressTypeNone + 1; compressIdx < LENGTH_OF(compressHelperLocal); compressIdx++) { const struct CompressHelperLocal *compress = &compressHelperLocal[compressIdx]; if (filterType == compress->compressType) { ASSERT(filterParam != NULL); PackRead *const paramRead = pckReadNew(filterParam); const int level = pckReadI32P(paramRead); const bool raw = pckReadBoolP(paramRead); result = ioFilterMove(compress->compressNew(level, raw), memContextPrior()); break; } else if (filterType == compress->decompressType) { result = ioFilterMove(compress->decompressNew(pckReadBoolP(pckReadNew(filterParam))), memContextPrior()); break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(IO_FILTER, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * decompressFilter(const CompressType type, const DecompressFilterParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(BOOL, param.raw); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); ASSERT(type != compressTypeNone); compressTypePresent(type); FUNCTION_TEST_RETURN(IO_FILTER, compressHelperLocal[type].decompressNew(param.raw)); } /**********************************************************************************************************************************/ FN_EXTERN const String * compressExtStr(const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(type < LENGTH_OF(compressHelperLocal)); FUNCTION_TEST_RETURN_CONST(STRING, compressHelperLocal[type].ext); } /**********************************************************************************************************************************/ FN_EXTERN void compressExtCat(String *const file, const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, file); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(file != NULL); strCat(file, compressExtStr(type)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * compressExtStrip(const String *const file, const CompressType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, file); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_END(); ASSERT(file != NULL); if (!strEndsWith(file, compressExtStr(type))) THROW_FMT(FormatError, "'%s' must have '%s' extension", strZ(file), strZ(compressExtStr(type))); FUNCTION_TEST_RETURN(STRING, strSubN(file, 0, strSize(file) - strSize(compressExtStr(type)))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/helper.h0000664000175100017510000001215715240547630020763 0ustar00runnerrunner/*********************************************************************************************************************************** Compression Helper Abstract compression types so that the calling code does not need to worry about the individual characteristics of the compression type. All compress calls should use this module rather than the individual filters, even if a specific compression type is required. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_HELPER_H #define COMMON_COMPRESS_HELPER_H #include /*********************************************************************************************************************************** Available compression types ***********************************************************************************************************************************/ typedef enum { compressTypeNone, // No compression compressTypeBz2, // bzip2 compressTypeGz, // gzip compressTypeLz4, // lz4 compressTypeZst, // zstandard // These types have not been implemented but are included here so older versions can identify compression types added by future // versions. In that sense this list is speculative, but these seem to be all the types that are likely to be added in the // foreseeable future. compressTypeXz, // xz/lzma } CompressType; #include #include /*********************************************************************************************************************************** Compression types as a regexp. In the future this regexp will be generated automatically at build time but we want to wait until the build code is migrated to C to do that. For now just define it statically since we don't expect it to change very often. If a supported type is not in this list then it should cause an integration test to fail. ***********************************************************************************************************************************/ #define COMPRESS_TYPE_REGEXP "(\\.gz|\\.lz4|\\.zst|\\.xz|\\.bz2)" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get enum from a compression type string FN_EXTERN CompressType compressTypeEnum(StringId type); // Get string representation of a compression type. This is the extension without the period. FN_EXTERN const String *compressTypeStr(CompressType type); // Get compression type from a (file) name by checking the extension. If the extension is not a supported compression type then // compressType none is returned, even if the file is compressed with some unknown type. FN_EXTERN CompressType compressTypeFromName(const String *name); // Compression filter for the specified type. Error when compress type is none or invalid. typedef struct CompressFilterParam { VAR_PARAM_HEADER; bool raw; // Omit headers, checksum, etc. when possible } CompressFilterParam; #define compressFilterP(type, level, ...) \ compressFilter(type, level, (CompressFilterParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoFilter *compressFilter(CompressType type, int level, CompressFilterParam param); // Compression/decompression filter based on string type and a parameter list. This is useful when a filter must be created on a // remote system since the filter type and parameters can be passed through a protocol. FN_EXTERN IoFilter *compressFilterPack(StringId filterType, const Pack *filterParam); // Decompression filter for the specified type. Error when compress type is none or invalid. typedef struct DecompressFilterParam { VAR_PARAM_HEADER; bool raw; // Omit headers, checksum, etc. when possible } DecompressFilterParam; #define decompressFilterP(type, ...) \ decompressFilter(type, (DecompressFilterParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoFilter *decompressFilter(CompressType type, DecompressFilterParam param); // Get extension for the current compression type FN_EXTERN const String *compressExtStr(CompressType type); // Add extension for current compression type to the file FN_EXTERN void compressExtCat(String *file, CompressType type); // Remove the specified compression extension. Error when the extension is not correct. FN_EXTERN String *compressExtStrip(const String *file, CompressType type); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/lz4/0000775000175100017510000000000015240547630020036 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/lz4/common.c0000664000175100017510000000154415240547630021476 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Common ***********************************************************************************************************************************/ #include #include #include "common/compress/lz4/common.h" #include "common/debug.h" /**********************************************************************************************************************************/ FN_EXTERN LZ4F_errorCode_t lz4Error(const LZ4F_errorCode_t error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, error); FUNCTION_TEST_END(); if (LZ4F_isError(error)) THROW_FMT(FormatError, "lz4 error: [%zd] %s", (ssize_t)error, LZ4F_getErrorName(error)); FUNCTION_TEST_RETURN_TYPE(LZ4F_errorCode_t, error); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/lz4/common.h0000664000175100017510000000203615240547630021500 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Common ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_LZ4_COMMON_H #define COMMON_COMPRESS_LZ4_COMMON_H /*********************************************************************************************************************************** LZ4 extension ***********************************************************************************************************************************/ #define LZ4_EXT "lz4" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process lz4 errors FN_EXTERN size_t lz4Error(size_t error); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/lz4/compress.c0000664000175100017510000002576515240547630022054 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Compress Developed against version r131 using the documentation in https://github.com/lz4/lz4/blob/r131/lib/lz4frame.h. ***********************************************************************************************************************************/ #include #include #include #include #include "common/compress/common.h" #include "common/compress/lz4/common.h" #include "common/compress/lz4/compress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Older versions of lz4 do not define the max header size. This seems to be the max for any version. ***********************************************************************************************************************************/ #ifndef LZ4F_HEADER_SIZE_MAX #define LZ4F_HEADER_SIZE_MAX 19 #endif /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Lz4Compress { LZ4F_compressionContext_t context; // LZ4 compression context LZ4F_preferences_t prefs; // Preferences -- just compress level set IoFilter *filter; // Filter interface Buffer *buffer; // For when the output buffer can't accept all compressed data bool first; // Is this the first call to process? bool inputSame; // Is the same input required on the next process call? bool flushing; // Is input complete and flushing in progress? } Lz4Compress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void lz4CompressToLog(const Lz4Compress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{level: %d, first: %s, inputSame: %s, flushing: %s}", this->prefs.compressionLevel, cvtBoolToConstZ(this->first), cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->flushing)); } #define FUNCTION_LOG_LZ4_COMPRESS_TYPE \ Lz4Compress * #define FUNCTION_LOG_LZ4_COMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, lz4CompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free compression context ***********************************************************************************************************************************/ static void lz4CompressFreeResource(THIS_VOID) { THIS(Lz4Compress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(LZ4_COMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); LZ4F_freeCompressionContext(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Compress data ***********************************************************************************************************************************/ // Helper to return a buffer where output will be written. If there is enough space in the provided output buffer then use it, // otherwise allocate an internal buffer to hold the compressed data. Once we start using the internal buffer we'll need to continue // using it until it is completely flushed. static Buffer * lz4CompressBuffer(Lz4Compress *const this, const size_t required, Buffer *const output) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_COMPRESS, this); FUNCTION_TEST_PARAM(SIZE, required); FUNCTION_TEST_PARAM(BUFFER, output); FUNCTION_TEST_END(); Buffer *result = output; // Is an internal buffer required to hold the compressed data? if (!bufEmpty(this->buffer) || required >= bufRemains(output)) { // Reallocate buffer if it is not large enough if (required >= bufRemains(this->buffer)) bufResize(this->buffer, bufUsed(this->buffer) + required); result = this->buffer; } FUNCTION_TEST_RETURN(BUFFER, result); } // Helper to flush output data to compressed buffer static void lz4CompressFlush(Lz4Compress *const this, Buffer *const output, Buffer *const compressed) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_COMPRESS, this); FUNCTION_TEST_PARAM(BUFFER, output); FUNCTION_TEST_PARAM(BUFFER, compressed); FUNCTION_TEST_END(); // If the compressed buffer can hold all the output if (bufRemains(compressed) >= bufUsed(output)) { bufCat(compressed, output); bufUsedZero(output); this->inputSame = false; } // Else flush as much as possible and set inputSame to flush more once the compressed buffer has been emptied else { size_t catSize = bufRemains(compressed); bufCatSub(compressed, output, 0, catSize); memmove(bufPtr(output), bufPtr(output) + catSize, bufUsed(output) - catSize); bufUsedSet(output, bufUsed(output) - catSize); this->inputSame = true; } FUNCTION_TEST_RETURN_VOID(); } static void lz4CompressProcess(THIS_VOID, const Buffer *const uncompressed, Buffer *const compressed) { THIS(Lz4Compress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(LZ4_COMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!(this->flushing && !this->inputSame)); ASSERT(this->context != NULL); ASSERT(compressed != NULL); ASSERT(!this->flushing || uncompressed == NULL); // Flush overflow output to the compressed buffer if (this->inputSame) { lz4CompressFlush(this, this->buffer, compressed); } else { Buffer *output = NULL; // If first call to process then begin compression if (this->first) { output = lz4CompressBuffer(this, LZ4F_HEADER_SIZE_MAX, compressed); bufUsedInc( output, lz4Error(LZ4F_compressBegin(this->context, bufRemainsPtr(output), bufRemains(output), &this->prefs))); this->first = false; } // Normal processing call if (uncompressed != NULL) { output = lz4CompressBuffer(this, lz4Error(LZ4F_compressBound(bufUsed(uncompressed), &this->prefs)), compressed); bufUsedInc( output, lz4Error( LZ4F_compressUpdate( this->context, bufRemainsPtr(output), bufRemains(output), bufPtrConst(uncompressed), bufUsed(uncompressed), NULL))); } // Else flush remaining output else { // Pass 1 as src size to help allocate enough space for the final flush. This is required for some versions that don't // allocate enough memory unless autoFlush is enabled. Other versions fail if autoFlush is only enabled before the final // flush. This will hopefully work across all versions even if it does allocate a larger buffer than needed. output = lz4CompressBuffer(this, lz4Error(LZ4F_compressBound(1, &this->prefs)), compressed); bufUsedInc(output, lz4Error(LZ4F_compressEnd(this->context, bufRemainsPtr(output), bufRemains(output), NULL))); this->flushing = true; } // If the output buffer was allocated locally it will need to be flushed to the compressed buffer if (output != compressed) lz4CompressFlush(this, output, compressed); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is compress done? ***********************************************************************************************************************************/ static bool lz4CompressDone(const THIS_VOID) { THIS(const Lz4Compress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->flushing && !this->inputSame); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool lz4CompressInputSame(const THIS_VOID) { THIS(const Lz4Compress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * lz4CompressNew(const int level, const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, level); FUNCTION_LOG_PARAM(BOOL, raw); FUNCTION_LOG_END(); ASSERT(level >= LZ4_COMPRESS_LEVEL_MIN && level <= LZ4_COMPRESS_LEVEL_MAX); OBJ_NEW_BEGIN(Lz4Compress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Lz4Compress) { .prefs = { .compressionLevel = level, .frameInfo = {.contentChecksumFlag = raw ? LZ4F_noContentChecksum : LZ4F_contentChecksumEnabled}, }, .first = true, .buffer = bufNew(0), }; // Create lz4 context lz4Error(LZ4F_createCompressionContext(&this->context, LZ4F_VERSION)); // Set callback to ensure lz4 context is freed memContextCallbackSet(objMemContext(this), lz4CompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( LZ4_COMPRESS_FILTER_TYPE, this, compressParamList(level, raw), .done = lz4CompressDone, .inOut = lz4CompressProcess, .inputSame = lz4CompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/lz4/compress.h0000664000175100017510000000306315240547630022044 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Compress Compress IO to the lz4 format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_LZ4_COMPRESS_H #define COMMON_COMPRESS_LZ4_COMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define LZ4_COMPRESS_FILTER_TYPE STRID6("lz4-cmp", 0x103436e068c1) /*********************************************************************************************************************************** Level constants ***********************************************************************************************************************************/ #define LZ4_COMPRESS_LEVEL_MIN -5 #define LZ4_COMPRESS_LEVEL_MAX 12 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *lz4CompressNew(int level, bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/lz4/decompress.c0000664000175100017510000001601315240547630022347 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Decompress ***********************************************************************************************************************************/ #include #include #include #include "common/compress/common.h" #include "common/compress/lz4/common.h" #include "common/compress/lz4/decompress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Lz4Decompress { LZ4F_decompressionContext_t context; // LZ4 decompression context IoFilter *filter; // Filter interface bool inputSame; // Is the same input required on the next process call? size_t inputOffset; // Current offset from the start of the buffer bool frameDone; // Has the current frame completed? bool done; // Is decompression done? } Lz4Decompress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void lz4DecompressToLog(const Lz4Decompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, inputOffset: %zu, frameDone %s, done: %s}", cvtBoolToConstZ(this->inputSame), this->inputOffset, cvtBoolToConstZ(this->frameDone), cvtBoolToConstZ(this->done)); } #define FUNCTION_LOG_LZ4_DECOMPRESS_TYPE \ Lz4Decompress * #define FUNCTION_LOG_LZ4_DECOMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, lz4DecompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free decompression context ***********************************************************************************************************************************/ static void lz4DecompressFreeResource(THIS_VOID) { THIS(Lz4Decompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(LZ4_DECOMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); LZ4F_freeDecompressionContext(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Decompress data ***********************************************************************************************************************************/ static void lz4DecompressProcess(THIS_VOID, const Buffer *const compressed, Buffer *const decompressed) { THIS(Lz4Decompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(LZ4_DECOMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_PARAM(BUFFER, decompressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->context != NULL); ASSERT(decompressed != NULL); // When there is no more input then decompression is done if (compressed == NULL) { // If the current frame being decompressed was not completed then error if (!this->frameDone) THROW(FormatError, "unexpected eof in compressed data"); this->done = true; } else { // Decompress as much data as possible size_t srcSize = bufUsed(compressed) - this->inputOffset; size_t dstSize = bufRemains(decompressed); this->frameDone = lz4Error( LZ4F_decompress( this->context, bufRemainsPtr(decompressed), &dstSize, bufPtrConst(compressed) + this->inputOffset, &srcSize, NULL)) == 0; bufUsedInc(decompressed, dstSize); // If the compressed data was not fully processed then update the offset and set inputSame if (srcSize < bufUsed(compressed) - this->inputOffset) { this->inputOffset += srcSize; this->inputSame = true; } // Else all compressed data was processed else { this->inputOffset = 0; this->inputSame = false; } } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is decompress done? ***********************************************************************************************************************************/ static bool lz4DecompressDone(const THIS_VOID) { THIS(const Lz4Decompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool lz4DecompressInputSame(const THIS_VOID) { THIS(const Lz4Decompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LZ4_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * lz4DecompressNew(const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); (void)raw; // Not required for decompress FUNCTION_LOG_END(); OBJ_NEW_BEGIN(Lz4Decompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Lz4Decompress){0}; // Create lz4 context lz4Error(LZ4F_createDecompressionContext(&this->context, LZ4F_VERSION)); // Set callback to ensure lz4 context is freed memContextCallbackSet(objMemContext(this), lz4DecompressFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( LZ4_DECOMPRESS_FILTER_TYPE, this, decompressParamList(raw), .done = lz4DecompressDone, .inOut = lz4DecompressProcess, .inputSame = lz4DecompressInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/lz4/decompress.h0000664000175100017510000000221515240547630022353 0ustar00runnerrunner/*********************************************************************************************************************************** LZ4 Decompress Decompress IO from the lz4 format. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_LZ4_DECOMPRESS_H #define COMMON_COMPRESS_LZ4_DECOMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define LZ4_DECOMPRESS_FILTER_TYPE STRID6("lz4-dcmp", 0x40d0c46e068c1) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *lz4DecompressNew(bool raw); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/zst/0000775000175100017510000000000015240547630020145 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/zst/common.c0000664000175100017510000000200615240547630021577 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Common ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBZST #include // Check the version -- this is done in configure but it makes sense to be sure #if ZSTD_VERSION_MAJOR < 1 #error "ZSTD_VERSION_MAJOR must be >= 1" #endif #include "common/compress/zst/common.h" #include "common/debug.h" /**********************************************************************************************************************************/ FN_EXTERN size_t zstError(const size_t error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, error); FUNCTION_TEST_END(); if (ZSTD_isError(error)) THROW_FMT(FormatError, "zst error: [%zd] %s", (ssize_t)error, ZSTD_getErrorName(error)); FUNCTION_TEST_RETURN(SIZE, error); } #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/zst/common.h0000664000175100017510000000224715240547630021613 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Common Developed using the documentation in https://github.com/facebook/zstd/blob/v1.0.0/lib/zstd.h. ***********************************************************************************************************************************/ #ifndef COMMON_COMPRESS_ZST_COMMON_H #define COMMON_COMPRESS_ZST_COMMON_H #include /*********************************************************************************************************************************** ZST extension ***********************************************************************************************************************************/ #define ZST_EXT "zst" #ifdef HAVE_LIBZST /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN size_t zstError(size_t error); #endif // HAVE_LIBZST #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/zst/compress.c0000664000175100017510000001651215240547630022151 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Compress ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBZST #include #include "common/compress/common.h" #include "common/compress/zst/common.h" #include "common/compress/zst/compress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ZstCompress { ZSTD_CStream *context; // Compression context int level; // Compression level IoFilter *filter; // Filter interface bool inputSame; // Is the same input required on the next process call? size_t inputOffset; // Current offset in input buffer bool flushing; // Is input complete and flushing in progress? } ZstCompress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void zstCompressToLog(const ZstCompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{level: %d, inputSame: %s, inputOffset: %zu, flushing: %s}", this->level, cvtBoolToConstZ(this->inputSame), this->inputOffset, cvtBoolToConstZ(this->flushing)); } #define FUNCTION_LOG_ZST_COMPRESS_TYPE \ ZstCompress * #define FUNCTION_LOG_ZST_COMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, zstCompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free compression context ***********************************************************************************************************************************/ static void zstCompressFreeResource(THIS_VOID) { THIS(ZstCompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ZST_COMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ZSTD_freeCStream(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Compress data ***********************************************************************************************************************************/ static void zstCompressProcess(THIS_VOID, const Buffer *const uncompressed, Buffer *const compressed) { THIS(ZstCompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ZST_COMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, uncompressed); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!(this->flushing && !this->inputSame)); ASSERT(this->context != NULL); ASSERT(compressed != NULL); ASSERT(!this->flushing || uncompressed == NULL); // Initialize output buffer ZSTD_outBuffer out = {.dst = bufRemainsPtr(compressed), .size = bufRemains(compressed)}; // If input is NULL then start flushing if (uncompressed == NULL) { this->flushing = true; this->inputSame = zstError(ZSTD_endStream(this->context, &out)) != 0; } // Else still have input data else { // Initialize input buffer ZSTD_inBuffer in = { .src = bufPtrConst(uncompressed) + this->inputOffset, .size = bufUsed(uncompressed) - this->inputOffset, }; // Perform compression zstError(ZSTD_compressStream(this->context, &out, &in)); // If the input buffer was not entirely consumed then set inputSame and store the offset where processing will restart if (in.pos < in.size) { // Output buffer should be completely full ASSERT(out.pos == out.size); this->inputSame = true; this->inputOffset += in.pos; } // Else ready for more input else { this->inputSame = false; this->inputOffset = 0; } } bufUsedInc(compressed, out.pos); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is compress done? ***********************************************************************************************************************************/ static bool zstCompressDone(const THIS_VOID) { THIS(const ZstCompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ZST_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->flushing && !this->inputSame); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool zstCompressInputSame(const THIS_VOID) { THIS(const ZstCompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ZST_COMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * zstCompressNew(const int level, const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, level); (void)raw; // Raw unsupported FUNCTION_LOG_END(); ASSERT(level >= ZST_COMPRESS_LEVEL_MIN && level <= ZST_COMPRESS_LEVEL_MAX); OBJ_NEW_BEGIN(ZstCompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (ZstCompress) { .context = ZSTD_createCStream(), .level = level, }; // Set callback to ensure zst context is freed memContextCallbackSet(objMemContext(this), zstCompressFreeResource, this); // Initialize context zstError(ZSTD_initCStream(this->context, this->level)); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( ZST_COMPRESS_FILTER_TYPE, this, compressParamList(level, raw), .done = zstCompressDone, .inOut = zstCompressProcess, .inputSame = zstCompressInputSame)); } #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/zst/compress.h0000664000175100017510000000313415240547630022152 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Compress Compress IO to the zst format. ***********************************************************************************************************************************/ #ifdef HAVE_LIBZST #ifndef COMMON_COMPRESS_ZST_COMPRESS_H #define COMMON_COMPRESS_ZST_COMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define ZST_COMPRESS_FILTER_TYPE STRID5("zst-cmp", 0x41a3dd27a0) /*********************************************************************************************************************************** Level constants ***********************************************************************************************************************************/ #define ZST_COMPRESS_LEVEL_MIN -7 #define ZST_COMPRESS_LEVEL_MAX 22 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *zstCompressNew(int level, bool raw); #endif #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/zst/decompress.c0000664000175100017510000001624215240547630022462 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Decompress ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBZST #include #include "common/compress/common.h" #include "common/compress/zst/common.h" #include "common/compress/zst/decompress.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ZstDecompress { ZSTD_DStream *context; // Decompression context IoFilter *filter; // Filter interface bool inputSame; // Is the same input required on the next process call? size_t inputOffset; // Current offset in input buffer bool frameDone; // Has the current frame completed? bool done; // Is decompression done? } ZstDecompress; /*********************************************************************************************************************************** Render as string for logging ***********************************************************************************************************************************/ static void zstDecompressToLog(const ZstDecompress *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, inputOffset: %zu, frameDone %s, done: %s}", cvtBoolToConstZ(this->inputSame), this->inputOffset, cvtBoolToConstZ(this->frameDone), cvtBoolToConstZ(this->done)); } #define FUNCTION_LOG_ZST_DECOMPRESS_TYPE \ ZstDecompress * #define FUNCTION_LOG_ZST_DECOMPRESS_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, zstDecompressToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free decompression context ***********************************************************************************************************************************/ static void zstDecompressFreeResource(THIS_VOID) { THIS(ZstDecompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ZST_DECOMPRESS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ZSTD_freeDStream(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Decompress data ***********************************************************************************************************************************/ static void zstDecompressProcess(THIS_VOID, const Buffer *const compressed, Buffer *const decompressed) { THIS(ZstDecompress); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ZST_DECOMPRESS, this); FUNCTION_LOG_PARAM(BUFFER, compressed); FUNCTION_LOG_PARAM(BUFFER, decompressed); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->context != NULL); ASSERT(decompressed != NULL); // When there is no more input then decompression is done if (compressed == NULL) { // If the current frame being decompressed was not completed then error if (!this->frameDone) THROW(FormatError, "unexpected eof in compressed data"); this->done = true; } else { // Initialize input/output buffer ZSTD_inBuffer in = {.src = bufPtrConst(compressed) + this->inputOffset, .size = bufUsed(compressed) - this->inputOffset}; ZSTD_outBuffer out = {.dst = bufRemainsPtr(decompressed), .size = bufRemains(decompressed)}; // Perform decompression. Track frame done so we can detect unexpected EOF. this->frameDone = zstError(ZSTD_decompressStream(this->context, &out, &in)) == 0; bufUsedInc(decompressed, out.pos); // If the input buffer was not entirely consumed then set inputSame and store the offset where processing will restart if (in.pos < in.size) { // Output buffer should be completely full ASSERT(out.pos == out.size); this->inputSame = true; this->inputOffset += in.pos; } // Else ready for more input else { this->inputOffset = 0; this->inputSame = false; } } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is decompress done? ***********************************************************************************************************************************/ static bool zstDecompressDone(const THIS_VOID) { THIS(const ZstDecompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ZST_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done); } /*********************************************************************************************************************************** Is the same input required on the next process call? ***********************************************************************************************************************************/ static bool zstDecompressInputSame(const THIS_VOID) { THIS(const ZstDecompress); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ZST_DECOMPRESS, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * zstDecompressNew(const bool raw) { FUNCTION_LOG_BEGIN(logLevelTrace); (void)raw; // Raw unsupported FUNCTION_LOG_END(); OBJ_NEW_BEGIN(ZstDecompress, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (ZstDecompress) { .context = ZSTD_createDStream(), }; // Set callback to ensure zst context is freed memContextCallbackSet(objMemContext(this), zstDecompressFreeResource, this); // Initialize context zstError(ZSTD_initDStream(this->context)); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( ZST_DECOMPRESS_FILTER_TYPE, this, decompressParamList(raw), .done = zstDecompressDone, .inOut = zstDecompressProcess, .inputSame = zstDecompressInputSame)); } #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/compress/zst/decompress.h0000664000175100017510000000226615240547630022470 0ustar00runnerrunner/*********************************************************************************************************************************** ZST Decompress Decompress IO from the zst format. ***********************************************************************************************************************************/ #ifdef HAVE_LIBZST #ifndef COMMON_COMPRESS_ZST_DECOMPRESS_H #define COMMON_COMPRESS_ZST_DECOMPRESS_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define ZST_DECOMPRESS_FILTER_TYPE STRID5("zst-dcmp", 0x83464dd27a0) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *zstDecompressNew(bool raw); #endif #endif // HAVE_LIBZST ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/0000775000175100017510000000000015240547630017012 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/cipherBlock.c0000664000175100017510000004703315240547630021412 0ustar00runnerrunner/*********************************************************************************************************************************** Block Cipher ***********************************************************************************************************************************/ #include #include #include #include #include "common/crypto/cipherBlock.h" #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Header constants and sizes ***********************************************************************************************************************************/ // Magic constant for salted encrypt. Only salted encrypt is done here, but this constant is required for compatibility with the // openssl command-line tool. #define CIPHER_BLOCK_MAGIC "Salted__" #define CIPHER_BLOCK_MAGIC_SIZE (sizeof(CIPHER_BLOCK_MAGIC) - 1) // Total length of cipher header #define CIPHER_BLOCK_HEADER_SIZE (CIPHER_BLOCK_MAGIC_SIZE + PKCS5_SALT_LEN) /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct CipherBlock { CipherMode mode; // Mode encrypt/decrypt bool raw; // Omit header magic to save space bool saltDone; // Has the salt been read/generated? bool processDone; // Has any data been processed? const Buffer *pass; // Passphrase used to generate encryption key size_t headerSize; // Size of header read during decrypt uint8_t header[CIPHER_BLOCK_HEADER_SIZE]; // Buffer to hold partial header during decrypt const EVP_CIPHER *cipher; // Cipher object const EVP_MD *digest; // Message digest object EVP_CIPHER_CTX *cipherContext; // Encrypt/decrypt context Buffer *buffer; // Internal buffer in case destination buffer isn't large enough bool inputSame; // Is the same input required on next process call? bool done; // Is processing done? } CipherBlock; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void cipherBlockToLog(const CipherBlock *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{inputSame: %s, done: %s}", cvtBoolToConstZ(this->inputSame), cvtBoolToConstZ(this->done)); } #define FUNCTION_LOG_CIPHER_BLOCK_TYPE \ CipherBlock * #define FUNCTION_LOG_CIPHER_BLOCK_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, cipherBlockToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free cipher context ***********************************************************************************************************************************/ static void cipherBlockFreeResource(THIS_VOID) { THIS(CipherBlock); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_END(); ASSERT(this != NULL); EVP_CIPHER_CTX_free(this->cipherContext); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Determine how large the destination buffer should be ***********************************************************************************************************************************/ static size_t cipherBlockProcessSize(const CipherBlock *const this, const size_t sourceSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_PARAM(SIZE, sourceSize); FUNCTION_LOG_END(); ASSERT(this != NULL); // Destination size is source size plus one extra block size_t destinationSize = sourceSize + EVP_MAX_BLOCK_LENGTH; // On encrypt the header size must be included before the first block if (this->mode == cipherModeEncrypt && !this->saltDone) destinationSize += CIPHER_BLOCK_MAGIC_SIZE + PKCS5_SALT_LEN; FUNCTION_LOG_RETURN(SIZE, destinationSize); } /*********************************************************************************************************************************** Encrypt/decrypt data ***********************************************************************************************************************************/ static size_t cipherBlockProcessBlock(CipherBlock *const this, const uint8_t *source, size_t sourceSize, uint8_t *destination) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_PARAM_P(BYTEDATA, source); FUNCTION_LOG_PARAM(SIZE, sourceSize); FUNCTION_LOG_PARAM_P(BYTEDATA, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(source != NULL || sourceSize == 0); ASSERT(destination != NULL); // Actual destination size size_t destinationSize = 0; // If the salt has not been generated/read yet if (!this->saltDone) { const uint8_t *salt = NULL; // On encrypt the salt is generated if (this->mode == cipherModeEncrypt) { // Add magic to the destination buffer so openssl knows the file is salted if (!this->raw) { memcpy(destination, CIPHER_BLOCK_MAGIC, CIPHER_BLOCK_MAGIC_SIZE); destination += CIPHER_BLOCK_MAGIC_SIZE; destinationSize += CIPHER_BLOCK_MAGIC_SIZE; } // Add salt to the destination buffer cryptoRandomBytes(destination, PKCS5_SALT_LEN); salt = destination; destination += PKCS5_SALT_LEN; destinationSize += PKCS5_SALT_LEN; } // On decrypt the salt is read from the header else if (sourceSize > 0) { // Check if the entire header has been read const size_t headerExpected = this->raw ? PKCS5_SALT_LEN : CIPHER_BLOCK_HEADER_SIZE; if (this->headerSize + sourceSize >= headerExpected) { // Copy header (or remains of header) from source into the header buffer memcpy(this->header + this->headerSize, source, headerExpected - this->headerSize); salt = this->header + (this->raw ? 0 : CIPHER_BLOCK_MAGIC_SIZE); // Advance source and source size by the number of bytes read source += headerExpected - this->headerSize; sourceSize -= headerExpected - this->headerSize; // The first bytes of the file to decrypt should be equal to the magic. If not then this is not an encrypted file, // or at least not in a format we recognize. if (!this->raw && memcmp(this->header, CIPHER_BLOCK_MAGIC, CIPHER_BLOCK_MAGIC_SIZE) != 0) THROW(CryptoError, "cipher header invalid"); } // Else copy what was provided into the header buffer and return 0 else { memcpy(this->header + this->headerSize, source, sourceSize); this->headerSize += sourceSize; // Indicate that there is nothing left to process sourceSize = 0; } } // If salt generation/read is done if (salt) { // Generate key and initialization vector uint8_t key[EVP_MAX_KEY_LENGTH]; uint8_t initVector[EVP_MAX_IV_LENGTH]; EVP_BytesToKey(this->cipher, this->digest, salt, bufPtrConst(this->pass), (int)bufSize(this->pass), 1, key, initVector); // Create context to track cipher cryptoError(!(this->cipherContext = EVP_CIPHER_CTX_new()), "unable to create context"); // Set free callback to ensure cipher context is freed memContextCallbackSet(objMemContext(this), cipherBlockFreeResource, this); // Initialize cipher cryptoError( !EVP_CipherInit_ex(this->cipherContext, this->cipher, NULL, key, initVector, this->mode == cipherModeEncrypt), "unable to initialize cipher"); this->saltDone = true; } } // Recheck that source size > 0 as the bytes may have been consumed reading the header if (sourceSize > 0) { // Process the data int destinationUpdateSize = 0; cryptoError( !EVP_CipherUpdate(this->cipherContext, destination, &destinationUpdateSize, source, (int)sourceSize), "unable to process cipher"); destinationSize += (size_t)destinationUpdateSize; // Note that data has been processed so flush is valid this->processDone = true; } // Return actual destination size FUNCTION_LOG_RETURN(SIZE, destinationSize); } /*********************************************************************************************************************************** Flush the remaining data ***********************************************************************************************************************************/ static size_t cipherBlockFlush(CipherBlock *const this, Buffer *const destination) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_PARAM(BUFFER, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(destination != NULL); // Actual destination size int destinationSize = 0; // If no header was processed then error if (!this->saltDone) THROW(CryptoError, "cipher header missing"); // Only flush remaining data if some data was processed if (!EVP_CipherFinal(this->cipherContext, bufRemainsPtr(destination), &destinationSize)) THROW(CryptoError, "unable to flush"); // Return actual destination size FUNCTION_LOG_RETURN(SIZE, (size_t)destinationSize); } /*********************************************************************************************************************************** Process function used by C filter ***********************************************************************************************************************************/ static void cipherBlockProcess(THIS_VOID, const Buffer *const source, Buffer *const destination) { THIS(CipherBlock); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CIPHER_BLOCK, this); FUNCTION_LOG_PARAM(BUFFER, source); FUNCTION_LOG_PARAM(BUFFER, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(destination != NULL); ASSERT(bufRemains(destination) > 0); // Copy already buffered bytes if (this->buffer != NULL && !bufEmpty(this->buffer)) { if (bufRemains(destination) >= bufUsed(this->buffer)) { bufCat(destination, this->buffer); bufUsedZero(this->buffer); this->inputSame = false; } else { const size_t catSize = bufRemains(destination); bufCatSub(destination, this->buffer, 0, catSize); memmove(bufPtr(this->buffer), bufPtr(this->buffer) + catSize, bufUsed(this->buffer) - catSize); bufUsedSet(this->buffer, bufUsed(this->buffer) - catSize); this->inputSame = true; } } else { ASSERT(this->buffer == NULL || bufEmpty(this->buffer)); // Determine how much space is required in the output buffer Buffer *outputActual = destination; const size_t destinationSize = cipherBlockProcessSize(this, source == NULL ? 0 : bufUsed(source)); if (destinationSize > bufRemains(destination)) { // Allocate the buffer if needed MEM_CONTEXT_OBJ_BEGIN(this) { if (this->buffer == NULL) { this->buffer = bufNew(destinationSize); } // Resize buffer if needed else bufResize(this->buffer, destinationSize); } MEM_CONTEXT_OBJ_END(); outputActual = this->buffer; } // Encrypt/decrypt bytes size_t destinationSizeActual; if (source == NULL) { // If salt was not generated it means that process() was never called with any data. It's OK to encrypt a zero byte file // but we need to call process to generate the header. if (!this->saltDone) { destinationSizeActual = cipherBlockProcessBlock(this, NULL, 0, bufRemainsPtr(outputActual)); bufUsedInc(outputActual, destinationSizeActual); } destinationSizeActual = cipherBlockFlush(this, outputActual); this->done = true; } else { destinationSizeActual = cipherBlockProcessBlock( this, bufPtrConst(source), bufUsed(source), bufRemainsPtr(outputActual)); } bufUsedInc(outputActual, destinationSizeActual); // Copy from buffer to destination if needed if (this->buffer != NULL && !bufEmpty(this->buffer)) cipherBlockProcess(this, source, destination); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is cipher done? ***********************************************************************************************************************************/ static bool cipherBlockDone(const THIS_VOID) { THIS(const CipherBlock); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_BLOCK, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->done && !this->inputSame); } /*********************************************************************************************************************************** Should the same input be provided again? ***********************************************************************************************************************************/ static bool cipherBlockInputSame(const THIS_VOID) { THIS(const CipherBlock); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CIPHER_BLOCK, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * cipherBlockNew(const CipherMode mode, const CipherType cipherType, const Buffer *const pass, const CipherBlockNewParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, mode); FUNCTION_LOG_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(BUFFER, pass); // Use FUNCTION_TEST so passphrase is not logged FUNCTION_LOG_PARAM(STRING, param.digest); FUNCTION_LOG_PARAM(BOOL, param.raw); FUNCTION_LOG_END(); ASSERT(pass != NULL); ASSERT(!bufEmpty(pass)); // Init crypto subsystem cryptoInit(); // Lookup cipher by name. This means the ciphers passed in must exactly match a name expected by OpenSSL. This is a good thing // since the name required by the openssl command-line tool will match what is used by pgBackRest. char *const cipherTypeZ = zNewStrId(cipherType); const EVP_CIPHER *cipher = EVP_get_cipherbyname(cipherTypeZ); if (!cipher) THROW_FMT(AssertError, "unable to load cipher '%s'", cipherTypeZ); zFree(cipherTypeZ); // Lookup digest. If not defined it will be set to sha1. const EVP_MD *digest = NULL; if (param.digest) digest = EVP_get_digestbyname(strZ(param.digest)); else digest = EVP_sha1(); if (!digest) THROW_FMT(AssertError, "unable to load digest '%s'", strZ(param.digest)); OBJ_NEW_BEGIN(CipherBlock, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (CipherBlock) { .mode = mode, .raw = param.raw, .cipher = cipher, .digest = digest, .pass = bufDup(pass), }; } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, mode); pckWriteU64P(packWrite, cipherType); pckWriteBinP(packWrite, pass); pckWriteStrP(packWrite, param.digest); pckWriteBoolP(packWrite, param.raw); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP( CIPHER_BLOCK_FILTER_TYPE, this, paramList, .done = cipherBlockDone, .inOut = cipherBlockProcess, .inputSame = cipherBlockInputSame)); } FN_EXTERN IoFilter * cipherBlockNewPack(const Pack *const paramList) { IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const paramListPack = pckReadNew(paramList); const CipherMode cipherMode = (CipherMode)pckReadU64P(paramListPack); const CipherType cipherType = (CipherType)pckReadU64P(paramListPack); const Buffer *const pass = pckReadBinP(paramListPack); const String *const digest = pckReadStrP(paramListPack); const bool raw = pckReadBoolP(paramListPack); result = ioFilterMove(cipherBlockNewP(cipherMode, cipherType, pass, .digest = digest, .raw = raw), memContextPrior()); } MEM_CONTEXT_TEMP_END(); return result; } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * cipherBlockFilterGroupAdd(IoFilterGroup *const filterGroup, const CipherType type, const CipherMode mode, const String *const pass) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING_ID, mode); FUNCTION_TEST_PARAM(STRING, pass); // Use FUNCTION_TEST so passphrase is not logged FUNCTION_LOG_END(); ASSERT(filterGroup != NULL); ASSERT((type == cipherTypeNone && pass == NULL) || (type != cipherTypeNone && pass != NULL)); if (type != cipherTypeNone) ioFilterGroupAdd(filterGroup, cipherBlockNewP(mode, type, BUFSTR(pass))); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, filterGroup); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/cipherBlock.h0000664000175100017510000000434615240547630021417 0ustar00runnerrunner/*********************************************************************************************************************************** Block Cipher Header ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_CIPHERBLOCK_H #define COMMON_CRYPTO_CIPHERBLOCK_H #include "common/crypto/common.h" #include "common/io/filter/group.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define CIPHER_BLOCK_FILTER_TYPE STRID5("cipher-blk", 0x16c16e45441230) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct CipherBlockNewParam { VAR_PARAM_HEADER; const String *digest; // Digest to use (defaults to SHA-1) bool raw; // Omit header magic to save space } CipherBlockNewParam; #define cipherBlockNewP(mode, cipherType, pass, ...) \ cipherBlockNew(mode, cipherType, pass, (CipherBlockNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoFilter *cipherBlockNew(CipherMode mode, CipherType cipherType, const Buffer *pass, CipherBlockNewParam param); FN_EXTERN IoFilter *cipherBlockNewPack(const Pack *paramList); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Add a block cipher to an io object FN_EXTERN IoFilterGroup *cipherBlockFilterGroupAdd( IoFilterGroup *filterGroup, CipherType type, CipherMode mode, const String *pass); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/common.c0000664000175100017510000000542615240547630020455 0ustar00runnerrunner/*********************************************************************************************************************************** Crypto Common ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/crypto/common.h" #include "common/debug.h" #include "common/log.h" /*********************************************************************************************************************************** Flag to indicate if OpenSSL has already been initialized ***********************************************************************************************************************************/ static bool cryptoInitDone = false; /**********************************************************************************************************************************/ FN_EXTERN void cryptoError(const bool error, const char *const description) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, error); FUNCTION_TEST_PARAM(STRINGZ, description); FUNCTION_TEST_END(); if (error) cryptoErrorCode(ERR_get_error(), description); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void cryptoErrorCode(const unsigned long code, const char *const description) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, code); FUNCTION_TEST_PARAM(STRINGZ, description); FUNCTION_TEST_END(); const char *const errorMessage = ERR_reason_error_string(code); THROW_FMT(CryptoError, "%s: [%lu] %s", description, code, errorMessage == NULL ? "no details available" : errorMessage); FUNCTION_TEST_NO_RETURN(); } /**********************************************************************************************************************************/ FN_EXTERN void cryptoInit(void) { FUNCTION_LOG_VOID(logLevelTrace); if (!cryptoInitDone) { // Load crypto strings and algorithms ERR_load_crypto_strings(); OpenSSL_add_all_algorithms(); // Initialization OPENSSL_init_ssl(OPENSSL_INIT_LOAD_CONFIG, NULL); // Mark crypto as initialized cryptoInitDone = true; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cryptoRandomBytes(uint8_t *const buffer, const size_t size) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(BYTEDATA, buffer); FUNCTION_LOG_PARAM(SIZE, size); FUNCTION_LOG_END(); ASSERT(buffer != NULL); ASSERT(size > 0); RAND_bytes(buffer, (int)size); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/common.h0000664000175100017510000000434015240547630020454 0ustar00runnerrunner/*********************************************************************************************************************************** Crypto Common ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_COMMON_H #define COMMON_CRYPTO_COMMON_H #include /*********************************************************************************************************************************** Cipher modes ***********************************************************************************************************************************/ typedef enum { cipherModeEncrypt = STRID5("encrypt", 0x521990dc50), cipherModeDecrypt = STRID5("decrypt", 0x521990ca40), } CipherMode; /*********************************************************************************************************************************** Cipher types ***********************************************************************************************************************************/ typedef enum { cipherTypeNone = STRID5("none", 0x2b9ee0), cipherTypeAes256Cbc = STRID5("aes-256-cbc", 0xc43dfbbcdcca10), } CipherType; /*********************************************************************************************************************************** Hash types ***********************************************************************************************************************************/ typedef enum { hashTypeMd5 = STRID5("md5", 0x748d0), hashTypeSha1 = STRID6("sha1", 0x7412131), hashTypeSha256 = STRID5("sha256", 0x3dde05130), } HashType; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize crypto FN_EXTERN void cryptoInit(void); // Throw crypto errors FN_EXTERN void cryptoError(bool error, const char *description); FN_EXTERN noreturn void cryptoErrorCode(unsigned long code, const char *description); // Generate random bytes FN_EXTERN void cryptoRandomBytes(uint8_t *buffer, size_t size); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/hash.c0000664000175100017510000002624315240547630020110 0ustar00runnerrunner/*********************************************************************************************************************************** Cryptographic Hash ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/crypto/common.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Hashes for zero-length files (i.e., seed value) ***********************************************************************************************************************************/ BUFFER_EXTERN( HASH_TYPE_SHA1_ZERO_BUF, 0xda, 0x39, 0xa3, 0xee, 0x5e, 0x6b, 0x4b, 0x0d, 0x32, 0x55, 0xbf, 0xef, 0x95, 0x60, 0x18, 0x90, 0xaf, 0xd8, 0x07, 0x09); BUFFER_EXTERN( HASH_TYPE_SHA256_ZERO_BUF, 0xe3, 0xb0, 0xc4, 0x42, 0x98, 0xfc, 0x1c, 0x14, 0x9a, 0xfb, 0xf4, 0xc8, 0x99, 0x6f, 0xb9, 0x24, 0x27, 0xae, 0x41, 0xe4, 0x64, 0x9b, 0x93, 0x4c, 0xa4, 0x95, 0x99, 0x1b, 0x78, 0x52, 0xb8, 0x55); /*********************************************************************************************************************************** Include local MD5 code ***********************************************************************************************************************************/ #include "common/crypto/md5.vendor.c.inc" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct CryptoHash { const EVP_MD *hashType; // Hash type (sha1, md5, etc.) EVP_MD_CTX *hashContext; // Message hash context MD5_CTX md5Context; // MD5 context (used to bypass FIPS restrictions) Buffer *hash; // Hash in binary form } CryptoHash; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_CRYPTO_HASH_TYPE \ CryptoHash * #define FUNCTION_LOG_CRYPTO_HASH_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "CryptoHash", buffer, bufferSize) /*********************************************************************************************************************************** Free hash context ***********************************************************************************************************************************/ static void cryptoHashFreeResource(THIS_VOID) { THIS(CryptoHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CRYPTO_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); EVP_MD_CTX_destroy(this->hashContext); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Add message data to the hash from a Buffer ***********************************************************************************************************************************/ static void cryptoHashProcess(THIS_VOID, const Buffer *const message) { THIS(CryptoHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CRYPTO_HASH, this); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->hash == NULL); ASSERT(message != NULL); // Standard OpenSSL implementation if (this->hashContext != NULL) { cryptoError(!EVP_DigestUpdate(this->hashContext, bufPtrConst(message), bufUsed(message)), "unable to process message hash"); } // Else local MD5 implementation else MD5_Update(&this->md5Context, bufPtrConst(message), bufUsed(message)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get binary representation of the hash ***********************************************************************************************************************************/ static const Buffer * cryptoHash(CryptoHash *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CRYPTO_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); if (this->hash == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { // Standard OpenSSL implementation if (this->hashContext != NULL) { this->hash = bufNew((size_t)EVP_MD_size(this->hashType)); cryptoError(!EVP_DigestFinal_ex(this->hashContext, bufPtr(this->hash), NULL), "unable to finalize message hash"); } // Else local MD5 implementation else { this->hash = bufNew(HASH_TYPE_M5_SIZE); MD5_Final(bufPtr(this->hash), &this->md5Context); } bufUsedSet(this->hash, bufSize(this->hash)); } MEM_CONTEXT_OBJ_END(); } FUNCTION_LOG_RETURN(BUFFER, this->hash); } /*********************************************************************************************************************************** Get string representation of the hash as a filter result ***********************************************************************************************************************************/ static Pack * cryptoHashResult(THIS_VOID) { THIS(CryptoHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(CRYPTO_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteBinP(packWrite, cryptoHash(this)); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * cryptoHashNew(const HashType type) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_END(); ASSERT(type != 0); // Init crypto subsystem cryptoInit(); OBJ_NEW_BEGIN(CryptoHash, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (CryptoHash){0}; // Use local MD5 implementation since FIPS-enabled systems do not allow MD5. This is a bit misguided since there are valid // cases for using MD5 which do not involve, for example, password hashes. Since popular object stores, e.g. S3, require // MD5 for verifying payload integrity we are simply forced to provide MD5 functionality. if (type == hashTypeMd5) { MD5_Init(&this->md5Context); } // Else use the standard OpenSSL implementation else { // Lookup digest char typeZ[STRID_MAX + 1]; strIdToZ(type, typeZ); if ((this->hashType = EVP_get_digestbyname(typeZ)) == NULL) THROW_FMT(AssertError, "unable to load hash '%s'", typeZ); // Create context cryptoError((this->hashContext = EVP_MD_CTX_create()) == NULL, "unable to create hash context"); // Set free callback to ensure hash context is freed memContextCallbackSet(objMemContext(this), cryptoHashFreeResource, this); // Initialize context cryptoError(!EVP_DigestInit_ex(this->hashContext, this->hashType, NULL), "unable to initialize hash context"); } } OBJ_NEW_END(); // Create param list Pack *paramList; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteStrIdP(packWrite, type); pckWriteEndP(packWrite); paramList = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP(CRYPTO_HASH_FILTER_TYPE, this, paramList, .in = cryptoHashProcess, .result = cryptoHashResult)); } FN_EXTERN IoFilter * cryptoHashNewPack(const Pack *const paramList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK, paramList); FUNCTION_TEST_END(); IoFilter *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { result = ioFilterMove(cryptoHashNew(pckReadStrIdP(pckReadNew(paramList))), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(IO_FILTER, result); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * cryptoHashOne(const HashType type, const Buffer *const message) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(type != 0); ASSERT(message != NULL); Buffer *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { IoFilter *const hash = cryptoHashNew(type); if (!bufEmpty(message)) ioFilterProcessIn(hash, message); const Buffer *const buffer = cryptoHash((CryptoHash *)ioFilterDriver(hash)); MEM_CONTEXT_PRIOR_BEGIN() { result = bufDup(buffer); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * cryptoHmacOne(const HashType type, const Buffer *const key, const Buffer *const message) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(BUFFER, key); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(type != 0); ASSERT(key != NULL); ASSERT(message != NULL); // Init crypto subsystem cryptoInit(); // Lookup digest char typeZ[STRID_MAX + 1]; strIdToZ(type, typeZ); const EVP_MD *const hashType = EVP_get_digestbyname(typeZ); ASSERT(hashType != NULL); // Allocate a buffer to hold the hmac Buffer *const result = bufNew((size_t)EVP_MD_size(hashType)); bufUsedSet(result, bufSize(result)); // Calculate the HMAC HMAC(hashType, bufPtrConst(key), (int)bufUsed(key), bufPtrConst(message), bufUsed(message), bufPtr(result), NULL); FUNCTION_LOG_RETURN(BUFFER, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/hash.h0000664000175100017510000000652515240547630020116 0ustar00runnerrunner/*********************************************************************************************************************************** Cryptographic Hash Generate a hash (sha1, md5, etc.) from a string, Buffer, or using an IoFilter. ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_HASH_H #define COMMON_CRYPTO_HASH_H #include "common/crypto/common.h" #include "common/io/filter/filter.h" #include "common/type/string.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define CRYPTO_HASH_FILTER_TYPE STRID5("hash", 0x44c280) /*********************************************************************************************************************************** Hashes for zero-length files (i.e., starting hash) ***********************************************************************************************************************************/ #define HASH_TYPE_MD5_ZERO "d41d8cd98f00b204e9800998ecf8427e" #define HASH_TYPE_SHA1_ZERO "da39a3ee5e6b4b0d3255bfef95601890afd80709" BUFFER_DECLARE(HASH_TYPE_SHA1_ZERO_BUF); #define HASH_TYPE_SHA256_ZERO \ "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" BUFFER_DECLARE(HASH_TYPE_SHA256_ZERO_BUF); /*********************************************************************************************************************************** Hash type sizes ***********************************************************************************************************************************/ #define HASH_TYPE_M5_SIZE 16 #define HASH_TYPE_MD5_SIZE_HEX (HASH_TYPE_M5_SIZE * 2) #define HASH_TYPE_SHA1_SIZE 20 #define HASH_TYPE_SHA1_SIZE_HEX (HASH_TYPE_SHA1_SIZE * 2) #define HASH_TYPE_SHA256_SIZE 32 #define HASH_TYPE_SHA256_SIZE_HEX (HASH_TYPE_SHA256_SIZE * 2) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *cryptoHashNew(HashType type); FN_EXTERN IoFilter *cryptoHashNewPack(const Pack *paramList); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Get hash for one buffer FN_EXTERN Buffer *cryptoHashOne(HashType type, const Buffer *message); // Get hmac for one message/key FN_EXTERN Buffer *cryptoHmacOne(HashType type, const Buffer *key, const Buffer *message); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/md5.vendor.c.inc0000664000175100017510000002131515240547630021711 0ustar00runnerrunner/* * This is an OpenSSL-compatible implementation of the RSA Data Security, Inc. * MD5 Message-Digest Algorithm (RFC 1321). * * Homepage: * http://openwall.info/wiki/people/solar/software/public-domain-source-code/md5 * * Author: * Alexander Peslyak, better known as Solar Designer * * This software was written by Alexander Peslyak in 2001. No copyright is * claimed, and the software is hereby placed in the public domain. * In case this attempt to disclaim copyright and place the software in the * public domain is deemed null and void, then the software is * Copyright (c) 2001 Alexander Peslyak and it is hereby released to the * general public under the following terms: * * Redistribution and use in source and binary forms, with or without * modification, are permitted. * * There's ABSOLUTELY NO WARRANTY, express or implied. * * (This is a heavily cut-down "BSD license".) * * This differs from Colin Plumb's older public domain implementation in that * no exactly 32-bit integer data type is required (any 32-bit or wider * unsigned integer data type will do), there's no compile-time endianness * configuration, and the function prototypes match OpenSSL's. No code from * Colin Plumb's implementation has been reused; this comment merely compares * the properties of the two independent implementations. * * The primary goals of this implementation are portability and ease of use. * It is meant to be fast, but not as fast as possible. Some known * optimizations are not included to reduce source code size and avoid * compile-time configuration. */ #include #include /* Any 32-bit or wider unsigned integer data type will do */ typedef unsigned int MD5_u32plus; typedef struct { MD5_u32plus lo, hi; MD5_u32plus a, b, c, d; uint8_t buffer[64]; MD5_u32plus block[16]; } MD5_CTX; /* * The basic MD5 functions. * * F and G are optimized compared to their RFC 1321 definitions for * architectures that lack an AND-NOT instruction, just like in Colin Plumb's * implementation. */ #define F(x, y, z) ((z) ^ ((x) & ((y) ^ (z)))) #define G(x, y, z) ((y) ^ ((z) & ((x) ^ (y)))) #define H(x, y, z) (((x) ^ (y)) ^ (z)) #define H2(x, y, z) ((x) ^ ((y) ^ (z))) #define I(x, y, z) ((y) ^ ((x) | ~(z))) /* * The MD5 transformation for all four rounds. */ #define STEP(f, a, b, c, d, x, t, s) \ (a) += f((b), (c), (d)) + (x) + (t); \ (a) = (((a) << (s)) | (((a) & 0xffffffff) >> (32 - (s)))); \ (a) += (b); /* * SET reads 4 input bytes in little-endian byte order and stores them in a * properly aligned word in host byte order. * * The check for little-endian architectures that tolerate unaligned memory * accesses is just an optimization. Nothing will break if it fails to detect * a suitable architecture. * * Unfortunately, this optimization may be a C strict aliasing rules violation * if the caller's data buffer has effective type that cannot be aliased by * MD5_u32plus. In practice, this problem may occur if these MD5 routines are * inlined into a calling function, or with future and dangerously advanced * link-time optimizations. For the time being, keeping these MD5 routines in * their own translation unit avoids the problem. */ #if defined(__i386__) || defined(__x86_64__) || defined(__vax__) #define SET(n) \ (*(const MD5_u32plus *)&ptr[(n) * 4]) #define GET(n) \ SET(n) #else #define SET(n) \ (ctx->block[(n)] = \ (MD5_u32plus)ptr[(n) * 4] | \ ((MD5_u32plus)ptr[(n) * 4 + 1] << 8) | \ ((MD5_u32plus)ptr[(n) * 4 + 2] << 16) | \ ((MD5_u32plus)ptr[(n) * 4 + 3] << 24)) #define GET(n) \ (ctx->block[(n)]) #endif /* * This processes one or more 64-byte data blocks, but does NOT update the bit * counters. There are no alignment requirements. */ static const void *body(MD5_CTX *ctx, const void *data, unsigned long size) { const uint8_t *ptr; MD5_u32plus a, b, c, d; MD5_u32plus saved_a, saved_b, saved_c, saved_d; ptr = (const uint8_t *)data; a = ctx->a; b = ctx->b; c = ctx->c; d = ctx->d; do { saved_a = a; saved_b = b; saved_c = c; saved_d = d; /* Round 1 */ STEP(F, a, b, c, d, SET(0), 0xd76aa478, 7) STEP(F, d, a, b, c, SET(1), 0xe8c7b756, 12) STEP(F, c, d, a, b, SET(2), 0x242070db, 17) STEP(F, b, c, d, a, SET(3), 0xc1bdceee, 22) STEP(F, a, b, c, d, SET(4), 0xf57c0faf, 7) STEP(F, d, a, b, c, SET(5), 0x4787c62a, 12) STEP(F, c, d, a, b, SET(6), 0xa8304613, 17) STEP(F, b, c, d, a, SET(7), 0xfd469501, 22) STEP(F, a, b, c, d, SET(8), 0x698098d8, 7) STEP(F, d, a, b, c, SET(9), 0x8b44f7af, 12) STEP(F, c, d, a, b, SET(10), 0xffff5bb1, 17) STEP(F, b, c, d, a, SET(11), 0x895cd7be, 22) STEP(F, a, b, c, d, SET(12), 0x6b901122, 7) STEP(F, d, a, b, c, SET(13), 0xfd987193, 12) STEP(F, c, d, a, b, SET(14), 0xa679438e, 17) STEP(F, b, c, d, a, SET(15), 0x49b40821, 22) /* Round 2 */ STEP(G, a, b, c, d, GET(1), 0xf61e2562, 5) STEP(G, d, a, b, c, GET(6), 0xc040b340, 9) STEP(G, c, d, a, b, GET(11), 0x265e5a51, 14) STEP(G, b, c, d, a, GET(0), 0xe9b6c7aa, 20) STEP(G, a, b, c, d, GET(5), 0xd62f105d, 5) STEP(G, d, a, b, c, GET(10), 0x02441453, 9) STEP(G, c, d, a, b, GET(15), 0xd8a1e681, 14) STEP(G, b, c, d, a, GET(4), 0xe7d3fbc8, 20) STEP(G, a, b, c, d, GET(9), 0x21e1cde6, 5) STEP(G, d, a, b, c, GET(14), 0xc33707d6, 9) STEP(G, c, d, a, b, GET(3), 0xf4d50d87, 14) STEP(G, b, c, d, a, GET(8), 0x455a14ed, 20) STEP(G, a, b, c, d, GET(13), 0xa9e3e905, 5) STEP(G, d, a, b, c, GET(2), 0xfcefa3f8, 9) STEP(G, c, d, a, b, GET(7), 0x676f02d9, 14) STEP(G, b, c, d, a, GET(12), 0x8d2a4c8a, 20) /* Round 3 */ STEP(H, a, b, c, d, GET(5), 0xfffa3942, 4) STEP(H2, d, a, b, c, GET(8), 0x8771f681, 11) STEP(H, c, d, a, b, GET(11), 0x6d9d6122, 16) STEP(H2, b, c, d, a, GET(14), 0xfde5380c, 23) STEP(H, a, b, c, d, GET(1), 0xa4beea44, 4) STEP(H2, d, a, b, c, GET(4), 0x4bdecfa9, 11) STEP(H, c, d, a, b, GET(7), 0xf6bb4b60, 16) STEP(H2, b, c, d, a, GET(10), 0xbebfbc70, 23) STEP(H, a, b, c, d, GET(13), 0x289b7ec6, 4) STEP(H2, d, a, b, c, GET(0), 0xeaa127fa, 11) STEP(H, c, d, a, b, GET(3), 0xd4ef3085, 16) STEP(H2, b, c, d, a, GET(6), 0x04881d05, 23) STEP(H, a, b, c, d, GET(9), 0xd9d4d039, 4) STEP(H2, d, a, b, c, GET(12), 0xe6db99e5, 11) STEP(H, c, d, a, b, GET(15), 0x1fa27cf8, 16) STEP(H2, b, c, d, a, GET(2), 0xc4ac5665, 23) /* Round 4 */ STEP(I, a, b, c, d, GET(0), 0xf4292244, 6) STEP(I, d, a, b, c, GET(7), 0x432aff97, 10) STEP(I, c, d, a, b, GET(14), 0xab9423a7, 15) STEP(I, b, c, d, a, GET(5), 0xfc93a039, 21) STEP(I, a, b, c, d, GET(12), 0x655b59c3, 6) STEP(I, d, a, b, c, GET(3), 0x8f0ccc92, 10) STEP(I, c, d, a, b, GET(10), 0xffeff47d, 15) STEP(I, b, c, d, a, GET(1), 0x85845dd1, 21) STEP(I, a, b, c, d, GET(8), 0x6fa87e4f, 6) STEP(I, d, a, b, c, GET(15), 0xfe2ce6e0, 10) STEP(I, c, d, a, b, GET(6), 0xa3014314, 15) STEP(I, b, c, d, a, GET(13), 0x4e0811a1, 21) STEP(I, a, b, c, d, GET(4), 0xf7537e82, 6) STEP(I, d, a, b, c, GET(11), 0xbd3af235, 10) STEP(I, c, d, a, b, GET(2), 0x2ad7d2bb, 15) STEP(I, b, c, d, a, GET(9), 0xeb86d391, 21) a += saved_a; b += saved_b; c += saved_c; d += saved_d; ptr += 64; } while (size -= 64); ctx->a = a; ctx->b = b; ctx->c = c; ctx->d = d; return ptr; } static void MD5_Init(MD5_CTX *ctx) { *ctx = (MD5_CTX) { .a = 0x67452301, .b = 0xefcdab89, .c = 0x98badcfe, .d = 0x10325476, }; } static void MD5_Update(MD5_CTX *ctx, const void *data, unsigned long size) { MD5_u32plus saved_lo; unsigned long used, available; saved_lo = ctx->lo; if ((ctx->lo = (saved_lo + size) & 0x1fffffff) < saved_lo) ctx->hi++; ctx->hi += (MD5_u32plus)(size >> 29); used = saved_lo & 0x3f; if (used) { available = 64 - used; if (size < available) { memcpy(&ctx->buffer[used], data, size); return; } memcpy(&ctx->buffer[used], data, available); data = (const uint8_t *)data + available; size -= available; body(ctx, ctx->buffer, 64); } if (size >= 64) { data = body(ctx, data, size & ~(unsigned long)0x3f); size &= 0x3f; } memcpy(ctx->buffer, data, size); } #define OUT(dst, src) \ (dst)[0] = (uint8_t)(src); \ (dst)[1] = (uint8_t)((src) >> 8); \ (dst)[2] = (uint8_t)((src) >> 16); \ (dst)[3] = (uint8_t)((src) >> 24); static void MD5_Final(uint8_t *result, MD5_CTX *ctx) { unsigned long used, available; used = ctx->lo & 0x3f; ctx->buffer[used++] = 0x80; available = 64 - used; if (available < 8) { memset(&ctx->buffer[used], 0, available); body(ctx, ctx->buffer, 64); used = 0; available = 64; } memset(&ctx->buffer[used], 0, available - 8); ctx->lo <<= 3; OUT(&ctx->buffer[56], ctx->lo) OUT(&ctx->buffer[60], ctx->hi) body(ctx, ctx->buffer, 64); OUT(&result[0], ctx->a) OUT(&result[4], ctx->b) OUT(&result[8], ctx->c) OUT(&result[12], ctx->d) memset(ctx, 0, sizeof(*ctx)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/xxhash.c0000664000175100017510000001317015240547630020463 0ustar00runnerrunner/*********************************************************************************************************************************** xxHash Interface ***********************************************************************************************************************************/ #include #include "common/crypto/xxhash.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Include local xxHash code ***********************************************************************************************************************************/ #define XXH_INLINE_ALL #include "common/crypto/xxhash.vendor.c.inc" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct XxHash { size_t size; // Size of hash to return XXH3_state_t *state; // xxHash state } XxHash; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_XX_HASH_TYPE \ XxHash * #define FUNCTION_LOG_XX_HASH_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "XxHash", buffer, bufferSize) /*********************************************************************************************************************************** Free hash context ***********************************************************************************************************************************/ static void xxHashFreeResource(THIS_VOID) { THIS(XxHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(XX_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); XXH3_freeState(this->state); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Add message data to the hash from a Buffer ***********************************************************************************************************************************/ static void xxHashProcess(THIS_VOID, const Buffer *const message) { THIS(XxHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(XX_HASH, this); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(message != NULL); XXH3_128bits_update(this->state, bufPtrConst(message), bufUsed(message)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get string representation of the hash as a filter result ***********************************************************************************************************************************/ static Pack * xxHashResult(THIS_VOID) { THIS(XxHash); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(XX_HASH, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); XXH128_canonical_t canonical; XXH128_canonicalFromHash(&canonical, XXH3_128bits_digest(this->state)); pckWriteBinP(packWrite, BUF(canonical.digest, this->size)); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * xxHashNew(const size_t size) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SIZE, size); FUNCTION_LOG_END(); ASSERT(size >= 1 && size <= XX_HASH_SIZE_MAX); OBJ_NEW_BEGIN(XxHash, .callbackQty = 1) { *this = (XxHash){.size = size}; this->state = XXH3_createState(); XXH3_128bits_reset(this->state); // Set free callback to ensure hash context is freed memContextCallbackSet(objMemContext(this), xxHashFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER, ioFilterNewP(XX_HASH_FILTER_TYPE, this, NULL, .in = xxHashProcess, .result = xxHashResult)); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * xxHashOne(const size_t size, const Buffer *const message) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SIZE, size); FUNCTION_LOG_PARAM(BUFFER, message); FUNCTION_LOG_END(); ASSERT(size >= 1 && size <= XX_HASH_SIZE_MAX); ASSERT(message != NULL); Buffer *const result = bufNew(size); XXH128_canonical_t canonical; XXH128_canonicalFromHash(&canonical, XXH3_128bits(bufPtrConst(message), bufUsed(message))); memcpy(bufPtr(result), canonical.digest, size); bufUsedSet(result, size); FUNCTION_LOG_RETURN(BUFFER, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/xxhash.h0000664000175100017510000000456615240547630020501 0ustar00runnerrunner/*********************************************************************************************************************************** xxHash Interface Generate a 128-bit hash and return the specified number of bytes from the canonical representation. The 128-bit variant is only slightly slower than the 64-bit variant so this makes getting up to 16 bytes of hash value fairly simple. We consider this safe because xxHash claims to have excellent dispersion, zstd uses the lower 32-bits of a 64-bit hash for content checksums, xxHash has been tested with SMHasher, and the "128-bit variant has better mixing and strength than the 64-bit variant, even without counting the significantly larger output size". ***********************************************************************************************************************************/ #ifndef COMMON_CRYPTO_XXHASH_H #define COMMON_CRYPTO_XXHASH_H #include "common/io/filter/filter.h" #include "common/type/buffer.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define XX_HASH_FILTER_TYPE STRID5("xxhash", 0x1130a3180) /*********************************************************************************************************************************** Max size of hash ***********************************************************************************************************************************/ #define XX_HASH_SIZE_MAX 16 /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *xxHashNew(size_t size); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Get hash for one buffer FN_EXTERN Buffer *xxHashOne(size_t size, const Buffer *message); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/crypto/xxhash.vendor.c.inc0000664000175100017510000075625015240547630022544 0ustar00runnerrunner/* * xxHash - Extremely Fast Hash algorithm * Header File * Copyright (C) 2012-2021 Yann Collet * * BSD 2-Clause License (https://www.opensource.org/licenses/bsd-license.php) * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions are * met: * * * Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * Redistributions in binary form must reproduce the above * copyright notice, this list of conditions and the following disclaimer * in the documentation and/or other materials provided with the * distribution. * * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. * * You can contact the author at: * - xxHash homepage: https://www.xxhash.com * - xxHash source repository: https://github.com/Cyan4973/xxHash */ /*! * @mainpage xxHash * * xxHash is an extremely fast non-cryptographic hash algorithm, working at RAM speed * limits. * * It is proposed in four flavors, in three families: * 1. @ref XXH32_family * - Classic 32-bit hash function. Simple, compact, and runs on almost all * 32-bit and 64-bit systems. * 2. @ref XXH64_family * - Classic 64-bit adaptation of XXH32. Just as simple, and runs well on most * 64-bit systems (but _not_ 32-bit systems). * 3. @ref XXH3_family * - Modern 64-bit and 128-bit hash function family which features improved * strength and performance across the board, especially on smaller data. * It benefits greatly from SIMD and 64-bit without requiring it. * * Benchmarks * --- * The reference system uses an Intel i7-9700K CPU, and runs Ubuntu x64 20.04. * The open source benchmark program is compiled with clang v10.0 using -O3 flag. * * | Hash Name | ISA ext | Width | Large Data Speed | Small Data Velocity | * | -------------------- | ------- | ----: | ---------------: | ------------------: | * | XXH3_64bits() | @b AVX2 | 64 | 59.4 GB/s | 133.1 | * | MeowHash | AES-NI | 128 | 58.2 GB/s | 52.5 | * | XXH3_128bits() | @b AVX2 | 128 | 57.9 GB/s | 118.1 | * | CLHash | PCLMUL | 64 | 37.1 GB/s | 58.1 | * | XXH3_64bits() | @b SSE2 | 64 | 31.5 GB/s | 133.1 | * | XXH3_128bits() | @b SSE2 | 128 | 29.6 GB/s | 118.1 | * | RAM sequential read | | N/A | 28.0 GB/s | N/A | * | ahash | AES-NI | 64 | 22.5 GB/s | 107.2 | * | City64 | | 64 | 22.0 GB/s | 76.6 | * | T1ha2 | | 64 | 22.0 GB/s | 99.0 | * | City128 | | 128 | 21.7 GB/s | 57.7 | * | FarmHash | AES-NI | 64 | 21.3 GB/s | 71.9 | * | XXH64() | | 64 | 19.4 GB/s | 71.0 | * | SpookyHash | | 64 | 19.3 GB/s | 53.2 | * | Mum | | 64 | 18.0 GB/s | 67.0 | * | CRC32C | SSE4.2 | 32 | 13.0 GB/s | 57.9 | * | XXH32() | | 32 | 9.7 GB/s | 71.9 | * | City32 | | 32 | 9.1 GB/s | 66.0 | * | Blake3* | @b AVX2 | 256 | 4.4 GB/s | 8.1 | * | Murmur3 | | 32 | 3.9 GB/s | 56.1 | * | SipHash* | | 64 | 3.0 GB/s | 43.2 | * | Blake3* | @b SSE2 | 256 | 2.4 GB/s | 8.1 | * | HighwayHash | | 64 | 1.4 GB/s | 6.0 | * | FNV64 | | 64 | 1.2 GB/s | 62.7 | * | Blake2* | | 256 | 1.1 GB/s | 5.1 | * | SHA1* | | 160 | 0.8 GB/s | 5.6 | * | MD5* | | 128 | 0.6 GB/s | 7.8 | * @note * - Hashes which require a specific ISA extension are noted. SSE2 is also noted, * even though it is mandatory on x64. * - Hashes with an asterisk are cryptographic. Note that MD5 is non-cryptographic * by modern standards. * - Small data velocity is a rough average of algorithm's efficiency for small * data. For more accurate information, see the wiki. * - More benchmarks and strength tests are found on the wiki: * https://github.com/Cyan4973/xxHash/wiki * * Usage * ------ * All xxHash variants use a similar API. Changing the algorithm is a trivial * substitution. * * @pre * For functions which take an input and length parameter, the following * requirements are assumed: * - The range from [`input`, `input + length`) is valid, readable memory. * - The only exception is if the `length` is `0`, `input` may be `NULL`. * - For C++, the objects must have the *TriviallyCopyable* property, as the * functions access bytes directly as if it was an array of `unsigned char`. * * @anchor single_shot_example * **Single Shot** * * These functions are stateless functions which hash a contiguous block of memory, * immediately returning the result. They are the easiest and usually the fastest * option. * * XXH32(), XXH64(), XXH3_64bits(), XXH3_128bits() * * @code{.c} * #include * #include "xxhash.h" * * // Example for a function which hashes a null terminated string with XXH32(). * XXH32_hash_t hash_string(const char* string, XXH32_hash_t seed) * { * // NULL pointers are only valid if the length is zero * size_t length = (string == NULL) ? 0 : strlen(string); * return XXH32(string, length, seed); * } * @endcode * * @anchor streaming_example * **Streaming** * * These groups of functions allow incremental hashing of unknown size, even * more than what would fit in a size_t. * * XXH32_reset(), XXH64_reset(), XXH3_64bits_reset(), XXH3_128bits_reset() * * @code{.c} * #include * #include * #include "xxhash.h" * // Example for a function which hashes a FILE incrementally with XXH3_64bits(). * XXH64_hash_t hashFile(FILE* f) * { * // Allocate a state struct. Do not just use malloc() or new. * XXH3_state_t* state = XXH3_createState(); * assert(state != NULL && "Out of memory!"); * // Reset the state to start a new hashing session. * XXH3_64bits_reset(state); * char buffer[4096]; * size_t count; * // Read the file in chunks * while ((count = fread(buffer, 1, sizeof(buffer), f)) != 0) { * // Run update() as many times as necessary to process the data * XXH3_64bits_update(state, buffer, count); * } * // Retrieve the finalized hash. This will not change the state. * XXH64_hash_t result = XXH3_64bits_digest(state); * // Free the state. Do not use free(). * XXH3_freeState(state); * return result; * } * @endcode * * @file xxhash.h * xxHash prototypes and implementation */ #if defined (__cplusplus) extern "C" { #endif /* **************************** * INLINE mode ******************************/ /*! * @defgroup public Public API * Contains details on the public xxHash functions. * @{ */ #ifdef XXH_DOXYGEN /*! * @brief Gives access to internal state declaration, required for static allocation. * * Incompatible with dynamic linking, due to risks of ABI changes. * * Usage: * @code{.c} * #define XXH_STATIC_LINKING_ONLY * #include "xxhash.h" * @endcode */ # define XXH_STATIC_LINKING_ONLY /* Do not undef XXH_STATIC_LINKING_ONLY for Doxygen */ /*! * @brief Gives access to internal definitions. * * Usage: * @code{.c} * #define XXH_STATIC_LINKING_ONLY * #define XXH_IMPLEMENTATION * #include "xxhash.h" * @endcode */ # define XXH_IMPLEMENTATION /* Do not undef XXH_IMPLEMENTATION for Doxygen */ /*! * @brief Exposes the implementation and marks all functions as `inline`. * * Use these build macros to inline xxhash into the target unit. * Inlining improves performance on small inputs, especially when the length is * expressed as a compile-time constant: * * https://fastcompression.blogspot.com/2018/03/xxhash-for-small-keys-impressive-power.html * * It also keeps xxHash symbols private to the unit, so they are not exported. * * Usage: * @code{.c} * #define XXH_INLINE_ALL * #include "xxhash.h" * @endcode * Do not compile and link xxhash.o as a separate object, as it is not useful. */ # define XXH_INLINE_ALL # undef XXH_INLINE_ALL /*! * @brief Exposes the implementation without marking functions as inline. */ # define XXH_PRIVATE_API # undef XXH_PRIVATE_API /*! * @brief Emulate a namespace by transparently prefixing all symbols. * * If you want to include _and expose_ xxHash functions from within your own * library, but also want to avoid symbol collisions with other libraries which * may also include xxHash, you can use @ref XXH_NAMESPACE to automatically prefix * any public symbol from xxhash library with the value of @ref XXH_NAMESPACE * (therefore, avoid empty or numeric values). * * Note that no change is required within the calling program as long as it * includes `xxhash.h`: Regular symbol names will be automatically translated * by this header. */ # define XXH_NAMESPACE /* YOUR NAME HERE */ # undef XXH_NAMESPACE #endif #if (defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API)) \ && !defined(XXH_INLINE_ALL_31684351384) /* this section should be traversed only once */ # define XXH_INLINE_ALL_31684351384 /* give access to the advanced API, required to compile implementations */ # undef XXH_STATIC_LINKING_ONLY /* avoid macro redef */ # define XXH_STATIC_LINKING_ONLY /* make all functions private */ # undef XXH_PUBLIC_API # if defined(__GNUC__) # define XXH_PUBLIC_API static __inline __attribute__((unused)) # elif defined (__cplusplus) || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) # define XXH_PUBLIC_API static inline # elif defined(_MSC_VER) # define XXH_PUBLIC_API static __inline # else /* note: this version may generate warnings for unused static functions */ # define XXH_PUBLIC_API static # endif /* * This part deals with the special case where a unit wants to inline xxHash, * but "xxhash.h" has previously been included without XXH_INLINE_ALL, * such as part of some previously included *.h header file. * Without further action, the new include would just be ignored, * and functions would effectively _not_ be inlined (silent failure). * The following macros solve this situation by prefixing all inlined names, * avoiding naming collision with previous inclusions. */ /* Before that, we unconditionally #undef all symbols, * in case they were already defined with XXH_NAMESPACE. * They will then be redefined for XXH_INLINE_ALL */ # undef XXH_versionNumber /* XXH32 */ # undef XXH32 # undef XXH32_createState # undef XXH32_freeState # undef XXH32_reset # undef XXH32_update # undef XXH32_digest # undef XXH32_copyState # undef XXH32_canonicalFromHash # undef XXH32_hashFromCanonical /* XXH64 */ # undef XXH64 # undef XXH64_createState # undef XXH64_freeState # undef XXH64_reset # undef XXH64_update # undef XXH64_digest # undef XXH64_copyState # undef XXH64_canonicalFromHash # undef XXH64_hashFromCanonical /* XXH3_64bits */ # undef XXH3_64bits # undef XXH3_64bits_withSecret # undef XXH3_64bits_withSeed # undef XXH3_64bits_withSecretandSeed # undef XXH3_createState # undef XXH3_freeState # undef XXH3_copyState # undef XXH3_64bits_reset # undef XXH3_64bits_reset_withSeed # undef XXH3_64bits_reset_withSecret # undef XXH3_64bits_update # undef XXH3_64bits_digest # undef XXH3_generateSecret /* XXH3_128bits */ # undef XXH128 # undef XXH3_128bits # undef XXH3_128bits_withSeed # undef XXH3_128bits_withSecret # undef XXH3_128bits_reset # undef XXH3_128bits_reset_withSeed # undef XXH3_128bits_reset_withSecret # undef XXH3_128bits_reset_withSecretandSeed # undef XXH3_128bits_update # undef XXH3_128bits_digest # undef XXH128_isEqual # undef XXH128_cmp # undef XXH128_canonicalFromHash # undef XXH128_hashFromCanonical /* Finally, free the namespace itself */ # undef XXH_NAMESPACE /* employ the namespace for XXH_INLINE_ALL */ # define XXH_NAMESPACE XXH_INLINE_ /* * Some identifiers (enums, type names) are not symbols, * but they must nonetheless be renamed to avoid redeclaration. * Alternative solution: do not redeclare them. * However, this requires some #ifdefs, and has a more dispersed impact. * Meanwhile, renaming can be achieved in a single place. */ # define XXH_IPREF(Id) XXH_NAMESPACE ## Id # define XXH_OK XXH_IPREF(XXH_OK) # define XXH_ERROR XXH_IPREF(XXH_ERROR) # define XXH_errorcode XXH_IPREF(XXH_errorcode) # define XXH32_canonical_t XXH_IPREF(XXH32_canonical_t) # define XXH64_canonical_t XXH_IPREF(XXH64_canonical_t) # define XXH128_canonical_t XXH_IPREF(XXH128_canonical_t) # define XXH32_state_s XXH_IPREF(XXH32_state_s) # define XXH32_state_t XXH_IPREF(XXH32_state_t) # define XXH64_state_s XXH_IPREF(XXH64_state_s) # define XXH64_state_t XXH_IPREF(XXH64_state_t) # define XXH3_state_s XXH_IPREF(XXH3_state_s) # define XXH3_state_t XXH_IPREF(XXH3_state_t) # define XXH128_hash_t XXH_IPREF(XXH128_hash_t) /* Ensure the header is parsed again, even if it was previously included */ # undef XXHASH_H_5627135585666179 # undef XXHASH_H_STATIC_13879238742 #endif /* XXH_INLINE_ALL || XXH_PRIVATE_API */ /* **************************************************************** * Stable API *****************************************************************/ #ifndef XXHASH_H_5627135585666179 #define XXHASH_H_5627135585666179 1 /*! @brief Marks a global symbol. */ #if !defined(XXH_INLINE_ALL) && !defined(XXH_PRIVATE_API) # if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT)) # ifdef XXH_EXPORT # define XXH_PUBLIC_API __declspec(dllexport) # elif XXH_IMPORT # define XXH_PUBLIC_API __declspec(dllimport) # endif # else # define XXH_PUBLIC_API /* do nothing */ # endif #endif #ifdef XXH_NAMESPACE # define XXH_CAT(A,B) A##B # define XXH_NAME2(A,B) XXH_CAT(A,B) # define XXH_versionNumber XXH_NAME2(XXH_NAMESPACE, XXH_versionNumber) /* XXH32 */ # define XXH32 XXH_NAME2(XXH_NAMESPACE, XXH32) # define XXH32_createState XXH_NAME2(XXH_NAMESPACE, XXH32_createState) # define XXH32_freeState XXH_NAME2(XXH_NAMESPACE, XXH32_freeState) # define XXH32_reset XXH_NAME2(XXH_NAMESPACE, XXH32_reset) # define XXH32_update XXH_NAME2(XXH_NAMESPACE, XXH32_update) # define XXH32_digest XXH_NAME2(XXH_NAMESPACE, XXH32_digest) # define XXH32_copyState XXH_NAME2(XXH_NAMESPACE, XXH32_copyState) # define XXH32_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH32_canonicalFromHash) # define XXH32_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH32_hashFromCanonical) /* XXH64 */ # define XXH64 XXH_NAME2(XXH_NAMESPACE, XXH64) # define XXH64_createState XXH_NAME2(XXH_NAMESPACE, XXH64_createState) # define XXH64_freeState XXH_NAME2(XXH_NAMESPACE, XXH64_freeState) # define XXH64_reset XXH_NAME2(XXH_NAMESPACE, XXH64_reset) # define XXH64_update XXH_NAME2(XXH_NAMESPACE, XXH64_update) # define XXH64_digest XXH_NAME2(XXH_NAMESPACE, XXH64_digest) # define XXH64_copyState XXH_NAME2(XXH_NAMESPACE, XXH64_copyState) # define XXH64_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH64_canonicalFromHash) # define XXH64_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH64_hashFromCanonical) /* XXH3_64bits */ # define XXH3_64bits XXH_NAME2(XXH_NAMESPACE, XXH3_64bits) # define XXH3_64bits_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSecret) # define XXH3_64bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSeed) # define XXH3_64bits_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_withSecretandSeed) # define XXH3_createState XXH_NAME2(XXH_NAMESPACE, XXH3_createState) # define XXH3_freeState XXH_NAME2(XXH_NAMESPACE, XXH3_freeState) # define XXH3_copyState XXH_NAME2(XXH_NAMESPACE, XXH3_copyState) # define XXH3_64bits_reset XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset) # define XXH3_64bits_reset_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSeed) # define XXH3_64bits_reset_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSecret) # define XXH3_64bits_reset_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_reset_withSecretandSeed) # define XXH3_64bits_update XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_update) # define XXH3_64bits_digest XXH_NAME2(XXH_NAMESPACE, XXH3_64bits_digest) # define XXH3_generateSecret XXH_NAME2(XXH_NAMESPACE, XXH3_generateSecret) # define XXH3_generateSecret_fromSeed XXH_NAME2(XXH_NAMESPACE, XXH3_generateSecret_fromSeed) /* XXH3_128bits */ # define XXH128 XXH_NAME2(XXH_NAMESPACE, XXH128) # define XXH3_128bits XXH_NAME2(XXH_NAMESPACE, XXH3_128bits) # define XXH3_128bits_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSeed) # define XXH3_128bits_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSecret) # define XXH3_128bits_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_withSecretandSeed) # define XXH3_128bits_reset XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset) # define XXH3_128bits_reset_withSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSeed) # define XXH3_128bits_reset_withSecret XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSecret) # define XXH3_128bits_reset_withSecretandSeed XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_reset_withSecretandSeed) # define XXH3_128bits_update XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_update) # define XXH3_128bits_digest XXH_NAME2(XXH_NAMESPACE, XXH3_128bits_digest) # define XXH128_isEqual XXH_NAME2(XXH_NAMESPACE, XXH128_isEqual) # define XXH128_cmp XXH_NAME2(XXH_NAMESPACE, XXH128_cmp) # define XXH128_canonicalFromHash XXH_NAME2(XXH_NAMESPACE, XXH128_canonicalFromHash) # define XXH128_hashFromCanonical XXH_NAME2(XXH_NAMESPACE, XXH128_hashFromCanonical) #endif /* ************************************* * Compiler specifics ***************************************/ /* specific declaration modes for Windows */ #if !defined(XXH_INLINE_ALL) && !defined(XXH_PRIVATE_API) # if defined(WIN32) && defined(_MSC_VER) && (defined(XXH_IMPORT) || defined(XXH_EXPORT)) # ifdef XXH_EXPORT # define XXH_PUBLIC_API __declspec(dllexport) # elif XXH_IMPORT # define XXH_PUBLIC_API __declspec(dllimport) # endif # else # define XXH_PUBLIC_API /* do nothing */ # endif #endif #if defined (__GNUC__) # define XXH_CONSTF __attribute__((const)) # define XXH_PUREF __attribute__((pure)) # define XXH_MALLOCF __attribute__((malloc)) #else # define XXH_CONSTF /* disable */ # define XXH_PUREF # define XXH_MALLOCF #endif /* ************************************* * Version ***************************************/ #define XXH_VERSION_MAJOR 0 #define XXH_VERSION_MINOR 8 #define XXH_VERSION_RELEASE 2 /*! @brief Version number, encoded as two digits each */ #define XXH_VERSION_NUMBER (XXH_VERSION_MAJOR *100*100 + XXH_VERSION_MINOR *100 + XXH_VERSION_RELEASE) /*! * @brief Obtains the xxHash version. * * This is mostly useful when xxHash is compiled as a shared library, * since the returned value comes from the library, as opposed to header file. * * @return @ref XXH_VERSION_NUMBER of the invoked library. */ XXH_PUBLIC_API XXH_CONSTF unsigned XXH_versionNumber (void); /* **************************** * Common basic types ******************************/ #include /* size_t */ /*! * @brief Exit code for the streaming API. */ typedef enum { XXH_OK = 0, /*!< OK */ XXH_ERROR /*!< Error */ } XXH_errorcode; /*-********************************************************************** * 32-bit hash ************************************************************************/ #if defined(XXH_DOXYGEN) /* Don't show include */ /*! * @brief An unsigned 32-bit integer. * * Not necessarily defined to `uint32_t` but functionally equivalent. */ typedef uint32_t XXH32_hash_t; #elif !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint32_t XXH32_hash_t; #else # include # if UINT_MAX == 0xFFFFFFFFUL typedef unsigned int XXH32_hash_t; # elif ULONG_MAX == 0xFFFFFFFFUL typedef unsigned long XXH32_hash_t; # else # error "unsupported platform: need a 32-bit type" # endif #endif /*! * @} * * @defgroup XXH32_family XXH32 family * @ingroup public * Contains functions used in the classic 32-bit xxHash algorithm. * * @note * XXH32 is useful for older platforms, with no or poor 64-bit performance. * Note that the @ref XXH3_family provides competitive speed for both 32-bit * and 64-bit systems, and offers true 64/128 bit hash results. * * @see @ref XXH64_family, @ref XXH3_family : Other xxHash families * @see @ref XXH32_impl for implementation details * @{ */ /*! * @brief Calculates the 32-bit hash of @p input using xxHash32. * * Speed on Core 2 Duo @ 3 GHz (single thread, SMHasher benchmark): 5.4 GB/s * * See @ref single_shot_example "Single Shot Example" for an example. * * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * @param seed The 32-bit seed to alter the hash's output predictably. * * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return The calculated 32-bit hash value. * * @see * XXH64(), XXH3_64bits_withSeed(), XXH3_128bits_withSeed(), XXH128(): * Direct equivalents for the other variants of xxHash. * @see * XXH32_createState(), XXH32_update(), XXH32_digest(): Streaming version. */ XXH_PUBLIC_API XXH_PUREF XXH32_hash_t XXH32 (const void* input, size_t length, XXH32_hash_t seed); #ifndef XXH_NO_STREAM /*! * Streaming functions generate the xxHash value from an incremental input. * This method is slower than single-call functions, due to state management. * For small inputs, prefer `XXH32()` and `XXH64()`, which are better optimized. * * An XXH state must first be allocated using `XXH*_createState()`. * * Start a new hash by initializing the state with a seed using `XXH*_reset()`. * * Then, feed the hash state by calling `XXH*_update()` as many times as necessary. * * The function returns an error code, with 0 meaning OK, and any other value * meaning there is an error. * * Finally, a hash value can be produced anytime, by using `XXH*_digest()`. * This function returns the nn-bits hash as an int or long long. * * It's still possible to continue inserting input into the hash state after a * digest, and generate new hash values later on by invoking `XXH*_digest()`. * * When done, release the state using `XXH*_freeState()`. * * @see streaming_example at the top of @ref xxhash.h for an example. */ /*! * @typedef struct XXH32_state_s XXH32_state_t * @brief The opaque state struct for the XXH32 streaming API. * * @see XXH32_state_s for details. */ typedef struct XXH32_state_s XXH32_state_t; /*! * @brief Allocates an @ref XXH32_state_t. * * Must be freed with XXH32_freeState(). * @return An allocated XXH32_state_t on success, `NULL` on failure. */ XXH_PUBLIC_API XXH_MALLOCF XXH32_state_t* XXH32_createState(void); /*! * @brief Frees an @ref XXH32_state_t. * * Must be allocated with XXH32_createState(). * @param statePtr A pointer to an @ref XXH32_state_t allocated with @ref XXH32_createState(). * @return XXH_OK. */ XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr); /*! * @brief Copies one @ref XXH32_state_t to another. * * @param dst_state The state to copy to. * @param src_state The state to copy from. * @pre * @p dst_state and @p src_state must not be `NULL` and must not overlap. */ XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dst_state, const XXH32_state_t* src_state); /*! * @brief Resets an @ref XXH32_state_t to begin a new hash. * * This function resets and seeds a state. Call it before @ref XXH32_update(). * * @param statePtr The state struct to reset. * @param seed The 32-bit seed to alter the hash result predictably. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH32_reset (XXH32_state_t* statePtr, XXH32_hash_t seed); /*! * @brief Consumes a block of @p input to an @ref XXH32_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* statePtr, const void* input, size_t length); /*! * @brief Returns the calculated hash value from an @ref XXH32_state_t. * * @note * Calling XXH32_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated xxHash32 value from that state. */ XXH_PUBLIC_API XXH_PUREF XXH32_hash_t XXH32_digest (const XXH32_state_t* statePtr); #endif /* !XXH_NO_STREAM */ /******* Canonical representation *******/ /* * The default return values from XXH functions are unsigned 32 and 64 bit * integers. * This the simplest and fastest format for further post-processing. * * However, this leaves open the question of what is the order on the byte level, * since little and big endian conventions will store the same number differently. * * The canonical representation settles this issue by mandating big-endian * convention, the same convention as human-readable numbers (large digits first). * * When writing hash values to storage, sending them over a network, or printing * them, it's highly recommended to use the canonical representation to ensure * portability across a wider range of systems, present and future. * * The following functions allow transformation of hash values to and from * canonical format. */ /*! * @brief Canonical (big endian) representation of @ref XXH32_hash_t. */ typedef struct { unsigned char digest[4]; /*!< Hash bytes, big endian */ } XXH32_canonical_t; /*! * @brief Converts an @ref XXH32_hash_t to a big endian @ref XXH32_canonical_t. * * @param dst The @ref XXH32_canonical_t pointer to be stored to. * @param hash The @ref XXH32_hash_t to be converted. * * @pre * @p dst must not be `NULL`. */ XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash); /*! * @brief Converts an @ref XXH32_canonical_t to a native @ref XXH32_hash_t. * * @param src The @ref XXH32_canonical_t to convert. * * @pre * @p src must not be `NULL`. * * @return The converted hash. */ XXH_PUBLIC_API XXH_PUREF XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src); /*! @cond Doxygen ignores this part */ #ifdef __has_attribute # define XXH_HAS_ATTRIBUTE(x) __has_attribute(x) #else # define XXH_HAS_ATTRIBUTE(x) 0 #endif /*! @endcond */ /*! @cond Doxygen ignores this part */ /* * C23 __STDC_VERSION__ number hasn't been specified yet. For now * leave as `201711L` (C17 + 1). * TODO: Update to correct value when its been specified. */ #define XXH_C23_VN 201711L /*! @endcond */ /*! @cond Doxygen ignores this part */ /* C-language Attributes are added in C23. */ #if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= XXH_C23_VN) && defined(__has_c_attribute) # define XXH_HAS_C_ATTRIBUTE(x) __has_c_attribute(x) #else # define XXH_HAS_C_ATTRIBUTE(x) 0 #endif /*! @endcond */ /*! @cond Doxygen ignores this part */ #if defined(__cplusplus) && defined(__has_cpp_attribute) # define XXH_HAS_CPP_ATTRIBUTE(x) __has_cpp_attribute(x) #else # define XXH_HAS_CPP_ATTRIBUTE(x) 0 #endif /*! @endcond */ /*! @cond Doxygen ignores this part */ /* * Define XXH_FALLTHROUGH macro for annotating switch case with the 'fallthrough' attribute * introduced in CPP17 and C23. * CPP17 : https://en.cppreference.com/w/cpp/language/attributes/fallthrough * C23 : https://en.cppreference.com/w/c/language/attributes/fallthrough */ #if XXH_HAS_C_ATTRIBUTE(fallthrough) || XXH_HAS_CPP_ATTRIBUTE(fallthrough) # define XXH_FALLTHROUGH [[fallthrough]] #elif XXH_HAS_ATTRIBUTE(__fallthrough__) # define XXH_FALLTHROUGH __attribute__ ((__fallthrough__)) #else # define XXH_FALLTHROUGH /* fallthrough */ #endif /*! @endcond */ /*! @cond Doxygen ignores this part */ /* * Define XXH_NOESCAPE for annotated pointers in public API. * https://clang.llvm.org/docs/AttributeReference.html#noescape * As of writing this, only supported by clang. */ #if XXH_HAS_ATTRIBUTE(noescape) # define XXH_NOESCAPE __attribute__((noescape)) #else # define XXH_NOESCAPE #endif /*! @endcond */ /*! * @} * @ingroup public * @{ */ #ifndef XXH_NO_LONG_LONG /*-********************************************************************** * 64-bit hash ************************************************************************/ #if defined(XXH_DOXYGEN) /* don't include */ /*! * @brief An unsigned 64-bit integer. * * Not necessarily defined to `uint64_t` but functionally equivalent. */ typedef uint64_t XXH64_hash_t; #elif !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint64_t XXH64_hash_t; #else # include # if defined(__LP64__) && ULONG_MAX == 0xFFFFFFFFFFFFFFFFULL /* LP64 ABI says uint64_t is unsigned long */ typedef unsigned long XXH64_hash_t; # else /* the following type must have a width of 64-bit */ typedef unsigned long long XXH64_hash_t; # endif #endif /*! * @} * * @defgroup XXH64_family XXH64 family * @ingroup public * @{ * Contains functions used in the classic 64-bit xxHash algorithm. * * @note * XXH3 provides competitive speed for both 32-bit and 64-bit systems, * and offers true 64/128 bit hash results. * It provides better speed for systems with vector processing capabilities. */ /*! * @brief Calculates the 64-bit hash of @p input using xxHash64. * * This function usually runs faster on 64-bit systems, but slower on 32-bit * systems (see benchmark). * * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * @param seed The 64-bit seed to alter the hash's output predictably. * * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return The calculated 64-bit hash. * * @see * XXH32(), XXH3_64bits_withSeed(), XXH3_128bits_withSeed(), XXH128(): * Direct equivalents for the other variants of xxHash. * @see * XXH64_createState(), XXH64_update(), XXH64_digest(): Streaming version. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH64(XXH_NOESCAPE const void* input, size_t length, XXH64_hash_t seed); /******* Streaming *******/ #ifndef XXH_NO_STREAM /*! * @brief The opaque state struct for the XXH64 streaming API. * * @see XXH64_state_s for details. */ typedef struct XXH64_state_s XXH64_state_t; /* incomplete type */ /*! * @brief Allocates an @ref XXH64_state_t. * * Must be freed with XXH64_freeState(). * @return An allocated XXH64_state_t on success, `NULL` on failure. */ XXH_PUBLIC_API XXH_MALLOCF XXH64_state_t* XXH64_createState(void); /*! * @brief Frees an @ref XXH64_state_t. * * Must be allocated with XXH64_createState(). * @param statePtr A pointer to an @ref XXH64_state_t allocated with @ref XXH64_createState(). * @return XXH_OK. */ XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr); /*! * @brief Copies one @ref XXH64_state_t to another. * * @param dst_state The state to copy to. * @param src_state The state to copy from. * @pre * @p dst_state and @p src_state must not be `NULL` and must not overlap. */ XXH_PUBLIC_API void XXH64_copyState(XXH_NOESCAPE XXH64_state_t* dst_state, const XXH64_state_t* src_state); /*! * @brief Resets an @ref XXH64_state_t to begin a new hash. * * This function resets and seeds a state. Call it before @ref XXH64_update(). * * @param statePtr The state struct to reset. * @param seed The 64-bit seed to alter the hash result predictably. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH64_reset (XXH_NOESCAPE XXH64_state_t* statePtr, XXH64_hash_t seed); /*! * @brief Consumes a block of @p input to an @ref XXH64_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH_NOESCAPE XXH64_state_t* statePtr, XXH_NOESCAPE const void* input, size_t length); /*! * @brief Returns the calculated hash value from an @ref XXH64_state_t. * * @note * Calling XXH64_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated xxHash64 value from that state. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH64_digest (XXH_NOESCAPE const XXH64_state_t* statePtr); #endif /* !XXH_NO_STREAM */ /******* Canonical representation *******/ /*! * @brief Canonical (big endian) representation of @ref XXH64_hash_t. */ typedef struct { unsigned char digest[sizeof(XXH64_hash_t)]; } XXH64_canonical_t; /*! * @brief Converts an @ref XXH64_hash_t to a big endian @ref XXH64_canonical_t. * * @param dst The @ref XXH64_canonical_t pointer to be stored to. * @param hash The @ref XXH64_hash_t to be converted. * * @pre * @p dst must not be `NULL`. */ XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH_NOESCAPE XXH64_canonical_t* dst, XXH64_hash_t hash); /*! * @brief Converts an @ref XXH64_canonical_t to a native @ref XXH64_hash_t. * * @param src The @ref XXH64_canonical_t to convert. * * @pre * @p src must not be `NULL`. * * @return The converted hash. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH64_hashFromCanonical(XXH_NOESCAPE const XXH64_canonical_t* src); #ifndef XXH_NO_XXH3 /*! * @} * ************************************************************************ * @defgroup XXH3_family XXH3 family * @ingroup public * @{ * * XXH3 is a more recent hash algorithm featuring: * - Improved speed for both small and large inputs * - True 64-bit and 128-bit outputs * - SIMD acceleration * - Improved 32-bit viability * * Speed analysis methodology is explained here: * * https://fastcompression.blogspot.com/2019/03/presenting-xxh3.html * * Compared to XXH64, expect XXH3 to run approximately * ~2x faster on large inputs and >3x faster on small ones, * exact differences vary depending on platform. * * XXH3's speed benefits greatly from SIMD and 64-bit arithmetic, * but does not require it. * Most 32-bit and 64-bit targets that can run XXH32 smoothly can run XXH3 * at competitive speeds, even without vector support. Further details are * explained in the implementation. * * XXH3 has a fast scalar implementation, but it also includes accelerated SIMD * implementations for many common platforms: * - AVX512 * - AVX2 * - SSE2 * - ARM NEON * - WebAssembly SIMD128 * - POWER8 VSX * - s390x ZVector * This can be controlled via the @ref XXH_VECTOR macro, but it automatically * selects the best version according to predefined macros. For the x86 family, an * automatic runtime dispatcher is included separately in @ref xxh_x86dispatch.c. * * XXH3 implementation is portable: * it has a generic C90 formulation that can be compiled on any platform, * all implementations generate exactly the same hash value on all platforms. * Starting from v0.8.0, it's also labelled "stable", meaning that * any future version will also generate the same hash value. * * XXH3 offers 2 variants, _64bits and _128bits. * * When only 64 bits are needed, prefer invoking the _64bits variant, as it * reduces the amount of mixing, resulting in faster speed on small inputs. * It's also generally simpler to manipulate a scalar return type than a struct. * * The API supports one-shot hashing, streaming mode, and custom secrets. */ /*-********************************************************************** * XXH3 64-bit variant ************************************************************************/ /*! * @brief 64-bit unseeded variant of XXH3. * * This is equivalent to @ref XXH3_64bits_withSeed() with a seed of 0, however * it may have slightly better performance due to constant propagation of the * defaults. * * @see * XXH32(), XXH64(), XXH3_128bits(): equivalent for the other xxHash algorithms * @see * XXH3_64bits_withSeed(), XXH3_64bits_withSecret(): other seeding variants * @see * XXH3_64bits_reset(), XXH3_64bits_update(), XXH3_64bits_digest(): Streaming version. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits(XXH_NOESCAPE const void* input, size_t length); /*! * @brief 64-bit seeded variant of XXH3 * * This variant generates a custom secret on the fly based on default secret * altered using the `seed` value. * * While this operation is decently fast, note that it's not completely free. * * @note * seed == 0 produces the same results as @ref XXH3_64bits(). * * @param input The data to hash * @param length The length * @param seed The 64-bit seed to alter the state. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits_withSeed(XXH_NOESCAPE const void* input, size_t length, XXH64_hash_t seed); /*! * The bare minimum size for a custom secret. * * @see * XXH3_64bits_withSecret(), XXH3_64bits_reset_withSecret(), * XXH3_128bits_withSecret(), XXH3_128bits_reset_withSecret(). */ #define XXH3_SECRET_SIZE_MIN 136 /*! * @brief 64-bit variant of XXH3 with a custom "secret". * * It's possible to provide any blob of bytes as a "secret" to generate the hash. * This makes it more difficult for an external actor to prepare an intentional collision. * The main condition is that secretSize *must* be large enough (>= XXH3_SECRET_SIZE_MIN). * However, the quality of the secret impacts the dispersion of the hash algorithm. * Therefore, the secret _must_ look like a bunch of random bytes. * Avoid "trivial" or structured data such as repeated sequences or a text document. * Whenever in doubt about the "randomness" of the blob of bytes, * consider employing "XXH3_generateSecret()" instead (see below). * It will generate a proper high entropy secret derived from the blob of bytes. * Another advantage of using XXH3_generateSecret() is that * it guarantees that all bits within the initial blob of bytes * will impact every bit of the output. * This is not necessarily the case when using the blob of bytes directly * because, when hashing _small_ inputs, only a portion of the secret is employed. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits_withSecret(XXH_NOESCAPE const void* data, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize); /******* Streaming *******/ #ifndef XXH_NO_STREAM /* * Streaming requires state maintenance. * This operation costs memory and CPU. * As a consequence, streaming is slower than one-shot hashing. * For better performance, prefer one-shot functions whenever applicable. */ /*! * @brief The state struct for the XXH3 streaming API. * * @see XXH3_state_s for details. */ typedef struct XXH3_state_s XXH3_state_t; XXH_PUBLIC_API XXH_MALLOCF XXH3_state_t* XXH3_createState(void); XXH_PUBLIC_API XXH_errorcode XXH3_freeState(XXH3_state_t* statePtr); /*! * @brief Copies one @ref XXH3_state_t to another. * * @param dst_state The state to copy to. * @param src_state The state to copy from. * @pre * @p dst_state and @p src_state must not be `NULL` and must not overlap. */ XXH_PUBLIC_API void XXH3_copyState(XXH_NOESCAPE XXH3_state_t* dst_state, XXH_NOESCAPE const XXH3_state_t* src_state); /*! * @brief Resets an @ref XXH3_state_t to begin a new hash. * * This function resets `statePtr` and generate a secret with default parameters. Call it before @ref XXH3_64bits_update(). * Digest will be equivalent to `XXH3_64bits()`. * * @param statePtr The state struct to reset. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. * */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset(XXH_NOESCAPE XXH3_state_t* statePtr); /*! * @brief Resets an @ref XXH3_state_t with 64-bit seed to begin a new hash. * * This function resets `statePtr` and generate a secret from `seed`. Call it before @ref XXH3_64bits_update(). * Digest will be equivalent to `XXH3_64bits_withSeed()`. * * @param statePtr The state struct to reset. * @param seed The 64-bit seed to alter the state. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. * */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH64_hash_t seed); /*! * XXH3_64bits_reset_withSecret(): * `secret` is referenced, it _must outlive_ the hash streaming session. * Similar to one-shot API, `secretSize` must be >= `XXH3_SECRET_SIZE_MIN`, * and the quality of produced hash values depends on secret's entropy * (secret's content should look like a bunch of random bytes). * When in doubt about the randomness of a candidate `secret`, * consider employing `XXH3_generateSecret()` instead (see below). */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecret(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize); /*! * @brief Consumes a block of @p input to an @ref XXH3_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_update (XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* input, size_t length); /*! * @brief Returns the calculated XXH3 64-bit hash value from an @ref XXH3_state_t. * * @note * Calling XXH3_64bits_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated XXH3 64-bit hash value from that state. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits_digest (XXH_NOESCAPE const XXH3_state_t* statePtr); #endif /* !XXH_NO_STREAM */ /* note : canonical representation of XXH3 is the same as XXH64 * since they both produce XXH64_hash_t values */ /*-********************************************************************** * XXH3 128-bit variant ************************************************************************/ /*! * @brief The return value from 128-bit hashes. * * Stored in little endian order, although the fields themselves are in native * endianness. */ typedef struct { XXH64_hash_t low64; /*!< `value & 0xFFFFFFFFFFFFFFFF` */ XXH64_hash_t high64; /*!< `value >> 64` */ } XXH128_hash_t; /*! * @brief Unseeded 128-bit variant of XXH3 * * The 128-bit variant of XXH3 has more strength, but it has a bit of overhead * for shorter inputs. * * This is equivalent to @ref XXH3_128bits_withSeed() with a seed of 0, however * it may have slightly better performance due to constant propagation of the * defaults. * * @see * XXH32(), XXH64(), XXH3_64bits(): equivalent for the other xxHash algorithms * @see * XXH3_128bits_withSeed(), XXH3_128bits_withSecret(): other seeding variants * @see * XXH3_128bits_reset(), XXH3_128bits_update(), XXH3_128bits_digest(): Streaming version. */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits(XXH_NOESCAPE const void* data, size_t len); /*! @brief Seeded 128-bit variant of XXH3. @see XXH3_64bits_withSeed(). */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits_withSeed(XXH_NOESCAPE const void* data, size_t len, XXH64_hash_t seed); /*! @brief Custom secret 128-bit variant of XXH3. @see XXH3_64bits_withSecret(). */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits_withSecret(XXH_NOESCAPE const void* data, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize); /******* Streaming *******/ #ifndef XXH_NO_STREAM /* * Streaming requires state maintenance. * This operation costs memory and CPU. * As a consequence, streaming is slower than one-shot hashing. * For better performance, prefer one-shot functions whenever applicable. * * XXH3_128bits uses the same XXH3_state_t as XXH3_64bits(). * Use already declared XXH3_createState() and XXH3_freeState(). * * All reset and streaming functions have same meaning as their 64-bit counterpart. */ /*! * @brief Resets an @ref XXH3_state_t to begin a new hash. * * This function resets `statePtr` and generate a secret with default parameters. Call it before @ref XXH3_128bits_update(). * Digest will be equivalent to `XXH3_128bits()`. * * @param statePtr The state struct to reset. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. * */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset(XXH_NOESCAPE XXH3_state_t* statePtr); /*! * @brief Resets an @ref XXH3_state_t with 64-bit seed to begin a new hash. * * This function resets `statePtr` and generate a secret from `seed`. Call it before @ref XXH3_128bits_update(). * Digest will be equivalent to `XXH3_128bits_withSeed()`. * * @param statePtr The state struct to reset. * @param seed The 64-bit seed to alter the state. * * @pre * @p statePtr must not be `NULL`. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. * */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH64_hash_t seed); /*! @brief Custom secret 128-bit variant of XXH3. @see XXH_64bits_reset_withSecret(). */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecret(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize); /*! * @brief Consumes a block of @p input to an @ref XXH3_state_t. * * Call this to incrementally consume blocks of data. * * @param statePtr The state struct to update. * @param input The block of data to be hashed, at least @p length bytes in size. * @param length The length of @p input, in bytes. * * @pre * @p statePtr must not be `NULL`. * @pre * The memory between @p input and @p input + @p length must be valid, * readable, contiguous memory. However, if @p length is `0`, @p input may be * `NULL`. In C++, this also must be *TriviallyCopyable*. * * @return @ref XXH_OK on success, @ref XXH_ERROR on failure. */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_update (XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* input, size_t length); /*! * @brief Returns the calculated XXH3 128-bit hash value from an @ref XXH3_state_t. * * @note * Calling XXH3_128bits_digest() will not affect @p statePtr, so you can update, * digest, and update again. * * @param statePtr The state struct to calculate the hash from. * * @pre * @p statePtr must not be `NULL`. * * @return The calculated XXH3 128-bit hash value from that state. */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits_digest (XXH_NOESCAPE const XXH3_state_t* statePtr); #endif /* !XXH_NO_STREAM */ /* Following helper functions make it possible to compare XXH128_hast_t values. * Since XXH128_hash_t is a structure, this capability is not offered by the language. * Note: For better performance, these functions can be inlined using XXH_INLINE_ALL */ /*! * XXH128_isEqual(): * Return: 1 if `h1` and `h2` are equal, 0 if they are not. */ XXH_PUBLIC_API XXH_PUREF int XXH128_isEqual(XXH128_hash_t h1, XXH128_hash_t h2); /*! * @brief Compares two @ref XXH128_hash_t * This comparator is compatible with stdlib's `qsort()`/`bsearch()`. * * @return: >0 if *h128_1 > *h128_2 * =0 if *h128_1 == *h128_2 * <0 if *h128_1 < *h128_2 */ XXH_PUBLIC_API XXH_PUREF int XXH128_cmp(XXH_NOESCAPE const void* h128_1, XXH_NOESCAPE const void* h128_2); /******* Canonical representation *******/ typedef struct { unsigned char digest[sizeof(XXH128_hash_t)]; } XXH128_canonical_t; /*! * @brief Converts an @ref XXH128_hash_t to a big endian @ref XXH128_canonical_t. * * @param dst The @ref XXH128_canonical_t pointer to be stored to. * @param hash The @ref XXH128_hash_t to be converted. * * @pre * @p dst must not be `NULL`. */ XXH_PUBLIC_API void XXH128_canonicalFromHash(XXH_NOESCAPE XXH128_canonical_t* dst, XXH128_hash_t hash); /*! * @brief Converts an @ref XXH128_canonical_t to a native @ref XXH128_hash_t. * * @param src The @ref XXH128_canonical_t to convert. * * @pre * @p src must not be `NULL`. * * @return The converted hash. */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH128_hashFromCanonical(XXH_NOESCAPE const XXH128_canonical_t* src); #endif /* !XXH_NO_XXH3 */ #endif /* XXH_NO_LONG_LONG */ /*! * @} */ #endif /* XXHASH_H_5627135585666179 */ #if defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742) #define XXHASH_H_STATIC_13879238742 /* **************************************************************************** * This section contains declarations which are not guaranteed to remain stable. * They may change in future versions, becoming incompatible with a different * version of the library. * These declarations should only be used with static linking. * Never use them in association with dynamic linking! ***************************************************************************** */ /* * These definitions are only present to allow static allocation * of XXH states, on stack or in a struct, for example. * Never **ever** access their members directly. */ /*! * @internal * @brief Structure for XXH32 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is * an opaque type. This allows fields to safely be changed. * * Typedef'd to @ref XXH32_state_t. * Do not access the members of this struct directly. * @see XXH64_state_s, XXH3_state_s */ struct XXH32_state_s { XXH32_hash_t total_len_32; /*!< Total length hashed, modulo 2^32 */ XXH32_hash_t large_len; /*!< Whether the hash is >= 16 (handles @ref total_len_32 overflow) */ XXH32_hash_t v[4]; /*!< Accumulator lanes */ XXH32_hash_t mem32[4]; /*!< Internal buffer for partial reads. Treated as unsigned char[16]. */ XXH32_hash_t memsize; /*!< Amount of data in @ref mem32 */ XXH32_hash_t reserved; /*!< Reserved field. Do not read nor write to it. */ }; /* typedef'd to XXH32_state_t */ #ifndef XXH_NO_LONG_LONG /* defined when there is no 64-bit support */ /*! * @internal * @brief Structure for XXH64 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. Otherwise it is * an opaque type. This allows fields to safely be changed. * * Typedef'd to @ref XXH64_state_t. * Do not access the members of this struct directly. * @see XXH32_state_s, XXH3_state_s */ struct XXH64_state_s { XXH64_hash_t total_len; /*!< Total length hashed. This is always 64-bit. */ XXH64_hash_t v[4]; /*!< Accumulator lanes */ XXH64_hash_t mem64[4]; /*!< Internal buffer for partial reads. Treated as unsigned char[32]. */ XXH32_hash_t memsize; /*!< Amount of data in @ref mem64 */ XXH32_hash_t reserved32; /*!< Reserved field, needed for padding anyways*/ XXH64_hash_t reserved64; /*!< Reserved field. Do not read or write to it. */ }; /* typedef'd to XXH64_state_t */ #ifndef XXH_NO_XXH3 #if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* >= C11 */ # include # define XXH_ALIGN(n) alignas(n) #elif defined(__cplusplus) && (__cplusplus >= 201103L) /* >= C++11 */ /* In C++ alignas() is a keyword */ # define XXH_ALIGN(n) alignas(n) #elif defined(__GNUC__) # define XXH_ALIGN(n) __attribute__ ((aligned(n))) #elif defined(_MSC_VER) # define XXH_ALIGN(n) __declspec(align(n)) #else # define XXH_ALIGN(n) /* disabled */ #endif /* Old GCC versions only accept the attribute after the type in structures. */ #if !(defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L)) /* C11+ */ \ && ! (defined(__cplusplus) && (__cplusplus >= 201103L)) /* >= C++11 */ \ && defined(__GNUC__) # define XXH_ALIGN_MEMBER(align, type) type XXH_ALIGN(align) #else # define XXH_ALIGN_MEMBER(align, type) XXH_ALIGN(align) type #endif /*! * @brief The size of the internal XXH3 buffer. * * This is the optimal update size for incremental hashing. * * @see XXH3_64b_update(), XXH3_128b_update(). */ #define XXH3_INTERNALBUFFER_SIZE 256 /*! * @internal * @brief Default size of the secret buffer (and @ref XXH3_kSecret). * * This is the size used in @ref XXH3_kSecret and the seeded functions. * * Not to be confused with @ref XXH3_SECRET_SIZE_MIN. */ #define XXH3_SECRET_DEFAULT_SIZE 192 /*! * @internal * @brief Structure for XXH3 streaming API. * * @note This is only defined when @ref XXH_STATIC_LINKING_ONLY, * @ref XXH_INLINE_ALL, or @ref XXH_IMPLEMENTATION is defined. * Otherwise it is an opaque type. * Never use this definition in combination with dynamic library. * This allows fields to safely be changed in the future. * * @note ** This structure has a strict alignment requirement of 64 bytes!! ** * Do not allocate this with `malloc()` or `new`, * it will not be sufficiently aligned. * Use @ref XXH3_createState() and @ref XXH3_freeState(), or stack allocation. * * Typedef'd to @ref XXH3_state_t. * Do never access the members of this struct directly. * * @see XXH3_INITSTATE() for stack initialization. * @see XXH3_createState(), XXH3_freeState(). * @see XXH32_state_s, XXH64_state_s */ struct XXH3_state_s { XXH_ALIGN_MEMBER(64, XXH64_hash_t acc[8]); /*!< The 8 accumulators. See @ref XXH32_state_s::v and @ref XXH64_state_s::v */ XXH_ALIGN_MEMBER(64, unsigned char customSecret[XXH3_SECRET_DEFAULT_SIZE]); /*!< Used to store a custom secret generated from a seed. */ XXH_ALIGN_MEMBER(64, unsigned char buffer[XXH3_INTERNALBUFFER_SIZE]); /*!< The internal buffer. @see XXH32_state_s::mem32 */ XXH32_hash_t bufferedSize; /*!< The amount of memory in @ref buffer, @see XXH32_state_s::memsize */ XXH32_hash_t useSeed; /*!< Reserved field. Needed for padding on 64-bit. */ size_t nbStripesSoFar; /*!< Number or stripes processed. */ XXH64_hash_t totalLen; /*!< Total length hashed. 64-bit even on 32-bit targets. */ size_t nbStripesPerBlock; /*!< Number of stripes per block. */ size_t secretLimit; /*!< Size of @ref customSecret or @ref extSecret */ XXH64_hash_t seed; /*!< Seed for _withSeed variants. Must be zero otherwise, @see XXH3_INITSTATE() */ XXH64_hash_t reserved64; /*!< Reserved field. */ const unsigned char* extSecret; /*!< Reference to an external secret for the _withSecret variants, NULL * for other variants. */ /* note: there may be some padding at the end due to alignment on 64 bytes */ }; /* typedef'd to XXH3_state_t */ #undef XXH_ALIGN_MEMBER /*! * @brief Initializes a stack-allocated `XXH3_state_s`. * * When the @ref XXH3_state_t structure is merely emplaced on stack, * it should be initialized with XXH3_INITSTATE() or a memset() * in case its first reset uses XXH3_NNbits_reset_withSeed(). * This init can be omitted if the first reset uses default or _withSecret mode. * This operation isn't necessary when the state is created with XXH3_createState(). * Note that this doesn't prepare the state for a streaming operation, * it's still necessary to use XXH3_NNbits_reset*() afterwards. */ #define XXH3_INITSTATE(XXH3_state_ptr) \ do { \ XXH3_state_t* tmp_xxh3_state_ptr = (XXH3_state_ptr); \ tmp_xxh3_state_ptr->seed = 0; \ tmp_xxh3_state_ptr->extSecret = NULL; \ } while(0) /*! * simple alias to pre-selected XXH3_128bits variant */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH128(XXH_NOESCAPE const void* data, size_t len, XXH64_hash_t seed); /* === Experimental API === */ /* Symbols defined below must be considered tied to a specific library version. */ /*! * XXH3_generateSecret(): * * Derive a high-entropy secret from any user-defined content, named customSeed. * The generated secret can be used in combination with `*_withSecret()` functions. * The `_withSecret()` variants are useful to provide a higher level of protection * than 64-bit seed, as it becomes much more difficult for an external actor to * guess how to impact the calculation logic. * * The function accepts as input a custom seed of any length and any content, * and derives from it a high-entropy secret of length @p secretSize into an * already allocated buffer @p secretBuffer. * * The generated secret can then be used with any `*_withSecret()` variant. * The functions @ref XXH3_128bits_withSecret(), @ref XXH3_64bits_withSecret(), * @ref XXH3_128bits_reset_withSecret() and @ref XXH3_64bits_reset_withSecret() * are part of this list. They all accept a `secret` parameter * which must be large enough for implementation reasons (>= @ref XXH3_SECRET_SIZE_MIN) * _and_ feature very high entropy (consist of random-looking bytes). * These conditions can be a high bar to meet, so @ref XXH3_generateSecret() can * be employed to ensure proper quality. * * @p customSeed can be anything. It can have any size, even small ones, * and its content can be anything, even "poor entropy" sources such as a bunch * of zeroes. The resulting `secret` will nonetheless provide all required qualities. * * @pre * - @p secretSize must be >= @ref XXH3_SECRET_SIZE_MIN * - When @p customSeedSize > 0, supplying NULL as customSeed is undefined behavior. * * Example code: * @code{.c} * #include * #include * #include * #define XXH_STATIC_LINKING_ONLY // expose unstable API * #include "xxhash.h" * // Hashes argv[2] using the entropy from argv[1]. * int main(int argc, char* argv[]) * { * char secret[XXH3_SECRET_SIZE_MIN]; * if (argv != 3) { return 1; } * XXH3_generateSecret(secret, sizeof(secret), argv[1], strlen(argv[1])); * XXH64_hash_t h = XXH3_64bits_withSecret( * argv[2], strlen(argv[2]), * secret, sizeof(secret) * ); * printf("%016llx\n", (unsigned long long) h); * } * @endcode */ XXH_PUBLIC_API XXH_errorcode XXH3_generateSecret(XXH_NOESCAPE void* secretBuffer, size_t secretSize, XXH_NOESCAPE const void* customSeed, size_t customSeedSize); /*! * @brief Generate the same secret as the _withSeed() variants. * * The generated secret can be used in combination with *`*_withSecret()` and `_withSecretandSeed()` variants. * * Example C++ `std::string` hash class: * @code{.cpp} * #include * #define XXH_STATIC_LINKING_ONLY // expose unstable API * #include "xxhash.h" * // Slow, seeds each time * class HashSlow { * XXH64_hash_t seed; * public: * HashSlow(XXH64_hash_t s) : seed{s} {} * size_t operator()(const std::string& x) const { * return size_t{XXH3_64bits_withSeed(x.c_str(), x.length(), seed)}; * } * }; * // Fast, caches the seeded secret for future uses. * class HashFast { * unsigned char secret[XXH3_SECRET_SIZE_MIN]; * public: * HashFast(XXH64_hash_t s) { * XXH3_generateSecret_fromSeed(secret, seed); * } * size_t operator()(const std::string& x) const { * return size_t{ * XXH3_64bits_withSecret(x.c_str(), x.length(), secret, sizeof(secret)) * }; * } * }; * @endcode * @param secretBuffer A writable buffer of @ref XXH3_SECRET_SIZE_MIN bytes * @param seed The seed to seed the state. */ XXH_PUBLIC_API void XXH3_generateSecret_fromSeed(XXH_NOESCAPE void* secretBuffer, XXH64_hash_t seed); /*! * These variants generate hash values using either * @p seed for "short" keys (< XXH3_MIDSIZE_MAX = 240 bytes) * or @p secret for "large" keys (>= XXH3_MIDSIZE_MAX). * * This generally benefits speed, compared to `_withSeed()` or `_withSecret()`. * `_withSeed()` has to generate the secret on the fly for "large" keys. * It's fast, but can be perceptible for "not so large" keys (< 1 KB). * `_withSecret()` has to generate the masks on the fly for "small" keys, * which requires more instructions than _withSeed() variants. * Therefore, _withSecretandSeed variant combines the best of both worlds. * * When @p secret has been generated by XXH3_generateSecret_fromSeed(), * this variant produces *exactly* the same results as `_withSeed()` variant, * hence offering only a pure speed benefit on "large" input, * by skipping the need to regenerate the secret for every large input. * * Another usage scenario is to hash the secret to a 64-bit hash value, * for example with XXH3_64bits(), which then becomes the seed, * and then employ both the seed and the secret in _withSecretandSeed(). * On top of speed, an added benefit is that each bit in the secret * has a 50% chance to swap each bit in the output, via its impact to the seed. * * This is not guaranteed when using the secret directly in "small data" scenarios, * because only portions of the secret are employed for small data. */ XXH_PUBLIC_API XXH_PUREF XXH64_hash_t XXH3_64bits_withSecretandSeed(XXH_NOESCAPE const void* data, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed); /*! @copydoc XXH3_64bits_withSecretandSeed() */ XXH_PUBLIC_API XXH_PUREF XXH128_hash_t XXH3_128bits_withSecretandSeed(XXH_NOESCAPE const void* input, size_t length, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed64); #ifndef XXH_NO_STREAM /*! @copydoc XXH3_64bits_withSecretandSeed() */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecretandSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed64); /*! @copydoc XXH3_64bits_withSecretandSeed() */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecretandSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed64); #endif /* !XXH_NO_STREAM */ #endif /* !XXH_NO_XXH3 */ #endif /* XXH_NO_LONG_LONG */ #if defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) # define XXH_IMPLEMENTATION #endif #endif /* defined(XXH_STATIC_LINKING_ONLY) && !defined(XXHASH_H_STATIC_13879238742) */ /* ======================================================================== */ /* ======================================================================== */ /* ======================================================================== */ /*-********************************************************************** * xxHash implementation *-********************************************************************** * xxHash's implementation used to be hosted inside xxhash.c. * * However, inlining requires implementation to be visible to the compiler, * hence be included alongside the header. * Previously, implementation was hosted inside xxhash.c, * which was then #included when inlining was activated. * This construction created issues with a few build and install systems, * as it required xxhash.c to be stored in /include directory. * * xxHash implementation is now directly integrated within xxhash.h. * As a consequence, xxhash.c is no longer needed in /include. * * xxhash.c is still available and is still useful. * In a "normal" setup, when xxhash is not inlined, * xxhash.h only exposes the prototypes and public symbols, * while xxhash.c can be built into an object file xxhash.o * which can then be linked into the final binary. ************************************************************************/ #if ( defined(XXH_INLINE_ALL) || defined(XXH_PRIVATE_API) \ || defined(XXH_IMPLEMENTATION) ) && !defined(XXH_IMPLEM_13a8737387) # define XXH_IMPLEM_13a8737387 /* ************************************* * Tuning parameters ***************************************/ /*! * @defgroup tuning Tuning parameters * @{ * * Various macros to control xxHash's behavior. */ #ifdef XXH_DOXYGEN /*! * @brief Define this to disable 64-bit code. * * Useful if only using the @ref XXH32_family and you have a strict C90 compiler. */ # define XXH_NO_LONG_LONG # undef XXH_NO_LONG_LONG /* don't actually */ /*! * @brief Controls how unaligned memory is accessed. * * By default, access to unaligned memory is controlled by `memcpy()`, which is * safe and portable. * * Unfortunately, on some target/compiler combinations, the generated assembly * is sub-optimal. * * The below switch allow selection of a different access method * in the search for improved performance. * * @par Possible options: * * - `XXH_FORCE_MEMORY_ACCESS=0` (default): `memcpy` * @par * Use `memcpy()`. Safe and portable. Note that most modern compilers will * eliminate the function call and treat it as an unaligned access. * * - `XXH_FORCE_MEMORY_ACCESS=1`: `__attribute__((aligned(1)))` * @par * Depends on compiler extensions and is therefore not portable. * This method is safe _if_ your compiler supports it, * and *generally* as fast or faster than `memcpy`. * * - `XXH_FORCE_MEMORY_ACCESS=2`: Direct cast * @par * Casts directly and dereferences. This method doesn't depend on the * compiler, but it violates the C standard as it directly dereferences an * unaligned pointer. It can generate buggy code on targets which do not * support unaligned memory accesses, but in some circumstances, it's the * only known way to get the most performance. * * - `XXH_FORCE_MEMORY_ACCESS=3`: Byteshift * @par * Also portable. This can generate the best code on old compilers which don't * inline small `memcpy()` calls, and it might also be faster on big-endian * systems which lack a native byteswap instruction. However, some compilers * will emit literal byteshifts even if the target supports unaligned access. * * * @warning * Methods 1 and 2 rely on implementation-defined behavior. Use these with * care, as what works on one compiler/platform/optimization level may cause * another to read garbage data or even crash. * * See https://fastcompression.blogspot.com/2015/08/accessing-unaligned-memory.html for details. * * Prefer these methods in priority order (0 > 3 > 1 > 2) */ # define XXH_FORCE_MEMORY_ACCESS 0 /*! * @def XXH_SIZE_OPT * @brief Controls how much xxHash optimizes for size. * * xxHash, when compiled, tends to result in a rather large binary size. This * is mostly due to heavy usage to forced inlining and constant folding of the * @ref XXH3_family to increase performance. * * However, some developers prefer size over speed. This option can * significantly reduce the size of the generated code. When using the `-Os` * or `-Oz` options on GCC or Clang, this is defined to 1 by default, * otherwise it is defined to 0. * * Most of these size optimizations can be controlled manually. * * This is a number from 0-2. * - `XXH_SIZE_OPT` == 0: Default. xxHash makes no size optimizations. Speed * comes first. * - `XXH_SIZE_OPT` == 1: Default for `-Os` and `-Oz`. xxHash is more * conservative and disables hacks that increase code size. It implies the * options @ref XXH_NO_INLINE_HINTS == 1, @ref XXH_FORCE_ALIGN_CHECK == 0, * and @ref XXH3_NEON_LANES == 8 if they are not already defined. * - `XXH_SIZE_OPT` == 2: xxHash tries to make itself as small as possible. * Performance may cry. For example, the single shot functions just use the * streaming API. */ # define XXH_SIZE_OPT 0 /*! * @def XXH_FORCE_ALIGN_CHECK * @brief If defined to non-zero, adds a special path for aligned inputs (XXH32() * and XXH64() only). * * This is an important performance trick for architectures without decent * unaligned memory access performance. * * It checks for input alignment, and when conditions are met, uses a "fast * path" employing direct 32-bit/64-bit reads, resulting in _dramatically * faster_ read speed. * * The check costs one initial branch per hash, which is generally negligible, * but not zero. * * Moreover, it's not useful to generate an additional code path if memory * access uses the same instruction for both aligned and unaligned * addresses (e.g. x86 and aarch64). * * In these cases, the alignment check can be removed by setting this macro to 0. * Then the code will always use unaligned memory access. * Align check is automatically disabled on x86, x64, ARM64, and some ARM chips * which are platforms known to offer good unaligned memory accesses performance. * * It is also disabled by default when @ref XXH_SIZE_OPT >= 1. * * This option does not affect XXH3 (only XXH32 and XXH64). */ # define XXH_FORCE_ALIGN_CHECK 0 /*! * @def XXH_NO_INLINE_HINTS * @brief When non-zero, sets all functions to `static`. * * By default, xxHash tries to force the compiler to inline almost all internal * functions. * * This can usually improve performance due to reduced jumping and improved * constant folding, but significantly increases the size of the binary which * might not be favorable. * * Additionally, sometimes the forced inlining can be detrimental to performance, * depending on the architecture. * * XXH_NO_INLINE_HINTS marks all internal functions as static, giving the * compiler full control on whether to inline or not. * * When not optimizing (-O0), using `-fno-inline` with GCC or Clang, or if * @ref XXH_SIZE_OPT >= 1, this will automatically be defined. */ # define XXH_NO_INLINE_HINTS 0 /*! * @def XXH3_INLINE_SECRET * @brief Determines whether to inline the XXH3 withSecret code. * * When the secret size is known, the compiler can improve the performance * of XXH3_64bits_withSecret() and XXH3_128bits_withSecret(). * * However, if the secret size is not known, it doesn't have any benefit. This * happens when xxHash is compiled into a global symbol. Therefore, if * @ref XXH_INLINE_ALL is *not* defined, this will be defined to 0. * * Additionally, this defaults to 0 on GCC 12+, which has an issue with function pointers * that are *sometimes* force inline on -Og, and it is impossible to automatically * detect this optimization level. */ # define XXH3_INLINE_SECRET 0 /*! * @def XXH32_ENDJMP * @brief Whether to use a jump for `XXH32_finalize`. * * For performance, `XXH32_finalize` uses multiple branches in the finalizer. * This is generally preferable for performance, * but depending on exact architecture, a jmp may be preferable. * * This setting is only possibly making a difference for very small inputs. */ # define XXH32_ENDJMP 0 /*! * @internal * @brief Redefines old internal names. * * For compatibility with code that uses xxHash's internals before the names * were changed to improve namespacing. There is no other reason to use this. */ # define XXH_OLD_NAMES # undef XXH_OLD_NAMES /* don't actually use, it is ugly. */ /*! * @def XXH_NO_STREAM * @brief Disables the streaming API. * * When xxHash is not inlined and the streaming functions are not used, disabling * the streaming functions can improve code size significantly, especially with * the @ref XXH3_family which tends to make constant folded copies of itself. */ # define XXH_NO_STREAM # undef XXH_NO_STREAM /* don't actually */ #endif /* XXH_DOXYGEN */ /*! * @} */ #ifndef XXH_FORCE_MEMORY_ACCESS /* can be defined externally, on command line for example */ /* prefer __packed__ structures (method 1) for GCC * < ARMv7 with unaligned access (e.g. Raspbian armhf) still uses byte shifting, so we use memcpy * which for some reason does unaligned loads. */ # if defined(__GNUC__) && !(defined(__ARM_ARCH) && __ARM_ARCH < 7 && defined(__ARM_FEATURE_UNALIGNED)) # define XXH_FORCE_MEMORY_ACCESS 1 # endif #endif #ifndef XXH_SIZE_OPT /* default to 1 for -Os or -Oz */ # if (defined(__GNUC__) || defined(__clang__)) && defined(__OPTIMIZE_SIZE__) # define XXH_SIZE_OPT 1 # else # define XXH_SIZE_OPT 0 # endif #endif #ifndef XXH_FORCE_ALIGN_CHECK /* can be defined externally */ /* don't check on sizeopt, x86, aarch64, or arm when unaligned access is available */ # if XXH_SIZE_OPT >= 1 || \ defined(__i386) || defined(__x86_64__) || defined(__aarch64__) || defined(__ARM_FEATURE_UNALIGNED) \ || defined(_M_IX86) || defined(_M_X64) || defined(_M_ARM64) || defined(_M_ARM) /* visual */ # define XXH_FORCE_ALIGN_CHECK 0 # else # define XXH_FORCE_ALIGN_CHECK 1 # endif #endif #ifndef XXH_NO_INLINE_HINTS # if XXH_SIZE_OPT >= 1 || defined(__NO_INLINE__) /* -O0, -fno-inline */ # define XXH_NO_INLINE_HINTS 1 # else # define XXH_NO_INLINE_HINTS 0 # endif #endif #ifndef XXH3_INLINE_SECRET # if (defined(__GNUC__) && !defined(__clang__) && __GNUC__ >= 12) \ || !defined(XXH_INLINE_ALL) # define XXH3_INLINE_SECRET 0 # else # define XXH3_INLINE_SECRET 1 # endif #endif #ifndef XXH32_ENDJMP /* generally preferable for performance */ # define XXH32_ENDJMP 0 #endif /*! * @defgroup impl Implementation * @{ */ /* ************************************* * Includes & Memory related functions ***************************************/ #if defined(XXH_NO_STREAM) /* nothing */ #elif defined(XXH_NO_STDLIB) /* When requesting to disable any mention of stdlib, * the library loses the ability to invoked malloc / free. * In practice, it means that functions like `XXH*_createState()` * will always fail, and return NULL. * This flag is useful in situations where * xxhash.h is integrated into some kernel, embedded or limited environment * without access to dynamic allocation. */ static XXH_CONSTF void* XXH_malloc(size_t s) { (void)s; return NULL; } static void XXH_free(void* p) { (void)p; } #else /* * Modify the local functions below should you wish to use * different memory routines for malloc() and free() */ #include /*! * @internal * @brief Modify this function to use a different routine than malloc(). */ static XXH_MALLOCF void* XXH_malloc(size_t s) { return malloc(s); } /*! * @internal * @brief Modify this function to use a different routine than free(). */ static void XXH_free(void* p) { free(p); } #endif /* XXH_NO_STDLIB */ #include /*! * @internal * @brief Modify this function to use a different routine than memcpy(). */ static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcpy(dest,src,size); } #include /* ULLONG_MAX */ /* ************************************* * Compiler Specific Options ***************************************/ #ifdef _MSC_VER /* Visual Studio warning fix */ # pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */ #endif #if XXH_NO_INLINE_HINTS /* disable inlining hints */ # if defined(__GNUC__) || defined(__clang__) # define XXH_FORCE_INLINE static __attribute__((unused)) # else # define XXH_FORCE_INLINE static # endif # define XXH_NO_INLINE static /* enable inlining hints */ #elif defined(__GNUC__) || defined(__clang__) # define XXH_FORCE_INLINE static __inline__ __attribute__((always_inline, unused)) # define XXH_NO_INLINE static __attribute__((noinline)) #elif defined(_MSC_VER) /* Visual Studio */ # define XXH_FORCE_INLINE static __forceinline # define XXH_NO_INLINE static __declspec(noinline) #elif defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L)) /* C99 */ # define XXH_FORCE_INLINE static inline # define XXH_NO_INLINE static #else # define XXH_FORCE_INLINE static # define XXH_NO_INLINE static #endif #if XXH3_INLINE_SECRET # define XXH3_WITH_SECRET_INLINE XXH_FORCE_INLINE #else # define XXH3_WITH_SECRET_INLINE XXH_NO_INLINE #endif /* ************************************* * Debug ***************************************/ /*! * @ingroup tuning * @def XXH_DEBUGLEVEL * @brief Sets the debugging level. * * XXH_DEBUGLEVEL is expected to be defined externally, typically via the * compiler's command line options. The value must be a number. */ #ifndef XXH_DEBUGLEVEL # ifdef DEBUGLEVEL /* backwards compat */ # define XXH_DEBUGLEVEL DEBUGLEVEL # else # define XXH_DEBUGLEVEL 0 # endif #endif #if (XXH_DEBUGLEVEL>=1) # include /* note: can still be disabled with NDEBUG */ # define XXH_ASSERT(c) assert(c) #else # if defined(__INTEL_COMPILER) # define XXH_ASSERT(c) XXH_ASSUME((unsigned char) (c)) # else # define XXH_ASSERT(c) XXH_ASSUME(c) # endif #endif /* note: use after variable declarations */ #ifndef XXH_STATIC_ASSERT # if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= 201112L) /* C11 */ # define XXH_STATIC_ASSERT_WITH_MESSAGE(c,m) do { _Static_assert((c),m); } while(0) # elif defined(__cplusplus) && (__cplusplus >= 201103L) /* C++11 */ # define XXH_STATIC_ASSERT_WITH_MESSAGE(c,m) do { static_assert((c),m); } while(0) # else # define XXH_STATIC_ASSERT_WITH_MESSAGE(c,m) do { struct xxh_sa { char x[(c) ? 1 : -1]; }; } while(0) # endif # define XXH_STATIC_ASSERT(c) XXH_STATIC_ASSERT_WITH_MESSAGE((c),#c) #endif /*! * @internal * @def XXH_COMPILER_GUARD(var) * @brief Used to prevent unwanted optimizations for @p var. * * It uses an empty GCC inline assembly statement with a register constraint * which forces @p var into a general purpose register (eg eax, ebx, ecx * on x86) and marks it as modified. * * This is used in a few places to avoid unwanted autovectorization (e.g. * XXH32_round()). All vectorization we want is explicit via intrinsics, * and _usually_ isn't wanted elsewhere. * * We also use it to prevent unwanted constant folding for AArch64 in * XXH3_initCustomSecret_scalar(). */ #if defined(__GNUC__) || defined(__clang__) # define XXH_COMPILER_GUARD(var) __asm__("" : "+r" (var)) #else # define XXH_COMPILER_GUARD(var) ((void)0) #endif /* Specifically for NEON vectors which use the "w" constraint, on * Clang. */ #if defined(__clang__) && defined(__ARM_ARCH) && !defined(__wasm__) # define XXH_COMPILER_GUARD_CLANG_NEON(var) __asm__("" : "+w" (var)) #else # define XXH_COMPILER_GUARD_CLANG_NEON(var) ((void)0) #endif /* ************************************* * Basic Types ***************************************/ #if !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) # include typedef uint8_t xxh_u8; #else typedef unsigned char xxh_u8; #endif typedef XXH32_hash_t xxh_u32; #ifdef XXH_OLD_NAMES # warning "XXH_OLD_NAMES is planned to be removed starting v0.9. If the program depends on it, consider moving away from it by employing newer type names directly" # define BYTE xxh_u8 # define U8 xxh_u8 # define U32 xxh_u32 #endif /* *** Memory access *** */ /*! * @internal * @fn xxh_u32 XXH_read32(const void* ptr) * @brief Reads an unaligned 32-bit integer from @p ptr in native endianness. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit native endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readLE32(const void* ptr) * @brief Reads an unaligned 32-bit little endian integer from @p ptr. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit little endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readBE32(const void* ptr) * @brief Reads an unaligned 32-bit big endian integer from @p ptr. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * * @param ptr The pointer to read from. * @return The 32-bit big endian integer from the bytes at @p ptr. */ /*! * @internal * @fn xxh_u32 XXH_readLE32_align(const void* ptr, XXH_alignment align) * @brief Like @ref XXH_readLE32(), but has an option for aligned reads. * * Affected by @ref XXH_FORCE_MEMORY_ACCESS. * Note that when @ref XXH_FORCE_ALIGN_CHECK == 0, the @p align parameter is * always @ref XXH_alignment::XXH_unaligned. * * @param ptr The pointer to read from. * @param align Whether @p ptr is aligned. * @pre * If @p align == @ref XXH_alignment::XXH_aligned, @p ptr must be 4 byte * aligned. * @return The 32-bit little endian integer from the bytes at @p ptr. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) /* * Manual byteshift. Best for old compilers which don't inline memcpy. * We actually directly use XXH_readLE32 and XXH_readBE32. */ #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) /* * Force direct memory access. Only works on CPU which support unaligned memory * access in hardware. */ static xxh_u32 XXH_read32(const void* memPtr) { return *(const xxh_u32*) memPtr; } #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) /* * __attribute__((aligned(1))) is supported by gcc and clang. Originally the * documentation claimed that it only increased the alignment, but actually it * can decrease it on gcc, clang, and icc: * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69502, * https://gcc.godbolt.org/z/xYez1j67Y. */ #ifdef XXH_OLD_NAMES typedef union { xxh_u32 u32; } __attribute__((packed)) unalign; #endif static xxh_u32 XXH_read32(const void* ptr) { typedef __attribute__((aligned(1))) xxh_u32 xxh_unalign32; return *((const xxh_unalign32*)ptr); } #else /* * Portable and safe solution. Generally efficient. * see: https://fastcompression.blogspot.com/2015/08/accessing-unaligned-memory.html */ static xxh_u32 XXH_read32(const void* memPtr) { xxh_u32 val; XXH_memcpy(&val, memPtr, sizeof(val)); return val; } #endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ /* *** Endianness *** */ /*! * @ingroup tuning * @def XXH_CPU_LITTLE_ENDIAN * @brief Whether the target is little endian. * * Defined to 1 if the target is little endian, or 0 if it is big endian. * It can be defined externally, for example on the compiler command line. * * If it is not defined, * a runtime check (which is usually constant folded) is used instead. * * @note * This is not necessarily defined to an integer constant. * * @see XXH_isLittleEndian() for the runtime check. */ #ifndef XXH_CPU_LITTLE_ENDIAN /* * Try to detect endianness automatically, to avoid the nonstandard behavior * in `XXH_isLittleEndian()` */ # if defined(_WIN32) /* Windows is always little endian */ \ || defined(__LITTLE_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) # define XXH_CPU_LITTLE_ENDIAN 1 # elif defined(__BIG_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) # define XXH_CPU_LITTLE_ENDIAN 0 # else /*! * @internal * @brief Runtime check for @ref XXH_CPU_LITTLE_ENDIAN. * * Most compilers will constant fold this. */ static int XXH_isLittleEndian(void) { /* * Portable and well-defined behavior. * Don't use static: it is detrimental to performance. */ const union { xxh_u32 u; xxh_u8 c[4]; } one = { 1 }; return one.c[0]; } # define XXH_CPU_LITTLE_ENDIAN XXH_isLittleEndian() # endif #endif /* **************************************** * Compiler-specific Functions and Macros ******************************************/ #define XXH_GCC_VERSION (__GNUC__ * 100 + __GNUC_MINOR__) #ifdef __has_builtin # define XXH_HAS_BUILTIN(x) __has_builtin(x) #else # define XXH_HAS_BUILTIN(x) 0 #endif /* * C23 and future versions have standard "unreachable()". * Once it has been implemented reliably we can add it as an * additional case: * * ``` * #if defined(__STDC_VERSION__) && (__STDC_VERSION__ >= XXH_C23_VN) * # include * # ifdef unreachable * # define XXH_UNREACHABLE() unreachable() * # endif * #endif * ``` * * Note C++23 also has std::unreachable() which can be detected * as follows: * ``` * #if defined(__cpp_lib_unreachable) && (__cpp_lib_unreachable >= 202202L) * # include * # define XXH_UNREACHABLE() std::unreachable() * #endif * ``` * NB: `__cpp_lib_unreachable` is defined in the `` header. * We don't use that as including `` in `extern "C"` blocks * doesn't work on GCC12 */ #if XXH_HAS_BUILTIN(__builtin_unreachable) # define XXH_UNREACHABLE() __builtin_unreachable() #elif defined(_MSC_VER) # define XXH_UNREACHABLE() __assume(0) #else # define XXH_UNREACHABLE() #endif #if XXH_HAS_BUILTIN(__builtin_assume) # define XXH_ASSUME(c) __builtin_assume(c) #else # define XXH_ASSUME(c) if (!(c)) { XXH_UNREACHABLE(); } #endif /*! * @internal * @def XXH_rotl32(x,r) * @brief 32-bit rotate left. * * @param x The 32-bit integer to be rotated. * @param r The number of bits to rotate. * @pre * @p r > 0 && @p r < 32 * @note * @p x and @p r may be evaluated multiple times. * @return The rotated result. */ #if !defined(NO_CLANG_BUILTIN) && XXH_HAS_BUILTIN(__builtin_rotateleft32) \ && XXH_HAS_BUILTIN(__builtin_rotateleft64) # define XXH_rotl32 __builtin_rotateleft32 # define XXH_rotl64 __builtin_rotateleft64 /* Note: although _rotl exists for minGW (GCC under windows), performance seems poor */ #elif defined(_MSC_VER) # define XXH_rotl32(x,r) _rotl(x,r) # define XXH_rotl64(x,r) _rotl64(x,r) #else # define XXH_rotl32(x,r) (((x) << (r)) | ((x) >> (32 - (r)))) # define XXH_rotl64(x,r) (((x) << (r)) | ((x) >> (64 - (r)))) #endif /*! * @internal * @fn xxh_u32 XXH_swap32(xxh_u32 x) * @brief A 32-bit byteswap. * * @param x The 32-bit integer to byteswap. * @return @p x, byteswapped. */ #if defined(_MSC_VER) /* Visual Studio */ # define XXH_swap32 _byteswap_ulong #elif XXH_GCC_VERSION >= 403 # define XXH_swap32 __builtin_bswap32 #else static xxh_u32 XXH_swap32 (xxh_u32 x) { return ((x << 24) & 0xff000000 ) | ((x << 8) & 0x00ff0000 ) | ((x >> 8) & 0x0000ff00 ) | ((x >> 24) & 0x000000ff ); } #endif /* *************************** * Memory reads *****************************/ /*! * @internal * @brief Enum to indicate whether a pointer is aligned. */ typedef enum { XXH_aligned, /*!< Aligned */ XXH_unaligned /*!< Possibly unaligned */ } XXH_alignment; /* * XXH_FORCE_MEMORY_ACCESS==3 is an endian-independent byteshift load. * * This is ideal for older compilers which don't inline memcpy. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) XXH_FORCE_INLINE xxh_u32 XXH_readLE32(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[0] | ((xxh_u32)bytePtr[1] << 8) | ((xxh_u32)bytePtr[2] << 16) | ((xxh_u32)bytePtr[3] << 24); } XXH_FORCE_INLINE xxh_u32 XXH_readBE32(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[3] | ((xxh_u32)bytePtr[2] << 8) | ((xxh_u32)bytePtr[1] << 16) | ((xxh_u32)bytePtr[0] << 24); } #else XXH_FORCE_INLINE xxh_u32 XXH_readLE32(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr)); } static xxh_u32 XXH_readBE32(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr); } #endif XXH_FORCE_INLINE xxh_u32 XXH_readLE32_align(const void* ptr, XXH_alignment align) { if (align==XXH_unaligned) { return XXH_readLE32(ptr); } else { return XXH_CPU_LITTLE_ENDIAN ? *(const xxh_u32*)ptr : XXH_swap32(*(const xxh_u32*)ptr); } } /* ************************************* * Misc ***************************************/ /*! @ingroup public */ XXH_PUBLIC_API unsigned XXH_versionNumber (void) { return XXH_VERSION_NUMBER; } /* ******************************************************************* * 32-bit hash functions *********************************************************************/ /*! * @} * @defgroup XXH32_impl XXH32 implementation * @ingroup impl * * Details on the XXH32 implementation. * @{ */ /* #define instead of static const, to be used as initializers */ #define XXH_PRIME32_1 0x9E3779B1U /*!< 0b10011110001101110111100110110001 */ #define XXH_PRIME32_2 0x85EBCA77U /*!< 0b10000101111010111100101001110111 */ #define XXH_PRIME32_3 0xC2B2AE3DU /*!< 0b11000010101100101010111000111101 */ #define XXH_PRIME32_4 0x27D4EB2FU /*!< 0b00100111110101001110101100101111 */ #define XXH_PRIME32_5 0x165667B1U /*!< 0b00010110010101100110011110110001 */ #ifdef XXH_OLD_NAMES # define PRIME32_1 XXH_PRIME32_1 # define PRIME32_2 XXH_PRIME32_2 # define PRIME32_3 XXH_PRIME32_3 # define PRIME32_4 XXH_PRIME32_4 # define PRIME32_5 XXH_PRIME32_5 #endif /*! * @internal * @brief Normal stripe processing routine. * * This shuffles the bits so that any bit from @p input impacts several bits in * @p acc. * * @param acc The accumulator lane. * @param input The stripe of input to mix. * @return The mixed accumulator lane. */ static xxh_u32 XXH32_round(xxh_u32 acc, xxh_u32 input) { acc += input * XXH_PRIME32_2; acc = XXH_rotl32(acc, 13); acc *= XXH_PRIME32_1; #if (defined(__SSE4_1__) || defined(__aarch64__) || defined(__wasm_simd128__)) && !defined(XXH_ENABLE_AUTOVECTORIZE) /* * UGLY HACK: * A compiler fence is the only thing that prevents GCC and Clang from * autovectorizing the XXH32 loop (pragmas and attributes don't work for some * reason) without globally disabling SSE4.1. * * The reason we want to avoid vectorization is because despite working on * 4 integers at a time, there are multiple factors slowing XXH32 down on * SSE4: * - There's a ridiculous amount of lag from pmulld (10 cycles of latency on * newer chips!) making it slightly slower to multiply four integers at * once compared to four integers independently. Even when pmulld was * fastest, Sandy/Ivy Bridge, it is still not worth it to go into SSE * just to multiply unless doing a long operation. * * - Four instructions are required to rotate, * movqda tmp, v // not required with VEX encoding * pslld tmp, 13 // tmp <<= 13 * psrld v, 19 // x >>= 19 * por v, tmp // x |= tmp * compared to one for scalar: * roll v, 13 // reliably fast across the board * shldl v, v, 13 // Sandy Bridge and later prefer this for some reason * * - Instruction level parallelism is actually more beneficial here because * the SIMD actually serializes this operation: While v1 is rotating, v2 * can load data, while v3 can multiply. SSE forces them to operate * together. * * This is also enabled on AArch64, as Clang is *very aggressive* in vectorizing * the loop. NEON is only faster on the A53, and with the newer cores, it is less * than half the speed. * * Additionally, this is used on WASM SIMD128 because it JITs to the same * SIMD instructions and has the same issue. */ XXH_COMPILER_GUARD(acc); #endif return acc; } /*! * @internal * @brief Mixes all bits to finalize the hash. * * The final mix ensures that all input bits have a chance to impact any bit in * the output digest, resulting in an unbiased distribution. * * @param hash The hash to avalanche. * @return The avalanched hash. */ static xxh_u32 XXH32_avalanche(xxh_u32 hash) { hash ^= hash >> 15; hash *= XXH_PRIME32_2; hash ^= hash >> 13; hash *= XXH_PRIME32_3; hash ^= hash >> 16; return hash; } #define XXH_get32bits(p) XXH_readLE32_align(p, align) /*! * @internal * @brief Processes the last 0-15 bytes of @p ptr. * * There may be up to 15 bytes remaining to consume from the input. * This final stage will digest them to ensure that all input bytes are present * in the final mix. * * @param hash The hash to finalize. * @param ptr The pointer to the remaining input. * @param len The remaining length, modulo 16. * @param align Whether @p ptr is aligned. * @return The finalized hash. * @see XXH64_finalize(). */ static XXH_PUREF xxh_u32 XXH32_finalize(xxh_u32 hash, const xxh_u8* ptr, size_t len, XXH_alignment align) { #define XXH_PROCESS1 do { \ hash += (*ptr++) * XXH_PRIME32_5; \ hash = XXH_rotl32(hash, 11) * XXH_PRIME32_1; \ } while (0) #define XXH_PROCESS4 do { \ hash += XXH_get32bits(ptr) * XXH_PRIME32_3; \ ptr += 4; \ hash = XXH_rotl32(hash, 17) * XXH_PRIME32_4; \ } while (0) if (ptr==NULL) XXH_ASSERT(len == 0); /* Compact rerolled version; generally faster */ if (!XXH32_ENDJMP) { len &= 15; while (len >= 4) { XXH_PROCESS4; len -= 4; } while (len > 0) { XXH_PROCESS1; --len; } return XXH32_avalanche(hash); } else { switch(len&15) /* or switch(bEnd - p) */ { case 12: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 8: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 4: XXH_PROCESS4; return XXH32_avalanche(hash); case 13: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 9: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 5: XXH_PROCESS4; XXH_PROCESS1; return XXH32_avalanche(hash); case 14: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 10: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 6: XXH_PROCESS4; XXH_PROCESS1; XXH_PROCESS1; return XXH32_avalanche(hash); case 15: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 11: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 7: XXH_PROCESS4; XXH_FALLTHROUGH; /* fallthrough */ case 3: XXH_PROCESS1; XXH_FALLTHROUGH; /* fallthrough */ case 2: XXH_PROCESS1; XXH_FALLTHROUGH; /* fallthrough */ case 1: XXH_PROCESS1; XXH_FALLTHROUGH; /* fallthrough */ case 0: return XXH32_avalanche(hash); } XXH_ASSERT(0); return hash; /* reaching this point is deemed impossible */ } } #ifdef XXH_OLD_NAMES # define PROCESS1 XXH_PROCESS1 # define PROCESS4 XXH_PROCESS4 #else # undef XXH_PROCESS1 # undef XXH_PROCESS4 #endif /*! * @internal * @brief The implementation for @ref XXH32(). * * @param input , len , seed Directly passed from @ref XXH32(). * @param align Whether @p input is aligned. * @return The calculated hash. */ XXH_FORCE_INLINE XXH_PUREF xxh_u32 XXH32_endian_align(const xxh_u8* input, size_t len, xxh_u32 seed, XXH_alignment align) { xxh_u32 h32; if (input==NULL) XXH_ASSERT(len == 0); if (len>=16) { const xxh_u8* const bEnd = input + len; const xxh_u8* const limit = bEnd - 15; xxh_u32 v1 = seed + XXH_PRIME32_1 + XXH_PRIME32_2; xxh_u32 v2 = seed + XXH_PRIME32_2; xxh_u32 v3 = seed + 0; xxh_u32 v4 = seed - XXH_PRIME32_1; do { v1 = XXH32_round(v1, XXH_get32bits(input)); input += 4; v2 = XXH32_round(v2, XXH_get32bits(input)); input += 4; v3 = XXH32_round(v3, XXH_get32bits(input)); input += 4; v4 = XXH32_round(v4, XXH_get32bits(input)); input += 4; } while (input < limit); h32 = XXH_rotl32(v1, 1) + XXH_rotl32(v2, 7) + XXH_rotl32(v3, 12) + XXH_rotl32(v4, 18); } else { h32 = seed + XXH_PRIME32_5; } h32 += (xxh_u32)len; return XXH32_finalize(h32, input, len&15, align); } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32 (const void* input, size_t len, XXH32_hash_t seed) { #if !defined(XXH_NO_STREAM) && XXH_SIZE_OPT >= 2 /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ XXH32_state_t state; XXH32_reset(&state, seed); XXH32_update(&state, (const xxh_u8*)input, len); return XXH32_digest(&state); #else if (XXH_FORCE_ALIGN_CHECK) { if ((((size_t)input) & 3) == 0) { /* Input is 4-bytes aligned, leverage the speed benefit */ return XXH32_endian_align((const xxh_u8*)input, len, seed, XXH_aligned); } } return XXH32_endian_align((const xxh_u8*)input, len, seed, XXH_unaligned); #endif } /******* Hash streaming *******/ #ifndef XXH_NO_STREAM /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH32_state_t* XXH32_createState(void) { return (XXH32_state_t*)XXH_malloc(sizeof(XXH32_state_t)); } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_freeState(XXH32_state_t* statePtr) { XXH_free(statePtr); return XXH_OK; } /*! @ingroup XXH32_family */ XXH_PUBLIC_API void XXH32_copyState(XXH32_state_t* dstState, const XXH32_state_t* srcState) { XXH_memcpy(dstState, srcState, sizeof(*dstState)); } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_reset(XXH32_state_t* statePtr, XXH32_hash_t seed) { XXH_ASSERT(statePtr != NULL); memset(statePtr, 0, sizeof(*statePtr)); statePtr->v[0] = seed + XXH_PRIME32_1 + XXH_PRIME32_2; statePtr->v[1] = seed + XXH_PRIME32_2; statePtr->v[2] = seed + 0; statePtr->v[3] = seed - XXH_PRIME32_1; return XXH_OK; } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH_errorcode XXH32_update(XXH32_state_t* state, const void* input, size_t len) { if (input==NULL) { XXH_ASSERT(len == 0); return XXH_OK; } { const xxh_u8* p = (const xxh_u8*)input; const xxh_u8* const bEnd = p + len; state->total_len_32 += (XXH32_hash_t)len; state->large_len |= (XXH32_hash_t)((len>=16) | (state->total_len_32>=16)); if (state->memsize + len < 16) { /* fill in tmp buffer */ XXH_memcpy((xxh_u8*)(state->mem32) + state->memsize, input, len); state->memsize += (XXH32_hash_t)len; return XXH_OK; } if (state->memsize) { /* some data left from previous update */ XXH_memcpy((xxh_u8*)(state->mem32) + state->memsize, input, 16-state->memsize); { const xxh_u32* p32 = state->mem32; state->v[0] = XXH32_round(state->v[0], XXH_readLE32(p32)); p32++; state->v[1] = XXH32_round(state->v[1], XXH_readLE32(p32)); p32++; state->v[2] = XXH32_round(state->v[2], XXH_readLE32(p32)); p32++; state->v[3] = XXH32_round(state->v[3], XXH_readLE32(p32)); } p += 16-state->memsize; state->memsize = 0; } if (p <= bEnd-16) { const xxh_u8* const limit = bEnd - 16; do { state->v[0] = XXH32_round(state->v[0], XXH_readLE32(p)); p+=4; state->v[1] = XXH32_round(state->v[1], XXH_readLE32(p)); p+=4; state->v[2] = XXH32_round(state->v[2], XXH_readLE32(p)); p+=4; state->v[3] = XXH32_round(state->v[3], XXH_readLE32(p)); p+=4; } while (p<=limit); } if (p < bEnd) { XXH_memcpy(state->mem32, p, (size_t)(bEnd-p)); state->memsize = (unsigned)(bEnd-p); } } return XXH_OK; } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32_digest(const XXH32_state_t* state) { xxh_u32 h32; if (state->large_len) { h32 = XXH_rotl32(state->v[0], 1) + XXH_rotl32(state->v[1], 7) + XXH_rotl32(state->v[2], 12) + XXH_rotl32(state->v[3], 18); } else { h32 = state->v[2] /* == seed */ + XXH_PRIME32_5; } h32 += state->total_len_32; return XXH32_finalize(h32, (const xxh_u8*)state->mem32, state->memsize, XXH_aligned); } #endif /* !XXH_NO_STREAM */ /******* Canonical representation *******/ /*! * @ingroup XXH32_family * The default return values from XXH functions are unsigned 32 and 64 bit * integers. * * The canonical representation uses big endian convention, the same convention * as human-readable numbers (large digits first). * * This way, hash values can be written into a file or buffer, remaining * comparable across different systems. * * The following functions allow transformation of hash values to and from their * canonical format. */ XXH_PUBLIC_API void XXH32_canonicalFromHash(XXH32_canonical_t* dst, XXH32_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH32_canonical_t) == sizeof(XXH32_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap32(hash); XXH_memcpy(dst, &hash, sizeof(*dst)); } /*! @ingroup XXH32_family */ XXH_PUBLIC_API XXH32_hash_t XXH32_hashFromCanonical(const XXH32_canonical_t* src) { return XXH_readBE32(src); } #ifndef XXH_NO_LONG_LONG /* ******************************************************************* * 64-bit hash functions *********************************************************************/ /*! * @} * @ingroup impl * @{ */ /******* Memory access *******/ typedef XXH64_hash_t xxh_u64; #ifdef XXH_OLD_NAMES # define U64 xxh_u64 #endif #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) /* * Manual byteshift. Best for old compilers which don't inline memcpy. * We actually directly use XXH_readLE64 and XXH_readBE64. */ #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==2)) /* Force direct memory access. Only works on CPU which support unaligned memory access in hardware */ static xxh_u64 XXH_read64(const void* memPtr) { return *(const xxh_u64*) memPtr; } #elif (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==1)) /* * __attribute__((aligned(1))) is supported by gcc and clang. Originally the * documentation claimed that it only increased the alignment, but actually it * can decrease it on gcc, clang, and icc: * https://gcc.gnu.org/bugzilla/show_bug.cgi?id=69502, * https://gcc.godbolt.org/z/xYez1j67Y. */ #ifdef XXH_OLD_NAMES typedef union { xxh_u32 u32; xxh_u64 u64; } __attribute__((packed)) unalign64; #endif static xxh_u64 XXH_read64(const void* ptr) { typedef __attribute__((aligned(1))) xxh_u64 xxh_unalign64; return *((const xxh_unalign64*)ptr); } #else /* * Portable and safe solution. Generally efficient. * see: https://fastcompression.blogspot.com/2015/08/accessing-unaligned-memory.html */ static xxh_u64 XXH_read64(const void* memPtr) { xxh_u64 val; XXH_memcpy(&val, memPtr, sizeof(val)); return val; } #endif /* XXH_FORCE_DIRECT_MEMORY_ACCESS */ #if defined(_MSC_VER) /* Visual Studio */ # define XXH_swap64 _byteswap_uint64 #elif XXH_GCC_VERSION >= 403 # define XXH_swap64 __builtin_bswap64 #else static xxh_u64 XXH_swap64(xxh_u64 x) { return ((x << 56) & 0xff00000000000000ULL) | ((x << 40) & 0x00ff000000000000ULL) | ((x << 24) & 0x0000ff0000000000ULL) | ((x << 8) & 0x000000ff00000000ULL) | ((x >> 8) & 0x00000000ff000000ULL) | ((x >> 24) & 0x0000000000ff0000ULL) | ((x >> 40) & 0x000000000000ff00ULL) | ((x >> 56) & 0x00000000000000ffULL); } #endif /* XXH_FORCE_MEMORY_ACCESS==3 is an endian-independent byteshift load. */ #if (defined(XXH_FORCE_MEMORY_ACCESS) && (XXH_FORCE_MEMORY_ACCESS==3)) XXH_FORCE_INLINE xxh_u64 XXH_readLE64(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[0] | ((xxh_u64)bytePtr[1] << 8) | ((xxh_u64)bytePtr[2] << 16) | ((xxh_u64)bytePtr[3] << 24) | ((xxh_u64)bytePtr[4] << 32) | ((xxh_u64)bytePtr[5] << 40) | ((xxh_u64)bytePtr[6] << 48) | ((xxh_u64)bytePtr[7] << 56); } XXH_FORCE_INLINE xxh_u64 XXH_readBE64(const void* memPtr) { const xxh_u8* bytePtr = (const xxh_u8 *)memPtr; return bytePtr[7] | ((xxh_u64)bytePtr[6] << 8) | ((xxh_u64)bytePtr[5] << 16) | ((xxh_u64)bytePtr[4] << 24) | ((xxh_u64)bytePtr[3] << 32) | ((xxh_u64)bytePtr[2] << 40) | ((xxh_u64)bytePtr[1] << 48) | ((xxh_u64)bytePtr[0] << 56); } #else XXH_FORCE_INLINE xxh_u64 XXH_readLE64(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr)); } static xxh_u64 XXH_readBE64(const void* ptr) { return XXH_CPU_LITTLE_ENDIAN ? XXH_swap64(XXH_read64(ptr)) : XXH_read64(ptr); } #endif XXH_FORCE_INLINE xxh_u64 XXH_readLE64_align(const void* ptr, XXH_alignment align) { if (align==XXH_unaligned) return XXH_readLE64(ptr); else return XXH_CPU_LITTLE_ENDIAN ? *(const xxh_u64*)ptr : XXH_swap64(*(const xxh_u64*)ptr); } /******* xxh64 *******/ /*! * @} * @defgroup XXH64_impl XXH64 implementation * @ingroup impl * * Details on the XXH64 implementation. * @{ */ /* #define rather that static const, to be used as initializers */ #define XXH_PRIME64_1 0x9E3779B185EBCA87ULL /*!< 0b1001111000110111011110011011000110000101111010111100101010000111 */ #define XXH_PRIME64_2 0xC2B2AE3D27D4EB4FULL /*!< 0b1100001010110010101011100011110100100111110101001110101101001111 */ #define XXH_PRIME64_3 0x165667B19E3779F9ULL /*!< 0b0001011001010110011001111011000110011110001101110111100111111001 */ #define XXH_PRIME64_4 0x85EBCA77C2B2AE63ULL /*!< 0b1000010111101011110010100111011111000010101100101010111001100011 */ #define XXH_PRIME64_5 0x27D4EB2F165667C5ULL /*!< 0b0010011111010100111010110010111100010110010101100110011111000101 */ #ifdef XXH_OLD_NAMES # define PRIME64_1 XXH_PRIME64_1 # define PRIME64_2 XXH_PRIME64_2 # define PRIME64_3 XXH_PRIME64_3 # define PRIME64_4 XXH_PRIME64_4 # define PRIME64_5 XXH_PRIME64_5 #endif /*! @copydoc XXH32_round */ static xxh_u64 XXH64_round(xxh_u64 acc, xxh_u64 input) { acc += input * XXH_PRIME64_2; acc = XXH_rotl64(acc, 31); acc *= XXH_PRIME64_1; return acc; } static xxh_u64 XXH64_mergeRound(xxh_u64 acc, xxh_u64 val) { val = XXH64_round(0, val); acc ^= val; acc = acc * XXH_PRIME64_1 + XXH_PRIME64_4; return acc; } /*! @copydoc XXH32_avalanche */ static xxh_u64 XXH64_avalanche(xxh_u64 hash) { hash ^= hash >> 33; hash *= XXH_PRIME64_2; hash ^= hash >> 29; hash *= XXH_PRIME64_3; hash ^= hash >> 32; return hash; } #define XXH_get64bits(p) XXH_readLE64_align(p, align) /*! * @internal * @brief Processes the last 0-31 bytes of @p ptr. * * There may be up to 31 bytes remaining to consume from the input. * This final stage will digest them to ensure that all input bytes are present * in the final mix. * * @param hash The hash to finalize. * @param ptr The pointer to the remaining input. * @param len The remaining length, modulo 32. * @param align Whether @p ptr is aligned. * @return The finalized hash * @see XXH32_finalize(). */ static XXH_PUREF xxh_u64 XXH64_finalize(xxh_u64 hash, const xxh_u8* ptr, size_t len, XXH_alignment align) { if (ptr==NULL) XXH_ASSERT(len == 0); len &= 31; while (len >= 8) { xxh_u64 const k1 = XXH64_round(0, XXH_get64bits(ptr)); ptr += 8; hash ^= k1; hash = XXH_rotl64(hash,27) * XXH_PRIME64_1 + XXH_PRIME64_4; len -= 8; } if (len >= 4) { hash ^= (xxh_u64)(XXH_get32bits(ptr)) * XXH_PRIME64_1; ptr += 4; hash = XXH_rotl64(hash, 23) * XXH_PRIME64_2 + XXH_PRIME64_3; len -= 4; } while (len > 0) { hash ^= (*ptr++) * XXH_PRIME64_5; hash = XXH_rotl64(hash, 11) * XXH_PRIME64_1; --len; } return XXH64_avalanche(hash); } #ifdef XXH_OLD_NAMES # define PROCESS1_64 XXH_PROCESS1_64 # define PROCESS4_64 XXH_PROCESS4_64 # define PROCESS8_64 XXH_PROCESS8_64 #else # undef XXH_PROCESS1_64 # undef XXH_PROCESS4_64 # undef XXH_PROCESS8_64 #endif /*! * @internal * @brief The implementation for @ref XXH64(). * * @param input , len , seed Directly passed from @ref XXH64(). * @param align Whether @p input is aligned. * @return The calculated hash. */ XXH_FORCE_INLINE XXH_PUREF xxh_u64 XXH64_endian_align(const xxh_u8* input, size_t len, xxh_u64 seed, XXH_alignment align) { xxh_u64 h64; if (input==NULL) XXH_ASSERT(len == 0); if (len>=32) { const xxh_u8* const bEnd = input + len; const xxh_u8* const limit = bEnd - 31; xxh_u64 v1 = seed + XXH_PRIME64_1 + XXH_PRIME64_2; xxh_u64 v2 = seed + XXH_PRIME64_2; xxh_u64 v3 = seed + 0; xxh_u64 v4 = seed - XXH_PRIME64_1; do { v1 = XXH64_round(v1, XXH_get64bits(input)); input+=8; v2 = XXH64_round(v2, XXH_get64bits(input)); input+=8; v3 = XXH64_round(v3, XXH_get64bits(input)); input+=8; v4 = XXH64_round(v4, XXH_get64bits(input)); input+=8; } while (input= 2 /* Simple version, good for code maintenance, but unfortunately slow for small inputs */ XXH64_state_t state; XXH64_reset(&state, seed); XXH64_update(&state, (const xxh_u8*)input, len); return XXH64_digest(&state); #else if (XXH_FORCE_ALIGN_CHECK) { if ((((size_t)input) & 7)==0) { /* Input is aligned, let's leverage the speed advantage */ return XXH64_endian_align((const xxh_u8*)input, len, seed, XXH_aligned); } } return XXH64_endian_align((const xxh_u8*)input, len, seed, XXH_unaligned); #endif } /******* Hash Streaming *******/ #ifndef XXH_NO_STREAM /*! @ingroup XXH64_family*/ XXH_PUBLIC_API XXH64_state_t* XXH64_createState(void) { return (XXH64_state_t*)XXH_malloc(sizeof(XXH64_state_t)); } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_freeState(XXH64_state_t* statePtr) { XXH_free(statePtr); return XXH_OK; } /*! @ingroup XXH64_family */ XXH_PUBLIC_API void XXH64_copyState(XXH_NOESCAPE XXH64_state_t* dstState, const XXH64_state_t* srcState) { XXH_memcpy(dstState, srcState, sizeof(*dstState)); } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH_NOESCAPE XXH64_state_t* statePtr, XXH64_hash_t seed) { XXH_ASSERT(statePtr != NULL); memset(statePtr, 0, sizeof(*statePtr)); statePtr->v[0] = seed + XXH_PRIME64_1 + XXH_PRIME64_2; statePtr->v[1] = seed + XXH_PRIME64_2; statePtr->v[2] = seed + 0; statePtr->v[3] = seed - XXH_PRIME64_1; return XXH_OK; } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH_NOESCAPE XXH64_state_t* state, XXH_NOESCAPE const void* input, size_t len) { if (input==NULL) { XXH_ASSERT(len == 0); return XXH_OK; } { const xxh_u8* p = (const xxh_u8*)input; const xxh_u8* const bEnd = p + len; state->total_len += len; if (state->memsize + len < 32) { /* fill in tmp buffer */ XXH_memcpy(((xxh_u8*)state->mem64) + state->memsize, input, len); state->memsize += (xxh_u32)len; return XXH_OK; } if (state->memsize) { /* tmp buffer is full */ XXH_memcpy(((xxh_u8*)state->mem64) + state->memsize, input, 32-state->memsize); state->v[0] = XXH64_round(state->v[0], XXH_readLE64(state->mem64+0)); state->v[1] = XXH64_round(state->v[1], XXH_readLE64(state->mem64+1)); state->v[2] = XXH64_round(state->v[2], XXH_readLE64(state->mem64+2)); state->v[3] = XXH64_round(state->v[3], XXH_readLE64(state->mem64+3)); p += 32 - state->memsize; state->memsize = 0; } if (p+32 <= bEnd) { const xxh_u8* const limit = bEnd - 32; do { state->v[0] = XXH64_round(state->v[0], XXH_readLE64(p)); p+=8; state->v[1] = XXH64_round(state->v[1], XXH_readLE64(p)); p+=8; state->v[2] = XXH64_round(state->v[2], XXH_readLE64(p)); p+=8; state->v[3] = XXH64_round(state->v[3], XXH_readLE64(p)); p+=8; } while (p<=limit); } if (p < bEnd) { XXH_memcpy(state->mem64, p, (size_t)(bEnd-p)); state->memsize = (unsigned)(bEnd-p); } } return XXH_OK; } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH64_hash_t XXH64_digest(XXH_NOESCAPE const XXH64_state_t* state) { xxh_u64 h64; if (state->total_len >= 32) { h64 = XXH_rotl64(state->v[0], 1) + XXH_rotl64(state->v[1], 7) + XXH_rotl64(state->v[2], 12) + XXH_rotl64(state->v[3], 18); h64 = XXH64_mergeRound(h64, state->v[0]); h64 = XXH64_mergeRound(h64, state->v[1]); h64 = XXH64_mergeRound(h64, state->v[2]); h64 = XXH64_mergeRound(h64, state->v[3]); } else { h64 = state->v[2] /*seed*/ + XXH_PRIME64_5; } h64 += (xxh_u64) state->total_len; return XXH64_finalize(h64, (const xxh_u8*)state->mem64, (size_t)state->total_len, XXH_aligned); } #endif /* !XXH_NO_STREAM */ /******* Canonical representation *******/ /*! @ingroup XXH64_family */ XXH_PUBLIC_API void XXH64_canonicalFromHash(XXH_NOESCAPE XXH64_canonical_t* dst, XXH64_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH64_canonical_t) == sizeof(XXH64_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) hash = XXH_swap64(hash); XXH_memcpy(dst, &hash, sizeof(*dst)); } /*! @ingroup XXH64_family */ XXH_PUBLIC_API XXH64_hash_t XXH64_hashFromCanonical(XXH_NOESCAPE const XXH64_canonical_t* src) { return XXH_readBE64(src); } #ifndef XXH_NO_XXH3 /* ********************************************************************* * XXH3 * New generation hash designed for speed on small keys and vectorization ************************************************************************ */ /*! * @} * @defgroup XXH3_impl XXH3 implementation * @ingroup impl * @{ */ /* === Compiler specifics === */ #if ((defined(sun) || defined(__sun)) && __cplusplus) /* Solaris includes __STDC_VERSION__ with C++. Tested with GCC 5.5 */ # define XXH_RESTRICT /* disable */ #elif defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* >= C99 */ # define XXH_RESTRICT restrict #elif (defined (__GNUC__) && ((__GNUC__ > 3) || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1))) \ || (defined (__clang__)) \ || (defined (_MSC_VER) && (_MSC_VER >= 1400)) \ || (defined (__INTEL_COMPILER) && (__INTEL_COMPILER >= 1300)) /* * There are a LOT more compilers that recognize __restrict but this * covers the major ones. */ # define XXH_RESTRICT __restrict #else # define XXH_RESTRICT /* disable */ #endif #if (defined(__GNUC__) && (__GNUC__ >= 3)) \ || (defined(__INTEL_COMPILER) && (__INTEL_COMPILER >= 800)) \ || defined(__clang__) # define XXH_likely(x) __builtin_expect(x, 1) # define XXH_unlikely(x) __builtin_expect(x, 0) #else # define XXH_likely(x) (x) # define XXH_unlikely(x) (x) #endif #ifndef XXH_HAS_INCLUDE # ifdef __has_include # define XXH_HAS_INCLUDE(x) __has_include(x) # else # define XXH_HAS_INCLUDE(x) 0 # endif #endif #if defined(__GNUC__) || defined(__clang__) # if defined(__ARM_FEATURE_SVE) # include # endif # if defined(__ARM_NEON__) || defined(__ARM_NEON) \ || (defined(_M_ARM) && _M_ARM >= 7) \ || defined(_M_ARM64) || defined(_M_ARM64EC) \ || (defined(__wasm_simd128__) && XXH_HAS_INCLUDE()) /* WASM SIMD128 via SIMDe */ # define inline __inline__ /* circumvent a clang bug */ # include # undef inline # elif defined(__AVX2__) # include # elif defined(__SSE2__) # include # endif #endif #if defined(_MSC_VER) # include #endif /* * One goal of XXH3 is to make it fast on both 32-bit and 64-bit, while * remaining a true 64-bit/128-bit hash function. * * This is done by prioritizing a subset of 64-bit operations that can be * emulated without too many steps on the average 32-bit machine. * * For example, these two lines seem similar, and run equally fast on 64-bit: * * xxh_u64 x; * x ^= (x >> 47); // good * x ^= (x >> 13); // bad * * However, to a 32-bit machine, there is a major difference. * * x ^= (x >> 47) looks like this: * * x.lo ^= (x.hi >> (47 - 32)); * * while x ^= (x >> 13) looks like this: * * // note: funnel shifts are not usually cheap. * x.lo ^= (x.lo >> 13) | (x.hi << (32 - 13)); * x.hi ^= (x.hi >> 13); * * The first one is significantly faster than the second, simply because the * shift is larger than 32. This means: * - All the bits we need are in the upper 32 bits, so we can ignore the lower * 32 bits in the shift. * - The shift result will always fit in the lower 32 bits, and therefore, * we can ignore the upper 32 bits in the xor. * * Thanks to this optimization, XXH3 only requires these features to be efficient: * * - Usable unaligned access * - A 32-bit or 64-bit ALU * - If 32-bit, a decent ADC instruction * - A 32 or 64-bit multiply with a 64-bit result * - For the 128-bit variant, a decent byteswap helps short inputs. * * The first two are already required by XXH32, and almost all 32-bit and 64-bit * platforms which can run XXH32 can run XXH3 efficiently. * * Thumb-1, the classic 16-bit only subset of ARM's instruction set, is one * notable exception. * * First of all, Thumb-1 lacks support for the UMULL instruction which * performs the important long multiply. This means numerous __aeabi_lmul * calls. * * Second of all, the 8 functional registers are just not enough. * Setup for __aeabi_lmul, byteshift loads, pointers, and all arithmetic need * Lo registers, and this shuffling results in thousands more MOVs than A32. * * A32 and T32 don't have this limitation. They can access all 14 registers, * do a 32->64 multiply with UMULL, and the flexible operand allowing free * shifts is helpful, too. * * Therefore, we do a quick sanity check. * * If compiling Thumb-1 for a target which supports ARM instructions, we will * emit a warning, as it is not a "sane" platform to compile for. * * Usually, if this happens, it is because of an accident and you probably need * to specify -march, as you likely meant to compile for a newer architecture. * * Credit: large sections of the vectorial and asm source code paths * have been contributed by @easyaspi314 */ #if defined(__thumb__) && !defined(__thumb2__) && defined(__ARM_ARCH_ISA_ARM) # warning "XXH3 is highly inefficient without ARM or Thumb-2." #endif /* ========================================== * Vectorization detection * ========================================== */ #ifdef XXH_DOXYGEN /*! * @ingroup tuning * @brief Overrides the vectorization implementation chosen for XXH3. * * Can be defined to 0 to disable SIMD or any of the values mentioned in * @ref XXH_VECTOR_TYPE. * * If this is not defined, it uses predefined macros to determine the best * implementation. */ # define XXH_VECTOR XXH_SCALAR /*! * @ingroup tuning * @brief Possible values for @ref XXH_VECTOR. * * Note that these are actually implemented as macros. * * If this is not defined, it is detected automatically. * internal macro XXH_X86DISPATCH overrides this. */ enum XXH_VECTOR_TYPE /* fake enum */ { XXH_SCALAR = 0, /*!< Portable scalar version */ XXH_SSE2 = 1, /*!< * SSE2 for Pentium 4, Opteron, all x86_64. * * @note SSE2 is also guaranteed on Windows 10, macOS, and * Android x86. */ XXH_AVX2 = 2, /*!< AVX2 for Haswell and Bulldozer */ XXH_AVX512 = 3, /*!< AVX512 for Skylake and Icelake */ XXH_NEON = 4, /*!< * NEON for most ARMv7-A, all AArch64, and WASM SIMD128 * via the SIMDeverywhere polyfill provided with the * Emscripten SDK. */ XXH_VSX = 5, /*!< VSX and ZVector for POWER8/z13 (64-bit) */ XXH_SVE = 6, /*!< SVE for some ARMv8-A and ARMv9-A */ }; /*! * @ingroup tuning * @brief Selects the minimum alignment for XXH3's accumulators. * * When using SIMD, this should match the alignment required for said vector * type, so, for example, 32 for AVX2. * * Default: Auto detected. */ # define XXH_ACC_ALIGN 8 #endif /* Actual definition */ #ifndef XXH_DOXYGEN # define XXH_SCALAR 0 # define XXH_SSE2 1 # define XXH_AVX2 2 # define XXH_AVX512 3 # define XXH_NEON 4 # define XXH_VSX 5 # define XXH_SVE 6 #endif #ifndef XXH_VECTOR /* can be defined on command line */ # if defined(__ARM_FEATURE_SVE) # define XXH_VECTOR XXH_SVE # elif ( \ defined(__ARM_NEON__) || defined(__ARM_NEON) /* gcc */ \ || defined(_M_ARM) || defined(_M_ARM64) || defined(_M_ARM64EC) /* msvc */ \ || (defined(__wasm_simd128__) && XXH_HAS_INCLUDE()) /* wasm simd128 via SIMDe */ \ ) && ( \ defined(_WIN32) || defined(__LITTLE_ENDIAN__) /* little endian only */ \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__) \ ) # define XXH_VECTOR XXH_NEON # elif defined(__AVX512F__) # define XXH_VECTOR XXH_AVX512 # elif defined(__AVX2__) # define XXH_VECTOR XXH_AVX2 # elif defined(__SSE2__) || defined(_M_AMD64) || defined(_M_X64) || (defined(_M_IX86_FP) && (_M_IX86_FP == 2)) # define XXH_VECTOR XXH_SSE2 # elif (defined(__PPC64__) && defined(__POWER8_VECTOR__)) \ || (defined(__s390x__) && defined(__VEC__)) \ && defined(__GNUC__) /* TODO: IBM XL */ # define XXH_VECTOR XXH_VSX # else # define XXH_VECTOR XXH_SCALAR # endif #endif /* __ARM_FEATURE_SVE is only supported by GCC & Clang. */ #if (XXH_VECTOR == XXH_SVE) && !defined(__ARM_FEATURE_SVE) # ifdef _MSC_VER # pragma warning(once : 4606) # else # warning "__ARM_FEATURE_SVE isn't supported. Use SCALAR instead." # endif # undef XXH_VECTOR # define XXH_VECTOR XXH_SCALAR #endif /* * Controls the alignment of the accumulator, * for compatibility with aligned vector loads, which are usually faster. */ #ifndef XXH_ACC_ALIGN # if defined(XXH_X86DISPATCH) # define XXH_ACC_ALIGN 64 /* for compatibility with avx512 */ # elif XXH_VECTOR == XXH_SCALAR /* scalar */ # define XXH_ACC_ALIGN 8 # elif XXH_VECTOR == XXH_SSE2 /* sse2 */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_AVX2 /* avx2 */ # define XXH_ACC_ALIGN 32 # elif XXH_VECTOR == XXH_NEON /* neon */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_VSX /* vsx */ # define XXH_ACC_ALIGN 16 # elif XXH_VECTOR == XXH_AVX512 /* avx512 */ # define XXH_ACC_ALIGN 64 # elif XXH_VECTOR == XXH_SVE /* sve */ # define XXH_ACC_ALIGN 64 # endif #endif #if defined(XXH_X86DISPATCH) || XXH_VECTOR == XXH_SSE2 \ || XXH_VECTOR == XXH_AVX2 || XXH_VECTOR == XXH_AVX512 # define XXH_SEC_ALIGN XXH_ACC_ALIGN #elif XXH_VECTOR == XXH_SVE # define XXH_SEC_ALIGN XXH_ACC_ALIGN #else # define XXH_SEC_ALIGN 8 #endif #if defined(__GNUC__) || defined(__clang__) # define XXH_ALIASING __attribute__((may_alias)) #else # define XXH_ALIASING /* nothing */ #endif /* * UGLY HACK: * GCC usually generates the best code with -O3 for xxHash. * * However, when targeting AVX2, it is overzealous in its unrolling resulting * in code roughly 3/4 the speed of Clang. * * There are other issues, such as GCC splitting _mm256_loadu_si256 into * _mm_loadu_si128 + _mm256_inserti128_si256. This is an optimization which * only applies to Sandy and Ivy Bridge... which don't even support AVX2. * * That is why when compiling the AVX2 version, it is recommended to use either * -O2 -mavx2 -march=haswell * or * -O2 -mavx2 -mno-avx256-split-unaligned-load * for decent performance, or to use Clang instead. * * Fortunately, we can control the first one with a pragma that forces GCC into * -O2, but the other one we can't control without "failed to inline always * inline function due to target mismatch" warnings. */ #if XXH_VECTOR == XXH_AVX2 /* AVX2 */ \ && defined(__GNUC__) && !defined(__clang__) /* GCC, not Clang */ \ && defined(__OPTIMIZE__) && XXH_SIZE_OPT <= 0 /* respect -O0 and -Os */ # pragma GCC push_options # pragma GCC optimize("-O2") #endif #if XXH_VECTOR == XXH_NEON /* * UGLY HACK: While AArch64 GCC on Linux does not seem to care, on macOS, GCC -O3 * optimizes out the entire hashLong loop because of the aliasing violation. * * However, GCC is also inefficient at load-store optimization with vld1q/vst1q, * so the only option is to mark it as aliasing. */ typedef uint64x2_t xxh_aliasing_uint64x2_t XXH_ALIASING; /*! * @internal * @brief `vld1q_u64` but faster and alignment-safe. * * On AArch64, unaligned access is always safe, but on ARMv7-a, it is only * *conditionally* safe (`vld1` has an alignment bit like `movdq[ua]` in x86). * * GCC for AArch64 sees `vld1q_u8` as an intrinsic instead of a load, so it * prohibits load-store optimizations. Therefore, a direct dereference is used. * * Otherwise, `vld1q_u8` is used with `vreinterpretq_u8_u64` to do a safe * unaligned load. */ #if defined(__aarch64__) && defined(__GNUC__) && !defined(__clang__) XXH_FORCE_INLINE uint64x2_t XXH_vld1q_u64(void const* ptr) /* silence -Wcast-align */ { return *(xxh_aliasing_uint64x2_t const *)ptr; } #else XXH_FORCE_INLINE uint64x2_t XXH_vld1q_u64(void const* ptr) { return vreinterpretq_u64_u8(vld1q_u8((uint8_t const*)ptr)); } #endif /*! * @internal * @brief `vmlal_u32` on low and high halves of a vector. * * This is a workaround for AArch64 GCC < 11 which implemented arm_neon.h with * inline assembly and were therefore incapable of merging the `vget_{low, high}_u32` * with `vmlal_u32`. */ #if defined(__aarch64__) && defined(__GNUC__) && !defined(__clang__) && __GNUC__ < 11 XXH_FORCE_INLINE uint64x2_t XXH_vmlal_low_u32(uint64x2_t acc, uint32x4_t lhs, uint32x4_t rhs) { /* Inline assembly is the only way */ __asm__("umlal %0.2d, %1.2s, %2.2s" : "+w" (acc) : "w" (lhs), "w" (rhs)); return acc; } XXH_FORCE_INLINE uint64x2_t XXH_vmlal_high_u32(uint64x2_t acc, uint32x4_t lhs, uint32x4_t rhs) { /* This intrinsic works as expected */ return vmlal_high_u32(acc, lhs, rhs); } #else /* Portable intrinsic versions */ XXH_FORCE_INLINE uint64x2_t XXH_vmlal_low_u32(uint64x2_t acc, uint32x4_t lhs, uint32x4_t rhs) { return vmlal_u32(acc, vget_low_u32(lhs), vget_low_u32(rhs)); } /*! @copydoc XXH_vmlal_low_u32 * Assume the compiler converts this to vmlal_high_u32 on aarch64 */ XXH_FORCE_INLINE uint64x2_t XXH_vmlal_high_u32(uint64x2_t acc, uint32x4_t lhs, uint32x4_t rhs) { return vmlal_u32(acc, vget_high_u32(lhs), vget_high_u32(rhs)); } #endif /*! * @ingroup tuning * @brief Controls the NEON to scalar ratio for XXH3 * * This can be set to 2, 4, 6, or 8. * * ARM Cortex CPUs are _very_ sensitive to how their pipelines are used. * * For example, the Cortex-A73 can dispatch 3 micro-ops per cycle, but only 2 of those * can be NEON. If you are only using NEON instructions, you are only using 2/3 of the CPU * bandwidth. * * This is even more noticeable on the more advanced cores like the Cortex-A76 which * can dispatch 8 micro-ops per cycle, but still only 2 NEON micro-ops at once. * * Therefore, to make the most out of the pipeline, it is beneficial to run 6 NEON lanes * and 2 scalar lanes, which is chosen by default. * * This does not apply to Apple processors or 32-bit processors, which run better with * full NEON. These will default to 8. Additionally, size-optimized builds run 8 lanes. * * This change benefits CPUs with large micro-op buffers without negatively affecting * most other CPUs: * * | Chipset | Dispatch type | NEON only | 6:2 hybrid | Diff. | * |:----------------------|:--------------------|----------:|-----------:|------:| * | Snapdragon 730 (A76) | 2 NEON/8 micro-ops | 8.8 GB/s | 10.1 GB/s | ~16% | * | Snapdragon 835 (A73) | 2 NEON/3 micro-ops | 5.1 GB/s | 5.3 GB/s | ~5% | * | Marvell PXA1928 (A53) | In-order dual-issue | 1.9 GB/s | 1.9 GB/s | 0% | * | Apple M1 | 4 NEON/8 micro-ops | 37.3 GB/s | 36.1 GB/s | ~-3% | * * It also seems to fix some bad codegen on GCC, making it almost as fast as clang. * * When using WASM SIMD128, if this is 2 or 6, SIMDe will scalarize 2 of the lanes meaning * it effectively becomes worse 4. * * @see XXH3_accumulate_512_neon() */ # ifndef XXH3_NEON_LANES # if (defined(__aarch64__) || defined(__arm64__) || defined(_M_ARM64) || defined(_M_ARM64EC)) \ && !defined(__APPLE__) && XXH_SIZE_OPT <= 0 # define XXH3_NEON_LANES 6 # else # define XXH3_NEON_LANES XXH_ACC_NB # endif # endif #endif /* XXH_VECTOR == XXH_NEON */ /* * VSX and Z Vector helpers. * * This is very messy, and any pull requests to clean this up are welcome. * * There are a lot of problems with supporting VSX and s390x, due to * inconsistent intrinsics, spotty coverage, and multiple endiannesses. */ #if XXH_VECTOR == XXH_VSX /* Annoyingly, these headers _may_ define three macros: `bool`, `vector`, * and `pixel`. This is a problem for obvious reasons. * * These keywords are unnecessary; the spec literally says they are * equivalent to `__bool`, `__vector`, and `__pixel` and may be undef'd * after including the header. * * We use pragma push_macro/pop_macro to keep the namespace clean. */ # pragma push_macro("bool") # pragma push_macro("vector") # pragma push_macro("pixel") /* silence potential macro redefined warnings */ # undef bool # undef vector # undef pixel # if defined(__s390x__) # include # else # include # endif /* Restore the original macro values, if applicable. */ # pragma pop_macro("pixel") # pragma pop_macro("vector") # pragma pop_macro("bool") typedef __vector unsigned long long xxh_u64x2; typedef __vector unsigned char xxh_u8x16; typedef __vector unsigned xxh_u32x4; /* * UGLY HACK: Similar to aarch64 macOS GCC, s390x GCC has the same aliasing issue. */ typedef xxh_u64x2 xxh_aliasing_u64x2 XXH_ALIASING; # ifndef XXH_VSX_BE # if defined(__BIG_ENDIAN__) \ || (defined(__BYTE_ORDER__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) # define XXH_VSX_BE 1 # elif defined(__VEC_ELEMENT_REG_ORDER__) && __VEC_ELEMENT_REG_ORDER__ == __ORDER_BIG_ENDIAN__ # warning "-maltivec=be is not recommended. Please use native endianness." # define XXH_VSX_BE 1 # else # define XXH_VSX_BE 0 # endif # endif /* !defined(XXH_VSX_BE) */ # if XXH_VSX_BE # if defined(__POWER9_VECTOR__) || (defined(__clang__) && defined(__s390x__)) # define XXH_vec_revb vec_revb # else /*! * A polyfill for POWER9's vec_revb(). */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_revb(xxh_u64x2 val) { xxh_u8x16 const vByteSwap = { 0x07, 0x06, 0x05, 0x04, 0x03, 0x02, 0x01, 0x00, 0x0F, 0x0E, 0x0D, 0x0C, 0x0B, 0x0A, 0x09, 0x08 }; return vec_perm(val, val, vByteSwap); } # endif # endif /* XXH_VSX_BE */ /*! * Performs an unaligned vector load and byte swaps it on big endian. */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_loadu(const void *ptr) { xxh_u64x2 ret; XXH_memcpy(&ret, ptr, sizeof(xxh_u64x2)); # if XXH_VSX_BE ret = XXH_vec_revb(ret); # endif return ret; } /* * vec_mulo and vec_mule are very problematic intrinsics on PowerPC * * These intrinsics weren't added until GCC 8, despite existing for a while, * and they are endian dependent. Also, their meaning swap depending on version. * */ # if defined(__s390x__) /* s390x is always big endian, no issue on this platform */ # define XXH_vec_mulo vec_mulo # define XXH_vec_mule vec_mule # elif defined(__clang__) && XXH_HAS_BUILTIN(__builtin_altivec_vmuleuw) && !defined(__ibmxl__) /* Clang has a better way to control this, we can just use the builtin which doesn't swap. */ /* The IBM XL Compiler (which defined __clang__) only implements the vec_* operations */ # define XXH_vec_mulo __builtin_altivec_vmulouw # define XXH_vec_mule __builtin_altivec_vmuleuw # else /* gcc needs inline assembly */ /* Adapted from https://github.com/google/highwayhash/blob/master/highwayhash/hh_vsx.h. */ XXH_FORCE_INLINE xxh_u64x2 XXH_vec_mulo(xxh_u32x4 a, xxh_u32x4 b) { xxh_u64x2 result; __asm__("vmulouw %0, %1, %2" : "=v" (result) : "v" (a), "v" (b)); return result; } XXH_FORCE_INLINE xxh_u64x2 XXH_vec_mule(xxh_u32x4 a, xxh_u32x4 b) { xxh_u64x2 result; __asm__("vmuleuw %0, %1, %2" : "=v" (result) : "v" (a), "v" (b)); return result; } # endif /* XXH_vec_mulo, XXH_vec_mule */ #endif /* XXH_VECTOR == XXH_VSX */ #if XXH_VECTOR == XXH_SVE #define ACCRND(acc, offset) \ do { \ svuint64_t input_vec = svld1_u64(mask, xinput + offset); \ svuint64_t secret_vec = svld1_u64(mask, xsecret + offset); \ svuint64_t mixed = sveor_u64_x(mask, secret_vec, input_vec); \ svuint64_t swapped = svtbl_u64(input_vec, kSwap); \ svuint64_t mixed_lo = svextw_u64_x(mask, mixed); \ svuint64_t mixed_hi = svlsr_n_u64_x(mask, mixed, 32); \ svuint64_t mul = svmad_u64_x(mask, mixed_lo, mixed_hi, swapped); \ acc = svadd_u64_x(mask, acc, mul); \ } while (0) #endif /* XXH_VECTOR == XXH_SVE */ /* prefetch * can be disabled, by declaring XXH_NO_PREFETCH build macro */ #if defined(XXH_NO_PREFETCH) # define XXH_PREFETCH(ptr) (void)(ptr) /* disabled */ #else # if XXH_SIZE_OPT >= 1 # define XXH_PREFETCH(ptr) (void)(ptr) # elif defined(_MSC_VER) && (defined(_M_X64) || defined(_M_IX86)) /* _mm_prefetch() not defined outside of x86/x64 */ # include /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */ # define XXH_PREFETCH(ptr) _mm_prefetch((const char*)(ptr), _MM_HINT_T0) # elif defined(__GNUC__) && ( (__GNUC__ >= 4) || ( (__GNUC__ == 3) && (__GNUC_MINOR__ >= 1) ) ) # define XXH_PREFETCH(ptr) __builtin_prefetch((ptr), 0 /* rw==read */, 3 /* locality */) # else # define XXH_PREFETCH(ptr) (void)(ptr) /* disabled */ # endif #endif /* XXH_NO_PREFETCH */ /* ========================================== * XXH3 default settings * ========================================== */ #define XXH_SECRET_DEFAULT_SIZE 192 /* minimum XXH3_SECRET_SIZE_MIN */ #if (XXH_SECRET_DEFAULT_SIZE < XXH3_SECRET_SIZE_MIN) # error "default keyset is not large enough" #endif /*! Pseudorandom secret taken directly from FARSH. */ XXH_ALIGN(64) static const xxh_u8 XXH3_kSecret[XXH_SECRET_DEFAULT_SIZE] = { 0xb8, 0xfe, 0x6c, 0x39, 0x23, 0xa4, 0x4b, 0xbe, 0x7c, 0x01, 0x81, 0x2c, 0xf7, 0x21, 0xad, 0x1c, 0xde, 0xd4, 0x6d, 0xe9, 0x83, 0x90, 0x97, 0xdb, 0x72, 0x40, 0xa4, 0xa4, 0xb7, 0xb3, 0x67, 0x1f, 0xcb, 0x79, 0xe6, 0x4e, 0xcc, 0xc0, 0xe5, 0x78, 0x82, 0x5a, 0xd0, 0x7d, 0xcc, 0xff, 0x72, 0x21, 0xb8, 0x08, 0x46, 0x74, 0xf7, 0x43, 0x24, 0x8e, 0xe0, 0x35, 0x90, 0xe6, 0x81, 0x3a, 0x26, 0x4c, 0x3c, 0x28, 0x52, 0xbb, 0x91, 0xc3, 0x00, 0xcb, 0x88, 0xd0, 0x65, 0x8b, 0x1b, 0x53, 0x2e, 0xa3, 0x71, 0x64, 0x48, 0x97, 0xa2, 0x0d, 0xf9, 0x4e, 0x38, 0x19, 0xef, 0x46, 0xa9, 0xde, 0xac, 0xd8, 0xa8, 0xfa, 0x76, 0x3f, 0xe3, 0x9c, 0x34, 0x3f, 0xf9, 0xdc, 0xbb, 0xc7, 0xc7, 0x0b, 0x4f, 0x1d, 0x8a, 0x51, 0xe0, 0x4b, 0xcd, 0xb4, 0x59, 0x31, 0xc8, 0x9f, 0x7e, 0xc9, 0xd9, 0x78, 0x73, 0x64, 0xea, 0xc5, 0xac, 0x83, 0x34, 0xd3, 0xeb, 0xc3, 0xc5, 0x81, 0xa0, 0xff, 0xfa, 0x13, 0x63, 0xeb, 0x17, 0x0d, 0xdd, 0x51, 0xb7, 0xf0, 0xda, 0x49, 0xd3, 0x16, 0x55, 0x26, 0x29, 0xd4, 0x68, 0x9e, 0x2b, 0x16, 0xbe, 0x58, 0x7d, 0x47, 0xa1, 0xfc, 0x8f, 0xf8, 0xb8, 0xd1, 0x7a, 0xd0, 0x31, 0xce, 0x45, 0xcb, 0x3a, 0x8f, 0x95, 0x16, 0x04, 0x28, 0xaf, 0xd7, 0xfb, 0xca, 0xbb, 0x4b, 0x40, 0x7e, }; static const xxh_u64 PRIME_MX1 = 0x165667919E3779F9ULL; /*!< 0b0001011001010110011001111001000110011110001101110111100111111001 */ static const xxh_u64 PRIME_MX2 = 0x9FB21C651E98DF25ULL; /*!< 0b1001111110110010000111000110010100011110100110001101111100100101 */ #ifdef XXH_OLD_NAMES # define kSecret XXH3_kSecret #endif #ifdef XXH_DOXYGEN /*! * @brief Calculates a 32-bit to 64-bit long multiply. * * Implemented as a macro. * * Wraps `__emulu` on MSVC x86 because it tends to call `__allmul` when it doesn't * need to (but it shouldn't need to anyways, it is about 7 instructions to do * a 64x64 multiply...). Since we know that this will _always_ emit `MULL`, we * use that instead of the normal method. * * If you are compiling for platforms like Thumb-1 and don't have a better option, * you may also want to write your own long multiply routine here. * * @param x, y Numbers to be multiplied * @return 64-bit product of the low 32 bits of @p x and @p y. */ XXH_FORCE_INLINE xxh_u64 XXH_mult32to64(xxh_u64 x, xxh_u64 y) { return (x & 0xFFFFFFFF) * (y & 0xFFFFFFFF); } #elif defined(_MSC_VER) && defined(_M_IX86) # define XXH_mult32to64(x, y) __emulu((unsigned)(x), (unsigned)(y)) #else /* * Downcast + upcast is usually better than masking on older compilers like * GCC 4.2 (especially 32-bit ones), all without affecting newer compilers. * * The other method, (x & 0xFFFFFFFF) * (y & 0xFFFFFFFF), will AND both operands * and perform a full 64x64 multiply -- entirely redundant on 32-bit. */ # define XXH_mult32to64(x, y) ((xxh_u64)(xxh_u32)(x) * (xxh_u64)(xxh_u32)(y)) #endif /*! * @brief Calculates a 64->128-bit long multiply. * * Uses `__uint128_t` and `_umul128` if available, otherwise uses a scalar * version. * * @param lhs , rhs The 64-bit integers to be multiplied * @return The 128-bit result represented in an @ref XXH128_hash_t. */ static XXH128_hash_t XXH_mult64to128(xxh_u64 lhs, xxh_u64 rhs) { /* * GCC/Clang __uint128_t method. * * On most 64-bit targets, GCC and Clang define a __uint128_t type. * This is usually the best way as it usually uses a native long 64-bit * multiply, such as MULQ on x86_64 or MUL + UMULH on aarch64. * * Usually. * * Despite being a 32-bit platform, Clang (and emscripten) define this type * despite not having the arithmetic for it. This results in a laggy * compiler builtin call which calculates a full 128-bit multiply. * In that case it is best to use the portable one. * https://github.com/Cyan4973/xxHash/issues/211#issuecomment-515575677 */ #if (defined(__GNUC__) || defined(__clang__)) && !defined(__wasm__) \ && defined(__SIZEOF_INT128__) \ || (defined(_INTEGRAL_MAX_BITS) && _INTEGRAL_MAX_BITS >= 128) __uint128_t const product = (__uint128_t)lhs * (__uint128_t)rhs; XXH128_hash_t r128; r128.low64 = (xxh_u64)(product); r128.high64 = (xxh_u64)(product >> 64); return r128; /* * MSVC for x64's _umul128 method. * * xxh_u64 _umul128(xxh_u64 Multiplier, xxh_u64 Multiplicand, xxh_u64 *HighProduct); * * This compiles to single operand MUL on x64. */ #elif (defined(_M_X64) || defined(_M_IA64)) && !defined(_M_ARM64EC) #ifndef _MSC_VER # pragma intrinsic(_umul128) #endif xxh_u64 product_high; xxh_u64 const product_low = _umul128(lhs, rhs, &product_high); XXH128_hash_t r128; r128.low64 = product_low; r128.high64 = product_high; return r128; /* * MSVC for ARM64's __umulh method. * * This compiles to the same MUL + UMULH as GCC/Clang's __uint128_t method. */ #elif defined(_M_ARM64) || defined(_M_ARM64EC) #ifndef _MSC_VER # pragma intrinsic(__umulh) #endif XXH128_hash_t r128; r128.low64 = lhs * rhs; r128.high64 = __umulh(lhs, rhs); return r128; #else /* * Portable scalar method. Optimized for 32-bit and 64-bit ALUs. * * This is a fast and simple grade school multiply, which is shown below * with base 10 arithmetic instead of base 0x100000000. * * 9 3 // D2 lhs = 93 * x 7 5 // D2 rhs = 75 * ---------- * 1 5 // D2 lo_lo = (93 % 10) * (75 % 10) = 15 * 4 5 | // D2 hi_lo = (93 / 10) * (75 % 10) = 45 * 2 1 | // D2 lo_hi = (93 % 10) * (75 / 10) = 21 * + 6 3 | | // D2 hi_hi = (93 / 10) * (75 / 10) = 63 * --------- * 2 7 | // D2 cross = (15 / 10) + (45 % 10) + 21 = 27 * + 6 7 | | // D2 upper = (27 / 10) + (45 / 10) + 63 = 67 * --------- * 6 9 7 5 // D4 res = (27 * 10) + (15 % 10) + (67 * 100) = 6975 * * The reasons for adding the products like this are: * 1. It avoids manual carry tracking. Just like how * (9 * 9) + 9 + 9 = 99, the same applies with this for UINT64_MAX. * This avoids a lot of complexity. * * 2. It hints for, and on Clang, compiles to, the powerful UMAAL * instruction available in ARM's Digital Signal Processing extension * in 32-bit ARMv6 and later, which is shown below: * * void UMAAL(xxh_u32 *RdLo, xxh_u32 *RdHi, xxh_u32 Rn, xxh_u32 Rm) * { * xxh_u64 product = (xxh_u64)*RdLo * (xxh_u64)*RdHi + Rn + Rm; * *RdLo = (xxh_u32)(product & 0xFFFFFFFF); * *RdHi = (xxh_u32)(product >> 32); * } * * This instruction was designed for efficient long multiplication, and * allows this to be calculated in only 4 instructions at speeds * comparable to some 64-bit ALUs. * * 3. It isn't terrible on other platforms. Usually this will be a couple * of 32-bit ADD/ADCs. */ /* First calculate all of the cross products. */ xxh_u64 const lo_lo = XXH_mult32to64(lhs & 0xFFFFFFFF, rhs & 0xFFFFFFFF); xxh_u64 const hi_lo = XXH_mult32to64(lhs >> 32, rhs & 0xFFFFFFFF); xxh_u64 const lo_hi = XXH_mult32to64(lhs & 0xFFFFFFFF, rhs >> 32); xxh_u64 const hi_hi = XXH_mult32to64(lhs >> 32, rhs >> 32); /* Now add the products together. These will never overflow. */ xxh_u64 const cross = (lo_lo >> 32) + (hi_lo & 0xFFFFFFFF) + lo_hi; xxh_u64 const upper = (hi_lo >> 32) + (cross >> 32) + hi_hi; xxh_u64 const lower = (cross << 32) | (lo_lo & 0xFFFFFFFF); XXH128_hash_t r128; r128.low64 = lower; r128.high64 = upper; return r128; #endif } /*! * @brief Calculates a 64-bit to 128-bit multiply, then XOR folds it. * * The reason for the separate function is to prevent passing too many structs * around by value. This will hopefully inline the multiply, but we don't force it. * * @param lhs , rhs The 64-bit integers to multiply * @return The low 64 bits of the product XOR'd by the high 64 bits. * @see XXH_mult64to128() */ static xxh_u64 XXH3_mul128_fold64(xxh_u64 lhs, xxh_u64 rhs) { XXH128_hash_t product = XXH_mult64to128(lhs, rhs); return product.low64 ^ product.high64; } /*! Seems to produce slightly better code on GCC for some reason. */ XXH_FORCE_INLINE XXH_CONSTF xxh_u64 XXH_xorshift64(xxh_u64 v64, int shift) { XXH_ASSERT(0 <= shift && shift < 64); return v64 ^ (v64 >> shift); } /* * This is a fast avalanche stage, * suitable when input bits are already partially mixed */ static XXH64_hash_t XXH3_avalanche(xxh_u64 h64) { h64 = XXH_xorshift64(h64, 37); h64 *= PRIME_MX1; h64 = XXH_xorshift64(h64, 32); return h64; } /* * This is a stronger avalanche, * inspired by Pelle Evensen's rrmxmx * preferable when input has not been previously mixed */ static XXH64_hash_t XXH3_rrmxmx(xxh_u64 h64, xxh_u64 len) { /* this mix is inspired by Pelle Evensen's rrmxmx */ h64 ^= XXH_rotl64(h64, 49) ^ XXH_rotl64(h64, 24); h64 *= PRIME_MX2; h64 ^= (h64 >> 35) + len ; h64 *= PRIME_MX2; return XXH_xorshift64(h64, 28); } /* ========================================== * Short keys * ========================================== * One of the shortcomings of XXH32 and XXH64 was that their performance was * sub-optimal on short lengths. It used an iterative algorithm which strongly * favored lengths that were a multiple of 4 or 8. * * Instead of iterating over individual inputs, we use a set of single shot * functions which piece together a range of lengths and operate in constant time. * * Additionally, the number of multiplies has been significantly reduced. This * reduces latency, especially when emulating 64-bit multiplies on 32-bit. * * Depending on the platform, this may or may not be faster than XXH32, but it * is almost guaranteed to be faster than XXH64. */ /* * At very short lengths, there isn't enough input to fully hide secrets, or use * the entire secret. * * There is also only a limited amount of mixing we can do before significantly * impacting performance. * * Therefore, we use different sections of the secret and always mix two secret * samples with an XOR. This should have no effect on performance on the * seedless or withSeed variants because everything _should_ be constant folded * by modern compilers. * * The XOR mixing hides individual parts of the secret and increases entropy. * * This adds an extra layer of strength for custom secrets. */ XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_1to3_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(1 <= len && len <= 3); XXH_ASSERT(secret != NULL); /* * len = 1: combined = { input[0], 0x01, input[0], input[0] } * len = 2: combined = { input[1], 0x02, input[0], input[1] } * len = 3: combined = { input[2], 0x03, input[0], input[1] } */ { xxh_u8 const c1 = input[0]; xxh_u8 const c2 = input[len >> 1]; xxh_u8 const c3 = input[len - 1]; xxh_u32 const combined = ((xxh_u32)c1 << 16) | ((xxh_u32)c2 << 24) | ((xxh_u32)c3 << 0) | ((xxh_u32)len << 8); xxh_u64 const bitflip = (XXH_readLE32(secret) ^ XXH_readLE32(secret+4)) + seed; xxh_u64 const keyed = (xxh_u64)combined ^ bitflip; return XXH64_avalanche(keyed); } } XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_4to8_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(4 <= len && len <= 8); seed ^= (xxh_u64)XXH_swap32((xxh_u32)seed) << 32; { xxh_u32 const input1 = XXH_readLE32(input); xxh_u32 const input2 = XXH_readLE32(input + len - 4); xxh_u64 const bitflip = (XXH_readLE64(secret+8) ^ XXH_readLE64(secret+16)) - seed; xxh_u64 const input64 = input2 + (((xxh_u64)input1) << 32); xxh_u64 const keyed = input64 ^ bitflip; return XXH3_rrmxmx(keyed, len); } } XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_9to16_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(9 <= len && len <= 16); { xxh_u64 const bitflip1 = (XXH_readLE64(secret+24) ^ XXH_readLE64(secret+32)) + seed; xxh_u64 const bitflip2 = (XXH_readLE64(secret+40) ^ XXH_readLE64(secret+48)) - seed; xxh_u64 const input_lo = XXH_readLE64(input) ^ bitflip1; xxh_u64 const input_hi = XXH_readLE64(input + len - 8) ^ bitflip2; xxh_u64 const acc = len + XXH_swap64(input_lo) + input_hi + XXH3_mul128_fold64(input_lo, input_hi); return XXH3_avalanche(acc); } } XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_0to16_64b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(len <= 16); { if (XXH_likely(len > 8)) return XXH3_len_9to16_64b(input, len, secret, seed); if (XXH_likely(len >= 4)) return XXH3_len_4to8_64b(input, len, secret, seed); if (len) return XXH3_len_1to3_64b(input, len, secret, seed); return XXH64_avalanche(seed ^ (XXH_readLE64(secret+56) ^ XXH_readLE64(secret+64))); } } /* * DISCLAIMER: There are known *seed-dependent* multicollisions here due to * multiplication by zero, affecting hashes of lengths 17 to 240. * * However, they are very unlikely. * * Keep this in mind when using the unseeded XXH3_64bits() variant: As with all * unseeded non-cryptographic hashes, it does not attempt to defend itself * against specially crafted inputs, only random inputs. * * Compared to classic UMAC where a 1 in 2^31 chance of 4 consecutive bytes * cancelling out the secret is taken an arbitrary number of times (addressed * in XXH3_accumulate_512), this collision is very unlikely with random inputs * and/or proper seeding: * * This only has a 1 in 2^63 chance of 8 consecutive bytes cancelling out, in a * function that is only called up to 16 times per hash with up to 240 bytes of * input. * * This is not too bad for a non-cryptographic hash function, especially with * only 64 bit outputs. * * The 128-bit variant (which trades some speed for strength) is NOT affected * by this, although it is always a good idea to use a proper seed if you care * about strength. */ XXH_FORCE_INLINE xxh_u64 XXH3_mix16B(const xxh_u8* XXH_RESTRICT input, const xxh_u8* XXH_RESTRICT secret, xxh_u64 seed64) { #if defined(__GNUC__) && !defined(__clang__) /* GCC, not Clang */ \ && defined(__i386__) && defined(__SSE2__) /* x86 + SSE2 */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable like XXH32 hack */ /* * UGLY HACK: * GCC for x86 tends to autovectorize the 128-bit multiply, resulting in * slower code. * * By forcing seed64 into a register, we disrupt the cost model and * cause it to scalarize. See `XXH32_round()` * * FIXME: Clang's output is still _much_ faster -- On an AMD Ryzen 3600, * XXH3_64bits @ len=240 runs at 4.6 GB/s with Clang 9, but 3.3 GB/s on * GCC 9.2, despite both emitting scalar code. * * GCC generates much better scalar code than Clang for the rest of XXH3, * which is why finding a more optimal codepath is an interest. */ XXH_COMPILER_GUARD(seed64); #endif { xxh_u64 const input_lo = XXH_readLE64(input); xxh_u64 const input_hi = XXH_readLE64(input+8); return XXH3_mul128_fold64( input_lo ^ (XXH_readLE64(secret) + seed64), input_hi ^ (XXH_readLE64(secret+8) - seed64) ); } } /* For mid range keys, XXH3 uses a Mum-hash variant. */ XXH_FORCE_INLINE XXH_PUREF XXH64_hash_t XXH3_len_17to128_64b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(16 < len && len <= 128); { xxh_u64 acc = len * XXH_PRIME64_1; #if XXH_SIZE_OPT >= 1 /* Smaller and cleaner, but slightly slower. */ unsigned int i = (unsigned int)(len - 1) / 32; do { acc += XXH3_mix16B(input+16 * i, secret+32*i, seed); acc += XXH3_mix16B(input+len-16*(i+1), secret+32*i+16, seed); } while (i-- != 0); #else if (len > 32) { if (len > 64) { if (len > 96) { acc += XXH3_mix16B(input+48, secret+96, seed); acc += XXH3_mix16B(input+len-64, secret+112, seed); } acc += XXH3_mix16B(input+32, secret+64, seed); acc += XXH3_mix16B(input+len-48, secret+80, seed); } acc += XXH3_mix16B(input+16, secret+32, seed); acc += XXH3_mix16B(input+len-32, secret+48, seed); } acc += XXH3_mix16B(input+0, secret+0, seed); acc += XXH3_mix16B(input+len-16, secret+16, seed); #endif return XXH3_avalanche(acc); } } #define XXH3_MIDSIZE_MAX 240 XXH_NO_INLINE XXH_PUREF XXH64_hash_t XXH3_len_129to240_64b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); #define XXH3_MIDSIZE_STARTOFFSET 3 #define XXH3_MIDSIZE_LASTOFFSET 17 { xxh_u64 acc = len * XXH_PRIME64_1; xxh_u64 acc_end; unsigned int const nbRounds = (unsigned int)len / 16; unsigned int i; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); for (i=0; i<8; i++) { acc += XXH3_mix16B(input+(16*i), secret+(16*i), seed); } /* last bytes */ acc_end = XXH3_mix16B(input + len - 16, secret + XXH3_SECRET_SIZE_MIN - XXH3_MIDSIZE_LASTOFFSET, seed); XXH_ASSERT(nbRounds >= 8); acc = XXH3_avalanche(acc); #if defined(__clang__) /* Clang */ \ && (defined(__ARM_NEON) || defined(__ARM_NEON__)) /* NEON */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable */ /* * UGLY HACK: * Clang for ARMv7-A tries to vectorize this loop, similar to GCC x86. * In everywhere else, it uses scalar code. * * For 64->128-bit multiplies, even if the NEON was 100% optimal, it * would still be slower than UMAAL (see XXH_mult64to128). * * Unfortunately, Clang doesn't handle the long multiplies properly and * converts them to the nonexistent "vmulq_u64" intrinsic, which is then * scalarized into an ugly mess of VMOV.32 instructions. * * This mess is difficult to avoid without turning autovectorization * off completely, but they are usually relatively minor and/or not * worth it to fix. * * This loop is the easiest to fix, as unlike XXH32, this pragma * _actually works_ because it is a loop vectorization instead of an * SLP vectorization. */ #pragma clang loop vectorize(disable) #endif for (i=8 ; i < nbRounds; i++) { /* * Prevents clang for unrolling the acc loop and interleaving with this one. */ XXH_COMPILER_GUARD(acc); acc_end += XXH3_mix16B(input+(16*i), secret+(16*(i-8)) + XXH3_MIDSIZE_STARTOFFSET, seed); } return XXH3_avalanche(acc + acc_end); } } /* ======= Long Keys ======= */ #define XXH_STRIPE_LEN 64 #define XXH_SECRET_CONSUME_RATE 8 /* nb of secret bytes consumed at each accumulation */ #define XXH_ACC_NB (XXH_STRIPE_LEN / sizeof(xxh_u64)) #ifdef XXH_OLD_NAMES # define STRIPE_LEN XXH_STRIPE_LEN # define ACC_NB XXH_ACC_NB #endif #ifndef XXH_PREFETCH_DIST # ifdef __clang__ # define XXH_PREFETCH_DIST 320 # else # if (XXH_VECTOR == XXH_AVX512) # define XXH_PREFETCH_DIST 512 # else # define XXH_PREFETCH_DIST 384 # endif # endif /* __clang__ */ #endif /* XXH_PREFETCH_DIST */ /* * These macros are to generate an XXH3_accumulate() function. * The two arguments select the name suffix and target attribute. * * The name of this symbol is XXH3_accumulate_() and it calls * XXH3_accumulate_512_(). * * It may be useful to hand implement this function if the compiler fails to * optimize the inline function. */ #define XXH3_ACCUMULATE_TEMPLATE(name) \ void \ XXH3_accumulate_##name(xxh_u64* XXH_RESTRICT acc, \ const xxh_u8* XXH_RESTRICT input, \ const xxh_u8* XXH_RESTRICT secret, \ size_t nbStripes) \ { \ size_t n; \ for (n = 0; n < nbStripes; n++ ) { \ const xxh_u8* const in = input + n*XXH_STRIPE_LEN; \ XXH_PREFETCH(in + XXH_PREFETCH_DIST); \ XXH3_accumulate_512_##name( \ acc, \ in, \ secret + n*XXH_SECRET_CONSUME_RATE); \ } \ } XXH_FORCE_INLINE void XXH_writeLE64(void* dst, xxh_u64 v64) { if (!XXH_CPU_LITTLE_ENDIAN) v64 = XXH_swap64(v64); XXH_memcpy(dst, &v64, sizeof(v64)); } /* Several intrinsic functions below are supposed to accept __int64 as argument, * as documented in https://software.intel.com/sites/landingpage/IntrinsicsGuide/ . * However, several environments do not define __int64 type, * requiring a workaround. */ #if !defined (__VMS) \ && (defined (__cplusplus) \ || (defined (__STDC_VERSION__) && (__STDC_VERSION__ >= 199901L) /* C99 */) ) typedef int64_t xxh_i64; #else /* the following type must have a width of 64-bit */ typedef long long xxh_i64; #endif /* * XXH3_accumulate_512 is the tightest loop for long inputs, and it is the most optimized. * * It is a hardened version of UMAC, based off of FARSH's implementation. * * This was chosen because it adapts quite well to 32-bit, 64-bit, and SIMD * implementations, and it is ridiculously fast. * * We harden it by mixing the original input to the accumulators as well as the product. * * This means that in the (relatively likely) case of a multiply by zero, the * original input is preserved. * * On 128-bit inputs, we swap 64-bit pairs when we add the input to improve * cross-pollination, as otherwise the upper and lower halves would be * essentially independent. * * This doesn't matter on 64-bit hashes since they all get merged together in * the end, so we skip the extra step. * * Both XXH3_64bits and XXH3_128bits use this subroutine. */ #if (XXH_VECTOR == XXH_AVX512) \ || (defined(XXH_DISPATCH_AVX512) && XXH_DISPATCH_AVX512 != 0) #ifndef XXH_TARGET_AVX512 # define XXH_TARGET_AVX512 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_accumulate_512_avx512(void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { __m512i* const xacc = (__m512i *) acc; XXH_ASSERT((((size_t)acc) & 63) == 0); XXH_STATIC_ASSERT(XXH_STRIPE_LEN == sizeof(__m512i)); { /* data_vec = input[0]; */ __m512i const data_vec = _mm512_loadu_si512 (input); /* key_vec = secret[0]; */ __m512i const key_vec = _mm512_loadu_si512 (secret); /* data_key = data_vec ^ key_vec; */ __m512i const data_key = _mm512_xor_si512 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m512i const data_key_lo = _mm512_srli_epi64 (data_key, 32); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m512i const product = _mm512_mul_epu32 (data_key, data_key_lo); /* xacc[0] += swap(data_vec); */ __m512i const data_swap = _mm512_shuffle_epi32(data_vec, (_MM_PERM_ENUM)_MM_SHUFFLE(1, 0, 3, 2)); __m512i const sum = _mm512_add_epi64(*xacc, data_swap); /* xacc[0] += product; */ *xacc = _mm512_add_epi64(product, sum); } } XXH_FORCE_INLINE XXH_TARGET_AVX512 XXH3_ACCUMULATE_TEMPLATE(avx512) /* * XXH3_scrambleAcc: Scrambles the accumulators to improve mixing. * * Multiplication isn't perfect, as explained by Google in HighwayHash: * * // Multiplication mixes/scrambles bytes 0-7 of the 64-bit result to * // varying degrees. In descending order of goodness, bytes * // 3 4 2 5 1 6 0 7 have quality 228 224 164 160 100 96 36 32. * // As expected, the upper and lower bytes are much worse. * * Source: https://github.com/google/highwayhash/blob/0aaf66b/highwayhash/hh_avx2.h#L291 * * Since our algorithm uses a pseudorandom secret to add some variance into the * mix, we don't need to (or want to) mix as often or as much as HighwayHash does. * * This isn't as tight as XXH3_accumulate, but still written in SIMD to avoid * extraction. * * Both XXH3_64bits and XXH3_128bits use this subroutine. */ XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_scrambleAcc_avx512(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 63) == 0); XXH_STATIC_ASSERT(XXH_STRIPE_LEN == sizeof(__m512i)); { __m512i* const xacc = (__m512i*) acc; const __m512i prime32 = _mm512_set1_epi32((int)XXH_PRIME32_1); /* xacc[0] ^= (xacc[0] >> 47) */ __m512i const acc_vec = *xacc; __m512i const shifted = _mm512_srli_epi64 (acc_vec, 47); /* xacc[0] ^= secret; */ __m512i const key_vec = _mm512_loadu_si512 (secret); __m512i const data_key = _mm512_ternarylogic_epi32(key_vec, acc_vec, shifted, 0x96 /* key_vec ^ acc_vec ^ shifted */); /* xacc[0] *= XXH_PRIME32_1; */ __m512i const data_key_hi = _mm512_srli_epi64 (data_key, 32); __m512i const prod_lo = _mm512_mul_epu32 (data_key, prime32); __m512i const prod_hi = _mm512_mul_epu32 (data_key_hi, prime32); *xacc = _mm512_add_epi64(prod_lo, _mm512_slli_epi64(prod_hi, 32)); } } XXH_FORCE_INLINE XXH_TARGET_AVX512 void XXH3_initCustomSecret_avx512(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 63) == 0); XXH_STATIC_ASSERT(XXH_SEC_ALIGN == 64); XXH_ASSERT(((size_t)customSecret & 63) == 0); (void)(&XXH_writeLE64); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / sizeof(__m512i); __m512i const seed_pos = _mm512_set1_epi64((xxh_i64)seed64); __m512i const seed = _mm512_mask_sub_epi64(seed_pos, 0xAA, _mm512_set1_epi8(0), seed_pos); const __m512i* const src = (const __m512i*) ((const void*) XXH3_kSecret); __m512i* const dest = ( __m512i*) customSecret; int i; XXH_ASSERT(((size_t)src & 63) == 0); /* control alignment */ XXH_ASSERT(((size_t)dest & 63) == 0); for (i=0; i < nbRounds; ++i) { dest[i] = _mm512_add_epi64(_mm512_load_si512(src + i), seed); } } } #endif #if (XXH_VECTOR == XXH_AVX2) \ || (defined(XXH_DISPATCH_AVX2) && XXH_DISPATCH_AVX2 != 0) #ifndef XXH_TARGET_AVX2 # define XXH_TARGET_AVX2 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_accumulate_512_avx2( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 31) == 0); { __m256i* const xacc = (__m256i *) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xinput = (const __m256i *) input; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xsecret = (const __m256i *) secret; size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m256i); i++) { /* data_vec = xinput[i]; */ __m256i const data_vec = _mm256_loadu_si256 (xinput+i); /* key_vec = xsecret[i]; */ __m256i const key_vec = _mm256_loadu_si256 (xsecret+i); /* data_key = data_vec ^ key_vec; */ __m256i const data_key = _mm256_xor_si256 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m256i const data_key_lo = _mm256_srli_epi64 (data_key, 32); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m256i const product = _mm256_mul_epu32 (data_key, data_key_lo); /* xacc[i] += swap(data_vec); */ __m256i const data_swap = _mm256_shuffle_epi32(data_vec, _MM_SHUFFLE(1, 0, 3, 2)); __m256i const sum = _mm256_add_epi64(xacc[i], data_swap); /* xacc[i] += product; */ xacc[i] = _mm256_add_epi64(product, sum); } } } XXH_FORCE_INLINE XXH_TARGET_AVX2 XXH3_ACCUMULATE_TEMPLATE(avx2) XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_scrambleAcc_avx2(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 31) == 0); { __m256i* const xacc = (__m256i*) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm256_loadu_si256 requires a const __m256i * pointer for some reason. */ const __m256i* const xsecret = (const __m256i *) secret; const __m256i prime32 = _mm256_set1_epi32((int)XXH_PRIME32_1); size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m256i); i++) { /* xacc[i] ^= (xacc[i] >> 47) */ __m256i const acc_vec = xacc[i]; __m256i const shifted = _mm256_srli_epi64 (acc_vec, 47); __m256i const data_vec = _mm256_xor_si256 (acc_vec, shifted); /* xacc[i] ^= xsecret; */ __m256i const key_vec = _mm256_loadu_si256 (xsecret+i); __m256i const data_key = _mm256_xor_si256 (data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1; */ __m256i const data_key_hi = _mm256_srli_epi64 (data_key, 32); __m256i const prod_lo = _mm256_mul_epu32 (data_key, prime32); __m256i const prod_hi = _mm256_mul_epu32 (data_key_hi, prime32); xacc[i] = _mm256_add_epi64(prod_lo, _mm256_slli_epi64(prod_hi, 32)); } } } XXH_FORCE_INLINE XXH_TARGET_AVX2 void XXH3_initCustomSecret_avx2(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 31) == 0); XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE / sizeof(__m256i)) == 6); XXH_STATIC_ASSERT(XXH_SEC_ALIGN <= 64); (void)(&XXH_writeLE64); XXH_PREFETCH(customSecret); { __m256i const seed = _mm256_set_epi64x((xxh_i64)(0U - seed64), (xxh_i64)seed64, (xxh_i64)(0U - seed64), (xxh_i64)seed64); const __m256i* const src = (const __m256i*) ((const void*) XXH3_kSecret); __m256i* dest = ( __m256i*) customSecret; # if defined(__GNUC__) || defined(__clang__) /* * On GCC & Clang, marking 'dest' as modified will cause the compiler: * - do not extract the secret from sse registers in the internal loop * - use less common registers, and avoid pushing these reg into stack */ XXH_COMPILER_GUARD(dest); # endif XXH_ASSERT(((size_t)src & 31) == 0); /* control alignment */ XXH_ASSERT(((size_t)dest & 31) == 0); /* GCC -O2 need unroll loop manually */ dest[0] = _mm256_add_epi64(_mm256_load_si256(src+0), seed); dest[1] = _mm256_add_epi64(_mm256_load_si256(src+1), seed); dest[2] = _mm256_add_epi64(_mm256_load_si256(src+2), seed); dest[3] = _mm256_add_epi64(_mm256_load_si256(src+3), seed); dest[4] = _mm256_add_epi64(_mm256_load_si256(src+4), seed); dest[5] = _mm256_add_epi64(_mm256_load_si256(src+5), seed); } } #endif /* x86dispatch always generates SSE2 */ #if (XXH_VECTOR == XXH_SSE2) || defined(XXH_X86DISPATCH) #ifndef XXH_TARGET_SSE2 # define XXH_TARGET_SSE2 /* disable attribute target */ #endif XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_accumulate_512_sse2( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { /* SSE2 is just a half-scale version of the AVX2 version. */ XXH_ASSERT((((size_t)acc) & 15) == 0); { __m128i* const xacc = (__m128i *) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xinput = (const __m128i *) input; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xsecret = (const __m128i *) secret; size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m128i); i++) { /* data_vec = xinput[i]; */ __m128i const data_vec = _mm_loadu_si128 (xinput+i); /* key_vec = xsecret[i]; */ __m128i const key_vec = _mm_loadu_si128 (xsecret+i); /* data_key = data_vec ^ key_vec; */ __m128i const data_key = _mm_xor_si128 (data_vec, key_vec); /* data_key_lo = data_key >> 32; */ __m128i const data_key_lo = _mm_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); /* product = (data_key & 0xffffffff) * (data_key_lo & 0xffffffff); */ __m128i const product = _mm_mul_epu32 (data_key, data_key_lo); /* xacc[i] += swap(data_vec); */ __m128i const data_swap = _mm_shuffle_epi32(data_vec, _MM_SHUFFLE(1,0,3,2)); __m128i const sum = _mm_add_epi64(xacc[i], data_swap); /* xacc[i] += product; */ xacc[i] = _mm_add_epi64(product, sum); } } } XXH_FORCE_INLINE XXH_TARGET_SSE2 XXH3_ACCUMULATE_TEMPLATE(sse2) XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_scrambleAcc_sse2(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { __m128i* const xacc = (__m128i*) acc; /* Unaligned. This is mainly for pointer arithmetic, and because * _mm_loadu_si128 requires a const __m128i * pointer for some reason. */ const __m128i* const xsecret = (const __m128i *) secret; const __m128i prime32 = _mm_set1_epi32((int)XXH_PRIME32_1); size_t i; for (i=0; i < XXH_STRIPE_LEN/sizeof(__m128i); i++) { /* xacc[i] ^= (xacc[i] >> 47) */ __m128i const acc_vec = xacc[i]; __m128i const shifted = _mm_srli_epi64 (acc_vec, 47); __m128i const data_vec = _mm_xor_si128 (acc_vec, shifted); /* xacc[i] ^= xsecret[i]; */ __m128i const key_vec = _mm_loadu_si128 (xsecret+i); __m128i const data_key = _mm_xor_si128 (data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1; */ __m128i const data_key_hi = _mm_shuffle_epi32 (data_key, _MM_SHUFFLE(0, 3, 0, 1)); __m128i const prod_lo = _mm_mul_epu32 (data_key, prime32); __m128i const prod_hi = _mm_mul_epu32 (data_key_hi, prime32); xacc[i] = _mm_add_epi64(prod_lo, _mm_slli_epi64(prod_hi, 32)); } } } XXH_FORCE_INLINE XXH_TARGET_SSE2 void XXH3_initCustomSecret_sse2(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 15) == 0); (void)(&XXH_writeLE64); { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / sizeof(__m128i); # if defined(_MSC_VER) && defined(_M_IX86) && _MSC_VER < 1900 /* MSVC 32bit mode does not support _mm_set_epi64x before 2015 */ XXH_ALIGN(16) const xxh_i64 seed64x2[2] = { (xxh_i64)seed64, (xxh_i64)(0U - seed64) }; __m128i const seed = _mm_load_si128((__m128i const*)seed64x2); # else __m128i const seed = _mm_set_epi64x((xxh_i64)(0U - seed64), (xxh_i64)seed64); # endif int i; const void* const src16 = XXH3_kSecret; __m128i* dst16 = (__m128i*) customSecret; # if defined(__GNUC__) || defined(__clang__) /* * On GCC & Clang, marking 'dest' as modified will cause the compiler: * - do not extract the secret from sse registers in the internal loop * - use less common registers, and avoid pushing these reg into stack */ XXH_COMPILER_GUARD(dst16); # endif XXH_ASSERT(((size_t)src16 & 15) == 0); /* control alignment */ XXH_ASSERT(((size_t)dst16 & 15) == 0); for (i=0; i < nbRounds; ++i) { dst16[i] = _mm_add_epi64(_mm_load_si128((const __m128i *)src16+i), seed); } } } #endif #if (XXH_VECTOR == XXH_NEON) /* forward declarations for the scalar routines */ XXH_FORCE_INLINE void XXH3_scalarRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT input, void const* XXH_RESTRICT secret, size_t lane); XXH_FORCE_INLINE void XXH3_scalarScrambleRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT secret, size_t lane); /*! * @internal * @brief The bulk processing loop for NEON and WASM SIMD128. * * The NEON code path is actually partially scalar when running on AArch64. This * is to optimize the pipelining and can have up to 15% speedup depending on the * CPU, and it also mitigates some GCC codegen issues. * * @see XXH3_NEON_LANES for configuring this and details about this optimization. * * NEON's 32-bit to 64-bit long multiply takes a half vector of 32-bit * integers instead of the other platforms which mask full 64-bit vectors, * so the setup is more complicated than just shifting right. * * Additionally, there is an optimization for 4 lanes at once noted below. * * Since, as stated, the most optimal amount of lanes for Cortexes is 6, * there needs to be *three* versions of the accumulate operation used * for the remaining 2 lanes. * * WASM's SIMD128 uses SIMDe's arm_neon.h polyfill because the intrinsics overlap * nearly perfectly. */ XXH_FORCE_INLINE void XXH3_accumulate_512_neon( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); XXH_STATIC_ASSERT(XXH3_NEON_LANES > 0 && XXH3_NEON_LANES <= XXH_ACC_NB && XXH3_NEON_LANES % 2 == 0); { /* GCC for darwin arm64 does not like aliasing here */ xxh_aliasing_uint64x2_t* const xacc = (xxh_aliasing_uint64x2_t*) acc; /* We don't use a uint32x4_t pointer because it causes bus errors on ARMv7. */ uint8_t const* xinput = (const uint8_t *) input; uint8_t const* xsecret = (const uint8_t *) secret; size_t i; #ifdef __wasm_simd128__ /* * On WASM SIMD128, Clang emits direct address loads when XXH3_kSecret * is constant propagated, which results in it converting it to this * inside the loop: * * a = v128.load(XXH3_kSecret + 0 + $secret_offset, offset = 0) * b = v128.load(XXH3_kSecret + 16 + $secret_offset, offset = 0) * ... * * This requires a full 32-bit address immediate (and therefore a 6 byte * instruction) as well as an add for each offset. * * Putting an asm guard prevents it from folding (at the cost of losing * the alignment hint), and uses the free offset in `v128.load` instead * of adding secret_offset each time which overall reduces code size by * about a kilobyte and improves performance. */ XXH_COMPILER_GUARD(xsecret); #endif /* Scalar lanes use the normal scalarRound routine */ for (i = XXH3_NEON_LANES; i < XXH_ACC_NB; i++) { XXH3_scalarRound(acc, input, secret, i); } i = 0; /* 4 NEON lanes at a time. */ for (; i+1 < XXH3_NEON_LANES / 2; i+=2) { /* data_vec = xinput[i]; */ uint64x2_t data_vec_1 = XXH_vld1q_u64(xinput + (i * 16)); uint64x2_t data_vec_2 = XXH_vld1q_u64(xinput + ((i+1) * 16)); /* key_vec = xsecret[i]; */ uint64x2_t key_vec_1 = XXH_vld1q_u64(xsecret + (i * 16)); uint64x2_t key_vec_2 = XXH_vld1q_u64(xsecret + ((i+1) * 16)); /* data_swap = swap(data_vec) */ uint64x2_t data_swap_1 = vextq_u64(data_vec_1, data_vec_1, 1); uint64x2_t data_swap_2 = vextq_u64(data_vec_2, data_vec_2, 1); /* data_key = data_vec ^ key_vec; */ uint64x2_t data_key_1 = veorq_u64(data_vec_1, key_vec_1); uint64x2_t data_key_2 = veorq_u64(data_vec_2, key_vec_2); /* * If we reinterpret the 64x2 vectors as 32x4 vectors, we can use a * de-interleave operation for 4 lanes in 1 step with `vuzpq_u32` to * get one vector with the low 32 bits of each lane, and one vector * with the high 32 bits of each lane. * * The intrinsic returns a double vector because the original ARMv7-a * instruction modified both arguments in place. AArch64 and SIMD128 emit * two instructions from this intrinsic. * * [ dk11L | dk11H | dk12L | dk12H ] -> [ dk11L | dk12L | dk21L | dk22L ] * [ dk21L | dk21H | dk22L | dk22H ] -> [ dk11H | dk12H | dk21H | dk22H ] */ uint32x4x2_t unzipped = vuzpq_u32( vreinterpretq_u32_u64(data_key_1), vreinterpretq_u32_u64(data_key_2) ); /* data_key_lo = data_key & 0xFFFFFFFF */ uint32x4_t data_key_lo = unzipped.val[0]; /* data_key_hi = data_key >> 32 */ uint32x4_t data_key_hi = unzipped.val[1]; /* * Then, we can split the vectors horizontally and multiply which, as for most * widening intrinsics, have a variant that works on both high half vectors * for free on AArch64. A similar instruction is available on SIMD128. * * sum = data_swap + (u64x2) data_key_lo * (u64x2) data_key_hi */ uint64x2_t sum_1 = XXH_vmlal_low_u32(data_swap_1, data_key_lo, data_key_hi); uint64x2_t sum_2 = XXH_vmlal_high_u32(data_swap_2, data_key_lo, data_key_hi); /* * Clang reorders * a += b * c; // umlal swap.2d, dkl.2s, dkh.2s * c += a; // add acc.2d, acc.2d, swap.2d * to * c += a; // add acc.2d, acc.2d, swap.2d * c += b * c; // umlal acc.2d, dkl.2s, dkh.2s * * While it would make sense in theory since the addition is faster, * for reasons likely related to umlal being limited to certain NEON * pipelines, this is worse. A compiler guard fixes this. */ XXH_COMPILER_GUARD_CLANG_NEON(sum_1); XXH_COMPILER_GUARD_CLANG_NEON(sum_2); /* xacc[i] = acc_vec + sum; */ xacc[i] = vaddq_u64(xacc[i], sum_1); xacc[i+1] = vaddq_u64(xacc[i+1], sum_2); } /* Operate on the remaining NEON lanes 2 at a time. */ for (; i < XXH3_NEON_LANES / 2; i++) { /* data_vec = xinput[i]; */ uint64x2_t data_vec = XXH_vld1q_u64(xinput + (i * 16)); /* key_vec = xsecret[i]; */ uint64x2_t key_vec = XXH_vld1q_u64(xsecret + (i * 16)); /* acc_vec_2 = swap(data_vec) */ uint64x2_t data_swap = vextq_u64(data_vec, data_vec, 1); /* data_key = data_vec ^ key_vec; */ uint64x2_t data_key = veorq_u64(data_vec, key_vec); /* For two lanes, just use VMOVN and VSHRN. */ /* data_key_lo = data_key & 0xFFFFFFFF; */ uint32x2_t data_key_lo = vmovn_u64(data_key); /* data_key_hi = data_key >> 32; */ uint32x2_t data_key_hi = vshrn_n_u64(data_key, 32); /* sum = data_swap + (u64x2) data_key_lo * (u64x2) data_key_hi; */ uint64x2_t sum = vmlal_u32(data_swap, data_key_lo, data_key_hi); /* Same Clang workaround as before */ XXH_COMPILER_GUARD_CLANG_NEON(sum); /* xacc[i] = acc_vec + sum; */ xacc[i] = vaddq_u64 (xacc[i], sum); } } } XXH_FORCE_INLINE XXH3_ACCUMULATE_TEMPLATE(neon) XXH_FORCE_INLINE void XXH3_scrambleAcc_neon(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { xxh_aliasing_uint64x2_t* xacc = (xxh_aliasing_uint64x2_t*) acc; uint8_t const* xsecret = (uint8_t const*) secret; size_t i; /* WASM uses operator overloads and doesn't need these. */ #ifndef __wasm_simd128__ /* { prime32_1, prime32_1 } */ uint32x2_t const kPrimeLo = vdup_n_u32(XXH_PRIME32_1); /* { 0, prime32_1, 0, prime32_1 } */ uint32x4_t const kPrimeHi = vreinterpretq_u32_u64(vdupq_n_u64((xxh_u64)XXH_PRIME32_1 << 32)); #endif /* AArch64 uses both scalar and neon at the same time */ for (i = XXH3_NEON_LANES; i < XXH_ACC_NB; i++) { XXH3_scalarScrambleRound(acc, secret, i); } for (i=0; i < XXH3_NEON_LANES / 2; i++) { /* xacc[i] ^= (xacc[i] >> 47); */ uint64x2_t acc_vec = xacc[i]; uint64x2_t shifted = vshrq_n_u64(acc_vec, 47); uint64x2_t data_vec = veorq_u64(acc_vec, shifted); /* xacc[i] ^= xsecret[i]; */ uint64x2_t key_vec = XXH_vld1q_u64(xsecret + (i * 16)); uint64x2_t data_key = veorq_u64(data_vec, key_vec); /* xacc[i] *= XXH_PRIME32_1 */ #ifdef __wasm_simd128__ /* SIMD128 has multiply by u64x2, use it instead of expanding and scalarizing */ xacc[i] = data_key * XXH_PRIME32_1; #else /* * Expanded version with portable NEON intrinsics * * lo(x) * lo(y) + (hi(x) * lo(y) << 32) * * prod_hi = hi(data_key) * lo(prime) << 32 * * Since we only need 32 bits of this multiply a trick can be used, reinterpreting the vector * as a uint32x4_t and multiplying by { 0, prime, 0, prime } to cancel out the unwanted bits * and avoid the shift. */ uint32x4_t prod_hi = vmulq_u32 (vreinterpretq_u32_u64(data_key), kPrimeHi); /* Extract low bits for vmlal_u32 */ uint32x2_t data_key_lo = vmovn_u64(data_key); /* xacc[i] = prod_hi + lo(data_key) * XXH_PRIME32_1; */ xacc[i] = vmlal_u32(vreinterpretq_u64_u32(prod_hi), data_key_lo, kPrimeLo); #endif } } } #endif #if (XXH_VECTOR == XXH_VSX) XXH_FORCE_INLINE void XXH3_accumulate_512_vsx( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { /* presumed aligned */ xxh_aliasing_u64x2* const xacc = (xxh_aliasing_u64x2*) acc; xxh_u8 const* const xinput = (xxh_u8 const*) input; /* no alignment restriction */ xxh_u8 const* const xsecret = (xxh_u8 const*) secret; /* no alignment restriction */ xxh_u64x2 const v32 = { 32, 32 }; size_t i; for (i = 0; i < XXH_STRIPE_LEN / sizeof(xxh_u64x2); i++) { /* data_vec = xinput[i]; */ xxh_u64x2 const data_vec = XXH_vec_loadu(xinput + 16*i); /* key_vec = xsecret[i]; */ xxh_u64x2 const key_vec = XXH_vec_loadu(xsecret + 16*i); xxh_u64x2 const data_key = data_vec ^ key_vec; /* shuffled = (data_key << 32) | (data_key >> 32); */ xxh_u32x4 const shuffled = (xxh_u32x4)vec_rl(data_key, v32); /* product = ((xxh_u64x2)data_key & 0xFFFFFFFF) * ((xxh_u64x2)shuffled & 0xFFFFFFFF); */ xxh_u64x2 const product = XXH_vec_mulo((xxh_u32x4)data_key, shuffled); /* acc_vec = xacc[i]; */ xxh_u64x2 acc_vec = xacc[i]; acc_vec += product; /* swap high and low halves */ #ifdef __s390x__ acc_vec += vec_permi(data_vec, data_vec, 2); #else acc_vec += vec_xxpermdi(data_vec, data_vec, 2); #endif xacc[i] = acc_vec; } } XXH_FORCE_INLINE XXH3_ACCUMULATE_TEMPLATE(vsx) XXH_FORCE_INLINE void XXH3_scrambleAcc_vsx(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { XXH_ASSERT((((size_t)acc) & 15) == 0); { xxh_aliasing_u64x2* const xacc = (xxh_aliasing_u64x2*) acc; const xxh_u8* const xsecret = (const xxh_u8*) secret; /* constants */ xxh_u64x2 const v32 = { 32, 32 }; xxh_u64x2 const v47 = { 47, 47 }; xxh_u32x4 const prime = { XXH_PRIME32_1, XXH_PRIME32_1, XXH_PRIME32_1, XXH_PRIME32_1 }; size_t i; for (i = 0; i < XXH_STRIPE_LEN / sizeof(xxh_u64x2); i++) { /* xacc[i] ^= (xacc[i] >> 47); */ xxh_u64x2 const acc_vec = xacc[i]; xxh_u64x2 const data_vec = acc_vec ^ (acc_vec >> v47); /* xacc[i] ^= xsecret[i]; */ xxh_u64x2 const key_vec = XXH_vec_loadu(xsecret + 16*i); xxh_u64x2 const data_key = data_vec ^ key_vec; /* xacc[i] *= XXH_PRIME32_1 */ /* prod_lo = ((xxh_u64x2)data_key & 0xFFFFFFFF) * ((xxh_u64x2)prime & 0xFFFFFFFF); */ xxh_u64x2 const prod_even = XXH_vec_mule((xxh_u32x4)data_key, prime); /* prod_hi = ((xxh_u64x2)data_key >> 32) * ((xxh_u64x2)prime >> 32); */ xxh_u64x2 const prod_odd = XXH_vec_mulo((xxh_u32x4)data_key, prime); xacc[i] = prod_odd + (prod_even << v32); } } } #endif #if (XXH_VECTOR == XXH_SVE) XXH_FORCE_INLINE void XXH3_accumulate_512_sve( void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { uint64_t *xacc = (uint64_t *)acc; const uint64_t *xinput = (const uint64_t *)(const void *)input; const uint64_t *xsecret = (const uint64_t *)(const void *)secret; svuint64_t kSwap = sveor_n_u64_z(svptrue_b64(), svindex_u64(0, 1), 1); uint64_t element_count = svcntd(); if (element_count >= 8) { svbool_t mask = svptrue_pat_b64(SV_VL8); svuint64_t vacc = svld1_u64(mask, xacc); ACCRND(vacc, 0); svst1_u64(mask, xacc, vacc); } else if (element_count == 2) { /* sve128 */ svbool_t mask = svptrue_pat_b64(SV_VL2); svuint64_t acc0 = svld1_u64(mask, xacc + 0); svuint64_t acc1 = svld1_u64(mask, xacc + 2); svuint64_t acc2 = svld1_u64(mask, xacc + 4); svuint64_t acc3 = svld1_u64(mask, xacc + 6); ACCRND(acc0, 0); ACCRND(acc1, 2); ACCRND(acc2, 4); ACCRND(acc3, 6); svst1_u64(mask, xacc + 0, acc0); svst1_u64(mask, xacc + 2, acc1); svst1_u64(mask, xacc + 4, acc2); svst1_u64(mask, xacc + 6, acc3); } else { svbool_t mask = svptrue_pat_b64(SV_VL4); svuint64_t acc0 = svld1_u64(mask, xacc + 0); svuint64_t acc1 = svld1_u64(mask, xacc + 4); ACCRND(acc0, 0); ACCRND(acc1, 4); svst1_u64(mask, xacc + 0, acc0); svst1_u64(mask, xacc + 4, acc1); } } XXH_FORCE_INLINE void XXH3_accumulate_sve(xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT input, const xxh_u8* XXH_RESTRICT secret, size_t nbStripes) { if (nbStripes != 0) { uint64_t *xacc = (uint64_t *)acc; const uint64_t *xinput = (const uint64_t *)(const void *)input; const uint64_t *xsecret = (const uint64_t *)(const void *)secret; svuint64_t kSwap = sveor_n_u64_z(svptrue_b64(), svindex_u64(0, 1), 1); uint64_t element_count = svcntd(); if (element_count >= 8) { svbool_t mask = svptrue_pat_b64(SV_VL8); svuint64_t vacc = svld1_u64(mask, xacc + 0); do { /* svprfd(svbool_t, void *, enum svfprop); */ svprfd(mask, xinput + 128, SV_PLDL1STRM); ACCRND(vacc, 0); xinput += 8; xsecret += 1; nbStripes--; } while (nbStripes != 0); svst1_u64(mask, xacc + 0, vacc); } else if (element_count == 2) { /* sve128 */ svbool_t mask = svptrue_pat_b64(SV_VL2); svuint64_t acc0 = svld1_u64(mask, xacc + 0); svuint64_t acc1 = svld1_u64(mask, xacc + 2); svuint64_t acc2 = svld1_u64(mask, xacc + 4); svuint64_t acc3 = svld1_u64(mask, xacc + 6); do { svprfd(mask, xinput + 128, SV_PLDL1STRM); ACCRND(acc0, 0); ACCRND(acc1, 2); ACCRND(acc2, 4); ACCRND(acc3, 6); xinput += 8; xsecret += 1; nbStripes--; } while (nbStripes != 0); svst1_u64(mask, xacc + 0, acc0); svst1_u64(mask, xacc + 2, acc1); svst1_u64(mask, xacc + 4, acc2); svst1_u64(mask, xacc + 6, acc3); } else { svbool_t mask = svptrue_pat_b64(SV_VL4); svuint64_t acc0 = svld1_u64(mask, xacc + 0); svuint64_t acc1 = svld1_u64(mask, xacc + 4); do { svprfd(mask, xinput + 128, SV_PLDL1STRM); ACCRND(acc0, 0); ACCRND(acc1, 4); xinput += 8; xsecret += 1; nbStripes--; } while (nbStripes != 0); svst1_u64(mask, xacc + 0, acc0); svst1_u64(mask, xacc + 4, acc1); } } } #endif /* scalar variants - universal */ #if defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) /* * In XXH3_scalarRound(), GCC and Clang have a similar codegen issue, where they * emit an excess mask and a full 64-bit multiply-add (MADD X-form). * * While this might not seem like much, as AArch64 is a 64-bit architecture, only * big Cortex designs have a full 64-bit multiplier. * * On the little cores, the smaller 32-bit multiplier is used, and full 64-bit * multiplies expand to 2-3 multiplies in microcode. This has a major penalty * of up to 4 latency cycles and 2 stall cycles in the multiply pipeline. * * Thankfully, AArch64 still provides the 32-bit long multiply-add (UMADDL) which does * not have this penalty and does the mask automatically. */ XXH_FORCE_INLINE xxh_u64 XXH_mult32to64_add64(xxh_u64 lhs, xxh_u64 rhs, xxh_u64 acc) { xxh_u64 ret; /* note: %x = 64-bit register, %w = 32-bit register */ __asm__("umaddl %x0, %w1, %w2, %x3" : "=r" (ret) : "r" (lhs), "r" (rhs), "r" (acc)); return ret; } #else XXH_FORCE_INLINE xxh_u64 XXH_mult32to64_add64(xxh_u64 lhs, xxh_u64 rhs, xxh_u64 acc) { return XXH_mult32to64((xxh_u32)lhs, (xxh_u32)rhs) + acc; } #endif /*! * @internal * @brief Scalar round for @ref XXH3_accumulate_512_scalar(). * * This is extracted to its own function because the NEON path uses a combination * of NEON and scalar. */ XXH_FORCE_INLINE void XXH3_scalarRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT input, void const* XXH_RESTRICT secret, size_t lane) { xxh_u64* xacc = (xxh_u64*) acc; xxh_u8 const* xinput = (xxh_u8 const*) input; xxh_u8 const* xsecret = (xxh_u8 const*) secret; XXH_ASSERT(lane < XXH_ACC_NB); XXH_ASSERT(((size_t)acc & (XXH_ACC_ALIGN-1)) == 0); { xxh_u64 const data_val = XXH_readLE64(xinput + lane * 8); xxh_u64 const data_key = data_val ^ XXH_readLE64(xsecret + lane * 8); xacc[lane ^ 1] += data_val; /* swap adjacent lanes */ xacc[lane] = XXH_mult32to64_add64(data_key /* & 0xFFFFFFFF */, data_key >> 32, xacc[lane]); } } /*! * @internal * @brief Processes a 64 byte block of data using the scalar path. */ XXH_FORCE_INLINE void XXH3_accumulate_512_scalar(void* XXH_RESTRICT acc, const void* XXH_RESTRICT input, const void* XXH_RESTRICT secret) { size_t i; /* ARM GCC refuses to unroll this loop, resulting in a 24% slowdown on ARMv6. */ #if defined(__GNUC__) && !defined(__clang__) \ && (defined(__arm__) || defined(__thumb2__)) \ && defined(__ARM_FEATURE_UNALIGNED) /* no unaligned access just wastes bytes */ \ && XXH_SIZE_OPT <= 0 # pragma GCC unroll 8 #endif for (i=0; i < XXH_ACC_NB; i++) { XXH3_scalarRound(acc, input, secret, i); } } XXH_FORCE_INLINE XXH3_ACCUMULATE_TEMPLATE(scalar) /*! * @internal * @brief Scalar scramble step for @ref XXH3_scrambleAcc_scalar(). * * This is extracted to its own function because the NEON path uses a combination * of NEON and scalar. */ XXH_FORCE_INLINE void XXH3_scalarScrambleRound(void* XXH_RESTRICT acc, void const* XXH_RESTRICT secret, size_t lane) { xxh_u64* const xacc = (xxh_u64*) acc; /* presumed aligned */ const xxh_u8* const xsecret = (const xxh_u8*) secret; /* no alignment restriction */ XXH_ASSERT((((size_t)acc) & (XXH_ACC_ALIGN-1)) == 0); XXH_ASSERT(lane < XXH_ACC_NB); { xxh_u64 const key64 = XXH_readLE64(xsecret + lane * 8); xxh_u64 acc64 = xacc[lane]; acc64 = XXH_xorshift64(acc64, 47); acc64 ^= key64; acc64 *= XXH_PRIME32_1; xacc[lane] = acc64; } } /*! * @internal * @brief Scrambles the accumulators after a large chunk has been read */ XXH_FORCE_INLINE void XXH3_scrambleAcc_scalar(void* XXH_RESTRICT acc, const void* XXH_RESTRICT secret) { size_t i; for (i=0; i < XXH_ACC_NB; i++) { XXH3_scalarScrambleRound(acc, secret, i); } } XXH_FORCE_INLINE void XXH3_initCustomSecret_scalar(void* XXH_RESTRICT customSecret, xxh_u64 seed64) { /* * We need a separate pointer for the hack below, * which requires a non-const pointer. * Any decent compiler will optimize this out otherwise. */ const xxh_u8* kSecretPtr = XXH3_kSecret; XXH_STATIC_ASSERT((XXH_SECRET_DEFAULT_SIZE & 15) == 0); #if defined(__GNUC__) && defined(__aarch64__) /* * UGLY HACK: * GCC and Clang generate a bunch of MOV/MOVK pairs for aarch64, and they are * placed sequentially, in order, at the top of the unrolled loop. * * While MOVK is great for generating constants (2 cycles for a 64-bit * constant compared to 4 cycles for LDR), it fights for bandwidth with * the arithmetic instructions. * * I L S * MOVK * MOVK * MOVK * MOVK * ADD * SUB STR * STR * By forcing loads from memory (as the asm line causes the compiler to assume * that XXH3_kSecretPtr has been changed), the pipelines are used more * efficiently: * I L S * LDR * ADD LDR * SUB STR * STR * * See XXH3_NEON_LANES for details on the pipsline. * * XXH3_64bits_withSeed, len == 256, Snapdragon 835 * without hack: 2654.4 MB/s * with hack: 3202.9 MB/s */ XXH_COMPILER_GUARD(kSecretPtr); #endif { int const nbRounds = XXH_SECRET_DEFAULT_SIZE / 16; int i; for (i=0; i < nbRounds; i++) { /* * The asm hack causes the compiler to assume that kSecretPtr aliases with * customSecret, and on aarch64, this prevented LDP from merging two * loads together for free. Putting the loads together before the stores * properly generates LDP. */ xxh_u64 lo = XXH_readLE64(kSecretPtr + 16*i) + seed64; xxh_u64 hi = XXH_readLE64(kSecretPtr + 16*i + 8) - seed64; XXH_writeLE64((xxh_u8*)customSecret + 16*i, lo); XXH_writeLE64((xxh_u8*)customSecret + 16*i + 8, hi); } } } typedef void (*XXH3_f_accumulate)(xxh_u64* XXH_RESTRICT, const xxh_u8* XXH_RESTRICT, const xxh_u8* XXH_RESTRICT, size_t); typedef void (*XXH3_f_scrambleAcc)(void* XXH_RESTRICT, const void*); typedef void (*XXH3_f_initCustomSecret)(void* XXH_RESTRICT, xxh_u64); #if (XXH_VECTOR == XXH_AVX512) #define XXH3_accumulate_512 XXH3_accumulate_512_avx512 #define XXH3_accumulate XXH3_accumulate_avx512 #define XXH3_scrambleAcc XXH3_scrambleAcc_avx512 #define XXH3_initCustomSecret XXH3_initCustomSecret_avx512 #elif (XXH_VECTOR == XXH_AVX2) #define XXH3_accumulate_512 XXH3_accumulate_512_avx2 #define XXH3_accumulate XXH3_accumulate_avx2 #define XXH3_scrambleAcc XXH3_scrambleAcc_avx2 #define XXH3_initCustomSecret XXH3_initCustomSecret_avx2 #elif (XXH_VECTOR == XXH_SSE2) #define XXH3_accumulate_512 XXH3_accumulate_512_sse2 #define XXH3_accumulate XXH3_accumulate_sse2 #define XXH3_scrambleAcc XXH3_scrambleAcc_sse2 #define XXH3_initCustomSecret XXH3_initCustomSecret_sse2 #elif (XXH_VECTOR == XXH_NEON) #define XXH3_accumulate_512 XXH3_accumulate_512_neon #define XXH3_accumulate XXH3_accumulate_neon #define XXH3_scrambleAcc XXH3_scrambleAcc_neon #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #elif (XXH_VECTOR == XXH_VSX) #define XXH3_accumulate_512 XXH3_accumulate_512_vsx #define XXH3_accumulate XXH3_accumulate_vsx #define XXH3_scrambleAcc XXH3_scrambleAcc_vsx #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #elif (XXH_VECTOR == XXH_SVE) #define XXH3_accumulate_512 XXH3_accumulate_512_sve #define XXH3_accumulate XXH3_accumulate_sve #define XXH3_scrambleAcc XXH3_scrambleAcc_scalar #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #else /* scalar */ #define XXH3_accumulate_512 XXH3_accumulate_512_scalar #define XXH3_accumulate XXH3_accumulate_scalar #define XXH3_scrambleAcc XXH3_scrambleAcc_scalar #define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #endif #if XXH_SIZE_OPT >= 1 /* don't do SIMD for initialization */ # undef XXH3_initCustomSecret # define XXH3_initCustomSecret XXH3_initCustomSecret_scalar #endif XXH_FORCE_INLINE void XXH3_hashLong_internal_loop(xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { size_t const nbStripesPerBlock = (secretSize - XXH_STRIPE_LEN) / XXH_SECRET_CONSUME_RATE; size_t const block_len = XXH_STRIPE_LEN * nbStripesPerBlock; size_t const nb_blocks = (len - 1) / block_len; size_t n; XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); for (n = 0; n < nb_blocks; n++) { f_acc(acc, input + n*block_len, secret, nbStripesPerBlock); f_scramble(acc, secret + secretSize - XXH_STRIPE_LEN); } /* last partial block */ XXH_ASSERT(len > XXH_STRIPE_LEN); { size_t const nbStripes = ((len - 1) - (block_len * nb_blocks)) / XXH_STRIPE_LEN; XXH_ASSERT(nbStripes <= (secretSize / XXH_SECRET_CONSUME_RATE)); f_acc(acc, input + nb_blocks*block_len, secret, nbStripes); /* last stripe */ { const xxh_u8* const p = input + len - XXH_STRIPE_LEN; #define XXH_SECRET_LASTACC_START 7 /* not aligned on 8, last secret is different from acc & scrambler */ XXH3_accumulate_512(acc, p, secret + secretSize - XXH_STRIPE_LEN - XXH_SECRET_LASTACC_START); } } } XXH_FORCE_INLINE xxh_u64 XXH3_mix2Accs(const xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT secret) { return XXH3_mul128_fold64( acc[0] ^ XXH_readLE64(secret), acc[1] ^ XXH_readLE64(secret+8) ); } static XXH64_hash_t XXH3_mergeAccs(const xxh_u64* XXH_RESTRICT acc, const xxh_u8* XXH_RESTRICT secret, xxh_u64 start) { xxh_u64 result64 = start; size_t i = 0; for (i = 0; i < 4; i++) { result64 += XXH3_mix2Accs(acc+2*i, secret + 16*i); #if defined(__clang__) /* Clang */ \ && (defined(__arm__) || defined(__thumb__)) /* ARMv7 */ \ && (defined(__ARM_NEON) || defined(__ARM_NEON__)) /* NEON */ \ && !defined(XXH_ENABLE_AUTOVECTORIZE) /* Define to disable */ /* * UGLY HACK: * Prevent autovectorization on Clang ARMv7-a. Exact same problem as * the one in XXH3_len_129to240_64b. Speeds up shorter keys > 240b. * XXH3_64bits, len == 256, Snapdragon 835: * without hack: 2063.7 MB/s * with hack: 2560.7 MB/s */ XXH_COMPILER_GUARD(result64); #endif } return XXH3_avalanche(result64); } #define XXH3_INIT_ACC { XXH_PRIME32_3, XXH_PRIME64_1, XXH_PRIME64_2, XXH_PRIME64_3, \ XXH_PRIME64_4, XXH_PRIME32_2, XXH_PRIME64_5, XXH_PRIME32_1 } XXH_FORCE_INLINE XXH64_hash_t XXH3_hashLong_64b_internal(const void* XXH_RESTRICT input, size_t len, const void* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[XXH_ACC_NB] = XXH3_INIT_ACC; XXH3_hashLong_internal_loop(acc, (const xxh_u8*)input, len, (const xxh_u8*)secret, secretSize, f_acc, f_scramble); /* converge into final hash */ XXH_STATIC_ASSERT(sizeof(acc) == 64); /* do not align on 8, so that the secret is different from the accumulator */ #define XXH_SECRET_MERGEACCS_START 11 XXH_ASSERT(secretSize >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); return XXH3_mergeAccs(acc, (const xxh_u8*)secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)len * XXH_PRIME64_1); } /* * It's important for performance to transmit secret's size (when it's static) * so that the compiler can properly optimize the vectorized loop. * This makes a big performance difference for "medium" keys (<1 KB) when using AVX instruction set. * When the secret size is unknown, or on GCC 12 where the mix of NO_INLINE and FORCE_INLINE * breaks -Og, this is XXH_NO_INLINE. */ XXH3_WITH_SECRET_INLINE XXH64_hash_t XXH3_hashLong_64b_withSecret(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; return XXH3_hashLong_64b_internal(input, len, secret, secretLen, XXH3_accumulate, XXH3_scrambleAcc); } /* * It's preferable for performance that XXH3_hashLong is not inlined, * as it results in a smaller function for small data, easier to the instruction cache. * Note that inside this no_inline function, we do inline the internal loop, * and provide a statically defined secret size to allow optimization of vector loop. */ XXH_NO_INLINE XXH_PUREF XXH64_hash_t XXH3_hashLong_64b_default(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; (void)secret; (void)secretLen; return XXH3_hashLong_64b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_accumulate, XXH3_scrambleAcc); } /* * XXH3_hashLong_64b_withSeed(): * Generate a custom key based on alteration of default XXH3_kSecret with the seed, * and then use this key for long mode hashing. * * This operation is decently fast but nonetheless costs a little bit of time. * Try to avoid it whenever possible (typically when seed==0). * * It's important for performance that XXH3_hashLong is not inlined. Not sure * why (uop cache maybe?), but the difference is large and easily measurable. */ XXH_FORCE_INLINE XXH64_hash_t XXH3_hashLong_64b_withSeed_internal(const void* input, size_t len, XXH64_hash_t seed, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble, XXH3_f_initCustomSecret f_initSec) { #if XXH_SIZE_OPT <= 0 if (seed == 0) return XXH3_hashLong_64b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), f_acc, f_scramble); #endif { XXH_ALIGN(XXH_SEC_ALIGN) xxh_u8 secret[XXH_SECRET_DEFAULT_SIZE]; f_initSec(secret, seed); return XXH3_hashLong_64b_internal(input, len, secret, sizeof(secret), f_acc, f_scramble); } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH64_hash_t XXH3_hashLong_64b_withSeed(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed, const xxh_u8* XXH_RESTRICT secret, size_t secretLen) { (void)secret; (void)secretLen; return XXH3_hashLong_64b_withSeed_internal(input, len, seed, XXH3_accumulate, XXH3_scrambleAcc, XXH3_initCustomSecret); } typedef XXH64_hash_t (*XXH3_hashLong64_f)(const void* XXH_RESTRICT, size_t, XXH64_hash_t, const xxh_u8* XXH_RESTRICT, size_t); XXH_FORCE_INLINE XXH64_hash_t XXH3_64bits_internal(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen, XXH3_hashLong64_f f_hashLong) { XXH_ASSERT(secretLen >= XXH3_SECRET_SIZE_MIN); /* * If an action is to be taken if `secretLen` condition is not respected, * it should be done here. * For now, it's a contract pre-condition. * Adding a check and a branch here would cost performance at every hash. * Also, note that function signature doesn't offer room to return an error. */ if (len <= 16) return XXH3_len_0to16_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, seed64); if (len <= 128) return XXH3_len_17to128_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); if (len <= XXH3_MIDSIZE_MAX) return XXH3_len_129to240_64b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); return f_hashLong(input, len, seed64, (const xxh_u8*)secret, secretLen); } /* === Public entry point === */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits(XXH_NOESCAPE const void* input, size_t length) { return XXH3_64bits_internal(input, length, 0, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_64b_default); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecret(XXH_NOESCAPE const void* input, size_t length, XXH_NOESCAPE const void* secret, size_t secretSize) { return XXH3_64bits_internal(input, length, 0, secret, secretSize, XXH3_hashLong_64b_withSecret); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSeed(XXH_NOESCAPE const void* input, size_t length, XXH64_hash_t seed) { return XXH3_64bits_internal(input, length, seed, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_64b_withSeed); } XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_withSecretandSeed(XXH_NOESCAPE const void* input, size_t length, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed) { if (length <= XXH3_MIDSIZE_MAX) return XXH3_64bits_internal(input, length, seed, XXH3_kSecret, sizeof(XXH3_kSecret), NULL); return XXH3_hashLong_64b_withSecret(input, length, seed, (const xxh_u8*)secret, secretSize); } /* === XXH3 streaming === */ #ifndef XXH_NO_STREAM /* * Malloc's a pointer that is always aligned to align. * * This must be freed with `XXH_alignedFree()`. * * malloc typically guarantees 16 byte alignment on 64-bit systems and 8 byte * alignment on 32-bit. This isn't enough for the 32 byte aligned loads in AVX2 * or on 32-bit, the 16 byte aligned loads in SSE2 and NEON. * * This underalignment previously caused a rather obvious crash which went * completely unnoticed due to XXH3_createState() not actually being tested. * Credit to RedSpah for noticing this bug. * * The alignment is done manually: Functions like posix_memalign or _mm_malloc * are avoided: To maintain portability, we would have to write a fallback * like this anyways, and besides, testing for the existence of library * functions without relying on external build tools is impossible. * * The method is simple: Overallocate, manually align, and store the offset * to the original behind the returned pointer. * * Align must be a power of 2 and 8 <= align <= 128. */ static XXH_MALLOCF void* XXH_alignedMalloc(size_t s, size_t align) { XXH_ASSERT(align <= 128 && align >= 8); /* range check */ XXH_ASSERT((align & (align-1)) == 0); /* power of 2 */ XXH_ASSERT(s != 0 && s < (s + align)); /* empty/overflow */ { /* Overallocate to make room for manual realignment and an offset byte */ xxh_u8* base = (xxh_u8*)XXH_malloc(s + align); if (base != NULL) { /* * Get the offset needed to align this pointer. * * Even if the returned pointer is aligned, there will always be * at least one byte to store the offset to the original pointer. */ size_t offset = align - ((size_t)base & (align - 1)); /* base % align */ /* Add the offset for the now-aligned pointer */ xxh_u8* ptr = base + offset; XXH_ASSERT((size_t)ptr % align == 0); /* Store the offset immediately before the returned pointer. */ ptr[-1] = (xxh_u8)offset; return ptr; } return NULL; } } /* * Frees an aligned pointer allocated by XXH_alignedMalloc(). Don't pass * normal malloc'd pointers, XXH_alignedMalloc has a specific data layout. */ static void XXH_alignedFree(void* p) { if (p != NULL) { xxh_u8* ptr = (xxh_u8*)p; /* Get the offset byte we added in XXH_malloc. */ xxh_u8 offset = ptr[-1]; /* Free the original malloc'd pointer */ xxh_u8* base = ptr - offset; XXH_free(base); } } /*! @ingroup XXH3_family */ /*! * @brief Allocate an @ref XXH3_state_t. * * Must be freed with XXH3_freeState(). * @return An allocated XXH3_state_t on success, `NULL` on failure. */ XXH_PUBLIC_API XXH3_state_t* XXH3_createState(void) { XXH3_state_t* const state = (XXH3_state_t*)XXH_alignedMalloc(sizeof(XXH3_state_t), 64); if (state==NULL) return NULL; XXH3_INITSTATE(state); return state; } /*! @ingroup XXH3_family */ /*! * @brief Frees an @ref XXH3_state_t. * * Must be allocated with XXH3_createState(). * @param statePtr A pointer to an @ref XXH3_state_t allocated with @ref XXH3_createState(). * @return XXH_OK. */ XXH_PUBLIC_API XXH_errorcode XXH3_freeState(XXH3_state_t* statePtr) { XXH_alignedFree(statePtr); return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API void XXH3_copyState(XXH_NOESCAPE XXH3_state_t* dst_state, XXH_NOESCAPE const XXH3_state_t* src_state) { XXH_memcpy(dst_state, src_state, sizeof(*dst_state)); } static void XXH3_reset_internal(XXH3_state_t* statePtr, XXH64_hash_t seed, const void* secret, size_t secretSize) { size_t const initStart = offsetof(XXH3_state_t, bufferedSize); size_t const initLength = offsetof(XXH3_state_t, nbStripesPerBlock) - initStart; XXH_ASSERT(offsetof(XXH3_state_t, nbStripesPerBlock) > initStart); XXH_ASSERT(statePtr != NULL); /* set members from bufferedSize to nbStripesPerBlock (excluded) to 0 */ memset((char*)statePtr + initStart, 0, initLength); statePtr->acc[0] = XXH_PRIME32_3; statePtr->acc[1] = XXH_PRIME64_1; statePtr->acc[2] = XXH_PRIME64_2; statePtr->acc[3] = XXH_PRIME64_3; statePtr->acc[4] = XXH_PRIME64_4; statePtr->acc[5] = XXH_PRIME32_2; statePtr->acc[6] = XXH_PRIME64_5; statePtr->acc[7] = XXH_PRIME32_1; statePtr->seed = seed; statePtr->useSeed = (seed != 0); statePtr->extSecret = (const unsigned char*)secret; XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); statePtr->secretLimit = secretSize - XXH_STRIPE_LEN; statePtr->nbStripesPerBlock = statePtr->secretLimit / XXH_SECRET_CONSUME_RATE; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset(XXH_NOESCAPE XXH3_state_t* statePtr) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, XXH3_kSecret, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecret(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize) { if (statePtr == NULL) return XXH_ERROR; XXH3_reset_internal(statePtr, 0, secret, secretSize); if (secret == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH64_hash_t seed) { if (statePtr == NULL) return XXH_ERROR; if (seed==0) return XXH3_64bits_reset(statePtr); if ((seed != statePtr->seed) || (statePtr->extSecret != NULL)) XXH3_initCustomSecret(statePtr->customSecret, seed); XXH3_reset_internal(statePtr, seed, NULL, XXH_SECRET_DEFAULT_SIZE); return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_reset_withSecretandSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed64) { if (statePtr == NULL) return XXH_ERROR; if (secret == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; XXH3_reset_internal(statePtr, seed64, secret, secretSize); statePtr->useSeed = 1; /* always, even if seed64==0 */ return XXH_OK; } /*! * @internal * @brief Processes a large input for XXH3_update() and XXH3_digest_long(). * * Unlike XXH3_hashLong_internal_loop(), this can process data that overlaps a block. * * @param acc Pointer to the 8 accumulator lanes * @param nbStripesSoFarPtr In/out pointer to the number of leftover stripes in the block* * @param nbStripesPerBlock Number of stripes in a block * @param input Input pointer * @param nbStripes Number of stripes to process * @param secret Secret pointer * @param secretLimit Offset of the last block in @p secret * @param f_acc Pointer to an XXH3_accumulate implementation * @param f_scramble Pointer to an XXH3_scrambleAcc implementation * @return Pointer past the end of @p input after processing */ XXH_FORCE_INLINE const xxh_u8 * XXH3_consumeStripes(xxh_u64* XXH_RESTRICT acc, size_t* XXH_RESTRICT nbStripesSoFarPtr, size_t nbStripesPerBlock, const xxh_u8* XXH_RESTRICT input, size_t nbStripes, const xxh_u8* XXH_RESTRICT secret, size_t secretLimit, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { const xxh_u8* initialSecret = secret + *nbStripesSoFarPtr * XXH_SECRET_CONSUME_RATE; /* Process full blocks */ if (nbStripes >= (nbStripesPerBlock - *nbStripesSoFarPtr)) { /* Process the initial partial block... */ size_t nbStripesThisIter = nbStripesPerBlock - *nbStripesSoFarPtr; do { /* Accumulate and scramble */ f_acc(acc, input, initialSecret, nbStripesThisIter); f_scramble(acc, secret + secretLimit); input += nbStripesThisIter * XXH_STRIPE_LEN; nbStripes -= nbStripesThisIter; /* Then continue the loop with the full block size */ nbStripesThisIter = nbStripesPerBlock; initialSecret = secret; } while (nbStripes >= nbStripesPerBlock); *nbStripesSoFarPtr = 0; } /* Process a partial block */ if (nbStripes > 0) { f_acc(acc, input, initialSecret, nbStripes); input += nbStripes * XXH_STRIPE_LEN; *nbStripesSoFarPtr += nbStripes; } /* Return end pointer */ return input; } #ifndef XXH3_STREAM_USE_STACK # if XXH_SIZE_OPT <= 0 && !defined(__clang__) /* clang doesn't need additional stack space */ # define XXH3_STREAM_USE_STACK 1 # endif #endif /* * Both XXH3_64bits_update and XXH3_128bits_update use this routine. */ XXH_FORCE_INLINE XXH_errorcode XXH3_update(XXH3_state_t* XXH_RESTRICT const state, const xxh_u8* XXH_RESTRICT input, size_t len, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { if (input==NULL) { XXH_ASSERT(len == 0); return XXH_OK; } XXH_ASSERT(state != NULL); { const xxh_u8* const bEnd = input + len; const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; #if defined(XXH3_STREAM_USE_STACK) && XXH3_STREAM_USE_STACK >= 1 /* For some reason, gcc and MSVC seem to suffer greatly * when operating accumulators directly into state. * Operating into stack space seems to enable proper optimization. * clang, on the other hand, doesn't seem to need this trick */ XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[8]; XXH_memcpy(acc, state->acc, sizeof(acc)); #else xxh_u64* XXH_RESTRICT const acc = state->acc; #endif state->totalLen += len; XXH_ASSERT(state->bufferedSize <= XXH3_INTERNALBUFFER_SIZE); /* small input : just fill in tmp buffer */ if (len <= XXH3_INTERNALBUFFER_SIZE - state->bufferedSize) { XXH_memcpy(state->buffer + state->bufferedSize, input, len); state->bufferedSize += (XXH32_hash_t)len; return XXH_OK; } /* total input is now > XXH3_INTERNALBUFFER_SIZE */ #define XXH3_INTERNALBUFFER_STRIPES (XXH3_INTERNALBUFFER_SIZE / XXH_STRIPE_LEN) XXH_STATIC_ASSERT(XXH3_INTERNALBUFFER_SIZE % XXH_STRIPE_LEN == 0); /* clean multiple */ /* * Internal buffer is partially filled (always, except at beginning) * Complete it, then consume it. */ if (state->bufferedSize) { size_t const loadSize = XXH3_INTERNALBUFFER_SIZE - state->bufferedSize; XXH_memcpy(state->buffer + state->bufferedSize, input, loadSize); input += loadSize; XXH3_consumeStripes(acc, &state->nbStripesSoFar, state->nbStripesPerBlock, state->buffer, XXH3_INTERNALBUFFER_STRIPES, secret, state->secretLimit, f_acc, f_scramble); state->bufferedSize = 0; } XXH_ASSERT(input < bEnd); if (bEnd - input > XXH3_INTERNALBUFFER_SIZE) { size_t nbStripes = (size_t)(bEnd - 1 - input) / XXH_STRIPE_LEN; input = XXH3_consumeStripes(acc, &state->nbStripesSoFar, state->nbStripesPerBlock, input, nbStripes, secret, state->secretLimit, f_acc, f_scramble); XXH_memcpy(state->buffer + sizeof(state->buffer) - XXH_STRIPE_LEN, input - XXH_STRIPE_LEN, XXH_STRIPE_LEN); } /* Some remaining input (always) : buffer it */ XXH_ASSERT(input < bEnd); XXH_ASSERT(bEnd - input <= XXH3_INTERNALBUFFER_SIZE); XXH_ASSERT(state->bufferedSize == 0); XXH_memcpy(state->buffer, input, (size_t)(bEnd-input)); state->bufferedSize = (XXH32_hash_t)(bEnd-input); #if defined(XXH3_STREAM_USE_STACK) && XXH3_STREAM_USE_STACK >= 1 /* save stack accumulators into state */ XXH_memcpy(state->acc, acc, sizeof(acc)); #endif } return XXH_OK; } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_64bits_update(XXH_NOESCAPE XXH3_state_t* state, XXH_NOESCAPE const void* input, size_t len) { return XXH3_update(state, (const xxh_u8*)input, len, XXH3_accumulate, XXH3_scrambleAcc); } XXH_FORCE_INLINE void XXH3_digest_long (XXH64_hash_t* acc, const XXH3_state_t* state, const unsigned char* secret) { xxh_u8 lastStripe[XXH_STRIPE_LEN]; const xxh_u8* lastStripePtr; /* * Digest on a local copy. This way, the state remains unaltered, and it can * continue ingesting more input afterwards. */ XXH_memcpy(acc, state->acc, sizeof(state->acc)); if (state->bufferedSize >= XXH_STRIPE_LEN) { /* Consume remaining stripes then point to remaining data in buffer */ size_t const nbStripes = (state->bufferedSize - 1) / XXH_STRIPE_LEN; size_t nbStripesSoFar = state->nbStripesSoFar; XXH3_consumeStripes(acc, &nbStripesSoFar, state->nbStripesPerBlock, state->buffer, nbStripes, secret, state->secretLimit, XXH3_accumulate, XXH3_scrambleAcc); lastStripePtr = state->buffer + state->bufferedSize - XXH_STRIPE_LEN; } else { /* bufferedSize < XXH_STRIPE_LEN */ /* Copy to temp buffer */ size_t const catchupSize = XXH_STRIPE_LEN - state->bufferedSize; XXH_ASSERT(state->bufferedSize > 0); /* there is always some input buffered */ XXH_memcpy(lastStripe, state->buffer + sizeof(state->buffer) - catchupSize, catchupSize); XXH_memcpy(lastStripe + catchupSize, state->buffer, state->bufferedSize); lastStripePtr = lastStripe; } /* Last stripe */ XXH3_accumulate_512(acc, lastStripePtr, secret + state->secretLimit - XXH_SECRET_LASTACC_START); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH64_hash_t XXH3_64bits_digest (XXH_NOESCAPE const XXH3_state_t* state) { const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; if (state->totalLen > XXH3_MIDSIZE_MAX) { XXH_ALIGN(XXH_ACC_ALIGN) XXH64_hash_t acc[XXH_ACC_NB]; XXH3_digest_long(acc, state, secret); return XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)state->totalLen * XXH_PRIME64_1); } /* totalLen <= XXH3_MIDSIZE_MAX: digesting a short input */ if (state->useSeed) return XXH3_64bits_withSeed(state->buffer, (size_t)state->totalLen, state->seed); return XXH3_64bits_withSecret(state->buffer, (size_t)(state->totalLen), secret, state->secretLimit + XXH_STRIPE_LEN); } #endif /* !XXH_NO_STREAM */ /* ========================================== * XXH3 128 bits (a.k.a XXH128) * ========================================== * XXH3's 128-bit variant has better mixing and strength than the 64-bit variant, * even without counting the significantly larger output size. * * For example, extra steps are taken to avoid the seed-dependent collisions * in 17-240 byte inputs (See XXH3_mix16B and XXH128_mix32B). * * This strength naturally comes at the cost of some speed, especially on short * lengths. Note that longer hashes are about as fast as the 64-bit version * due to it using only a slight modification of the 64-bit loop. * * XXH128 is also more oriented towards 64-bit machines. It is still extremely * fast for a _128-bit_ hash on 32-bit (it usually clears XXH64). */ XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_1to3_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { /* A doubled version of 1to3_64b with different constants. */ XXH_ASSERT(input != NULL); XXH_ASSERT(1 <= len && len <= 3); XXH_ASSERT(secret != NULL); /* * len = 1: combinedl = { input[0], 0x01, input[0], input[0] } * len = 2: combinedl = { input[1], 0x02, input[0], input[1] } * len = 3: combinedl = { input[2], 0x03, input[0], input[1] } */ { xxh_u8 const c1 = input[0]; xxh_u8 const c2 = input[len >> 1]; xxh_u8 const c3 = input[len - 1]; xxh_u32 const combinedl = ((xxh_u32)c1 <<16) | ((xxh_u32)c2 << 24) | ((xxh_u32)c3 << 0) | ((xxh_u32)len << 8); xxh_u32 const combinedh = XXH_rotl32(XXH_swap32(combinedl), 13); xxh_u64 const bitflipl = (XXH_readLE32(secret) ^ XXH_readLE32(secret+4)) + seed; xxh_u64 const bitfliph = (XXH_readLE32(secret+8) ^ XXH_readLE32(secret+12)) - seed; xxh_u64 const keyed_lo = (xxh_u64)combinedl ^ bitflipl; xxh_u64 const keyed_hi = (xxh_u64)combinedh ^ bitfliph; XXH128_hash_t h128; h128.low64 = XXH64_avalanche(keyed_lo); h128.high64 = XXH64_avalanche(keyed_hi); return h128; } } XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_4to8_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(4 <= len && len <= 8); seed ^= (xxh_u64)XXH_swap32((xxh_u32)seed) << 32; { xxh_u32 const input_lo = XXH_readLE32(input); xxh_u32 const input_hi = XXH_readLE32(input + len - 4); xxh_u64 const input_64 = input_lo + ((xxh_u64)input_hi << 32); xxh_u64 const bitflip = (XXH_readLE64(secret+16) ^ XXH_readLE64(secret+24)) + seed; xxh_u64 const keyed = input_64 ^ bitflip; /* Shift len to the left to ensure it is even, this avoids even multiplies. */ XXH128_hash_t m128 = XXH_mult64to128(keyed, XXH_PRIME64_1 + (len << 2)); m128.high64 += (m128.low64 << 1); m128.low64 ^= (m128.high64 >> 3); m128.low64 = XXH_xorshift64(m128.low64, 35); m128.low64 *= PRIME_MX2; m128.low64 = XXH_xorshift64(m128.low64, 28); m128.high64 = XXH3_avalanche(m128.high64); return m128; } } XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_9to16_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(input != NULL); XXH_ASSERT(secret != NULL); XXH_ASSERT(9 <= len && len <= 16); { xxh_u64 const bitflipl = (XXH_readLE64(secret+32) ^ XXH_readLE64(secret+40)) - seed; xxh_u64 const bitfliph = (XXH_readLE64(secret+48) ^ XXH_readLE64(secret+56)) + seed; xxh_u64 const input_lo = XXH_readLE64(input); xxh_u64 input_hi = XXH_readLE64(input + len - 8); XXH128_hash_t m128 = XXH_mult64to128(input_lo ^ input_hi ^ bitflipl, XXH_PRIME64_1); /* * Put len in the middle of m128 to ensure that the length gets mixed to * both the low and high bits in the 128x64 multiply below. */ m128.low64 += (xxh_u64)(len - 1) << 54; input_hi ^= bitfliph; /* * Add the high 32 bits of input_hi to the high 32 bits of m128, then * add the long product of the low 32 bits of input_hi and XXH_PRIME32_2 to * the high 64 bits of m128. * * The best approach to this operation is different on 32-bit and 64-bit. */ if (sizeof(void *) < sizeof(xxh_u64)) { /* 32-bit */ /* * 32-bit optimized version, which is more readable. * * On 32-bit, it removes an ADC and delays a dependency between the two * halves of m128.high64, but it generates an extra mask on 64-bit. */ m128.high64 += (input_hi & 0xFFFFFFFF00000000ULL) + XXH_mult32to64((xxh_u32)input_hi, XXH_PRIME32_2); } else { /* * 64-bit optimized (albeit more confusing) version. * * Uses some properties of addition and multiplication to remove the mask: * * Let: * a = input_hi.lo = (input_hi & 0x00000000FFFFFFFF) * b = input_hi.hi = (input_hi & 0xFFFFFFFF00000000) * c = XXH_PRIME32_2 * * a + (b * c) * Inverse Property: x + y - x == y * a + (b * (1 + c - 1)) * Distributive Property: x * (y + z) == (x * y) + (x * z) * a + (b * 1) + (b * (c - 1)) * Identity Property: x * 1 == x * a + b + (b * (c - 1)) * * Substitute a, b, and c: * input_hi.hi + input_hi.lo + ((xxh_u64)input_hi.lo * (XXH_PRIME32_2 - 1)) * * Since input_hi.hi + input_hi.lo == input_hi, we get this: * input_hi + ((xxh_u64)input_hi.lo * (XXH_PRIME32_2 - 1)) */ m128.high64 += input_hi + XXH_mult32to64((xxh_u32)input_hi, XXH_PRIME32_2 - 1); } /* m128 ^= XXH_swap64(m128 >> 64); */ m128.low64 ^= XXH_swap64(m128.high64); { /* 128x64 multiply: h128 = m128 * XXH_PRIME64_2; */ XXH128_hash_t h128 = XXH_mult64to128(m128.low64, XXH_PRIME64_2); h128.high64 += m128.high64 * XXH_PRIME64_2; h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = XXH3_avalanche(h128.high64); return h128; } } } /* * Assumption: `secret` size is >= XXH3_SECRET_SIZE_MIN */ XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_0to16_128b(const xxh_u8* input, size_t len, const xxh_u8* secret, XXH64_hash_t seed) { XXH_ASSERT(len <= 16); { if (len > 8) return XXH3_len_9to16_128b(input, len, secret, seed); if (len >= 4) return XXH3_len_4to8_128b(input, len, secret, seed); if (len) return XXH3_len_1to3_128b(input, len, secret, seed); { XXH128_hash_t h128; xxh_u64 const bitflipl = XXH_readLE64(secret+64) ^ XXH_readLE64(secret+72); xxh_u64 const bitfliph = XXH_readLE64(secret+80) ^ XXH_readLE64(secret+88); h128.low64 = XXH64_avalanche(seed ^ bitflipl); h128.high64 = XXH64_avalanche( seed ^ bitfliph); return h128; } } } /* * A bit slower than XXH3_mix16B, but handles multiply by zero better. */ XXH_FORCE_INLINE XXH128_hash_t XXH128_mix32B(XXH128_hash_t acc, const xxh_u8* input_1, const xxh_u8* input_2, const xxh_u8* secret, XXH64_hash_t seed) { acc.low64 += XXH3_mix16B (input_1, secret+0, seed); acc.low64 ^= XXH_readLE64(input_2) + XXH_readLE64(input_2 + 8); acc.high64 += XXH3_mix16B (input_2, secret+16, seed); acc.high64 ^= XXH_readLE64(input_1) + XXH_readLE64(input_1 + 8); return acc; } XXH_FORCE_INLINE XXH_PUREF XXH128_hash_t XXH3_len_17to128_128b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(16 < len && len <= 128); { XXH128_hash_t acc; acc.low64 = len * XXH_PRIME64_1; acc.high64 = 0; #if XXH_SIZE_OPT >= 1 { /* Smaller, but slightly slower. */ unsigned int i = (unsigned int)(len - 1) / 32; do { acc = XXH128_mix32B(acc, input+16*i, input+len-16*(i+1), secret+32*i, seed); } while (i-- != 0); } #else if (len > 32) { if (len > 64) { if (len > 96) { acc = XXH128_mix32B(acc, input+48, input+len-64, secret+96, seed); } acc = XXH128_mix32B(acc, input+32, input+len-48, secret+64, seed); } acc = XXH128_mix32B(acc, input+16, input+len-32, secret+32, seed); } acc = XXH128_mix32B(acc, input, input+len-16, secret, seed); #endif { XXH128_hash_t h128; h128.low64 = acc.low64 + acc.high64; h128.high64 = (acc.low64 * XXH_PRIME64_1) + (acc.high64 * XXH_PRIME64_4) + ((len - seed) * XXH_PRIME64_2); h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = (XXH64_hash_t)0 - XXH3_avalanche(h128.high64); return h128; } } } XXH_NO_INLINE XXH_PUREF XXH128_hash_t XXH3_len_129to240_128b(const xxh_u8* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH64_hash_t seed) { XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); (void)secretSize; XXH_ASSERT(128 < len && len <= XXH3_MIDSIZE_MAX); { XXH128_hash_t acc; unsigned i; acc.low64 = len * XXH_PRIME64_1; acc.high64 = 0; /* * We set as `i` as offset + 32. We do this so that unchanged * `len` can be used as upper bound. This reaches a sweet spot * where both x86 and aarch64 get simple agen and good codegen * for the loop. */ for (i = 32; i < 160; i += 32) { acc = XXH128_mix32B(acc, input + i - 32, input + i - 16, secret + i - 32, seed); } acc.low64 = XXH3_avalanche(acc.low64); acc.high64 = XXH3_avalanche(acc.high64); /* * NB: `i <= len` will duplicate the last 32-bytes if * len % 32 was zero. This is an unfortunate necessity to keep * the hash result stable. */ for (i=160; i <= len; i += 32) { acc = XXH128_mix32B(acc, input + i - 32, input + i - 16, secret + XXH3_MIDSIZE_STARTOFFSET + i - 160, seed); } /* last bytes */ acc = XXH128_mix32B(acc, input + len - 16, input + len - 32, secret + XXH3_SECRET_SIZE_MIN - XXH3_MIDSIZE_LASTOFFSET - 16, (XXH64_hash_t)0 - seed); { XXH128_hash_t h128; h128.low64 = acc.low64 + acc.high64; h128.high64 = (acc.low64 * XXH_PRIME64_1) + (acc.high64 * XXH_PRIME64_4) + ((len - seed) * XXH_PRIME64_2); h128.low64 = XXH3_avalanche(h128.low64); h128.high64 = (XXH64_hash_t)0 - XXH3_avalanche(h128.high64); return h128; } } } XXH_FORCE_INLINE XXH128_hash_t XXH3_hashLong_128b_internal(const void* XXH_RESTRICT input, size_t len, const xxh_u8* XXH_RESTRICT secret, size_t secretSize, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble) { XXH_ALIGN(XXH_ACC_ALIGN) xxh_u64 acc[XXH_ACC_NB] = XXH3_INIT_ACC; XXH3_hashLong_internal_loop(acc, (const xxh_u8*)input, len, secret, secretSize, f_acc, f_scramble); /* converge into final hash */ XXH_STATIC_ASSERT(sizeof(acc) == 64); XXH_ASSERT(secretSize >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); { XXH128_hash_t h128; h128.low64 = XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)len * XXH_PRIME64_1); h128.high64 = XXH3_mergeAccs(acc, secret + secretSize - sizeof(acc) - XXH_SECRET_MERGEACCS_START, ~((xxh_u64)len * XXH_PRIME64_2)); return h128; } } /* * It's important for performance that XXH3_hashLong() is not inlined. */ XXH_NO_INLINE XXH_PUREF XXH128_hash_t XXH3_hashLong_128b_default(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; (void)secret; (void)secretLen; return XXH3_hashLong_128b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_accumulate, XXH3_scrambleAcc); } /* * It's important for performance to pass @p secretLen (when it's static) * to the compiler, so that it can properly optimize the vectorized loop. * * When the secret size is unknown, or on GCC 12 where the mix of NO_INLINE and FORCE_INLINE * breaks -Og, this is XXH_NO_INLINE. */ XXH3_WITH_SECRET_INLINE XXH128_hash_t XXH3_hashLong_128b_withSecret(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)seed64; return XXH3_hashLong_128b_internal(input, len, (const xxh_u8*)secret, secretLen, XXH3_accumulate, XXH3_scrambleAcc); } XXH_FORCE_INLINE XXH128_hash_t XXH3_hashLong_128b_withSeed_internal(const void* XXH_RESTRICT input, size_t len, XXH64_hash_t seed64, XXH3_f_accumulate f_acc, XXH3_f_scrambleAcc f_scramble, XXH3_f_initCustomSecret f_initSec) { if (seed64 == 0) return XXH3_hashLong_128b_internal(input, len, XXH3_kSecret, sizeof(XXH3_kSecret), f_acc, f_scramble); { XXH_ALIGN(XXH_SEC_ALIGN) xxh_u8 secret[XXH_SECRET_DEFAULT_SIZE]; f_initSec(secret, seed64); return XXH3_hashLong_128b_internal(input, len, (const xxh_u8*)secret, sizeof(secret), f_acc, f_scramble); } } /* * It's important for performance that XXH3_hashLong is not inlined. */ XXH_NO_INLINE XXH128_hash_t XXH3_hashLong_128b_withSeed(const void* input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen) { (void)secret; (void)secretLen; return XXH3_hashLong_128b_withSeed_internal(input, len, seed64, XXH3_accumulate, XXH3_scrambleAcc, XXH3_initCustomSecret); } typedef XXH128_hash_t (*XXH3_hashLong128_f)(const void* XXH_RESTRICT, size_t, XXH64_hash_t, const void* XXH_RESTRICT, size_t); XXH_FORCE_INLINE XXH128_hash_t XXH3_128bits_internal(const void* input, size_t len, XXH64_hash_t seed64, const void* XXH_RESTRICT secret, size_t secretLen, XXH3_hashLong128_f f_hl128) { XXH_ASSERT(secretLen >= XXH3_SECRET_SIZE_MIN); /* * If an action is to be taken if `secret` conditions are not respected, * it should be done here. * For now, it's a contract pre-condition. * Adding a check and a branch here would cost performance at every hash. */ if (len <= 16) return XXH3_len_0to16_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, seed64); if (len <= 128) return XXH3_len_17to128_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); if (len <= XXH3_MIDSIZE_MAX) return XXH3_len_129to240_128b((const xxh_u8*)input, len, (const xxh_u8*)secret, secretLen, seed64); return f_hl128(input, len, seed64, secret, secretLen); } /* === Public XXH128 API === */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits(XXH_NOESCAPE const void* input, size_t len) { return XXH3_128bits_internal(input, len, 0, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_128b_default); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecret(XXH_NOESCAPE const void* input, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize) { return XXH3_128bits_internal(input, len, 0, (const xxh_u8*)secret, secretSize, XXH3_hashLong_128b_withSecret); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSeed(XXH_NOESCAPE const void* input, size_t len, XXH64_hash_t seed) { return XXH3_128bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), XXH3_hashLong_128b_withSeed); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_withSecretandSeed(XXH_NOESCAPE const void* input, size_t len, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed) { if (len <= XXH3_MIDSIZE_MAX) return XXH3_128bits_internal(input, len, seed, XXH3_kSecret, sizeof(XXH3_kSecret), NULL); return XXH3_hashLong_128b_withSecret(input, len, seed, secret, secretSize); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH128(XXH_NOESCAPE const void* input, size_t len, XXH64_hash_t seed) { return XXH3_128bits_withSeed(input, len, seed); } /* === XXH3 128-bit streaming === */ #ifndef XXH_NO_STREAM /* * All initialization and update functions are identical to 64-bit streaming variant. * The only difference is the finalization routine. */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset(XXH_NOESCAPE XXH3_state_t* statePtr) { return XXH3_64bits_reset(statePtr); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecret(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize) { return XXH3_64bits_reset_withSecret(statePtr, secret, secretSize); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH64_hash_t seed) { return XXH3_64bits_reset_withSeed(statePtr, seed); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_reset_withSecretandSeed(XXH_NOESCAPE XXH3_state_t* statePtr, XXH_NOESCAPE const void* secret, size_t secretSize, XXH64_hash_t seed) { return XXH3_64bits_reset_withSecretandSeed(statePtr, secret, secretSize, seed); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_128bits_update(XXH_NOESCAPE XXH3_state_t* state, XXH_NOESCAPE const void* input, size_t len) { return XXH3_64bits_update(state, input, len); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH3_128bits_digest (XXH_NOESCAPE const XXH3_state_t* state) { const unsigned char* const secret = (state->extSecret == NULL) ? state->customSecret : state->extSecret; if (state->totalLen > XXH3_MIDSIZE_MAX) { XXH_ALIGN(XXH_ACC_ALIGN) XXH64_hash_t acc[XXH_ACC_NB]; XXH3_digest_long(acc, state, secret); XXH_ASSERT(state->secretLimit + XXH_STRIPE_LEN >= sizeof(acc) + XXH_SECRET_MERGEACCS_START); { XXH128_hash_t h128; h128.low64 = XXH3_mergeAccs(acc, secret + XXH_SECRET_MERGEACCS_START, (xxh_u64)state->totalLen * XXH_PRIME64_1); h128.high64 = XXH3_mergeAccs(acc, secret + state->secretLimit + XXH_STRIPE_LEN - sizeof(acc) - XXH_SECRET_MERGEACCS_START, ~((xxh_u64)state->totalLen * XXH_PRIME64_2)); return h128; } } /* len <= XXH3_MIDSIZE_MAX : short code */ if (state->seed) return XXH3_128bits_withSeed(state->buffer, (size_t)state->totalLen, state->seed); return XXH3_128bits_withSecret(state->buffer, (size_t)(state->totalLen), secret, state->secretLimit + XXH_STRIPE_LEN); } #endif /* !XXH_NO_STREAM */ /* 128-bit utility functions */ #include /* memcmp, memcpy */ /* return : 1 is equal, 0 if different */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API int XXH128_isEqual(XXH128_hash_t h1, XXH128_hash_t h2) { /* note : XXH128_hash_t is compact, it has no padding byte */ return !(memcmp(&h1, &h2, sizeof(h1))); } /* This prototype is compatible with stdlib's qsort(). * @return : >0 if *h128_1 > *h128_2 * <0 if *h128_1 < *h128_2 * =0 if *h128_1 == *h128_2 */ /*! @ingroup XXH3_family */ XXH_PUBLIC_API int XXH128_cmp(XXH_NOESCAPE const void* h128_1, XXH_NOESCAPE const void* h128_2) { XXH128_hash_t const h1 = *(const XXH128_hash_t*)h128_1; XXH128_hash_t const h2 = *(const XXH128_hash_t*)h128_2; int const hcmp = (h1.high64 > h2.high64) - (h2.high64 > h1.high64); /* note : bets that, in most cases, hash values are different */ if (hcmp) return hcmp; return (h1.low64 > h2.low64) - (h2.low64 > h1.low64); } /*====== Canonical representation ======*/ /*! @ingroup XXH3_family */ XXH_PUBLIC_API void XXH128_canonicalFromHash(XXH_NOESCAPE XXH128_canonical_t* dst, XXH128_hash_t hash) { XXH_STATIC_ASSERT(sizeof(XXH128_canonical_t) == sizeof(XXH128_hash_t)); if (XXH_CPU_LITTLE_ENDIAN) { hash.high64 = XXH_swap64(hash.high64); hash.low64 = XXH_swap64(hash.low64); } XXH_memcpy(dst, &hash.high64, sizeof(hash.high64)); XXH_memcpy((char*)dst + sizeof(hash.high64), &hash.low64, sizeof(hash.low64)); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH128_hash_t XXH128_hashFromCanonical(XXH_NOESCAPE const XXH128_canonical_t* src) { XXH128_hash_t h; h.high64 = XXH_readBE64(src); h.low64 = XXH_readBE64(src->digest + 8); return h; } /* ========================================== * Secret generators * ========================================== */ #define XXH_MIN(x, y) (((x) > (y)) ? (y) : (x)) XXH_FORCE_INLINE void XXH3_combine16(void* dst, XXH128_hash_t h128) { XXH_writeLE64( dst, XXH_readLE64(dst) ^ h128.low64 ); XXH_writeLE64( (char*)dst+8, XXH_readLE64((char*)dst+8) ^ h128.high64 ); } /*! @ingroup XXH3_family */ XXH_PUBLIC_API XXH_errorcode XXH3_generateSecret(XXH_NOESCAPE void* secretBuffer, size_t secretSize, XXH_NOESCAPE const void* customSeed, size_t customSeedSize) { #if (XXH_DEBUGLEVEL >= 1) XXH_ASSERT(secretBuffer != NULL); XXH_ASSERT(secretSize >= XXH3_SECRET_SIZE_MIN); #else /* production mode, assert() are disabled */ if (secretBuffer == NULL) return XXH_ERROR; if (secretSize < XXH3_SECRET_SIZE_MIN) return XXH_ERROR; #endif if (customSeedSize == 0) { customSeed = XXH3_kSecret; customSeedSize = XXH_SECRET_DEFAULT_SIZE; } #if (XXH_DEBUGLEVEL >= 1) XXH_ASSERT(customSeed != NULL); #else if (customSeed == NULL) return XXH_ERROR; #endif /* Fill secretBuffer with a copy of customSeed - repeat as needed */ { size_t pos = 0; while (pos < secretSize) { size_t const toCopy = XXH_MIN((secretSize - pos), customSeedSize); memcpy((char*)secretBuffer + pos, customSeed, toCopy); pos += toCopy; } } { size_t const nbSeg16 = secretSize / 16; size_t n; XXH128_canonical_t scrambler; XXH128_canonicalFromHash(&scrambler, XXH128(customSeed, customSeedSize, 0)); for (n=0; n #include #include "common/debug.h" /**********************************************************************************************************************************/ FN_EXTERN size_t objToLog(const void *const object, ObjToLogFormat formatFunc, char *const buffer, const size_t bufferSize) { StringStatic debugLog = strStcInit(buffer, bufferSize); if (object == NULL) strStcCat(&debugLog, NULL_Z); else formatFunc(object, &debugLog); return strStcResultSize(&debugLog); } /**********************************************************************************************************************************/ FN_EXTERN size_t objNameToLog(const void *object, const char *objectName, char *buffer, size_t bufferSize) { StringStatic debugLog = strStcInit(buffer, bufferSize); if (object == NULL) strStcCat(&debugLog, NULL_Z); else strStcFmt(&debugLog, "{%s}", objectName); return strStcResultSize(&debugLog); } /**********************************************************************************************************************************/ FN_EXTERN size_t ptrToLog(const void *pointer, const char *pointerName, char *buffer, size_t bufferSize) { StringStatic debugLog = strStcInit(buffer, bufferSize); if (pointer == NULL) strStcCat(&debugLog, NULL_Z); else strStcFmt(&debugLog, "(%s)", pointerName); return strStcResultSize(&debugLog); } /**********************************************************************************************************************************/ FN_EXTERN size_t typeToLog(const char *typeName, char *buffer, size_t bufferSize) { StringStatic debugLog = strStcInit(buffer, bufferSize); strStcFmt(&debugLog, "%s", typeName); return strStcResultSize(&debugLog); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/debug.h0000664000175100017510000011347115240547630016740 0ustar00runnerrunner/*********************************************************************************************************************************** Debug Routines ***********************************************************************************************************************************/ #ifndef COMMON_DEBUG_H #define COMMON_DEBUG_H #include "common/assert.h" #include "common/stackTrace.h" #include "common/type/convert.h" #include "common/type/stringStatic.h" /*********************************************************************************************************************************** These functions allow auditing of child mem contexts and allocations that are left in the calling context when a function exits. This helps find leaks, i.e. child mem contexts or allocations created in the calling context (and not freed) accidentally. The FUNCTION_AUDIT_*() macros can be used to annotate functions that do that follow the default behavior, i.e. that a single value is returned and that is the only value created in the calling context. ***********************************************************************************************************************************/ #if defined(DEBUG_MEM) && defined(DEBUG_TEST_TRACE) #include "common/macro.h" #include "common/memContext.h" // Begin the audit #define FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN() \ MemContextAuditState MEM_CONTEXT_AUDIT_param = {.memContext = memContextCurrent()}; \ memContextAuditBegin(&MEM_CONTEXT_AUDIT_param) // End the audit #define FUNCTION_TEST_MEM_CONTEXT_AUDIT_END(returnType) \ memContextAuditEnd(&MEM_CONTEXT_AUDIT_param, returnType) // Allow any new mem contexts or allocations in the calling context. These should be fixed and this macro eventually removed. #define FUNCTION_AUDIT_IF(condition) \ do \ { \ if (!(condition)) \ MEM_CONTEXT_AUDIT_param.returnTypeAny = true; \ } \ while (0) // Callbacks are difficult to audit so ignore them. Eventually they should all be removed. #define FUNCTION_AUDIT_CALLBACK() MEM_CONTEXT_AUDIT_param.returnTypeAny = true // Helper function that creates new mem contexts or allocations in the calling context. These functions should be static (except // for interface helpers) but it is not clear that anything else needs to be done. #define FUNCTION_AUDIT_HELPER() MEM_CONTEXT_AUDIT_param.returnTypeAny = true // Function returns a struct that has new mem contexts or allocations in the calling context. Find a way to fix these. #define FUNCTION_AUDIT_STRUCT() MEM_CONTEXT_AUDIT_param.returnTypeAny = true #else #define FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN() #define FUNCTION_TEST_MEM_CONTEXT_AUDIT_END(returnType) #define FUNCTION_AUDIT_IF(condition) #define FUNCTION_AUDIT_CALLBACK() #define FUNCTION_AUDIT_HELPER() #define FUNCTION_AUDIT_STRUCT() #endif /*********************************************************************************************************************************** Base function debugging macros In debug mode parameters will always be recorded in the stack trace while in production mode they will only be recorded when the log level is set to debug or trace. ***********************************************************************************************************************************/ #define FUNCTION_LOG_LEVEL() \ FUNCTION_LOG_logLevel #ifdef DEBUG_TEST_TRACE #define FUNCTION_LOG_BEGIN_BASE(logLevel) \ LogLevel FUNCTION_LOG_LEVEL() = STACK_TRACE_PUSH(logLevel); \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN(); \ \ { \ stackTraceParamLog(); \ stackTraceTestStop() #define FUNCTION_LOG_END_BASE() \ stackTraceTestStart(); \ LOG_FMT(FUNCTION_LOG_LEVEL(), 0, "(%s)", stackTraceParam()); \ } #else #define FUNCTION_LOG_BEGIN_BASE(logLevel) \ LogLevel FUNCTION_LOG_LEVEL() = STACK_TRACE_PUSH(logLevel); \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN(); \ \ if (logAny(FUNCTION_LOG_LEVEL())) \ { \ stackTraceParamLog() #define FUNCTION_LOG_END_BASE() \ LOG_FMT(FUNCTION_LOG_LEVEL(), 0, "(%s)", stackTraceParam()); \ } #endif /*********************************************************************************************************************************** General purpose function debugging macros FUNCTION_LOG_VOID() is provided as a shortcut for functions that have no parameters. ***********************************************************************************************************************************/ #define FUNCTION_LOG_BEGIN(logLevel) \ FUNCTION_LOG_BEGIN_BASE(logLevel) #define FUNCTION_LOG_END() \ FUNCTION_LOG_END_BASE() #define FUNCTION_LOG_VOID(logLevel) \ FUNCTION_LOG_BEGIN_BASE(logLevel); \ FUNCTION_LOG_END_BASE() #define FUNCTION_LOG_PARAM(typeMacroPrefix, param) \ stackTraceParamAdd(FUNCTION_LOG_##typeMacroPrefix##_FORMAT(param, stackTraceParamBuffer(#param), STACK_TRACE_PARAM_MAX)) #define FUNCTION_LOG_PARAM_P(typeMacroPrefix, param) \ do \ { \ char *buffer = stackTraceParamBuffer(#param); \ \ if (param == NULL) \ stackTraceParamAdd(typeToLog(NULL_Z, buffer, STACK_TRACE_PARAM_MAX)); \ else \ { \ buffer[0] = '*'; \ stackTraceParamAdd(FUNCTION_LOG_##typeMacroPrefix##_FORMAT(*param, buffer + 1, STACK_TRACE_PARAM_MAX - 1) + 1); \ } \ } \ while (0) #define FUNCTION_LOG_PARAM_PP(typeMacroPrefix, param) \ do \ { \ char *buffer = stackTraceParamBuffer(#param); \ \ if (param == NULL) \ stackTraceParamAdd(typeToLog(NULL_Z, buffer, STACK_TRACE_PARAM_MAX)); \ else if (*param == NULL) \ stackTraceParamAdd(typeToLog("*null", buffer, STACK_TRACE_PARAM_MAX)); \ else \ { \ buffer[0] = '*'; \ buffer[1] = '*'; \ stackTraceParamAdd(FUNCTION_LOG_##typeMacroPrefix##_FORMAT(**param, buffer + 2, STACK_TRACE_PARAM_MAX - 2) + 2); \ } \ } \ while (0) /*********************************************************************************************************************************** Functions and macros to render various data types ***********************************************************************************************************************************/ // Convert object to a zero-terminated string for logging typedef void (*ObjToLogFormat)(const void *object, StringStatic *debugLog); FN_EXTERN size_t objToLog(const void *object, ObjToLogFormat formatFunc, char *buffer, size_t bufferSize); #define FUNCTION_LOG_OBJECT_FORMAT(object, formatFunc, buffer, bufferSize) \ objToLog(object, (ObjToLogFormat)formatFunc, buffer, bufferSize) // Convert object name to a zero-terminated string for logging FN_EXTERN size_t objNameToLog(const void *object, const char *objectName, char *buffer, size_t bufferSize); // Convert pointer to a zero-terminated string for logging FN_EXTERN size_t ptrToLog(const void *pointer, const char *pointerName, char *buffer, size_t bufferSize); // Convert zero-terminated string for logging FN_INLINE_ALWAYS size_t strzToLog(const char *const string, char *const buffer, const size_t bufferSize) { return (size_t)snprintf(buffer, bufferSize, string == NULL ? "%s" : "\"%s\"", string == NULL ? NULL_Z : string); } // Convert a type name to a zero-terminated string for logging FN_EXTERN size_t typeToLog(const char *typeName, char *buffer, size_t bufferSize); #define FUNCTION_LOG_BOOL_TYPE \ bool #define FUNCTION_LOG_BOOL_FORMAT(value, buffer, bufferSize) \ cvtBoolToZ(value, buffer, bufferSize) #define FUNCTION_LOG_CHAR_TYPE \ char #define FUNCTION_LOG_CHAR_FORMAT(value, buffer, bufferSize) \ cvtCharToZ(value, buffer, bufferSize) #define FUNCTION_LOG_CHARDATA_TYPE \ char #define FUNCTION_LOG_CHARDATA_FORMAT(value, buffer, bufferSize) \ typeToLog("(char)", buffer, bufferSize) #define FUNCTION_LOG_CHARPY_TYPE \ char *[] #define FUNCTION_LOG_CHARPY_FORMAT(value, buffer, bufferSize) \ ptrToLog(value, "char *[]", buffer, bufferSize) #define FUNCTION_LOG_INT_TYPE \ int #define FUNCTION_LOG_INT_FORMAT(value, buffer, bufferSize) \ cvtIntToZ(value, buffer, bufferSize) #define FUNCTION_LOG_INT64_TYPE \ int64_t #define FUNCTION_LOG_INT64_FORMAT(value, buffer, bufferSize) \ cvtInt64ToZ(value, buffer, bufferSize) #define FUNCTION_LOG_ENUM_TYPE \ unsigned int #define FUNCTION_LOG_ENUM_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_UINT_FORMAT(value, buffer, bufferSize) #define FUNCTION_LOG_FUNCTIONP_FORMAT(value, buffer, bufferSize) \ ptrToLog(value == NULL ? NULL : (void *)1, "function *", buffer, bufferSize) #define FUNCTION_LOG_MODE_TYPE \ mode_t #define FUNCTION_LOG_MODE_FORMAT(value, buffer, bufferSize) \ cvtModeToZ(value, buffer, bufferSize) #define FUNCTION_LOG_TIMEMSEC_TYPE \ TimeMSec #define FUNCTION_LOG_TIMEMSEC_FORMAT(value, buffer, bufferSize) \ cvtUInt64ToZ(value, buffer, bufferSize) #define FUNCTION_LOG_BYTEDATA_TYPE \ uint8_t #define FUNCTION_LOG_BYTEDATA_FORMAT(value, buffer, bufferSize) \ typeToLog("(uint8_t)", buffer, bufferSize) #define FUNCTION_LOG_SIZE_TYPE \ size_t #define FUNCTION_LOG_SIZE_FORMAT(value, buffer, bufferSize) \ cvtSizeToZ(value, buffer, bufferSize) #define FUNCTION_LOG_TIME_TYPE \ time_t #define FUNCTION_LOG_TIME_FORMAT(value, buffer, bufferSize) \ cvtTimeToZP("%s", value, buffer, bufferSize) #define FUNCTION_LOG_UINT_TYPE \ unsigned int #define FUNCTION_LOG_UINT_FORMAT(value, buffer, bufferSize) \ cvtUIntToZ(value, buffer, bufferSize) #define FUNCTION_LOG_UINT16_TYPE \ uint16_t #define FUNCTION_LOG_UINT16_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_UINT_FORMAT(value, buffer, bufferSize) #define FUNCTION_LOG_UINT32_TYPE \ uint32_t #define FUNCTION_LOG_UINT32_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_UINT_FORMAT(value, buffer, bufferSize) #define FUNCTION_LOG_UINT64_TYPE \ uint64_t #define FUNCTION_LOG_UINT64_FORMAT(value, buffer, bufferSize) \ cvtUInt64ToZ(value, buffer, bufferSize) #define FUNCTION_LOG_VOID_TYPE \ void #define FUNCTION_LOG_VOID_FORMAT(value, buffer, bufferSize) \ typeToLog("void", buffer, bufferSize) #define FUNCTION_LOG_STRINGZ_TYPE \ char * #define FUNCTION_LOG_STRINGZ_FORMAT(value, buffer, bufferSize) \ strzToLog(value, buffer, bufferSize) #define FUNCTION_LOG_SSIZE_TYPE \ ssize_t #define FUNCTION_LOG_SSIZE_FORMAT(value, buffer, bufferSize) \ cvtInt64ToZ(value, buffer, bufferSize) /*********************************************************************************************************************************** Macros to return function results (or void) ***********************************************************************************************************************************/ #define FUNCTION_LOG_RETURN_BASE(typePre, typeMacroPrefix, typePost, ...) \ do \ { \ typePre FUNCTION_LOG_##typeMacroPrefix##_TYPE typePost FUNCTION_LOG_RETURN_result = __VA_ARGS__; \ \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END(STRINGIFY(FUNCTION_LOG_##typeMacroPrefix##_TYPE)); \ STACK_TRACE_POP(false); \ \ IF_LOG_ANY(FUNCTION_LOG_LEVEL()) \ { \ char buffer[STACK_TRACE_PARAM_MAX]; \ \ FUNCTION_LOG_##typeMacroPrefix##_FORMAT(FUNCTION_LOG_RETURN_result, buffer, sizeof(buffer)); \ LOG_FMT(FUNCTION_LOG_LEVEL(), 0, "=> %s", buffer); \ } \ \ return FUNCTION_LOG_RETURN_result; \ } \ while (0) #define FUNCTION_LOG_RETURN(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(, typeMacroPrefix, , __VA_ARGS__) #define FUNCTION_LOG_RETURN_P(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(, typeMacroPrefix, *, __VA_ARGS__) #define FUNCTION_LOG_RETURN_PP(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(, typeMacroPrefix, **, __VA_ARGS__) #define FUNCTION_LOG_RETURN_CONST(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(const, typeMacroPrefix, , __VA_ARGS__) #define FUNCTION_LOG_RETURN_CONST_P(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(const, typeMacroPrefix, *, __VA_ARGS__) #define FUNCTION_LOG_RETURN_CONST_PP(typeMacroPrefix, ...) \ FUNCTION_LOG_RETURN_BASE(const, typeMacroPrefix, **, __VA_ARGS__) #define FUNCTION_LOG_RETURN_STRUCT(...) \ do \ { \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END("struct"); \ STACK_TRACE_POP(false); \ \ IF_LOG_ANY(FUNCTION_LOG_LEVEL()) \ LOG_FMT(FUNCTION_LOG_LEVEL(), 0, "=> struct"); \ \ return __VA_ARGS__; \ } \ while (0) #define FUNCTION_LOG_RETURN_VOID() \ do \ { \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END("void"); \ STACK_TRACE_POP(false); \ \ LOG(FUNCTION_LOG_LEVEL(), 0, "=> void"); \ return; \ } \ while (0) /*********************************************************************************************************************************** Function Test Macros In debug builds these macros will update the stack trace with function names and parameters but will not log. In production builds all test macros are compiled out (except for return statements). Ignore DEBUG_TEST_TRACE_MACRO if DEBUG is not defined because the underlying functions that support the macros will not be present. ***********************************************************************************************************************************/ #ifdef DEBUG #ifdef DEBUG_TEST_TRACE #define DEBUG_TEST_TRACE_MACRO #endif // DEBUG_TEST_TRACE #endif // DEBUG // Annotate functions that do not return so it is clear why there is no FUNCTION_TEST_RETURN*() macro #define FUNCTION_TEST_NO_RETURN() #ifdef DEBUG_TEST_TRACE_MACRO #define FUNCTION_TEST_BEGIN() \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_BEGIN(); \ \ /* Ensure that FUNCTION_LOG_BEGIN() and FUNCTION_TEST_BEGIN() are not both used in a single function by declaring the */ \ /* same variable that FUNCTION_LOG_BEGIN() uses to track logging */ \ LogLevel FUNCTION_LOG_LEVEL(); \ (void)FUNCTION_LOG_LEVEL(); \ \ /* Ensure that FUNCTION_TEST_RETURN*() is not used with FUNCTION_LOG_BEGIN*() by declaring a variable that will be */ \ /* referenced in FUNCTION_TEST_RETURN*() */ \ bool FUNCTION_TEST_BEGIN_exists; \ \ if (stackTraceTest()) \ { \ STACK_TRACE_PUSH(logLevelDebug); \ stackTraceParamLog(); \ stackTraceTestStop() #define FUNCTION_TEST_PARAM(typeMacroPrefix, param) \ FUNCTION_LOG_PARAM(typeMacroPrefix, param) #define FUNCTION_TEST_PARAM_P(typeName, param) \ FUNCTION_LOG_PARAM_P(typeName, param) #define FUNCTION_TEST_PARAM_PP(typeName, param) \ FUNCTION_LOG_PARAM_PP(typeName, param) #define FUNCTION_TEST_END() \ (void)FUNCTION_TEST_BEGIN_exists; /* CHECK for presence of FUNCTION_TEST_BEGIN*() */ \ \ stackTraceTestStart(); \ } #define FUNCTION_TEST_VOID() \ FUNCTION_TEST_BEGIN(); \ FUNCTION_TEST_END(); #define FUNCTION_TEST_RETURN_TYPE_BASE(typePre, type, typePost, ...) \ do \ { \ (void)FUNCTION_TEST_BEGIN_exists; /* CHECK for presence of FUNCTION_TEST_BEGIN*() */ \ \ typePre type typePost FUNCTION_TEST_result = __VA_ARGS__; \ \ STACK_TRACE_POP(true); \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END(STRINGIFY(type)); \ \ return FUNCTION_TEST_result; \ } \ while (0) #define FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(typePre, typeMacroPrefix, typePost, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(typePre, FUNCTION_LOG_##typeMacroPrefix##_TYPE, typePost, __VA_ARGS__) #define FUNCTION_TEST_RETURN(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(, typeMacroPrefix, , __VA_ARGS__) #define FUNCTION_TEST_RETURN_P(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(, typeMacroPrefix, *, __VA_ARGS__) #define FUNCTION_TEST_RETURN_PP(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(, typeMacroPrefix, **, __VA_ARGS__) #define FUNCTION_TEST_RETURN_CONST(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(const, typeMacroPrefix, , __VA_ARGS__) #define FUNCTION_TEST_RETURN_CONST_P(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(const, typeMacroPrefix, *, __VA_ARGS__) #define FUNCTION_TEST_RETURN_CONST_PP(typeMacroPrefix, ...) \ FUNCTION_TEST_RETURN_TYPE_MACRO_BASE(const, typeMacroPrefix, **, __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(, type, , __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_P(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(, type, *, __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_PP(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(, type, **, __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_CONST(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(const, type, , __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_CONST_P(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(const, type, *, __VA_ARGS__) #define FUNCTION_TEST_RETURN_TYPE_CONST_PP(type, ...) \ FUNCTION_TEST_RETURN_TYPE_BASE(const, type, **, __VA_ARGS__) #define FUNCTION_TEST_RETURN_VOID() \ do \ { \ (void)FUNCTION_TEST_BEGIN_exists; /* CHECK for presence of FUNCTION_TEST_BEGIN*() */ \ \ FUNCTION_TEST_MEM_CONTEXT_AUDIT_END("void"); \ STACK_TRACE_POP(true); \ return; \ } \ while (0) #else #define FUNCTION_TEST_BEGIN() #define FUNCTION_TEST_PARAM(typeMacroPrefix, param) #define FUNCTION_TEST_PARAM_P(typeMacroPrefix, param) #define FUNCTION_TEST_PARAM_PP(typeMacroPrefix, param) #define FUNCTION_TEST_END() #define FUNCTION_TEST_VOID() #define FUNCTION_TEST_RETURN(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_P(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_PP(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_CONST(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_CONST_P(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_CONST_PP(typeMacroPrefix, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_P(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_PP(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_CONST(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_CONST_P(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_TYPE_CONST_PP(type, ...) \ return __VA_ARGS__ #define FUNCTION_TEST_RETURN_VOID() \ return #endif // DEBUG_TEST_TRACE_MACRO #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/encode.c0000664000175100017510000005065115240547630017102 0ustar00runnerrunner/*********************************************************************************************************************************** Binary to String Encode/Decode ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/encode.h" /*********************************************************************************************************************************** Assert that encoding type is valid. This needs to be kept up to date with the last item in the enum. ***********************************************************************************************************************************/ #define ASSERT_ENCODE_TYPE_VALID(type) \ ASSERT(type <= encodingHex); /*********************************************************************************************************************************** Base64 encoding/decoding ***********************************************************************************************************************************/ VR_NON_STRING static const char encodeBase64Lookup[64] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"; static void encodeToStrBase64(const uint8_t *const source, const size_t sourceSize, char *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(BYTEDATA, source); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_PARAM_P(CHARDATA, destination); FUNCTION_TEST_END(); ASSERT(source != NULL); ASSERT(destination != NULL); unsigned int destinationIdx = 0; // Encode the string from three bytes to four characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx += 3) { // First encoded character is always used completely destination[destinationIdx++] = encodeBase64Lookup[source[sourceIdx] >> 2]; // If there is only one byte to encode then the second encoded character is only partly used and the third and fourth // encoded characters are padded. if (sourceSize - sourceIdx == 1) { destination[destinationIdx++] = encodeBase64Lookup[(source[sourceIdx] & 0x03) << 4]; destination[destinationIdx++] = 0x3d; destination[destinationIdx++] = 0x3d; } // Else if more than one byte to encode else { // If there is more than one byte to encode then the second encoded character is used completely destination[destinationIdx++] = encodeBase64Lookup[((source[sourceIdx] & 0x03) << 4) | ((source[sourceIdx + 1] & 0xf0) >> 4)]; // If there are only two bytes to encode then the third encoded character is only partly used and the fourth encoded // character is padded. if (sourceSize - sourceIdx == 2) { destination[destinationIdx++] = encodeBase64Lookup[(source[sourceIdx + 1] & 0x0f) << 2]; destination[destinationIdx++] = 0x3d; } // Else the third and fourth encoded characters are used completely else { destination[destinationIdx++] = encodeBase64Lookup[((source[sourceIdx + 1] & 0x0f) << 2) | ((source[sourceIdx + 2] & 0xc0) >> 6)]; destination[destinationIdx++] = encodeBase64Lookup[source[sourceIdx + 2] & 0x3f]; } } } // Zero-terminate the string destination[destinationIdx] = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t encodeToStrSizeBase64(const size_t sourceSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_END(); // Four characters are needed to encode each 3 byte group (plus up to two bytes of padding) FUNCTION_TEST_RETURN(SIZE, (sourceSize + 2) / 3 * 4); } /**********************************************************************************************************************************/ static const int8_t decodeBase64Lookup[256] = { -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 62, -1, -1, -1, 63, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, -1, -1, -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, -1, -1, -1, -1, -1, -1, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, }; static void decodeToBinValidateBase64(const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); // Check for the correct length const size_t sourceSize = strlen(source); if (sourceSize % 4 != 0) THROW_FMT(FormatError, "base64 size %zu is not evenly divisible by 4", sourceSize); // Check all characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx++) { // Check terminators if (source[sourceIdx] == 0x3d) { // Make sure they are only in the last two positions if (sourceIdx < sourceSize - 2) THROW(FormatError, "base64 '=' character may only appear in last two positions"); // If second to last char is = then last char must also be if (sourceIdx == sourceSize - 2 && source[sourceSize - 1] != 0x3d) THROW(FormatError, "base64 last character must be '=' if second to last is"); } else { // Error on any invalid characters if (decodeBase64Lookup[(int)source[sourceIdx]] == -1) THROW_FMT(FormatError, "base64 invalid character found at position %u", sourceIdx); } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static void decodeToBinBase64(const char *const source, uint8_t *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_PARAM_P(BYTEDATA, destination); FUNCTION_TEST_END(); // Validate encoded string decodeToBinValidateBase64(source); int destinationIdx = 0; // Decode the binary data from four characters to three bytes for (unsigned int sourceIdx = 0; sourceIdx < strlen(source); sourceIdx += 4) { // Always decode the first character destination[destinationIdx++] = (uint8_t)(decodeBase64Lookup[(int)source[sourceIdx]] << 2 | decodeBase64Lookup[(int)source[sourceIdx + 1]] >> 4); // Second character is optional if (source[sourceIdx + 2] != 0x3d) { destination[destinationIdx++] = (uint8_t) ((decodeBase64Lookup[(int)source[sourceIdx + 1]] << 4) | (decodeBase64Lookup[(int)source[sourceIdx + 2]] >> 2)); } // Third character is optional if (source[sourceIdx + 3] != 0x3d) { destination[destinationIdx++] = (uint8_t) (((decodeBase64Lookup[(int)source[sourceIdx + 2]] << 6) & 0xc0) | decodeBase64Lookup[(int)source[sourceIdx + 3]]); } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t decodeToBinSizeBase64(const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); // Validate encoded string decodeToBinValidateBase64(source); // Start with size calculated directly from source length const size_t sourceSize = strlen(source); size_t destinationSize = sourceSize / 4 * 3; // Subtract last character if it is not present if (source[sourceSize - 1] == 0x3d) { destinationSize--; // Subtract second to last character if it is not present if (source[sourceSize - 2] == 0x3d) destinationSize--; } FUNCTION_TEST_RETURN(SIZE, destinationSize); } /*********************************************************************************************************************************** Base64Url encoding ***********************************************************************************************************************************/ VR_NON_STRING static const char encodeBase64LookupUrl[64] = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_"; static void encodeToStrBase64Url(const uint8_t *const source, const size_t sourceSize, char *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(BYTEDATA, source); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_PARAM_P(CHARDATA, destination); FUNCTION_TEST_END(); ASSERT(source != NULL); ASSERT(destination != NULL); unsigned int destinationIdx = 0; // Encode the string from three bytes to four characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx += 3) { // First encoded character is always used completely destination[destinationIdx++] = encodeBase64LookupUrl[source[sourceIdx] >> 2]; // If there is only one byte to encode then the second encoded character is only partly used if (sourceSize - sourceIdx == 1) { destination[destinationIdx++] = encodeBase64LookupUrl[(source[sourceIdx] & 0x03) << 4]; } // Else if more than one byte to encode else { // If there is more than one byte to encode then the second encoded character is used completely destination[destinationIdx++] = encodeBase64LookupUrl[((source[sourceIdx] & 0x03) << 4) | ((source[sourceIdx + 1] & 0xf0) >> 4)]; // If there are only two bytes to encode then the third encoded character is only partly used if (sourceSize - sourceIdx == 2) { destination[destinationIdx++] = encodeBase64LookupUrl[(source[sourceIdx + 1] & 0x0f) << 2]; } // Else the third and fourth encoded characters are used completely else { destination[destinationIdx++] = encodeBase64LookupUrl[((source[sourceIdx + 1] & 0x0f) << 2) | ((source[sourceIdx + 2] & 0xc0) >> 6)]; destination[destinationIdx++] = encodeBase64LookupUrl[source[sourceIdx + 2] & 0x3f]; } } } // Zero-terminate the string destination[destinationIdx] = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t encodeToStrSizeBase64Url(const size_t sourceSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_END(); // Four characters are needed to encode each 3 byte group, three characters for 2 bytes, and two characters for 1 byte FUNCTION_TEST_RETURN(SIZE, sourceSize / 3 * 4 + (sourceSize % 3 == 0 ? 0 : sourceSize % 3 + 1)); } /*********************************************************************************************************************************** Hex encoding/decoding ***********************************************************************************************************************************/ VR_NON_STRING static const char encodeHexLookup[512] = "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f" "202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f" "404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f" "606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f" "808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9f" "a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf" "c0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedf" "e0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff"; static void encodeToStrHex(const uint8_t *const source, const size_t sourceSize, char *destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(BYTEDATA, source); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_PARAM_P(CHARDATA, destination); FUNCTION_TEST_END(); ASSERT(source != NULL); ASSERT(destination != NULL); // Encode the string from one bytes to two characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx += 1) { memcpy(destination, &encodeHexLookup[source[sourceIdx] * 2], 2); destination += 2; } // Zero-terminate the string *destination = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t encodeToStrSizeHex(const size_t sourceSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(SIZE, sourceSize * 2); } /**********************************************************************************************************************************/ // {uncrustify_off - array alignment} static const int8_t decodeHexLookup[256] = { -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, -1, -1, -1, -1, -1, -1, -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 10, 11, 12, 13, 14, 15, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, }; // {uncrustify_on} static void decodeToBinValidateHex(const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); // Check for the correct length const size_t sourceSize = strlen(source); if (sourceSize % 2 != 0) THROW_FMT(FormatError, "hex size %zu is not evenly divisible by 2", sourceSize); // Check all characters for (unsigned int sourceIdx = 0; sourceIdx < sourceSize; sourceIdx++) { if (decodeHexLookup[(int)source[sourceIdx]] == -1) THROW_FMT(FormatError, "hex invalid character found at position %u", sourceIdx); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static void decodeToBinHex(const char *const source, uint8_t *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_PARAM_P(BYTEDATA, destination); FUNCTION_TEST_END(); // Validate encoded string decodeToBinValidateHex(source); int destinationIdx = 0; // Decode the binary data from two characters to one byte for (unsigned int sourceIdx = 0; sourceIdx < strlen(source); sourceIdx += 2) { destination[destinationIdx++] = (uint8_t)(decodeHexLookup[(int)source[sourceIdx]] << 4 | decodeHexLookup[(int)source[sourceIdx + 1]]); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static size_t decodeToBinSizeHex(const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); // Validate encoded string decodeToBinValidateHex(source); FUNCTION_TEST_RETURN(SIZE, strlen(source) / 2); } /*********************************************************************************************************************************** Generic encoding/decoding ***********************************************************************************************************************************/ FN_EXTERN void encodeToStr(const EncodingType type, const uint8_t *const source, const size_t sourceSize, char *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM_P(BYTEDATA, source); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_PARAM_P(CHARDATA, destination); FUNCTION_TEST_END(); ASSERT_ENCODE_TYPE_VALID(type); switch (type) { case encodingBase64: encodeToStrBase64(source, sourceSize, destination); break; case encodingBase64Url: encodeToStrBase64Url(source, sourceSize, destination); break; case encodingHex: encodeToStrHex(source, sourceSize, destination); break; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN size_t encodeToStrSize(const EncodingType type, const size_t sourceSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(SIZE, sourceSize); FUNCTION_TEST_END(); ASSERT_ENCODE_TYPE_VALID(type); size_t destinationSize = SIZE_MAX; switch (type) { case encodingBase64: destinationSize = encodeToStrSizeBase64(sourceSize); break; case encodingBase64Url: destinationSize = encodeToStrSizeBase64Url(sourceSize); break; case encodingHex: destinationSize = encodeToStrSizeHex(sourceSize); break; } FUNCTION_TEST_RETURN(SIZE, destinationSize); } /**********************************************************************************************************************************/ FN_EXTERN void decodeToBin(const EncodingType type, const char *const source, uint8_t *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_PARAM_P(BYTEDATA, destination); FUNCTION_TEST_END(); ASSERT_ENCODE_TYPE_VALID(type); switch (type) { case encodingBase64: decodeToBinBase64(source, destination); break; case encodingHex: decodeToBinHex(source, destination); break; default: ASSERT_MSG("unsupported"); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN size_t decodeToBinSize(const EncodingType type, const char *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(STRINGZ, source); FUNCTION_TEST_END(); ASSERT_ENCODE_TYPE_VALID(type); size_t destinationSize = SIZE_MAX; switch (type) { case encodingBase64: destinationSize = decodeToBinSizeBase64(source); break; case encodingHex: destinationSize = decodeToBinSizeHex(source); break; default: ASSERT_MSG("unsupported"); } FUNCTION_TEST_RETURN(SIZE, destinationSize); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/encode.h0000664000175100017510000000301415240547630017076 0ustar00runnerrunner/*********************************************************************************************************************************** Binary to String Encode/Decode ***********************************************************************************************************************************/ #ifndef COMMON_ENCODE_H #define COMMON_ENCODE_H #include /*********************************************************************************************************************************** Encoding types ***********************************************************************************************************************************/ typedef enum { encodingBase64, encodingBase64Url, encodingHex, } EncodingType; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Encode binary data to a printable string FN_EXTERN void encodeToStr(EncodingType type, const uint8_t *source, size_t sourceSize, char *destination); // Size of the string returned by encodeToStr() FN_EXTERN size_t encodeToStrSize(EncodingType type, size_t sourceSize); // Decode a string to binary data FN_EXTERN void decodeToBin(EncodingType type, const char *source, uint8_t *destination); // Size of the binary data returned by decodeToBin() FN_EXTERN size_t decodeToBinSize(EncodingType type, const char *source); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/error/0000775000175100017510000000000015240547630016623 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/error/error.auto.c.inc0000664000175100017510000001463115240547630021644 0ustar00runnerrunner/*********************************************************************************************************************************** Error Type Definition Automatically generated by 'build-code error' -- do not modify directly. ***********************************************************************************************************************************/ /*********************************************************************************************************************************** Error type definitions ***********************************************************************************************************************************/ ERROR_DEFINE( 25, AssertError, true, RuntimeError); ERROR_DEFINE( 26, ChecksumError, false, RuntimeError); ERROR_DEFINE( 27, ConfigError, false, RuntimeError); ERROR_DEFINE( 28, FileInvalidError, false, RuntimeError); ERROR_DEFINE( 29, FormatError, false, RuntimeError); ERROR_DEFINE( 30, CommandRequiredError, false, RuntimeError); ERROR_DEFINE( 31, OptionInvalidError, false, RuntimeError); ERROR_DEFINE( 32, OptionInvalidValueError, false, RuntimeError); ERROR_DEFINE( 37, OptionRequiredError, false, RuntimeError); ERROR_DEFINE( 38, PgRunningError, false, RuntimeError); ERROR_DEFINE( 39, ProtocolError, false, RuntimeError); ERROR_DEFINE( 40, PathNotEmptyError, false, RuntimeError); ERROR_DEFINE( 41, FileOpenError, false, RuntimeError); ERROR_DEFINE( 42, FileReadError, false, RuntimeError); ERROR_DEFINE( 43, ParamRequiredError, false, RuntimeError); ERROR_DEFINE( 44, ArchiveMismatchError, false, RuntimeError); ERROR_DEFINE( 45, ArchiveDuplicateError, false, RuntimeError); ERROR_DEFINE( 46, VersionNotSupportedError, false, RuntimeError); ERROR_DEFINE( 47, PathCreateError, false, RuntimeError); ERROR_DEFINE( 48, CommandInvalidError, false, RuntimeError); ERROR_DEFINE( 49, HostConnectError, false, RuntimeError); ERROR_DEFINE( 50, LockAcquireError, false, RuntimeError); ERROR_DEFINE( 51, BackupMismatchError, false, RuntimeError); ERROR_DEFINE( 52, FileSyncError, false, RuntimeError); ERROR_DEFINE( 53, PathOpenError, false, RuntimeError); ERROR_DEFINE( 54, PathSyncError, false, RuntimeError); ERROR_DEFINE( 55, FileMissingError, false, RuntimeError); ERROR_DEFINE( 56, DbConnectError, false, RuntimeError); ERROR_DEFINE( 57, DbQueryError, false, RuntimeError); ERROR_DEFINE( 58, DbMismatchError, false, RuntimeError); ERROR_DEFINE( 60, FileRemoveError, false, RuntimeError); ERROR_DEFINE( 61, PathRemoveError, false, RuntimeError); ERROR_DEFINE( 62, StopError, false, RuntimeError); ERROR_DEFINE( 63, TermError, false, RuntimeError); ERROR_DEFINE( 64, FileWriteError, false, RuntimeError); ERROR_DEFINE( 67, FeatureNotSupportedError, false, RuntimeError); ERROR_DEFINE( 68, ArchiveCommandInvalidError, false, RuntimeError); ERROR_DEFINE( 69, LinkExpectedError, false, RuntimeError); ERROR_DEFINE( 70, LinkDestinationError, false, RuntimeError); ERROR_DEFINE( 72, HostInvalidError, false, RuntimeError); ERROR_DEFINE( 73, PathMissingError, false, RuntimeError); ERROR_DEFINE( 74, FileMoveError, false, RuntimeError); ERROR_DEFINE( 75, BackupSetInvalidError, false, RuntimeError); ERROR_DEFINE( 76, TablespaceMapError, false, RuntimeError); ERROR_DEFINE( 78, LinkMapError, false, RuntimeError); ERROR_DEFINE( 79, FileCloseError, false, RuntimeError); ERROR_DEFINE( 80, DbMissingError, false, RuntimeError); ERROR_DEFINE( 81, DbInvalidError, false, RuntimeError); ERROR_DEFINE( 82, ArchiveTimeoutError, false, RuntimeError); ERROR_DEFINE( 83, FileModeError, false, RuntimeError); ERROR_DEFINE( 87, ArchiveDisabledError, false, RuntimeError); ERROR_DEFINE( 88, FileOwnerError, false, RuntimeError); ERROR_DEFINE( 89, UserMissingError, false, RuntimeError); ERROR_DEFINE( 93, FileExistsError, false, RuntimeError); ERROR_DEFINE( 94, MemoryError, true, RuntimeError); ERROR_DEFINE( 95, CryptoError, false, RuntimeError); ERROR_DEFINE( 96, ParamInvalidError, false, RuntimeError); ERROR_DEFINE( 97, PathCloseError, false, RuntimeError); ERROR_DEFINE( 98, FileInfoError, false, RuntimeError); ERROR_DEFINE( 99, JsonFormatError, false, RuntimeError); ERROR_DEFINE(100, KernelError, false, RuntimeError); ERROR_DEFINE(101, ServiceError, false, RuntimeError); ERROR_DEFINE(102, ExecuteError, false, RuntimeError); ERROR_DEFINE(103, RepoInvalidError, false, RuntimeError); ERROR_DEFINE(104, CommandError, false, RuntimeError); ERROR_DEFINE(105, AccessError, false, RuntimeError); ERROR_DEFINE(106, ClockError, false, RuntimeError); ERROR_DEFINE(122, RuntimeError, false, RuntimeError); ERROR_DEFINE(124, UnhandledError, false, RuntimeError); ERROR_DEFINE(125, UnknownError, false, RuntimeError); /*********************************************************************************************************************************** Error type array ***********************************************************************************************************************************/ static const ErrorType *const errorTypeList[] = { &AssertError, &ChecksumError, &ConfigError, &FileInvalidError, &FormatError, &CommandRequiredError, &OptionInvalidError, &OptionInvalidValueError, &OptionRequiredError, &PgRunningError, &ProtocolError, &PathNotEmptyError, &FileOpenError, &FileReadError, &ParamRequiredError, &ArchiveMismatchError, &ArchiveDuplicateError, &VersionNotSupportedError, &PathCreateError, &CommandInvalidError, &HostConnectError, &LockAcquireError, &BackupMismatchError, &FileSyncError, &PathOpenError, &PathSyncError, &FileMissingError, &DbConnectError, &DbQueryError, &DbMismatchError, &FileRemoveError, &PathRemoveError, &StopError, &TermError, &FileWriteError, &FeatureNotSupportedError, &ArchiveCommandInvalidError, &LinkExpectedError, &LinkDestinationError, &HostInvalidError, &PathMissingError, &FileMoveError, &BackupSetInvalidError, &TablespaceMapError, &LinkMapError, &FileCloseError, &DbMissingError, &DbInvalidError, &ArchiveTimeoutError, &FileModeError, &ArchiveDisabledError, &FileOwnerError, &UserMissingError, &FileExistsError, &MemoryError, &CryptoError, &ParamInvalidError, &PathCloseError, &FileInfoError, &JsonFormatError, &KernelError, &ServiceError, &ExecuteError, &RepoInvalidError, &CommandError, &AccessError, &ClockError, &RuntimeError, &UnhandledError, &UnknownError, NULL, }; ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/error/error.auto.h0000664000175100017510000000564615240547630021107 0ustar00runnerrunner/*********************************************************************************************************************************** Error Type Definition Automatically generated by 'build-code error' -- do not modify directly. ***********************************************************************************************************************************/ #ifndef COMMON_ERROR_ERROR_AUTO_H #define COMMON_ERROR_ERROR_AUTO_H /*********************************************************************************************************************************** Error type declarations ***********************************************************************************************************************************/ ERROR_DECLARE(AssertError); ERROR_DECLARE(ChecksumError); ERROR_DECLARE(ConfigError); ERROR_DECLARE(FileInvalidError); ERROR_DECLARE(FormatError); ERROR_DECLARE(CommandRequiredError); ERROR_DECLARE(OptionInvalidError); ERROR_DECLARE(OptionInvalidValueError); ERROR_DECLARE(OptionRequiredError); ERROR_DECLARE(PgRunningError); ERROR_DECLARE(ProtocolError); ERROR_DECLARE(PathNotEmptyError); ERROR_DECLARE(FileOpenError); ERROR_DECLARE(FileReadError); ERROR_DECLARE(ParamRequiredError); ERROR_DECLARE(ArchiveMismatchError); ERROR_DECLARE(ArchiveDuplicateError); ERROR_DECLARE(VersionNotSupportedError); ERROR_DECLARE(PathCreateError); ERROR_DECLARE(CommandInvalidError); ERROR_DECLARE(HostConnectError); ERROR_DECLARE(LockAcquireError); ERROR_DECLARE(BackupMismatchError); ERROR_DECLARE(FileSyncError); ERROR_DECLARE(PathOpenError); ERROR_DECLARE(PathSyncError); ERROR_DECLARE(FileMissingError); ERROR_DECLARE(DbConnectError); ERROR_DECLARE(DbQueryError); ERROR_DECLARE(DbMismatchError); ERROR_DECLARE(FileRemoveError); ERROR_DECLARE(PathRemoveError); ERROR_DECLARE(StopError); ERROR_DECLARE(TermError); ERROR_DECLARE(FileWriteError); ERROR_DECLARE(FeatureNotSupportedError); ERROR_DECLARE(ArchiveCommandInvalidError); ERROR_DECLARE(LinkExpectedError); ERROR_DECLARE(LinkDestinationError); ERROR_DECLARE(HostInvalidError); ERROR_DECLARE(PathMissingError); ERROR_DECLARE(FileMoveError); ERROR_DECLARE(BackupSetInvalidError); ERROR_DECLARE(TablespaceMapError); ERROR_DECLARE(LinkMapError); ERROR_DECLARE(FileCloseError); ERROR_DECLARE(DbMissingError); ERROR_DECLARE(DbInvalidError); ERROR_DECLARE(ArchiveTimeoutError); ERROR_DECLARE(FileModeError); ERROR_DECLARE(ArchiveDisabledError); ERROR_DECLARE(FileOwnerError); ERROR_DECLARE(UserMissingError); ERROR_DECLARE(FileExistsError); ERROR_DECLARE(MemoryError); ERROR_DECLARE(CryptoError); ERROR_DECLARE(ParamInvalidError); ERROR_DECLARE(PathCloseError); ERROR_DECLARE(FileInfoError); ERROR_DECLARE(JsonFormatError); ERROR_DECLARE(KernelError); ERROR_DECLARE(ServiceError); ERROR_DECLARE(ExecuteError); ERROR_DECLARE(RepoInvalidError); ERROR_DECLARE(CommandError); ERROR_DECLARE(AccessError); ERROR_DECLARE(ClockError); ERROR_DECLARE(RuntimeError); ERROR_DECLARE(UnhandledError); ERROR_DECLARE(UnknownError); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/error/error.c0000664000175100017510000004327115240547630020127 0ustar00runnerrunner/*********************************************************************************************************************************** Error Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/error/error.h" #include "common/macro.h" #include "common/stackTrace.h" /*********************************************************************************************************************************** Represents an error type ***********************************************************************************************************************************/ struct ErrorType { const int code; const bool fatal; const char *name; const struct ErrorType *parentType; }; // Macro for defining new error types #define ERROR_DEFINE(code, name, fatal, parentType) \ VR_EXTERN_DEFINE const ErrorType name = {code, fatal, #name, &parentType} // Define test error #ifdef DEBUG ERROR_DEFINE(1, TestError, false, RuntimeError); #endif // Include error type definitions #include "common/error/error.auto.c.inc" /*********************************************************************************************************************************** Maximum allowed number of nested try blocks ***********************************************************************************************************************************/ #define ERROR_TRY_MAX 64 /*********************************************************************************************************************************** States for each try ***********************************************************************************************************************************/ typedef enum {errorStateTry, errorStateCatch, errorStateEnd} ErrorState; /*********************************************************************************************************************************** Track error handling ***********************************************************************************************************************************/ typedef struct Error { const ErrorType *errorType; // Error type const char *fileName; // Source file where the error occurred const char *functionName; // Function where the error occurred int fileLine; // Source file line where the error occurred const char *message; // Description of the error const char *stackTrace; // Stack trace } Error; static struct { // Array of jump buffers jmp_buf jumpList[ERROR_TRY_MAX]; // Handler list const ErrorHandlerFunction *handlerList; unsigned int handlerTotal; // State of each try int tryTotal; struct { ErrorState state; bool uncaught; } tryList[ERROR_TRY_MAX + 1]; // Last error Error error; } errorContext; /*********************************************************************************************************************************** Message buffer and buffer size The message buffer is statically allocated so there is some space to store error messages. Not being able to allocate such a small amount of memory seems pretty unlikely so just keep the code simple and let the loader deal with massively constrained memory situations. The temp buffer is required because the error message being passed might be the error already stored in the message buffer. ***********************************************************************************************************************************/ #ifndef ERROR_MESSAGE_BUFFER_SIZE #define ERROR_MESSAGE_BUFFER_SIZE 8192 #endif static char messageBuffer[ERROR_MESSAGE_BUFFER_SIZE]; static char messageBufferTemp[ERROR_MESSAGE_BUFFER_SIZE]; static char stackTraceBuffer[ERROR_MESSAGE_BUFFER_SIZE]; /**********************************************************************************************************************************/ FN_EXTERN void errorHandlerSet(const ErrorHandlerFunction *const list, const unsigned int total) { assert(total == 0 || list != NULL); errorContext.handlerList = list; errorContext.handlerTotal = total; } /**********************************************************************************************************************************/ FN_EXTERN int errorTypeCode(const ErrorType *const errorType) { return errorType->code; } /**********************************************************************************************************************************/ FN_EXTERN bool errorTypeFatal(const ErrorType *const errorType) { return errorType->fatal; } /**********************************************************************************************************************************/ FN_EXTERN const ErrorType * errorTypeFromCode(const int code) { // Search for error type by code int errorTypeIdx = 0; const ErrorType *result = errorTypeList[errorTypeIdx]; while (result != NULL) { if (result->code == code) break; result = errorTypeList[++errorTypeIdx]; } // Error if type was not found if (result == NULL) result = &UnknownError; return result; } /**********************************************************************************************************************************/ FN_EXTERN const char * errorTypeName(const ErrorType *const errorType) { return errorType->name; } /**********************************************************************************************************************************/ FN_EXTERN const ErrorType * errorTypeParent(const ErrorType *const errorType) { return errorType->parentType; } /**********************************************************************************************************************************/ FN_EXTERN unsigned int errorTryDepth(void) { return (unsigned int)errorContext.tryTotal; } /**********************************************************************************************************************************/ FN_EXTERN bool errorTypeExtends(const ErrorType *const child, const ErrorType *const parent) { const ErrorType *find = child; do { find = errorTypeParent(find); // Parent was found if (find == parent) return true; } while (find != errorTypeParent(find)); // Parent was not found return false; } /**********************************************************************************************************************************/ FN_EXTERN const ErrorType * errorType(void) { assert(errorContext.error.errorType != NULL); return errorContext.error.errorType; } /**********************************************************************************************************************************/ FN_EXTERN int errorCode(void) { return errorTypeCode(errorType()); } /**********************************************************************************************************************************/ FN_EXTERN bool errorFatal(void) { return errorTypeFatal(errorType()); } /**********************************************************************************************************************************/ FN_EXTERN const char * errorFileName(void) { assert(errorContext.error.fileName != NULL); return errorContext.error.fileName; } /**********************************************************************************************************************************/ FN_EXTERN const char * errorFunctionName(void) { assert(errorContext.error.functionName != NULL); return errorContext.error.functionName; } /**********************************************************************************************************************************/ FN_EXTERN int errorFileLine(void) { assert(errorContext.error.fileLine != 0); return errorContext.error.fileLine; } /**********************************************************************************************************************************/ FN_EXTERN const char * errorMessage(void) { assert(errorContext.error.message != NULL); return errorContext.error.message; } /**********************************************************************************************************************************/ FN_EXTERN const char * errorName(void) { return errorTypeName(errorType()); } /**********************************************************************************************************************************/ FN_EXTERN const char * errorStackTrace(void) { assert(errorContext.error.stackTrace != NULL); return errorContext.error.stackTrace; } /**********************************************************************************************************************************/ FN_EXTERN bool errorInstanceOf(const ErrorType *const errorTypeTest) { return errorType() == errorTypeTest || errorTypeExtends(errorType(), errorTypeTest); } /*********************************************************************************************************************************** Return current error context state ***********************************************************************************************************************************/ static ErrorState errorInternalState(void) { return errorContext.tryList[errorContext.tryTotal].state; } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalTryBegin(const char *const fileName, const char *const functionName, const int fileLine) { // If try total has been exceeded then throw an error if (errorContext.tryTotal >= ERROR_TRY_MAX) errorInternalThrowFmt(&AssertError, fileName, functionName, fileLine, "too many nested try blocks"); // Increment try total errorContext.tryTotal++; // Setup try errorContext.tryList[errorContext.tryTotal].state = errorStateTry; errorContext.tryList[errorContext.tryTotal].uncaught = false; } /**********************************************************************************************************************************/ FN_EXTERN jmp_buf * errorInternalJump(void) { return &errorContext.jumpList[errorContext.tryTotal - 1]; } /**********************************************************************************************************************************/ FN_EXTERN bool errorInternalCatch(const ErrorType *const errorTypeCatch, const bool fatalCatch) { assert(fatalCatch || !errorTypeFatal(errorTypeCatch)); // If just entering error state clean up the stack if (errorInternalState() == errorStateTry) { for (unsigned int handlerIdx = 0; handlerIdx < errorContext.handlerTotal; handlerIdx++) errorContext.handlerList[handlerIdx](errorTryDepth(), fatalCatch); errorContext.tryList[errorContext.tryTotal].state++; } if (errorInternalState() == errorStateCatch && errorInstanceOf(errorTypeCatch) && (fatalCatch || !errorFatal())) { errorContext.tryList[errorContext.tryTotal].uncaught = false; errorContext.tryList[errorContext.tryTotal].state++; return true; } return false; } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalPropagate(void) { assert(errorType() != NULL); // Mark the error as uncaught errorContext.tryList[errorContext.tryTotal].uncaught = true; // If there is a parent try then jump to it if (errorContext.tryTotal > 0) longjmp(errorContext.jumpList[errorContext.tryTotal - 1], 1); // If there was no try to catch this error then output to stderr fprintf(stderr, "\nUncaught %s: %s\n thrown at %s:%d\n\n", errorName(), errorMessage(), errorFileName(), errorFileLine()); fflush(stderr); // Exit with failure exit(UnhandledError.code); } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalTryEnd(void) { // Any catch blocks have been processed and none of them called RETHROW() so clear the error if (errorInternalState() == errorStateEnd && !errorContext.tryList[errorContext.tryTotal].uncaught) errorContext.error = (Error){0}; // Remove the try errorContext.tryTotal--; // If not caught in the last try then propagate if (errorContext.tryList[errorContext.tryTotal + 1].uncaught) errorInternalPropagate(); } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalThrow( const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const message, const char *const stackTrace) { // Setup error data errorContext.error.errorType = errorType; errorContext.error.fileName = fileName; errorContext.error.functionName = functionName; errorContext.error.fileLine = fileLine; // Assign message to the error. If errorMessage() is passed as the message there is no need to make a copy. if (message != messageBuffer) snprintf(messageBuffer, sizeof(messageBuffer), "%s", message); errorContext.error.message = (const char *)messageBuffer; // If a stack trace was provided if (stackTrace != NULL) snprintf(stackTraceBuffer, sizeof(stackTraceBuffer), "%s", stackTrace); // Else generate the stack trace for the error else if ( stackTraceToZ( stackTraceBuffer, sizeof(stackTraceBuffer), errorFileName(), errorFunctionName(), (unsigned int)fileLine) >= sizeof(stackTraceBuffer)) { // Indicate that the stack trace was truncated } errorContext.error.stackTrace = (const char *)stackTraceBuffer; // Propagate the error errorInternalPropagate(); } FN_EXTERN void errorInternalThrowFmt( const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const format, ...) { // Format message va_list argument; va_start(argument, format); vsnprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, format, argument); va_end(argument); errorInternalThrow(errorType, fileName, functionName, fileLine, messageBufferTemp, NULL); } /**********************************************************************************************************************************/ FN_EXTERN void errorInternalThrowSys( const int errNo, const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const message) { // Format message with system message appended if (errNo == 0) snprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, "%s", message); else snprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, "%s: [%d] %s", message, errNo, strerror(errNo)); errorInternalThrow(errorType, fileName, functionName, fileLine, messageBufferTemp, NULL); } #ifdef DEBUG_COVERAGE FN_EXTERN void errorInternalThrowOnSys( const bool error, const int errNo, const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const message) { if (error) errorInternalThrowSys(errNo, errorType, fileName, functionName, fileLine, message); } #endif FN_EXTERN void errorInternalThrowSysFmt( const int errNo, const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const format, ...) { // Format message va_list argument; va_start(argument, format); size_t messageSize = (size_t)vsnprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, format, argument); va_end(argument); // vsnprintf() returns the untruncated length; limit it so the appended system message stays in bounds if (messageSize > ERROR_MESSAGE_BUFFER_SIZE - 1) messageSize = ERROR_MESSAGE_BUFFER_SIZE - 1; // Append the system message if (errNo != 0) snprintf(messageBufferTemp + messageSize, ERROR_MESSAGE_BUFFER_SIZE - messageSize, ": [%d] %s", errNo, strerror(errNo)); errorInternalThrow(errorType, fileName, functionName, fileLine, messageBufferTemp, NULL); } #ifdef DEBUG_COVERAGE FN_EXTERN void errorInternalThrowOnSysFmt( const bool error, const int errNo, const ErrorType *const errorType, const char *const fileName, const char *const functionName, const int fileLine, const char *const format, ...) { if (error) { // Format message va_list argument; va_start(argument, format); size_t messageSize = (size_t)vsnprintf(messageBufferTemp, ERROR_MESSAGE_BUFFER_SIZE, format, argument); va_end(argument); // vsnprintf() returns the untruncated length; limit it so the appended system message stays in bounds if (messageSize > ERROR_MESSAGE_BUFFER_SIZE - 1) messageSize = ERROR_MESSAGE_BUFFER_SIZE - 1; // Append the system message if (errNo != 0) { snprintf( messageBufferTemp + messageSize, ERROR_MESSAGE_BUFFER_SIZE - messageSize, ": [%d] %s", errNo, strerror(errNo)); } errorInternalThrow(errorType, fileName, functionName, fileLine, messageBufferTemp, NULL); } } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/error/error.h0000664000175100017510000005707115240547630020137 0ustar00runnerrunner/*********************************************************************************************************************************** Error Handler Implement a try ... catch ... finally error handler. TRY_BEGIN() { } CATCH(Error1) { } CATCH(Error2) { } CATCH_ANY() { } FINALLY() { } TRY_END(); The CATCH() and FINALLY() blocks are optional but at least one must be specified. There is no need for a TRY block by itself because errors will automatically be propagated to the nearest try block in the call stack. IMPORTANT: If a local variable of the function containing a TRY block is modified in the TRY_BEGIN() block and used later in the function after an error is thrown, that variable must be declared "volatile" if the preserving the value is important. Beware that gcc's -Wclobbered warnings are almost entirely useless for catching such issues. IMPORTANT: Never call return from within any of the error-handling blocks. ***********************************************************************************************************************************/ #ifndef COMMON_ERROR_ERROR_H #define COMMON_ERROR_ERROR_H #include #include #include /*********************************************************************************************************************************** Error type object ***********************************************************************************************************************************/ typedef struct ErrorType ErrorType; // Macro for declaring new error types #define ERROR_DECLARE(name) \ VR_EXTERN_DECLARE const ErrorType name // Include error type declarations #include "common/error/error.auto.h" // Declare test error #ifdef DEBUG ERROR_DECLARE(TestError); #else // Must always be defined since it might be needed to compile (though not used) during profiling #define TestError AssertError #endif /*********************************************************************************************************************************** Functions to get information about a generic error type ***********************************************************************************************************************************/ // Error type code FN_EXTERN int errorTypeCode(const ErrorType *errorType); // Is the error type fatal FN_EXTERN bool errorTypeFatal(const ErrorType *errorType); // Get error type using a code FN_EXTERN const ErrorType *errorTypeFromCode(int code); // Error type name FN_EXTERN const char *errorTypeName(const ErrorType *errorType); // Error type parent FN_EXTERN const ErrorType *errorTypeParent(const ErrorType *errorType); // Does the child error type extend the parent error type? FN_EXTERN bool errorTypeExtends(const ErrorType *child, const ErrorType *parent); /*********************************************************************************************************************************** Functions to get information about the current error within a CATCH() block. Invalid outside a CATCH() block. ***********************************************************************************************************************************/ // Error type FN_EXTERN const ErrorType *errorType(void); // Error code (pulled from error type) FN_EXTERN int errorCode(void); // Is the error fatal? FN_EXTERN bool errorFatal(void); // Error filename FN_EXTERN const char *errorFileName(void); // Error function name FN_EXTERN const char *errorFunctionName(void); // Error file line number FN_EXTERN int errorFileLine(void); // Is this error an instance of the error type? FN_EXTERN bool errorInstanceOf(const ErrorType *errorTypeTest); // Error message FN_EXTERN const char *errorMessage(void); // Error name (pulled from error type) FN_EXTERN const char *errorName(void); // Error stack trace FN_EXTERN const char *errorStackTrace(void); /*********************************************************************************************************************************** Try stack getters/setters ***********************************************************************************************************************************/ // Get the depth of the current try statement (0 if none) FN_EXTERN unsigned int errorTryDepth(void); // Add a handler to be called when an error occurs typedef void (*const ErrorHandlerFunction)(unsigned int, bool); FN_EXTERN void errorHandlerSet(const ErrorHandlerFunction *list, unsigned int total); /*********************************************************************************************************************************** Begin a block where errors can be thrown ***********************************************************************************************************************************/ #define TRY_BEGIN() \ do \ { \ volatile bool TRY_finally = false; \ (void)TRY_finally; /* Will be unused if there is no finally block */ \ \ errorInternalTryBegin(__FILE__, __func__, __LINE__); \ \ if (setjmp(*errorInternalJump()) == 0) \ { /*********************************************************************************************************************************** Catch a specific error thrown in the try block. Fatal errors cannot be caught with this block. ***********************************************************************************************************************************/ #define CATCH(errorTypeCatch) \ } \ else if (errorInternalCatch(&errorTypeCatch, false)) \ { /*********************************************************************************************************************************** Catch any non-fatal error thrown in the try block ***********************************************************************************************************************************/ #define CATCH_ANY() \ } \ else if (errorInternalCatch(&RuntimeError, false)) \ { /*********************************************************************************************************************************** Catch any error thrown in the try block including fatal errors. To maximize the chance of being able to report the error caught, memory contexts will not be freed and mem context callbacks will not be called. If the TRY block is enclosed in a mem context then child mem contexts will be freed and child mem context callbacks called when the enclosing mem context is freed. ***********************************************************************************************************************************/ #define CATCH_FATAL() \ } \ else if (errorInternalCatch(&RuntimeError, true)) \ { /*********************************************************************************************************************************** Code to run whether the try block was successful or not ***********************************************************************************************************************************/ #define FINALLY() \ } \ \ if (!TRY_finally) \ { \ TRY_finally = true; /*********************************************************************************************************************************** End the try block ***********************************************************************************************************************************/ #define TRY_END() \ } \ \ errorInternalTryEnd(); \ } \ while (0) /*********************************************************************************************************************************** Throw an error Errors can be thrown any time, but if there is no TRY block somewhere in the call stack then the program will exit and print the error information to stderr. The seldom used "THROWP" variants allow an error to be thrown with a pointer to the error type. ***********************************************************************************************************************************/ #define THROW(errorType, message) \ errorInternalThrow(&errorType, __FILE__, __func__, __LINE__, message, NULL) #define THROW_FMT(errorType, ...) \ errorInternalThrowFmt(&errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROWP(errorType, message) \ errorInternalThrow(errorType, __FILE__, __func__, __LINE__, message, NULL) #define THROWP_FMT(errorType, ...) \ errorInternalThrowFmt(errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROW_CODE(errorCode, message) \ errorInternalThrow(errorTypeFromCode(errorCode), __FILE__, __func__, __LINE__, message, NULL) #define THROW_CODE_FMT(errorCode, ...) \ errorInternalThrowFmt(errorTypeFromCode(errorCode), __FILE__, __func__, __LINE__, __VA_ARGS__) /*********************************************************************************************************************************** Throw an error when a system call fails ***********************************************************************************************************************************/ #define THROW_SYS_ERROR(errorType, message) \ errorInternalThrowSys(errno, &errorType, __FILE__, __func__, __LINE__, message) #define THROW_SYS_ERROR_FMT(errorType, ...) \ errorInternalThrowSysFmt(errno, &errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROWP_SYS_ERROR(errorType, message) \ errorInternalThrowSys(errno, errorType, __FILE__, __func__, __LINE__, message) #define THROWP_SYS_ERROR_FMT(errorType, ...) \ errorInternalThrowSysFmt(errno, errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROW_SYS_ERROR_CODE(errNo, errorType, message) \ errorInternalThrowSys(errNo, &errorType, __FILE__, __func__, __LINE__, message) #define THROW_SYS_ERROR_CODE_FMT(errNo, errorType, ...) \ errorInternalThrowSysFmt(errNo, &errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) #define THROWP_SYS_ERROR_CODE(errNo, errorType, message) \ errorInternalThrowSys(errNo, errorType, __FILE__, __func__, __LINE__, message) #define THROWP_SYS_ERROR_CODE_FMT(errNo, errorType, ...) \ errorInternalThrowSysFmt(errNo, errorType, __FILE__, __func__, __LINE__, __VA_ARGS__) // When coverage testing, define special versions of the macros that don't contain branches. These macros are less efficient because // they need to call errorInternalThrowOnSys*() before determining if there is an error or not, but they allow coverage testing for // THROW*_ON*() calls that contain conditionals. #ifdef DEBUG_COVERAGE // The expression can't be passed directly to errorInternalThrowSys*() because we need to be sure it is evaluated before passing // errno. Depending on optimization that might not happen. #define THROW_ON_SYS_ERROR(expression, errorType, message) \ do \ { \ bool error = expression; \ errorInternalThrowOnSys(error, errno, &errorType, __FILE__, __func__, __LINE__, message); \ } while (0) #define THROW_ON_SYS_ERROR_FMT(expression, errorType, ...) \ do \ { \ bool error = expression; \ errorInternalThrowOnSysFmt(error, errno, &errorType, __FILE__, __func__, __LINE__, __VA_ARGS__); \ } while (0) #define THROWP_ON_SYS_ERROR(expression, errorType, message) \ do \ { \ bool error = expression; \ errorInternalThrowOnSys(error, errno, errorType, __FILE__, __func__, __LINE__, message); \ } while (0) #define THROWP_ON_SYS_ERROR_FMT(expression, errorType, ...) \ do \ { \ bool error = expression; \ errorInternalThrowOnSysFmt(error, errno, errorType, __FILE__, __func__, __LINE__, __VA_ARGS__); \ } while (0) // Else define the normal macros which check for an error first #else #define THROW_ON_SYS_ERROR(expression, errorType, message) \ do \ { \ if (expression) \ errorInternalThrowSys(errno, &errorType, __FILE__, __func__, __LINE__, message); \ } while (0) #define THROW_ON_SYS_ERROR_FMT(expression, errorType, ...) \ do \ { \ if (expression) \ errorInternalThrowSysFmt(errno, &errorType, __FILE__, __func__, __LINE__, __VA_ARGS__); \ } while (0) #define THROWP_ON_SYS_ERROR(expression, errorType, message) \ do \ { \ if (expression) \ errorInternalThrowSys(errno, errorType, __FILE__, __func__, __LINE__, message); \ } while (0) #define THROWP_ON_SYS_ERROR_FMT(expression, errorType, ...) \ do \ { \ if (expression) \ errorInternalThrowSysFmt(errno, errorType, __FILE__, __func__, __LINE__, __VA_ARGS__); \ } while (0) #endif /*********************************************************************************************************************************** Rethrow the current error ***********************************************************************************************************************************/ #define RETHROW() \ errorInternalPropagate() /*********************************************************************************************************************************** Internal functions These functions are used by the macros to implement the error handler and should never be called independently. ***********************************************************************************************************************************/ // Begin the try block FN_EXTERN void errorInternalTryBegin(const char *fileName, const char *functionName, int fileLine); // Return jump buffer for current try FN_EXTERN jmp_buf *errorInternalJump(void); // True when in catch state and the expected error matches FN_EXTERN bool errorInternalCatch(const ErrorType *errorTypeCatch, bool fatalCatch); // Propagate the error up so it can be caught FN_EXTERN noreturn void errorInternalPropagate(void); // End the try block FN_EXTERN void errorInternalTryEnd(void); // Throw an error FN_EXTERN noreturn void errorInternalThrow( const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *message, const char *stackTrace); FN_EXTERN noreturn FN_PRINTF(5, 6) void errorInternalThrowFmt( const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *format, ...); // Throw a system error FN_EXTERN noreturn void errorInternalThrowSys( int errNo, const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *message); // Throw a formatted system error FN_EXTERN FN_PRINTF(6, 7) noreturn void errorInternalThrowSysFmt( int errNo, const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *format, ...); // Versions of the above for coverage testing which checks the error condition inside the function #ifdef DEBUG_COVERAGE FN_EXTERN void errorInternalThrowOnSys( bool error, int errNo, const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *message); FN_EXTERN FN_PRINTF(7, 8) void errorInternalThrowOnSysFmt( bool error, int errNo, const ErrorType *errorType, const char *fileName, const char *functionName, int fileLine, const char *format, ...); #endif /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_ERROR_TYPE_TYPE \ ErrorType * #define FUNCTION_LOG_ERROR_TYPE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ErrorType", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/error/retry.c0000664000175100017510000001177515240547630020147 0ustar00runnerrunner/*********************************************************************************************************************************** Error Retry Message ***********************************************************************************************************************************/ #include #include "string.h" #include "common/debug.h" #include "common/error/retry.h" #include "common/time.h" #include "common/type/list.h" #include "common/type/string.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct ErrorRetry { ErrorRetryPub pub; // Publicly accessible variables const String *message; // First error message List *list; // List of retry errors TimeMSec timeBegin; // Time error retries began }; // Keep track of retries with the same error typedef struct ErrorRetryItem { String *message; // Error message unsigned int total; // Total errors with this message const ErrorType *type; // Error type TimeMSec retryFirst; // First retry TimeMSec retryLast; // Last Retry } ErrorRetryItem; /**********************************************************************************************************************************/ FN_EXTERN ErrorRetry * errRetryNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(ErrorRetry, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (ErrorRetry) { .timeBegin = timeMSec(), .list = lstNewP(sizeof(ErrorRetryItem), .comparator = lstComparatorStr), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(ERROR_RETRY, this); } /**********************************************************************************************************************************/ FN_EXTERN String * errRetryMessage(const ErrorRetry *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ERROR_RETRY, this); FUNCTION_TEST_END(); ASSERT(this->message != NULL); // Always include first message String *const result = strCat(strNew(), this->message); // Add retries, varying the message depending on how many retries there were for (unsigned int listIdx = 0; listIdx < lstSize(this->list); listIdx++) { const ErrorRetryItem *const error = lstGet(this->list, listIdx); strCatFmt(result, "\n [%s] ", errorTypeName(error->type)); if (error->retryFirst == error->retryLast) strCatFmt(result, "on retry at %" PRIu64, error->retryFirst); else strCatFmt(result, "on %u retries from %" PRIu64 "-%" PRIu64, error->total, error->retryFirst, error->retryLast); strCatFmt(result, "ms: %s", strZ(error->message)); } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void errRetryAdd(ErrorRetry *const this, const ErrRetryAddParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ERROR_RETRY, this); FUNCTION_TEST_PARAM(ERROR_TYPE, param.type); FUNCTION_TEST_PARAM(STRING, param.message); FUNCTION_TEST_END(); // Set defaults const ErrorType *const type = param.type == NULL ? errorType() : param.type; const char *const message = param.message == NULL ? errorMessage() : strZ(param.message); // Set type on first error if (this->pub.type == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.type = type; this->message = strNewZ(message); } MEM_CONTEXT_OBJ_END(); } else { // If error is not found then add it const String *const messageFind = STR(message); const TimeMSec retryTime = timeMSec() - this->timeBegin; ErrorRetryItem *const error = lstFind(this->list, &messageFind); if (error == NULL) { MEM_CONTEXT_OBJ_BEGIN(this->list) { const ErrorRetryItem errorNew = { .type = type, .total = 1, .message = strDup(messageFind), .retryFirst = retryTime, .retryLast = retryTime, }; lstAdd(this->list, &errorNew); } MEM_CONTEXT_OBJ_END(); } // Else update the error found else { error->total++; error->retryLast = retryTime; } } FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/error/retry.h0000664000175100017510000000741615240547630020151 0ustar00runnerrunner/*********************************************************************************************************************************** Error Retry Message Accumulate errors during retries to provide a single, coherent error message to the user that includes information about the original error and all the failed retries. Retries with the same error message are aggregated to keep the error message as short as possible. ***********************************************************************************************************************************/ #ifndef COMMON_ERROR_RETRY_H #define COMMON_ERROR_RETRY_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ErrorRetry ErrorRetry; #include "common/error/error.h" #include "common/type/object.h" #include "common/type/string.h" /*********************************************************************************************************************************** Constructor ***********************************************************************************************************************************/ FN_EXTERN ErrorRetry *errRetryNew(void); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ErrorRetryPub { const ErrorType *type; // Error type } ErrorRetryPub; // Get error type FN_INLINE_ALWAYS const ErrorType * errRetryType(const ErrorRetry *const this) { return THIS_PUB(ErrorRetry)->type; } // Get error message FN_EXTERN String *errRetryMessage(const ErrorRetry *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add error typedef struct ErrRetryAddParam { VAR_PARAM_HEADER; const ErrorType *type; // Error type (defaults to errorType()) const String *message; // Error message (defaults to errorMessage()) } ErrRetryAddParam; #define errRetryAddP(this, ...) \ errRetryAdd(this, (ErrRetryAddParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void errRetryAdd(ErrorRetry *const this, ErrRetryAddParam param); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void errRetryFree(ErrorRetry *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_ERROR_RETRY_TYPE \ ErrorRetry * #define FUNCTION_LOG_ERROR_RETRY_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ErrorRetry", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/exec.c0000664000175100017510000004165315240547630016573 0ustar00runnerrunner/*********************************************************************************************************************************** Execute Process ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include "common/debug.h" #include "common/exec.h" #include "common/fork.h" #include "common/io/fdRead.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/io/write.h" #include "common/log.h" #include "common/wait.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Exec { ExecPub pub; // Publicly accessible variables StringList *param; // List of parameters to pass to command const String *name; // Name to display in log/error messages TimeMSec timeout; // Timeout for any i/o operation (read, write, etc.) pid_t processId; // Process id of the child process int fdRead; // Read file descriptor int fdWrite; // Write file descriptor int fdError; // Error file descriptor IoRead *ioReadFd; // File descriptor read interface IoWrite *ioWriteFd; // File descriptor write interface }; /*********************************************************************************************************************************** Macro to close file descriptors after dup2() in the child process If the parent process is daemomized and has closed stdout, stdin, and stderr or some combination of them, then the newly created descriptors might overlap stdout, stdin, or stderr. In that case we don't want to accidentally close the descriptor that we have just copied. Note that this is pretty specific to the way that file descriptors are handled in this module and may not be generally applicable in other code. ***********************************************************************************************************************************/ #define PIPE_DUP2(pipe, pipeIdx, fd) \ do \ { \ dup2(pipe[pipeIdx], fd); \ \ if (pipe[0] != fd) \ close(pipe[0]); \ \ if (pipe[1] != fd) \ close(pipe[1]); \ } \ while (0); /*********************************************************************************************************************************** Free exec file descriptors and ensure process is shut down ***********************************************************************************************************************************/ static void execFreeResource(THIS_VOID) { THIS(Exec); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close file descriptors close(this->fdRead); close(this->fdWrite); close(this->fdError); // Wait for the child to exit. We don't really care how it exits as long as it does. if (this->processId != 0) { MEM_CONTEXT_TEMP_BEGIN() { int processResult = 0; Wait *const wait = waitNew(this->timeout); do { THROW_ON_SYS_ERROR( (processResult = waitpid(this->processId, NULL, WNOHANG)) == -1, ExecuteError, "unable to wait on child process"); } while (processResult == 0 && waitMore(wait)); // If the process did not exit then error -- else we may end up with a collection of zombie processes if (processResult == 0) THROW_FMT(ExecuteError, "%s did not exit when expected", strZ(this->name)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Exec * execNew(const String *const command, const StringList *const param, const String *const name, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, command); FUNCTION_LOG_PARAM(STRING_LIST, param); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(name != NULL); ASSERT(timeout > 0); OBJ_NEW_BEGIN(Exec, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Exec) { .name = strDup(name), .timeout = timeout, // Parameter list is optional but if not specified we need to build one with the command .param = param == NULL ? strLstNew() : strLstDup(param), }; // If command is NULL then the first item in the param list is the command if (command == NULL) { ASSERT(param != NULL && strLstSize(param) != 0); this->pub.command = strLstGet(param, 0); } // Else copy command and insert it as the first parameter else { this->pub.command = strDup(command); strLstInsert(this->param, 0, execCommand(this)); } } OBJ_NEW_END(); FUNCTION_LOG_RETURN(EXEC, this); } /*********************************************************************************************************************************** Check if the process is still running This should be called when anything unexpected happens while reading or writing, including errors and eof. If this function returns then the original error should be rethrown. ***********************************************************************************************************************************/ // Helper to throw status error static void execCheckStatusError(Exec *const this, int status, const String *const output) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(EXEC, this); FUNCTION_TEST_PARAM(INT, status); FUNCTION_TEST_PARAM(STRING, output); FUNCTION_TEST_END(); // Throw the error with as much information as is available THROWP_FMT( errorTypeFromCode(WEXITSTATUS(status)), "%s terminated unexpectedly [%d]%s%s", strZ(this->name), WEXITSTATUS(status), strSize(output) > 0 ? ": " : "", strSize(output) > 0 ? strZ(output) : ""); FUNCTION_TEST_RETURN_VOID(); } // Helper to throw signal error static void execCheckSignalError(Exec *const this, int status) { THROW_FMT(ExecuteError, "%s terminated unexpectedly on signal %d", strZ(this->name), WTERMSIG(status)); } static void execCheck(Exec *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_END(); ASSERT(this != NULL); int processStatus; int processResult; THROW_ON_SYS_ERROR( (processResult = waitpid(this->processId, &processStatus, WNOHANG)) == -1, ExecuteError, "unable to wait on child process"); if (processResult != 0) { // Clear the process id so we don't try to wait for this process on free this->processId = 0; // If the process exited normally if (WIFEXITED(processStatus)) { execCheckStatusError( this, processStatus, strTrim(strNewBuf(ioReadBuf(ioFdReadNewOpen(strNewFmt("%s error", strZ(this->name)), this->fdError, 0))))); } // If the process did not exit normally then it must have been a signal execCheckSignalError(this, processStatus); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Read from the process ***********************************************************************************************************************************/ static size_t execRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(Exec); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); size_t result; TRY_BEGIN() { result = ioReadInterface(this->ioReadFd)->read(ioReadDriver(this->ioReadFd), buffer, block); } CATCH_ANY() { execCheck(this); RETHROW(); } TRY_END(); FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Write to the process ***********************************************************************************************************************************/ static void execWrite(THIS_VOID, const Buffer *const buffer) { THIS(Exec); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); TRY_BEGIN() { ioWrite(this->ioWriteFd, buffer); ioWriteFlush(this->ioWriteFd); } CATCH_ANY() { execCheck(this); RETHROW(); } TRY_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is the process eof? ***********************************************************************************************************************************/ static bool execEof(THIS_VOID) { THIS(Exec); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Check if the process is still running on eof if (ioReadInterface(this->ioReadFd)->eof(ioReadDriver(this->ioReadFd))) execCheck(this); FUNCTION_LOG_RETURN(BOOL, false); } /*********************************************************************************************************************************** Get the read file descriptor ***********************************************************************************************************************************/ static int execFdRead(const THIS_VOID) { THIS(const Exec); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(EXEC, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fdRead); } /**********************************************************************************************************************************/ FN_EXTERN void execOpen(Exec *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(EXEC, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Create pipes to communicate with the subprocess. The names of the pipes are from the perspective of the parent process since // the child process will use them only briefly before exec'ing. int pipeRead[2]; int pipeWrite[2]; int pipeError[2]; THROW_ON_SYS_ERROR(pipe(pipeRead) == -1, KernelError, "unable to create read pipe"); THROW_ON_SYS_ERROR(pipe(pipeWrite) == -1, KernelError, "unable to create write pipe"); THROW_ON_SYS_ERROR(pipe(pipeError) == -1, KernelError, "unable to create error pipe"); // Fork the subprocess this->processId = forkSafe(); // Exec command in the child process if (this->processId == 0) { // Disable logging and close log file logClose(); // Assign stdout to the input side of the read pipe PIPE_DUP2(pipeRead, 1, STDOUT_FILENO); // Assign stdin to the output side of the write pipe PIPE_DUP2(pipeWrite, 0, STDIN_FILENO); // Assign stderr to the input side of the error pipe PIPE_DUP2(pipeError, 1, STDERR_FILENO); // Execute the binary. This statement will not return if it is successful. execvp() requires non-const parameters because it // modifies them after the fork. #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" execvp(strZ(execCommand(this)), UNCONSTIFY(char **, strLstPtr(this->param))); #pragma GCC diagnostic pop // If we got here then there was an error. We can't use a throw as we normally would because we have already shutdown // logging and we don't want to execute exit paths that might free parent resources which we still have references to. fprintf(stderr, "unable to execute '%s': [%d] %s\n", strZ(execCommand(this)), errno, strerror(errno)); exit(errorTypeCode(&ExecuteError)); } // Close the unused file descriptors close(pipeRead[1]); close(pipeWrite[0]); close(pipeError[1]); // Store the file descriptors we'll use and need to close when the process terminates this->fdRead = pipeRead[0]; this->fdWrite = pipeWrite[1]; this->fdError = pipeError[0]; MEM_CONTEXT_OBJ_BEGIN(this) { // Assign file descriptors to io interfaces this->ioReadFd = ioFdReadNew(strNewFmt("%s read", strZ(this->name)), this->fdRead, this->timeout); this->ioWriteFd = ioFdWriteNewOpen(strNewFmt("%s write", strZ(this->name)), this->fdWrite, this->timeout); // Create wrapper interfaces that check process state this->pub.ioReadExec = ioReadNewP(this, .block = true, .read = execRead, .eof = execEof, .fd = execFdRead); ioReadOpen(execIoRead(this)); this->pub.ioWriteExec = ioWriteNewP(this, .write = execWrite); ioWriteOpen(execIoWrite(this)); } MEM_CONTEXT_OBJ_END(); // Set a callback so the file descriptors will get freed memContextCallbackSet(objMemContext(this), execFreeResource, this); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * execOne(const StringList *const list) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_LIST, list); FUNCTION_LOG_END(); ASSERT(list != NULL); ASSERT(strLstSize(list) != 0); String *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Exec command Exec *const exec = execNew(NULL, list, strLstGet(list, 0), ioTimeoutMs()); execOpen(exec); // Read all output from stdout ioReadOpen(exec->ioReadFd); String *const output = strNewBuf(ioReadBuf(exec->ioReadFd)); // Wait for the process to exit int processStatus; THROW_ON_SYS_ERROR( waitpid(exec->processId, &processStatus, 0) == -1, ExecuteError, "unable to wait on child process"); // Clear the process id so we don't try to wait for this process on free exec->processId = 0; // Check exit status if (WIFEXITED(processStatus)) { if (WEXITSTATUS(processStatus) != 0) execCheckStatusError(exec, processStatus, strTrim(output)); } // Else if the process did not exit normally then it must have been a signal else execCheckSignalError(exec, processStatus); MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(strTrim(output)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/exec.h0000664000175100017510000001033015240547630016564 0ustar00runnerrunner/*********************************************************************************************************************************** Execute Process Executes a child process and allows the calling process to communicate with it using read/write io. This object is specially tailored to implement the protocol layer and may or may not be generally applicable to general purpose execution. ***********************************************************************************************************************************/ #ifndef COMMON_EXEC_H #define COMMON_EXEC_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Exec Exec; #include "common/io/read.h" #include "common/io/write.h" #include "common/time.h" #include "common/type/object.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Exec *execNew(const String *command, const StringList *param, const String *name, TimeMSec timeout); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ExecPub { String *command; // Command to execute IoRead *ioReadExec; // Wrapper for file descriptor read interface IoWrite *ioWriteExec; // Wrapper for file descriptor write interface } ExecPub; // Exec command FN_INLINE_ALWAYS const String * execCommand(const Exec *const this) { return THIS_PUB(Exec)->command; } // Read interface FN_INLINE_ALWAYS IoRead * execIoRead(Exec *const this) { return THIS_PUB(Exec)->ioReadExec; } // Write interface FN_INLINE_ALWAYS IoWrite * execIoWrite(Exec *const this) { return THIS_PUB(Exec)->ioWriteExec; } // Exec MemContext FN_INLINE_ALWAYS MemContext * execMemContext(Exec *const this) { return objMemContext(this); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Execute command FN_EXTERN void execOpen(Exec *this); /*********************************************************************************************************************************** Helper Functions ***********************************************************************************************************************************/ // Execute a command with an explicit parameter list and return stdout output. The first item in the list is the command and the // remaining items are parameters. FN_EXTERN String *execOne(const StringList *list); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void execFree(Exec *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_EXEC_TYPE \ Exec * #define FUNCTION_LOG_EXEC_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Exec", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/fork.c0000664000175100017510000000463415240547630016606 0ustar00runnerrunner/*********************************************************************************************************************************** Fork Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/fork.h" #include "common/log.h" /**********************************************************************************************************************************/ FN_EXTERN int forkSafe(void) { FUNCTION_LOG_VOID(logLevelTrace); const int result = fork(); THROW_ON_SYS_ERROR(result == -1, KernelError, "unable to fork"); FUNCTION_LOG_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN void forkDetach(void) { FUNCTION_LOG_VOID(logLevelTrace); // Make this process a group leader so the parent process won't block waiting for it to finish THROW_ON_SYS_ERROR(setsid() == -1, KernelError, "unable to create new session group"); // The process should never receive a SIGHUP but ignore it just in case signal(SIGHUP, SIG_IGN); // There should be no way the child process can exit first (after the next fork) but just in case ignore SIGCHLD. This means // that the child process will automatically be reaped by the kernel should it finish first rather than becoming defunct. signal(SIGCHLD, SIG_IGN); // Fork again and let the parent process terminate to ensure that we get rid of the session leading process. Only session // leaders may get a TTY again. if (forkSafe() != 0) exit(0); // Reset SIGCHLD to the default handler so waitpid() calls in the process will work as expected signal(SIGCHLD, SIG_DFL); // Change the working directory to / so there is no dependency on the original working directory THROW_ON_SYS_ERROR(chdir("/") == -1, PathMissingError, "unable to change directory to '/'"); // Close standard file descriptors THROW_ON_SYS_ERROR(close(STDIN_FILENO) == -1, FileCloseError, "unable to close stdin"); THROW_ON_SYS_ERROR(close(STDOUT_FILENO) == -1, FileCloseError, "unable to close stdout"); THROW_ON_SYS_ERROR(close(STDERR_FILENO) == -1, FileCloseError, "unable to close stderr"); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/fork.h0000664000175100017510000000164715240547630016614 0ustar00runnerrunner/*********************************************************************************************************************************** Fork Handler ***********************************************************************************************************************************/ #ifndef COMMON_FORK_H #define COMMON_FORK_H /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Fork a new process and throw an error if it fails FN_EXTERN int forkSafe(void); // Detach a forked process so it can continue running after the parent process has exited. This is not a typical daemon startup // because the parent process may continue to run and perform work for some time. FN_EXTERN void forkDetach(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/ini.c0000664000175100017510000003106015240547630016415 0ustar00runnerrunner/*********************************************************************************************************************************** Ini Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/ini.h" #include "common/log.h" #include "common/type/json.h" #include "common/type/keyValue.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Ini { KeyValue *store; // Key value store that contains the ini data IoRead *read; // Read object for ini data IniValue value; // Current value unsigned int lineIdx; // Current line used for error reporting bool strict; // Expect all values to be JSON and do not trim }; /**********************************************************************************************************************************/ FN_EXTERN Ini * iniNew(IoRead *const read, const IniNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(BOOL, param.strict); FUNCTION_LOG_PARAM(BOOL, param.store); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(Ini, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Ini) { .read = read, .value = { .section = strNew(), .key = strNew(), .value = strNew(), }, .strict = param.strict, }; ioReadOpen(this->read); // Build KeyValue store if (param.store) { this->store = kvNew(); MEM_CONTEXT_TEMP_RESET_BEGIN() { const IniValue *value = iniValueNext(this); // Add values while not done while (value != NULL) { const Variant *const sectionKey = VARSTR(value->section); KeyValue *sectionKv = varKv(kvGet(this->store, sectionKey)); if (sectionKv == NULL) sectionKv = kvPutKv(this->store, sectionKey); kvAdd(sectionKv, VARSTR(value->key), VARSTR(value->value)); MEM_CONTEXT_TEMP_RESET(1000); value = iniValueNext(this); } } MEM_CONTEXT_TEMP_END(); } } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INI, this); } /**********************************************************************************************************************************/ FN_EXTERN const IniValue * iniValueNext(Ini *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_END(); ASSERT(this != NULL); const IniValue *result = NULL; MEM_CONTEXT_TEMP_RESET_BEGIN() { // Read until a value is found or eof while (result == NULL && !ioReadEof(this->read)) { // Read the line and trim if needed String *const line = ioReadLineParam(this->read, true); if (!this->strict) strTrim(line); const char *const linePtr = strZ(line); this->lineIdx++; // Only interested in lines that are not blank or comments if (strSize(line) > 0 && (this->strict || linePtr[0] != '#')) { // The line is a section. Since the value must be valid JSON this means that the value must never be an array. if (linePtr[0] == '[' && (!this->strict || linePtr[strSize(line) - 1] == ']')) { // Make sure the section ends with ] if (!this->strict && linePtr[strSize(line) - 1] != ']') THROW_FMT(FormatError, "ini section should end with ] at line %u: %s", this->lineIdx, linePtr); // Store the section strCatZN(strTrunc(this->value.section), linePtr + 1, strSize(line) - 2); if (strEmpty(this->value.section)) THROW_FMT(FormatError, "invalid empty section at line %u: %s", this->lineIdx, linePtr); } // Else it is a key/value else { if (strEmpty(this->value.section)) THROW_FMT(FormatError, "key/value found outside of section at line %u: %s", this->lineIdx, linePtr); // Find the = const char *lineEqual = strstr(linePtr, "="); if (lineEqual == NULL) THROW_FMT(FormatError, "missing '=' in key/value at line %u: %s", this->lineIdx, linePtr); // Extract the key/value. For strict this may require some retries if the key includes an = character since this // is also the separator. We know the value must be valid JSON so if it isn't then add the characters up to the // next = to the key and try to parse the value as JSON again. If the value never becomes valid JSON then an // error is thrown. bool retry; do { retry = false; // Get key/value strCatZN(strTrunc(this->value.key), linePtr, (size_t)(lineEqual - linePtr)); strCatZ(strTrunc(this->value.value), lineEqual + 1); // Value is expected to be valid JSON if (this->strict) { TRY_BEGIN() { jsonValidate(this->value.value); } CATCH(JsonFormatError) { // If value is not valid JSON look for another =. If not found then nothing to retry. lineEqual = strstr(lineEqual + 1, "="); if (lineEqual == NULL) { THROW_FMT( FormatError, "invalid JSON value at line %u '%s': %s", this->lineIdx, linePtr, errorMessage()); } // Try again with = in new position retry = true; } TRY_END(); } // Else just trim else { strTrim(this->value.key); strTrim(this->value.value); } } while (retry); // Key may not be zero-length if (strSize(this->value.key) == 0) THROW_FMT(FormatError, "key is zero-length at line %u: %s", this->lineIdx, linePtr); // A value was found so return it result = &this->value; } } MEM_CONTEXT_TEMP_RESET(1000); } // If no more values found then close if (result == NULL) ioReadClose(this->read); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_CONST(INI_VALUE, result); } /**********************************************************************************************************************************/ FN_EXTERN void iniValid(Ini *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_END(); while (iniValueNext(this) != NULL); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Internal function to get an ini value ***********************************************************************************************************************************/ static const Variant * iniGetInternal(const Ini *const this, const String *const section, const String *const key, const bool required) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(BOOL, required); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->store != NULL); ASSERT(section != NULL); ASSERT(key != NULL); const Variant *result = NULL; // Get the section const KeyValue *const sectionKv = varKv(kvGet(this->store, VARSTR(section))); // Section must exist to get the value if (sectionKv != NULL) result = kvGet(sectionKv, VARSTR(key)); // If value is null and required then error if (result == NULL && required) THROW_FMT(FormatError, "section '%s', key '%s' does not exist", strZ(section), strZ(key)); FUNCTION_TEST_RETURN_CONST(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN const String * iniGet(const Ini *const this, const String *const section, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, varStr(iniGetInternal(this, section, key, true))); } /**********************************************************************************************************************************/ FN_EXTERN StringList * iniGetList(const Ini *const this, const String *const section, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); // Get the value const Variant *const result = iniGetInternal(this, section, key, false); FUNCTION_TEST_RETURN(STRING_LIST, result == NULL ? NULL : strLstNewVarLst(varVarLst(result))); } /**********************************************************************************************************************************/ FN_EXTERN bool iniSectionKeyIsList(const Ini *const this, const String *const section, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); // Get the value const Variant *const result = iniGetInternal(this, section, key, true); FUNCTION_TEST_RETURN(BOOL, varType(result) == varTypeVariantList); } /**********************************************************************************************************************************/ FN_EXTERN StringList * iniSectionList(const Ini *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->store != NULL); FUNCTION_TEST_RETURN(STRING_LIST, strLstNewVarLst(kvKeyList(this->store))); } /**********************************************************************************************************************************/ FN_EXTERN StringList * iniSectionKeyList(const Ini *const this, const String *const section) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INI, this); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->store != NULL); ASSERT(section != NULL); StringList *result; MEM_CONTEXT_TEMP_BEGIN() { // Get the section const KeyValue *const sectionKv = varKv(kvGet(this->store, VARSTR(section))); // Return key list if the section exists if (sectionKv != NULL) { result = strLstNewVarLst(kvKeyList(sectionKv)); } // Otherwise return an empty list else result = strLstNew(); strLstMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/ini.h0000664000175100017510000001146515240547630016431 0ustar00runnerrunner/*********************************************************************************************************************************** Ini Handler ***********************************************************************************************************************************/ #ifndef COMMON_INI_H #define COMMON_INI_H /*********************************************************************************************************************************** Ini object ***********************************************************************************************************************************/ typedef struct Ini Ini; #include "common/io/read.h" #include "common/io/write.h" #include "common/type/object.h" #include "common/type/variant.h" /*********************************************************************************************************************************** Ini section/key/value ***********************************************************************************************************************************/ typedef struct IniValue { String *section; String *key; String *value; } IniValue; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct IniNewParam { VAR_PARAM_HEADER; bool strict; // Expect all values to be JSON and do not trim bool store; // Store in KeyValue so functions can be used to query } IniNewParam; #define iniNewP(read, ...) \ iniNew(read, (IniNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Ini *iniNew(IoRead *read, IniNewParam param); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS Ini * iniMove(Ini *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Check that the ini is valid FN_EXTERN void iniValid(Ini *this); // Get the next value in the ini file. Note that members of IniValue are reused between calls so the members of a previous call may // change after the next call. Any members that need to be preserved should be copied before the next call. FN_EXTERN const IniValue *iniValueNext(Ini *this); /*********************************************************************************************************************************** Getters ***********************************************************************************************************************************/ // Get an ini value -- error if it does not exist FN_EXTERN const String *iniGet(const Ini *this, const String *section, const String *key); // Ini key list FN_EXTERN StringList *iniGetList(const Ini *this, const String *section, const String *key); // The key's value is a list FN_EXTERN bool iniSectionKeyIsList(const Ini *this, const String *section, const String *key); // List of sections FN_EXTERN StringList *iniSectionList(const Ini *const this); // List of keys for a section FN_EXTERN StringList *iniSectionKeyList(const Ini *this, const String *section); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void iniFree(Ini *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_INI_TYPE \ Ini * #define FUNCTION_LOG_INI_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Ini", buffer, bufferSize) #define FUNCTION_LOG_INI_VALUE_TYPE \ IniValue * #define FUNCTION_LOG_INI_VALUE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IniValue", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/0000775000175100017510000000000015240547630016101 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/bufferRead.c0000664000175100017510000001000515240547630020306 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer IO Read ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoBufferRead { const Buffer *read; // Buffer to read data from size_t readPos; // Current position in the read buffer bool eof; // Has the end of the buffer been reached? } IoBufferRead; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_BUFFER_READ_TYPE \ IoBufferRead * #define FUNCTION_LOG_IO_BUFFER_READ_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoBufferRead", buffer, bufferSize) /*********************************************************************************************************************************** Read data from the buffer ***********************************************************************************************************************************/ static size_t ioBufferRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(IoBufferRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_BUFFER_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); size_t actualBytes = 0; if (!this->eof) { // Determine how many bytes can actually be read from the buffer (if all bytes will be read then EOF) actualBytes = bufUsed(this->read) - this->readPos; if (actualBytes > bufRemains(buffer)) actualBytes = bufRemains(buffer); else this->eof = true; // Copy bytes to buffer bufCatSub(buffer, this->read, this->readPos, actualBytes); this->readPos += actualBytes; } FUNCTION_LOG_RETURN(SIZE, actualBytes); } /*********************************************************************************************************************************** Have all bytes been read from the buffer? ***********************************************************************************************************************************/ static bool ioBufferReadEof(THIS_VOID) { THIS(IoBufferRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_BUFFER_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->eof); } /**********************************************************************************************************************************/ FN_EXTERN IoRead * ioBufferReadNew(const Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(buffer != NULL); OBJ_NEW_BEGIN(IoBufferRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoBufferRead) { .read = buffer, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_READ, ioReadNewP(this, .eof = ioBufferReadEof, .read = ioBufferRead)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/bufferRead.h0000664000175100017510000000172415240547630020323 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer IO Read Read from a Buffer object using the IoRead interface. ***********************************************************************************************************************************/ #ifndef COMMON_IO_BUFFERREAD_H #define COMMON_IO_BUFFERREAD_H #include "common/io/read.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoRead *ioBufferReadNew(const Buffer *buffer); // Construct and open buffer read FN_INLINE_ALWAYS IoRead * ioBufferReadNewOpen(const Buffer *const buffer) { IoRead *const result = ioBufferReadNew(buffer); ioReadOpen(result); return result; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/bufferWrite.c0000664000175100017510000000531315240547630020533 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer IO Write ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/bufferWrite.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoBufferWrite { Buffer *write; // Buffer to write into } IoBufferWrite; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_BUFFER_WRITE_TYPE \ IoBufferWrite * #define FUNCTION_LOG_IO_BUFFER_WRITE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoBufferWrite", buffer, bufferSize) /*********************************************************************************************************************************** Write to the buffer ***********************************************************************************************************************************/ static void ioBufferWrite(THIS_VOID, const Buffer *const buffer) { THIS(IoBufferWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_BUFFER_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); bufCat(this->write, buffer); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN IoWrite * ioBufferWriteNew(Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(buffer != NULL); OBJ_NEW_BEGIN(IoBufferWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoBufferWrite) { .write = buffer, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_WRITE, ioWriteNewP(this, .write = ioBufferWrite)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/bufferWrite.h0000664000175100017510000000172415240547630020542 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer IO Write Write to a Buffer object using the IoWrite interface. ***********************************************************************************************************************************/ #ifndef COMMON_IO_BUFFERWRITE_H #define COMMON_IO_BUFFERWRITE_H #include "common/io/write.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoWrite *ioBufferWriteNew(Buffer *buffer); // Construct and open buffer write FN_INLINE_ALWAYS IoWrite * ioBufferWriteNewOpen(Buffer *const buffer) { IoWrite *const result = ioBufferWriteNew(buffer); ioWriteOpen(result); return result; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/client.c0000664000175100017510000000433215240547630017525 0ustar00runnerrunner/*********************************************************************************************************************************** Io Client Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/client.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoClient { IoClientPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN IoClient * ioClientNew(void *const driver, const IoClientInterface *const interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_CLIENT_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface != NULL); ASSERT(interface->type != 0); ASSERT(interface->name != NULL); ASSERT(interface->open != NULL); ASSERT(interface->toLog != NULL); OBJ_NEW_BEGIN(IoClient, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoClient) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_CLIENT, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioClientToLog(const IoClient *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(this->pub.interface->type, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", driver: "); this->pub.interface->toLog(this->pub.driver, debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/client.h0000664000175100017510000000635015240547630017534 0ustar00runnerrunner/*********************************************************************************************************************************** Io Client Interface Create sessions for protocol clients. For example, a TLS client can be created with tlsClientNewP() and then new TLS sessions can be opened with ioClientOpen(). ***********************************************************************************************************************************/ #ifndef COMMON_IO_CLIENT_H #define COMMON_IO_CLIENT_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoClient IoClient; #include "common/io/client.intern.h" #include "common/io/session.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoClientPub { void *driver; // Driver object const IoClientInterface *interface; // Driver interface } IoClientPub; // Name that identifies the client FN_INLINE_ALWAYS const String * ioClientName(const IoClient *const this) { return THIS_PUB(IoClient)->interface->name(THIS_PUB(IoClient)->driver); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS IoClient * ioClientMove(IoClient *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Open session FN_INLINE_ALWAYS IoSession * ioClientOpen(IoClient *const this) { return THIS_PUB(IoClient)->interface->open(THIS_PUB(IoClient)->driver); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioClientFree(IoClient *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioClientToLog(const IoClient *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_CLIENT_TYPE \ IoClient * #define FUNCTION_LOG_IO_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioClientToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/client.intern.h0000664000175100017510000000412715240547630021032 0ustar00runnerrunner/*********************************************************************************************************************************** Io Client Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_CLIENT_INTERN_H #define COMMON_IO_CLIENT_INTERN_H #include "common/io/client.h" #include "common/io/session.h" #include "common/type/string.h" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct IoClientInterface { // Type used to identify the client StringId type; // Client name, usually host:port or some other unique identifier const String *(*name)(void *driver); // Open a session IoSession *(*open)(void *driver); // Driver log function void (*toLog)(const void *driver, StringStatic *debugLog); } IoClientInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoClient *ioClientNew(void *driver, const IoClientInterface *interface); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_CLIENT_INTERFACE_TYPE \ IoClientInterface * #define FUNCTION_LOG_IO_CLIENT_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoClientInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/fd.c0000664000175100017510000000640715240547630016645 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Functions ***********************************************************************************************************************************/ #include #ifdef __sun__ // Illumos needs sys/siginfo for sigset_t inside poll.h #include #endif #include #include "common/debug.h" #include "common/io/fd.h" #include "common/log.h" /*********************************************************************************************************************************** Use poll() to determine when data is ready to read/write on a socket. Retry after EINTR with whatever time is left on the timer. ***********************************************************************************************************************************/ // Helper to determine when poll() should be retried static bool fdReadyRetry(const int pollResult, const int errNo, const bool first, TimeMSec *const timeout, const TimeMSec timeEnd) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, pollResult); FUNCTION_TEST_PARAM(INT, errNo); FUNCTION_TEST_PARAM(BOOL, first); FUNCTION_TEST_PARAM_P(TIME_MSEC, timeout); FUNCTION_TEST_PARAM(TIME_MSEC, timeEnd); FUNCTION_TEST_END(); ASSERT(timeout != NULL); // No retry by default bool result = false; // Process errors if (pollResult == -1) { // Don't error on an interrupt. If the interrupt lasts long enough there may be a timeout, though. if (errNo != EINTR) THROW_SYS_ERROR_CODE(errNo, KernelError, "unable to poll socket"); // Always retry on the first iteration if (first) { result = true; } // Else retry if there is time left else { TimeMSec timeCurrent = timeMSec(); if (timeEnd > timeCurrent) { *timeout = timeEnd - timeCurrent; result = true; } } } FUNCTION_TEST_RETURN(BOOL, result); } FN_EXTERN bool fdReady(const int fd, const bool read, const bool write, TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(BOOL, read); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(fd >= 0); ASSERT(read || write); ASSERT(timeout < INT_MAX); // Poll settings struct pollfd inputFd = {.fd = fd}; if (read) inputFd.events |= POLLIN; if (write) inputFd.events |= POLLOUT; // Wait for ready or timeout TimeMSec timeEnd = timeMSec() + timeout; bool first = true; // Initialize result and errno to look like a retryable error. We have no good way to test this function with interrupts so this // at least ensures that the condition is retried. int result = -1; int errNo = EINTR; while (fdReadyRetry(result, errNo, first, &timeout, timeEnd)) { result = poll(&inputFd, 1, (int)timeout); errNo = errno; first = false; } FUNCTION_LOG_RETURN(BOOL, result > 0); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/fd.h0000664000175100017510000000221215240547630016640 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Functions ***********************************************************************************************************************************/ #ifndef COMMON_IO_FD_H #define COMMON_IO_FD_H #include "common/time.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Wait until the file descriptor is ready to read/write or timeout FN_EXTERN bool fdReady(int fd, bool read, bool write, TimeMSec timeout); // Wait until the file descriptor is ready to read or timeout FN_INLINE_ALWAYS bool fdReadyRead(const int fd, const TimeMSec timeout) { return fdReady(fd, true, false, timeout); } // Wait until the file descriptor is ready to write or timeout FN_INLINE_ALWAYS bool fdReadyWrite(const int fd, const TimeMSec timeout) { return fdReady(fd, false, true, timeout); } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/fdRead.c0000664000175100017510000001422615240547630017437 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Io Read ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/fd.h" #include "common/io/fdRead.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoFdRead { const String *name; // File descriptor name for error messages int fd; // File descriptor to read data from TimeMSec timeout; // Timeout for read operation bool eof; // Has the end of the stream been reached? } IoFdRead; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_FD_READ_TYPE \ IoFdRead * #define FUNCTION_LOG_IO_FD_READ_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoFdRead", buffer, bufferSize) /*********************************************************************************************************************************** Are there bytes ready to read immediately? ***********************************************************************************************************************************/ static bool ioFdReadReady(THIS_VOID, const bool error) { THIS(IoFdRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_READ, this); FUNCTION_LOG_PARAM(BOOL, error); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = true; // Check if the file descriptor is ready to read if (!fdReadyRead(this->fd, this->timeout)) { // Error if requested if (error) THROW_FMT(FileReadError, "timeout after %" PRIu64 "ms waiting for read from '%s'", this->timeout, strZ(this->name)); // File descriptor is not ready to read result = false; } FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Read data from the file descriptor ***********************************************************************************************************************************/ static size_t ioFdRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(IoFdRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(!bufFull(buffer)); ssize_t actualBytes = 0; // If blocking keep reading until buffer is full or eof if (!this->eof) { do { // Check if there is data to be read and error if not ioFdReadReady(this, true); // Read and handle errors THROW_ON_SYS_ERROR_FMT( (actualBytes = read(this->fd, bufRemainsPtr(buffer), bufRemains(buffer))) == -1, FileReadError, "unable to read from %s", strZ(this->name)); // Update amount of buffer used bufUsedInc(buffer, (size_t)actualBytes); // If zero bytes were returned then eof if (actualBytes == 0) this->eof = true; } while (bufRemains(buffer) > 0 && !this->eof && block); } FUNCTION_LOG_RETURN(SIZE, (size_t)actualBytes); } /*********************************************************************************************************************************** Have all bytes been read from the buffer? ***********************************************************************************************************************************/ static bool ioFdReadEof(THIS_VOID) { THIS(IoFdRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->eof); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int ioFdReadFd(const THIS_VOID) { THIS(const IoFdRead); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FD_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ FN_EXTERN IoRead * ioFdReadNew(const String *const name, const int fd, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(fd != -1); OBJ_NEW_BEGIN(IoFdRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoFdRead) { .name = strDup(name), .fd = fd, .timeout = timeout, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_READ, ioReadNewP(this, .block = true, .eof = ioFdReadEof, .fd = ioFdReadFd, .read = ioFdRead, .ready = ioFdReadReady)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/fdRead.h0000664000175100017510000000204415240547630017437 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Read Read from a file descriptor using the IoRead interface. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FDREAD_H #define COMMON_IO_FDREAD_H #include "common/io/read.h" #include "common/time.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoRead *ioFdReadNew(const String *name, int fd, TimeMSec timeout); // Construct and open read fd FN_INLINE_ALWAYS IoRead * ioFdReadNewOpen(const String *const name, const int fd, const TimeMSec timeout) { IoRead *const result = ioFdReadNew(name, fd, timeout); ioReadOpen(result); return result; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/fdWrite.c0000664000175100017510000001475215240547630017662 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Io Write ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/fd.h" #include "common/io/fdWrite.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoFdWrite { const String *name; // File descriptor name for error messages int fd; // File descriptor to write to TimeMSec timeout; // Timeout for write operation } IoFdWrite; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_FD_WRITE_TYPE \ IoFdWrite * #define FUNCTION_LOG_IO_FD_WRITE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoFdWrite", buffer, bufferSize) /*********************************************************************************************************************************** // Can bytes be written immediately? ***********************************************************************************************************************************/ static bool ioFdWriteReady(THIS_VOID, const bool error) { THIS(IoFdWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_WRITE, this); FUNCTION_LOG_PARAM(BOOL, error); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = true; // Check if the file descriptor is ready to write if (!fdReadyWrite(this->fd, this->timeout)) { // Error if requested if (error) THROW_FMT(FileWriteError, "timeout after %" PRIu64 "ms waiting for write to '%s'", this->timeout, strZ(this->name)); // File descriptor is not ready to write result = false; } FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Write to the file descriptor ***********************************************************************************************************************************/ // Helper wrapper around write() system call for unit testing static ssize_t ioFdWriteInternal(const int fd, const void *const buffer, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, fd); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(fd >= 0); ASSERT(buffer != NULL); FUNCTION_TEST_RETURN(SSIZE, write(fd, buffer, size)); } static void ioFdWrite(THIS_VOID, const Buffer *const buffer) { THIS(IoFdWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FD_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Handle partial writes and non-blocking socket operations size_t totalWritten = 0; size_t bufferRemaining = bufUsed(buffer); while (bufferRemaining > 0) { const ssize_t result = ioFdWriteInternal(this->fd, bufPtrConst(buffer) + totalWritten, bufferRemaining); if (result == -1) { // Handle non-blocking socket case where write buffer is full if (errno == EAGAIN) { // Wait for socket to become writable (will throw on timeout) ioFdWriteReady(this, true); continue; } // Treat all other errors as fatal THROW_SYS_ERROR_FMT( FileWriteError, "unable to finish write to %s (wrote %zu/%zu bytes)", strZ(this->name), totalWritten, bufUsed(buffer)); } totalWritten += (size_t)result; bufferRemaining -= (size_t)result; } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int ioFdWriteFd(const THIS_VOID) { THIS(const IoFdWrite); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FD_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ FN_EXTERN IoWrite * ioFdWriteNew(const String *const name, const int fd, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(IoFdWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoFdWrite) { .name = strDup(name), .fd = fd, .timeout = timeout, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_WRITE, ioWriteNewP(this, .fd = ioFdWriteFd, .ready = ioFdWriteReady, .write = ioFdWrite)); } /**********************************************************************************************************************************/ FN_EXTERN void ioFdWriteOneStr(const int fd, const String *const string) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(STRING, string); FUNCTION_LOG_END(); ASSERT(string != NULL); if (write(fd, strZ(string), strSize(string)) != (int)strSize(string)) THROW_SYS_ERROR(FileWriteError, "unable to write to fd"); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/fdWrite.h0000664000175100017510000000270015240547630017655 0ustar00runnerrunner/*********************************************************************************************************************************** File Descriptor Io Write Write to a file descriptor using the IoWrite interface. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FDWRITE_H #define COMMON_IO_FDWRITE_H #include "common/io/write.h" #include "common/time.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoWrite *ioFdWriteNew(const String *name, int fd, TimeMSec timeout); // Construct and open write fd FN_INLINE_ALWAYS IoWrite * ioFdWriteNewOpen(const String *const name, const int fd, const TimeMSec timeout) { IoWrite *const result = ioFdWriteNew(name, fd, timeout); ioWriteOpen(result); return result; } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Write a string to the specified file descriptor FN_EXTERN void ioFdWriteOneStr(int fd, const String *string); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/0000775000175100017510000000000015240547630017366 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/buffer.c0000664000175100017510000001133315240547630021004 0ustar00runnerrunner/*********************************************************************************************************************************** IO Buffer Filter ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/filter/buffer.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define BUFFER_FILTER_TYPE STRID5("buffer", 0x24531aa20) /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoBuffer { size_t inputPos; // Position in input buffer bool inputSame; // Is the same input required again? } IoBuffer; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void ioBufferToLog(const IoBuffer *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{inputSame: %s, inputPos: %zu}", cvtBoolToConstZ(this->inputSame), this->inputPos); } #define FUNCTION_LOG_IO_BUFFER_TYPE \ IoBuffer * #define FUNCTION_LOG_IO_BUFFER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioBufferToLog, buffer, bufferSize) /*********************************************************************************************************************************** Move data from the input buffer to the output buffer ***********************************************************************************************************************************/ static void ioBufferProcess(THIS_VOID, const Buffer *const input, Buffer *const output) { THIS(IoBuffer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_BUFFER, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_PARAM(BUFFER, output); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); ASSERT(output != NULL); // Determine how much data needs to be copied and reduce if there is not enough space in the output size_t copySize = bufUsed(input) - this->inputPos; if (copySize > bufRemains(output)) copySize = bufRemains(output); // Copy data to the output buffer bufCatSub(output, input, this->inputPos, copySize); // If all data was copied then reset inputPos and allow new input if (this->inputPos + copySize == bufUsed(input)) { this->inputSame = false; this->inputPos = 0; } // Else update inputPos and indicate that the same input should be passed again else { this->inputSame = true; this->inputPos += copySize; } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Is the same input required again? If the remaining space in the output buffer is smaller than the used space in the input buffer then the same input must be provided again. ***********************************************************************************************************************************/ static bool ioBufferInputSame(const THIS_VOID) { THIS(const IoBuffer); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_BUFFER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->inputSame); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * ioBufferNew(void) { FUNCTION_LOG_VOID(logLevelTrace); OBJ_NEW_BEGIN(IoBuffer) { *this = (IoBuffer){0}; } OBJ_NEW_END(); FUNCTION_LOG_RETURN( IO_FILTER, ioFilterNewP(BUFFER_FILTER_TYPE, this, NULL, .inOut = ioBufferProcess, .inputSame = ioBufferInputSame)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/buffer.h0000664000175100017510000000175615240547630021021 0ustar00runnerrunner/*********************************************************************************************************************************** IO Buffer Filter Move data from the input buffer to the output buffer without overflowing the output buffer. Automatically used as the last filter in a FilterGroup if the last filter is not already an InOut filter, so there is no reason to add it manually to a FilterGroup. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_BUFFER_H #define COMMON_IO_FILTER_BUFFER_H #include "common/io/filter/filter.h" #include "common/type/buffer.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *ioBufferNew(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/filter.c0000664000175100017510000001474615240547630021033 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/memContext.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoFilter { IoFilterPub pub; // Publicly accessible variables bool flushing; // Has the filter started flushing? }; /*********************************************************************************************************************************** Allocations will be in the memory context of the caller. ***********************************************************************************************************************************/ FN_EXTERN IoFilter * ioFilterNew(const StringId type, void *const driver, Pack *const paramList, const IoFilterInterface interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(PACK, paramList); FUNCTION_LOG_PARAM(IO_FILTER_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(type != 0); ASSERT(driver != NULL); // One of processIn or processInOut must be set ASSERT(interface.in != NULL || interface.inOut != NULL); // But not both of them ASSERT(!(interface.in != NULL && interface.inOut != NULL)); // If the filter does not produce output then it should produce a result ASSERT(interface.in == NULL || (interface.result != NULL && interface.done == NULL && interface.inputSame == NULL)); OBJ_NEW_BEGIN(IoFilter, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoFilter) { .pub = { .type = type, .driver = objMoveToInterface(driver, this, memContextPrior()), .paramList = pckMove(paramList, objMemContext(this)), .interface = interface, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterProcessIn(IoFilter *const this, const Buffer *const input) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_PARAM(BUFFER, input); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->pub.interface.in != NULL); CHECK(AssertError, input == NULL || !bufEmpty(input), "expected data or flush"); CHECK(AssertError, !this->flushing || input == NULL, "no data allowed after flush"); if (input == NULL) this->flushing = true; else this->pub.interface.in(this->pub.driver, input); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterProcessInOut(IoFilter *const this, const Buffer *const input, Buffer *const output) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_PARAM(BUFFER, input); FUNCTION_TEST_PARAM(BUFFER, output); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(output != NULL); ASSERT(this->pub.interface.inOut != NULL); CHECK(AssertError, input == NULL || !bufEmpty(input), "expected data or flush"); CHECK(AssertError, !this->flushing || input == NULL, "no data allowed after flush"); if (input == NULL && !this->flushing) this->flushing = true; if (!ioFilterDone(this)) this->pub.interface.inOut(this->pub.driver, input, output); // If input same is requested then there must be some output otherwise there is no point in requesting the same input CHECK(AssertError, !ioFilterInputSame(this) || !bufEmpty(output), "expected input to be consumed or some output"); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** If done is not defined by the filter then check inputSame. If inputSame is true then the filter is not done. Even if the filter is done the interface will not report done until the interface is flushing. ***********************************************************************************************************************************/ FN_EXTERN bool ioFilterDone(const IoFilter *this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); bool result = false; if (this->flushing) result = this->pub.interface.done != NULL ? this->pub.interface.done(this->pub.driver) : !ioFilterInputSame(this); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN bool ioFilterInputSame(const IoFilter *this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->pub.interface.inputSame != NULL ? this->pub.interface.inputSame(this->pub.driver) : false); } /**********************************************************************************************************************************/ FN_EXTERN Pack * ioFilterResult(const IoFilter *this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(PACK, this->pub.interface.result ? this->pub.interface.result(this->pub.driver) : NULL); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterToLog(const IoFilter *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(ioFilterType(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/filter.h0000664000175100017510000000537615240547630021037 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Interface Filters can modify an i/o stream (e.g. GzCompress, GzDecompress), generate a result (e.g. IoSize, CryptoHash), or even do both. A filter is created using a constructor implemented by each filter (e.g. ioBufferNew). Filter processing is managed by IoFilterGroup so the only user facing functions are ioFilterResult() and ioFilterType(). Information on implementing a filter is in filter.internal.h. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_FILTER_H #define COMMON_IO_FILTER_FILTER_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoFilter IoFilter; #include "common/io/filter/filter.intern.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Get filter result FN_EXTERN Pack *ioFilterResult(const IoFilter *this); // Identifies the filter and is used when pulling results from the filter group FN_INLINE_ALWAYS StringId ioFilterType(const IoFilter *const this) { return THIS_PUB(IoFilter)->type; } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioFilterFree(IoFilter *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioFilterToLog(const IoFilter *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_FILTER_TYPE \ IoFilter * #define FUNCTION_LOG_IO_FILTER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioFilterToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/filter.intern.h0000664000175100017510000001503215240547630022323 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Interface Internal Two types of filters are implemented using this interface: In and InOut. In filters accept input and produce a result, but do not modify the input. An example is the IoSize filter which counts all bytes that pass through it. InOut filters accept input and produce output (and perhaps a result). Because the input/output buffers may not be the same size the filter must be prepared to accept the same input again (by implementing IoFilterInputSame) if the output buffer is too small to accept all processed data. If the filter holds state even when inputSame is false then it may also implement IoFilterDone to indicate that the filter should be flushed (by passing NULL inputs) after all input has been processed. InOut filters should strive to fill the output buffer as much as possible, i.e., if the output buffer is not full after processing then inputSame should be false. An example is the IoBuffer filter which buffers data between unequally sized input/output buffers. Each filter has a type that allows it to be identified in the filter list. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_FILTER_INTERN_H #define COMMON_IO_FILTER_FILTER_INTERN_H #include "common/type/buffer.h" #include "common/type/pack.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct IoFilterInterface { // Indicates that filter processing is done. This is used for filters that have additional data to be flushed even after all // input has been processed. Compression and encryption filters will usually need to implement done. If done is not implemented // then it will always return true if all input has been consumed, i.e. if inputSame returns false. bool (*done)(const void *driver); // Processing function for filters that do not produce output. Note that result must be implemented in this case (or else what // would be the point. void (*in)(void *driver, const Buffer *); // Processing function for filters that produce output. InOut filters will typically implement inputSame and may also implement // done. void (*inOut)(void *driver, const Buffer *, Buffer *); // InOut filters must be prepared for an output buffer that is too small to accept all the processed output. In this case the // filter must implement inputSame and set it to true when there is more output to be produced for a given input. On the next // call to inOut the same input will be passed along with a fresh output buffer with space for more processed output. bool (*inputSame)(const void *driver); // If the filter produces a result then this function must be implemented to return the result. A result can be anything that // is not processed output, e.g. a count of total bytes or a cryptographic hash. The returned buffer must be a pack containing // the result. Pack *(*result)(void *driver); } IoFilterInterface; #define ioFilterNewP(type, driver, paramList, ...) \ ioFilterNew(type, driver, paramList, (IoFilterInterface){__VA_ARGS__}) FN_EXTERN IoFilter *ioFilterNew(StringId type, void *driver, Pack *paramList, IoFilterInterface); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoFilterPub { StringId type; // Filter type IoFilterInterface interface; // Filter interface void *driver; // Filter driver const Pack *paramList; // Filter parameters } IoFilterPub; // Is the filter done? FN_EXTERN bool ioFilterDone(const IoFilter *this); // Driver for the filter FN_INLINE_ALWAYS void * ioFilterDriver(IoFilter *const this) { return THIS_PUB(IoFilter)->driver; } // Does the filter need the same input again? If the filter cannot get all its output into the output buffer then it may need access // to the same input again. FN_EXTERN bool ioFilterInputSame(const IoFilter *this); // Interface for the filter FN_INLINE_ALWAYS const IoFilterInterface * ioFilterInterface(const IoFilter *const this) { return &THIS_PUB(IoFilter)->interface; } // Does filter produce output? All In filters produce output. FN_INLINE_ALWAYS bool ioFilterOutput(const IoFilter *const this) { return THIS_PUB(IoFilter)->interface.inOut != NULL; } // List of filter parameters FN_INLINE_ALWAYS const Pack * ioFilterParamList(const IoFilter *const this) { return THIS_PUB(IoFilter)->paramList; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Filter input only (a result is expected) FN_EXTERN void ioFilterProcessIn(IoFilter *this, const Buffer *input); // Filter input and produce output FN_EXTERN void ioFilterProcessInOut(IoFilter *this, const Buffer *input, Buffer *output); // Move filter to a new parent mem context FN_INLINE_ALWAYS IoFilter * ioFilterMove(IoFilter *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_FILTER_INTERFACE_TYPE \ IoFilterInterface * #define FUNCTION_LOG_IO_FILTER_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoFilterInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/group.c0000664000175100017510000005112215240547630020667 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Group ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/filter/buffer.h" #include "common/io/filter/filter.h" #include "common/io/filter/group.h" #include "common/io/io.h" #include "common/log.h" #include "common/type/list.h" /*********************************************************************************************************************************** Filter and buffer structure Contains the filter object and inout/output buffers. ***********************************************************************************************************************************/ typedef struct IoFilterData { const Buffer *const *input; // Input buffer for filter Buffer *inputLocal; // Non-null if a locally created buffer that can be cleared IoFilter *filter; // Filter to apply Buffer *output; // Output buffer for filter } IoFilterData; // Macros for logging #define FUNCTION_LOG_IO_FILTER_DATA_TYPE \ IoFilterData * #define FUNCTION_LOG_IO_FILTER_DATA_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoFilterData", buffer, bufferSize) /*********************************************************************************************************************************** Filter results ***********************************************************************************************************************************/ typedef struct IoFilterResult { StringId type; // Filter type Pack *result; // Filter result } IoFilterResult; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoFilterGroup { IoFilterGroupPub pub; // Publicly accessible variables const Buffer *input; // Input buffer passed in for processing List *filterResult; // Filter results (if any) #ifdef DEBUG bool flushing; // Is output being flushed? #endif }; /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * ioFilterGroupNew(void) { FUNCTION_LOG_VOID(logLevelTrace); OBJ_NEW_BEGIN(IoFilterGroup, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoFilterGroup) { .pub = { .done = false, .filterList = lstNewP(sizeof(IoFilterData)), }, .filterResult = lstNewP(sizeof(IoFilterResult)), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, this); } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * ioFilterGroupAdd(IoFilterGroup *const this, IoFilter *const filter) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(IO_FILTER, filter); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened && !this->pub.closed); ASSERT(filter != NULL); // Move the filter to this object's mem context ioFilterMove(filter, objMemContext(this)); // Add the filter IoFilterData filterData = {.filter = filter}; lstAdd(this->pub.filterList, &filterData); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, this); } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * ioFilterGroupInsert(IoFilterGroup *const this, const unsigned int listIdx, IoFilter *const filter) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(IO_FILTER, filter); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened && !this->pub.closed); ASSERT(filter != NULL); // Move the filter to this object's mem context ioFilterMove(filter, objMemContext(this)); // Add the filter IoFilterData filterData = {.filter = filter}; lstInsert(this->pub.filterList, listIdx, &filterData); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, this); } /*********************************************************************************************************************************** Get a filter ***********************************************************************************************************************************/ static IoFilterData * ioFilterGroupGet(const IoFilterGroup *const this, const unsigned int filterIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER_GROUP, this); FUNCTION_TEST_PARAM(UINT, filterIdx); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_FILTER_DATA, (IoFilterData *)lstGet(this->pub.filterList, filterIdx)); } /**********************************************************************************************************************************/ FN_EXTERN IoFilterGroup * ioFilterGroupClear(IoFilterGroup *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened); for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) ioFilterFree(ioFilterGroupGet(this, filterIdx)->filter); lstClear(this->pub.filterList); FUNCTION_LOG_RETURN(IO_FILTER_GROUP, this); } /*********************************************************************************************************************************** Setup the filter group and allocate any required buffers ***********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupOpen(IoFilterGroup *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { // If the last filter is not an output filter then add a filter to buffer/copy data. Input filters won't copy to an output // buffer so we need some way to get the data to the output buffer. if (ioFilterGroupSize(this) == 0 || !ioFilterOutput((ioFilterGroupGet(this, ioFilterGroupSize(this) - 1))->filter)) { ioFilterGroupAdd(this, ioBufferNew()); } // Create filter input/output buffers. Input filters do not get an output buffer since they don't produce output. Buffer **lastOutputBuffer = NULL; for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) { IoFilterData *const filterData = ioFilterGroupGet(this, filterIdx); // If there is no last output buffer yet, then use the input buffer that will be provided by the caller if (lastOutputBuffer == NULL) { filterData->input = &this->input; } // Else assign the last output buffer to the input else { // This cast is required because the compiler can't guarantee the const-ness of this object, i.e. it could be // modified in other parts of the code. This is actually expected and the only reason we need this const is to match // the const-ness of the input buffer provided by the caller. filterData->input = (const Buffer *const *)lastOutputBuffer; filterData->inputLocal = *lastOutputBuffer; } // If this is not the last output filter then create a new output buffer for it. The output buffer for the last filter // will be provided to the process function. ASSERT(ioFilterGroupSize(this) != 0); if (ioFilterOutput(filterData->filter) && filterIdx < ioFilterGroupSize(this) - 1) { filterData->output = bufNew(ioBufferSize()); lastOutputBuffer = &filterData->output; } } } MEM_CONTEXT_OBJ_END(); // Filter group is open #ifdef DEBUG this->pub.opened = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupProcess(IoFilterGroup *const this, const Buffer *const input, Buffer *const output) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_PARAM(BUFFER, output); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); ASSERT(input == NULL || !bufEmpty(input)); ASSERT(!this->flushing || input == NULL); ASSERT(output != NULL); ASSERT(bufRemains(output) > 0); // Once input goes to NULL then flushing has started #ifdef DEBUG if (input == NULL) this->flushing = true; #endif // Assign input and output buffers this->input = input; (ioFilterGroupGet(this, ioFilterGroupSize(this) - 1))->output = output; // do { // Start from the first filter by default unsigned int filterIdx = 0; // Search from the end of the list for a filter that needs the same input. This indicates that the filter was not able to // empty the input buffer on the last call. Maybe it won't this time either -- we can but try. if (ioFilterGroupInputSame(this)) { this->pub.inputSame = false; filterIdx = ioFilterGroupSize(this); do { filterIdx--; if (ioFilterInputSame((ioFilterGroupGet(this, filterIdx))->filter)) { this->pub.inputSame = true; break; } } while (filterIdx != 0); // If no filter is found that needs the same input that means we are done with the current input. So end the loop and // get some more input. if (!ioFilterGroupInputSame(this)) break; } // Process forward from the filter that has input to process. This may be a filter that needs the same input or it may be // new input for the first filter. for (; filterIdx < ioFilterGroupSize(this); filterIdx++) { IoFilterData *const filterData = ioFilterGroupGet(this, filterIdx); // Process the filter if it is not done if (!ioFilterDone(filterData->filter)) { // If the filter produces output if (ioFilterOutput(filterData->filter)) { ioFilterProcessInOut(filterData->filter, *filterData->input, filterData->output); // If inputSame is set then the output buffer for this filter is full and it will need to be re-processed with // the same input once the output buffer is cleared if (ioFilterInputSame(filterData->filter)) { this->pub.inputSame = true; } // Else clear the buffer if it was locally allocated. If the input buffer was passed in then the caller is // responsible for clearing it. else if (filterData->inputLocal != NULL) bufUsedZero(filterData->inputLocal); // If the output buffer is not full and the filter is not done then more data is required if (!bufFull(filterData->output) && !ioFilterDone(filterData->filter)) break; } // Else the filter does not produce output else ioFilterProcessIn(filterData->filter, *filterData->input); } // If the filter is done and has no more output then null the output buffer. Downstream filters have a pointer to this // buffer so their inputs will also change to null and they'll flush. if (filterData->output != NULL && ioFilterDone(filterData->filter) && bufUsed(filterData->output) == 0) filterData->output = NULL; } } while (!bufFull(output) && ioFilterGroupInputSame(this)); // Scan the filter list to determine if inputSame is set or done is not set for any filter. We can't trust // ioFilterGroupInputSame() when it is true without going through the loop above again. We need to scan to set this->pub.done // anyway so set this->pub.inputSame in the same loop. this->pub.done = true; this->pub.inputSame = false; for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) { const IoFilterData *const filterData = ioFilterGroupGet(this, filterIdx); // When inputSame then this->pub.done = false and we can exit the loop immediately if (ioFilterInputSame(filterData->filter)) { this->pub.done = false; this->pub.inputSame = true; break; } // Set this->pub.done = false if any filter is not done if (!ioFilterDone(filterData->filter)) this->pub.done = false; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupClose(IoFilterGroup *this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Gather results from the filters for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) { const IoFilter *const filter = ioFilterGroupGet(this, filterIdx)->filter; MEM_CONTEXT_BEGIN(lstMemContext(this->filterResult)) { lstAdd(this->filterResult, &(IoFilterResult){.type = ioFilterType(filter), .result = ioFilterResult(filter)}); } MEM_CONTEXT_END(); } // Filter group is open #ifdef DEBUG this->pub.closed = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Pack * ioFilterGroupParamAll(const IoFilterGroup *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened); ASSERT(this->pub.filterList != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); for (unsigned int filterIdx = 0; filterIdx < ioFilterGroupSize(this); filterIdx++) { IoFilter *filter = ioFilterGroupGet(this, filterIdx)->filter; pckWriteStrIdP(packWrite, ioFilterType(filter)); pckWritePackP(packWrite, ioFilterParamList(filter)); } pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN const Pack * ioFilterGroupResultPack(const IoFilterGroup *const this, const StringId filterType, const IoFilterGroupResultParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(STRING_ID, filterType); FUNCTION_LOG_PARAM(UINT, param.idx); FUNCTION_LOG_END(); ASSERT(this->pub.opened); ASSERT(filterType != 0); const Pack *result = NULL; // Search for the result unsigned int foundIdx = 0; for (unsigned int filterResultIdx = 0; filterResultIdx < lstSize(this->filterResult); filterResultIdx++) { const IoFilterResult *const filterResult = lstGet(this->filterResult, filterResultIdx); // If the filter matches check the index if (filterResult->type == filterType) { // If the index matches return the result if (foundIdx == param.idx) { result = filterResult->result; break; } // Increment the index and keep searching foundIdx++; } } FUNCTION_LOG_RETURN_CONST(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * ioFilterGroupResult(const IoFilterGroup *const this, const StringId filterType, const IoFilterGroupResultParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(STRING_ID, filterType); FUNCTION_LOG_PARAM(UINT, param.idx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(PACK_READ, pckReadNew(ioFilterGroupResultPack(this, filterType, param))); } /**********************************************************************************************************************************/ FN_EXTERN Pack * ioFilterGroupResultAll(const IoFilterGroup *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.closed); Pack *result = NULL; // Pack the result list MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); for (unsigned int filterResultIdx = 0; filterResultIdx < lstSize(this->filterResult); filterResultIdx++) { const IoFilterResult *const filterResult = lstGet(this->filterResult, filterResultIdx); pckWriteStrIdP(packWrite, filterResult->type); pckWritePackP(packWrite, filterResult->result); } pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupResultAllSet(IoFilterGroup *const this, const Pack *const filterResultPack) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_FILTER_GROUP, this); FUNCTION_LOG_PARAM(PACK, filterResultPack); FUNCTION_LOG_END(); ASSERT(this != NULL); if (filterResultPack != NULL) { // Unpack the results into a list MEM_CONTEXT_TEMP_BEGIN() { PackRead *const packRead = pckReadNew(filterResultPack); MEM_CONTEXT_BEGIN(lstMemContext(this->filterResult)) { while (!pckReadNullP(packRead)) { const StringId type = pckReadStrIdP(packRead); Pack *const result = pckReadPackP(packRead); lstAdd(this->filterResult, &(IoFilterResult){.type = type, .result = result}); } } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupToLog(const IoFilterGroup *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{inputSame: %s, done: %s" #ifdef DEBUG ", opened %s, flushing %s, closed %s" #endif "}", cvtBoolToConstZ(this->pub.inputSame), cvtBoolToConstZ(this->pub.done) #ifdef DEBUG , cvtBoolToConstZ(this->pub.opened), cvtBoolToConstZ(this->flushing), cvtBoolToConstZ(this->pub.closed) #endif ); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/group.h0000664000175100017510000001460515240547630020701 0ustar00runnerrunner/*********************************************************************************************************************************** IO Filter Group Process data through an arbitrary group of filters in the order added by the user using ioFilterGroupAdd(). After processing results can be gathered using ioFilterGroupResult() for any filters that produce results. Processing is complex and asymmetric for read/write so should be done via the IoRead and IoWrite objects. General users need only call ioFilterGroupNew(), ioFilterGroupAdd(), and ioFilterGroupResult(). ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_GROUP_H #define COMMON_IO_FILTER_GROUP_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoFilterGroup IoFilterGroup; #include "common/io/filter/filter.h" #include "common/type/list.h" #include "common/type/object.h" #include "common/type/pack.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilterGroup *ioFilterGroupNew(void); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoFilterGroupPub { List *filterList; // List of filters to apply bool inputSame; // Same input required again? bool done; // Is processing done? #ifdef DEBUG bool opened; // Has the filter set been opened? bool closed; // Has the filter set been closed? #endif } IoFilterGroupPub; // Is the filter group done processing? FN_INLINE_ALWAYS bool ioFilterGroupDone(const IoFilterGroup *const this) { ASSERT_INLINE(THIS_PUB(IoFilterGroup)->opened && !THIS_PUB(IoFilterGroup)->closed); return THIS_PUB(IoFilterGroup)->done; } // Should the same input be passed again? A buffer of input can produce multiple buffers of output, e.g. when a file containing all // zeroes is being decompressed. FN_INLINE_ALWAYS bool ioFilterGroupInputSame(const IoFilterGroup *const this) { ASSERT_INLINE(THIS_PUB(IoFilterGroup)->opened && !THIS_PUB(IoFilterGroup)->closed); return THIS_PUB(IoFilterGroup)->inputSame; } // Get all filters and their parameters so they can be passed to a remote FN_EXTERN Pack *ioFilterGroupParamAll(const IoFilterGroup *this); // Get filter results. If the same filter was used more than once then idx can be used to specify which one to get. typedef struct IoFilterGroupResultParam { VAR_PARAM_HEADER; unsigned int idx; } IoFilterGroupResultParam; #define ioFilterGroupResultP(this, filterType, ...) \ ioFilterGroupResult(this, filterType, (IoFilterGroupResultParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *ioFilterGroupResult(const IoFilterGroup *this, StringId filterType, IoFilterGroupResultParam param); #define ioFilterGroupResultPackP(this, filterType, ...) \ ioFilterGroupResultPack(this, filterType, (IoFilterGroupResultParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN const Pack *ioFilterGroupResultPack(const IoFilterGroup *this, StringId filterType, IoFilterGroupResultParam param); // Get/set all filter results FN_EXTERN Pack *ioFilterGroupResultAll(const IoFilterGroup *this); FN_EXTERN void ioFilterGroupResultAllSet(IoFilterGroup *this, const Pack *filterResult); // Return total number of filters FN_INLINE_ALWAYS unsigned int ioFilterGroupSize(const IoFilterGroup *const this) { return lstSize(THIS_PUB(IoFilterGroup)->filterList); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a filter FN_EXTERN IoFilterGroup *ioFilterGroupAdd(IoFilterGroup *this, IoFilter *filter); // Insert a filter before an index FN_EXTERN IoFilterGroup *ioFilterGroupInsert(IoFilterGroup *this, unsigned int listIdx, IoFilter *filter); // Clear filters FN_EXTERN IoFilterGroup *ioFilterGroupClear(IoFilterGroup *this); // Open filter group FN_EXTERN void ioFilterGroupOpen(IoFilterGroup *this); // Process filters FN_EXTERN void ioFilterGroupProcess(IoFilterGroup *this, const Buffer *input, Buffer *output); // Close filter group and gather results FN_EXTERN void ioFilterGroupClose(IoFilterGroup *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioFilterGroupFree(IoFilterGroup *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioFilterGroupToLog(const IoFilterGroup *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_FILTER_GROUP_TYPE \ IoFilterGroup * #define FUNCTION_LOG_IO_FILTER_GROUP_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioFilterGroupToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/sink.c0000664000175100017510000000501415240547630020476 0ustar00runnerrunner/*********************************************************************************************************************************** IO Sink Filter ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/io/filter/sink.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoSink { bool dummy; // Struct requires one member } IoSink; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_SINK_TYPE \ IoSink * #define FUNCTION_LOG_IO_SINK_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoSink", buffer, bufferSize) /*********************************************************************************************************************************** Discard all input ***********************************************************************************************************************************/ static void ioSinkProcess(THIS_VOID, const Buffer *const input, Buffer *const output) { THIS(IoSink); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_SINK, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_PARAM(BUFFER, output); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); ASSERT(output != NULL); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * ioSinkNew(void) { FUNCTION_LOG_VOID(logLevelTrace); OBJ_NEW_BEGIN(IoSink) { } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER, ioFilterNewP(SINK_FILTER_TYPE, this, NULL, .inOut = ioSinkProcess)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/sink.h0000664000175100017510000000233215240547630020503 0ustar00runnerrunner/*********************************************************************************************************************************** IO Sink Filter Consume all bytes sent to the filter without passing any on. This filter is useful when running size/hash filters on a remote when no data should be returned. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_SINK_H #define COMMON_IO_FILTER_SINK_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define SINK_FILTER_TYPE STRID5("sink", 0x5b9330) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *ioSinkNew(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/size.c0000664000175100017510000000702215240547630020505 0ustar00runnerrunner/*********************************************************************************************************************************** IO Size Filter ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/io/filter/size.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoSize { uint64_t size; // Total size of all input } IoSize; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void ioSizeToLog(const IoSize *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{size: %" PRIu64 "}", this->size); } #define FUNCTION_LOG_IO_SIZE_TYPE \ IoSize * #define FUNCTION_LOG_IO_SIZE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioSizeToLog, buffer, bufferSize) /*********************************************************************************************************************************** Count bytes in the input ***********************************************************************************************************************************/ static void ioSizeProcess(THIS_VOID, const Buffer *const input) { THIS(IoSize); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_SIZE, this); FUNCTION_LOG_PARAM(BUFFER, input); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(input != NULL); this->size += bufUsed(input); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Return filter result ***********************************************************************************************************************************/ static Pack * ioSizeResult(THIS_VOID) { THIS(IoSize); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_SIZE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = pckWriteNewP(); pckWriteU64P(packWrite, this->size); pckWriteEndP(packWrite); result = pckMove(pckWriteResult(packWrite), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN IoFilter * ioSizeNew(void) { FUNCTION_LOG_VOID(logLevelTrace); OBJ_NEW_BEGIN(IoSize) { *this = (IoSize){0}; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_FILTER, ioFilterNewP(SIZE_FILTER_TYPE, this, NULL, .in = ioSizeProcess, .result = ioSizeResult)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/filter/size.h0000664000175100017510000000233415240547630020513 0ustar00runnerrunner/*********************************************************************************************************************************** IO Size Filter Count all bytes that pass through the filter. Useful for getting file/IO size if added first in a FilterGroup with IoRead or last in a FilterGroup with IoWrite. ***********************************************************************************************************************************/ #ifndef COMMON_IO_FILTER_SIZE_H #define COMMON_IO_FILTER_SIZE_H #include "common/io/filter/filter.h" /*********************************************************************************************************************************** Filter type constant ***********************************************************************************************************************************/ #define SIZE_FILTER_TYPE STRID5("size", 0x2e9330) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoFilter *ioSizeNew(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/0000775000175100017510000000000015240547630017060 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/client.c0000664000175100017510000001101215240547630020475 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Client ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/client.h" #include "common/io/http/client.h" #include "common/log.h" #include "common/stat.h" #include "common/type/object.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(HTTP_STAT_CLIENT_STR, HTTP_STAT_CLIENT); STRING_EXTERN(HTTP_STAT_CLOSE_STR, HTTP_STAT_CLOSE); STRING_EXTERN(HTTP_STAT_REQUEST_STR, HTTP_STAT_REQUEST); STRING_EXTERN(HTTP_STAT_RETRY_STR, HTTP_STAT_RETRY); STRING_EXTERN(HTTP_STAT_SESSION_STR, HTTP_STAT_SESSION); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpClient { HttpClientPub pub; // Publicly accessible variables IoClient *ioClient; // Io client (e.g. TLS or socket client) List *sessionReuseList; // List of HTTP sessions that can be reused }; /**********************************************************************************************************************************/ FN_EXTERN HttpClient * httpClientNew(IoClient *const ioClient, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_CLIENT, ioClient); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(ioClient != NULL); OBJ_NEW_BEGIN(HttpClient, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpClient) { .pub = { .timeout = timeout, }, .ioClient = ioClient, .sessionReuseList = lstNewP(sizeof(HttpSession *)), }; statInc(HTTP_STAT_CLIENT_STR); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(HTTP_CLIENT, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpSession * httpClientOpen(HttpClient *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); HttpSession *result = NULL; // Check if there is a reusable session if (!lstEmpty(this->sessionReuseList)) { // Remove session from reusable list result = *(HttpSession **)lstGet(this->sessionReuseList, 0); lstRemoveIdx(this->sessionReuseList, 0); // Move session to the calling context httpSessionMove(result, memContextCurrent()); } // Else create a new session else { result = httpSessionNew(this, ioClientOpen(this->ioClient)); statInc(HTTP_STAT_SESSION_STR); } FUNCTION_LOG_RETURN(HTTP_SESSION, result); } /**********************************************************************************************************************************/ FN_EXTERN void httpClientReuse(HttpClient *const this, HttpSession *const session) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_CLIENT, this); FUNCTION_LOG_PARAM(HTTP_SESSION, session); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(session != NULL); httpSessionMove(session, lstMemContext(this->sessionReuseList)); lstAdd(this->sessionReuseList, &session); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void httpClientToLog(const HttpClient *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{ioClient: "); ioClientToLog(this->ioClient, debugLog); strStcFmt(debugLog, ", reusable: %u, timeout: %" PRIu64 "}", lstSize(this->sessionReuseList), httpClientTimeout(this)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/client.h0000664000175100017510000001124715240547630020514 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Client A robust HTTP client with connection reuse and automatic retries. Using a single object to make multiple requests is more efficient because connections are reused whenever possible. Requests are automatically retried when the connection has been closed by the server. Any 5xx response is also retried. Only the HTTPS protocol is currently supported. IMPORTANT NOTE: HttpClient should have a longer lifetime than any active HttpSession objects. This does not apply to HttpSession objects that are freed, i.e. if an error occurs it does not matter in what order HttpClient and HttpSession objects are destroyed, or HttpSession objects that have been returned to the client with httpClientReuse(). The danger is when an active HttpResponse completes and tries to call httpClientReuse() on an HttpClient that has been freed thus causing a segfault. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_CLIENT_H #define COMMON_IO_HTTP_CLIENT_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpClient HttpClient; #include "common/io/client.h" #include "common/io/http/session.h" #include "common/time.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define HTTP_STAT_CLIENT "http.client" // Clients created STRING_DECLARE(HTTP_STAT_CLIENT_STR); #define HTTP_STAT_CLOSE "http.close" // Closes forced by server STRING_DECLARE(HTTP_STAT_CLOSE_STR); #define HTTP_STAT_REQUEST "http.request" // Requests (i.e. calls to httpRequestNew()) STRING_DECLARE(HTTP_STAT_REQUEST_STR); #define HTTP_STAT_RETRY "http.retry" // Request retries STRING_DECLARE(HTTP_STAT_RETRY_STR); #define HTTP_STAT_SESSION "http.session" // Sessions created STRING_DECLARE(HTTP_STAT_SESSION_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpClient *httpClientNew(IoClient *ioClient, TimeMSec timeout); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct HttpClientPub { TimeMSec timeout; // Request timeout } HttpClientPub; FN_INLINE_ALWAYS TimeMSec httpClientTimeout(const HttpClient *const this) { return THIS_PUB(HttpClient)->timeout; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open a new session FN_EXTERN HttpSession *httpClientOpen(HttpClient *this); // Request/response finished cleanly so session can be reused FN_EXTERN void httpClientReuse(HttpClient *this, HttpSession *session); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpClientToLog(const HttpClient *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_CLIENT_TYPE \ HttpClient * #define FUNCTION_LOG_HTTP_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpClientToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/common.c0000664000175100017510000001151415240547630020516 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Common ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/http/common.h" #include "common/time.h" /*********************************************************************************************************************************** Convert the time using the format specified in https://tools.ietf.org/html/rfc7231#section-7.1.1.1 which is used by HTTP 1.1 (the only version we support). ***********************************************************************************************************************************/ static const char *const httpCommonMonthList[] = { "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec" }; static const char *const httpCommonDayList[] = {"Sun", "Mon", "Tue", "Wed", "Thu", "Fri", "Sat"}; FN_EXTERN time_t httpDateToTime(const String *const lastModified) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, lastModified); FUNCTION_TEST_END(); // Find the month const char *const month = strZ(lastModified) + 8; unsigned int monthIdx = 0; for (; monthIdx < LENGTH_OF(httpCommonMonthList); monthIdx++) { if (strncmp(month, httpCommonMonthList[monthIdx], 3) == 0) break; } if (monthIdx == LENGTH_OF(httpCommonMonthList)) THROW_FMT(FormatError, "invalid month '%.3s'", month); // Convert to time_t FUNCTION_TEST_RETURN( TIME, epochFromParts( cvtZSubNToInt(strZ(lastModified), 12, 4), (int)monthIdx + 1, cvtZSubNToInt(strZ(lastModified), 5, 2), cvtZSubNToInt(strZ(lastModified), 17, 2), cvtZSubNToInt(strZ(lastModified), 20, 2), cvtZSubNToInt(strZ(lastModified), 23, 2), 0)); } FN_EXTERN String * httpDateFromTime(const time_t time) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TIME, time); FUNCTION_TEST_END(); struct tm timePart; gmtime_r(&time, &timePart); FUNCTION_TEST_RETURN( STRING, strNewFmt( "%s, %02d %s %04d %02d:%02d:%02d GMT", httpCommonDayList[timePart.tm_wday], timePart.tm_mday, httpCommonMonthList[timePart.tm_mon], timePart.tm_year + 1900, timePart.tm_hour, timePart.tm_min, timePart.tm_sec)); } /**********************************************************************************************************************************/ FN_EXTERN String * httpUriDecode(const String *const uri) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, uri); FUNCTION_TEST_END(); String *result = NULL; // Decode if the string is not null if (uri != NULL) { result = strNew(); // Iterate all characters in the string for (unsigned uriIdx = 0; uriIdx < strSize(uri); uriIdx++) { char uriChar = strZ(uri)[uriIdx]; // Convert escaped characters if (uriChar == '%') { // Sequence must be exactly three characters (% and two hex digits) if (strSize(uri) - uriIdx < 3) THROW_FMT(FormatError, "invalid escape sequence length in '%s'", strZ(uri)); // Convert hex digits uriChar = (char)cvtZSubNToUIntBase(strZ(uri), uriIdx + 1, 2, 16); // Skip to next character or escape uriIdx += 2; } strCatChr(result, uriChar); } } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * httpUriEncode(const String *const uri, const bool path) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, uri); FUNCTION_TEST_PARAM(BOOL, path); FUNCTION_TEST_END(); String *result = NULL; // Encode if the string is not null if (uri != NULL) { result = strNew(); // Iterate all characters in the string for (unsigned uriIdx = 0; uriIdx < strSize(uri); uriIdx++) { const char uriChar = strZ(uri)[uriIdx]; // These characters are reproduced verbatim if ((uriChar >= 'A' && uriChar <= 'Z') || (uriChar >= 'a' && uriChar <= 'z') || (uriChar >= '0' && uriChar <= '9') || uriChar == '_' || uriChar == '-' || uriChar == '~' || uriChar == '.' || (path && uriChar == '/')) { strCatChr(result, uriChar); } // All other characters are hex-encoded else strCatFmt(result, "%%%02X", (uint8_t)uriChar); } } FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/common.h0000664000175100017510000000214115240547630020517 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Common ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_COMMON_H #define COMMON_IO_HTTP_COMMON_H #include #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Convert HTTP date to time_t and vice versa FN_EXTERN time_t httpDateToTime(const String *lastModified); FN_EXTERN String *httpDateFromTime(time_t time); // Encode string to conform with URI specifications. If a path is being encoded then / characters won't be encoded. FN_EXTERN String *httpUriEncode(const String *uri, bool path); // Decode string that conforms to URI specifications FN_EXTERN String *httpUriDecode(const String *uri); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/header.c0000664000175100017510000001646415240547630020467 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Header ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/header.h" #include "common/io/http/request.h" #include "common/type/keyValue.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpHeader { const StringList *redactList; // List of headers to redact during logging KeyValue *kv; // KeyValue store }; /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderNew(const StringList *const redactList) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(HttpHeader, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpHeader) { .redactList = strLstDup(redactList), .kv = kvNew(), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderDup(const HttpHeader *const header, const StringList *const redactList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, header); FUNCTION_TEST_PARAM(STRING_LIST, redactList); FUNCTION_TEST_END(); if (header == NULL) FUNCTION_TEST_RETURN(HTTP_HEADER, NULL); OBJ_NEW_BEGIN(HttpHeader, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpHeader) { .redactList = redactList == NULL ? strLstDup(header->redactList) : strLstDup(redactList), .kv = kvDup(header->kv), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderAdd(HttpHeader *const this, const String *const key, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT(value != NULL); // Check if the key already exists const Variant *const keyVar = VARSTR(key); const Variant *const valueVar = kvGet(this->kv, keyVar); // If the key exists then append the new value. The HTTP spec (RFC 2616, Section 4.2) says that if a header appears more than // once then it is equivalent to a single comma-separated header. There appear to be a few exceptions such as Set-Cookie, but // they should not be of concern to us here. if (valueVar != NULL) { MEM_CONTEXT_TEMP_BEGIN() { String *const valueAppend = strCat(strNew(), varStr(valueVar)); strCatZ(valueAppend, ", "); strCatZ(valueAppend, strZ(value)); kvPut(this->kv, keyVar, VARSTR(valueAppend)); } MEM_CONTEXT_TEMP_END(); } // Else store the key else kvPut(this->kv, keyVar, VARSTR(value)); FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN const String * httpHeaderGet(const HttpHeader *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN_CONST(STRING, varStr(kvGet(this->kv, VARSTR(key)))); } /**********************************************************************************************************************************/ FN_EXTERN StringList * httpHeaderList(const HttpHeader *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING_LIST, strLstSort(strLstNewVarLst(kvKeyList(this->kv)), sortOrderAsc)); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderPut(HttpHeader *const this, const String *const key, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT(value != NULL); // Store the key kvPut(this->kv, VARSTR(key), VARSTR(value)); FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpHeaderPutRange(HttpHeader *const this, const uint64_t offset, const Variant *const limit) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(UINT64, offset); FUNCTION_TEST_PARAM(VARIANT, limit); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(limit == NULL || varType(limit) == varTypeUInt64); if (offset != 0 || limit != NULL) { String *const range = strCatFmt(strNew(), HTTP_HEADER_RANGE_BYTES "=%" PRIu64 "-", offset); if (limit != NULL) strCatFmt(range, "%" PRIu64, offset + varUInt64(limit) - 1); httpHeaderPut(this, HTTP_HEADER_RANGE_STR, range); strFree(range); } FUNCTION_TEST_RETURN(HTTP_HEADER, this); } /**********************************************************************************************************************************/ FN_EXTERN bool httpHeaderRedact(const HttpHeader *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_HEADER, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(BOOL, this->redactList != NULL && strLstExists(this->redactList, key)); } /**********************************************************************************************************************************/ FN_EXTERN void httpHeaderToLog(const HttpHeader *const this, StringStatic *const debugLog) { const VariantList *const keyList = kvKeyList(this->kv); strStcCatChr(debugLog, '{'); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { const String *const key = varStr(varLstGet(keyList, keyIdx)); if (keyIdx != 0) strStcCat(debugLog, ", "); if (httpHeaderRedact(this, key)) strStcFmt(debugLog, "%s: ", strZ(key)); else strStcFmt(debugLog, "%s: '%s'", strZ(key), strZ(httpHeaderGet(this, key))); } strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/header.h0000664000175100017510000000664315240547630020472 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Header Object to track HTTP headers. Headers can be marked as redacted so they are not logged. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_HEADER_H #define COMMON_IO_HTTP_HEADER_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpHeader HttpHeader; #include "common/type/object.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpHeader *httpHeaderNew(const StringList *redactList); FN_EXTERN HttpHeader *httpHeaderDup(const HttpHeader *header, const StringList *redactList); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a header FN_EXTERN HttpHeader *httpHeaderAdd(HttpHeader *this, const String *key, const String *value); // Get a value using the key FN_EXTERN const String *httpHeaderGet(const HttpHeader *this, const String *key); // Get list of keys FN_EXTERN StringList *httpHeaderList(const HttpHeader *this); // Move to a new parent mem context FN_INLINE_ALWAYS HttpHeader * httpHeaderMove(HttpHeader *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Put a header FN_EXTERN HttpHeader *httpHeaderPut(HttpHeader *this, const String *header, const String *value); // Put range header when needed FN_EXTERN HttpHeader *httpHeaderPutRange(HttpHeader *this, uint64_t offset, const Variant *limit); // Should the header be redacted when logging? FN_EXTERN bool httpHeaderRedact(const HttpHeader *this, const String *key); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpHeaderFree(HttpHeader *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpHeaderToLog(const HttpHeader *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_HEADER_TYPE \ HttpHeader * #define FUNCTION_LOG_HTTP_HEADER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpHeaderToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/query.c0000664000175100017510000002327215240547630020377 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Query ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/common.h" #include "common/io/http/query.h" #include "common/type/keyValue.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpQuery { KeyValue *kv; // KeyValue store const StringList *redactList; // List of keys to redact values for }; /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryNew(const HttpQueryNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, param.kv); FUNCTION_TEST_PARAM(STRING_LIST, param.redactList); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(HttpQuery, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpQuery) { .kv = param.kv != NULL ? kvDup(param.kv) : kvNew(), .redactList = strLstDup(param.redactList), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryNewStr(const String *query) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, query); FUNCTION_TEST_END(); ASSERT(query != NULL); OBJ_NEW_BEGIN(HttpQuery, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpQuery) { .kv = kvNew(), }; MEM_CONTEXT_TEMP_BEGIN() { // Remove initial ? when present if (strBeginsWithZ(query, "?")) query = strSub(query, 1); // Split query into individual key value pairs const StringList *const keyValueList = strLstNewSplitZ(query, "&"); for (unsigned int keyValueIdx = 0; keyValueIdx < strLstSize(keyValueList); keyValueIdx++) { // Add each key/value pair StringList *keyValue = strLstNewSplitZ(strLstGet(keyValueList, keyValueIdx), "="); if (strLstSize(keyValue) != 2) { THROW_FMT( FormatError, "invalid key/value '%s' in query '%s'", strZ(strLstGet(keyValueList, keyValueIdx)), strZ(query)); } httpQueryAdd(this, httpUriDecode(strLstGet(keyValue, 0)), httpUriDecode(strLstGet(keyValue, 1))); } } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryDup(const HttpQuery *const query, const HttpQueryDupParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_PARAM(STRING_LIST, param.redactList); FUNCTION_TEST_END(); if (query == NULL) FUNCTION_TEST_RETURN(HTTP_QUERY, NULL); OBJ_NEW_BEGIN(HttpQuery, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpQuery) { .kv = kvDup(query->kv), .redactList = param.redactList != NULL ? strLstDup(param.redactList) : strLstDup(query->redactList), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryAdd(HttpQuery *const this, const String *const key, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT(value != NULL); // Make sure the key does not already exist const Variant *keyVar = VARSTR(key); if (kvGet(this->kv, keyVar) != NULL) THROW_FMT(AssertError, "key '%s' already exists", strZ(key)); // Store the key kvPut(this->kv, keyVar, VARSTR(value)); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN const String * httpQueryGet(const HttpQuery *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN_CONST(STRING, varStr(kvGet(this->kv, VARSTR(key)))); } /**********************************************************************************************************************************/ FN_EXTERN StringList * httpQueryList(const HttpQuery *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING_LIST, strLstSort(strLstNewVarLst(kvKeyList(this->kv)), sortOrderAsc)); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryMerge(HttpQuery *const this, const HttpQuery *const query) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(query != NULL); MEM_CONTEXT_TEMP_BEGIN() { const VariantList *const keyList = kvKeyList(query->kv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { const Variant *const key = varLstGet(keyList, keyIdx); httpQueryAdd(this, varStr(key), varStr(kvGet(query->kv, key))); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpQuery * httpQueryPut(HttpQuery *const this, const String *const key, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT(value != NULL); // Store the key kvPut(this->kv, VARSTR(key), VARSTR(value)); FUNCTION_TEST_RETURN(HTTP_QUERY, this); } /**********************************************************************************************************************************/ FN_EXTERN bool httpQueryRedact(const HttpQuery *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(BOOL, this->redactList != NULL && strLstExists(this->redactList, key)); } /**********************************************************************************************************************************/ FN_EXTERN String * httpQueryRender(const HttpQuery *const this, const HttpQueryRenderParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_QUERY, this); FUNCTION_TEST_PARAM(BOOL, param.redact); FUNCTION_TEST_END(); String *result = NULL; if (this != NULL) { MEM_CONTEXT_TEMP_BEGIN() { const StringList *const keyList = httpQueryList(this); if (!strLstEmpty(keyList)) { MEM_CONTEXT_PRIOR_BEGIN() { result = strNew(); } MEM_CONTEXT_PRIOR_END(); for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { const String *const key = strLstGet(keyList, keyIdx); if (strSize(result) != 0) strCatZ(result, "&"); strCatFmt( result, "%s=%s", strZ(httpUriEncode(key, false)), param.redact && httpQueryRedact(this, key) ? "" : strZ(httpUriEncode(httpQueryGet(this, key), false))); } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void httpQueryToLog(const HttpQuery *const this, StringStatic *const debugLog) { const VariantList *const keyList = kvKeyList(this->kv); strStcCatChr(debugLog, '{'); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { const String *const key = varStr(varLstGet(keyList, keyIdx)); if (keyIdx != 0) strStcCat(debugLog, ", "); if (httpQueryRedact(this, key)) strStcFmt(debugLog, "%s: ", strZ(key)); else strStcFmt(debugLog, "%s: '%s'", strZ(key), strZ(httpQueryGet(this, key))); } strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/query.h0000664000175100017510000001163015240547630020377 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Query Object to track HTTP queries and output them with proper escaping. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_QUERY_H #define COMMON_IO_HTTP_QUERY_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpQuery HttpQuery; #include "common/type/object.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct HttpQueryNewParam { VAR_PARAM_HEADER; const KeyValue *kv; // Initial query key/value list const StringList *redactList; // List of keys to redact values for } HttpQueryNewParam; #define httpQueryNewP(...) \ httpQueryNew((HttpQueryNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpQuery *httpQueryNew(HttpQueryNewParam param); // New from encoded query string FN_EXTERN HttpQuery *httpQueryNewStr(const String *query); // Duplicate typedef struct HttpQueryDupParam { VAR_PARAM_HEADER; const StringList *redactList; // List of keys to redact values for } HttpQueryDupParam; #define httpQueryDupP(query, ...) \ httpQueryDup(query, (HttpQueryDupParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpQuery *httpQueryDup(const HttpQuery *query, HttpQueryDupParam param); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a query item FN_EXTERN HttpQuery *httpQueryAdd(HttpQuery *this, const String *key, const String *value); // Get a value using the key FN_EXTERN const String *httpQueryGet(const HttpQuery *this, const String *key); // Get list of keys FN_EXTERN StringList *httpQueryList(const HttpQuery *this); // Merge the contents of another query into this one FN_EXTERN HttpQuery *httpQueryMerge(HttpQuery *this, const HttpQuery *query); // Move to a new parent mem context FN_INLINE_ALWAYS HttpQuery * httpQueryMove(HttpQuery *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Put a query item FN_EXTERN HttpQuery *httpQueryPut(HttpQuery *this, const String *header, const String *value); // Should the query key be redacted when logging? FN_EXTERN bool httpQueryRedact(const HttpQuery *this, const String *key); // Render the query for inclusion in an HTTP request typedef struct HttpQueryRenderParam { VAR_PARAM_HEADER; bool redact; // Redact user-visible query string } HttpQueryRenderParam; #define httpQueryRenderP(this, ...) \ httpQueryRender(this, (HttpQueryRenderParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN String *httpQueryRender(const HttpQuery *this, HttpQueryRenderParam param); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpQueryFree(HttpQuery *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpQueryToLog(const HttpQuery *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_QUERY_TYPE \ HttpQuery * #define FUNCTION_LOG_HTTP_QUERY_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpQueryToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/request.c0000664000175100017510000005316715240547630020730 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Request ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/error/retry.h" #include "common/io/http/common.h" #include "common/io/http/request.h" #include "common/log.h" #include "common/stat.h" #include "common/wait.h" #include "version.h" /*********************************************************************************************************************************** HTTP constants ***********************************************************************************************************************************/ STRING_EXTERN(HTTP_VERSION_STR, HTTP_VERSION); STRING_EXTERN(HTTP_VERSION_10_STR, HTTP_VERSION_10); STRING_EXTERN(HTTP_VERB_DELETE_STR, HTTP_VERB_DELETE); STRING_EXTERN(HTTP_VERB_GET_STR, HTTP_VERB_GET); STRING_EXTERN(HTTP_VERB_HEAD_STR, HTTP_VERB_HEAD); STRING_EXTERN(HTTP_VERB_POST_STR, HTTP_VERB_POST); STRING_EXTERN(HTTP_VERB_PUT_STR, HTTP_VERB_PUT); STRING_EXTERN(HTTP_HEADER_AUTHORIZATION_STR, HTTP_HEADER_AUTHORIZATION); STRING_EXTERN(HTTP_HEADER_CONTENT_ID_STR, HTTP_HEADER_CONTENT_ID); STRING_EXTERN(HTTP_HEADER_CONTENT_LENGTH_STR, HTTP_HEADER_CONTENT_LENGTH); STRING_EXTERN(HTTP_HEADER_CONTENT_MD5_STR, HTTP_HEADER_CONTENT_MD5); STRING_EXTERN(HTTP_HEADER_CONTENT_RANGE_STR, HTTP_HEADER_CONTENT_RANGE); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_STR, HTTP_HEADER_CONTENT_TYPE); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL_STR, HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_HTTP_STR, HTTP_HEADER_CONTENT_TYPE_HTTP); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_JSON_STR, HTTP_HEADER_CONTENT_TYPE_JSON); STRING_EXTERN(HTTP_HEADER_CONTENT_TYPE_XML_STR, HTTP_HEADER_CONTENT_TYPE_XML); STRING_EXTERN(HTTP_HEADER_ETAG_STR, HTTP_HEADER_ETAG); STRING_EXTERN(HTTP_HEADER_DATE_STR, HTTP_HEADER_DATE); STRING_EXTERN(HTTP_HEADER_HOST_STR, HTTP_HEADER_HOST); STRING_EXTERN(HTTP_HEADER_LAST_MODIFIED_STR, HTTP_HEADER_LAST_MODIFIED); STRING_EXTERN(HTTP_HEADER_RANGE_STR, HTTP_HEADER_RANGE); #define HTTP_HEADER_USER_AGENT "user-agent" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpRequest { HttpRequestPub pub; // Publicly accessible variables HttpClient *client; // HTTP client const Buffer *content; // HTTP content HttpSession *session; // Session for async requests }; struct HttpRequestMulti { List *contentList; // Multipart content size_t contentSize; // Size of all content (excluding boundaries) Buffer *boundaryRaw; // Multipart boundary (not yet formatted for output) }; /*********************************************************************************************************************************** Format the request (excluding content) ***********************************************************************************************************************************/ static String * httpRequestFmt( const String *const verb, const String *const path, const HttpQuery *const query, const HttpHeader *const header, const bool agent, const bool headerSpace) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, verb); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_PARAM(HTTP_HEADER, header); FUNCTION_TEST_PARAM(BOOL, agent); FUNCTION_TEST_PARAM(BOOL, headerSpace); FUNCTION_TEST_END(); ASSERT(verb != NULL); ASSERT(path != NULL); ASSERT(header != NULL); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { // Add verb, path, and query strCatFmt( result, "%s %s%s%s " HTTP_VERSION "\r\n", strZ(verb), strZ(path), query == NULL ? "" : "?", query == NULL ? "" : strZ(httpQueryRenderP(query))); // Add user agent if (agent) strCatZ(result, HTTP_HEADER_USER_AGENT ":" PROJECT_NAME "/" PROJECT_VERSION "\r\n"); // Add headers. A space after the colon is used for multipart sub-requests since Azure's batch parser requires the canonical // "Name: value" form, but is omitted for normal requests to preserve existing behavior. StringList *const headerList = httpHeaderList(header); for (unsigned int headerIdx = 0; headerIdx < strLstSize(headerList); headerIdx++) { const String *const headerKey = strLstGet(headerList, headerIdx); strCatFmt( result, headerSpace ? "%s: %s\r\n" : "%s:%s\r\n", strZ(headerKey), strZ(httpHeaderGet(header, headerKey))); } // Add blank line to end of headers strCat(result, CRLF_STR); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Process the request ***********************************************************************************************************************************/ static HttpResponse * httpRequestProcess(HttpRequest *const this, const bool waitForResponse, const bool contentCache) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, this); FUNCTION_LOG_PARAM(BOOL, waitForResponse); FUNCTION_LOG_PARAM(BOOL, contentCache); FUNCTION_LOG_END(); ASSERT(this != NULL); // HTTP Response HttpResponse *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { bool retry; ErrorRetry *const errRetry = errRetryNew(); Wait *const wait = waitNew(httpClientTimeout(this->client)); do { // Create a session to process the request. This is done outside exception handling to avoid multiplying retries. HttpSession *session = NULL; bool send = true; // If a session is saved then the request was already successfully sent if (this->session != NULL) { session = httpSessionMove(this->session, memContextCurrent()); this->session = NULL; send = false; } // Else the request has not been sent yet or this is a retry else session = httpClientOpen(this->client); // Assume there will be no retry retry = false; TRY_BEGIN() { MEM_CONTEXT_TEMP_BEGIN() { // Send the request if (send) { // Write the request as a buffer so secrets do not show up in logs ioWrite( httpSessionIoWrite(session), BUFSTR( httpRequestFmt( httpRequestVerb(this), httpRequestPath(this), httpRequestQuery(this), httpRequestHeader(this), true, false))); // Write out content if any if (this->content != NULL) ioWrite(httpSessionIoWrite(session), this->content); // Flush all writes ioWriteFlush(httpSessionIoWrite(session)); // If not waiting for the response then move the session to the object context if (!waitForResponse) this->session = httpSessionMove(session, objMemContext(this)); } // Wait for response if (waitForResponse) { result = httpResponseNew(session, httpRequestVerb(this), contentCache); // Error on response codes that should be retried to engage the retry logic if (httpResponseCodeRetry(result)) THROW_FMT(ServiceError, "[%u] %s", httpResponseCode(result), strZ(httpResponseReason(result))); // Move response to outer temp context httpResponseMove(result, memContextPrior()); } } MEM_CONTEXT_TEMP_END(); } CATCH_ANY() { // Add the error retry info errRetryAddP(errRetry); // Sleep and then retry unless the total wait time has expired if (waitMore(wait)) { LOG_DEBUG_FMT("retry %s: %s", errorTypeName(errorType()), errorMessage()); retry = true; statInc(HTTP_STAT_RETRY_STR); } else THROWP(errRetryType(errRetry), strZ(errRetryMessage(errRetry))); } TRY_END(); } while (retry); // Move response to calling context httpResponseMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /**********************************************************************************************************************************/ FN_EXTERN HttpRequest * httpRequestNew(HttpClient *const client, const String *const verb, const String *const path, const HttpRequestNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_CLIENT, client); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_END(); ASSERT(verb != NULL); ASSERT(path != NULL); OBJ_NEW_BEGIN(HttpRequest, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpRequest) { .pub = { .verb = strDup(verb), .path = strDup(path), .query = httpQueryDupP(param.query), .header = param.header == NULL ? httpHeaderNew(NULL) : httpHeaderDup(param.header, NULL), }, .client = client, .content = param.content == NULL ? NULL : bufDup(param.content), }; } OBJ_NEW_END(); // Send the request httpRequestProcess(this, false, false); statInc(HTTP_STAT_REQUEST_STR); FUNCTION_LOG_RETURN(HTTP_REQUEST, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpResponse * httpRequestResponse(HttpRequest *const this, const bool contentCache) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, this); FUNCTION_LOG_PARAM(BOOL, contentCache); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(HTTP_RESPONSE, httpRequestProcess(this, true, contentCache)); } /**********************************************************************************************************************************/ FN_EXTERN void httpRequestError(const HttpRequest *const this, HttpResponse *const response) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_REQUEST, this); FUNCTION_LOG_PARAM(HTTP_RESPONSE, response); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(response != NULL); // Error code String *const error = strCatFmt(strNew(), "HTTP request failed with %u", httpResponseCode(response)); // Add reason when present if (strSize(httpResponseReason(response)) > 0) strCatFmt(error, " (%s)", strZ(httpResponseReason(response))); // Output path/query strCatZ(error, ":\n*** Path/Query ***:"); strCatFmt(error, "\n%s %s", strZ(httpRequestVerb(this)), strZ(httpRequestPath(this))); if (httpRequestQuery(this) != NULL) strCatFmt(error, "?%s", strZ(httpQueryRenderP(httpRequestQuery(this), .redact = true))); // Output request headers const StringList *const requestHeaderList = httpHeaderList(httpRequestHeader(this)); if (!strLstEmpty(requestHeaderList)) { strCatZ(error, "\n*** Request Headers ***:"); for (unsigned int requestHeaderIdx = 0; requestHeaderIdx < strLstSize(requestHeaderList); requestHeaderIdx++) { const String *const key = strLstGet(requestHeaderList, requestHeaderIdx); strCatFmt( error, "\n%s: %s", strZ(key), httpHeaderRedact(httpRequestHeader(this), key) ? "" : strZ(httpHeaderGet(httpRequestHeader(this), key))); } } // Output response headers const HttpHeader *const responseHeader = httpResponseHeader(response); const StringList *const responseHeaderList = httpHeaderList(responseHeader); if (!strLstEmpty(responseHeaderList)) { strCatZ(error, "\n*** Response Headers ***:"); for (unsigned int responseHeaderIdx = 0; responseHeaderIdx < strLstSize(responseHeaderList); responseHeaderIdx++) { const String *const key = strLstGet(responseHeaderList, responseHeaderIdx); strCatFmt(error, "\n%s: %s", strZ(key), strZ(httpHeaderGet(responseHeader, key))); } } // Add response content, if any if (!bufEmpty(httpResponseContent(response))) { strCatZ(error, "\n*** Response Content ***:\n"); strCat(error, strNewBuf(httpResponseContent(response))); } THROW(ProtocolError, strZ(error)); } /**********************************************************************************************************************************/ #define HTTP_MULTIPART_BOUNDARY_INIT_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_INIT) - 1) FN_EXTERN HttpRequestMulti * httpRequestMultiNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(HttpRequestMulti, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpRequestMulti) { .boundaryRaw = bufNewC(HTTP_MULTIPART_BOUNDARY_INIT, HTTP_MULTIPART_BOUNDARY_INIT_SIZE), .contentList = lstNewP(sizeof(Buffer *)), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_REQUEST_MULTI, this); } /**********************************************************************************************************************************/ #define HTTP_MULTIPART_BOUNDARY_EXTRA_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_EXTRA) - 1) FN_EXTERN void httpRequestMultiAdd( HttpRequestMulti *const this, const String *const contentId, const String *const verb, const String *const path, const HttpRequestNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_REQUEST_MULTI, this); FUNCTION_TEST_PARAM(STRING, contentId); FUNCTION_TEST_PARAM(STRING, verb); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(HTTP_QUERY, param.query); FUNCTION_TEST_PARAM(HTTP_HEADER, param.header); FUNCTION_TEST_PARAM(BUFFER, param.content); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(contentId != NULL); ASSERT(verb != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Construct request header. The MIME part headers use the canonical "Name: value" form (capitalized name, space after the // colon) since Azure's batch parser is stricter than the MIME/OData spec and rejects other forms. String *const request = strNew(); strCatZ(request, "Content-Type: " HTTP_HEADER_CONTENT_TYPE_HTTP "\r\n"); strCatZ(request, "Content-Transfer-Encoding: " HTTP_HEADER_CONTENT_TRANSFER_ENCODING_BINARY "\r\n"); strCatFmt(request, "Content-ID: %s\r\n\r\n", strZ(contentId)); strCat(request, httpRequestFmt(verb, path, param.query, param.header, false, true)); MEM_CONTEXT_OBJ_BEGIN(this->contentList) { // Add content Buffer *const content = bufNew(strSize(request) + (param.content == NULL ? 0 : bufUsed(param.content))); bufCat(content, BUFSTR(request)); bufCat(content, param.content); // Find a boundary that is not used in the content while (bufFindP(content, this->boundaryRaw) != NULL) { if (bufUsed(this->boundaryRaw) == HTTP_MULTIPART_BOUNDARY_INIT_SIZE + HTTP_MULTIPART_BOUNDARY_EXTRA_SIZE) THROW(AssertError, "unable to construct unique boundary"); bufCatC( this->boundaryRaw, (const uint8_t *)HTTP_MULTIPART_BOUNDARY_EXTRA, bufUsed(this->boundaryRaw) - HTTP_MULTIPART_BOUNDARY_INIT_SIZE, HTTP_MULTIPART_BOUNDARY_NEXT); } // Add to list lstAdd(this->contentList, &content); this->contentSize += bufUsed(content); } MEM_CONTEXT_OBJ_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ #define HTTP_MULTIPART_BOUNDARY_PRE_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_PRE) - 1) #define HTTP_MULTIPART_BOUNDARY_POST_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_POST) - 1) #define HTTP_MULTIPART_BOUNDARY_POST_LAST_SIZE (sizeof(HTTP_MULTIPART_BOUNDARY_POST_LAST) - 1) FN_EXTERN Buffer * httpRequestMultiContent(HttpRequestMulti *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_REQUEST_MULTI, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(!lstEmpty(this->contentList)); const size_t boundarySize = bufUsed(this->boundaryRaw) + HTTP_MULTIPART_BOUNDARY_PRE_SIZE; Buffer *const result = bufNew( this->contentSize + (lstSize(this->contentList) * (boundarySize + HTTP_MULTIPART_BOUNDARY_POST_SIZE)) + HTTP_MULTIPART_BOUNDARY_POST_LAST_SIZE); MEM_CONTEXT_TEMP_BEGIN() { // Generate boundary. The leading CRLF is conceptually part of the delimiter that precedes each subsequent part, but the // opening delimiter is emitted without it below since Azure's batch parser requires the body to begin with the delimiter. Buffer *const boundary = bufNew(boundarySize); bufCat(boundary, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_PRE)); bufCat(boundary, this->boundaryRaw); // Add first boundary (without the leading CRLF so the body begins with the delimiter) bufCat(result, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_OPEN)); bufCat(result, this->boundaryRaw); bufCat(result, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_POST)); // Add content and boundaries for (unsigned int contentIdx = 0; contentIdx < lstSize(this->contentList); contentIdx++) { bufCat(result, *(Buffer **)lstGet(this->contentList, contentIdx)); bufCat(result, boundary); if (contentIdx < lstSize(this->contentList) - 1) bufCat(result, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_POST)); else bufCat(result, BUFSTRDEF(HTTP_MULTIPART_BOUNDARY_POST_LAST)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN HttpHeader * httpRequestMultiHeaderAdd(HttpRequestMulti *const this, HttpHeader *const header) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_REQUEST_MULTI, this); FUNCTION_TEST_PARAM(HTTP_HEADER, header); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(header != NULL); MEM_CONTEXT_TEMP_BEGIN() { httpHeaderAdd( header, HTTP_HEADER_CONTENT_TYPE_STR, strNewFmt(HTTP_HEADER_CONTENT_TYPE_MULTIPART "%s", strZ(strNewBuf(this->boundaryRaw)))); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(HTTP_HEADER, header); } /**********************************************************************************************************************************/ FN_EXTERN void httpRequestToLog(const HttpRequest *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{verb: %s, path: %s, contentSize: %zu, query: ", strZ(httpRequestVerb(this)), strZ(httpRequestPath(this)), this->content == NULL ? 0 : bufUsed(this->content)); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT( httpRequestQuery(this), httpQueryToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", header: "), httpHeaderToLog(httpRequestHeader(this), debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/request.h0000664000175100017510000002471715240547630020734 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Request Send a request to an HTTP server and get a response. The interface is natively asynchronous, i.e. httpRequestNew() sends a request and httpRequestResponse() waits for a response. These can be called together for synchronous behavior or separately for asynchronous behavior. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_REQUEST_H #define COMMON_IO_HTTP_REQUEST_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpRequest HttpRequest; typedef struct HttpRequestMulti HttpRequestMulti; #include "common/io/http/header.h" #include "common/io/http/query.h" #include "common/io/http/response.h" #include "common/type/object.h" /*********************************************************************************************************************************** HTTP Constants ***********************************************************************************************************************************/ #define HTTP_VERSION "HTTP/1.1" STRING_DECLARE(HTTP_VERSION_STR); #define HTTP_VERSION_10 "HTTP/1.0" STRING_DECLARE(HTTP_VERSION_10_STR); #define HTTP_VERB_DELETE "DELETE" STRING_DECLARE(HTTP_VERB_DELETE_STR); #define HTTP_VERB_GET "GET" STRING_DECLARE(HTTP_VERB_GET_STR); #define HTTP_VERB_HEAD "HEAD" STRING_DECLARE(HTTP_VERB_HEAD_STR); #define HTTP_VERB_POST "POST" STRING_DECLARE(HTTP_VERB_POST_STR); #define HTTP_VERB_PUT "PUT" STRING_DECLARE(HTTP_VERB_PUT_STR); #define HTTP_HEADER_AUTHORIZATION "authorization" STRING_DECLARE(HTTP_HEADER_AUTHORIZATION_STR); #define HTTP_HEADER_CONTENT_ID "content-id" STRING_DECLARE(HTTP_HEADER_CONTENT_ID_STR); #define HTTP_HEADER_CONTENT_LENGTH "content-length" STRING_DECLARE(HTTP_HEADER_CONTENT_LENGTH_STR); #define HTTP_HEADER_CONTENT_MD5 "content-md5" STRING_DECLARE(HTTP_HEADER_CONTENT_MD5_STR); #define HTTP_HEADER_CONTENT_RANGE "content-range" STRING_DECLARE(HTTP_HEADER_CONTENT_RANGE_STR); #define HTTP_HEADER_CONTENT_TRANSFER_ENCODING "content-transfer-encoding" #define HTTP_HEADER_CONTENT_TRANSFER_ENCODING_BINARY "binary" #define HTTP_HEADER_CONTENT_TYPE "content-type" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_STR); #define HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL "application/x-www-form-urlencoded" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL_STR); #define HTTP_HEADER_CONTENT_TYPE_HTTP "application/http" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_HTTP_STR); #define HTTP_HEADER_CONTENT_TYPE_JSON "application/json" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_JSON_STR); #define HTTP_HEADER_CONTENT_TYPE_MULTIPART "multipart/mixed; boundary=" #define HTTP_HEADER_CONTENT_TYPE_XML "application/xml" STRING_DECLARE(HTTP_HEADER_CONTENT_TYPE_XML_STR); #define HTTP_HEADER_CONTENT_RANGE_BYTES "bytes" #define HTTP_HEADER_DATE "date" STRING_DECLARE(HTTP_HEADER_DATE_STR); #define HTTP_HEADER_ETAG "etag" STRING_DECLARE(HTTP_HEADER_ETAG_STR); #define HTTP_HEADER_HOST "host" STRING_DECLARE(HTTP_HEADER_HOST_STR); #define HTTP_HEADER_LAST_MODIFIED "last-modified" STRING_DECLARE(HTTP_HEADER_LAST_MODIFIED_STR); #define HTTP_HEADER_RANGE "range" STRING_DECLARE(HTTP_HEADER_RANGE_STR); #define HTTP_HEADER_RANGE_BYTES "bytes" #define HTTP_MULTIPART_BOUNDARY_INIT "QKX4EYg4" #define HTTP_MULTIPART_BOUNDARY_NEXT 4 #define HTTP_MULTIPART_BOUNDARY_EXTRA "LARJ52gF4F239iNVrrC5w5aYskcGrWCXIFlMp5IxswggIhcX2A0gF9nrgN8q" #define HTTP_MULTIPART_BOUNDARY_OPEN "--" #define HTTP_MULTIPART_BOUNDARY_PRE "\r\n--" #define HTTP_MULTIPART_BOUNDARY_POST "\r\n" #define HTTP_MULTIPART_BOUNDARY_POST_LAST "--\r\n" /*********************************************************************************************************************************** Request Constructors ***********************************************************************************************************************************/ typedef struct HttpRequestNewParam { VAR_PARAM_HEADER; const HttpQuery *query; const HttpHeader *header; const Buffer *content; } HttpRequestNewParam; #define httpRequestNewP(client, verb, path, ...) \ httpRequestNew(client, verb, path, (HttpRequestNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpRequest *httpRequestNew(HttpClient *client, const String *verb, const String *path, HttpRequestNewParam param); /*********************************************************************************************************************************** Request Getters/Setters ***********************************************************************************************************************************/ typedef struct HttpRequestPub { const String *verb; // HTTP verb (GET, POST, etc.) const String *path; // HTTP path const HttpQuery *query; // HTTP query const HttpHeader *header; // HTTP headers } HttpRequestPub; // Request path FN_INLINE_ALWAYS const String * httpRequestPath(const HttpRequest *const this) { return THIS_PUB(HttpRequest)->path; } // Request query FN_INLINE_ALWAYS const HttpQuery * httpRequestQuery(const HttpRequest *const this) { return THIS_PUB(HttpRequest)->query; } // Request headers FN_INLINE_ALWAYS const HttpHeader * httpRequestHeader(const HttpRequest *const this) { return THIS_PUB(HttpRequest)->header; } // Request verb FN_INLINE_ALWAYS const String * httpRequestVerb(const HttpRequest *const this) { return THIS_PUB(HttpRequest)->verb; } /*********************************************************************************************************************************** Request Functions ***********************************************************************************************************************************/ // Wait for a response from the request FN_EXTERN HttpResponse *httpRequestResponse(HttpRequest *this, bool contentCache); // Throw an error if the request failed FN_EXTERN noreturn void httpRequestError(const HttpRequest *this, HttpResponse *response); // Move to a new parent mem context FN_INLINE_ALWAYS HttpRequest * httpRequestMove(HttpRequest *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Request Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpRequestFree(HttpRequest *const this) { objFree(this); } /*********************************************************************************************************************************** Request Multi Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpRequestMulti *httpRequestMultiNew(void); /*********************************************************************************************************************************** Request Multi Functions ***********************************************************************************************************************************/ // Add a request to multipart content #define httpRequestMultiAddP(this, contentId, verb, path, ...) \ httpRequestMultiAdd(this, contentId, verb, path, (HttpRequestNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void httpRequestMultiAdd( HttpRequestMulti *this, const String *contentId, const String *verb, const String *path, HttpRequestNewParam param); // Concatenate multipart content FN_EXTERN Buffer *httpRequestMultiContent(HttpRequestMulti *this); // Add multipart header FN_EXTERN HttpHeader *httpRequestMultiHeaderAdd(HttpRequestMulti *this, HttpHeader *header); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpRequestToLog(const HttpRequest *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_REQUEST_TYPE \ HttpRequest * #define FUNCTION_LOG_HTTP_REQUEST_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpRequestToLog, buffer, bufferSize) #define FUNCTION_LOG_HTTP_REQUEST_MULTI_TYPE \ HttpRequestMulti * #define FUNCTION_LOG_HTTP_REQUEST_MULTI_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "HttpRequestMulti", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/response.c0000664000175100017510000006036115240547630021070 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Response ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/bufferWrite.h" #include "common/io/http/client.h" #include "common/io/http/common.h" #include "common/io/http/request.h" #include "common/io/http/response.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/log.h" #include "common/stat.h" #include "common/wait.h" /*********************************************************************************************************************************** HTTP constants ***********************************************************************************************************************************/ #define HTTP_HEADER_CONNECTION "connection" STRING_STATIC(HTTP_HEADER_CONNECTION_STR, HTTP_HEADER_CONNECTION); #define HTTP_HEADER_TRANSFER_ENCODING "transfer-encoding" STRING_STATIC(HTTP_HEADER_TRANSFER_ENCODING_STR, HTTP_HEADER_TRANSFER_ENCODING); #define HTTP_VALUE_CONNECTION_CLOSE "close" STRING_STATIC(HTTP_VALUE_CONNECTION_CLOSE_STR, HTTP_VALUE_CONNECTION_CLOSE); #define HTTP_VALUE_TRANSFER_ENCODING_CHUNKED "chunked" STRING_STATIC(HTTP_VALUE_TRANSFER_ENCODING_CHUNKED_STR, HTTP_VALUE_TRANSFER_ENCODING_CHUNKED); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpResponse { HttpResponsePub pub; // Publicly accessible variables HttpSession *session; // HTTP session bool contentChunked; // Is the response content chunked? bool contentSizeSet; // Was content size specified? uint64_t contentSize; // Content size (ignored for chunked) uint64_t contentRemaining; // Content remaining (per chunk if chunked) bool closeOnContentEof; // Will server close after content is sent? bool contentExists; // Does content exist? bool contentEof; // Has all content been read? Buffer *content; // Caches content once requested }; struct HttpResponseMulti { const Buffer *content; // Response content Buffer *boundary; // Multipart boundary const uint8_t *boundaryLast; // Last boundary location }; /*********************************************************************************************************************************** When response is done close/reuse the connection ***********************************************************************************************************************************/ static void httpResponseDone(HttpResponse *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); // If close was requested by the server then free the session if (this->closeOnContentEof) { httpSessionFree(this->session); // Only update the close stats after a successful response so it is not counted if there was an error/retry statInc(HTTP_STAT_CLOSE_STR); } // Else return it to the client so it can be reused else httpSessionDone(this->session); this->session = NULL; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Read content ***********************************************************************************************************************************/ // Helper to determine if it is OK to accept unexpected EOF, e.g. the server closed the socket with properly closing the TLS // connection. The idea is that since these types of responses can be validated we should be able to detect a short read. static bool httpResponseReadIgnoreUnexpectedEof(const HttpResponse *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE, this); FUNCTION_LOG_END(); const String *const contentType = httpHeaderGet(this->pub.header, HTTP_HEADER_CONTENT_TYPE_STR); FUNCTION_LOG_RETURN( BOOL, strEq(contentType, HTTP_HEADER_CONTENT_TYPE_XML_STR) || strEq(contentType, HTTP_HEADER_CONTENT_TYPE_JSON_STR)); } static size_t httpResponseRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(HttpResponse); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(!bufFull(buffer)); ASSERT(this->contentEof || this->session != NULL); // Read if EOF has not been reached size_t actualBytes = 0; if (!this->contentEof) { MEM_CONTEXT_TEMP_BEGIN() { // If close was requested and no content specified then the server may send content up until the eof if (this->closeOnContentEof && !this->contentChunked && this->contentSize == 0) { IoRead *const rawRead = httpSessionIoReadP( this->session, .ignoreUnexpectedEof = httpResponseReadIgnoreUnexpectedEof(this)); ioRead(rawRead, buffer); this->contentEof = ioReadEof(rawRead); } // Else read using specified encoding or size else { IoRead *const rawRead = httpSessionIoReadP(this->session); do { // If chunked content and no content remaining if (this->contentChunked && this->contentRemaining == 0) { // Read length of next chunk const String *chunkHeader = strTrim(ioReadLine(rawRead)); const int chunkExtPos = strChr(chunkHeader, ';'); if (chunkExtPos != -1) chunkHeader = strTrim(strSubN(chunkHeader, 0, (size_t)chunkExtPos)); this->contentRemaining = cvtZToUInt64Base(strZ(chunkHeader), 16); // A zero-size chunk terminates the response if (this->contentRemaining == 0) { // Consume chunk trailers and the blank line that terminates them. while (strSize(strTrim(ioReadLine(rawRead))) != 0); this->contentEof = true; } } // Read if there is content remaining if (this->contentRemaining > 0) { // If the buffer is larger than the content that needs to be read then limit the buffer size so the read // won't block or read too far. Casting to size_t is safe on 32-bit because we know the max buffer size is // defined as less than 2^32 so content remaining can't be more than that. if (bufRemains(buffer) > this->contentRemaining) bufLimitSet(buffer, bufSize(buffer) - (bufRemains(buffer) - (size_t)this->contentRemaining)); actualBytes = bufRemains(buffer); this->contentRemaining -= ioRead(rawRead, buffer); // Error if EOF but content read is not complete if (ioReadEof(rawRead)) THROW(FileReadError, "unexpected EOF reading HTTP content"); // Clear limit (this works even if the limit was not set and it is easier than checking) bufLimitClear(buffer); } // If no content remaining if (this->contentRemaining == 0) { // If chunked then consume the blank line that follows every chunk. There might be more chunk data so loop // back around to check. if (this->contentChunked && !this->contentEof) { ioReadLine(rawRead); } // If total content size was provided then this is eof else this->contentEof = true; } } while (!bufFull(buffer) && !this->contentEof); } // If all content has been read if (this->contentEof) httpResponseDone(this); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(SIZE, (size_t)actualBytes); } /*********************************************************************************************************************************** Has all content been read? ***********************************************************************************************************************************/ static bool httpResponseEof(THIS_VOID) { THIS(HttpResponse); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->contentEof); } /*********************************************************************************************************************************** Read response status ***********************************************************************************************************************************/ static void httpResponseStatusRead(HttpResponse *const this, IoRead *const read) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_RESPONSE, this); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // Read status String *status = ioReadLine(read); // Check status ends with a CR and remove it to make error formatting easier and more accurate if (!strEndsWith(status, CR_STR)) THROW_FMT(FormatError, "HTTP response status '%s' should be CR-terminated", strZ(status)); status = strSubN(status, 0, strSize(status) - 1); // Check status is at least the minimum required length to avoid harder to interpret errors later on if (strSize(status) < sizeof(HTTP_VERSION) + 4) THROW_FMT(FormatError, "HTTP response '%s' has invalid length", strZ(strTrim(status))); // If HTTP/1.0 then the connection will be closed on content eof since connections are not reused by default if (strBeginsWith(status, HTTP_VERSION_10_STR)) { this->closeOnContentEof = true; } // Else check that the version is the default (1.1) else if (!strBeginsWith(status, HTTP_VERSION_STR)) THROW_FMT(FormatError, "HTTP version of response '%s' must be " HTTP_VERSION " or " HTTP_VERSION_10, strZ(status)); // Read status code status = strSub(status, sizeof(HTTP_VERSION)); int spacePos = strChr(status, ' '); if (spacePos != 3) THROW_FMT(FormatError, "response status '%s' must have a space after the status code", strZ(status)); this->pub.code = cvtZSubNToUInt(strZ(status), 0, (size_t)spacePos); // Read reason phrase. A missing reason phrase will be represented as an empty string. MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.reason = strSub(status, (size_t)spacePos + 1); } MEM_CONTEXT_OBJ_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Read response headers ***********************************************************************************************************************************/ static void httpResponseHeaderRead(HttpResponse *const this, IoRead *const read, const bool allowEof) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_RESPONSE, this); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_PARAM(BOOL, allowEof); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { do { // Read the next header. In a multipart part the headers may be terminated by the boundary rather than a blank line (the // boundary's leading CRLF doubles as the terminator), so allow eof to end the header block in that case. String *const header = strTrim(ioReadLineParam(read, allowEof)); // If the header is empty then we have reached the end of the headers if (strSize(header) == 0) break; // Split the header and store it int colonPos = strChr(header, ':'); if (colonPos < 0) THROW_FMT(FormatError, "header '%s' missing colon", strZ(strTrim(header))); const String *const headerKey = strLower(strTrim(strSubN(header, 0, (size_t)colonPos))); String *const headerValue = strTrim(strSub(header, (size_t)colonPos + 1)); // Read transfer encoding (only chunked is supported) if (strEq(headerKey, HTTP_HEADER_TRANSFER_ENCODING_STR)) { // Error if transfer encoding is not chunked if (!strEq(strLower(headerValue), HTTP_VALUE_TRANSFER_ENCODING_CHUNKED_STR)) { THROW_FMT( FormatError, "only '%s' is supported for '%s' header", HTTP_VALUE_TRANSFER_ENCODING_CHUNKED, HTTP_HEADER_TRANSFER_ENCODING); } this->contentChunked = true; } // Read content size if (strEq(headerKey, HTTP_HEADER_CONTENT_LENGTH_STR)) { this->contentSizeSet = true; this->contentSize = cvtZToUInt64(strZ(headerValue)); this->contentRemaining = this->contentSize; } // If the server notified of a closed connection then close the client connection after reading content. This // prevents doing a retry on the next request when using the closed connection. if (strEq(headerKey, HTTP_HEADER_CONNECTION_STR) && strEq(strLower(headerValue), HTTP_VALUE_CONNECTION_CLOSE_STR)) this->closeOnContentEof = true; // Add after header checks in case the value was modified httpHeaderAdd(this->pub.header, headerKey, headerValue); } while (1); // Error if transfer encoding and content length are both set if (this->contentChunked && this->contentSizeSet) { THROW_FMT( FormatError, "'%s' and '%s' headers are both set", HTTP_HEADER_TRANSFER_ENCODING, HTTP_HEADER_CONTENT_LENGTH); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ // Constructor helper static HttpResponse * httpResponseNewInternal(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(HttpResponse, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpResponse) { .pub = { .header = httpHeaderNew(NULL), }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_RESPONSE, this); } FN_EXTERN HttpResponse * httpResponseNew(HttpSession *const session, const String *const verb, const bool contentCache) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_SESSION, session); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(BOOL, contentCache); FUNCTION_LOG_END(); ASSERT(session != NULL); ASSERT(verb != NULL); HttpResponse *const this = httpResponseNewInternal(); this->session = httpSessionMove(session, objMemContext(this)); // Read status httpResponseStatusRead(this, httpSessionIoReadP(this->session)); // Read headers httpResponseHeaderRead(this, httpSessionIoReadP(this->session), false); // Was content returned in the response? HEAD will report content but not actually return any. this->contentExists = (this->contentChunked || this->contentSize > 0 || this->closeOnContentEof) && !strEq(verb, HTTP_VERB_HEAD_STR); this->contentEof = !this->contentExists; // Create an io object, even if there is no content. This makes the logic for readers easier -- they can just check eof // rather than also checking if the io object exists. MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.contentRead = ioReadNewP(this, .eof = httpResponseEof, .read = httpResponseRead); ioReadOpen(httpResponseIoRead(this)); } MEM_CONTEXT_OBJ_END(); // If there is no content then we are done with the client if (!this->contentExists) { httpResponseDone(this); } // Else cache content when requested or on error else if (contentCache || !httpResponseCodeOk(this)) { MEM_CONTEXT_OBJ_BEGIN(this) { httpResponseContent(this); } MEM_CONTEXT_OBJ_END(); } FUNCTION_LOG_RETURN(HTTP_RESPONSE, this); } /**********************************************************************************************************************************/ FN_EXTERN const Buffer * httpResponseContent(HttpResponse *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_RESPONSE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (this->content == NULL) { this->content = bufNew(0); if (this->contentExists) { do { bufResize(this->content, bufSize(this->content) + ioBufferSize()); httpResponseRead(this, this->content, true); } while (!httpResponseEof(this)); bufResize(this->content, bufUsed(this->content)); } } FUNCTION_TEST_RETURN(BUFFER, this->content); } /**********************************************************************************************************************************/ FN_EXTERN bool httpResponseCodeRetry(const HttpResponse *const this) { return httpResponseCode(this) / 100 == HTTP_RESPONSE_CODE_CLASS_RETRY || httpResponseCode(this) == HTTP_RESPONSE_CODE_REQUEST_TIMEOUT || httpResponseCode(this) == HTTP_RESPONSE_CODE_TOO_MANY_REQUESTS; } /**********************************************************************************************************************************/ FN_EXTERN HttpResponseMulti * httpResponseMultiNew(const Buffer *const content, const String *const contentType) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, content); FUNCTION_LOG_PARAM(STRING, contentType); FUNCTION_LOG_END(); ASSERT(content != NULL); OBJ_NEW_BEGIN(HttpResponseMulti, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpResponseMulti) { .content = content, }; MEM_CONTEXT_TEMP_BEGIN() { // Extract boundary from content type header if (contentType == NULL || !strBeginsWith(contentType, STRDEF(HTTP_HEADER_CONTENT_TYPE_MULTIPART))) THROW(FormatError, "expected multipart content type"); const String *boundary = strSub(contentType, sizeof(HTTP_HEADER_CONTENT_TYPE_MULTIPART) - 1); if (strZ(boundary)[0] == '"') boundary = strSubN(boundary, 1, strSize(boundary) - 2); const String *const boundaryNext = strNewFmt(HTTP_MULTIPART_BOUNDARY_PRE "%s", strZ(boundary)); MEM_CONTEXT_PRIOR_BEGIN() { this->boundary = bufNewC(strZ(boundaryNext), strSize(boundaryNext)); } MEM_CONTEXT_PRIOR_END(); // Find first boundary const String *const boundaryFirst = strNewFmt("--%s" HTTP_MULTIPART_BOUNDARY_POST, strZ(boundary)); ASSERT(strSize(boundaryFirst) == strSize(boundaryNext)); this->boundaryLast = bufFindP(this->content, BUFSTR(boundaryFirst)); if (this->boundaryLast == NULL) THROW_FMT(FormatError, "multipart boundary '%s' not found", strZ(boundary)); } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE_MULTI, this); } /**********************************************************************************************************************************/ FN_EXTERN HttpResponse * httpResponseMultiNext(HttpResponseMulti *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(HTTP_RESPONSE_MULTI, this); FUNCTION_LOG_END(); ASSERT(this != NULL); HttpResponse *result = NULL; // Find next boundary this->boundaryLast += bufSize(this->boundary); const uint8_t *const boundaryNext = bufFindP(this->content, this->boundary, .begin = this->boundaryLast); if (boundaryNext != NULL) { result = httpResponseNewInternal(); MEM_CONTEXT_TEMP_BEGIN() { // Create a buffer for the response Buffer *const response = bufNewC(this->boundaryLast, (size_t)(boundaryNext - this->boundaryLast)); IoRead *const responseIo = ioBufferReadNewOpen(response); // Read multipart headers httpResponseHeaderRead(result, responseIo, true); CHECK( FormatError, strEq(httpHeaderGet(httpResponseHeader(result), HTTP_HEADER_CONTENT_TYPE_STR), HTTP_HEADER_CONTENT_TYPE_HTTP_STR), "expected '" HTTP_HEADER_CONTENT_TYPE ":" HTTP_HEADER_CONTENT_TYPE_HTTP "'"); // Read status httpResponseStatusRead(result, responseIo); // Read headers (eof is allowed since the boundary, not a blank line, may terminate the embedded response headers) httpResponseHeaderRead(result, responseIo, true); // Read content CHECK(FormatError, !result->contentChunked, "chunked encoding not supported in multipart"); MEM_CONTEXT_OBJ_BEGIN(result) { result->content = bufNew((size_t)result->contentSize); } MEM_CONTEXT_OBJ_END(); IoWrite *const write = ioBufferWriteNewOpen(result->content); ioCopyP(responseIo, write); ioWriteClose(write); // Set last boundary this->boundaryLast = boundaryNext + 2; } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /**********************************************************************************************************************************/ FN_EXTERN void httpResponseToLog(const HttpResponse *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{code: %u, reason: %s, contentChunked: %s, contentSize: %" PRIu64 ", contentRemaining: %" PRIu64 ", closeOnContentEof: %s" ", contentExists: %s, contentEof: %s, contentCached: %s}", httpResponseCode(this), httpResponseReason(this) == NULL ? NULL_Z : strZ(httpResponseReason(this)), cvtBoolToConstZ(this->contentChunked), this->contentSize, this->contentRemaining, cvtBoolToConstZ(this->closeOnContentEof), cvtBoolToConstZ(this->contentExists), cvtBoolToConstZ(this->contentEof), cvtBoolToConstZ(this->content != NULL)); strStcCat(debugLog, ", header: "), httpHeaderToLog(httpResponseHeader(this), debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/response.h0000664000175100017510000001566015240547630021077 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Response Response created after a successful request. Once the content is read the underlying connection may be recycled but the headers, cached content, etc. will still be available for the lifetime of the object. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_RESPONSE_H #define COMMON_IO_HTTP_RESPONSE_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpResponse HttpResponse; typedef struct HttpResponseMulti HttpResponseMulti; #include "common/io/http/header.h" #include "common/io/http/session.h" #include "common/io/read.h" #include "common/type/object.h" /*********************************************************************************************************************************** HTTP Response Constants ***********************************************************************************************************************************/ #define HTTP_RESPONSE_CODE_PERMANENT_REDIRECT 308 #define HTTP_RESPONSE_CODE_FORBIDDEN 403 #define HTTP_RESPONSE_CODE_NOT_FOUND 404 #define HTTP_RESPONSE_CODE_REQUEST_TIMEOUT 408 #define HTTP_RESPONSE_CODE_TOO_MANY_REQUESTS 429 // 2xx indicates success #define HTTP_RESPONSE_CODE_CLASS_OK 2 // 5xx errors that should always be retried #define HTTP_RESPONSE_CODE_CLASS_RETRY 5 /*********************************************************************************************************************************** Response Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpResponse *httpResponseNew(HttpSession *session, const String *verb, bool contentCache); /*********************************************************************************************************************************** Response Getters/Setters ***********************************************************************************************************************************/ typedef struct HttpResponsePub { IoRead *contentRead; // Read interface for response content unsigned int code; // Response code (e.g. 200, 404) HttpHeader *header; // Response headers String *reason; // Response reason e.g. (OK, Not Found) } HttpResponsePub; // Read interface used to get the response content. This is intended for reading content that may be very large and will not be held // in memory all at once. If the content must be loaded completely for processing (e.g. XML) then httpResponseContent() is simpler. FN_INLINE_ALWAYS IoRead * httpResponseIoRead(HttpResponse *const this) { return THIS_PUB(HttpResponse)->contentRead; } // Response code FN_INLINE_ALWAYS unsigned int httpResponseCode(const HttpResponse *const this) { return THIS_PUB(HttpResponse)->code; } // Response headers FN_INLINE_ALWAYS const HttpHeader * httpResponseHeader(const HttpResponse *const this) { return THIS_PUB(HttpResponse)->header; } // Response reason FN_INLINE_ALWAYS const String * httpResponseReason(const HttpResponse *const this) { return THIS_PUB(HttpResponse)->reason; } /*********************************************************************************************************************************** Response Functions ***********************************************************************************************************************************/ // Is this response code OK, i.e. 2XX? FN_INLINE_ALWAYS bool httpResponseCodeOk(const HttpResponse *const this) { return httpResponseCode(this) / 100 == HTTP_RESPONSE_CODE_CLASS_OK; } // Should the request be retried? FN_EXTERN bool httpResponseCodeRetry(const HttpResponse *this); // Fetch all response content. Content will be cached so it can be retrieved again without additional cost. FN_EXTERN const Buffer *httpResponseContent(HttpResponse *this); // Move to a new parent mem context FN_INLINE_ALWAYS HttpResponse * httpResponseMove(HttpResponse *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Response Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpResponseFree(HttpResponse *const this) { objFree(this); } /*********************************************************************************************************************************** Response Multi Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpResponseMulti *httpResponseMultiNew(const Buffer *content, const String *contentType); /*********************************************************************************************************************************** Response Multi Functions ***********************************************************************************************************************************/ // Get next response FN_EXTERN HttpResponse *httpResponseMultiNext(HttpResponseMulti *this); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void httpResponseToLog(const HttpResponse *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_RESPONSE_TYPE \ HttpResponse * #define FUNCTION_LOG_HTTP_RESPONSE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpResponseToLog, buffer, bufferSize) #define FUNCTION_LOG_HTTP_RESPONSE_MULTI_TYPE \ HttpResponseMulti * #define FUNCTION_LOG_HTTP_RESPONSE_MULTI_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "HttpResponseMulti", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/session.c0000664000175100017510000000570015240547630020711 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Session ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/session.h" #include "common/io/io.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpSession { HttpClient *httpClient; // HTTP client IoSession *ioSession; // IO session }; /**********************************************************************************************************************************/ FN_EXTERN HttpSession * httpSessionNew(HttpClient *const httpClient, IoSession *const ioSession) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_CLIENT, httpClient); FUNCTION_LOG_PARAM(IO_SESSION, ioSession); FUNCTION_LOG_END(); ASSERT(httpClient != NULL); ASSERT(ioSession != NULL); OBJ_NEW_BEGIN(HttpSession, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpSession) { .httpClient = httpClient, .ioSession = ioSessionMove(ioSession, memContextCurrent()), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(HTTP_SESSION, this); } /**********************************************************************************************************************************/ FN_EXTERN void httpSessionDone(HttpSession *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); httpClientReuse(this->httpClient, this); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN IoRead * httpSessionIoRead(HttpSession *const this, const HttpSessionIoReadParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_SESSION, this); FUNCTION_TEST_PARAM(BOOL, param.ignoreUnexpectedEof); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_READ, ioSessionIoReadP(this->ioSession, .ignoreUnexpectedEof = param.ignoreUnexpectedEof)); } /**********************************************************************************************************************************/ FN_EXTERN IoWrite * httpSessionIoWrite(HttpSession *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_WRITE, ioSessionIoWrite(this->ioSession)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/session.h0000664000175100017510000000714515240547630020723 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP Session HTTP sessions are created by calling httpClientOpen(), which is currently done exclusively by the HttpRequest object. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_SESSION_H #define COMMON_IO_HTTP_SESSION_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpSession HttpSession; #include "common/io/http/client.h" #include "common/io/read.h" #include "common/io/session.h" #include "common/io/write.h" #include "common/type/object.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN HttpSession *httpSessionNew(HttpClient *client, IoSession *session); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS HttpSession * httpSessionMove(HttpSession *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Work with the session has finished cleanly and it can be reused FN_EXTERN void httpSessionDone(HttpSession *this); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Read interface typedef struct HttpSessionIoReadParam { VAR_PARAM_HEADER; bool ignoreUnexpectedEof; } HttpSessionIoReadParam; #define httpSessionIoReadP(this, ...) \ httpSessionIoRead(this, (HttpSessionIoReadParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoRead *httpSessionIoRead(HttpSession *this, HttpSessionIoReadParam param); // Write interface FN_EXTERN IoWrite *httpSessionIoWrite(HttpSession *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpSessionFree(HttpSession *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_HTTP_SESSION_TYPE \ HttpSession * #define FUNCTION_LOG_HTTP_SESSION_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "HttpSession", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/url.c0000664000175100017510000001723715240547630020040 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP URL ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/url.h" #include "common/regExp.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Regular expression for URLs. This is not intended to be completely comprehensive, e.g. it is still possible to enter bad hostnames. The goal is to make sure the syntax is correct enough for the rest of the parsing to succeed. ***********************************************************************************************************************************/ STRING_STATIC( HTTP_URL_REGEXP_STR, "^(http[s]{0,1}:\\/\\/){0,1}" // Optional protocol (http or https) "([^\\[\\:\\/?]+|\\[[a-fA-F0-9:]+\\])" // host/ipv4/ipv6 "(:[1-9][0-9]{0,4}){0,1}" // Optional port "(\\/[^?\\/]*)*$"); // Optional path /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct HttpUrl { HttpUrlPub pub; }; /*********************************************************************************************************************************** Convert protocol type to a string ***********************************************************************************************************************************/ STRING_STATIC(HTTP_PROTOCOL_HTTP_STR, "http"); STRING_STATIC(HTTP_PROTOCOL_HTTPS_STR, "https"); static const String * httpProtocolTypeStr(const HttpProtocolType type) { switch (type) { case httpProtocolTypeHttp: return HTTP_PROTOCOL_HTTP_STR; case httpProtocolTypeHttps: return HTTP_PROTOCOL_HTTPS_STR; default: return NULL; } } /**********************************************************************************************************************************/ FN_EXTERN HttpUrl * httpUrlNewParse(const String *const url, const HttpUrlNewParseParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, url); FUNCTION_TEST_PARAM(ENUM, param.type); FUNCTION_TEST_PARAM(ENUM, param.defaultType); FUNCTION_TEST_END(); ASSERT(url != NULL); OBJ_NEW_BEGIN(HttpUrl, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (HttpUrl) { .pub = { .url = strDup(url), }, }; MEM_CONTEXT_TEMP_BEGIN() { // Check that URL format is one we accept if (!regExpMatchOne(HTTP_URL_REGEXP_STR, url)) THROW_FMT(FormatError, "invalid URL '%s'", strZ(url)); // Determine whether the first part is protocol or host StringList *const splitUrl = strLstNewSplitZ(url, "/"); if (strEqZ(strLstGet(splitUrl, 0), "http:")) this->pub.type = httpProtocolTypeHttp; else if (strEqZ(strLstGet(splitUrl, 0), "https:")) this->pub.type = httpProtocolTypeHttps; // If no protocol found then the first part is the host if (this->pub.type == httpProtocolTypeAny) { // Must have either an explicitly set protocol via the type parameter or a default protocol if the type is any ASSERT(param.type != httpProtocolTypeAny || param.defaultType != httpProtocolTypeAny); this->pub.type = param.type != httpProtocolTypeAny ? param.type : param.defaultType; } // Else protocol was found else { // Protocol must match expected if (param.type != httpProtocolTypeAny && this->pub.type != param.type) THROW_FMT(FormatError, "expected protocol '%s' in URL '%s'", strZ(httpProtocolTypeStr(param.type)), strZ(url)); // Remove protocol parts from split strLstRemoveIdx(splitUrl, 0); strLstRemoveIdx(splitUrl, 0); } // Get host const String *host = strLstGet(splitUrl, 0); const String *port = NULL; // If an IPv6 address if (strBeginsWithZ(host, "[")) { // Split closing bracket const StringList *const splitHost = strLstNewSplitZ(host, "]"); ASSERT(strLstSize(splitHost) == 2); // Remove opening bracket host = strSub(strLstGet(splitHost, 0), 1); // Get port if specified if (strSize(strLstGet(splitHost, 1)) > 0) port = strSub(strLstGet(splitHost, 1), 1); } // Else IPv4 or host name else { // Split on colon const StringList *const splitHost = strLstNewSplitZ(host, ":"); ASSERT(strLstSize(splitHost) != 0); // First part is the host host = strLstGet(splitHost, 0); // Second part is the port, if it exists if (strLstSize(splitHost) > 1) { ASSERT(strLstSize(splitHost) == 2); port = strLstGet(splitHost, 1); } } // Copy host into object context MEM_CONTEXT_PRIOR_BEGIN() { this->pub.host = strDup(host); } MEM_CONTEXT_PRIOR_END(); // Get port if specified if (port != NULL) { this->pub.port = cvtZToUInt(strZ(port)); } // Else set default port based on the protocol else { ASSERT(this->pub.type != httpProtocolTypeAny); if (this->pub.type == httpProtocolTypeHttp) this->pub.port = 80; else this->pub.port = 443; } // Check for path if (strLstSize(splitUrl) > 1) { // Remove host part so it is easier to construct the path strLstRemoveIdx(splitUrl, 0); // Construct path and copy into local context const String *const path = strLstJoin(splitUrl, "/"); MEM_CONTEXT_PRIOR_BEGIN() { this->pub.path = strNewFmt("/%s", strZ(path)); } MEM_CONTEXT_PRIOR_END(); } // Else default path is / else this->pub.path = FSLASH_STR; } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(HTTP_URL, this); } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void httpUrlToLog(const HttpUrl *const this, StringStatic *const debugLog) { // Is IPv6 address? bool ipv6 = strChr(this->pub.host, ':') != -1; strStcFmt( debugLog, "{%s://%s%s%s:%u%s}", strZ(httpProtocolTypeStr(this->pub.type)), ipv6 ? "[" : "", strZ(this->pub.host), ipv6 ? "]" : "", this->pub.port, strZ(this->pub.path)); } #endif // DEBUG ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/http/url.h0000664000175100017510000001075215240547630020040 0ustar00runnerrunner/*********************************************************************************************************************************** HTTP URL Parse a URL into component parts. ***********************************************************************************************************************************/ #ifndef COMMON_IO_HTTP_URL_H #define COMMON_IO_HTTP_URL_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct HttpUrl HttpUrl; #include "common/type/object.h" #include "common/type/param.h" #include "common/type/string.h" /*********************************************************************************************************************************** HTTP protocol type ***********************************************************************************************************************************/ typedef enum { httpProtocolTypeAny = 0, httpProtocolTypeHttp = 1, httpProtocolTypeHttps = 2, } HttpProtocolType; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct HttpUrlNewParseParam { VAR_PARAM_HEADER; HttpProtocolType type; // Expected protocol type (httpProtocolTypeAny if any) HttpProtocolType defaultType; // Default protocol type if explicitly set type is any } HttpUrlNewParseParam; #define httpUrlNewParseP(url, ...) \ httpUrlNewParse(url, (HttpUrlNewParseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpUrl *httpUrlNewParse(const String *const url, HttpUrlNewParseParam param); /*********************************************************************************************************************************** Getters/setters ***********************************************************************************************************************************/ typedef struct HttpUrlPub { const String *url; // Original URL HttpProtocolType type; // Protocol type, e.g. http const String *host; // Host unsigned int port; // Port const String *path; // Path } HttpUrlPub; // Protocol type FN_INLINE_ALWAYS HttpProtocolType httpUrlProtocolType(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->type; } // Host FN_INLINE_ALWAYS const String * httpUrlHost(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->host; } // Path FN_INLINE_ALWAYS const String * httpUrlPath(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->path; } // Port FN_INLINE_ALWAYS unsigned int httpUrlPort(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->port; } // URL (exactly as originally passed) FN_INLINE_ALWAYS const String * httpUrl(const HttpUrl *const this) { return THIS_PUB(HttpUrl)->url; } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void httpUrlFree(HttpUrl *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void httpUrlToLog(const HttpUrl *this, StringStatic *debugLog); #define FUNCTION_LOG_HTTP_URL_TYPE \ HttpUrl * #define FUNCTION_LOG_HTTP_URL_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, httpUrlToLog, buffer, bufferSize) #endif // DEBUG #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/io.c0000664000175100017510000001224115240547630016654 0ustar00runnerrunner/*********************************************************************************************************************************** IO Functions ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/filter/sink.h" #include "common/io/io.h" #include "common/log.h" /*********************************************************************************************************************************** Buffer size This buffer size will be used for all IO operations that require buffers not passed by the caller. Initially it is set to a conservative default with the expectation that it will be changed to a new value after options have been loaded. In general callers should set their buffer size using ioBufferSize() but there may be cases where an alternative buffer size makes sense. ***********************************************************************************************************************************/ #define IO_BUFFER_BLOCK_SIZE (8 * 1024) static size_t bufferSize = (8 * IO_BUFFER_BLOCK_SIZE); // I/O timeout in milliseconds static TimeMSec timeoutMs = 60000; /**********************************************************************************************************************************/ FN_EXTERN size_t ioBufferSize(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(SIZE, bufferSize); } FN_EXTERN void ioBufferSizeSet(const size_t bufferSizeParam) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, bufferSizeParam); FUNCTION_TEST_END(); bufferSize = bufferSizeParam; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN TimeMSec ioTimeoutMs(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(TIME_MSEC, timeoutMs); } FN_EXTERN void ioTimeoutMsSet(const TimeMSec timeoutMsParam) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TIME_MSEC, timeoutMsParam); FUNCTION_TEST_END(); timeoutMs = timeoutMsParam; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * ioReadBuf(IoRead *const read) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_END(); ASSERT(read != NULL); Buffer *const result = bufNew(0); MEM_CONTEXT_TEMP_BEGIN() { // Read IO into the buffer do { bufResize(result, bufSize(result) + ioBufferSize()); ioRead(read, result); } while (!ioReadEof(read)); // Resize the buffer and move to prior context bufResize(result, bufUsed(result)); bufMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN void ioCopy(IoRead *const source, IoWrite *const destination, const IoCopyParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, source); FUNCTION_TEST_PARAM(IO_WRITE, destination); FUNCTION_TEST_PARAM(VARIANT, param.limit); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { Buffer *const buffer = bufNew(ioBufferSize()); uint64_t copied = 0; bool limitReached = false; do { // If a limit was specified then limit the buffer size on the last iteration if (param.limit != NULL) { const uint64_t bufferLimit = varUInt64(param.limit) - copied; if (bufferLimit < bufSize(buffer)) { bufLimitSet(buffer, (size_t)bufferLimit); limitReached = true; } } // Copy bytes ioRead(source, buffer); ioWrite(destination, buffer); // Update bytes copied and clear the buffer copied += bufUsed(buffer); bufUsedZero(buffer); } while (!limitReached && !ioReadEof(source)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool ioReadDrain(IoRead *const read) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_END(); ASSERT(read != NULL); // Add a sink filter so we only need one read ioFilterGroupAdd(ioReadFilterGroup(read), ioSinkNew()); // Check if the IO can be opened bool result = ioReadOpen(read); if (result) { MEM_CONTEXT_TEMP_BEGIN() { // A single read that returns zero bytes ioRead(read, bufNew(1)); ASSERT(ioReadEof(read)); // Close the IO ioReadClose(read); } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/io.h0000664000175100017510000000424115240547630016662 0ustar00runnerrunner/*********************************************************************************************************************************** IO Functions Common IO functions. ***********************************************************************************************************************************/ #ifndef COMMON_IO_IO_H #define COMMON_IO_IO_H #include #include #include #include /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Copy data from source to destination (both must be open and neither will be closed) typedef struct IoCopyParam { VAR_PARAM_HEADER; const Variant *limit; // Limit bytes to copy from source } IoCopyParam; #define ioCopyP(source, destination, ...) \ ioCopy(source, destination, (IoCopyParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void ioCopy(IoRead *source, IoWrite *destination, IoCopyParam param); // Read all IO into a buffer FN_EXTERN Buffer *ioReadBuf(IoRead *read); // Read all IO but don't store it. Useful for calculating checksums, size, etc. FN_EXTERN bool ioReadDrain(IoRead *read); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Buffer size FN_EXTERN size_t ioBufferSize(void); FN_EXTERN void ioBufferSizeSet(size_t bufferSize); // I/O timeout in milliseconds. Used to timeout on connections and read/write operations. Note that an *entire* read/write operation // does not need to take place within this timeout but at least some progress needs to be made, even if it is only a byte. FN_EXTERN TimeMSec ioTimeoutMs(void); FN_EXTERN void ioTimeoutMsSet(TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/limitRead.c0000664000175100017510000000763415240547630020171 0ustar00runnerrunner/*********************************************************************************************************************************** Read Chunked I/O ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/limitRead.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoLimitRead { IoRead *read; // IoRead to read data from uint64_t limit; // Limit of data to read uint64_t current; // Current out of data read } IoLimitRead; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_LIMIT_READ_TYPE \ IoLimitRead * #define FUNCTION_LOG_IO_LIMIT_READ_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoLimitRead", buffer, bufferSize) /*********************************************************************************************************************************** Read next chunk or partial chunk ***********************************************************************************************************************************/ static size_t ioLimitRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(IoLimitRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_LIMIT_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Continue reading until limit const size_t result = this->limit - this->current >= bufRemains(buffer) ? bufRemains(buffer) : (size_t)(this->limit - this->current); bufLimitSet(buffer, bufUsed(buffer) + result); ioRead(this->read, buffer); // Update current read this->current += result; FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Have all chunks been read? ***********************************************************************************************************************************/ static bool ioLimitReadEof(THIS_VOID) { THIS(IoLimitRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_LIMIT_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->limit == this->current); } /**********************************************************************************************************************************/ FN_EXTERN IoRead * ioLimitReadNew(IoRead *const read, const uint64_t limit) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(UINT64, limit); FUNCTION_LOG_END(); ASSERT(read != NULL); OBJ_NEW_BEGIN(IoLimitRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoLimitRead) { .read = read, .limit = limit, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_READ, ioReadNewP(this, .eof = ioLimitReadEof, .read = ioLimitRead)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/limitRead.h0000664000175100017510000000137315240547630020170 0ustar00runnerrunner/*********************************************************************************************************************************** Read Limited Data Read only as much data as specified. ***********************************************************************************************************************************/ #ifndef COMMON_IO_LIMITREAD_H #define COMMON_IO_LIMITREAD_H #include "common/io/read.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoRead *ioLimitReadNew(IoRead *read, uint64_t limit); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/read.c0000664000175100017510000004255615240547630017174 0ustar00runnerrunner/*********************************************************************************************************************************** IO Read Interface ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoRead { IoReadPub pub; // Publicly accessible variables Buffer *input; // Input buffer Buffer *output; // Internal output buffer (extra output from buffered reads) size_t outputPos; // Current position in the internal output buffer }; /**********************************************************************************************************************************/ FN_EXTERN IoRead * ioReadNew(void *const driver, const IoReadInterface interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_READ_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface.read != NULL); OBJ_NEW_BEGIN(IoRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoRead) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, .filterGroup = ioFilterGroupNew(), }, .input = bufNew(ioBufferSize()), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_READ, this); } /**********************************************************************************************************************************/ FN_EXTERN bool ioReadOpen(IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.opened && !this->pub.closed); ASSERT(ioFilterGroupSize(this->pub.filterGroup) == 0 || !ioReadBlock(this)); // Open if the driver has an open function const bool result = ioReadInterface(this)->open != NULL ? ioReadInterface(this)->open(ioReadDriver(this)) : true; // Only open the filter group if the read was opened if (result) ioFilterGroupOpen(this->pub.filterGroup); #ifdef DEBUG this->pub.opened = result; #endif FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Is the driver at EOF? This is different from the overall eof because filters may still be holding buffered data. ***********************************************************************************************************************************/ static bool ioReadEofDriver(const IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); FUNCTION_LOG_RETURN(BOOL, ioReadInterface(this)->eof != NULL ? ioReadInterface(this)->eof(ioReadDriver(this)) : false); } /**********************************************************************************************************************************/ static void ioReadInternal(IoRead *const this, Buffer *const buffer, const bool block) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Loop until EOF or the output buffer is full const size_t bufferUsedBegin = bufUsed(buffer); while (!ioReadEof(this) && bufRemains(buffer) > 0) { // Process input buffer again to get more output if (ioFilterGroupInputSame(this->pub.filterGroup)) { ioFilterGroupProcess(this->pub.filterGroup, this->input, buffer); } // Else new input can be accepted else { // Read if not EOF if (this->input != NULL) { if (!ioReadEofDriver(this)) { bufUsedZero(this->input); // If blocking then limit the amount of data requested if (ioReadBlock(this) && bufRemains(this->input) > bufRemains(buffer)) bufLimitSet(this->input, bufRemains(buffer)); ioReadInterface(this)->read(ioReadDriver(this), this->input, block); bufLimitClear(this->input); } // Set input to NULL and flush (no need to actually free the buffer here as it will be freed with the mem context) else this->input = NULL; } // Process the input buffer (or flush if NULL) if (this->input == NULL || !bufEmpty(this->input)) ioFilterGroupProcess(this->pub.filterGroup, this->input, buffer); // Stop if not blocking -- we don't need to fill the buffer as long as we got some data if (!block && bufUsed(buffer) > bufferUsedBegin) break; } // Eof when no more input and the filter group is done this->pub.eofAll = ioReadEofDriver(this) && ioFilterGroupDone(this->pub.filterGroup); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Read data and use buffered line read output when present ***********************************************************************************************************************************/ FN_EXTERN size_t ioRead(IoRead *const this, Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BUFFER, this->output); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Store size of remaining portion of buffer to calculate total read at the end const size_t outputRemains = bufRemains(buffer); // Copy any data in the internal output buffer if (this->output != NULL && bufUsed(this->output) - this->outputPos > 0 && bufRemains(buffer) > 0) { // Internal output buffer remains taking into account the position const size_t outputInternalRemains = bufUsed(this->output) - this->outputPos; // Determine how much data should be copied const size_t size = outputInternalRemains > bufRemains(buffer) ? bufRemains(buffer) : outputInternalRemains; // Copy data to the output buffer bufCatSub(buffer, this->output, this->outputPos, size); this->outputPos += size; } // Read data ioReadInternal(this, buffer, true); FUNCTION_LOG_RETURN(SIZE, outputRemains - bufRemains(buffer)); } /**********************************************************************************************************************************/ FN_EXTERN size_t ioReadSmall(IoRead *const this, Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, this); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Allocate the internal output buffer if it has not already been allocated if (this->output == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->output = bufNew(ioBufferSize()); } MEM_CONTEXT_OBJ_END(); } // Store size of remaining portion of buffer to calculate total read at the end size_t outputRemains = bufRemains(buffer); do { // Internal output buffer remains taking into account the position const size_t outputInternalRemains = bufUsed(this->output) - this->outputPos; // Use any data in the internal output buffer if (outputInternalRemains > 0) { // Determine how much data should be copied const size_t size = outputInternalRemains > bufRemains(buffer) ? bufRemains(buffer) : outputInternalRemains; // Copy data to the output buffer bufCatSub(buffer, this->output, this->outputPos, size); this->outputPos += size; } // If more data is required if (!bufFull(buffer)) { // If the data required is the same size as the internal output buffer then just read into the external buffer if (bufRemains(buffer) >= bufSize(this->output)) { ioReadInternal(this, buffer, true); } // Else read as much data as is available. If it is not enough we will try again later. else { // Clear the internal output buffer since all data was copied already bufUsedZero(this->output); this->outputPos = 0; ioReadInternal(this, this->output, false); } } } while (!bufFull(buffer) && !ioReadEof(this)); FUNCTION_TEST_RETURN(SIZE, outputRemains - bufRemains(buffer)); } /*********************************************************************************************************************************** The entire string to search for must fit within a single buffer. ***********************************************************************************************************************************/ FN_EXTERN String * ioReadLineParam(IoRead *const this, const bool allowEof) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BOOL, allowEof); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Allocate the output buffer if it has not already been allocated. This buffer is not allocated at object creation because it // is not always used. if (this->output == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->output = bufNew(ioBufferSize()); } MEM_CONTEXT_OBJ_END(); } // Search for a linefeed String *result = NULL; do { // Internal output buffer remains taking into account the position size_t outputInternalRemains = bufUsed(this->output) - this->outputPos; if (outputInternalRemains > 0) { // Internal output buffer pointer taking into account the position const char *const outputPtr = (char *)bufPtr(this->output) + this->outputPos; // Search for a linefeed in the buffer const char *const linefeed = memchr(outputPtr, '\n', outputInternalRemains); // A linefeed was found so get the string if (linefeed != NULL) { // Get the string size const size_t size = (size_t)(linefeed - outputPtr); // Create the string result = strNewZN(outputPtr, size); this->outputPos += size + 1; } } // Read data if no linefeed was found in the existing buffer if (result == NULL) { // If there is remaining data left in the internal output buffer then trim off the used data if (outputInternalRemains > 0) { memmove( bufPtr(this->output), bufPtr(this->output) + (bufUsed(this->output) - outputInternalRemains), outputInternalRemains); } // Set used bytes and reset position bufUsedSet(this->output, outputInternalRemains); this->outputPos = 0; // If the buffer is full then the linefeed (if it exists) is outside the buffer if (bufFull(this->output)) THROW_FMT(FileReadError, "unable to find line in %zu byte buffer", bufUsed(this->output)); if (ioReadEof(this)) { if (allowEof) result = strNewZN((char *)bufPtr(this->output), bufUsed(this->output)); else THROW(FileReadError, "unexpected eof while reading line"); } else ioReadInternal(this, this->output, false); } } while (result == NULL); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t ioReadVarIntU64(IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); // Allocate the internal output buffer if it has not already been allocated if (this->output == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->output = bufNew(ioBufferSize()); } MEM_CONTEXT_OBJ_END(); } uint64_t result = 0; uint8_t byte; // Convert bytes from varint-128 encoding to a uint64 for (unsigned int bufferIdx = 0; bufferIdx < CVT_VARINT128_BUFFER_SIZE; bufferIdx++) { // Get more bytes if needed if (bufUsed(this->output) - this->outputPos == 0) { // Clear the internal output buffer since all data was copied already bufUsedZero(this->output); this->outputPos = 0; // Read into the internal output buffer ioReadInternal(this, this->output, false); // Error on eof if (bufUsed(this->output) == 0) THROW(FileReadError, "unexpected eof"); } // Get the next encoded byte byte = bufPtr(this->output)[this->outputPos++]; // Shift the lower order 7 encoded bits into the uint64 in reverse order result |= (uint64_t)(byte & 0x7f) << (7 * bufferIdx); // Done if the high order bit is not set to indicate more data if (byte < 0x80) break; } // By this point all bytes should have been read so error if this is not the case. This could be due to a coding error or // corrupton in the data stream. if (byte >= 0x80) THROW(FormatError, "unterminated base-128 integer"); FUNCTION_LOG_RETURN(UINT64, result); } /**********************************************************************************************************************************/ FN_EXTERN bool ioReadReady(IoRead *const this, const IoReadReadyParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BOOL, param.error); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = true; if (ioReadInterface(this)->ready != NULL) result = ioReadInterface(this)->ready(ioReadDriver(this), param.error); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t ioReadFlush(IoRead *const this, const IoReadFlushParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_PARAM(BOOL, param.errorOnBytes); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Flush remaining data uint64_t result = 0; if (!ioReadEof(this)) { Buffer *const buffer = bufNew(ioBufferSize()); do { result += ioRead(this, buffer); bufUsedZero(buffer); } while (!ioReadEof(this)); bufFree(buffer); } // Error when bytes found and error requested if (result != 0 && param.errorOnBytes) THROW_FMT(FileReadError, "expected EOF but flushed %" PRIu64 " byte(s)", result); FUNCTION_LOG_RETURN(UINT64, result); } /**********************************************************************************************************************************/ FN_EXTERN void ioReadClose(IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.opened && !this->pub.closed); // Close the filter group and gather results ioFilterGroupClose(this->pub.filterGroup); // Close the driver if there is a close function if (ioReadInterface(this)->close != NULL) ioReadInterface(this)->close(ioReadDriver(this)); #ifdef DEBUG this->pub.closed = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int ioReadFd(const IoRead *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INT, ioReadInterface(this)->fd == NULL ? -1 : ioReadInterface(this)->fd(ioReadDriver(this))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/read.h0000664000175100017510000001251415240547630017170 0ustar00runnerrunner/*********************************************************************************************************************************** IO Read Interface Objects that read from some IO source (file, socket, etc.) are implemented using this interface. All objects are required to implement IoReadProcess and can optionally implement IoReadOpen, IoReadClose, or IoReadEof. IoReadOpen and IoReadClose can be used to allocate/open or deallocate/free resources. If IoReadEof is not implemented then ioReadEof() will always return false. An example of an IoRead object is IoBufferRead. ***********************************************************************************************************************************/ #ifndef COMMON_IO_READ_H #define COMMON_IO_READ_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoRead IoRead; #include "common/io/filter/group.h" #include "common/io/read.intern.h" #include "common/type/buffer.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Do reads block when more bytes are requested than are available to read? FN_INLINE_ALWAYS bool ioReadBlock(const IoRead *const this) { return THIS_PUB(IoRead)->interface.block; } // Is IO at EOF? All driver reads are complete and all data has been flushed from the filters (if any). FN_INLINE_ALWAYS bool ioReadEof(const IoRead *const this) { ASSERT_INLINE(THIS_PUB(IoRead)->opened && !THIS_PUB(IoRead)->closed); return THIS_PUB(IoRead)->eofAll; } // Get filter group if filters need to be added FN_INLINE_ALWAYS IoFilterGroup * ioReadFilterGroup(IoRead *const this) { return THIS_PUB(IoRead)->filterGroup; } // File descriptor for the read object. Not all read objects have a file descriptor and -1 will be returned in that case. FN_EXTERN int ioReadFd(const IoRead *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open the IO FN_EXTERN bool ioReadOpen(IoRead *this); // Read data from IO and process filters FN_EXTERN size_t ioRead(IoRead *this, Buffer *buffer); // Same as ioRead() but optimized for small reads (intended for making repetitive reads that are smaller than ioBufferSize()) FN_EXTERN size_t ioReadSmall(IoRead *this, Buffer *buffer); // Read linefeed-terminated string and optionally error on eof FN_EXTERN String *ioReadLineParam(IoRead *this, bool allowEof); // Read linefeed-terminated string FN_INLINE_ALWAYS String * ioReadLine(IoRead *const this) { return ioReadLineParam(this, false); } // Read varint-128 encoding FN_EXTERN uint64_t ioReadVarIntU64(IoRead *this); // Are there bytes ready to read immediately? There are no guarantees on how much data is available to read but it must be at least // one byte. typedef struct IoReadReadyParam { VAR_PARAM_HEADER; bool error; // Error when read not ready } IoReadReadyParam; #define ioReadReadyP(this, ...) \ ioReadReady(this, (IoReadReadyParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool ioReadReady(IoRead *this, IoReadReadyParam param); // Flush all remaining bytes and return bytes flushed. Optionally error when bytes are flushed. typedef struct IoReadFlushParam { VAR_PARAM_HEADER; bool errorOnBytes; // Error when any bytes are found during flush } IoReadFlushParam; #define ioReadFlushP(this, ...) \ ioReadFlush(this, (IoReadFlushParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN uint64_t ioReadFlush(IoRead *this, IoReadFlushParam param); // Close the IO FN_EXTERN void ioReadClose(IoRead *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioReadFree(IoRead *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_READ_TYPE \ IoRead * #define FUNCTION_LOG_IO_READ_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoRead", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/read.intern.h0000664000175100017510000000712515240547630020470 0ustar00runnerrunner/*********************************************************************************************************************************** IO Read Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_READ_INTERN_H #define COMMON_IO_READ_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ #include "common/io/read.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct IoReadInterface { bool block; // Do reads block when buffer is larger than available bytes? bool (*eof)(void *driver); void (*close)(void *driver); bool (*open)(void *driver); int (*fd)(const void *driver); size_t (*read)(void *driver, Buffer *buffer, bool block); // Are there bytes ready to read immediately? There are no guarantees on how much data is available to read but it must be at // least one byte. Optionally error when read is not ready. bool (*ready)(void *driver, bool error); } IoReadInterface; #define ioReadNewP(driver, ...) \ ioReadNew(driver, (IoReadInterface){__VA_ARGS__}) FN_EXTERN IoRead *ioReadNew(void *driver, IoReadInterface interface); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoReadPub { void *driver; // Driver object IoReadInterface interface; // Driver interface IoFilterGroup *filterGroup; // IO filters bool eofAll; // Is the read done (read and filters complete)? #ifdef DEBUG bool opened; // Has the io been opened? bool closed; // Has the io been closed? #endif } IoReadPub; // Driver for the read object FN_INLINE_ALWAYS void * ioReadDriver(const IoRead *const this) { return THIS_PUB(IoRead)->driver; } // Interface for the read object FN_INLINE_ALWAYS const IoReadInterface * ioReadInterface(const IoRead *const this) { return &THIS_PUB(IoRead)->interface; } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_READ_INTERFACE_TYPE \ IoReadInterface #define FUNCTION_LOG_IO_READ_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoReadInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/server.c0000664000175100017510000000437315240547630017562 0ustar00runnerrunner/*********************************************************************************************************************************** Io Server Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/server.h" #include "common/log.h" #include "common/memContext.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoServer { IoServerPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN IoServer * ioServerNew(void *const driver, const IoServerInterface *const interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_SERVER_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface != NULL); ASSERT(interface->type != 0); ASSERT(interface->name != NULL); ASSERT(interface->accept != NULL); ASSERT(interface->toLog != NULL); OBJ_NEW_BEGIN(IoServer, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoServer) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_SERVER, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioServerToLog(const IoServer *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(this->pub.interface->type, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", driver: "); this->pub.interface->toLog(this->pub.driver, debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/server.h0000664000175100017510000000642215240547630017564 0ustar00runnerrunner/*********************************************************************************************************************************** Io Server Interface Create sessions for protocol servers. For example, a TLS server can be created with tlsServerNew() and then new TLS sessions can be accepted with ioServerAccept(). ***********************************************************************************************************************************/ #ifndef COMMON_IO_SERVER_H #define COMMON_IO_SERVER_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoServer IoServer; #include "common/io/server.intern.h" #include "common/io/session.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoServerPub { void *driver; // Driver object const IoServerInterface *interface; // Driver interface } IoServerPub; // Name that identifies the server FN_INLINE_ALWAYS const String * ioServerName(const IoServer *const this) { return THIS_PUB(IoServer)->interface->name(THIS_PUB(IoServer)->driver); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS IoServer * ioServerMove(IoServer *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Open session FN_INLINE_ALWAYS IoSession * ioServerAccept(IoServer *const this, IoSession *const session) { return THIS_PUB(IoServer)->interface->accept(THIS_PUB(IoServer)->driver, session); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioServerFree(IoServer *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioServerToLog(const IoServer *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_SERVER_TYPE \ IoServer * #define FUNCTION_LOG_IO_SERVER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioServerToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/server.intern.h0000664000175100017510000000416215240547630021061 0ustar00runnerrunner/*********************************************************************************************************************************** Io Server Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_SERVER_INTERN_H #define COMMON_IO_SERVER_INTERN_H #include "common/io/server.h" #include "common/io/session.h" #include "common/type/string.h" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct IoServerInterface { // Type used to identify the server StringId type; // Server name, usually address:port or some other unique identifier const String *(*name)(void *driver); // Accept a session IoSession *(*accept)(void *driver, IoSession *session); // Driver log function void (*toLog)(const void *driver, StringStatic *debugLog); } IoServerInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoServer *ioServerNew(void *driver, const IoServerInterface *interface); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_SERVER_INTERFACE_TYPE \ IoServerInterface * #define FUNCTION_LOG_IO_SERVER_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoServerInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/session.c0000664000175100017510000001142115240547630017727 0ustar00runnerrunner/*********************************************************************************************************************************** Io Session Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/session.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoSession { IoSessionPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN IoSession * ioSessionNew(void *const driver, const IoSessionInterface *const interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_SESSION_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface != NULL); ASSERT(interface->type != 0); ASSERT(interface->close != NULL); ASSERT(interface->ioRead != NULL); ASSERT(interface->ioWrite != NULL); ASSERT(interface->role != NULL); ASSERT(interface->toLog != NULL); OBJ_NEW_BEGIN(IoSession, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoSession) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_SESSION, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioSessionAuthenticatedSet(IoSession *const this, const bool authenticated) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_SESSION, this); FUNCTION_TEST_PARAM(BOOL, authenticated); FUNCTION_TEST_END(); this->pub.authenticated = authenticated; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int ioSessionFd(IoSession *this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->pub.interface->fd == NULL ? -1 : this->pub.interface->fd(this->pub.driver)); } /**********************************************************************************************************************************/ FN_EXTERN void ioSessionPeerNameSet(IoSession *const this, const String *const peerName) // {vm_covered} { FUNCTION_TEST_BEGIN(); // {vm_covered} FUNCTION_TEST_PARAM(IO_SESSION, this); // {vm_covered} FUNCTION_TEST_PARAM(STRING, peerName); // {vm_covered} FUNCTION_TEST_END(); // {vm_covered} MEM_CONTEXT_OBJ_BEGIN(this) // {vm_covered} { this->pub.peerName = strDup(peerName); // {vm_covered} } MEM_CONTEXT_END(); // {vm_covered} FUNCTION_TEST_RETURN_VOID(); // {vm_covered} } // {vm_covered} /**********************************************************************************************************************************/ FN_EXTERN void ioSessionToLog(const IoSession *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(this->pub.interface->type, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", role: "); strStcResultSizeInc(debugLog, strIdToLog(ioSessionRole(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", driver: "); this->pub.interface->toLog(this->pub.driver, debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/session.h0000664000175100017510000001267515240547630017750 0ustar00runnerrunner/*********************************************************************************************************************************** Io Session Interface Provides access to IoRead and IoWrite interfaces for interacting with the session returned by ioClientOpen(). Sessions should always be closed when work with them is done but they also contain destructors to do cleanup if there is an error. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SESSION_H #define COMMON_IO_SESSION_H #include "common/type/stringId.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoSession IoSession; /*********************************************************************************************************************************** Session roles ***********************************************************************************************************************************/ typedef enum { ioSessionRoleClient = STRID5("client", 0x28e2a5830), // Client session ioSessionRoleServer = STRID5("server", 0x245b48b30), // Server session } IoSessionRole; #include "common/io/read.h" #include "common/io/session.intern.h" #include "common/io/write.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoSessionPub { void *driver; // Driver object const IoSessionInterface *interface; // Driver interface const String *peerName; // Name of peer (exact meaning depends on driver) bool authenticated; // Is the session authenticated? } IoSessionPub; // Is the session authenticated? The exact meaning of "authenticated" will vary by driver type. FN_INLINE_ALWAYS bool ioSessionAuthenticated(const IoSession *const this) { return THIS_PUB(IoSession)->authenticated; } // Session file descriptor, -1 if none FN_EXTERN int ioSessionFd(IoSession *this); // Read interface typedef struct IoSessionIoReadParam { VAR_PARAM_HEADER; bool ignoreUnexpectedEof; // Allow unexpected EOF, e.g. TLS session improperly terminated } IoSessionIoReadParam; #define ioSessionIoReadP(this, ...) \ ioSessionIoRead(this, (IoSessionIoReadParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_INLINE_ALWAYS IoRead * ioSessionIoRead(IoSession *const this, const IoSessionIoReadParam param) { return THIS_PUB(IoSession)->interface->ioRead(THIS_PUB(IoSession)->driver, param.ignoreUnexpectedEof); } // Write interface FN_INLINE_ALWAYS IoWrite * ioSessionIoWrite(IoSession *const this) { return THIS_PUB(IoSession)->interface->ioWrite(THIS_PUB(IoSession)->driver); } // The peer name, if any. The exact meaning will vary by driver type, for example the peer name will be the client CN for the // TlsSession driver. FN_INLINE_ALWAYS const String * ioSessionPeerName(const IoSession *const this) { return THIS_PUB(IoSession)->peerName; } // Session role FN_INLINE_ALWAYS IoSessionRole ioSessionRole(const IoSession *const this) { return THIS_PUB(IoSession)->interface->role(THIS_PUB(IoSession)->driver); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Close the session FN_INLINE_ALWAYS void ioSessionClose(IoSession *const this) { return THIS_PUB(IoSession)->interface->close(THIS_PUB(IoSession)->driver); } // Move to a new parent mem context FN_INLINE_ALWAYS IoSession * ioSessionMove(IoSession *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioSessionFree(IoSession *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void ioSessionToLog(const IoSession *this, StringStatic *debugLog); #define FUNCTION_LOG_IO_SESSION_TYPE \ IoSession * #define FUNCTION_LOG_IO_SESSION_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, ioSessionToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/session.intern.h0000664000175100017510000000540615240547630021240 0ustar00runnerrunner/*********************************************************************************************************************************** Io Session Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_SESSION_INTERN_H #define COMMON_IO_SESSION_INTERN_H #include "common/io/session.h" #include "common/io/write.h" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct IoSessionInterface { // Type used to identify the session StringId type; // Close the session void (*close)(void *driver); // Session file descriptor, if any int (*fd)(void *driver); // IoRead interface for the session IoRead *(*ioRead)(void *driver, bool ignoreUnexpectedEof); // IoWrite interface for the session IoWrite *(*ioWrite)(void *driver); // Session role IoSessionRole (*role)(const void *driver); // Driver log function void (*toLog)(const void *driver, StringStatic *debugLog); } IoSessionInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoSession *ioSessionNew(void *driver, const IoSessionInterface *interface); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Has the session been authenticated? FN_EXTERN void ioSessionAuthenticatedSet(IoSession *this, bool authenticated); // Set the peer name FN_EXTERN void ioSessionPeerNameSet(IoSession *this, const String *peerName); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_SESSION_INTERFACE_TYPE \ IoSessionInterface * #define FUNCTION_LOG_IO_SESSION_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoSessionInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/0000775000175100017510000000000015240547630017371 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/address.c0000664000175100017510000003014115240547630021161 0ustar00runnerrunner/*********************************************************************************************************************************** Address Info ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/io/socket/address.h" #include "common/log.h" #include "common/memContext.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct AddressInfo { AddressInfoPub pub; // Publicly accessible variables struct addrinfo *info; // Linked list of addresses }; /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ // Store preferred addresses for hosts typedef struct AddressInfoPreference { const String *const host; // Host String *address; // Preferred address for host } AddressInfoPreference; static struct AddressInfoLocal { MemContext *memContext; // Mem context List *prefList; // List of preferred addresses for hosts } addressInfoLocal; /*********************************************************************************************************************************** Free addrinfo linked list allocated by getaddrinfo() ***********************************************************************************************************************************/ static void addrInfoFreeResource(THIS_VOID) { THIS(AddressInfo); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ADDRESS_INFO, this); FUNCTION_LOG_END(); ASSERT(this != NULL); freeaddrinfo(this->info); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void addrInfoSort(AddressInfo *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ADDRESS_INFO, this); FUNCTION_TEST_END(); // Only sort if there is more than one address if (lstSize(this->pub.list) > 1) { // By default start with IPv6 and first address int family = AF_INET6; unsigned int addrIdx = 0; // If a preferred address is in the list then move it to the first position and update family const AddressInfoPreference *const addrPref = lstFind(addressInfoLocal.prefList, &this->pub.host); if (addrPref != NULL) { AddressInfoItem *const addrFindItem = lstFind(this->pub.list, &addrPref->address); if (addrFindItem != NULL) { // Swap with first address if the address is not already first AddressInfoItem *const addrFirstItem = lstGet(this->pub.list, 0); if (addrFirstItem != addrFindItem) { const AddressInfoItem addrCopyItem = *addrFirstItem; *addrFirstItem = *addrFindItem; *addrFindItem = addrCopyItem; } // Set family and skip first address family = addrFirstItem->info->ai_family == AF_INET6 ? AF_INET : AF_INET6; addrIdx++; } } // Alternate IPv6 and IPv4 addresses for (; addrIdx < lstSize(this->pub.list); addrIdx++) { AddressInfoItem *const addrItem = lstGet(this->pub.list, addrIdx); // If not the family we expect, search for one if (addrItem->info->ai_family != family) { unsigned int addrFindIdx = addrIdx + 1; for (; addrFindIdx < lstSize(this->pub.list); addrFindIdx++) { AddressInfoItem *const addrFindItem = lstGet(this->pub.list, addrFindIdx); if (addrFindItem->info->ai_family == family) { // Swap addresses const AddressInfoItem addrCopyItem = *addrItem; *addrItem = *addrFindItem; *addrFindItem = addrCopyItem; // Move on to next address break; } } // Address of the required family not found so sorting is done if (addrFindIdx == lstSize(this->pub.list)) break; } // Switch family family = family == AF_INET6 ? AF_INET : AF_INET6; } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN AddressInfo * addrInfoNew(const String *const host, const unsigned int port) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_END(); ASSERT(host != NULL); ASSERT(port != 0); // Initialize mem context and preference list if (addressInfoLocal.memContext == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(AddressInfo, .childQty = MEM_CONTEXT_QTY_MAX) { addressInfoLocal.memContext = MEM_CONTEXT_NEW(); addressInfoLocal.prefList = lstNewP(sizeof(AddressInfoPreference), .comparator = lstComparatorStr); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } OBJ_NEW_BEGIN(AddressInfo, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (AddressInfo) { .pub = { .host = strDup(host), .port = port, .list = lstNewP(sizeof(AddressInfoItem), .comparator = lstComparatorStr), }, }; MEM_CONTEXT_TEMP_BEGIN() { // Set hints that narrow the type of address we are looking for -- we'll take ipv4 or ipv6 struct addrinfo hints = (struct addrinfo) { .ai_family = AF_UNSPEC, .ai_flags = AI_PASSIVE | AI_NUMERICSERV, .ai_socktype = SOCK_STREAM, .ai_protocol = IPPROTO_TCP, }; // Convert the port to a zero-terminated string for use with getaddrinfo() char portZ[CVT_BASE10_BUFFER_SIZE]; cvtUIntToZ(port, portZ, sizeof(portZ)); // Do the lookup int error; if ((error = getaddrinfo(strZ(host), portZ, &hints, &this->info)) != 0) THROW_FMT(HostConnectError, "unable to get address for '%s': [%d] %s", strZ(host), error, gai_strerror(error)); // Convert address linked list to list MEM_CONTEXT_OBJ_BEGIN(this->pub.list) { struct addrinfo *info = this->info; lstAdd(this->pub.list, &(AddressInfoItem){.name = addrInfoToStr(info), .info = info}); // Set free callback to ensure address info is freed memContextCallbackSet(objMemContext(this), addrInfoFreeResource, this); while (info->ai_next != NULL) { info = info->ai_next; lstAdd(this->pub.list, &(AddressInfoItem){.name = addrInfoToStr(info), .info = info}); } } MEM_CONTEXT_OBJ_END(); } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(ADDRESS_INFO, this); } /**********************************************************************************************************************************/ FN_EXTERN void addrInfoPrefer(AddressInfo *const this, const unsigned int index) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(ADDRESS_INFO, this); FUNCTION_LOG_PARAM(UINT, index); FUNCTION_LOG_END(); AddressInfoPreference *const addrPref = lstFind(addressInfoLocal.prefList, &this->pub.host); const String *const address = addrInfoGet(this, index)->name; MEM_CONTEXT_OBJ_BEGIN(addressInfoLocal.prefList) { // If no preference set yet if (addrPref == NULL) { lstAdd(addressInfoLocal.prefList, &(AddressInfoPreference){.host = strDup(this->pub.host), .address = strDup(address)}); lstSort(addressInfoLocal.prefList, sortOrderAsc); } // Else update address else { // Free the old address strFree(addrPref->address); // Assign new address addrPref->address = strDup(address); } } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Convert address to a zero-terminated string ***********************************************************************************************************************************/ #define ADDR_INFO_STR_BUFFER_SIZE 48 #define ADDR_INFO_STR_INVALID "invalid" static void addrInfoToZ(const struct addrinfo *const addrInfo, char *const address, const size_t addressSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, addrInfo); FUNCTION_TEST_PARAM_P(CHAR, address); FUNCTION_TEST_PARAM(SIZE, addressSize); FUNCTION_TEST_END(); ASSERT(addrInfo != NULL); ASSERT(address != NULL); ASSERT(addressSize >= sizeof(ADDR_INFO_STR_INVALID)); if (getnameinfo(addrInfo->ai_addr, addrInfo->ai_addrlen, address, (socklen_t)addressSize, 0, 0, NI_NUMERICHOST) != 0) memcpy(address, ADDR_INFO_STR_INVALID, sizeof(ADDR_INFO_STR_INVALID)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * addrInfoToName(const String *const host, const unsigned int port, const struct addrinfo *const addrInfo) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, host); FUNCTION_TEST_PARAM(UINT, port); FUNCTION_TEST_PARAM_P(VOID, addrInfo); FUNCTION_TEST_END(); String *const result = strCatFmt(strNew(), "%s:%u", strZ(host), port); String *const address = addrInfoToStr(addrInfo); if (!strEq(host, address)) strCatFmt(result, " (%s)", strZ(address)); strFree(address); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * addrInfoToStr(const struct addrinfo *const addrInfo) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, addrInfo); FUNCTION_TEST_END(); char address[ADDR_INFO_STR_BUFFER_SIZE]; addrInfoToZ(addrInfo, address, sizeof(address)); FUNCTION_TEST_RETURN(STRING, strNewZ(address)); } /**********************************************************************************************************************************/ FN_EXTERN void addrInfoToLog(const AddressInfo *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{host: "); strToLog(addrInfoHost(this), debugLog); strStcFmt(debugLog, ", port: %u, list: [", addrInfoPort(this)); for (unsigned int listIdx = 0; listIdx < addrInfoSize(this); listIdx++) { const AddressInfoItem *const addrItem = addrInfoGet(this, listIdx); if (listIdx != 0) strStcCat(debugLog, ", "); strStcCat(debugLog, strZ(addrItem->name)); } strStcCat(debugLog, "]}"); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/address.h0000664000175100017510000001157615240547630021201 0ustar00runnerrunner/*********************************************************************************************************************************** Address Info Get address info for a host/address. If the input is a host then there may be multiple addresses that map to the host. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_ADDRESSINFO_H #define COMMON_IO_SOCKET_ADDRESSINFO_H #include /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct AddressInfo AddressInfo; #include "common/type/list.h" #include "common/type/object.h" #include "common/type/string.h" /*********************************************************************************************************************************** Constructor ***********************************************************************************************************************************/ FN_EXTERN AddressInfo *addrInfoNew(const String *const host, unsigned int port); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct AddressInfoItem { String *name; // Address as a string struct addrinfo *info; // Full address info } AddressInfoItem; typedef struct AddressInfoPub { const String *host; // Host for address info lookup unsigned int port; // Port for address info lookup List *list; // List of addresses for the host } AddressInfoPub; // Get address FN_INLINE_ALWAYS const AddressInfoItem * addrInfoGet(const AddressInfo *const this, const unsigned int index) { return (const AddressInfoItem *)lstGet(THIS_PUB(AddressInfo)->list, index); } // Get lookup host FN_INLINE_ALWAYS const String * addrInfoHost(const AddressInfo *const this) { return THIS_PUB(AddressInfo)->host; } // Get lookup port FN_INLINE_ALWAYS unsigned int addrInfoPort(const AddressInfo *const this) { return THIS_PUB(AddressInfo)->port; } // Size of address list FN_INLINE_ALWAYS unsigned int addrInfoSize(const AddressInfo *const this) { return lstSize(THIS_PUB(AddressInfo)->list); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Sort addresses alternating between IPv6 and IPv4 FN_EXTERN void addrInfoSort(AddressInfo *this); // Set preferred address for host, which will be preferred in future lookups FN_EXTERN void addrInfoPrefer(AddressInfo *this, unsigned int index); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void addrInfoFree(AddressInfo *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Convert address info to string FN_EXTERN String *addrInfoToStr(const struct addrinfo *addrInfo); // Generate name for the host/port/address combination (address is omitted if it equals host) FN_EXTERN String *addrInfoToName(const String *host, unsigned int port, const struct addrinfo *addrInfo); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void addrInfoToLog(const AddressInfo *this, StringStatic *debugLog); #define FUNCTION_LOG_ADDRESS_INFO_TYPE \ AddressInfo * #define FUNCTION_LOG_ADDRESS_INFO_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, addrInfoToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/client.c0000664000175100017510000003577315240547630021032 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Client ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/error/retry.h" #include "common/io/client.h" #include "common/io/fd.h" #include "common/io/socket/address.h" #include "common/io/socket/client.h" #include "common/io/socket/common.h" #include "common/io/socket/session.h" #include "common/log.h" #include "common/stat.h" #include "common/type/object.h" #include "common/wait.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(SOCKET_STAT_CLIENT_STR, SOCKET_STAT_CLIENT); STRING_EXTERN(SOCKET_STAT_RETRY_STR, SOCKET_STAT_RETRY); STRING_EXTERN(SOCKET_STAT_SESSION_STR, SOCKET_STAT_SESSION); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct SocketClient { String *host; // Hostname or IP address unsigned int port; // Port to connect to host on String *name; // Socket name (host:port) TimeMSec timeoutConnect; // Timeout for connection TimeMSec timeoutSession; // Timeout passed to session } SocketClient; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void sckClientToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const SocketClient); strStcFmt( debugLog, "{host: %s, port: %u, timeoutConnect: %" PRIu64 ", timeoutSession: %" PRIu64 "}", strZ(this->host), this->port, this->timeoutConnect, this->timeoutSession); } #define FUNCTION_LOG_SOCKET_CLIENT_TYPE \ SocketClient * #define FUNCTION_LOG_SOCKET_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, sckClientToLog, buffer, bufferSize) /*********************************************************************************************************************************** Connections are established using the "happy eyeballs" approach from RFC 8305, i.e. new addresses (if available) are tried if the prior address has already had a reasonable time to connect. This prevents waiting too long on a failed connection but does not try all the addresses at once. Prior connections that are still waiting are rechecked periodically if no subsequent connection is successful. ***********************************************************************************************************************************/ typedef struct SckClientOpenData { const struct addrinfo *const address; // IP address const char *name; // Combination of host, address, port int fd; // File descriptor for socket int errNo; // Current error (may just indicate to keep waiting) } SckClientOpenData; // Helper to determine if non-blocking connection has been established static bool sckClientOpenWait(SckClientOpenData *const openData, const TimeMSec timeout) { ASSERT(openData != NULL); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRINGZ, openData->name); FUNCTION_LOG_PARAM(INT, openData->fd); FUNCTION_LOG_PARAM(INT, openData->errNo); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(openData->name != NULL); ASSERT(openData->fd > 0); ASSERT(openData->errNo != 0); bool result = true; // The connection has started but since we are in non-blocking mode it has not completed yet if (openData->errNo == EINPROGRESS || openData->errNo == EINTR) { // Wait for write-ready if (fdReadyWrite(openData->fd, timeout)) { // Check for success or error. If the connection was successful this will set errNo to 0. socklen_t errNoLen = sizeof(openData->errNo); THROW_ON_SYS_ERROR_FMT( getsockopt(openData->fd, SOL_SOCKET, SO_ERROR, &openData->errNo, &errNoLen) == -1, HostConnectError, "unable to get socket error for '%s'", openData->name); } // Else still waiting on connection else result = false; } // Throw error if it is still set if (result && openData->errNo != 0) THROW_SYS_ERROR_CODE_FMT(openData->errNo, HostConnectError, "unable to connect to '%s'", openData->name); FUNCTION_LOG_RETURN(BOOL, result); } static IoSession * sckClientOpen(THIS_VOID) { THIS(SocketClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); IoSession *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { Wait *const wait = waitNew(this->timeoutConnect); ErrorRetry *const errRetry = errRetryNew(); bool retry; // Get an address list for the host and sort it AddressInfo *const addrInfo = addrInfoNew(this->host, this->port); addrInfoSort(addrInfo); // Try addresses until success or timeout List *const openDataList = lstNewP(sizeof(SckClientOpenData)); volatile unsigned int addrInfoIdx = 0; do { // Assume there will be no retry SckClientOpenData *volatile openData = NULL; const ErrorType *errLastType = NULL; String *errLastMessage = NULL; retry = false; // Try connection TRY_BEGIN() { // Iterate waiting connections and see if any of them have completed for (unsigned int openDataIdx = 0; openDataIdx < lstSize(openDataList); openDataIdx++) { SckClientOpenData *const openDataWait = lstGet(openDataList, openDataIdx); if (openDataWait->fd != 0) { // Set so the connection will be closed on error openData = openDataWait; // Check if the connection has completed without waiting if (sckClientOpenWait(openData, 0)) break; // Reset to NULL if the connection is still waiting so another connection can be attempted openData = NULL; } } // Try or retry a connection since none of the waiting connections completed if (openData == NULL) { // If connection does not exist yet then create it if (addrInfoIdx == lstSize(openDataList)) { openData = lstAdd(openDataList, &(SckClientOpenData){.address = addrInfoGet(addrInfo, addrInfoIdx)->info}); openData->name = strZ(addrInfoToName(this->host, this->port, openData->address)); } // Else search for a connection that is not waiting else { for (; addrInfoIdx < lstSize(openDataList); addrInfoIdx++) { SckClientOpenData *const openDataNoWait = lstGet(openDataList, addrInfoIdx); if (openDataNoWait->fd == 0) { openData = openDataNoWait; break; } } } // Attempt connection to host if (openData != NULL) { openData->fd = socket( openData->address->ai_family, openData->address->ai_socktype, openData->address->ai_protocol); THROW_ON_SYS_ERROR(openData->fd == -1, HostConnectError, "unable to create socket"); sckOptionSet(openData->fd); if (connect(openData->fd, openData->address->ai_addr, openData->address->ai_addrlen) == -1) { // Save the error and wait for connection. The initial wait time will be fixed at 250ms, the default // value recommended by RFC 8305. openData->errNo = errno; sckClientOpenWait(openData, 250); } } } // Connection was successful so open session and set result if (openData != NULL && openData->errNo == 0) { // Create the session MEM_CONTEXT_PRIOR_BEGIN() { result = sckSessionNew(ioSessionRoleClient, openData->fd, this->host, this->port, this->timeoutSession); } MEM_CONTEXT_PRIOR_END(); // Update client name to include address strTrunc(this->name); strCatZ(this->name, openData->name); // Set preferred address addrInfoPrefer(addrInfo, addrInfoIdx); // Clear socket so it will not be freed later openData->fd = 0; } } CATCH_ANY() { // Cast is required to prevent Coverity from complaining about side effects of using volatile in an assertion ASSERT((SckClientOpenData *)openData != NULL); // Close socket close(openData->fd); // Clear socket so the connection can be retried openData->fd = 0; openData->errNo = 0; // Add the error retry info errRetryAddP(errRetry); // Store error info for later logging errLastType = errorType(); errLastMessage = strNewZ(errorMessage()); } TRY_END(); // If the connection was not completed then wait and/or retry if (result == NULL) { // Increment address info index and reset if the end has been reached. When the end has been reached sleep for a bit // to hopefully have better chance at succeeding, otherwise continue right to the next address. addrInfoIdx++; if (addrInfoIdx >= addrInfoSize(addrInfo)) { addrInfoIdx = 0; retry = waitMore(wait); } else retry = true; // General timeout errors when no retries remain if (!retry) { for (unsigned int openDataIdx = 0; openDataIdx < lstSize(openDataList); openDataIdx++) { SckClientOpenData *const openDataTimeout = lstGet(openDataList, openDataIdx); if (openDataTimeout->fd != 0) { errRetryAddP( errRetry, .type = &HostConnectError, .message = strNewFmt("timeout connecting to '%s'", openDataTimeout->name)); } } } // Log retry when there was an error if (errLastType != NULL) { LOG_DEBUG_FMT("retry %s: %s", errorTypeName(errLastType), strZ(errLastMessage)); strFree(errLastMessage); statInc(SOCKET_STAT_RETRY_STR); } } } while (retry); // Free any connections that were in progress but never completed for (unsigned int openDataIdx = 0; openDataIdx < lstSize(openDataList); openDataIdx++) { SckClientOpenData *const openDataFree = lstGet(openDataList, openDataIdx); if (openDataFree->fd != 0) close(openDataFree->fd); } // Error when no result if (result == NULL) THROWP(errRetryType(errRetry), strZ(errRetryMessage(errRetry))); statInc(SOCKET_STAT_SESSION_STR); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(IO_SESSION, result); } /**********************************************************************************************************************************/ static const String * sckClientName(THIS_VOID) { THIS(SocketClient); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_CLIENT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_CONST(STRING, this->name); } /**********************************************************************************************************************************/ static const IoClientInterface sckClientInterface = { .type = IO_CLIENT_SOCKET_TYPE, .name = sckClientName, .open = sckClientOpen, .toLog = sckClientToLog, }; FN_EXTERN IoClient * sckClientNew(const String *const host, const unsigned int port, const TimeMSec timeoutConnect, const TimeMSec timeoutSession) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeoutConnect); FUNCTION_LOG_PARAM(TIME_MSEC, timeoutSession); FUNCTION_LOG_END(); ASSERT(host != NULL); OBJ_NEW_BEGIN(SocketClient, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (SocketClient) { .host = strDup(host), .port = port, .name = strCatFmt(strNew(), "%s:%u", strZ(host), port), .timeoutConnect = timeoutConnect, .timeoutSession = timeoutSession, }; } OBJ_NEW_END(); statInc(SOCKET_STAT_CLIENT_STR); FUNCTION_LOG_RETURN(IO_CLIENT, ioClientNew(this, &sckClientInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/client.h0000664000175100017510000000376015240547630021026 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Client A simple socket client intended to allow access to services that are exposed via a socket. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_CLIENT_H #define COMMON_IO_SOCKET_CLIENT_H #include "common/io/client.h" #include "common/time.h" /*********************************************************************************************************************************** Io client type ***********************************************************************************************************************************/ #define IO_CLIENT_SOCKET_TYPE STRID5("socket", 0x28558df30) /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define SOCKET_STAT_CLIENT "socket.client" // Clients created STRING_DECLARE(SOCKET_STAT_CLIENT_STR); #define SOCKET_STAT_RETRY "socket.retry" // Connection retries STRING_DECLARE(SOCKET_STAT_RETRY_STR); #define SOCKET_STAT_SESSION "socket.session" // Sessions created STRING_DECLARE(SOCKET_STAT_SESSION_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoClient *sckClientNew(const String *host, unsigned int port, TimeMSec timeoutConnect, TimeMSec timeoutSession); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/common.c0000664000175100017510000001234215240547630021027 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Common Functions ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/io/socket/address.h" #include "common/io/socket/common.h" #include "common/log.h" #include "common/wait.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct SocketLocal { bool init; // sckInit() has been called bool block; // Use blocking mode socket bool keepAlive; // Is socket keep-alive enabled? int tcpKeepAliveCount; // TCP keep alive count (0 disables) int tcpKeepAliveIdle; // TCP keep alive idle (0 disables) int tcpKeepAliveInterval; // TCP keep alive interval (0 disables) } socketLocal; /**********************************************************************************************************************************/ FN_EXTERN void sckInit( const bool block, const bool keepAlive, const int tcpKeepAliveCount, const int tcpKeepAliveIdle, const int tcpKeepAliveInterval) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_PARAM(BOOL, keepAlive); FUNCTION_LOG_PARAM(INT, tcpKeepAliveCount); FUNCTION_LOG_PARAM(INT, tcpKeepAliveIdle); FUNCTION_LOG_PARAM(INT, tcpKeepAliveInterval); FUNCTION_LOG_END(); ASSERT(tcpKeepAliveCount >= 0); ASSERT(tcpKeepAliveIdle >= 0); ASSERT(tcpKeepAliveInterval >= 0); socketLocal.init = true; socketLocal.block = block; socketLocal.keepAlive = keepAlive; socketLocal.tcpKeepAliveCount = tcpKeepAliveCount; socketLocal.tcpKeepAliveIdle = tcpKeepAliveIdle; socketLocal.tcpKeepAliveInterval = tcpKeepAliveInterval; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void sckOptionSet(const int fd) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, fd); FUNCTION_TEST_END(); ASSERT(socketLocal.init); // Disable the Nagle algorithm. This means that segments are always sent as soon as possible, even if there is only a small // amount of data. Our internal buffering minimizes the benefit of this optimization so lower latency is preferred. #ifdef TCP_NODELAY int socketValue = 1; THROW_ON_SYS_ERROR( setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set TCP_NODELAY"); #endif // Put the socket in non-blocking mode if (!socketLocal.block) { int flags; THROW_ON_SYS_ERROR((flags = fcntl(fd, F_GETFL)) == -1, ProtocolError, "unable to get flags"); THROW_ON_SYS_ERROR(fcntl(fd, F_SETFL, flags | O_NONBLOCK) == -1, ProtocolError, "unable to set O_NONBLOCK"); } // Automatically close the socket (in the child process) on a successful execve() call. Connections are never shared between // processes so there is no reason to leave them open. #ifdef F_SETFD THROW_ON_SYS_ERROR(fcntl(fd, F_SETFD, FD_CLOEXEC) == -1, ProtocolError, "unable to set FD_CLOEXEC"); #endif // Enable TCP keepalives if (socketLocal.keepAlive) { int socketValue = 1; THROW_ON_SYS_ERROR( setsockopt(fd, SOL_SOCKET, SO_KEEPALIVE, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set SO_KEEPALIVE"); // Set TCP_KEEPCNT when available #ifdef TCP_KEEPIDLE if (socketLocal.tcpKeepAliveCount > 0) { socketValue = socketLocal.tcpKeepAliveCount; THROW_ON_SYS_ERROR( setsockopt(fd, IPPROTO_TCP, TCP_KEEPCNT, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set TCP_KEEPCNT"); } #endif // Set TCP_KEEPIDLE when available #ifdef TCP_KEEPIDLE if (socketLocal.tcpKeepAliveIdle > 0) { socketValue = socketLocal.tcpKeepAliveIdle; THROW_ON_SYS_ERROR( setsockopt(fd, IPPROTO_TCP, TCP_KEEPIDLE, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set SO_KEEPIDLE"); } #endif // Set TCP_KEEPINTVL when available #ifdef TCP_KEEPIDLE if (socketLocal.tcpKeepAliveInterval > 0) { socketValue = socketLocal.tcpKeepAliveInterval; THROW_ON_SYS_ERROR( setsockopt(fd, IPPROTO_TCP, TCP_KEEPINTVL, &socketValue, sizeof(int)) == -1, ProtocolError, "unable to set SO_KEEPINTVL"); } #endif } FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/common.h0000664000175100017510000000173015240547630021033 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Common Functions ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_COMMON_H #define COMMON_IO_SOCKET_COMMON_H #include #include "common/time.h" #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize settings for socket connections (some are used only for TCP) FN_EXTERN void sckInit(bool block, bool keepAlive, int tcpKeepAliveCount, int tcpKeepAliveIdle, int tcpKeepAliveInterval); // Set options on a socket FN_EXTERN void sckOptionSet(int fd); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/server.c0000664000175100017510000002056415240547630021052 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Server ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/io/server.h" #include "common/io/socket/address.h" #include "common/io/socket/common.h" #include "common/io/socket/server.h" #include "common/io/socket/session.h" #include "common/log.h" #include "common/memContext.h" #include "common/stat.h" #include "common/type/object.h" #include "common/wait.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(SOCKET_STAT_SERVER_STR, SOCKET_STAT_SERVER); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct SocketServer { MemContext *memContext; // Mem context String *address; // Address to listen on unsigned int port; // Port to listen on int socket; // Socket used to listen for new connections String *name; // Socket name (address:port) TimeMSec timeout; // Timeout for any i/o operation (connect, read, etc.) } SocketServer; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void sckServerToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const SocketServer); strStcFmt(debugLog, "{address: %s, port: %u, timeout: %" PRIu64 "}", strZ(this->address), this->port, this->timeout); } #define FUNCTION_LOG_SOCKET_SERVER_TYPE \ SocketServer * #define FUNCTION_LOG_SOCKET_SERVER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, sckServerToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free connection ***********************************************************************************************************************************/ static void sckServerFreeResource(THIS_VOID) { THIS(SocketServer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_SERVER, this); FUNCTION_LOG_END(); ASSERT(this != NULL); close(this->socket); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static IoSession * sckServerAccept(THIS_VOID, IoSession *const session) { THIS(SocketServer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_SERVER, this); (void)session; // Not used by this server FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(session == NULL); IoSession *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Accept the socket connection struct sockaddr_in addr; unsigned int len = sizeof(addr); int serverSocket = accept(this->socket, (struct sockaddr *)&addr, &len); if (serverSocket != -1) { // Create socket session sckOptionSet(serverSocket); MEM_CONTEXT_PRIOR_BEGIN() { result = sckSessionNew(ioSessionRoleServer, serverSocket, this->address, this->port, this->timeout); } MEM_CONTEXT_PRIOR_END(); statInc(SOCKET_STAT_SESSION_STR); } else THROW_ON_SYS_ERROR(errno != EINTR, FileOpenError, "unable to accept socket"); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(IO_SESSION, result); } /**********************************************************************************************************************************/ static const String * sckServerName(THIS_VOID) // {vm_covered} { THIS(SocketServer); // {vm_covered} FUNCTION_TEST_BEGIN(); // {vm_covered} FUNCTION_TEST_PARAM(SOCKET_SERVER, this); // {vm_covered} FUNCTION_TEST_END(); // {vm_covered} ASSERT(this != NULL); // {vm_covered} FUNCTION_TEST_RETURN_CONST(STRING, this->name); // {vm_covered} } /**********************************************************************************************************************************/ static const IoServerInterface sckServerInterface = { .type = IO_SERVER_SOCKET_TYPE, .name = sckServerName, .accept = sckServerAccept, .toLog = sckServerToLog, }; FN_EXTERN IoServer * sckServerNew(const String *const address, const unsigned int port, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, address); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(address != NULL); ASSERT(port > 0); OBJ_NEW_BEGIN(SocketServer, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (SocketServer) { .address = strDup(address), .port = port, .name = strCatFmt(strNew(), "%s:%u", strZ(address), port), .timeout = timeout, }; // Lookup address const struct addrinfo *const addressFound = addrInfoGet(addrInfoNew(this->address, this->port), 0)->info; // Create socket THROW_ON_SYS_ERROR( (this->socket = socket(addressFound->ai_family, SOCK_STREAM, 0)) == -1, FileOpenError, "unable to create socket"); // Set the address as reusable so we can bind again quickly after a restart or crash int reuseAddr = 1; THROW_ON_SYS_ERROR( setsockopt(this->socket, SOL_SOCKET, SO_REUSEADDR, &reuseAddr, sizeof(reuseAddr)) == -1, ProtocolError, "unable to set SO_REUSEADDR"); // Ensure file descriptor is closed memContextCallbackSet(objMemContext(this), sckServerFreeResource, this); // Update server name to include address strTrunc(this->name); strCat(this->name, addrInfoToName(this->address, this->port, addressFound)); // Bind the address Wait *const wait = waitNew(timeout); int result; do { result = bind(this->socket, addressFound->ai_addr, addressFound->ai_addrlen); if (result == -1 && !waitMore(wait)) THROW_SYS_ERROR(FileOpenError, "unable to bind socket"); } while (result == -1); // Listen for client connections. It might be a good idea to make the backlog configurable but this value seems OK for now. THROW_ON_SYS_ERROR(listen(this->socket, 100) == -1, FileOpenError, "unable to listen on socket"); } OBJ_NEW_END(); statInc(SOCKET_STAT_SERVER_STR); FUNCTION_LOG_RETURN(IO_SERVER, ioServerNew(this, &sckServerInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/server.h0000664000175100017510000000325515240547630021055 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Server A simple socket server intended to expose services by accepting a socket. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_SERVER_H #define COMMON_IO_SOCKET_SERVER_H #include "common/io/server.h" #include "common/io/socket/client.h" #include "common/time.h" /*********************************************************************************************************************************** Io server type ***********************************************************************************************************************************/ #define IO_SERVER_SOCKET_TYPE IO_CLIENT_SOCKET_TYPE /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define SOCKET_STAT_SERVER "socket.server" // Servers created STRING_DECLARE(SOCKET_STAT_SERVER_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoServer *sckServerNew(const String *address, unsigned int port, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/session.c0000664000175100017510000001547515240547630021234 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Session ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/fdRead.h" #include "common/io/fdWrite.h" #include "common/io/session.h" #include "common/io/socket/client.h" #include "common/io/socket/session.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct SocketSession { IoSessionRole role; // Role (server or client) int fd; // File descriptor String *host; // Hostname or IP address unsigned int port; // Port to connect to host on TimeMSec timeout; // Timeout for any i/o operation (read, write, etc.) IoRead *read; // IoRead interface to the file descriptor IoWrite *write; // IoWrite interface to the file descriptor } SocketSession; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void sckSessionToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const SocketSession); strStcFmt( debugLog, "{host: %s, port: %u, fd: %d, timeout: %" PRIu64 "}", strZ(this->host), this->port, this->fd, this->timeout); } #define FUNCTION_LOG_SOCKET_SESSION_TYPE \ SocketSession * #define FUNCTION_LOG_SOCKET_SESSION_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, sckSessionToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free connection ***********************************************************************************************************************************/ static void sckSessionFreeResource(THIS_VOID) { THIS(SocketSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); close(this->fd); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void sckSessionClose(THIS_VOID) { THIS(SocketSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(SOCKET_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If not already closed if (this->fd != -1) { // Clear the callback to close the socket memContextCallbackClear(objMemContext(this)); // Close the socket close(this->fd); this->fd = -1; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static int sckSessionFd(THIS_VOID) { THIS(SocketSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ static IoRead * sckSessionIoRead(THIS_VOID, const bool ignoreUnexpectedEof) { THIS(SocketSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_SESSION, this); (void)ignoreUnexpectedEof; // Unused FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_READ, this->read); } /**********************************************************************************************************************************/ static IoWrite * sckSessionIoWrite(THIS_VOID) { THIS(SocketSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_WRITE, this->write); } /**********************************************************************************************************************************/ static IoSessionRole sckSessionRole(const THIS_VOID) { THIS(const SocketSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SOCKET_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING_ID, this->role); } /**********************************************************************************************************************************/ static const IoSessionInterface sckSessionInterface = { .type = IO_CLIENT_SOCKET_TYPE, .close = sckSessionClose, .fd = sckSessionFd, .ioRead = sckSessionIoRead, .ioWrite = sckSessionIoWrite, .role = sckSessionRole, .toLog = sckSessionToLog, }; FN_EXTERN IoSession * sckSessionNew(const IoSessionRole role, const int fd, const String *const host, const unsigned int port, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, role); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(fd != -1); ASSERT(host != NULL); OBJ_NEW_BEGIN(SocketSession, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { String *const name = strNewFmt("%s:%u", strZ(host), port); *this = (SocketSession) { .role = role, .fd = fd, .host = strDup(host), .port = port, .timeout = timeout, .read = ioFdReadNewOpen(name, fd, timeout), .write = ioFdWriteNewOpen(name, fd, timeout), }; strFree(name); // Ensure file descriptor is closed memContextCallbackSet(objMemContext(this), sckSessionFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_SESSION, ioSessionNew(this, &sckSessionInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/socket/session.h0000664000175100017510000000161615240547630021231 0ustar00runnerrunner/*********************************************************************************************************************************** Socket Session A simple socket session intended to allow access to services that are exposed via a socket. ***********************************************************************************************************************************/ #ifndef COMMON_IO_SOCKET_SESSION_H #define COMMON_IO_SOCKET_SESSION_H #include "common/io/session.h" #include "common/time.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoSession *sckSessionNew(IoSessionRole role, int fd, const String *host, unsigned int port, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/tls/0000775000175100017510000000000015240547630016703 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/tls/client.c0000664000175100017510000005267115240547630020340 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Client ***********************************************************************************************************************************/ #include // {uncrustify_off - header order required for FreeBSD} #include #include #include #include // {uncrustify_on} #include #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/client.h" #include "common/io/io.h" #include "common/io/tls/client.h" #include "common/io/tls/common.h" #include "common/io/tls/session.h" #include "common/log.h" #include "common/stat.h" #include "common/type/object.h" #include "common/wait.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(TLS_STAT_CLIENT_STR, TLS_STAT_CLIENT); STRING_EXTERN(TLS_STAT_RETRY_STR, TLS_STAT_RETRY); STRING_EXTERN(TLS_STAT_SESSION_STR, TLS_STAT_SESSION); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct TlsClient { const String *host; // Host to use for peer verification TimeMSec timeoutConnect; // Timeout for connection TimeMSec timeoutSession; // Timeout passed to session bool verifyPeer; // Should the peer (server) certificate be verified? IoClient *ioClient; // Underlying client (usually a SocketClient) SSL_CTX *context; // TLS context } TlsClient; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void tlsClientToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const TlsClient); strStcCat(debugLog, "{ioClient: "); ioClientToLog(this->ioClient, debugLog); strStcFmt( debugLog, ", timeoutConnect: %" PRIu64 ", timeoutSession: %" PRIu64 ", verifyPeer: %s}", this->timeoutConnect, this->timeoutSession, cvtBoolToConstZ(this->verifyPeer)); } #define FUNCTION_LOG_TLS_CLIENT_TYPE \ TlsClient * #define FUNCTION_LOG_TLS_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, tlsClientToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free connection ***********************************************************************************************************************************/ static void tlsClientFreeResource(THIS_VOID) { THIS(TlsClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); SSL_CTX_free(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Check if a name from the server certificate matches the hostname Matching is always case-insensitive since DNS is case insensitive. ***********************************************************************************************************************************/ static bool tlsClientHostVerifyName(const String *const host, const String *const name) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_END(); ASSERT(host != NULL); ASSERT(name != NULL); // Check for NULLs in the name tlsCertNameVerify(name); bool result = false; // Try an exact match if (strcasecmp(strZ(name), strZ(host)) == 0) // {vm_covered} { result = true; // {vm_covered} } // Else check if a wildcard certificate matches the host name // // The rules are: // 1. Only match the '*' character as wildcard // 2. Only match wildcards at the start of the string // 3. The '*' character does *not* match '.', meaning that we match only a single pathname component. // 4. Don't support more than one '*' in a single pattern. // // This is roughly in line with RFC2818, but contrary to what most browsers appear to be implementing (point 3 being the // difference) else if (strZ(name)[0] == '*' && strZ(name)[1] == '.' && strSize(name) > 2 && // {vm_covered} strSize(name) < strSize(host) && // {vm_covered} strcasecmp(strZ(name) + 1, strZ(host) + strSize(host) - strSize(name) + 1) == 0 && // {vm_covered} strChr(host, '.') >= (int)(strSize(host) - strSize(name))) // {vm_covered} { result = true; // {vm_covered} } FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Check if an IP address from the server certificate matches the hostname Adapted from libpq/fe-secure-common.c. ***********************************************************************************************************************************/ static bool tlsClientHostVerifyIPAddr(const String *const host, const Buffer *const address) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(BUFFER, address); FUNCTION_LOG_END(); ASSERT(host != NULL); ASSERT(address != NULL); bool result = false; // The data from the certificate is in network byte order. Convert our host string to network-ordered bytes as well for // comparison. The host string is not guaranteed to be an IP address so if this conversion fails consider it a mismatch rather // than an error. Use the size of the address parameter to determine ipv4 or ipv6 checking. const size_t addrSize = bufSize(address); // IPv4 if (addrSize == sizeof(struct in_addr)) { struct in_addr addr; if (inet_pton(AF_INET, strZ(host), &addr) == 1 && // {vm_covered} memcmp(bufPtrConst(address), &addr.s_addr, addrSize) == 0) // {vm_covered} { result = true; // {vm_covered} } } // Else IPv6 else if (addrSize == sizeof(struct in6_addr)) { struct in6_addr addr; if (inet_pton(AF_INET6, strZ(host), &addr) == 1 && // {vm_covered} memcmp(bufPtrConst(address), &addr.s6_addr, addrSize) == 0) // {vm_covered} { result = true; // {vm_covered} } } FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Verify that the server certificate matches the hostname we connected to The certificate's Common Name and Subject Alternative Names are considered. ***********************************************************************************************************************************/ static bool tlsClientHostVerify(const String *const host, X509 *const certificate) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM_P(VOID, certificate); FUNCTION_LOG_END(); ASSERT(host != NULL); bool result = false; // Error if the certificate is NULL if (certificate == NULL) // {vm_covered} THROW(CryptoError, "No certificate presented by the TLS server"); // {vm_covered} MEM_CONTEXT_TEMP_BEGIN() // {vm_covered} { // First get the subject alternative names from the certificate and compare them against the hostname STACK_OF(GENERAL_NAME) *const altNameStack = (STACK_OF(GENERAL_NAME) *)X509_get_ext_d2i( // {vm_covered} certificate, NID_subject_alt_name, NULL, NULL); // {vm_covered} bool altNameFound = false; // {vm_covered} if (altNameStack) // {vm_covered} { for (int altNameIdx = 0; altNameIdx < sk_GENERAL_NAME_num(altNameStack); altNameIdx++) // {vm_covered} { const GENERAL_NAME *const name = sk_GENERAL_NAME_value(altNameStack, altNameIdx); // {vm_covered} altNameFound = true; // {vm_covered} if (name->type == GEN_DNS) // {vm_covered} result = tlsClientHostVerifyName(host, tlsAsn1ToStr(name->d.dNSName)); // {vm_covered} else if (name->type == GEN_IPADD) // {vm_covered} result = tlsClientHostVerifyIPAddr(host, tlsAsn1ToBuf(name->d.dNSName)); // {vm_covered} if (result != false) // {vm_covered} break; // {vm_covered} } sk_GENERAL_NAME_pop_free(altNameStack, GENERAL_NAME_free); // {vm_covered} } // If no subject alternative name was found then check the common name. Per RFC 2818 and RFC 6125, if the subjectAltName // extension of type dNSName is present the CN must be ignored. if (!altNameFound) // {vm_covered} result = tlsClientHostVerifyName(host, tlsCertCommonName(certificate)); // {vm_covered} } MEM_CONTEXT_TEMP_END(); // {vm_covered} FUNCTION_LOG_RETURN(BOOL, result); // {vm_covered} } /*********************************************************************************************************************************** Authenticate server Adapted from PostgreSQL open_client_SSL() in src/interfaces/libpq/fe-secure-openssl.c. ***********************************************************************************************************************************/ static bool tlsClientAuth(const TlsClient *const this, SSL *const tlsSession) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_CLIENT, this); FUNCTION_LOG_PARAM_P(VOID, tlsSession); FUNCTION_LOG_END(); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // Verify that the certificate presented by the server is valid if (this->verifyPeer) // {vm_covered} { // Verify that the chain of trust leads to a valid CA const long int verifyResult = SSL_get_verify_result(tlsSession); // {vm_covered} if (verifyResult != X509_V_OK) // {vm_covered} { THROW_FMT( // {vm_covered} CryptoError, "unable to verify certificate presented by '%s': [%ld] %s", // {vm_covered} strZ(ioClientName(this->ioClient)), verifyResult, // {vm_covered} X509_verify_cert_error_string(verifyResult)); // {vm_covered} } // Verify that the hostname appears in the certificate X509 *const certificate = SSL_get_peer_certificate(tlsSession); // {vm_covered} const bool nameResult = tlsClientHostVerify(this->host, certificate); // {vm_covered} X509_free(certificate); // {vm_covered} if (!nameResult) // {vm_covered} { THROW_FMT( // {vm_covered} CryptoError, // {vm_covered} "unable to find hostname '%s' in certificate common name or subject alternative names", // {vm_covered} strZ(this->host)); // {vm_covered} } } result = true; } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Open TLS session on a socket ***********************************************************************************************************************************/ static IoSession * tlsClientOpen(THIS_VOID) { THIS(TlsClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); IoSession *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { bool retry; Wait *const wait = waitNew(this->timeoutConnect); SSL *tlsSession = NULL; do { // Assume there will be no retry retry = false; // Create the TLS session tlsSession = SSL_new(this->context); cryptoError(tlsSession == NULL, "unable to create TLS session"); // Set server host name used for validation. The exception here is necessary for MacOS which for some reason defines the // host name parameter as void * rather than const char * as on most platforms. #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wcast-qual" cryptoError(SSL_set_tlsext_host_name(tlsSession, strZ(this->host)) != 1, "unable to set TLS host name"); #pragma GCC diagnostic pop // Open the underlying session first since this is mostly likely to fail. This is done outside exception handling to // avoid multiplying retries. IoSession *ioSession = NULL; TRY_BEGIN() { ioSession = ioClientOpen(this->ioClient); } CATCH_ANY() { SSL_free(tlsSession); RETHROW(); } TRY_END(); // Open TLS session TRY_BEGIN() { result = tlsSessionNew(tlsSession, ioSession, this->timeoutSession); } CATCH_ANY() { result = NULL; // Retry if wait time has not expired if (waitMore(wait)) { LOG_DEBUG_FMT("retry %s: %s", errorTypeName(errorType()), errorMessage()); retry = true; statInc(TLS_STAT_RETRY_STR); } else RETHROW(); } TRY_END(); } while (retry); // Authenticate TLS session ASSERT(result != NULL); ioSessionAuthenticatedSet(result, tlsClientAuth(this, tlsSession)); // Move session ioSessionMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); statInc(TLS_STAT_SESSION_STR); FUNCTION_LOG_RETURN(IO_SESSION, result); } /**********************************************************************************************************************************/ static const String * tlsClientName(THIS_VOID) { THIS(TlsClient); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TLS_CLIENT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_CONST(STRING, ioClientName(this->ioClient)); } /*********************************************************************************************************************************** Initialize TLS session with all required security features Adapted from PostgreSQL initialize_SSL() in src/interfaces/libpq/fe-secure-openssl.c. ***********************************************************************************************************************************/ static const IoClientInterface tlsClientInterface = { .type = IO_CLIENT_TLS_TYPE, .name = tlsClientName, .open = tlsClientOpen, .toLog = tlsClientToLog, }; FN_EXTERN IoClient * tlsClientNew( IoClient *const ioClient, const String *const host, const TimeMSec timeoutConnect, const TimeMSec timeoutSession, const bool verifyPeer, const TlsClientNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_CLIENT, ioClient); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(TIME_MSEC, timeoutConnect); FUNCTION_LOG_PARAM(TIME_MSEC, timeoutSession); FUNCTION_LOG_PARAM(BOOL, verifyPeer); FUNCTION_LOG_PARAM(STRING, param.caFile); FUNCTION_LOG_PARAM(STRING, param.caPath); FUNCTION_LOG_PARAM(STRING, param.certFile); FUNCTION_LOG_PARAM(STRING, param.keyFile); FUNCTION_LOG_END(); ASSERT(ioClient != NULL); OBJ_NEW_BEGIN(TlsClient, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (TlsClient) { .ioClient = ioClientMove(ioClient, objMemContext(this)), .host = strDup(host), .timeoutConnect = timeoutConnect, .timeoutSession = timeoutSession, .verifyPeer = verifyPeer, .context = tlsContext(verifyPeer), }; // Set callback to free context memContextCallbackSet(objMemContext(this), tlsClientFreeResource, this); // Enable safe compatibility options SSL_CTX_set_options(this->context, SSL_OP_ALL); // Set location of CA certificates if the server certificate will be verified if (this->verifyPeer) { // If the user specified a location if (param.caFile != NULL || param.caPath != NULL) // {vm_covered} { cryptoError( // {vm_covered} SSL_CTX_load_verify_locations( // {vm_covered} this->context, strZNull(param.caFile), strZNull(param.caPath)) != 1, // {vm_covered} "unable to set user-defined CA certificate location"); // {vm_covered} } // Else use the defaults else { cryptoError( SSL_CTX_set_default_verify_paths(this->context) != 1, "unable to set default CA certificate location"); } } // Load certificate and key, if specified tlsCertKeyLoad(this->context, param.certFile, param.keyFile); } OBJ_NEW_END(); statInc(TLS_STAT_CLIENT_STR); FUNCTION_LOG_RETURN(IO_CLIENT, ioClientNew(this, &tlsClientInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/tls/client.h0000664000175100017510000000577615240547630020351 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Client A simple, secure TLS client intended to allow access to services that are exposed via HTTPS. We call it TLS instead of SSL because SSL methods are disabled so only TLS connections are allowed. This object is intended to be used for multiple TLS sessions so ioClientOpen() can be called each time a new session is needed. ***********************************************************************************************************************************/ #ifndef COMMON_IO_TLS_CLIENT_H #define COMMON_IO_TLS_CLIENT_H #include "common/io/client.h" #include "common/time.h" /*********************************************************************************************************************************** Io client type ***********************************************************************************************************************************/ #define IO_CLIENT_TLS_TYPE STRID5("tls", 0x4d940) /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define TLS_STAT_CLIENT "tls.client" // Clients created STRING_DECLARE(TLS_STAT_CLIENT_STR); #define TLS_STAT_RETRY "tls.retry" // Connection retries STRING_DECLARE(TLS_STAT_RETRY_STR); #define TLS_STAT_SESSION "tls.session" // Sessions created STRING_DECLARE(TLS_STAT_SESSION_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct TlsClientNewParam { VAR_PARAM_HEADER; const String *caFile; const String *caPath; const String *certFile; const String *keyFile; } TlsClientNewParam; #define tlsClientNewP(ioClient, host, timeoutConnect, timeoutSession, verifyPeer, ...) \ tlsClientNew(ioClient, host, timeoutConnect, timeoutSession, verifyPeer, (TlsClientNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN IoClient *tlsClientNew( IoClient *ioClient, const String *host, TimeMSec timeoutConnect, TimeMSec timeoutSession, bool verifyPeer, TlsClientNewParam param); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Statistics as a formatted string String *tlsClientStatStr(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/tls/common.c0000664000175100017510000002540515240547630020345 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Common ***********************************************************************************************************************************/ #include #include #include #include #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/tls/common.h" #include "common/user.h" #include "storage/posix/storage.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct TlsCommonLocal { MemContext *memContext; // Mem context for TLS common String *tlsCipher12; // Allowed ciphers for TLSv1.2 String *tlsCipher13; // Allowed ciphers for TLSv1.3 } tlsCommonLocal; /**********************************************************************************************************************************/ FN_EXTERN Buffer * tlsAsn1ToBuf(const ASN1_STRING *const nameAsn1) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, nameAsn1); FUNCTION_TEST_END(); // The name should not be null if (nameAsn1 == NULL) // {vm_covered} THROW(CryptoError, "TLS certificate name entry is missing"); FUNCTION_TEST_RETURN( // {vm_covered} BUFFER, bufNewC( (const char *)ASN1_STRING_get0_data(nameAsn1), (size_t)ASN1_STRING_length(nameAsn1))); } /**********************************************************************************************************************************/ FN_EXTERN String * tlsAsn1ToStr(const ASN1_STRING *const nameAsn1) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, nameAsn1); FUNCTION_TEST_END(); // The name should not be null if (nameAsn1 == NULL) // {vm_covered} THROW(CryptoError, "TLS certificate name entry is missing"); FUNCTION_TEST_RETURN( // {vm_covered} STRING, strNewZN( (const char *)ASN1_STRING_get0_data(nameAsn1), (size_t)ASN1_STRING_length(nameAsn1))); } /**********************************************************************************************************************************/ FN_EXTERN void tlsCertNameVerify(const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); // Check for embedded NULLs if (strlen(strZ(name)) != strSize(name)) THROW(CryptoError, "TLS certificate name contains embedded null"); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * tlsCertCommonName(X509 *const certificate) // {vm_covered} { FUNCTION_TEST_BEGIN(); // {vm_covered} FUNCTION_TEST_PARAM_P(VOID, certificate); // {vm_covered} FUNCTION_TEST_END(); // {vm_covered} X509_NAME *const subjectName = X509_get_subject_name(certificate); // {vm_covered} CHECK(FormatError, subjectName != NULL, "subject name is missing"); // {vm_covered} const int commonNameIndex = X509_NAME_get_index_by_NID(subjectName, NID_commonName, -1); // {vm_covered} CHECK(FormatError, commonNameIndex >= 0, "common name is missing"); // {vm_covered} String *result = tlsAsn1ToStr(X509_NAME_ENTRY_get_data(X509_NAME_get_entry(subjectName, commonNameIndex))); // {vm_covered} // Check for NULLs in the name tlsCertNameVerify(result); // {vm_covered} FUNCTION_TEST_RETURN(STRING, result); // {vm_covered} } /**********************************************************************************************************************************/ // Callback to process cert passwords static int tlsCertPwd(char *const buffer, const int size, const int rwFlag, void *const userData) { (void)rwFlag; (void)userData; CHECK(ServiceError, size > 0, "buffer has zero size"); // No password is currently supplied buffer[0] = '\0'; return 0; } FN_EXTERN void tlsCertKeyLoad(SSL_CTX *const context, const String *const certFile, const String *const keyFile) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, context); FUNCTION_TEST_PARAM(STRING, certFile); FUNCTION_TEST_PARAM(STRING, keyFile); FUNCTION_TEST_END(); ASSERT(context != NULL); ASSERT((certFile == NULL && keyFile == NULL) || (certFile != NULL && keyFile != NULL)); if (certFile != NULL) { userInit(); MEM_CONTEXT_TEMP_BEGIN() { // Set cert password callback SSL_CTX_set_default_passwd_cb(context, tlsCertPwd); // Load certificate cryptoError( SSL_CTX_use_certificate_chain_file(context, strZ(certFile)) != 1, zNewFmt("unable to load cert file '%s'", strZ(certFile))); // Check that key has the correct permissions const StorageInfo keyInfo = storageInfoP( storagePosixNewP(FSLASH_STR), keyFile, .ignoreMissing = true, .followLink = true); if (keyInfo.exists) { if (keyInfo.userId != userId() && keyInfo.userId != 0) // {vm_covered} { THROW_FMT( // {vm_covered} FileReadError, "key file '%s' must be owned by the '%s' user or root", strZ(keyFile), strZ(userName())); } if ((keyInfo.userId == userId() && keyInfo.mode & (S_IRWXG | S_IRWXO)) || // {vm_covered} (keyInfo.userId == 0 && keyInfo.mode & (S_IWGRP | S_IXGRP | S_IRWXO))) // {vm_covered} { THROW_FMT( FileReadError, "key file '%s' has group or other permissions\n" "HINT: file must have permissions u=rw (0600) or less if owned by the '%s' user\n" "HINT: file must have permissions u=rw, g=r (0640) or less if owned by root\n", strZ(keyFile), strZ(userName())); } } // Load key and verify that the key and cert go together cryptoError( SSL_CTX_use_PrivateKey_file(context, strZ(keyFile), SSL_FILETYPE_PEM) != 1, zNewFmt("unable to load key file '%s'", strZ(keyFile))); // Verify again that the cert and key go together. It is not clear why this is needed since the key has already been // verified in SSL_CTX_use_PrivateKey_file(), but it may be that older versions of OpenSSL need it. cryptoError( SSL_CTX_check_private_key(context) != 1, zNewFmt("cert '%s' and key '%s' do not match", strZ(certFile), strZ(keyFile))); } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void tlsInit(const String *const tlsCipher12, const String *const tlsCipher13) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, tlsCipher12); FUNCTION_TEST_PARAM(STRING, tlsCipher13); FUNCTION_TEST_END(); // Free old mem context (if any) if (tlsCommonLocal.memContext != NULL) memContextFree(tlsCommonLocal.memContext); // Initialize mem context MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(TlsCommon, .childQty = MEM_CONTEXT_QTY_MAX) { tlsCommonLocal.memContext = MEM_CONTEXT_NEW(); tlsCommonLocal.tlsCipher12 = strDup(tlsCipher12); tlsCommonLocal.tlsCipher13 = strDup(tlsCipher13); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN SSL_CTX * tlsContext(const bool verifyPeer) { FUNCTION_TEST_VOID(); cryptoInit(); // Select the TLS method to use. To maintain compatibility with older versions of OpenSSL we need to use an SSL method, but // SSL versions will be excluded in SSL_CTX_set_options(). const SSL_METHOD *const method = SSLv23_method(); cryptoError(method == NULL, "unable to load TLS method"); // Create TLS context SSL_CTX *const result = SSL_CTX_new(method); cryptoError(result == NULL, "unable to create TLS context"); // Disable compression SSL_CTX_set_options(result, SSL_OP_NO_COMPRESSION); // Disable auto-retry to prevent SSL_read() from hanging SSL_CTX_clear_mode(result, SSL_MODE_AUTO_RETRY); // Enhance security when verifyPeer is enabled if (verifyPeer) { // Set minimum TLS version to 1.2 cryptoError(SSL_CTX_set_min_proto_version(result, TLS1_2_VERSION) != 1, "unable to set minumum TLS version to 1.2"); // Set allowed ciphers for TLSv1.2 when configured if (tlsCommonLocal.tlsCipher12 != NULL) cryptoError(SSL_CTX_set_cipher_list(result, strZ(tlsCommonLocal.tlsCipher12)) != 1, "unable to set TLSv1.2 ciphers"); // Set allowed ciphers for TLSv1.3 when configured if (tlsCommonLocal.tlsCipher13 != NULL) cryptoError(SSL_CTX_set_ciphersuites(result, strZ(tlsCommonLocal.tlsCipher13)) != 1, "unable to set TLSv1.3 ciphers"); } FUNCTION_TEST_RETURN_TYPE_P(SSL_CTX, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/tls/common.h0000664000175100017510000000276215240547630020353 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Common ***********************************************************************************************************************************/ #ifndef COMMON_IO_TLS_COMMON_H #define COMMON_IO_TLS_COMMON_H #include #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize TLS with allowed ciphers FN_EXTERN void tlsInit(const String *tlsCipher12, const String *tlsCipher13); // Convert an ASN1 string used in certificates to a Buffer FN_EXTERN Buffer *tlsAsn1ToBuf(const ASN1_STRING *nameAsn1); // Convert an ASN1 string used in certificates to a String FN_EXTERN String *tlsAsn1ToStr(const ASN1_STRING *nameAsn1); // Get common name from a certificate FN_EXTERN String *tlsCertCommonName(X509 *certificate); // Reject embedded nulls in certificate common or alternative name to prevent attacks like CVE-2009-4034 FN_EXTERN void tlsCertNameVerify(const String *name); // Load certificate and key and check that they match FN_EXTERN void tlsCertKeyLoad(SSL_CTX *context, const String *certFile, const String *keyFile); // Create TLS context FN_EXTERN SSL_CTX *tlsContext(bool verifyPeer); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/tls/server.c0000664000175100017510000004000115240547630020350 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Server ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/server.h" #include "common/io/tls/common.h" #include "common/io/tls/server.h" #include "common/io/tls/session.h" #include "common/log.h" #include "common/stat.h" #include "common/type/object.h" /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_EXTERN(TLS_STAT_SERVER_STR, TLS_STAT_SERVER); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct TlsServer { String *host; // Host SSL_CTX *context; // TLS context TimeMSec timeout; // Timeout for any i/o operation (connect, read, etc.) bool verifyPeer; // Will the client cert be verified? } TlsServer; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void tlsServerToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const TlsServer); strStcFmt(debugLog, "{host: %s, timeout: %" PRIu64 "}", strZ(this->host), this->timeout); } #define FUNCTION_LOG_TLS_SERVER_TYPE \ TlsServer * #define FUNCTION_LOG_TLS_SERVER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, tlsServerToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free context ***********************************************************************************************************************************/ static void tlsServerFreeResource(THIS_VOID) { THIS(TlsServer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SERVER, this); FUNCTION_LOG_END(); ASSERT(this != NULL); SSL_CTX_free(this->context); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Set DH parameters for generating ephemeral DH keys. The DH parameters can take a long time to compute, so they must be precomputed using parameters provided by the OpenSSL project. These values can be static since the OpenSSL library can efficiently generate random keys from the information provided. Adapted from PostgreSQL initialize_dh() in src/backend/libpq/be-secure-openssl.c. Also see https://weakdh.org and https://en.wikipedia.org/wiki/Logjam_(computer_security). ***********************************************************************************************************************************/ // Hardcoded DH parameters, used in ephemeral DH keying. This is the 2048-bit DH parameter from RFC 3526. The generation of the // prime is specified in RFC 2412 Appendix E, which also discusses the design choice of the generator. Note that when loaded with // OpenSSL this causes DH_check() to fail on DH_NOT_SUITABLE_GENERATOR, where leaking a bit is preferred. // {uncrustify_off - comment inside string} #define DH_2048 \ "-----BEGIN DH PARAMETERS-----\n" \ "MIIBCAKCAQEA///////////JD9qiIWjCNMTGYouA3BzRKQJOCIpnzHQCC76mOxOb\n" \ "IlFKCHmONATd75UZs806QxswKwpt8l8UN0/hNW1tUcJF5IW1dmJefsb0TELppjft\n" \ "awv/XLb0Brft7jhr+1qJn6WunyQRfEsf5kkoZlHs5Fs9wgB8uKFjvwWY2kg2HFXT\n" \ "mmkWP6j9JM9fg2VdI9yjrZYcYvNWIIVSu57VKQdwlpZtZww1Tkq8mATxdGwIyhgh\n" \ "fDKQXkYuNs474553LBgOhgObJ4Oi7Aeij7XFXfBvTFLJ3ivL9pVYFxg5lUl86pVq\n" \ "5RXSJhiY+gUQFXKOWoqsqmj//////////wIBAg==\n" \ "-----END DH PARAMETERS-----" // {uncrustify_on} static void tlsServerDh(SSL_CTX *const context) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, context); FUNCTION_TEST_END(); SSL_CTX_set_options(context, SSL_OP_SINGLE_DH_USE); BIO *const bio = BIO_new_mem_buf(DH_2048, sizeof(DH_2048)); cryptoError(bio == NULL, "unable create buffer for DH parameters"); TRY_BEGIN() { #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wdeprecated-declarations" DH *const dh = PEM_read_bio_DHparams(bio, NULL, NULL, NULL); #pragma GCC diagnostic pop TRY_BEGIN() { cryptoError(SSL_CTX_set_tmp_dh(context, dh) != 1, "unable to set temp dh parameters"); } FINALLY() { #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wdeprecated-declarations" DH_free(dh); #pragma GCC diagnostic pop } TRY_END(); } FINALLY() { BIO_free(bio); } TRY_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set ECDH parameters for generating ephemeral Elliptic Curve DH keys. Adapted from PostgreSQL initialize_ecdh() in src/backend/libpq/be-secure-openssl.c. ***********************************************************************************************************************************/ #define ECHD_CURVE "prime256v1" static void tlsServerEcdh(SSL_CTX *const context) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, context); FUNCTION_TEST_END(); const int nid = OBJ_sn2nid(ECHD_CURVE); cryptoError(nid == NID_undef, "unrecognized ECDH curve " ECHD_CURVE); #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wdeprecated-declarations" EC_KEY *const ecdh = EC_KEY_new_by_curve_name(nid); #pragma GCC diagnostic pop cryptoError(ecdh == NULL, "could not create ecdh key"); SSL_CTX_set_options(context, SSL_OP_SINGLE_ECDH_USE); TRY_BEGIN() { cryptoError(SSL_CTX_set_tmp_ecdh(context, ecdh) != 1, "unable to set temp ecdh key"); } FINALLY() { #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wdeprecated-declarations" EC_KEY_free(ecdh); #pragma GCC diagnostic pop } TRY_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Authenticate client Adapted from PostgreSQL be_tls_open_server() in src/backend/libpq/be-secure-openssl.c. ***********************************************************************************************************************************/ static void tlsServerAuth(const TlsServer *const this, IoSession *const ioSession, SSL *const tlsSession) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SERVER, this); FUNCTION_LOG_PARAM(IO_SESSION, ioSession); FUNCTION_LOG_PARAM_P(VOID, tlsSession); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // If peer verification requested if (this->verifyPeer) // {vm_covered} { // If the client cert was presented then the session is authenticated. An error will be thrown automatically if the // client cert is not valid. X509 *const clientCert = SSL_get_peer_certificate(tlsSession); // {vm_covered} ioSessionAuthenticatedSet(ioSession, clientCert != NULL); // {vm_covered} // Set the peer name to the client cert common name if (clientCert != NULL) // {vm_covered} ioSessionPeerNameSet(ioSession, tlsCertCommonName(clientCert)); // {vm_covered} // Free the cert X509_free(clientCert); // {vm_covered} } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static IoSession * tlsServerAccept(THIS_VOID, IoSession *const ioSession) { THIS(TlsServer); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SERVER, this); FUNCTION_LOG_PARAM(IO_SESSION, ioSession); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(ioSession != NULL); IoSession *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Open TLS session SSL *tlsSession = SSL_new(this->context); result = tlsSessionNew(tlsSession, ioSession, this->timeout); // Authenticate TLS session tlsServerAuth(this, result, tlsSession); // Move session ioSessionMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); statInc(TLS_STAT_SESSION_STR); FUNCTION_LOG_RETURN(IO_SESSION, result); } /**********************************************************************************************************************************/ static const String * tlsServerName(THIS_VOID) // {vm_covered} { THIS(TlsServer); // {vm_covered} FUNCTION_TEST_BEGIN(); // {vm_covered} FUNCTION_TEST_PARAM(TLS_SERVER, this); // {vm_covered} FUNCTION_TEST_END(); // {vm_covered} ASSERT(this != NULL); // {vm_covered} FUNCTION_TEST_RETURN_CONST(STRING, this->host); // {vm_covered} } /*********************************************************************************************************************************** Initialize TLS context with all required security features Adapted from PostgreSQL be_tls_init() in src/backend/libpq/be-secure-openssl.c. ***********************************************************************************************************************************/ static const IoServerInterface tlsServerInterface = { .type = IO_SERVER_TLS_TYPE, .name = tlsServerName, .accept = tlsServerAccept, .toLog = tlsServerToLog, }; FN_EXTERN IoServer * tlsServerNew( const String *const host, const String *const caFile, const String *const keyFile, const String *const certFile, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, keyFile); FUNCTION_LOG_PARAM(STRING, certFile); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(host != NULL); ASSERT(keyFile != NULL); ASSERT(certFile != NULL); OBJ_NEW_BEGIN(TlsServer, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (TlsServer) { .host = strDup(host), .context = tlsContext(true), .timeout = timeout, }; // Set callback to free context memContextCallbackSet(objMemContext(this), tlsServerFreeResource, this); // Set options SSL_CTX_set_options( this->context, // Let server set cipher order SSL_OP_CIPHER_SERVER_PREFERENCE | #ifdef SSL_OP_NO_RENEGOTIATION // Disable renegotiation, available since 1.1.0h. This affects only TLSv1.2 and older protocol versions as TLSv1.3 has // no support for renegotiation. SSL_OP_NO_RENEGOTIATION | #endif // Disable session tickets SSL_OP_NO_TICKET); // Disable session caching SSL_CTX_set_session_cache_mode(this->context, SSL_SESS_CACHE_OFF); // Setup ephemeral DH and ECDH keys tlsServerDh(this->context); tlsServerEcdh(this->context); // Load certificate and key tlsCertKeyLoad(this->context, certFile, keyFile); // If a CA store is specified then client certificates will be verified // ------------------------------------------------------------------------------------------------------------------------- if (caFile != NULL) // {vm_covered} { // Load CA store cryptoError( // {vm_covered} SSL_CTX_load_verify_locations(this->context, strZ(caFile), NULL) != 1, // {vm_covered} zNewFmt("unable to load CA file '%s'", strZ(caFile))); // {vm_covered} // Tell OpenSSL to send the list of root certs we trust to clients in CertificateRequests. This lets a client with a // keystore select the appropriate client certificate to send to us. Also, this ensures that the SSL context will own // the rootCertList and free it when no longer needed. STACK_OF(X509_NAME) *rootCertList = SSL_load_client_CA_file(strZ(caFile)); // {vm_covered} cryptoError(rootCertList == NULL, zNewFmt("unable to generate CA list from '%s'", strZ(caFile))); // {vm_covered} SSL_CTX_set_client_CA_list(this->context, rootCertList); // {vm_covered} // Always ask for SSL client cert, but don't fail when not presented. In this case the server will disconnect after // sending a data end message to the client. The client can use this to verify that the server is running without the // need to authenticate. SSL_CTX_set_verify(this->context, SSL_VERIFY_PEER | SSL_VERIFY_CLIENT_ONCE, NULL); // {vm_covered} // Set a flag so the client cert will be checked later this->verifyPeer = true; // {vm_covered} } } OBJ_NEW_END(); statInc(TLS_STAT_SERVER_STR); FUNCTION_LOG_RETURN(IO_SERVER, ioServerNew(this, &tlsServerInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/tls/server.h0000664000175100017510000000333115240547630020362 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Server A simple TLS server intended to expose services by accepting TLS over an IoSession. ***********************************************************************************************************************************/ #ifndef COMMON_IO_TLS_SERVER_H #define COMMON_IO_TLS_SERVER_H #include "common/io/server.h" #include "common/io/tls/client.h" #include "common/time.h" /*********************************************************************************************************************************** Io server type ***********************************************************************************************************************************/ #define IO_SERVER_TLS_TYPE IO_CLIENT_TLS_TYPE /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ #define TLS_STAT_SERVER "tls.server" // Servers created STRING_DECLARE(TLS_STAT_SERVER_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN IoServer *tlsServerNew( const String *host, const String *caFile, const String *keyFile, const String *certFile, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/tls/session.c0000664000175100017510000003342315240547630020537 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Session ***********************************************************************************************************************************/ #include #include #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/io/session.h" #include "common/io/tls/client.h" #include "common/io/tls/session.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct TlsSession { IoSession *ioSession; // Io session SSL *session; // TLS session on the file descriptor TimeMSec timeout; // Timeout for any i/o operation (read, write, etc.) bool shutdownOnClose; // Shutdown the TLS connection when closing the session bool ignoreUnexpectedEof; // Ignore unexpected EOF when requested IoRead *read; // Read interface IoWrite *write; // Write interface } TlsSession; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ static void tlsSessionToLog(const THIS_VOID, StringStatic *const debugLog) { THIS(const TlsSession); strStcCat(debugLog, "{ioSession: "); ioSessionToLog(this->ioSession, debugLog); strStcFmt(debugLog, ", timeout: %" PRIu64 ", shutdownOnClose: %s}", this->timeout, cvtBoolToConstZ(this->shutdownOnClose)); } #define FUNCTION_LOG_TLS_SESSION_TYPE \ TlsSession * #define FUNCTION_LOG_TLS_SESSION_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, tlsSessionToLog, buffer, bufferSize) /*********************************************************************************************************************************** Free connection ***********************************************************************************************************************************/ static void tlsSessionFreeResource(THIS_VOID) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); SSL_free(this->session); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void tlsSessionClose(THIS_VOID) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If not already closed if (this->session != NULL) { // Shutdown on request if (this->shutdownOnClose) SSL_shutdown(this->session); // Close the io session ioSessionClose(this->ioSession); // Free the TLS session memContextCallbackClear(objMemContext(this)); tlsSessionFreeResource(this); this->session = NULL; } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Process result from SSL_read(), SSL_write(), SSL_connect(), and SSL_accept(). Returns: 0 if the function should be tried again with the same parameters -1 if the connection was closed gracefully > 0 with the read/write size if SSL_read()/SSL_write() was called ***********************************************************************************************************************************/ // Helper to process error conditions static int tlsSessionResultProcess( TlsSession *const this, const int errorTls, const long unsigned int errorTlsDetail, const int errorSys, const bool closeOk) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_PARAM(INT, errorTls); FUNCTION_LOG_PARAM(UINT64, errorTlsDetail); FUNCTION_LOG_PARAM(INT, errorSys); FUNCTION_LOG_PARAM(BOOL, closeOk); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); int result = -1; switch (errorTls) { // Try again after waiting for read ready case SSL_ERROR_WANT_READ: ioReadReadyP(ioSessionIoReadP(this->ioSession), .error = true); result = 0; break; // Try again after waiting for write ready case SSL_ERROR_WANT_WRITE: ioWriteReadyP(ioSessionIoWrite(this->ioSession), .error = true); result = 0; break; // Handle graceful termination by the server or unexpected EOF/error default: { // Close connection on graceful termination by the server or unexpected EOF/error when allowed if (errorTls == SSL_ERROR_ZERO_RETURN || this->ignoreUnexpectedEof) { if (!closeOk) THROW(ProtocolError, "unexpected TLS eof"); this->shutdownOnClose = false; tlsSessionClose(this); } // Else error else { // Get detailed error message when available const char *const errorTlsDetailMessage = ERR_reason_error_string(errorTlsDetail); THROW_FMT( ServiceError, "TLS error [%d:%lu] %s", errorTls, errorTlsDetail, errorTlsDetailMessage == NULL ? "no details available" : errorTlsDetailMessage); } break; } } FUNCTION_LOG_RETURN(INT, result); } static int tlsSessionResult(TlsSession *const this, int result, const bool closeOk) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_PARAM(INT, result); FUNCTION_LOG_PARAM(BOOL, closeOk); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); // Process errors if (result <= 0) { // Get TLS error and store errno in case of syscall error const int errorTls = SSL_get_error(this->session, result); const long unsigned int errorTlsDetail = ERR_get_error(); const int errorSys = errno; result = tlsSessionResultProcess(this, errorTls, errorTlsDetail, errorSys, closeOk); } FUNCTION_LOG_RETURN(INT, result); } /*********************************************************************************************************************************** Read from the TLS session ***********************************************************************************************************************************/ static size_t tlsSessionRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); ASSERT(buffer != NULL); ASSERT(!bufFull(buffer)); int result = 0; // If blocking read keep reading until buffer is full do { // If no TLS data pending then check the io to reduce blocking if (!SSL_pending(this->session)) ioReadReadyP(ioSessionIoReadP(this->ioSession), .error = true); // Read and handle errors. The error queue must be cleared before this operation. ERR_clear_error(); result = tlsSessionResult(this, SSL_read(this->session, bufRemainsPtr(buffer), (int)bufRemains(buffer)), true); // Update amount of buffer used if (result > 0) { bufUsedInc(buffer, (size_t)result); } // If the connection was closed then we are at eof. It is up to the layer above TLS to decide if this is an error. else if (result == -1) break; } while (block && bufRemains(buffer) > 0); FUNCTION_LOG_RETURN(SIZE, (size_t)result); } /*********************************************************************************************************************************** Write to the TLS session ***********************************************************************************************************************************/ static void tlsSessionWrite(THIS_VOID, const Buffer *const buffer) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); ASSERT(buffer != NULL); int result = 0; while (result == 0) { // Write and handle errors. The error queue must be cleared before this operation. ERR_clear_error(); result = tlsSessionResult(this, SSL_write(this->session, bufPtrConst(buffer), (int)bufUsed(buffer)), false); CHECK(ServiceError, result == 0 || (size_t)result == bufUsed(buffer), "expected retry or complete write"); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Has session been closed by the server? ***********************************************************************************************************************************/ static bool tlsSessionEof(THIS_VOID) { THIS(TlsSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TLS_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(BOOL, this->session == NULL); } /**********************************************************************************************************************************/ static IoRead * tlsSessionIoRead(THIS_VOID, const bool ignoreUnexpectedEof) { THIS(TlsSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TLS_SESSION, this); FUNCTION_TEST_PARAM(BOOL, ignoreUnexpectedEof); FUNCTION_TEST_END(); ASSERT(this != NULL); this->ignoreUnexpectedEof = ignoreUnexpectedEof; FUNCTION_TEST_RETURN(IO_READ, this->read); } /**********************************************************************************************************************************/ static IoWrite * tlsSessionIoWrite(THIS_VOID) { THIS(TlsSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TLS_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(IO_WRITE, this->write); } /**********************************************************************************************************************************/ static IoSessionRole tlsSessionRole(const THIS_VOID) { THIS(const TlsSession); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(TLS_SESSION, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING_ID, ioSessionRole(this->ioSession)); } /**********************************************************************************************************************************/ static const IoSessionInterface tlsSessionInterface = { .type = IO_CLIENT_TLS_TYPE, .close = tlsSessionClose, .ioRead = tlsSessionIoRead, .ioWrite = tlsSessionIoWrite, .role = tlsSessionRole, .toLog = tlsSessionToLog, }; FN_EXTERN IoSession * tlsSessionNew(SSL *const session, IoSession *const ioSession, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM_P(VOID, session); FUNCTION_LOG_PARAM(IO_SESSION, ioSession); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(session != NULL); ASSERT(ioSession != NULL); OBJ_NEW_BEGIN(TlsSession, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (TlsSession) { .session = session, .ioSession = ioSessionMove(ioSession, objMemContext(this)), .timeout = timeout, .shutdownOnClose = true, }; // Ensure session is freed memContextCallbackSet(objMemContext(this), tlsSessionFreeResource, this); // Assign file descriptor to TLS session cryptoError(SSL_set_fd(this->session, ioSessionFd(this->ioSession)) != 1, "unable to add fd to TLS session"); // Negotiate TLS session. The error queue must be cleared before this operation. int result = 0; while (result == 0) { ERR_clear_error(); if (ioSessionRole(this->ioSession) == ioSessionRoleClient) result = tlsSessionResult(this, SSL_connect(this->session), false); else result = tlsSessionResult(this, SSL_accept(this->session), false); } // Create read and write interfaces this->write = ioWriteNewP(this, .write = tlsSessionWrite); ioWriteOpen(this->write); this->read = ioReadNewP(this, .block = true, .eof = tlsSessionEof, .read = tlsSessionRead); ioReadOpen(this->read); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_SESSION, ioSessionNew(this, &tlsSessionInterface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/tls/session.h0000664000175100017510000000222215240547630020535 0ustar00runnerrunner/*********************************************************************************************************************************** TLS Session TLS sessions are created by calling ioClientOpen(). TLS sessions are generally reused so the user must be prepared to retry their transaction on a read/write error if the server closes the session before it is reused. If this behavior is not desirable then ioSessionFree()/ioClientOpen() can be called to get a new session. ***********************************************************************************************************************************/ #ifndef COMMON_IO_TLS_SESSION_H #define COMMON_IO_TLS_SESSION_H #include #include "common/io/session.h" #include "common/time.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Only called by TLS client/server code FN_EXTERN IoSession *tlsSessionNew(SSL *session, IoSession *ioSession, TimeMSec timeout); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/write.c0000664000175100017510000002344615240547630017410 0ustar00runnerrunner/*********************************************************************************************************************************** IO Write Interface ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/io/write.h" #include "common/log.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct IoWrite { IoWritePub pub; // Publicly accessible variables Buffer *output; // Output buffer uint64_t position; // Current write position #ifdef DEBUG bool filterGroupSet; // Were filters set? bool opened; // Has the io been opened? bool closed; // Has the io been closed? #endif }; /**********************************************************************************************************************************/ FN_EXTERN IoWrite * ioWriteNew(void *const driver, const IoWriteInterface interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(IO_WRITE_INTERFACE, interface); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(interface.write != NULL); OBJ_NEW_BEGIN(IoWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (IoWrite) { .pub = { .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, .filterGroup = ioFilterGroupNew(), }, .output = bufNew(ioBufferSize()), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(IO_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteOpen(IoWrite *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->opened && !this->closed); if (ioWriteInterface(this)->open != NULL) ioWriteInterface(this)->open(ioWriteDriver(this)); // Track whether filters were added to prevent seek()/flush() from being called later since they won't work with most filters #ifdef DEBUG this->filterGroupSet = ioFilterGroupSize(this->pub.filterGroup) > 0; #endif // Open the filter group ioFilterGroupOpen(this->pub.filterGroup); #ifdef DEBUG this->opened = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWrite(IoWrite *const this, const Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->opened && !this->closed); // Only write if there is data to write if (buffer != NULL && !bufEmpty(buffer)) { do { ioFilterGroupProcess(this->pub.filterGroup, buffer, this->output); // Write data if the buffer is full if (bufRemains(this->output) == 0) { ioWriteInterface(this)->write(ioWriteDriver(this), this->output); this->position += bufUsed(this->output); bufUsedZero(this->output); } } while (ioFilterGroupInputSame(this->pub.filterGroup)); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteLine(IoWrite *const this, const Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ioWrite(this, buffer); ioWrite(this, LF_BUF); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool ioWriteReady(IoWrite *const this, const IoWriteReadyParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(BOOL, param.error); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = true; if (ioWriteInterface(this)->ready != NULL) result = ioWriteInterface(this)->ready(ioWriteDriver(this), param.error); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteStr(IoWrite *const this, const String *const string) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(STRING, string); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(string != NULL); ioWrite(this, BUFSTR(string)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteStrLine(IoWrite *const this, const String *const string) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(STRING, string); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(string != NULL); ioWrite(this, BUFSTR(string)); ioWrite(this, LF_BUF); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteVarIntU64(IoWrite *const this, const uint64_t value) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(UINT64, value); FUNCTION_LOG_END(); ASSERT(this != NULL); uint8_t buffer[CVT_VARINT128_BUFFER_SIZE]; size_t bufferPos = 0; cvtUInt64ToVarInt128(value, buffer, &bufferPos, sizeof(buffer)); ioWrite(this, BUF(buffer, bufferPos)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteFlush(IoWrite *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->opened && !this->closed); ASSERT(!this->filterGroupSet); if (!bufEmpty(this->output)) { ioWriteInterface(this)->write(ioWriteDriver(this), this->output); bufUsedZero(this->output); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteSeek(IoWrite *const this, const uint64_t position) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_PARAM(UINT64, position); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->opened && !this->closed); ASSERT(!this->filterGroupSet); ASSERT(ioWriteInterface(this)->write != NULL); // If the requested position is not the current position then flush writes and seek. Writes must be flushed first so they will // not later end up in the wrong location. if (position != this->position) { ioWriteFlush(this); ioWriteInterface(this)->seek(ioWriteDriver(this), position); // Update current position this->position = position; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void ioWriteClose(IoWrite *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->opened && !this->closed); // Flush remaining data do { ioFilterGroupProcess(this->pub.filterGroup, NULL, this->output); // Write data if the buffer is full or if this is the last buffer to be written if (bufRemains(this->output) == 0 || (ioFilterGroupDone(this->pub.filterGroup) && !bufEmpty(this->output))) { ioWriteInterface(this)->write(ioWriteDriver(this), this->output); bufUsedZero(this->output); } } while (!ioFilterGroupDone(this->pub.filterGroup)); // Close the filter group and gather results ioFilterGroupClose(this->pub.filterGroup); // Close the driver if there is a close function if (ioWriteInterface(this)->close != NULL) ioWriteInterface(this)->close(ioWriteDriver(this)); #ifdef DEBUG this->closed = true; #endif FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int ioWriteFd(const IoWrite *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(IO_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INT, ioWriteInterface(this)->fd == NULL ? -1 : ioWriteInterface(this)->fd(ioWriteDriver(this))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/write.h0000664000175100017510000001073315240547630017410 0ustar00runnerrunner/*********************************************************************************************************************************** IO Write Interface Objects that write to some IO destination (file, socket, etc.) are implemented using this interface. All objects are required to implement IoWriteProcess and can optionally implement IoWriteOpen or IoWriteClose. IoWriteOpen and IoWriteClose can be used to allocate/open or deallocate/free resources. An example of an IoWrite object is IoBufferWrite. ***********************************************************************************************************************************/ #ifndef COMMON_IO_WRITE_H #define COMMON_IO_WRITE_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct IoWrite IoWrite; #include "common/io/filter/group.h" #include "common/io/write.intern.h" #include "common/type/buffer.h" #include "common/type/object.h" /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Filter group. Filters must be set before open and cannot be reset FN_INLINE_ALWAYS IoFilterGroup * ioWriteFilterGroup(IoWrite *const this) { return THIS_PUB(IoWrite)->filterGroup; } // File descriptor for the write object. Not all write objects have a file descriptor and -1 will be returned in that case. FN_EXTERN int ioWriteFd(const IoWrite *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open the IO FN_EXTERN void ioWriteOpen(IoWrite *this); // Write data to IO and process filters FN_EXTERN void ioWrite(IoWrite *this, const Buffer *buffer); // Write linefeed-terminated buffer FN_EXTERN void ioWriteLine(IoWrite *this, const Buffer *buffer); // Can bytes be written immediately? There are no guarantees on how much data can be written but it must be at least one byte. typedef struct IoWriteReadyParam { VAR_PARAM_HEADER; bool error; // Error when write not ready } IoWriteReadyParam; #define ioWriteReadyP(this, ...) \ ioWriteReady(this, (IoWriteReadyParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool ioWriteReady(IoWrite *this, IoWriteReadyParam param); // Write string FN_EXTERN void ioWriteStr(IoWrite *this, const String *string); // Write linefeed-terminated string FN_EXTERN void ioWriteStrLine(IoWrite *this, const String *string); // Write varint-128 encoding FN_EXTERN void ioWriteVarIntU64(IoWrite *this, uint64_t value); // Flush any data in the output buffer. This does not end writing and will not work if filters are present. FN_EXTERN void ioWriteFlush(IoWrite *this); // Seek to specified position relative to beginning of write FN_EXTERN void ioWriteSeek(IoWrite *this, uint64_t position); // Close the IO and write any additional data that has not been written yet FN_EXTERN void ioWriteClose(IoWrite *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void ioWriteFree(IoWrite *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_WRITE_TYPE \ IoWrite * #define FUNCTION_LOG_IO_WRITE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "IoWrite", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/io/write.intern.h0000664000175100017510000000576415240547630020716 0ustar00runnerrunner/*********************************************************************************************************************************** IO Write Interface Internal ***********************************************************************************************************************************/ #ifndef COMMON_IO_WRITE_INTERN_H #define COMMON_IO_WRITE_INTERN_H #include "common/io/write.h" #include "common/type/buffer.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct IoWriteInterface { void (*close)(void *driver); int (*fd)(const void *driver); void (*open)(void *driver); // Can bytes be written immediately? There are no guarantees on how much data can be written but it must be at least one byte. // Optionally error when write is not ready. bool (*ready)(void *driver, bool error); // Seek to specified position relative to beginning of write void (*seek)(void *driver, uint64_t position); void (*write)(void *driver, const Buffer *buffer); } IoWriteInterface; #define ioWriteNewP(driver, ...) \ ioWriteNew(driver, (IoWriteInterface){__VA_ARGS__}) FN_EXTERN IoWrite *ioWriteNew(void *driver, IoWriteInterface interface); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct IoWritePub { void *driver; // Driver object IoWriteInterface interface; // Driver interface IoFilterGroup *filterGroup; // IO filters } IoWritePub; // Driver for the write object FN_INLINE_ALWAYS void * ioWriteDriver(const IoWrite *const this) { return THIS_PUB(IoWrite)->driver; } // Interface for the write object FN_INLINE_ALWAYS const IoWriteInterface * ioWriteInterface(const IoWrite *const this) { return &THIS_PUB(IoWrite)->interface; } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_IO_WRITE_INTERFACE_TYPE \ IoWriteInterface #define FUNCTION_LOG_IO_WRITE_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "IoWriteInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/lock.c0000664000175100017510000004111115240547630016564 0ustar00runnerrunner/*********************************************************************************************************************************** Lock Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/bufferWrite.h" #include "common/io/fdRead.h" #include "common/io/fdWrite.h" #include "common/io/io.h" #include "common/lock.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/json.h" #include "common/user.h" #include "common/wait.h" #include "storage/posix/storage.h" #include "storage/storage.intern.h" #include "version.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ // Indicates a lock that was made by matching exec-id rather than holding an actual lock. This distinguishes it from -1, which is a // general system error. #define LOCK_ON_EXEC_ID -2 #define LOCK_KEY_EXEC_ID STRID6("execId", 0x12e0c56051) #define LOCK_KEY_PERCENT_COMPLETE STRID6("pctCplt", 0x14310a140d01) #define LOCK_KEY_PROCESS_ID STRID5("pid", 0x11300) #define LOCK_KEY_SIZE_COMPLETE STRID6("szCplt", 0x50c4286931) #define LOCK_KEY_SIZE STRID5("sz", 0x3530) /*********************************************************************************************************************************** Mem context and local variables ***********************************************************************************************************************************/ typedef struct LockFile { String *name; // Name of lock file int fd; // File descriptor for lock file bool written; // Has the lock file been written? } LockFile; static struct LockLocal { MemContext *memContext; // Mem context for locks const String *path; // Lock path const String *execId; // Process exec id List *lockList; // List of locks held const Storage *storage; // Storage object for lock path } lockLocal; /**********************************************************************************************************************************/ FN_EXTERN void lockInit(const String *const path, const String *const execId) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(STRING, execId); FUNCTION_LOG_END(); ASSERT(lockLocal.memContext == NULL); // Allocate a mem context to hold lock info MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(Lock, .childQty = MEM_CONTEXT_QTY_MAX) { lockLocal.memContext = MEM_CONTEXT_NEW(); lockLocal.path = strDup(path); lockLocal.execId = strDup(execId); lockLocal.lockList = lstNewP(sizeof(LockFile), .comparator = lstComparatorStr); lockLocal.storage = storagePosixNewP(lockLocal.path, .write = true); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const String * lockExecId(void) { FUNCTION_TEST_VOID(); ASSERT(lockLocal.memContext != NULL); FUNCTION_TEST_RETURN_CONST(STRING, lockLocal.execId); } /*********************************************************************************************************************************** Read contents of lock file ***********************************************************************************************************************************/ // Size of initial buffer used to load lock file #define LOCK_BUFFER_SIZE 128 // Helper to read data static LockData lockReadData(const String *const lockFileName, const int fd) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, lockFileName); FUNCTION_LOG_PARAM(INT, fd); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(lockFileName != NULL); ASSERT(fd != -1); LockData result = {0}; TRY_BEGIN() { MEM_CONTEXT_TEMP_BEGIN() { // Read contents of file Buffer *const buffer = bufNew(LOCK_BUFFER_SIZE); IoWrite *const write = ioBufferWriteNewOpen(buffer); ioCopyP(ioFdReadNewOpen(lockFileName, fd, 0), write); ioWriteClose(write); JsonRead *const json = jsonReadNew(strNewBuf(buffer)); jsonReadObjectBegin(json); MEM_CONTEXT_PRIOR_BEGIN() { result.execId = jsonReadStr(jsonReadKeyRequireStrId(json, LOCK_KEY_EXEC_ID)); if (jsonReadKeyExpectStrId(json, LOCK_KEY_PERCENT_COMPLETE)) result.percentComplete = varNewUInt(jsonReadUInt(json)); result.processId = jsonReadInt(jsonReadKeyRequireStrId(json, LOCK_KEY_PROCESS_ID)); if (jsonReadKeyExpectStrId(json, LOCK_KEY_SIZE)) result.size = varNewUInt64(jsonReadUInt64(json)); if (jsonReadKeyExpectStrId(json, LOCK_KEY_SIZE_COMPLETE)) result.sizeComplete = varNewUInt64(jsonReadUInt64(json)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } CATCH_ANY() { THROWP_FMT(errorType(), "unable to read lock file '%s': %s", strZ(lockFileName), errorMessage()); } TRY_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN bool lockAcquire(const String *const lockFileName, const LockAcquireParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, lockFileName); FUNCTION_LOG_PARAM(BOOL, param.returnOnNoLock); FUNCTION_LOG_END(); ASSERT(lockFileName != NULL); ASSERT(lockLocal.memContext != NULL); bool result = false; if (lstFind(lockLocal.lockList, &lockFileName) != NULL) THROW_FMT(AssertError, "lock on file '%s' already held", strZ(lockFileName)); MEM_CONTEXT_TEMP_BEGIN() { const String *const lockFilePath = storagePathP(lockLocal.storage, lockFileName); bool retry; int errNo = 0; int fd = -1; do { // Assume there will be no retry retry = false; // Attempt to open the file if ((fd = open(strZ(lockFilePath), O_RDWR | O_CREAT, STORAGE_MODE_FILE_DEFAULT)) == -1) { // Save the error for reporting outside the loop errNo = errno; // If the path does not exist then create it if (errNo == ENOENT) { storagePathCreateP(storagePosixNewP(FSLASH_STR, .write = true), strPath(lockFilePath)); retry = true; } } else { // Attempt to lock the file if (flock(fd, LOCK_EX | LOCK_NB) == -1) { // Save the error for reporting outside the loop errNo = errno; // Get execId from lock file and close it const String *execId = NULL; TRY_BEGIN() { execId = lockReadData(lockFileName, fd).execId; } CATCH_ANY() { // Any errors will be reported as unable to acquire a lock. If a process is trying to get a lock but is not // synchronized with the process holding the actual lock, it is possible that it could see a short read or // have some other problem reading. Reporting the error will likely be misleading when the actual problem is // that another process owns the lock file. } FINALLY() { close(fd); } TRY_END(); // Even though we were unable to lock the file, it may be that it is already locked by another process with the // same exec-id, i.e. spawned by the same original main process. If so, report the lock as successful. if (strEq(execId, lockLocal.execId)) fd = LOCK_ON_EXEC_ID; else fd = -1; } } } while (fd == -1 && retry); // If the lock was not successful if (fd == -1) { // Error when requested if (!param.returnOnNoLock || errNo != EWOULDBLOCK) { const String *errorHint = NULL; if (errNo == EWOULDBLOCK) errorHint = strNewZ("\nHINT: is another " PROJECT_NAME " process running?"); else if (errNo == EACCES) { // Get information for the current user userInit(); errorHint = strNewFmt( "\nHINT: does '%s:%s' running " PROJECT_NAME " have permissions on the '%s' file?", strZ(userName()), strZ(groupName()), strZ(lockFilePath)); } THROW_FMT( LockAcquireError, "unable to acquire lock on file '%s': %s%s", strZ(lockFilePath), strerror(errNo), errorHint == NULL ? "" : strZ(errorHint)); } } // Else add lock to list else { MEM_CONTEXT_OBJ_BEGIN(lockLocal.lockList) { lstAdd(lockLocal.lockList, &(LockFile){.name = strDup(lockFileName), .fd = fd}); } MEM_CONTEXT_OBJ_END(); // Write lock data unless lock was acquired by matching execId if (fd != LOCK_ON_EXEC_ID) lockWriteP(lockFileName); // Success result = true; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN LockReadResult lockRead(const String *const lockFileName, const LockReadParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, lockFileName); FUNCTION_LOG_PARAM(BOOL, param.remove); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(lockLocal.memContext != NULL); ASSERT(lockFileName != NULL); LockReadResult result = {.status = lockReadStatusValid}; MEM_CONTEXT_TEMP_BEGIN() { const String *const lockFilePath = storagePathP(lockLocal.storage, lockFileName); // If we cannot open the lock file for any reason then warn int fd = -1; if ((fd = open(strZ(lockFilePath), O_RDONLY, 0)) == -1) { result.status = lockReadStatusMissing; } else { // Attempt a lock on the file - if a lock can be acquired that means the original process died without removing the lock if (flock(fd, LOCK_EX | LOCK_NB) == 0) { result.status = lockReadStatusUnlocked; } // Else attempt to read the file else { TRY_BEGIN() { MEM_CONTEXT_PRIOR_BEGIN() { result.data = lockReadData(lockFilePath, fd); } MEM_CONTEXT_PRIOR_END(); } CATCH_ANY() { result.status = lockReadStatusInvalid; } TRY_END(); } // Remove lock file if requested but do not report failures if (param.remove) unlink(strZ(lockFilePath)); // Close after unlinking to prevent a race condition where another process creates the file as we remove it close(fd); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN void lockWrite(const String *const lockFileName, const LockWriteParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, lockFileName); FUNCTION_LOG_PARAM(VARIANT, param.percentComplete); FUNCTION_LOG_PARAM(VARIANT, param.sizeComplete); FUNCTION_LOG_PARAM(VARIANT, param.size); FUNCTION_LOG_END(); ASSERT(lockFileName != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build the json object JsonWrite *const json = jsonWriteNewP(); jsonWriteObjectBegin(json); jsonWriteStr(jsonWriteKeyStrId(json, LOCK_KEY_EXEC_ID), lockLocal.execId); if (param.percentComplete != NULL) jsonWriteUInt(jsonWriteKeyStrId(json, LOCK_KEY_PERCENT_COMPLETE), varUInt(param.percentComplete)); jsonWriteInt(jsonWriteKeyStrId(json, LOCK_KEY_PROCESS_ID), getpid()); if (param.size != NULL) jsonWriteUInt64(jsonWriteKeyStrId(json, LOCK_KEY_SIZE), varUInt64(param.size)); if (param.sizeComplete != NULL) jsonWriteUInt64(jsonWriteKeyStrId(json, LOCK_KEY_SIZE_COMPLETE), varUInt64(param.sizeComplete)); jsonWriteObjectEnd(json); // Write to lock file LockFile *const lockFile = lstFind(lockLocal.lockList, &lockFileName); if (lockFile == NULL) THROW_FMT(AssertError, "lock file '%s' not found", strZ(lockFileName)); const String *const lockFilePath = storagePathP(lockLocal.storage, lockFileName); if (lockFile->written) { // Seek to beginning of lock file THROW_ON_SYS_ERROR_FMT( lseek(lockFile->fd, 0, SEEK_SET) == -1, FileOpenError, STORAGE_ERROR_READ_SEEK, (uint64_t)0, strZ(lockFilePath)); // In case the current write is shorter than before THROW_ON_SYS_ERROR_FMT(ftruncate(lockFile->fd, 0) == -1, FileWriteError, "unable to truncate '%s'", strZ(lockFilePath)); } // Write lock file data IoWrite *const write = ioFdWriteNewOpen(lockFilePath, lockFile->fd, 0); ioCopyP(ioBufferReadNewOpen(BUFSTR(jsonWriteResult(json))), write); ioWriteClose(write); lockFile->written = true; } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool lockRelease(const LockReleaseParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BOOL, param.returnOnNoLock); FUNCTION_LOG_END(); ASSERT(lockLocal.memContext != NULL); bool result = false; // Nothing to do if lock list is empty if (lstEmpty(lockLocal.lockList)) { // Fail if requested if (!param.returnOnNoLock) THROW(AssertError, "no lock is held by this process"); } // Else release all locks else { // Release until list is empty while (!lstEmpty(lockLocal.lockList)) { LockFile *const lockFile = lstGet(lockLocal.lockList, 0); // Remove lock file if this lock was not acquired by matching execId if (lockFile->fd != LOCK_ON_EXEC_ID) { storageRemoveP(lockLocal.storage, lockFile->name); close(lockFile->fd); } strFree(lockFile->name); lstRemoveIdx(lockLocal.lockList, 0); } // Success result = true; } FUNCTION_LOG_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/lock.h0000664000175100017510000001164615240547630016603 0ustar00runnerrunner/*********************************************************************************************************************************** Lock Handler ***********************************************************************************************************************************/ #ifndef COMMON_LOCK_H #define COMMON_LOCK_H #include /*********************************************************************************************************************************** Lock data ***********************************************************************************************************************************/ #include "common/type/variant.h" // Lock data typedef struct LockData { pid_t processId; // Process holding the lock const String *execId; // Exec id of process holding the lock const Variant *percentComplete; // Percentage of backup complete * 100 (when not NULL) const Variant *sizeComplete; // Completed size of the backup in bytes const Variant *size; // Total size of the backup in bytes } LockData; #include "common/time.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define LOCK_FILE_EXT ".lock" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize lock module FN_EXTERN void lockInit(const String *path, const String *execId); // Exec id for this process, as set by lockInit() FN_EXTERN const String *lockExecId(void); // Acquire a lock typedef struct LockAcquireParam { VAR_PARAM_HEADER; bool returnOnNoLock; // Return when no lock acquired (rather than throw an error) } LockAcquireParam; #define lockAcquireP(lockFileName, ...) \ lockAcquire(lockFileName, (LockAcquireParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool lockAcquire(const String *lockFileName, LockAcquireParam param); // Write data to a lock file typedef struct LockWriteParam { VAR_PARAM_HEADER; const Variant *percentComplete; // Percentage of backup complete * 100 (when not NULL) const Variant *sizeComplete; // Completed size of the backup in bytes const Variant *size; // Total size of the backup in bytes } LockWriteParam; #define lockWriteP(lockFileName, ...) \ lockWrite(lockFileName, (LockWriteParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void lockWrite(const String *lockFileName, LockWriteParam param); // Read a lock file held by another process to get information about what the process is doing typedef enum { lockReadStatusMissing, // File is missing lockReadStatusUnlocked, // File is not locked lockReadStatusInvalid, // File contents are invalid lockReadStatusValid, // File is locked and contexts are valid } LockReadStatus; typedef struct LockReadResult { LockReadStatus status; // Status of file read LockData data; // Lock data } LockReadResult; typedef struct LockReadParam { VAR_PARAM_HEADER; bool remove; // Remove the lock file after locking/reading } LockReadParam; #define lockReadP(lockFileName, ...) \ lockRead(lockFileName, (LockReadParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN LockReadResult lockRead(const String *lockFileName, LockReadParam param); // Release lock(s) typedef struct LockReleaseParam { VAR_PARAM_HEADER; bool returnOnNoLock; // Return when no lock is held (rather than throw an error) } LockReleaseParam; #define lockReleaseP(...) \ lockRelease((LockReleaseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool lockRelease(LockReleaseParam param); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/log.c0000664000175100017510000005140215240547630016421 0ustar00runnerrunner/*********************************************************************************************************************************** Log Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include #include #include "common/debug.h" #include "common/log.h" #include "common/macro.h" #include "common/time.h" #include "common/type/convert.h" /*********************************************************************************************************************************** Module variables ***********************************************************************************************************************************/ // Log levels static LogLevel logLevelStdOut = logLevelError; static LogLevel logLevelStdErr = logLevelError; static LogLevel logLevelFile = logLevelOff; static LogLevel logLevelAny = logLevelError; // Log file descriptors static int logFdStdOut = STDOUT_FILENO; static int logFdStdErr = STDERR_FILENO; static int logFdFile = -1; // Has the log file banner been written yet? static bool logFileBanner = false; // Is the timestamp printed in the log? static bool logTimestamp = false; // Default process id if none is specified static unsigned int logProcessId = 0; // Size of the process id field static int logProcessSize = 2; // Prefix DRY-RUN to log messages static bool logDryRun = false; /*********************************************************************************************************************************** Dry run prefix ***********************************************************************************************************************************/ #define DRY_RUN_PREFIX "[DRY-RUN] " /*********************************************************************************************************************************** Test Asserts ***********************************************************************************************************************************/ #define ASSERT_LOG_LEVEL(logLevel) \ ASSERT(logLevel >= LOG_LEVEL_MIN && logLevel <= LOG_LEVEL_MAX) /*********************************************************************************************************************************** Log buffer -- used to format log header and message ***********************************************************************************************************************************/ static char logBuffer[LOG_BUFFER_SIZE]; /**********************************************************************************************************************************/ #define LOG_LEVEL_TOTAL (LOG_LEVEL_MAX + 1) static const char *const logLevelList[LOG_LEVEL_TOTAL] = { "OFF", "ASSERT", "ERROR", "WARN", "INFO", "DETAIL", "DEBUG", "TRACE", }; FN_EXTERN LogLevel logLevelEnum(unsigned int logLevelSeq) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, logLevelSeq); FUNCTION_TEST_END(); ASSERT(logLevelSeq < LOG_LEVEL_MAX); logLevelSeq += logLevelSeq > 0; FUNCTION_TEST_RETURN(ENUM, logLevelSeq); } FN_EXTERN const char * logLevelStr(const LogLevel logLevel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_END(); ASSERT(logLevel <= LOG_LEVEL_MAX); FUNCTION_TEST_RETURN_CONST(STRINGZ, logLevelList[logLevel]); } /**********************************************************************************************************************************/ static void logAnySet(void) { FUNCTION_TEST_VOID(); logLevelAny = logLevelStdOut; if (logLevelStdErr > logLevelAny) logLevelAny = logLevelStdErr; if (logLevelFile > logLevelAny && logFdFile != -1) logLevelAny = logLevelFile; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN bool logAny(const LogLevel logLevel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_END(); ASSERT_LOG_LEVEL(logLevel); FUNCTION_TEST_RETURN(BOOL, logLevel <= logLevelAny); } /**********************************************************************************************************************************/ FN_EXTERN void logInit( const LogLevel logLevelStdOutParam, const LogLevel logLevelStdErrParam, const LogLevel logLevelFileParam, const bool logTimestampParam, const unsigned int processId, const unsigned int logProcessMax, const bool dryRunParam) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevelStdOutParam); FUNCTION_TEST_PARAM(ENUM, logLevelStdErrParam); FUNCTION_TEST_PARAM(ENUM, logLevelFileParam); FUNCTION_TEST_PARAM(BOOL, logTimestampParam); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_PARAM(UINT, logProcessMax); FUNCTION_TEST_PARAM(BOOL, dryRunParam); FUNCTION_TEST_END(); ASSERT(logLevelStdOutParam <= LOG_LEVEL_MAX); ASSERT(logLevelStdErrParam <= LOG_LEVEL_MAX); ASSERT(logLevelFileParam <= LOG_LEVEL_MAX); ASSERT(processId <= 999); ASSERT(logProcessMax <= 999); logLevelStdOut = logLevelStdOutParam; logLevelStdErr = logLevelStdErrParam; logLevelFile = logLevelFileParam; logTimestamp = logTimestampParam; logProcessId = processId; logProcessSize = logProcessMax > 99 ? 3 : 2; logDryRun = dryRunParam; logAnySet(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Close the log file ***********************************************************************************************************************************/ static void logFileClose(void) { FUNCTION_TEST_VOID(); // Close the file descriptor if it is open if (logFdFile != -1) { close(logFdFile); logFdFile = -1; } logAnySet(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool logFileSet(const char *const logFile) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, logFile); FUNCTION_TEST_END(); ASSERT(logFile != NULL); // Close the log file if it is already open logFileClose(); // Only open the file if there is a chance to log something bool result = true; if (logLevelFile != logLevelOff) { // Open the file and handle errors logFdFile = open(logFile, O_CREAT | O_APPEND | O_WRONLY, 0640); if (logFdFile == -1) { const int errNo = errno; LOG_WARN_FMT( "unable to open log file '%s': %s\nNOTE: process will continue without log file.", logFile, strerror(errNo)); result = false; } // Output the banner on first log message logFileBanner = false; logAnySet(); } logAnySet(); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void logClose(void) { FUNCTION_TEST_VOID(); // Disable all logging logInit(logLevelOff, logLevelOff, logLevelOff, false, 0, 1, false); // Close the log file if it is open logFileClose(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Determine if the log level is in the specified range ***********************************************************************************************************************************/ static bool logRange(const LogLevel logLevel, const LogLevel logRangeMin, const LogLevel logRangeMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(ENUM, logRangeMin); FUNCTION_TEST_PARAM(ENUM, logRangeMax); FUNCTION_TEST_END(); ASSERT_LOG_LEVEL(logLevel); ASSERT_LOG_LEVEL(logRangeMin); ASSERT_LOG_LEVEL(logRangeMax); ASSERT(logRangeMin <= logRangeMax); FUNCTION_TEST_RETURN(BOOL, logLevel >= logRangeMin && logLevel <= logRangeMax); } /*********************************************************************************************************************************** Internal write function that handles errors ***********************************************************************************************************************************/ static void logWrite(const int fd, const char *const message, const size_t messageSize, const char *const errorDetail) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, fd); FUNCTION_TEST_PARAM(STRINGZ, message); FUNCTION_TEST_PARAM(SIZE, messageSize); FUNCTION_TEST_PARAM(STRINGZ, errorDetail); FUNCTION_TEST_END(); ASSERT(fd != -1); ASSERT(message != NULL); ASSERT(messageSize != 0); ASSERT(errorDetail != NULL); if ((size_t)write(fd, message, messageSize) != messageSize) THROW_SYS_ERROR_FMT(FileWriteError, "unable to write %s", errorDetail); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Write out log message and indent subsequent lines ***********************************************************************************************************************************/ static void logWriteIndent(const int fd, const char *message, const size_t indentSize, const char *const errorDetail) { // Indent buffer -- used to write out indent space without having to loop static const char indentBuffer[] = " "; FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, fd); FUNCTION_TEST_PARAM(STRINGZ, message); FUNCTION_TEST_PARAM(SIZE, indentSize); FUNCTION_TEST_PARAM(STRINGZ, errorDetail); FUNCTION_TEST_END(); ASSERT(fd != -1); ASSERT(message != NULL); ASSERT(indentSize > 0 && indentSize < sizeof(indentBuffer)); ASSERT(errorDetail != NULL); // Indent all lines after the first const char *linefeedPtr = strchr(message, '\n'); bool first = true; while (linefeedPtr != NULL) { if (!first) logWrite(fd, indentBuffer, indentSize, errorDetail); else first = false; logWrite(fd, message, (size_t)(linefeedPtr - message + 1), errorDetail); message += (size_t)(linefeedPtr - message + 1); linefeedPtr = strchr(message, '\n'); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Generate the log header and anything else that needs to happen before the message is output ***********************************************************************************************************************************/ typedef struct LogPreResult { size_t bufferPos; char *logBufferStdErr; size_t indentSize; } LogPreResult; static LogPreResult logPre( const LogLevel logLevel, const unsigned int processId, const char *const fileName, const char *const functionName, const int code) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_PARAM(STRINGZ, fileName); FUNCTION_TEST_PARAM(STRINGZ, functionName); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_END(); ASSERT_LOG_LEVEL(logLevel); ASSERT(fileName != NULL); ASSERT(functionName != NULL); ASSERT( (code == 0 && logLevel > logLevelError) || (logLevel == logLevelError && code != errorTypeCode(&AssertError)) || (logLevel == logLevelAssert && code == errorTypeCode(&AssertError))); // Initialize buffer position LogPreResult result = {.bufferPos = 0}; // Add time if (logTimestamp) { const TimeMSec logTimeMSec = timeMSec(); const time_t logTimeSec = (time_t)(logTimeMSec / MSEC_PER_SEC); result.bufferPos += cvtTimeToZP( "%Y-%m-%d %H:%M:%S", logTimeSec, logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos); result.bufferPos += (size_t)snprintf( logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos, ".%03d ", (int)(logTimeMSec % 1000)); } // Add process and aligned log level result.bufferPos += (size_t)snprintf( logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos, "P%0*u %*s: ", logProcessSize, processId == (unsigned int)-1 ? logProcessId : processId, 6, logLevelStr(logLevel)); // When writing to stderr the timestamp, process, and log level alignment will be skipped result.logBufferStdErr = logBuffer + result.bufferPos - strlen(logLevelStr(logLevel)) - 2; // Set the indent size -- this will need to be adjusted for stderr result.indentSize = result.bufferPos; // Add error code if (code != 0) result.bufferPos += (size_t)snprintf(logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos, "[%03d]: ", code); // Add dry-run prefix if (logDryRun) result.bufferPos += (size_t)snprintf(logBuffer + result.bufferPos, sizeof(logBuffer) - result.bufferPos, DRY_RUN_PREFIX); // Add debug info if (logLevel >= logLevelDebug) { // Adding padding for debug and trace levels. Cast to handle compilers (e.g. MSVC) that coerce to signed after subtraction. for (unsigned int paddingIdx = 0; paddingIdx < (unsigned int)((logLevel - logLevelDebug + 1) * 4); paddingIdx++) { logBuffer[result.bufferPos++] = ' '; result.indentSize++; } result.bufferPos += (size_t)snprintf( logBuffer + result.bufferPos, LOG_BUFFER_SIZE - result.bufferPos, "%.*s::%s: ", (int)strlen(fileName) - 2, fileName, functionName); } FUNCTION_TEST_RETURN_TYPE(LogPreResult, result); } /*********************************************************************************************************************************** Finalize formatting and log after the message has been added to the buffer ***********************************************************************************************************************************/ #define LOG_BANNER "-------------------PROCESS START-------------------\n" static void logPost(LogPreResult *const logData, const LogLevel logLevel, const LogLevel logRangeMin, const LogLevel logRangeMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, logData->bufferPos); FUNCTION_TEST_PARAM(STRINGZ, logData->logBufferStdErr); FUNCTION_TEST_PARAM(SIZE, logData->indentSize); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(ENUM, logRangeMin); FUNCTION_TEST_PARAM(ENUM, logRangeMax); FUNCTION_TEST_END(); ASSERT_LOG_LEVEL(logLevel); ASSERT_LOG_LEVEL(logRangeMin); ASSERT_LOG_LEVEL(logRangeMax); ASSERT(logRangeMin <= logRangeMax); // Reserve room for the linefeed and null terminator. snprintf() returns the untruncated length, so a message longer than the // buffer can leave bufferPos past the end; limit it so the writes below stay in bounds. if (logData->bufferPos > LOG_BUFFER_SIZE - 2) logData->bufferPos = LOG_BUFFER_SIZE - 2; // Add linefeed logBuffer[logData->bufferPos++] = '\n'; logBuffer[logData->bufferPos] = 0; // Determine where to log the message based on log-level-stderr if (logLevel <= logLevelStdErr) { if ((logLevelStdErr > logLevelStdOut && logRange(logLevelStdErr, logRangeMin, logRangeMax)) || (logLevelStdErr <= logLevelStdOut && logRange(logLevelStdOut, logRangeMin, logRangeMax))) { logWriteIndent( logFdStdErr, logData->logBufferStdErr, logData->indentSize - (size_t)(logData->logBufferStdErr - logBuffer), "log to stderr"); } } else if (logLevel <= logLevelStdOut && logRange(logLevelStdOut, logRangeMin, logRangeMax)) logWriteIndent(logFdStdOut, logBuffer, logData->indentSize, "log to stdout"); // Log to file if (logLevel <= logLevelFile && logFdFile != -1 && logRange(logLevelFile, logRangeMin, logRangeMax)) { // If the banner has not been written if (!logFileBanner) { // Add a blank line if the file already has content if (lseek(logFdFile, 0, SEEK_END) > 0) logWrite(logFdFile, "\n", 1, "banner spacing to file"); // Write process start banner logWrite(logFdFile, LOG_BANNER, sizeof(LOG_BANNER) - 1, "banner to file"); // Mark banner as written logFileBanner = true; } logWriteIndent(logFdFile, logBuffer, logData->indentSize, "log to file"); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ #define LOG_SIGNAL_MESSAGE_PRE "terminated on signal " FN_EXTERN void logSignal(const LogLevel logLevel, const char *const signalName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(STRINGZ, signalName); FUNCTION_TEST_END(); ASSERT(signalName != NULL); static_assert(LOG_BUFFER_SIZE >= sizeof(LOG_SIGNAL_MESSAGE_PRE), "invalid log buffer size"); // Initialize log buffer and data with static signal message memcpy(logBuffer, LOG_SIGNAL_MESSAGE_PRE, sizeof(LOG_SIGNAL_MESSAGE_PRE) - 1); LogPreResult logData = {.bufferPos = sizeof(LOG_SIGNAL_MESSAGE_PRE) - 1, .logBufferStdErr = logBuffer, .indentSize = 4}; // Add signal name and update buffer position (memcpy includes the null terminator). A signal name is always a short internal // constant that fits the buffer and memcpy is async-signal-safe, unlike snprintf(). const size_t signalNameSize = strlen(signalName); memcpy(logBuffer + logData.bufferPos, signalName, signalNameSize + 1); logData.bufferPos += signalNameSize; logPost(&logData, logLevel, LOG_LEVEL_MIN, LOG_LEVEL_MAX); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void logInternal( const LogLevel logLevel, const LogLevel logRangeMin, const LogLevel logRangeMax, const unsigned int processId, const char *const fileName, const char *const functionName, const int code, const char *const message) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(ENUM, logRangeMin); FUNCTION_TEST_PARAM(ENUM, logRangeMax); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_PARAM(STRINGZ, fileName); FUNCTION_TEST_PARAM(STRINGZ, functionName); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_PARAM(STRINGZ, message); FUNCTION_TEST_END(); ASSERT(message != NULL); LogPreResult logData = logPre(logLevel, processId, fileName, functionName, code); // Copy message into buffer and update buffer position logData.bufferPos += (size_t)snprintf(logBuffer + logData.bufferPos, LOG_BUFFER_SIZE - logData.bufferPos, "%s", message); logPost(&logData, logLevel, logRangeMin, logRangeMax); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void logInternalFmt( const LogLevel logLevel, const LogLevel logRangeMin, const LogLevel logRangeMax, const unsigned int processId, const char *const fileName, const char *const functionName, const int code, const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, logLevel); FUNCTION_TEST_PARAM(ENUM, logRangeMin); FUNCTION_TEST_PARAM(ENUM, logRangeMax); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_PARAM(STRINGZ, fileName); FUNCTION_TEST_PARAM(STRINGZ, functionName); FUNCTION_TEST_PARAM(INT, code); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(format != NULL); LogPreResult logData = logPre(logLevel, processId, fileName, functionName, code); // Format message into buffer and update buffer position va_list argumentList; va_start(argumentList, format); logData.bufferPos += (size_t)vsnprintf( logBuffer + logData.bufferPos, LOG_BUFFER_SIZE - logData.bufferPos, format, argumentList); va_end(argumentList); logPost(&logData, logLevel, logRangeMin, logRangeMax); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/log.h0000664000175100017510000002723715240547630016437 0ustar00runnerrunner/*********************************************************************************************************************************** Log Handler ***********************************************************************************************************************************/ #ifndef COMMON_LOG_H #define COMMON_LOG_H #include #include #include "common/logLevel.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Max size allowed for a single log message including header ***********************************************************************************************************************************/ #ifndef LOG_BUFFER_SIZE #define LOG_BUFFER_SIZE ((size_t)(32 * 1024)) #endif /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize the log system FN_EXTERN void logInit( LogLevel logLevelStdOutParam, LogLevel logLevelStdErrParam, LogLevel logLevelFileParam, bool logTimestampParam, unsigned int processId, unsigned int logProcessMax, bool dryRun); // Close the log system FN_EXTERN void logClose(void); // Set the log file. Returns true if file logging is off or the log file was successfully opened, false if the log file could not be // opened. FN_EXTERN bool logFileSet(const char *logFile); // Check if a log level will be logged to any output. This is useful for log messages that are expensive to generate and should be // skipped if they will be discarded. FN_EXTERN bool logAny(LogLevel logLevel); // Convert log level to string and vice versa FN_EXTERN LogLevel logLevelEnum(unsigned int logLevelSeq); FN_EXTERN const char *logLevelStr(LogLevel logLevel); // Log exit on signal FN_EXTERN void logSignal(LogLevel logLevel, const char *signalName); /*********************************************************************************************************************************** Macros Macros only call logInternal()/logInternalFmt() if the message will be logged to one of the available outputs. Also simplify each call site by supplying commonly-used values. Note that it's possible that not all the macros below will appear in the code. In particular the ERROR and ASSERT macros should not be used directly. They are included for completeness and future usage. ***********************************************************************************************************************************/ // Define a macro to test logAny() that can be removed when performing coverage testing. Checking logAny() saves a function call for // logging calls that won't be output anywhere, but since the macro contains a branch it causes coverage problems. #ifdef DEBUG_COVERAGE #define IF_LOG_ANY(logLevel) #else #define IF_LOG_ANY(logLevel) \ if (logAny(logLevel)) #endif #define LOG_INTERNAL(logLevel, logRangeMin, logRangeMax, processId, code, message) \ do \ { \ IF_LOG_ANY(logLevel) \ { \ logInternal(logLevel, logRangeMin, logRangeMax, processId, __FILE__, __func__, code, message); \ } \ } while (0) #define LOG_INTERNAL_FMT(logLevel, logRangeMin, logRangeMax, processId, code, ...) \ do \ { \ IF_LOG_ANY(logLevel) \ { \ logInternalFmt(logLevel, logRangeMin, logRangeMax, processId, __FILE__, __func__, code, __VA_ARGS__); \ } \ } while (0) #define LOG_PID(logLevel, processId, code, message) \ LOG_INTERNAL(logLevel, LOG_LEVEL_MIN, LOG_LEVEL_MAX, processId, code, message) #define LOG_ASSERT_PID(processId, message) \ LOG_PID(logLevelAssert, processId, errorTypeCode(&AssertError), message) #define LOG_ERROR_PID(processId, code, message) \ LOG_PID(logLevelError, processId, code, message) #define LOG_WARN_PID(processId, message) \ LOG_PID(logLevelWarn, processId, 0, message) #define LOG_INFO_PID(processId, message) \ LOG_PID(logLevelInfo, processId, 0, message) #define LOG_DETAIL_PID(processId, message) \ LOG_PID(logLevelDetail, processId, 0, message) #define LOG_DEBUG_PID(processId, message) \ LOG_PID(logLevelDebug, processId, 0, message) #define LOG_TRACE_PID(processId, message) \ LOG_PID(logLevelTrace, processId, 0, message) #define LOG_PID_FMT(logLevel, processId, code, ...) \ LOG_INTERNAL_FMT(logLevel, LOG_LEVEL_MIN, LOG_LEVEL_MAX, processId, code, __VA_ARGS__) #define LOG_ASSERT_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelAssert, processId, errorTypeCode(&AssertError), __VA_ARGS__) #define LOG_ERROR_PID_FMT(processId, code, ...) \ LOG_PID_FMT(logLevelError, processId, code, __VA_ARGS__) #define LOG_WARN_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelWarn, processId, 0, __VA_ARGS__) #define LOG_INFO_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelInfo, processId, 0, __VA_ARGS__) #define LOG_DETAIL_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelDetail, processId, 0, __VA_ARGS__) #define LOG_DEBUG_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelDebug, processId, 0, __VA_ARGS__) #define LOG_TRACE_PID_FMT(processId, ...) \ LOG_PID_FMT(logLevelTrace, processId, 0, __VA_ARGS__) #define LOG(logLevel, code, message) \ LOG_PID(logLevel, UINT_MAX, code, message) #define LOG_ASSERT(message) \ LOG(logLevelAssert, errorTypeCode(&AssertError), message) #define LOG_ERROR(code, message) \ LOG(logLevelError, code, message) #define LOG_WARN(message) \ LOG(logLevelWarn, 0, message) #define LOG_INFO(message) \ LOG(logLevelInfo, 0, message) #define LOG_DETAIL(message) \ LOG(logLevelDetail, 0, message) #define LOG_DEBUG(message) \ LOG(logLevelDebug, 0, message) #define LOG_TRACE(message) \ LOG(logLevelTrace, 0, message) #define LOG_FMT(logLevel, code, ...) \ LOG_PID_FMT(logLevel, UINT_MAX, code, __VA_ARGS__) #define LOG_ASSERT_FMT(...) \ LOG_FMT(logLevelAssert, errorTypeCode(&AssertError), __VA_ARGS__) #define LOG_ERROR_FMT(code, ...) \ LOG_FMT(logLevelError, code, __VA_ARGS__) #define LOG_WARN_FMT(...) \ LOG_FMT(logLevelWarn, 0, __VA_ARGS__) #define LOG_INFO_FMT(...) \ LOG_FMT(logLevelInfo, 0, __VA_ARGS__) #define LOG_DETAIL_FMT(...) \ LOG_FMT(logLevelDetail, 0, __VA_ARGS__) #define LOG_DEBUG_FMT(...) \ LOG_FMT(logLevelDebug, 0, __VA_ARGS__) #define LOG_TRACE_FMT(...) \ LOG_FMT(logLevelTrace, 0, __VA_ARGS__) /*********************************************************************************************************************************** Internal Functions (in virtually all cases the macros above should be used in preference to these functions) ***********************************************************************************************************************************/ // Log function FN_EXTERN void logInternal( LogLevel logLevel, LogLevel logRangeMin, LogLevel logRangeMax, unsigned int processId, const char *fileName, const char *functionName, int code, const char *message); // Log function with formatting FN_EXTERN FN_PRINTF(8, 9) void logInternalFmt( LogLevel logLevel, LogLevel logRangeMin, LogLevel logRangeMax, unsigned int processId, const char *fileName, const char *functionName, int code, const char *format, ...); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/logLevel.h0000664000175100017510000000167515240547630017425 0ustar00runnerrunner/*********************************************************************************************************************************** Log Levels ***********************************************************************************************************************************/ #ifndef COMMON_LOGLEVEL_H #define COMMON_LOGLEVEL_H /*********************************************************************************************************************************** Log types ***********************************************************************************************************************************/ typedef enum { logLevelOff, logLevelAssert, logLevelError, logLevelWarn, logLevelInfo, logLevelDetail, logLevelDebug, logLevelTrace, } LogLevel; #define LOG_LEVEL_MIN logLevelAssert #define LOG_LEVEL_MAX logLevelTrace #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/macro.h0000664000175100017510000001346715240547630016757 0ustar00runnerrunner/*********************************************************************************************************************************** General Macros Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group Portions Copyright (c) 1994, Regents of the University of California ***********************************************************************************************************************************/ #ifndef COMMON_MACRO_H #define COMMON_MACRO_H #include #include /*********************************************************************************************************************************** Convert the parameter to a zero-terminated string Useful for converting non-string types (e.g. int) to strings for inclusion in messages. ***********************************************************************************************************************************/ #define STRINGIFY_HELPER(param) #param #define STRINGIFY(param) STRINGIFY_HELPER(param) /*********************************************************************************************************************************** If param2 > param1 then assign it to param1 Useful for ensuring coverage in cases where compared values may be always ascending or descending. ***********************************************************************************************************************************/ #define MAX_ASSIGN(param1, param2) \ do \ { \ if (param2 > param1) \ param1 = param2; \ } \ while (0) /*********************************************************************************************************************************** C11 supports static_assert but there are some syntactic placement restrictions. STATIC_ASSERT_EXP() makes it safe to use in an expression. ***********************************************************************************************************************************/ #define STATIC_ASSERT_EXPR(condition, message) \ ((void)({static_assert(condition, message); true;})) /*********************************************************************************************************************************** Allows casting const-ness away from an expression, but doesn't allow changing the type. Enforcement of the latter currently only works for gcc-like compilers. Note that it is not safe to cast const-ness away if the result will ever be modified (it would be undefined behavior). Doing so can cause compiler mis-optimizations or runtime crashes (by modifying read-only memory). It is only safe to use when the result will not be modified, but API design or language restrictions prevent you from declaring that (e.g. because a function returns both const and non-const variables). Note that this only works in function scope, not for global variables (it would be nice, but not trivial, to improve that). Adapted from PostgreSQL src/include/c.h. ***********************************************************************************************************************************/ #ifdef HAVE_BUILTIN_TYPES_COMPATIBLE_P #define UNCONSTIFY(type, expression) \ (STATIC_ASSERT_EXPR(__builtin_types_compatible_p(__typeof(expression), const type), "invalid cast"), (type)(expression)) #else #define UNCONSTIFY(type, expression) \ ((type)(expression)) #endif /*********************************************************************************************************************************** Determine the byte offset required to align a type after an arbitrary number of bytes This is useful for determining how to correctly align a type in a buffer that is being dynamically built up like a struct. ***********************************************************************************************************************************/ #define ALIGN_OFFSET(type, bytes) (alignof(type) - ((bytes) % alignof(type))) /*********************************************************************************************************************************** Determine size of a type in a struct Using sizeof() on a struct member requires additional syntax. ***********************************************************************************************************************************/ #define SIZE_OF_STRUCT_MEMBER(struct, member) sizeof(((struct){0}).member) /*********************************************************************************************************************************** Determine the length of an array that can be determined at compile time For this macro to work correctly the array must be declared like: int intList[] = {0, 1}; It will not work for an array declared like: int *intList; ***********************************************************************************************************************************/ #define LENGTH_OF(array) (sizeof(array) / sizeof((array)[0])) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/memContext.c0000664000175100017510000014676515240547630020004 0ustar00runnerrunner/*********************************************************************************************************************************** Memory Context Manager ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/macro.h" #include "common/memContext.h" /*********************************************************************************************************************************** Contains information about a memory allocation. This header is placed at the beginning of every memory allocation returned to the user by memNew(), etc. The advantage is that when an allocation is passed back by the user we know the location of the allocation header by doing some pointer arithmetic. This is much faster than searching through a list. ***********************************************************************************************************************************/ typedef struct MemContextAlloc { unsigned int allocIdx : 32; // Index in the allocation list unsigned int size : 32; // Allocation size (4GB max) } MemContextAlloc; // Get the allocation buffer pointer given the allocation header pointer #define MEM_CONTEXT_ALLOC_BUFFER(header) ((MemContextAlloc *)header + 1) // Get the allocation header pointer given the allocation buffer pointer #define MEM_CONTEXT_ALLOC_HEADER(buffer) ((MemContextAlloc *)buffer - 1) // Make sure the allocation is valid for the current memory context. This check only works correctly if the allocation is valid and // allocated as one of many but belongs to another context. Otherwise, there is likely to be a segfault. #define ASSERT_ALLOC_MANY_VALID(alloc) \ ASSERT( \ alloc != NULL && (uintptr_t)alloc != (uintptr_t)-sizeof(MemContextAlloc) && \ alloc->allocIdx < memContextAllocMany(memContextStack[memContextCurrentStackIdx].memContext)->listSize && \ memContextAllocMany(memContextStack[memContextCurrentStackIdx].memContext)->list[alloc->allocIdx]); /*********************************************************************************************************************************** Contains information about the memory context ***********************************************************************************************************************************/ // Quantity of child contexts, allocations, or callbacks typedef enum { memQtyNone = 0, // None for this type memQtyOne = 1, // One for this type memQtyMany = 2, // Many for this type } MemQty; // Main structure required by every mem context struct MemContext { #ifdef DEBUG const char *name; // Indicates what the context is being used for uint64_t sequenceNew; // Sequence when this context was created (used for audit) bool active : 1; // Is the context currently active? #endif MemQty childQty : 2; // How many child contexts can this context have? bool childInitialized : 1; // Has the child context list been initialized? MemQty allocQty : 2; // How many allocations can this context have? bool allocInitialized : 1; // Has the allocation list been initialized? MemQty callbackQty : 2; // How many callbacks can this context have? bool callbackInitialized : 1; // Has the callback been initialized? size_t allocExtra : 16; // Size of extra allocation (1kB max) unsigned int contextParentIdx; // Index in the parent context list MemContext *contextParent; // All contexts have a parent except top }; // Mem context with one allocation typedef struct MemContextAllocOne { MemContextAlloc *alloc; // Memory allocation created in this context } MemContextAllocOne; // Mem context with many allocations typedef struct MemContextAllocMany { MemContextAlloc **list; // List of memory allocations created in this context unsigned int listSize; // Size of alloc list (not the actual count of allocations) unsigned int freeIdx; // Index of first free space in the alloc list } MemContextAllocMany; // Mem context with one child context typedef struct MemContextChildOne { MemContext *context; // Context created in this context } MemContextChildOne; // Mem context with many child contexts typedef struct MemContextChildMany { MemContext **list; // List of contexts created in this context unsigned int listSize; // Size of child context list (not the actual count of contexts) unsigned int freeIdx; // Index of first free space in the context list } MemContextChildMany; // Mem context with one callback typedef struct MemContextCallbackOne { void (*function)(void *); // Function to call before the context is freed void *argument; // Argument to pass to callback function } MemContextCallbackOne; /*********************************************************************************************************************************** Possible sizes for the manifest based on options ***********************************************************************************************************************************/ // {uncrustify_off - formatting compressed to save space} static const uint8_t memContextSizePossible[memQtyMany + 1][memQtyMany + 1][memQtyOne + 1] = { // child none {// alloc none {/* callback none */ 0, /* callback one */ sizeof(MemContextCallbackOne)}, // alloc one {/* callback none */ sizeof(MemContextAllocOne), /* callback one */ sizeof(MemContextAllocOne) + sizeof(MemContextCallbackOne)}, // alloc many {/* callback none */ sizeof(MemContextAllocMany), /* callback one */ sizeof(MemContextAllocMany) + sizeof(MemContextCallbackOne)}}, // child one {// alloc none {/* callback none */ sizeof(MemContextChildOne), /* callback one */ sizeof(MemContextChildOne) + sizeof(MemContextCallbackOne)}, // alloc one {/* callback none */ sizeof(MemContextChildOne) + sizeof(MemContextAllocOne), /* callback one */ sizeof(MemContextChildOne) + sizeof(MemContextAllocOne) + sizeof(MemContextCallbackOne)}, // alloc many {/* callback none */ sizeof(MemContextChildOne) + sizeof(MemContextAllocMany), /* callback one */ sizeof(MemContextChildOne) + sizeof(MemContextAllocMany) + sizeof(MemContextCallbackOne)}}, // child many {// alloc none {/* callback none */ sizeof(MemContextChildMany), /* callback one */ sizeof(MemContextChildMany) + sizeof(MemContextCallbackOne)}, // alloc one {/* callback none */ sizeof(MemContextChildMany) + sizeof(MemContextAllocOne), /* callback one */ sizeof(MemContextChildMany) + sizeof(MemContextAllocOne) + sizeof(MemContextCallbackOne)}, // alloc many {/* callback none */ sizeof(MemContextChildMany) + sizeof(MemContextAllocMany), /* callback one */ sizeof(MemContextChildMany) + sizeof(MemContextAllocMany) + sizeof(MemContextCallbackOne)}}, }; // {uncrustify_on} /*********************************************************************************************************************************** Get pointers to optional parts of the manifest ***********************************************************************************************************************************/ // Get pointer to child part #define MEM_CONTEXT_CHILD_OFFSET(memContext) ((uint8_t *)(memContext + 1) + memContext->allocExtra) static MemContextChildOne * memContextChildOne(MemContext *const memContext) { return (MemContextChildOne *)MEM_CONTEXT_CHILD_OFFSET(memContext); } static MemContextChildMany * memContextChildMany(MemContext *const memContext) { return (MemContextChildMany *)MEM_CONTEXT_CHILD_OFFSET(memContext); } // Get pointer to allocation part #define MEM_CONTEXT_ALLOC_OFFSET(memContext) \ ((uint8_t *)(memContext + 1) + memContextSizePossible[memContext->childQty][0][0] + memContext->allocExtra) static MemContextAllocOne * memContextAllocOne(MemContext *const memContext) { return (MemContextAllocOne *)MEM_CONTEXT_ALLOC_OFFSET(memContext); } static MemContextAllocMany * memContextAllocMany(MemContext *const memContext) { return (MemContextAllocMany *)MEM_CONTEXT_ALLOC_OFFSET(memContext); } // Get pointer to callback part static MemContextCallbackOne * memContextCallbackOne(MemContext *const memContext) { return (MemContextCallbackOne *) ((uint8_t *)(memContext + 1) + memContextSizePossible[memContext->childQty][memContext->allocQty][0] + memContext->allocExtra); } /*********************************************************************************************************************************** Top context The top context always exists and can never be freed. All other contexts are children of the top context. The top context is generally used to allocate memory that exists for the life of the program. ***********************************************************************************************************************************/ static struct MemContextTop { MemContext memContext; MemContextChildMany memContextChildMany; MemContextAllocMany memContextAllocMany; } contextTop = { .memContext = { #ifdef DEBUG .name = "TOP", .active = true, #endif .childQty = memQtyMany, .allocQty = memQtyMany, }, }; /*********************************************************************************************************************************** Memory context stack types ***********************************************************************************************************************************/ typedef enum { memContextStackTypeSwitch = 0, // Context can be switched to allocate mem for new variables memContextStackTypeNew, // Context to be tracked for error handling - cannot switch to } MemContextStackType; /*********************************************************************************************************************************** Mem context stack used to pop mem contexts and cleanup after an error ***********************************************************************************************************************************/ #define MEM_CONTEXT_STACK_MAX 128 static struct MemContextStack { MemContext *memContext; MemContextStackType type; unsigned int tryDepth; } memContextStack[MEM_CONTEXT_STACK_MAX] = {{.memContext = (MemContext *)&contextTop}}; static unsigned int memContextCurrentStackIdx = 0; static unsigned int memContextMaxStackIdx = 0; /*********************************************************************************************************************************** ***********************************************************************************************************************************/ #ifdef DEBUG static uint64_t memContextSequence = 0; FN_EXTERN void memContextAuditBegin(MemContextAuditState *const state) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, state); FUNCTION_TEST_END(); ASSERT(state != NULL); ASSERT(state->memContext != NULL); ASSERT(state->memContext == memContextTop() || state->memContext->sequenceNew != 0); if (state->memContext->childInitialized) { ASSERT(state->memContext->childQty != memQtyNone); if (state->memContext->childQty == memQtyOne) { MemContextChildOne *const memContextChild = memContextChildOne(state->memContext); if (memContextChild->context != NULL) state->sequenceContextNew = memContextChild->context->sequenceNew; } else { ASSERT(state->memContext->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(state->memContext); for (unsigned int contextIdx = 0; contextIdx < memContextChild->listSize; contextIdx++) { if (memContextChild->list[contextIdx] != NULL && memContextChild->list[contextIdx]->sequenceNew > state->sequenceContextNew) { state->sequenceContextNew = memContextChild->list[contextIdx]->sequenceNew; } } } } FUNCTION_TEST_RETURN_VOID(); } static bool memContextAuditNameMatch(const char *const actual, const char *const expected) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, actual); FUNCTION_TEST_PARAM(STRINGZ, expected); FUNCTION_TEST_END(); ASSERT(actual != NULL); ASSERT(expected != NULL); unsigned int actualIdx = 0; while (actual[actualIdx] != '\0' && actual[actualIdx] == expected[actualIdx]) actualIdx++; FUNCTION_TEST_RETURN( BOOL, (actual[actualIdx] == '\0' || strncmp(actual + actualIdx, "::", 2) == 0) && (expected[actualIdx] == '\0' || strcmp(expected + actualIdx, " *") == 0)); } FN_EXTERN void memContextAuditEnd(const MemContextAuditState *const state, const char *const returnType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, state); FUNCTION_TEST_PARAM(STRINGZ, returnType); FUNCTION_TEST_END(); if (state->returnTypeAny) FUNCTION_TEST_RETURN_VOID(); if (state->memContext->childInitialized) { ASSERT(state->memContext->childQty != memQtyNone); const char *returnTypeInvalid = NULL; const char *returnTypeFound = NULL; if (state->memContext->childQty == memQtyOne) { MemContextChildOne *const memContextChild = memContextChildOne(state->memContext); if (memContextChild->context != NULL && memContextChild->context->sequenceNew > state->sequenceContextNew && !memContextAuditNameMatch(memContextChild->context->name, returnType)) { returnTypeInvalid = memContextChild->context->name; } } else { ASSERT(state->memContext->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(state->memContext); for (unsigned int contextIdx = 0; contextIdx < memContextChild->listSize; contextIdx++) { if (memContextChild->list[contextIdx] != NULL && memContextChild->list[contextIdx]->sequenceNew > state->sequenceContextNew) { if (memContextAuditNameMatch(memContextChild->list[contextIdx]->name, returnType)) { if (returnTypeFound != NULL) { returnTypeInvalid = memContextChild->list[contextIdx]->name; break; } returnTypeFound = memContextChild->list[contextIdx]->name; } else { returnTypeInvalid = memContextChild->list[contextIdx]->name; break; } } } } if (returnTypeInvalid != NULL) { if (returnTypeFound != NULL) { THROW_FMT( AssertError, "expected return type '%s' already found but also found '%s'", returnTypeFound, returnTypeInvalid); } else { THROW_FMT( AssertError, "expected return type '%s' but found '%s'", returnType, returnTypeInvalid); } } } FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void * memContextAuditAllocExtraName(void *const allocExtra, const char *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, allocExtra); FUNCTION_TEST_PARAM(STRINGZ, name); FUNCTION_TEST_END(); memContextFromAllocExtra(allocExtra)->name = name; FUNCTION_TEST_RETURN_P(VOID, allocExtra); } #endif /*********************************************************************************************************************************** Wrapper around malloc() with error handling ***********************************************************************************************************************************/ static void * memAllocInternal(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate memory void *const buffer = malloc(size); // Error when malloc fails if (buffer == NULL) THROW_FMT(MemoryError, "unable to allocate %zu bytes", size); // Return the buffer FUNCTION_TEST_RETURN_P(VOID, buffer); } /*********************************************************************************************************************************** Allocate an array of pointers and set all entries to NULL ***********************************************************************************************************************************/ static void * memAllocPtrArrayInternal(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate memory void **const buffer = memAllocInternal(size * sizeof(void *)); // Set all pointers to NULL for (size_t ptrIdx = 0; ptrIdx < size; ptrIdx++) buffer[ptrIdx] = NULL; // Return the buffer FUNCTION_TEST_RETURN_P(VOID, buffer); } /*********************************************************************************************************************************** Wrapper around realloc() with error handling ***********************************************************************************************************************************/ static void * memReAllocInternal(void *const bufferOld, const size_t sizeNew) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, bufferOld); FUNCTION_TEST_PARAM(SIZE, sizeNew); FUNCTION_TEST_END(); ASSERT(bufferOld != NULL); // Allocate memory void *const bufferNew = realloc(bufferOld, sizeNew); // Error when realloc fails if (bufferNew == NULL) THROW_FMT(MemoryError, "unable to reallocate %zu bytes", sizeNew); // Return the buffer FUNCTION_TEST_RETURN_P(VOID, bufferNew); } /*********************************************************************************************************************************** Wrapper around realloc() with error handling ***********************************************************************************************************************************/ static void * memReAllocPtrArrayInternal(void *const bufferOld, const size_t sizeOld, const size_t sizeNew) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, bufferOld); FUNCTION_TEST_PARAM(SIZE, sizeOld); FUNCTION_TEST_PARAM(SIZE, sizeNew); FUNCTION_TEST_END(); // Allocate memory void **const bufferNew = memReAllocInternal(bufferOld, sizeNew * sizeof(void *)); // Set all new pointers to NULL for (size_t ptrIdx = sizeOld; ptrIdx < sizeNew; ptrIdx++) bufferNew[ptrIdx] = NULL; // Return the buffer FUNCTION_TEST_RETURN_P(VOID, bufferNew); } /*********************************************************************************************************************************** Wrapper around free() ***********************************************************************************************************************************/ static void memFreeInternal(void *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); free(buffer); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Find space for a new mem context ***********************************************************************************************************************************/ static unsigned int memContextNewIndex(MemContext *const memContext, MemContextChildMany *const memContextChild) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, memContext); FUNCTION_TEST_PARAM_P(VOID, memContextChild); FUNCTION_TEST_END(); ASSERT(memContext != NULL); ASSERT(memContextChild != NULL); // Initialize (free space will always be index 0) if (!memContext->childInitialized) { *memContextChild = (MemContextChildMany) { .list = memAllocPtrArrayInternal(MEM_CONTEXT_INITIAL_SIZE), .listSize = MEM_CONTEXT_INITIAL_SIZE, }; memContext->childInitialized = true; } else { // Try to find space for the new context for (; memContextChild->freeIdx < memContextChild->listSize; memContextChild->freeIdx++) { if (memContextChild->list[memContextChild->freeIdx] == NULL) break; } // If no space was found then allocate more if (memContextChild->freeIdx == memContextChild->listSize) { // Calculate new list size const unsigned int listSizeNew = memContextChild->listSize * 2; // ReAllocate memory before modifying anything else in case there is an error memContextChild->list = memReAllocPtrArrayInternal(memContextChild->list, memContextChild->listSize, listSizeNew); // Set new list size memContextChild->listSize = listSizeNew; } } FUNCTION_TEST_RETURN(UINT, memContextChild->freeIdx); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextNew( #ifdef DEBUG const char *const name, #endif const MemContextNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, name); FUNCTION_TEST_PARAM(UINT, param.childQty); FUNCTION_TEST_PARAM(UINT, param.allocQty); FUNCTION_TEST_PARAM(UINT, param.callbackQty); FUNCTION_TEST_PARAM(SIZE, param.allocExtra); FUNCTION_TEST_END(); ASSERT(name != NULL); ASSERT(param.childQty <= 1 || param.childQty == UINT8_MAX); ASSERT(param.allocQty <= 1 || param.allocQty == UINT8_MAX); ASSERT(param.callbackQty <= 1); // Check context name length ASSERT(name[0] != '\0'); // Pad allocExtra so that optional structures will be aligned. There may not be any optional structures but it doesn't seem // worth the cost of checking since memory allocations are aligned so the extra bytes would be wasted anyway. size_t allocExtra = param.allocExtra; if (allocExtra % alignof(void *) != 0) allocExtra += ALIGN_OFFSET(void *, allocExtra); // Create the new context MemContext *const contextCurrent = memContextStack[memContextCurrentStackIdx].memContext; ASSERT(contextCurrent->childQty != memQtyNone); const MemQty childQty = param.childQty > 1 ? memQtyMany : (MemQty)param.childQty; const MemQty allocQty = param.allocQty > 1 ? memQtyMany : (MemQty)param.allocQty; const MemQty callbackQty = (MemQty)param.callbackQty; MemContext *const this = memAllocInternal( sizeof(MemContext) + allocExtra + memContextSizePossible[childQty][allocQty][callbackQty]); *this = (MemContext) { #ifdef DEBUG // Set the context name .name = name, // Set audit sequence .sequenceNew = ++memContextSequence, // Set new context active .active = true, #endif // Set flags .childQty = childQty, .allocQty = allocQty, .callbackQty = callbackQty, // Set extra allocation .allocExtra = (uint16_t)allocExtra, // Set current context as the parent .contextParent = contextCurrent, }; // Find space for the new context if (contextCurrent->childQty == memQtyOne) { ASSERT(!contextCurrent->childInitialized || memContextChildOne(contextCurrent)->context == NULL); memContextChildOne(contextCurrent)->context = this; contextCurrent->childInitialized = true; } else { ASSERT(contextCurrent->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(contextCurrent); this->contextParentIdx = memContextNewIndex(contextCurrent, memContextChild); memContextChild->list[this->contextParentIdx] = this; // Possible free context must be in the next position memContextChild->freeIdx++; } // Add to the mem context stack so it will be automatically freed on error if memContextKeep() has not been called memContextMaxStackIdx++; memContextStack[memContextMaxStackIdx] = (struct MemContextStack) { .memContext = this, .type = memContextStackTypeNew, .tryDepth = errorTryDepth(), }; // Return context FUNCTION_TEST_RETURN(MEM_CONTEXT, this); } /**********************************************************************************************************************************/ FN_EXTERN void * memContextAllocExtra(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->allocExtra != 0); FUNCTION_TEST_RETURN_P(VOID, this + 1); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextFromAllocExtra(void *const allocExtra) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, allocExtra); FUNCTION_TEST_END(); ASSERT(allocExtra != NULL); ASSERT(((MemContext *)allocExtra - 1)->allocExtra != 0); FUNCTION_TEST_RETURN(MEM_CONTEXT, (MemContext *)allocExtra - 1); } /**********************************************************************************************************************************/ FN_EXTERN void memContextCallbackSet(MemContext *const this, void (*const callbackFunction)(void *), void *const callbackArgument) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_PARAM(FUNCTIONP, callbackFunction); FUNCTION_TEST_PARAM_P(VOID, callbackArgument); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->active); ASSERT(callbackFunction != NULL); ASSERT(this->active); ASSERT(this->callbackQty != memQtyNone); #ifdef DEBUG // Error if callback has already been set - there may be valid use cases for this in the future but error until one is found if (this->callbackInitialized) THROW_FMT(AssertError, "callback is already set for context '%s'", this->name); #endif // Set callback function and argument *(memContextCallbackOne(this)) = (MemContextCallbackOne){.function = callbackFunction, .argument = callbackArgument}; this->callbackInitialized = true; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextCallbackClear(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->callbackQty != memQtyNone); ASSERT(this->active); // Clear callback function and argument *(memContextCallbackOne(this)) = (MemContextCallbackOne){0}; this->callbackInitialized = false; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Find an available slot in the memory context's allocation list and allocate memory ***********************************************************************************************************************************/ static MemContextAlloc * memContextAllocNew(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate memory MemContextAlloc *const result = memAllocInternal(sizeof(MemContextAlloc) + size); // Find space for the new allocation MemContext *const contextCurrent = memContextStack[memContextCurrentStackIdx].memContext; ASSERT(contextCurrent->allocQty != memQtyNone); if (contextCurrent->allocQty == memQtyOne) { MemContextAllocOne *const contextAlloc = memContextAllocOne(contextCurrent); ASSERT(!contextCurrent->allocInitialized || contextAlloc->alloc == NULL); // Initialize allocation header *result = (MemContextAlloc){.size = (unsigned int)(sizeof(MemContextAlloc) + size)}; // Set pointer in allocation contextAlloc->alloc = result; contextCurrent->allocInitialized = true; } else { ASSERT(contextCurrent->allocQty == memQtyMany); MemContextAllocMany *const contextAlloc = memContextAllocMany(contextCurrent); // Initialize (free space will always be index 0) if (!contextCurrent->allocInitialized) { *contextAlloc = (MemContextAllocMany) { .list = memAllocPtrArrayInternal(MEM_CONTEXT_ALLOC_INITIAL_SIZE), .listSize = contextAlloc->listSize = MEM_CONTEXT_ALLOC_INITIAL_SIZE, }; contextCurrent->allocInitialized = true; } else { for (; contextAlloc->freeIdx < contextAlloc->listSize; contextAlloc->freeIdx++) if (contextAlloc->list[contextAlloc->freeIdx] == NULL) break; // If no space was found then allocate more if (contextAlloc->freeIdx == contextAlloc->listSize) { // Calculate new list size const unsigned int listSizeNew = contextAlloc->listSize * 2; // Reallocate memory before modifying anything else in case there is an error contextAlloc->list = memReAllocPtrArrayInternal(contextAlloc->list, contextAlloc->listSize, listSizeNew); // Set new size contextAlloc->listSize = listSizeNew; } } // Initialize allocation header *result = (MemContextAlloc){.allocIdx = contextAlloc->freeIdx, .size = (unsigned int)(sizeof(MemContextAlloc) + size)}; // Set pointer in allocation list contextAlloc->list[contextAlloc->freeIdx] = result; // Update free index to next location. This location may not be free but it is where the search should start next time. contextAlloc->freeIdx++; } FUNCTION_TEST_RETURN_TYPE_P(MemContextAlloc, result); } /*********************************************************************************************************************************** Resize memory that has already been allocated ***********************************************************************************************************************************/ static MemContextAlloc * memContextAllocResize(MemContextAlloc *alloc, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, alloc); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Resize the allocation alloc = memReAllocInternal(alloc, sizeof(MemContextAlloc) + size); alloc->size = (unsigned int)(sizeof(MemContextAlloc) + size); // Update pointer in allocation list in case the realloc moved the allocation MemContext *const currentContext = memContextStack[memContextCurrentStackIdx].memContext; ASSERT(currentContext->allocQty != memQtyNone); ASSERT(currentContext->allocInitialized); if (currentContext->allocQty == memQtyOne) { ASSERT(memContextAllocOne(currentContext)->alloc != NULL); memContextAllocOne(currentContext)->alloc = alloc; } else { ASSERT(currentContext->allocQty == memQtyMany); ASSERT_ALLOC_MANY_VALID(alloc); memContextAllocMany(currentContext)->list[alloc->allocIdx] = alloc; } FUNCTION_TEST_RETURN_TYPE_P(MemContextAlloc, alloc); } /**********************************************************************************************************************************/ FN_EXTERN void * memNew(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_P(VOID, MEM_CONTEXT_ALLOC_BUFFER(memContextAllocNew(size))); } /**********************************************************************************************************************************/ FN_EXTERN void * memNewPtrArray(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate pointer array void **const buffer = (void **const)MEM_CONTEXT_ALLOC_BUFFER(memContextAllocNew(size * sizeof(void *))); // Set pointers to NULL for (size_t ptrIdx = 0; ptrIdx < size; ptrIdx++) buffer[ptrIdx] = NULL; FUNCTION_TEST_RETURN_P(VOID, buffer); } /**********************************************************************************************************************************/ FN_EXTERN void * memResize(void *const buffer, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_P(VOID, MEM_CONTEXT_ALLOC_BUFFER(memContextAllocResize(MEM_CONTEXT_ALLOC_HEADER(buffer), size))); } /**********************************************************************************************************************************/ FN_EXTERN void memFree(void *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_END(); // Get the allocation MemContext *const contextCurrent = memContextStack[memContextCurrentStackIdx].memContext; ASSERT(contextCurrent->allocQty != memQtyNone); ASSERT(contextCurrent->allocInitialized); MemContextAlloc *const alloc = MEM_CONTEXT_ALLOC_HEADER(buffer); // Remove allocation from the context if (contextCurrent->allocQty == memQtyOne) { ASSERT(memContextAllocOne(contextCurrent)->alloc == alloc); memContextAllocOne(contextCurrent)->alloc = NULL; } else { ASSERT(contextCurrent->allocQty == memQtyMany); ASSERT_ALLOC_MANY_VALID(alloc); // If this allocation is before the current free allocation then make it the current free allocation MemContextAllocMany *const contextAlloc = memContextAllocMany(contextCurrent); if (alloc->allocIdx < contextAlloc->freeIdx) contextAlloc->freeIdx = alloc->allocIdx; // Null the allocation contextAlloc->list[alloc->allocIdx] = NULL; } // Free the allocation memFreeInternal(alloc); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextMove(MemContext *const this, MemContext *const parentNew) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_PARAM(MEM_CONTEXT, parentNew); FUNCTION_TEST_END(); ASSERT(parentNew != NULL); // Only move if a valid mem context is provided and the old and new parents are not the same if (this != NULL && this->contextParent != parentNew) { ASSERT(this->active); ASSERT(this->contextParent->active); ASSERT(this->contextParent->childQty != memQtyNone); ASSERT(this->contextParent->childInitialized); // Null out the context in the old parent if (this->contextParent->childQty == memQtyOne) { ASSERT(memContextChildOne(this->contextParent)->context != NULL); memContextChildOne(this->contextParent)->context = NULL; } else { ASSERT(this->contextParent->childQty == memQtyMany); ASSERT(memContextChildMany(this->contextParent)->list[this->contextParentIdx] == this); memContextChildMany(this->contextParent)->list[this->contextParentIdx] = NULL; } // Find a place in the new parent context and assign it ASSERT(parentNew->active); ASSERT(parentNew->childQty != memQtyNone); if (parentNew->childQty == memQtyOne) { ASSERT(!parentNew->childInitialized || memContextChildOne(parentNew)->context == NULL); memContextChildOne(parentNew)->context = this; parentNew->childInitialized = true; } else { ASSERT(parentNew->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(parentNew); // The child list may move while finding a new index so store the index and use it with (what might be) the new pointer this->contextParentIdx = memContextNewIndex(parentNew, memContextChild); memContextChild->list[this->contextParentIdx] = this; } // Set new parent this->contextParent = parentNew; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextSwitch(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->active); ASSERT(memContextCurrentStackIdx < MEM_CONTEXT_STACK_MAX - 1); memContextMaxStackIdx++; memContextCurrentStackIdx = memContextMaxStackIdx; // Add memContext to the stack as a context that can be used for memory allocation memContextStack[memContextCurrentStackIdx] = (struct MemContextStack) { .memContext = this, .type = memContextStackTypeSwitch, .tryDepth = errorTryDepth(), }; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextSwitchBack(void) { FUNCTION_TEST_VOID(); ASSERT(memContextCurrentStackIdx > 0); // Generate a detailed error to help with debugging #ifdef DEBUG if (memContextStack[memContextMaxStackIdx].type == memContextStackTypeNew) { THROW_FMT( AssertError, "current context expected but new context '%s' found", memContextStack[memContextMaxStackIdx].memContext->name); } #endif ASSERT(memContextCurrentStackIdx == memContextMaxStackIdx); memContextMaxStackIdx--; memContextCurrentStackIdx--; // memContext of type New cannot be the current context so keep going until we find a memContext we can switch to as the current // context while (memContextStack[memContextCurrentStackIdx].type == memContextStackTypeNew) memContextCurrentStackIdx--; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextKeep(void) { FUNCTION_TEST_VOID(); // Generate a detailed error to help with debugging #ifdef DEBUG if (memContextStack[memContextMaxStackIdx].type != memContextStackTypeNew) { THROW_FMT( AssertError, "new context expected but current context '%s' found", memContextStack[memContextMaxStackIdx].memContext->name); } #endif memContextMaxStackIdx--; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void memContextDiscard(void) { FUNCTION_TEST_VOID(); // Generate a detailed error to help with debugging #ifdef DEBUG if (memContextStack[memContextMaxStackIdx].type != memContextStackTypeNew) { THROW_FMT( AssertError, "new context expected but current context '%s' found", memContextStack[memContextMaxStackIdx].memContext->name); } #endif memContextFree(memContextStack[memContextMaxStackIdx].memContext); memContextMaxStackIdx--; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextTop(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(MEM_CONTEXT, (MemContext *)&contextTop); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextCurrent(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(MEM_CONTEXT, memContextStack[memContextCurrentStackIdx].memContext); } /**********************************************************************************************************************************/ FN_EXTERN MemContext * memContextPrior(void) { FUNCTION_TEST_VOID(); ASSERT(memContextCurrentStackIdx > 0); unsigned int priorIdx = 1; while (memContextStack[memContextCurrentStackIdx - priorIdx].type == memContextStackTypeNew) priorIdx++; FUNCTION_TEST_RETURN(MEM_CONTEXT, memContextStack[memContextCurrentStackIdx - priorIdx].memContext); } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN size_t memContextSize(const MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->active); // Size of struct and extra size_t total = 0; const uint8_t *offset = (const uint8_t *)(this + 1) + this->allocExtra; // Size of child contexts if (this->childQty == memQtyOne) { if (this->childInitialized) { const MemContextChildOne *const contextChild = (const MemContextChildOne *const)offset; if (contextChild->context != NULL) total += memContextSize(contextChild->context); } offset += sizeof(MemContextChildOne); } else if (this->childQty == memQtyMany) { if (this->childInitialized) { const MemContextChildMany *const contextChild = (const MemContextChildMany *const)offset; for (unsigned int contextIdx = 0; contextIdx < contextChild->listSize; contextIdx++) { if (contextChild->list[contextIdx] != NULL) total += memContextSize(contextChild->list[contextIdx]); } total += contextChild->listSize * sizeof(MemContextChildMany *); } offset += sizeof(MemContextChildMany); } // Size of allocations if (this->allocQty == memQtyOne) { if (this->allocInitialized) { const MemContextAllocOne *const contextAlloc = (const MemContextAllocOne *const)offset; if (contextAlloc->alloc != NULL) total += contextAlloc->alloc->size; } offset += sizeof(MemContextAllocOne); } else if (this->allocQty == memQtyMany) { if (this->allocInitialized) { const MemContextAllocMany *const contextAlloc = (const MemContextAllocMany *const)offset; for (unsigned int allocIdx = 0; allocIdx < contextAlloc->listSize; allocIdx++) { if (contextAlloc->list[allocIdx] != NULL) total += contextAlloc->list[allocIdx]->size; } total += contextAlloc->listSize * sizeof(MemContextAllocMany *); } offset += sizeof(MemContextAllocMany); } // Size of callback if (this->callbackQty != memQtyNone) offset += sizeof(MemContextCallbackOne); FUNCTION_TEST_RETURN(SIZE, (size_t)(offset - (const uint8_t *)this) + total); } #endif // DEBUG /**********************************************************************************************************************************/ FN_EXTERN void memContextClean(const unsigned int tryDepth, const bool fatal) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, tryDepth); FUNCTION_TEST_PARAM(BOOL, false); FUNCTION_TEST_END(); ASSERT(tryDepth > 0); // Iterate through everything pushed to the stack since the last try while (memContextStack[memContextMaxStackIdx].tryDepth >= tryDepth) { // Free memory contexts that were not kept. Skip this for fatal errors to avoid calling destructors that could error and // mask the original error. if (memContextStack[memContextMaxStackIdx].type == memContextStackTypeNew) { if (!fatal) memContextFree(memContextStack[memContextMaxStackIdx].memContext); } // Else find the prior context and make it the current context else { memContextCurrentStackIdx--; while (memContextStack[memContextCurrentStackIdx].type == memContextStackTypeNew) memContextCurrentStackIdx--; } memContextMaxStackIdx--; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to execute callbacks for the context and all its children static void memContextCallbackRecurse(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); #ifdef DEBUG // Certain actions against the context are no longer allowed this->active = false; #endif // Callback if (this->callbackInitialized) { MemContextCallbackOne *const callback = memContextCallbackOne(this); callback->function(callback->argument); this->callbackInitialized = false; } // Child callbacks if (this->childInitialized) { if (this->childQty == memQtyOne) { MemContextChildOne *const memContextChild = memContextChildOne(this); if (memContextChild->context != NULL) memContextCallbackRecurse(memContextChild->context); } else { ASSERT(this->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(this); for (unsigned int contextIdx = 0; contextIdx < memContextChild->listSize; contextIdx++) { if (memContextChild->list[contextIdx] != NULL) memContextCallbackRecurse(memContextChild->list[contextIdx]); } } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static void memContextFreeRecurse(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); #ifdef DEBUG // Current context cannot be freed unless it is top (top is never really freed, just the stuff under it) if (this == memContextStack[memContextCurrentStackIdx].memContext && this != memContextTop()) THROW_FMT(AssertError, "cannot free current context '%s'", this->name); #endif // Free child contexts if (this->childInitialized) { if (this->childQty == memQtyOne) { MemContextChildOne *const memContextChild = memContextChildOne(this); if (memContextChild->context != NULL) memContextFreeRecurse(memContextChild->context); } else { ASSERT(this->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(this); for (unsigned int contextIdx = 0; contextIdx < memContextChild->listSize; contextIdx++) { if (memContextChild->list[contextIdx] != NULL) memContextFreeRecurse(memContextChild->list[contextIdx]); } // Free child context allocation list memFreeInternal(memContextChildMany(this)->list); } } // Free memory allocations and list if (this->allocInitialized) { ASSERT(this->allocQty != memQtyNone); if (this->allocQty == memQtyOne) { MemContextAllocOne *const contextAlloc = memContextAllocOne(this); if (contextAlloc->alloc != NULL) memFreeInternal(contextAlloc->alloc); } else { ASSERT(this->allocQty == memQtyMany); MemContextAllocMany *const contextAlloc = memContextAllocMany(this); for (unsigned int allocIdx = 0; allocIdx < contextAlloc->listSize; allocIdx++) if (contextAlloc->list[allocIdx] != NULL) memFreeInternal(contextAlloc->list[allocIdx]); memFreeInternal(contextAlloc->list); } } // Free the memory context so the slot can be reused (if not the top mem context) if (this != memContextTop()) { ASSERT(this->contextParent != NULL); if (this->contextParent->childQty == memQtyOne) memContextChildOne(this->contextParent)->context = NULL; else { ASSERT(this->contextParent->childQty == memQtyMany); MemContextChildMany *const memContextChild = memContextChildMany(this->contextParent); // If the context index is lower than the current free index in the parent then replace it if (this->contextParentIdx < memContextChild->freeIdx) memContextChild->freeIdx = this->contextParentIdx; memContextChildMany(this->contextParent)->list[this->contextParentIdx] = NULL; } memFreeInternal(this); } // Else reset top context. In practice it is uncommon for the top mem context to be freed and then used again. else { this->childInitialized = false; this->allocInitialized = false; #ifdef DEBUG // Make the top mem context active again this->active = true; #endif } FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void memContextFree(MemContext *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->active); // Execute callbacks TRY_BEGIN() { memContextCallbackRecurse(this); } // Free context even if a callback fails FINALLY() { memContextFreeRecurse(this); } TRY_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/memContext.h0000664000175100017510000005016415240547630017774 0ustar00runnerrunner/*********************************************************************************************************************************** Memory Context Manager Memory is allocated inside contexts and all allocations (and child memory contexts) are freed when the context is freed. The goal is to make memory management both easier and more performant. Memory context management is encapsulated in macros so there is rarely any need to call the functions directly. Memory allocations are mostly performed in the constructors of objects and reallocated as needed. See the sections on memory context management and memory allocations below for more details. ***********************************************************************************************************************************/ #ifndef COMMON_MEMCONTEXT_H #define COMMON_MEMCONTEXT_H #include #include #include /*********************************************************************************************************************************** Memory context object ***********************************************************************************************************************************/ typedef struct MemContext MemContext; #include "common/type/param.h" /*********************************************************************************************************************************** Define initial number of memory contexts No space is reserved for child contexts when a new context is created because most contexts will be leaves. When a child context is requested then space will be reserved for this many child contexts initially. When more space is needed the size will be doubled. ***********************************************************************************************************************************/ #define MEM_CONTEXT_INITIAL_SIZE 4 /*********************************************************************************************************************************** Define initial number of memory allocations Space is reserved for this many allocations when a context is created. When more space is needed the size will be doubled. ***********************************************************************************************************************************/ #define MEM_CONTEXT_ALLOC_INITIAL_SIZE 4 /*********************************************************************************************************************************** Functions and macros to audit a mem context by detecting new child contexts/allocations that were created begin the begin/end but are not the expected return type. ***********************************************************************************************************************************/ #if defined(DEBUG) typedef struct MemContextAuditState { MemContext *memContext; // Mem context to audit bool returnTypeAny; // Skip auditing for this mem context uint64_t sequenceContextNew; // Max sequence for new contexts at beginning } MemContextAuditState; // Begin the audit FN_EXTERN void memContextAuditBegin(MemContextAuditState *state); // End the audit and make sure the return type is as expected FN_EXTERN void memContextAuditEnd(const MemContextAuditState *state, const char *returnTypeDefault); // Rename a mem context using the extra allocation pointer #define MEM_CONTEXT_AUDIT_ALLOC_EXTRA_NAME(this, name) memContextAuditAllocExtraName(this, #name) FN_EXTERN void *memContextAuditAllocExtraName(void *allocExtra, const char *name); #else #define MEM_CONTEXT_AUDIT_ALLOC_EXTRA_NAME(this, name) this #endif /*********************************************************************************************************************************** Memory management functions All these functions operate in the current memory context, including memResize() and memFree(). ***********************************************************************************************************************************/ // Allocate memory in the current memory context FN_EXTERN void *memNew(size_t size); // Allocate requested number of pointers and initialize them to NULL FN_EXTERN void *memNewPtrArray(size_t size); // Reallocate to the new size. Original buffer pointer is undefined on return. FN_EXTERN void *memResize(void *buffer, size_t size); // Free memory allocation FN_EXTERN void memFree(void *buffer); /*********************************************************************************************************************************** Ensure that the prior memory context is restored after the block executes (even on error) MEM_CONTEXT_BEGIN(memContext) { } MEM_CONTEXT_END(); ***********************************************************************************************************************************/ #define MEM_CONTEXT_BEGIN(memContext) \ do \ { \ memContextSwitch(memContext); /* Switch to the new memory context */ #define MEM_CONTEXT_END() \ memContextSwitchBack(); /* Switch back to the prior context */ \ } \ while (0) /*********************************************************************************************************************************** Switch to prior context and ensure that the previous prior memory context is restored after the block executes (even on error) MEM_CONTEXT_PRIOR_BEGIN(memContext) { } MEM_CONTEXT_PRIOR_END(); ***********************************************************************************************************************************/ #define MEM_CONTEXT_PRIOR_BEGIN() \ MEM_CONTEXT_BEGIN(memContextPrior()) #define MEM_CONTEXT_PRIOR_END() \ MEM_CONTEXT_END() /*********************************************************************************************************************************** Create a new context and make sure it is freed on error and prior context is restored in all cases MEM_CONTEXT_NEW_BEGIN(memContextName, ...) { } MEM_CONTEXT_NEW_END(); Note that memory context names are expected to live for the lifetime of the context -- no copy is made. ***********************************************************************************************************************************/ #define MEM_CONTEXT_NEW() \ MEM_CONTEXT_NEW_memContext #define MEM_CONTEXT_NEW_BEGIN(memContextName, ...) \ do \ { \ MemContext *MEM_CONTEXT_NEW() = memContextNewP(STRINGIFY(memContextName), __VA_ARGS__); \ memContextSwitch(MEM_CONTEXT_NEW()); #define MEM_CONTEXT_NEW_ALLOC() \ memContextAllocExtra(MEM_CONTEXT_NEW()) #define MEM_CONTEXT_NEW_END() \ memContextSwitchBack(); \ memContextKeep(); \ } \ while (0) /*********************************************************************************************************************************** Create a temporary memory context and make sure it is freed when done (even on error) MEM_CONTEXT_TEMP_BEGIN() { } MEM_CONTEXT_TEMP_END(); ***********************************************************************************************************************************/ #define MEM_CONTEXT_TEMP() \ MEM_CONTEXT_TEMP_memContext #define MEM_CONTEXT_TEMP_BEGIN() \ do \ { \ MemContext *MEM_CONTEXT_TEMP() = memContextNewP( \ "temporary", .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX); \ memContextSwitch(MEM_CONTEXT_TEMP()); #define MEM_CONTEXT_TEMP_RESET_BEGIN() \ MEM_CONTEXT_TEMP_BEGIN() \ unsigned int MEM_CONTEXT_TEMP_loopTotal = 0; #define MEM_CONTEXT_TEMP_RESET(resetTotal) \ do \ { \ MEM_CONTEXT_TEMP_loopTotal++; \ \ if (MEM_CONTEXT_TEMP_loopTotal >= resetTotal) \ { \ memContextSwitchBack(); \ memContextDiscard(); \ MEM_CONTEXT_TEMP() = memContextNewP("temporary", .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX); \ memContextSwitch(MEM_CONTEXT_TEMP()); \ MEM_CONTEXT_TEMP_loopTotal = 0; \ } \ } \ while (0) #define MEM_CONTEXT_TEMP_END() \ memContextSwitchBack(); \ memContextDiscard(); \ } \ while (0) /*********************************************************************************************************************************** Memory context management functions memContextSwitch(memContextNewP()); memContextSwitchBack(); memContextKeep()/memContextDiscard(); There is no need to implement any error handling. The mem context system will automatically clean up any mem contexts that were created but not marked as keep when an error occurs and reset the current mem context to whatever it was at the beginning of the nearest try block. Use the MEM_CONTEXT*() macros when possible rather than reimplement the boilerplate for every memory context block. ***********************************************************************************************************************************/ // Create a new mem context in the current mem context. The new context must be either kept with memContextKeep() or discarded with // memContextDiscard() before switching back from the parent context. typedef struct MemContextNewParam { VAR_PARAM_HEADER; uint8_t childQty; // How many child contexts can this context have? uint8_t allocQty; // How many allocations can this context have? uint8_t callbackQty; // How many callbacks can this context have? uint16_t allocExtra; // Extra memory to allocate with the context } MemContextNewParam; // Maximum amount of extra memory that can be allocated with the context using allocExtra #define MEM_CONTEXT_ALLOC_EXTRA_MAX UINT16_MAX // Specify maximum quantity of child contexts or allocations using childQty or allocQty #define MEM_CONTEXT_QTY_MAX UINT8_MAX #ifdef DEBUG #define memContextNewP(name, ...) \ memContextNew(name, (MemContextNewParam){VAR_PARAM_INIT, __VA_ARGS__}) #else #define memContextNewP(name, ...) \ memContextNew((MemContextNewParam){VAR_PARAM_INIT, __VA_ARGS__}) #endif FN_EXTERN MemContext *memContextNew( #ifdef DEBUG const char *name, #endif MemContextNewParam param); // Switch to a context making it the current mem context FN_EXTERN void memContextSwitch(MemContext *this); // Switch back to the context that was current before the last switch. If the last function called was memContextNewP(), then // memContextKeep()/memContextDiscard() must be called before calling memContextSwitchBack(), otherwise an error will occur in // debug builds and the behavior is undefined in production builds. FN_EXTERN void memContextSwitchBack(void); // Keep a context created by memContextNewP() so that it will not be automatically freed if an error occurs. If the context was // switched after the call to memContextNewP(), then it must be switched back before calling memContextKeep() or an error will occur // in debug builds and the behavior is undefined in production builds. FN_EXTERN void memContextKeep(void); // Discard a context created by memContextNewP(). If the context was switched after the call to memContextNewP(), then it must be // switched back before calling memContextDiscard() or an error will occur in debug builds and the behavior is undefined in // production builds. FN_EXTERN void memContextDiscard(void); // Move mem context to a new parent. This is generally used to move objects to a new context once they have been successfully // created. FN_EXTERN void memContextMove(MemContext *this, MemContext *parentNew); // Set a function that will be called when this mem context is freed FN_EXTERN void memContextCallbackSet(MemContext *this, void (*callbackFunction)(void *), void *); // Clear the callback function so it won't be called when the mem context is freed. This is usually done in the object free method // after resources have been freed but before memContextFree() is called. The goal is to prevent the object free method from being // called more than once. FN_EXTERN void memContextCallbackClear(MemContext *this); // Free a memory context FN_EXTERN void memContextFree(MemContext *this); /*********************************************************************************************************************************** Memory context getters ***********************************************************************************************************************************/ // Pointer to the extra memory allocated with the mem context FN_EXTERN void *memContextAllocExtra(MemContext *this); // Get mem context using pointer to the memory allocated with the mem context FN_EXTERN MemContext *memContextFromAllocExtra(void *allocExtra); // Current memory context FN_EXTERN MemContext *memContextCurrent(void); // Prior context, i.e. the context that was current before the last memContextSwitch() FN_EXTERN MemContext *memContextPrior(void); // "top" context. This context is created at initialization and is always present, i.e. it is never freed. The top context is a good // place to put long-lived mem contexts since they won't be automatically freed until the program exits. FN_EXTERN MemContext *memContextTop(void); // Get total size of mem context and all children #ifdef DEBUG FN_EXTERN size_t memContextSize(const MemContext *this); #endif // DEBUG /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_MEM_CONTEXT_TYPE \ MemContext * #define FUNCTION_LOG_MEM_CONTEXT_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "MemContext", buffer, bufferSize) /*********************************************************************************************************************************** Internal functions ***********************************************************************************************************************************/ // Clean up mem contexts after an error. Should only be called from error handling routines. FN_EXTERN void memContextClean(unsigned int tryDepth, bool fatal); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/regExp.c0000664000175100017510000001412315240547630017071 0ustar00runnerrunner/*********************************************************************************************************************************** Regular Expression Handler ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/regExp.h" /*********************************************************************************************************************************** Contains information about the regular expression handler ***********************************************************************************************************************************/ struct RegExp { regex_t regExp; }; /*********************************************************************************************************************************** Free regular expression ***********************************************************************************************************************************/ static void regExpFreeResource(THIS_VOID) { THIS(RegExp); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(REGEXP, this); FUNCTION_TEST_END(); ASSERT(this != NULL); regfree(&this->regExp); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Handle errors ***********************************************************************************************************************************/ static void regExpError(const int error) { char buffer[4096]; regerror(error, NULL, buffer, sizeof(buffer)); THROW(FormatError, buffer); } static void regExpErrorCheck(const int error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, error); FUNCTION_TEST_END(); if (error != 0 && error != REG_NOMATCH) regExpError(error); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN RegExp * regExpNew(const String *const expression) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_END(); ASSERT(expression != NULL); OBJ_NEW_BEGIN(RegExp, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (RegExp){{0}}; // Extra braces are required for older gcc versions // Compile the regexp and process errors int result = 0; if ((result = regcomp(&this->regExp, strZ(expression), REG_EXTENDED)) != 0) regExpError(result); // Set free callback to ensure cipher context is freed memContextCallbackSet(objMemContext(this), regExpFreeResource, this); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(REGEXP, this); } /**********************************************************************************************************************************/ FN_EXTERN bool regExpMatch(RegExp *const this, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(REGEXP, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(string != NULL); // Test for a match regmatch_t matchPtr; const int result = regexec(&this->regExp, strZ(string), 1, &matchPtr, 0); // Check for an error regExpErrorCheck(result); FUNCTION_TEST_RETURN(BOOL, result == 0); } /**********************************************************************************************************************************/ FN_EXTERN bool regExpMatchOne(const String *const expression, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(expression != NULL); ASSERT(string != NULL); bool result; MEM_CONTEXT_TEMP_BEGIN() { result = regExpMatch(regExpNew(expression), string); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN String * regExpPrefix(const String *const expression) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_END(); String *result = NULL; // Only generate prefix if expression is defined and has a beginning anchor if (expression != NULL && strZ(expression)[0] == '^') { const char *const expressionZ = strZ(expression); const size_t expressionSize = strSize(expression); unsigned int expressionIdx = 1; for (; expressionIdx < expressionSize; expressionIdx++) { char expressionChr = expressionZ[expressionIdx]; // Search for characters that will end the prefix if (expressionChr == '.' || expressionChr == '^' || expressionChr == '$' || expressionChr == '*' || expressionChr == '+' || expressionChr == '-' || expressionChr == '?' || expressionChr == '(' || expressionChr == '[' || expressionChr == '{' || expressionChr == ' ' || expressionChr == '|' || expressionChr == '\\') { break; } } // Will there be any characters in the prefix? if (expressionIdx > 1) { // Search the rest of the string for another begin anchor unsigned int anchorIdx = expressionIdx; for (; anchorIdx < expressionSize; anchorIdx++) { // [^ and \^ are not begin anchors if (expressionZ[anchorIdx] == '^' && expressionZ[anchorIdx - 1] != '[' && expressionZ[anchorIdx - 1] != '\\') break; } // If no other begin anchor was found then the prefix is usable if (anchorIdx == expressionSize) result = strSubN(expression, 1, expressionIdx - 1); } } FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/regExp.h0000664000175100017510000000617015240547630017101 0ustar00runnerrunner/*********************************************************************************************************************************** Regular Expression Handler ***********************************************************************************************************************************/ #ifndef COMMON_REGEXP_H #define COMMON_REGEXP_H /*********************************************************************************************************************************** RegExp object ***********************************************************************************************************************************/ typedef struct RegExp RegExp; #include "common/type/object.h" #include "common/type/string.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN RegExp *regExpNew(const String *expression); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Match on a regular expression FN_EXTERN bool regExpMatch(RegExp *this, const String *string); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Match a regular expression in one call for brevity FN_EXTERN bool regExpMatchOne(const String *expression, const String *string); // Return the common prefix of a regular expression, if it has one. The common prefix consists of fixed characters that must always // be found at the beginning of the string to be matched. Escaped characters will not be included in the prefix. If there is no // usable prefix then NULL is returned. FN_EXTERN String *regExpPrefix(const String *expression); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void regExpFree(RegExp *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_REGEXP_TYPE \ RegExp * #define FUNCTION_LOG_REGEXP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "RegExp", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/stackTrace.c0000664000175100017510000003465015240547630017732 0ustar00runnerrunner/*********************************************************************************************************************************** Stack Trace Handler ***********************************************************************************************************************************/ #include #include #include #include #ifdef HAVE_LIBBACKTRACE #include #endif #include "common/assert.h" #include "common/macro.h" #include "common/stackTrace.h" /*********************************************************************************************************************************** Max call stack depth ***********************************************************************************************************************************/ #define STACK_TRACE_MAX 128 /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ // Stack trace function data typedef struct StackTraceData { const char *fileName; const char *functionName; unsigned int fileLine; LogLevel functionLogLevel; unsigned int tryDepth; bool paramOverflow; bool paramLog; char *param; size_t paramSize; } StackTraceData; static struct StackTraceLocal { int stackSize; // Stack size StackTraceData stack[STACK_TRACE_MAX]; // Stack data char functionParamBuffer[32 * 1024]; // Buffer to hold function parameters #ifdef HAVE_LIBBACKTRACE struct backtrace_state *backTraceState; // Backtrace state struct #endif } stackTraceLocal; /**********************************************************************************************************************************/ #ifdef DEBUG static struct StackTraceTestLocal { bool testFlag; // Don't log in parameter logging functions to avoid recursion } stackTraceTestLocal = {.testFlag = true}; FN_EXTERN void stackTraceTestStart(void) { stackTraceTestLocal.testFlag = true; } FN_EXTERN void stackTraceTestStop(void) { stackTraceTestLocal.testFlag = false; } FN_EXTERN bool stackTraceTest(void) { return stackTraceTestLocal.testFlag; } FN_EXTERN void stackTraceTestFileLineSet(unsigned int fileLine) { ASSERT(stackTraceLocal.stackSize > 0); stackTraceLocal.stack[stackTraceLocal.stackSize - 1].fileLine = fileLine; } #endif /**********************************************************************************************************************************/ FN_EXTERN LogLevel stackTracePush(const char *const fileName, const char *const functionName, const LogLevel functionLogLevel) { ASSERT(stackTraceLocal.stackSize < STACK_TRACE_MAX - 1); // Set function info StackTraceData *const data = &stackTraceLocal.stack[stackTraceLocal.stackSize]; *data = (StackTraceData) { .fileName = fileName, .functionName = functionName, .tryDepth = errorTryDepth(), }; // Set param pointer if (stackTraceLocal.stackSize == 0) { data->param = stackTraceLocal.functionParamBuffer; data->functionLogLevel = functionLogLevel; } else { StackTraceData *const dataPrior = &stackTraceLocal.stack[stackTraceLocal.stackSize - 1]; data->param = dataPrior->param + dataPrior->paramSize + 1; // Log level cannot be lower than the prior function if (functionLogLevel < dataPrior->functionLogLevel) data->functionLogLevel = dataPrior->functionLogLevel; else data->functionLogLevel = functionLogLevel; } stackTraceLocal.stackSize++; return data->functionLogLevel; } /**********************************************************************************************************************************/ static const char * stackTraceParamIdx(const int stackIdx) { ASSERT(stackTraceLocal.stackSize > 0); ASSERT(stackIdx < stackTraceLocal.stackSize); StackTraceData *const data = &stackTraceLocal.stack[stackIdx]; if (data->paramLog) { if (data->paramOverflow) return "buffer full - parameters not available"; if (data->paramSize == 0) return "void"; return data->param; } // If no parameters return the log level required to get them #define LOG_LEVEL_REQUIRED " log level required for parameters" return data->functionLogLevel == logLevelTrace ? "trace" LOG_LEVEL_REQUIRED : "debug" LOG_LEVEL_REQUIRED; } FN_EXTERN const char * stackTraceParam(void) { return stackTraceParamIdx(stackTraceLocal.stackSize - 1); } /**********************************************************************************************************************************/ FN_EXTERN char * stackTraceParamBuffer(const char *const paramName) { ASSERT(stackTraceLocal.stackSize > 0); StackTraceData *const data = &stackTraceLocal.stack[stackTraceLocal.stackSize - 1]; const size_t paramNameSize = strlen(paramName); // Make sure that adding this parameter will not overflow the buffer if ((size_t)(data->param - stackTraceLocal.functionParamBuffer) + data->paramSize + paramNameSize + 4 > sizeof(stackTraceLocal.functionParamBuffer) - (STACK_TRACE_PARAM_MAX * 2)) { // Set overflow to true data->paramOverflow = true; // There's no way to stop the parameter from being formatted so we reserve a space at the end where the format can safely // take place and not disturb the rest of the buffer. Hopefully overflows just won't happen but we need to be prepared in // case of runaway recursion or some other issue that fills the buffer because we don't want a segfault. return stackTraceLocal.functionParamBuffer + sizeof(stackTraceLocal.functionParamBuffer) - STACK_TRACE_PARAM_MAX; } // Add a comma if a parameter is already in the list if (data->paramSize != 0) { data->param[data->paramSize++] = ','; data->param[data->paramSize++] = ' '; } // Add the parameter name memcpy(data->param + data->paramSize, paramName, paramNameSize); data->paramSize += paramNameSize; // Add param/value separator data->param[data->paramSize++] = ':'; data->param[data->paramSize++] = ' '; return data->param + data->paramSize; } /**********************************************************************************************************************************/ FN_EXTERN void stackTraceParamAdd(const size_t bufferSize) { ASSERT(stackTraceLocal.stackSize > 0); StackTraceData *data = &stackTraceLocal.stack[stackTraceLocal.stackSize - 1]; if (!data->paramOverflow) data->paramSize += bufferSize; } /**********************************************************************************************************************************/ FN_EXTERN void stackTraceParamLog(void) { ASSERT(stackTraceLocal.stackSize > 0); stackTraceLocal.stack[stackTraceLocal.stackSize - 1].paramLog = true; } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void stackTracePop(const char *const fileName, const char *const functionName, const bool test) { ASSERT(stackTraceLocal.stackSize > 0); if (!test || stackTraceTest()) { stackTraceLocal.stackSize--; StackTraceData *data = &stackTraceLocal.stack[stackTraceLocal.stackSize]; if (strcmp(data->fileName, fileName) != 0 || strcmp(data->functionName, functionName) != 0) THROW_FMT(AssertError, "popping %s:%s but expected %s:%s", fileName, functionName, data->fileName, data->functionName); } } #else FN_EXTERN void stackTracePop(void) { stackTraceLocal.stackSize--; } #endif /*********************************************************************************************************************************** Stack trace format ***********************************************************************************************************************************/ static FN_PRINTF(4, 5) size_t stackTraceFmt(char *const buffer, const size_t bufferSize, const size_t bufferUsed, const char *const format, ...) { va_list argumentList; va_start(argumentList, format); const int result = vsnprintf( buffer + bufferUsed, bufferUsed < bufferSize ? bufferSize - bufferUsed : 0, format, argumentList); va_end(argumentList); return (size_t)result; } /*********************************************************************************************************************************** Helper to trim off extra path before the src path ***********************************************************************************************************************************/ static const char * stackTraceTrimSrc(const char *const fileName) { const char *const src = strstr(fileName, "src/"); return src == NULL ? fileName : src + 4; } /**********************************************************************************************************************************/ #ifdef HAVE_LIBBACKTRACE typedef struct StackTraceBackData { bool firstCall; bool firstLine; int stackIdx; size_t result; char *const buffer; const size_t bufferSize; } StackTraceBackData; // Callback to add backtrace data when available static int stackTraceBackCallback( void *const dataVoid, const uintptr_t pc, const char *fileName, const int fileLine, const char *const functionName) { (void)pc; StackTraceBackData *const data = dataVoid; // Catch any unset parameters which indicates the debug data is not available if (fileName == NULL || fileLine == 0 || functionName == NULL) { // If this is the first call then stop because the top of the backtrace must be one of our functions if (data->firstCall) return true; // Else return but do not stop data->firstCall = false; return false; } // Reset first call data->firstCall = false; // If the function name matches combine backtrace data with stack data if (data->stackIdx >= 0 && strcmp(functionName, stackTraceLocal.stack[data->stackIdx].functionName) == 0) { data->result += stackTraceFmt( data->buffer, data->bufferSize, data->result, "%s%s:%s:%d:(%s)", data->firstLine ? "" : "\n", stackTraceTrimSrc(stackTraceLocal.stack[data->stackIdx].fileName), functionName, fileLine, stackTraceParamIdx(data->stackIdx)); data->stackIdx--; } // Else just use stack data. Skip any functions in the error module since they are not useful for the user else { fileName = stackTraceTrimSrc(fileName); if (strcmp(fileName, "common/error/error.c") == 0) return false; data->result += stackTraceFmt( data->buffer, data->bufferSize, data->result, "%s%s:%s:%d:(no parameters available)", data->firstLine ? "" : "\n", fileName, functionName, fileLine); } // Reset first line data->firstLine = false; // Stop when the main function has been processed return strcmp(functionName, "main") == 0; } // Dummy error callback. If there is an error just generate the default stack trace. static void stackTraceBackErrorCallback(void *const data, const char *const msg, const int errnum) { (void)data; (void)msg; (void)errnum; } #endif FN_EXTERN size_t stackTraceToZ( char *const buffer, const size_t bufferSize, const char *fileName, const char *const functionName, const unsigned int fileLine) { #ifdef HAVE_LIBBACKTRACE // Attempt to use backtrace data StackTraceBackData data = { .firstCall = true, .firstLine = true, .stackIdx = stackTraceLocal.stackSize - 1, .result = 0, .buffer = buffer, .bufferSize = bufferSize, }; if (stackTraceLocal.backTraceState == NULL) stackTraceLocal.backTraceState = backtrace_create_state(NULL, false, NULL, NULL); backtrace_full(stackTraceLocal.backTraceState, 2, stackTraceBackCallback, stackTraceBackErrorCallback, &data); if (data.result != 0) return data.result; #endif // HAVE_LIBBACKTRACE size_t result = 0; const char *param = "test build required for parameters"; int stackIdx = stackTraceLocal.stackSize - 1; // If the current function passed in is the same as the top function on the stack then use the parameters for that function fileName = stackTraceTrimSrc(fileName); if (stackTraceLocal.stackSize > 0 && strcmp(fileName, stackTraceTrimSrc(stackTraceLocal.stack[stackIdx].fileName)) == 0 && strcmp(functionName, stackTraceLocal.stack[stackIdx].functionName) == 0) { param = stackTraceParamIdx(stackTraceLocal.stackSize - 1); stackIdx--; } // Output the current function result = stackTraceFmt(buffer, bufferSize, 0, "%s:%s:%u:(%s)", fileName, functionName, fileLine, param); // Output stack if there is anything on it if (stackIdx >= 0) { // If the function passed in was not at the top of the stack then some functions are missing if (stackIdx == stackTraceLocal.stackSize - 1) result += stackTraceFmt(buffer, bufferSize, result, "\n ... function(s) omitted ..."); // Output the rest of the stack for (; stackIdx >= 0; stackIdx--) { const StackTraceData *const data = &stackTraceLocal.stack[stackIdx]; result += stackTraceFmt(buffer, bufferSize, result, "\n%s:%s", stackTraceTrimSrc(data->fileName), data->functionName); if (data->fileLine > 0) result += stackTraceFmt(buffer, bufferSize, result, ":%u", data->fileLine); result += stackTraceFmt(buffer, bufferSize, result, ":(%s)", stackTraceParamIdx(stackIdx)); } } return result; } /**********************************************************************************************************************************/ FN_EXTERN void stackTraceClean(const unsigned int tryDepth, const bool fatal) { (void)fatal; // Cleanup is the same for fatal errors while (stackTraceLocal.stackSize > 0 && stackTraceLocal.stack[stackTraceLocal.stackSize - 1].tryDepth >= tryDepth) stackTraceLocal.stackSize--; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/stackTrace.h0000664000175100017510000000677415240547630017745 0ustar00runnerrunner/*********************************************************************************************************************************** Stack Trace Handler ***********************************************************************************************************************************/ #ifndef COMMON_STACKTRACE_H #define COMMON_STACKTRACE_H #include #include #include "common/logLevel.h" /*********************************************************************************************************************************** Maximum size of a single parameter (including NULL terminator) ***********************************************************************************************************************************/ #define STACK_TRACE_PARAM_MAX 4096 /*********************************************************************************************************************************** Macros to access internal functions ***********************************************************************************************************************************/ #define STACK_TRACE_PUSH(logLevel) \ stackTracePush(__FILE__, __func__, logLevel) #ifdef DEBUG #define STACK_TRACE_POP(test) \ stackTracePop(__FILE__, __func__, test); #else #define STACK_TRACE_POP(test) \ stackTracePop(); #endif /*********************************************************************************************************************************** Internal Functions ***********************************************************************************************************************************/ #ifdef DEBUG // Enable/disable test function logging FN_EXTERN void stackTraceTestStart(void); FN_EXTERN void stackTraceTestStop(void); FN_EXTERN bool stackTraceTest(void); // Set line number for the current function on the stack FN_EXTERN void stackTraceTestFileLineSet(unsigned int fileLine); #else // Must always be valid since it might be needed to compile (though not used) during profiling #define stackTraceTestFileLineSet(fileLine) #endif // Push a new function onto the trace stack FN_EXTERN LogLevel stackTracePush(const char *fileName, const char *functionName, LogLevel functionLogLevel); // Pop a function from the trace stack #ifdef DEBUG FN_EXTERN void stackTracePop(const char *fileName, const char *functionName, bool test); #else FN_EXTERN void stackTracePop(void); #endif // Generate the stack trace FN_EXTERN size_t stackTraceToZ( char *buffer, size_t bufferSize, const char *fileName, const char *functionName, unsigned int fileLine); // Mark that parameters are being logged -- if none appear then the function is void FN_EXTERN void stackTraceParamLog(void); // Get parameters for the top function on the stack FN_EXTERN const char *stackTraceParam(void); // Get the next location where a parameter can be added in the param buffer FN_EXTERN char *stackTraceParamBuffer(const char *param); // Add a parameter to the function on the top of the stack FN_EXTERN void stackTraceParamAdd(size_t bufferSize); // Clean the stack at and below the try level. Called by the error to cleanup the stack when an exception occurs. FN_EXTERN void stackTraceClean(unsigned int tryDepth, bool fatal); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/stat.c0000664000175100017510000001101415240547630016606 0ustar00runnerrunner/*********************************************************************************************************************************** Statistics Collector ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/memContext.h" #include "common/stat.h" #include "common/type/json.h" #include "common/type/list.h" /*********************************************************************************************************************************** Cumulative statistics ***********************************************************************************************************************************/ typedef struct Stat { const String *key; uint64_t total; } Stat; /*********************************************************************************************************************************** Local data ***********************************************************************************************************************************/ static struct { MemContext *memContext; // Mem context to store data in this struct List *stat; // Cumulative stats } statLocalData; /**********************************************************************************************************************************/ FN_EXTERN void statInit(void) { FUNCTION_TEST_VOID(); ASSERT(statLocalData.memContext == NULL); MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(StatLocalData, .childQty = MEM_CONTEXT_QTY_MAX) { statLocalData.memContext = MEM_CONTEXT_NEW(); statLocalData.stat = lstNewP(sizeof(Stat), .sortOrder = sortOrderAsc, .comparator = lstComparatorStr); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get the specified stat. If it doesn't already exist it will be created. ***********************************************************************************************************************************/ static Stat * statGetOrCreate(const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(key != NULL); // Attempt to find the stat Stat *stat = lstFind(statLocalData.stat, &key); // If not found then create it if (stat == NULL) { // Add the new stat MEM_CONTEXT_BEGIN(lstMemContext(statLocalData.stat)) { lstAdd(statLocalData.stat, &(Stat){.key = strDup(key)}); } MEM_CONTEXT_END(); // Sort stats so this stat will be easier to find later lstSort(statLocalData.stat, sortOrderAsc); // The stat might have moved so we'll need to find it and make sure we have the correct pointer stat = lstFind(statLocalData.stat, &key); ASSERT(stat != NULL); } FUNCTION_TEST_RETURN_TYPE_P(Stat, stat); } /**********************************************************************************************************************************/ FN_EXTERN void statInc(const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(statLocalData.memContext != NULL); ASSERT(key != NULL); statGetOrCreate(key)->total++; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * statToJson(void) { FUNCTION_TEST_VOID(); ASSERT(statLocalData.memContext != NULL); String *result = NULL; if (!lstEmpty(statLocalData.stat)) { result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP(.json = result)); for (unsigned int statIdx = 0; statIdx < lstSize(statLocalData.stat); statIdx++) { const Stat *const stat = lstGet(statLocalData.stat, statIdx); jsonWriteObjectBegin(jsonWriteKey(json, stat->key)); jsonWriteUInt64(jsonWriteKeyZ(json, "total"), stat->total); jsonWriteObjectEnd(json); } jsonWriteObjectEnd(json); } MEM_CONTEXT_TEMP_END(); } FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/stat.h0000664000175100017510000000273515240547630016625 0ustar00runnerrunner/*********************************************************************************************************************************** Statistics Collector Collect simple statistics that can be output to a KeyValue for processing and logging. Each stat has a String that identifies it uniquely and will also be used in the output. Individual stats do not need to be created in advance since they will be created as needed at runtime. However, statInit() must be called before any other stat*() functions. NOTE: Statistics are held in a sorted list so there is some cost involved in each lookup. In general, statistics should be used for relatively important or high-latency operations where measurements are critical. For instance, using statistics to count the iterations of a loop would likely be a bad idea. ***********************************************************************************************************************************/ #ifndef COMMON_STAT_H #define COMMON_STAT_H #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize the stats collector FN_EXTERN void statInit(void); // Increment stat by one FN_EXTERN void statInc(const String *key); // Output stats to JSON FN_EXTERN String *statToJson(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/time.c0000664000175100017510000001602315240547630016576 0ustar00runnerrunner/*********************************************************************************************************************************** Time Management ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/time.h" /*********************************************************************************************************************************** Constants describing number of sub-units in an interval ***********************************************************************************************************************************/ #define MSEC_PER_USEC ((TimeMSec)1000) /**********************************************************************************************************************************/ FN_EXTERN TimeMSec timeMSec(void) { FUNCTION_TEST_VOID(); struct timeval currentTime; gettimeofday(¤tTime, NULL); FUNCTION_TEST_RETURN(TIME_MSEC, ((TimeMSec)currentTime.tv_sec * MSEC_PER_SEC) + (TimeMSec)currentTime.tv_usec / MSEC_PER_USEC); } /**********************************************************************************************************************************/ FN_EXTERN void sleepMSec(const TimeMSec sleepMSec) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, sleepMSec); FUNCTION_TEST_END(); if (sleepMSec > 0) { const struct timespec delay = { .tv_sec = (time_t)(sleepMSec / MSEC_PER_SEC), .tv_nsec = (long)(sleepMSec % MSEC_PER_SEC * 1000000), }; nanosleep(&delay, NULL); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void datePartsValid(const int year, const int month, const int day) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, year); FUNCTION_TEST_PARAM(INT, month); FUNCTION_TEST_PARAM(INT, day); FUNCTION_TEST_END(); static const int daysPerMonth[2][12] = { {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, {31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}, }; if (!(year >= 1970 && month > 0 && month <= 12 && day > 0 && day <= daysPerMonth[yearIsLeap(year) ? 1 : 0][month - 1])) THROW_FMT(FormatError, "invalid date %04d-%02d-%02d", year, month, day); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void timePartsValid(const int hour, const int minute, const int second) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, hour); FUNCTION_TEST_PARAM(INT, minute); FUNCTION_TEST_PARAM(INT, second); FUNCTION_TEST_END(); if (!(hour >= 0 && hour < 24 && minute >= 0 && minute < 60 && second >= 0 && second < 60)) THROW_FMT(FormatError, "invalid time %02d:%02d:%02d", hour, minute, second); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void tzPartsValid(const int tzHour, const int tzMinute) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, tzHour); // signed hour part of timezone FUNCTION_TEST_PARAM(INT, tzMinute); // minutes part of timezone FUNCTION_TEST_END(); // Valid time zones range from GMT-12 all the way to GMT+14 (i.e. -1200 and +1400 are the min/max). // ??? This is only a sanity check for basic validity of timezone offset of 15 minute intervals until the timezone // database is implemented. if (!(((tzHour > -12 && tzHour < 14) && (tzMinute % 15 == 0)) || (tzHour == -12 && tzMinute == 0) || (tzHour == 14 && tzMinute == 0))) { THROW_FMT(FormatError, "invalid timezone %02d%02d", tzHour, tzMinute); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN int tzOffsetSeconds(int tzHour, const int tzMinute) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, tzHour); // signed hour part of timezone (e.g. -7) FUNCTION_TEST_PARAM(INT, tzMinute); // minutes part of timezone FUNCTION_TEST_END(); // Validate the timezone hour and minute tzPartsValid(tzHour, tzMinute); int sign = 1; // Preserve the sign and convert the hours to a positive number for calculating seconds if (tzHour < 0) { sign = -1; tzHour = sign * tzHour; } FUNCTION_TEST_RETURN(INT, sign * (tzHour * 3600 + tzMinute * 60)); } /**********************************************************************************************************************************/ FN_EXTERN bool yearIsLeap(const int year) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, year); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, (year % 4 == 0 && year % 100 != 0) || (year % 400 == 0)); } /**********************************************************************************************************************************/ FN_EXTERN int dayOfYear(const int year, const int month, const int day) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, year); FUNCTION_TEST_PARAM(INT, month); FUNCTION_TEST_PARAM(INT, day); FUNCTION_TEST_END(); datePartsValid(year, month, day); static const int cumulativeDaysPerMonth[2][12] = { {0, 31, 59, 90, 120, 151, 181, 212, 243, 273, 304, 334}, {0, 31, 60, 91, 121, 152, 182, 213, 244, 274, 305, 335}, }; FUNCTION_TEST_RETURN(INT, cumulativeDaysPerMonth[yearIsLeap(year) ? 1 : 0][month - 1] + day); } /**********************************************************************************************************************************/ FN_EXTERN time_t epochFromParts( const int year, const int month, const int day, const int hour, const int minute, const int second, const int tzOffsetSecond) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, year); FUNCTION_TEST_PARAM(INT, month); FUNCTION_TEST_PARAM(INT, day); FUNCTION_TEST_PARAM(INT, hour); FUNCTION_TEST_PARAM(INT, minute); FUNCTION_TEST_PARAM(INT, second); FUNCTION_TEST_PARAM(INT, tzOffsetSecond); FUNCTION_TEST_END(); timePartsValid(hour, minute, second); // Return epoch using calculation from https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_16 FUNCTION_TEST_RETURN( TIME, -1 * tzOffsetSecond + second + minute * 60 + hour * 3600 + (dayOfYear(year, month, day) - 1) * 86400 + (year - 1900 - 70) * 31536000 + ((year - 1900 - 69) / 4) * 86400 - ((year - 1900 - 1) / 100) * 86400 + ((year - 1900 + 299) / 400) * 86400); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/time.h0000664000175100017510000000636415240547630016612 0ustar00runnerrunner/*********************************************************************************************************************************** Time Management ***********************************************************************************************************************************/ #ifndef COMMON_TIME_H #define COMMON_TIME_H #include #include #include /*********************************************************************************************************************************** Time types ***********************************************************************************************************************************/ typedef uint64_t TimeMSec; /*********************************************************************************************************************************** Constants describing number of sub-units in an interval ***********************************************************************************************************************************/ #define MSEC_PER_SEC ((TimeMSec)1000) #define SEC_PER_DAY ((time_t)86400) /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Sleep for specified milliseconds FN_EXTERN void sleepMSec(TimeMSec sleepMSec); // Epoch time in milliseconds FN_EXTERN TimeMSec timeMSec(void); // Are the date parts valid? (year >= 1970, month 1-12, day 1-31) FN_EXTERN void datePartsValid(int year, int month, int day); // Are the time parts valid? FN_EXTERN void timePartsValid(int hour, int minute, int second); // Are the timezone offset parts valid? FN_EXTERN void tzPartsValid(int tzHour, int tzMinute); // Given the hours, minutes and sign of a time zone (e.g. -0700 => -7, 0) return the signed number or seconds (e.g. -25200) FN_EXTERN int tzOffsetSeconds(int tzHour, int tzMinute); // Is the year a leap year? FN_EXTERN bool yearIsLeap(int year); // Get days since the beginning of the year (year >= 1970, month 1-12, day 1-31), returns 1-366 FN_EXTERN int dayOfYear(int year, int month, int day); // Return epoch time from date/time parts (year >= 1970, month 1-12, day 1-31, hour 0-23, minute 0-59, second 0-59, tzOffsetSecond // is the number of seconds plus or minus (+/-) the provided time - if 0, then the datetime is assumed to be UTC) FN_EXTERN time_t epochFromParts(int year, int month, int day, int hour, int minute, int second, int tzOffsetSecond); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_TIME_MSEC_TYPE \ TimeMSec #define FUNCTION_LOG_TIME_MSEC_FORMAT(value, buffer, bufferSize) \ cvtUInt64ToZ(value, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/0000775000175100017510000000000015240547630016453 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/blob.c0000664000175100017510000000511115240547630017533 0ustar00runnerrunner/*********************************************************************************************************************************** Blob Handler ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/blob.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Blob { char *block; // Current block for writing size_t pos; // Position in current block }; /**********************************************************************************************************************************/ FN_EXTERN Blob * blbNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(Blob, .allocQty = MEM_CONTEXT_QTY_MAX) { *this = (Blob) { .block = memNew(BLOB_BLOCK_SIZE), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(BLOB, this); } /**********************************************************************************************************************************/ FN_EXTERN const void * blbAdd(Blob *const this, const void *const data, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BLOB, this); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); void *result = NULL; MEM_CONTEXT_OBJ_BEGIN(this) { // If data is >= block size then allocate a special block for the data if (size >= BLOB_BLOCK_SIZE) { result = memNew(size); memcpy(result, data, size); } else { // If data will fit in the current block if (BLOB_BLOCK_SIZE - this->pos >= size) { result = (uint8_t *)this->block + this->pos; memcpy(result, data, size); this->pos += size; } // Else allocate a new block else { this->block = memNew(BLOB_BLOCK_SIZE); result = this->block; memcpy(result, data, size); this->pos = size; } } } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN_P(VOID, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/blob.h0000664000175100017510000000627215240547630017551 0ustar00runnerrunner/*********************************************************************************************************************************** Blob Handler Efficiently store large numbers of small allocations by packing them onto larger blocks. Note that the allocations will not be aligned unless all of them are aligned. A usage example is to store zero-terminated strings, which do not need to be aligned, and where alignment might waste significant space. Note that this code is fairly primitive, and this should probably be a capability of memory contexts. However, for now it reduces memory requirements for large numbers of zero-terminated strings. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_BLOB_H #define COMMON_TYPE_BLOB_H /*********************************************************************************************************************************** Size of blocks allocated for blob data ***********************************************************************************************************************************/ #ifndef BLOB_BLOCK_SIZE #define BLOB_BLOCK_SIZE (64 * 1024) #endif /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Blob Blob; #include "common/type/object.h" /*********************************************************************************************************************************** Constructor ***********************************************************************************************************************************/ FN_EXTERN Blob *blbNew(void); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add data to the blob FN_EXTERN const void *blbAdd(Blob *this, const void *data, size_t size); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void strBlbFree(Blob *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_BLOB_TYPE \ Blob * #define FUNCTION_LOG_BLOB_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Blob", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/buffer.c0000664000175100017510000002745415240547630020104 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer Handler ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/type/buffer.h" /*********************************************************************************************************************************** Constant buffers that are generally useful ***********************************************************************************************************************************/ BUFFER_STRDEF_EXTERN(BRACEL_BUF, "{"); BUFFER_STRDEF_EXTERN(BRACER_BUF, "}"); BUFFER_STRDEF_EXTERN(BRACKETL_BUF, "["); BUFFER_STRDEF_EXTERN(BRACKETR_BUF, "]"); BUFFER_STRDEF_EXTERN(COMMA_BUF, ","); BUFFER_STRDEF_EXTERN(EQ_BUF, "="); BUFFER_STRDEF_EXTERN(LF_BUF, "\n"); BUFFER_STRDEF_EXTERN(QUOTED_BUF, "\""); /*********************************************************************************************************************************** Contains information about the buffer ***********************************************************************************************************************************/ struct Buffer { BufferPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufNew(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(Buffer, .allocQty = 1) { *this = (Buffer) { .pub = { .sizeAlloc = size, .size = size, }, }; // Allocate buffer if (size > 0) this->pub.buffer = memNew(this->pub.sizeAlloc); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufNewC(const void *const buffer, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Create object and copy data Buffer *const this = bufNew(size); memcpy(this->pub.buffer, buffer, bufSize(this)); this->pub.used = bufSize(this); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufNewDecode(const EncodingType type, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); Buffer *const this = bufNew(decodeToBinSize(type, strZ(string))); decodeToBin(type, strZ(string), bufPtr(this)); bufUsedSet(this, bufSize(this)); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufDup(const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Create object and copy data Buffer *const this = bufNew(bufUsed(buffer)); if (bufUsed(buffer) != 0) memcpy(this->pub.buffer, buffer->pub.buffer, bufSize(this)); this->pub.used = bufSize(this); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufCat(Buffer *const this, const Buffer *const cat) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(BUFFER, cat); FUNCTION_TEST_END(); ASSERT(this != NULL); if (cat != NULL) bufCatC(this, cat->pub.buffer, 0, bufUsed(cat)); FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufCatC(Buffer *const this, const uint8_t *const cat, const size_t catOffset, const size_t catSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM_P(BYTEDATA, cat); FUNCTION_TEST_PARAM(SIZE, catOffset); FUNCTION_TEST_PARAM(SIZE, catSize); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(catSize == 0 || cat != NULL); if (catSize > 0) { if (bufUsed(this) + catSize > bufSize(this)) bufResize(this, bufUsed(this) + catSize); // Just here to silence nonnull warnings from clang static analyzer ASSERT(bufPtr(this) != NULL); memcpy(bufPtr(this) + bufUsed(this), cat + catOffset, catSize); this->pub.used += catSize; } FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufCatSub(Buffer *const this, const Buffer *const cat, const size_t catOffset, const size_t catSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(BUFFER, cat); FUNCTION_TEST_PARAM(SIZE, catOffset); FUNCTION_TEST_PARAM(SIZE, catSize); FUNCTION_TEST_END(); ASSERT(this != NULL); if (cat != NULL) { ASSERT(catOffset <= bufUsed(cat)); ASSERT(catSize <= bufUsed(cat) - catOffset); bufCatC(this, cat->pub.buffer, catOffset, catSize); } FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN bool bufEq(const Buffer *const this, const Buffer *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(BUFFER, compare); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(compare != NULL); if (bufUsed(this) == bufUsed(compare)) FUNCTION_TEST_RETURN(BOOL, memcmp(bufPtrConst(this), bufPtrConst(compare), bufUsed(compare)) == 0); FUNCTION_TEST_RETURN(BOOL, false); } /**********************************************************************************************************************************/ FN_EXTERN const uint8_t * bufFind(const Buffer *const this, const Buffer *const find, const BufFindParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(BUFFER, find); FUNCTION_TEST_PARAM_P(VOID, param.begin); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(find != NULL); ASSERT(param.begin == NULL || (param.begin >= bufPtrConst(this) && param.begin - bufPtrConst(this) <= (off_t)bufUsed(this))); const void *result = NULL; if (bufUsed(this) >= bufUsed(find)) { const uint8_t *haystack = param.begin != NULL ? param.begin : bufPtrConst(this); unsigned int findIdx = (unsigned int)(haystack - bufPtrConst(this)); for (; findIdx <= bufUsed(this) - bufUsed(find); findIdx++) { if (memcmp(haystack, bufPtrConst(find), bufSize(find)) == 0) { result = haystack; break; } haystack++; } } FUNCTION_TEST_RETURN_CONST_P(BYTEDATA, result); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * bufResize(Buffer *const this, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); // Only resize if it the new size is different if (bufSizeAlloc(this) != size) { // If new size is zero then free memory if allocated if (size == 0) { // When setting size down to 0 the buffer should always be allocated ASSERT(bufPtrConst(this) != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { memFree(bufPtr(this)); } MEM_CONTEXT_OBJ_END(); this->pub.buffer = NULL; this->pub.sizeAlloc = 0; } // Else allocate or resize else { MEM_CONTEXT_OBJ_BEGIN(this) { if (bufPtrConst(this) == NULL) this->pub.buffer = memNew(size); else this->pub.buffer = memResize(bufPtr(this), size); } MEM_CONTEXT_OBJ_END(); this->pub.sizeAlloc = size; } if (bufUsed(this) > bufSizeAlloc(this)) this->pub.used = bufSizeAlloc(this); if (!bufSizeLimit(this)) this->pub.size = bufSizeAlloc(this); else if (bufSize(this) > bufSizeAlloc(this)) this->pub.size = bufSizeAlloc(this); } FUNCTION_TEST_RETURN(BUFFER, this); } /**********************************************************************************************************************************/ FN_EXTERN void bufLimitClear(Buffer *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); this->pub.sizeLimit = false; this->pub.size = bufSizeAlloc(this); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void bufLimitSet(Buffer *const this, const size_t limit) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(SIZE, limit); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(limit <= bufSizeAlloc(this)); ASSERT(limit >= bufUsed(this)); this->pub.size = limit; this->pub.sizeLimit = limit != bufSizeAlloc(this); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void bufUsedInc(Buffer *const this, const size_t inc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(SIZE, inc); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(bufUsed(this) + inc <= bufSize(this)); this->pub.used += inc; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void bufUsedSet(Buffer *const this, const size_t used) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_PARAM(SIZE, used); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(used <= bufSize(this)); this->pub.used = used; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void bufUsedZero(Buffer *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, this); FUNCTION_TEST_END(); ASSERT(this != NULL); this->pub.used = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void bufToLog(const Buffer *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{used: %zu, size: %zu", bufUsed(this), bufSize(this)); if (bufSizeLimit(this)) strStcFmt(debugLog, ", sizeAlloc: %zu", bufSizeAlloc(this)); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/buffer.h0000664000175100017510000002506315240547630020103 0ustar00runnerrunner/*********************************************************************************************************************************** Buffer Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_BUFFER_H #define COMMON_TYPE_BUFFER_H /*********************************************************************************************************************************** Buffer object ***********************************************************************************************************************************/ typedef struct Buffer Buffer; #include "common/memContext.h" #include "common/type/object.h" #include "common/type/string.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Buffer *bufNew(size_t size); // Create a new buffer from a C buffer FN_EXTERN Buffer *bufNewC(const void *buffer, size_t size); // Create a new buffer from a string encoded with the specified type FN_EXTERN Buffer *bufNewDecode(EncodingType type, const String *string); // Duplicate a buffer FN_EXTERN Buffer *bufDup(const Buffer *buffer); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct BufferPub { size_t sizeAlloc; // Allocated size of the buffer size_t size; // Reported size of the buffer bool sizeLimit; // Is the size limited to make the buffer appear smaller? size_t used; // Amount of buffer used uint8_t *buffer; // Buffer } BufferPub; // Amount of the buffer actually used. This will be updated automatically when possible but if the buffer is modified by using // bufPtr() then the user is responsible for updating used. FN_INLINE_ALWAYS size_t bufUsed(const Buffer *const this) { return THIS_PUB(Buffer)->used; } // Is the buffer empty? FN_INLINE_ALWAYS bool bufEmpty(const Buffer *const this) { return bufUsed(this) == 0; } // Buffer size FN_INLINE_ALWAYS size_t bufSize(const Buffer *const this) { return THIS_PUB(Buffer)->size; } // Is the buffer full? FN_INLINE_ALWAYS bool bufFull(const Buffer *const this) { return bufUsed(this) == bufSize(this); } // Buffer pointer FN_INLINE_ALWAYS uint8_t * bufPtr(Buffer *const this) { return THIS_PUB(Buffer)->buffer; } // Const buffer pointer FN_INLINE_ALWAYS const uint8_t * bufPtrConst(const Buffer *const this) { return THIS_PUB(Buffer)->buffer; } // Remaining space in the buffer FN_INLINE_ALWAYS size_t bufRemains(const Buffer *const this) { return bufSize(this) - bufUsed(this); } // Pointer to remaining buffer space (after used space) FN_INLINE_ALWAYS uint8_t * bufRemainsPtr(Buffer *const this) { return bufPtr(this) + bufUsed(this); } // Allocated buffer size. This may be different from bufSize() if a limit has been set. FN_INLINE_ALWAYS size_t bufSizeAlloc(const Buffer *const this) { return THIS_PUB(Buffer)->sizeAlloc; } // Is the size limited to make the buffer appear smaller? FN_INLINE_ALWAYS bool bufSizeLimit(const Buffer *const this) { return THIS_PUB(Buffer)->sizeLimit; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Append the contents of another buffer FN_EXTERN Buffer *bufCat(Buffer *this, const Buffer *cat); // Append a C buffer FN_EXTERN Buffer *bufCatC(Buffer *this, const uint8_t *cat, size_t catOffset, size_t catSize); // Append a subset of another buffer FN_EXTERN Buffer *bufCatSub(Buffer *this, const Buffer *cat, size_t catOffset, size_t catSize); // Are two buffers equal? FN_EXTERN bool bufEq(const Buffer *this, const Buffer *compare); // Find a buffer in another buffer typedef struct BufFindParam { VAR_PARAM_HEADER; const uint8_t *begin; // Begin find from this address } BufFindParam; #define bufFindP(this, find, ...) \ bufFind(this, find, (BufFindParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN const uint8_t *bufFind(const Buffer *this, const Buffer *find, BufFindParam param); // Move to a new parent mem context FN_INLINE_ALWAYS Buffer * bufMove(Buffer *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Resize the buffer FN_EXTERN Buffer *bufResize(Buffer *this, size_t size); // Manage buffer limits FN_EXTERN void bufLimitClear(Buffer *this); FN_EXTERN void bufLimitSet(Buffer *this, size_t limit); FN_EXTERN void bufUsedInc(Buffer *this, size_t inc); FN_EXTERN void bufUsedSet(Buffer *this, size_t used); FN_EXTERN void bufUsedZero(Buffer *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void bufFree(Buffer *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for constant buffers Frequently used constant buffers can be declared with these macros at compile time rather than dynamically at run time. Note that buffers created in this way are declared as const so can't be modified or freed by the buf*() methods. Casting to Buffer * will result in a segfault. By convention all buffer constant identifiers are appended with _BUF. ***********************************************************************************************************************************/ // This struct must be kept in sync with BufferPub (except for const qualifiers) typedef struct BufferPubConst { size_t sizeAlloc; // Allocated size of the buffer size_t size; // Reported size of the buffer bool sizeLimit; // Is the size limited to make the buffer appear smaller? size_t used; // Amount of buffer used const uint8_t *buffer; // Buffer } BufferPubConst; // Create a buffer constant inline from an uint8_t[] #define BUF(bufferParam, sizeParam) \ ((const Buffer *)&(const BufferPubConst){.size = sizeParam, .used = sizeParam, .buffer = (const uint8_t *)bufferParam}) // Create a buffer constant inline from a non-constant zero-terminated string #define BUFSTRZ(stringz) \ BUF((const uint8_t *)stringz, strlen(stringz)) // Create a buffer constant inline from a String #define BUFSTR(string) \ BUF((const uint8_t *)strZ(string), strSize(string)) // Create a buffer constant inline from a constant zero-terminated string #define BUFSTRDEF(stringdef) \ BUF((const uint8_t *)stringdef, (sizeof(stringdef) - 1)) // Used to define buffer constants that will be externed using BUFFER_DECLARE(). Must be used in a .c file. #define BUFFER_EXTERN(name, ...) \ static const uint8_t name##_RAW[] = {__VA_ARGS__}; \ VR_EXTERN_DEFINE const Buffer *const name = BUF(name##_RAW, sizeof(name##_RAW)); // Used to define String Buffer constants that will be externed using BUFFER_DECLARE(). Must be used in a .c file. #define BUFFER_STRDEF_EXTERN(name, string) \ VR_EXTERN_DEFINE const Buffer *const name = BUFSTRDEF(string) // Used to define String Buffer constants that will be local to the .c file. Must be used in a .c file. #define BUFFER_STRDEF_STATIC(name, string) \ static const Buffer *const name = BUFSTRDEF(string) // Used to declare externed Buffer constants defined with BUFFER*EXTERN(). Must be used in a .h file. #define BUFFER_DECLARE(name) \ VR_EXTERN_DECLARE const Buffer *const name /*********************************************************************************************************************************** Constant buffers that are generally useful ***********************************************************************************************************************************/ BUFFER_DECLARE(BRACEL_BUF); BUFFER_DECLARE(BRACER_BUF); BUFFER_DECLARE(BRACKETL_BUF); BUFFER_DECLARE(BRACKETR_BUF); BUFFER_DECLARE(COMMA_BUF); BUFFER_DECLARE(EQ_BUF); BUFFER_DECLARE(LF_BUF); BUFFER_DECLARE(QUOTED_BUF); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void bufToLog(const Buffer *this, StringStatic *debugLog); #define FUNCTION_LOG_BUFFER_TYPE \ Buffer * #define FUNCTION_LOG_BUFFER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, bufToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/convert.c0000664000175100017510000007001415240547630020301 0ustar00runnerrunner/*********************************************************************************************************************************** Convert C Types ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include #include "common/debug.h" #include "common/macro.h" #include "common/time.h" #include "common/type/convert.h" /*********************************************************************************************************************************** Check results of strto*() function for: * leading/trailing spaces * invalid characters * blank string * error in errno ***********************************************************************************************************************************/ static void cvtZToIntValid(const int errNo, const int base, const char *const value, const char *const endPtr, const char *const type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, errNo); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(STRINGZ, endPtr); FUNCTION_TEST_PARAM(STRINGZ, type); FUNCTION_TEST_END(); ASSERT(value != NULL); ASSERT(endPtr != NULL); if (errNo != 0 || *value == '\0' || isspace(*value) || *endPtr != '\0') THROW_FMT(FormatError, "unable to convert base %d string '%s' to %s", base, value, type); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Convert zero-terminated string to int64 and validate result ***********************************************************************************************************************************/ static int64_t cvtZToInt64Internal(const char *const value, const char *const type, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(STRINGZ, type); FUNCTION_TEST_END(); ASSERT(value != NULL); ASSERT(type != NULL); // Convert from string errno = 0; char *endPtr = NULL; const int64_t result = strtoll(value, &endPtr, base); // Validate the result cvtZToIntValid(errno, base, value, endPtr, type); FUNCTION_TEST_RETURN(INT64, result); } /*********************************************************************************************************************************** Convert zero-terminated string to uint64 and validate result ***********************************************************************************************************************************/ static uint64_t cvtZToUInt64Internal(const char *const value, const char *const type, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(STRINGZ, type); FUNCTION_TEST_END(); ASSERT(value != NULL); ASSERT(type != NULL); // Convert from string errno = 0; char *endPtr = NULL; const uint64_t result = strtoull(value, &endPtr, base); // Validate the result cvtZToIntValid(errno, base, value, endPtr, type); FUNCTION_TEST_RETURN(UINT64, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtBoolToZ(const bool value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%s", cvtBoolToConstZ(value)); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN const char * cvtBoolToConstZ(bool value) { return value ? TRUE_Z : FALSE_Z; } /*********************************************************************************************************************************** Round an integer contained in a string ***********************************************************************************************************************************/ static size_t cvtRound(size_t result, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, result); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferSize > 0); for (int roundIdx = (int)result - 2; roundIdx >= 0; roundIdx--) { // Round when the rounding digit is >= 5 (current digit needs to be incremented) or the current digit > 9 (prior carry // overflowed and needs to be carried again) if (((roundIdx == (int)result - 2) && buffer[roundIdx + 1] >= '5') || buffer[roundIdx] > '9') { // Carry to the prior digit if (buffer[roundIdx] >= '9') { // If this is the first digit then add a new prior digit to carry to. Since it will start as zero we can just // set it to one. if (roundIdx == 0) { if (result + 1 >= bufferSize) THROW(AssertError, "buffer overflow"); memmove(buffer + 1, buffer, ++result); buffer[1] = '0'; buffer[0] = '1'; } // Else set current digit to zero and carry to prior digit. An overflow is handled on the next iteration. else { buffer[roundIdx] = '0'; buffer[roundIdx - 1]++; } } // Else increment current digit else buffer[roundIdx]++; } // Else stop rounding else break; } // Remove rightmost digit used to start rounding result--; buffer[result] = '\0'; // Return string length FUNCTION_TEST_RETURN(SIZE, result); } /*********************************************************************************************************************************** Separate the fractional part of an integer container in a string ***********************************************************************************************************************************/ static size_t cvtFraction(size_t result, const unsigned int precision, const bool trim, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, result); FUNCTION_TEST_PARAM(UINT, precision); FUNCTION_TEST_PARAM(BOOL, trim); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferSize > 0); // Add decimal point if (result + 1 >= bufferSize) THROW(AssertError, "buffer overflow"); memmove(buffer + result - precision + 1, buffer + result - precision, precision + 1); buffer[result - precision] = '.'; buffer[++result] = '\0'; // Strip off any final 0s and the decimal point if there are no non-zero digits after it if (trim) { char *end = buffer + result - 1; while (*end == '0' || *end == '.') { // It should not be possible to go past the beginning because a decimal point is always written ASSERT(end > buffer); end--; if (*(end + 1) == '.') break; } // Zero terminate the string end[1] = 0; // Calculate length result = (size_t)(end - buffer + 1); } // Return string length FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cvtPctToUInt(const uint64_t dividend, const uint64_t divisor) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_END(); ASSERT(dividend <= divisor); // If 100% then return a fixed value to avoid any rounding throwing off the result if (dividend == divisor) FUNCTION_TEST_RETURN(UINT, 10000); // Calculate percentage char buffer[CVT_PCT_BUFFER_SIZE]; size_t size = (size_t)snprintf(buffer, sizeof(buffer), "%04" PRIu64, (uint64_t)((double)dividend / (double)divisor * 100000)); // Round cvtRound(size, buffer, sizeof(buffer)); FUNCTION_TEST_RETURN(UINT, cvtZToUInt(buffer)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtPctToZ(const uint64_t dividend, const uint64_t divisor, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferSize > 0); ASSERT(dividend <= divisor); // Calculate percentage as an integer size_t result = (size_t)snprintf(buffer, bufferSize, "%03u", cvtPctToUInt(dividend, divisor)); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); // Separate fractional part result = cvtFraction(result, 2, false, buffer, bufferSize); // Add percent sign if (result + 1 >= bufferSize) THROW(AssertError, "buffer overflow"); buffer[result++] = '%'; buffer[result] = '\0'; // Return string length FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtDivToZ( const uint64_t dividend, const uint64_t divisor, const unsigned int precision, const bool trim, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_PARAM(UINT, precision); FUNCTION_TEST_PARAM(BOOL, trim); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferSize > 0); // Determine multiplier for precision digits unsigned int multiplier = 1; switch (precision) { case 0: break; case 1: multiplier = 10; break; case 2: multiplier = 100; break; default: CHECK_FMT(AssertError, precision <= 3, "precision %u is invalid", precision); multiplier = 1000; break; } CHECK_FMT(AssertError, dividend <= UINT64_MAX / multiplier, "dividend %" PRIu64 " is too large", dividend); // If possible add a digit for rounding bool round = false; if (dividend <= UINT64_MAX / (multiplier * 10)) { round = true; multiplier *= 10; } // Convert to string size_t result = (size_t)snprintf(buffer, bufferSize, "%0*" PRIu64, (int)precision, dividend * multiplier / divisor); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); // Round if (round) result = cvtRound(result, buffer, bufferSize); // Separate fractional part if (precision > 0) result = cvtFraction(result, precision, trim, buffer, bufferSize); // Return string length FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtIntToZ(const int value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%d", value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN int cvtZToIntBase(const char *const value, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); const int64_t result = cvtZToInt64Internal(value, "int", base); if (result > INT_MAX || result < INT_MIN) THROW_FMT(FormatError, "unable to convert base %d string '%s' to int", base, value); FUNCTION_TEST_RETURN(INT, (int)result); } FN_EXTERN int cvtZToInt(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(INT, cvtZToIntBase(value, 10)); } FN_EXTERN int cvtZSubNToIntBase(const char *const value, const size_t offset, const size_t size, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(INT, base); FUNCTION_TEST_END(); ASSERT(value != NULL); char buffer[CVT_BASE10_BUFFER_SIZE + 1]; ASSERT(size <= CVT_BASE10_BUFFER_SIZE); memcpy(buffer, value + offset, size); buffer[size] = '\0'; FUNCTION_TEST_RETURN(INT, cvtZToIntBase(buffer, base)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtInt64ToZ(const int64_t value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT64, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%" PRId64, value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN int64_t cvtZToInt64Base(const char *const value, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(INT64, cvtZToInt64Internal(value, "int64", base)); } FN_EXTERN int64_t cvtZToInt64(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(INT64, cvtZToInt64Base(value, 10)); } FN_EXTERN int64_t cvtZSubNToInt64Base(const char *const value, const size_t offset, const size_t size, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(INT, base); FUNCTION_TEST_END(); ASSERT(value != NULL); char buffer[CVT_BASE10_BUFFER_SIZE + 1]; ASSERT(size <= CVT_BASE10_BUFFER_SIZE); memcpy(buffer, value + offset, size); buffer[size] = '\0'; FUNCTION_TEST_RETURN(INT64, cvtZToInt64Base(buffer, base)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtModeToZ(const mode_t value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MODE, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%04o", value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN mode_t cvtZToMode(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(MODE, (mode_t)cvtZToUIntBase(value, 8)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtSizeToZ(const size_t value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%zu", value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtTimeToZ(const char *const format, const time_t value, char *const buffer, const size_t bufferSize, const CvtTimeToZParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_PARAM(TIME, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_PARAM(BOOL, param.utc); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Musl libc does not behave like other C libraries when formatting %s as output from gmtime_r() so forbid it entirely, see // https://www.openwall.com/lists/musl/2025/06/02/3 for details. ASSERT(!param.utc || strstr(format, "%s") == NULL); struct tm timePart; // We can ignore this warning here since the format parameter of cvtTimeToZP() is checked #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wformat-nonliteral" const size_t result = strftime( buffer, bufferSize, format, param.utc ? gmtime_r(&value, &timePart) : localtime_r(&value, &timePart)); #pragma GCC diagnostic pop if (result == 0) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } /**********************************************************************************************************************************/ // Helper to convert a time part, e.g. year static int cvtZNToTimePart(const char *const time, const char *const part, const size_t partSize) { int result = 0; int power = 1; for (size_t partIdx = partSize - 1; partIdx < partSize; partIdx--) { if (!isdigit(part[partIdx])) THROW_FMT(FormatError, "invalid date/time %s", time); result += (part[partIdx] - '0') * power; power *= 10; } return result; } FN_EXTERN time_t cvtZToTime(const char *const time) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, time); FUNCTION_TEST_END(); ASSERT(time != NULL); // Validate structure of date/time if (strlen(time) < 19 || time[4] != '-' || time[7] != '-' || time[10] != ' ' || time[13] != ':' || time[16] != ':') THROW_FMT(FormatError, "invalid date/time %s", time); // Parse date/time const int year = cvtZNToTimePart(time, time, 4); const int month = cvtZNToTimePart(time, time + 5, 2); const int day = cvtZNToTimePart(time, time + 8, 2); const int hour = cvtZNToTimePart(time, time + 11, 2); const int minute = cvtZNToTimePart(time, time + 14, 2); const int second = cvtZNToTimePart(time, time + 17, 2); // Confirm date and time parts are valid datePartsValid(year, month, day); timePartsValid(hour, minute, second); // Consume milliseconds when present (they are omitted from the result) const char *part = time + 19; if ((*part == '.' || *part == ',') && strlen(part) >= 2) { part++; while (*part != 0 && isdigit(*part)) part++; } // Add timezone offset when present if ((*part == '+' || *part == '-') && strlen(part) >= 3) { const int offsetHour = cvtZNToTimePart(time, part + 1, 2) * (*part == '-' ? -1 : 1); part += 3; // Offset separator is optional if (*part == ':') part++; // Offset minutes are optional int offsetMinute = 0; if (strlen(part) == 2) { offsetMinute = cvtZNToTimePart(time, part, 2); part += 2; } // Make sure there is nothing left over if (*part != 0) THROW_FMT(FormatError, "invalid date/time %s", time); FUNCTION_TEST_RETURN( TIME, epochFromParts(year, month, day, hour, minute, second, tzOffsetSeconds(offsetHour, offsetMinute))); } // Make sure there is nothing left over if (*part != 0) THROW_FMT(FormatError, "invalid date/time %s", time); // If no timezone was specified then use the current timezone. Set tm_isdst to -1 to force mktime to consider if DST. For // example, if system time is America/New_York then 2019-09-14 20:02:49 was a time in DST so the Epoch value should be // 1568505769 (and not 1568509369 which would be 2019-09-14 21:02:49 - an hour too late). struct tm timePart = { .tm_year = year - 1900, .tm_mon = month - 1, .tm_mday = day, .tm_hour = hour, .tm_min = minute, .tm_sec = second, .tm_isdst = -1, }; FUNCTION_TEST_RETURN(TIME, mktime(&timePart)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtUIntToZ(const unsigned int value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%u", value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN unsigned int cvtZToUIntBase(const char *const value, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); const uint64_t result = cvtZToUInt64Internal(value, "unsigned int", base); // Don't allow negative numbers even though strtoull() does and check max value if (*value == '-' || result > UINT_MAX) THROW_FMT(FormatError, "unable to convert base %d string '%s' to unsigned int", base, value); FUNCTION_TEST_RETURN(UINT, (unsigned int)result); } FN_EXTERN unsigned int cvtZToUInt(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(UINT, cvtZToUIntBase(value, 10)); } FN_EXTERN unsigned int cvtZSubNToUIntBase(const char *const value, const size_t offset, const size_t size, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(INT, base); FUNCTION_TEST_END(); ASSERT(value != NULL); char buffer[CVT_BASE10_BUFFER_SIZE + 1]; ASSERT(size <= CVT_BASE10_BUFFER_SIZE); memcpy(buffer, value + offset, size); buffer[size] = '\0'; FUNCTION_TEST_RETURN(UINT, cvtZToUIntBase(buffer, base)); } /**********************************************************************************************************************************/ FN_EXTERN size_t cvtUInt64ToZ(const uint64_t value, char *const buffer, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_PARAM_P(CHARDATA, buffer); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); const size_t result = (size_t)snprintf(buffer, bufferSize, "%" PRIu64, value); if (result >= bufferSize) THROW(AssertError, "buffer overflow"); FUNCTION_TEST_RETURN(SIZE, result); } FN_EXTERN uint64_t cvtZToUInt64Base(const char *const value, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); const uint64_t result = cvtZToUInt64Internal(value, "uint64", base); // Don't allow negative numbers even though strtoull() does if (*value == '-') THROW_FMT(FormatError, "unable to convert base %d string '%s' to uint64", base, value); FUNCTION_TEST_RETURN(UINT64, result); } FN_EXTERN uint64_t cvtZToUInt64(const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(value != NULL); FUNCTION_TEST_RETURN(UINT64, cvtZToUInt64Base(value, 10)); } FN_EXTERN uint64_t cvtZSubNToUInt64Base(const char *const value, const size_t offset, const size_t size, const int base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(INT, base); FUNCTION_TEST_END(); ASSERT(value != NULL); char buffer[CVT_BASE10_BUFFER_SIZE + 1]; ASSERT(size <= CVT_BASE10_BUFFER_SIZE); memcpy(buffer, value + offset, size); buffer[size] = '\0'; FUNCTION_TEST_RETURN(UINT64, cvtZToUInt64Base(buffer, base)); } /**********************************************************************************************************************************/ FN_EXTERN void cvtUInt64ToVarInt128(uint64_t value, uint8_t *const buffer, size_t *const bufferPos, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(UINT64, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferPos != NULL); ASSERT(bufferSize > *bufferPos); // Keep encoding bytes while the remaining value is greater than 7 bits while (value >= 0x80) { // Encode the lower order 7 bits, adding the continuation bit to indicate there is more data buffer[*bufferPos] = (uint8_t)value | 0x80; // Shift the value to remove bits that have been encoded value >>= 7; // Keep track of size so we know how many bytes to write out (*bufferPos)++; // Make sure the buffer won't overflow if (*bufferPos >= bufferSize) THROW(AssertError, "buffer overflow"); } // Encode the last 7 bits of value buffer[*bufferPos] = (uint8_t)value; (*bufferPos)++; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN uint64_t cvtUInt64FromVarInt128(const uint8_t *const buffer, size_t *const bufferPos, const size_t bufferSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM_P(SIZE, bufferPos); FUNCTION_TEST_PARAM(SIZE, bufferSize); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(bufferPos != NULL); // Decode all bytes uint64_t result = 0; uint8_t byte; for (unsigned int bufferIdx = 0; bufferIdx < CVT_VARINT128_BUFFER_SIZE; bufferIdx++) { // Error if the buffer position is beyond the buffer size if (*bufferPos >= bufferSize) THROW(FormatError, "buffer position is beyond buffer size"); // Get the next encoded byte byte = buffer[*bufferPos]; // Shift the lower order 7 encoded bits into the uint64 in reverse order result |= (uint64_t)(byte & 0x7f) << (7 * bufferIdx); // Increment buffer position to indicate that the byte has been processed (*bufferPos)++; // Done if the high order bit is not set to indicate more data if (byte < 0x80) break; } // By this point all bytes should have been read so error if this is not the case. This could be due to a coding error or // corruption in the data stream. if (byte >= 0x80) THROW(FormatError, "unterminated varint-128 integer"); FUNCTION_TEST_RETURN(UINT64, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/convert.h0000664000175100017510000001600015240547630020301 0ustar00runnerrunner/*********************************************************************************************************************************** Convert C Types Contains conversions to/from native C types. Conversions of project types should be defined in their respective type modules. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_CONVERT_H #define COMMON_TYPE_CONVERT_H #include #include #include #include #include "common/assert.h" #include "common/type/param.h" /*********************************************************************************************************************************** Required buffer sizes ***********************************************************************************************************************************/ #define CVT_BOOL_BUFFER_SIZE 6 #define CVT_BASE10_BUFFER_SIZE 64 #define CVT_DIV_BUFFER_SIZE 68 #define CVT_PCT_BUFFER_SIZE 8 #define CVT_VARINT128_BUFFER_SIZE 10 /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Convert char to zero-terminated string FN_INLINE_ALWAYS size_t cvtCharToZ(const char value, char *const buffer, const size_t bufferSize) { ASSERT_INLINE(buffer != NULL); ASSERT_INLINE(bufferSize >= 2); return (size_t)snprintf(buffer, bufferSize, "%c", value); } // Convert percentage to an integer. This is useful for exporting a percentage in a way that does not incur rounding. FN_EXTERN unsigned int cvtPctToUInt(uint64_t dividend, uint64_t divisor); // Convert percentage to zero-terminated string. This is more reproducible than formatting the results of floating point division. FN_EXTERN size_t cvtPctToZ(uint64_t dividend, uint64_t divisor, char *buffer, size_t bufferSize); // Convert division to zero-terminated string. This is more reproducible than formatting the results of floating point division. FN_EXTERN size_t cvtDivToZ(uint64_t dividend, uint64_t divisor, unsigned int precision, bool trim, char *buffer, size_t bufferSize); // Convert int to zero-terminated string and vice versa FN_EXTERN size_t cvtIntToZ(const int value, char *buffer, size_t bufferSize); FN_EXTERN int cvtZToInt(const char *value); FN_EXTERN int cvtZToIntBase(const char *value, int base); FN_EXTERN int cvtZSubNToIntBase(const char *value, size_t offset, size_t size, int base); FN_INLINE_ALWAYS int cvtZSubNToInt(const char *const value, const size_t offset, const size_t size) { return cvtZSubNToIntBase(value, offset, size, 10); } // Convert int64 to zero-terminated string and vice versa FN_EXTERN size_t cvtInt64ToZ(int64_t value, char *buffer, size_t bufferSize); FN_EXTERN int64_t cvtZToInt64(const char *value); FN_EXTERN int64_t cvtZToInt64Base(const char *value, int base); FN_EXTERN int64_t cvtZSubNToInt64Base(const char *value, size_t offset, size_t size, int base); FN_INLINE_ALWAYS int64_t cvtZSubNToInt64(const char *const value, const size_t offset, const size_t size) { return cvtZSubNToInt64Base(value, offset, size, 10); } // Convert int32/64 to uint32/64 using zigzag encoding and vice versa. Zigzag encoding places the sign in the least significant bit // so that signed and unsigned values alternate, e.g. 0 = 0, -1 = 1, 1 = 2, -2 = 3, 2 = 4, -3 = 5, 3 = 6, etc. This moves as many // bits as possible into the low order bits which is good for other types of encoding, e.g. base-128. See // http://neurocline.github.io/dev/2015/09/17/zig-zag-encoding.html for details. FN_INLINE_ALWAYS uint32_t cvtInt32ToZigZag(const int32_t value) { return ((uint32_t)value << 1) ^ (uint32_t)(value >> 31); } FN_INLINE_ALWAYS int32_t cvtInt32FromZigZag(const uint32_t value) { return (int32_t)((value >> 1) ^ (~(value & 1) + 1)); } FN_INLINE_ALWAYS uint64_t cvtInt64ToZigZag(const int64_t value) { return ((uint64_t)value << 1) ^ (uint64_t)(value >> 63); } FN_INLINE_ALWAYS int64_t cvtInt64FromZigZag(const uint64_t value) { return (int64_t)((value >> 1) ^ (~(value & 1) + 1)); } // Convert mode to zero-terminated string and vice versa FN_EXTERN size_t cvtModeToZ(mode_t value, char *buffer, size_t bufferSize); FN_EXTERN mode_t cvtZToMode(const char *value); // Convert size/ssize to zero-terminated string FN_EXTERN size_t cvtSizeToZ(size_t value, char *buffer, size_t bufferSize); // Convert time_t to zero-terminated string typedef struct CvtTimeToZParam { VAR_PARAM_HEADER; bool utc; // Use UTC instead of local time? } CvtTimeToZParam; #define cvtTimeToZP(format, value, buffer, bufferSize, ...) \ cvtTimeToZ(format, value, buffer, bufferSize, (CvtTimeToZParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN FN_STRFTIME(1) size_t cvtTimeToZ( const char *format, time_t value, char *buffer, size_t bufferSize, CvtTimeToZParam param); // Convert zero-terminated string with the following format to time_t: // ^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}((\,|\.)[0-9]{1,6})?((\+|\-)[0-9]{2}(:?)([0-9]{2})?)?$ FN_EXTERN time_t cvtZToTime(const char *time); // Convert uint to zero-terminated string and vice versa FN_EXTERN size_t cvtUIntToZ(unsigned int value, char *buffer, size_t bufferSize); FN_EXTERN unsigned int cvtZToUInt(const char *value); FN_EXTERN unsigned int cvtZToUIntBase(const char *value, int base); FN_EXTERN unsigned int cvtZSubNToUIntBase(const char *value, size_t offset, size_t size, int base); FN_INLINE_ALWAYS unsigned int cvtZSubNToUInt(const char *const value, const size_t offset, const size_t size) { return cvtZSubNToUIntBase(value, offset, size, 10); } // Convert uint64 to zero-terminated string and vice versa FN_EXTERN size_t cvtUInt64ToZ(uint64_t value, char *buffer, size_t bufferSize); FN_EXTERN uint64_t cvtZToUInt64(const char *value); FN_EXTERN uint64_t cvtZToUInt64Base(const char *value, int base); FN_EXTERN uint64_t cvtZSubNToUInt64Base(const char *value, size_t offset, size_t size, int base); FN_INLINE_ALWAYS uint64_t cvtZSubNToUInt64(const char *const value, const size_t offset, const size_t size) { return cvtZSubNToUInt64Base(value, offset, size, 10); } // Convert uint64 to base-128 varint and vice versa FN_EXTERN void cvtUInt64ToVarInt128(uint64_t value, uint8_t *buffer, size_t *bufferPos, size_t bufferSize); FN_EXTERN uint64_t cvtUInt64FromVarInt128(const uint8_t *buffer, size_t *bufferPos, size_t bufferSize); // Convert boolean to zero-terminated string. Use cvtBoolToConstZ() whenever possible since it is more efficient. FN_EXTERN size_t cvtBoolToZ(bool value, char *buffer, size_t bufferSize); FN_EXTERN const char *cvtBoolToConstZ(bool value); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/json.c0000664000175100017510000016130215240547630017573 0ustar00runnerrunner/*********************************************************************************************************************************** Convert JSON to/from KeyValue ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/type/json.h" /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ typedef struct JsonReadStack { JsonType type; // Container type bool first; // First element has been read } JsonReadStack; struct JsonRead { const char *json; // JSON string List *stack; // Stack of object/array tags bool key; // Was a key read for an object value? bool complete; // JSON read is complete }; typedef struct JsonWriteStack { JsonType type; // Container type bool first; // First element has been written char keyLast[67]; // Last key (may need to be adjusted for alignment) } JsonWriteStack; struct JsonWrite { String *json; // JSON string List *stack; // Stack of object/array tags bool key; // Is a key set for an object value? bool complete; // JSON write is complete }; /*********************************************************************************************************************************** Identify container and scalar types ***********************************************************************************************************************************/ static bool jsonTypeScalar(const JsonType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); switch (type) { case jsonTypeBool: case jsonTypeNull: case jsonTypeNumber: case jsonTypeString: FUNCTION_TEST_RETURN(BOOL, true); default: FUNCTION_TEST_RETURN(BOOL, false); } } static bool jsonTypeContainer(const JsonType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(STRING_ID, !jsonTypeScalar(type)); } /**********************************************************************************************************************************/ FN_EXTERN JsonRead * jsonReadNew(const String *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, json); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(JsonRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (JsonRead) { .json = strZ(json), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(JSON_READ, this); } /*********************************************************************************************************************************** Consume whitespace ***********************************************************************************************************************************/ static void jsonReadConsumeWhiteSpace(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); bool whitespace = true; do { switch (*this->json) { case ' ': case '\t': case '\n': case '\r': this->json++; break; default: whitespace = false; break; } } while (whitespace); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN JsonType jsonReadTypeNext(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadConsumeWhiteSpace(this); // There should be some data if (*this->json == '\0') THROW(JsonFormatError, "expected data"); // Determine data type JsonType result; switch (*this->json) { // String case '"': result = jsonTypeString; break; // Integer case '-': case '0' ... '9': result = jsonTypeNumber; break; // Boolean case 't': case 'f': result = jsonTypeBool; break; // Null case 'n': { result = jsonTypeNull; break; } // Array case '[': result = jsonTypeArrayBegin; break; case ']': result = jsonTypeArrayEnd; break; // Object case '{': result = jsonTypeObjectBegin; break; case '}': result = jsonTypeObjectEnd; break; // Invalid type default: THROW_FMT(JsonFormatError, "invalid type at: %s", this->json); break; } FUNCTION_TEST_RETURN(STRING_ID, result); } /*********************************************************************************************************************************** Read the next type ignoring a single comma before the type ***********************************************************************************************************************************/ static JsonType jsonReadTypeNextIgnoreComma(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadConsumeWhiteSpace(this); if (*this->json != ',') FUNCTION_TEST_RETURN(STRING_ID, jsonReadTypeNext(this)); const char *const jsonBeforeComma = this->json; this->json++; const JsonType result = jsonReadTypeNext(this); this->json = jsonBeforeComma; FUNCTION_TEST_RETURN(STRING_ID, result); } /*********************************************************************************************************************************** Push/pop a type on/off the stack ***********************************************************************************************************************************/ static void jsonReadPush(JsonRead *const this, const JsonType type, const bool key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(BOOL, key); FUNCTION_TEST_END(); ASSERT(this != NULL); // Cannot read after complete if (this->complete) THROW(FormatError, "JSON read is complete"); // Check that the requested type matches the actual type. The actual type needs to be assigned to a local variable or Coverity // will complain about parameter order when jsonReadTypeNextIgnoreComma() is called again in THROW_FMT() even though it is a // noop. const JsonType typeActual = jsonReadTypeNextIgnoreComma(this); if (typeActual != type) THROW_FMT(JsonFormatError, "expected '%s' but found '%s' at: %s", zNewStrId(type), zNewStrId(typeActual), this->json); // If the container stack has not been created yet if (this->stack == NULL) { ASSERT(!key); // If a scalar then no need to create the stack since the read is complete if (jsonTypeScalar(type)) { this->complete = true; FUNCTION_TEST_RETURN_VOID(); } // Create the container stack else { MEM_CONTEXT_OBJ_BEGIN(this) { this->stack = lstNewP(sizeof(JsonReadStack)); } MEM_CONTEXT_OBJ_END(); } } // If there is a container on the stack ASSERT((jsonTypeContainer(type) && !key) || !lstEmpty(this->stack)); if (!lstEmpty(this->stack)) { JsonWriteStack *const item = lstGetLast(this->stack); // If this is a key check that a key has not been read since the last value if (key) { ASSERT(item->type == jsonTypeObjectBegin); if (this->key) THROW_FMT(AssertError, "key has already been read at: %s", this->json); this->key = true; } // Else make sure a key has been read else if (item->type == jsonTypeObjectBegin) { if (!this->key) THROW_FMT(AssertError, "key has not been read at: %s", this->json); this->key = false; } // Consume comma if (item->first && (item->type != jsonTypeObjectBegin || key)) { if (*this->json != ',') THROW_FMT(JsonFormatError, "missing comma at: %s", this->json); this->json++; jsonReadConsumeWhiteSpace(this); } else item->first = true; } // Add container to the stack if (jsonTypeContainer(type)) lstAdd(this->stack, &(JsonReadStack){.type = type}); FUNCTION_TEST_RETURN_VOID(); } static void jsonReadPop(JsonRead *const this, const JsonType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(jsonTypeContainer(type)); // Cannot read after complete if (this->complete) THROW(FormatError, "JSON read is complete"); ASSERT(this->stack != NULL); ASSERT(!lstEmpty(this->stack)); ASSERT_DECLARE(const JsonReadStack *const container = lstGetLast(this->stack)); ASSERT( (container->type == jsonTypeArrayBegin && type == jsonTypeArrayEnd) || (container->type == jsonTypeObjectBegin && type == jsonTypeObjectEnd)); // Check that the container has ended if (jsonReadTypeNext(this) != type) { THROW_FMT( JsonFormatError, "expected %c but found %c at: %s", type == jsonTypeArrayEnd ? ']' : '}', *this->json, this->json); } // Remove container lstRemoveLast(this->stack); // Nothing more to read if this is the last container end if (lstEmpty(this->stack)) this->complete = true; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Read JSON number ***********************************************************************************************************************************/ typedef enum { jsonNumberTypeU64, jsonNumberTypeI64, } JsonNumberType; typedef struct JsonReadNumberResult { JsonNumberType type; union { int64_t i64; uint64_t u64; }; } JsonReadNumberResult; static JsonReadNumberResult jsonReadNumber(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); // Consume the - when present const bool intSigned = *this->json == '-'; // Consume all digits size_t digits = 0; while (isdigit(this->json[digits + intSigned])) digits++; // Invalid if no digits were found if (digits == 0) THROW_FMT(JsonFormatError, "no digits found at: %s", this->json); // Copy to buffer if (digits >= CVT_BASE10_BUFFER_SIZE) THROW_FMT(JsonFormatError, "invalid number at: %s", this->json); const size_t size = digits + intSigned; this->json += size; // Return result if (intSigned) { FUNCTION_TEST_RETURN_TYPE( JsonReadNumberResult, (JsonReadNumberResult){.type = jsonNumberTypeI64, .i64 = cvtZSubNToInt64(this->json - size, 0, size)}); } FUNCTION_TEST_RETURN_TYPE( JsonReadNumberResult, (JsonReadNumberResult){.type = jsonNumberTypeU64, .u64 = cvtZSubNToUInt64(this->json - size, 0, size)}); } /**********************************************************************************************************************************/ FN_EXTERN void jsonReadArrayBegin(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeArrayBegin, false); this->json++; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void jsonReadArrayEnd(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPop(this, jsonTypeArrayEnd); this->json++; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool jsonReadBool(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeBool, false); if (strncmp(this->json, TRUE_Z, sizeof(TRUE_Z) - 1) == 0) { this->json += sizeof(TRUE_Z) - 1; FUNCTION_TEST_RETURN(BOOL, true); } if (strncmp(this->json, FALSE_Z, sizeof(FALSE_Z) - 1) == 0) { this->json += sizeof(FALSE_Z) - 1; FUNCTION_TEST_RETURN(BOOL, false); } THROW_FMT(JsonFormatError, "expected boolean at: %s", this->json); } /**********************************************************************************************************************************/ FN_EXTERN int jsonReadInt(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeU64) { if (number.u64 > INT_MAX) THROW_FMT(JsonFormatError, "%" PRIu64 " is out of range for int", number.u64); FUNCTION_TEST_RETURN(INT, (int)number.u64); } if (number.i64 < INT_MIN) THROW_FMT(JsonFormatError, "%" PRId64 " is out of range for int", number.i64); FUNCTION_TEST_RETURN(INT, (int)number.i64); } /**********************************************************************************************************************************/ FN_EXTERN int64_t jsonReadInt64(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeI64) FUNCTION_TEST_RETURN(INT64, number.i64); if (number.u64 > INT64_MAX) THROW_FMT(JsonFormatError, "%" PRIu64 " is out of range for int64", number.u64); FUNCTION_TEST_RETURN(INT64, (int64_t)number.u64); } /**********************************************************************************************************************************/ typedef struct JsonReadKeyInternalResult { const char *buffer; size_t size; } JsonReadKeyInternalResult; static JsonReadKeyInternalResult jsonReadKeyZN(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Skip the beginning " ASSERT(*this->json == '"'); this->json++; // Read string JsonReadKeyInternalResult result = {.buffer = this->json}; while (*this->json != '"') { if (*this->json == '\\') THROW_FMT(JsonFormatError, "escape character not allowed in key at: %s", this->json); else if (*this->json == '\0') THROW(JsonFormatError, "expected '\"' but found null delimiter"); this->json++; } // Set key size result.size = (size_t)(this->json - result.buffer); // Advance the character array pointer to the next element after the string this->json++; // Consume the : after the key jsonReadConsumeWhiteSpace(this); if (*this->json != ':') THROW_FMT(JsonFormatError, "expected : after key '%.*s' at: %s", (int)result.size, result.buffer, this->json); this->json++; FUNCTION_TEST_RETURN_TYPE(JsonReadKeyInternalResult, result); } FN_EXTERN String * jsonReadKey(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeString, true); JsonReadKeyInternalResult key = jsonReadKeyZN(this); FUNCTION_TEST_RETURN(STRING, strNewZN(key.buffer, key.size)); } /**********************************************************************************************************************************/ FN_EXTERN bool jsonReadKeyExpect(JsonRead *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); bool result = false; // Check that we are currently reading an object jsonReadConsumeWhiteSpace(this); JsonWriteStack *const item = lstGetLast(this->stack); ASSERT(item->type == jsonTypeObjectBegin); // Read until object end while (jsonReadTypeNextIgnoreComma(this) != jsonTypeObjectEnd) { // Save state before reading the key const char *const jsonBeforeKey = this->json; const bool firstBeforeKey = item->first; jsonReadPush(this, jsonTypeString, true); // Get and compare next key const JsonReadKeyInternalResult keyNext = jsonReadKeyZN(this); const int compare = strncmp(strZ(key), keyNext.buffer, keyNext.size); // Return true if key matches if (compare == 0) { result = true; break; } // Return false and reset state if requested key is after the actual key else if (compare < 0) { this->json = jsonBeforeKey; this->key = false; item->first = firstBeforeKey; break; } // Skip the value and continue jsonReadSkip(this); } FUNCTION_TEST_RETURN(BOOL, result); } FN_EXTERN bool jsonReadKeyExpectStrId(JsonRead *const this, const StringId key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING_ID, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != 0); char buffer[STRID_MAX + 1]; FUNCTION_TEST_RETURN(BOOL, jsonReadKeyExpect(this, STR_SIZE(buffer, strIdToZ(key, buffer)))); } FN_EXTERN bool jsonReadKeyExpectZ(JsonRead *const this, const char *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRINGZ, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(BOOL, jsonReadKeyExpect(this, STR(key))); } FN_EXTERN JsonRead * jsonReadKeyRequire(JsonRead *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); if (!jsonReadKeyExpect(this, key)) THROW_FMT(JsonFormatError, "required key '%s' not found", strZ(key)); FUNCTION_TEST_RETURN(JSON_READ, this); } FN_EXTERN JsonRead * jsonReadKeyRequireStrId(JsonRead *const this, const StringId key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRING_ID, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != 0); char buffer[STRID_MAX + 1]; FUNCTION_TEST_RETURN(JSON_READ, jsonReadKeyRequire(this, STR_SIZE(buffer, strIdToZ(key, buffer)))); } FN_EXTERN JsonRead * jsonReadKeyRequireZ(JsonRead *const this, const char *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_PARAM(STRINGZ, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(JSON_READ, jsonReadKeyRequire(this, STR(key))); } /**********************************************************************************************************************************/ FN_EXTERN void jsonReadNull(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNull, false); if (strncmp(this->json, NULL_Z, sizeof(NULL_Z) - 1) == 0) { this->json += sizeof(NULL_Z) - 1; FUNCTION_TEST_RETURN_VOID(); } THROW_FMT(JsonFormatError, "expected null at: %s", this->json); } /**********************************************************************************************************************************/ FN_EXTERN void jsonReadObjectBegin(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeObjectBegin, false); this->json++; FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void jsonReadObjectEnd(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPop(this, jsonTypeObjectEnd); this->json++; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ // Quickly skip over a valid JSON string static void jsonReadSkipStr(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Skip the beginning " ASSERT(*this->json == '"'); this->json++; // Read string while (*this->json != '"') { if (*this->json == '\\') { this->json++; switch (*this->json) { // Unicode character case 'u': { this->json++; // Expect four digits unsigned int digitIdx = 0; for (; digitIdx < 4; digitIdx++) { if (!isdigit(this->json[digitIdx])) break; } if (digitIdx != 4) THROW_FMT(JsonFormatError, "unable to decode at: %s", this->json - 2); this->json += 4; break; } // Any other escape default: break; } } else if (*this->json == '\0') THROW(JsonFormatError, "expected '\"' but found null delimiter"); this->json++; } // Advance the character array pointer to the next element after the string this->json++; FUNCTION_TEST_RETURN_VOID(); } static void jsonReadSkipRecurse(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); switch (jsonReadTypeNextIgnoreComma(this)) { case jsonTypeBool: jsonReadBool(this); break; case jsonTypeNull: jsonReadNull(this); break; case jsonTypeNumber: jsonReadPush(this, jsonTypeNumber, false); jsonReadNumber(this); break; case jsonTypeString: jsonReadPush(this, jsonTypeString, false); jsonReadSkipStr(this); break; case jsonTypeArrayBegin: { jsonReadArrayBegin(this); while (jsonReadTypeNextIgnoreComma(this) != jsonTypeArrayEnd) jsonReadSkipRecurse(this); jsonReadArrayEnd(this); break; } default: { jsonReadObjectBegin(this); while (jsonReadTypeNextIgnoreComma(this) != jsonTypeObjectEnd) { // Read key jsonReadPush(this, jsonTypeString, true); jsonReadSkipStr(this); // Consume the : after the key jsonReadConsumeWhiteSpace(this); if (*this->json != ':') THROW_FMT(JsonFormatError, "expected : after key at: %s", this->json); this->json++; // Skip value jsonReadSkipRecurse(this); } jsonReadObjectEnd(this); break; } } FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void jsonReadSkip(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadSkipRecurse(this); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * jsonReadStr(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // If NULL if (jsonReadTypeNextIgnoreComma(this) == jsonTypeNull) { jsonReadNull(this); FUNCTION_TEST_RETURN(STRING, NULL); } // Read string jsonReadPush(this, jsonTypeString, false); String *const result = strNew(); // Skip the beginning " ASSERT(*this->json == '"'); this->json++; // Track portion of string with no escapes const char *noEscape = NULL; size_t noEscapeSize = 0; while (*this->json != '"') { if (*this->json == '\\') { // Copy portion of string without escapes if (noEscapeSize > 0) { strCatZN(result, noEscape, noEscapeSize); noEscapeSize = 0; } this->json++; switch (*this->json) { case '"': strCatChr(result, '"'); break; case '\\': strCatChr(result, '\\'); break; case '/': strCatChr(result, '/'); break; case 'n': strCatChr(result, '\n'); break; case 'r': strCatChr(result, '\r'); break; case 't': strCatChr(result, '\t'); break; case 'b': strCatChr(result, '\b'); break; case 'f': strCatChr(result, '\f'); break; case 'u': { this->json++; // We don't know how to decode anything except ASCII so fail if it looks like Unicode if (strncmp(this->json, "00", 2) != 0) THROW_FMT(JsonFormatError, "unable to decode at: %s", this->json - 2); // Decode char this->json += 2; strCatChr(result, (char)cvtZSubNToUIntBase(this->json, 0, 2, 16)); this->json++; break; } default: THROW_FMT(JsonFormatError, "invalid escape character at: %s", this->json - 1); } } else { if (*this->json == '\0') THROW(JsonFormatError, "expected '\"' but found null delimiter"); // If escape string is zero size then start it if (noEscapeSize == 0) noEscape = this->json; noEscapeSize++; } this->json++; } // Copy portion of string without escapes if (noEscapeSize > 0) strCatZN(result, noEscape, noEscapeSize); // Advance the character array pointer to the next element after the string this->json++; FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN StringId jsonReadStrId(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); StringId result = 0; MEM_CONTEXT_TEMP_BEGIN() { result = strStrId(jsonReadStr(this)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_ID, result); } /**********************************************************************************************************************************/ FN_EXTERN StringList * jsonReadStrLst(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { jsonReadArrayBegin(this); while (jsonReadTypeNextIgnoreComma(this) != jsonTypeArrayEnd) strLstAdd(result, jsonReadStr(this)); jsonReadArrayEnd(this); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int jsonReadUInt(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeI64) THROW_FMT(JsonFormatError, "%" PRId64 " is out of range for uint", number.i64); if (number.u64 > UINT_MAX) THROW_FMT(JsonFormatError, "%" PRIu64 " is out of range for uint", number.u64); FUNCTION_TEST_RETURN(UINT, (unsigned int)number.u64); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t jsonReadUInt64(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeI64) THROW_FMT(JsonFormatError, "%" PRId64 " is out of range for uint64", number.i64); FUNCTION_TEST_RETURN(UINT64, number.u64); } /**********************************************************************************************************************************/ static Variant * jsonReadVarRecurse(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); switch (jsonReadTypeNextIgnoreComma(this)) { case jsonTypeBool: FUNCTION_TEST_RETURN(VARIANT, varNewBool(jsonReadBool(this))); case jsonTypeNull: jsonReadNull(this); FUNCTION_TEST_RETURN(VARIANT, NULL); case jsonTypeNumber: { jsonReadPush(this, jsonTypeNumber, false); JsonReadNumberResult number = jsonReadNumber(this); if (number.type == jsonNumberTypeU64) FUNCTION_TEST_RETURN(VARIANT, varNewUInt64(number.u64)); FUNCTION_TEST_RETURN(VARIANT, varNewInt64(number.i64)); } case jsonTypeString: { String *const string = jsonReadStr(this); Variant *const result = varNewStr(string); strFree(string); FUNCTION_TEST_RETURN(VARIANT, result); } case jsonTypeArrayBegin: { VariantList *const list = varLstNew(); jsonReadArrayBegin(this); MEM_CONTEXT_BEGIN(lstMemContext((List *)list)) { while (jsonReadTypeNextIgnoreComma(this) != jsonTypeArrayEnd) varLstAdd(list, jsonReadVarRecurse(this)); } MEM_CONTEXT_END(); jsonReadArrayEnd(this); Variant *const result = varNewVarLst(list); varLstFree(list); FUNCTION_TEST_RETURN(VARIANT, result); } default: { KeyValue *const result = kvNew(); jsonReadObjectBegin(this); while (jsonReadTypeNextIgnoreComma(this) != jsonTypeObjectEnd) { String *const key = jsonReadKey(this); Variant *const value = jsonReadVarRecurse(this); kvPut(result, VARSTR(key), value); strFree(key); varFree(value); } jsonReadObjectEnd(this); FUNCTION_TEST_RETURN(VARIANT, varNewKv(result)); } } } FN_EXTERN Variant * jsonReadVar(JsonRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_READ, this); FUNCTION_TEST_END(); Variant *const result = jsonReadVarRecurse(this); FUNCTION_TEST_RETURN(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * jsonToVar(const String *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, json); FUNCTION_TEST_END(); Variant *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { JsonRead *const read = jsonReadNew(json); MEM_CONTEXT_PRIOR_BEGIN() { result = jsonReadVar(read); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN void jsonValidate(const String *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, json); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { JsonRead *const read = jsonReadNew(json); MEM_CONTEXT_PRIOR_BEGIN() { jsonReadSkip(read); jsonReadConsumeWhiteSpace(read); if (*read->json != '\0') THROW_FMT(JsonFormatError, "characters after JSON at: %s", read->json); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void jsonReadToLog(const JsonRead *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{json: %s}", this->json); } #endif // DEBUG /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteNew(const JsonWriteNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, param.json); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(JsonWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (JsonWrite) { .json = param.json == NULL ? strNew() : param.json, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /*********************************************************************************************************************************** Push/pop a type on/off the stack ***********************************************************************************************************************************/ static void jsonWritePush(JsonWrite *const this, const JsonType type, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); // Cannot write after complete if (this->complete) THROW(JsonFormatError, "JSON write is complete"); // If the container stack jas not been created yet if (this->stack == NULL) { ASSERT(key == NULL); // If a scalar then no need to create the stack since the write is complete if (jsonTypeScalar(type)) { this->complete = true; FUNCTION_TEST_RETURN_VOID(); } // Create the container stack else { MEM_CONTEXT_OBJ_BEGIN(this) { this->stack = lstNewP(sizeof(JsonWriteStack)); } MEM_CONTEXT_OBJ_END(); } } // If there is a container on the stack ASSERT((jsonTypeContainer(type) && key == NULL) || !lstEmpty(this->stack)); if (!lstEmpty(this->stack)) { JsonWriteStack *const item = lstGetLast(this->stack); // If this is a key check that a key has not been written since the last value if (key != NULL) { if (this->key) THROW_FMT(JsonFormatError, "key has already been written"); // Also check that the key is after the last key if (item->keyLast[0] != '\0' && strCmpZ(key, item->keyLast) <= 0) THROW_FMT(JsonFormatError, "key '%s' is not after prior key '%s'", strZ(key), item->keyLast); // Copy key to a buffer to avoid needing to allocate memory if (strSize(key) >= SIZE_OF_STRUCT_MEMBER(JsonWriteStack, keyLast)) { THROW_FMT( AssertError, "key '%s' must not be longer than %zu bytes", strZ(key), SIZE_OF_STRUCT_MEMBER(JsonWriteStack, keyLast) - 1); } memcpy(item->keyLast, strZ(key), strSize(key) + 1); this->key = true; } // Else make sure a key has been written else { if (item->type == jsonTypeObjectBegin) { if (!this->key) THROW_FMT(JsonFormatError, "key has not been written"); this->key = false; } } // Write comma if (item->first && (item->type != jsonTypeObjectBegin || key != NULL)) strCatChr(this->json, ','); else item->first = true; } // Add container to the stack if (jsonTypeContainer(type)) lstAdd(this->stack, &(JsonWriteStack){.type = type}); FUNCTION_TEST_RETURN_VOID(); } static void jsonWritePop(JsonWrite *const this ASSERT_PARAM(const JsonType type)) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->stack != NULL); ASSERT(!lstEmpty(this->stack)); ASSERT_DECLARE(const JsonWriteStack *const container = lstGetLast(this->stack)); ASSERT( (container->type == jsonTypeArrayBegin && type == jsonTypeArrayEnd) || (container->type == jsonTypeObjectBegin && type == jsonTypeObjectEnd)); ASSERT(container->type != jsonTypeObjectEnd || this->key == false); // Remove container lstRemoveLast(this->stack); // Nothing more to write if this is the last container end if (lstEmpty(this->stack)) this->complete = true; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteArrayBegin(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeArrayBegin, NULL); strCatChr(this->json, '['); FUNCTION_TEST_RETURN(JSON_WRITE, this); } FN_EXTERN JsonWrite * jsonWriteArrayEnd(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePop(this ASSERT_PARAM(jsonTypeArrayEnd)); strCatChr(this->json, ']'); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteBool(JsonWrite *const this, const bool value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(BOOL, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeBool, NULL); strCat(this->json, value ? TRUE_STR : FALSE_STR); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteInt(JsonWrite *const this, const int value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(INT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNumber, NULL); char working[CVT_BASE10_BUFFER_SIZE]; strCatZN(this->json, working, cvtIntToZ(value, working, sizeof(working))); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteInt64(JsonWrite *const this, const int64_t value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(INT64, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNumber, NULL); char working[CVT_BASE10_BUFFER_SIZE]; strCatZN(this->json, working, cvtInt64ToZ(value, working, sizeof(working))); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteJson(JsonWrite *const this, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); if (value == NULL) jsonWriteNull(this); else { jsonWritePush(this, jsonTypeString, NULL); strCat(this->json, value); } FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteKey(JsonWrite *const this, const String *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); ASSERT( strchr(strZ(key), '\f') == NULL && strchr(strZ(key), '\\') == NULL && strchr(strZ(key), '\n') == NULL && strchr(strZ(key), '\r') == NULL && strchr(strZ(key), '\t') == NULL && strchr(strZ(key), '\b') == NULL && strchr(strZ(key), '"') == NULL); jsonWritePush(this, jsonTypeString, key); strCatChr(this->json, '"'); strCat(this->json, key); strCatZ(this->json, "\":"); FUNCTION_TEST_RETURN(JSON_WRITE, this); } FN_EXTERN JsonWrite * jsonWriteKeyStrId(JsonWrite *const this, const StringId key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_ID, key); FUNCTION_TEST_END(); char buffer[STRID_MAX + 4]; size_t size = strIdToZ(key, buffer + 1); jsonWritePush(this, jsonTypeString, STR_SIZE(buffer + 1, size)); buffer[0] = '"'; buffer[++size] = '"'; buffer[++size] = ':'; strCatZN(this->json, buffer, size + 1); FUNCTION_TEST_RETURN(JSON_WRITE, this); } FN_EXTERN JsonWrite * jsonWriteKeyZ(JsonWrite *const this, const char *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRINGZ, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_TEST_RETURN(JSON_WRITE, jsonWriteKey(this, STR(key))); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteNull(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNull, NULL); strCat(this->json, NULL_STR); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteObjectBegin(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeObjectBegin, NULL); strCatChr(this->json, '{'); FUNCTION_TEST_RETURN(JSON_WRITE, this); } FN_EXTERN JsonWrite * jsonWriteObjectEnd(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePop(this ASSERT_PARAM(jsonTypeObjectEnd)); strCatChr(this->json, '}'); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteStr(JsonWrite *const this, const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(this != NULL); if (value == NULL) { jsonWriteNull(this); FUNCTION_TEST_RETURN(JSON_WRITE, this); } jsonWritePush(this, jsonTypeString, NULL); strCatChr(this->json, '"'); // Track portion of string with no escapes const char *valuePtr = strZ(value); const char *noEscape = NULL; size_t noEscapeSize = 0; for (unsigned int valueIdx = 0; valueIdx < strSize(value); valueIdx++) { switch (*valuePtr) { case '"': case '\\': case '\n': case '\r': case '\t': case '\b': case '\f': { // Copy portion of string without escapes if (noEscapeSize > 0) { strCatZN(this->json, noEscape, noEscapeSize); noEscapeSize = 0; } switch (*valuePtr) { case '"': strCatZ(this->json, "\\\""); break; case '\\': strCatZ(this->json, "\\\\"); break; case '\n': strCatZ(this->json, "\\n"); break; case '\r': strCatZ(this->json, "\\r"); break; case '\t': strCatZ(this->json, "\\t"); break; case '\b': strCatZ(this->json, "\\b"); break; case '\f': strCatZ(this->json, "\\f"); break; } break; } default: { // If escape string is zero size then start it if (noEscapeSize == 0) noEscape = valuePtr; noEscapeSize++; break; } } valuePtr++; } // Copy portion of string without escapes if (noEscapeSize > 0) strCatZN(this->json, noEscape, noEscapeSize); strCatChr(this->json, '"'); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteStrFmt(JsonWrite *const this, const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // Determine how long the allocated string needs to be va_list argumentList; va_start(argumentList, format); const size_t size = (size_t)vsnprintf(NULL, 0, format, argumentList); va_end(argumentList); // Format string va_start(argumentList, format); char *const buffer = memNew(size + 1); vsnprintf(buffer, size + 1, format, argumentList); va_end(argumentList); jsonWriteStr(this, STR_SIZE(buffer, size)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteStrId(JsonWrite *const this, const StringId value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_ID, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeString, NULL); char buffer[STRID_MAX + 3]; buffer[0] = '"'; size_t size = strIdToZN(value, buffer + 1) + 1; buffer[size++] = '"'; strCatZN(this->json, buffer, size); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteStrLst(JsonWrite *const this, const StringList *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRING_LIST, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(value != NULL); jsonWriteArrayBegin(this); for (unsigned int valueIdx = 0; valueIdx < strLstSize(value); valueIdx++) jsonWriteStr(this, strLstGet(value, valueIdx)); jsonWriteArrayEnd(this); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteUInt(JsonWrite *const this, const unsigned int value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(UINT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNumber, NULL); char working[CVT_BASE10_BUFFER_SIZE]; strCatZN(this->json, working, cvtUIntToZ(value, working, sizeof(working))); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteUInt64(JsonWrite *const this, const uint64_t value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWritePush(this, jsonTypeNumber, NULL); char working[CVT_BASE10_BUFFER_SIZE]; strCatZN(this->json, working, cvtUInt64ToZ(value, working, sizeof(working))); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ static void jsonWriteVarRecurse(JsonWrite *const this, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); MEM_CONTEXT_TEMP_BEGIN() { // If VariantList then process each item in the array. Currently the list must be KeyValue types. if (value == NULL) jsonWriteNull(this); else { switch (varType(value)) { case varTypeBool: jsonWriteBool(this, varBool(value)); break; case varTypeInt: jsonWriteInt(this, varInt(value)); break; case varTypeInt64: jsonWriteInt64(this, varInt64(value)); break; case varTypeKeyValue: { const KeyValue *const keyValue = varKv(value); if (keyValue == NULL) jsonWriteNull(this); else { const StringList *const keyList = strLstSort(strLstNewVarLst(kvKeyList(keyValue)), sortOrderAsc); jsonWriteObjectBegin(this); for (unsigned int keyListIdx = 0; keyListIdx < strLstSize(keyList); keyListIdx++) { // Write key const String *const key = strLstGet(keyList, keyListIdx); jsonWriteKey(this, key); // Write value jsonWriteVarRecurse(this, kvGet(keyValue, VARSTR(key))); } jsonWriteObjectEnd(this); } break; } case varTypeString: { jsonWriteStr(this, varStr(value)); break; } case varTypeUInt: jsonWriteUInt(this, varUInt(value)); break; case varTypeUInt64: jsonWriteUInt64(this, varUInt64(value)); break; default: { ASSERT(varType(value) == varTypeVariantList); const VariantList *const valueList = varVarLst(value); if (valueList == NULL) jsonWriteNull(this); else { jsonWriteArrayBegin(this); for (unsigned int valueListIdx = 0; valueListIdx < varLstSize(valueList); valueListIdx++) jsonWriteVarRecurse(this, varLstGet(valueList, valueListIdx)); jsonWriteArrayEnd(this); } } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN JsonWrite * jsonWriteVar(JsonWrite *const this, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWriteVarRecurse(this, value); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN JsonWrite * jsonWriteZ(JsonWrite *const this, const char *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_PARAM(STRINGZ, value); FUNCTION_TEST_END(); ASSERT(this != NULL); jsonWriteStr(this, value == NULL ? NULL : STR(value)); FUNCTION_TEST_RETURN(JSON_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN const String * jsonWriteResult(JsonWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(JSON_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->complete); FUNCTION_TEST_RETURN(STRING, this->json); } /**********************************************************************************************************************************/ FN_EXTERN String * jsonFromVar(const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { jsonWriteVar(jsonWriteNewP(.json = result), value); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void jsonWriteToLog(const JsonWrite *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{size: %zu}", strSize(this->json)); } #endif // DEBUG ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/json.h0000664000175100017510000002245715240547630017607 0ustar00runnerrunner/*********************************************************************************************************************************** JSON read/write ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_JSON_H #define COMMON_TYPE_JSON_H #include "common/type/stringId.h" #include "common/type/variant.h" /*********************************************************************************************************************************** JSON types ***********************************************************************************************************************************/ typedef enum { jsonTypeBool = STRID5("bool", 0x63de20), // Boolean jsonTypeNull = STRID5("null", 0x632ae0), // Null jsonTypeNumber = STRID5("nmbr", 0x909ae0), // Number jsonTypeString = STRID5("str", 0x4a930), // String jsonTypeArrayBegin = STRID5("ary-b", 0x2de6410), // Array begin jsonTypeArrayEnd = STRID5("ary-e", 0x5de6410), // Array end jsonTypeObjectBegin = STRID5("obj-b", 0x2da84f0), // Object begin jsonTypeObjectEnd = STRID5("obj-e", 0x5da84f0), // Object end } JsonType; /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ typedef struct JsonRead JsonRead; typedef struct JsonWrite JsonWrite; /*********************************************************************************************************************************** Read Constructors ***********************************************************************************************************************************/ FN_EXTERN JsonRead *jsonReadNew(const String *string); /*********************************************************************************************************************************** Read Functions ***********************************************************************************************************************************/ // Read next JSON type. This is based on an examination of the first character so there may be an error when the type is read, but // the type will not change. FN_EXTERN JsonType jsonReadTypeNext(JsonRead *this); // Read array begin/end FN_EXTERN void jsonReadArrayBegin(JsonRead *this); FN_EXTERN void jsonReadArrayEnd(JsonRead *this); // Read boolean FN_EXTERN bool jsonReadBool(JsonRead *this); // Read integer FN_EXTERN int jsonReadInt(JsonRead *this); FN_EXTERN int64_t jsonReadInt64(JsonRead *this); FN_EXTERN unsigned int jsonReadUInt(JsonRead *this); FN_EXTERN uint64_t jsonReadUInt64(JsonRead *this); // Read key FN_EXTERN String *jsonReadKey(JsonRead *this); // Read expected key FN_EXTERN bool jsonReadKeyExpect(JsonRead *this, const String *key); FN_EXTERN bool jsonReadKeyExpectStrId(JsonRead *this, StringId key); FN_EXTERN bool jsonReadKeyExpectZ(JsonRead *this, const char *key); // Read required key FN_EXTERN JsonRead *jsonReadKeyRequire(JsonRead *this, const String *key); FN_EXTERN JsonRead *jsonReadKeyRequireStrId(JsonRead *this, StringId key); FN_EXTERN JsonRead *jsonReadKeyRequireZ(JsonRead *this, const char *key); // Read null FN_EXTERN void jsonReadNull(JsonRead *this); // Read object begin/end FN_EXTERN void jsonReadObjectBegin(JsonRead *this); FN_EXTERN void jsonReadObjectEnd(JsonRead *this); // Skip value FN_EXTERN void jsonReadSkip(JsonRead *this); // Read string FN_EXTERN String *jsonReadStr(JsonRead *this); FN_EXTERN StringId jsonReadStrId(JsonRead *this); FN_EXTERN StringList *jsonReadStrLst(JsonRead *this); // Read variant FN_EXTERN Variant *jsonReadVar(JsonRead *this); /*********************************************************************************************************************************** Read Helper Functions ***********************************************************************************************************************************/ // Convert JSON to Variant FN_EXTERN Variant *jsonToVar(const String *json); // Validate JSON FN_EXTERN void jsonValidate(const String *json); /*********************************************************************************************************************************** Read Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void jsonReadFree(JsonRead *const this) { objFree(this); } /*********************************************************************************************************************************** Write Constructors ***********************************************************************************************************************************/ typedef struct JsonWriteNewParam { VAR_PARAM_HEADER; String *json; } JsonWriteNewParam; #define jsonWriteNewP(...) \ jsonWriteNew((JsonWriteNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN JsonWrite *jsonWriteNew(JsonWriteNewParam param); /*********************************************************************************************************************************** Write Functions ***********************************************************************************************************************************/ // Write array begin/end FN_EXTERN JsonWrite *jsonWriteArrayBegin(JsonWrite *this); FN_EXTERN JsonWrite *jsonWriteArrayEnd(JsonWrite *this); // Write boolean FN_EXTERN JsonWrite *jsonWriteBool(JsonWrite *this, bool value); // Write integer FN_EXTERN JsonWrite *jsonWriteInt(JsonWrite *this, int value); FN_EXTERN JsonWrite *jsonWriteInt64(JsonWrite *this, int64_t value); FN_EXTERN JsonWrite *jsonWriteUInt(JsonWrite *this, unsigned int value); FN_EXTERN JsonWrite *jsonWriteUInt64(JsonWrite *this, uint64_t value); // Write JSON FN_EXTERN JsonWrite *jsonWriteJson(JsonWrite *this, const String *value); // Write key FN_EXTERN JsonWrite *jsonWriteKey(JsonWrite *this, const String *key); FN_EXTERN JsonWrite *jsonWriteKeyStrId(JsonWrite *this, StringId key); FN_EXTERN JsonWrite *jsonWriteKeyZ(JsonWrite *this, const char *key); // Write null FN_EXTERN JsonWrite *jsonWriteNull(JsonWrite *this); // Write object begin/end FN_EXTERN JsonWrite *jsonWriteObjectBegin(JsonWrite *this); FN_EXTERN JsonWrite *jsonWriteObjectEnd(JsonWrite *this); // Write string FN_EXTERN JsonWrite *jsonWriteStr(JsonWrite *this, const String *value); FN_EXTERN FN_PRINTF(2, 3) JsonWrite *jsonWriteStrFmt(JsonWrite *this, const char *format, ...); FN_EXTERN JsonWrite *jsonWriteStrId(JsonWrite *this, StringId value); FN_EXTERN JsonWrite *jsonWriteStrLst(JsonWrite *this, const StringList *value); FN_EXTERN JsonWrite *jsonWriteZ(JsonWrite *this, const char *value); // Write variant FN_EXTERN JsonWrite *jsonWriteVar(JsonWrite *this, const Variant *value); /*********************************************************************************************************************************** Write Getters/Setters ***********************************************************************************************************************************/ // Get JSON result FN_EXTERN const String *jsonWriteResult(JsonWrite *this); /*********************************************************************************************************************************** Write Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void jsonWriteFree(JsonWrite *const this) { objFree(this); } /*********************************************************************************************************************************** Write Helper Functions ***********************************************************************************************************************************/ // Convert Variant to JSON FN_EXTERN String *jsonFromVar(const Variant *value); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #ifdef DEBUG FN_EXTERN void jsonReadToLog(const JsonRead *this, StringStatic *debugLog); #define FUNCTION_LOG_JSON_READ_TYPE \ JsonRead * #define FUNCTION_LOG_JSON_READ_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, jsonReadToLog, buffer, bufferSize) FN_EXTERN void jsonWriteToLog(const JsonWrite *this, StringStatic *debugLog); #define FUNCTION_LOG_JSON_WRITE_TYPE \ JsonWrite * #define FUNCTION_LOG_JSON_WRITE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, jsonWriteToLog, buffer, bufferSize) #endif // DEBUG #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/keyValue.c0000664000175100017510000002653515240547630020417 0ustar00runnerrunner/*********************************************************************************************************************************** Key Value Handler ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/type/keyValue.h" #include "common/type/list.h" #include "common/type/variantList.h" /*********************************************************************************************************************************** Contains information about the key value store ***********************************************************************************************************************************/ struct KeyValue { KeyValuePub pub; // Publicly accessible variables List *list; // List of keys/values }; /*********************************************************************************************************************************** Contains information about an individual key/value pair ***********************************************************************************************************************************/ typedef struct KeyValuePair { Variant *key; // The key Variant *value; // The value (this may be NULL) } KeyValuePair; /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(KeyValue, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (KeyValue) { .pub = { .keyList = varLstNew(), }, .list = lstNewP(sizeof(KeyValuePair)), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(KEY_VALUE, this); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvDup(const KeyValue *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, source); FUNCTION_TEST_END(); ASSERT(source != NULL); KeyValue *const this = kvNew(); MEM_CONTEXT_OBJ_BEGIN(this) { // Duplicate all key/values for (unsigned int listIdx = 0; listIdx < lstSize(source->list); listIdx++) { const KeyValuePair *const sourcePair = (const KeyValuePair *)lstGet(source->list, listIdx); lstAdd(this->list, &(KeyValuePair){.key = varDup(sourcePair->key), .value = varDup(sourcePair->value)}); } // Duplicate key list this->pub.keyList = varLstDup(kvKeyList(source)); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(KEY_VALUE, this); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int kvGetIdx(const KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); // Search for the key unsigned int result = KEY_NOT_FOUND; for (unsigned int listIdx = 0; listIdx < lstSize(this->list); listIdx++) { const KeyValuePair *const pair = (const KeyValuePair *)lstGet(this->list, listIdx); // Break if the key matches if (varEq(key, pair->key)) { result = listIdx; break; } } FUNCTION_TEST_RETURN(UINT, result); } /*********************************************************************************************************************************** Internal put key/value pair Handles the common logic for the external put functions. The correct mem context should be set before calling this function. ***********************************************************************************************************************************/ static void kvPutInternal(KeyValue *const this, const Variant *const key, Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); FUNCTION_AUDIT_HELPER(); // Find the key const unsigned int listIdx = kvGetIdx(this, key); // If the key was not found then add it if (listIdx == KEY_NOT_FOUND) { // Copy the pair const KeyValuePair pair = {.key = varDup(key), .value = value}; // Add to the list lstAdd(this->list, &pair); // Add to the key list varLstAdd(this->pub.keyList, varDup(key)); } // Else update it else { KeyValuePair *pair = (KeyValuePair *)lstGet(this->list, listIdx); if (pair->value != NULL) varFree(pair->value); pair->value = value; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvPut(KeyValue *const this, const Variant *const key, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { kvPutInternal(this, key, varDup(value)); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(KEY_VALUE, this); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvAdd(KeyValue *const this, const Variant *const key, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { // Find the key const unsigned int listIdx = kvGetIdx(this, key); // If the key was not found then add it if (listIdx == KEY_NOT_FOUND) { kvPutInternal(this, key, varDup(value)); } // Else create or add to the variant list else { KeyValuePair *const pair = (KeyValuePair *)lstGet(this->list, listIdx); if (pair->value == NULL) pair->value = varDup(value); else if (varType(pair->value) != varTypeVariantList) { Variant *valueList = varNewVarLst(varLstNew()); varLstAdd(varVarLst(valueList), pair->value); varLstAdd(varVarLst(valueList), varDup(value)); pair->value = valueList; } else varLstAdd(varVarLst(pair->value), varDup(value)); } } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(KEY_VALUE, this); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvPutKv(KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); KeyValue *const result = kvNew(); MEM_CONTEXT_OBJ_BEGIN(this) { kvPutInternal(this, key, varNewKv(result)); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(KEY_VALUE, result); } /**********************************************************************************************************************************/ FN_EXTERN const Variant * kvGet(const KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); Variant *result = NULL; // Find the key const unsigned int listIdx = kvGetIdx(this, key); if (listIdx != KEY_NOT_FOUND) result = ((KeyValuePair *)lstGet(this->list, listIdx))->value; FUNCTION_TEST_RETURN(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN const Variant * kvGetDefault(const KeyValue *const this, const Variant *const key, const Variant *const defaultValue) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_PARAM(VARIANT, defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); // Find the key const unsigned int listIdx = kvGetIdx(this, key); // If key not found then return default, else return the value if (listIdx == KEY_NOT_FOUND) FUNCTION_TEST_RETURN_CONST(VARIANT, defaultValue); FUNCTION_TEST_RETURN(VARIANT, ((KeyValuePair *)lstGet(this->list, listIdx))->value); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * kvGetList(const KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); // Get the value const Variant *const value = kvGet(this, key); // Return the list if it already is one if (value != NULL && varType(value) == varTypeVariantList) FUNCTION_TEST_RETURN(VARIANT_LIST, varLstDup(varVarLst(value))); // Create a list to return VariantList *result; VariantList *const list = varLstNew(); MEM_CONTEXT_OBJ_BEGIN(list) { result = varLstAdd(list, varDup(value)); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(VARIANT_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * kvRemove(KeyValue *const this, const Variant *const key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, this); FUNCTION_TEST_PARAM(VARIANT, key); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(key != NULL); // Find the key const unsigned int listIdx = kvGetIdx(this, key); // If the key was found, remove it if (listIdx != KEY_NOT_FOUND) { // Free the key/value being removed and remove from the list const KeyValuePair *const pair = (KeyValuePair *)lstGet(this->list, listIdx); varFree(pair->key); varFree(pair->value); lstRemoveIdx(this->list, listIdx); // Remove from the key list (index must be the same as the key/value list) ASSERT(varEq(key, varLstGet(this->pub.keyList, listIdx))); varFree(varLstGet(this->pub.keyList, listIdx)); lstRemoveIdx((List *)this->pub.keyList, listIdx); } FUNCTION_TEST_RETURN(KEY_VALUE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/keyValue.h0000664000175100017510000001173415240547630020417 0ustar00runnerrunner/*********************************************************************************************************************************** Key Value Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_KEYVALUE_H #define COMMON_TYPE_KEYVALUE_H /*********************************************************************************************************************************** KeyValue object ***********************************************************************************************************************************/ typedef struct KeyValue KeyValue; #include "common/type/object.h" #include "common/type/variantList.h" /*********************************************************************************************************************************** Constant to indicate key not found ***********************************************************************************************************************************/ #define KEY_NOT_FOUND UINT_MAX /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN KeyValue *kvNew(void); FN_EXTERN KeyValue *kvDup(const KeyValue *source); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct KeyValuePub { VariantList *keyList; // List of keys } KeyValuePub; // List of keys FN_INLINE_ALWAYS const VariantList * kvKeyList(const KeyValue *const this) { return THIS_PUB(KeyValue)->keyList; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add a value to an existing key. If the key does not exist then this works the same as kvPut(). If the key does exist then the // value is converted to a VariantList with the existing value and the new value as list items. Subsequent calls expand the list. // // NOTE: this function is seldom required and kvPut() should be used when a key is expected to have a single value. FN_EXTERN KeyValue *kvAdd(KeyValue *this, const Variant *key, const Variant *value); // Move to a new parent mem context FN_INLINE_ALWAYS KeyValue * kvMove(KeyValue *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Put key/value pair FN_EXTERN KeyValue *kvPut(KeyValue *this, const Variant *key, const Variant *value); // Put key/value store. If this is called on an existing key it will replace the key with an empty kev/value store, even if the key // already contains a key/value store. FN_EXTERN KeyValue *kvPutKv(KeyValue *this, const Variant *key); // Get a value using the key FN_EXTERN const Variant *kvGet(const KeyValue *this, const Variant *key); // Get a value using the key and return a default if not found FN_EXTERN const Variant *kvGetDefault(const KeyValue *this, const Variant *key, const Variant *defaultValue); // Get key index if it exists FN_EXTERN unsigned int kvGetIdx(const KeyValue *this, const Variant *key); // Does the key exist (even if the value is NULL) FN_INLINE_ALWAYS bool kvKeyExists(const KeyValue *const this, const Variant *const key) { return kvGetIdx(this, key) != KEY_NOT_FOUND; } // Get a value as a list (even if there is only one value) using the key FN_EXTERN VariantList *kvGetList(const KeyValue *this, const Variant *key); // Remove a key/value pair FN_EXTERN KeyValue *kvRemove(KeyValue *this, const Variant *key); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void kvFree(KeyValue *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_KEY_VALUE_TYPE \ KeyValue * #define FUNCTION_LOG_KEY_VALUE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "KeyValue", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/list.c0000664000175100017510000003540715240547630017603 0ustar00runnerrunner/*********************************************************************************************************************************** List Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/type/list.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct List { ListPub pub; // Publicly accessible variables unsigned int listSizeMax; SortOrder sortOrder; uint8_t *listAlloc; // Pointer to memory allocated for the list ListComparator *comparator; }; /**********************************************************************************************************************************/ FN_EXTERN List * lstNew(const size_t itemSize, const ListParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, itemSize); FUNCTION_TEST_PARAM(FUNCTIONP, param.comparator); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(List, .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { *this = (List) { .pub = { .itemSize = itemSize, }, .sortOrder = param.sortOrder, .comparator = param.comparator, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN List * lstClear(List *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (this->pub.list != NULL) { MEM_CONTEXT_BEGIN(lstMemContext(this)) { memFree(this->pub.list); } MEM_CONTEXT_END(); this->pub.list = NULL; this->pub.listSize = 0; this->listSizeMax = 0; } FUNCTION_TEST_RETURN(LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN int lstComparatorStr(const void *const item1, const void *const item2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, item1); FUNCTION_TEST_PARAM_P(VOID, item2); FUNCTION_TEST_END(); ASSERT(item1 != NULL); ASSERT(item2 != NULL); FUNCTION_TEST_RETURN(INT, strCmp(*(const String *const *)item1, *(const String *const *)item2)); } /**********************************************************************************************************************************/ FN_EXTERN int lstComparatorInt(const void *const intPtr1, const void *const intPtr2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, intPtr1); FUNCTION_TEST_PARAM_P(VOID, intPtr2); FUNCTION_TEST_END(); ASSERT(intPtr1 != NULL); ASSERT(intPtr2 != NULL); const int int1 = *(const int *)intPtr1; const int int2 = *(const int *)intPtr2; FUNCTION_TEST_RETURN(INT, LST_COMPARATOR_CMP(int1, int2)); } /**********************************************************************************************************************************/ FN_EXTERN int lstComparatorUInt(const void *const uintPtr1, const void *const uintPtr2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, uintPtr1); FUNCTION_TEST_PARAM_P(VOID, uintPtr2); FUNCTION_TEST_END(); ASSERT(uintPtr1 != NULL); ASSERT(uintPtr2 != NULL); const unsigned int uint1 = *(const unsigned int *)uintPtr1; const unsigned int uint2 = *(const unsigned int *)uintPtr2; FUNCTION_TEST_RETURN(INT, LST_COMPARATOR_CMP(uint1, uint2)); } /**********************************************************************************************************************************/ FN_EXTERN int lstComparatorZ(const void *const item1, const void *const item2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, item1); FUNCTION_TEST_PARAM_P(VOID, item2); FUNCTION_TEST_END(); ASSERT(item1 != NULL); ASSERT(item2 != NULL); FUNCTION_TEST_RETURN(INT, strcmp(*(const char *const *)item1, *(const char *const *)item2)); } /*********************************************************************************************************************************** General function for a descending comparator that simply switches the parameters on the main comparator (which should be asc) ***********************************************************************************************************************************/ static const List *comparatorDescList = NULL; static int lstComparatorDesc(const void *const item1, const void *const item2) { return comparatorDescList->comparator(item2, item1); } /**********************************************************************************************************************************/ FN_EXTERN void * lstGet(const List *const this, const unsigned int listIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM(UINT, listIdx); FUNCTION_TEST_END(); ASSERT(this != NULL); // Ensure list index is in range if (listIdx >= lstSize(this)) THROW_FMT(AssertError, "cannot get index %u from list with %u value(s)", listIdx, lstSize(this)); // Return pointer to list item FUNCTION_TEST_RETURN_P(VOID, this->pub.list + (listIdx * this->pub.itemSize)); } FN_EXTERN void * lstGetLast(const List *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Ensure there are items in the list if (lstSize(this) == 0) THROW(AssertError, "cannot get last from list with no values"); // Return pointer to list item FUNCTION_TEST_RETURN_P(VOID, lstGet(this, lstSize(this) - 1)); } /**********************************************************************************************************************************/ FN_EXTERN void * lstFind(const List *const this, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->comparator != NULL); ASSERT(item != NULL); if (this->pub.list != NULL) { if (this->sortOrder == sortOrderAsc) FUNCTION_TEST_RETURN_P(VOID, bsearch(item, this->pub.list, lstSize(this), this->pub.itemSize, this->comparator)); else if (this->sortOrder == sortOrderDesc) { // Assign the list for the descending comparator to use comparatorDescList = this; FUNCTION_TEST_RETURN_P(VOID, bsearch(item, this->pub.list, lstSize(this), this->pub.itemSize, lstComparatorDesc)); } // Fall back on an iterative search for (unsigned int listIdx = 0; listIdx < lstSize(this); listIdx++) { if (this->comparator(item, lstGet(this, listIdx)) == 0) FUNCTION_TEST_RETURN_P(VOID, lstGet(this, listIdx)); } } FUNCTION_TEST_RETURN_P(VOID, NULL); } FN_EXTERN unsigned int lstFindIdx(const List *const this, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(item != NULL); void *const result = lstFind(this, item); FUNCTION_TEST_RETURN(UINT, result == NULL ? LIST_NOT_FOUND : lstIdx(this, result)); } FN_EXTERN const void * lstFindDefault(const List *const this, const void *const item, const void *const itemDefault) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_PARAM_P(VOID, itemDefault); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(item != NULL); void *const result = lstFind(this, item); FUNCTION_TEST_RETURN_CONST_P(VOID, result == NULL ? itemDefault : result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int lstIdx(const List *const this, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(item != NULL); // Item pointers should always be aligned with the beginning of an item in the list ASSERT((size_t)((const uint8_t *const)item - this->pub.list) % this->pub.itemSize == 0); const size_t result = (size_t)((const uint8_t *const)item - this->pub.list) / this->pub.itemSize; // Item pointers should always be in range ASSERT(result < lstSize(this)); FUNCTION_TEST_RETURN(UINT, (unsigned int)result); } /**********************************************************************************************************************************/ FN_EXTERN void * lstInsert(List *const this, const unsigned int listIdx, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(listIdx <= lstSize(this)); ASSERT(item != NULL); // If list size = max then allocate more space if (lstSize(this) == this->listSizeMax) { MEM_CONTEXT_BEGIN(lstMemContext(this)) { // If nothing has been allocated yet if (this->listSizeMax == 0) { this->listSizeMax = LIST_INITIAL_SIZE; this->pub.list = memNew(this->listSizeMax * this->pub.itemSize); } // Else the list needs to be extended else { this->listSizeMax *= 2; this->pub.list = memResize(this->pub.list, this->listSizeMax * this->pub.itemSize); } this->listAlloc = this->pub.list; } MEM_CONTEXT_END(); } // Else if there is space before the beginning of the list then move the list down else if ( (this->pub.list != this->listAlloc) && (lstSize(this) + ((uintptr_t)(this->pub.list - this->listAlloc) / this->pub.itemSize) == this->listSizeMax)) { memmove(this->listAlloc, this->pub.list, lstSize(this) * this->pub.itemSize); this->pub.list = this->listAlloc; } // Calculate the position where this item will be copied void *const itemPtr = this->pub.list + (listIdx * this->pub.itemSize); // If not inserting at the end then move items down to make space if (listIdx != lstSize(this)) memmove(this->pub.list + ((listIdx + 1) * this->pub.itemSize), itemPtr, (lstSize(this) - listIdx) * this->pub.itemSize); // Copy item into the list this->sortOrder = sortOrderNone; memcpy(itemPtr, item, this->pub.itemSize); this->pub.listSize++; FUNCTION_TEST_RETURN_P(VOID, itemPtr); } /**********************************************************************************************************************************/ FN_EXTERN List * lstRemoveIdx(List *const this, const unsigned int listIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM(UINT, listIdx); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(listIdx <= lstSize(this)); // Decrement the list size this->pub.listSize--; // If this is the first item then move the list pointer up to avoid moving all the items if (listIdx == 0) { this->pub.list += this->pub.itemSize; } // Else remove the item by moving the items after it down else { memmove( this->pub.list + (listIdx * this->pub.itemSize), this->pub.list + ((listIdx + 1) * this->pub.itemSize), (lstSize(this) - listIdx) * this->pub.itemSize); } FUNCTION_TEST_RETURN(LIST, this); } FN_EXTERN bool lstRemove(List *const this, const void *const item) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM_P(VOID, item); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(item != NULL); const unsigned int listIdx = lstFindIdx(this, item); if (listIdx != LIST_NOT_FOUND) { lstRemoveIdx(this, listIdx); FUNCTION_TEST_RETURN(BOOL, true); } FUNCTION_TEST_RETURN(BOOL, false); } FN_EXTERN List * lstRemoveLast(List *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (lstSize(this) == 0) THROW(AssertError, "cannot remove last from list with no values"); FUNCTION_TEST_RETURN(LIST, lstRemoveIdx(this, lstSize(this) - 1)); } /**********************************************************************************************************************************/ FN_EXTERN List * lstSort(List *const this, const SortOrder sortOrder) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM(ENUM, sortOrder); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->comparator != NULL); if (this->pub.list != NULL) { switch (sortOrder) { case sortOrderAsc: qsort(this->pub.list, lstSize(this), this->pub.itemSize, this->comparator); break; case sortOrderDesc: { // Assign the list that will be sorted for the comparator function to use comparatorDescList = this; qsort(this->pub.list, lstSize(this), this->pub.itemSize, lstComparatorDesc); break; } case sortOrderNone: break; } } this->sortOrder = sortOrder; FUNCTION_TEST_RETURN(LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN List * lstComparatorSet(List *const this, ListComparator *const comparator) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(LIST, this); FUNCTION_TEST_PARAM(FUNCTIONP, comparator); FUNCTION_TEST_END(); ASSERT(this != NULL); this->comparator = comparator; this->sortOrder = sortOrderNone; FUNCTION_TEST_RETURN(LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN void lstToLog(const List *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{size: %u}", lstSize(this)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/list.h0000664000175100017510000002217315240547630017604 0ustar00runnerrunner/*********************************************************************************************************************************** List Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_LIST_H #define COMMON_TYPE_LIST_H #include /*********************************************************************************************************************************** List object ***********************************************************************************************************************************/ typedef struct List List; #include "common/memContext.h" #include "common/type/object.h" #include "common/type/param.h" #include "common/type/string.h" /*********************************************************************************************************************************** Sort orders ***********************************************************************************************************************************/ typedef enum { sortOrderNone, sortOrderAsc, sortOrderDesc, } SortOrder; /*********************************************************************************************************************************** Define initial size of a list ***********************************************************************************************************************************/ #define LIST_INITIAL_SIZE 8 /*********************************************************************************************************************************** Item was not found in the list ***********************************************************************************************************************************/ #define LIST_NOT_FOUND UINT_MAX /*********************************************************************************************************************************** Function type for comparing items in the list The return value should be -1 when item1 < item2, 0 when item1 == item2, and 1 when item2 > item1. ***********************************************************************************************************************************/ typedef int ListComparator(const void *item1, const void *item2); // General purpose list comparator for Strings or structs with a String as the first member FN_EXTERN int lstComparatorStr(const void *item1, const void *item2); // General purpose list comparator for unsigned int or structs with an unsigned int as the first member FN_EXTERN int lstComparatorUInt(const void *item1, const void *item2); // General purpose list comparator for int or structs with an int as the first member FN_EXTERN int lstComparatorInt(const void *item1, const void *item2); // General purpose list comparator for zero-terminated strings or structs with a zero-terminated string as the first member FN_EXTERN int lstComparatorZ(const void *item1, const void *item2); // Macro to compare two values in a branchless and transitive fashion #define LST_COMPARATOR_CMP(item1, item2) \ (((item1) > (item2)) - ((item1) < (item2))) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct ListParam { VAR_PARAM_HEADER; SortOrder sortOrder; ListComparator *comparator; } ListParam; #define lstNewP(itemSize, ...) \ lstNew(itemSize, (ListParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN List *lstNew(size_t itemSize, ListParam param); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ListPub { unsigned int listSize; // List size size_t itemSize; // Size of item stored in the list uint8_t *list; // Pointer to the current start of the list } ListPub; // Set a new comparator FN_EXTERN List *lstComparatorSet(List *this, ListComparator *comparator); // Memory context for this list FN_INLINE_ALWAYS MemContext * lstMemContext(List *const this) { return objMemContext(this); } // List size FN_INLINE_ALWAYS unsigned int lstSize(const List *const this) { return THIS_PUB(List)->listSize; } // Is the list empty? FN_INLINE_ALWAYS bool lstEmpty(const List *const this) { return lstSize(this) == 0; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Clear items from a list FN_EXTERN List *lstClear(List *this); // Get an item from the list FN_EXTERN void *lstGet(const List *this, unsigned int listIdx); FN_EXTERN void *lstGetLast(const List *this); // Find an item in the list FN_EXTERN void *lstFind(const List *this, const void *item); FN_EXTERN const void *lstFindDefault(const List *this, const void *item, const void *itemDefault); FN_EXTERN unsigned int lstFindIdx(const List *this, const void *item); // Does an item exist in the list? FN_INLINE_ALWAYS bool lstExists(const List *const this, const void *const item) { return lstFind(this, item) != NULL; } // Get the index of a list item FN_EXTERN unsigned int lstIdx(const List *this, const void *item); // Insert an item into the list FN_EXTERN void *lstInsert(List *this, unsigned int listIdx, const void *item); // Add an item to the end of the list FN_INLINE_ALWAYS void * lstAdd(List *const this, const void *const item) { return lstInsert(this, lstSize(this), item); } // Move to a new parent mem context FN_INLINE_ALWAYS List * lstMove(List *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Remove an item from the list FN_EXTERN bool lstRemove(List *this, const void *item); FN_EXTERN List *lstRemoveIdx(List *this, unsigned int listIdx); FN_EXTERN List *lstRemoveLast(List *this); // List sort FN_EXTERN List *lstSort(List *this, SortOrder sortOrder); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void lstFree(List *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for constant lists Frequently used constant lists can be declared with these macros at compile time rather than dynamically at run time. Note that lists created in this way are declared as const so can't be modified or freed by the lst*() methods. Casting to List * will result in a segfault due to modifying read-only memory. By convention all List constant identifiers are appended with _LST. ***********************************************************************************************************************************/ // This struct must be kept in sync with ListPub (except for const qualifiers) typedef struct ListPubConst { unsigned int listSize; // List size size_t itemSize; // Size of item stored in the list const uint8_t *list; // Pointer to the current start of the list } ListPubConst; // Create a List constant inline from a constant array #define LSTDEF(listParam) \ (const List *)&(ListPubConst){ \ .list = (const uint8_t *)listParam, .itemSize = sizeof((listParam)[0]), .listSize = LENGTH_OF(listParam)} /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void lstToLog(const List *this, StringStatic *debugLog); #define FUNCTION_LOG_LIST_TYPE \ List * #define FUNCTION_LOG_LIST_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, lstToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/object.c0000664000175100017510000000240615240547630020067 0ustar00runnerrunner/*********************************************************************************************************************************** Object Helper Macros and Functions ***********************************************************************************************************************************/ #include #include "common/type/object.h" /**********************************************************************************************************************************/ FN_EXTERN void * objMove(THIS_VOID, MemContext *const parentNew) { if (thisVoid != NULL) memContextMove(memContextFromAllocExtra(thisVoid), parentNew); return thisVoid; } /**********************************************************************************************************************************/ FN_EXTERN void * objMoveToInterface(THIS_VOID, void *const interfaceVoid, const MemContext *const current) { return objMemContext(thisVoid) != current ? objMove(thisVoid, objMemContext(interfaceVoid)) : thisVoid; } /**********************************************************************************************************************************/ FN_EXTERN void objFree(THIS_VOID) { if (thisVoid != NULL) memContextFree(memContextFromAllocExtra(thisVoid)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/object.h0000664000175100017510000001632115240547630020075 0ustar00runnerrunner/*********************************************************************************************************************************** Object Helper Macros and Functions These macros and functions implement common object functionality. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_OBJECT_H #define COMMON_TYPE_OBJECT_H #include "common/assert.h" #include "common/macro.h" #include "common/memContext.h" /*********************************************************************************************************************************** Create a new object This is a thin wrapper on MEM_CONTEXT_NEW_*() to do object specific initialization. The general pattern for a new object is: OBJ_NEW_BEGIN(MyObj) { *this = (MyObj) { .data = ... }; } OBJ_NEW_END(); The this variable is variable is automatically created with the specified type. The BASE and EXTRA variants are available when a custom variable must be used or different amount of memory must be allocated. ***********************************************************************************************************************************/ #define OBJ_NEW_BASE_EXTRA_BEGIN(type, extra, ...) \ MEM_CONTEXT_NEW_BEGIN(type, .allocExtra = extra, __VA_ARGS__) #define OBJ_NEW_EXTRA_BEGIN(type, extra, ...) \ memContextSwitch(memContextNewP(STRINGIFY(type), .allocExtra = extra, __VA_ARGS__)); \ type *const this = OBJ_NEW_ALLOC(); \ \ do \ { #define OBJ_NEW_BASE_BEGIN(type, ...) \ OBJ_NEW_BASE_EXTRA_BEGIN(type, sizeof(type), __VA_ARGS__) #define OBJ_NEW_BEGIN(type, ...) \ OBJ_NEW_EXTRA_BEGIN(type, sizeof(type), __VA_ARGS__) #define OBJ_NEW_ALLOC() \ memContextAllocExtra(memContextCurrent()) #define OBJ_NEW_END() \ MEM_CONTEXT_NEW_END() /*********************************************************************************************************************************** Rename an object for auditing purposes. The original name for an object is based on the type used to create the object but a different name might be needed to identify it during auditing. For example, a filter named IoSink would need to be renamed to IoFilter to identify it as a filter for auditing. This only has an effect in test builds. ***********************************************************************************************************************************/ #define OBJ_NAME(this, name) MEM_CONTEXT_AUDIT_ALLOC_EXTRA_NAME(this, name) /*********************************************************************************************************************************** Used in interface function parameter lists to discourage use of the untyped thisVoid parameter, e.g.: size_t bufferRead(THIS_VOID, Buffer *buffer) This macro should not be used unless the function is assigned to an interface or passed as a parameter. ***********************************************************************************************************************************/ #define THIS_VOID void *thisVoid /*********************************************************************************************************************************** Create a local "this" variable of the correct type from a THIS_VOID parameter ***********************************************************************************************************************************/ #define THIS(type) type *this = thisVoid /*********************************************************************************************************************************** Cast this private struct, e.g. List, to the associated public struct, e.g. ListPub. Note that the public struct must be the first member of the private struct. For example: FN_INLINE_ALWAYS unsigned int lstSize(const List *const this) { return THIS_PUB(List)->listSize; } The macro also ensures that this != NULL so there is no need to do that in the calling function. ***********************************************************************************************************************************/ #define THIS_PUB(type) ((const type##Pub *)thisNotNull(this)) FN_INLINE_ALWAYS const void * thisNotNull(const void *const this) { ASSERT_INLINE(this != NULL); return this; } /*********************************************************************************************************************************** Switch to the object memory context and ensure that the prior memory context is restored after the block executes (even on error) ***********************************************************************************************************************************/ #define MEM_CONTEXT_OBJ_BEGIN(this) MEM_CONTEXT_BEGIN(objMemContext(this)) #define MEM_CONTEXT_OBJ_END() MEM_CONTEXT_END() /*********************************************************************************************************************************** Functions To ensure proper type checking, these functions are meant to be called from inline functions created specifically for each object: FN_INLINE_ALWAYS void storageFree(Storage *this) { objFree(this); } ***********************************************************************************************************************************/ // Get the object mem context FN_INLINE_ALWAYS MemContext * objMemContext(void *const this) { return memContextFromAllocExtra(this); } // Move an object to a new context if this != NULL FN_EXTERN void *objMove(THIS_VOID, MemContext *parentNew); // Move driver object into an interface object when required. This is the general case, but sometimes an object will expose its // interfaces in a different way, e.g. methods, so the object retains ownership of the interfaces. If this function is called in the // driver object context then the driver object will retain ownership of the interface. Otherwise, the interface object will become // the owner of the driver object. FN_EXTERN void *objMoveToInterface(THIS_VOID, void *interfaceVoid, const MemContext *current); // Free the object mem context if this != NULL FN_EXTERN void objFree(THIS_VOID); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/pack.c0000664000175100017510000020337515240547630017547 0ustar00runnerrunner/*********************************************************************************************************************************** Pack Type Each pack field begins with a one byte tag. The four high order bits of the tag contain the field type (PackTypeMap). To allow more types than four bits will allow, the type bits can be set to 0xF and the rest of the type (- 0xF) will be stored by a base-128 encoded integer immediately following the tag. The four lower order bits vary by type. When the "more ID delta" indicator is set then the tag (and type, if any) will be followed by a base-128 encoded integer with the higher order ID delta bits. The ID delta represents the delta from the ID of the previous field. When the "more value indicator" then the tag (and the type and ID delta, if any) will be followed by a base-128 encoded integer with the high order value bits, i.e. the bits that were not stored directly in the tag. For integer types the value is the integer being stored but for string and binary types the value is 1 if the size is greater than 0 and 0 if the size is 0. When the size is greater than 0 the tag is immediately followed by (or after the delta ID if "more ID delta" is set) the base-128 encoded size and then by the string/binary bytes. For string and binary types the value bit indicates if there is data, not the length of the data, which is why the length is stored immediately following the tag when the value bit is set. This prevents storing an additional byte when the string/binary length is zero. The following are definitions for the pack tag field and examples of how it is interpreted. Integer types (packTypeMapData[type].valueMultiBit) when an unsigned value is <= 1 or a signed value is >= -1 and <= 0: 3 - more value indicator bit set to 0 2 - value low order bit 1 - more ID delta indicator bit 0 - ID delta low order bit Example: b704 b = unsigned int 64 type 7 = tag byte low bits: 0 1 1 1 meaning: "value low order bit" - the value of the u64 field is 1 "more ID delta indicator bit" - there exists a gap (i.e. NULLs are not stored so there is a gap between the stored IDs) "ID delta low order bit" - gaps are interpreted as the currently stored ID minus previously stored ID minus 1, therefore if the previously store ID is 1 and the ID of this u64 field is 11 then a gap of 10 exists. 10 is represented internally as 9 since there is always at least a gap of 1 which never needs to be recorded (it is a given). 9 in bit format is 1 0 0 1 - the low-order bit is 1 so the "ID delta low order bit" is set. 04 = since the low order bit of the internal ID delta was already set in bit 0 of the tag byte, then the remain bits are shifted right by one and represented in this second byte as 4. To get the ID delta for 04, shift the 4 back to the left one and then add back the "ID delta low order bit" to give a binary representation of 1 0 0 1 = 9. Add back the 1 which is never recorded and the ID gap is 10. Integer types (packTypeMapData[type].valueMultiBit) when an unsigned value is > 1 or a signed value is < -1 or > 0: 3 - more value indicator bit set to 1 2 - more ID delta indicator bit 0-1 - ID delta low order bits Example: 5e021f 5 = signed int 64 type e = tag byte low bits: 1 1 1 0 meaning: "more value indicator bit set to 1" - the actual value is < -1 or > 0 "more ID delta indicator bit" - there exists a gap (i.e. NULLs are not stored so there is a gap between the stored IDs) "ID delta low order bits" - here the bit 1 is set to 1 and bit 0 is not so the ID delta has the second low order bit set but not the first 02 = since bit 0 and bit 1 of the tag byte are accounted for then the 02 is the result of shifting the ID delta right by 2. Shifting the 2 back to the left by 2 and adding back the second low order bit as 1 and the first low order bit as 0 then the bit representation would be 1 0 1 0 which is ten (10) so the gap between the IDs is 11. 1f = signed, zigzag representation of -16 (the actual value) String, binary types, and boolean (packTypeMapData[type].valueSingleBit): 3 - value bit 2 - more ID delta indicator bit 0-1 - ID delta low order bits Note: binary type is interpreted the same way as string type Example: 8c090673616d706c65 8 = string type c = tag byte low bits: 1 1 0 0 meaning: "value bit" - there is data "more ID delta indicator bit" - there exists a gap (i.e. NULLs are not stored so there is a gap between the stored IDs) 09 = since neither "ID delta low order bits" is set in the tag, they are both 0, so shifting 9 left by 2, the 2 low order bits are now 0 so the result is 0x24 = 36 in decimal. Add back the 1 which is never recorded and the ID gap is 37. 06 = the length of the string is 6 bytes 73616d706c65 = the 6 bytes of the string value ("sample") Example: 30 3 = boolean type 0 = "value bit" 0 means the value is false Note that if the boolean had been pack written with .defaultWrite = false there would have been a gap instead of the 30. Array and object types: 3 - more ID delta indicator bit 0-2 - ID delta low order bits Note: arrays and objects are merely containers for the other pack types. Example: 1801 (container begin) 1 = array type 8 = "more ID delta indicator bit" - there exists a gap (i.e. NULLs are not stored so there is a gap between the stored IDs) 01 = since there are three "ID delta low order bits", the 01 will be shifted left by 3 with zeros, resulting in 8. Add back the 1 which is never recorded and the ID gap is 9. ... 00 = container end - the array/object container end will occur when a 0 byte (00) is encountered that is not part of a pack field within the array/object ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/io/bufferWrite.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/io/write.h" #include "common/type/convert.h" #include "common/type/pack.h" /*********************************************************************************************************************************** Map PackType types to the types that will be written into the pack. This hides the details of the type IDs from the user and allows the IDs used in the pack to differ from the IDs the user sees. ***********************************************************************************************************************************/ typedef enum { pckTypeMapUnknown = 0, // Used internally when the type is not known pckTypeMapArray = 1, // Maps to pckTypeArray pckTypeMapBool = 2, // Maps to pckTypeBool pckTypeMapI32 = 3, // Maps to pckTypeI32 pckTypeMapI64 = 4, // Maps to pckTypeI64 pckTypeMapObj = 5, // Maps to pckTypeObj // The empty position 6 can be used for a new type that will be encoded entirely in the tag pckTypeMapStr = 7, // Maps to pckTypeStr pckTypeMapU32 = 8, // Maps to pckTypeU32 pckTypeMapU64 = 9, // Maps to pckTypeU64 pckTypeMapStrId = 10, // Maps to pckTypeStrId // The empty positions before 15 can be used for new types that will be encoded entirely in the tag pckTypeMapTime = 15, // Maps to pckTypeTime pckTypeMapBin = 16, // Maps to pckTypeBin pckTypeMapPack = 17, // Maps to pckTypePack pckTypeMapMode = 18, // Maps to pckTypeMode } PackTypeMap; typedef struct PackTypeMapData { PackType type; // Data type bool valueSingleBit; // Can the value be stored in a single bit (e.g. bool) bool valueMultiBit; // Can the value require multiple bits (e.g. integer) bool size; // Does the type require a size (e.g. string) } PackTypeMapData; static const PackTypeMapData packTypeMapData[] = { // Unknown type map data should not be used [pckTypeMapUnknown] = {0}, // Types that can be encoded entirely in the tag [pckTypeMapArray] = { .type = pckTypeArray, }, [pckTypeMapBool] = { .type = pckTypeBool, .valueSingleBit = true, }, [pckTypeMapI32] = { .type = pckTypeI32, .valueMultiBit = true, }, [pckTypeMapI64] = { .type = pckTypeI64, .valueMultiBit = true, }, [pckTypeMapObj] = { .type = pckTypeObj, }, [pckTypeMapStr] = { .type = pckTypeStr, .valueSingleBit = true, .size = true, }, [pckTypeMapU32] = { .type = pckTypeU32, .valueMultiBit = true, }, [pckTypeMapU64] = { .type = pckTypeU64, .valueMultiBit = true, }, [pckTypeMapStrId] = { .type = pckTypeStrId, .valueMultiBit = true, }, // Types that require an extra byte to encode [pckTypeMapTime] = { .type = pckTypeTime, .valueMultiBit = true, }, [pckTypeMapBin] = { .type = pckTypeBin, .valueSingleBit = true, .size = true, }, [pckTypeMapPack] = { .type = pckTypePack, .size = true, }, [pckTypeMapMode] = { .type = pckTypeMode, .valueMultiBit = true, }, }; /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ typedef struct PackTagStackItem { PackTypeMap typeMap; // Tag type map unsigned int idLast; // Last id in the container unsigned int nullTotal; // Total nulls since last tag written } PackTagStackItem; typedef struct PackTagStack { List *stack; // Stack of object/array tags PackTagStackItem bottom; // Static bottom of the stack PackTagStackItem *top; // Top tag on the stack unsigned int depth; // Stack depth } PackTagStack; struct PackRead { IoRead *read; // Read pack from Buffer *buffer; // Buffer containing read data const uint8_t *bufferPtr; // Pointer to buffer size_t bufferPos; // Position in the buffer size_t bufferUsed; // Amount of data in the buffer unsigned int tagNextId; // Next tag id PackTypeMap tagNextTypeMap; // Next tag type map uint64_t tagNextValue; // Next tag value size_t tagNextSize; // Next tag size PackTagStack tagStack; // Stack of object/array tags }; struct PackWrite { IoWrite *write; // Write pack to Buffer *buffer; // Buffer to contain write data PackTagStack tagStack; // Stack of object/array tags }; /*********************************************************************************************************************************** Push a container onto the tag stack ***********************************************************************************************************************************/ static void pckTagStackPush(MemContext *const memContext, PackTagStack *const tagStack, const PackTypeMap typeMap) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MEM_CONTEXT, memContext); FUNCTION_TEST_PARAM_P(VOID, tagStack); FUNCTION_TEST_PARAM(ENUM, typeMap); FUNCTION_TEST_END(); ASSERT(tagStack != NULL); ASSERT(typeMap == pckTypeMapArray || typeMap == pckTypeMapObj); if (tagStack->stack == NULL) { MEM_CONTEXT_BEGIN(memContext) { tagStack->stack = lstNewP(sizeof(PackTagStackItem)); } MEM_CONTEXT_END(); } tagStack->top = lstAdd(tagStack->stack, &(PackTagStackItem){.typeMap = typeMap}); tagStack->depth++; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Check that a container can be popped off the stack ***********************************************************************************************************************************/ static void pckTagStackPopCheck(PackTagStack *const tagStack, const PackTypeMap typeMap) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, tagStack); FUNCTION_TEST_PARAM(ENUM, typeMap); FUNCTION_TEST_END(); ASSERT(tagStack != NULL); ASSERT(typeMap == pckTypeMapArray || typeMap == pckTypeMapObj); if (tagStack->depth == 0 || tagStack->top->typeMap != typeMap) THROW_FMT(FormatError, "not in %s", zNewStrId(packTypeMapData[typeMap].type)); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Pop a container off the tag stack ***********************************************************************************************************************************/ static void pckTagStackPop(PackTagStack *const tagStack) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, tagStack); FUNCTION_TEST_END(); ASSERT(tagStack != NULL); ASSERT(tagStack->depth != 0); ASSERT(tagStack->stack != NULL); lstRemoveLast(tagStack->stack); tagStack->depth--; if (tagStack->depth == 0) tagStack->top = &tagStack->bottom; else tagStack->top = lstGetLast(tagStack->stack); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to create common data static PackRead * pckReadNewInternal(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(PackRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (PackRead) { .tagStack = {.bottom = {.typeMap = pckTypeMapObj}}, }; this->tagStack.top = &this->tagStack.bottom; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(PACK_READ, this); } FN_EXTERN PackRead * pckReadNewIo(IoRead *const read) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_READ, read); FUNCTION_TEST_END(); ASSERT(read != NULL); PackRead *const this = pckReadNewInternal(); MEM_CONTEXT_OBJ_BEGIN(this) { this->read = read; this->buffer = bufNew(ioBufferSize()); this->bufferPtr = bufPtr(this->buffer); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(PACK_READ, this); } FN_EXTERN PackRead * pckReadNew(const Pack *const pack) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK, pack); FUNCTION_TEST_END(); if (pack == NULL) FUNCTION_TEST_RETURN(PACK_READ, NULL); FUNCTION_TEST_RETURN(PACK_READ, pckReadNewC(bufPtrConst((const Buffer *)pack), bufUsed((const Buffer *)pack))); } FN_EXTERN PackRead * pckReadNewC(const uint8_t *const buffer, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(buffer != NULL); PackRead *const this = pckReadNewInternal(); this->bufferPtr = buffer; this->bufferUsed = size; FUNCTION_TEST_RETURN(PACK_READ, this); } /*********************************************************************************************************************************** Read bytes from the buffer IMPORTANT NOTE: To avoid having dynamically created return buffers the current buffer position (this->bufferPos) is stored in the object. Therefore this function should not be used as a parameter in other function calls since the value of this->bufferPos will change. ***********************************************************************************************************************************/ static size_t pckReadBuffer(PackRead *const this, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); size_t remaining = this->bufferUsed - this->bufferPos; if (remaining < size) { if (this->read != NULL) { // Nothing can be remaining since each read fetches exactly the number of bytes required ASSERT(remaining == 0); bufUsedZero(this->buffer); // Limit the buffer for the next read so we don't read past the end of the pack bufLimitSet(this->buffer, size < bufSizeAlloc(this->buffer) ? size : bufSizeAlloc(this->buffer)); // Read bytes ioReadSmall(this->read, this->buffer); this->bufferPos = 0; this->bufferUsed = bufUsed(this->buffer); remaining = this->bufferUsed; } if (remaining < 1) THROW(FormatError, "unexpected EOF"); FUNCTION_TEST_RETURN(SIZE, remaining < size ? remaining : size); } FUNCTION_TEST_RETURN(SIZE, size); } /*********************************************************************************************************************************** Consume buffer left in tagNextSize ***********************************************************************************************************************************/ static void pckReadConsumeBuffer(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); while (this->tagNextSize != 0) { const size_t size = pckReadBuffer(this, this->tagNextSize); this->bufferPos += size; this->tagNextSize -= size; } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Unpack an unsigned 64-bit integer from base-128 varint encoding ***********************************************************************************************************************************/ static uint64_t pckReadU64Internal(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (this->read != NULL) { // Internal buffer should be empty ASSERT(this->bufferUsed == this->bufferPos); FUNCTION_TEST_RETURN(UINT64, ioReadVarIntU64(this->read)); } FUNCTION_TEST_RETURN(UINT64, cvtUInt64FromVarInt128(this->bufferPtr, &this->bufferPos, this->bufferUsed)); } /*********************************************************************************************************************************** Read next field tag ***********************************************************************************************************************************/ static bool pckReadTagNext(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); bool result = false; // Read the tag byte pckReadBuffer(this, 1); const unsigned int tag = this->bufferPtr[this->bufferPos]; this->bufferPos++; // If the current container is complete (e.g. object) if (tag == 0) { this->tagNextId = UINT_MAX; } // Else a regular tag else { // Read field type (e.g. int64, string) this->tagNextTypeMap = tag >> 4; if (this->tagNextTypeMap == 0xF) this->tagNextTypeMap = (unsigned int)pckReadU64Internal(this) + 0xF; CHECK( FormatError, this->tagNextTypeMap < LENGTH_OF(packTypeMapData) && packTypeMapData[this->tagNextTypeMap].type != 0, "invalid tag type"); // If the value can contain multiple bits (e.g. integer) if (packTypeMapData[this->tagNextTypeMap].valueMultiBit) { // If the value is stored following the tag (value > 1 bit) if (tag & 0x8) { // Read low order bits of the field ID delta this->tagNextId = tag & 0x3; // Read high order bits of the field ID delta when specified if (tag & 0x4) this->tagNextId |= (unsigned int)pckReadU64Internal(this) << 2; // Read value this->tagNextValue = pckReadU64Internal(this); } // Else the value is stored in the tag (value == 1 bit) else { // Read low order bit of the field ID delta this->tagNextId = tag & 0x1; // Read high order bits of the field ID delta when specified if (tag & 0x2) this->tagNextId |= (unsigned int)pckReadU64Internal(this) << 1; // Read value this->tagNextValue = (tag >> 2) & 0x3; } } // Else the value is a single bit (e.g. boolean) else if (packTypeMapData[this->tagNextTypeMap].valueSingleBit) { // Read low order bits of the field ID delta this->tagNextId = tag & 0x3; // Read high order bits of the field ID delta when specified if (tag & 0x4) this->tagNextId |= (unsigned int)pckReadU64Internal(this) << 2; // Read value this->tagNextValue = (tag >> 3) & 0x1; } // Else the value is multiple tags (e.g. container) else { // Read low order bits of the field ID delta this->tagNextId = tag & 0x7; // Read high order bits of the field ID delta when specified if (tag & 0x8) this->tagNextId |= (unsigned int)pckReadU64Internal(this) << 3; // Value length is variable so is stored after the tag this->tagNextValue = 0; } // Increment the next tag id this->tagNextId += this->tagStack.top->idLast + 1; // Get tag size if it exists if (packTypeMapData[this->tagNextTypeMap].size && (this->tagNextValue > 0 || !packTypeMapData[this->tagNextTypeMap].valueSingleBit)) { this->tagNextSize = (size_t)pckReadU64Internal(this); } else this->tagNextSize = 0; // Tag was found result = true; } FUNCTION_TEST_RETURN(BOOL, result); } /*********************************************************************************************************************************** Read field tag Some tags and data may be skipped based on the value of the id parameter. ***********************************************************************************************************************************/ static uint64_t pckReadTag(PackRead *const this, unsigned int *const id, const PackTypeMap typeMap, const bool peek) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM_P(UINT, id); FUNCTION_TEST_PARAM(ENUM, typeMap); FUNCTION_TEST_PARAM(BOOL, peek); // Look at the next tag without advancing the field id FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(id != NULL); ASSERT((peek && typeMap == pckTypeMapUnknown) || (!peek && typeMap != pckTypeMapUnknown)); // Increment the id by one if no id was specified if (*id == 0) { *id = this->tagStack.top->idLast + 1; } // Else check that the id has been incremented else if (*id <= this->tagStack.top->idLast) THROW_FMT(FormatError, "field %u was already read", *id); // Search for the requested id do { // Get the next tag if it has not been read yet if (this->tagNextId == 0) pckReadTagNext(this); // Return if the id does not exist if (*id < this->tagNextId) { break; } // Else the id exists else if (*id == this->tagNextId) { // When not peeking the next tag (just to see what it is) then error if the type map is not as specified if (!peek) { if (this->tagNextTypeMap != typeMap) { THROW_FMT( FormatError, "field %u is type '%s' but expected '%s'", this->tagNextId, zNewStrId(packTypeMapData[this->tagNextTypeMap].type), zNewStrId(packTypeMapData[typeMap].type)); } this->tagStack.top->idLast = this->tagNextId; this->tagNextId = 0; } break; } // Read data for the field being skipped if this is not the field requested pckReadConsumeBuffer(this); // Increment the last id to the id just read this->tagStack.top->idLast = this->tagNextId; // Read tag on the next iteration this->tagNextId = 0; } while (1); FUNCTION_TEST_RETURN(UINT64, this->tagNextValue); } /**********************************************************************************************************************************/ FN_EXTERN bool pckReadNext(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->tagNextId == 0); FUNCTION_TEST_RETURN(BOOL, pckReadTagNext(this)); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int pckReadId(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(UINT, this->tagNextId); } /**********************************************************************************************************************************/ FN_EXTERN size_t pckReadSize(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(packTypeMapData[this->tagNextTypeMap].size); FUNCTION_TEST_RETURN(SIZE, this->tagNextSize); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadConsume(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); unsigned int id = 0; pckReadTag(this, &id, pckTypeMapUnknown, true); pckReadTag(this, &id, this->tagNextTypeMap, false); pckReadConsumeBuffer(this); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const uint8_t * pckReadBufPtr(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(packTypeMapData[this->tagNextTypeMap].size); ASSERT(this->buffer == NULL); FUNCTION_TEST_RETURN_CONST_P(BYTEDATA, this->bufferPtr + this->bufferPos); } /**********************************************************************************************************************************/ // Internal version of pckReadNull() that does not require a PackIdParam struct. Some functions already have an id variable so // assigning that to a PackIdParam struct and then copying it back is wasteful. static inline bool pckReadNullInternal(PackRead *const this, unsigned int *const id) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM_P(UINT, id); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(id != NULL); // Read tag at specified id pckReadTag(this, id, pckTypeMapUnknown, true); // If the field is NULL then set idLast (to avoid rechecking the same id on the next call) and return true if (*id < this->tagNextId) { this->tagStack.top->idLast = *id; FUNCTION_TEST_RETURN(BOOL, true); } // The field is not NULL FUNCTION_TEST_RETURN(BOOL, false); } FN_EXTERN bool pckReadNull(PackRead *const this, PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, pckReadNullInternal(this, ¶m.id)); } /**********************************************************************************************************************************/ FN_EXTERN PackType pckReadType(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING_ID, packTypeMapData[this->tagNextTypeMap].type); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadArrayBegin(PackRead *const this, PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); // Read array begin pckReadTag(this, ¶m.id, pckTypeMapArray, false); // Add array to the tag stack so IDs can be tracked separately from the parent container pckTagStackPush(objMemContext(this), &this->tagStack, pckTypeMapArray); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void pckReadArrayEnd(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Check tag stack pop pckTagStackPopCheck(&this->tagStack, pckTypeMapArray); // Make sure we are at the end of the array unsigned int id = UINT_MAX - 1; pckReadTag(this, &id, pckTypeMapUnknown, true); // Pop array off the stack pckTagStackPop(&this->tagStack); // Reset tagNextId to keep reading this->tagNextId = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * pckReadBin(PackRead *const this, PckReadBinParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(BUFFER, NULL); Buffer *result; // If buffer size > 0 if (pckReadTag(this, ¶m.id, pckTypeMapBin, false)) { // Get the buffer size result = bufNew(this->tagNextSize); // Read the buffer out in chunks while (bufUsed(result) < bufSize(result)) { size_t size = pckReadBuffer(this, bufRemains(result)); bufCatC(result, this->bufferPtr, this->bufferPos, size); this->bufferPos += size; } } // Else return a zero-sized buffer else result = bufNew(0); FUNCTION_TEST_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN bool pckReadBool(PackRead *const this, PckReadBoolParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(BOOL, param.defaultValue); FUNCTION_TEST_RETURN(BOOL, pckReadTag(this, ¶m.id, pckTypeMapBool, false)); } /**********************************************************************************************************************************/ FN_EXTERN int32_t pckReadI32(PackRead *const this, PckReadI32Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(INT, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(INT, param.defaultValue); FUNCTION_TEST_RETURN(INT, cvtInt32FromZigZag((uint32_t)pckReadTag(this, ¶m.id, pckTypeMapI32, false))); } /**********************************************************************************************************************************/ FN_EXTERN int64_t pckReadI64(PackRead *const this, PckReadI64Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(INT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(INT64, param.defaultValue); FUNCTION_TEST_RETURN(INT64, cvtInt64FromZigZag(pckReadTag(this, ¶m.id, pckTypeMapI64, false))); } /**********************************************************************************************************************************/ FN_EXTERN mode_t pckReadMode(PackRead *const this, PckReadModeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(MODE, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(MODE, param.defaultValue); FUNCTION_TEST_RETURN(MODE, (mode_t)pckReadTag(this, ¶m.id, pckTypeMapMode, false)); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadObjBegin(PackRead *const this, PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); // Read object begin pckReadTag(this, ¶m.id, pckTypeMapObj, false); // Add object to the tag stack so IDs can be tracked separately from the parent container pckTagStackPush(objMemContext(this), &this->tagStack, pckTypeMapObj); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void pckReadObjEnd(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Check tag stack pop pckTagStackPopCheck(&this->tagStack, pckTypeMapObj); // Make sure we are at the end of the object unsigned id = UINT_MAX - 1; pckReadTag(this, &id, pckTypeMapUnknown, true); // Pop object off the stack pckTagStackPop(&this->tagStack); // Reset tagNextId to keep reading this->tagNextId = 0; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * pckReadPackRead(PackRead *const this, PckReadPackParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); Pack *const pack = pckReadPack(this, param); PackRead *const result = pckReadNew(pack); if (result != NULL) pckMove(pack, objMemContext(result)); FUNCTION_TEST_RETURN(PACK_READ, result); } FN_EXTERN PackRead * pckReadPackReadConst(PackRead *const this, PckReadPackParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(PACK_READ, NULL); // Read the tag pckReadTag(this, ¶m.id, pckTypeMapPack, false); PackRead *const result = pckReadNewC(pckReadBufPtr(this), pckReadSize(this)); pckReadConsumeBuffer(this); FUNCTION_TEST_RETURN(PACK_READ, result); } FN_EXTERN Pack * pckReadPack(PackRead *const this, PckReadPackParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(PACK, NULL); // Read the tag pckReadTag(this, ¶m.id, pckTypeMapPack, false); // Get the pack size Buffer *const result = OBJ_NAME(bufNew(this->tagNextSize), Pack::Buffer); // Read the pack out in chunks while (bufUsed(result) < bufSize(result)) { size_t size = pckReadBuffer(this, bufRemains(result)); bufCatC(result, this->bufferPtr, this->bufferPos, size); this->bufferPos += size; } FUNCTION_TEST_RETURN(PACK, (Pack *)result); } /**********************************************************************************************************************************/ FN_EXTERN String * pckReadStr(PackRead *const this, PckReadStrParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(STRING, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(STRING, strDup(param.defaultValue)); String *result; // If string size > 0 if (pckReadTag(this, ¶m.id, pckTypeMapStr, false)) { // Read the string size const size_t sizeExpected = (size_t)this->tagNextSize; // Read the string out in chunks result = strNew(); while (strSize(result) != sizeExpected) { const size_t sizeRead = pckReadBuffer(this, sizeExpected - strSize(result)); strCatZN(result, (const char *)this->bufferPtr + this->bufferPos, sizeRead); this->bufferPos += sizeRead; } } // Else return an empty string else result = strNew(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN StringId pckReadStrId(PackRead *const this, PckReadStrIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(UINT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(STRING_ID, param.defaultValue); FUNCTION_TEST_RETURN(STRING_ID, pckReadTag(this, ¶m.id, pckTypeMapStrId, false)); } /**********************************************************************************************************************************/ FN_EXTERN StringList * pckReadStrLst(PackRead *const this, PckReadStrLstParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(STRING_LIST, NULL); pckReadArrayBeginP(this, .id = param.id); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { while (!pckReadNullP(this)) strLstAdd(result, pckReadStrP(this)); } MEM_CONTEXT_TEMP_END(); pckReadArrayEndP(this); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN time_t pckReadTime(PackRead *const this, PckReadTimeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(TIME, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(TIME, param.defaultValue); FUNCTION_TEST_RETURN(TIME, (time_t)cvtInt64FromZigZag(pckReadTag(this, ¶m.id, pckTypeMapTime, false))); } /**********************************************************************************************************************************/ FN_EXTERN uint32_t pckReadU32(PackRead *const this, PckReadU32Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(UINT32, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(UINT32, param.defaultValue); FUNCTION_TEST_RETURN(UINT32, (uint32_t)pckReadTag(this, ¶m.id, pckTypeMapU32, false)); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t pckReadU64(PackRead *const this, PckReadU64Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(UINT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (pckReadNullInternal(this, ¶m.id)) FUNCTION_TEST_RETURN(UINT64, param.defaultValue); FUNCTION_TEST_RETURN(UINT64, pckReadTag(this, ¶m.id, pckTypeMapU64, false)); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadEnd(PackRead *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); CHECK(FormatError, this->tagStack.depth == 0, "invalid pack end read in container"); // Make sure we are at the end of the pack unsigned int id = UINT_MAX - 1; pckReadTag(this, &id, pckTypeMapUnknown, true); this->tagStack.top = NULL; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void pckReadToLog(const PackRead *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{depth: %u, idLast: %u, tagNextId: %u, tagNextType: %u, tagNextValue %" PRIu64 "}", this->tagStack.depth, this->tagStack.top->idLast, this->tagNextId, this->tagNextTypeMap, this->tagNextValue); } /**********************************************************************************************************************************/ // Helper to create common data static PackWrite * pckWriteNewInternal(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(PackWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (PackWrite) { .tagStack = {.bottom = {.typeMap = pckTypeMapObj}}, }; this->tagStack.top = &this->tagStack.bottom; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(PACK_WRITE, this); } FN_EXTERN PackWrite * pckWriteNew(const PckWriteNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, param.size); FUNCTION_TEST_END(); PackWrite *this = pckWriteNewInternal(); MEM_CONTEXT_OBJ_BEGIN(this) { this->buffer = bufNew(param.size == 0 ? PACK_EXTRA_MIN : param.size); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(PACK_WRITE, this); } FN_EXTERN PackWrite * pckWriteNewIo(IoWrite *const write) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_WRITE, write); FUNCTION_TEST_END(); ASSERT(write != NULL); PackWrite *const this = pckWriteNewInternal(); MEM_CONTEXT_OBJ_BEGIN(this) { this->write = write; this->buffer = bufNew(ioBufferSize()); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /*********************************************************************************************************************************** Write to io or buffer ***********************************************************************************************************************************/ static void pckWriteBuffer(PackWrite *const this, const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(this != NULL); // If writing directly to a buffer if (this->write == NULL) { // Add space in the buffer to write and add extra space so future writes won't always need to resize the buffer if (bufRemains(this->buffer) < bufUsed(buffer)) bufResize(this->buffer, (bufSizeAlloc(this->buffer) + bufUsed(buffer)) + PACK_EXTRA_MIN); // Write to the buffer bufCat(this->buffer, buffer); } // Else writing to io else { // If there's enough space to write to the internal buffer then do that if (bufRemains(this->buffer) >= bufUsed(buffer)) bufCat(this->buffer, buffer); else { // Flush the internal buffer if it has data if (!bufEmpty(this->buffer)) { ioWrite(this->write, this->buffer); bufUsedZero(this->buffer); } // If there's enough space to write to the internal buffer then do that if (bufRemains(this->buffer) >= bufUsed(buffer)) { bufCat(this->buffer, buffer); } // Else write directly to io else ioWrite(this->write, buffer); } } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Pack an unsigned 64-bit integer to base-128 varint encoding ***********************************************************************************************************************************/ static void pckWriteU64Internal(PackWrite *const this, const uint64_t value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_END(); ASSERT(this != NULL); uint8_t buffer[CVT_VARINT128_BUFFER_SIZE]; size_t size = 0; cvtUInt64ToVarInt128(value, buffer, &size, sizeof(buffer)); // Write encoded bytes to the buffer pckWriteBuffer(this, BUF(buffer, size)); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Write field tag ***********************************************************************************************************************************/ static void pckWriteTag(PackWrite *const this, const PackTypeMap typeMap, unsigned int id, uint64_t value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(ENUM, typeMap); FUNCTION_TEST_PARAM(UINT, id); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_END(); ASSERT(this != NULL); // If id is not specified then add one to previous tag (and include all NULLs) if (id == 0) { id = this->tagStack.top->idLast + this->tagStack.top->nullTotal + 1; } // Else the id must be greater than the last one else CHECK(AssertError, id > this->tagStack.top->idLast, "field id must be greater than last id"); // Clear NULLs now that field id has been calculated this->tagStack.top->nullTotal = 0; // Calculate field ID delta unsigned int tagId = id - this->tagStack.top->idLast - 1; // Write field type map (e.g. int64, string) uint64_t tag = typeMap >= 0xF ? 0xF0 : typeMap << 4; // If the value can contain multiple bits (e.g. integer) if (packTypeMapData[typeMap].valueMultiBit) { // If the value is stored in the tag (value == 1 bit) if (value < 2) { // Write low order bit of the value tag |= (value & 0x1) << 2; value >>= 1; // Write low order bit of the field ID delta tag |= tagId & 0x1; tagId >>= 1; // Set bit to indicate that high order bits of the field ID delta are be written after the tag if (tagId > 0) tag |= 0x2; } // Else the value is stored following the tag (value > 1 bit) else { // Set bit to indicate that the value is written after the tag tag |= 0x8; // Write low order bits of the field ID delta tag |= tagId & 0x3; tagId >>= 2; // Set bit to indicate that high order bits of the field ID delta are be written after the tag if (tagId > 0) tag |= 0x4; } } // Else the value is a single bit (e.g. boolean) else if (packTypeMapData[typeMap].valueSingleBit) { // Write value tag |= (value & 0x1) << 3; value >>= 1; // Write low order bits of the field ID delta tag |= tagId & 0x3; tagId >>= 2; // Set bit to indicate that high order bits of the field ID delta are be written after the tag if (tagId > 0) tag |= 0x4; } else { // No value expected ASSERT(value == 0); // Write low order bits of the field ID delta tag |= tagId & 0x7; tagId >>= 3; // Set bit to indicate that high order bits of the field ID delta must be written after the tag if (tagId > 0) tag |= 0x8; } // Write tag uint8_t tagByte = (uint8_t)tag; pckWriteBuffer(this, BUF(&tagByte, 1)); // Write remaining type map if (typeMap >= 0xF) pckWriteU64Internal(this, typeMap - 0xF); // Write low order bits of the field ID delta if (tagId > 0) pckWriteU64Internal(this, tagId); // Write low order bits of the value if (value > 0) pckWriteU64Internal(this, value); // Set last field id this->tagStack.top->idLast = id; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Write a default as NULL (missing) ***********************************************************************************************************************************/ static inline bool pckWriteDefaultNull(PackWrite *const this, const bool defaultWrite, const bool defaultEqual) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BOOL, defaultWrite); FUNCTION_TEST_PARAM(BOOL, defaultEqual); FUNCTION_TEST_END(); ASSERT(this != NULL); // Write a NULL if not forcing the default to be written and the value passed equals the default if (!defaultWrite && defaultEqual) { this->tagStack.top->nullTotal++; FUNCTION_TEST_RETURN(BOOL, true); } // Let the caller know that it should write the value FUNCTION_TEST_RETURN(BOOL, false); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteNull(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); this->tagStack.top->nullTotal++; FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteArrayBegin(PackWrite *const this, const PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); // Write the array tag pckWriteTag(this, pckTypeMapArray, param.id, 0); // Add array to the tag stack so IDs can be tracked separately from the parent container pckTagStackPush(objMemContext(this), &this->tagStack, pckTypeMapArray); FUNCTION_TEST_RETURN(PACK_WRITE, this); } FN_EXTERN PackWrite * pckWriteArrayEnd(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Check tag stack pop pckTagStackPopCheck(&this->tagStack, pckTypeMapArray); // Write end of array tag pckWriteU64Internal(this, 0); // Pop array off the stack to revert to ID tracking for the prior container pckTagStackPop(&this->tagStack); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteBin(PackWrite *const this, const Buffer *const value, const PckWriteBinParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BUFFER, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, false, value == NULL)) { ASSERT(value != NULL); // Write buffer size if > 0 pckWriteTag(this, pckTypeMapBin, param.id, !bufEmpty(value)); // Write buffer data if size > 0 if (!bufEmpty(value)) { pckWriteU64Internal(this, bufUsed(value)); pckWriteBuffer(this, value); } } FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteBool(PackWrite *const this, const bool value, const PckWriteBoolParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BOOL, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(BOOL, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapBool, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteI32(PackWrite *const this, const int32_t value, const PckWriteI32Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(INT, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(INT, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapI32, param.id, cvtInt32ToZigZag(value)); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteI64(PackWrite *const this, const int64_t value, const PckWriteI64Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(INT64, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(INT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapI64, param.id, cvtInt64ToZigZag(value)); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteMode(PackWrite *const this, const mode_t value, const PckWriteModeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT32, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(MODE, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapMode, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteObjBegin(PackWrite *const this, const PackIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); // Write the object tag pckWriteTag(this, pckTypeMapObj, param.id, 0); // Add object to the tag stack so IDs can be tracked separately from the parent container pckTagStackPush(objMemContext(this), &this->tagStack, pckTypeMapObj); FUNCTION_TEST_RETURN(PACK_WRITE, this); } FN_EXTERN PackWrite * pckWriteObjEnd(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Check tag stack pop pckTagStackPopCheck(&this->tagStack, pckTypeMapObj); // Write end of object tag pckWriteU64Internal(this, 0); // Pop object off the stack to revert to ID tracking for the prior container pckTagStackPop(&this->tagStack); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWritePack(PackWrite *const this, const Pack *const value, const PckWritePackParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(BUFFER, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, false, value == NULL)) { ASSERT(value != NULL); pckWriteTag(this, pckTypeMapPack, param.id, 0); pckWriteU64Internal(this, bufUsed((const Buffer *)value)); pckWriteBuffer(this, (const Buffer *)value); } FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteStr(PackWrite *this, const String *const value, const PckWriteStrParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(STRING, param.defaultValue); FUNCTION_TEST_END(); if (!pckWriteDefaultNull(this, param.defaultWrite, strEq(value, param.defaultValue))) { ASSERT(value != NULL); // Write string size if > 0 pckWriteTag(this, pckTypeMapStr, param.id, strSize(value) > 0); // Write string data if size > 0 if (strSize(value) > 0) { pckWriteU64Internal(this, strSize(value)); pckWriteBuffer(this, BUF(strZ(value), strSize(value))); } } FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteStrId(PackWrite *const this, const uint64_t value, const PckWriteStrIdParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(UINT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapStrId, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteStrLst(PackWrite *const this, const StringList *const value, const PckWriteStrLstParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(STRING_LIST, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, false, value == NULL)) { ASSERT(value != NULL); pckWriteArrayBeginP(this, .id = param.id); for (unsigned int valueIdx = 0; valueIdx < strLstSize(value); valueIdx++) pckWriteStrP(this, strLstGet(value, valueIdx)); pckWriteArrayEndP(this); } FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteTime(PackWrite *const this, const time_t value, const PckWriteTimeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(TIME, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(TIME, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapTime, param.id, cvtInt64ToZigZag(value)); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteU32(PackWrite *const this, const uint32_t value, const PckWriteU32Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT32, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(UINT32, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapU32, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteU64(PackWrite *const this, const uint64_t value, const PckWriteU64Param param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_PARAM(UINT64, value); FUNCTION_TEST_PARAM(UINT, param.id); FUNCTION_TEST_PARAM(BOOL, param.defaultWrite); FUNCTION_TEST_PARAM(UINT64, param.defaultValue); FUNCTION_TEST_END(); ASSERT(this != NULL); if (!pckWriteDefaultNull(this, param.defaultWrite, value == param.defaultValue)) pckWriteTag(this, pckTypeMapU64, param.id, value); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * pckWriteEnd(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->tagStack.depth == 0); pckWriteU64Internal(this, 0); this->tagStack.top = NULL; // If writing to io flush the internal buffer if (this->write != NULL) { if (!bufEmpty(this->buffer)) ioWrite(this->write, this->buffer); } // Else resize the external buffer to trim off extra space added during processing else bufResize(this->buffer, bufUsed(this->buffer)); FUNCTION_TEST_RETURN(PACK_WRITE, this); } /**********************************************************************************************************************************/ FN_EXTERN Pack * pckWriteResult(PackWrite *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_WRITE, this); FUNCTION_TEST_END(); Pack *result = NULL; if (this != NULL) { ASSERT(this->tagStack.top == NULL); result = (Pack *)OBJ_NAME(this->buffer, Pack::Buffer); } FUNCTION_TEST_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN void pckWriteToLog(const PackWrite *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{depth: %u, idLast: %u}", this->tagStack.depth, this->tagStack.top == NULL ? 0 : this->tagStack.top->idLast); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/pack.h0000664000175100017510000006705315240547630017555 0ustar00runnerrunner/*********************************************************************************************************************************** Pack Type The pack type encodes binary data compactly while still maintaining structure and strict typing. The idea is based on Thrift, ProtocolBuffers, and Avro, compared here: https://medium.com/better-programming/use-binary-encoding-instead-of-json-dec745ec09b6. The pack type has been further optimized to balance between purely in-memory structures and those intended to be passed via a protocol or saved in a file. Integers are stored with base-128 varint encoding which is equivalent to network byte order, i.e., the endianness of the sending and receiving host don't matter. The overall idea is similar to JSON but IDs are used instead of names, typing is more granular, and the representation is far more compact. A pack can readily be converted to JSON but the reverse is not as precise due to loose typing in JSON. A pack is a stream format, i.e. it is intended to be read in order from beginning to end. Fields in a pack are identified by IDs. A field ID is stored as a delta from the previous ID, which is very efficient, but means that reading from the middle is generally not practical. The size of the gap between field IDs is important -- a gap of 1 never incurs extra cost, but depending on the field type larger gaps may require additional bytes to store the field ID delta. The standard default is the C default for that type (e.g. bool = false, int = 0) but can be changed with the .defaultValue parameter. For example, pckWriteBoolP(write, false, .defaultWrite = true) will write a 0 (i.e. false) with a field ID into the pack, but pckWriteBoolP(write, false) will not write to the pack, it will simply skip the ID. Note that pckWriteStrP(packWrite, NULL, .defaultWrite = true) is not valid since there is no way to explicitly write a NULL. NULLs are not stored in a pack and are therefore not typed. A NULL is essentially just a gap in the field IDs. Fields that are frequently NULL are best stored at the end of an object. When using read functions the default will always be returned when the field is NULL (i.e. missing). There are times when NULL must be explicitly passed, for example: pckWriteStrP(pack, strEmpty(checksumSha1) ? NULL : checksumSha1); In this case, NULL is declared since checksumSha1 might be empty and, following the rules for NULLs the field ID is skipped when strEmpty(checksumSha1). Upon reading, we can declare an empty string when a NULL (field ID gap) is encountered, e.g. pckReadStrP(pack, .defaultValue = EMPTY_STR)). A pack is an object by default. Objects can store fields, objects, or arrays. Objects and arrays will be referred to collectively as containers. Fields contain data to be stored, e.g. integers, strings, etc. Here is a simple example of a pack: PackWrite *write = pckWriteNewP(); pckWriteU64P(write, 77); pckWriteBoolP(write, false, .defaultWrite = true); pckWriteI32P(write, -1, .defaultValue = -1); pckWriteStringP(write, STRDEF("sample")); pckWriteEndP(); A string representation of this pack is `1:uint64:77,2:bool:false,4:str:sample`. The boolean was stored even though it was the default because a write was explicitly requested. The int32 field was not stored because the value matched the explicitly set default. Note that there is a gap in the ID stream, which represents the NULL/default value. This pack can be read with: PackRead *read = pckReadNew(pack); pckReadU64P(read); pckReadBoolP(read); pckReadI32P(read, .defaultValue = -1); pckReadStringP(read); pckReadEndP(); Note that defaults are not stored in the pack so any defaults that were applied when writing (by setting .defaultValue) must be applied again when reading by setting .defaultValue if the default value is not a standard C default. If we don't care about the NULL/default, another way to read is: PackRead *read = pckReadNew(pack); pckReadU64P(read); pckReadBoolP(read); pckReadStringP(read, .id = 4); pckReadEndP(); By default each read/write advances the field ID by one. If an ID is specified it must be unique and increasing, because it will advance the field ID beyond the value specified. An error will occur if an ID is attempted to be read/written but the field ID has advanced beyond it. An array can be read with: pckReadArrayBeginP(read); while (pckReadNext(read)) { // Read array element } pckReadArrayEndP(read); Note that any container (i.e. array or object) resets the field ID to one so there is no need for the caller to maintain a cumulative field ID. At the end of a container the numbering will continue from wherever the outer container left off. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_PACK_H #define COMMON_TYPE_PACK_H #include #include /*********************************************************************************************************************************** Minimum number of extra bytes to allocate for packs that are growing or are likely to grow ***********************************************************************************************************************************/ #ifndef PACK_EXTRA_MIN #define PACK_EXTRA_MIN 128 #endif /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ typedef struct Pack Pack; typedef struct PackRead PackRead; typedef struct PackWrite PackWrite; #include "common/io/read.h" #include "common/io/write.h" #include "common/type/object.h" #include "common/type/stringId.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Pack data type ***********************************************************************************************************************************/ typedef enum { pckTypeArray = STRID5("array", 0x190ca410), pckTypeBin = STRID5("bin", 0x39220), pckTypeBool = STRID5("bool", 0x63de20), pckTypeI32 = STRID6("i32", 0x1e7c91), pckTypeI64 = STRID6("i64", 0x208891), pckTypeObj = STRID5("obj", 0x284f0), pckTypeMode = STRID5("mode", 0x291ed0), pckTypePack = STRID5("pack", 0x58c300), pckTypeStr = STRID5("str", 0x4a930), pckTypeStrId = STRID5("strid", 0x44ca930), pckTypeTime = STRID5("time", 0x2b5340), pckTypeU32 = STRID6("u32", 0x1e7d51), pckTypeU64 = STRID6("u64", 0x208951), } PackType; /*********************************************************************************************************************************** Pack Functions ***********************************************************************************************************************************/ // Duplicate pack FN_INLINE_ALWAYS Pack * pckDup(const Pack *const this) { return (Pack *)OBJ_NAME(bufDup((const Buffer *)this), Pack::Buffer); } // Cast Buffer to Pack FN_INLINE_ALWAYS const Pack * pckFromBuf(const Buffer *const buffer) { return (const Pack *)buffer; } // Move to a new parent mem context FN_INLINE_ALWAYS Pack * pckMove(Pack *const this, MemContext *const parentNew) { return (Pack *)bufMove((Buffer *)this, parentNew); } // Cast Pack to Buffer FN_INLINE_ALWAYS const Buffer * pckToBuf(const Pack *const pack) { return (const Buffer *)pack; } /*********************************************************************************************************************************** Read Constructors ***********************************************************************************************************************************/ // Note that the pack is not moved into the PackRead mem context and must be moved explicitly if the PackRead object is moved. FN_EXTERN PackRead *pckReadNew(const Pack *pack); FN_EXTERN PackRead *pckReadNewC(const uint8_t *const buffer, size_t size); FN_EXTERN PackRead *pckReadNewIo(IoRead *read); /*********************************************************************************************************************************** Read Functions ***********************************************************************************************************************************/ typedef struct PackIdParam { VAR_PARAM_HEADER; unsigned int id; } PackIdParam; // Read next field. This is useful when the type of the next field is unknown, i.e. a completely dynamic data structure, or for // debugging. If you just need to know if the field exists or not, then use pckReadNullP(). FN_EXTERN bool pckReadNext(PackRead *this); // Current field buffer for fields that have a size, e.g. bin. Can only be used when the pack was created with pckReadNew(). Note // that this pointer is tied to the buffer the pack was created with so be careful not to free it too soon. FN_EXTERN const uint8_t *pckReadBufPtr(PackRead *this); // Consume the next field regardless of type. FN_EXTERN void pckReadConsume(PackRead *this); // Current field id. Set after a call to pckReadNext(). FN_EXTERN unsigned int pckReadId(PackRead *this); // Current field size for fields that have a size, e.g. bin FN_EXTERN size_t pckReadSize(PackRead *this); // Current field type. Set after a call to pckReadNext(). FN_EXTERN PackType pckReadType(PackRead *this); // Is the field NULL? If the field is NULL the id will be advanced so the field does not need to be read explicitly. If the field is // not NULL then the id is not advanced since a subsequent read is expected. #define pckReadNullP(this, ...) \ pckReadNull(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool pckReadNull(PackRead *this, PackIdParam param); // Read array begin/end #define pckReadArrayBeginP(this, ...) \ pckReadArrayBegin(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void pckReadArrayBegin(PackRead *this, PackIdParam param); #define pckReadArrayEndP(this) \ pckReadArrayEnd(this) FN_EXTERN void pckReadArrayEnd(PackRead *this); // Read binary typedef struct PckReadBinParam { VAR_PARAM_HEADER; unsigned int id; } PckReadBinParam; #define pckReadBinP(this, ...) \ pckReadBin(this, (PckReadBinParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Buffer *pckReadBin(PackRead *this, PckReadBinParam param); // Read boolean typedef struct PckReadBoolParam { VAR_PARAM_HEADER; unsigned int id; uint32_t defaultValue; } PckReadBoolParam; #define pckReadBoolP(this, ...) \ pckReadBool(this, (PckReadBoolParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool pckReadBool(PackRead *this, PckReadBoolParam param); // Read 32-bit signed integer typedef struct PckReadI32Param { VAR_PARAM_HEADER; unsigned int id; int32_t defaultValue; } PckReadI32Param; #define pckReadI32P(this, ...) \ pckReadI32(this, (PckReadI32Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN int32_t pckReadI32(PackRead *this, PckReadI32Param param); // Read 64-bit signed integer typedef struct PckReadI64Param { VAR_PARAM_HEADER; unsigned int id; int64_t defaultValue; } PckReadI64Param; #define pckReadI64P(this, ...) \ pckReadI64(this, (PckReadI64Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN int64_t pckReadI64(PackRead *this, PckReadI64Param param); // Read mode typedef struct PckReadModeParam { VAR_PARAM_HEADER; unsigned int id; mode_t defaultValue; } PckReadModeParam; #define pckReadModeP(this, ...) \ pckReadMode(this, (PckReadModeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN mode_t pckReadMode(PackRead *this, PckReadModeParam param); // Move to a new parent mem context FN_INLINE_ALWAYS PackRead * pckReadMove(PackRead *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Read object begin/end #define pckReadObjBeginP(this, ...) \ pckReadObjBegin(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void pckReadObjBegin(PackRead *this, PackIdParam param); #define pckReadObjEndP(this) \ pckReadObjEnd(this) FN_EXTERN void pckReadObjEnd(PackRead *this); // Read pack typedef struct PckReadPackParam { VAR_PARAM_HEADER; unsigned int id; } PckReadPackParam; #define pckReadPackReadP(this, ...) \ pckReadPackRead(this, (PckReadPackParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *pckReadPackRead(PackRead *this, PckReadPackParam param); // Read pack using the same buffer passed to pckReadNew(). Note that this pointer is tied to the buffer the pack was created with so // be careful not to free it too soon. #define pckReadPackReadConstP(this, ...) \ pckReadPackReadConst(this, (PckReadPackParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *pckReadPackReadConst(PackRead *this, PckReadPackParam param); // Read pack buffer #define pckReadPackP(this, ...) \ pckReadPack(this, (PckReadPackParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Pack *pckReadPack(PackRead *this, PckReadPackParam param); // Read string typedef struct PckReadStrParam { VAR_PARAM_HEADER; unsigned int id; const String *defaultValue; } PckReadStrParam; #define pckReadStrP(this, ...) \ pckReadStr(this, (PckReadStrParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN String *pckReadStr(PackRead *this, PckReadStrParam param); // Read string id typedef struct PckReadStrIdParam { VAR_PARAM_HEADER; unsigned int id; StringId defaultValue; } PckReadStrIdParam; #define pckReadStrIdP(this, ...) \ pckReadStrId(this, (PckReadStrIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN uint64_t pckReadStrId(PackRead *this, PckReadStrIdParam param); // Read string list typedef struct PckReadStrLstParam { VAR_PARAM_HEADER; unsigned int id; } PckReadStrLstParam; #define pckReadStrLstP(this, ...) \ pckReadStrLst(this, (PckReadStrLstParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StringList *pckReadStrLst(PackRead *const this, PckReadStrLstParam param); // Read time typedef struct PckReadTimeParam { VAR_PARAM_HEADER; unsigned int id; time_t defaultValue; } PckReadTimeParam; #define pckReadTimeP(this, ...) \ pckReadTime(this, (PckReadTimeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN time_t pckReadTime(PackRead *this, PckReadTimeParam param); // Read 32-bit unsigned integer typedef struct PckReadU32Param { VAR_PARAM_HEADER; unsigned int id; uint32_t defaultValue; } PckReadU32Param; #define pckReadU32P(this, ...) \ pckReadU32(this, (PckReadU32Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN uint32_t pckReadU32(PackRead *this, PckReadU32Param param); // Read 64-bit unsigned integer typedef struct PckReadU64Param { VAR_PARAM_HEADER; unsigned int id; uint64_t defaultValue; } PckReadU64Param; #define pckReadU64P(this, ...) \ pckReadU64(this, (PckReadU64Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN uint64_t pckReadU64(PackRead *this, PckReadU64Param param); // Read end #define pckReadEndP(this) \ pckReadEnd(this) FN_EXTERN void pckReadEnd(PackRead *this); /*********************************************************************************************************************************** Read Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void pckReadFree(PackRead *const this) { objFree(this); } /*********************************************************************************************************************************** Write Constructors ***********************************************************************************************************************************/ typedef struct PckWriteNewParam { VAR_PARAM_HEADER; size_t size; } PckWriteNewParam; #define pckWriteNewP(...) \ pckWriteNew((PckWriteNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteNew(PckWriteNewParam param); FN_EXTERN PackWrite *pckWriteNewIo(IoWrite *write); /*********************************************************************************************************************************** Write Functions ***********************************************************************************************************************************/ // Write array begin/end #define pckWriteArrayBeginP(this, ...) \ pckWriteArrayBegin(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteArrayBegin(PackWrite *this, PackIdParam param); #define pckWriteArrayEndP(this) \ pckWriteArrayEnd(this) FN_EXTERN PackWrite *pckWriteArrayEnd(PackWrite *this); // Write binary typedef struct PckWriteBinParam { VAR_PARAM_HEADER; unsigned int id; } PckWriteBinParam; #define pckWriteBinP(this, value, ...) \ pckWriteBin(this, value, (PckWriteBinParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteBin(PackWrite *this, const Buffer *value, PckWriteBinParam param); // Write boolean typedef struct PckWriteBoolParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; uint32_t defaultValue; } PckWriteBoolParam; #define pckWriteBoolP(this, value, ...) \ pckWriteBool(this, value, (PckWriteBoolParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteBool(PackWrite *this, bool value, PckWriteBoolParam param); // Write 32-bit signed integer typedef struct PckWriteI32Param { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; int32_t defaultValue; } PckWriteI32Param; #define pckWriteI32P(this, value, ...) \ pckWriteI32(this, value, (PckWriteI32Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteI32(PackWrite *this, int32_t value, PckWriteI32Param param); // Write 64-bit signed integer typedef struct PckWriteI64Param { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; int64_t defaultValue; } PckWriteI64Param; #define pckWriteI64P(this, value, ...) \ pckWriteI64(this, value, (PckWriteI64Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteI64(PackWrite *this, int64_t value, PckWriteI64Param param); // Write mode typedef struct PckWriteModeParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; mode_t defaultValue; } PckWriteModeParam; #define pckWriteModeP(this, value, ...) \ pckWriteMode(this, value, (PckWriteModeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteMode(PackWrite *this, mode_t value, PckWriteModeParam param); // Move to a new parent mem context FN_INLINE_ALWAYS PackWrite * pckWriteMove(PackWrite *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Write null #define pckWriteNullP(this) \ pckWriteNull(this) FN_EXTERN PackWrite *pckWriteNull(PackWrite *this); // Write object begin/end #define pckWriteObjBeginP(this, ...) \ pckWriteObjBegin(this, (PackIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteObjBegin(PackWrite *this, PackIdParam param); #define pckWriteObjEndP(this) \ pckWriteObjEnd(this) FN_EXTERN PackWrite *pckWriteObjEnd(PackWrite *this); // Write pack typedef struct PckWritePackParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; } PckWritePackParam; #define pckWritePackP(this, value, ...) \ pckWritePack(this, value, (PckWritePackParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWritePack(PackWrite *this, const Pack *value, PckWritePackParam param); // Write string typedef struct PckWriteStrParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; const String *defaultValue; } PckWriteStrParam; #define pckWriteStrP(this, value, ...) \ pckWriteStr(this, value, (PckWriteStrParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteStr(PackWrite *this, const String *value, PckWriteStrParam param); // Write string id typedef struct PckWriteStrIdParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; StringId defaultValue; } PckWriteStrIdParam; #define pckWriteStrIdP(this, value, ...) \ pckWriteStrId(this, value, (PckWriteStrIdParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteStrId(PackWrite *this, uint64_t value, PckWriteStrIdParam param); // Write string list typedef struct PckWriteStrLstParam { VAR_PARAM_HEADER; unsigned int id; } PckWriteStrLstParam; #define pckWriteStrLstP(this, value, ...) \ pckWriteStrLst(this, value, (PckWriteStrLstParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteStrLst(PackWrite *const this, const StringList *const value, const PckWriteStrLstParam param); // Write time typedef struct PckWriteTimeParam { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; time_t defaultValue; } PckWriteTimeParam; #define pckWriteTimeP(this, value, ...) \ pckWriteTime(this, value, (PckWriteTimeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteTime(PackWrite *this, time_t value, PckWriteTimeParam param); // Write 32-bit unsigned integer typedef struct PckWriteU32Param { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; uint32_t defaultValue; } PckWriteU32Param; #define pckWriteU32P(this, value, ...) \ pckWriteU32(this, value, (PckWriteU32Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteU32(PackWrite *this, uint32_t value, PckWriteU32Param param); // Write 64-bit unsigned integer typedef struct PckWriteU64Param { VAR_PARAM_HEADER; bool defaultWrite; unsigned int id; uint64_t defaultValue; } PckWriteU64Param; #define pckWriteU64P(this, value, ...) \ pckWriteU64(this, value, (PckWriteU64Param){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackWrite *pckWriteU64(PackWrite *this, uint64_t value, PckWriteU64Param param); // Write end #define pckWriteEndP(this) \ pckWriteEnd(this) FN_EXTERN PackWrite *pckWriteEnd(PackWrite *this); /*********************************************************************************************************************************** Write Getters/Setters ***********************************************************************************************************************************/ // Get Pack the PackWrite was writing to (returns NULL if pckWriteNew() was not used to construct the object). This function is only // valid after pckWriteEndP() has been called. FN_EXTERN Pack *pckWriteResult(PackWrite *this); /*********************************************************************************************************************************** Write Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void pckWriteFree(PackWrite *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_PACK_TYPE \ Pack * #define FUNCTION_LOG_PACK_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Pack", buffer, bufferSize) FN_EXTERN void pckReadToLog(const PackRead *this, StringStatic *debugLog); #define FUNCTION_LOG_PACK_READ_TYPE \ PackRead * #define FUNCTION_LOG_PACK_READ_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, pckReadToLog, buffer, bufferSize) FN_EXTERN void pckWriteToLog(const PackWrite *this, StringStatic *debugLog); #define FUNCTION_LOG_PACK_WRITE_TYPE \ PackWrite * #define FUNCTION_LOG_PACK_WRITE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, pckWriteToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/param.h0000664000175100017510000000216415240547630017727 0ustar00runnerrunner/*********************************************************************************************************************************** Variable Parameter Helper Macros ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_PARAM_H #define COMMON_TYPE_PARAM_H /*********************************************************************************************************************************** Macros to help with implementing functions with variable parameters using structs ***********************************************************************************************************************************/ // This macro goes at the top of the parameter struct #define VAR_PARAM_HEADER \ bool dummy // This macro goes in the struct parameter list right before __VA_ARGS__ #define VAR_PARAM_INIT \ .dummy = false #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/string.c0000664000175100017510000010727415240547630020140 0ustar00runnerrunner/*********************************************************************************************************************************** String Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include "common/debug.h" #include "common/macro.h" #include "common/memContext.h" #include "common/type/string.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Constant strings that are generally useful ***********************************************************************************************************************************/ STRING_EXTERN(CR_STR, "\r"); STRING_EXTERN(CRLF_STR, "\r\n"); STRING_EXTERN(DOT_STR, "."); STRING_EXTERN(DOTDOT_STR, ".."); STRING_EXTERN(EMPTY_STR, ""); STRING_EXTERN(FALSE_STR, FALSE_Z); STRING_EXTERN(FSLASH_STR, "/"); STRING_EXTERN(LF_STR, "\n"); STRING_EXTERN(N_STR, "n"); STRING_EXTERN(NULL_STR, NULL_Z); STRING_EXTERN(TRUE_STR, TRUE_Z); STRING_EXTERN(Y_STR, "y"); STRING_EXTERN(ZERO_STR, "0"); /*********************************************************************************************************************************** Buffer macros ***********************************************************************************************************************************/ // Fixed size buffer allocated at the end of the object allocation #define STR_FIXED_BUFFER (char *)(this + 1) // Is the string using the fixed size buffer? #define STR_IS_FIXED_BUFFER() (this->pub.buffer == STR_FIXED_BUFFER) // Empty buffer #define STR_EMPTY_BUFFER (EMPTY_STR->pub.buffer) // Is the string using the empty buffer? #define STR_IS_EMPTY_BUFFER() (this->pub.buffer == STR_EMPTY_BUFFER) /*********************************************************************************************************************************** Maximum size of a string ***********************************************************************************************************************************/ #define STRING_SIZE_MAX 1073741824 #define CHECK_SIZE(size) \ do \ { \ if ((size) > STRING_SIZE_MAX) \ THROW(AssertError, "string size must be <= " STRINGIFY(STRING_SIZE_MAX) " bytes"); \ } \ while (0) /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct String { StringPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN String * strNew(void) { FUNCTION_TEST_VOID(); OBJ_NEW_BEGIN(String, .allocQty = 1) { *this = (String) { .pub = { // Set empty so nothing is allocated until needed .buffer = STR_EMPTY_BUFFER, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STRING, this); } /*********************************************************************************************************************************** Create fixed size String ***********************************************************************************************************************************/ static String * strNewFixed(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); CHECK_SIZE(size); // If the string is larger than the extra allowed with a mem context then allocate the buffer separately const size_t allocExtra = sizeof(String) + size + 1; if (allocExtra > MEM_CONTEXT_ALLOC_EXTRA_MAX) { OBJ_NEW_BEGIN(String, .allocQty = 1) { *this = (String) { .pub = { .size = (unsigned int)size, .buffer = memNew(size + 1), }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STRING, this); } OBJ_NEW_EXTRA_BEGIN(String, (uint16_t)(allocExtra)) { *this = (String) { .pub = { .size = (unsigned int)size, .buffer = STR_FIXED_BUFFER, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewZ(const char *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, string); FUNCTION_TEST_END(); ASSERT(string != NULL); // Create object String *const this = strNewFixed(strlen(string)); // Assign string memcpy(this->pub.buffer, string, strSize(this)); this->pub.buffer[strSize(this)] = '\0'; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewDiv(const uint64_t dividend, const uint64_t divisor, StrNewDivParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_PARAM(UINT, param.precision); FUNCTION_TEST_PARAM(BOOL, param.trim); FUNCTION_TEST_END(); char working[CVT_DIV_BUFFER_SIZE]; size_t resultSize = cvtDivToZ(dividend, divisor, param.precision, param.trim, working, sizeof(working)); FUNCTION_TEST_RETURN(STRING, strNewZN(working, resultSize)); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewPct(const uint64_t dividend, const uint64_t divisor) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, dividend); FUNCTION_TEST_PARAM(UINT64, divisor); FUNCTION_TEST_END(); char working[CVT_PCT_BUFFER_SIZE]; size_t resultSize = cvtPctToZ(dividend, divisor, working, sizeof(working)); FUNCTION_TEST_RETURN(STRING, strNewZN(working, resultSize)); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewStrId(const StringId strId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_END(); char buffer[STRID_MAX + 1]; const size_t size = strIdToZN(strId, buffer); buffer[size] = '\0'; FUNCTION_TEST_RETURN(STRING, strNewZN(buffer, size)); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewTime(const char *const format, const time_t timestamp, const StrNewTimeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_PARAM(TIME, timestamp); FUNCTION_TEST_PARAM(BOOL, param.utc); FUNCTION_TEST_END(); char buffer[64]; // We can ignore this warning here since the format parameter of strNewTimeP() is checked #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wformat-nonliteral" cvtTimeToZP(format, timestamp, buffer, sizeof(buffer), .utc = param.utc); #pragma GCC diagnostic pop FUNCTION_TEST_RETURN(STRING, strNewZ(buffer)); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewBuf(const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Create object String *const this = strNewFixed(bufUsed(buffer)); // Assign string if (strSize(this) != 0) memcpy(this->pub.buffer, bufPtrConst(buffer), strSize(this)); this->pub.buffer[strSize(this)] = 0; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewEncode(const EncodingType type, const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); // Create object String *const this = strNewFixed(encodeToStrSize(type, bufUsed(buffer))); // Encode buffer if (bufUsed(buffer) > 0) { encodeToStr(type, bufPtrConst(buffer), bufUsed(buffer), this->pub.buffer); } // Else zero-terminate else this->pub.buffer[0] = '\0'; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewFmt(const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(format != NULL); // Determine how long the allocated string needs to be and create object va_list argumentList; va_start(argumentList, format); String *const this = strNewFixed((size_t)vsnprintf(NULL, 0, format, argumentList)); va_end(argumentList); // Format string va_start(argumentList, format); vsnprintf(this->pub.buffer, strSize(this) + 1, format, argumentList); va_end(argumentList); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strNewZN(const char *const string, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(CHARDATA, string); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(string != NULL); // Create object String *const this = strNewFixed(size); // Assign string memcpy(this->pub.buffer, string, strSize(this)); this->pub.buffer[strSize(this)] = 0; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strBase(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING, strNewZ(strBaseZ(this))); } FN_EXTERN const char * strBaseZ(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); const char *end = this->pub.buffer + strSize(this); while (end > this->pub.buffer && *(end - 1) != '/') end--; FUNCTION_TEST_RETURN_CONST(STRINGZ, end); } /**********************************************************************************************************************************/ FN_EXTERN bool strBeginsWith(const String *const this, const String *const beginsWith) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, beginsWith); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(beginsWith != NULL); FUNCTION_TEST_RETURN(BOOL, strBeginsWithZ(this, strZ(beginsWith))); } FN_EXTERN bool strBeginsWithZ(const String *const this, const char *const beginsWith) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, beginsWith); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(beginsWith != NULL); bool result = false; const unsigned int beginsWithSize = (unsigned int)strlen(beginsWith); if (strSize(this) >= beginsWithSize) result = strncmp(strZ(this), beginsWith, beginsWithSize) == 0; FUNCTION_TEST_RETURN(BOOL, result); } /*********************************************************************************************************************************** Resize the string to allow the requested number of characters to be appended ***********************************************************************************************************************************/ static void strResize(String *const this, const size_t requested) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(SIZE, requested); FUNCTION_TEST_END(); if (requested > this->pub.extra) { // Fixed size strings may not be resized CHECK(AssertError, !STR_IS_FIXED_BUFFER(), "resize of fixed size string"); // Check size CHECK_SIZE(strSize(this) + requested); // Calculate new extra needs to satisfy request and leave extra space for new growth this->pub.extra = (unsigned int)(requested + ((strSize(this) + requested) / 2)); // Adding too little extra space usually leads to immediate resizing so enforce a minimum if (this->pub.extra < STRING_EXTRA_MIN) this->pub.extra = STRING_EXTRA_MIN; MEM_CONTEXT_OBJ_BEGIN(this) { if (STR_IS_EMPTY_BUFFER()) this->pub.buffer = memNew(strSize(this) + this->pub.extra + 1); else this->pub.buffer = memResize(this->pub.buffer, strSize(this) + this->pub.extra + 1); } MEM_CONTEXT_OBJ_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * strCat(String *const this, const String *const cat) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, cat); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(cat != NULL); FUNCTION_TEST_RETURN(STRING, strCatZN(this, strZ(cat), strSize(cat))); } FN_EXTERN String * strCatZ(String *const this, const char *const cat) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, cat); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(cat != NULL); // Determine length of string to append const size_t sizeGrow = strlen(cat); if (sizeGrow != 0) { // Ensure there is enough space to grow the string strResize(this, sizeGrow); // Append the string memcpy(this->pub.buffer + strSize(this), cat, sizeGrow + 1); this->pub.size += (unsigned int)sizeGrow; this->pub.extra -= (unsigned int)sizeGrow; } FUNCTION_TEST_RETURN(STRING, this); } FN_EXTERN String * strCatZN(String *const this, const char *const cat, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, cat); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); if (size != 0) { ASSERT(cat != NULL); // Ensure there is enough space to grow the string strResize(this, size); // Append the string memcpy(this->pub.buffer + strSize(this), cat, size); this->pub.buffer[strSize(this) + size] = '\0'; // Update size/extra this->pub.size += (unsigned int)size; this->pub.extra -= (unsigned int)size; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatBuf(String *const this, const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); FUNCTION_TEST_RETURN(STRING, strCatZN(this, (const char *)bufPtrConst(buffer), bufUsed(buffer))); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatChr(String *const this, const char cat) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(CHAR, cat); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(cat != 0); // Ensure there is enough space to grow the string strResize(this, 1); // Append the character this->pub.buffer[this->pub.size++] = cat; this->pub.buffer[this->pub.size] = 0; this->pub.extra--; FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatEncode(String *const this, const EncodingType type, const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); const size_t encodeSize = encodeToStrSize(type, bufUsed(buffer)); if (encodeSize != 0) { // Ensure there is enough space to grow the string strResize(this, encodeSize); // Append the encoded string encodeToStr(type, bufPtrConst(buffer), bufUsed(buffer), this->pub.buffer + strSize(this)); // Update size/extra this->pub.size += (unsigned int)encodeSize; this->pub.extra -= (unsigned int)encodeSize; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatTime(String *const this, const char *const format, const time_t timestamp, const StrCatTimeParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_PARAM(TIME, timestamp); FUNCTION_TEST_PARAM(BOOL, param.utc); FUNCTION_TEST_END(); char buffer[64]; // We can ignore this warning here since the format parameter of strCatTimeP() is checked #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wformat-nonliteral" cvtTimeToZP(format, timestamp, buffer, sizeof(buffer), .utc = param.utc); #pragma GCC diagnostic pop FUNCTION_TEST_RETURN(STRING, strCatZ(this, buffer)); } /**********************************************************************************************************************************/ FN_EXTERN String * strCatFmt(String *const this, const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(format != NULL); // Determine how long the allocated string needs to be va_list argumentList; va_start(argumentList, format); const size_t sizeGrow = (size_t)vsnprintf(NULL, 0, format, argumentList); va_end(argumentList); if (sizeGrow != 0) { // Ensure there is enough space to grow the string strResize(this, sizeGrow); // Append the formatted string va_start(argumentList, format); vsnprintf(this->pub.buffer + strSize(this), sizeGrow + 1, format, argumentList); va_end(argumentList); this->pub.size += (unsigned int)sizeGrow; this->pub.extra -= (unsigned int)sizeGrow; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN int strCmp(const String *const this, const String *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, compare); FUNCTION_TEST_END(); if (this != NULL && compare != NULL) FUNCTION_TEST_RETURN(INT, strcmp(strZ(this), strZ(compare))); else if (this == NULL) { if (compare == NULL) FUNCTION_TEST_RETURN(INT, 0); FUNCTION_TEST_RETURN(INT, -1); } FUNCTION_TEST_RETURN(INT, 1); } FN_EXTERN int strCmpZ(const String *const this, const char *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, compare); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(INT, strCmp(this, compare == NULL ? NULL : STR(compare))); } /**********************************************************************************************************************************/ FN_EXTERN String * strDup(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); String *result = NULL; if (this != NULL) result = strNewZ(strZ(this)); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN bool strEmpty(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, strSize(this) == 0); } /**********************************************************************************************************************************/ FN_EXTERN bool strEndsWith(const String *const this, const String *const endsWith) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, endsWith); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(endsWith != NULL); FUNCTION_TEST_RETURN(BOOL, strEndsWithZ(this, strZ(endsWith))); } FN_EXTERN bool strEndsWithZ(const String *const this, const char *const endsWith) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, endsWith); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(endsWith != NULL); bool result = false; const unsigned int endsWithSize = (unsigned int)strlen(endsWith); if (strSize(this) >= endsWithSize) result = strcmp(strZ(this) + (strSize(this) - endsWithSize), endsWith) == 0; FUNCTION_TEST_RETURN(BOOL, result); } /*********************************************************************************************************************************** There are two separate implementations because string objects can get the size very efficiently whereas the zero-terminated strings would need a call to strlen(). ***********************************************************************************************************************************/ FN_EXTERN bool strEq(const String *const this, const String *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, compare); FUNCTION_TEST_END(); bool result = false; if (this != NULL && compare != NULL) { if (strSize(this) == strSize(compare)) result = strcmp(strZ(this), strZ(compare)) == 0; } else result = this == NULL && compare == NULL; FUNCTION_TEST_RETURN(BOOL, result); } FN_EXTERN bool strEqZ(const String *const this, const char *const compare) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRINGZ, compare); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(compare != NULL); FUNCTION_TEST_RETURN(BOOL, strcmp(strZ(this), compare) == 0); } /**********************************************************************************************************************************/ FN_EXTERN String * strFirstUpper(String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (strSize(this) > 0) this->pub.buffer[0] = (char)toupper(this->pub.buffer[0]); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strFirstLower(String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); if (strSize(this) > 0) this->pub.buffer[0] = (char)tolower(this->pub.buffer[0]); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strLower(String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); for (size_t idx = 0; idx < strSize(this); idx++) this->pub.buffer[idx] = (char)tolower(this->pub.buffer[idx]); FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strPath(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); const char *end = this->pub.buffer + strSize(this); while (end > this->pub.buffer && *(end - 1) != '/') end--; FUNCTION_TEST_RETURN( STRING, strNewZN( this->pub.buffer, end - this->pub.buffer <= 1 ? (size_t)(end - this->pub.buffer) : (size_t)(end - this->pub.buffer - 1))); } /**********************************************************************************************************************************/ FN_EXTERN String * strPathAbsolute(const String *const this, const String *const base) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(STRING, base); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; // Path is already absolute so just return it if (strBeginsWith(this, FSLASH_STR)) { result = strDup(this); } // Else we'll need to construct the absolute path. You would hope we could use realpath() here but it is so broken in the Posix // spec that is seems best avoided. else { ASSERT(base != NULL); // Base must be absolute to start if (!strBeginsWith(base, FSLASH_STR)) THROW_FMT(AssertError, "base path '%s' is not absolute", strZ(base)); MEM_CONTEXT_TEMP_BEGIN() { StringList *const baseList = strLstNewSplit(base, FSLASH_STR); StringList *const pathList = strLstNewSplit(this, FSLASH_STR); while (!strLstEmpty(pathList)) { const String *const pathPart = strLstGet(pathList, 0); // If the last part is empty if (strSize(pathPart) == 0) { // Allow when this is the last part since it just means there was a trailing / if (strLstSize(pathList) == 1) { strLstRemoveIdx(pathList, 0); break; } THROW_FMT(AssertError, "'%s' is not a valid relative path", strZ(this)); } if (strEq(pathPart, DOTDOT_STR)) { const String *const basePart = strLstGet(baseList, strLstSize(baseList) - 1); if (strSize(basePart) == 0) THROW_FMT(AssertError, "relative path '%s' goes back too far in base path '%s'", strZ(this), strZ(base)); strLstRemoveIdx(baseList, strLstSize(baseList) - 1); } else if (!strEq(pathPart, DOT_STR)) strLstAdd(baseList, pathPart); strLstRemoveIdx(pathList, 0); } MEM_CONTEXT_PRIOR_BEGIN() { if (strLstSize(baseList) == 1) result = strDup(FSLASH_STR); else result = strLstJoin(baseList, "/"); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } // There should not be any stray .. or // in the final result if (strstr(strZ(result), "/..") != NULL || strstr(strZ(result), "//") != NULL) THROW_FMT(AssertError, "result path '%s' is not absolute", strZ(result)); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN const char * strZNull(const String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRINGZ, this == NULL ? NULL : strZ(this)); } /**********************************************************************************************************************************/ FN_EXTERN String * strReplaceChr(String *const this, const char find, const char replace) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(CHAR, find); FUNCTION_TEST_PARAM(CHAR, replace); FUNCTION_TEST_END(); ASSERT(this != NULL); for (size_t stringIdx = 0; stringIdx < strSize(this); stringIdx++) { if (this->pub.buffer[stringIdx] == find) this->pub.buffer[stringIdx] = replace; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strSub(const String *const this, const size_t start) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(SIZE, start); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(start <= this->pub.size); FUNCTION_TEST_RETURN(STRING, strSubN(this, start, strSize(this) - start)); } /**********************************************************************************************************************************/ FN_EXTERN String * strSubN(const String *const this, const size_t start, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(SIZE, start); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(start <= strSize(this)); ASSERT(start + size <= strSize(this)); FUNCTION_TEST_RETURN(STRING, strNewZN(strZ(this) + start, size)); } /**********************************************************************************************************************************/ FN_EXTERN String * strTrim(String *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // Nothing to trim if size is zero if (strSize(this) > 0) { // Find the beginning of the string skipping all whitespace char *begin = this->pub.buffer; while (*begin != 0 && (*begin == ' ' || *begin == '\t' || *begin == '\r' || *begin == '\n')) begin++; // Find the end of the string skipping all whitespace char *end = this->pub.buffer + (strSize(this) - 1); while (end > begin && (*end == ' ' || *end == '\t' || *end == '\r' || *end == '\n')) end--; // Is there anything to trim? const size_t newSize = (size_t)(end - begin + 1); if (begin != this->pub.buffer || newSize < strSize(this)) { // Calculate new size and extra this->pub.extra = (unsigned int)(strSize(this) - newSize); this->pub.size = (unsigned int)newSize; // Move the substr to the beginning of the buffer memmove(this->pub.buffer, begin, strSize(this)); this->pub.buffer[strSize(this)] = 0; } } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN int strChr(const String *const this, const char chr) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_PARAM(CHAR, chr); FUNCTION_TEST_END(); ASSERT(this != NULL); int result = -1; if (strSize(this) > 0) { const char *const ptr = strchr(this->pub.buffer, chr); if (ptr != NULL) result = (int)(ptr - this->pub.buffer); } FUNCTION_TEST_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN String * strTruncIdx(String *const this, const int idx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(idx >= 0 && (size_t)idx <= strSize(this)); if (strSize(this) > 0) { // Reset the size to end at the index this->pub.extra = (unsigned int)(strSize(this) - (size_t)idx); this->pub.size = (unsigned int)idx; this->pub.buffer[strSize(this)] = 0; } FUNCTION_TEST_RETURN(STRING, this); } /**********************************************************************************************************************************/ FN_EXTERN void strToLog(const String *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{\"%s\"}", strZ(this)); } /**********************************************************************************************************************************/ FN_EXTERN String * strSizeFormat(const uint64_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, size); FUNCTION_TEST_END(); String *result; if (size < 1024) result = strNewFmt("%" PRIu64 "B", size); else { char working[CVT_DIV_BUFFER_SIZE]; uint64_t divisor = 1024 * 1024 * 1024; unsigned int precision = 1; const char *suffix = "GB"; if (size < (1024 * 1024)) { divisor = 1024; suffix = "KB"; } else if (size < (1024 * 1024 * 1024)) { divisor = 1024 * 1024; suffix = "MB"; } // Skip precision when it would cause overflow if (size > UINT64_MAX / 10) precision = 0; // Format size cvtDivToZ(size, divisor, precision, true, working, sizeof(working)); result = strNewFmt("%s%s", working, suffix); } FUNCTION_TEST_RETURN(STRING, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/string.h0000664000175100017510000003675515240547630020152 0ustar00runnerrunner/*********************************************************************************************************************************** String Handler Strings are lightweight objects in that they do not have their own memory context, instead they exist in the current context in which they are instantiated. If a string is needed outside the current memory context, the memory context must be switched to the old context and then back. Below is a simplified example: String *result = NULL; <--- is created in the current memory context (referred to as "old context" below) MEM_CONTEXT_TEMP_BEGIN() <--- begins a new temporary context { String *resultStr = strNewZN("myNewStr"); <--- creates a string in the temporary memory context MEM_CONTEXT_PRIOR_BEGIN() <--- switch to the old context so the duplication of the string is in that context { result = strDup(resultStr); <--- recreates a copy of the string in the old context where "result" was created } MEM_CONTEXT_PRIOR_END(); <--- switch back to the temporary context } MEM_CONTEXT_TEMP_END(); <-- frees everything created inside this temporary memory context - i.e resultStr ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRING_H #define COMMON_TYPE_STRING_H #include #include "common/debug.h" /*********************************************************************************************************************************** Minimum number of extra bytes to allocate for strings that are growing or are likely to grow ***********************************************************************************************************************************/ #ifndef STRING_EXTRA_MIN #define STRING_EXTRA_MIN 64 #endif /*********************************************************************************************************************************** String object ***********************************************************************************************************************************/ typedef struct String String; #include "common/assert.h" #include "common/encode.h" #include "common/type/buffer.h" #include "common/type/stringId.h" #include "common/type/stringZ.h" /*********************************************************************************************************************************** Constructors Only strings created with strNew() can use strCat*() functions. If a string needs to be concatenated then start with, .e.g.: String *example = strCatZ(strNew(), "example"); This syntax signals that the string will be modified so the string is allocated separately from the object so it can be resized when needed. Most strings never need to be modified and can be stored more efficiently by allocating their memory with the object. ***********************************************************************************************************************************/ // Create a new empty string for concatenation FN_EXTERN String *strNew(void); // Create a new fixed length string from a zero-terminated string FN_EXTERN String *strNewZ(const char *const string); // Create a new fixed length string from a zero-terminated string with a specific length. The string may or may not be // zero-terminated but we'll use that nomenclature since we're not concerned about the end of the string. FN_EXTERN String *strNewZN(const char *string, size_t size); // Create a new fixed length string from a buffer. If the buffer has a NULL character this may not work as expected. All the data // will be copied but only the data before the NULL character will be used as a string. FN_EXTERN String *strNewBuf(const Buffer *buffer); // Create a new fixed length string by performing division typedef struct StrNewDivParam { VAR_PARAM_HEADER; unsigned int precision; // Digits of precision after the decimal bool trim; // Trim trailing zeros and decimal point? } StrNewDivParam; #define strNewDivP(dividend, divisor, ...) \ strNewDiv(dividend, divisor, (StrNewDivParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN String *strNewDiv(uint64_t dividend, uint64_t divisor, StrNewDivParam param); // Create a new fixed length string containing a percentage FN_EXTERN String *strNewPct(uint64_t dividend, uint64_t divisor); // Create string from string id FN_EXTERN String *strNewStrId(StringId strId); // Create a new fixed length string by converting a timestamp typedef struct StrNewTimeParam { VAR_PARAM_HEADER; bool utc; // Use UTC instead of local time? } StrNewTimeParam; #define strNewTimeP(format, timestamp, ...) \ strNewTime(format, timestamp, (StrNewTimeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN FN_STRFTIME(1) String *strNewTime(const char *format, time_t timestamp, StrNewTimeParam param); // Create a new fixed length string encoded with the specified type (e.g. encodingBase64) from a buffer FN_EXTERN String *strNewEncode(EncodingType type, const Buffer *buffer); // Create a new fixed length string from a format string with parameters (i.e. sprintf) FN_EXTERN FN_PRINTF(1, 2) String *strNewFmt(const char *format, ...); // Create a new fixed length string from a string FN_EXTERN String *strDup(const String *this); /*********************************************************************************************************************************** Getters/setters ***********************************************************************************************************************************/ typedef struct StringPub { uint64_t size : 32; // Actual size of the string uint64_t extra : 32; // Extra space allocated for expansion char *buffer; // String buffer } StringPub; // String size minus null-terminator, i.e. the same value that strlen() would return FN_INLINE_ALWAYS size_t strSize(const String *this) { return THIS_PUB(String)->size; } // Pointer to zero-terminated string. strZNull() returns NULL when the String is NULL. FN_INLINE_ALWAYS const char * strZ(const String *this) { return THIS_PUB(String)->buffer; } FN_EXTERN const char *strZNull(const String *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Return the file part of a string (i.e. everything after the last / or the entire string if there is no /). strBaseZ() does not // make a copy of the string so it may be more appropriate for large loops where saving memory is important. FN_EXTERN String *strBase(const String *this); FN_EXTERN const char *strBaseZ(const String *this); // Does the string begin with the specified string? FN_EXTERN bool strBeginsWith(const String *this, const String *beginsWith); FN_EXTERN bool strBeginsWithZ(const String *this, const char *beginsWith); // Append a string or zero-terminated string FN_EXTERN String *strCat(String *this, const String *cat); FN_EXTERN String *strCatZ(String *this, const char *cat); // Append a buffer FN_EXTERN String *strCatBuf(String *this, const Buffer *buffer); // Append a character FN_EXTERN String *strCatChr(String *this, char cat); // Append a string encoded with the specified type (e.g. encodingBase64) from a buffer FN_EXTERN String *strCatEncode(String *this, EncodingType type, const Buffer *buffer); // Append a timestamp typedef struct StrCatTimeParam { VAR_PARAM_HEADER; bool utc; // Use UTC instead of local time? } StrCatTimeParam; #define strCatTimeP(this, format, timestamp, ...) \ strCatTime(this, format, timestamp, (StrCatTimeParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN FN_STRFTIME(2) String *strCatTime(String *this, const char *format, time_t timestamp, StrCatTimeParam param); // Append a formatted string FN_EXTERN FN_PRINTF(2, 3) String *strCatFmt(String *this, const char *format, ...); // Append N characters from a zero-terminated string. Note that the string does not actually need to be zero-terminated as long as // N is <= the end of the string being concatenated. FN_EXTERN String *strCatZN(String *this, const char *cat, size_t size); FN_INLINE_ALWAYS String * strCatN(String *this, const String *const cat, const size_t size) { ASSERT_INLINE(cat != NULL); return strCatZN(this, strZ(cat), size); } // Return the index to the location of the first occurrence of a character within a string, else -1 FN_EXTERN int strChr(const String *this, char chr); // C-style string compare FN_EXTERN int strCmp(const String *this, const String *compare); FN_EXTERN int strCmpZ(const String *this, const char *compare); // Is the string empty? FN_EXTERN bool strEmpty(const String *this); // Does the string end with the specified string? FN_EXTERN bool strEndsWith(const String *this, const String *endsWith); FN_EXTERN bool strEndsWithZ(const String *this, const char *endsWith); // Are two strings equal? FN_EXTERN bool strEq(const String *this, const String *compare); FN_EXTERN bool strEqZ(const String *this, const char *compare); // Upper-case the first letter FN_EXTERN String *strFirstUpper(String *this); // Lower-case the first letter FN_EXTERN String *strFirstLower(String *this); // Lower-case entire string FN_EXTERN String *strLower(String *this); // Return the path part of a string (i.e. everything before the last / or "" if there is no /) FN_EXTERN String *strPath(const String *this); // Combine with a base path to get an absolute path FN_EXTERN String *strPathAbsolute(const String *this, const String *base); // Replace a character with another character FN_EXTERN String *strReplaceChr(String *this, char find, char replace); // Format sizes (file, buffer, etc.) in human-readable form FN_EXTERN String *strSizeFormat(const uint64_t fileSize); // Convert string to string id FN_INLINE_ALWAYS StringId strStrId(const String *const this) { return strIdFromZN(strZ(this), strSize(this), STRING_ID_SEQ_NONE); } // Return a substring given only the start position FN_EXTERN String *strSub(const String *this, size_t start); // Return a substring given the start position and size FN_EXTERN String *strSubN(const String *this, size_t start, size_t size); // Trim whitespace from the beginning and end of a string FN_EXTERN String *strTrim(String *this); // Truncate the end of a string from the index provided to the current end (e.g. 123KB pass index of K returns 123) FN_EXTERN String *strTruncIdx(String *this, int idx); // Truncate the string to zero size FN_INLINE_ALWAYS String * strTrunc(String *const this) { return strTruncIdx(this, 0); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void strFree(String *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for constant strings Frequently used constant strings can be declared with these macros at compile time rather than dynamically at run time. Note that strings created in this way are declared as const so can't be modified or freed by the str*() methods. Casting to String * will result in a segfault due to modifying read-only memory. By convention all string constant identifiers are appended with _STR. ***********************************************************************************************************************************/ // This struct must be kept in sync with StringPub (except for const qualifiers) typedef struct StringPubConst { uint64_t size : 32; // Actual size of the string uint64_t extra : 32; // Extra space allocated for expansion const char *buffer; // String buffer } StringPubConst; #define STR_SIZE(bufferParam, sizeParam) \ ((const String *const)&(const StringPubConst){.buffer = (const char *const)(bufferParam), .size = (unsigned int)(sizeParam)}) // Create a String constant inline from any zero-terminated string #define STR(buffer) \ STR_SIZE(buffer, strlen(buffer)) // Create a String constant inline from a #define or inline string constant #define STRDEF(buffer) \ STR_SIZE(buffer, sizeof(buffer) - 1) // Used to define String constants that will be externed using STRING_DECLARE(). Must be used in a .c file. #define STRING_EXTERN(name, buffer) \ VR_EXTERN_DEFINE const String *const name = STRDEF(buffer) // Used to define String constants that will be local to the .c file. Must be used in a .c file. #define STRING_STATIC(name, buffer) \ static const String *const name = STRDEF(buffer) // Used to declare externed String constants defined with STRING_EXTERN(). Must be used in a .h file. #define STRING_DECLARE(name) \ VR_EXTERN_DECLARE const String *const name /*********************************************************************************************************************************** Constant strings that are generally useful ***********************************************************************************************************************************/ STRING_DECLARE(CR_STR); STRING_DECLARE(CRLF_STR); STRING_DECLARE(DOT_STR); STRING_DECLARE(DOTDOT_STR); STRING_DECLARE(EMPTY_STR); STRING_DECLARE(FALSE_STR); STRING_DECLARE(FSLASH_STR); STRING_DECLARE(LF_STR); STRING_DECLARE(N_STR); STRING_DECLARE(NULL_STR); STRING_DECLARE(TRUE_STR); STRING_DECLARE(Y_STR); STRING_DECLARE(ZERO_STR); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void strToLog(const String *this, StringStatic *debugLog); #define FUNCTION_LOG_STRING_TYPE \ String * #define FUNCTION_LOG_STRING_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, strToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/stringId.c0000664000175100017510000004177615240547630020421 0ustar00runnerrunner/*********************************************************************************************************************************** Represent Short Strings as Integers ***********************************************************************************************************************************/ #include #include "common/assert.h" #include "common/debug.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Number of bits to use for encoding. The number of bits affects the character set that can be encoded. ***********************************************************************************************************************************/ typedef enum { stringIdBit5 = 0, // 5-bit encoding for a-z, 2, 5, 6, and - characters stringIdBit6 = 1, // 6-bit encoding for a-z, 0-9, A-Z, and - characters } StringIdBit; /*********************************************************************************************************************************** Constants used to extract information from the header ***********************************************************************************************************************************/ #define STRING_ID_BIT 0x1 // Bit to indicate 5/6 bits #define STRING_ID_HEADER_SHIFT 4 // Shift for standard header // It is possible to store a max sequence of 69 if all bits are used in 6-bit encoding -- but this seems like overkill for now #define STRING_ID_SEQ_MAX 37 // Max sequence #define STRING_ID_SEQ_HIGH_MIN 6 // Sequence >= must be stored in high sequence bits #define STRING_ID_SEQ_HIGH_SHIFT 5 // Shift for high sequence bits #define STRING_ID_SEQ_LOW_MASK 0xE // Mask for low sequence bits #define STRING_ID_SEQ_HIGH_MASK 0x1F0 // Mask for high sequence bits /**********************************************************************************************************************************/ // Helper to do encoding for specified number of bits static StringId strIdBitFromZN(const StringIdBit bit, const char *const buffer, size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, bit); FUNCTION_TEST_PARAM(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(size > 0); // Encoding type switch (bit) { // 5-bit encoding case stringIdBit5: { // Map to convert characters to encoding // {uncrustify_off - array alignment} static const uint8_t map[256] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 0, 0, 28, 0, 0, 29, 30, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, }; // {uncrustify_on} // If size is greater than can be encoded then error if (size > STRID5_MAX) FUNCTION_TEST_RETURN(STRING_ID, 0); // Make sure the string is valid for this encoding for (size_t bufferIdx = 0; bufferIdx < size; bufferIdx++) { if (map[(uint8_t)buffer[bufferIdx]] == 0) FUNCTION_TEST_RETURN(STRING_ID, 0); } // Set encoding in header uint64_t result = stringIdBit5; // Encode based on the number of characters that need to be encoded switch (size) { case 12: result |= (uint64_t)map[(uint8_t)buffer[11]] << 59; __attribute__((fallthrough)); case 11: result |= (uint64_t)map[(uint8_t)buffer[10]] << 54; __attribute__((fallthrough)); case 10: result |= (uint64_t)map[(uint8_t)buffer[9]] << 49; __attribute__((fallthrough)); case 9: result |= (uint64_t)map[(uint8_t)buffer[8]] << 44; __attribute__((fallthrough)); case 8: result |= (uint64_t)map[(uint8_t)buffer[7]] << 39; __attribute__((fallthrough)); case 7: result |= (uint64_t)map[(uint8_t)buffer[6]] << 34; __attribute__((fallthrough)); case 6: result |= (uint64_t)map[(uint8_t)buffer[5]] << 29; __attribute__((fallthrough)); case 5: result |= (uint64_t)map[(uint8_t)buffer[4]] << 24; __attribute__((fallthrough)); case 4: result |= (uint64_t)map[(uint8_t)buffer[3]] << 19; __attribute__((fallthrough)); case 3: result |= (uint64_t)map[(uint8_t)buffer[2]] << 14; __attribute__((fallthrough)); case 2: result |= (uint64_t)map[(uint8_t)buffer[1]] << 9; __attribute__((fallthrough)); default: result |= (uint64_t)map[(uint8_t)buffer[0]] << 4; } FUNCTION_TEST_RETURN(STRING_ID, result); } // 6-bit encoding default: { ASSERT(bit == stringIdBit6); // Map to convert characters to encoding // {uncrustify_off - array alignment} static const uint8_t map[256] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 27, 0, 0, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 0, 0, 0, 0, 0, 0, 0, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, }; // {uncrustify_on} // If size is greater than can be encoded then error if (size > STRID6_MAX) FUNCTION_TEST_RETURN(STRING_ID, 0); // Make sure the string is valid for this encoding for (size_t bufferIdx = 0; bufferIdx < size; bufferIdx++) { if (map[(uint8_t)buffer[bufferIdx]] == 0) FUNCTION_TEST_RETURN(STRING_ID, 0); } // Set encoding in header uint64_t result = stringIdBit6; // Encode based on the number of characters that need to be encoded switch (size) { case 10: result |= (uint64_t)map[(uint8_t)buffer[9]] << 58; __attribute__((fallthrough)); case 9: result |= (uint64_t)map[(uint8_t)buffer[8]] << 52; __attribute__((fallthrough)); case 8: result |= (uint64_t)map[(uint8_t)buffer[7]] << 46; __attribute__((fallthrough)); case 7: result |= (uint64_t)map[(uint8_t)buffer[6]] << 40; __attribute__((fallthrough)); case 6: result |= (uint64_t)map[(uint8_t)buffer[5]] << 34; __attribute__((fallthrough)); case 5: result |= (uint64_t)map[(uint8_t)buffer[4]] << 28; __attribute__((fallthrough)); case 4: result |= (uint64_t)map[(uint8_t)buffer[3]] << 22; __attribute__((fallthrough)); case 3: result |= (uint64_t)map[(uint8_t)buffer[2]] << 16; __attribute__((fallthrough)); case 2: result |= (uint64_t)map[(uint8_t)buffer[1]] << 10; __attribute__((fallthrough)); default: result |= (uint64_t)map[(uint8_t)buffer[0]] << 4; } FUNCTION_TEST_RETURN(STRING_ID, result); } } } FN_EXTERN StringId strIdFromZN(const char *const buffer, const size_t size, const unsigned int sequence) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VOID, buffer); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_PARAM(UINT, sequence); FUNCTION_TEST_END(); StringId result = strIdBitFromZN(stringIdBit5, buffer, size); // If 5-bit encoding fails try 6-bit if (result == 0) { result = strIdBitFromZN(stringIdBit6, buffer, size); // Error when 6-bit encoding also fails if (result == 0) THROW_FMT(FormatError, "'%s' contains invalid characters", buffer); } // Add sequence when specified if (sequence != STRING_ID_SEQ_NONE) { // If sequence can fit in header if (sequence < STRING_ID_SEQ_HIGH_MIN) { result |= (sequence + 1) << STRING_ID_BIT; } // Else sequence must be stored in high bits else { const StringIdBit bit = (StringIdBit)(result & STRING_ID_BIT); ASSERT(size < (bit ? STRID6_MAX : STRID5_MAX)); ASSERT(sequence <= STRING_ID_SEQ_MAX); // Shift to remove the bit marker (will be added back later) and make space for high sequence result >>= STRING_ID_BIT; result <<= STRING_ID_SEQ_HIGH_SHIFT + STRING_ID_BIT; // Add the sequence result |= (sequence - STRING_ID_SEQ_HIGH_MIN) << STRING_ID_HEADER_SHIFT | STRING_ID_SEQ_LOW_MASK | bit; } } FUNCTION_TEST_RETURN(STRING_ID, result); } /**********************************************************************************************************************************/ FN_EXTERN size_t strIdToZN(StringId strId, char *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_PARAM(VOID, buffer); FUNCTION_TEST_END(); ASSERT(strId != 0); ASSERT(buffer != NULL); // Extract bits used to encode the characters StringIdBit bit = (StringIdBit)(strId & STRING_ID_BIT); // Remove header to get the encoded characters if ((strId & STRING_ID_SEQ_LOW_MASK) == STRING_ID_SEQ_LOW_MASK) strId >>= STRING_ID_SEQ_HIGH_SHIFT + STRING_ID_HEADER_SHIFT; else strId >>= STRING_ID_HEADER_SHIFT; // Decoding type switch (bit) { // 5-bit decoding case stringIdBit5: { // Map to convert encoding to characters VR_NON_STRING static const char map[32] = "!abcdefghijklmnopqrstuvwxyz-256!"; // Macro to decode all but the last character #define STR5ID_TO_ZN_IDX(idx) \ buffer[idx] = map[strId & 0x1F]; \ strId >>= 5; \ \ if (strId == 0) \ FUNCTION_TEST_RETURN(SIZE, idx + 1) // Char 1-11 STR5ID_TO_ZN_IDX(0); STR5ID_TO_ZN_IDX(1); STR5ID_TO_ZN_IDX(2); STR5ID_TO_ZN_IDX(3); STR5ID_TO_ZN_IDX(4); STR5ID_TO_ZN_IDX(5); STR5ID_TO_ZN_IDX(6); STR5ID_TO_ZN_IDX(7); STR5ID_TO_ZN_IDX(8); STR5ID_TO_ZN_IDX(9); STR5ID_TO_ZN_IDX(10); // Char 12 buffer[11] = map[strId & 0x1F]; ASSERT(strId >> 5 == 0); FUNCTION_TEST_RETURN(SIZE, 12); } // 6-bit decoding default: { ASSERT(bit == stringIdBit6); // Map to convert encoding to characters VR_NON_STRING static const char map[64] = "!abcdefghijklmnopqrstuvwxyz-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"; // Macro to decode all but the last character #define STR6ID_TO_ZN_IDX(idx) \ buffer[idx] = map[strId & 0x3F]; \ strId >>= 6; \ \ if (strId == 0) \ FUNCTION_TEST_RETURN(SIZE, idx + 1) // Char 1-9 STR6ID_TO_ZN_IDX(0); STR6ID_TO_ZN_IDX(1); STR6ID_TO_ZN_IDX(2); STR6ID_TO_ZN_IDX(3); STR6ID_TO_ZN_IDX(4); STR6ID_TO_ZN_IDX(5); STR6ID_TO_ZN_IDX(6); STR6ID_TO_ZN_IDX(7); STR6ID_TO_ZN_IDX(8); // Char 10 buffer[9] = map[strId & 0x3F]; ASSERT(strId >> 6 == 0); FUNCTION_TEST_RETURN(SIZE, 10); } } } /**********************************************************************************************************************************/ FN_EXTERN size_t strIdToZ(const StringId strId, char *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_PARAM(VOID, buffer); FUNCTION_TEST_END(); size_t size = strIdToZN(strId, buffer); buffer[size] = '\0'; FUNCTION_TEST_RETURN(SIZE, size); } FN_EXTERN unsigned int strIdSeq(const StringId strId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_END(); const unsigned int sequence = (unsigned int)strId & STRING_ID_SEQ_LOW_MASK; if (sequence != STRING_ID_SEQ_LOW_MASK) { ASSERT(sequence != 0); FUNCTION_TEST_RETURN(UINT, (sequence >> STRING_ID_BIT) - 1); } FUNCTION_TEST_RETURN( UINT, (((unsigned int)strId & STRING_ID_SEQ_HIGH_MASK) >> STRING_ID_HEADER_SHIFT) + STRING_ID_SEQ_HIGH_MIN); } /**********************************************************************************************************************************/ FN_EXTERN size_t strIdToLog(const StringId strId, char *const buffer, const size_t bufferSize) { ASSERT(bufferSize > STRID_MAX); (void)bufferSize; // Treat 0 as if it were null since this can never be a valid StringId if (strId == 0) return (size_t)snprintf(buffer, bufferSize, NULL_Z); return strIdToZ(strId, buffer); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/stringId.h0000664000175100017510000001524715240547630020420 0ustar00runnerrunner/*********************************************************************************************************************************** Represent Short Strings as Integers It is often useful to represent identifiers as strings when they cannot easily be represented as an enum/integer, e.g. because they are distributed among a number of unrelated modules or need to be passed to remote processes. Strings are also more helpful in debugging since they can be recognized without cross-referencing the source. However, strings are awkward to work with in C since they cannot be directly used in switch statements leading to less efficient if-else structures. A StringId encodes a short string into an integer so it can be used in switch statements but may also be readily converted back into a string for debugging purposes. StringIds may also be suitable for matching user input providing the strings are short enough. strIdFromZ("mytest0123") will return the StringId 0x7de75c51315464d1. Using the value, the string representation can be retrieved with strIdToZ(0x7de75c51315464d1, ...) which returns "mytest0123+" where the plus at the end signals that the original string was equal to or longer than the maximum allowed. When assigning a StringId to an enum, it will be necessary to cast the StringId to the enum type if the enum contains all 32-bit values, since some compilers will complain about the implicit conversion without a cast. The enum will be 32-bit if all values of the enum are <= 0xffffffff. See bldStrId() for information on generating StringId constants. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRINGID_H #define COMMON_TYPE_STRINGID_H #include #include #include #include /*********************************************************************************************************************************** Maximum number of characters in the string representation of a StringId. When the string representation is equal to or greater than the MAX, then a plus sign + will be assigned as the last (MAX) character in the string representation. This is a safe buffer size when calling strIdToZN. If the buffer needs to be zero-terminated then an extra byte should be added. ***********************************************************************************************************************************/ // Maximum for specific encodings (e.g. 5-bit, 6-bit) #define STRID5_MAX 12 #define STRID6_MAX 10 // Maximum for any encoding #define STRID_MAX STRID5_MAX /*********************************************************************************************************************************** Constants used to extract information from the header ***********************************************************************************************************************************/ #define STRING_ID_BIT_MASK 3 /*********************************************************************************************************************************** Indicator for no sequence value ***********************************************************************************************************************************/ #define STRING_ID_SEQ_NONE UINT_MAX /*********************************************************************************************************************************** StringId typedef to make them more recognizable in the code ***********************************************************************************************************************************/ typedef uint64_t StringId; /*********************************************************************************************************************************** Macros to define constant StringIds. ALWAYS use bldStrId() to create these macros. The parameters in the macros are not verified against each other so the str parameter is included only for documentation purposes. ***********************************************************************************************************************************/ #define STRID5(str, strId) strId #define STRID5S(str, seq, strId) strId #define STRID6(str, strId) strId #define STRID6S(str, seq, strId) strId /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Convert N chars to StringId and optionally add sequence FN_EXTERN StringId strIdFromZN(const char *buffer, size_t size, unsigned int sequence); // Convert zero-terminated string to StringId and add sequence FN_INLINE_ALWAYS StringId strIdSeqFromZ(const char *const str, const unsigned int sequence) { return strIdFromZN(str, strlen(str), sequence); } // Convert zero-terminated string to StringId FN_INLINE_ALWAYS StringId strIdFromZ(const char *const str) { return strIdSeqFromZ(str, STRING_ID_SEQ_NONE); } // Write StringId to characters without zero-terminating. The buffer at ptr must have enough space to write the entire StringId, // which could be eight characters. However, the caller may know the exact (or max length) in advance and act accordingly. The // actual number of bytes written is returned. FN_EXTERN size_t strIdToZN(StringId strId, char *const buffer); // Convert StringId to zero-terminated string. See strIdToZN() for buffer sizing and return value. FN_EXTERN size_t strIdToZ(StringId strId, char *const buffer); // Get the sequence for a StringId (or error if none) FN_EXTERN unsigned int strIdSeq(StringId strId); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN size_t strIdToLog(StringId strId, char *buffer, size_t bufferSize); #define FUNCTION_LOG_STRING_ID_TYPE \ StringId #define FUNCTION_LOG_STRING_ID_FORMAT(value, buffer, bufferSize) \ strIdToLog(value, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/stringList.c0000664000175100017510000003334415240547630020770 0ustar00runnerrunner/*********************************************************************************************************************************** String List Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Internal add -- the string must have been created in the list's mem context before being passed ***********************************************************************************************************************************/ FN_INLINE_ALWAYS String * strLstAddInternal(StringList *const this, String *const string) { return *(String **)lstAdd((List *)this, &string); } /*********************************************************************************************************************************** Internal insert -- the string must have been created in the list's mem context before being passed ***********************************************************************************************************************************/ FN_INLINE_ALWAYS String * strLstInsertInternal(StringList *const this, const unsigned int listIdx, String *const string) { return *(String **)lstInsert((List *)this, listIdx, &string); } /**********************************************************************************************************************************/ FN_EXTERN StringList * strLstNewSplitZ(const String *const string, const char *const delimiter) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_PARAM(STRINGZ, delimiter); FUNCTION_TEST_END(); ASSERT(string != NULL); ASSERT(delimiter != NULL); // Create the list StringList *const this = strLstNew(); // Base points to the beginning of the string that is being searched const char *stringBase = strZ(string); // Match points to the next delimiter match that has been found const char *stringMatch = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { do { // Find a delimiter match stringMatch = strstr(stringBase, delimiter); // If a match was found then add the string if (stringMatch != NULL) { strLstAddInternal(this, strNewZN(stringBase, (size_t)(stringMatch - stringBase))); stringBase = stringMatch + strlen(delimiter); } // Else make whatever is left the last string else strLstAddInternal(this, strNewZ(stringBase)); } while (stringMatch != NULL); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN StringList * strLstNewVarLst(const VariantList *const sourceList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, sourceList); FUNCTION_TEST_END(); // Create the list StringList *this = NULL; if (sourceList != NULL) { this = strLstNew(); // Copy variants MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { for (unsigned int listIdx = 0; listIdx < varLstSize(sourceList); listIdx++) strLstAddInternal(this, strDup(varStr(varLstGet(sourceList, listIdx)))); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN(STRING_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN StringList * strLstDup(const StringList *const sourceList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, sourceList); FUNCTION_TEST_END(); StringList *this = NULL; if (sourceList != NULL) { // Create the list this = strLstNew(); // Copy strings MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { for (unsigned int listIdx = 0; listIdx < strLstSize(sourceList); listIdx++) strLstAddInternal(this, strDup(strLstGet(sourceList, listIdx))); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN(STRING_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN String * strLstAdd(StringList *const this, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strDup(string)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * strLstAddSubN(StringList *const this, const String *const string, const size_t offset, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(string != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strSubN(string, offset, size)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * strLstAddFmt(StringList *const this, const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(format != NULL); // Determine how long the allocated string needs to be va_list argumentList; va_start(argumentList, format); const size_t size = (size_t)vsnprintf(NULL, 0, format, argumentList); va_end(argumentList); // Format string va_start(argumentList, format); char *const buffer = memNew(size + 1); vsnprintf(buffer, size + 1, format, argumentList); va_end(argumentList); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strNewZN(buffer, size)); } MEM_CONTEXT_END(); memFree(buffer); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * strLstAddIfMissing(StringList *const this, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(this != NULL); String *const *const result = lstFind((List *)this, &string); if (result == NULL) FUNCTION_TEST_RETURN(STRING, strLstAdd(this, string)); FUNCTION_TEST_RETURN(STRING, *result); } FN_EXTERN String * strLstAddZ(StringList *const this, const char *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRINGZ, string); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strNewZ(string)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * strLstAddZSubN(StringList *const this, const char *const string, const size_t offset, const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRINGZ, string); FUNCTION_TEST_PARAM(SIZE, offset); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(string != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstAddInternal(this, strSubN(STR(string), offset, size)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int strLstFindIdx(const StringList *const this, const String *const string, const StrLstFindIdxParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_PARAM(BOOL, param.required); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(string != NULL); const unsigned int result = lstFindIdx((const List *)this, &string); if (result == LIST_NOT_FOUND && param.required) THROW_FMT(AssertError, "unable to find '%s' in string list", strZ(string)); FUNCTION_TEST_RETURN(UINT, result); } /**********************************************************************************************************************************/ FN_EXTERN String * strLstInsert(StringList *const this, const unsigned int listIdx, const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(UINT, listIdx); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { result = strLstInsertInternal(this, listIdx, strDup(string)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * strLstJoinQuote(const StringList *const this, const char *const separator, const char *const quote) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRINGZ, separator); FUNCTION_TEST_PARAM(STRINGZ, quote); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(separator != NULL); ASSERT(quote != NULL); String *const join = strNew(); for (unsigned int listIdx = 0; listIdx < strLstSize(this); listIdx++) { if (listIdx != 0) strCatZ(join, separator); if (strLstGet(this, listIdx) == NULL) strCatZ(join, "[NULL]"); else strCatFmt(join, "%s%s%s", quote, strZ(strLstGet(this, listIdx)), quote); } FUNCTION_TEST_RETURN(STRING, join); } /**********************************************************************************************************************************/ FN_EXTERN StringList * strLstMergeAnti(const StringList *const this, const StringList *const anti) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_PARAM(STRING_LIST, anti); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(anti != NULL); StringList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { result = strLstNew(); unsigned int antiIdx = 0; // Check every item in this for (unsigned int thisIdx = 0; thisIdx < strLstSize(this); thisIdx++) { bool add = true; const String *const listItem = strLstGet(this, thisIdx); ASSERT(listItem != NULL); // If anything left in anti look for matches while (antiIdx < strLstSize(anti)) { ASSERT(strLstGet(anti, antiIdx) != NULL); const int compare = strCmp(listItem, strLstGet(anti, antiIdx)); // If the current item in this is less than the current item in anti then it will be added if (compare < 0) { break; } // If they are equal it will not be added else if (compare == 0) { add = false; antiIdx++; break; } // Otherwise keep searching anti for a match else antiIdx++; } // Add to the result list if no match found if (add) strLstAdd(result, listItem); } strLstMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN const char ** strLstPtr(const StringList *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, this); FUNCTION_TEST_END(); ASSERT(this != NULL); const char **const list = memNew((strLstSize(this) + 1) * sizeof(char *)); for (unsigned int listIdx = 0; listIdx < strLstSize(this); listIdx++) { if (strLstGet(this, listIdx) == NULL) list[listIdx] = NULL; else list[listIdx] = strZ(strLstGet(this, listIdx)); } list[strLstSize(this)] = NULL; FUNCTION_TEST_RETURN_CONST_P(STRINGZ, list); } /**********************************************************************************************************************************/ FN_EXTERN void strLstToLog(const StringList *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{["); for (unsigned int strLstIdx = 0; strLstIdx < strLstSize(this); strLstIdx++) { const String *const value = strLstGet(this, strLstIdx); if (strLstIdx != 0) strStcCat(debugLog, ", "); if (value == NULL) strStcCat(debugLog, NULL_Z); else strStcFmt(debugLog, "\"%s\"", strZ(value)); } strStcCat(debugLog, "]}"); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/stringList.h0000664000175100017510000001636615240547630021002 0ustar00runnerrunner/*********************************************************************************************************************************** String List Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRINGLIST_H #define COMMON_TYPE_STRINGLIST_H /*********************************************************************************************************************************** StringList object ***********************************************************************************************************************************/ typedef struct StringList StringList; #include "common/memContext.h" #include "common/type/list.h" #include "common/type/string.h" #include "common/type/variantList.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Create empty StringList FN_INLINE_ALWAYS StringList * strLstNew(void) { return (StringList *)OBJ_NAME(lstNewP(sizeof(String *), .comparator = lstComparatorStr), StringList::List); } // Split a string into a string list based on a delimiter FN_EXTERN StringList *strLstNewSplitZ(const String *string, const char *delimiter); FN_INLINE_ALWAYS StringList * strLstNewSplit(const String *const string, const String *const delimiter) { return strLstNewSplitZ(string, strZ(delimiter)); } // New string list from a variant list -- all variants in the list must be type string FN_EXTERN StringList *strLstNewVarLst(const VariantList *sourceList); // Duplicate a string list FN_EXTERN StringList *strLstDup(const StringList *sourceList); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Set a new comparator FN_INLINE_ALWAYS StringList * strLstComparatorSet(StringList *const this, ListComparator *const comparator) { return (StringList *)lstComparatorSet((List *)this, comparator); } // List size FN_INLINE_ALWAYS unsigned int strLstSize(const StringList *const this) { return lstSize((const List *)this); } // Is the list empty? FN_INLINE_ALWAYS bool strLstEmpty(const StringList *const this) { return strLstSize(this) == 0; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add String to the list FN_EXTERN String *strLstAdd(StringList *this, const String *string); FN_EXTERN String *strLstAddSubN(StringList *this, const String *string, size_t offset, size_t size); FN_INLINE_ALWAYS String * strLstAddSub(StringList *const this, const String *const string, const size_t size) { return strLstAddSubN(this, string, 0, size); } FN_EXTERN FN_PRINTF(2, 3) String *strLstAddFmt(StringList *this, const char *format, ...); FN_EXTERN String *strLstAddZ(StringList *this, const char *string); FN_EXTERN String *strLstAddZSubN(StringList *this, const char *string, size_t offset, size_t size); FN_INLINE_ALWAYS String * strLstAddZSub(StringList *const this, const char *const string, const size_t size) { return strLstAddZSubN(this, string, 0, size); } FN_EXTERN String *strLstAddIfMissing(StringList *this, const String *string); // Does the specified string exist in the list? FN_INLINE_ALWAYS bool strLstExists(const StringList *const this, const String *const string) { return lstExists((const List *)this, &string); } // Find string index in the list typedef struct StrLstFindIdxParam { VAR_PARAM_HEADER; bool required; } StrLstFindIdxParam; #define strLstFindIdxP(this, string, ...) \ strLstFindIdx(this, string, (StrLstFindIdxParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN unsigned int strLstFindIdx(const StringList *this, const String *string, StrLstFindIdxParam param); // Insert into the list FN_EXTERN String *strLstInsert(StringList *this, unsigned int listIdx, const String *string); // Get a string by index FN_INLINE_ALWAYS String * strLstGet(const StringList *const this, const unsigned int listIdx) { return *(String **)lstGet((const List *)this, listIdx); } // Join a list of strings into a single string using the specified separator and quote with specified quote character FN_EXTERN String *strLstJoinQuote(const StringList *this, const char *separator, const char *quote); // Join a list of strings into a single string using the specified separator FN_INLINE_ALWAYS String * strLstJoin(const StringList *const this, const char *const separator) { return strLstJoinQuote(this, separator, ""); } // Return all items in this list which are not in the anti list. The input lists must *both* be sorted ascending or the results will // be undefined. FN_EXTERN StringList *strLstMergeAnti(const StringList *this, const StringList *anti); // Move to a new parent mem context FN_INLINE_ALWAYS StringList * strLstMove(StringList *const this, MemContext *const parentNew) { return (StringList *)lstMove((List *)this, parentNew); } // Return a null-terminated array of pointers to the zero-terminated strings in this list. DO NOT override const and modify any of // the strings in this array, though it is OK to modify the array itself. FN_EXTERN const char **strLstPtr(const StringList *this); // Remove an item from the list FN_INLINE_ALWAYS bool strLstRemove(StringList *const this, const String *const item) { return lstRemove((List *)this, &item); } FN_INLINE_ALWAYS StringList * strLstRemoveIdx(StringList *const this, const unsigned int listIdx) { return (StringList *)lstRemoveIdx((List *)this, listIdx); } // List sort FN_INLINE_ALWAYS StringList * strLstSort(StringList *const this, const SortOrder sortOrder) { return (StringList *)lstSort((List *)this, sortOrder); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void strLstFree(StringList *const this) { lstFree((List *)this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void strLstToLog(const StringList *this, StringStatic *debugLog); #define FUNCTION_LOG_STRING_LIST_TYPE \ StringList * #define FUNCTION_LOG_STRING_LIST_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, strLstToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/stringStatic.c0000664000175100017510000000421615240547630021300 0ustar00runnerrunner/*********************************************************************************************************************************** Static String Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/type/stringStatic.h" /**********************************************************************************************************************************/ FN_EXTERN StringStatic * strStcFmt(StringStatic *const debugLog, const char *const format, ...) { // Proceed if there is space for at least one character if (strStcRemainsSize(debugLog) > 1) { va_list argument; va_start(argument, format); size_t result = (size_t)vsnprintf(strStcRemains(debugLog), strStcRemainsSize(debugLog), format, argument); va_end(argument); if (result >= strStcRemainsSize(debugLog)) debugLog->resultSize = debugLog->bufferSize - 1; else debugLog->resultSize += result; } return debugLog; } /**********************************************************************************************************************************/ FN_EXTERN void strStcCat(StringStatic *const debugLog, const char *const cat) { // Proceed if there is space for at least one character const size_t remainsSize = strStcRemainsSize(debugLog); if (remainsSize > 1) { const size_t catSize = strlen(cat); const size_t resultSize = catSize > remainsSize - 1 ? remainsSize - 1 : catSize; memcpy(strStcRemains(debugLog), cat, resultSize); debugLog->resultSize += resultSize; debugLog->buffer[debugLog->resultSize] = '\0'; } } /**********************************************************************************************************************************/ FN_EXTERN void strStcCatChr(StringStatic *const debugLog, const char cat) { if (strStcRemainsSize(debugLog) > 1) { debugLog->buffer[debugLog->resultSize] = cat; debugLog->buffer[++debugLog->resultSize] = '\0'; } } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/stringStatic.h0000664000175100017510000000664315240547630021313 0ustar00runnerrunner/*********************************************************************************************************************************** Static String Handler Format a statically-size char buffer with automatic truncation and size tracking. These are convenience methods on top of the built-in format and memcpy functions to make the interface simpler. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRINGSTATIC_H #define COMMON_TYPE_STRINGSTATIC_H #include /*********************************************************************************************************************************** Zero-terminated strings that are generally useful ***********************************************************************************************************************************/ #define FALSE_Z "false" #define NULL_Z "null" #define TRUE_Z "true" /*********************************************************************************************************************************** Type ***********************************************************************************************************************************/ typedef struct StringStatic { char *buffer; size_t bufferSize; size_t resultSize; } StringStatic; /*********************************************************************************************************************************** Constructor ***********************************************************************************************************************************/ // Init static string FN_INLINE_ALWAYS StringStatic strStcInit(char *const buffer, const size_t bufferSize) { return (StringStatic){.buffer = buffer, .bufferSize = bufferSize}; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Cat formatted string to static string FN_EXTERN FN_PRINTF(2, 3) StringStatic *strStcFmt(StringStatic *debugLog, const char *format, ...); // Cat zero-terminated string to static string FN_EXTERN void strStcCat(StringStatic *debugLog, const char *cat); // Cat character to static string FN_EXTERN void strStcCatChr(StringStatic *debugLog, char cat); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Pointer to remaining buffer FN_INLINE_ALWAYS char * strStcRemains(const StringStatic *const debugLog) { return debugLog->buffer + debugLog->resultSize; } // Remaining buffer size FN_INLINE_ALWAYS size_t strStcRemainsSize(const StringStatic *const debugLog) { return debugLog->bufferSize - debugLog->resultSize; } // Result size FN_INLINE_ALWAYS size_t strStcResultSize(const StringStatic *const debugLog) { return debugLog->resultSize; } // Increment result size FN_INLINE_ALWAYS void strStcResultSizeInc(StringStatic *const debugLog, const size_t inc) { debugLog->resultSize += inc; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/stringZ.c0000664000175100017510000000470515240547630020265 0ustar00runnerrunner/*********************************************************************************************************************************** Zero-Terminated String Handler ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/object.h" #include "common/type/stringId.h" #include "common/type/stringZ.h" /**********************************************************************************************************************************/ static char * zNewInternal(const size_t size) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, size); FUNCTION_TEST_END(); // Allocate the string as extra or a separate allocation based on how large it is char *result; OBJ_NEW_BASE_EXTRA_BEGIN( char *, size > MEM_CONTEXT_ALLOC_EXTRA_MAX ? 0 : (uint16_t)size, .allocQty = size > MEM_CONTEXT_ALLOC_EXTRA_MAX ? 1 : 0) { result = size > MEM_CONTEXT_ALLOC_EXTRA_MAX ? memNew(size) : OBJ_NEW_ALLOC(); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STRINGZ, result); } /**********************************************************************************************************************************/ FN_EXTERN char * zNewFmt(const char *const format, ...) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, format); FUNCTION_TEST_END(); ASSERT(format != NULL); // Determine how long the allocated string needs to be va_list argumentList; va_start(argumentList, format); const size_t size = (size_t)vsnprintf(NULL, 0, format, argumentList) + 1; va_end(argumentList); // Format string char *const result = zNewInternal(size); va_start(argumentList, format); vsnprintf(result, size, format, argumentList); va_end(argumentList); FUNCTION_TEST_RETURN(STRINGZ, result); } /**********************************************************************************************************************************/ FN_EXTERN char * zNewStrId(const StringId strId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, strId); FUNCTION_TEST_END(); ASSERT(strId != 0); ASSERT(MEM_CONTEXT_ALLOC_EXTRA_MAX >= STRID_MAX + 1); char *const result = zNewInternal(STRID_MAX + 1); result[strIdToZN(strId, result)] = '\0'; FUNCTION_TEST_RETURN(STRINGZ, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/stringZ.h0000664000175100017510000000270715240547630020272 0ustar00runnerrunner/*********************************************************************************************************************************** Zero-Terminated String Handler These functions are shortcuts for creating/modifying zero-terminated strings without needing to wrap a strNew*() call in strZ(). The benefit is simpler code with less indentation, but be aware that memory is still being allocated. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_STRINGZ_H #define COMMON_TYPE_STRINGZ_H #include "common/type/object.h" #include "common/type/stringId.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Format a zero-terminated string FN_EXTERN FN_PRINTF(1, 2) char *zNewFmt(const char *format, ...); // Convert StringId to zero-terminated string FN_EXTERN char *zNewStrId(StringId strId); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void zFree(char *const this) { objFree(this); } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/variant.c0000664000175100017510000007331415240547630020273 0ustar00runnerrunner/*********************************************************************************************************************************** Variant Data Type ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/convert.h" #include "common/type/variant.h" /*********************************************************************************************************************************** Constant variants that are generally useful ***********************************************************************************************************************************/ // Used to define Bool Variant constants that will be externed using VARIANT_DECLARE(). #define VARIANT_BOOL_EXTERN(name, dataParam) \ VR_EXTERN_DEFINE const Variant *const name = ((const Variant *)&(const VariantBoolPub){.type = varTypeBool, .data = dataParam}) VARIANT_BOOL_EXTERN(BOOL_FALSE_VAR, false); VARIANT_BOOL_EXTERN(BOOL_TRUE_VAR, true); /*********************************************************************************************************************************** Information about the variant ***********************************************************************************************************************************/ struct Variant { VariantPub pub; // Publicly accessible variables }; typedef struct VariantBool { VariantBoolPub pub; // Publicly accessible variables } VariantBool; typedef struct VariantInt { VariantIntPub pub; // Publicly accessible variables } VariantInt; typedef struct VariantInt64 { VariantInt64Pub pub; // Publicly accessible variables } VariantInt64; typedef struct VariantKeyValue { VARIANT_COMMON KeyValue *data; // KeyValue data } VariantKeyValue; typedef struct VariantString { VariantStringPub pub; // Publicly accessible variables } VariantString; typedef struct VariantUInt { VariantUIntPub pub; // Publicly accessible variables } VariantUInt; typedef struct VariantUInt64 { VariantUInt64Pub pub; // Publicly accessible variables } VariantUInt64; typedef struct VariantVariantList { VARIANT_COMMON VariantList *data; // VariantList data } VariantVariantList; /*********************************************************************************************************************************** Variant type names ***********************************************************************************************************************************/ static const char *const variantTypeName[] = { "bool", // varTypeBool "int", // varTypeInt "int64", // varTypeInt64 "KeyValue", // varTypeKeyValue "String", // varTypeString "unsigned int", // varTypeUInt "uint64", // varTypeUInt64 "VariantList", // varTypeVariantList }; /**********************************************************************************************************************************/ FN_EXTERN Variant * varDup(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); Variant *result = NULL; if (this != NULL) { switch (varType(this)) { case varTypeBool: result = varNewBool(varBool(this)); break; case varTypeInt: result = varNewInt(varInt(this)); break; case varTypeInt64: result = varNewInt64(varInt64(this)); break; case varTypeKeyValue: result = varNewKv(kvDup(varKv(this))); break; case varTypeString: result = varNewStr(varStr(this)); break; case varTypeUInt: result = varNewUInt(varUInt(this)); break; case varTypeUInt64: result = varNewUInt64(varUInt64(this)); break; case varTypeVariantList: result = varNewVarLst(varVarLst(this)); break; } } FUNCTION_TEST_RETURN(VARIANT, result); } /**********************************************************************************************************************************/ FN_EXTERN bool varEq(const Variant *const this1, const Variant *const this2) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this1); FUNCTION_TEST_PARAM(VARIANT, this2); FUNCTION_TEST_END(); bool result = false; // Test if both variants are non-null if (this1 != NULL && this2 != NULL) { // Test if both variants are of the same type if (varType(this1) == varType(this2)) { switch (varType(this1)) { case varTypeBool: result = varBool(this1) == varBool(this2); break; case varTypeInt: result = varInt(this1) == varInt(this2); break; case varTypeInt64: result = varInt64(this1) == varInt64(this2); break; case varTypeString: result = strEq(varStr(this1), varStr(this2)); break; case varTypeUInt: result = varUInt(this1) == varUInt(this2); break; case varTypeUInt64: result = varUInt64(this1) == varUInt64(this2); break; default: ASSERT(varType(this1) == varTypeKeyValue || varType(this1) == varTypeVariantList); THROW_FMT(AssertError, "unable to test equality for %s", variantTypeName[varType(this1)]); } } } // Else they are equal if they are both null else result = this1 == NULL && this2 == NULL; FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewBool(const bool data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantBool) { *this = (VariantBool) { .pub = { .type = varTypeBool, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantBool)); } /**********************************************************************************************************************************/ FN_EXTERN bool varBool(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeBool); FUNCTION_TEST_RETURN(BOOL, ((const VariantBool *)this)->pub.data); } FN_EXTERN bool varBoolForce(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); bool result = false; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: result = varInt(this) != 0; break; case varTypeInt64: result = varInt64(this) != 0; break; case varTypeString: { // List of false/true boolean string values. Note that false/true values must be equal. static const char *const boolString[] = { "n", "f", "0", "no", FALSE_Z, "off", "y", "t", "1", "yes", TRUE_Z, "on", }; // Search for the string const char *const string = strZ(varStr(this)); unsigned int boolIdx = 0; for (; boolIdx < LENGTH_OF(boolString); boolIdx++) if (strcasecmp(string, boolString[boolIdx]) == 0) break; // If string was not found then not a boolean if (boolIdx == LENGTH_OF(boolString)) THROW_FMT(FormatError, "unable to convert str '%s' to bool", string); // False if in first half of list, true if in second half result = boolIdx / (LENGTH_OF(boolString) / 2); break; } case varTypeUInt: result = varUInt(this) != 0; break; case varTypeUInt64: result = varUInt64(this) != 0; break; default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeBool]); } FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewInt(const int data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantInt) { *this = (VariantInt) { .pub = { .type = varTypeInt, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantInt)); } /**********************************************************************************************************************************/ FN_EXTERN int varInt(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeInt); FUNCTION_TEST_RETURN(INT, ((const VariantInt *)this)->pub.data); } FN_EXTERN int varIntForce(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); int result = 0; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: result = varInt(this); break; case varTypeInt64: { const int64_t resultTest = varInt64(this); // Make sure the value fits into a normal 32-bit int range since 32-bit platforms are supported if (resultTest > INT32_MAX || resultTest < INT32_MIN) THROW_FMT( FormatError, "unable to convert %s %" PRId64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeInt]); result = (int)resultTest; break; } case varTypeString: result = cvtZToInt(strZ(varStr(this))); break; case varTypeUInt: { const unsigned int resultTest = varUInt(this); // Make sure the value fits into a normal 32-bit int range if (resultTest > INT32_MAX) THROW_FMT( FormatError, "unable to convert %s %u to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeInt]); result = (int)resultTest; break; } case varTypeUInt64: { const uint64_t resultTest = varUInt64(this); // Make sure the value fits into a normal 32-bit int range if (resultTest > INT32_MAX) THROW_FMT( FormatError, "unable to convert %s %" PRIu64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeInt]); result = (int)resultTest; break; } default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeInt]); } FUNCTION_TEST_RETURN(INT, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewInt64(const int64_t data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT64, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantInt64) { *this = (VariantInt64) { .pub = { .type = varTypeInt64, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantInt64)); } /**********************************************************************************************************************************/ FN_EXTERN int64_t varInt64(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeInt64); FUNCTION_TEST_RETURN(INT64, ((const VariantInt64 *)this)->pub.data); } FN_EXTERN int64_t varInt64Force(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); int64_t result = 0; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: result = (int64_t)varInt(this); break; case varTypeInt64: result = varInt64(this); break; case varTypeString: result = cvtZToInt64(strZ(varStr(this))); break; case varTypeUInt: result = varUInt(this); break; case varTypeUInt64: { const uint64_t resultTest = varUInt64(this); // If max number of unsigned 64-bit integer is greater than max 64-bit signed integer can hold, then error if (resultTest <= INT64_MAX) result = (int64_t)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %" PRIu64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeInt64]); } break; } default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeInt64]); } FUNCTION_TEST_RETURN(INT64, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewUInt(const unsigned int data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantUInt) { *this = (VariantUInt) { .pub = { .type = varTypeUInt, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantUInt)); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int varUInt(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeUInt); FUNCTION_TEST_RETURN(UINT, ((const VariantUInt *)this)->pub.data); } FN_EXTERN unsigned int varUIntForce(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); unsigned int result = 0; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: { const int resultTest = varInt(this); // If integer is a negative number, throw an error since the resulting conversion would be a different number if (resultTest >= 0) result = (unsigned int)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %d to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt]); } break; } case varTypeInt64: { const int64_t resultTest = varInt64(this); // If integer is a negative number or too large, throw an error since the resulting conversion would be out of bounds if (resultTest >= 0 && resultTest <= UINT_MAX) result = (unsigned int)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %" PRId64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt]); } break; } case varTypeUInt: result = varUInt(this); break; case varTypeUInt64: { const uint64_t resultTest = varUInt64(this); // If integer is too large, throw an error since the resulting conversion would be out of bounds if (resultTest <= UINT_MAX) result = (unsigned int)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %" PRIu64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt]); } break; } case varTypeString: result = cvtZToUInt(strZ(varStr(this))); break; default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeUInt]); } FUNCTION_TEST_RETURN(UINT, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewUInt64(const uint64_t data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantUInt64) { *this = (VariantUInt64) { .pub = { .type = varTypeUInt64, .data = data, }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantUInt64)); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t varUInt64(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(varType(this) == varTypeUInt64); FUNCTION_TEST_RETURN(UINT64, ((const VariantUInt64 *)this)->pub.data); } FN_EXTERN uint64_t varUInt64Force(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); uint64_t result = 0; switch (varType(this)) { case varTypeBool: result = varBool(this); break; case varTypeInt: { const int resultTest = varInt(this); // If integer is a negative number, throw an error since the resulting conversion would be a different number if (resultTest >= 0) result = (uint64_t)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %d to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt64]); } break; } case varTypeInt64: { const int64_t resultTest = varInt64(this); // If integer is a negative number, throw an error since the resulting conversion would be out of bounds if (resultTest >= 0) result = (uint64_t)resultTest; else { THROW_FMT( FormatError, "unable to convert %s %" PRId64 " to %s", variantTypeName[varType(this)], resultTest, variantTypeName[varTypeUInt64]); } break; } case varTypeString: result = cvtZToUInt64(strZ(varStr(this))); break; case varTypeUInt: result = varUInt(this); break; case varTypeUInt64: result = varUInt64(this); break; default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(AssertError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeUInt64]); } FUNCTION_TEST_RETURN(UINT64, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewKv(KeyValue *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(KEY_VALUE, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantKeyValue, .childQty = 1) { *this = (VariantKeyValue) { .type = varTypeKeyValue, }; if (data != NULL) this->data = kvMove(data, memContextCurrent()); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantKeyValue)); } /**********************************************************************************************************************************/ FN_EXTERN KeyValue * varKv(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); KeyValue *result = NULL; if (this != NULL) { ASSERT(varType(this) == varTypeKeyValue); result = ((const VariantKeyValue *)this)->data; } FUNCTION_TEST_RETURN(KEY_VALUE, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewStr(const String *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, data); FUNCTION_TEST_END(); // If the variant is larger than the extra allowed with a mem context then allocate the buffer separately const size_t allocExtra = sizeof(VariantString) + (data != NULL ? sizeof(StringPub) + strSize(data) + 1 : 0); if (allocExtra > MEM_CONTEXT_ALLOC_EXTRA_MAX) { ASSERT(data != NULL); OBJ_NEW_BEGIN(VariantString, .childQty = 1) { *this = (VariantString) { .pub = { .type = varTypeString, .data = strDup(data), }, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantString)); } OBJ_NEW_EXTRA_BEGIN(VariantString, (uint16_t)(allocExtra)) { *this = (VariantString) { .pub = { .type = varTypeString, }, }; if (data != NULL) { // Point the String after the VariantString struct StringPub *const pubData = (StringPub *)(this + 1); this->pub.data = (String *)pubData; // Assign the string size and buffer (after StringPub struct) *pubData = (StringPub) { .size = (unsigned int)strSize(data), .buffer = (char *)(pubData + 1), }; // Assign the string memcpy(pubData->buffer, strZ(data), strSize(data)); pubData->buffer[strSize(data)] = '\0'; } } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantString)); } FN_EXTERN Variant * varNewStrZ(const char *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, data); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(VARIANT, varNewStr(data == NULL ? NULL : STR(data))); } /**********************************************************************************************************************************/ FN_EXTERN const String * varStr(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); String *result = NULL; if (this != NULL) { ASSERT(varType(this) == varTypeString); result = ((const VariantString *)this)->pub.data; } FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN String * varStrForce(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result = NULL; switch (varType(this)) { case varTypeBool: result = strNewZ(cvtBoolToConstZ(varBool(this))); break; case varTypeInt: { char working[CVT_BASE10_BUFFER_SIZE]; cvtIntToZ(varInt(this), working, sizeof(working)); result = strNewZ(working); break; } case varTypeInt64: { char working[CVT_BASE10_BUFFER_SIZE]; cvtInt64ToZ(varInt64(this), working, sizeof(working)); result = strNewZ(working); break; } case varTypeString: result = strDup(varStr(this)); break; case varTypeUInt: { char working[CVT_BASE10_BUFFER_SIZE]; cvtUIntToZ(varUInt(this), working, sizeof(working)); result = strNewZ(working); break; } case varTypeUInt64: { char working[CVT_BASE10_BUFFER_SIZE]; cvtUInt64ToZ(varUInt64(this), working, sizeof(working)); result = strNewZ(working); break; } default: ASSERT(varType(this) == varTypeKeyValue || varType(this) == varTypeVariantList); THROW_FMT(FormatError, "unable to force %s to %s", variantTypeName[varType(this)], variantTypeName[varTypeString]); } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN Variant * varNewVarLst(const VariantList *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, data); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(VariantVariantList, .childQty = 1) { *this = (VariantVariantList) { .type = varTypeVariantList, }; if (data != NULL) this->data = varLstDup(data); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(VARIANT, (Variant *)OBJ_NAME(this, Variant::VariantVariantList)); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * varVarLst(const Variant *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, this); FUNCTION_TEST_END(); VariantList *result = NULL; if (this != NULL) { ASSERT(varType(this) == varTypeVariantList); result = ((const VariantVariantList *)this)->data; } FUNCTION_TEST_RETURN(VARIANT_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN void varToLog(const Variant *const this, StringStatic *const debugLog) { switch (varType(this)) { case varTypeString: strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT(varStr(this), strToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); break; case varTypeKeyValue: strStcCat(debugLog, "{KeyValue}"); break; case varTypeVariantList: strStcCat(debugLog, "{VariantList}"); break; case varTypeBool: strStcFmt(debugLog, "{%s}", cvtBoolToConstZ(varBool(this))); break; case varTypeInt: strStcFmt(debugLog, "{%d}", varInt(this)); break; case varTypeInt64: strStcFmt(debugLog, "{%" PRId64 "}", varInt64(this)); break; case varTypeUInt: strStcFmt(debugLog, "{%u}", varUInt(this)); break; case varTypeUInt64: strStcFmt(debugLog, "{%" PRIu64 "}", varUInt64(this)); break; } } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/variant.h0000664000175100017510000002656215240547630020303 0ustar00runnerrunner/*********************************************************************************************************************************** Variant Data Type Variants are lightweight objects in that they do not have their own memory context, instead they exist in the current context in which they are instantiated. If a variant is needed outside the current memory context, the memory context must be switched to the old context and then back. Below is a simplified example: Variant *result = NULL; <--- is created in the current memory context (referred to as "old context" below) MEM_CONTEXT_TEMP_BEGIN() <--- begins a new temporary context { String *resultStr = strNewZN("myNewStr"); <--- creates a string in the temporary memory context MEM_CONTEXT_PRIOR_BEGIN() <--- switch to old context so creation of the variant from the string is in old context { result = varNewUInt64(cvtZToUInt64(strZ(resultStr))); <--- recreates variant from the string in the old context. } MEM_CONTEXT_PRIOR_END(); <--- switch back to the temporary context } MEM_CONTEXT_TEMP_END(); <-- frees everything created inside this temporary memory context - i.e resultStr ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_VARIANT_H #define COMMON_TYPE_VARIANT_H #include /*********************************************************************************************************************************** Variant object ***********************************************************************************************************************************/ typedef struct Variant Variant; /*********************************************************************************************************************************** Variant type ***********************************************************************************************************************************/ typedef enum { varTypeBool, varTypeInt, varTypeInt64, varTypeKeyValue, varTypeString, varTypeUInt, varTypeUInt64, varTypeVariantList, } VariantType; #include "common/type/keyValue.h" #include "common/type/string.h" #include "common/type/variantList.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Variant *varNewBool(bool data); FN_EXTERN Variant *varNewInt(int data); FN_EXTERN Variant *varNewInt64(int64_t data); // Note that the KeyValue is not duplicated because it this a heavy-weight operation. It is merely moved into the same MemContext as // the Variant. FN_EXTERN Variant *varNewKv(KeyValue *data); FN_EXTERN Variant *varNewStr(const String *data); FN_EXTERN Variant *varNewStrZ(const char *data); FN_EXTERN Variant *varNewUInt(unsigned int data); FN_EXTERN Variant *varNewUInt64(uint64_t data); FN_EXTERN Variant *varNewVarLst(const VariantList *data); FN_EXTERN Variant *varDup(const Variant *this); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ #define VARIANT_COMMON \ VariantType type; /* Variant type */ typedef struct VariantPub { VARIANT_COMMON } VariantPub; typedef struct VariantBoolPub { VARIANT_COMMON bool data; // Boolean data } VariantBoolPub; typedef struct VariantIntPub { VARIANT_COMMON int data; // Signed integer data } VariantIntPub; typedef struct VariantInt64Pub { VARIANT_COMMON int64_t data; // 64-bit signed integer data } VariantInt64Pub; typedef struct VariantStringPub { VARIANT_COMMON String *data; // String data } VariantStringPub; typedef struct VariantUIntPub { VARIANT_COMMON unsigned int data; // Unsigned integer data } VariantUIntPub; typedef struct VariantUInt64Pub { VARIANT_COMMON uint64_t data; // 64-bit unsigned integer data } VariantUInt64Pub; FN_EXTERN bool varBool(const Variant *this); FN_EXTERN bool varBoolForce(const Variant *this); FN_EXTERN int varInt(const Variant *this); FN_EXTERN int varIntForce(const Variant *this); FN_EXTERN int64_t varInt64(const Variant *this); FN_EXTERN int64_t varInt64Force(const Variant *this); FN_EXTERN KeyValue *varKv(const Variant *this); FN_EXTERN const String *varStr(const Variant *this); FN_EXTERN String *varStrForce(const Variant *this); FN_EXTERN unsigned int varUInt(const Variant *this); FN_EXTERN unsigned int varUIntForce(const Variant *this); FN_EXTERN uint64_t varUInt64(const Variant *this); FN_EXTERN uint64_t varUInt64Force(const Variant *this); FN_EXTERN VariantList *varVarLst(const Variant *this); // Variant type FN_INLINE_ALWAYS VariantType varType(const Variant *const this) { return THIS_PUB(Variant)->type; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Test if Variants are equal FN_EXTERN bool varEq(const Variant *this1, const Variant *this2); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void varFree(Variant *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for constant variants Frequently used constant variants can be declared with these macros at compile time rather than dynamically at run time. Note that variants created in this way are declared as const so can't be modified or freed by the var*() methods. Casting to Variant * will generally result in a segfault. By convention all variant constant identifiers are appended with _VAR. ***********************************************************************************************************************************/ // Create a Bool Variant constant inline from a bool #define VARBOOL(dataParam) \ ((const Variant *)&(const VariantBoolPub){.type = varTypeBool, .data = dataParam}) // Create an Int Variant constant inline from an int #define VARINT(dataParam) \ ((const Variant *)&(const VariantIntPub){.type = varTypeInt, .data = dataParam}) // Create an Int64 Variant constant inline from an int64_t #define VARINT64(dataParam) \ ((const Variant *)&(const VariantInt64Pub){.type = varTypeInt64, .data = dataParam}) // Create a String Variant constant inline from any zero-terminated string. The struct must be kept in sync with VariantStringPub // (except for const qualifiers) typedef struct VariantStringPubConst { VARIANT_COMMON const String *data; // String data } VariantStringPubConst; #define VARSTRZ(dataParam) \ ((const Variant *)&(const VariantStringPubConst){.type = varTypeString, .data = (const String *)STR(dataParam)}) // Create a String Variant constant inline from a #define or inline string constant #define VARSTRDEF(dataParam) \ ((const Variant *)&(const VariantStringPubConst){.type = varTypeString, .data = (const String *)STRDEF(dataParam)}) // Create a String Variant constant inline from a String constant #define VARSTR(dataParam) \ ((const Variant *)&(const VariantStringPubConst){.type = varTypeString, .data = (const String *)(dataParam)}) // Used to define String Variant constants that will be externed using VARIANT_DECLARE(). Must be used in a .c file. #define VARIANT_STRDEF_EXTERN(name, dataParam) \ VR_EXTERN_DEFINE const Variant *const name = VARSTRDEF(dataParam) // Used to define String Variant constants that will be local to the .c file. Must be used in a .c file. #define VARIANT_STRDEF_STATIC(name, dataParam) \ static const Variant *const name = VARSTRDEF(dataParam) // Create a UInt Variant constant inline from an unsigned int #define VARUINT(dataParam) \ ((const Variant *)&(const VariantUIntPub){.type = varTypeUInt, .data = dataParam}) // Create a UInt64 Variant constant inline from a uint64_t #define VARUINT64(dataParam) \ ((const Variant *)&(const VariantUInt64Pub){.type = varTypeUInt64, .data = dataParam}) // Used to declare externed String Variant constants defined with VARIANT*EXTERN(). Must be used in a .h file. #define VARIANT_DECLARE(name) \ VR_EXTERN_DECLARE const Variant *const name /*********************************************************************************************************************************** Constant variants that are generally useful ***********************************************************************************************************************************/ VARIANT_DECLARE(BOOL_FALSE_VAR); VARIANT_DECLARE(BOOL_TRUE_VAR); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void varToLog(const Variant *this, StringStatic *debugLog); #define FUNCTION_LOG_VARIANT_TYPE \ Variant * #define FUNCTION_LOG_VARIANT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, varToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/variantList.c0000664000175100017510000000355715240547630021131 0ustar00runnerrunner/*********************************************************************************************************************************** Variant List Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/type/list.h" #include "common/type/variantList.h" /**********************************************************************************************************************************/ FN_EXTERN VariantList * varLstNewStrLst(const StringList *const stringList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_LIST, stringList); FUNCTION_TEST_END(); VariantList *this = NULL; if (stringList != NULL) { this = varLstNew(); MEM_CONTEXT_OBJ_BEGIN(this) { for (unsigned int listIdx = 0; listIdx < strLstSize(stringList); listIdx++) varLstAdd(this, varNewStr(strLstGet(stringList, listIdx))); } MEM_CONTEXT_OBJ_END(); } FUNCTION_TEST_RETURN(VARIANT_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * varLstDup(const VariantList *const source) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT_LIST, source); FUNCTION_TEST_END(); VariantList *this = NULL; if (source != NULL) { this = varLstNew(); MEM_CONTEXT_OBJ_BEGIN(this) { for (unsigned int listIdx = 0; listIdx < varLstSize(source); listIdx++) varLstAdd(this, varDup(varLstGet(source, listIdx))); } MEM_CONTEXT_OBJ_END(); } FUNCTION_TEST_RETURN(VARIANT_LIST, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/variantList.h0000664000175100017510000000724615240547630021135 0ustar00runnerrunner/*********************************************************************************************************************************** Variant List Handler ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_VARIANTLIST_H #define COMMON_TYPE_VARIANTLIST_H /*********************************************************************************************************************************** Variant list object ***********************************************************************************************************************************/ typedef struct VariantList VariantList; #include "common/type/stringList.h" #include "common/type/variant.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Create empty VariantList FN_INLINE_ALWAYS VariantList * varLstNew(void) { return (VariantList *)OBJ_NAME(lstNewP(sizeof(Variant *)), VariantList::List); } // Create VariantList from StringList FN_EXTERN VariantList *varLstNewStrLst(const StringList *stringList); // Duplicate a variant list FN_EXTERN VariantList *varLstDup(const VariantList *source); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // List size FN_INLINE_ALWAYS unsigned int varLstSize(const VariantList *const this) { return lstSize((const List *)this); } // Is the list empty? FN_INLINE_ALWAYS bool varLstEmpty(const VariantList *const this) { return varLstSize(this) == 0; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add to list FN_INLINE_ALWAYS VariantList * varLstAdd(VariantList *const this, Variant *const data) { lstAdd((List *)this, &data); return this; } // Get by index FN_INLINE_ALWAYS Variant * varLstGet(const VariantList *const this, const unsigned int listIdx) { return *(Variant **)lstGet((const List *)this, listIdx); } // Move to new parent mem context FN_INLINE_ALWAYS VariantList * varLstMove(VariantList *const this, MemContext *const parentNew) { return (VariantList *)lstMove((List *)this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void varLstFree(VariantList *const this) { lstFree((List *)this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_VARIANT_LIST_TYPE \ VariantList * #define FUNCTION_LOG_VARIANT_LIST_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "VariantList", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/xml.c0000664000175100017510000003013315240547630017417 0ustar00runnerrunner/*********************************************************************************************************************************** Xml Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/xml.h" /*********************************************************************************************************************************** Encoding constants ***********************************************************************************************************************************/ #define XML_ENCODING_TYPE_UTF8 "UTF-8" /*********************************************************************************************************************************** Node type ***********************************************************************************************************************************/ struct XmlNode { xmlNodePtr node; }; /*********************************************************************************************************************************** Document type ***********************************************************************************************************************************/ struct XmlDocument { XmlDocumentPub pub; // Publicly accessible variables xmlDocPtr xml; }; /*********************************************************************************************************************************** Error handler For now this is a noop until more detailed error messages are needed. The function is called multiple times per error, so the messages need to be accumulated and then returned together. This empty function is required because without it libxml2 will dump errors to stdout. Really. ***********************************************************************************************************************************/ static void xmlErrorHandler(void *const ctx, const char *const format, ...) { (void)ctx; (void)format; } /*********************************************************************************************************************************** Initialize xml ***********************************************************************************************************************************/ static void xmlInit(void) { FUNCTION_TEST_VOID(); // Initialize xml if it is not already initialized static bool xmlInit = false; if (!xmlInit) { LIBXML_TEST_VERSION; // Set error handler xmlSetGenericErrorFunc(NULL, xmlErrorHandler); xmlInit = true; } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ static XmlNodeList * xmlNodeLstNew(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(XML_NODE_LIST, (XmlNodeList *)OBJ_NAME(lstNewP(sizeof(XmlNode *)), XmlNodeList::List)); } /**********************************************************************************************************************************/ static XmlNode * xmlNodeNew(xmlNodePtr const node) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, node); FUNCTION_TEST_END(); ASSERT(node != NULL); OBJ_NEW_BEGIN(XmlNode) { *this = (XmlNode) { .node = node, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(XML_NODE, this); } /**********************************************************************************************************************************/ FN_EXTERN XmlNode * xmlNodeAdd(XmlNode *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); XmlNode *const result = xmlNodeNew(xmlNewNode(NULL, (const xmlChar *)strZ(name))); xmlAddChild(this->node, result->node); FUNCTION_TEST_RETURN(XML_NODE, result); } /*********************************************************************************************************************************** Add a node to a node list ***********************************************************************************************************************************/ static XmlNodeList * xmlNodeLstAdd(XmlNodeList *const this, xmlNodePtr const node) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE_LIST, this); FUNCTION_TEST_PARAM_P(VOID, node); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(node != NULL); MEM_CONTEXT_BEGIN(lstMemContext((List *)this)) { XmlNode *const item = xmlNodeNew(node); lstAdd((List *)this, &item); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(XML_NODE_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN String * xmlNodeContent(const XmlNode *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_END(); String *result = NULL; if (this != NULL) { xmlChar *const content = xmlNodeGetContent(this->node); result = strNewZ((char *)content); xmlFree(content); } FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN void xmlNodeContentSet(XmlNode *const this, const String *const content) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING, content); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(content != NULL); xmlAddChild(this->node, xmlNewText((const xmlChar *)strZ(content))); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * xmlNodeName(const XmlNode *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(STRING, strNewZ((const char *)this->node->name)); } /**********************************************************************************************************************************/ FN_EXTERN XmlNodeList * xmlNodeChildList(const XmlNode *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); XmlNodeList *const list = xmlNodeLstNew(); for (xmlNodePtr currentNode = this->node->children; currentNode != NULL; currentNode = currentNode->next) { if (currentNode->type == XML_ELEMENT_NODE && strEqZ(name, (const char *)currentNode->name)) xmlNodeLstAdd(list, currentNode); } FUNCTION_TEST_RETURN(XML_NODE_LIST, list); } /**********************************************************************************************************************************/ FN_EXTERN XmlNodeList * xmlNodeChildListMulti(const XmlNode *const this, const StringList *const nameList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING_LIST, nameList); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(nameList != NULL); XmlNodeList *const list = xmlNodeLstNew(); for (xmlNodePtr currentNode = this->node->children; currentNode != NULL; currentNode = currentNode->next) { if (currentNode->type == XML_ELEMENT_NODE && strLstExists(nameList, STR((const char *)currentNode->name))) xmlNodeLstAdd(list, currentNode); } FUNCTION_TEST_RETURN(XML_NODE_LIST, list); } /**********************************************************************************************************************************/ FN_EXTERN XmlNode * xmlNodeChildN(const XmlNode *const this, const String *const name, const unsigned int index, const bool errorOnMissing) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_NODE, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_PARAM(UINT, index); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); XmlNode *child = NULL; unsigned int childIdx = 0; for (xmlNodePtr currentNode = this->node->children; currentNode != NULL; currentNode = currentNode->next) { if (currentNode->type == XML_ELEMENT_NODE && strEqZ(name, (const char *)currentNode->name)) { if (childIdx == index) { child = xmlNodeNew(currentNode); break; } childIdx++; } } if (child == NULL && errorOnMissing) THROW_FMT(FormatError, "unable to find child '%s':%u in node '%s'", strZ(name), index, this->node->name); FUNCTION_TEST_RETURN(XML_NODE, child); } /*********************************************************************************************************************************** Free document ***********************************************************************************************************************************/ static void xmlDocumentFreeResource(THIS_VOID) { THIS(XmlDocument); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(XML_DOCUMENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); xmlFreeDoc(this->xml); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN XmlDocument * xmlDocumentNew(const String *const rootName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, rootName); FUNCTION_TEST_END(); ASSERT(rootName != NULL); xmlInit(); OBJ_NEW_BEGIN(XmlDocument, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (XmlDocument) { .xml = xmlNewDoc((const xmlChar *)"1.0"), }; // Set callback to ensure xml document is freed memContextCallbackSet(objMemContext(this), xmlDocumentFreeResource, this); this->pub.root = xmlNodeNew(xmlNewNode(NULL, (const xmlChar *)strZ(rootName))); xmlDocSetRootElement(this->xml, xmlDocumentRoot(this)->node); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(XML_DOCUMENT, this); } /**********************************************************************************************************************************/ FN_EXTERN XmlDocument * xmlDocumentNewBuf(const Buffer *const buffer) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, buffer); FUNCTION_TEST_END(); ASSERT(buffer != NULL); ASSERT(!bufEmpty(buffer)); xmlInit(); OBJ_NEW_BEGIN(XmlDocument, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (XmlDocument){{0}}; // Extra braces are required for older gcc versions if ((this->xml = xmlReadMemory((const char *)bufPtrConst(buffer), (int)bufUsed(buffer), "noname.xml", NULL, 0)) == NULL) THROW_FMT(FormatError, "invalid xml in %zu byte(s):\n%s", bufUsed(buffer), strZ(strNewBuf(buffer))); // Set callback to ensure xml document is freed memContextCallbackSet(objMemContext(this), xmlDocumentFreeResource, this); // Get the root node this->pub.root = xmlNodeNew(xmlDocGetRootElement(this->xml)); } OBJ_NEW_END(); FUNCTION_TEST_RETURN(XML_DOCUMENT, this); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * xmlDocumentBuf(const XmlDocument *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(XML_DOCUMENT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); xmlChar *xml; int xmlSize; xmlDocDumpMemoryEnc(this->xml, &xml, &xmlSize, XML_ENCODING_TYPE_UTF8); Buffer *result = bufNewC(xml, (size_t)xmlSize); xmlFree(xml); FUNCTION_TEST_RETURN(BUFFER, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/type/xml.h0000664000175100017510000001422415240547630017427 0ustar00runnerrunner/*********************************************************************************************************************************** Xml Handler A thin wrapper around the libxml2 library. There are many capabilities of libxml2 that are not exposed here and may need to be added to when implementing new features. ***********************************************************************************************************************************/ #ifndef COMMON_TYPE_XML_H #define COMMON_TYPE_XML_H /*********************************************************************************************************************************** Objects ***********************************************************************************************************************************/ typedef struct XmlDocument XmlDocument; typedef struct XmlNode XmlNode; typedef struct XmlNodeList XmlNodeList; #include "common/memContext.h" #include "common/type/list.h" #include "common/type/object.h" #include "common/type/stringList.h" /*********************************************************************************************************************************** Document Constructors ***********************************************************************************************************************************/ // Document with the specified root node FN_EXTERN XmlDocument *xmlDocumentNew(const String *rootNode); // Document from Buffer FN_EXTERN XmlDocument *xmlDocumentNewBuf(const Buffer *); /*********************************************************************************************************************************** Document Getters ***********************************************************************************************************************************/ typedef struct XmlDocumentPub { XmlNode *root; // Root node } XmlDocumentPub; // Dump document to a buffer FN_EXTERN Buffer *xmlDocumentBuf(const XmlDocument *this); // Root node FN_INLINE_ALWAYS XmlNode * xmlDocumentRoot(const XmlDocument *const this) { return ((const XmlDocumentPub *const)this)->root; } /*********************************************************************************************************************************** Document Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void xmlDocumentFree(XmlDocument *const this) { objFree(this); } /*********************************************************************************************************************************** Node Functions ***********************************************************************************************************************************/ // Add a node FN_EXTERN XmlNode *xmlNodeAdd(XmlNode *this, const String *name); /*********************************************************************************************************************************** Node Getters/Setters ***********************************************************************************************************************************/ // Node child (by name or index) FN_EXTERN XmlNode *xmlNodeChildN(const XmlNode *this, const String *name, unsigned int index, bool errorOnMissing); FN_INLINE_ALWAYS XmlNode * xmlNodeChild(const XmlNode *const this, const String *const name, const bool errorOnMissing) { return xmlNodeChildN(this, name, 0, errorOnMissing); } // List of child nodes FN_EXTERN XmlNodeList *xmlNodeChildList(const XmlNode *this, const String *name); // List of child nodes using multiple names FN_EXTERN XmlNodeList *xmlNodeChildListMulti(const XmlNode *this, const StringList *nameList); // Node content FN_EXTERN String *xmlNodeContent(const XmlNode *this); FN_EXTERN void xmlNodeContentSet(XmlNode *this, const String *content); // Node name FN_EXTERN String *xmlNodeName(const XmlNode *this); /*********************************************************************************************************************************** Node List Getters ***********************************************************************************************************************************/ // Get a node from the list FN_INLINE_ALWAYS XmlNode * xmlNodeLstGet(const XmlNodeList *const this, const unsigned int listIdx) { return *(XmlNode **)lstGet((const List *const)this, listIdx); } // Node list size FN_INLINE_ALWAYS unsigned int xmlNodeLstSize(const XmlNodeList *const this) { return lstSize((const List *const)this); } /*********************************************************************************************************************************** Node List Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void xmlNodeLstFree(XmlNodeList *const this) { lstFree((List *const)this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_XML_DOCUMENT_TYPE \ XmlDocument * #define FUNCTION_LOG_XML_DOCUMENT_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "XmlDocument", buffer, bufferSize) #define FUNCTION_LOG_XML_NODE_TYPE \ XmlNode * #define FUNCTION_LOG_XML_NODE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "XmlNode", buffer, bufferSize) #define FUNCTION_LOG_XML_NODE_LIST_TYPE \ XmlNodeList * #define FUNCTION_LOG_XML_NODE_LIST_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "XmlNodeList", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/user.c0000664000175100017510000002552215240547630016622 0ustar00runnerrunner/*********************************************************************************************************************************** System User/Group Management ***********************************************************************************************************************************/ #include #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "common/user.h" /*********************************************************************************************************************************** Per-process uid->name and gid->name caches. On systems where uid/gid lookups are routed to a remote name service (sssd, systemd-userdbd, LDAP via nss-pam-ldapd, etc.), every call costs a Unix-socket round-trip plus its own audit-and-syscall overhead. Profiling shows this can dominate the manifest-build phase for clusters with millions of files even though the data files almost always share a single owner (typically "postgres"). The cache is a small fixed-size array because the working set of unique ids on a PG host is tiny (usually 1-3). Linear scan is faster than a hash table at this size and avoids any allocation in the hot path. The cached String lives in memContextTop() so it survives the caller's context. Cache misses past USER_NAME_CACHE_SIZE fall through to an uncached lookup (correct but slow); in practice we never exceed the table for a backup of a normally-configured cluster. ***********************************************************************************************************************************/ #define USER_NAME_CACHE_SIZE 16 typedef struct UserNameCache { uid_t id; // The uid that was looked up const String *name; // User name returned from lookup, NULL if unresolvable } UserNameCache; typedef struct GroupNameCache { gid_t id; // The gid that was looked up const String *name; // Group name returned from lookup, NULL if unresolvable } GroupNameCache; /*********************************************************************************************************************************** User group info ***********************************************************************************************************************************/ static struct { MemContext *memContext; // Mem context to store data in this struct uid_t userId; // Real user id of the calling process from getuid() bool userRoot; // Is this the root user? const String *userName; // User name if it exists UserNameCache userNameCache[USER_NAME_CACHE_SIZE]; // User name cache unsigned int userNameCacheTotal; // User cache total gid_t groupId; // Real group id of the calling process from getgid() const String *groupName; // Group name if it exists GroupNameCache groupNameCache[USER_NAME_CACHE_SIZE]; // Group name cache unsigned int groupNameCacheTotal; // Group name cache total #ifdef HAVE_LIBSSH2 const String *userHome; // User home directory #endif // HAVE_LIBSSH2 } userLocalData; /**********************************************************************************************************************************/ static void userInitInternal(void) { FUNCTION_TEST_VOID(); MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(UserLocalData, .childQty = MEM_CONTEXT_QTY_MAX) { userLocalData.memContext = MEM_CONTEXT_NEW(); userLocalData.userId = getuid(); userLocalData.userName = userNameFromId(userLocalData.userId); #ifdef HAVE_LIBSSH2 userLocalData.userHome = userHomeFromId(userLocalData.userId); #endif // HAVE_LIBSSH2 userLocalData.userRoot = userLocalData.userId == 0; userLocalData.groupId = getgid(); userLocalData.groupName = groupNameFromId(userLocalData.groupId); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void userInit(void) { FUNCTION_TEST_VOID(); if (!userLocalData.memContext) userInitInternal(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN gid_t groupId(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_TYPE(gid_t, userLocalData.groupId); } /**********************************************************************************************************************************/ FN_EXTERN gid_t groupIdFromName(const String *const groupName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, groupName); FUNCTION_TEST_END(); if (groupName != NULL) { const struct group *const groupData = getgrnam(strZ(groupName)); if (groupData != NULL) FUNCTION_TEST_RETURN_TYPE(gid_t, groupData->gr_gid); } FUNCTION_TEST_RETURN_TYPE(gid_t, (gid_t)-1); } /**********************************************************************************************************************************/ FN_EXTERN const String * groupName(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STRING, userLocalData.groupName); } /**********************************************************************************************************************************/ FN_EXTERN String * groupNameFromId(const gid_t groupId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, groupId); FUNCTION_TEST_END(); // Cache lookup on a small fixed-size table -- linear scan is fastest at this size for (unsigned int groupIdx = 0; groupIdx < userLocalData.groupNameCacheTotal; groupIdx++) { if (userLocalData.groupNameCache[groupIdx].id == groupId) FUNCTION_TEST_RETURN(STRING, strDup(userLocalData.groupNameCache[groupIdx].name)); } // Cache miss -- do the actual lookup const struct group *const groupData = getgrgid(groupId); String *const result = groupData != NULL ? strNewZ(groupData->gr_name) : NULL; // Record the answer (including the negative case) so we never look this id up again. Past USER_NAME_CACHE_SIZE we just fall // back to per-call lookup, which is correct -- only slower for the unusual case of many distinct owners. if (userLocalData.groupNameCacheTotal < USER_NAME_CACHE_SIZE) { MEM_CONTEXT_BEGIN(memContextTop()) { userLocalData.groupNameCache[userLocalData.groupNameCacheTotal] = (GroupNameCache) { .id = groupId, .name = strDup(result), }; } MEM_CONTEXT_END(); userLocalData.groupNameCacheTotal++; } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ // Currently userHome() and userHomeFromId() are only used if we are building with libssh2 #ifdef HAVE_LIBSSH2 FN_EXTERN const String * userHome(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STRING, userLocalData.userHome); } /**********************************************************************************************************************************/ FN_EXTERN String * userHomeFromId(const uid_t userId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, userId); FUNCTION_TEST_END(); const struct passwd *const userData = getpwuid(userId); if (userData != NULL) FUNCTION_TEST_RETURN(STRING, strNewZ(userData->pw_dir)); FUNCTION_TEST_RETURN(STRING, NULL); } #endif // HAVE_LIBSSH2 /**********************************************************************************************************************************/ FN_EXTERN uid_t userId(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_TYPE(uid_t, userLocalData.userId); } /**********************************************************************************************************************************/ FN_EXTERN uid_t userIdFromName(const String *const userName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, userName); FUNCTION_TEST_END(); if (userName != NULL) { const struct passwd *const userData = getpwnam(strZ(userName)); if (userData != NULL) FUNCTION_TEST_RETURN_TYPE(uid_t, userData->pw_uid); } FUNCTION_TEST_RETURN_TYPE(uid_t, (uid_t)-1); } /**********************************************************************************************************************************/ FN_EXTERN const String * userName(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STRING, userLocalData.userName); } /**********************************************************************************************************************************/ FN_EXTERN String * userNameFromId(const uid_t userId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, userId); FUNCTION_TEST_END(); // Cache lookup on a small fixed-size table -- linear scan is fastest at this size for (unsigned int userIdx = 0; userIdx < userLocalData.userNameCacheTotal; userIdx++) { if (userLocalData.userNameCache[userIdx].id == userId) FUNCTION_TEST_RETURN(STRING, strDup(userLocalData.userNameCache[userIdx].name)); } // Cache miss -- do the actual lookup const struct passwd *const userData = getpwuid(userId); String *const result = userData != NULL ? strNewZ(userData->pw_name) : NULL; // Record the answer (including the negative case) so we never look this id up again. Past USER_NAME_CACHE_SIZE we just fall // back to per-call lookup, which is correct -- only slower for the unusual case of many distinct owners. if (userLocalData.userNameCacheTotal < USER_NAME_CACHE_SIZE) { MEM_CONTEXT_BEGIN(memContextTop()) { userLocalData.userNameCache[userLocalData.userNameCacheTotal] = (UserNameCache) { .id = userId, .name = strDup(result), }; } MEM_CONTEXT_END(); userLocalData.userNameCacheTotal++; } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN bool userRoot(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(BOOL, userLocalData.userRoot); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/user.h0000664000175100017510000000415215240547630016623 0ustar00runnerrunner/*********************************************************************************************************************************** System User/Group Management ***********************************************************************************************************************************/ #ifndef COMMON_USER_H #define COMMON_USER_H #include #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Call this initializer before using any of the functions below. Safe to call more than once. FN_EXTERN void userInit(void); // Get the primary group id of the current user FN_EXTERN gid_t groupId(void); // Get the id of the specified group. Returns (gid_t)-1 if not found. FN_EXTERN gid_t groupIdFromName(const String *groupName); // Get the primary group name of the current user. Returns NULL if there is no mapping. FN_EXTERN const String *groupName(void); // Get the group name from a group id. Returns NULL if the group id is invalid or there is no mapping. FN_EXTERN String *groupNameFromId(gid_t groupId); #ifdef HAVE_LIBSSH2 // Get the home directory of the current user. Returns NULL if there is no mapping. FN_EXTERN const String *userHome(void); // Get the user home directory from a user id. Returns NULL if the user id is invalid or there is no mapping. FN_EXTERN String *userHomeFromId(uid_t userId); #endif // HAVE_LIBSSH2 // Get the id of the current user FN_EXTERN uid_t userId(void); // Get the id of the specified user. Returns (uid_t)-1 if not found. FN_EXTERN uid_t userIdFromName(const String *userName); // Get the name of the current user. Returns NULL if there is no mapping. FN_EXTERN const String *userName(void); // Get the user name from a user id. Returns NULL if the user id is invalid or there is no mapping. FN_EXTERN String *userNameFromId(uid_t userId); // Is the current user the root user? FN_EXTERN bool userRoot(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/wait.c0000664000175100017510000001024215240547630016601 0ustar00runnerrunner/*********************************************************************************************************************************** Wait Handler ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/wait.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Wait { TimeMSec waitTime; // Total time to wait (in usec) TimeMSec sleepTime; // Next sleep time (in usec) TimeMSec sleepPrevTime; // Previous time slept (in usec) TimeMSec beginTime; // Time the wait began (in epoch usec) unsigned int retry; // Retries remaining }; /**********************************************************************************************************************************/ FN_EXTERN Wait * waitNew(const TimeMSec waitTime) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(TIMEMSEC, waitTime); FUNCTION_LOG_END(); ASSERT(waitTime <= 999999000); OBJ_NEW_BEGIN(Wait, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Wait) { .waitTime = waitTime, .retry = 2, }; // Calculate first sleep time -- start with 1/10th of a second for anything >= 1 second if (this->waitTime >= MSEC_PER_SEC) { this->sleepTime = MSEC_PER_SEC / 10; } // Unless the wait time is really small -- in that case divide wait time by 10 else this->sleepTime = this->waitTime / 10; // Get beginning time this->beginTime = timeMSec(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(WAIT, this); } /**********************************************************************************************************************************/ FN_EXTERN TimeMSec waitRemains(Wait *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(WAIT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); TimeMSec result = 0; // If sleep is 0 then the wait time has already ended if (this->sleepTime > 0) { // Get the elapsed time const TimeMSec elapsedTime = timeMSec() - this->beginTime; // Is there more time to go? if (elapsedTime < this->waitTime) { // Calculate remaining time result = this->waitTime - elapsedTime; // Calculate sleep time as a sum of current and last (a Fibonacci-type sequence) TimeMSec sleepTime = this->sleepTime + this->sleepPrevTime; // Make sure sleep time does not go beyond remaining time (this won't be negative because of the if condition above) if (sleepTime > result) sleepTime = result; // Store new sleep times this->sleepPrevTime = this->sleepTime; this->sleepTime = sleepTime; } // Else are there retries left? else if (this->retry != 0) { result = this->sleepTime; } // Else set sleep to zero else this->sleepTime = 0; // Decrement retries if (this->retry > 0) this->retry--; } FUNCTION_LOG_RETURN(TIME_MSEC, result); } /**********************************************************************************************************************************/ FN_EXTERN bool waitMore(Wait *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(WAIT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = false; // If time remains in the wait then sleep if (waitRemains(this) > 0) { sleepMSec(this->sleepTime); result = true; } FUNCTION_LOG_RETURN(BOOL, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/common/wait.h0000664000175100017510000000552715240547630016620 0ustar00runnerrunner/*********************************************************************************************************************************** Wait Handler Used for operations that may fail due to an error or some unexpected condition such as file missing. When waitMore() is called it will wait (based on a Fibonacci backoff) before returning to give the error or condition time to clear. Even when the wait time has expired before waitMore() is called, there will still be two retries to compensate for operations that use up the entire time limit. Any number of retries are allowed within the time limit. ***********************************************************************************************************************************/ #ifndef COMMON_WAIT_H #define COMMON_WAIT_H /*********************************************************************************************************************************** Wait object ***********************************************************************************************************************************/ typedef struct Wait Wait; #include "common/time.h" #include "common/type/object.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Wait *waitNew(TimeMSec waitTime); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Return the remaining time left FN_EXTERN TimeMSec waitRemains(Wait *this); // Wait and return true if the caller has more time/retries left FN_EXTERN bool waitMore(Wait *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void waitFree(Wait *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_WAIT_TYPE \ Wait * #define FUNCTION_LOG_WAIT_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Wait", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/0000775000175100017510000000000015240547630015447 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/common.c0000664000175100017510000001276415240547630017115 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Common ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/regExp.h" #include "common/time.h" #include "config/common.h" /**********************************************************************************************************************************/ // Helper to get the multiplier based on the qualifier static int64_t cfgParseSizeQualifier(const char qualifier) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(CHAR, qualifier); FUNCTION_TEST_END(); int64_t result; switch (qualifier) { case 'b': result = 1; break; case 'k': result = 1024; break; case 'm': result = 1024 * 1024; break; case 'g': result = 1024 * 1024 * 1024; break; case 't': result = 1024LL * 1024LL * 1024LL * 1024LL; break; case 'p': result = 1024LL * 1024LL * 1024LL * 1024LL * 1024LL; break; default: THROW_FMT(AssertError, "'%c' is not a valid size qualifier", qualifier); } FUNCTION_TEST_RETURN(INT64, result); } FN_EXTERN int64_t cfgParseSize(const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(value != NULL); // Lowercase the value String *const valueLower = strLower(strDup(value)); // Match the value against possible values if (regExpMatchOne(STRDEF("^[0-9]+(kib|kb|k|mib|mb|m|gib|gb|g|tib|tb|t|pib|pb|p|b)*$"), valueLower)) { // Get the character array and size const char *const strArray = strZ(valueLower); const size_t size = strSize(valueLower); int chrPos = -1; // If there is a 'b' on the end, then see if the previous character is a number if (strArray[size - 1] == 'b') { // If the previous character is a number, then the letter to look at is 'b' which is the last position else it is in the // next to last position (e.g. kb - so the 'k' is the position of interest). Only need to test for <= 9 since the regex // enforces the format. Also allow an 'i' before the 'b'. if (strArray[size - 2] <= '9') chrPos = (int)(size - 1); else if (strArray[size - 2] == 'i') chrPos = (int)(size - 3); else chrPos = (int)(size - 2); } // Else if there is no 'b' at the end but the last position is not a number then it must be one of the letters, e.g. 'k' else if (strArray[size - 1] > '9') chrPos = (int)(size - 1); int64_t multiplier = 1; // If a letter was found calculate multiplier, else do nothing since assumed value is already in bytes if (chrPos != -1) { multiplier = cfgParseSizeQualifier(strArray[chrPos]); // Remove any letters strTruncIdx(valueLower, chrPos); } // Convert string to bytes const int64_t valueInt = cvtZToInt64(strZ(valueLower)); if (valueInt > INT64_MAX / multiplier) THROW_FMT(FormatError, "value '%s' is out of range", strZ(value)); strFree(valueLower); FUNCTION_TEST_RETURN(INT64, valueInt * multiplier); } THROW_FMT(FormatError, "value '%s' is not valid", strZ(value)); } /**********************************************************************************************************************************/ FN_EXTERN int64_t cfgParseTime(const String *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, value); FUNCTION_TEST_END(); ASSERT(value != NULL); // Get value ptr and size const char *const valuePtr = strZ(value); const size_t size = strSize(value); size_t qualifierSize = 0; int64_t multiplier = 1000; // Check if this is ms (the only two character qualifier) if (size > 1 && tolower(valuePtr[size - 2]) == 'm' && tolower(valuePtr[size - 1]) == 's') { qualifierSize = 2; multiplier = 1; } // Else check for single character qualifier else if (size > 0) { switch (tolower(valuePtr[size - 1])) { case 's': qualifierSize = 1; multiplier = 1000; break; case 'm': qualifierSize = 1; multiplier = 60 * 1000; break; case 'h': qualifierSize = 1; multiplier = 60 * 60 * 1000; break; case 'd': qualifierSize = 1; multiplier = 24 * 60 * 60 * 1000; break; case 'w': qualifierSize = 1; multiplier = 7 * 24 * 60 * 60 * 1000; break; } } // Only proceed if there are numbers to parse if (size - qualifierSize > 0) { // Convert string to time const int64_t valueInt = cvtZSubNToInt64Base(valuePtr, 0, size - qualifierSize, 10); if (valueInt <= INT64_MAX / multiplier) FUNCTION_TEST_RETURN(INT64, valueInt * multiplier); } THROW_FMT(FormatError, "value '%s' is not valid", strZ(value)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/common.h0000664000175100017510000000146615240547630017117 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Common ***********************************************************************************************************************************/ #ifndef CONFIG_COMMON_H #define CONFIG_COMMON_H #include "common/type/string.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Parse option size, e.g. 23m FN_EXTERN int64_t cfgParseSize(const String *value); // Parse option time, e.g. 900 FN_EXTERN int64_t cfgParseTime(const String *value); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/config.auto.h0000664000175100017510000011557115240547630020046 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Configuration Automatically generated by 'build-code config' -- do not modify directly. ***********************************************************************************************************************************/ #ifndef CONFIG_CONFIG_AUTO_H #define CONFIG_CONFIG_AUTO_H /*********************************************************************************************************************************** Command constants ***********************************************************************************************************************************/ #define CFGCMD_ANNOTATE "annotate" #define CFGCMD_ARCHIVE_GET "archive-get" #define CFGCMD_ARCHIVE_PUSH "archive-push" #define CFGCMD_BACKUP "backup" #define CFGCMD_CHECK "check" #define CFGCMD_EXPIRE "expire" #define CFGCMD_HELP "help" #define CFGCMD_INFO "info" #define CFGCMD_MANIFEST "manifest" #define CFGCMD_REPO_GET "repo-get" #define CFGCMD_REPO_LS "repo-ls" #define CFGCMD_REPO_PUT "repo-put" #define CFGCMD_REPO_RM "repo-rm" #define CFGCMD_RESTORE "restore" #define CFGCMD_SERVER "server" #define CFGCMD_SERVER_PING "server-ping" #define CFGCMD_STANZA_CREATE "stanza-create" #define CFGCMD_STANZA_DELETE "stanza-delete" #define CFGCMD_STANZA_UPGRADE "stanza-upgrade" #define CFGCMD_START "start" #define CFGCMD_STOP "stop" #define CFGCMD_VERIFY "verify" #define CFGCMD_VERSION "version" #define CFG_COMMAND_TOTAL 23 /*********************************************************************************************************************************** Option group constants ***********************************************************************************************************************************/ #define CFG_OPTION_GROUP_TOTAL 2 /*********************************************************************************************************************************** Option constants ***********************************************************************************************************************************/ #define CFGOPT_ALLOW_ROOT "allow-root" #define CFGOPT_ANNOTATION "annotation" #define CFGOPT_ARCHIVE_ASYNC "archive-async" #define CFGOPT_ARCHIVE_CHECK "archive-check" #define CFGOPT_ARCHIVE_COPY "archive-copy" #define CFGOPT_ARCHIVE_EXPIRE_BEFORE "archive-expire-before" #define CFGOPT_ARCHIVE_GET_QUEUE_MAX "archive-get-queue-max" #define CFGOPT_ARCHIVE_HEADER_CHECK "archive-header-check" #define CFGOPT_ARCHIVE_MISSING_RETRY "archive-missing-retry" #define CFGOPT_ARCHIVE_MODE "archive-mode" #define CFGOPT_ARCHIVE_MODE_CHECK "archive-mode-check" #define CFGOPT_ARCHIVE_PUSH_BATCH_SIZE "archive-push-batch-size" #define CFGOPT_ARCHIVE_PUSH_QUEUE_MAX "archive-push-queue-max" #define CFGOPT_ARCHIVE_TIMEOUT "archive-timeout" #define CFGOPT_BACKUP_STANDBY "backup-standby" #define CFGOPT_BETA "beta" #define CFGOPT_BUFFER_SIZE "buffer-size" #define CFGOPT_CHECKSUM_PAGE "checksum-page" #define CFGOPT_CIPHER_PASS "cipher-pass" #define CFGOPT_CMD "cmd" #define CFGOPT_CMD_SSH "cmd-ssh" #define CFGOPT_COMPRESS "compress" #define CFGOPT_COMPRESS_LEVEL "compress-level" #define CFGOPT_COMPRESS_LEVEL_NETWORK "compress-level-network" #define CFGOPT_COMPRESS_TYPE "compress-type" #define CFGOPT_CONFIG "config" #define CFGOPT_CONFIG_INCLUDE_PATH "config-include-path" #define CFGOPT_CONFIG_PATH "config-path" #define CFGOPT_DB_EXCLUDE "db-exclude" #define CFGOPT_DB_INCLUDE "db-include" #define CFGOPT_DB_TIMEOUT "db-timeout" #define CFGOPT_DELTA "delta" #define CFGOPT_DETAIL_LEVEL "detail-level" #define CFGOPT_DRY_RUN "dry-run" #define CFGOPT_EXCLUDE "exclude" #define CFGOPT_EXEC_ID "exec-id" #define CFGOPT_EXPIRE_AUTO "expire-auto" #define CFGOPT_FILTER "filter" #define CFGOPT_FORCE "force" #define CFGOPT_HELP "help" #define CFGOPT_IGNORE_MISSING "ignore-missing" #define CFGOPT_IO_TIMEOUT "io-timeout" #define CFGOPT_JOB_RETRY "job-retry" #define CFGOPT_JOB_RETRY_INTERVAL "job-retry-interval" #define CFGOPT_LINK_ALL "link-all" #define CFGOPT_LINK_MAP "link-map" #define CFGOPT_LOCK "lock" #define CFGOPT_LOCK_PATH "lock-path" #define CFGOPT_LOG_LEVEL_CONSOLE "log-level-console" #define CFGOPT_LOG_LEVEL_FILE "log-level-file" #define CFGOPT_LOG_LEVEL_STDERR "log-level-stderr" #define CFGOPT_LOG_PATH "log-path" #define CFGOPT_LOG_SUBPROCESS "log-subprocess" #define CFGOPT_LOG_TIMESTAMP "log-timestamp" #define CFGOPT_MANIFEST_SAVE_THRESHOLD "manifest-save-threshold" #define CFGOPT_NEUTRAL_UMASK "neutral-umask" #define CFGOPT_OLDEST "oldest" #define CFGOPT_ONLINE "online" #define CFGOPT_OUTPUT "output" #define CFGOPT_PAGE_HEADER_CHECK "page-header-check" #define CFGOPT_PG "pg" #define CFGOPT_PG_VERSION_FORCE "pg-version-force" #define CFGOPT_PRIORITY "priority" #define CFGOPT_PROCESS "process" #define CFGOPT_PROCESS_MAX "process-max" #define CFGOPT_PROTOCOL_TIMEOUT "protocol-timeout" #define CFGOPT_RAW "raw" #define CFGOPT_RECOVERY_OPTION "recovery-option" #define CFGOPT_RECURSE "recurse" #define CFGOPT_REFERENCE "reference" #define CFGOPT_REMOTE_TYPE "remote-type" #define CFGOPT_REPO "repo" #define CFGOPT_REPO_TARGET_TIME "repo-target-time" #define CFGOPT_REPORT "report" #define CFGOPT_RESUME "resume" #define CFGOPT_SCK_BLOCK "sck-block" #define CFGOPT_SCK_KEEP_ALIVE "sck-keep-alive" #define CFGOPT_SET "set" #define CFGOPT_SORT "sort" #define CFGOPT_SPOOL_PATH "spool-path" #define CFGOPT_STANZA "stanza" #define CFGOPT_START_FAST "start-fast" #define CFGOPT_STOP_AUTO "stop-auto" #define CFGOPT_TABLESPACE_MAP "tablespace-map" #define CFGOPT_TABLESPACE_MAP_ALL "tablespace-map-all" #define CFGOPT_TARGET "target" #define CFGOPT_TARGET_ACTION "target-action" #define CFGOPT_TARGET_EXCLUSIVE "target-exclusive" #define CFGOPT_TARGET_TIMELINE "target-timeline" #define CFGOPT_TCP_KEEP_ALIVE_COUNT "tcp-keep-alive-count" #define CFGOPT_TCP_KEEP_ALIVE_IDLE "tcp-keep-alive-idle" #define CFGOPT_TCP_KEEP_ALIVE_INTERVAL "tcp-keep-alive-interval" #define CFGOPT_TLS_CIPHER_12 "tls-cipher-12" #define CFGOPT_TLS_CIPHER_13 "tls-cipher-13" #define CFGOPT_TLS_SERVER_ADDRESS "tls-server-address" #define CFGOPT_TLS_SERVER_AUTH "tls-server-auth" #define CFGOPT_TLS_SERVER_CA_FILE "tls-server-ca-file" #define CFGOPT_TLS_SERVER_CERT_FILE "tls-server-cert-file" #define CFGOPT_TLS_SERVER_KEY_FILE "tls-server-key-file" #define CFGOPT_TLS_SERVER_PORT "tls-server-port" #define CFGOPT_TYPE "type" #define CFGOPT_VERBOSE "verbose" #define CFGOPT_VERSION "version" #define CFG_OPTION_TOTAL 198 /*********************************************************************************************************************************** Option value constants ***********************************************************************************************************************************/ #define CFGOPTVAL_ARCHIVE_MODE_PRESERVE 0 #define CFGOPTVAL_ARCHIVE_MODE_PRESERVE_STRID STRID5S("preserve", 0, 0x2da45996502) #define CFGOPTVAL_ARCHIVE_MODE_PRESERVE_Z "preserve" #define CFGOPTVAL_ARCHIVE_MODE_OFF 1 #define CFGOPTVAL_ARCHIVE_MODE_OFF_STRID STRID5S("off", 1, 0x18cf4) #define CFGOPTVAL_ARCHIVE_MODE_OFF_Z "off" #define CFGOPTVAL_BACKUP_STANDBY_N 0 #define CFGOPTVAL_BACKUP_STANDBY_N_STRID STRID5S("n", 0, 0xe2) #define CFGOPTVAL_BACKUP_STANDBY_N_Z "n" #define CFGOPTVAL_BACKUP_STANDBY_PREFER 1 #define CFGOPTVAL_BACKUP_STANDBY_PREFER_STRID STRID5S("prefer", 1, 0x245316504) #define CFGOPTVAL_BACKUP_STANDBY_PREFER_Z "prefer" #define CFGOPTVAL_BACKUP_STANDBY_Y 2 #define CFGOPTVAL_BACKUP_STANDBY_Y_STRID STRID5S("y", 2, 0x196) #define CFGOPTVAL_BACKUP_STANDBY_Y_Z "y" #define CFGOPTVAL_COMPRESS_TYPE_BZ2 STRID5("bz2", 0x73420) #define CFGOPTVAL_COMPRESS_TYPE_BZ2_Z "bz2" #define CFGOPTVAL_COMPRESS_TYPE_GZ STRID5("gz", 0x3470) #define CFGOPTVAL_COMPRESS_TYPE_GZ_Z "gz" #define CFGOPTVAL_COMPRESS_TYPE_LZ4 STRID6("lz4", 0x2068c1) #define CFGOPTVAL_COMPRESS_TYPE_LZ4_Z "lz4" #define CFGOPTVAL_COMPRESS_TYPE_NONE STRID5("none", 0x2b9ee0) #define CFGOPTVAL_COMPRESS_TYPE_NONE_Z "none" #define CFGOPTVAL_COMPRESS_TYPE_ZST STRID5("zst", 0x527a0) #define CFGOPTVAL_COMPRESS_TYPE_ZST_Z "zst" #define CFGOPTVAL_DETAIL_LEVEL_FULL 0 #define CFGOPTVAL_DETAIL_LEVEL_FULL_STRID STRID5S("full", 0, 0x632a62) #define CFGOPTVAL_DETAIL_LEVEL_FULL_Z "full" #define CFGOPTVAL_DETAIL_LEVEL_PROGRESS 1 #define CFGOPTVAL_DETAIL_LEVEL_PROGRESS_STRID STRID5S("progress", 1, 0x9ccb23be504) #define CFGOPTVAL_DETAIL_LEVEL_PROGRESS_Z "progress" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_OFF 0 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_OFF_STRID STRID5S("off", 0, 0x18cf2) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_OFF_Z "off" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_ERROR 1 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_ERROR_STRID STRID5S("error", 1, 0x127ca454) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_ERROR_Z "error" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_WARN 2 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_WARN_STRID STRID5S("warn", 2, 0x748376) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_WARN_Z "warn" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_INFO 3 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_INFO_STRID STRID5S("info", 3, 0x799c98) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_INFO_Z "info" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DETAIL 4 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DETAIL_STRID STRID5S("detail", 4, 0x1890d0a4a) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DETAIL_Z "detail" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DEBUG 5 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DEBUG_STRID STRID5S("debug", 5, 0x7a88a4c) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_DEBUG_Z "debug" #define CFGOPTVAL_LOG_LEVEL_CONSOLE_TRACE 6 #define CFGOPTVAL_LOG_LEVEL_CONSOLE_TRACE_STRID STRID5S("trace", 6, 0xa30ca80e) #define CFGOPTVAL_LOG_LEVEL_CONSOLE_TRACE_Z "trace" #define CFGOPTVAL_LOG_LEVEL_FILE_OFF 0 #define CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID STRID5S("off", 0, 0x18cf2) #define CFGOPTVAL_LOG_LEVEL_FILE_OFF_Z "off" #define CFGOPTVAL_LOG_LEVEL_FILE_ERROR 1 #define CFGOPTVAL_LOG_LEVEL_FILE_ERROR_STRID STRID5S("error", 1, 0x127ca454) #define CFGOPTVAL_LOG_LEVEL_FILE_ERROR_Z "error" #define CFGOPTVAL_LOG_LEVEL_FILE_WARN 2 #define CFGOPTVAL_LOG_LEVEL_FILE_WARN_STRID STRID5S("warn", 2, 0x748376) #define CFGOPTVAL_LOG_LEVEL_FILE_WARN_Z "warn" #define CFGOPTVAL_LOG_LEVEL_FILE_INFO 3 #define CFGOPTVAL_LOG_LEVEL_FILE_INFO_STRID STRID5S("info", 3, 0x799c98) #define CFGOPTVAL_LOG_LEVEL_FILE_INFO_Z "info" #define CFGOPTVAL_LOG_LEVEL_FILE_DETAIL 4 #define CFGOPTVAL_LOG_LEVEL_FILE_DETAIL_STRID STRID5S("detail", 4, 0x1890d0a4a) #define CFGOPTVAL_LOG_LEVEL_FILE_DETAIL_Z "detail" #define CFGOPTVAL_LOG_LEVEL_FILE_DEBUG 5 #define CFGOPTVAL_LOG_LEVEL_FILE_DEBUG_STRID STRID5S("debug", 5, 0x7a88a4c) #define CFGOPTVAL_LOG_LEVEL_FILE_DEBUG_Z "debug" #define CFGOPTVAL_LOG_LEVEL_FILE_TRACE 6 #define CFGOPTVAL_LOG_LEVEL_FILE_TRACE_STRID STRID5S("trace", 6, 0xa30ca80e) #define CFGOPTVAL_LOG_LEVEL_FILE_TRACE_Z "trace" #define CFGOPTVAL_LOG_LEVEL_STDERR_OFF 0 #define CFGOPTVAL_LOG_LEVEL_STDERR_OFF_STRID STRID5S("off", 0, 0x18cf2) #define CFGOPTVAL_LOG_LEVEL_STDERR_OFF_Z "off" #define CFGOPTVAL_LOG_LEVEL_STDERR_ERROR 1 #define CFGOPTVAL_LOG_LEVEL_STDERR_ERROR_STRID STRID5S("error", 1, 0x127ca454) #define CFGOPTVAL_LOG_LEVEL_STDERR_ERROR_Z "error" #define CFGOPTVAL_LOG_LEVEL_STDERR_WARN 2 #define CFGOPTVAL_LOG_LEVEL_STDERR_WARN_STRID STRID5S("warn", 2, 0x748376) #define CFGOPTVAL_LOG_LEVEL_STDERR_WARN_Z "warn" #define CFGOPTVAL_LOG_LEVEL_STDERR_INFO 3 #define CFGOPTVAL_LOG_LEVEL_STDERR_INFO_STRID STRID5S("info", 3, 0x799c98) #define CFGOPTVAL_LOG_LEVEL_STDERR_INFO_Z "info" #define CFGOPTVAL_LOG_LEVEL_STDERR_DETAIL 4 #define CFGOPTVAL_LOG_LEVEL_STDERR_DETAIL_STRID STRID5S("detail", 4, 0x1890d0a4a) #define CFGOPTVAL_LOG_LEVEL_STDERR_DETAIL_Z "detail" #define CFGOPTVAL_LOG_LEVEL_STDERR_DEBUG 5 #define CFGOPTVAL_LOG_LEVEL_STDERR_DEBUG_STRID STRID5S("debug", 5, 0x7a88a4c) #define CFGOPTVAL_LOG_LEVEL_STDERR_DEBUG_Z "debug" #define CFGOPTVAL_LOG_LEVEL_STDERR_TRACE 6 #define CFGOPTVAL_LOG_LEVEL_STDERR_TRACE_STRID STRID5S("trace", 6, 0xa30ca80e) #define CFGOPTVAL_LOG_LEVEL_STDERR_TRACE_Z "trace" #define CFGOPTVAL_OUTPUT_TEXT 0 #define CFGOPTVAL_OUTPUT_TEXT_STRID STRID5S("text", 0, 0xa60b42) #define CFGOPTVAL_OUTPUT_TEXT_Z "text" #define CFGOPTVAL_OUTPUT_JSON 1 #define CFGOPTVAL_OUTPUT_JSON_STRID STRID5S("json", 1, 0x73e6a4) #define CFGOPTVAL_OUTPUT_JSON_Z "json" #define CFGOPTVAL_VERIFY_OUTPUT_NONE 0 #define CFGOPTVAL_VERIFY_OUTPUT_NONE_STRID STRID5S("none", 0, 0x2b9ee2) #define CFGOPTVAL_VERIFY_OUTPUT_NONE_Z "none" #define CFGOPTVAL_VERIFY_OUTPUT_TEXT 1 #define CFGOPTVAL_VERIFY_OUTPUT_TEXT_STRID STRID5S("text", 1, 0xa60b44) #define CFGOPTVAL_VERIFY_OUTPUT_TEXT_Z "text" #define CFGOPTVAL_VERSION_OUTPUT_TEXT 0 #define CFGOPTVAL_VERSION_OUTPUT_TEXT_STRID STRID5S("text", 0, 0xa60b42) #define CFGOPTVAL_VERSION_OUTPUT_TEXT_Z "text" #define CFGOPTVAL_VERSION_OUTPUT_NUM 1 #define CFGOPTVAL_VERSION_OUTPUT_NUM_STRID STRID5S("num", 1, 0x36ae4) #define CFGOPTVAL_VERSION_OUTPUT_NUM_Z "num" #define CFGOPTVAL_PG_HOST_TYPE_SSH STRID5("ssh", 0x22730) #define CFGOPTVAL_PG_HOST_TYPE_SSH_Z "ssh" #define CFGOPTVAL_PG_HOST_TYPE_TLS STRID5("tls", 0x4d940) #define CFGOPTVAL_PG_HOST_TYPE_TLS_Z "tls" #define CFGOPTVAL_REMOTE_TYPE_PG STRID5("pg", 0xf00) #define CFGOPTVAL_REMOTE_TYPE_PG_Z "pg" #define CFGOPTVAL_REMOTE_TYPE_REPO STRID5("repo", 0x7c0b20) #define CFGOPTVAL_REMOTE_TYPE_REPO_Z "repo" #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SHARED 0 #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SHARED_STRID STRID5S("shared", 0, 0x85905132) #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SHARED_Z "shared" #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SAS 1 #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SAS_STRID STRID5S("sas", 1, 0x4c334) #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_SAS_Z "sas" #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_AUTO 2 #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_AUTO_STRID STRID5S("auto", 2, 0x7d2a16) #define CFGOPTVAL_REPO_AZURE_KEY_TYPE_AUTO_Z "auto" #define CFGOPTVAL_REPO_AZURE_URI_STYLE_HOST 0 #define CFGOPTVAL_REPO_AZURE_URI_STYLE_HOST_STRID STRID5S("host", 0, 0xa4de82) #define CFGOPTVAL_REPO_AZURE_URI_STYLE_HOST_Z "host" #define CFGOPTVAL_REPO_AZURE_URI_STYLE_PATH 1 #define CFGOPTVAL_REPO_AZURE_URI_STYLE_PATH_STRID STRID5S("path", 1, 0x450304) #define CFGOPTVAL_REPO_AZURE_URI_STYLE_PATH_Z "path" #define CFGOPTVAL_REPO_CIPHER_TYPE_AES_256_CBC STRID5("aes-256-cbc", 0xc43dfbbcdcca10) #define CFGOPTVAL_REPO_CIPHER_TYPE_AES_256_CBC_Z "aes-256-cbc" #define CFGOPTVAL_REPO_CIPHER_TYPE_NONE STRID5("none", 0x2b9ee0) #define CFGOPTVAL_REPO_CIPHER_TYPE_NONE_Z "none" #define CFGOPTVAL_REPO_GCS_KEY_TYPE_SERVICE 0 #define CFGOPTVAL_REPO_GCS_KEY_TYPE_SERVICE_STRID STRID5S("service", 0, 0x1469b48b32) #define CFGOPTVAL_REPO_GCS_KEY_TYPE_SERVICE_Z "service" #define CFGOPTVAL_REPO_GCS_KEY_TYPE_TOKEN 1 #define CFGOPTVAL_REPO_GCS_KEY_TYPE_TOKEN_STRID STRID5S("token", 1, 0xe2adf44) #define CFGOPTVAL_REPO_GCS_KEY_TYPE_TOKEN_Z "token" #define CFGOPTVAL_REPO_GCS_KEY_TYPE_AUTO 2 #define CFGOPTVAL_REPO_GCS_KEY_TYPE_AUTO_STRID STRID5S("auto", 2, 0x7d2a16) #define CFGOPTVAL_REPO_GCS_KEY_TYPE_AUTO_Z "auto" #define CFGOPTVAL_REPO_HOST_TYPE_SSH STRID5("ssh", 0x22730) #define CFGOPTVAL_REPO_HOST_TYPE_SSH_Z "ssh" #define CFGOPTVAL_REPO_HOST_TYPE_TLS STRID5("tls", 0x4d940) #define CFGOPTVAL_REPO_HOST_TYPE_TLS_Z "tls" #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_DIFF STRID5("diff", 0x319240) #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_DIFF_Z "diff" #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_FULL STRID5("full", 0x632a60) #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_FULL_Z "full" #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_INCR STRID5("incr", 0x90dc90) #define CFGOPTVAL_REPO_RETENTION_ARCHIVE_TYPE_INCR_Z "incr" #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT 0 #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT_STRID STRID5S("count", 0, 0x14755e32) #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT_Z "count" #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_TIME 1 #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_TIME_STRID STRID5S("time", 1, 0x2b5344) #define CFGOPTVAL_REPO_RETENTION_FULL_TYPE_TIME_Z "time" #define CFGOPTVAL_REPO_S3_KEY_TYPE_SHARED 0 #define CFGOPTVAL_REPO_S3_KEY_TYPE_SHARED_STRID STRID5S("shared", 0, 0x85905132) #define CFGOPTVAL_REPO_S3_KEY_TYPE_SHARED_Z "shared" #define CFGOPTVAL_REPO_S3_KEY_TYPE_AUTO 1 #define CFGOPTVAL_REPO_S3_KEY_TYPE_AUTO_STRID STRID5S("auto", 1, 0x7d2a14) #define CFGOPTVAL_REPO_S3_KEY_TYPE_AUTO_Z "auto" #define CFGOPTVAL_REPO_S3_KEY_TYPE_WEB_ID 2 #define CFGOPTVAL_REPO_S3_KEY_TYPE_WEB_ID_STRID STRID5S("web-id", 2, 0x89d88b76) #define CFGOPTVAL_REPO_S3_KEY_TYPE_WEB_ID_Z "web-id" #define CFGOPTVAL_REPO_S3_KEY_TYPE_POD_ID 3 #define CFGOPTVAL_REPO_S3_KEY_TYPE_POD_ID_STRID STRID5S("pod-id", 3, 0x89d91f08) #define CFGOPTVAL_REPO_S3_KEY_TYPE_POD_ID_Z "pod-id" #define CFGOPTVAL_REPO_S3_KEY_TYPE_PROCESS 4 #define CFGOPTVAL_REPO_S3_KEY_TYPE_PROCESS_STRID STRID5S("process", 4, 0x4e651be50a) #define CFGOPTVAL_REPO_S3_KEY_TYPE_PROCESS_Z "process" #define CFGOPTVAL_REPO_S3_URI_STYLE_HOST 0 #define CFGOPTVAL_REPO_S3_URI_STYLE_HOST_STRID STRID5S("host", 0, 0xa4de82) #define CFGOPTVAL_REPO_S3_URI_STYLE_HOST_Z "host" #define CFGOPTVAL_REPO_S3_URI_STYLE_PATH 1 #define CFGOPTVAL_REPO_S3_URI_STYLE_PATH_STRID STRID5S("path", 1, 0x450304) #define CFGOPTVAL_REPO_S3_URI_STYLE_PATH_Z "path" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_ACCEPT_NEW STRID5("accept-new", 0x2e576e9028c610) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_ACCEPT_NEW_Z "accept-new" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT STRID5("fingerprint", 0x51c9942453b9260) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z "fingerprint" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_NONE STRID5("none", 0x2b9ee0) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_NONE_Z "none" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_STRICT STRID5("strict", 0x2834ca930) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_STRICT_Z "strict" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_MD5 STRID5("md5", 0x748d0) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_MD5_Z "md5" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_SHA1 STRID6("sha1", 0x7412131) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_SHA1_Z "sha1" #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_SHA256 STRID5("sha256", 0x3dde05130) #define CFGOPTVAL_REPO_SFTP_HOST_KEY_HASH_TYPE_SHA256_Z "sha256" #define CFGOPTVAL_REPO_TYPE_AZURE STRID5("azure", 0x5957410) #define CFGOPTVAL_REPO_TYPE_AZURE_Z "azure" #define CFGOPTVAL_REPO_TYPE_CIFS STRID5("cifs", 0x999230) #define CFGOPTVAL_REPO_TYPE_CIFS_Z "cifs" #define CFGOPTVAL_REPO_TYPE_GCS STRID5("gcs", 0x4c670) #define CFGOPTVAL_REPO_TYPE_GCS_Z "gcs" #define CFGOPTVAL_REPO_TYPE_POSIX STRID5("posix", 0x184cdf00) #define CFGOPTVAL_REPO_TYPE_POSIX_Z "posix" #define CFGOPTVAL_REPO_TYPE_S3 STRID6("s3", 0x7d31) #define CFGOPTVAL_REPO_TYPE_S3_Z "s3" #define CFGOPTVAL_REPO_TYPE_SFTP STRID5("sftp", 0x850d30) #define CFGOPTVAL_REPO_TYPE_SFTP_Z "sftp" #define CFGOPTVAL_SORT_NONE 0 #define CFGOPTVAL_SORT_NONE_STRID STRID5S("none", 0, 0x2b9ee2) #define CFGOPTVAL_SORT_NONE_Z "none" #define CFGOPTVAL_SORT_ASC 1 #define CFGOPTVAL_SORT_ASC_STRID STRID5S("asc", 1, 0xe614) #define CFGOPTVAL_SORT_ASC_Z "asc" #define CFGOPTVAL_SORT_DESC 2 #define CFGOPTVAL_SORT_DESC_STRID STRID5S("desc", 2, 0x1cca46) #define CFGOPTVAL_SORT_DESC_Z "desc" #define CFGOPTVAL_TARGET_ACTION_PAUSE STRID5("pause", 0x59d4300) #define CFGOPTVAL_TARGET_ACTION_PAUSE_Z "pause" #define CFGOPTVAL_TARGET_ACTION_PROMOTE STRID5("promote", 0x168f6be500) #define CFGOPTVAL_TARGET_ACTION_PROMOTE_Z "promote" #define CFGOPTVAL_TARGET_ACTION_SHUTDOWN STRID5("shutdown", 0x75de4a55130) #define CFGOPTVAL_TARGET_ACTION_SHUTDOWN_Z "shutdown" #define CFGOPTVAL_TYPE_DIFF STRID5("diff", 0x319240) #define CFGOPTVAL_TYPE_DIFF_Z "diff" #define CFGOPTVAL_TYPE_FULL STRID5("full", 0x632a60) #define CFGOPTVAL_TYPE_FULL_Z "full" #define CFGOPTVAL_TYPE_INCR STRID5("incr", 0x90dc90) #define CFGOPTVAL_TYPE_INCR_Z "incr" #define CFGOPTVAL_RESTORE_TYPE_DEFAULT 0 #define CFGOPTVAL_RESTORE_TYPE_DEFAULT_STRID STRID5S("default", 0, 0x5195098a42) #define CFGOPTVAL_RESTORE_TYPE_DEFAULT_Z "default" #define CFGOPTVAL_RESTORE_TYPE_PRESERVE 1 #define CFGOPTVAL_RESTORE_TYPE_PRESERVE_STRID STRID5S("preserve", 1, 0x2da45996504) #define CFGOPTVAL_RESTORE_TYPE_PRESERVE_Z "preserve" #define CFGOPTVAL_RESTORE_TYPE_IMMEDIATE 2 #define CFGOPTVAL_RESTORE_TYPE_IMMEDIATE_STRID STRID5S("immediate", 2, 0x5a05242b5a96) #define CFGOPTVAL_RESTORE_TYPE_IMMEDIATE_Z "immediate" #define CFGOPTVAL_RESTORE_TYPE_STANDBY 3 #define CFGOPTVAL_RESTORE_TYPE_STANDBY_STRID STRID5S("standby", 3, 0x6444706938) #define CFGOPTVAL_RESTORE_TYPE_STANDBY_Z "standby" #define CFGOPTVAL_RESTORE_TYPE_NONE 4 #define CFGOPTVAL_RESTORE_TYPE_NONE_STRID STRID5S("none", 4, 0x2b9eea) #define CFGOPTVAL_RESTORE_TYPE_NONE_Z "none" #define CFGOPTVAL_RESTORE_TYPE_NAME 5 #define CFGOPTVAL_RESTORE_TYPE_NAME_STRID STRID5S("name", 5, 0x2b42ec) #define CFGOPTVAL_RESTORE_TYPE_NAME_Z "name" #define CFGOPTVAL_RESTORE_TYPE_TIME 6 #define CFGOPTVAL_RESTORE_TYPE_TIME_STRID STRID5S("time", 6, 0x56a680e) #define CFGOPTVAL_RESTORE_TYPE_TIME_Z "time" #define CFGOPTVAL_RESTORE_TYPE_LSN 7 #define CFGOPTVAL_RESTORE_TYPE_LSN_STRID STRID5S("lsn", 7, 0x74d81e) #define CFGOPTVAL_RESTORE_TYPE_LSN_Z "lsn" #define CFGOPTVAL_RESTORE_TYPE_XID 8 #define CFGOPTVAL_RESTORE_TYPE_XID_STRID STRID5S("xid", 8, 0x22702e) #define CFGOPTVAL_RESTORE_TYPE_XID_Z "xid" /*********************************************************************************************************************************** Command enum ***********************************************************************************************************************************/ typedef enum { cfgCmdAnnotate, cfgCmdArchiveGet, cfgCmdArchivePush, cfgCmdBackup, cfgCmdCheck, cfgCmdExpire, cfgCmdHelp, cfgCmdInfo, cfgCmdManifest, cfgCmdRepoGet, cfgCmdRepoLs, cfgCmdRepoPut, cfgCmdRepoRm, cfgCmdRestore, cfgCmdServer, cfgCmdServerPing, cfgCmdStanzaCreate, cfgCmdStanzaDelete, cfgCmdStanzaUpgrade, cfgCmdStart, cfgCmdStop, cfgCmdVerify, cfgCmdVersion, } ConfigCommand; /*********************************************************************************************************************************** Option group enum ***********************************************************************************************************************************/ typedef enum { cfgOptGrpPg, cfgOptGrpRepo, } ConfigOptionGroup; /*********************************************************************************************************************************** Option enum ***********************************************************************************************************************************/ typedef enum { cfgOptAllowRoot, cfgOptAnnotation, cfgOptArchiveAsync, cfgOptArchiveCheck, cfgOptArchiveCopy, cfgOptArchiveExpireBefore, cfgOptArchiveGetQueueMax, cfgOptArchiveHeaderCheck, cfgOptArchiveMissingRetry, cfgOptArchiveMode, cfgOptArchiveModeCheck, cfgOptArchivePushBatchSize, cfgOptArchivePushQueueMax, cfgOptArchiveTimeout, cfgOptBackupStandby, cfgOptBeta, cfgOptBufferSize, cfgOptChecksumPage, cfgOptCipherPass, cfgOptCmd, cfgOptCmdSsh, cfgOptCompress, cfgOptCompressLevel, cfgOptCompressLevelNetwork, cfgOptCompressType, cfgOptConfig, cfgOptConfigIncludePath, cfgOptConfigPath, cfgOptDbExclude, cfgOptDbInclude, cfgOptDbTimeout, cfgOptDelta, cfgOptDetailLevel, cfgOptDryRun, cfgOptExclude, cfgOptExecId, cfgOptExpireAuto, cfgOptFilter, cfgOptForce, cfgOptHelp, cfgOptIgnoreMissing, cfgOptIoTimeout, cfgOptJobRetry, cfgOptJobRetryInterval, cfgOptLinkAll, cfgOptLinkMap, cfgOptLock, cfgOptLockPath, cfgOptLogLevelConsole, cfgOptLogLevelFile, cfgOptLogLevelStderr, cfgOptLogPath, cfgOptLogSubprocess, cfgOptLogTimestamp, cfgOptManifestSaveThreshold, cfgOptNeutralUmask, cfgOptOldest, cfgOptOnline, cfgOptOutput, cfgOptPageHeaderCheck, cfgOptPg, cfgOptPgDatabase, cfgOptPgHost, cfgOptPgHostCaFile, cfgOptPgHostCaPath, cfgOptPgHostCertFile, cfgOptPgHostCmd, cfgOptPgHostConfig, cfgOptPgHostConfigIncludePath, cfgOptPgHostConfigPath, cfgOptPgHostKeyFile, cfgOptPgHostPort, cfgOptPgHostType, cfgOptPgHostUser, cfgOptPgLocal, cfgOptPgPath, cfgOptPgPort, cfgOptPgSocketPath, cfgOptPgUser, cfgOptPgVersionForce, cfgOptPriority, cfgOptProcess, cfgOptProcessMax, cfgOptProtocolTimeout, cfgOptRaw, cfgOptRecoveryOption, cfgOptRecurse, cfgOptReference, cfgOptRemoteType, cfgOptRepo, cfgOptRepoAzureAccount, cfgOptRepoAzureContainer, cfgOptRepoAzureEndpoint, cfgOptRepoAzureKey, cfgOptRepoAzureKeyType, cfgOptRepoAzureUriStyle, cfgOptRepoBlock, cfgOptRepoBlockAgeMap, cfgOptRepoBlockChecksumSizeMap, cfgOptRepoBlockSizeMap, cfgOptRepoBlockSizeSuper, cfgOptRepoBlockSizeSuperFull, cfgOptRepoBundle, cfgOptRepoBundleLimit, cfgOptRepoBundleSize, cfgOptRepoCipherPass, cfgOptRepoCipherType, cfgOptRepoGcsBucket, cfgOptRepoGcsEndpoint, cfgOptRepoGcsKey, cfgOptRepoGcsKeyType, cfgOptRepoGcsUserProject, cfgOptRepoHardlink, cfgOptRepoHost, cfgOptRepoHostCaFile, cfgOptRepoHostCaPath, cfgOptRepoHostCertFile, cfgOptRepoHostCmd, cfgOptRepoHostConfig, cfgOptRepoHostConfigIncludePath, cfgOptRepoHostConfigPath, cfgOptRepoHostKeyFile, cfgOptRepoHostPort, cfgOptRepoHostType, cfgOptRepoHostUser, cfgOptRepoLocal, cfgOptRepoPath, cfgOptRepoRetentionArchive, cfgOptRepoRetentionArchiveType, cfgOptRepoRetentionDiff, cfgOptRepoRetentionFull, cfgOptRepoRetentionFullType, cfgOptRepoRetentionHistory, cfgOptRepoS3Bucket, cfgOptRepoS3Endpoint, cfgOptRepoS3Key, cfgOptRepoS3KeySecret, cfgOptRepoS3KeyType, cfgOptRepoS3KmsKeyId, cfgOptRepoS3ProcessCmd, cfgOptRepoS3Region, cfgOptRepoS3RequesterPays, cfgOptRepoS3Role, cfgOptRepoS3Service, cfgOptRepoS3SseCustomerKey, cfgOptRepoS3StsHost, cfgOptRepoS3Token, cfgOptRepoS3UriStyle, cfgOptRepoSftpHost, cfgOptRepoSftpHostFingerprint, cfgOptRepoSftpHostKeyCheckType, cfgOptRepoSftpHostKeyHashType, cfgOptRepoSftpHostPort, cfgOptRepoSftpHostUser, cfgOptRepoSftpKnownHost, cfgOptRepoSftpPrivateKeyFile, cfgOptRepoSftpPrivateKeyPassphrase, cfgOptRepoSftpPublicKeyFile, cfgOptRepoStorageCaFile, cfgOptRepoStorageCaPath, cfgOptRepoStorageHost, cfgOptRepoStoragePort, cfgOptRepoStorageTag, cfgOptRepoStorageUploadChunkSize, cfgOptRepoStorageVerifyTls, cfgOptRepoSymlink, cfgOptRepoTargetTime, cfgOptRepoType, cfgOptReport, cfgOptResume, cfgOptSckBlock, cfgOptSckKeepAlive, cfgOptSet, cfgOptSort, cfgOptSpoolPath, cfgOptStanza, cfgOptStartFast, cfgOptStopAuto, cfgOptTablespaceMap, cfgOptTablespaceMapAll, cfgOptTarget, cfgOptTargetAction, cfgOptTargetExclusive, cfgOptTargetTimeline, cfgOptTcpKeepAliveCount, cfgOptTcpKeepAliveIdle, cfgOptTcpKeepAliveInterval, cfgOptTlsCipher12, cfgOptTlsCipher13, cfgOptTlsServerAddress, cfgOptTlsServerAuth, cfgOptTlsServerCaFile, cfgOptTlsServerCertFile, cfgOptTlsServerKeyFile, cfgOptTlsServerPort, cfgOptType, cfgOptVerbose, cfgOptVersion, } ConfigOption; #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/config.c0000664000175100017510000011213715240547630017065 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Configuration ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/memContext.h" #include "config/config.intern.h" #include "config/parse.h" /*********************************************************************************************************************************** Data for the currently loaded configuration ***********************************************************************************************************************************/ static Config *configLocal = NULL; /**********************************************************************************************************************************/ FN_EXTERN void cfgInit(Config *const config) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, config); FUNCTION_TEST_END(); ASSERT(config != NULL); // Free the old context if (configLocal != NULL) memContextFree(configLocal->memContext); // Set config and move context to top so it persists for the life of the program configLocal = config; memContextMove(configLocal->memContext, memContextTop()); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN bool cfgInited(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(ENUM, configLocal != NULL); } /**********************************************************************************************************************************/ FN_EXTERN ConfigCommand cfgCommand(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); FUNCTION_TEST_RETURN(ENUM, configLocal->command); } FN_EXTERN ConfigCommandRole cfgCommandRole(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); FUNCTION_TEST_RETURN(ENUM, configLocal->commandRole); } FN_EXTERN void cfgCommandSet(ConfigCommand commandId, ConfigCommandRole commandRoleId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_PARAM(ENUM, commandRoleId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(commandId < CFG_COMMAND_TOTAL); configLocal->command = commandId; configLocal->commandRole = commandRoleId; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgCommandHelp(void) { FUNCTION_TEST_VOID(); ASSERT(configLocal != NULL); FUNCTION_TEST_RETURN(BOOL, configLocal->help); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * cfgCommandJobRetry(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); // Return NULL if no retries const unsigned int retryTotal = cfgOptionUInt(cfgOptJobRetry); if (retryTotal == 0) FUNCTION_TEST_RETURN(VARIANT_LIST, NULL); // Build retry list VariantList *const result = varLstNew(); MEM_CONTEXT_BEGIN(lstMemContext((List *)result)) { for (unsigned int retryIdx = 0; retryIdx < cfgOptionUInt(cfgOptJobRetry); retryIdx++) varLstAdd(result, varNewUInt64(retryIdx == 0 ? 0 : cfgOptionUInt64(cfgOptJobRetryInterval))); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN(VARIANT_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgCommandName(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN_CONST(STRINGZ, cfgParseCommandName(configLocal->command)); } FN_EXTERN String * cfgCommandRoleName(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(STRING, cfgParseCommandRoleName(cfgCommand(), cfgCommandRole())); } /**********************************************************************************************************************************/ FN_EXTERN const StringList * cfgCommandParam(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); if (configLocal->paramList == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { configLocal->paramList = strLstNew(); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN(STRING_LIST, configLocal->paramList); } /**********************************************************************************************************************************/ FN_EXTERN const String * cfgBin(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); FUNCTION_TEST_RETURN(STRING, configLocal->bin); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgLockRequired(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); // Local roles never take a lock and the remote role has special logic for locking FUNCTION_TEST_RETURN( BOOL, // If a lock is required for the command and the role is main (configLocal->lockRequired && cfgCommandRole() == cfgCmdRoleMain) || // Or any command when the role is async cfgCommandRole() == cfgCmdRoleAsync); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgLockRemoteRequired(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN(BOOL, configLocal->lockRemoteRequired); } /**********************************************************************************************************************************/ FN_EXTERN LockType cfgLockType(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN(ENUM, configLocal->lockType); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgLogFile(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN( BOOL, // If the command always logs to a file configLocal->logFile || // Or log-level-file was explicitly set as a param/env var (cfgOptionValid(cfgOptLogLevelFile) && cfgOptionSource(cfgOptLogLevelFile) == cfgSourceParam) || // Or the role is async cfgCommandRole() == cfgCmdRoleAsync); } /**********************************************************************************************************************************/ FN_EXTERN LogLevel cfgLogLevelDefault(void) { FUNCTION_TEST_VOID(); ASSERT(cfgInited()); ASSERT(configLocal->command < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN(ENUM, configLocal->logLevelDefault); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionGroup(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].group); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgOptionGroupName(const ConfigOptionGroup groupId, const unsigned int groupIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, groupId); FUNCTION_TEST_PARAM(UINT, groupIdx); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(groupId < CFG_OPTION_GROUP_TOTAL); ASSERT(groupIdx < configLocal->optionGroup[groupId].indexTotal); // Generate names for the group the first time one is requested ConfigOptionGroupData *const group = &configLocal->optionGroup[groupId]; if (group->indexName == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { group->indexName = memNew(sizeof(String *) * group->indexTotal); for (unsigned int groupIdx = 0; groupIdx < group->indexTotal; groupIdx++) group->indexName[groupIdx] = strNewFmt("%s%u", group->name, group->indexMap[groupIdx] + 1); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STRINGZ, strZ(group->indexName[groupIdx])); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionGroupId(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(configLocal->option[optionId].group); FUNCTION_TEST_RETURN(UINT, configLocal->option[optionId].groupId); } /**********************************************************************************************************************************/ static bool cfgOptionSequence(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].sequence); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionGroupIdxDefault(const ConfigOptionGroup groupId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, groupId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(groupId < CFG_OPTION_GROUP_TOTAL); ASSERT(configLocal->optionGroup[groupId].indexDefaultExists); FUNCTION_TEST_RETURN(UINT, configLocal->optionGroup[groupId].indexDefault); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionGroupIdxToKey(const ConfigOptionGroup groupId, const unsigned int groupIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, groupId); FUNCTION_TEST_PARAM(UINT, groupIdx); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(groupId < CFG_OPTION_GROUP_TOTAL); ASSERT(groupIdx < configLocal->optionGroup[groupId].indexTotal); FUNCTION_TEST_RETURN(UINT, configLocal->optionGroup[groupId].indexMap[groupIdx] + 1); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionKeyToIdx(const ConfigOption optionId, const unsigned int key) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, key); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); unsigned int result = 0; // If then option is in a group then search for the key, else the index is 0 if (cfgOptionGroup(optionId)) { const unsigned int groupId = cfgOptionGroupId(optionId); // Search the group for the key for (; result < cfgOptionGroupIdxTotal(groupId); result++) { if (configLocal->optionGroup[groupId].indexMap[result] == key - 1) break; } // Error when the key is not found if (result == cfgOptionGroupIdxTotal(groupId)) THROW_FMT(AssertError, "key '%u' is not valid for '%s' option", key, configLocal->option[optionId].name); } FUNCTION_TEST_RETURN(UINT, result); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionGroupIdxTotal(const ConfigOptionGroup groupId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, groupId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(groupId < CFG_OPTION_GROUP_TOTAL); FUNCTION_TEST_RETURN(UINT, configLocal->optionGroup[groupId].indexTotal); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionIdxDefault(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT( !configLocal->option[optionId].group || configLocal->optionGroup[configLocal->option[optionId].groupId].indexDefaultExists); const ConfigOptionData *const option = &configLocal->option[optionId]; FUNCTION_TEST_RETURN(UINT, option->group ? configLocal->optionGroup[option->groupId].indexDefault : 0); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionIdxTotal(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); const ConfigOptionData *const option = &configLocal->option[optionId]; FUNCTION_TEST_RETURN(UINT, option->group ? configLocal->optionGroup[option->groupId].indexTotal : 1); } /**********************************************************************************************************************************/ FN_EXTERN const String * cfgOptionIdxDefaultValue(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); FUNCTION_TEST_RETURN_CONST(STRING, configLocal->option[optionId].index[optionIdx].defaultValue); } /**********************************************************************************************************************************/ FN_EXTERN const String * cfgOptionDisplayVar(const Variant *const value, const ConfigOptionType optionType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_PARAM(UINT, optionType); FUNCTION_TEST_END(); ASSERT(value != NULL); ASSERT(optionType != cfgOptTypeHash && optionType != cfgOptTypeList); if (varType(value) == varTypeString) { FUNCTION_TEST_RETURN_CONST(STRING, varStr(value)); } else if (optionType == cfgOptTypeBoolean) { FUNCTION_TEST_RETURN_CONST(STRING, varBool(value) ? TRUE_STR : FALSE_STR); } else if (optionType == cfgOptTypeTime) { FUNCTION_TEST_RETURN_CONST(STRING, strNewDivP(varUInt64Force(value), MSEC_PER_SEC, .precision = 3, .trim = true)); } else if (optionType == cfgOptTypeStringId) { FUNCTION_TEST_RETURN_CONST(STRING, strNewStrId(varUInt64(value))); } FUNCTION_TEST_RETURN(STRING, varStrForce(value)); } FN_EXTERN const String * cfgOptionIdxDisplay(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); // Check that the option is valid for the current command if (!cfgOptionValid(optionId)) THROW_FMT(AssertError, "option '%s' is not valid for the current command", cfgOptionIdxName(optionId, optionIdx)); // If there is already a display value set then return that ConfigOptionValue *const option = &configLocal->option[optionId].index[optionIdx]; if (option->display != NULL) FUNCTION_TEST_RETURN_CONST(STRING, option->display); // Generate the display value based on the type MEM_CONTEXT_BEGIN(configLocal->memContext) { option->display = cfgOptionDisplayVar(cfgOptionIdxVar(optionId, optionIdx), cfgParseOptionType(optionId)); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_CONST(STRING, option->display); } FN_EXTERN const String * cfgOptionDisplay(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, cfgOptionIdxDisplay(optionId, cfgOptionIdxDefault(optionId))); } /*********************************************************************************************************************************** Get option name by id ***********************************************************************************************************************************/ FN_EXTERN const char * cfgOptionName(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN_CONST(STRINGZ, cfgOptionIdxName(optionId, cfgOptionIdxDefault(optionId))); } FN_EXTERN const char * cfgOptionIdxName(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); // If an indexed option ConfigOptionData *const option = &configLocal->option[optionId]; if (option->group) { const ConfigOptionGroupData *const group = &configLocal->optionGroup[option->groupId]; // Generate indexed names for the option the first time one is requested if (option->indexName == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { option->indexName = memNew(sizeof(String *) * group->indexTotal); for (unsigned int optionIdx = 0; optionIdx < group->indexTotal; optionIdx++) { option->indexName[optionIdx] = strNewFmt( "%s%u%s", group->name, group->indexMap[optionIdx] + 1, option->name + strlen(group->name)); } } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STRINGZ, strZ(option->indexName[optionIdx])); } // Else not indexed FUNCTION_TEST_RETURN_CONST(STRINGZ, option->name); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionIdxNegate(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].index[optionIdx].negate); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionIdxReset(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].index[optionIdx].reset); } /**********************************************************************************************************************************/ // Helper to enforce constraints when getting options static ConfigOptionValue * cfgOptionIdxInternal( const ConfigOption optionId, const unsigned int optionIdx, const ConfigOptionDataType typeRequested, const bool nullAllowed) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_PARAM(ENUM, typeRequested); FUNCTION_TEST_PARAM(BOOL, nullAllowed); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); // Check that the option is valid for the current command if (!cfgOptionValid(optionId)) THROW_FMT(AssertError, "option '%s' is not valid for the current command", cfgOptionIdxName(optionId, optionIdx)); // If the option is not NULL then check it is the requested type const ConfigOptionData *const option = &configLocal->option[optionId]; ConfigOptionValue *const result = &option->index[optionIdx]; if (result->set) { if (option->dataType != typeRequested) { THROW_FMT( AssertError, "option '%s' is type %u but %u was requested", cfgOptionIdxName(optionId, optionIdx), option->dataType, typeRequested); } } // Else check the option is allowed to be NULL else if (!nullAllowed) THROW_FMT(AssertError, "option '%s' is null but non-null was requested", cfgOptionIdxName(optionId, optionIdx)); FUNCTION_TEST_RETURN_TYPE_P(ConfigOptionValue, result); } FN_EXTERN Variant * cfgOptionIdxVar(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); const ConfigOptionData *const option = &configLocal->option[optionId]; if (option->index[optionIdx].set) { const ConfigOptionValueType *const optionValueType = &option->index[optionIdx].value; switch (option->dataType) { case cfgOptDataTypeBoolean: FUNCTION_TEST_RETURN(VARIANT, varNewBool(optionValueType->boolean)); case cfgOptDataTypeHash: FUNCTION_TEST_RETURN(VARIANT, varNewKv(kvDup(optionValueType->keyValue))); case cfgOptDataTypeInteger: FUNCTION_TEST_RETURN(VARIANT, varNewInt64(optionValueType->integer)); case cfgOptDataTypeList: FUNCTION_TEST_RETURN(VARIANT, varNewVarLst(optionValueType->list)); case cfgOptDataTypeStringId: cfgOptionIdxStrId(optionId, optionIdx); FUNCTION_TEST_RETURN(VARIANT, varNewUInt64(optionValueType->stringId)); default: ASSERT(option->dataType == cfgOptDataTypeString); break; } FUNCTION_TEST_RETURN(VARIANT, varNewStr(optionValueType->string)); } FUNCTION_TEST_RETURN(VARIANT, NULL); } FN_EXTERN bool cfgOptionIdxBool(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(BOOL, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeBoolean, false)->value.boolean); } FN_EXTERN int cfgOptionIdxInt(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer >= INT_MIN); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer <= INT_MAX); FUNCTION_LOG_RETURN(INT, (int)cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer); } FN_EXTERN int64_t cfgOptionIdxInt64(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(INT64, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer); } FN_EXTERN unsigned int cfgOptionIdxUInt(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer >= 0); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer <= UINT_MAX); FUNCTION_LOG_RETURN(UINT, (unsigned int)cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer); } FN_EXTERN uint64_t cfgOptionIdxUInt64(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer >= 0); FUNCTION_LOG_RETURN(UINT64, (uint64_t)cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeInteger, false)->value.integer); } FN_EXTERN const KeyValue * cfgOptionIdxKv(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN_CONST(KEY_VALUE, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeHash, false)->value.keyValue); } FN_EXTERN const KeyValue * cfgOptionIdxKvNull(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN_CONST(KEY_VALUE, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeHash, true)->value.keyValue); } FN_EXTERN const VariantList * cfgOptionIdxLst(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgInited()); const VariantList *optionValue = cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeList, true)->value.list; if (optionValue == NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { optionValue = varLstNew(); configLocal->option[optionId].index[optionIdx].value.list = optionValue; } MEM_CONTEXT_END(); } FUNCTION_LOG_RETURN_CONST(VARIANT_LIST, optionValue); } FN_EXTERN const String * cfgOptionIdxStr(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN_CONST(STRING, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeString, false)->value.string); } FN_EXTERN const String * cfgOptionIdxStrNull(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN_CONST(STRING, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeString, true)->value.string); } FN_EXTERN StringId cfgOptionIdxStrId(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ConfigOptionValue *const optionValue = cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeStringId, false); ConfigOptionValueType *const optionValueType = &optionValue->value; if (optionValueType->stringId == 0) { optionValueType->stringId = strIdFromZN( strZ(optionValue->display), strSize(optionValue->display), cfgOptionSequence(optionId) ? optionValue->sequence : STRING_ID_SEQ_NONE); } FUNCTION_LOG_RETURN(STRING_ID, optionValueType->stringId); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int cfgOptionIdxSeq(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, optionId); FUNCTION_LOG_PARAM(UINT, optionIdx); FUNCTION_LOG_END(); ASSERT(cfgOptionSequence(optionId)); FUNCTION_LOG_RETURN(UINT, cfgOptionIdxInternal(optionId, optionIdx, cfgOptDataTypeStringId, false)->sequence); } /**********************************************************************************************************************************/ FN_EXTERN void cfgOptionSet(const ConfigOption optionId, const ConfigSource source, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(ENUM, source); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); cfgOptionIdxSet(optionId, cfgOptionIdxDefault(optionId), source, value); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void cfgOptionIdxSet( const ConfigOption optionId, const unsigned int optionIdx, const ConfigSource source, const Variant *const value) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_PARAM(ENUM, source); FUNCTION_TEST_PARAM(VARIANT, value); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); // Set the source const ConfigOptionData *const option = &configLocal->option[optionId]; ConfigOptionValue *const optionValue = &option->index[optionIdx]; optionValue->source = source; // Only set value if it is not null ConfigOptionValueType *const optionValueType = &optionValue->value; if (value != NULL) { switch (option->dataType) { case cfgOptDataTypeBoolean: optionValueType->boolean = varBool(value); break; case cfgOptDataTypeInteger: optionValueType->integer = varInt64(value); break; case cfgOptDataTypeString: { if (varType(value) == varTypeString) { MEM_CONTEXT_BEGIN(configLocal->memContext) { optionValueType->string = strDup(varStr(value)); } MEM_CONTEXT_END(); } else { THROW_FMT( AssertError, "option '%s' must be set with String variant", cfgOptionIdxName(optionId, optionIdx)); } break; } case cfgOptDataTypeStringId: { if (varType(value) == varTypeUInt64) { optionValueType->stringId = varUInt64(value); if (option->sequence) optionValue->sequence = (uint8_t)strIdSeq(optionValueType->stringId); } else { ASSERT(!option->sequence); optionValueType->stringId = strStrId(varStr(value)); } break; } default: THROW_FMT(AssertError, "set not available for option data type %u", option->dataType); } optionValue->set = true; } else { optionValue->set = false; // Pointer values need to be set to null since they can be accessed when the option is not set, e.g. cfgOptionStrNull(). // Setting string to NULL suffices to set the other pointers in the union to NULL. optionValueType->string = NULL; } // Set the default when specified if (source == cfgSourceDefault) { if (value != NULL) { MEM_CONTEXT_BEGIN(configLocal->memContext) { optionValue->defaultValue = cfgOptionDisplayVar(value, cfgParseOptionType(optionId)); } MEM_CONTEXT_END(); } else optionValue->defaultValue = NULL; } // Clear the display value, which will be generated when needed optionValue->display = NULL; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ConfigSource cfgOptionSource(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(ENUM, cfgOptionIdxSource(optionId, cfgOptionIdxDefault(optionId))); } FN_EXTERN ConfigSource cfgOptionIdxSource(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT((!group && optionIdx == 0) || (group && optionIdx < indexTotal)); FUNCTION_TEST_RETURN(ENUM, configLocal->option[optionId].index[optionIdx].source); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionTest(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, cfgOptionIdxTest(optionId, cfgOptionIdxDefault(optionId))); } FN_EXTERN bool cfgOptionIdxTest(const ConfigOption optionId, const unsigned int optionIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, optionIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); ASSERT_DECLARE(const bool group = configLocal->option[optionId].group); ASSERT_DECLARE(const unsigned int indexTotal = configLocal->optionGroup[configLocal->option[optionId].groupId].indexTotal); ASSERT(!cfgOptionValid(optionId) || ((!group && optionIdx == 0) || (group && optionIdx < indexTotal))); FUNCTION_TEST_RETURN(BOOL, cfgOptionValid(optionId) && configLocal->option[optionId].index[optionIdx].set); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgOptionValid(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); FUNCTION_TEST_RETURN(BOOL, configLocal->option[optionId].valid); } FN_EXTERN void cfgOptionInvalidate(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT(cfgInited()); configLocal->option[optionId].valid = false; FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/config.h0000664000175100017510000002701115240547630017066 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Configuration This module serves as a database for the configuration options. The configuration rules reside in config/define.c and config/parse.c sets the command and options and determines which options are valid for a command. ***********************************************************************************************************************************/ #ifndef CONFIG_CONFIG_H #define CONFIG_CONFIG_H #include "common/log.h" #include "common/type/stringId.h" #include "common/type/stringList.h" #include "config/config.auto.h" /*********************************************************************************************************************************** Command Role Enum Commands may have multiple processes that work together to implement their functionality. These roles allow each process to know what it is supposed to do. ***********************************************************************************************************************************/ typedef enum { // Called directly by the user. This is the main process of the command that may or may not spawn other command roles. cfgCmdRoleMain = 0, // Async worker that is spawned so the main process can return a result while work continues. An async worker may spawn local or // remote workers. cfgCmdRoleAsync, // Local worker for parallelizing jobs. A local work may spawn a remote worker. cfgCmdRoleLocal, // Remote worker for accessing resources on another host cfgCmdRoleRemote, } ConfigCommandRole; // Constants for the command roles #define CONFIG_COMMAND_ROLE_ASYNC "async" #define CONFIG_COMMAND_ROLE_LOCAL "local" #define CONFIG_COMMAND_ROLE_REMOTE "remote" #define CFG_COMMAND_ROLE_TOTAL 4 /*********************************************************************************************************************************** Lock types ***********************************************************************************************************************************/ typedef enum { lockTypeArchive, lockTypeBackup, lockTypeRestore, lockTypeAll, lockTypeNone, } LockType; /*********************************************************************************************************************************** Command Functions Access the current command and command parameters. ***********************************************************************************************************************************/ // Current command FN_EXTERN ConfigCommand cfgCommand(void); // Current command role (async, local, remote) FN_EXTERN ConfigCommandRole cfgCommandRole(void); // Get command name FN_EXTERN const char *cfgCommandName(void); // Get command:role name FN_EXTERN String *cfgCommandRoleName(void); // Does this command require an immediate lock? FN_EXTERN bool cfgLockRequired(void); // Does the command require a remote lock? FN_EXTERN bool cfgLockRemoteRequired(void); // Lock type required for this command FN_EXTERN LockType cfgLockType(void); // Does this command log to a file? FN_EXTERN bool cfgLogFile(void); // Default log level -- used for log messages that are common to all commands FN_EXTERN LogLevel cfgLogLevelDefault(void); // Command parameters, if any FN_EXTERN const StringList *cfgCommandParam(void); /*********************************************************************************************************************************** Option Group Functions ***********************************************************************************************************************************/ // Format group name for display to the user. Useful for messages that do not show an option name but must use an group name that // the user will recognize. FN_EXTERN const char *cfgOptionGroupName(ConfigOptionGroup groupId, unsigned int groupIdx); // Get the default index for this group, i.e. the index that will be used if a non-indexed function like cfgOptionTest() is called. FN_EXTERN unsigned int cfgOptionGroupIdxDefault(ConfigOptionGroup groupId); // Convert the group index to a key, i.e. the key that was used in the original configuration file, command-line, etc. FN_EXTERN unsigned int cfgOptionGroupIdxToKey(ConfigOptionGroup groupId, unsigned int groupIdx); // Total indexes, 0 if the group is not valid. Will be the total configured indexes, no matter which raw indexes were used during // configuration. e.g., if pg1-path and pg8-path are configured then this function will return 2. FN_EXTERN unsigned int cfgOptionGroupIdxTotal(ConfigOptionGroup groupId); /*********************************************************************************************************************************** Option Functions Access option values, indexes, and determine if an option is valid for the current command. Most functions have a variant that accepts an index, which currently work with non-indexed options (with optionIdx 0) but they may not always do so. ***********************************************************************************************************************************/ // Get default index for a group option FN_EXTERN unsigned int cfgOptionIdxDefault(ConfigOption optionId); // Get boolean config option FN_EXTERN bool cfgOptionIdxBool(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS bool cfgOptionBool(const ConfigOption optionId) { return cfgOptionIdxBool(optionId, cfgOptionIdxDefault(optionId)); } // Get int config option FN_EXTERN int cfgOptionIdxInt(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS int cfgOptionInt(const ConfigOption optionId) { return cfgOptionIdxInt(optionId, cfgOptionIdxDefault(optionId)); } // Get int64 config option FN_EXTERN int64_t cfgOptionIdxInt64(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS int64_t cfgOptionInt64(const ConfigOption optionId) { return cfgOptionIdxInt64(optionId, cfgOptionIdxDefault(optionId)); } // Get kv config option FN_EXTERN const KeyValue *cfgOptionIdxKv(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const KeyValue * cfgOptionKv(const ConfigOption optionId) { return cfgOptionIdxKv(optionId, cfgOptionIdxDefault(optionId)); } FN_EXTERN const KeyValue *cfgOptionIdxKvNull(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const KeyValue * cfgOptionKvNull(const ConfigOption optionId) { return cfgOptionIdxKvNull(optionId, cfgOptionIdxDefault(optionId)); } // Get list config option FN_EXTERN const VariantList *cfgOptionIdxLst(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const VariantList * cfgOptionLst(const ConfigOption optionId) { return cfgOptionIdxLst(optionId, cfgOptionIdxDefault(optionId)); } // Get String config option FN_EXTERN const String *cfgOptionIdxStr(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const String * cfgOptionStr(const ConfigOption optionId) { return cfgOptionIdxStr(optionId, cfgOptionIdxDefault(optionId)); } FN_EXTERN const String *cfgOptionIdxStrNull(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS const String * cfgOptionStrNull(const ConfigOption optionId) { return cfgOptionIdxStrNull(optionId, cfgOptionIdxDefault(optionId)); } // Get StringId config option FN_EXTERN StringId cfgOptionIdxStrId(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS StringId cfgOptionStrId(const ConfigOption optionId) { return cfgOptionIdxStrId(optionId, cfgOptionIdxDefault(optionId)); } // Get config option sequence FN_EXTERN unsigned int cfgOptionIdxSeq(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS unsigned int cfgOptionSeq(const ConfigOption optionId) { return cfgOptionIdxSeq(optionId, cfgOptionIdxDefault(optionId)); } // Get uint config option FN_EXTERN unsigned int cfgOptionIdxUInt(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS unsigned int cfgOptionUInt(const ConfigOption optionId) { return cfgOptionIdxUInt(optionId, cfgOptionIdxDefault(optionId)); } // Get uint64 config option FN_EXTERN uint64_t cfgOptionIdxUInt64(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS uint64_t cfgOptionUInt64(const ConfigOption optionId) { return cfgOptionIdxUInt64(optionId, cfgOptionIdxDefault(optionId)); } // Format the configuration value for display to the user or passing on a command line. If the value was set by the user via the // command line, config, etc., then that exact value will be displayed. This makes it easier for the user to recognize the value and // saves having to format it into something reasonable, especially for time and size option types. Note that cfgOptTypeHash and // cfgOptTypeList option types are not supported by these functions, but they are generally not displayed to the user as a whole. FN_EXTERN const String *cfgOptionDisplay(const ConfigOption optionId); FN_EXTERN const String *cfgOptionIdxDisplay(const ConfigOption optionId, const unsigned int optionIdx); // Option name by id FN_EXTERN const char *cfgOptionName(ConfigOption optionId); FN_EXTERN const char *cfgOptionIdxName(ConfigOption optionId, unsigned int optionIdx); // Is the option valid for this command? FN_EXTERN bool cfgOptionValid(ConfigOption optionId); // Is the option valid for the command and also has a value? FN_EXTERN bool cfgOptionTest(ConfigOption optionId); FN_EXTERN bool cfgOptionIdxTest(ConfigOption optionId, unsigned int optionIdx); /*********************************************************************************************************************************** Option Source Enum Defines where an option was sourced from. The source is only needed when determining what parameters should be passed to a remote process. ***********************************************************************************************************************************/ typedef enum { cfgSourceDefault, // Default value cfgSourceParam, // Passed as command-line parameter cfgSourceConfig, // From configuration file or environment variable } ConfigSource; /*********************************************************************************************************************************** Load Functions Used primarily by modules that need to manipulate the configuration. These modules include, but are not limited to, config/parse.c, config/load.c. ***********************************************************************************************************************************/ // Was help requested? FN_EXTERN bool cfgCommandHelp(void); FN_EXTERN void cfgCommandSet(ConfigCommand commandId, ConfigCommandRole commandRoleId); // Path to current pgbackrest binary FN_EXTERN const String *cfgBin(void); // Was the option negated? FN_EXTERN bool cfgOptionIdxNegate(ConfigOption optionId, unsigned int optionIdx); // Was the option reset? FN_EXTERN bool cfgOptionIdxReset(ConfigOption optionId, unsigned int optionIdx); // Set config option FN_EXTERN void cfgOptionSet(ConfigOption optionId, ConfigSource source, const Variant *value); FN_EXTERN void cfgOptionIdxSet(ConfigOption optionId, unsigned int optionIdx, ConfigSource source, const Variant *value); // How was the option set (default, param, config)? FN_EXTERN ConfigSource cfgOptionSource(ConfigOption optionId); FN_EXTERN ConfigSource cfgOptionIdxSource(ConfigOption optionId, unsigned int optionIdx); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/config.intern.h0000664000175100017510000002130715240547630020366 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Configuration Internals These structures and functions are generally used by modules that create configurations, e.g. config/parse, or modules that manipulate the configuration as a whole, e.g. protocol/helper, in order to communicate with other processes. The general-purpose functions for querying the current configuration are found in config.h. ***********************************************************************************************************************************/ #ifndef CONFIG_CONFIG_INTERN_H #define CONFIG_CONFIG_INTERN_H #include "config/config.h" #include "config/parse.h" /*********************************************************************************************************************************** Configuration data. These structures are not directly user-created or accessible. cfgParseP() creates the structures and uses cfgInit() to load it as the current configuration. Various cfg*() functions provide access. ***********************************************************************************************************************************/ // Group options that are related to allow valid and test checks across all options in the group typedef struct ConfigOptionGroupData { const char *name; // Name bool valid; // Is option group valid for the current command? unsigned int indexTotal; // Total number of indexes with values in option group bool indexDefaultExists; // Is there a default index for non-indexed functions? unsigned int indexDefault; // Default index (usually 0) unsigned int *indexMap; // List of index to key index mappings const String **indexName; // List of index names } ConfigOptionGroupData; // Option data typedef union ConfigOptionValueType { bool boolean; // Boolean int64_t integer; // Integer const KeyValue *keyValue; // KeyValue const VariantList *list; // VariantList const String *string; // String StringId stringId; // StringId } ConfigOptionValueType; typedef struct ConfigOptionValue { bool set; // Is the option set? bool negate; // Is the option negated? bool reset; // Is the option reset? uint8_t sequence; // Sequence, if any unsigned int source; // Where the option came from, i.e. ConfigSource enum const String *display; // Current display value, if any. Used for messages, etc. const String *defaultValue; // Default value ConfigOptionValueType value; // Option value } ConfigOptionValue; typedef struct ConfigOptionData { const char *name; // Name bool valid; // Is option valid for current command? bool group; // In a group? bool sequence; // Is there a sequence? unsigned int groupId; // Id if in a group ConfigOptionDataType dataType; // Underlying data type ConfigOptionValue *index; // List of indexed values (only 1 unless the option is indexed) const String **indexName; // Index names (e.g. repo1-path, repo2-path) } ConfigOptionData; typedef struct Config { MemContext *memContext; // Mem context for config data // Generally set by the command parser but can also be set during execute to change commands, i.e. backup -> expire ConfigCommand command; // Current command ConfigCommandRole commandRole; // Current command role String *bin; // Path to current pgbackrest binary bool help; // Was help requested for the command? bool lockRequired; // Is an immediate lock required? bool lockRemoteRequired; // Is a lock required on the remote? LockType lockType; // Lock type required bool logFile; // Will the command log to a file? LogLevel logLevelDefault; // Default log level StringList *paramList; // Parameters passed to the command (if any) ConfigOptionGroupData optionGroup[CFG_OPTION_GROUP_TOTAL]; // Option group data ConfigOptionData option[CFG_OPTION_TOTAL]; // Option data } Config; /*********************************************************************************************************************************** Init Function ***********************************************************************************************************************************/ // Init with new configuration FN_EXTERN void cfgInit(Config *config); // Has the configuration been initialized? FN_EXTERN bool cfgInited(void); /*********************************************************************************************************************************** Command Functions ***********************************************************************************************************************************/ // List of retry intervals for local jobs. The interval for the first retry will always be 0. NULL if there are no retries. FN_EXTERN VariantList *cfgCommandJobRetry(void); /*********************************************************************************************************************************** Option Group Functions ***********************************************************************************************************************************/ // Is the option in a group? FN_EXTERN bool cfgOptionGroup(ConfigOption optionId); // Group id if the option is in a group FN_EXTERN unsigned int cfgOptionGroupId(ConfigOption optionId); /*********************************************************************************************************************************** Option Functions ***********************************************************************************************************************************/ // Option default - should only be called by the help command FN_EXTERN const String *cfgOptionIdxDefaultValue(ConfigOption optionId, unsigned int optionIdx); // Format a variant for display using the supplied option type. cfgOptionDisplay()/cfgOptionIdxDisplay() should be used whenever // possible, but sometimes the variant needs to be manipulated before being formatted. FN_EXTERN const String *cfgOptionDisplayVar(const Variant *const value, const ConfigOptionType optionType); // Convert the key used in the original configuration to a group index. This is used when an option key must be translated into the // local group index, e.g. during parsing or when getting the value of specific options from a remote. FN_EXTERN unsigned int cfgOptionKeyToIdx(ConfigOption optionId, unsigned int key); // Total indexes for the option if in a group, 1 otherwise. FN_EXTERN unsigned int cfgOptionIdxTotal(ConfigOption optionId); // Get config option as a Variant FN_EXTERN Variant *cfgOptionIdxVar(ConfigOption optionId, unsigned int optionIdx); FN_INLINE_ALWAYS Variant * cfgOptionVar(const ConfigOption optionId) { return cfgOptionIdxVar(optionId, cfgOptionIdxDefault(optionId)); } // Invalidate an option so it will not be passed to other processes. This is used to manage deprecated options that have a newer // option that should be used when possible, e.g. compress and compress-type. FN_EXTERN void cfgOptionInvalidate(ConfigOption optionId); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/exec.c0000664000175100017510000001464315240547630016547 0ustar00runnerrunner/*********************************************************************************************************************************** Exec Configuration ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/log.h" #include "config/config.intern.h" #include "config/exec.h" #include "config/parse.h" /**********************************************************************************************************************************/ FN_EXTERN StringList * cfgExecParam( const ConfigCommand commandId, const ConfigCommandRole commandRoleId, const KeyValue *const optionReplace, const bool local, const bool quote) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, commandId); FUNCTION_LOG_PARAM(ENUM, commandRoleId); FUNCTION_LOG_PARAM(KEY_VALUE, optionReplace); FUNCTION_LOG_PARAM(BOOL, local); // Will the new process be running on the same host? FUNCTION_LOG_PARAM(BOOL, quote); // Do parameters with spaces need to be quoted? FUNCTION_LOG_END(); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Loop though options and add the ones that apply to the specified command for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { // Skip the option if it is not valid for the original/specified command or if is secure. Also skip repo1-cipher-type // because there's no point of passing it if the other process doesn't have access to repo1-cipher-pass. There is // probably a better way to do this last part... if (!cfgParseOptionValid(commandId, commandRoleId, optionId) || cfgParseOptionSecure(optionId) || optionId == cfgOptRepoCipherType) { continue; } // Loop through option indexes const unsigned int optionIdxTotal = cfgOptionGroup(optionId) ? cfgOptionGroupIdxTotal(cfgOptionGroupId(optionId)) : 1; for (unsigned int optionIdx = 0; optionIdx < optionIdxTotal; optionIdx++) { // First check for a replacement const Variant *const key = VARSTRZ(cfgOptionIdxName(optionId, optionIdx)); const Variant *value = NULL; bool exists = false; // If an option is requested to be replaced (usually because remote processes do not have access to the config) // then if the option exists, get the new value for replacement if (optionReplace != NULL) { exists = kvKeyExists(optionReplace, key); if (exists) value = kvGet(optionReplace, key); } // If the key exists but its value is NULL then skip this option if (exists && value == NULL) continue; // If no replacement then see if this option is not default if (value == NULL && cfgOptionValid(optionId)) { if (cfgOptionIdxNegate(optionId, optionIdx)) value = BOOL_FALSE_VAR; else if (cfgOptionIdxSource(optionId, optionIdx) != cfgSourceDefault) value = cfgOptionIdxVar(optionId, optionIdx); } // If the option was reset if (value == NULL && cfgOptionValid(optionId) && cfgOptionIdxReset(optionId, optionIdx)) { strLstAddFmt(result, "--reset-%s", cfgOptionIdxName(optionId, optionIdx)); } // Else format the value if found, even if the option is not valid for the command else if (value != NULL && (!local || exists || cfgOptionIdxSource(optionId, optionIdx) == cfgSourceParam)) { if (varType(value) == varTypeBool) strLstAddFmt(result, "--%s%s", varBool(value) ? "" : "no-", cfgOptionIdxName(optionId, optionIdx)); else { StringList *valueList = NULL; if (varType(value) == varTypeKeyValue) { valueList = strLstNew(); const KeyValue *const optionKv = varKv(value); const VariantList *const keyList = kvKeyList(optionKv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(keyList); keyIdx++) { strLstAddFmt( valueList, "%s=%s", strZ(varStr(varLstGet(keyList, keyIdx))), strZ(varStrForce(kvGet(optionKv, varLstGet(keyList, keyIdx))))); } } else if (varType(value) == varTypeVariantList) { valueList = strLstNewVarLst(varVarLst(value)); } // Else only one value else { valueList = strLstNew(); strLstAdd( valueList, exists ? cfgOptionDisplayVar(value, cfgParseOptionType(optionId)) : cfgOptionIdxDisplay(optionId, optionIdx)); } // Output options and values for (unsigned int valueListIdx = 0; valueListIdx < strLstSize(valueList); valueListIdx++) { const String *value = strLstGet(valueList, valueListIdx); if (quote && strchr(strZ(value), ' ') != NULL) value = strNewFmt("\"%s\"", strZ(value)); strLstAddFmt(result, "--%s=%s", cfgOptionIdxName(optionId, optionIdx), strZ(value)); } } } } } // Add the command strLstAdd(result, cfgParseCommandRoleName(commandId, commandRoleId)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/exec.h0000664000175100017510000000240615240547630016546 0ustar00runnerrunner/*********************************************************************************************************************************** Exec Configuration ***********************************************************************************************************************************/ #ifndef CONFIG_EXEC_H #define CONFIG_EXEC_H #include "common/type/stringList.h" #include "config/config.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Generate a list of options required for execution of a new command, overriding options with the values in optionReplace when // present. If local is set then the new command must have access to the local configuration files and environment since only // options originally passed on the command-line will be added to the list. Use quote when the options will be used as a // concatenated string rather than being passed directly to exec*() as a list. FN_EXTERN StringList *cfgExecParam( ConfigCommand commandId, ConfigCommandRole commandRoleId, const KeyValue *optionReplace, bool local, bool quote); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/load.c0000664000175100017510000006666315240547630016553 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Load ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/command.h" #include "command/lock.h" #include "common/crypto/common.h" #include "common/debug.h" #include "common/io/io.h" #include "common/io/socket/common.h" #include "common/io/tls/common.h" #include "common/log.h" #include "common/memContext.h" #include "common/user.h" #include "config/config.intern.h" #include "config/load.h" #include "config/parse.h" #include "info/infoBackup.h" #include "storage/cifs/storage.h" #include "storage/helper.h" #include "storage/posix/storage.h" #include "storage/sftp/storage.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct ConfigLoadLocal { unsigned int argListSize; // Argument list size const char **argList; // Argument list } configLoadLocal; /*********************************************************************************************************************************** Load log settings ***********************************************************************************************************************************/ static void cfgLoadLogSetting(void) { FUNCTION_LOG_VOID(logLevelTrace); // Initialize logging LogLevel logLevelConsole = logLevelOff; LogLevel logLevelStdErr = logLevelOff; LogLevel logLevelFile = logLevelOff; bool logTimestamp = true; unsigned int logProcessMax = 1; if (cfgOptionValid(cfgOptLogLevelConsole)) logLevelConsole = logLevelEnum(cfgOptionSeq(cfgOptLogLevelConsole)); if (cfgOptionValid(cfgOptLogLevelStderr)) logLevelStdErr = logLevelEnum(cfgOptionSeq(cfgOptLogLevelStderr)); if (cfgOptionValid(cfgOptLogLevelFile)) logLevelFile = logLevelEnum(cfgOptionSeq(cfgOptLogLevelFile)); if (cfgOptionValid(cfgOptLogTimestamp)) logTimestamp = cfgOptionBool(cfgOptLogTimestamp); if (cfgOptionValid(cfgOptProcessMax)) logProcessMax = cfgOptionUInt(cfgOptProcessMax); logInit( logLevelConsole, logLevelStdErr, logLevelFile, logTimestamp, 0, logProcessMax, cfgOptionValid(cfgOptDryRun) && cfgOptionBool(cfgOptDryRun)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cfgLoadUpdateOption(void) { FUNCTION_LOG_VOID(logLevelTrace); FUNCTION_AUDIT_HELPER(); // Make sure the repo option is set for the stanza-delete command when more than one repo is configured or the first configured // repo is not key 1. if (!cfgCommandHelp() && cfgOptionValid(cfgOptRepo) && cfgCommand() == cfgCmdStanzaDelete && !cfgOptionTest(cfgOptRepo) && (cfgOptionGroupIdxTotal(cfgOptGrpRepo) > 1 || cfgOptionGroupIdxToKey(cfgOptGrpRepo, 0) != 1)) { THROW_FMT( OptionRequiredError, "%s command requires option: " CFGOPT_REPO "\n" "HINT: this command requires a specific repository to operate on", cfgCommandName()); } // If there is more than one repo configured if (cfgOptionGroupIdxTotal(cfgOptGrpRepo) > 1) { for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); optionIdx++) { // If the repo is local and either posix or cifs if (!cfgOptionIdxTest(cfgOptRepoHost, optionIdx) && (cfgOptionIdxStrId(cfgOptRepoType, optionIdx) == STORAGE_POSIX_TYPE || cfgOptionIdxStrId(cfgOptRepoType, optionIdx) == STORAGE_CIFS_TYPE)) { // Ensure a local repo does not have the same path as another local repo of the same type for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { if (optionIdx != repoIdx && !(cfgOptionIdxTest(cfgOptRepoHost, repoIdx)) && cfgOptionIdxStrId(cfgOptRepoType, optionIdx) == cfgOptionIdxStrId(cfgOptRepoType, repoIdx) && strEq(cfgOptionIdxStr(cfgOptRepoPath, optionIdx), cfgOptionIdxStr(cfgOptRepoPath, repoIdx))) { THROW_FMT( OptionInvalidValueError, "local %s and %s paths are both '%s' but must be different", cfgOptionGroupName(cfgOptGrpRepo, optionIdx), cfgOptionGroupName(cfgOptGrpRepo, repoIdx), strZ(cfgOptionIdxDisplay(cfgOptRepoPath, repoIdx))); } } } } } // Protocol timeout should be greater than db timeout if (cfgOptionTest(cfgOptDbTimeout) && cfgOptionTest(cfgOptProtocolTimeout) && cfgOptionInt64(cfgOptProtocolTimeout) <= cfgOptionInt64(cfgOptDbTimeout)) { // If protocol-timeout is default then increase it to be greater than db-timeout if (cfgOptionSource(cfgOptProtocolTimeout) == cfgSourceDefault) { cfgOptionSet( cfgOptProtocolTimeout, cfgSourceDefault, VARINT64(cfgOptionInt64(cfgOptDbTimeout) + (int64_t)(30 * MSEC_PER_SEC))); } else if (cfgOptionSource(cfgOptDbTimeout) == cfgSourceDefault) { const int64_t dbTimeout = cfgOptionInt64(cfgOptProtocolTimeout) - (int64_t)(30 * MSEC_PER_SEC); // Normally the protocol time will be greater than 45 seconds so db timeout can be at least 15 seconds if (dbTimeout >= (int64_t)(15 * MSEC_PER_SEC)) { cfgOptionSet(cfgOptDbTimeout, cfgSourceDefault, VARINT64(dbTimeout)); } // But in some test cases the protocol timeout will be very small so make db timeout half of protocol timeout else cfgOptionSet(cfgOptDbTimeout, cfgSourceDefault, VARINT64(cfgOptionInt64(cfgOptProtocolTimeout) / 2)); } else { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for '" CFGOPT_PROTOCOL_TIMEOUT "' option\nHINT '" CFGOPT_PROTOCOL_TIMEOUT "' option (%s)" " should be greater than '" CFGOPT_DB_TIMEOUT "' option (%s).", strZ(cfgOptionDisplay(cfgOptProtocolTimeout)), strZ(cfgOptionDisplay(cfgOptProtocolTimeout)), strZ(cfgOptionDisplay(cfgOptDbTimeout))); } } // Make sure that repo and pg host settings are not both set - cannot both be remote if (cfgOptionValid(cfgOptPgHost) && cfgOptionValid(cfgOptRepoHost)) { bool pgHostFound = false; for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpPg); optionIdx++) { if (cfgOptionIdxTest(cfgOptPgHost, optionIdx)) { pgHostFound = true; break; } } // If a pg-host was found, see if a repo-host is configured if (pgHostFound == true) { for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); optionIdx++) { if (cfgOptionIdxTest(cfgOptRepoHost, optionIdx)) THROW_FMT(ConfigError, "pg and repo hosts cannot both be configured as remote"); } } } // Warn when repo-retention-full is not set on a configured repo if (!cfgCommandHelp() && cfgOptionValid(cfgOptRepoRetentionFullType)) { for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); optionIdx++) { if (!(cfgOptionIdxTest(cfgOptRepoRetentionFull, optionIdx))) { LOG_WARN_FMT( "option '%s' is not set for '%s=%s', the repository may run out of space" "\nHINT: to retain full backups indefinitely (without warning), set option '%s' to the maximum.", cfgOptionIdxName(cfgOptRepoRetentionFull, optionIdx), cfgOptionIdxName(cfgOptRepoRetentionFullType, optionIdx), strZ(cfgOptionIdxDisplay(cfgOptRepoRetentionFullType, optionIdx)), cfgOptionIdxName(cfgOptRepoRetentionFull, optionIdx)); } } } // If archive retention is valid for the command, then set archive settings if (cfgOptionValid(cfgOptRepoRetentionArchive)) { // For each possible repo, check and adjust the settings as appropriate for (unsigned int optionIdx = 0; optionIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); optionIdx++) { const BackupType archiveRetentionType = (BackupType)cfgOptionIdxStrId(cfgOptRepoRetentionArchiveType, optionIdx); // If the archive retention is not explicitly set then determine what it should be defaulted to if (!cfgOptionIdxTest(cfgOptRepoRetentionArchive, optionIdx)) { // If repo-retention-archive-type is default (full), then if repo-retention-full is set, set the // repo-retention-archive to this value when retention-full-type is 'count', else ignore archiving. If // retention-full-type is 'time' then the expire command will default the archive retention accordingly. const String *const msgArchiveOff = strNewFmt( "WAL segments will not be expired: option '%s=%s' but", cfgOptionIdxName(cfgOptRepoRetentionArchiveType, optionIdx), zNewStrId(archiveRetentionType)); switch (archiveRetentionType) { case backupTypeFull: { if (cfgOptionIdxSeq(cfgOptRepoRetentionFullType, optionIdx) == CFGOPTVAL_REPO_RETENTION_FULL_TYPE_COUNT && cfgOptionIdxTest(cfgOptRepoRetentionFull, optionIdx)) { cfgOptionIdxSet( cfgOptRepoRetentionArchive, optionIdx, cfgSourceDefault, VARINT64(cfgOptionIdxInt64(cfgOptRepoRetentionFull, optionIdx))); } break; } case backupTypeDiff: { // if repo-retention-diff is set then user must have set it if (cfgOptionIdxTest(cfgOptRepoRetentionDiff, optionIdx)) { cfgOptionIdxSet( cfgOptRepoRetentionArchive, optionIdx, cfgSourceDefault, VARINT64(cfgOptionIdxInt64(cfgOptRepoRetentionDiff, optionIdx))); } else { LOG_WARN_FMT( "%s neither option '%s' nor option '%s' is set", strZ(msgArchiveOff), cfgOptionIdxName(cfgOptRepoRetentionArchive, optionIdx), cfgOptionIdxName(cfgOptRepoRetentionDiff, optionIdx)); } break; } case backupTypeIncr: LOG_WARN_FMT( "%s option '%s' is not set", strZ(msgArchiveOff), cfgOptionIdxName(cfgOptRepoRetentionArchive, optionIdx)); break; } } else { // If repo-retention-archive is set then check repo-retention-archive-type and issue a warning if the // corresponding setting is UNDEF since UNDEF means backups will not be expired but they should be in the // practice of setting this value even though expiring the archive itself is OK and will be performed. if (archiveRetentionType == backupTypeDiff && !cfgOptionIdxTest(cfgOptRepoRetentionDiff, optionIdx)) { LOG_WARN_FMT( "option '%s' is not set for '%s=" CFGOPTVAL_TYPE_DIFF_Z "'\n" "HINT: to retain differential backups indefinitely (without warning), set option '%s' to the maximum.", cfgOptionIdxName(cfgOptRepoRetentionDiff, optionIdx), cfgOptionIdxName(cfgOptRepoRetentionArchiveType, optionIdx), cfgOptionIdxName(cfgOptRepoRetentionDiff, optionIdx)); } } } } // For each possible repo, error if an S3 bucket name contains dots when using host style URIs for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { if (cfgOptionIdxTest(cfgOptRepoS3Bucket, repoIdx) && cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx) && cfgOptionIdxSeq(cfgOptRepoS3UriStyle, repoIdx) == CFGOPTVAL_REPO_S3_URI_STYLE_HOST && strChr(cfgOptionIdxStr(cfgOptRepoS3Bucket, repoIdx), '.') != -1) { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for option '%s'\n" "HINT: RFC-2818 forbids dots in wildcard matches.\n" "HINT: TLS/SSL verification cannot proceed with this bucket name.\n" "HINT: remove dots from the bucket name.", strZ(cfgOptionIdxDisplay(cfgOptRepoS3Bucket, repoIdx)), cfgOptionIdxName(cfgOptRepoS3Bucket, repoIdx)); } } // Check/update compress-type if compress is valid. There should be no references to the compress option outside this block. if (cfgOptionValid(cfgOptCompress)) { if (cfgOptionSource(cfgOptCompress) != cfgSourceDefault) { if (cfgOptionSource(cfgOptCompressType) != cfgSourceDefault) { LOG_WARN( "'" CFGOPT_COMPRESS "' and '" CFGOPT_COMPRESS_TYPE "' options should not both be set\n" "HINT: '" CFGOPT_COMPRESS_TYPE "' is preferred and '" CFGOPT_COMPRESS "' is deprecated."); } // Set compress-type to none. Eventually the compress option will be deprecated and removed so this reduces code churn // when that happens. if (!cfgOptionBool(cfgOptCompress) && cfgOptionSource(cfgOptCompressType) == cfgSourceDefault) { cfgOptionSet(cfgOptCompressType, cfgSourceParam, VARUINT64(CFGOPTVAL_COMPRESS_TYPE_NONE)); cfgOptionSet(cfgOptCompressLevel, cfgSourceDefault, VARINT64(0)); } } // Now invalidate compress so it can't be used and won't be passed to child processes cfgOptionInvalidate(cfgOptCompress); cfgOptionSet(cfgOptCompress, cfgSourceDefault, NULL); } // Error if repo-sftp--host-key-check-type is explicitly set to anything other than fingerprint and repo-sftp-host-fingerprint // is also specified. For backward compatibility we need to allow repo-sftp-host-fingerprint when // repo-sftp-host-key-check-type defaults to yes, but emit a warning to let the user know to change the configuration. Also // set repo-sftp-host-key-check-type=fingerprint so other code does not need to know about this exception. for (unsigned int repoIdx = 0; repoIdx < cfgOptionGroupIdxTotal(cfgOptGrpRepo); repoIdx++) { if (cfgOptionIdxTest(cfgOptRepoSftpHostKeyCheckType, repoIdx)) { if (cfgOptionIdxTest(cfgOptRepoSftpHostFingerprint, repoIdx)) { if (cfgOptionIdxSource(cfgOptRepoSftpHostKeyCheckType, repoIdx) == cfgSourceDefault) { LOG_WARN_FMT( "option '%s' without option '%s' = '" CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z "' is" " deprecated\n" "HINT: set option '%s=" CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z "'", cfgOptionIdxName(cfgOptRepoSftpHostFingerprint, repoIdx), cfgOptionIdxName(cfgOptRepoSftpHostKeyCheckType, repoIdx), cfgOptionIdxName(cfgOptRepoSftpHostKeyCheckType, repoIdx)); cfgOptionIdxSet( cfgOptRepoSftpHostKeyCheckType, repoIdx, cfgSourceDefault, VARSTRZ(CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z)); } else { if (cfgOptionIdxStrId(cfgOptRepoSftpHostKeyCheckType, repoIdx) != CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT) { THROW_FMT( OptionInvalidError, "option '%s' not valid without option '%s' = '" CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT_Z "'", cfgOptionIdxName(cfgOptRepoSftpHostFingerprint, repoIdx), cfgOptionIdxName(cfgOptRepoSftpHostKeyCheckType, repoIdx)); } } } else { if (cfgOptionIdxStrId(cfgOptRepoSftpHostKeyCheckType, repoIdx) == CFGOPTVAL_REPO_SFTP_HOST_KEY_CHECK_TYPE_FINGERPRINT) { THROW_FMT( OptionRequiredError, "%s command requires option: %s", cfgCommandName(), cfgOptionIdxName(cfgOptRepoSftpHostFingerprint, repoIdx)); } } } } // A repo must be specified when targeting time. Not all repo types support versioning so rather than try to skip repos in that // case it seems to be easier to just target a specific repo. Also, depending on the type of corruption, different repos might // require different target times. if (cfgOptionTest(cfgOptRepoTargetTime) && cfgOptionSource(cfgOptRepo) == cfgSourceDefault) THROW_FMT(OptionInvalidError, "option '" CFGOPT_REPO_TARGET_TIME "' not valid without option '" CFGOPT_REPO "'"); // The stop-auto option is deprecated if (cfgOptionValid(cfgOptStopAuto) && cfgOptionSource(cfgOptStopAuto) != cfgSourceDefault) { LOG_WARN( "option '" CFGOPT_STOP_AUTO "' is deprecated\n" "HINT: all supported versions use non-exclusive backup.\n" "HINT: stop using this option to avoid an error when it is removed."); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * cfgLoadLogFileName(const ConfigCommandRole commandRole) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(ENUM, commandRole); FUNCTION_LOG_END(); // Construct log filename prefix String *const result = strCatFmt( strNew(), "%s/%s-%s", strZ(cfgOptionStr(cfgOptLogPath)), cfgOptionTest(cfgOptStanza) ? strZ(cfgOptionStr(cfgOptStanza)) : "all", cfgCommandName()); // ??? Append async for local/remote archive async commands. It would be good to find a more generic way to do this in case the // async role is added to more commands. if (commandRole == cfgCmdRoleLocal || commandRole == cfgCmdRoleRemote) { if (cfgOptionValid(cfgOptArchiveAsync) && cfgOptionBool(cfgOptArchiveAsync)) strCatZ(result, "-async"); } // Add command role if it is not main if (commandRole != cfgCmdRoleMain) strCatFmt(result, "-%s", strZ(cfgParseCommandRoleStr(commandRole))); // Add process id if local or remote role if (commandRole == cfgCmdRoleLocal || commandRole == cfgCmdRoleRemote) strCatFmt(result, "-%03u", cfgOptionUInt(cfgOptProcess)); // Add extension strCatZ(result, ".log"); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void cfgLoadLogFile(void) { if (cfgLogFile() && !cfgCommandHelp()) { MEM_CONTEXT_TEMP_BEGIN() { // Attempt to open log file if (!logFileSet(strZ(cfgLoadLogFileName(cfgCommandRole())))) cfgOptionSet(cfgOptLogLevelFile, cfgSourceParam, VARUINT64(CFGOPTVAL_LOG_LEVEL_CONSOLE_OFF_STRID)); } MEM_CONTEXT_TEMP_END(); } } /**********************************************************************************************************************************/ FN_EXTERN void cfgLoad(const unsigned int argListSize, const char *argList[]) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, argListSize); FUNCTION_LOG_PARAM(CHARPY, argList); FUNCTION_LOG_END(); ASSERT(argListSize > 0); ASSERT(argList != NULL); // Store arguments configLoadLocal.argListSize = argListSize; configLoadLocal.argList = argList; MEM_CONTEXT_TEMP_BEGIN() { // Parse config from command line and config file cfgParseP(storageLocal(), configLoadLocal.argListSize, configLoadLocal.argList); // Initialize dry-run mode for storage when valid for the current command storageHelperDryRunInit(cfgOptionValid(cfgOptDryRun) && cfgOptionBool(cfgOptDryRun)); // Load the log settings cfgLoadLogSetting(); // Error when running as the root user unless it is explicitly allowed. Running as root can create files owned by root that // the regular user is then unable to access, which is a common cause of hard-to-diagnose failures. The allow-root option // defaults to true only for the restore command, which is designed to run as root and manages file ownership carefully. The // check applies to all command roles so a non-root command cannot run as root on a remote. It is skipped for help/version // (where the option is not valid) since those may reasonably be run as root. if (!cfgCommandHelp() && cfgOptionValid(cfgOptAllowRoot) && !cfgOptionBool(cfgOptAllowRoot)) { userInit(); if (userRoot()) { THROW_FMT( OptionInvalidError, "the '%s' command must not be run as the root user\n" "HINT: running as root can create files that the regular user cannot access, causing later commands to fail.\n" "HINT: set the '" CFGOPT_ALLOW_ROOT "' option to allow running this command as root.", cfgCommandName()); } } // Neutralize the umask to make the repository file/path modes more consistent if (cfgOptionValid(cfgOptNeutralUmask) && cfgOptionBool(cfgOptNeutralUmask)) umask(0000); if (cfgOptionValid(cfgOptPriority) && cfgOptionSource(cfgOptPriority) != cfgSourceDefault) { // Sign conversion is ignored here due to type conflicts between setpriority() and getpid() on different platforms. // Linux returns _pid_t (int) from getpid() but accepts id_t (unsigned int) in setpriority(). FreeBSD (but not MacOS) // uses int for both. Presumably implicit conversion will work appropriately on each platform so just let that happen. #pragma GCC diagnostic push #pragma GCC diagnostic ignored "-Wsign-conversion" THROW_ON_SYS_ERROR( setpriority(PRIO_PROCESS, getpid(), cfgOptionInt(cfgOptPriority)) == -1, KernelError, "unable to set process priority"); #pragma GCC diagnostic pop } // Initialize TCP settings if (cfgOptionValid(cfgOptSckKeepAlive)) { sckInit( cfgOptionBool(cfgOptSckBlock), cfgOptionBool(cfgOptSckKeepAlive), cfgOptionTest(cfgOptTcpKeepAliveCount) ? cfgOptionInt(cfgOptTcpKeepAliveCount) : 0, cfgOptionTest(cfgOptTcpKeepAliveIdle) ? cfgOptionInt(cfgOptTcpKeepAliveIdle) : 0, cfgOptionTest(cfgOptTcpKeepAliveInterval) ? cfgOptionInt(cfgOptTcpKeepAliveInterval) : 0); } // Initialize TLS settings if (cfgOptionValid(cfgOptTlsCipher12)) { ASSERT(cfgOptionValid(cfgOptTlsCipher13)); tlsInit(cfgOptionStrNull(cfgOptTlsCipher12), cfgOptionStrNull(cfgOptTlsCipher13)); } // Set IO buffer size (use the default for help to lower memory usage) if (cfgOptionValid(cfgOptBufferSize) && !cfgCommandHelp()) ioBufferSizeSet(cfgOptionUInt(cfgOptBufferSize)); // Set IO timeout if (cfgOptionValid(cfgOptIoTimeout)) ioTimeoutMsSet(cfgOptionUInt64(cfgOptIoTimeout)); // Open the log file if this command logs to a file cfgLoadLogFile(); // Create the exec-id used to identify all locals and remotes spawned by this process. This allows lock contention to be // easily resolved and makes it easier to associate processes from various logs. if (cfgOptionValid(cfgOptExecId) && !cfgOptionTest(cfgOptExecId)) { // Generate some random bytes uint32_t execRandom; cryptoRandomBytes((uint8_t *)&execRandom, sizeof(execRandom)); // Format a string with the pid and the random bytes to serve as the exec id cfgOptionSet(cfgOptExecId, cfgSourceParam, VARSTR(strNewFmt("%d-%08x", getpid(), execRandom))); } // Begin the command cmdBegin(); // Initialize the lock module if (cfgOptionTest(cfgOptLockPath)) lockInit(cfgOptionStr(cfgOptLockPath), cfgOptionStr(cfgOptExecId)); // Acquire a lock if this command requires a lock if (cfgLockType() != lockTypeNone && !cfgCommandHelp() && cfgLockRequired()) cmdLockAcquireP(); // Update options that have complex rules cfgLoadUpdateOption(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void cfgLoadStanza(const String *const stanza) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, stanza); FUNCTION_LOG_END(); ASSERT(stanza != NULL); ASSERT(configLoadLocal.argListSize > 0); MEM_CONTEXT_TEMP_BEGIN() { // Store the exec id so it can be preserved after reload const Variant *const execId = varNewStr(cfgOptionStr(cfgOptExecId)); // Make a copy of the arguments and add the stanza (this assumes the stanza was not originally specified) StringList *const argListNew = strLstNew(); for (unsigned int argListIdx = 0; argListIdx < configLoadLocal.argListSize; argListIdx++) strLstAddZ(argListNew, configLoadLocal.argList[argListIdx]); strLstAddFmt(argListNew, "--" CFGOPT_STANZA "=%s", strZ(stanza)); // Parse config from command line and config file cfgParseP(storageLocal(), strLstSize(argListNew), strLstPtr(argListNew), .noConfigLoad = true, .noResetLogLevel = true); // Update options that have complex rules cfgLoadUpdateOption(); // Set execId to prior value cfgOptionSet(cfgOptExecId, cfgSourceParam, execId); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/load.h0000664000175100017510000000237515240547630016546 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Load ***********************************************************************************************************************************/ #ifndef CONFIG_LOAD_H #define CONFIG_LOAD_H #include #include "config/config.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Load the configuration FN_EXTERN void cfgLoad(unsigned int argListSize, const char *argList[]); // Load the configuration using the specified stanza FN_EXTERN void cfgLoadStanza(const String *stanza); // Generate log file path and name. Only the command role is configurable here because log settings may vary between commands. FN_EXTERN String *cfgLoadLogFileName(ConfigCommandRole commandRole); // Attempt to set the log file and turn file logging off if the file cannot be opened FN_EXTERN void cfgLoadLogFile(void); // Update options that have complex rules FN_EXTERN void cfgLoadUpdateOption(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/parse.auto.c.inc0000664000175100017510000611656115240547630020464 0ustar00runnerrunner/*********************************************************************************************************************************** Config Parse Rules Automatically generated by 'build-code config' -- do not modify directly. ***********************************************************************************************************************************/ /*********************************************************************************************************************************** Rule Strings ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_STR(value) PARSE_RULE_U32_2(parseRuleValStr##value) #define PARSE_RULE_VAL_STRID(value) PARSE_RULE_VAL_STR(QT_##value##_QT) static const StringPubConst parseRuleValueStr[] = { PARSE_RULE_STRPUB("-1"), // val/str PARSE_RULE_STRPUB("-20"), // val/str PARSE_RULE_STRPUB("-5"), // val/str PARSE_RULE_STRPUB("-7"), // val/str PARSE_RULE_STRPUB("/tmp/pgbackrest"), // val/str PARSE_RULE_STRPUB("/var/lib/pgbackrest"), // val/str PARSE_RULE_STRPUB("/var/log/pgbackrest"), // val/str PARSE_RULE_STRPUB("/var/spool/pgbackrest"), // val/str PARSE_RULE_STRPUB("0"), // val/str PARSE_RULE_STRPUB("0B"), // val/str PARSE_RULE_STRPUB("0s"), // val/str PARSE_RULE_STRPUB("1"), // val/str PARSE_RULE_STRPUB("100ms"), // val/str PARSE_RULE_STRPUB("1024"), // val/str PARSE_RULE_STRPUB("12"), // val/str PARSE_RULE_STRPUB("128KiB"), // val/str PARSE_RULE_STRPUB("128MiB"), // val/str PARSE_RULE_STRPUB("15m"), // val/str PARSE_RULE_STRPUB("15s"), // val/str PARSE_RULE_STRPUB("16GiB"), // val/str PARSE_RULE_STRPUB("16KiB"), // val/str PARSE_RULE_STRPUB("16MiB"), // val/str PARSE_RULE_STRPUB("19"), // val/str PARSE_RULE_STRPUB("1B"), // val/str PARSE_RULE_STRPUB("1GiB"), // val/str PARSE_RULE_STRPUB("1MiB"), // val/str PARSE_RULE_STRPUB("1PiB"), // val/str PARSE_RULE_STRPUB("1TiB"), // val/str PARSE_RULE_STRPUB("1d"), // val/str PARSE_RULE_STRPUB("1h"), // val/str PARSE_RULE_STRPUB("1m"), // val/str PARSE_RULE_STRPUB("2"), // val/str PARSE_RULE_STRPUB("20MiB"), // val/str PARSE_RULE_STRPUB("22"), // val/str PARSE_RULE_STRPUB("256"), // val/str PARSE_RULE_STRPUB("256KiB"), // val/str PARSE_RULE_STRPUB("2MiB"), // val/str PARSE_RULE_STRPUB("3"), // val/str PARSE_RULE_STRPUB("30m"), // val/str PARSE_RULE_STRPUB("31m"), // val/str PARSE_RULE_STRPUB("32"), // val/str PARSE_RULE_STRPUB("32KiB"), // val/str PARSE_RULE_STRPUB("360"), // val/str PARSE_RULE_STRPUB("3600"), // val/str PARSE_RULE_STRPUB("443"), // val/str PARSE_RULE_STRPUB("4MiB"), // val/str PARSE_RULE_STRPUB("4PiB"), // val/str PARSE_RULE_STRPUB("512KiB"), // val/str PARSE_RULE_STRPUB("5432"), // val/str PARSE_RULE_STRPUB("5MiB"), // val/str PARSE_RULE_STRPUB("6"), // val/str PARSE_RULE_STRPUB("64KiB"), // val/str PARSE_RULE_STRPUB("65535"), // val/str PARSE_RULE_STRPUB("7d"), // val/str PARSE_RULE_STRPUB("8432"), // val/str PARSE_RULE_STRPUB("8KiB"), // val/str PARSE_RULE_STRPUB("8MiB"), // val/str PARSE_RULE_STRPUB("9"), // val/str PARSE_RULE_STRPUB("900"), // val/str PARSE_RULE_STRPUB("999"), // val/str PARSE_RULE_STRPUB("9999999"), // val/str PARSE_RULE_STRPUB("accept-new"), // val/str PARSE_RULE_STRPUB("aes-256-cbc"), // val/str PARSE_RULE_STRPUB("asc"), // val/str PARSE_RULE_STRPUB("auto"), // val/str PARSE_RULE_STRPUB("azure"), // val/str PARSE_RULE_STRPUB("blob.core.windows.net"), // val/str PARSE_RULE_STRPUB("bz2"), // val/str PARSE_RULE_STRPUB("cifs"), // val/str PARSE_RULE_STRPUB("count"), // val/str PARSE_RULE_STRPUB("debug"), // val/str PARSE_RULE_STRPUB("default"), // val/str PARSE_RULE_STRPUB("desc"), // val/str PARSE_RULE_STRPUB("detail"), // val/str PARSE_RULE_STRPUB("diff"), // val/str PARSE_RULE_STRPUB("error"), // val/str PARSE_RULE_STRPUB("fingerprint"), // val/str PARSE_RULE_STRPUB("full"), // val/str PARSE_RULE_STRPUB("gcs"), // val/str PARSE_RULE_STRPUB("gz"), // val/str PARSE_RULE_STRPUB("host"), // val/str PARSE_RULE_STRPUB("immediate"), // val/str PARSE_RULE_STRPUB("incr"), // val/str PARSE_RULE_STRPUB("info"), // val/str PARSE_RULE_STRPUB("json"), // val/str PARSE_RULE_STRPUB("latest"), // val/str PARSE_RULE_STRPUB("localhost"), // val/str PARSE_RULE_STRPUB("lsn"), // val/str PARSE_RULE_STRPUB("lz4"), // val/str PARSE_RULE_STRPUB("md5"), // val/str PARSE_RULE_STRPUB("n"), // val/str PARSE_RULE_STRPUB("name"), // val/str PARSE_RULE_STRPUB("none"), // val/str PARSE_RULE_STRPUB("num"), // val/str PARSE_RULE_STRPUB("off"), // val/str PARSE_RULE_STRPUB("path"), // val/str PARSE_RULE_STRPUB("pause"), // val/str PARSE_RULE_STRPUB("pg"), // val/str PARSE_RULE_STRPUB("pgbackrest"), // val/str PARSE_RULE_STRPUB("pod-id"), // val/str PARSE_RULE_STRPUB("posix"), // val/str PARSE_RULE_STRPUB("postgres"), // val/str PARSE_RULE_STRPUB("prefer"), // val/str PARSE_RULE_STRPUB("preserve"), // val/str PARSE_RULE_STRPUB("process"), // val/str PARSE_RULE_STRPUB("progress"), // val/str PARSE_RULE_STRPUB("promote"), // val/str PARSE_RULE_STRPUB("repo"), // val/str PARSE_RULE_STRPUB("s3"), // val/str PARSE_RULE_STRPUB("sas"), // val/str PARSE_RULE_STRPUB("service"), // val/str PARSE_RULE_STRPUB("sftp"), // val/str PARSE_RULE_STRPUB("sha1"), // val/str PARSE_RULE_STRPUB("sha256"), // val/str PARSE_RULE_STRPUB("shared"), // val/str PARSE_RULE_STRPUB("shutdown"), // val/str PARSE_RULE_STRPUB("ssh"), // val/str PARSE_RULE_STRPUB("standby"), // val/str PARSE_RULE_STRPUB("storage.googleapis.com"), // val/str PARSE_RULE_STRPUB("strict"), // val/str PARSE_RULE_STRPUB("sts.amazonaws.com"), // val/str PARSE_RULE_STRPUB("text"), // val/str PARSE_RULE_STRPUB("time"), // val/str PARSE_RULE_STRPUB("tls"), // val/str PARSE_RULE_STRPUB("token"), // val/str PARSE_RULE_STRPUB("trace"), // val/str PARSE_RULE_STRPUB("warn"), // val/str PARSE_RULE_STRPUB("web-id"), // val/str PARSE_RULE_STRPUB("xid"), // val/str PARSE_RULE_STRPUB("y"), // val/str PARSE_RULE_STRPUB("zst"), // val/str PARSE_RULE_STRPUB(CFGOPTDEF_CONFIG_PATH), // val/str PARSE_RULE_STRPUB(CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_FILE), // val/str PARSE_RULE_STRPUB(CFGOPTDEF_CONFIG_PATH "/" PROJECT_CONFIG_INCLUDE_PATH), // val/str }; typedef enum { parseRuleValStrQT_DS_1_QT, // val/str/enum parseRuleValStrQT_DS_20_QT, // val/str/enum parseRuleValStrQT_DS_5_QT, // val/str/enum parseRuleValStrQT_DS_7_QT, // val/str/enum parseRuleValStrQT_FS_tmp_FS_pgbackrest_QT, // val/str/enum parseRuleValStrQT_FS_var_FS_lib_FS_pgbackrest_QT, // val/str/enum parseRuleValStrQT_FS_var_FS_log_FS_pgbackrest_QT, // val/str/enum parseRuleValStrQT_FS_var_FS_spool_FS_pgbackrest_QT, // val/str/enum parseRuleValStrQT_0_QT, // val/str/enum parseRuleValStrQT_0B_QT, // val/str/enum parseRuleValStrQT_0s_QT, // val/str/enum parseRuleValStrQT_1_QT, // val/str/enum parseRuleValStrQT_100ms_QT, // val/str/enum parseRuleValStrQT_1024_QT, // val/str/enum parseRuleValStrQT_12_QT, // val/str/enum parseRuleValStrQT_128KiB_QT, // val/str/enum parseRuleValStrQT_128MiB_QT, // val/str/enum parseRuleValStrQT_15m_QT, // val/str/enum parseRuleValStrQT_15s_QT, // val/str/enum parseRuleValStrQT_16GiB_QT, // val/str/enum parseRuleValStrQT_16KiB_QT, // val/str/enum parseRuleValStrQT_16MiB_QT, // val/str/enum parseRuleValStrQT_19_QT, // val/str/enum parseRuleValStrQT_1B_QT, // val/str/enum parseRuleValStrQT_1GiB_QT, // val/str/enum parseRuleValStrQT_1MiB_QT, // val/str/enum parseRuleValStrQT_1PiB_QT, // val/str/enum parseRuleValStrQT_1TiB_QT, // val/str/enum parseRuleValStrQT_1d_QT, // val/str/enum parseRuleValStrQT_1h_QT, // val/str/enum parseRuleValStrQT_1m_QT, // val/str/enum parseRuleValStrQT_2_QT, // val/str/enum parseRuleValStrQT_20MiB_QT, // val/str/enum parseRuleValStrQT_22_QT, // val/str/enum parseRuleValStrQT_256_QT, // val/str/enum parseRuleValStrQT_256KiB_QT, // val/str/enum parseRuleValStrQT_2MiB_QT, // val/str/enum parseRuleValStrQT_3_QT, // val/str/enum parseRuleValStrQT_30m_QT, // val/str/enum parseRuleValStrQT_31m_QT, // val/str/enum parseRuleValStrQT_32_QT, // val/str/enum parseRuleValStrQT_32KiB_QT, // val/str/enum parseRuleValStrQT_360_QT, // val/str/enum parseRuleValStrQT_3600_QT, // val/str/enum parseRuleValStrQT_443_QT, // val/str/enum parseRuleValStrQT_4MiB_QT, // val/str/enum parseRuleValStrQT_4PiB_QT, // val/str/enum parseRuleValStrQT_512KiB_QT, // val/str/enum parseRuleValStrQT_5432_QT, // val/str/enum parseRuleValStrQT_5MiB_QT, // val/str/enum parseRuleValStrQT_6_QT, // val/str/enum parseRuleValStrQT_64KiB_QT, // val/str/enum parseRuleValStrQT_65535_QT, // val/str/enum parseRuleValStrQT_7d_QT, // val/str/enum parseRuleValStrQT_8432_QT, // val/str/enum parseRuleValStrQT_8KiB_QT, // val/str/enum parseRuleValStrQT_8MiB_QT, // val/str/enum parseRuleValStrQT_9_QT, // val/str/enum parseRuleValStrQT_900_QT, // val/str/enum parseRuleValStrQT_999_QT, // val/str/enum parseRuleValStrQT_9999999_QT, // val/str/enum parseRuleValStrQT_accept_DS_new_QT, // val/str/enum parseRuleValStrQT_aes_DS_256_DS_cbc_QT, // val/str/enum parseRuleValStrQT_asc_QT, // val/str/enum parseRuleValStrQT_auto_QT, // val/str/enum parseRuleValStrQT_azure_QT, // val/str/enum parseRuleValStrQT_blob_DT_core_DT_windows_DT_net_QT, // val/str/enum parseRuleValStrQT_bz2_QT, // val/str/enum parseRuleValStrQT_cifs_QT, // val/str/enum parseRuleValStrQT_count_QT, // val/str/enum parseRuleValStrQT_debug_QT, // val/str/enum parseRuleValStrQT_default_QT, // val/str/enum parseRuleValStrQT_desc_QT, // val/str/enum parseRuleValStrQT_detail_QT, // val/str/enum parseRuleValStrQT_diff_QT, // val/str/enum parseRuleValStrQT_error_QT, // val/str/enum parseRuleValStrQT_fingerprint_QT, // val/str/enum parseRuleValStrQT_full_QT, // val/str/enum parseRuleValStrQT_gcs_QT, // val/str/enum parseRuleValStrQT_gz_QT, // val/str/enum parseRuleValStrQT_host_QT, // val/str/enum parseRuleValStrQT_immediate_QT, // val/str/enum parseRuleValStrQT_incr_QT, // val/str/enum parseRuleValStrQT_info_QT, // val/str/enum parseRuleValStrQT_json_QT, // val/str/enum parseRuleValStrQT_latest_QT, // val/str/enum parseRuleValStrQT_localhost_QT, // val/str/enum parseRuleValStrQT_lsn_QT, // val/str/enum parseRuleValStrQT_lz4_QT, // val/str/enum parseRuleValStrQT_md5_QT, // val/str/enum parseRuleValStrQT_n_QT, // val/str/enum parseRuleValStrQT_name_QT, // val/str/enum parseRuleValStrQT_none_QT, // val/str/enum parseRuleValStrQT_num_QT, // val/str/enum parseRuleValStrQT_off_QT, // val/str/enum parseRuleValStrQT_path_QT, // val/str/enum parseRuleValStrQT_pause_QT, // val/str/enum parseRuleValStrQT_pg_QT, // val/str/enum parseRuleValStrQT_pgbackrest_QT, // val/str/enum parseRuleValStrQT_pod_DS_id_QT, // val/str/enum parseRuleValStrQT_posix_QT, // val/str/enum parseRuleValStrQT_postgres_QT, // val/str/enum parseRuleValStrQT_prefer_QT, // val/str/enum parseRuleValStrQT_preserve_QT, // val/str/enum parseRuleValStrQT_process_QT, // val/str/enum parseRuleValStrQT_progress_QT, // val/str/enum parseRuleValStrQT_promote_QT, // val/str/enum parseRuleValStrQT_repo_QT, // val/str/enum parseRuleValStrQT_s3_QT, // val/str/enum parseRuleValStrQT_sas_QT, // val/str/enum parseRuleValStrQT_service_QT, // val/str/enum parseRuleValStrQT_sftp_QT, // val/str/enum parseRuleValStrQT_sha1_QT, // val/str/enum parseRuleValStrQT_sha256_QT, // val/str/enum parseRuleValStrQT_shared_QT, // val/str/enum parseRuleValStrQT_shutdown_QT, // val/str/enum parseRuleValStrQT_ssh_QT, // val/str/enum parseRuleValStrQT_standby_QT, // val/str/enum parseRuleValStrQT_storage_DT_googleapis_DT_com_QT, // val/str/enum parseRuleValStrQT_strict_QT, // val/str/enum parseRuleValStrQT_sts_DT_amazonaws_DT_com_QT, // val/str/enum parseRuleValStrQT_text_QT, // val/str/enum parseRuleValStrQT_time_QT, // val/str/enum parseRuleValStrQT_tls_QT, // val/str/enum parseRuleValStrQT_token_QT, // val/str/enum parseRuleValStrQT_trace_QT, // val/str/enum parseRuleValStrQT_warn_QT, // val/str/enum parseRuleValStrQT_web_DS_id_QT, // val/str/enum parseRuleValStrQT_xid_QT, // val/str/enum parseRuleValStrQT_y_QT, // val/str/enum parseRuleValStrQT_zst_QT, // val/str/enum parseRuleValStrCFGOPTDEF_CONFIG_PATH, // val/str/enum parseRuleValStrCFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE, // val/str/enum parseRuleValStrCFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH, // val/str/enum } ParseRuleValueStr; /*********************************************************************************************************************************** Rule Ints ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_INT(value) PARSE_RULE_U32_1(parseRuleValInt##value) static const int parseRuleValueInt[] = { -20, // val/int -7, // val/int -5, // val/int -1, // val/int 0, // val/int 1, // val/int 2, // val/int 3, // val/int 6, // val/int 9, // val/int 12, // val/int 19, // val/int 22, // val/int 32, // val/int 256, // val/int 360, // val/int 443, // val/int 900, // val/int 999, // val/int 1024, // val/int 3600, // val/int 5432, // val/int 8432, // val/int 65535, // val/int 9999999, // val/int }; static const uint8_t parseRuleValueIntStrMap[] = { parseRuleValStrQT_DS_20_QT, // val/int/strmap parseRuleValStrQT_DS_7_QT, // val/int/strmap parseRuleValStrQT_DS_5_QT, // val/int/strmap parseRuleValStrQT_DS_1_QT, // val/int/strmap parseRuleValStrQT_0_QT, // val/int/strmap parseRuleValStrQT_1_QT, // val/int/strmap parseRuleValStrQT_2_QT, // val/int/strmap parseRuleValStrQT_3_QT, // val/int/strmap parseRuleValStrQT_6_QT, // val/int/strmap parseRuleValStrQT_9_QT, // val/int/strmap parseRuleValStrQT_12_QT, // val/int/strmap parseRuleValStrQT_19_QT, // val/int/strmap parseRuleValStrQT_22_QT, // val/int/strmap parseRuleValStrQT_32_QT, // val/int/strmap parseRuleValStrQT_256_QT, // val/int/strmap parseRuleValStrQT_360_QT, // val/int/strmap parseRuleValStrQT_443_QT, // val/int/strmap parseRuleValStrQT_900_QT, // val/int/strmap parseRuleValStrQT_999_QT, // val/int/strmap parseRuleValStrQT_1024_QT, // val/int/strmap parseRuleValStrQT_3600_QT, // val/int/strmap parseRuleValStrQT_5432_QT, // val/int/strmap parseRuleValStrQT_8432_QT, // val/int/strmap parseRuleValStrQT_65535_QT, // val/int/strmap parseRuleValStrQT_9999999_QT, // val/int/strmap }; typedef enum { parseRuleValIntDS_20, // val/int/enum parseRuleValIntDS_7, // val/int/enum parseRuleValIntDS_5, // val/int/enum parseRuleValIntDS_1, // val/int/enum parseRuleValInt0, // val/int/enum parseRuleValInt1, // val/int/enum parseRuleValInt2, // val/int/enum parseRuleValInt3, // val/int/enum parseRuleValInt6, // val/int/enum parseRuleValInt9, // val/int/enum parseRuleValInt12, // val/int/enum parseRuleValInt19, // val/int/enum parseRuleValInt22, // val/int/enum parseRuleValInt32, // val/int/enum parseRuleValInt256, // val/int/enum parseRuleValInt360, // val/int/enum parseRuleValInt443, // val/int/enum parseRuleValInt900, // val/int/enum parseRuleValInt999, // val/int/enum parseRuleValInt1024, // val/int/enum parseRuleValInt3600, // val/int/enum parseRuleValInt5432, // val/int/enum parseRuleValInt8432, // val/int/enum parseRuleValInt65535, // val/int/enum parseRuleValInt9999999, // val/int/enum } ParseRuleValueInt; /*********************************************************************************************************************************** Rule Sizes ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_SIZE(value) PARSE_RULE_U32_1(parseRuleValSize##value) static const int64_t parseRuleValueSize[] = { 0, // val/size 1, // val/size 8192, // val/size 16384, // val/size 32768, // val/size 65536, // val/size 131072, // val/size 262144, // val/size 524288, // val/size 1048576, // val/size 2097152, // val/size 4194304, // val/size 5242880, // val/size 8388608, // val/size 16777216, // val/size 20971520, // val/size 134217728, // val/size 1073741824, // val/size 17179869184, // val/size 1099511627776, // val/size 1125899906842624, // val/size 4503599627370496, // val/size }; static const uint8_t parseRuleValueSizeStrMap[] = { parseRuleValStrQT_0B_QT, // val/size/strmap parseRuleValStrQT_1B_QT, // val/size/strmap parseRuleValStrQT_8KiB_QT, // val/size/strmap parseRuleValStrQT_16KiB_QT, // val/size/strmap parseRuleValStrQT_32KiB_QT, // val/size/strmap parseRuleValStrQT_64KiB_QT, // val/size/strmap parseRuleValStrQT_128KiB_QT, // val/size/strmap parseRuleValStrQT_256KiB_QT, // val/size/strmap parseRuleValStrQT_512KiB_QT, // val/size/strmap parseRuleValStrQT_1MiB_QT, // val/size/strmap parseRuleValStrQT_2MiB_QT, // val/size/strmap parseRuleValStrQT_4MiB_QT, // val/size/strmap parseRuleValStrQT_5MiB_QT, // val/size/strmap parseRuleValStrQT_8MiB_QT, // val/size/strmap parseRuleValStrQT_16MiB_QT, // val/size/strmap parseRuleValStrQT_20MiB_QT, // val/size/strmap parseRuleValStrQT_128MiB_QT, // val/size/strmap parseRuleValStrQT_1GiB_QT, // val/size/strmap parseRuleValStrQT_16GiB_QT, // val/size/strmap parseRuleValStrQT_1TiB_QT, // val/size/strmap parseRuleValStrQT_1PiB_QT, // val/size/strmap parseRuleValStrQT_4PiB_QT, // val/size/strmap }; typedef enum { parseRuleValSize0B, // val/size/enum parseRuleValSize1B, // val/size/enum parseRuleValSize8KiB, // val/size/enum parseRuleValSize16KiB, // val/size/enum parseRuleValSize32KiB, // val/size/enum parseRuleValSize64KiB, // val/size/enum parseRuleValSize128KiB, // val/size/enum parseRuleValSize256KiB, // val/size/enum parseRuleValSize512KiB, // val/size/enum parseRuleValSize1MiB, // val/size/enum parseRuleValSize2MiB, // val/size/enum parseRuleValSize4MiB, // val/size/enum parseRuleValSize5MiB, // val/size/enum parseRuleValSize8MiB, // val/size/enum parseRuleValSize16MiB, // val/size/enum parseRuleValSize20MiB, // val/size/enum parseRuleValSize128MiB, // val/size/enum parseRuleValSize1GiB, // val/size/enum parseRuleValSize16GiB, // val/size/enum parseRuleValSize1TiB, // val/size/enum parseRuleValSize1PiB, // val/size/enum parseRuleValSize4PiB, // val/size/enum } ParseRuleValueSize; /*********************************************************************************************************************************** Rule Times ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_TIME(value) PARSE_RULE_U32_1(parseRuleValTime##value) static const unsigned int parseRuleValueTime[] = { 0, // val/time 100, // val/time 15000, // val/time 60000, // val/time 900000, // val/time 1800000, // val/time 1860000, // val/time 3600000, // val/time 86400000, // val/time 604800000, // val/time }; static const uint8_t parseRuleValueTimeStrMap[] = { parseRuleValStrQT_0s_QT, // val/time/strmap parseRuleValStrQT_100ms_QT, // val/time/strmap parseRuleValStrQT_15s_QT, // val/time/strmap parseRuleValStrQT_1m_QT, // val/time/strmap parseRuleValStrQT_15m_QT, // val/time/strmap parseRuleValStrQT_30m_QT, // val/time/strmap parseRuleValStrQT_31m_QT, // val/time/strmap parseRuleValStrQT_1h_QT, // val/time/strmap parseRuleValStrQT_1d_QT, // val/time/strmap parseRuleValStrQT_7d_QT, // val/time/strmap }; typedef enum { parseRuleValTime0s, // val/time/enum parseRuleValTime100ms, // val/time/enum parseRuleValTime15s, // val/time/enum parseRuleValTime1m, // val/time/enum parseRuleValTime15m, // val/time/enum parseRuleValTime30m, // val/time/enum parseRuleValTime31m, // val/time/enum parseRuleValTime1h, // val/time/enum parseRuleValTime1d, // val/time/enum parseRuleValTime7d, // val/time/enum } ParseRuleValueTime; /*********************************************************************************************************************************** Dynamic default values ***********************************************************************************************************************************/ #define PARSE_RULE_DEFAULT_DYNAMIC(value) PARSE_RULE_U32_1(parseRuleDefaultDynamic##value) typedef enum { parseRuleDefaultDynamicBin, } ParseRuleDefaultDynamic; /*********************************************************************************************************************************** Command parse data ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_CMD(value) PARSE_RULE_U32_1(cfgCmd##value) static const ParseRuleCommand parseRuleCommand[CFG_COMMAND_TOTAL] = { PARSE_RULE_COMMAND // cmd/annotate ( // cmd/annotate PARSE_RULE_COMMAND_NAME("annotate"), // cmd/annotate PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/annotate PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/annotate PARSE_RULE_COMMAND_LOCK_TYPE(Backup), // cmd/annotate PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/annotate // cmd/annotate PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/annotate ( // cmd/annotate PARSE_RULE_COMMAND_ROLE(Main) // cmd/annotate PARSE_RULE_COMMAND_ROLE(Remote) // cmd/annotate ), // cmd/annotate ), // cmd/annotate // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/archive-get ( // cmd/archive-get PARSE_RULE_COMMAND_NAME("archive-get"), // cmd/archive-get PARSE_RULE_COMMAND_LOCK_TYPE(Archive), // cmd/archive-get PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/archive-get PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/archive-get // cmd/archive-get PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/archive-get ( // cmd/archive-get PARSE_RULE_COMMAND_ROLE(Async) // cmd/archive-get PARSE_RULE_COMMAND_ROLE(Local) // cmd/archive-get PARSE_RULE_COMMAND_ROLE(Main) // cmd/archive-get PARSE_RULE_COMMAND_ROLE(Remote) // cmd/archive-get ), // cmd/archive-get ), // cmd/archive-get // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/archive-push ( // cmd/archive-push PARSE_RULE_COMMAND_NAME("archive-push"), // cmd/archive-push PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/archive-push PARSE_RULE_COMMAND_LOCK_TYPE(Archive), // cmd/archive-push PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/archive-push PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/archive-push // cmd/archive-push PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/archive-push ( // cmd/archive-push PARSE_RULE_COMMAND_ROLE(Async) // cmd/archive-push PARSE_RULE_COMMAND_ROLE(Local) // cmd/archive-push PARSE_RULE_COMMAND_ROLE(Main) // cmd/archive-push PARSE_RULE_COMMAND_ROLE(Remote) // cmd/archive-push ), // cmd/archive-push ), // cmd/archive-push // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/backup ( // cmd/backup PARSE_RULE_COMMAND_NAME("backup"), // cmd/backup PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/backup PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/backup PARSE_RULE_COMMAND_LOCK_TYPE(Backup), // cmd/backup PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/backup PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/backup // cmd/backup PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/backup ( // cmd/backup PARSE_RULE_COMMAND_ROLE(Local) // cmd/backup PARSE_RULE_COMMAND_ROLE(Main) // cmd/backup PARSE_RULE_COMMAND_ROLE(Remote) // cmd/backup ), // cmd/backup ), // cmd/backup // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/check ( // cmd/check PARSE_RULE_COMMAND_NAME("check"), // cmd/check PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/check PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/check // cmd/check PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/check ( // cmd/check PARSE_RULE_COMMAND_ROLE(Main) // cmd/check PARSE_RULE_COMMAND_ROLE(Remote) // cmd/check ), // cmd/check ), // cmd/check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/expire ( // cmd/expire PARSE_RULE_COMMAND_NAME("expire"), // cmd/expire PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/expire PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/expire PARSE_RULE_COMMAND_LOCK_TYPE(Backup), // cmd/expire PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/expire PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/expire // cmd/expire PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/expire ( // cmd/expire PARSE_RULE_COMMAND_ROLE(Main) // cmd/expire PARSE_RULE_COMMAND_ROLE(Remote) // cmd/expire ), // cmd/expire ), // cmd/expire // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/help ( // cmd/help PARSE_RULE_COMMAND_NAME("help"), // cmd/help PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/help PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/help PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/help // cmd/help PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/help ( // cmd/help PARSE_RULE_COMMAND_ROLE(Main) // cmd/help ), // cmd/help ), // cmd/help // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/info ( // cmd/info PARSE_RULE_COMMAND_NAME("info"), // cmd/info PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/info PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/info // cmd/info PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/info ( // cmd/info PARSE_RULE_COMMAND_ROLE(Main) // cmd/info PARSE_RULE_COMMAND_ROLE(Remote) // cmd/info ), // cmd/info ), // cmd/info // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/manifest ( // cmd/manifest PARSE_RULE_COMMAND_NAME("manifest"), // cmd/manifest PARSE_RULE_COMMAND_INTERNAL(true) // cmd/manifest PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/manifest PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/manifest // cmd/manifest PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/manifest ( // cmd/manifest PARSE_RULE_COMMAND_ROLE(Main) // cmd/manifest PARSE_RULE_COMMAND_ROLE(Remote) // cmd/manifest ), // cmd/manifest ), // cmd/manifest // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/repo-get ( // cmd/repo-get PARSE_RULE_COMMAND_NAME("repo-get"), // cmd/repo-get PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/repo-get PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/repo-get PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/repo-get // cmd/repo-get PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/repo-get ( // cmd/repo-get PARSE_RULE_COMMAND_ROLE(Main) // cmd/repo-get PARSE_RULE_COMMAND_ROLE(Remote) // cmd/repo-get ), // cmd/repo-get ), // cmd/repo-get // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/repo-ls ( // cmd/repo-ls PARSE_RULE_COMMAND_NAME("repo-ls"), // cmd/repo-ls PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/repo-ls PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/repo-ls PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/repo-ls // cmd/repo-ls PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/repo-ls ( // cmd/repo-ls PARSE_RULE_COMMAND_ROLE(Main) // cmd/repo-ls PARSE_RULE_COMMAND_ROLE(Remote) // cmd/repo-ls ), // cmd/repo-ls ), // cmd/repo-ls // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/repo-put ( // cmd/repo-put PARSE_RULE_COMMAND_NAME("repo-put"), // cmd/repo-put PARSE_RULE_COMMAND_INTERNAL(true) // cmd/repo-put PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/repo-put PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/repo-put PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/repo-put // cmd/repo-put PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/repo-put ( // cmd/repo-put PARSE_RULE_COMMAND_ROLE(Main) // cmd/repo-put PARSE_RULE_COMMAND_ROLE(Remote) // cmd/repo-put ), // cmd/repo-put ), // cmd/repo-put // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/repo-rm ( // cmd/repo-rm PARSE_RULE_COMMAND_NAME("repo-rm"), // cmd/repo-rm PARSE_RULE_COMMAND_INTERNAL(true) // cmd/repo-rm PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/repo-rm PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/repo-rm PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/repo-rm // cmd/repo-rm PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/repo-rm ( // cmd/repo-rm PARSE_RULE_COMMAND_ROLE(Main) // cmd/repo-rm PARSE_RULE_COMMAND_ROLE(Remote) // cmd/repo-rm ), // cmd/repo-rm ), // cmd/repo-rm // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/restore ( // cmd/restore PARSE_RULE_COMMAND_NAME("restore"), // cmd/restore PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/restore PARSE_RULE_COMMAND_LOCK_TYPE(Restore), // cmd/restore PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/restore PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/restore // cmd/restore PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/restore ( // cmd/restore PARSE_RULE_COMMAND_ROLE(Local) // cmd/restore PARSE_RULE_COMMAND_ROLE(Main) // cmd/restore PARSE_RULE_COMMAND_ROLE(Remote) // cmd/restore ), // cmd/restore ), // cmd/restore // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/server ( // cmd/server PARSE_RULE_COMMAND_NAME("server"), // cmd/server PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/server PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/server PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/server // cmd/server PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/server ( // cmd/server PARSE_RULE_COMMAND_ROLE(Main) // cmd/server ), // cmd/server ), // cmd/server // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/server-ping ( // cmd/server-ping PARSE_RULE_COMMAND_NAME("server-ping"), // cmd/server-ping PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/server-ping PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/server-ping PARSE_RULE_COMMAND_PARAMETER_ALLOWED(true), // cmd/server-ping // cmd/server-ping PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/server-ping ( // cmd/server-ping PARSE_RULE_COMMAND_ROLE(Main) // cmd/server-ping ), // cmd/server-ping ), // cmd/server-ping // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/stanza-create ( // cmd/stanza-create PARSE_RULE_COMMAND_NAME("stanza-create"), // cmd/stanza-create PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/stanza-create PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/stanza-create PARSE_RULE_COMMAND_LOCK_TYPE(All), // cmd/stanza-create PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/stanza-create PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/stanza-create // cmd/stanza-create PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/stanza-create ( // cmd/stanza-create PARSE_RULE_COMMAND_ROLE(Main) // cmd/stanza-create PARSE_RULE_COMMAND_ROLE(Remote) // cmd/stanza-create ), // cmd/stanza-create ), // cmd/stanza-create // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/stanza-delete ( // cmd/stanza-delete PARSE_RULE_COMMAND_NAME("stanza-delete"), // cmd/stanza-delete PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/stanza-delete PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/stanza-delete PARSE_RULE_COMMAND_LOCK_TYPE(All), // cmd/stanza-delete PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/stanza-delete PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/stanza-delete // cmd/stanza-delete PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/stanza-delete ( // cmd/stanza-delete PARSE_RULE_COMMAND_ROLE(Main) // cmd/stanza-delete PARSE_RULE_COMMAND_ROLE(Remote) // cmd/stanza-delete ), // cmd/stanza-delete ), // cmd/stanza-delete // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/stanza-upgrade ( // cmd/stanza-upgrade PARSE_RULE_COMMAND_NAME("stanza-upgrade"), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOCK_REQUIRED(true), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(true), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOCK_TYPE(All), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/stanza-upgrade PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/stanza-upgrade // cmd/stanza-upgrade PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/stanza-upgrade ( // cmd/stanza-upgrade PARSE_RULE_COMMAND_ROLE(Main) // cmd/stanza-upgrade PARSE_RULE_COMMAND_ROLE(Remote) // cmd/stanza-upgrade ), // cmd/stanza-upgrade ), // cmd/stanza-upgrade // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/start ( // cmd/start PARSE_RULE_COMMAND_NAME("start"), // cmd/start PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/start PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/start PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/start // cmd/start PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/start ( // cmd/start PARSE_RULE_COMMAND_ROLE(Main) // cmd/start ), // cmd/start ), // cmd/start // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/stop ( // cmd/stop PARSE_RULE_COMMAND_NAME("stop"), // cmd/stop PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/stop PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/stop PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/stop // cmd/stop PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/stop ( // cmd/stop PARSE_RULE_COMMAND_ROLE(Main) // cmd/stop ), // cmd/stop ), // cmd/stop // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/verify ( // cmd/verify PARSE_RULE_COMMAND_NAME("verify"), // cmd/verify PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/verify PARSE_RULE_COMMAND_LOG_FILE(true), // cmd/verify PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Info), // cmd/verify // cmd/verify PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/verify ( // cmd/verify PARSE_RULE_COMMAND_ROLE(Local) // cmd/verify PARSE_RULE_COMMAND_ROLE(Main) // cmd/verify PARSE_RULE_COMMAND_ROLE(Remote) // cmd/verify ), // cmd/verify ), // cmd/verify // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_COMMAND // cmd/version ( // cmd/version PARSE_RULE_COMMAND_NAME("version"), // cmd/version PARSE_RULE_COMMAND_LOCK_TYPE(None), // cmd/version PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(Debug), // cmd/version // cmd/version PARSE_RULE_COMMAND_ROLE_VALID_LIST // cmd/version ( // cmd/version PARSE_RULE_COMMAND_ROLE(Main) // cmd/version ), // cmd/version ), // cmd/version }; /*********************************************************************************************************************************** Option group parse data ***********************************************************************************************************************************/ static const ParseRuleOptionGroup parseRuleOptionGroup[CFG_OPTION_GROUP_TOTAL] = { PARSE_RULE_OPTION_GROUP // opt-grp/pg ( // opt-grp/pg PARSE_RULE_OPTION_GROUP_NAME("pg"), // opt-grp/pg ), // opt-grp/pg // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION_GROUP // opt-grp/repo ( // opt-grp/repo PARSE_RULE_OPTION_GROUP_NAME("repo"), // opt-grp/repo ), // opt-grp/repo }; /*********************************************************************************************************************************** Option parse data ***********************************************************************************************************************************/ #define PARSE_RULE_VAL_OPT(value) PARSE_RULE_U32_2(cfgOpt##value) static const ParseRuleOption parseRuleOption[CFG_OPTION_TOTAL] = { PARSE_RULE_OPTION // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_NAME("allow-root"), // opt/allow-root PARSE_RULE_OPTION_TYPE(Boolean), // opt/allow-root PARSE_RULE_OPTION_NEGATE(true), // opt/allow-root PARSE_RULE_OPTION_RESET(true), // opt/allow-root PARSE_RULE_OPTION_REQUIRED(true), // opt/allow-root PARSE_RULE_OPTION_SECTION(Global), // opt/allow-root // opt/allow-root PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_COMMAND(Annotate) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Backup) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Check) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Expire) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Info) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Manifest) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Restore) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Server) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Start) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Stop) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Verify) // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Backup) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Restore) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Verify) // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTION_COMMAND(Annotate) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Backup) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Check) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Expire) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Info) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Manifest) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/allow-root PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Restore) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/allow-root PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/allow-root PARSE_RULE_OPTION_COMMAND(Verify) // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTIONAL // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTIONAL_GROUP // opt/allow-root ( // opt/allow-root PARSE_RULE_FILTER_CMD // opt/allow-root ( // opt/allow-root PARSE_RULE_VAL_CMD(Restore), // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTIONAL_DEFAULT // opt/allow-root ( // opt/allow-root PARSE_RULE_VAL_BOOL_TRUE, // opt/allow-root ), // opt/allow-root ), // opt/allow-root // opt/allow-root PARSE_RULE_OPTIONAL_GROUP // opt/allow-root ( // opt/allow-root PARSE_RULE_OPTIONAL_DEFAULT // opt/allow-root ( // opt/allow-root PARSE_RULE_VAL_BOOL_FALSE, // opt/allow-root ), // opt/allow-root ), // opt/allow-root ), // opt/allow-root ), // opt/allow-root // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/annotation ( // opt/annotation PARSE_RULE_OPTION_NAME("annotation"), // opt/annotation PARSE_RULE_OPTION_TYPE(Hash), // opt/annotation PARSE_RULE_OPTION_RESET(true), // opt/annotation PARSE_RULE_OPTION_REQUIRED(false), // opt/annotation PARSE_RULE_OPTION_SECTION(Global), // opt/annotation PARSE_RULE_OPTION_MULTI(true), // opt/annotation // opt/annotation PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/annotation ( // opt/annotation PARSE_RULE_OPTION_COMMAND(Annotate) // opt/annotation PARSE_RULE_OPTION_COMMAND(Backup) // opt/annotation ), // opt/annotation // opt/annotation PARSE_RULE_OPTIONAL // opt/annotation ( // opt/annotation PARSE_RULE_OPTIONAL_GROUP // opt/annotation ( // opt/annotation PARSE_RULE_FILTER_CMD // opt/annotation ( // opt/annotation PARSE_RULE_VAL_CMD(Annotate), // opt/annotation ), // opt/annotation // opt/annotation PARSE_RULE_OPTIONAL_REQUIRED(), // opt/annotation ), // opt/annotation ), // opt/annotation ), // opt/annotation // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_NAME("archive-async"), // opt/archive-async PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-async PARSE_RULE_OPTION_NEGATE(true), // opt/archive-async PARSE_RULE_OPTION_RESET(true), // opt/archive-async PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-async PARSE_RULE_OPTION_SECTION(Global), // opt/archive-async // opt/archive-async PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-async ), // opt/archive-async // opt/archive-async PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-async ), // opt/archive-async // opt/archive-async PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-async ), // opt/archive-async // opt/archive-async PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-async PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-async ), // opt/archive-async // opt/archive-async PARSE_RULE_OPTIONAL // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTIONAL_GROUP // opt/archive-async ( // opt/archive-async PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-async ( // opt/archive-async PARSE_RULE_VAL_BOOL_FALSE, // opt/archive-async ), // opt/archive-async ), // opt/archive-async ), // opt/archive-async ), // opt/archive-async // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTION_NAME("archive-check"), // opt/archive-check PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-check PARSE_RULE_OPTION_NEGATE(true), // opt/archive-check PARSE_RULE_OPTION_RESET(true), // opt/archive-check PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-check PARSE_RULE_OPTION_SECTION(Global), // opt/archive-check // opt/archive-check PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-check PARSE_RULE_OPTION_COMMAND(Backup) // opt/archive-check PARSE_RULE_OPTION_COMMAND(Check) // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTIONAL // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTIONAL_GROUP // opt/archive-check ( // opt/archive-check PARSE_RULE_FILTER_CMD // opt/archive-check ( // opt/archive-check PARSE_RULE_VAL_CMD(Backup), // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTIONAL_DEPEND // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/archive-check PARSE_RULE_VAL_OPT(Online), // opt/archive-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-check ( // opt/archive-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-check ), // opt/archive-check ), // opt/archive-check // opt/archive-check PARSE_RULE_OPTIONAL_GROUP // opt/archive-check ( // opt/archive-check PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-check ( // opt/archive-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-check ), // opt/archive-check ), // opt/archive-check ), // opt/archive-check ), // opt/archive-check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTION_NAME("archive-copy"), // opt/archive-copy PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-copy PARSE_RULE_OPTION_NEGATE(true), // opt/archive-copy PARSE_RULE_OPTION_RESET(true), // opt/archive-copy PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-copy PARSE_RULE_OPTION_SECTION(Global), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTION_COMMAND(Backup) // opt/archive-copy ), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTIONAL // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTIONAL_GROUP // opt/archive-copy ( // opt/archive-copy PARSE_RULE_FILTER_CMD // opt/archive-copy ( // opt/archive-copy PARSE_RULE_VAL_CMD(Backup), // opt/archive-copy ), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTIONAL_DEPEND // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/archive-copy PARSE_RULE_VAL_OPT(ArchiveCheck), // opt/archive-copy PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-copy ), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-copy ( // opt/archive-copy PARSE_RULE_VAL_BOOL_FALSE, // opt/archive-copy ), // opt/archive-copy ), // opt/archive-copy // opt/archive-copy PARSE_RULE_OPTIONAL_GROUP // opt/archive-copy ( // opt/archive-copy PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-copy ( // opt/archive-copy PARSE_RULE_VAL_BOOL_FALSE, // opt/archive-copy ), // opt/archive-copy ), // opt/archive-copy ), // opt/archive-copy ), // opt/archive-copy // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-expire-before ( // opt/archive-expire-before PARSE_RULE_OPTION_NAME("archive-expire-before"), // opt/archive-expire-before PARSE_RULE_OPTION_TYPE(String), // opt/archive-expire-before PARSE_RULE_OPTION_REQUIRED(false), // opt/archive-expire-before PARSE_RULE_OPTION_SECTION(CommandLine), // opt/archive-expire-before // opt/archive-expire-before PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-expire-before ( // opt/archive-expire-before PARSE_RULE_OPTION_COMMAND(Expire) // opt/archive-expire-before ), // opt/archive-expire-before ), // opt/archive-expire-before // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTION_NAME("archive-get-queue-max"), // opt/archive-get-queue-max PARSE_RULE_OPTION_TYPE(Size), // opt/archive-get-queue-max PARSE_RULE_OPTION_RESET(true), // opt/archive-get-queue-max PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-get-queue-max PARSE_RULE_OPTION_SECTION(Global), // opt/archive-get-queue-max // opt/archive-get-queue-max PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-get-queue-max ), // opt/archive-get-queue-max // opt/archive-get-queue-max PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-get-queue-max ), // opt/archive-get-queue-max // opt/archive-get-queue-max PARSE_RULE_OPTIONAL // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTIONAL_GROUP // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_VAL_SIZE(0B), // opt/archive-get-queue-max PARSE_RULE_VAL_SIZE(4PiB), // opt/archive-get-queue-max ), // opt/archive-get-queue-max // opt/archive-get-queue-max PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-get-queue-max ( // opt/archive-get-queue-max PARSE_RULE_VAL_SIZE(128MiB), // opt/archive-get-queue-max ), // opt/archive-get-queue-max ), // opt/archive-get-queue-max ), // opt/archive-get-queue-max ), // opt/archive-get-queue-max // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTION_NAME("archive-header-check"), // opt/archive-header-check PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-header-check PARSE_RULE_OPTION_NEGATE(true), // opt/archive-header-check PARSE_RULE_OPTION_RESET(true), // opt/archive-header-check PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-header-check PARSE_RULE_OPTION_SECTION(Global), // opt/archive-header-check // opt/archive-header-check PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-header-check ), // opt/archive-header-check // opt/archive-header-check PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-header-check ), // opt/archive-header-check // opt/archive-header-check PARSE_RULE_OPTIONAL // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTIONAL_GROUP // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-header-check ( // opt/archive-header-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-header-check ), // opt/archive-header-check ), // opt/archive-header-check ), // opt/archive-header-check ), // opt/archive-header-check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_OPTION_NAME("archive-missing-retry"), // opt/archive-missing-retry PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-missing-retry PARSE_RULE_OPTION_NEGATE(true), // opt/archive-missing-retry PARSE_RULE_OPTION_RESET(true), // opt/archive-missing-retry PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-missing-retry PARSE_RULE_OPTION_SECTION(Global), // opt/archive-missing-retry // opt/archive-missing-retry PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-missing-retry ), // opt/archive-missing-retry // opt/archive-missing-retry PARSE_RULE_OPTIONAL // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_OPTIONAL_GROUP // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-missing-retry ( // opt/archive-missing-retry PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-missing-retry ), // opt/archive-missing-retry ), // opt/archive-missing-retry ), // opt/archive-missing-retry ), // opt/archive-missing-retry // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-mode ( // opt/archive-mode PARSE_RULE_OPTION_NAME("archive-mode"), // opt/archive-mode PARSE_RULE_OPTION_TYPE(StringId), // opt/archive-mode PARSE_RULE_OPTION_SEQUENCE(true), // opt/archive-mode PARSE_RULE_OPTION_RESET(true), // opt/archive-mode PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-mode PARSE_RULE_OPTION_SECTION(Global), // opt/archive-mode // opt/archive-mode PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-mode ( // opt/archive-mode PARSE_RULE_OPTION_COMMAND(Restore) // opt/archive-mode ), // opt/archive-mode // opt/archive-mode PARSE_RULE_OPTIONAL // opt/archive-mode ( // opt/archive-mode PARSE_RULE_OPTIONAL_GROUP // opt/archive-mode ( // opt/archive-mode PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/archive-mode ( // opt/archive-mode PARSE_RULE_VAL_STRID(preserve), // opt/archive-mode PARSE_RULE_VAL_STRID(off), // opt/archive-mode ), // opt/archive-mode // opt/archive-mode PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-mode ( // opt/archive-mode PARSE_RULE_VAL_STRID(preserve), // opt/archive-mode ), // opt/archive-mode ), // opt/archive-mode ), // opt/archive-mode ), // opt/archive-mode // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTION_NAME("archive-mode-check"), // opt/archive-mode-check PARSE_RULE_OPTION_TYPE(Boolean), // opt/archive-mode-check PARSE_RULE_OPTION_NEGATE(true), // opt/archive-mode-check PARSE_RULE_OPTION_RESET(true), // opt/archive-mode-check PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-mode-check PARSE_RULE_OPTION_SECTION(Global), // opt/archive-mode-check // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND(Backup) // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND(Check) // opt/archive-mode-check ), // opt/archive-mode-check // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-mode-check ), // opt/archive-mode-check // opt/archive-mode-check PARSE_RULE_OPTIONAL // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTIONAL_GROUP // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTIONAL_DEPEND // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/archive-mode-check PARSE_RULE_VAL_OPT(ArchiveCheck), // opt/archive-mode-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-mode-check ), // opt/archive-mode-check // opt/archive-mode-check PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-mode-check ( // opt/archive-mode-check PARSE_RULE_VAL_BOOL_TRUE, // opt/archive-mode-check ), // opt/archive-mode-check ), // opt/archive-mode-check ), // opt/archive-mode-check ), // opt/archive-mode-check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTION_NAME("archive-push-batch-size"), // opt/archive-push-batch-size PARSE_RULE_OPTION_TYPE(Size), // opt/archive-push-batch-size PARSE_RULE_OPTION_RESET(true), // opt/archive-push-batch-size PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-push-batch-size PARSE_RULE_OPTION_SECTION(Global), // opt/archive-push-batch-size // opt/archive-push-batch-size PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-push-batch-size ), // opt/archive-push-batch-size // opt/archive-push-batch-size PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-push-batch-size ), // opt/archive-push-batch-size // opt/archive-push-batch-size PARSE_RULE_OPTIONAL // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTIONAL_GROUP // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_VAL_SIZE(1MiB), // opt/archive-push-batch-size PARSE_RULE_VAL_SIZE(4PiB), // opt/archive-push-batch-size ), // opt/archive-push-batch-size // opt/archive-push-batch-size PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-push-batch-size ( // opt/archive-push-batch-size PARSE_RULE_VAL_SIZE(16GiB), // opt/archive-push-batch-size ), // opt/archive-push-batch-size ), // opt/archive-push-batch-size ), // opt/archive-push-batch-size ), // opt/archive-push-batch-size // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTION_NAME("archive-push-queue-max"), // opt/archive-push-queue-max PARSE_RULE_OPTION_TYPE(Size), // opt/archive-push-queue-max PARSE_RULE_OPTION_RESET(true), // opt/archive-push-queue-max PARSE_RULE_OPTION_REQUIRED(false), // opt/archive-push-queue-max PARSE_RULE_OPTION_SECTION(Global), // opt/archive-push-queue-max // opt/archive-push-queue-max PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-push-queue-max ), // opt/archive-push-queue-max // opt/archive-push-queue-max PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-push-queue-max ), // opt/archive-push-queue-max // opt/archive-push-queue-max PARSE_RULE_OPTIONAL // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTIONAL_GROUP // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/archive-push-queue-max ( // opt/archive-push-queue-max PARSE_RULE_VAL_SIZE(0B), // opt/archive-push-queue-max PARSE_RULE_VAL_SIZE(4PiB), // opt/archive-push-queue-max ), // opt/archive-push-queue-max ), // opt/archive-push-queue-max ), // opt/archive-push-queue-max ), // opt/archive-push-queue-max // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_OPTION_NAME("archive-timeout"), // opt/archive-timeout PARSE_RULE_OPTION_TYPE(Time), // opt/archive-timeout PARSE_RULE_OPTION_RESET(true), // opt/archive-timeout PARSE_RULE_OPTION_REQUIRED(true), // opt/archive-timeout PARSE_RULE_OPTION_SECTION(Global), // opt/archive-timeout // opt/archive-timeout PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/archive-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/archive-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/archive-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/archive-timeout ), // opt/archive-timeout // opt/archive-timeout PARSE_RULE_OPTIONAL // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_OPTIONAL_GROUP // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_VAL_TIME(100ms), // opt/archive-timeout PARSE_RULE_VAL_TIME(1d), // opt/archive-timeout ), // opt/archive-timeout // opt/archive-timeout PARSE_RULE_OPTIONAL_DEFAULT // opt/archive-timeout ( // opt/archive-timeout PARSE_RULE_VAL_TIME(1m), // opt/archive-timeout ), // opt/archive-timeout ), // opt/archive-timeout ), // opt/archive-timeout ), // opt/archive-timeout // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/backup-standby ( // opt/backup-standby PARSE_RULE_OPTION_NAME("backup-standby"), // opt/backup-standby PARSE_RULE_OPTION_TYPE(StringId), // opt/backup-standby PARSE_RULE_OPTION_SEQUENCE(true), // opt/backup-standby PARSE_RULE_OPTION_BOOL_LIKE(true), // opt/backup-standby PARSE_RULE_OPTION_NEGATE(true), // opt/backup-standby PARSE_RULE_OPTION_RESET(true), // opt/backup-standby PARSE_RULE_OPTION_REQUIRED(true), // opt/backup-standby PARSE_RULE_OPTION_SECTION(Global), // opt/backup-standby // opt/backup-standby PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/backup-standby ( // opt/backup-standby PARSE_RULE_OPTION_COMMAND(Backup) // opt/backup-standby PARSE_RULE_OPTION_COMMAND(Check) // opt/backup-standby ), // opt/backup-standby // opt/backup-standby PARSE_RULE_OPTIONAL // opt/backup-standby ( // opt/backup-standby PARSE_RULE_OPTIONAL_GROUP // opt/backup-standby ( // opt/backup-standby PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/backup-standby ( // opt/backup-standby PARSE_RULE_VAL_STRID(n), // opt/backup-standby PARSE_RULE_VAL_STRID(prefer), // opt/backup-standby PARSE_RULE_VAL_STRID(y), // opt/backup-standby ), // opt/backup-standby // opt/backup-standby PARSE_RULE_OPTIONAL_DEFAULT // opt/backup-standby ( // opt/backup-standby PARSE_RULE_VAL_STRID(n), // opt/backup-standby ), // opt/backup-standby ), // opt/backup-standby ), // opt/backup-standby ), // opt/backup-standby // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/beta ( // opt/beta PARSE_RULE_OPTION_NAME("beta"), // opt/beta PARSE_RULE_OPTION_TYPE(Boolean), // opt/beta PARSE_RULE_OPTION_INTERNAL(true) // opt/beta PARSE_RULE_OPTION_BETA(true), // opt/beta PARSE_RULE_OPTION_NEGATE(true), // opt/beta PARSE_RULE_OPTION_RESET(true), // opt/beta PARSE_RULE_OPTION_REQUIRED(true), // opt/beta PARSE_RULE_OPTION_SECTION(Global), // opt/beta // opt/beta PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/beta ( // opt/beta PARSE_RULE_OPTION_COMMAND(Annotate) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/beta PARSE_RULE_OPTION_COMMAND(Backup) // opt/beta PARSE_RULE_OPTION_COMMAND(Check) // opt/beta PARSE_RULE_OPTION_COMMAND(Expire) // opt/beta PARSE_RULE_OPTION_COMMAND(Info) // opt/beta PARSE_RULE_OPTION_COMMAND(Manifest) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/beta PARSE_RULE_OPTION_COMMAND(Restore) // opt/beta PARSE_RULE_OPTION_COMMAND(Server) // opt/beta PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/beta PARSE_RULE_OPTION_COMMAND(Start) // opt/beta PARSE_RULE_OPTION_COMMAND(Stop) // opt/beta PARSE_RULE_OPTION_COMMAND(Verify) // opt/beta ), // opt/beta // opt/beta PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/beta ( // opt/beta PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/beta ), // opt/beta // opt/beta PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/beta ( // opt/beta PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/beta PARSE_RULE_OPTION_COMMAND(Backup) // opt/beta PARSE_RULE_OPTION_COMMAND(Restore) // opt/beta PARSE_RULE_OPTION_COMMAND(Verify) // opt/beta ), // opt/beta // opt/beta PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/beta ( // opt/beta PARSE_RULE_OPTION_COMMAND(Annotate) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/beta PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/beta PARSE_RULE_OPTION_COMMAND(Backup) // opt/beta PARSE_RULE_OPTION_COMMAND(Check) // opt/beta PARSE_RULE_OPTION_COMMAND(Expire) // opt/beta PARSE_RULE_OPTION_COMMAND(Info) // opt/beta PARSE_RULE_OPTION_COMMAND(Manifest) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/beta PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/beta PARSE_RULE_OPTION_COMMAND(Restore) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/beta PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/beta PARSE_RULE_OPTION_COMMAND(Verify) // opt/beta ), // opt/beta // opt/beta PARSE_RULE_OPTIONAL // opt/beta ( // opt/beta PARSE_RULE_OPTIONAL_GROUP // opt/beta ( // opt/beta PARSE_RULE_OPTIONAL_DEFAULT // opt/beta ( // opt/beta PARSE_RULE_VAL_BOOL_FALSE, // opt/beta ), // opt/beta ), // opt/beta ), // opt/beta ), // opt/beta // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_NAME("buffer-size"), // opt/buffer-size PARSE_RULE_OPTION_TYPE(Size), // opt/buffer-size PARSE_RULE_OPTION_RESET(true), // opt/buffer-size PARSE_RULE_OPTION_REQUIRED(true), // opt/buffer-size PARSE_RULE_OPTION_SECTION(Global), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Annotate) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Check) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Expire) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Info) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Manifest) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Server) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Annotate) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Check) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Expire) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Info) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Manifest) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/buffer-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTIONAL // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTIONAL_GROUP // opt/buffer-size ( // opt/buffer-size PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/buffer-size ( // opt/buffer-size PARSE_RULE_VAL_SIZE(16KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(32KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(64KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(128KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(256KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(512KiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(1MiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(2MiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(4MiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(8MiB), // opt/buffer-size PARSE_RULE_VAL_SIZE(16MiB), // opt/buffer-size ), // opt/buffer-size // opt/buffer-size PARSE_RULE_OPTIONAL_DEFAULT // opt/buffer-size ( // opt/buffer-size PARSE_RULE_VAL_SIZE(1MiB), // opt/buffer-size ), // opt/buffer-size ), // opt/buffer-size ), // opt/buffer-size ), // opt/buffer-size // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/checksum-page ( // opt/checksum-page PARSE_RULE_OPTION_NAME("checksum-page"), // opt/checksum-page PARSE_RULE_OPTION_TYPE(Boolean), // opt/checksum-page PARSE_RULE_OPTION_NEGATE(true), // opt/checksum-page PARSE_RULE_OPTION_RESET(true), // opt/checksum-page PARSE_RULE_OPTION_REQUIRED(false), // opt/checksum-page PARSE_RULE_OPTION_SECTION(Global), // opt/checksum-page // opt/checksum-page PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/checksum-page ( // opt/checksum-page PARSE_RULE_OPTION_COMMAND(Backup) // opt/checksum-page ), // opt/checksum-page ), // opt/checksum-page // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/cipher-pass ( // opt/cipher-pass PARSE_RULE_OPTION_NAME("cipher-pass"), // opt/cipher-pass PARSE_RULE_OPTION_TYPE(String), // opt/cipher-pass PARSE_RULE_OPTION_INTERNAL(true) // opt/cipher-pass PARSE_RULE_OPTION_REQUIRED(false), // opt/cipher-pass PARSE_RULE_OPTION_SECTION(CommandLine), // opt/cipher-pass // opt/cipher-pass PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/cipher-pass ( // opt/cipher-pass PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/cipher-pass PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/cipher-pass ), // opt/cipher-pass ), // opt/cipher-pass // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/cmd ( // opt/cmd PARSE_RULE_OPTION_NAME("cmd"), // opt/cmd PARSE_RULE_OPTION_TYPE(String), // opt/cmd PARSE_RULE_OPTION_DEFAULT_TYPE(Dynamic), // opt/cmd PARSE_RULE_OPTION_RESET(true), // opt/cmd PARSE_RULE_OPTION_REQUIRED(true), // opt/cmd PARSE_RULE_OPTION_SECTION(Global), // opt/cmd // opt/cmd PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/cmd ( // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/cmd ), // opt/cmd // opt/cmd PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/cmd ( // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd ), // opt/cmd // opt/cmd PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/cmd ( // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/cmd ), // opt/cmd // opt/cmd PARSE_RULE_OPTIONAL // opt/cmd ( // opt/cmd PARSE_RULE_OPTIONAL_GROUP // opt/cmd ( // opt/cmd PARSE_RULE_OPTIONAL_DEFAULT // opt/cmd ( // opt/cmd PARSE_RULE_DEFAULT_DYNAMIC(Bin), // opt/cmd ), // opt/cmd ), // opt/cmd ), // opt/cmd ), // opt/cmd // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTION_NAME("cmd-ssh"), // opt/cmd-ssh PARSE_RULE_OPTION_TYPE(String), // opt/cmd-ssh PARSE_RULE_OPTION_RESET(true), // opt/cmd-ssh PARSE_RULE_OPTION_REQUIRED(true), // opt/cmd-ssh PARSE_RULE_OPTION_SECTION(Global), // opt/cmd-ssh // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Annotate) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Backup) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Check) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Expire) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Info) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Manifest) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Restore) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Verify) // opt/cmd-ssh ), // opt/cmd-ssh // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd-ssh ), // opt/cmd-ssh // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Backup) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Restore) // opt/cmd-ssh PARSE_RULE_OPTION_COMMAND(Verify) // opt/cmd-ssh ), // opt/cmd-ssh // opt/cmd-ssh PARSE_RULE_OPTIONAL // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTIONAL_GROUP // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_OPTIONAL_DEFAULT // opt/cmd-ssh ( // opt/cmd-ssh PARSE_RULE_VAL_STR(QT_ssh_QT), // opt/cmd-ssh ), // opt/cmd-ssh ), // opt/cmd-ssh ), // opt/cmd-ssh ), // opt/cmd-ssh // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/compress ( // opt/compress PARSE_RULE_OPTION_NAME("compress"), // opt/compress PARSE_RULE_OPTION_TYPE(Boolean), // opt/compress PARSE_RULE_OPTION_NEGATE(true), // opt/compress PARSE_RULE_OPTION_RESET(true), // opt/compress PARSE_RULE_OPTION_REQUIRED(true), // opt/compress PARSE_RULE_OPTION_SECTION(Global), // opt/compress // opt/compress PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/compress ( // opt/compress PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress ), // opt/compress // opt/compress PARSE_RULE_OPTIONAL // opt/compress ( // opt/compress PARSE_RULE_OPTIONAL_GROUP // opt/compress ( // opt/compress PARSE_RULE_OPTIONAL_DEFAULT // opt/compress ( // opt/compress PARSE_RULE_VAL_BOOL_TRUE, // opt/compress ), // opt/compress ), // opt/compress ), // opt/compress ), // opt/compress // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTION_NAME("compress-level"), // opt/compress-level PARSE_RULE_OPTION_TYPE(Integer), // opt/compress-level PARSE_RULE_OPTION_RESET(true), // opt/compress-level PARSE_RULE_OPTION_REQUIRED(false), // opt/compress-level PARSE_RULE_OPTION_SECTION(Global), // opt/compress-level // opt/compress-level PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress-level ), // opt/compress-level // opt/compress-level PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level ), // opt/compress-level // opt/compress-level PARSE_RULE_OPTIONAL // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_GROUP // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_DEPEND // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_INT(0)), // opt/compress-level PARSE_RULE_VAL_OPT(CompressType), // opt/compress-level PARSE_RULE_VAL_STRID(bz2), // opt/compress-level PARSE_RULE_VAL_STRID(gz), // opt/compress-level PARSE_RULE_VAL_STRID(lz4), // opt/compress-level PARSE_RULE_VAL_STRID(zst), // opt/compress-level ), // opt/compress-level // opt/compress-level PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_ALLOW_RANGE_MAP // opt/compress-level ( // opt/compress-level PARSE_RULE_VAL_STRID(bz2), // opt/compress-level PARSE_RULE_VAL_INT(1), // opt/compress-level PARSE_RULE_VAL_INT(9), // opt/compress-level PARSE_RULE_VAL_STRID(gz), // opt/compress-level PARSE_RULE_VAL_INT(DS_1), // opt/compress-level PARSE_RULE_VAL_INT(9), // opt/compress-level PARSE_RULE_VAL_STRID(lz4), // opt/compress-level PARSE_RULE_VAL_INT(DS_5), // opt/compress-level PARSE_RULE_VAL_INT(12), // opt/compress-level PARSE_RULE_VAL_STRID(zst), // opt/compress-level PARSE_RULE_VAL_INT(DS_7), // opt/compress-level PARSE_RULE_VAL_INT(22), // opt/compress-level ), // opt/compress-level ), // opt/compress-level // opt/compress-level PARSE_RULE_OPTIONAL_DEFAULT // opt/compress-level ( // opt/compress-level PARSE_RULE_OPTIONAL_DEFAULT_MAP // opt/compress-level ( // opt/compress-level PARSE_RULE_VAL_STRID(bz2), // opt/compress-level PARSE_RULE_VAL_INT(9), // opt/compress-level PARSE_RULE_VAL_STRID(gz), // opt/compress-level PARSE_RULE_VAL_INT(6), // opt/compress-level PARSE_RULE_VAL_STRID(lz4), // opt/compress-level PARSE_RULE_VAL_INT(1), // opt/compress-level PARSE_RULE_VAL_STRID(zst), // opt/compress-level PARSE_RULE_VAL_INT(3), // opt/compress-level ), // opt/compress-level ), // opt/compress-level ), // opt/compress-level ), // opt/compress-level ), // opt/compress-level // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTION_NAME("compress-level-network"), // opt/compress-level-network PARSE_RULE_OPTION_TYPE(Integer), // opt/compress-level-network PARSE_RULE_OPTION_RESET(true), // opt/compress-level-network PARSE_RULE_OPTION_REQUIRED(true), // opt/compress-level-network PARSE_RULE_OPTION_SECTION(Global), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Annotate) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Check) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Expire) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Info) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Manifest) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Restore) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Verify) // opt/compress-level-network ), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level-network ), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Restore) // opt/compress-level-network PARSE_RULE_OPTION_COMMAND(Verify) // opt/compress-level-network ), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTIONAL // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTIONAL_GROUP // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_VAL_INT(DS_5), // opt/compress-level-network PARSE_RULE_VAL_INT(12), // opt/compress-level-network ), // opt/compress-level-network // opt/compress-level-network PARSE_RULE_OPTIONAL_DEFAULT // opt/compress-level-network ( // opt/compress-level-network PARSE_RULE_VAL_INT(1), // opt/compress-level-network ), // opt/compress-level-network ), // opt/compress-level-network ), // opt/compress-level-network ), // opt/compress-level-network // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTION_NAME("compress-type"), // opt/compress-type PARSE_RULE_OPTION_TYPE(StringId), // opt/compress-type PARSE_RULE_OPTION_RESET(true), // opt/compress-type PARSE_RULE_OPTION_REQUIRED(true), // opt/compress-type PARSE_RULE_OPTION_SECTION(Global), // opt/compress-type // opt/compress-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/compress-type ), // opt/compress-type // opt/compress-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/compress-type ), // opt/compress-type // opt/compress-type PARSE_RULE_OPTIONAL // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTIONAL_GROUP // opt/compress-type ( // opt/compress-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/compress-type ( // opt/compress-type PARSE_RULE_VAL_STRID(none), // opt/compress-type PARSE_RULE_VAL_STRID(bz2), // opt/compress-type PARSE_RULE_VAL_STRID(gz), // opt/compress-type PARSE_RULE_VAL_STRID(lz4), // opt/compress-type PARSE_RULE_VAL_STRID(zst), // opt/compress-type #ifndef HAVE_LIBZST // opt/compress-type PARSE_RULE_BOOL_FALSE, // opt/compress-type #endif // opt/compress-type ), // opt/compress-type // opt/compress-type PARSE_RULE_OPTIONAL_DEFAULT // opt/compress-type ( // opt/compress-type PARSE_RULE_VAL_STRID(gz), // opt/compress-type ), // opt/compress-type ), // opt/compress-type ), // opt/compress-type ), // opt/compress-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/config ( // opt/config PARSE_RULE_OPTION_NAME("config"), // opt/config PARSE_RULE_OPTION_TYPE(String), // opt/config PARSE_RULE_OPTION_NEGATE(true), // opt/config PARSE_RULE_OPTION_REQUIRED(true), // opt/config PARSE_RULE_OPTION_SECTION(CommandLine), // opt/config // opt/config PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/config ( // opt/config PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config PARSE_RULE_OPTION_COMMAND(Backup) // opt/config PARSE_RULE_OPTION_COMMAND(Check) // opt/config PARSE_RULE_OPTION_COMMAND(Expire) // opt/config PARSE_RULE_OPTION_COMMAND(Info) // opt/config PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/config PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config PARSE_RULE_OPTION_COMMAND(Restore) // opt/config PARSE_RULE_OPTION_COMMAND(Server) // opt/config PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config PARSE_RULE_OPTION_COMMAND(Start) // opt/config PARSE_RULE_OPTION_COMMAND(Stop) // opt/config PARSE_RULE_OPTION_COMMAND(Verify) // opt/config ), // opt/config // opt/config PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/config ( // opt/config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config ), // opt/config // opt/config PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/config ( // opt/config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config PARSE_RULE_OPTION_COMMAND(Backup) // opt/config PARSE_RULE_OPTION_COMMAND(Restore) // opt/config PARSE_RULE_OPTION_COMMAND(Verify) // opt/config ), // opt/config // opt/config PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/config ( // opt/config PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config PARSE_RULE_OPTION_COMMAND(Backup) // opt/config PARSE_RULE_OPTION_COMMAND(Check) // opt/config PARSE_RULE_OPTION_COMMAND(Expire) // opt/config PARSE_RULE_OPTION_COMMAND(Info) // opt/config PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/config PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config PARSE_RULE_OPTION_COMMAND(Restore) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config PARSE_RULE_OPTION_COMMAND(Verify) // opt/config ), // opt/config // opt/config PARSE_RULE_OPTIONAL // opt/config ( // opt/config PARSE_RULE_OPTIONAL_GROUP // opt/config ( // opt/config PARSE_RULE_OPTIONAL_DEFAULT // opt/config ( // opt/config PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE), // opt/config ), // opt/config ), // opt/config ), // opt/config ), // opt/config // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_NAME("config-include-path"), // opt/config-include-path PARSE_RULE_OPTION_TYPE(Path), // opt/config-include-path PARSE_RULE_OPTION_REQUIRED(true), // opt/config-include-path PARSE_RULE_OPTION_SECTION(CommandLine), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Check) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Info) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Server) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Start) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Stop) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-include-path ), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-include-path ), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-include-path ), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Check) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Info) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-include-path ), // opt/config-include-path // opt/config-include-path PARSE_RULE_OPTIONAL // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTIONAL_GROUP // opt/config-include-path ( // opt/config-include-path PARSE_RULE_OPTIONAL_DEFAULT // opt/config-include-path ( // opt/config-include-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH), // opt/config-include-path ), // opt/config-include-path ), // opt/config-include-path ), // opt/config-include-path ), // opt/config-include-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_NAME("config-path"), // opt/config-path PARSE_RULE_OPTION_TYPE(Path), // opt/config-path PARSE_RULE_OPTION_REQUIRED(true), // opt/config-path PARSE_RULE_OPTION_SECTION(CommandLine), // opt/config-path // opt/config-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-path PARSE_RULE_OPTION_COMMAND(Check) // opt/config-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/config-path PARSE_RULE_OPTION_COMMAND(Info) // opt/config-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-path PARSE_RULE_OPTION_COMMAND(Server) // opt/config-path PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config-path PARSE_RULE_OPTION_COMMAND(Start) // opt/config-path PARSE_RULE_OPTION_COMMAND(Stop) // opt/config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-path ), // opt/config-path // opt/config-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-path ), // opt/config-path // opt/config-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-path ), // opt/config-path // opt/config-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/config-path ( // opt/config-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/config-path PARSE_RULE_OPTION_COMMAND(Check) // opt/config-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/config-path PARSE_RULE_OPTION_COMMAND(Info) // opt/config-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/config-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/config-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/config-path ), // opt/config-path // opt/config-path PARSE_RULE_OPTIONAL // opt/config-path ( // opt/config-path PARSE_RULE_OPTIONAL_GROUP // opt/config-path ( // opt/config-path PARSE_RULE_OPTIONAL_DEFAULT // opt/config-path ( // opt/config-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH), // opt/config-path ), // opt/config-path ), // opt/config-path ), // opt/config-path ), // opt/config-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/db-exclude ( // opt/db-exclude PARSE_RULE_OPTION_NAME("db-exclude"), // opt/db-exclude PARSE_RULE_OPTION_TYPE(List), // opt/db-exclude PARSE_RULE_OPTION_RESET(true), // opt/db-exclude PARSE_RULE_OPTION_REQUIRED(false), // opt/db-exclude PARSE_RULE_OPTION_SECTION(Global), // opt/db-exclude PARSE_RULE_OPTION_MULTI(true), // opt/db-exclude // opt/db-exclude PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/db-exclude ( // opt/db-exclude PARSE_RULE_OPTION_COMMAND(Restore) // opt/db-exclude ), // opt/db-exclude ), // opt/db-exclude // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/db-include ( // opt/db-include PARSE_RULE_OPTION_NAME("db-include"), // opt/db-include PARSE_RULE_OPTION_TYPE(List), // opt/db-include PARSE_RULE_OPTION_RESET(true), // opt/db-include PARSE_RULE_OPTION_REQUIRED(false), // opt/db-include PARSE_RULE_OPTION_SECTION(Global), // opt/db-include PARSE_RULE_OPTION_MULTI(true), // opt/db-include // opt/db-include PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/db-include ( // opt/db-include PARSE_RULE_OPTION_COMMAND(Restore) // opt/db-include ), // opt/db-include ), // opt/db-include // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTION_NAME("db-timeout"), // opt/db-timeout PARSE_RULE_OPTION_TYPE(Time), // opt/db-timeout PARSE_RULE_OPTION_RESET(true), // opt/db-timeout PARSE_RULE_OPTION_REQUIRED(true), // opt/db-timeout PARSE_RULE_OPTION_SECTION(Global), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/db-timeout ), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/db-timeout ), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/db-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/db-timeout ), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTIONAL // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTIONAL_GROUP // opt/db-timeout ( // opt/db-timeout PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/db-timeout ( // opt/db-timeout PARSE_RULE_VAL_TIME(100ms), // opt/db-timeout PARSE_RULE_VAL_TIME(7d), // opt/db-timeout ), // opt/db-timeout // opt/db-timeout PARSE_RULE_OPTIONAL_DEFAULT // opt/db-timeout ( // opt/db-timeout PARSE_RULE_VAL_TIME(30m), // opt/db-timeout ), // opt/db-timeout ), // opt/db-timeout ), // opt/db-timeout ), // opt/db-timeout // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/delta ( // opt/delta PARSE_RULE_OPTION_NAME("delta"), // opt/delta PARSE_RULE_OPTION_TYPE(Boolean), // opt/delta PARSE_RULE_OPTION_NEGATE(true), // opt/delta PARSE_RULE_OPTION_RESET(true), // opt/delta PARSE_RULE_OPTION_REQUIRED(true), // opt/delta PARSE_RULE_OPTION_SECTION(Global), // opt/delta // opt/delta PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/delta ( // opt/delta PARSE_RULE_OPTION_COMMAND(Backup) // opt/delta PARSE_RULE_OPTION_COMMAND(Restore) // opt/delta ), // opt/delta // opt/delta PARSE_RULE_OPTIONAL // opt/delta ( // opt/delta PARSE_RULE_OPTIONAL_GROUP // opt/delta ( // opt/delta PARSE_RULE_OPTIONAL_DEFAULT // opt/delta ( // opt/delta PARSE_RULE_VAL_BOOL_FALSE, // opt/delta ), // opt/delta ), // opt/delta ), // opt/delta ), // opt/delta // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/detail-level ( // opt/detail-level PARSE_RULE_OPTION_NAME("detail-level"), // opt/detail-level PARSE_RULE_OPTION_TYPE(StringId), // opt/detail-level PARSE_RULE_OPTION_SEQUENCE(true), // opt/detail-level PARSE_RULE_OPTION_REQUIRED(true), // opt/detail-level PARSE_RULE_OPTION_SECTION(CommandLine), // opt/detail-level // opt/detail-level PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/detail-level ( // opt/detail-level PARSE_RULE_OPTION_COMMAND(Info) // opt/detail-level ), // opt/detail-level // opt/detail-level PARSE_RULE_OPTIONAL // opt/detail-level ( // opt/detail-level PARSE_RULE_OPTIONAL_GROUP // opt/detail-level ( // opt/detail-level PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/detail-level ( // opt/detail-level PARSE_RULE_VAL_STRID(full), // opt/detail-level PARSE_RULE_VAL_STRID(progress), // opt/detail-level ), // opt/detail-level // opt/detail-level PARSE_RULE_OPTIONAL_DEFAULT // opt/detail-level ( // opt/detail-level PARSE_RULE_VAL_STRID(full), // opt/detail-level ), // opt/detail-level ), // opt/detail-level ), // opt/detail-level ), // opt/detail-level // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/dry-run ( // opt/dry-run PARSE_RULE_OPTION_NAME("dry-run"), // opt/dry-run PARSE_RULE_OPTION_TYPE(Boolean), // opt/dry-run PARSE_RULE_OPTION_REQUIRED(true), // opt/dry-run PARSE_RULE_OPTION_SECTION(CommandLine), // opt/dry-run // opt/dry-run PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/dry-run ( // opt/dry-run PARSE_RULE_OPTION_COMMAND(Expire) // opt/dry-run ), // opt/dry-run // opt/dry-run PARSE_RULE_OPTIONAL // opt/dry-run ( // opt/dry-run PARSE_RULE_OPTIONAL_GROUP // opt/dry-run ( // opt/dry-run PARSE_RULE_OPTIONAL_DEFAULT // opt/dry-run ( // opt/dry-run PARSE_RULE_VAL_BOOL_FALSE, // opt/dry-run ), // opt/dry-run ), // opt/dry-run ), // opt/dry-run ), // opt/dry-run // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/exclude ( // opt/exclude PARSE_RULE_OPTION_NAME("exclude"), // opt/exclude PARSE_RULE_OPTION_TYPE(List), // opt/exclude PARSE_RULE_OPTION_RESET(true), // opt/exclude PARSE_RULE_OPTION_REQUIRED(false), // opt/exclude PARSE_RULE_OPTION_SECTION(Global), // opt/exclude PARSE_RULE_OPTION_MULTI(true), // opt/exclude // opt/exclude PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/exclude ( // opt/exclude PARSE_RULE_OPTION_COMMAND(Backup) // opt/exclude ), // opt/exclude ), // opt/exclude // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_NAME("exec-id"), // opt/exec-id PARSE_RULE_OPTION_TYPE(String), // opt/exec-id PARSE_RULE_OPTION_INTERNAL(true) // opt/exec-id PARSE_RULE_OPTION_REQUIRED(false), // opt/exec-id PARSE_RULE_OPTION_SECTION(CommandLine), // opt/exec-id // opt/exec-id PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_COMMAND(Annotate) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Check) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Expire) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Info) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Manifest) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Server) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Start) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Stop) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/exec-id ), // opt/exec-id // opt/exec-id PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/exec-id ), // opt/exec-id // opt/exec-id PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/exec-id ), // opt/exec-id // opt/exec-id PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/exec-id ( // opt/exec-id PARSE_RULE_OPTION_COMMAND(Annotate) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Check) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Expire) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Info) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Manifest) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/exec-id PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/exec-id PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/exec-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/exec-id ), // opt/exec-id ), // opt/exec-id // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/expire-auto ( // opt/expire-auto PARSE_RULE_OPTION_NAME("expire-auto"), // opt/expire-auto PARSE_RULE_OPTION_TYPE(Boolean), // opt/expire-auto PARSE_RULE_OPTION_NEGATE(true), // opt/expire-auto PARSE_RULE_OPTION_RESET(true), // opt/expire-auto PARSE_RULE_OPTION_REQUIRED(true), // opt/expire-auto PARSE_RULE_OPTION_SECTION(Global), // opt/expire-auto // opt/expire-auto PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/expire-auto ( // opt/expire-auto PARSE_RULE_OPTION_COMMAND(Backup) // opt/expire-auto ), // opt/expire-auto // opt/expire-auto PARSE_RULE_OPTIONAL // opt/expire-auto ( // opt/expire-auto PARSE_RULE_OPTIONAL_GROUP // opt/expire-auto ( // opt/expire-auto PARSE_RULE_OPTIONAL_DEFAULT // opt/expire-auto ( // opt/expire-auto PARSE_RULE_VAL_BOOL_TRUE, // opt/expire-auto ), // opt/expire-auto ), // opt/expire-auto ), // opt/expire-auto ), // opt/expire-auto // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/filter ( // opt/filter PARSE_RULE_OPTION_NAME("filter"), // opt/filter PARSE_RULE_OPTION_TYPE(String), // opt/filter PARSE_RULE_OPTION_REQUIRED(false), // opt/filter PARSE_RULE_OPTION_SECTION(CommandLine), // opt/filter // opt/filter PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/filter ( // opt/filter PARSE_RULE_OPTION_COMMAND(Manifest) // opt/filter PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/filter ), // opt/filter ), // opt/filter // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/force ( // opt/force PARSE_RULE_OPTION_NAME("force"), // opt/force PARSE_RULE_OPTION_TYPE(Boolean), // opt/force PARSE_RULE_OPTION_REQUIRED(true), // opt/force PARSE_RULE_OPTION_SECTION(CommandLine), // opt/force // opt/force PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST // opt/force ( // opt/force PARSE_RULE_OPTION_COMMAND_INTERNAL(StanzaCreate, true), // opt/force ) // opt/force // opt/force PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/force ( // opt/force PARSE_RULE_OPTION_COMMAND(Backup) // opt/force PARSE_RULE_OPTION_COMMAND(Restore) // opt/force PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/force PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/force PARSE_RULE_OPTION_COMMAND(Stop) // opt/force ), // opt/force // opt/force PARSE_RULE_OPTIONAL // opt/force ( // opt/force PARSE_RULE_OPTIONAL_GROUP // opt/force ( // opt/force PARSE_RULE_FILTER_CMD // opt/force ( // opt/force PARSE_RULE_VAL_CMD(Backup), // opt/force ), // opt/force // opt/force PARSE_RULE_OPTIONAL_DEPEND // opt/force ( // opt/force PARSE_RULE_VAL_OPT(Online), // opt/force PARSE_RULE_VAL_BOOL_FALSE, // opt/force ), // opt/force // opt/force PARSE_RULE_OPTIONAL_DEFAULT // opt/force ( // opt/force PARSE_RULE_VAL_BOOL_FALSE, // opt/force ), // opt/force ), // opt/force // opt/force PARSE_RULE_OPTIONAL_GROUP // opt/force ( // opt/force PARSE_RULE_OPTIONAL_DEFAULT // opt/force ( // opt/force PARSE_RULE_VAL_BOOL_FALSE, // opt/force ), // opt/force ), // opt/force ), // opt/force ), // opt/force // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/help ( // opt/help PARSE_RULE_OPTION_NAME("help"), // opt/help PARSE_RULE_OPTION_TYPE(Boolean), // opt/help PARSE_RULE_OPTION_REQUIRED(true), // opt/help PARSE_RULE_OPTION_SECTION(CommandLine), // opt/help // opt/help PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/help ( // opt/help PARSE_RULE_OPTION_COMMAND(Help) // opt/help ), // opt/help // opt/help PARSE_RULE_OPTIONAL // opt/help ( // opt/help PARSE_RULE_OPTIONAL_GROUP // opt/help ( // opt/help PARSE_RULE_OPTIONAL_DEFAULT // opt/help ( // opt/help PARSE_RULE_VAL_BOOL_FALSE, // opt/help ), // opt/help ), // opt/help ), // opt/help ), // opt/help // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_OPTION_NAME("ignore-missing"), // opt/ignore-missing PARSE_RULE_OPTION_TYPE(Boolean), // opt/ignore-missing PARSE_RULE_OPTION_REQUIRED(true), // opt/ignore-missing PARSE_RULE_OPTION_SECTION(CommandLine), // opt/ignore-missing // opt/ignore-missing PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/ignore-missing ), // opt/ignore-missing // opt/ignore-missing PARSE_RULE_OPTIONAL // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_OPTIONAL_GROUP // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_OPTIONAL_DEFAULT // opt/ignore-missing ( // opt/ignore-missing PARSE_RULE_VAL_BOOL_FALSE, // opt/ignore-missing ), // opt/ignore-missing ), // opt/ignore-missing ), // opt/ignore-missing ), // opt/ignore-missing // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_NAME("io-timeout"), // opt/io-timeout PARSE_RULE_OPTION_TYPE(Time), // opt/io-timeout PARSE_RULE_OPTION_RESET(true), // opt/io-timeout PARSE_RULE_OPTION_REQUIRED(true), // opt/io-timeout PARSE_RULE_OPTION_SECTION(Global), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Annotate) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Expire) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Info) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Manifest) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Server) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Annotate) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Expire) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Info) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Manifest) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/io-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTIONAL // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTIONAL_GROUP // opt/io-timeout ( // opt/io-timeout PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/io-timeout ( // opt/io-timeout PARSE_RULE_VAL_TIME(100ms), // opt/io-timeout PARSE_RULE_VAL_TIME(1h), // opt/io-timeout ), // opt/io-timeout // opt/io-timeout PARSE_RULE_OPTIONAL_DEFAULT // opt/io-timeout ( // opt/io-timeout PARSE_RULE_VAL_TIME(1m), // opt/io-timeout ), // opt/io-timeout ), // opt/io-timeout ), // opt/io-timeout ), // opt/io-timeout // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTION_NAME("job-retry"), // opt/job-retry PARSE_RULE_OPTION_TYPE(Integer), // opt/job-retry PARSE_RULE_OPTION_INTERNAL(true) // opt/job-retry PARSE_RULE_OPTION_RESET(true), // opt/job-retry PARSE_RULE_OPTION_REQUIRED(true), // opt/job-retry PARSE_RULE_OPTION_SECTION(Global), // opt/job-retry // opt/job-retry PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Backup) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Restore) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Verify) // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Backup) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Restore) // opt/job-retry PARSE_RULE_OPTION_COMMAND(Verify) // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTIONAL_GROUP // opt/job-retry ( // opt/job-retry PARSE_RULE_FILTER_CMD // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_CMD(ArchiveGet), // opt/job-retry PARSE_RULE_VAL_CMD(ArchivePush), // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_INT(0), // opt/job-retry PARSE_RULE_VAL_INT(360), // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL_DEFAULT // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_INT(1), // opt/job-retry ), // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL_GROUP // opt/job-retry ( // opt/job-retry PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_INT(0), // opt/job-retry PARSE_RULE_VAL_INT(360), // opt/job-retry ), // opt/job-retry // opt/job-retry PARSE_RULE_OPTIONAL_DEFAULT // opt/job-retry ( // opt/job-retry PARSE_RULE_VAL_INT(2), // opt/job-retry ), // opt/job-retry ), // opt/job-retry ), // opt/job-retry ), // opt/job-retry // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTION_NAME("job-retry-interval"), // opt/job-retry-interval PARSE_RULE_OPTION_TYPE(Time), // opt/job-retry-interval PARSE_RULE_OPTION_INTERNAL(true) // opt/job-retry-interval PARSE_RULE_OPTION_RESET(true), // opt/job-retry-interval PARSE_RULE_OPTION_REQUIRED(true), // opt/job-retry-interval PARSE_RULE_OPTION_SECTION(Global), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/job-retry-interval ), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry-interval ), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/job-retry-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/job-retry-interval ), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTIONAL // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTIONAL_GROUP // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_VAL_TIME(0s), // opt/job-retry-interval PARSE_RULE_VAL_TIME(15m), // opt/job-retry-interval ), // opt/job-retry-interval // opt/job-retry-interval PARSE_RULE_OPTIONAL_DEFAULT // opt/job-retry-interval ( // opt/job-retry-interval PARSE_RULE_VAL_TIME(15s), // opt/job-retry-interval ), // opt/job-retry-interval ), // opt/job-retry-interval ), // opt/job-retry-interval ), // opt/job-retry-interval // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/link-all ( // opt/link-all PARSE_RULE_OPTION_NAME("link-all"), // opt/link-all PARSE_RULE_OPTION_TYPE(Boolean), // opt/link-all PARSE_RULE_OPTION_NEGATE(true), // opt/link-all PARSE_RULE_OPTION_RESET(true), // opt/link-all PARSE_RULE_OPTION_REQUIRED(true), // opt/link-all PARSE_RULE_OPTION_SECTION(Global), // opt/link-all // opt/link-all PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/link-all ( // opt/link-all PARSE_RULE_OPTION_COMMAND(Restore) // opt/link-all ), // opt/link-all // opt/link-all PARSE_RULE_OPTIONAL // opt/link-all ( // opt/link-all PARSE_RULE_OPTIONAL_GROUP // opt/link-all ( // opt/link-all PARSE_RULE_OPTIONAL_DEFAULT // opt/link-all ( // opt/link-all PARSE_RULE_VAL_BOOL_FALSE, // opt/link-all ), // opt/link-all ), // opt/link-all ), // opt/link-all ), // opt/link-all // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/link-map ( // opt/link-map PARSE_RULE_OPTION_NAME("link-map"), // opt/link-map PARSE_RULE_OPTION_TYPE(Hash), // opt/link-map PARSE_RULE_OPTION_RESET(true), // opt/link-map PARSE_RULE_OPTION_REQUIRED(false), // opt/link-map PARSE_RULE_OPTION_SECTION(Global), // opt/link-map PARSE_RULE_OPTION_MULTI(true), // opt/link-map // opt/link-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/link-map ( // opt/link-map PARSE_RULE_OPTION_COMMAND(Restore) // opt/link-map ), // opt/link-map ), // opt/link-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/lock ( // opt/lock PARSE_RULE_OPTION_NAME("lock"), // opt/lock PARSE_RULE_OPTION_TYPE(List), // opt/lock PARSE_RULE_OPTION_INTERNAL(true) // opt/lock PARSE_RULE_OPTION_REQUIRED(false), // opt/lock PARSE_RULE_OPTION_SECTION(CommandLine), // opt/lock PARSE_RULE_OPTION_MULTI(true), // opt/lock // opt/lock PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/lock ( // opt/lock PARSE_RULE_OPTION_COMMAND(Annotate) // opt/lock PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock PARSE_RULE_OPTION_COMMAND(Backup) // opt/lock PARSE_RULE_OPTION_COMMAND(Expire) // opt/lock PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/lock PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/lock PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/lock ), // opt/lock ), // opt/lock // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_NAME("lock-path"), // opt/lock-path PARSE_RULE_OPTION_TYPE(Path), // opt/lock-path PARSE_RULE_OPTION_RESET(true), // opt/lock-path PARSE_RULE_OPTION_REQUIRED(true), // opt/lock-path PARSE_RULE_OPTION_SECTION(Global), // opt/lock-path // opt/lock-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Info) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Start) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Stop) // opt/lock-path ), // opt/lock-path // opt/lock-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock-path ), // opt/lock-path // opt/lock-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/lock-path ), // opt/lock-path // opt/lock-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/lock-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Info) // opt/lock-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/lock-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/lock-path ), // opt/lock-path // opt/lock-path PARSE_RULE_OPTIONAL // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTIONAL_GROUP // opt/lock-path ( // opt/lock-path PARSE_RULE_OPTIONAL_DEFAULT // opt/lock-path ( // opt/lock-path PARSE_RULE_VAL_STR(QT_FS_tmp_FS_pgbackrest_QT), // opt/lock-path ), // opt/lock-path ), // opt/lock-path ), // opt/lock-path ), // opt/lock-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_NAME("log-level-console"), // opt/log-level-console PARSE_RULE_OPTION_TYPE(StringId), // opt/log-level-console PARSE_RULE_OPTION_SEQUENCE(true), // opt/log-level-console PARSE_RULE_OPTION_RESET(true), // opt/log-level-console PARSE_RULE_OPTION_REQUIRED(true), // opt/log-level-console PARSE_RULE_OPTION_SECTION(Global), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Server) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Start) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-console PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTIONAL // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTIONAL_GROUP // opt/log-level-console ( // opt/log-level-console PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/log-level-console ( // opt/log-level-console PARSE_RULE_VAL_STRID(off), // opt/log-level-console PARSE_RULE_VAL_STRID(error), // opt/log-level-console PARSE_RULE_VAL_STRID(warn), // opt/log-level-console PARSE_RULE_VAL_STRID(info), // opt/log-level-console PARSE_RULE_VAL_STRID(detail), // opt/log-level-console PARSE_RULE_VAL_STRID(debug), // opt/log-level-console PARSE_RULE_VAL_STRID(trace), // opt/log-level-console ), // opt/log-level-console // opt/log-level-console PARSE_RULE_OPTIONAL_DEFAULT // opt/log-level-console ( // opt/log-level-console PARSE_RULE_VAL_STRID(warn), // opt/log-level-console PARSE_RULE_VAL_SEQ(2), // opt/log-level-console ), // opt/log-level-console ), // opt/log-level-console ), // opt/log-level-console ), // opt/log-level-console // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_NAME("log-level-file"), // opt/log-level-file PARSE_RULE_OPTION_TYPE(StringId), // opt/log-level-file PARSE_RULE_OPTION_SEQUENCE(true), // opt/log-level-file PARSE_RULE_OPTION_RESET(true), // opt/log-level-file PARSE_RULE_OPTION_REQUIRED(true), // opt/log-level-file PARSE_RULE_OPTION_SECTION(Global), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Server) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Start) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTIONAL // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTIONAL_GROUP // opt/log-level-file ( // opt/log-level-file PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/log-level-file ( // opt/log-level-file PARSE_RULE_VAL_STRID(off), // opt/log-level-file PARSE_RULE_VAL_STRID(error), // opt/log-level-file PARSE_RULE_VAL_STRID(warn), // opt/log-level-file PARSE_RULE_VAL_STRID(info), // opt/log-level-file PARSE_RULE_VAL_STRID(detail), // opt/log-level-file PARSE_RULE_VAL_STRID(debug), // opt/log-level-file PARSE_RULE_VAL_STRID(trace), // opt/log-level-file ), // opt/log-level-file // opt/log-level-file PARSE_RULE_OPTIONAL_DEFAULT // opt/log-level-file ( // opt/log-level-file PARSE_RULE_VAL_STRID(info), // opt/log-level-file PARSE_RULE_VAL_SEQ(3), // opt/log-level-file ), // opt/log-level-file ), // opt/log-level-file ), // opt/log-level-file ), // opt/log-level-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_NAME("log-level-stderr"), // opt/log-level-stderr PARSE_RULE_OPTION_TYPE(StringId), // opt/log-level-stderr PARSE_RULE_OPTION_SEQUENCE(true), // opt/log-level-stderr PARSE_RULE_OPTION_RESET(true), // opt/log-level-stderr PARSE_RULE_OPTION_REQUIRED(true), // opt/log-level-stderr PARSE_RULE_OPTION_SECTION(Global), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Server) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Start) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Check) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Info) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-level-stderr PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTIONAL // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTIONAL_GROUP // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_VAL_STRID(off), // opt/log-level-stderr PARSE_RULE_VAL_STRID(error), // opt/log-level-stderr PARSE_RULE_VAL_STRID(warn), // opt/log-level-stderr PARSE_RULE_VAL_STRID(info), // opt/log-level-stderr PARSE_RULE_VAL_STRID(detail), // opt/log-level-stderr PARSE_RULE_VAL_STRID(debug), // opt/log-level-stderr PARSE_RULE_VAL_STRID(trace), // opt/log-level-stderr ), // opt/log-level-stderr // opt/log-level-stderr PARSE_RULE_OPTIONAL_DEFAULT // opt/log-level-stderr ( // opt/log-level-stderr PARSE_RULE_VAL_STRID(off), // opt/log-level-stderr ), // opt/log-level-stderr ), // opt/log-level-stderr ), // opt/log-level-stderr ), // opt/log-level-stderr // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_NAME("log-path"), // opt/log-path PARSE_RULE_OPTION_TYPE(Path), // opt/log-path PARSE_RULE_OPTION_RESET(true), // opt/log-path PARSE_RULE_OPTION_REQUIRED(true), // opt/log-path PARSE_RULE_OPTION_SECTION(Global), // opt/log-path // opt/log-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-path PARSE_RULE_OPTION_COMMAND(Check) // opt/log-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-path PARSE_RULE_OPTION_COMMAND(Info) // opt/log-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-path PARSE_RULE_OPTION_COMMAND(Server) // opt/log-path PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-path PARSE_RULE_OPTION_COMMAND(Start) // opt/log-path PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-path ), // opt/log-path // opt/log-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-path ), // opt/log-path // opt/log-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-path ), // opt/log-path // opt/log-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-path ( // opt/log-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-path PARSE_RULE_OPTION_COMMAND(Check) // opt/log-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-path PARSE_RULE_OPTION_COMMAND(Info) // opt/log-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-path ), // opt/log-path // opt/log-path PARSE_RULE_OPTIONAL // opt/log-path ( // opt/log-path PARSE_RULE_OPTIONAL_GROUP // opt/log-path ( // opt/log-path PARSE_RULE_OPTIONAL_DEFAULT // opt/log-path ( // opt/log-path PARSE_RULE_VAL_STR(QT_FS_var_FS_log_FS_pgbackrest_QT), // opt/log-path ), // opt/log-path ), // opt/log-path ), // opt/log-path ), // opt/log-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_NAME("log-subprocess"), // opt/log-subprocess PARSE_RULE_OPTION_TYPE(Boolean), // opt/log-subprocess PARSE_RULE_OPTION_NEGATE(true), // opt/log-subprocess PARSE_RULE_OPTION_RESET(true), // opt/log-subprocess PARSE_RULE_OPTION_REQUIRED(true), // opt/log-subprocess PARSE_RULE_OPTION_SECTION(Global), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Check) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Info) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-subprocess ), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-subprocess ), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-subprocess ), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Check) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Info) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-subprocess PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-subprocess ), // opt/log-subprocess // opt/log-subprocess PARSE_RULE_OPTIONAL // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTIONAL_GROUP // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_OPTIONAL_DEFAULT // opt/log-subprocess ( // opt/log-subprocess PARSE_RULE_VAL_BOOL_FALSE, // opt/log-subprocess ), // opt/log-subprocess ), // opt/log-subprocess ), // opt/log-subprocess ), // opt/log-subprocess // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_NAME("log-timestamp"), // opt/log-timestamp PARSE_RULE_OPTION_TYPE(Boolean), // opt/log-timestamp PARSE_RULE_OPTION_NEGATE(true), // opt/log-timestamp PARSE_RULE_OPTION_RESET(true), // opt/log-timestamp PARSE_RULE_OPTION_REQUIRED(true), // opt/log-timestamp PARSE_RULE_OPTION_SECTION(Global), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Check) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Info) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Server) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Start) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Stop) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-timestamp ), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-timestamp ), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-timestamp ), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Annotate) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Backup) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Check) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Expire) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Info) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Manifest) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Restore) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/log-timestamp PARSE_RULE_OPTION_COMMAND(Verify) // opt/log-timestamp ), // opt/log-timestamp // opt/log-timestamp PARSE_RULE_OPTIONAL // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTIONAL_GROUP // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_OPTIONAL_DEFAULT // opt/log-timestamp ( // opt/log-timestamp PARSE_RULE_VAL_BOOL_TRUE, // opt/log-timestamp ), // opt/log-timestamp ), // opt/log-timestamp ), // opt/log-timestamp ), // opt/log-timestamp // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_OPTION_NAME("manifest-save-threshold"), // opt/manifest-save-threshold PARSE_RULE_OPTION_TYPE(Size), // opt/manifest-save-threshold PARSE_RULE_OPTION_RESET(true), // opt/manifest-save-threshold PARSE_RULE_OPTION_REQUIRED(true), // opt/manifest-save-threshold PARSE_RULE_OPTION_SECTION(Global), // opt/manifest-save-threshold // opt/manifest-save-threshold PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_OPTION_COMMAND(Backup) // opt/manifest-save-threshold ), // opt/manifest-save-threshold // opt/manifest-save-threshold PARSE_RULE_OPTIONAL // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_OPTIONAL_GROUP // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_VAL_SIZE(1B), // opt/manifest-save-threshold PARSE_RULE_VAL_SIZE(1TiB), // opt/manifest-save-threshold ), // opt/manifest-save-threshold // opt/manifest-save-threshold PARSE_RULE_OPTIONAL_DEFAULT // opt/manifest-save-threshold ( // opt/manifest-save-threshold PARSE_RULE_VAL_SIZE(1GiB), // opt/manifest-save-threshold ), // opt/manifest-save-threshold ), // opt/manifest-save-threshold ), // opt/manifest-save-threshold ), // opt/manifest-save-threshold // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_NAME("neutral-umask"), // opt/neutral-umask PARSE_RULE_OPTION_TYPE(Boolean), // opt/neutral-umask PARSE_RULE_OPTION_NEGATE(true), // opt/neutral-umask PARSE_RULE_OPTION_RESET(true), // opt/neutral-umask PARSE_RULE_OPTION_REQUIRED(true), // opt/neutral-umask PARSE_RULE_OPTION_SECTION(Global), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Annotate) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Backup) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Check) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Expire) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Restore) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Start) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Stop) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Verify) // opt/neutral-umask ), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/neutral-umask ), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Backup) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Restore) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Verify) // opt/neutral-umask ), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Annotate) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Backup) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Check) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Expire) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Restore) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/neutral-umask PARSE_RULE_OPTION_COMMAND(Verify) // opt/neutral-umask ), // opt/neutral-umask // opt/neutral-umask PARSE_RULE_OPTIONAL // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTIONAL_GROUP // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_OPTIONAL_DEFAULT // opt/neutral-umask ( // opt/neutral-umask PARSE_RULE_VAL_BOOL_TRUE, // opt/neutral-umask ), // opt/neutral-umask ), // opt/neutral-umask ), // opt/neutral-umask ), // opt/neutral-umask // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/oldest ( // opt/oldest PARSE_RULE_OPTION_NAME("oldest"), // opt/oldest PARSE_RULE_OPTION_TYPE(Boolean), // opt/oldest PARSE_RULE_OPTION_REQUIRED(true), // opt/oldest PARSE_RULE_OPTION_SECTION(CommandLine), // opt/oldest // opt/oldest PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/oldest ( // opt/oldest PARSE_RULE_OPTION_COMMAND(Expire) // opt/oldest ), // opt/oldest // opt/oldest PARSE_RULE_OPTIONAL // opt/oldest ( // opt/oldest PARSE_RULE_OPTIONAL_GROUP // opt/oldest ( // opt/oldest PARSE_RULE_FILTER_CMD // opt/oldest ( // opt/oldest PARSE_RULE_VAL_CMD(Expire), // opt/oldest ), // opt/oldest // opt/oldest PARSE_RULE_OPTIONAL_DEPEND // opt/oldest ( // opt/oldest PARSE_RULE_VAL_OPT(Stanza), // opt/oldest ), // opt/oldest // opt/oldest PARSE_RULE_OPTIONAL_DEFAULT // opt/oldest ( // opt/oldest PARSE_RULE_VAL_BOOL_FALSE, // opt/oldest ), // opt/oldest ), // opt/oldest // opt/oldest PARSE_RULE_OPTIONAL_GROUP // opt/oldest ( // opt/oldest PARSE_RULE_OPTIONAL_DEFAULT // opt/oldest ( // opt/oldest PARSE_RULE_VAL_BOOL_FALSE, // opt/oldest ), // opt/oldest ), // opt/oldest ), // opt/oldest ), // opt/oldest // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/online ( // opt/online PARSE_RULE_OPTION_NAME("online"), // opt/online PARSE_RULE_OPTION_TYPE(Boolean), // opt/online PARSE_RULE_OPTION_NEGATE(true), // opt/online PARSE_RULE_OPTION_REQUIRED(true), // opt/online PARSE_RULE_OPTION_SECTION(CommandLine), // opt/online // opt/online PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/online ( // opt/online PARSE_RULE_OPTION_COMMAND(Backup) // opt/online PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/online PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/online ), // opt/online // opt/online PARSE_RULE_OPTIONAL // opt/online ( // opt/online PARSE_RULE_OPTIONAL_GROUP // opt/online ( // opt/online PARSE_RULE_OPTIONAL_DEFAULT // opt/online ( // opt/online PARSE_RULE_VAL_BOOL_TRUE, // opt/online ), // opt/online ), // opt/online ), // opt/online ), // opt/online // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/output ( // opt/output PARSE_RULE_OPTION_NAME("output"), // opt/output PARSE_RULE_OPTION_TYPE(StringId), // opt/output PARSE_RULE_OPTION_SEQUENCE(true), // opt/output PARSE_RULE_OPTION_REQUIRED(true), // opt/output PARSE_RULE_OPTION_SECTION(CommandLine), // opt/output // opt/output PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/output ( // opt/output PARSE_RULE_OPTION_COMMAND(Info) // opt/output PARSE_RULE_OPTION_COMMAND(Manifest) // opt/output PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/output PARSE_RULE_OPTION_COMMAND(Verify) // opt/output PARSE_RULE_OPTION_COMMAND(Version) // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL // opt/output ( // opt/output PARSE_RULE_OPTIONAL_GROUP // opt/output ( // opt/output PARSE_RULE_FILTER_CMD // opt/output ( // opt/output PARSE_RULE_VAL_CMD(Verify), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/output ( // opt/output PARSE_RULE_VAL_STRID(none), // opt/output PARSE_RULE_VAL_STRID(text), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_DEFAULT // opt/output ( // opt/output PARSE_RULE_VAL_STRID(none), // opt/output ), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_GROUP // opt/output ( // opt/output PARSE_RULE_FILTER_CMD // opt/output ( // opt/output PARSE_RULE_VAL_CMD(Version), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/output ( // opt/output PARSE_RULE_VAL_STRID(text), // opt/output PARSE_RULE_VAL_STRID(num), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_DEFAULT // opt/output ( // opt/output PARSE_RULE_VAL_STRID(text), // opt/output ), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_GROUP // opt/output ( // opt/output PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/output ( // opt/output PARSE_RULE_VAL_STRID(text), // opt/output PARSE_RULE_VAL_STRID(json), // opt/output ), // opt/output // opt/output PARSE_RULE_OPTIONAL_DEFAULT // opt/output ( // opt/output PARSE_RULE_VAL_STRID(text), // opt/output ), // opt/output ), // opt/output ), // opt/output ), // opt/output // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/page-header-check ( // opt/page-header-check PARSE_RULE_OPTION_NAME("page-header-check"), // opt/page-header-check PARSE_RULE_OPTION_TYPE(Boolean), // opt/page-header-check PARSE_RULE_OPTION_NEGATE(true), // opt/page-header-check PARSE_RULE_OPTION_RESET(true), // opt/page-header-check PARSE_RULE_OPTION_REQUIRED(true), // opt/page-header-check PARSE_RULE_OPTION_SECTION(Global), // opt/page-header-check // opt/page-header-check PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/page-header-check ( // opt/page-header-check PARSE_RULE_OPTION_COMMAND(Backup) // opt/page-header-check ), // opt/page-header-check // opt/page-header-check PARSE_RULE_OPTIONAL // opt/page-header-check ( // opt/page-header-check PARSE_RULE_OPTIONAL_GROUP // opt/page-header-check ( // opt/page-header-check PARSE_RULE_OPTIONAL_DEFAULT // opt/page-header-check ( // opt/page-header-check PARSE_RULE_VAL_BOOL_TRUE, // opt/page-header-check ), // opt/page-header-check ), // opt/page-header-check ), // opt/page-header-check ), // opt/page-header-check // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg ( // opt/pg PARSE_RULE_OPTION_NAME("pg"), // opt/pg PARSE_RULE_OPTION_TYPE(Integer), // opt/pg PARSE_RULE_OPTION_INTERNAL(true) // opt/pg PARSE_RULE_OPTION_REQUIRED(false), // opt/pg PARSE_RULE_OPTION_SECTION(CommandLine), // opt/pg // opt/pg PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg ( // opt/pg PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg ), // opt/pg // opt/pg PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg ( // opt/pg PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg ), // opt/pg // opt/pg PARSE_RULE_OPTIONAL // opt/pg ( // opt/pg PARSE_RULE_OPTIONAL_GROUP // opt/pg ( // opt/pg PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/pg ( // opt/pg PARSE_RULE_VAL_INT(1), // opt/pg PARSE_RULE_VAL_INT(256), // opt/pg ), // opt/pg ), // opt/pg ), // opt/pg ), // opt/pg // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTION_NAME("pg-database"), // opt/pg-database PARSE_RULE_OPTION_TYPE(String), // opt/pg-database PARSE_RULE_OPTION_RESET(true), // opt/pg-database PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-database PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-database PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-database // opt/pg-database PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-database PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-database ), // opt/pg-database // opt/pg-database PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-database ), // opt/pg-database // opt/pg-database PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-database PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-database PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-database ), // opt/pg-database // opt/pg-database PARSE_RULE_OPTIONAL // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTIONAL_GROUP // opt/pg-database ( // opt/pg-database PARSE_RULE_OPTIONAL_DEPEND // opt/pg-database ( // opt/pg-database PARSE_RULE_VAL_OPT(PgPath), // opt/pg-database ), // opt/pg-database // opt/pg-database PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-database ( // opt/pg-database PARSE_RULE_VAL_STR(QT_postgres_QT), // opt/pg-database ), // opt/pg-database ), // opt/pg-database ), // opt/pg-database ), // opt/pg-database // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_NAME("pg-host"), // opt/pg-host PARSE_RULE_OPTION_TYPE(String), // opt/pg-host PARSE_RULE_OPTION_RESET(true), // opt/pg-host PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host // opt/pg-host PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_COMMAND_INTERNAL(ArchiveGet, true), // opt/pg-host PARSE_RULE_OPTION_COMMAND_INTERNAL(ArchivePush, true), // opt/pg-host PARSE_RULE_OPTION_COMMAND_INTERNAL(Restore, true), // opt/pg-host ) // opt/pg-host // opt/pg-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host ), // opt/pg-host // opt/pg-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-host ), // opt/pg-host // opt/pg-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-host ), // opt/pg-host // opt/pg-host PARSE_RULE_OPTIONAL // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTIONAL_GROUP // opt/pg-host ( // opt/pg-host PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host ( // opt/pg-host PARSE_RULE_VAL_OPT(PgLocal), // opt/pg-host PARSE_RULE_VAL_BOOL_FALSE, // opt/pg-host ), // opt/pg-host ), // opt/pg-host ), // opt/pg-host ), // opt/pg-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTION_NAME("pg-host-ca-file"), // opt/pg-host-ca-file PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-ca-file PARSE_RULE_OPTION_RESET(true), // opt/pg-host-ca-file PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host-ca-file PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-ca-file PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-ca-file // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-ca-file ), // opt/pg-host-ca-file // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-ca-file ), // opt/pg-host-ca-file // opt/pg-host-ca-file PARSE_RULE_OPTIONAL // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-ca-file ( // opt/pg-host-ca-file PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-ca-file PARSE_RULE_VAL_STRID(tls), // opt/pg-host-ca-file ), // opt/pg-host-ca-file ), // opt/pg-host-ca-file ), // opt/pg-host-ca-file ), // opt/pg-host-ca-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTION_NAME("pg-host-ca-path"), // opt/pg-host-ca-path PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-ca-path PARSE_RULE_OPTION_RESET(true), // opt/pg-host-ca-path PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host-ca-path PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-ca-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-ca-path // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-ca-path ), // opt/pg-host-ca-path // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-ca-path ), // opt/pg-host-ca-path // opt/pg-host-ca-path PARSE_RULE_OPTIONAL // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-ca-path ( // opt/pg-host-ca-path PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-ca-path PARSE_RULE_VAL_STRID(tls), // opt/pg-host-ca-path ), // opt/pg-host-ca-path ), // opt/pg-host-ca-path ), // opt/pg-host-ca-path ), // opt/pg-host-ca-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTION_NAME("pg-host-cert-file"), // opt/pg-host-cert-file PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-cert-file PARSE_RULE_OPTION_RESET(true), // opt/pg-host-cert-file PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-cert-file PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-cert-file PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-cert-file // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-cert-file ), // opt/pg-host-cert-file // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-cert-file ), // opt/pg-host-cert-file // opt/pg-host-cert-file PARSE_RULE_OPTIONAL // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-cert-file ( // opt/pg-host-cert-file PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-cert-file PARSE_RULE_VAL_STRID(tls), // opt/pg-host-cert-file ), // opt/pg-host-cert-file ), // opt/pg-host-cert-file ), // opt/pg-host-cert-file ), // opt/pg-host-cert-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTION_NAME("pg-host-cmd"), // opt/pg-host-cmd PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-cmd PARSE_RULE_OPTION_DEFAULT_TYPE(Dynamic), // opt/pg-host-cmd PARSE_RULE_OPTION_RESET(true), // opt/pg-host-cmd PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-cmd PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-cmd PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-cmd // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-cmd ), // opt/pg-host-cmd // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-cmd ), // opt/pg-host-cmd // opt/pg-host-cmd PARSE_RULE_OPTIONAL // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-cmd PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-cmd ), // opt/pg-host-cmd // opt/pg-host-cmd PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-cmd ( // opt/pg-host-cmd PARSE_RULE_DEFAULT_DYNAMIC(Bin), // opt/pg-host-cmd ), // opt/pg-host-cmd ), // opt/pg-host-cmd ), // opt/pg-host-cmd ), // opt/pg-host-cmd // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTION_NAME("pg-host-config"), // opt/pg-host-config PARSE_RULE_OPTION_TYPE(Path), // opt/pg-host-config PARSE_RULE_OPTION_RESET(true), // opt/pg-host-config PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-config PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-config PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-config // opt/pg-host-config PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-config ), // opt/pg-host-config // opt/pg-host-config PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config ), // opt/pg-host-config // opt/pg-host-config PARSE_RULE_OPTIONAL // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_VAL_OPT(PgHost), // opt/pg-host-config ), // opt/pg-host-config // opt/pg-host-config PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-config ( // opt/pg-host-config PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE), // opt/pg-host-config ), // opt/pg-host-config ), // opt/pg-host-config ), // opt/pg-host-config ), // opt/pg-host-config // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTION_NAME("pg-host-config-include-path"), // opt/pg-host-config-include-path PARSE_RULE_OPTION_TYPE(Path), // opt/pg-host-config-include-path PARSE_RULE_OPTION_RESET(true), // opt/pg-host-config-include-path PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-config-include-path PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-config-include-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-config-include-path // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path // opt/pg-host-config-include-path PARSE_RULE_OPTIONAL // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_VAL_OPT(PgHost), // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path // opt/pg-host-config-include-path PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-config-include-path ( // opt/pg-host-config-include-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH), ), // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path ), // opt/pg-host-config-include-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTION_NAME("pg-host-config-path"), // opt/pg-host-config-path PARSE_RULE_OPTION_TYPE(Path), // opt/pg-host-config-path PARSE_RULE_OPTION_RESET(true), // opt/pg-host-config-path PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-config-path PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-config-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-config-path // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-config-path ), // opt/pg-host-config-path // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-config-path ), // opt/pg-host-config-path // opt/pg-host-config-path PARSE_RULE_OPTIONAL // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_VAL_OPT(PgHost), // opt/pg-host-config-path ), // opt/pg-host-config-path // opt/pg-host-config-path PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-config-path ( // opt/pg-host-config-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH), // opt/pg-host-config-path ), // opt/pg-host-config-path ), // opt/pg-host-config-path ), // opt/pg-host-config-path ), // opt/pg-host-config-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTION_NAME("pg-host-key-file"), // opt/pg-host-key-file PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-key-file PARSE_RULE_OPTION_RESET(true), // opt/pg-host-key-file PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-key-file PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-key-file // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-key-file ), // opt/pg-host-key-file // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-key-file ), // opt/pg-host-key-file // opt/pg-host-key-file PARSE_RULE_OPTIONAL // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-key-file ( // opt/pg-host-key-file PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-key-file PARSE_RULE_VAL_STRID(tls), // opt/pg-host-key-file ), // opt/pg-host-key-file ), // opt/pg-host-key-file ), // opt/pg-host-key-file ), // opt/pg-host-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTION_NAME("pg-host-port"), // opt/pg-host-port PARSE_RULE_OPTION_TYPE(Integer), // opt/pg-host-port PARSE_RULE_OPTION_RESET(true), // opt/pg-host-port PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host-port PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-port PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-port ), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-port ), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTIONAL // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-port PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-port PARSE_RULE_VAL_STRID(tls), // opt/pg-host-port ), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_VAL_INT(0), // opt/pg-host-port PARSE_RULE_VAL_INT(65535), // opt/pg-host-port ), // opt/pg-host-port // opt/pg-host-port PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_OPTIONAL_DEFAULT_MAP // opt/pg-host-port ( // opt/pg-host-port PARSE_RULE_VAL_STRID(tls), // opt/pg-host-port PARSE_RULE_VAL_INT(8432), // opt/pg-host-port ), // opt/pg-host-port ), // opt/pg-host-port ), // opt/pg-host-port ), // opt/pg-host-port ), // opt/pg-host-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTION_NAME("pg-host-type"), // opt/pg-host-type PARSE_RULE_OPTION_TYPE(StringId), // opt/pg-host-type PARSE_RULE_OPTION_RESET(true), // opt/pg-host-type PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-host-type PARSE_RULE_OPTION_SECTION(Global), // opt/pg-host-type PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-type ), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-type ), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTIONAL // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_VAL_OPT(PgHost), // opt/pg-host-type ), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-type PARSE_RULE_VAL_STRID(tls), // opt/pg-host-type ), // opt/pg-host-type // opt/pg-host-type PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-type ( // opt/pg-host-type PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-type ), // opt/pg-host-type ), // opt/pg-host-type ), // opt/pg-host-type ), // opt/pg-host-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTION_NAME("pg-host-user"), // opt/pg-host-user PARSE_RULE_OPTION_TYPE(String), // opt/pg-host-user PARSE_RULE_OPTION_RESET(true), // opt/pg-host-user PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-host-user PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-host-user PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-host-user // opt/pg-host-user PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-host-user ), // opt/pg-host-user // opt/pg-host-user PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-host-user ), // opt/pg-host-user // opt/pg-host-user PARSE_RULE_OPTIONAL // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTIONAL_GROUP // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_OPTIONAL_DEPEND // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_VAL_OPT(PgHostType), // opt/pg-host-user PARSE_RULE_VAL_STRID(ssh), // opt/pg-host-user ), // opt/pg-host-user // opt/pg-host-user PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-host-user ( // opt/pg-host-user PARSE_RULE_VAL_STR(QT_postgres_QT), // opt/pg-host-user ), // opt/pg-host-user ), // opt/pg-host-user ), // opt/pg-host-user ), // opt/pg-host-user // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTION_NAME("pg-local"), // opt/pg-local PARSE_RULE_OPTION_TYPE(Boolean), // opt/pg-local PARSE_RULE_OPTION_INTERNAL(true) // opt/pg-local PARSE_RULE_OPTION_NEGATE(true), // opt/pg-local PARSE_RULE_OPTION_RESET(true), // opt/pg-local PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-local PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-local PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-local // opt/pg-local PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-local PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-local PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-local PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-local ), // opt/pg-local // opt/pg-local PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-local ), // opt/pg-local // opt/pg-local PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-local PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-local ), // opt/pg-local // opt/pg-local PARSE_RULE_OPTIONAL // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTIONAL_GROUP // opt/pg-local ( // opt/pg-local PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-local ( // opt/pg-local PARSE_RULE_VAL_BOOL_FALSE, // opt/pg-local ), // opt/pg-local ), // opt/pg-local ), // opt/pg-local ), // opt/pg-local // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_NAME("pg-path"), // opt/pg-path PARSE_RULE_OPTION_TYPE(Path), // opt/pg-path PARSE_RULE_OPTION_RESET(true), // opt/pg-path PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-path PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-path // opt/pg-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTIONAL // opt/pg-path ( // opt/pg-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-path ( // opt/pg-path PARSE_RULE_FILTER_CMD // opt/pg-path ( // opt/pg-path PARSE_RULE_VAL_CMD(ArchivePush), // opt/pg-path PARSE_RULE_VAL_CMD(Manifest), // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTIONAL_NOT_REQUIRED(), // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-path ( // opt/pg-path PARSE_RULE_FILTER_CMD // opt/pg-path ( // opt/pg-path PARSE_RULE_VAL_CMD(Check), // opt/pg-path ), // opt/pg-path // opt/pg-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-path ( // opt/pg-path PARSE_RULE_VAL_OPT(Stanza), // opt/pg-path ), // opt/pg-path ), // opt/pg-path ), // opt/pg-path ), // opt/pg-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTION_NAME("pg-port"), // opt/pg-port PARSE_RULE_OPTION_TYPE(Integer), // opt/pg-port PARSE_RULE_OPTION_RESET(true), // opt/pg-port PARSE_RULE_OPTION_REQUIRED(true), // opt/pg-port PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-port PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-port // opt/pg-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-port PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-port PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTIONAL // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTIONAL_GROUP // opt/pg-port ( // opt/pg-port PARSE_RULE_OPTIONAL_DEPEND // opt/pg-port ( // opt/pg-port PARSE_RULE_VAL_OPT(PgPath), // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/pg-port ( // opt/pg-port PARSE_RULE_VAL_INT(0), // opt/pg-port PARSE_RULE_VAL_INT(65535), // opt/pg-port ), // opt/pg-port // opt/pg-port PARSE_RULE_OPTIONAL_DEFAULT // opt/pg-port ( // opt/pg-port PARSE_RULE_VAL_INT(5432), // opt/pg-port ), // opt/pg-port ), // opt/pg-port ), // opt/pg-port ), // opt/pg-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTION_NAME("pg-socket-path"), // opt/pg-socket-path PARSE_RULE_OPTION_TYPE(String), // opt/pg-socket-path PARSE_RULE_OPTION_RESET(true), // opt/pg-socket-path PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-socket-path PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-socket-path PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-socket-path // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-socket-path ), // opt/pg-socket-path // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-socket-path ), // opt/pg-socket-path // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-socket-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-socket-path ), // opt/pg-socket-path // opt/pg-socket-path PARSE_RULE_OPTIONAL // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTIONAL_GROUP // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_OPTIONAL_DEPEND // opt/pg-socket-path ( // opt/pg-socket-path PARSE_RULE_VAL_OPT(PgPath), // opt/pg-socket-path ), // opt/pg-socket-path ), // opt/pg-socket-path ), // opt/pg-socket-path ), // opt/pg-socket-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTION_NAME("pg-user"), // opt/pg-user PARSE_RULE_OPTION_TYPE(String), // opt/pg-user PARSE_RULE_OPTION_RESET(true), // opt/pg-user PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-user PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-user PARSE_RULE_OPTION_GROUP_ID(Pg), // opt/pg-user // opt/pg-user PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-user PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-user ), // opt/pg-user // opt/pg-user PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-user ), // opt/pg-user // opt/pg-user PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-user PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-user ), // opt/pg-user // opt/pg-user PARSE_RULE_OPTIONAL // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTIONAL_GROUP // opt/pg-user ( // opt/pg-user PARSE_RULE_OPTIONAL_DEPEND // opt/pg-user ( // opt/pg-user PARSE_RULE_VAL_OPT(PgPath), // opt/pg-user ), // opt/pg-user ), // opt/pg-user ), // opt/pg-user ), // opt/pg-user // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_NAME("pg-version-force"), // opt/pg-version-force PARSE_RULE_OPTION_TYPE(String), // opt/pg-version-force PARSE_RULE_OPTION_RESET(true), // opt/pg-version-force PARSE_RULE_OPTION_REQUIRED(false), // opt/pg-version-force PARSE_RULE_OPTION_SECTION(Stanza), // opt/pg-version-force // opt/pg-version-force PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Verify) // opt/pg-version-force ), // opt/pg-version-force // opt/pg-version-force PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-version-force ), // opt/pg-version-force // opt/pg-version-force PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Verify) // opt/pg-version-force ), // opt/pg-version-force // opt/pg-version-force PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/pg-version-force ( // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Backup) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Check) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Manifest) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Restore) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/pg-version-force PARSE_RULE_OPTION_COMMAND(Verify) // opt/pg-version-force ), // opt/pg-version-force ), // opt/pg-version-force // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/priority ( // opt/priority PARSE_RULE_OPTION_NAME("priority"), // opt/priority PARSE_RULE_OPTION_TYPE(Integer), // opt/priority PARSE_RULE_OPTION_RESET(true), // opt/priority PARSE_RULE_OPTION_REQUIRED(false), // opt/priority PARSE_RULE_OPTION_SECTION(Global), // opt/priority // opt/priority PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/priority ( // opt/priority PARSE_RULE_OPTION_COMMAND(Annotate) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/priority PARSE_RULE_OPTION_COMMAND(Backup) // opt/priority PARSE_RULE_OPTION_COMMAND(Check) // opt/priority PARSE_RULE_OPTION_COMMAND(Expire) // opt/priority PARSE_RULE_OPTION_COMMAND(Info) // opt/priority PARSE_RULE_OPTION_COMMAND(Manifest) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/priority PARSE_RULE_OPTION_COMMAND(Restore) // opt/priority PARSE_RULE_OPTION_COMMAND(Server) // opt/priority PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/priority PARSE_RULE_OPTION_COMMAND(Start) // opt/priority PARSE_RULE_OPTION_COMMAND(Stop) // opt/priority PARSE_RULE_OPTION_COMMAND(Verify) // opt/priority ), // opt/priority // opt/priority PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/priority ( // opt/priority PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/priority ), // opt/priority // opt/priority PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/priority ( // opt/priority PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/priority PARSE_RULE_OPTION_COMMAND(Backup) // opt/priority PARSE_RULE_OPTION_COMMAND(Restore) // opt/priority PARSE_RULE_OPTION_COMMAND(Verify) // opt/priority ), // opt/priority // opt/priority PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/priority ( // opt/priority PARSE_RULE_OPTION_COMMAND(Annotate) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/priority PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/priority PARSE_RULE_OPTION_COMMAND(Backup) // opt/priority PARSE_RULE_OPTION_COMMAND(Check) // opt/priority PARSE_RULE_OPTION_COMMAND(Expire) // opt/priority PARSE_RULE_OPTION_COMMAND(Info) // opt/priority PARSE_RULE_OPTION_COMMAND(Manifest) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/priority PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/priority PARSE_RULE_OPTION_COMMAND(Restore) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/priority PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/priority PARSE_RULE_OPTION_COMMAND(Verify) // opt/priority ), // opt/priority // opt/priority PARSE_RULE_OPTIONAL // opt/priority ( // opt/priority PARSE_RULE_OPTIONAL_GROUP // opt/priority ( // opt/priority PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/priority ( // opt/priority PARSE_RULE_VAL_INT(DS_20), // opt/priority PARSE_RULE_VAL_INT(19), // opt/priority ), // opt/priority ), // opt/priority ), // opt/priority ), // opt/priority // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/process ( // opt/process PARSE_RULE_OPTION_NAME("process"), // opt/process PARSE_RULE_OPTION_TYPE(Integer), // opt/process PARSE_RULE_OPTION_INTERNAL(true) // opt/process PARSE_RULE_OPTION_REQUIRED(false), // opt/process PARSE_RULE_OPTION_SECTION(CommandLine), // opt/process // opt/process PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/process ( // opt/process PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/process PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/process PARSE_RULE_OPTION_COMMAND(Backup) // opt/process PARSE_RULE_OPTION_COMMAND(Restore) // opt/process PARSE_RULE_OPTION_COMMAND(Verify) // opt/process ), // opt/process // opt/process PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/process ( // opt/process PARSE_RULE_OPTION_COMMAND(Annotate) // opt/process PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/process PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/process PARSE_RULE_OPTION_COMMAND(Backup) // opt/process PARSE_RULE_OPTION_COMMAND(Check) // opt/process PARSE_RULE_OPTION_COMMAND(Expire) // opt/process PARSE_RULE_OPTION_COMMAND(Info) // opt/process PARSE_RULE_OPTION_COMMAND(Manifest) // opt/process PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/process PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/process PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/process PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/process PARSE_RULE_OPTION_COMMAND(Restore) // opt/process PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/process PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/process PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/process PARSE_RULE_OPTION_COMMAND(Verify) // opt/process ), // opt/process // opt/process PARSE_RULE_OPTIONAL // opt/process ( // opt/process PARSE_RULE_OPTIONAL_GROUP // opt/process ( // opt/process PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/process ( // opt/process PARSE_RULE_VAL_INT(0), // opt/process PARSE_RULE_VAL_INT(1024), // opt/process ), // opt/process ), // opt/process ), // opt/process ), // opt/process // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/process-max ( // opt/process-max PARSE_RULE_OPTION_NAME("process-max"), // opt/process-max PARSE_RULE_OPTION_TYPE(Integer), // opt/process-max PARSE_RULE_OPTION_RESET(true), // opt/process-max PARSE_RULE_OPTION_REQUIRED(true), // opt/process-max PARSE_RULE_OPTION_SECTION(Global), // opt/process-max // opt/process-max PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/process-max ( // opt/process-max PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/process-max PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/process-max PARSE_RULE_OPTION_COMMAND(Backup) // opt/process-max PARSE_RULE_OPTION_COMMAND(Restore) // opt/process-max PARSE_RULE_OPTION_COMMAND(Verify) // opt/process-max ), // opt/process-max // opt/process-max PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/process-max ( // opt/process-max PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/process-max PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/process-max ), // opt/process-max // opt/process-max PARSE_RULE_OPTIONAL // opt/process-max ( // opt/process-max PARSE_RULE_OPTIONAL_GROUP // opt/process-max ( // opt/process-max PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/process-max ( // opt/process-max PARSE_RULE_VAL_INT(1), // opt/process-max PARSE_RULE_VAL_INT(999), // opt/process-max ), // opt/process-max // opt/process-max PARSE_RULE_OPTIONAL_DEFAULT // opt/process-max ( // opt/process-max PARSE_RULE_VAL_INT(1), // opt/process-max ), // opt/process-max ), // opt/process-max ), // opt/process-max ), // opt/process-max // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_NAME("protocol-timeout"), // opt/protocol-timeout PARSE_RULE_OPTION_TYPE(Time), // opt/protocol-timeout PARSE_RULE_OPTION_RESET(true), // opt/protocol-timeout PARSE_RULE_OPTION_REQUIRED(true), // opt/protocol-timeout PARSE_RULE_OPTION_SECTION(Global), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Annotate) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Expire) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Info) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Manifest) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Server) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Annotate) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Backup) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Check) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Expire) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Info) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Manifest) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Restore) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/protocol-timeout PARSE_RULE_OPTION_COMMAND(Verify) // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTIONAL // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTIONAL_GROUP // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_VAL_TIME(100ms), // opt/protocol-timeout PARSE_RULE_VAL_TIME(7d), // opt/protocol-timeout ), // opt/protocol-timeout // opt/protocol-timeout PARSE_RULE_OPTIONAL_DEFAULT // opt/protocol-timeout ( // opt/protocol-timeout PARSE_RULE_VAL_TIME(31m), // opt/protocol-timeout ), // opt/protocol-timeout ), // opt/protocol-timeout ), // opt/protocol-timeout ), // opt/protocol-timeout // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/raw ( // opt/raw PARSE_RULE_OPTION_NAME("raw"), // opt/raw PARSE_RULE_OPTION_TYPE(Boolean), // opt/raw PARSE_RULE_OPTION_REQUIRED(true), // opt/raw PARSE_RULE_OPTION_SECTION(CommandLine), // opt/raw // opt/raw PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/raw ( // opt/raw PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/raw PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/raw ), // opt/raw // opt/raw PARSE_RULE_OPTIONAL // opt/raw ( // opt/raw PARSE_RULE_OPTIONAL_GROUP // opt/raw ( // opt/raw PARSE_RULE_OPTIONAL_DEFAULT // opt/raw ( // opt/raw PARSE_RULE_VAL_BOOL_FALSE, // opt/raw ), // opt/raw ), // opt/raw ), // opt/raw ), // opt/raw // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/recovery-option ( // opt/recovery-option PARSE_RULE_OPTION_NAME("recovery-option"), // opt/recovery-option PARSE_RULE_OPTION_TYPE(Hash), // opt/recovery-option PARSE_RULE_OPTION_RESET(true), // opt/recovery-option PARSE_RULE_OPTION_REQUIRED(false), // opt/recovery-option PARSE_RULE_OPTION_SECTION(Global), // opt/recovery-option PARSE_RULE_OPTION_MULTI(true), // opt/recovery-option // opt/recovery-option PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/recovery-option ( // opt/recovery-option PARSE_RULE_OPTION_COMMAND(Restore) // opt/recovery-option ), // opt/recovery-option // opt/recovery-option PARSE_RULE_OPTIONAL // opt/recovery-option ( // opt/recovery-option PARSE_RULE_OPTIONAL_GROUP // opt/recovery-option ( // opt/recovery-option PARSE_RULE_OPTIONAL_DEPEND // opt/recovery-option ( // opt/recovery-option PARSE_RULE_VAL_OPT(Type), // opt/recovery-option PARSE_RULE_VAL_STRID(default), // opt/recovery-option PARSE_RULE_VAL_STRID(immediate), // opt/recovery-option PARSE_RULE_VAL_STRID(lsn), // opt/recovery-option PARSE_RULE_VAL_STRID(name), // opt/recovery-option PARSE_RULE_VAL_STRID(time), // opt/recovery-option PARSE_RULE_VAL_STRID(standby), // opt/recovery-option PARSE_RULE_VAL_STRID(xid), // opt/recovery-option ), // opt/recovery-option ), // opt/recovery-option ), // opt/recovery-option ), // opt/recovery-option // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/recurse ( // opt/recurse PARSE_RULE_OPTION_NAME("recurse"), // opt/recurse PARSE_RULE_OPTION_TYPE(Boolean), // opt/recurse PARSE_RULE_OPTION_REQUIRED(true), // opt/recurse PARSE_RULE_OPTION_SECTION(CommandLine), // opt/recurse // opt/recurse PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/recurse ( // opt/recurse PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/recurse PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/recurse ), // opt/recurse // opt/recurse PARSE_RULE_OPTIONAL // opt/recurse ( // opt/recurse PARSE_RULE_OPTIONAL_GROUP // opt/recurse ( // opt/recurse PARSE_RULE_OPTIONAL_DEFAULT // opt/recurse ( // opt/recurse PARSE_RULE_VAL_BOOL_FALSE, // opt/recurse ), // opt/recurse ), // opt/recurse ), // opt/recurse ), // opt/recurse // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/reference ( // opt/reference PARSE_RULE_OPTION_NAME("reference"), // opt/reference PARSE_RULE_OPTION_TYPE(String), // opt/reference PARSE_RULE_OPTION_REQUIRED(false), // opt/reference PARSE_RULE_OPTION_SECTION(CommandLine), // opt/reference // opt/reference PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/reference ( // opt/reference PARSE_RULE_OPTION_COMMAND(Manifest) // opt/reference ), // opt/reference ), // opt/reference // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTION_NAME("remote-type"), // opt/remote-type PARSE_RULE_OPTION_TYPE(StringId), // opt/remote-type PARSE_RULE_OPTION_INTERNAL(true) // opt/remote-type PARSE_RULE_OPTION_REQUIRED(false), // opt/remote-type PARSE_RULE_OPTION_SECTION(CommandLine), // opt/remote-type // opt/remote-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/remote-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/remote-type ), // opt/remote-type // opt/remote-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/remote-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/remote-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Check) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Info) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/remote-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/remote-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/remote-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/remote-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/remote-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/remote-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/remote-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/remote-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/remote-type ), // opt/remote-type // opt/remote-type PARSE_RULE_OPTIONAL // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTIONAL_GROUP // opt/remote-type ( // opt/remote-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/remote-type ( // opt/remote-type PARSE_RULE_VAL_STRID(pg), // opt/remote-type PARSE_RULE_VAL_STRID(repo), // opt/remote-type ), // opt/remote-type ), // opt/remote-type ), // opt/remote-type ), // opt/remote-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo ( // opt/repo PARSE_RULE_OPTION_NAME("repo"), // opt/repo PARSE_RULE_OPTION_TYPE(Integer), // opt/repo PARSE_RULE_OPTION_REQUIRED(false), // opt/repo PARSE_RULE_OPTION_SECTION(CommandLine), // opt/repo // opt/repo PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo ( // opt/repo PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo PARSE_RULE_OPTION_COMMAND(Info) // opt/repo PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo ), // opt/repo // opt/repo PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo ( // opt/repo PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo ), // opt/repo // opt/repo PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo ( // opt/repo PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo ), // opt/repo // opt/repo PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo ( // opt/repo PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo PARSE_RULE_OPTION_COMMAND(Check) // opt/repo PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo PARSE_RULE_OPTION_COMMAND(Info) // opt/repo PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo ), // opt/repo // opt/repo PARSE_RULE_OPTIONAL // opt/repo ( // opt/repo PARSE_RULE_OPTIONAL_GROUP // opt/repo ( // opt/repo PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo ( // opt/repo PARSE_RULE_VAL_INT(1), // opt/repo PARSE_RULE_VAL_INT(256), // opt/repo ), // opt/repo ), // opt/repo ), // opt/repo ), // opt/repo // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_NAME("repo-azure-account"), // opt/repo-azure-account PARSE_RULE_OPTION_TYPE(String), // opt/repo-azure-account PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-account PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-account PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-account PARSE_RULE_OPTION_SECURE(true), // opt/repo-azure-account PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-account ), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-account ), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-account ), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-account PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-account ), // opt/repo-azure-account // opt/repo-azure-account PARSE_RULE_OPTIONAL // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-account ( // opt/repo-azure-account PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-account PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-account ), // opt/repo-azure-account ), // opt/repo-azure-account ), // opt/repo-azure-account ), // opt/repo-azure-account // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_NAME("repo-azure-container"), // opt/repo-azure-container PARSE_RULE_OPTION_TYPE(String), // opt/repo-azure-container PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-container PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-container PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-container PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-container ), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-container ), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-container ), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-container PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-container ), // opt/repo-azure-container // opt/repo-azure-container PARSE_RULE_OPTIONAL // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-container ( // opt/repo-azure-container PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-container PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-container ), // opt/repo-azure-container ), // opt/repo-azure-container ), // opt/repo-azure-container ), // opt/repo-azure-container // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_NAME("repo-azure-endpoint"), // opt/repo-azure-endpoint PARSE_RULE_OPTION_TYPE(String), // opt/repo-azure-endpoint PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-endpoint PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-endpoint PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-endpoint PARSE_RULE_OPTION_SECURE(true), // opt/repo-azure-endpoint PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTIONAL // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-endpoint PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // opt/repo-azure-endpoint PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-azure-endpoint ( // opt/repo-azure-endpoint PARSE_RULE_VAL_STR(QT_blob_DT_core_DT_windows_DT_net_QT), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint ), // opt/repo-azure-endpoint // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_NAME("repo-azure-key"), // opt/repo-azure-key PARSE_RULE_OPTION_TYPE(String), // opt/repo-azure-key PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-key PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-azure-key PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-key PARSE_RULE_OPTION_SECURE(true), // opt/repo-azure-key PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key ), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key ), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key ), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key ), // opt/repo-azure-key // opt/repo-azure-key PARSE_RULE_OPTIONAL // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-key ( // opt/repo-azure-key PARSE_RULE_VAL_OPT(RepoAzureKeyType), // opt/repo-azure-key PARSE_RULE_VAL_STRID(shared), // opt/repo-azure-key PARSE_RULE_VAL_STRID(sas), // opt/repo-azure-key ), // opt/repo-azure-key ), // opt/repo-azure-key ), // opt/repo-azure-key ), // opt/repo-azure-key // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_NAME("repo-azure-key-type"), // opt/repo-azure-key-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-azure-key-type PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-azure-key-type PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-key-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-key-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-key-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTIONAL // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(shared), // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(sas), // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(auto), // opt/repo-azure-key-type ), // opt/repo-azure-key-type // opt/repo-azure-key-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-azure-key-type ( // opt/repo-azure-key-type PARSE_RULE_VAL_STRID(shared), // opt/repo-azure-key-type ), // opt/repo-azure-key-type ), // opt/repo-azure-key-type ), // opt/repo-azure-key-type ), // opt/repo-azure-key-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_NAME("repo-azure-uri-style"), // opt/repo-azure-uri-style PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-azure-uri-style PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-azure-uri-style PARSE_RULE_OPTION_RESET(true), // opt/repo-azure-uri-style PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-azure-uri-style PARSE_RULE_OPTION_SECTION(Global), // opt/repo-azure-uri-style PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-azure-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL_GROUP // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL_DEPEND // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_VAL_OPT(RepoType), // opt/repo-azure-uri-style PARSE_RULE_VAL_STRID(azure), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_VAL_STRID(host), // opt/repo-azure-uri-style PARSE_RULE_VAL_STRID(path), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // opt/repo-azure-uri-style PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-azure-uri-style ( // opt/repo-azure-uri-style PARSE_RULE_VAL_STRID(host), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style ), // opt/repo-azure-uri-style // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTION_NAME("repo-block"), // opt/repo-block PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-block PARSE_RULE_OPTION_NEGATE(true), // opt/repo-block PARSE_RULE_OPTION_RESET(true), // opt/repo-block PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-block PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block // opt/repo-block PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block ), // opt/repo-block // opt/repo-block PARSE_RULE_OPTIONAL // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTIONAL_GROUP // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block ( // opt/repo-block PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/repo-block PARSE_RULE_VAL_OPT(RepoBundle), // opt/repo-block PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block ), // opt/repo-block // opt/repo-block PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-block ( // opt/repo-block PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-block ), // opt/repo-block ), // opt/repo-block ), // opt/repo-block ), // opt/repo-block // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_OPTION_NAME("repo-block-age-map"), // opt/repo-block-age-map PARSE_RULE_OPTION_TYPE(Hash), // opt/repo-block-age-map PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-age-map PARSE_RULE_OPTION_RESET(true), // opt/repo-block-age-map PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-block-age-map PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-age-map PARSE_RULE_OPTION_MULTI(true), // opt/repo-block-age-map PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-age-map // opt/repo-block-age-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-age-map ), // opt/repo-block-age-map // opt/repo-block-age-map PARSE_RULE_OPTIONAL // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-age-map ( // opt/repo-block-age-map PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-age-map PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-age-map ), // opt/repo-block-age-map ), // opt/repo-block-age-map ), // opt/repo-block-age-map ), // opt/repo-block-age-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_NAME("repo-block-checksum-size-map"), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_TYPE(Hash), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_RESET(true), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_MULTI(true), // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-checksum-size-map // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map // opt/repo-block-checksum-size-map PARSE_RULE_OPTIONAL // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-checksum-size-map ( // opt/repo-block-checksum-size-map PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-checksum-size-map PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map ), // opt/repo-block-checksum-size-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_OPTION_NAME("repo-block-size-map"), // opt/repo-block-size-map PARSE_RULE_OPTION_TYPE(Hash), // opt/repo-block-size-map PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-size-map PARSE_RULE_OPTION_RESET(true), // opt/repo-block-size-map PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-block-size-map PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-size-map PARSE_RULE_OPTION_MULTI(true), // opt/repo-block-size-map PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-size-map // opt/repo-block-size-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-size-map ), // opt/repo-block-size-map // opt/repo-block-size-map PARSE_RULE_OPTIONAL // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-size-map ( // opt/repo-block-size-map PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-size-map PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-size-map ), // opt/repo-block-size-map ), // opt/repo-block-size-map ), // opt/repo-block-size-map ), // opt/repo-block-size-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_OPTION_NAME("repo-block-size-super"), // opt/repo-block-size-super PARSE_RULE_OPTION_TYPE(Size), // opt/repo-block-size-super PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-size-super PARSE_RULE_OPTION_RESET(true), // opt/repo-block-size-super PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-block-size-super PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-size-super PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-size-super // opt/repo-block-size-super PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-size-super ), // opt/repo-block-size-super // opt/repo-block-size-super PARSE_RULE_OPTIONAL // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-size-super PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-size-super ), // opt/repo-block-size-super // opt/repo-block-size-super PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_VAL_SIZE(32KiB), // opt/repo-block-size-super PARSE_RULE_VAL_SIZE(16MiB), // opt/repo-block-size-super ), // opt/repo-block-size-super // opt/repo-block-size-super PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-block-size-super ( // opt/repo-block-size-super PARSE_RULE_VAL_SIZE(256KiB), // opt/repo-block-size-super ), // opt/repo-block-size-super ), // opt/repo-block-size-super ), // opt/repo-block-size-super ), // opt/repo-block-size-super // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_OPTION_NAME("repo-block-size-super-full"), // opt/repo-block-size-super-full PARSE_RULE_OPTION_TYPE(Size), // opt/repo-block-size-super-full PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-block-size-super-full PARSE_RULE_OPTION_RESET(true), // opt/repo-block-size-super-full PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-block-size-super-full PARSE_RULE_OPTION_SECTION(Global), // opt/repo-block-size-super-full PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-block-size-super-full // opt/repo-block-size-super-full PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL_GROUP // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL_DEPEND // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_VAL_OPT(RepoBlock), // opt/repo-block-size-super-full PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_VAL_SIZE(32KiB), // opt/repo-block-size-super-full PARSE_RULE_VAL_SIZE(16MiB), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full // opt/repo-block-size-super-full PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-block-size-super-full ( // opt/repo-block-size-super-full PARSE_RULE_VAL_SIZE(1MiB), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full ), // opt/repo-block-size-super-full // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_OPTION_NAME("repo-bundle"), // opt/repo-bundle PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-bundle PARSE_RULE_OPTION_NEGATE(true), // opt/repo-bundle PARSE_RULE_OPTION_RESET(true), // opt/repo-bundle PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-bundle PARSE_RULE_OPTION_SECTION(Global), // opt/repo-bundle PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-bundle // opt/repo-bundle PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-bundle ), // opt/repo-bundle // opt/repo-bundle PARSE_RULE_OPTIONAL // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_OPTIONAL_GROUP // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-bundle ( // opt/repo-bundle PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-bundle ), // opt/repo-bundle ), // opt/repo-bundle ), // opt/repo-bundle ), // opt/repo-bundle // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_OPTION_NAME("repo-bundle-limit"), // opt/repo-bundle-limit PARSE_RULE_OPTION_TYPE(Size), // opt/repo-bundle-limit PARSE_RULE_OPTION_RESET(true), // opt/repo-bundle-limit PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-bundle-limit PARSE_RULE_OPTION_SECTION(Global), // opt/repo-bundle-limit PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-bundle-limit // opt/repo-bundle-limit PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-bundle-limit ), // opt/repo-bundle-limit // opt/repo-bundle-limit PARSE_RULE_OPTIONAL // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_OPTIONAL_GROUP // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_OPTIONAL_DEPEND // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_VAL_OPT(RepoBundle), // opt/repo-bundle-limit PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-bundle-limit ), // opt/repo-bundle-limit // opt/repo-bundle-limit PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_VAL_SIZE(8KiB), // opt/repo-bundle-limit PARSE_RULE_VAL_SIZE(1PiB), // opt/repo-bundle-limit ), // opt/repo-bundle-limit // opt/repo-bundle-limit PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-bundle-limit ( // opt/repo-bundle-limit PARSE_RULE_VAL_SIZE(2MiB), // opt/repo-bundle-limit ), // opt/repo-bundle-limit ), // opt/repo-bundle-limit ), // opt/repo-bundle-limit ), // opt/repo-bundle-limit // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_OPTION_NAME("repo-bundle-size"), // opt/repo-bundle-size PARSE_RULE_OPTION_TYPE(Size), // opt/repo-bundle-size PARSE_RULE_OPTION_RESET(true), // opt/repo-bundle-size PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-bundle-size PARSE_RULE_OPTION_SECTION(Global), // opt/repo-bundle-size PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-bundle-size // opt/repo-bundle-size PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-bundle-size ), // opt/repo-bundle-size // opt/repo-bundle-size PARSE_RULE_OPTIONAL // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_OPTIONAL_GROUP // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_OPTIONAL_DEPEND // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_VAL_OPT(RepoBundle), // opt/repo-bundle-size PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-bundle-size ), // opt/repo-bundle-size // opt/repo-bundle-size PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_VAL_SIZE(1MiB), // opt/repo-bundle-size PARSE_RULE_VAL_SIZE(1PiB), // opt/repo-bundle-size ), // opt/repo-bundle-size // opt/repo-bundle-size PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-bundle-size ( // opt/repo-bundle-size PARSE_RULE_VAL_SIZE(20MiB), // opt/repo-bundle-size ), // opt/repo-bundle-size ), // opt/repo-bundle-size ), // opt/repo-bundle-size ), // opt/repo-bundle-size // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_NAME("repo-cipher-pass"), // opt/repo-cipher-pass PARSE_RULE_OPTION_TYPE(String), // opt/repo-cipher-pass PARSE_RULE_OPTION_RESET(true), // opt/repo-cipher-pass PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-cipher-pass PARSE_RULE_OPTION_SECTION(Global), // opt/repo-cipher-pass PARSE_RULE_OPTION_SECURE(true), // opt/repo-cipher-pass PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-cipher-pass PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-pass ), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-pass ), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-pass ), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-cipher-pass PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-pass ), // opt/repo-cipher-pass // opt/repo-cipher-pass PARSE_RULE_OPTIONAL // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTIONAL_GROUP // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_OPTIONAL_DEPEND // opt/repo-cipher-pass ( // opt/repo-cipher-pass PARSE_RULE_VAL_OPT(RepoCipherType), // opt/repo-cipher-pass PARSE_RULE_VAL_STRID(aes_DS_256_DS_cbc), // opt/repo-cipher-pass ), // opt/repo-cipher-pass ), // opt/repo-cipher-pass ), // opt/repo-cipher-pass ), // opt/repo-cipher-pass // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_NAME("repo-cipher-type"), // opt/repo-cipher-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-cipher-type PARSE_RULE_OPTION_RESET(true), // opt/repo-cipher-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-cipher-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-cipher-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-cipher-type PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-cipher-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTIONAL // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_VAL_STRID(none), // opt/repo-cipher-type PARSE_RULE_VAL_STRID(aes_DS_256_DS_cbc), // opt/repo-cipher-type ), // opt/repo-cipher-type // opt/repo-cipher-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-cipher-type ( // opt/repo-cipher-type PARSE_RULE_VAL_STRID(none), // opt/repo-cipher-type ), // opt/repo-cipher-type ), // opt/repo-cipher-type ), // opt/repo-cipher-type ), // opt/repo-cipher-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_NAME("repo-gcs-bucket"), // opt/repo-gcs-bucket PARSE_RULE_OPTION_TYPE(String), // opt/repo-gcs-bucket PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-bucket PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-gcs-bucket PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-bucket PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // opt/repo-gcs-bucket PARSE_RULE_OPTIONAL // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTIONAL_GROUP // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_OPTIONAL_DEPEND // opt/repo-gcs-bucket ( // opt/repo-gcs-bucket PARSE_RULE_VAL_OPT(RepoGcsKeyType), // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket ), // opt/repo-gcs-bucket // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_NAME("repo-gcs-endpoint"), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_TYPE(String), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-endpoint PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTIONAL // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTIONAL_GROUP // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_OPTIONAL_DEPEND // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_VAL_OPT(RepoGcsKeyType), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // opt/repo-gcs-endpoint PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-gcs-endpoint ( // opt/repo-gcs-endpoint PARSE_RULE_VAL_STR(QT_storage_DT_googleapis_DT_com_QT), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint ), // opt/repo-gcs-endpoint // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_NAME("repo-gcs-key"), // opt/repo-gcs-key PARSE_RULE_OPTION_TYPE(String), // opt/repo-gcs-key PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-key PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-gcs-key PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-key PARSE_RULE_OPTION_SECURE(true), // opt/repo-gcs-key PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key ), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key ), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key ), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key ), // opt/repo-gcs-key // opt/repo-gcs-key PARSE_RULE_OPTIONAL // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTIONAL_GROUP // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_OPTIONAL_DEPEND // opt/repo-gcs-key ( // opt/repo-gcs-key PARSE_RULE_VAL_OPT(RepoGcsKeyType), // opt/repo-gcs-key PARSE_RULE_VAL_STRID(service), // opt/repo-gcs-key PARSE_RULE_VAL_STRID(token), // opt/repo-gcs-key ), // opt/repo-gcs-key ), // opt/repo-gcs-key ), // opt/repo-gcs-key ), // opt/repo-gcs-key // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_NAME("repo-gcs-key-type"), // opt/repo-gcs-key-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-gcs-key-type PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-gcs-key-type PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-key-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-gcs-key-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-key-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(gcs), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(service), // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(token), // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(auto), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // opt/repo-gcs-key-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-gcs-key-type ( // opt/repo-gcs-key-type PARSE_RULE_VAL_STRID(service), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type ), // opt/repo-gcs-key-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_NAME("repo-gcs-user-project"), // opt/repo-gcs-user-project PARSE_RULE_OPTION_TYPE(String), // opt/repo-gcs-user-project PARSE_RULE_OPTION_RESET(true), // opt/repo-gcs-user-project PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-gcs-user-project PARSE_RULE_OPTION_SECTION(Global), // opt/repo-gcs-user-project PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-gcs-user-project // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-gcs-user-project ( // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-gcs-user-project PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project ), // opt/repo-gcs-user-project // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTION_NAME("repo-hardlink"), // opt/repo-hardlink PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-hardlink PARSE_RULE_OPTION_NEGATE(true), // opt/repo-hardlink PARSE_RULE_OPTION_RESET(true), // opt/repo-hardlink PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-hardlink PARSE_RULE_OPTION_SECTION(Global), // opt/repo-hardlink PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-hardlink // opt/repo-hardlink PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-hardlink ), // opt/repo-hardlink // opt/repo-hardlink PARSE_RULE_OPTIONAL // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTIONAL_GROUP // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTIONAL_DEPEND // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/repo-hardlink PARSE_RULE_VAL_OPT(RepoBundle), // opt/repo-hardlink PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-hardlink ), // opt/repo-hardlink // opt/repo-hardlink PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-hardlink ( // opt/repo-hardlink PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-hardlink ), // opt/repo-hardlink ), // opt/repo-hardlink ), // opt/repo-hardlink ), // opt/repo-hardlink // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTION_NAME("repo-host"), // opt/repo-host PARSE_RULE_OPTION_TYPE(String), // opt/repo-host PARSE_RULE_OPTION_RESET(true), // opt/repo-host PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host // opt/repo-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host ), // opt/repo-host // opt/repo-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host ), // opt/repo-host // opt/repo-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host ), // opt/repo-host // opt/repo-host PARSE_RULE_OPTIONAL // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-host ( // opt/repo-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host ( // opt/repo-host PARSE_RULE_VAL_OPT(RepoLocal), // opt/repo-host PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-host ), // opt/repo-host ), // opt/repo-host ), // opt/repo-host ), // opt/repo-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTION_NAME("repo-host-ca-file"), // opt/repo-host-ca-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-ca-file PARSE_RULE_OPTION_RESET(true), // opt/repo-host-ca-file PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host-ca-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-ca-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-ca-file // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-ca-file ), // opt/repo-host-ca-file // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-file ), // opt/repo-host-ca-file // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-ca-file ), // opt/repo-host-ca-file // opt/repo-host-ca-file PARSE_RULE_OPTIONAL // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-ca-file ( // opt/repo-host-ca-file PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-ca-file PARSE_RULE_VAL_STRID(tls), // opt/repo-host-ca-file ), // opt/repo-host-ca-file ), // opt/repo-host-ca-file ), // opt/repo-host-ca-file ), // opt/repo-host-ca-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTION_NAME("repo-host-ca-path"), // opt/repo-host-ca-path PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-ca-path PARSE_RULE_OPTION_RESET(true), // opt/repo-host-ca-path PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host-ca-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-ca-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-ca-path // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-ca-path ), // opt/repo-host-ca-path // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-path ), // opt/repo-host-ca-path // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-ca-path ), // opt/repo-host-ca-path // opt/repo-host-ca-path PARSE_RULE_OPTIONAL // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-ca-path ( // opt/repo-host-ca-path PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-ca-path PARSE_RULE_VAL_STRID(tls), // opt/repo-host-ca-path ), // opt/repo-host-ca-path ), // opt/repo-host-ca-path ), // opt/repo-host-ca-path ), // opt/repo-host-ca-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTION_NAME("repo-host-cert-file"), // opt/repo-host-cert-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-cert-file PARSE_RULE_OPTION_RESET(true), // opt/repo-host-cert-file PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-cert-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-cert-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-cert-file // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-cert-file ), // opt/repo-host-cert-file // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cert-file ), // opt/repo-host-cert-file // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-cert-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-cert-file ), // opt/repo-host-cert-file // opt/repo-host-cert-file PARSE_RULE_OPTIONAL // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-cert-file ( // opt/repo-host-cert-file PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-cert-file PARSE_RULE_VAL_STRID(tls), // opt/repo-host-cert-file ), // opt/repo-host-cert-file ), // opt/repo-host-cert-file ), // opt/repo-host-cert-file ), // opt/repo-host-cert-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTION_NAME("repo-host-cmd"), // opt/repo-host-cmd PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-cmd PARSE_RULE_OPTION_DEFAULT_TYPE(Dynamic), // opt/repo-host-cmd PARSE_RULE_OPTION_RESET(true), // opt/repo-host-cmd PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-cmd PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-cmd PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-cmd ), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cmd ), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-cmd ), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTIONAL // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-cmd PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-cmd ), // opt/repo-host-cmd // opt/repo-host-cmd PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-cmd ( // opt/repo-host-cmd PARSE_RULE_DEFAULT_DYNAMIC(Bin), // opt/repo-host-cmd ), // opt/repo-host-cmd ), // opt/repo-host-cmd ), // opt/repo-host-cmd ), // opt/repo-host-cmd // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTION_NAME("repo-host-config"), // opt/repo-host-config PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-config PARSE_RULE_OPTION_RESET(true), // opt/repo-host-config PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-config PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-config PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config ), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config ), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config ), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTIONAL // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_VAL_OPT(RepoHost), // opt/repo-host-config ), // opt/repo-host-config // opt/repo-host-config PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-config ( // opt/repo-host-config PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE), // opt/repo-host-config ), // opt/repo-host-config ), // opt/repo-host-config ), // opt/repo-host-config ), // opt/repo-host-config // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTION_NAME("repo-host-config-include-path"), // opt/repo-host-config-include-path PARSE_RULE_OPTION_TYPE(Path), // opt/repo-host-config-include-path PARSE_RULE_OPTION_RESET(true), // opt/repo-host-config-include-path PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-config-include-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-config-include-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config-include-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTIONAL // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_VAL_OPT(RepoHost), // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // opt/repo-host-config-include-path PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-config-include-path ( // opt/repo-host-config-include-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH), ), // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path ), // opt/repo-host-config-include-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTION_NAME("repo-host-config-path"), // opt/repo-host-config-path PARSE_RULE_OPTION_TYPE(Path), // opt/repo-host-config-path PARSE_RULE_OPTION_RESET(true), // opt/repo-host-config-path PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-config-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-config-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config-path ), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-path ), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-config-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-config-path ), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTIONAL // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_VAL_OPT(RepoHost), // opt/repo-host-config-path ), // opt/repo-host-config-path // opt/repo-host-config-path PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-config-path ( // opt/repo-host-config-path PARSE_RULE_VAL_STR(CFGOPTDEF_CONFIG_PATH), // opt/repo-host-config-path ), // opt/repo-host-config-path ), // opt/repo-host-config-path ), // opt/repo-host-config-path ), // opt/repo-host-config-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTION_NAME("repo-host-key-file"), // opt/repo-host-key-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-key-file PARSE_RULE_OPTION_RESET(true), // opt/repo-host-key-file PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-key-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-key-file // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-key-file ), // opt/repo-host-key-file // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-key-file ), // opt/repo-host-key-file // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-key-file ), // opt/repo-host-key-file // opt/repo-host-key-file PARSE_RULE_OPTIONAL // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-key-file ( // opt/repo-host-key-file PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-key-file PARSE_RULE_VAL_STRID(tls), // opt/repo-host-key-file ), // opt/repo-host-key-file ), // opt/repo-host-key-file ), // opt/repo-host-key-file ), // opt/repo-host-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTION_NAME("repo-host-port"), // opt/repo-host-port PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-host-port PARSE_RULE_OPTION_RESET(true), // opt/repo-host-port PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host-port PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-port PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTIONAL // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-port PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-port PARSE_RULE_VAL_STRID(tls), // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_VAL_INT(0), // opt/repo-host-port PARSE_RULE_VAL_INT(65535), // opt/repo-host-port ), // opt/repo-host-port // opt/repo-host-port PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_OPTIONAL_DEFAULT_MAP // opt/repo-host-port ( // opt/repo-host-port PARSE_RULE_VAL_STRID(tls), // opt/repo-host-port PARSE_RULE_VAL_INT(8432), // opt/repo-host-port ), // opt/repo-host-port ), // opt/repo-host-port ), // opt/repo-host-port ), // opt/repo-host-port ), // opt/repo-host-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTION_NAME("repo-host-type"), // opt/repo-host-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-host-type PARSE_RULE_OPTION_RESET(true), // opt/repo-host-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-host-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTIONAL // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_VAL_OPT(RepoHost), // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-type PARSE_RULE_VAL_STRID(tls), // opt/repo-host-type ), // opt/repo-host-type // opt/repo-host-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-type ( // opt/repo-host-type PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-type ), // opt/repo-host-type ), // opt/repo-host-type ), // opt/repo-host-type ), // opt/repo-host-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTION_NAME("repo-host-user"), // opt/repo-host-user PARSE_RULE_OPTION_TYPE(String), // opt/repo-host-user PARSE_RULE_OPTION_RESET(true), // opt/repo-host-user PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-host-user PARSE_RULE_OPTION_SECTION(Global), // opt/repo-host-user PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-user ), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-user ), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-host-user ), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTIONAL // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTIONAL_GROUP // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_OPTIONAL_DEPEND // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_VAL_OPT(RepoHostType), // opt/repo-host-user PARSE_RULE_VAL_STRID(ssh), // opt/repo-host-user ), // opt/repo-host-user // opt/repo-host-user PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-host-user ( // opt/repo-host-user PARSE_RULE_VAL_STR(QT_pgbackrest_QT), // opt/repo-host-user ), // opt/repo-host-user ), // opt/repo-host-user ), // opt/repo-host-user ), // opt/repo-host-user // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTION_NAME("repo-local"), // opt/repo-local PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-local PARSE_RULE_OPTION_INTERNAL(true) // opt/repo-local PARSE_RULE_OPTION_NEGATE(true), // opt/repo-local PARSE_RULE_OPTION_RESET(true), // opt/repo-local PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-local PARSE_RULE_OPTION_SECTION(Global), // opt/repo-local PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-local // opt/repo-local PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-local PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-local PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-local PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-local PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-local PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-local PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-local PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-local ), // opt/repo-local // opt/repo-local PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-local ), // opt/repo-local // opt/repo-local PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-local PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-local PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-local ), // opt/repo-local // opt/repo-local PARSE_RULE_OPTIONAL // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTIONAL_GROUP // opt/repo-local ( // opt/repo-local PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-local ( // opt/repo-local PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-local ), // opt/repo-local ), // opt/repo-local ), // opt/repo-local ), // opt/repo-local // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_NAME("repo-path"), // opt/repo-path PARSE_RULE_OPTION_TYPE(Path), // opt/repo-path PARSE_RULE_OPTION_RESET(true), // opt/repo-path PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-path PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-path // opt/repo-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-path ), // opt/repo-path // opt/repo-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-path ), // opt/repo-path // opt/repo-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-path ), // opt/repo-path // opt/repo-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-path ), // opt/repo-path // opt/repo-path PARSE_RULE_OPTIONAL // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-path ( // opt/repo-path PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-path ( // opt/repo-path PARSE_RULE_VAL_STR(QT_FS_var_FS_lib_FS_pgbackrest_QT), // opt/repo-path ), // opt/repo-path ), // opt/repo-path ), // opt/repo-path ), // opt/repo-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_OPTION_NAME("repo-retention-archive"), // opt/repo-retention-archive PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-retention-archive PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-archive PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-retention-archive PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-archive PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-archive // opt/repo-retention-archive PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-archive PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-archive ), // opt/repo-retention-archive // opt/repo-retention-archive PARSE_RULE_OPTIONAL // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-retention-archive ( // opt/repo-retention-archive PARSE_RULE_VAL_INT(1), // opt/repo-retention-archive PARSE_RULE_VAL_INT(9999999), // opt/repo-retention-archive ), // opt/repo-retention-archive ), // opt/repo-retention-archive ), // opt/repo-retention-archive ), // opt/repo-retention-archive // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_OPTION_NAME("repo-retention-archive-type"), // opt/repo-retention-archive-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-retention-archive-type PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-archive-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-retention-archive-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-archive-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-archive-type // opt/repo-retention-archive-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-archive-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type // opt/repo-retention-archive-type PARSE_RULE_OPTIONAL // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_VAL_STRID(full), // opt/repo-retention-archive-type PARSE_RULE_VAL_STRID(diff), // opt/repo-retention-archive-type PARSE_RULE_VAL_STRID(incr), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type // opt/repo-retention-archive-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-retention-archive-type ( // opt/repo-retention-archive-type PARSE_RULE_VAL_STRID(full), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type ), // opt/repo-retention-archive-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_OPTION_NAME("repo-retention-diff"), // opt/repo-retention-diff PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-retention-diff PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-diff PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-retention-diff PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-diff PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-diff // opt/repo-retention-diff PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-diff PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-diff ), // opt/repo-retention-diff // opt/repo-retention-diff PARSE_RULE_OPTIONAL // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-retention-diff ( // opt/repo-retention-diff PARSE_RULE_VAL_INT(1), // opt/repo-retention-diff PARSE_RULE_VAL_INT(9999999), // opt/repo-retention-diff ), // opt/repo-retention-diff ), // opt/repo-retention-diff ), // opt/repo-retention-diff ), // opt/repo-retention-diff // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_OPTION_NAME("repo-retention-full"), // opt/repo-retention-full PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-retention-full PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-full PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-retention-full PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-full PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-full // opt/repo-retention-full PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-full PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-full ), // opt/repo-retention-full // opt/repo-retention-full PARSE_RULE_OPTIONAL // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-retention-full ( // opt/repo-retention-full PARSE_RULE_VAL_INT(1), // opt/repo-retention-full PARSE_RULE_VAL_INT(9999999), // opt/repo-retention-full ), // opt/repo-retention-full ), // opt/repo-retention-full ), // opt/repo-retention-full ), // opt/repo-retention-full // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_OPTION_NAME("repo-retention-full-type"), // opt/repo-retention-full-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-retention-full-type PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-retention-full-type PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-full-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-retention-full-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-full-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-full-type // opt/repo-retention-full-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-full-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-full-type ), // opt/repo-retention-full-type // opt/repo-retention-full-type PARSE_RULE_OPTIONAL // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_VAL_STRID(count), // opt/repo-retention-full-type PARSE_RULE_VAL_STRID(time), // opt/repo-retention-full-type ), // opt/repo-retention-full-type // opt/repo-retention-full-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-retention-full-type ( // opt/repo-retention-full-type PARSE_RULE_VAL_STRID(count), // opt/repo-retention-full-type ), // opt/repo-retention-full-type ), // opt/repo-retention-full-type ), // opt/repo-retention-full-type ), // opt/repo-retention-full-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_OPTION_NAME("repo-retention-history"), // opt/repo-retention-history PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-retention-history PARSE_RULE_OPTION_RESET(true), // opt/repo-retention-history PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-retention-history PARSE_RULE_OPTION_SECTION(Global), // opt/repo-retention-history PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-retention-history // opt/repo-retention-history PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-retention-history PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-retention-history ), // opt/repo-retention-history // opt/repo-retention-history PARSE_RULE_OPTIONAL // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_OPTIONAL_GROUP // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-retention-history ( // opt/repo-retention-history PARSE_RULE_VAL_INT(0), // opt/repo-retention-history PARSE_RULE_VAL_INT(9999999), // opt/repo-retention-history ), // opt/repo-retention-history ), // opt/repo-retention-history ), // opt/repo-retention-history ), // opt/repo-retention-history // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_NAME("repo-s3-bucket"), // opt/repo-s3-bucket PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-bucket PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-bucket PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-bucket PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-bucket PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-bucket PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-bucket ), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-bucket ), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-bucket ), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-bucket PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-bucket ), // opt/repo-s3-bucket // opt/repo-s3-bucket PARSE_RULE_OPTIONAL // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-bucket ( // opt/repo-s3-bucket PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-bucket PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-bucket ), // opt/repo-s3-bucket ), // opt/repo-s3-bucket ), // opt/repo-s3-bucket ), // opt/repo-s3-bucket // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_NAME("repo-s3-endpoint"), // opt/repo-s3-endpoint PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-endpoint PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-endpoint PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-endpoint PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-endpoint PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-endpoint PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-endpoint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // opt/repo-s3-endpoint PARSE_RULE_OPTIONAL // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-endpoint ( // opt/repo-s3-endpoint PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-endpoint PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint ), // opt/repo-s3-endpoint // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_NAME("repo-s3-key"), // opt/repo-s3-key PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-key PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-key PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-key PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-key PARSE_RULE_OPTION_SECURE(true), // opt/repo-s3-key PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-key PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key ), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key ), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key ), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key ), // opt/repo-s3-key // opt/repo-s3-key PARSE_RULE_OPTIONAL // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-key ( // opt/repo-s3-key PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-key PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-key ), // opt/repo-s3-key ), // opt/repo-s3-key ), // opt/repo-s3-key ), // opt/repo-s3-key // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_NAME("repo-s3-key-secret"), // opt/repo-s3-key-secret PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-key-secret PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-key-secret PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-key-secret PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-key-secret PARSE_RULE_OPTION_SECURE(true), // opt/repo-s3-key-secret PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-key-secret PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key-secret PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // opt/repo-s3-key-secret PARSE_RULE_OPTIONAL // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-key-secret ( // opt/repo-s3-key-secret PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-key-secret PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret ), // opt/repo-s3-key-secret // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_NAME("repo-s3-key-type"), // opt/repo-s3-key-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-s3-key-type PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-s3-key-type PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-key-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-key-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-key-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-key-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTIONAL // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(auto), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(web_DS_id), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(pod_DS_id), // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(process), // opt/repo-s3-key-type ), // opt/repo-s3-key-type // opt/repo-s3-key-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-key-type ( // opt/repo-s3-key-type PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-key-type ), // opt/repo-s3-key-type ), // opt/repo-s3-key-type ), // opt/repo-s3-key-type ), // opt/repo-s3-key-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_NAME("repo-s3-kms-key-id"), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-kms-key-id PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // opt/repo-s3-kms-key-id PARSE_RULE_OPTIONAL // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-kms-key-id ( // opt/repo-s3-kms-key-id PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-kms-key-id PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id ), // opt/repo-s3-kms-key-id // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_NAME("repo-s3-process-cmd"), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_TYPE(List), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_MULTI(true), // opt/repo-s3-process-cmd PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-process-cmd PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // opt/repo-s3-process-cmd PARSE_RULE_OPTIONAL // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-process-cmd ( // opt/repo-s3-process-cmd PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-process-cmd PARSE_RULE_VAL_STRID(process), // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd ), // opt/repo-s3-process-cmd // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_NAME("repo-s3-region"), // opt/repo-s3-region PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-region PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-region PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-region PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-region PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-region PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-region ), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-region ), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-region ), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-region PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-region ), // opt/repo-s3-region // opt/repo-s3-region PARSE_RULE_OPTIONAL // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-region ( // opt/repo-s3-region PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-region PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-region ), // opt/repo-s3-region ), // opt/repo-s3-region ), // opt/repo-s3-region ), // opt/repo-s3-region // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_NAME("repo-s3-requester-pays"), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_NEGATE(true), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-requester-pays PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-requester-pays PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTIONAL // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-requester-pays PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // opt/repo-s3-requester-pays PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-requester-pays ( // opt/repo-s3-requester-pays PARSE_RULE_VAL_BOOL_FALSE, // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays ), // opt/repo-s3-requester-pays // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_NAME("repo-s3-role"), // opt/repo-s3-role PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-role PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-role PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-role PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-role PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-role ), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-role ), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-role ), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-role PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-role ), // opt/repo-s3-role // opt/repo-s3-role PARSE_RULE_OPTIONAL // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-role ( // opt/repo-s3-role PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-role PARSE_RULE_VAL_STRID(auto), // opt/repo-s3-role ), // opt/repo-s3-role ), // opt/repo-s3-role ), // opt/repo-s3-role ), // opt/repo-s3-role // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_NAME("repo-s3-service"), // opt/repo-s3-service PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-service PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-service PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-service PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-service PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-service PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTIONAL // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-service PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-service ), // opt/repo-s3-service // opt/repo-s3-service PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-service ( // opt/repo-s3-service PARSE_RULE_VAL_STR(QT_s3_QT), // opt/repo-s3-service ), // opt/repo-s3-service ), // opt/repo-s3-service ), // opt/repo-s3-service ), // opt/repo-s3-service // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_NAME("repo-s3-sse-customer-key"), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_SECURE(true), // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-sse-customer-key PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // opt/repo-s3-sse-customer-key PARSE_RULE_OPTIONAL // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-sse-customer-key ( // opt/repo-s3-sse-customer-key PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-sse-customer-key PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key ), // opt/repo-s3-sse-customer-key // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_NAME("repo-s3-sts-host"), // opt/repo-s3-sts-host PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-sts-host PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-sts-host PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-sts-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-sts-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-sts-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTIONAL // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-sts-host PARSE_RULE_VAL_STRID(web_DS_id), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // opt/repo-s3-sts-host PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-sts-host ( // opt/repo-s3-sts-host PARSE_RULE_VAL_STR(QT_sts_DT_amazonaws_DT_com_QT), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host ), // opt/repo-s3-sts-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_NAME("repo-s3-token"), // opt/repo-s3-token PARSE_RULE_OPTION_TYPE(String), // opt/repo-s3-token PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-token PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-s3-token PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-token PARSE_RULE_OPTION_SECURE(true), // opt/repo-s3-token PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-token ), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-token ), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-token ), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-token PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-token ), // opt/repo-s3-token // opt/repo-s3-token PARSE_RULE_OPTIONAL // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-token ( // opt/repo-s3-token PARSE_RULE_VAL_OPT(RepoS3KeyType), // opt/repo-s3-token PARSE_RULE_VAL_STRID(shared), // opt/repo-s3-token ), // opt/repo-s3-token ), // opt/repo-s3-token ), // opt/repo-s3-token ), // opt/repo-s3-token // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_NAME("repo-s3-uri-style"), // opt/repo-s3-uri-style PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-s3-uri-style PARSE_RULE_OPTION_SEQUENCE(true), // opt/repo-s3-uri-style PARSE_RULE_OPTION_RESET(true), // opt/repo-s3-uri-style PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-s3-uri-style PARSE_RULE_OPTION_SECTION(Global), // opt/repo-s3-uri-style PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-s3-uri-style PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL_GROUP // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL_DEPEND // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_VAL_OPT(RepoType), // opt/repo-s3-uri-style PARSE_RULE_VAL_STRID(s3), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_VAL_STRID(host), // opt/repo-s3-uri-style PARSE_RULE_VAL_STRID(path), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // opt/repo-s3-uri-style PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-s3-uri-style ( // opt/repo-s3-uri-style PARSE_RULE_VAL_STRID(host), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style ), // opt/repo-s3-uri-style // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_NAME("repo-sftp-host"), // opt/repo-sftp-host PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-host PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host ), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host ), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host ), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host ), // opt/repo-sftp-host // opt/repo-sftp-host PARSE_RULE_OPTIONAL // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host ( // opt/repo-sftp-host PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host ), // opt/repo-sftp-host ), // opt/repo-sftp-host ), // opt/repo-sftp-host ), // opt/repo-sftp-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_NAME("repo-sftp-host-fingerprint"), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTIONAL // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-fingerprint ( // opt/repo-sftp-host-fingerprint PARSE_RULE_VAL_OPT(RepoSftpHostKeyCheckType), // opt/repo-sftp-host-fingerprint PARSE_RULE_VAL_STRID(fingerprint), // opt/repo-sftp-host-fingerprint PARSE_RULE_VAL_STRID(strict), // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint ), // opt/repo-sftp-host-fingerprint // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_NAME("repo-sftp-host-key-check-type"), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(accept_DS_new), // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(fingerprint), // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(none), // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(strict), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // opt/repo-sftp-host-key-check-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-sftp-host-key-check-type ( // opt/repo-sftp-host-key-check-type PARSE_RULE_VAL_STRID(strict), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type ), // opt/repo-sftp-host-key-check-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_NAME("repo-sftp-host-key-hash-type"), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTIONAL // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // opt/repo-sftp-host-key-hash-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-sftp-host-key-hash-type ( // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_STRID(md5), // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_STRID(sha1), // opt/repo-sftp-host-key-hash-type PARSE_RULE_VAL_STRID(sha256), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type ), // opt/repo-sftp-host-key-hash-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_NAME("repo-sftp-host-port"), // opt/repo-sftp-host-port PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-sftp-host-port PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-port PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host-port PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-port PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host-port PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_VAL_INT(1), // opt/repo-sftp-host-port PARSE_RULE_VAL_INT(65535), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // opt/repo-sftp-host-port PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-sftp-host-port ( // opt/repo-sftp-host-port PARSE_RULE_VAL_INT(22), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port ), // opt/repo-sftp-host-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_NAME("repo-sftp-host-user"), // opt/repo-sftp-host-user PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-host-user PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-host-user PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-host-user PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-host-user PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-host-user PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // opt/repo-sftp-host-user PARSE_RULE_OPTIONAL // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-host-user ( // opt/repo-sftp-host-user PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-host-user PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user ), // opt/repo-sftp-host-user // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_NAME("repo-sftp-known-host"), // opt/repo-sftp-known-host PARSE_RULE_OPTION_TYPE(List), // opt/repo-sftp-known-host PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-known-host PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-sftp-known-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-known-host PARSE_RULE_OPTION_MULTI(true), // opt/repo-sftp-known-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-known-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // opt/repo-sftp-known-host PARSE_RULE_OPTIONAL // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-known-host ( // opt/repo-sftp-known-host PARSE_RULE_VAL_OPT(RepoSftpHostKeyCheckType), // opt/repo-sftp-known-host PARSE_RULE_VAL_STRID(accept_DS_new), // opt/repo-sftp-known-host PARSE_RULE_VAL_STRID(strict), // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host ), // opt/repo-sftp-known-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_NAME("repo-sftp-private-key-file"), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-private-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // opt/repo-sftp-private-key-file PARSE_RULE_OPTIONAL // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-private-key-file ( // opt/repo-sftp-private-key-file PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-private-key-file PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file ), // opt/repo-sftp-private-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_NAME("repo-sftp-private-key-passphrase"), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_SECURE(true), // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTIONAL // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-private-key-passphrase ( // opt/repo-sftp-private-key-passphrase PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-private-key-passphrase PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase ), // opt/repo-sftp-private-key-passphrase // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_NAME("repo-sftp-public-key-file"), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_RESET(true), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-sftp-public-key-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // opt/repo-sftp-public-key-file PARSE_RULE_OPTIONAL // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-sftp-public-key-file ( // opt/repo-sftp-public-key-file PARSE_RULE_VAL_OPT(RepoType), // opt/repo-sftp-public-key-file PARSE_RULE_VAL_STRID(sftp), // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file ), // opt/repo-sftp-public-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_NAME("repo-storage-ca-file"), // opt/repo-storage-ca-file PARSE_RULE_OPTION_TYPE(String), // opt/repo-storage-ca-file PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-ca-file PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-storage-ca-file PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-ca-file PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-ca-file PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // opt/repo-storage-ca-file PARSE_RULE_OPTIONAL // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-ca-file ( // opt/repo-storage-ca-file PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-ca-file PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-ca-file PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-ca-file PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file ), // opt/repo-storage-ca-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_NAME("repo-storage-ca-path"), // opt/repo-storage-ca-path PARSE_RULE_OPTION_TYPE(Path), // opt/repo-storage-ca-path PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-ca-path PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-storage-ca-path PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-ca-path PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-ca-path PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // opt/repo-storage-ca-path PARSE_RULE_OPTIONAL // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-ca-path ( // opt/repo-storage-ca-path PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-ca-path PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-ca-path PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-ca-path PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path ), // opt/repo-storage-ca-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_NAME("repo-storage-host"), // opt/repo-storage-host PARSE_RULE_OPTION_TYPE(String), // opt/repo-storage-host PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-host PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-storage-host PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-host PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-host ), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-host ), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-host ), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-host PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-host ), // opt/repo-storage-host // opt/repo-storage-host PARSE_RULE_OPTIONAL // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-host ( // opt/repo-storage-host PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-host PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-host PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-host ), // opt/repo-storage-host ), // opt/repo-storage-host ), // opt/repo-storage-host ), // opt/repo-storage-host // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_NAME("repo-storage-port"), // opt/repo-storage-port PARSE_RULE_OPTION_TYPE(Integer), // opt/repo-storage-port PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-port PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-storage-port PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-port PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-port PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTIONAL // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-port PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-port PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_VAL_INT(1), // opt/repo-storage-port PARSE_RULE_VAL_INT(65535), // opt/repo-storage-port ), // opt/repo-storage-port // opt/repo-storage-port PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-storage-port ( // opt/repo-storage-port PARSE_RULE_VAL_INT(443), // opt/repo-storage-port ), // opt/repo-storage-port ), // opt/repo-storage-port ), // opt/repo-storage-port ), // opt/repo-storage-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_NAME("repo-storage-tag"), // opt/repo-storage-tag PARSE_RULE_OPTION_TYPE(Hash), // opt/repo-storage-tag PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-tag PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-storage-tag PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-tag PARSE_RULE_OPTION_MULTI(true), // opt/repo-storage-tag PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-tag ), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tag ), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-tag ), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-tag PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-tag ), // opt/repo-storage-tag // opt/repo-storage-tag PARSE_RULE_OPTIONAL // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-tag ( // opt/repo-storage-tag PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-tag PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-tag PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-tag PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-tag ), // opt/repo-storage-tag ), // opt/repo-storage-tag ), // opt/repo-storage-tag ), // opt/repo-storage-tag // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_NAME("repo-storage-upload-chunk-size"), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_TYPE(Size), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_ALLOW_RANGE_MAP // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(4MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(1GiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(4MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(1GiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(5MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(1GiB), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_OPTIONAL_DEFAULT_MAP // opt/repo-storage-upload-chunk-size ( // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(4MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(4MiB), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-upload-chunk-size PARSE_RULE_VAL_SIZE(5MiB), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size ), // opt/repo-storage-upload-chunk-size // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_NAME("repo-storage-verify-tls"), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_NEGATE(true), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_RESET(true), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_SECTION(Global), // opt/repo-storage-verify-tls PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-storage-verify-tls PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTIONAL // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTIONAL_GROUP // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_OPTIONAL_DEPEND // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_VAL_OPT(RepoType), // opt/repo-storage-verify-tls PARSE_RULE_VAL_STRID(azure), // opt/repo-storage-verify-tls PARSE_RULE_VAL_STRID(gcs), // opt/repo-storage-verify-tls PARSE_RULE_VAL_STRID(s3), // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // opt/repo-storage-verify-tls PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-storage-verify-tls ( // opt/repo-storage-verify-tls PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls ), // opt/repo-storage-verify-tls // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTION_NAME("repo-symlink"), // opt/repo-symlink PARSE_RULE_OPTION_TYPE(Boolean), // opt/repo-symlink PARSE_RULE_OPTION_NEGATE(true), // opt/repo-symlink PARSE_RULE_OPTION_RESET(true), // opt/repo-symlink PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-symlink PARSE_RULE_OPTION_SECTION(Global), // opt/repo-symlink PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-symlink // opt/repo-symlink PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-symlink PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-symlink ), // opt/repo-symlink // opt/repo-symlink PARSE_RULE_OPTIONAL // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTIONAL_GROUP // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTIONAL_DEPEND // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(PARSE_RULE_VAL_BOOL_FALSE), // opt/repo-symlink PARSE_RULE_VAL_OPT(RepoType), // opt/repo-symlink PARSE_RULE_VAL_STRID(posix), // opt/repo-symlink ), // opt/repo-symlink // opt/repo-symlink PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-symlink ( // opt/repo-symlink PARSE_RULE_VAL_BOOL_TRUE, // opt/repo-symlink ), // opt/repo-symlink ), // opt/repo-symlink ), // opt/repo-symlink ), // opt/repo-symlink // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-target-time ( // opt/repo-target-time PARSE_RULE_OPTION_NAME("repo-target-time"), // opt/repo-target-time PARSE_RULE_OPTION_TYPE(String), // opt/repo-target-time PARSE_RULE_OPTION_RESET(true), // opt/repo-target-time PARSE_RULE_OPTION_REQUIRED(false), // opt/repo-target-time PARSE_RULE_OPTION_SECTION(Global), // opt/repo-target-time // opt/repo-target-time PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-target-time ( // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-target-time ), // opt/repo-target-time // opt/repo-target-time PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-target-time ( // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-target-time ), // opt/repo-target-time // opt/repo-target-time PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-target-time ( // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-target-time PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-target-time ), // opt/repo-target-time ), // opt/repo-target-time // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_NAME("repo-type"), // opt/repo-type PARSE_RULE_OPTION_TYPE(StringId), // opt/repo-type PARSE_RULE_OPTION_RESET(true), // opt/repo-type PARSE_RULE_OPTION_REQUIRED(true), // opt/repo-type PARSE_RULE_OPTION_SECTION(Global), // opt/repo-type PARSE_RULE_OPTION_GROUP_ID(Repo), // opt/repo-type PARSE_RULE_OPTION_DEPRECATE_MATCH(true), // opt/repo-type // opt/repo-type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTION_COMMAND(Annotate) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Backup) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Check) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Expire) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Info) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Manifest) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/repo-type PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Restore) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/repo-type PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/repo-type PARSE_RULE_OPTION_COMMAND(Verify) // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTIONAL // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTIONAL_GROUP // opt/repo-type ( // opt/repo-type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/repo-type ( // opt/repo-type PARSE_RULE_VAL_STRID(azure), // opt/repo-type PARSE_RULE_VAL_STRID(cifs), // opt/repo-type PARSE_RULE_VAL_STRID(gcs), // opt/repo-type PARSE_RULE_VAL_STRID(posix), // opt/repo-type PARSE_RULE_VAL_STRID(s3), // opt/repo-type PARSE_RULE_VAL_STRID(sftp), // opt/repo-type #ifndef HAVE_LIBSSH2 // opt/repo-type PARSE_RULE_BOOL_FALSE, // opt/repo-type #endif // opt/repo-type ), // opt/repo-type // opt/repo-type PARSE_RULE_OPTIONAL_DEFAULT // opt/repo-type ( // opt/repo-type PARSE_RULE_VAL_STRID(posix), // opt/repo-type ), // opt/repo-type ), // opt/repo-type ), // opt/repo-type ), // opt/repo-type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/report ( // opt/report PARSE_RULE_OPTION_NAME("report"), // opt/report PARSE_RULE_OPTION_TYPE(Boolean), // opt/report PARSE_RULE_OPTION_INTERNAL(true) // opt/report PARSE_RULE_OPTION_REQUIRED(true), // opt/report PARSE_RULE_OPTION_SECTION(CommandLine), // opt/report // opt/report PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/report ( // opt/report PARSE_RULE_OPTION_COMMAND(Check) // opt/report ), // opt/report // opt/report PARSE_RULE_OPTIONAL // opt/report ( // opt/report PARSE_RULE_OPTIONAL_GROUP // opt/report ( // opt/report PARSE_RULE_OPTIONAL_DEFAULT // opt/report ( // opt/report PARSE_RULE_VAL_BOOL_FALSE, // opt/report ), // opt/report ), // opt/report ), // opt/report ), // opt/report // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/resume ( // opt/resume PARSE_RULE_OPTION_NAME("resume"), // opt/resume PARSE_RULE_OPTION_TYPE(Boolean), // opt/resume PARSE_RULE_OPTION_NEGATE(true), // opt/resume PARSE_RULE_OPTION_RESET(true), // opt/resume PARSE_RULE_OPTION_REQUIRED(true), // opt/resume PARSE_RULE_OPTION_SECTION(Global), // opt/resume // opt/resume PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/resume ( // opt/resume PARSE_RULE_OPTION_COMMAND(Backup) // opt/resume ), // opt/resume // opt/resume PARSE_RULE_OPTIONAL // opt/resume ( // opt/resume PARSE_RULE_OPTIONAL_GROUP // opt/resume ( // opt/resume PARSE_RULE_OPTIONAL_DEFAULT // opt/resume ( // opt/resume PARSE_RULE_VAL_BOOL_TRUE, // opt/resume ), // opt/resume ), // opt/resume ), // opt/resume ), // opt/resume // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_NAME("sck-block"), // opt/sck-block PARSE_RULE_OPTION_TYPE(Boolean), // opt/sck-block PARSE_RULE_OPTION_INTERNAL(true) // opt/sck-block PARSE_RULE_OPTION_NEGATE(true), // opt/sck-block PARSE_RULE_OPTION_RESET(true), // opt/sck-block PARSE_RULE_OPTION_REQUIRED(true), // opt/sck-block PARSE_RULE_OPTION_SECTION(Global), // opt/sck-block // opt/sck-block PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_COMMAND(Annotate) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Check) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Expire) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Info) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Manifest) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Server) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-block ), // opt/sck-block // opt/sck-block PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-block ), // opt/sck-block // opt/sck-block PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-block ), // opt/sck-block // opt/sck-block PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTION_COMMAND(Annotate) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Check) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Expire) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Info) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Manifest) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/sck-block PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/sck-block PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/sck-block PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-block ), // opt/sck-block // opt/sck-block PARSE_RULE_OPTIONAL // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTIONAL_GROUP // opt/sck-block ( // opt/sck-block PARSE_RULE_OPTIONAL_DEFAULT // opt/sck-block ( // opt/sck-block PARSE_RULE_VAL_BOOL_FALSE, // opt/sck-block ), // opt/sck-block ), // opt/sck-block ), // opt/sck-block ), // opt/sck-block // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_NAME("sck-keep-alive"), // opt/sck-keep-alive PARSE_RULE_OPTION_TYPE(Boolean), // opt/sck-keep-alive PARSE_RULE_OPTION_NEGATE(true), // opt/sck-keep-alive PARSE_RULE_OPTION_RESET(true), // opt/sck-keep-alive PARSE_RULE_OPTION_REQUIRED(true), // opt/sck-keep-alive PARSE_RULE_OPTION_SECTION(Global), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Annotate) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Check) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Expire) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Info) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Manifest) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Server) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-keep-alive ), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-keep-alive ), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-keep-alive ), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Annotate) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Backup) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Check) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Expire) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Info) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Manifest) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Restore) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/sck-keep-alive PARSE_RULE_OPTION_COMMAND(Verify) // opt/sck-keep-alive ), // opt/sck-keep-alive // opt/sck-keep-alive PARSE_RULE_OPTIONAL // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTIONAL_GROUP // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_OPTIONAL_DEFAULT // opt/sck-keep-alive ( // opt/sck-keep-alive PARSE_RULE_VAL_BOOL_TRUE, // opt/sck-keep-alive ), // opt/sck-keep-alive ), // opt/sck-keep-alive ), // opt/sck-keep-alive ), // opt/sck-keep-alive // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/set ( // opt/set PARSE_RULE_OPTION_NAME("set"), // opt/set PARSE_RULE_OPTION_TYPE(String), // opt/set PARSE_RULE_OPTION_REQUIRED(false), // opt/set PARSE_RULE_OPTION_SECTION(CommandLine), // opt/set // opt/set PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/set ( // opt/set PARSE_RULE_OPTION_COMMAND(Annotate) // opt/set PARSE_RULE_OPTION_COMMAND(Expire) // opt/set PARSE_RULE_OPTION_COMMAND(Info) // opt/set PARSE_RULE_OPTION_COMMAND(Manifest) // opt/set PARSE_RULE_OPTION_COMMAND(Restore) // opt/set PARSE_RULE_OPTION_COMMAND(Verify) // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL // opt/set ( // opt/set PARSE_RULE_OPTIONAL_GROUP // opt/set ( // opt/set PARSE_RULE_FILTER_CMD // opt/set ( // opt/set PARSE_RULE_VAL_CMD(Annotate), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_REQUIRED(), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_GROUP // opt/set ( // opt/set PARSE_RULE_FILTER_CMD // opt/set ( // opt/set PARSE_RULE_VAL_CMD(Info), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_DEPEND // opt/set ( // opt/set PARSE_RULE_VAL_OPT(Stanza), // opt/set ), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_GROUP // opt/set ( // opt/set PARSE_RULE_FILTER_CMD // opt/set ( // opt/set PARSE_RULE_VAL_CMD(Manifest), // opt/set PARSE_RULE_VAL_CMD(Restore), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_DEFAULT // opt/set ( // opt/set PARSE_RULE_VAL_STR(QT_latest_QT), // opt/set ), // opt/set // opt/set PARSE_RULE_OPTIONAL_REQUIRED(), // opt/set ), // opt/set ), // opt/set ), // opt/set // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/sort ( // opt/sort PARSE_RULE_OPTION_NAME("sort"), // opt/sort PARSE_RULE_OPTION_TYPE(StringId), // opt/sort PARSE_RULE_OPTION_SEQUENCE(true), // opt/sort PARSE_RULE_OPTION_REQUIRED(true), // opt/sort PARSE_RULE_OPTION_SECTION(CommandLine), // opt/sort // opt/sort PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/sort ( // opt/sort PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/sort ), // opt/sort // opt/sort PARSE_RULE_OPTIONAL // opt/sort ( // opt/sort PARSE_RULE_OPTIONAL_GROUP // opt/sort ( // opt/sort PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/sort ( // opt/sort PARSE_RULE_VAL_STRID(none), // opt/sort PARSE_RULE_VAL_STRID(asc), // opt/sort PARSE_RULE_VAL_STRID(desc), // opt/sort ), // opt/sort // opt/sort PARSE_RULE_OPTIONAL_DEFAULT // opt/sort ( // opt/sort PARSE_RULE_VAL_STRID(asc), // opt/sort PARSE_RULE_VAL_SEQ(1), // opt/sort ), // opt/sort ), // opt/sort ), // opt/sort ), // opt/sort // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_NAME("spool-path"), // opt/spool-path PARSE_RULE_OPTION_TYPE(Path), // opt/spool-path PARSE_RULE_OPTION_RESET(true), // opt/spool-path PARSE_RULE_OPTION_REQUIRED(true), // opt/spool-path PARSE_RULE_OPTION_SECTION(Global), // opt/spool-path // opt/spool-path PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_COMMAND_INTERNAL(Restore, true), // opt/spool-path ) // opt/spool-path // opt/spool-path PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/spool-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/spool-path PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/spool-path PARSE_RULE_OPTION_COMMAND(Restore) // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTIONAL // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTIONAL_GROUP // opt/spool-path ( // opt/spool-path PARSE_RULE_FILTER_CMD // opt/spool-path ( // opt/spool-path PARSE_RULE_VAL_CMD(ArchiveGet), // opt/spool-path PARSE_RULE_VAL_CMD(ArchivePush), // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTIONAL_DEPEND // opt/spool-path ( // opt/spool-path PARSE_RULE_VAL_OPT(ArchiveAsync), // opt/spool-path PARSE_RULE_VAL_BOOL_TRUE, // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTIONAL_DEFAULT // opt/spool-path ( // opt/spool-path PARSE_RULE_VAL_STR(QT_FS_var_FS_spool_FS_pgbackrest_QT), // opt/spool-path ), // opt/spool-path ), // opt/spool-path // opt/spool-path PARSE_RULE_OPTIONAL_GROUP // opt/spool-path ( // opt/spool-path PARSE_RULE_OPTIONAL_DEFAULT // opt/spool-path ( // opt/spool-path PARSE_RULE_VAL_STR(QT_FS_var_FS_spool_FS_pgbackrest_QT), // opt/spool-path ), // opt/spool-path ), // opt/spool-path ), // opt/spool-path ), // opt/spool-path // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_NAME("stanza"), // opt/stanza PARSE_RULE_OPTION_TYPE(String), // opt/stanza PARSE_RULE_OPTION_REQUIRED(true), // opt/stanza PARSE_RULE_OPTION_SECTION(CommandLine), // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND_INTERNAL(RepoGet, true), // opt/stanza PARSE_RULE_OPTION_COMMAND_INTERNAL(RepoLs, true), // opt/stanza PARSE_RULE_OPTION_COMMAND_INTERNAL(RepoPut, true), // opt/stanza PARSE_RULE_OPTION_COMMAND_INTERNAL(RepoRm, true), // opt/stanza ) // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND(Annotate) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/stanza PARSE_RULE_OPTION_COMMAND(Backup) // opt/stanza PARSE_RULE_OPTION_COMMAND(Check) // opt/stanza PARSE_RULE_OPTION_COMMAND(Expire) // opt/stanza PARSE_RULE_OPTION_COMMAND(Info) // opt/stanza PARSE_RULE_OPTION_COMMAND(Manifest) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/stanza PARSE_RULE_OPTION_COMMAND(Restore) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/stanza PARSE_RULE_OPTION_COMMAND(Start) // opt/stanza PARSE_RULE_OPTION_COMMAND(Stop) // opt/stanza PARSE_RULE_OPTION_COMMAND(Verify) // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/stanza PARSE_RULE_OPTION_COMMAND(Backup) // opt/stanza PARSE_RULE_OPTION_COMMAND(Restore) // opt/stanza PARSE_RULE_OPTION_COMMAND(Verify) // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/stanza ( // opt/stanza PARSE_RULE_OPTION_COMMAND(Annotate) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/stanza PARSE_RULE_OPTION_COMMAND(Backup) // opt/stanza PARSE_RULE_OPTION_COMMAND(Check) // opt/stanza PARSE_RULE_OPTION_COMMAND(Expire) // opt/stanza PARSE_RULE_OPTION_COMMAND(Info) // opt/stanza PARSE_RULE_OPTION_COMMAND(Manifest) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/stanza PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/stanza PARSE_RULE_OPTION_COMMAND(Restore) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/stanza PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/stanza PARSE_RULE_OPTION_COMMAND(Verify) // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTIONAL // opt/stanza ( // opt/stanza PARSE_RULE_OPTIONAL_GROUP // opt/stanza ( // opt/stanza PARSE_RULE_FILTER_CMD // opt/stanza ( // opt/stanza PARSE_RULE_VAL_CMD(Check), // opt/stanza PARSE_RULE_VAL_CMD(Info), // opt/stanza PARSE_RULE_VAL_CMD(RepoGet), // opt/stanza PARSE_RULE_VAL_CMD(RepoLs), // opt/stanza PARSE_RULE_VAL_CMD(RepoPut), // opt/stanza PARSE_RULE_VAL_CMD(RepoRm), // opt/stanza PARSE_RULE_VAL_CMD(Start), // opt/stanza PARSE_RULE_VAL_CMD(Stop), // opt/stanza ), // opt/stanza // opt/stanza PARSE_RULE_OPTIONAL_NOT_REQUIRED(), // opt/stanza ), // opt/stanza ), // opt/stanza ), // opt/stanza // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/start-fast ( // opt/start-fast PARSE_RULE_OPTION_NAME("start-fast"), // opt/start-fast PARSE_RULE_OPTION_TYPE(Boolean), // opt/start-fast PARSE_RULE_OPTION_NEGATE(true), // opt/start-fast PARSE_RULE_OPTION_RESET(true), // opt/start-fast PARSE_RULE_OPTION_REQUIRED(true), // opt/start-fast PARSE_RULE_OPTION_SECTION(Global), // opt/start-fast // opt/start-fast PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/start-fast ( // opt/start-fast PARSE_RULE_OPTION_COMMAND(Backup) // opt/start-fast ), // opt/start-fast // opt/start-fast PARSE_RULE_OPTIONAL // opt/start-fast ( // opt/start-fast PARSE_RULE_OPTIONAL_GROUP // opt/start-fast ( // opt/start-fast PARSE_RULE_OPTIONAL_DEFAULT // opt/start-fast ( // opt/start-fast PARSE_RULE_VAL_BOOL_FALSE, // opt/start-fast ), // opt/start-fast ), // opt/start-fast ), // opt/start-fast ), // opt/start-fast // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/stop-auto ( // opt/stop-auto PARSE_RULE_OPTION_NAME("stop-auto"), // opt/stop-auto PARSE_RULE_OPTION_TYPE(Boolean), // opt/stop-auto PARSE_RULE_OPTION_INTERNAL(true) // opt/stop-auto PARSE_RULE_OPTION_NEGATE(true), // opt/stop-auto PARSE_RULE_OPTION_RESET(true), // opt/stop-auto PARSE_RULE_OPTION_REQUIRED(true), // opt/stop-auto PARSE_RULE_OPTION_SECTION(Global), // opt/stop-auto // opt/stop-auto PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/stop-auto ( // opt/stop-auto PARSE_RULE_OPTION_COMMAND(Backup) // opt/stop-auto ), // opt/stop-auto // opt/stop-auto PARSE_RULE_OPTIONAL // opt/stop-auto ( // opt/stop-auto PARSE_RULE_OPTIONAL_GROUP // opt/stop-auto ( // opt/stop-auto PARSE_RULE_OPTIONAL_DEFAULT // opt/stop-auto ( // opt/stop-auto PARSE_RULE_VAL_BOOL_FALSE, // opt/stop-auto ), // opt/stop-auto ), // opt/stop-auto ), // opt/stop-auto ), // opt/stop-auto // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tablespace-map ( // opt/tablespace-map PARSE_RULE_OPTION_NAME("tablespace-map"), // opt/tablespace-map PARSE_RULE_OPTION_TYPE(Hash), // opt/tablespace-map PARSE_RULE_OPTION_RESET(true), // opt/tablespace-map PARSE_RULE_OPTION_REQUIRED(false), // opt/tablespace-map PARSE_RULE_OPTION_SECTION(Global), // opt/tablespace-map PARSE_RULE_OPTION_MULTI(true), // opt/tablespace-map // opt/tablespace-map PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tablespace-map ( // opt/tablespace-map PARSE_RULE_OPTION_COMMAND(Restore) // opt/tablespace-map ), // opt/tablespace-map ), // opt/tablespace-map // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tablespace-map-all ( // opt/tablespace-map-all PARSE_RULE_OPTION_NAME("tablespace-map-all"), // opt/tablespace-map-all PARSE_RULE_OPTION_TYPE(String), // opt/tablespace-map-all PARSE_RULE_OPTION_RESET(true), // opt/tablespace-map-all PARSE_RULE_OPTION_REQUIRED(false), // opt/tablespace-map-all PARSE_RULE_OPTION_SECTION(Global), // opt/tablespace-map-all // opt/tablespace-map-all PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tablespace-map-all ( // opt/tablespace-map-all PARSE_RULE_OPTION_COMMAND(Restore) // opt/tablespace-map-all ), // opt/tablespace-map-all ), // opt/tablespace-map-all // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/target ( // opt/target PARSE_RULE_OPTION_NAME("target"), // opt/target PARSE_RULE_OPTION_TYPE(String), // opt/target PARSE_RULE_OPTION_REQUIRED(true), // opt/target PARSE_RULE_OPTION_SECTION(CommandLine), // opt/target // opt/target PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/target ( // opt/target PARSE_RULE_OPTION_COMMAND(Restore) // opt/target ), // opt/target // opt/target PARSE_RULE_OPTIONAL // opt/target ( // opt/target PARSE_RULE_OPTIONAL_GROUP // opt/target ( // opt/target PARSE_RULE_OPTIONAL_DEPEND // opt/target ( // opt/target PARSE_RULE_VAL_OPT(Type), // opt/target PARSE_RULE_VAL_STRID(lsn), // opt/target PARSE_RULE_VAL_STRID(name), // opt/target PARSE_RULE_VAL_STRID(time), // opt/target PARSE_RULE_VAL_STRID(xid), // opt/target ), // opt/target ), // opt/target ), // opt/target ), // opt/target // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/target-action ( // opt/target-action PARSE_RULE_OPTION_NAME("target-action"), // opt/target-action PARSE_RULE_OPTION_TYPE(StringId), // opt/target-action PARSE_RULE_OPTION_REQUIRED(true), // opt/target-action PARSE_RULE_OPTION_SECTION(CommandLine), // opt/target-action // opt/target-action PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/target-action ( // opt/target-action PARSE_RULE_OPTION_COMMAND(Restore) // opt/target-action ), // opt/target-action // opt/target-action PARSE_RULE_OPTIONAL // opt/target-action ( // opt/target-action PARSE_RULE_OPTIONAL_GROUP // opt/target-action ( // opt/target-action PARSE_RULE_OPTIONAL_DEPEND // opt/target-action ( // opt/target-action PARSE_RULE_VAL_OPT(Type), // opt/target-action PARSE_RULE_VAL_STRID(immediate), // opt/target-action PARSE_RULE_VAL_STRID(lsn), // opt/target-action PARSE_RULE_VAL_STRID(name), // opt/target-action PARSE_RULE_VAL_STRID(time), // opt/target-action PARSE_RULE_VAL_STRID(xid), // opt/target-action ), // opt/target-action // opt/target-action PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/target-action ( // opt/target-action PARSE_RULE_VAL_STRID(pause), // opt/target-action PARSE_RULE_VAL_STRID(promote), // opt/target-action PARSE_RULE_VAL_STRID(shutdown), // opt/target-action ), // opt/target-action // opt/target-action PARSE_RULE_OPTIONAL_DEFAULT // opt/target-action ( // opt/target-action PARSE_RULE_VAL_STRID(pause), // opt/target-action ), // opt/target-action ), // opt/target-action ), // opt/target-action ), // opt/target-action // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_OPTION_NAME("target-exclusive"), // opt/target-exclusive PARSE_RULE_OPTION_TYPE(Boolean), // opt/target-exclusive PARSE_RULE_OPTION_REQUIRED(true), // opt/target-exclusive PARSE_RULE_OPTION_SECTION(CommandLine), // opt/target-exclusive // opt/target-exclusive PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_OPTION_COMMAND(Restore) // opt/target-exclusive ), // opt/target-exclusive // opt/target-exclusive PARSE_RULE_OPTIONAL // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_OPTIONAL_GROUP // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_OPTIONAL_DEPEND // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_VAL_OPT(Type), // opt/target-exclusive PARSE_RULE_VAL_STRID(lsn), // opt/target-exclusive PARSE_RULE_VAL_STRID(time), // opt/target-exclusive PARSE_RULE_VAL_STRID(xid), // opt/target-exclusive ), // opt/target-exclusive // opt/target-exclusive PARSE_RULE_OPTIONAL_DEFAULT // opt/target-exclusive ( // opt/target-exclusive PARSE_RULE_VAL_BOOL_FALSE, // opt/target-exclusive ), // opt/target-exclusive ), // opt/target-exclusive ), // opt/target-exclusive ), // opt/target-exclusive // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/target-timeline ( // opt/target-timeline PARSE_RULE_OPTION_NAME("target-timeline"), // opt/target-timeline PARSE_RULE_OPTION_TYPE(String), // opt/target-timeline PARSE_RULE_OPTION_REQUIRED(false), // opt/target-timeline PARSE_RULE_OPTION_SECTION(CommandLine), // opt/target-timeline // opt/target-timeline PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/target-timeline ( // opt/target-timeline PARSE_RULE_OPTION_COMMAND(Restore) // opt/target-timeline ), // opt/target-timeline // opt/target-timeline PARSE_RULE_OPTIONAL // opt/target-timeline ( // opt/target-timeline PARSE_RULE_OPTIONAL_GROUP // opt/target-timeline ( // opt/target-timeline PARSE_RULE_OPTIONAL_DEPEND // opt/target-timeline ( // opt/target-timeline PARSE_RULE_VAL_OPT(Type), // opt/target-timeline PARSE_RULE_VAL_STRID(default), // opt/target-timeline PARSE_RULE_VAL_STRID(lsn), // opt/target-timeline PARSE_RULE_VAL_STRID(name), // opt/target-timeline PARSE_RULE_VAL_STRID(standby), // opt/target-timeline PARSE_RULE_VAL_STRID(time), // opt/target-timeline PARSE_RULE_VAL_STRID(xid), // opt/target-timeline ), // opt/target-timeline ), // opt/target-timeline ), // opt/target-timeline ), // opt/target-timeline // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_NAME("tcp-keep-alive-count"), // opt/tcp-keep-alive-count PARSE_RULE_OPTION_TYPE(Integer), // opt/tcp-keep-alive-count PARSE_RULE_OPTION_RESET(true), // opt/tcp-keep-alive-count PARSE_RULE_OPTION_REQUIRED(false), // opt/tcp-keep-alive-count PARSE_RULE_OPTION_SECTION(Global), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Server) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-count PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTIONAL // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTIONAL_GROUP // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_OPTIONAL_DEPEND // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_VAL_OPT(SckKeepAlive), // opt/tcp-keep-alive-count PARSE_RULE_VAL_BOOL_TRUE, // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // opt/tcp-keep-alive-count PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/tcp-keep-alive-count ( // opt/tcp-keep-alive-count PARSE_RULE_VAL_INT(1), // opt/tcp-keep-alive-count PARSE_RULE_VAL_INT(32), // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count ), // opt/tcp-keep-alive-count // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_NAME("tcp-keep-alive-idle"), // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_TYPE(Integer), // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_RESET(true), // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_REQUIRED(false), // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_SECTION(Global), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Server) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-idle PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTIONAL // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTIONAL_GROUP // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_OPTIONAL_DEPEND // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_VAL_OPT(SckKeepAlive), // opt/tcp-keep-alive-idle PARSE_RULE_VAL_BOOL_TRUE, // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // opt/tcp-keep-alive-idle PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/tcp-keep-alive-idle ( // opt/tcp-keep-alive-idle PARSE_RULE_VAL_INT(1), // opt/tcp-keep-alive-idle PARSE_RULE_VAL_INT(3600), // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle ), // opt/tcp-keep-alive-idle // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_NAME("tcp-keep-alive-interval"), // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_TYPE(Integer), // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_RESET(true), // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_REQUIRED(false), // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_SECTION(Global), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Server) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Backup) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Check) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Expire) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Info) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Restore) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tcp-keep-alive-interval PARSE_RULE_OPTION_COMMAND(Verify) // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTIONAL // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTIONAL_GROUP // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_OPTIONAL_DEPEND // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_VAL_OPT(SckKeepAlive), // opt/tcp-keep-alive-interval PARSE_RULE_VAL_BOOL_TRUE, // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // opt/tcp-keep-alive-interval PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/tcp-keep-alive-interval ( // opt/tcp-keep-alive-interval PARSE_RULE_VAL_INT(1), // opt/tcp-keep-alive-interval PARSE_RULE_VAL_INT(900), // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval ), // opt/tcp-keep-alive-interval // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_NAME("tls-cipher-12"), // opt/tls-cipher-12 PARSE_RULE_OPTION_TYPE(String), // opt/tls-cipher-12 PARSE_RULE_OPTION_RESET(true), // opt/tls-cipher-12 PARSE_RULE_OPTION_REQUIRED(false), // opt/tls-cipher-12 PARSE_RULE_OPTION_SECTION(Global), // opt/tls-cipher-12 // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Check) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Expire) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Info) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-12 ), // opt/tls-cipher-12 // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-12 ), // opt/tls-cipher-12 // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-12 ), // opt/tls-cipher-12 // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tls-cipher-12 ( // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Check) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Expire) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Info) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tls-cipher-12 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-12 ), // opt/tls-cipher-12 ), // opt/tls-cipher-12 // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_NAME("tls-cipher-13"), // opt/tls-cipher-13 PARSE_RULE_OPTION_TYPE(String), // opt/tls-cipher-13 PARSE_RULE_OPTION_RESET(true), // opt/tls-cipher-13 PARSE_RULE_OPTION_REQUIRED(false), // opt/tls-cipher-13 PARSE_RULE_OPTION_SECTION(Global), // opt/tls-cipher-13 // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Check) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Expire) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Info) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-13 ), // opt/tls-cipher-13 // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-13 ), // opt/tls-cipher-13 // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-13 ), // opt/tls-cipher-13 // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST // opt/tls-cipher-13 ( // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Annotate) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchiveGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(ArchivePush) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Backup) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Check) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Expire) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Info) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Manifest) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoGet) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoLs) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoPut) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(RepoRm) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Restore) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaCreate) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaDelete) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(StanzaUpgrade) // opt/tls-cipher-13 PARSE_RULE_OPTION_COMMAND(Verify) // opt/tls-cipher-13 ), // opt/tls-cipher-13 ), // opt/tls-cipher-13 // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_OPTION_NAME("tls-server-address"), // opt/tls-server-address PARSE_RULE_OPTION_TYPE(String), // opt/tls-server-address PARSE_RULE_OPTION_RESET(true), // opt/tls-server-address PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-address PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-address // opt/tls-server-address PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-address PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tls-server-address ), // opt/tls-server-address // opt/tls-server-address PARSE_RULE_OPTIONAL // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_OPTIONAL_GROUP // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_OPTIONAL_DEFAULT // opt/tls-server-address ( // opt/tls-server-address PARSE_RULE_VAL_STR(QT_localhost_QT), // opt/tls-server-address ), // opt/tls-server-address ), // opt/tls-server-address ), // opt/tls-server-address ), // opt/tls-server-address // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-auth ( // opt/tls-server-auth PARSE_RULE_OPTION_NAME("tls-server-auth"), // opt/tls-server-auth PARSE_RULE_OPTION_TYPE(Hash), // opt/tls-server-auth PARSE_RULE_OPTION_RESET(true), // opt/tls-server-auth PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-auth PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-auth PARSE_RULE_OPTION_MULTI(true), // opt/tls-server-auth // opt/tls-server-auth PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-auth ( // opt/tls-server-auth PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-auth ), // opt/tls-server-auth ), // opt/tls-server-auth // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-ca-file ( // opt/tls-server-ca-file PARSE_RULE_OPTION_NAME("tls-server-ca-file"), // opt/tls-server-ca-file PARSE_RULE_OPTION_TYPE(Path), // opt/tls-server-ca-file PARSE_RULE_OPTION_RESET(true), // opt/tls-server-ca-file PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-ca-file PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-ca-file // opt/tls-server-ca-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-ca-file ( // opt/tls-server-ca-file PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-ca-file ), // opt/tls-server-ca-file ), // opt/tls-server-ca-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-cert-file ( // opt/tls-server-cert-file PARSE_RULE_OPTION_NAME("tls-server-cert-file"), // opt/tls-server-cert-file PARSE_RULE_OPTION_TYPE(Path), // opt/tls-server-cert-file PARSE_RULE_OPTION_RESET(true), // opt/tls-server-cert-file PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-cert-file PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-cert-file // opt/tls-server-cert-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-cert-file ( // opt/tls-server-cert-file PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-cert-file ), // opt/tls-server-cert-file ), // opt/tls-server-cert-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-key-file ( // opt/tls-server-key-file PARSE_RULE_OPTION_NAME("tls-server-key-file"), // opt/tls-server-key-file PARSE_RULE_OPTION_TYPE(Path), // opt/tls-server-key-file PARSE_RULE_OPTION_RESET(true), // opt/tls-server-key-file PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-key-file PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-key-file // opt/tls-server-key-file PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-key-file ( // opt/tls-server-key-file PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-key-file ), // opt/tls-server-key-file ), // opt/tls-server-key-file // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_OPTION_NAME("tls-server-port"), // opt/tls-server-port PARSE_RULE_OPTION_TYPE(Integer), // opt/tls-server-port PARSE_RULE_OPTION_RESET(true), // opt/tls-server-port PARSE_RULE_OPTION_REQUIRED(true), // opt/tls-server-port PARSE_RULE_OPTION_SECTION(Global), // opt/tls-server-port // opt/tls-server-port PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_OPTION_COMMAND(Server) // opt/tls-server-port PARSE_RULE_OPTION_COMMAND(ServerPing) // opt/tls-server-port ), // opt/tls-server-port // opt/tls-server-port PARSE_RULE_OPTIONAL // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_OPTIONAL_GROUP // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_OPTIONAL_ALLOW_RANGE // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_VAL_INT(1), // opt/tls-server-port PARSE_RULE_VAL_INT(65535), // opt/tls-server-port ), // opt/tls-server-port // opt/tls-server-port PARSE_RULE_OPTIONAL_DEFAULT // opt/tls-server-port ( // opt/tls-server-port PARSE_RULE_VAL_INT(8432), // opt/tls-server-port ), // opt/tls-server-port ), // opt/tls-server-port ), // opt/tls-server-port ), // opt/tls-server-port // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/type ( // opt/type PARSE_RULE_OPTION_NAME("type"), // opt/type PARSE_RULE_OPTION_TYPE(StringId), // opt/type PARSE_RULE_OPTION_REQUIRED(true), // opt/type PARSE_RULE_OPTION_SECTION(CommandLine), // opt/type // opt/type PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/type ( // opt/type PARSE_RULE_OPTION_COMMAND(Backup) // opt/type PARSE_RULE_OPTION_COMMAND(Info) // opt/type PARSE_RULE_OPTION_COMMAND(Restore) // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL // opt/type ( // opt/type PARSE_RULE_OPTIONAL_GROUP // opt/type ( // opt/type PARSE_RULE_FILTER_CMD // opt/type ( // opt/type PARSE_RULE_VAL_CMD(Backup), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/type ( // opt/type PARSE_RULE_VAL_STRID(full), // opt/type PARSE_RULE_VAL_STRID(diff), // opt/type PARSE_RULE_VAL_STRID(incr), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_DEFAULT // opt/type ( // opt/type PARSE_RULE_VAL_STRID(incr), // opt/type ), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_GROUP // opt/type ( // opt/type PARSE_RULE_FILTER_CMD // opt/type ( // opt/type PARSE_RULE_VAL_CMD(Info), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/type ( // opt/type PARSE_RULE_VAL_STRID(full), // opt/type PARSE_RULE_VAL_STRID(diff), // opt/type PARSE_RULE_VAL_STRID(incr), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_NOT_REQUIRED(), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_GROUP // opt/type ( // opt/type PARSE_RULE_FILTER_CMD // opt/type ( // opt/type PARSE_RULE_VAL_CMD(Restore), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_SEQUENCE(), // opt/type // opt/type PARSE_RULE_OPTIONAL_ALLOW_LIST // opt/type ( // opt/type PARSE_RULE_VAL_STRID(default), // opt/type PARSE_RULE_VAL_STRID(preserve), // opt/type PARSE_RULE_VAL_STRID(immediate), // opt/type PARSE_RULE_VAL_STRID(standby), // opt/type PARSE_RULE_VAL_STRID(none), // opt/type PARSE_RULE_VAL_STRID(name), // opt/type PARSE_RULE_VAL_STRID(time), // opt/type PARSE_RULE_VAL_STRID(lsn), // opt/type PARSE_RULE_VAL_STRID(xid), // opt/type ), // opt/type // opt/type PARSE_RULE_OPTIONAL_DEFAULT // opt/type ( // opt/type PARSE_RULE_VAL_STRID(default), // opt/type ), // opt/type ), // opt/type ), // opt/type ), // opt/type // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/verbose ( // opt/verbose PARSE_RULE_OPTION_NAME("verbose"), // opt/verbose PARSE_RULE_OPTION_TYPE(Boolean), // opt/verbose PARSE_RULE_OPTION_REQUIRED(true), // opt/verbose PARSE_RULE_OPTION_SECTION(CommandLine), // opt/verbose // opt/verbose PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/verbose ( // opt/verbose PARSE_RULE_OPTION_COMMAND(Verify) // opt/verbose ), // opt/verbose // opt/verbose PARSE_RULE_OPTIONAL // opt/verbose ( // opt/verbose PARSE_RULE_OPTIONAL_GROUP // opt/verbose ( // opt/verbose PARSE_RULE_OPTIONAL_DEFAULT // opt/verbose ( // opt/verbose PARSE_RULE_VAL_BOOL_FALSE, // opt/verbose ), // opt/verbose ), // opt/verbose ), // opt/verbose ), // opt/verbose // ----------------------------------------------------------------------------------------------------------------------------- PARSE_RULE_OPTION // opt/version ( // opt/version PARSE_RULE_OPTION_NAME("version"), // opt/version PARSE_RULE_OPTION_TYPE(Boolean), // opt/version PARSE_RULE_OPTION_REQUIRED(true), // opt/version PARSE_RULE_OPTION_SECTION(CommandLine), // opt/version // opt/version PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST // opt/version ( // opt/version PARSE_RULE_OPTION_COMMAND(Help) // opt/version ), // opt/version // opt/version PARSE_RULE_OPTIONAL // opt/version ( // opt/version PARSE_RULE_OPTIONAL_GROUP // opt/version ( // opt/version PARSE_RULE_OPTIONAL_DEFAULT // opt/version ( // opt/version PARSE_RULE_VAL_BOOL_FALSE, // opt/version ), // opt/version ), // opt/version ), // opt/version ), // opt/version }; /*********************************************************************************************************************************** Option deprecations ***********************************************************************************************************************************/ #define CFG_OPTION_DEPRECATE_TOTAL 39 static const ParseRuleOptionDeprecate parseRuleOptionDeprecate[CFG_OPTION_DEPRECATE_TOTAL] = { { // opt-deprecate/archive-push-queue-max .name = "archive-queue-max", // opt-deprecate/archive-push-queue-max .id = cfgOptArchivePushQueueMax, // opt-deprecate/archive-push-queue-max .unindexed = true, // opt-deprecate/archive-push-queue-max }, // opt-deprecate/archive-push-queue-max // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host-cmd .name = "backup-cmd", // opt-deprecate/repo-host-cmd .id = cfgOptRepoHostCmd, // opt-deprecate/repo-host-cmd .unindexed = true, // opt-deprecate/repo-host-cmd }, // opt-deprecate/repo-host-cmd // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host-config .name = "backup-config", // opt-deprecate/repo-host-config .id = cfgOptRepoHostConfig, // opt-deprecate/repo-host-config .unindexed = true, // opt-deprecate/repo-host-config }, // opt-deprecate/repo-host-config // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host .name = "backup-host", // opt-deprecate/repo-host .id = cfgOptRepoHost, // opt-deprecate/repo-host .unindexed = true, // opt-deprecate/repo-host }, // opt-deprecate/repo-host // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host-port .name = "backup-ssh-port", // opt-deprecate/repo-host-port .id = cfgOptRepoHostPort, // opt-deprecate/repo-host-port .unindexed = true, // opt-deprecate/repo-host-port }, // opt-deprecate/repo-host-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-host-user .name = "backup-user", // opt-deprecate/repo-host-user .id = cfgOptRepoHostUser, // opt-deprecate/repo-host-user .unindexed = true, // opt-deprecate/repo-host-user }, // opt-deprecate/repo-host-user // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host-cmd .name = "db-cmd", // opt-deprecate/pg-host-cmd .id = cfgOptPgHostCmd, // opt-deprecate/pg-host-cmd .indexed = true, // opt-deprecate/pg-host-cmd .unindexed = true, // opt-deprecate/pg-host-cmd }, // opt-deprecate/pg-host-cmd // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host-config .name = "db-config", // opt-deprecate/pg-host-config .id = cfgOptPgHostConfig, // opt-deprecate/pg-host-config .indexed = true, // opt-deprecate/pg-host-config .unindexed = true, // opt-deprecate/pg-host-config }, // opt-deprecate/pg-host-config // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host .name = "db-host", // opt-deprecate/pg-host .id = cfgOptPgHost, // opt-deprecate/pg-host .indexed = true, // opt-deprecate/pg-host .unindexed = true, // opt-deprecate/pg-host }, // opt-deprecate/pg-host // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-path .name = "db-path", // opt-deprecate/pg-path .id = cfgOptPgPath, // opt-deprecate/pg-path .indexed = true, // opt-deprecate/pg-path .unindexed = true, // opt-deprecate/pg-path }, // opt-deprecate/pg-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-port .name = "db-port", // opt-deprecate/pg-port .id = cfgOptPgPort, // opt-deprecate/pg-port .indexed = true, // opt-deprecate/pg-port .unindexed = true, // opt-deprecate/pg-port }, // opt-deprecate/pg-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-socket-path .name = "db-socket-path", // opt-deprecate/pg-socket-path .id = cfgOptPgSocketPath, // opt-deprecate/pg-socket-path .indexed = true, // opt-deprecate/pg-socket-path .unindexed = true, // opt-deprecate/pg-socket-path }, // opt-deprecate/pg-socket-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host-port .name = "db-ssh-port", // opt-deprecate/pg-host-port .id = cfgOptPgHostPort, // opt-deprecate/pg-host-port .indexed = true, // opt-deprecate/pg-host-port .unindexed = true, // opt-deprecate/pg-host-port }, // opt-deprecate/pg-host-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/pg-host-user .name = "db-user", // opt-deprecate/pg-host-user .id = cfgOptPgHostUser, // opt-deprecate/pg-host-user .indexed = true, // opt-deprecate/pg-host-user .unindexed = true, // opt-deprecate/pg-host-user }, // opt-deprecate/pg-host-user // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-hardlink .name = "hardlink", // opt-deprecate/repo-hardlink .id = cfgOptRepoHardlink, // opt-deprecate/repo-hardlink .unindexed = true, // opt-deprecate/repo-hardlink }, // opt-deprecate/repo-hardlink // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-ca-file .name = "repo-azure-ca-file", // opt-deprecate/repo-storage-ca-file .id = cfgOptRepoStorageCaFile, // opt-deprecate/repo-storage-ca-file .indexed = true, // opt-deprecate/repo-storage-ca-file }, // opt-deprecate/repo-storage-ca-file // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-ca-path .name = "repo-azure-ca-path", // opt-deprecate/repo-storage-ca-path .id = cfgOptRepoStorageCaPath, // opt-deprecate/repo-storage-ca-path .indexed = true, // opt-deprecate/repo-storage-ca-path }, // opt-deprecate/repo-storage-ca-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-host .name = "repo-azure-host", // opt-deprecate/repo-storage-host .id = cfgOptRepoStorageHost, // opt-deprecate/repo-storage-host .indexed = true, // opt-deprecate/repo-storage-host }, // opt-deprecate/repo-storage-host // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-port .name = "repo-azure-port", // opt-deprecate/repo-storage-port .id = cfgOptRepoStoragePort, // opt-deprecate/repo-storage-port .indexed = true, // opt-deprecate/repo-storage-port }, // opt-deprecate/repo-storage-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-verify-tls .name = "repo-azure-verify-tls", // opt-deprecate/repo-storage-verify-tls .id = cfgOptRepoStorageVerifyTls, // opt-deprecate/repo-storage-verify-tls .indexed = true, // opt-deprecate/repo-storage-verify-tls }, // opt-deprecate/repo-storage-verify-tls // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-cipher-pass .name = "repo-cipher-pass", // opt-deprecate/repo-cipher-pass .id = cfgOptRepoCipherPass, // opt-deprecate/repo-cipher-pass .unindexed = true, // opt-deprecate/repo-cipher-pass }, // opt-deprecate/repo-cipher-pass // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-cipher-type .name = "repo-cipher-type", // opt-deprecate/repo-cipher-type .id = cfgOptRepoCipherType, // opt-deprecate/repo-cipher-type .unindexed = true, // opt-deprecate/repo-cipher-type }, // opt-deprecate/repo-cipher-type // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-path .name = "repo-path", // opt-deprecate/repo-path .id = cfgOptRepoPath, // opt-deprecate/repo-path .unindexed = true, // opt-deprecate/repo-path }, // opt-deprecate/repo-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-bucket .name = "repo-s3-bucket", // opt-deprecate/repo-s3-bucket .id = cfgOptRepoS3Bucket, // opt-deprecate/repo-s3-bucket .unindexed = true, // opt-deprecate/repo-s3-bucket }, // opt-deprecate/repo-s3-bucket // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-ca-file .name = "repo-s3-ca-file", // opt-deprecate/repo-storage-ca-file .id = cfgOptRepoStorageCaFile, // opt-deprecate/repo-storage-ca-file .indexed = true, // opt-deprecate/repo-storage-ca-file .unindexed = true, // opt-deprecate/repo-storage-ca-file }, // opt-deprecate/repo-storage-ca-file // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-ca-path .name = "repo-s3-ca-path", // opt-deprecate/repo-storage-ca-path .id = cfgOptRepoStorageCaPath, // opt-deprecate/repo-storage-ca-path .indexed = true, // opt-deprecate/repo-storage-ca-path .unindexed = true, // opt-deprecate/repo-storage-ca-path }, // opt-deprecate/repo-storage-ca-path // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-endpoint .name = "repo-s3-endpoint", // opt-deprecate/repo-s3-endpoint .id = cfgOptRepoS3Endpoint, // opt-deprecate/repo-s3-endpoint .unindexed = true, // opt-deprecate/repo-s3-endpoint }, // opt-deprecate/repo-s3-endpoint // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-host .name = "repo-s3-host", // opt-deprecate/repo-storage-host .id = cfgOptRepoStorageHost, // opt-deprecate/repo-storage-host .indexed = true, // opt-deprecate/repo-storage-host .unindexed = true, // opt-deprecate/repo-storage-host }, // opt-deprecate/repo-storage-host // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-key .name = "repo-s3-key", // opt-deprecate/repo-s3-key .id = cfgOptRepoS3Key, // opt-deprecate/repo-s3-key .unindexed = true, // opt-deprecate/repo-s3-key }, // opt-deprecate/repo-s3-key // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-key-secret .name = "repo-s3-key-secret", // opt-deprecate/repo-s3-key-secret .id = cfgOptRepoS3KeySecret, // opt-deprecate/repo-s3-key-secret .unindexed = true, // opt-deprecate/repo-s3-key-secret }, // opt-deprecate/repo-s3-key-secret // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-port .name = "repo-s3-port", // opt-deprecate/repo-storage-port .id = cfgOptRepoStoragePort, // opt-deprecate/repo-storage-port .indexed = true, // opt-deprecate/repo-storage-port }, // opt-deprecate/repo-storage-port // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-s3-region .name = "repo-s3-region", // opt-deprecate/repo-s3-region .id = cfgOptRepoS3Region, // opt-deprecate/repo-s3-region .unindexed = true, // opt-deprecate/repo-s3-region }, // opt-deprecate/repo-s3-region // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-verify-tls .name = "repo-s3-verify-ssl", // opt-deprecate/repo-storage-verify-tls .id = cfgOptRepoStorageVerifyTls, // opt-deprecate/repo-storage-verify-tls .indexed = true, // opt-deprecate/repo-storage-verify-tls .unindexed = true, // opt-deprecate/repo-storage-verify-tls }, // opt-deprecate/repo-storage-verify-tls // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-storage-verify-tls .name = "repo-s3-verify-tls", // opt-deprecate/repo-storage-verify-tls .id = cfgOptRepoStorageVerifyTls, // opt-deprecate/repo-storage-verify-tls .indexed = true, // opt-deprecate/repo-storage-verify-tls }, // opt-deprecate/repo-storage-verify-tls // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-type .name = "repo-type", // opt-deprecate/repo-type .id = cfgOptRepoType, // opt-deprecate/repo-type .unindexed = true, // opt-deprecate/repo-type }, // opt-deprecate/repo-type // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-retention-archive .name = "retention-archive", // opt-deprecate/repo-retention-archive .id = cfgOptRepoRetentionArchive, // opt-deprecate/repo-retention-archive .unindexed = true, // opt-deprecate/repo-retention-archive }, // opt-deprecate/repo-retention-archive // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-retention-archive-type .name = "retention-archive-type", // opt-deprecate/repo-retention-archive-type .id = cfgOptRepoRetentionArchiveType, // opt-deprecate/repo-retention-archive-type .unindexed = true, // opt-deprecate/repo-retention-archive-type }, // opt-deprecate/repo-retention-archive-type // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-retention-diff .name = "retention-diff", // opt-deprecate/repo-retention-diff .id = cfgOptRepoRetentionDiff, // opt-deprecate/repo-retention-diff .unindexed = true, // opt-deprecate/repo-retention-diff }, // opt-deprecate/repo-retention-diff // ----------------------------------------------------------------------------------------------------------------------------- { // opt-deprecate/repo-retention-full .name = "retention-full", // opt-deprecate/repo-retention-full .id = cfgOptRepoRetentionFull, // opt-deprecate/repo-retention-full .unindexed = true, // opt-deprecate/repo-retention-full }, // opt-deprecate/repo-retention-full }; /*********************************************************************************************************************************** Order for option parse resolution ***********************************************************************************************************************************/ static const uint8_t optionResolveOrder[] = { cfgOptStanza, // opt-resolve-order cfgOptAllowRoot, // opt-resolve-order cfgOptAnnotation, // opt-resolve-order cfgOptArchiveAsync, // opt-resolve-order cfgOptArchiveExpireBefore, // opt-resolve-order cfgOptArchiveGetQueueMax, // opt-resolve-order cfgOptArchiveHeaderCheck, // opt-resolve-order cfgOptArchiveMissingRetry, // opt-resolve-order cfgOptArchiveMode, // opt-resolve-order cfgOptArchivePushBatchSize, // opt-resolve-order cfgOptArchivePushQueueMax, // opt-resolve-order cfgOptArchiveTimeout, // opt-resolve-order cfgOptBackupStandby, // opt-resolve-order cfgOptBeta, // opt-resolve-order cfgOptBufferSize, // opt-resolve-order cfgOptChecksumPage, // opt-resolve-order cfgOptCipherPass, // opt-resolve-order cfgOptCmd, // opt-resolve-order cfgOptCmdSsh, // opt-resolve-order cfgOptCompress, // opt-resolve-order cfgOptCompressLevelNetwork, // opt-resolve-order cfgOptCompressType, // opt-resolve-order cfgOptConfig, // opt-resolve-order cfgOptConfigIncludePath, // opt-resolve-order cfgOptConfigPath, // opt-resolve-order cfgOptDbExclude, // opt-resolve-order cfgOptDbInclude, // opt-resolve-order cfgOptDbTimeout, // opt-resolve-order cfgOptDelta, // opt-resolve-order cfgOptDetailLevel, // opt-resolve-order cfgOptDryRun, // opt-resolve-order cfgOptExclude, // opt-resolve-order cfgOptExecId, // opt-resolve-order cfgOptExpireAuto, // opt-resolve-order cfgOptFilter, // opt-resolve-order cfgOptHelp, // opt-resolve-order cfgOptIgnoreMissing, // opt-resolve-order cfgOptIoTimeout, // opt-resolve-order cfgOptJobRetry, // opt-resolve-order cfgOptJobRetryInterval, // opt-resolve-order cfgOptLinkAll, // opt-resolve-order cfgOptLinkMap, // opt-resolve-order cfgOptLock, // opt-resolve-order cfgOptLockPath, // opt-resolve-order cfgOptLogLevelConsole, // opt-resolve-order cfgOptLogLevelFile, // opt-resolve-order cfgOptLogLevelStderr, // opt-resolve-order cfgOptLogPath, // opt-resolve-order cfgOptLogSubprocess, // opt-resolve-order cfgOptLogTimestamp, // opt-resolve-order cfgOptManifestSaveThreshold, // opt-resolve-order cfgOptNeutralUmask, // opt-resolve-order cfgOptOldest, // opt-resolve-order cfgOptOnline, // opt-resolve-order cfgOptOutput, // opt-resolve-order cfgOptPageHeaderCheck, // opt-resolve-order cfgOptPg, // opt-resolve-order cfgOptPgLocal, // opt-resolve-order cfgOptPgPath, // opt-resolve-order cfgOptPgPort, // opt-resolve-order cfgOptPgSocketPath, // opt-resolve-order cfgOptPgUser, // opt-resolve-order cfgOptPgVersionForce, // opt-resolve-order cfgOptPriority, // opt-resolve-order cfgOptProcess, // opt-resolve-order cfgOptProcessMax, // opt-resolve-order cfgOptProtocolTimeout, // opt-resolve-order cfgOptRaw, // opt-resolve-order cfgOptRecurse, // opt-resolve-order cfgOptReference, // opt-resolve-order cfgOptRemoteType, // opt-resolve-order cfgOptRepo, // opt-resolve-order cfgOptRepoBundle, // opt-resolve-order cfgOptRepoBundleLimit, // opt-resolve-order cfgOptRepoBundleSize, // opt-resolve-order cfgOptRepoCipherType, // opt-resolve-order cfgOptRepoGcsUserProject, // opt-resolve-order cfgOptRepoHardlink, // opt-resolve-order cfgOptRepoLocal, // opt-resolve-order cfgOptRepoPath, // opt-resolve-order cfgOptRepoRetentionArchive, // opt-resolve-order cfgOptRepoRetentionArchiveType, // opt-resolve-order cfgOptRepoRetentionDiff, // opt-resolve-order cfgOptRepoRetentionFull, // opt-resolve-order cfgOptRepoRetentionFullType, // opt-resolve-order cfgOptRepoRetentionHistory, // opt-resolve-order cfgOptRepoTargetTime, // opt-resolve-order cfgOptRepoType, // opt-resolve-order cfgOptReport, // opt-resolve-order cfgOptResume, // opt-resolve-order cfgOptSckBlock, // opt-resolve-order cfgOptSckKeepAlive, // opt-resolve-order cfgOptSet, // opt-resolve-order cfgOptSort, // opt-resolve-order cfgOptSpoolPath, // opt-resolve-order cfgOptStartFast, // opt-resolve-order cfgOptStopAuto, // opt-resolve-order cfgOptTablespaceMap, // opt-resolve-order cfgOptTablespaceMapAll, // opt-resolve-order cfgOptTcpKeepAliveCount, // opt-resolve-order cfgOptTcpKeepAliveIdle, // opt-resolve-order cfgOptTcpKeepAliveInterval, // opt-resolve-order cfgOptTlsCipher12, // opt-resolve-order cfgOptTlsCipher13, // opt-resolve-order cfgOptTlsServerAddress, // opt-resolve-order cfgOptTlsServerAuth, // opt-resolve-order cfgOptTlsServerCaFile, // opt-resolve-order cfgOptTlsServerCertFile, // opt-resolve-order cfgOptTlsServerKeyFile, // opt-resolve-order cfgOptTlsServerPort, // opt-resolve-order cfgOptType, // opt-resolve-order cfgOptVerbose, // opt-resolve-order cfgOptVersion, // opt-resolve-order cfgOptArchiveCheck, // opt-resolve-order cfgOptArchiveCopy, // opt-resolve-order cfgOptArchiveModeCheck, // opt-resolve-order cfgOptCompressLevel, // opt-resolve-order cfgOptForce, // opt-resolve-order cfgOptPgDatabase, // opt-resolve-order cfgOptPgHost, // opt-resolve-order cfgOptPgHostConfig, // opt-resolve-order cfgOptPgHostConfigIncludePath, // opt-resolve-order cfgOptPgHostConfigPath, // opt-resolve-order cfgOptPgHostType, // opt-resolve-order cfgOptPgHostUser, // opt-resolve-order cfgOptRecoveryOption, // opt-resolve-order cfgOptRepoAzureAccount, // opt-resolve-order cfgOptRepoAzureContainer, // opt-resolve-order cfgOptRepoAzureEndpoint, // opt-resolve-order cfgOptRepoAzureKeyType, // opt-resolve-order cfgOptRepoAzureUriStyle, // opt-resolve-order cfgOptRepoBlock, // opt-resolve-order cfgOptRepoBlockAgeMap, // opt-resolve-order cfgOptRepoBlockChecksumSizeMap, // opt-resolve-order cfgOptRepoBlockSizeMap, // opt-resolve-order cfgOptRepoBlockSizeSuper, // opt-resolve-order cfgOptRepoBlockSizeSuperFull, // opt-resolve-order cfgOptRepoCipherPass, // opt-resolve-order cfgOptRepoGcsKeyType, // opt-resolve-order cfgOptRepoHost, // opt-resolve-order cfgOptRepoHostConfig, // opt-resolve-order cfgOptRepoHostConfigIncludePath, // opt-resolve-order cfgOptRepoHostConfigPath, // opt-resolve-order cfgOptRepoHostType, // opt-resolve-order cfgOptRepoHostUser, // opt-resolve-order cfgOptRepoS3Bucket, // opt-resolve-order cfgOptRepoS3Endpoint, // opt-resolve-order cfgOptRepoS3KeyType, // opt-resolve-order cfgOptRepoS3KmsKeyId, // opt-resolve-order cfgOptRepoS3ProcessCmd, // opt-resolve-order cfgOptRepoS3Region, // opt-resolve-order cfgOptRepoS3RequesterPays, // opt-resolve-order cfgOptRepoS3Role, // opt-resolve-order cfgOptRepoS3Service, // opt-resolve-order cfgOptRepoS3SseCustomerKey, // opt-resolve-order cfgOptRepoS3StsHost, // opt-resolve-order cfgOptRepoS3Token, // opt-resolve-order cfgOptRepoS3UriStyle, // opt-resolve-order cfgOptRepoSftpHost, // opt-resolve-order cfgOptRepoSftpHostKeyCheckType, // opt-resolve-order cfgOptRepoSftpHostKeyHashType, // opt-resolve-order cfgOptRepoSftpHostPort, // opt-resolve-order cfgOptRepoSftpHostUser, // opt-resolve-order cfgOptRepoSftpKnownHost, // opt-resolve-order cfgOptRepoSftpPrivateKeyFile, // opt-resolve-order cfgOptRepoSftpPrivateKeyPassphrase, // opt-resolve-order cfgOptRepoSftpPublicKeyFile, // opt-resolve-order cfgOptRepoStorageCaFile, // opt-resolve-order cfgOptRepoStorageCaPath, // opt-resolve-order cfgOptRepoStorageHost, // opt-resolve-order cfgOptRepoStoragePort, // opt-resolve-order cfgOptRepoStorageTag, // opt-resolve-order cfgOptRepoStorageUploadChunkSize, // opt-resolve-order cfgOptRepoStorageVerifyTls, // opt-resolve-order cfgOptRepoSymlink, // opt-resolve-order cfgOptTarget, // opt-resolve-order cfgOptTargetAction, // opt-resolve-order cfgOptTargetExclusive, // opt-resolve-order cfgOptTargetTimeline, // opt-resolve-order cfgOptPgHostCaFile, // opt-resolve-order cfgOptPgHostCaPath, // opt-resolve-order cfgOptPgHostCertFile, // opt-resolve-order cfgOptPgHostCmd, // opt-resolve-order cfgOptPgHostKeyFile, // opt-resolve-order cfgOptPgHostPort, // opt-resolve-order cfgOptRepoAzureKey, // opt-resolve-order cfgOptRepoGcsBucket, // opt-resolve-order cfgOptRepoGcsEndpoint, // opt-resolve-order cfgOptRepoGcsKey, // opt-resolve-order cfgOptRepoHostCaFile, // opt-resolve-order cfgOptRepoHostCaPath, // opt-resolve-order cfgOptRepoHostCertFile, // opt-resolve-order cfgOptRepoHostCmd, // opt-resolve-order cfgOptRepoHostKeyFile, // opt-resolve-order cfgOptRepoHostPort, // opt-resolve-order cfgOptRepoS3Key, // opt-resolve-order cfgOptRepoS3KeySecret, // opt-resolve-order cfgOptRepoSftpHostFingerprint, // opt-resolve-order }; ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/parse.c0000664000175100017510000040751215240547630016736 0ustar00runnerrunner/*********************************************************************************************************************************** Command and Option Parse ***********************************************************************************************************************************/ #include #include #include #include #include #include #include "common/debug.h" #include "common/io/bufferRead.h" #include "common/log.h" #include "common/macro.h" #include "common/memContext.h" #include "common/regExp.h" #include "config/common.h" #include "config/config.intern.h" #include "config/parse.h" #include "version.h" /*********************************************************************************************************************************** The maximum number of keys that an indexed option can have, e.g. pg256-path would be the maximum pg-path option ***********************************************************************************************************************************/ #define CFG_OPTION_KEY_MAX 256 /*********************************************************************************************************************************** Section enum - defines which sections of the config an option can appear in ***********************************************************************************************************************************/ typedef enum { cfgSectionCommandLine, // Command-line only cfgSectionGlobal, // Command-line or in any config section cfgSectionStanza, // Command-line or in any config stanza section } ConfigSection; /*********************************************************************************************************************************** Default type enum ***********************************************************************************************************************************/ typedef enum { cfgDefaultTypeStatic, // A stored static string cfgDefaultTypeDynamic, // Determined at runtime } ConfigDefaultType; /*********************************************************************************************************************************** Standard config file name and old default path and name ***********************************************************************************************************************************/ #define PGBACKREST_CONFIG_ORIG_PATH_FILE "/etc/" PROJECT_CONFIG_FILE STRING_STATIC(PGBACKREST_CONFIG_ORIG_PATH_FILE_STR, PGBACKREST_CONFIG_ORIG_PATH_FILE); /*********************************************************************************************************************************** Mem context and local variables ***********************************************************************************************************************************/ static struct ConfigParseLocal { MemContext *memContext; // Mem context Ini *ini; // Parsed ini data } configParseLocal; /*********************************************************************************************************************************** Define how a command is parsed ***********************************************************************************************************************************/ typedef struct ParseRuleCommand { const char *name; // Name unsigned int commandRoleValid : CFG_COMMAND_ROLE_TOTAL; // Valid for the command role? bool lockRequired : 1; // Is an immediate lock required? bool lockRemoteRequired : 1; // Is a lock required on the remote? unsigned int lockType : 3; // Lock type required bool logFile : 1; // Will the command log to a file? unsigned int logLevelDefault : 4; // Default log level bool parameterAllowed : 1; // Command-line parameters are allowed } ParseRuleCommand; // Macros used to define parse rules in parse.auto.c.inc in a format that diffs well #define PARSE_RULE_COMMAND(...) \ {__VA_ARGS__} #define PARSE_RULE_COMMAND_NAME(nameParam) \ .name = nameParam #define PARSE_RULE_COMMAND_ROLE_VALID_LIST(...) \ .commandRoleValid = 0 __VA_ARGS__ #define PARSE_RULE_COMMAND_ROLE(commandRoleParam) \ | (1 << cfgCmdRole##commandRoleParam) #define PARSE_RULE_COMMAND_INTERNAL(internalParam) #define PARSE_RULE_COMMAND_LOCK_REQUIRED(lockRequiredParam) \ .lockRequired = lockRequiredParam #define PARSE_RULE_COMMAND_LOCK_REMOTE_REQUIRED(lockRemoteRequiredParam) \ .lockRemoteRequired = lockRemoteRequiredParam #define PARSE_RULE_COMMAND_LOCK_TYPE(lockTypeParam) \ .lockType = lockType##lockTypeParam #define PARSE_RULE_COMMAND_LOG_FILE(logFileParam) \ .logFile = logFileParam #define PARSE_RULE_COMMAND_LOG_LEVEL_DEFAULT(logLevelDefaultParam) \ .logLevelDefault = logLevel##logLevelDefaultParam #define PARSE_RULE_COMMAND_PARAMETER_ALLOWED(parameterAllowedParam) \ .parameterAllowed = parameterAllowedParam /*********************************************************************************************************************************** Define how an option group is parsed ***********************************************************************************************************************************/ typedef struct ParseRuleOptionGroup { const char *name; // All options in the group must be prefixed with this name } ParseRuleOptionGroup; // Macros used to define parse rules in parse.auto.c.inc in a format that diffs well #define PARSE_RULE_OPTION_GROUP(...) \ {__VA_ARGS__} #define PARSE_RULE_OPTION_GROUP_NAME(nameParam) \ .name = nameParam /*********************************************************************************************************************************** Define how an option is parsed and interacts with other options ***********************************************************************************************************************************/ typedef struct ParseRuleOption { const char *name; // Name unsigned int type : 4; // e.g. string, int, boolean unsigned int defaultType : 1; // e.g. static, dynamic bool boolLike : 1; // Option accepts y/n and can be treated as bool? bool beta : 1; // Is the option a beta feature? bool negate : 1; // Can the option be negated on the command line? bool reset : 1; // Can the option be reset on the command line? bool required : 1; // Is the option required? bool sequence : 1; // StringId contains sequence? unsigned int section : 2; // e.g. global, stanza, cmd-line bool secure : 1; // Needs to be redacted in logs and cmd-line? bool multi : 1; // Can be specified multiple times? bool group : 1; // In a group? unsigned int groupId : 1; // Id if in a group bool deprecateMatch : 1; // Does a deprecated name exactly match the option name? unsigned int packSize : 7; // Size of optional data in pack format uint32_t commandRoleValid[CFG_COMMAND_ROLE_TOTAL]; // Valid for the command role? const uint8_t *pack; // Optional data in pack format } ParseRuleOption; // Define additional types of data that can be associated with an option. Because these types are rare they are not given dedicated // fields and are instead packed and read at runtime. This may seem inefficient but they are only accessed a single time during // parse so space efficiency is more important than performance. typedef enum { parseRuleOptionalTypeValid = 1, parseRuleOptionalTypeSequence, parseRuleOptionalTypeAllowRange, parseRuleOptionalTypeAllowList, parseRuleOptionalTypeDefault, parseRuleOptionalTypeRequired, } ParseRuleOptionalType; // Optional rule filter types typedef enum { parseRuleFilterTypeCommand = 1, } ParseRuleFilterType; // Macros used to define parse rules in parse.auto.c.inc in a format that diffs well #define PARSE_RULE_OPTION(...) \ {__VA_ARGS__} #define PARSE_RULE_OPTION_NAME(nameParam) \ .name = nameParam #define PARSE_RULE_OPTION_TYPE(typeParam) \ .type = cfgOptType##typeParam #define PARSE_RULE_OPTION_INTERNAL(typeParam) #define PARSE_RULE_OPTION_COMMAND_INTERNAL_LIST(...) #define PARSE_RULE_OPTION_DEFAULT_TYPE(defaultTypeParam) \ .defaultType = cfgDefaultType##defaultTypeParam #define PARSE_RULE_OPTION_BOOL_LIKE(boolLikeParam) \ .boolLike = boolLikeParam #define PARSE_RULE_OPTION_BETA(betaParam) \ .beta = betaParam #define PARSE_RULE_OPTION_NEGATE(negateParam) \ .negate = negateParam #define PARSE_RULE_OPTION_RESET(resetParam) \ .reset = resetParam #define PARSE_RULE_OPTION_REQUIRED(requiredParam) \ .required = requiredParam #define PARSE_RULE_OPTION_SECTION(sectionParam) \ .section = cfgSection##sectionParam #define PARSE_RULE_OPTION_SECURE(secureParam) \ .secure = secureParam #define PARSE_RULE_OPTION_SEQUENCE(sequenceParam) \ .sequence = sequenceParam #define PARSE_RULE_OPTION_MULTI(typeMulti) \ .multi = typeMulti #define PARSE_RULE_OPTION_GROUP_ID(groupIdParam) \ .group = true, .groupId = cfgOptGrp##groupIdParam #define PARSE_RULE_OPTION_DEPRECATE_MATCH(deprecateMatchParam) \ .deprecateMatch = deprecateMatchParam #define PARSE_RULE_OPTION_COMMAND_ROLE_MAIN_VALID_LIST(...) \ .commandRoleValid[cfgCmdRoleMain] = 0 __VA_ARGS__ #define PARSE_RULE_OPTION_COMMAND_ROLE_ASYNC_VALID_LIST(...) \ .commandRoleValid[cfgCmdRoleAsync] = 0 __VA_ARGS__ #define PARSE_RULE_OPTION_COMMAND_ROLE_LOCAL_VALID_LIST(...) \ .commandRoleValid[cfgCmdRoleLocal] = 0 __VA_ARGS__ #define PARSE_RULE_OPTION_COMMAND_ROLE_REMOTE_VALID_LIST(...) \ .commandRoleValid[cfgCmdRoleRemote] = 0 __VA_ARGS__ #define PARSE_RULE_OPTION_COMMAND(commandParam) \ | (1 << cfgCmd##commandParam) #define PARSE_RULE_STRPUB(value) {.buffer = (const char *)value, .size = sizeof(value) - 1} // Macros used to define optional parse rules in pack format #define PARSE_RULE_VARINT_01(value) \ value #define PARSE_RULE_VARINT_02(value) \ 0x80 | (value & 0x7f), (value >> 7) & 0x7f #define PARSE_RULE_BOOL_TRUE 0x28 #define PARSE_RULE_BOOL_FALSE 0x20 #define PARSE_RULE_U32_1(value) 0x88, PARSE_RULE_VARINT_01(value) #define PARSE_RULE_U32_2(value) 0x88, PARSE_RULE_VARINT_02(value) #define PARSE_RULE_PACK(...) __VA_ARGS__ 0x00 #define PARSE_RULE_PACK_SIZE(...) \ 0xf0, 0x02, sizeof((const uint8_t []){PARSE_RULE_PACK(__VA_ARGS__)}), \ PARSE_RULE_PACK(__VA_ARGS__) #define PARSE_RULE_VAL_BOOL_TRUE PARSE_RULE_BOOL_TRUE #define PARSE_RULE_VAL_BOOL_FALSE PARSE_RULE_BOOL_FALSE #define PARSE_RULE_VAL_STR_IDX(valueIdx) ((const String *)&parseRuleValueStr[valueIdx]) #define PARSE_RULE_VAL_SEQ(value) PARSE_RULE_U32_1(value) #define PARSE_RULE_OPTIONAL(...) \ .packSize = sizeof((const uint8_t []){PARSE_RULE_PACK(__VA_ARGS__)}), \ .pack = (const uint8_t []){PARSE_RULE_PACK(__VA_ARGS__)} #define PARSE_RULE_OPTIONAL_GROUP(...) PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_FILTER_CMD(...) \ PARSE_RULE_PACK_SIZE(PARSE_RULE_U32_1(parseRuleFilterTypeCommand), __VA_ARGS__) #define PARSE_RULE_OPTIONAL_DEPEND(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeValid), PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_SEQUENCE(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeSequence) #define PARSE_RULE_OPTIONAL_NO_SEQUENCE(...) PARSE_RULE_OPTIONAL_SEQUENCE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_DEPEND_DEFAULT(value) PARSE_RULE_BOOL_TRUE, value #define PARSE_RULE_OPTIONAL_ALLOW_LIST(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeAllowList), PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_ALLOW_RANGE(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeAllowRange), PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_ALLOW_RANGE_MAP(...) \ 0x10, __VA_ARGS__ 0x00 #define PARSE_RULE_OPTIONAL_DEFAULT(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeDefault), PARSE_RULE_PACK_SIZE(__VA_ARGS__) #define PARSE_RULE_OPTIONAL_DEFAULT_MAP(...) \ 0x10, __VA_ARGS__ 0x00 #define PARSE_RULE_OPTIONAL_REQUIRED(...) \ PARSE_RULE_U32_1(parseRuleOptionalTypeRequired) #define PARSE_RULE_OPTIONAL_NOT_REQUIRED(...) PARSE_RULE_OPTIONAL_REQUIRED(__VA_ARGS__) /*********************************************************************************************************************************** Define option deprecations ***********************************************************************************************************************************/ typedef struct ParseRuleOptionDeprecate { const char *name; // Deprecated name ConfigOption id; // Option Id bool indexed; // Can the deprecation be indexed? bool unindexed; // Can the deprecation be unindexed? } ParseRuleOptionDeprecate; /*********************************************************************************************************************************** Include automatically generated parse data ***********************************************************************************************************************************/ #include "config/parse.auto.c.inc" /*********************************************************************************************************************************** Struct to hold options parsed from the command line ***********************************************************************************************************************************/ typedef struct ParseOptionValue { bool found : 1; // Was the option found? bool negate : 1; // Was the option negated on the command line? bool reset : 1; // Was the option reset on the command line? unsigned int source : 2; // Where was the option found? StringList *valueList; // List of values found } ParseOptionValue; typedef struct ParseOption { unsigned int indexListTotal; // Total options in indexed list ParseOptionValue *indexList; // List of indexed option values } ParseOption; #define FUNCTION_LOG_PARSE_OPTION_FORMAT(value, buffer, bufferSize) \ typeToLog("ParseOption", buffer, bufferSize) /*********************************************************************************************************************************** Get the indexed value, creating the array to contain it if needed ***********************************************************************************************************************************/ static ParseOptionValue * parseOptionIdxValue(ParseOption *const optionList, const unsigned int optionId, const unsigned int optionKeyIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(PARSE_OPTION, optionList); // Structure containing all options being parsed FUNCTION_TEST_PARAM(UINT, optionId); // Unique ID which also identifies the option in the parse list FUNCTION_TEST_PARAM(UINT, optionKeyIdx); // Zero-based key index (e.g. pg3-path => 2), 0 for non-indexed FUNCTION_TEST_END(); // If the requested index is beyond what has already been allocated if (optionKeyIdx >= optionList[optionId].indexListTotal) { // If the option is in a group if (parseRuleOption[optionId].group) { unsigned int optionOffset = 0; // Allocate enough memory to include the requested indexed or a fixed amount to avoid too many allocations if (optionList[optionId].indexListTotal == 0) { optionList[optionId].indexListTotal = optionKeyIdx >= (LIST_INITIAL_SIZE / 2) ? optionKeyIdx + 1 : (LIST_INITIAL_SIZE / 2); optionList[optionId].indexList = memNew(sizeof(ParseOptionValue) * optionList[optionId].indexListTotal); } // Allocate more memory when needed. This could be more efficient but the limited number of indexes currently allowed // makes it difficult to get coverage on a better implementation. else { optionOffset = optionList[optionId].indexListTotal; optionList[optionId].indexListTotal = optionKeyIdx + 1; optionList[optionId].indexList = memResize( optionList[optionId].indexList, sizeof(ParseOptionValue) * optionList[optionId].indexListTotal); } // Initialize the newly allocated memory for (unsigned int optKeyIdx = optionOffset; optKeyIdx < optionList[optionId].indexListTotal; optKeyIdx++) optionList[optionId].indexList[optKeyIdx] = (ParseOptionValue){0}; } // Else the option is not in a group so there can only be one value else { optionList[optionId].indexList = memNew(sizeof(ParseOptionValue)); optionList[optionId].indexListTotal = 1; optionList[optionId].indexList[0] = (ParseOptionValue){0}; } } // Return the indexed value FUNCTION_TEST_RETURN_TYPE_P(ParseOptionValue, &optionList[optionId].indexList[optionKeyIdx]); } /*********************************************************************************************************************************** Check that --beta is set if a beta option is used ***********************************************************************************************************************************/ static void parseOptionBeta( const unsigned int optionId, const unsigned int optionKeyIdx, const bool beta, const ParseOption *const parseOptionBeta) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, optionId); FUNCTION_TEST_PARAM(UINT, optionKeyIdx); FUNCTION_TEST_PARAM(BOOL, beta); FUNCTION_TEST_PARAM_P(PARSE_OPTION, parseOptionBeta); FUNCTION_TEST_END(); if (beta && optionId != cfgOptBeta && (parseOptionBeta->indexListTotal == 0 || parseOptionBeta->indexList[0].negate)) { THROW_FMT( OptionInvalidError, "option '%s' is not valid without option '" CFGOPT_BETA "'\n" "HINT: beta features require the --" CFGOPT_BETA " option to prevent accidental usage.", cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get command id by name ***********************************************************************************************************************************/ static ConfigCommand cfgParseCommandId(const char *const commandName) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, commandName); FUNCTION_TEST_END(); ASSERT(commandName != NULL); // If command has a colon only search part before the colon const char *const colonPtr = strchr(commandName, ':'); const size_t commandSize = colonPtr == NULL ? (size_t)strlen(commandName) : (size_t)(colonPtr - commandName); // Get command ConfigCommand commandId; for (commandId = 0; commandId < CFG_COMMAND_TOTAL; commandId++) { if (strncmp(commandName, parseRuleCommand[commandId].name, commandSize) == 0) break; } if (commandId == CFG_COMMAND_TOTAL) THROW_FMT(CommandInvalidError, "invalid command '%s'", commandName); FUNCTION_TEST_RETURN(ENUM, commandId); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgParseCommandName(const ConfigCommand commandId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_END(); ASSERT(commandId < CFG_COMMAND_TOTAL); FUNCTION_TEST_RETURN_CONST(STRINGZ, parseRuleCommand[commandId].name); } /*********************************************************************************************************************************** Convert command role from String to enum and vice versa ***********************************************************************************************************************************/ STRING_STATIC(CONFIG_COMMAND_ROLE_ASYNC_STR, CONFIG_COMMAND_ROLE_ASYNC); STRING_STATIC(CONFIG_COMMAND_ROLE_LOCAL_STR, CONFIG_COMMAND_ROLE_LOCAL); STRING_STATIC(CONFIG_COMMAND_ROLE_REMOTE_STR, CONFIG_COMMAND_ROLE_REMOTE); static ConfigCommandRole cfgParseCommandRoleEnum(const char *const commandRole) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRINGZ, commandRole); FUNCTION_TEST_END(); if (commandRole == NULL) FUNCTION_TEST_RETURN(ENUM, cfgCmdRoleMain); else if (strEqZ(CONFIG_COMMAND_ROLE_ASYNC_STR, commandRole)) FUNCTION_TEST_RETURN(ENUM, cfgCmdRoleAsync); else if (strEqZ(CONFIG_COMMAND_ROLE_LOCAL_STR, commandRole)) FUNCTION_TEST_RETURN(ENUM, cfgCmdRoleLocal); else if (strEqZ(CONFIG_COMMAND_ROLE_REMOTE_STR, commandRole)) FUNCTION_TEST_RETURN(ENUM, cfgCmdRoleRemote); THROW_FMT(CommandInvalidError, "invalid command role '%s'", commandRole); } FN_EXTERN const String * cfgParseCommandRoleStr(const ConfigCommandRole commandRole) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandRole); FUNCTION_TEST_END(); const String *result = NULL; switch (commandRole) { case cfgCmdRoleMain: break; case cfgCmdRoleAsync: result = CONFIG_COMMAND_ROLE_ASYNC_STR; break; case cfgCmdRoleLocal: result = CONFIG_COMMAND_ROLE_LOCAL_STR; break; case cfgCmdRoleRemote: result = CONFIG_COMMAND_ROLE_REMOTE_STR; break; } FUNCTION_TEST_RETURN_CONST(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN String * cfgParseCommandRoleName(const ConfigCommand commandId, const ConfigCommandRole commandRoleId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_PARAM(ENUM, commandRoleId); FUNCTION_TEST_END(); String *const result = strCatZ(strNew(), cfgParseCommandName(commandId)); if (commandRoleId != cfgCmdRoleMain) strCatFmt(result, ":%s", strZ(cfgParseCommandRoleStr(commandRoleId))); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN const Ini * cfgParseIni(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(INI, configParseLocal.ini); } /**********************************************************************************************************************************/ FN_EXTERN StringList * cfgParseStanzaList(void) { FUNCTION_TEST_VOID(); StringList *const result = strLstNew(); if (configParseLocal.ini != NULL) { MEM_CONTEXT_TEMP_BEGIN() { const StringList *const sectionList = strLstSort(iniSectionList(configParseLocal.ini), sortOrderAsc); for (unsigned int sectionIdx = 0; sectionIdx < strLstSize(sectionList); sectionIdx++) { const String *const section = strLstGet(sectionList, sectionIdx); // Skip global sections if (strEqZ(section, CFGDEF_SECTION_GLOBAL) || strBeginsWithZ(section, CFGDEF_SECTION_GLOBAL ":")) continue; // Extract stanza const StringList *const sectionPart = strLstNewSplitZ(section, ":"); ASSERT(strLstSize(sectionPart) <= 2); strLstAddIfMissing(result, strLstGet(sectionPart, 0)); } } MEM_CONTEXT_TEMP_END(); } strLstSort(result, sortOrderAsc); FUNCTION_TEST_RETURN(STRING_LIST, result); } /*********************************************************************************************************************************** Find an option by name in the option list ***********************************************************************************************************************************/ #define OPTION_PREFIX_NEGATE "no-" #define OPTION_PREFIX_RESET "reset-" #define OPTION_NAME_SIZE_MAX 64 FN_EXTERN CfgParseOptionResult cfgParseOption(const String *const optionCandidate, const CfgParseOptionParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, optionCandidate); FUNCTION_TEST_PARAM(BOOL, param.prefixMatch); FUNCTION_TEST_PARAM(BOOL, param.ignoreMissingIndex); FUNCTION_TEST_END(); ASSERT(optionCandidate != NULL); CfgParseOptionResult result = {0}; // Copy the option to a buffer so it can be efficiently manipulated char optionName[OPTION_NAME_SIZE_MAX + 1]; size_t optionNameSize = strSize(optionCandidate); if (optionNameSize > sizeof(optionName) - 1) { THROW_FMT( OptionInvalidError, "option '%s' exceeds maximum size of " STRINGIFY(OPTION_NAME_SIZE_MAX), strZ(optionCandidate)); } memcpy(optionName, strZ(optionCandidate), optionNameSize + 1); // If this looks like negate if (strncmp(optionName, OPTION_PREFIX_NEGATE, sizeof(OPTION_PREFIX_NEGATE) - 1) == 0) { result.negate = true; // Strip the negate prefix optionNameSize -= sizeof(OPTION_PREFIX_NEGATE) - 1; memmove(optionName, optionName + (sizeof(OPTION_PREFIX_NEGATE) - 1), optionNameSize + 1); } // Else if looks like reset else if (strncmp(optionName, OPTION_PREFIX_RESET, sizeof(OPTION_PREFIX_RESET) - 1) == 0) { result.reset = true; // Strip the reset prefix optionNameSize -= sizeof(OPTION_PREFIX_RESET) - 1; memmove(optionName, optionName + (sizeof(OPTION_PREFIX_RESET) - 1), optionNameSize + 1); } // Indexed options must have at least one dash char *const dashPtr = strchr(optionName, '-'); bool indexed = false; if (dashPtr != NULL) { if (dashPtr == optionName) THROW_FMT(OptionInvalidError, "option '%s' cannot begin with a dash", strZ(optionCandidate)); // Check if the first dash is preceded by a numeric key and keep a tally of the key char *numberPtr = dashPtr; unsigned int multiplier = 1; while (numberPtr > optionName && isdigit(*(numberPtr - 1))) { numberPtr--; result.keyIdx += (unsigned int)(*numberPtr - '0') * multiplier; multiplier *= 10; } if (numberPtr == optionName) THROW_FMT(OptionInvalidError, "option '%s' cannot begin with a number", strZ(optionCandidate)); // If there was a number then the option is indexed if (numberPtr != dashPtr) { indexed = true; // Strip the key to get the base option name optionNameSize -= (size_t)(dashPtr - numberPtr); memmove(numberPtr, dashPtr, optionNameSize + 1); // Check that the index is within bounds if (result.keyIdx < 1 || result.keyIdx > CFG_OPTION_KEY_MAX) { THROW_FMT( OptionInvalidError, "option '%s' key must be between 1 and " STRINGIFY(CFG_OPTION_KEY_MAX), strZ(optionCandidate)); } // Subtract one to represent a key index result.keyIdx--; } } // Search for an exact match. A copy of the option name must be made because bsearch() requires a reference. const char *const optionNamePtr = optionName; const ParseRuleOption *optionFound = bsearch( &optionNamePtr, parseRuleOption, CFG_OPTION_TOTAL, sizeof(ParseRuleOption), lstComparatorZ); // If the option was not found if (optionFound == NULL) { // Search for a single partial match (if requested) if (param.prefixMatch) { unsigned int findPartialIdx = 0; unsigned int findPartialTotal = 0; for (unsigned int findIdx = 0; findIdx < CFG_OPTION_TOTAL; findIdx++) { if (strncmp(parseRuleOption[findIdx].name, optionName, optionNameSize) == 0) { findPartialIdx = findIdx; findPartialTotal++; if (findPartialTotal > 1) break; } } // If a single partial match was found if (findPartialTotal == 1) optionFound = &parseRuleOption[findPartialIdx]; } // If the option was not found search deprecations if (optionFound == NULL) { // Search deprecations for an exact match const ParseRuleOptionDeprecate *deprecate = bsearch( &optionNamePtr, parseRuleOptionDeprecate, CFG_OPTION_DEPRECATE_TOTAL, sizeof(ParseRuleOptionDeprecate), lstComparatorZ); // If the option was not found then search deprecations for a single partial match (if requested) if (deprecate == NULL && param.prefixMatch) { unsigned int findPartialIdx = 0; unsigned int findPartialTotal = 0; for (unsigned int deprecateIdx = 0; deprecateIdx < CFG_OPTION_DEPRECATE_TOTAL; deprecateIdx++) { if (strncmp(parseRuleOptionDeprecate[deprecateIdx].name, optionName, optionNameSize) == 0) { findPartialIdx = deprecateIdx; findPartialTotal++; if (findPartialTotal > 1) break; } } // If a single partial match was found if (findPartialTotal == 1) deprecate = &parseRuleOptionDeprecate[findPartialIdx]; } // Deprecation was found if (deprecate != NULL) { // Error if the option is indexed but the deprecation is not if (indexed && !deprecate->indexed) THROW_FMT(OptionInvalidError, "deprecated option '%s' cannot have an index", strZ(optionCandidate)); // Error if the option is unindexed but the deprecation is not optionFound = &parseRuleOption[deprecate->id]; if (!indexed && !deprecate->unindexed && !param.ignoreMissingIndex) { ASSERT(dashPtr != NULL); const char *const groupName = parseRuleOptionGroup[optionFound->groupId].name; THROW_FMT( OptionInvalidError, "deprecated option '%s' requires an index\n" "HINT: add the required index, e.g. %.*s1%s.\n" "HINT: consider using the non-deprecated name, e.g. %s1%s.", strZ(optionCandidate), (int)(dashPtr - optionName), optionName, dashPtr, groupName, optionFound->name + strlen(groupName)); } result.deprecated = true; } } } // Option was found if (optionFound != NULL) { result.found = true; result.id = (unsigned int)(optionFound - parseRuleOption); // Error if negate is not allowed if (result.negate && !optionFound->negate) THROW_FMT(OptionInvalidError, "option '%s' cannot be negated", strZ(optionCandidate)); // Error if reset is not allowed if (result.reset && !optionFound->reset) THROW_FMT(OptionInvalidError, "option '%s' cannot be reset", strZ(optionCandidate)); // It is possible for an unindexed deprecation to match an indexed option name (without the index) exactly. For example, the // deprecated repo-path option now maps to repo-path index 0, which will yield an exact match. In this case we still need to // mark the option as deprecated. if (indexed == false && optionFound->deprecateMatch) result.deprecated = true; // If not deprecated make sure indexing matches. Deprecation indexing has already been checked because the rules are per // deprecation. if (!result.deprecated) { // Error if the option is indexed but should not be if (indexed && !optionFound->group) THROW_FMT(OptionInvalidError, "option '%s' cannot have an index", strZ(optionCandidate)); // Error if the option is unindexed but an index is required if (!indexed && optionFound->group && !param.ignoreMissingIndex) { const char *const groupName = parseRuleOptionGroup[optionFound->groupId].name; THROW_FMT( OptionInvalidError, "option '%s' requires an index\n" "HINT: add the required index, e.g. %s1%s.", strZ(optionCandidate), groupName, optionFound->name + strlen(groupName)); } } // Set flags result.beta = optionFound->beta; result.multi = optionFound->multi; FUNCTION_TEST_RETURN_TYPE(CfgParseOptionResult, result); } FUNCTION_TEST_RETURN_TYPE(CfgParseOptionResult, (CfgParseOptionResult){0}); } /**********************************************************************************************************************************/ FN_EXTERN ConfigOptionDataType cfgParseOptionDataType(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); switch (parseRuleOption[optionId].type) { case cfgOptTypeBoolean: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeBoolean); case cfgOptTypeHash: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeHash); case cfgOptTypeInteger: case cfgOptTypeSize: case cfgOptTypeTime: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeInteger); case cfgOptTypeList: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeList); case cfgOptTypeStringId: FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeStringId); default: break; } ASSERT(parseRuleOption[optionId].type == cfgOptTypePath || parseRuleOption[optionId].type == cfgOptTypeString); FUNCTION_TEST_RETURN(ENUM, cfgOptDataTypeString); } /*********************************************************************************************************************************** Get a value string by index ***********************************************************************************************************************************/ static const String * cfgParseOptionValueStr(const ConfigOptionType type, unsigned int valueIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(UINT, valueIdx); FUNCTION_TEST_END(); switch (type) { case cfgOptTypeInteger: valueIdx = parseRuleValueIntStrMap[valueIdx]; break; case cfgOptTypeSize: valueIdx = parseRuleValueSizeStrMap[valueIdx]; break; case cfgOptTypeTime: valueIdx = parseRuleValueTimeStrMap[valueIdx]; break; default: ASSERT(type == cfgOptTypePath || type == cfgOptTypeString || type == cfgOptTypeStringId); break; } FUNCTION_TEST_RETURN_CONST(STRING, PARSE_RULE_VAL_STR_IDX(valueIdx)); } /*********************************************************************************************************************************** Get an integer value by index ***********************************************************************************************************************************/ static int64_t cfgParseOptionValue(const ConfigOptionType type, unsigned int valueIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM(UINT, valueIdx); FUNCTION_TEST_END(); switch (type) { case cfgOptTypeInteger: FUNCTION_TEST_RETURN(INT64, parseRuleValueInt[valueIdx]); case cfgOptTypeSize: FUNCTION_TEST_RETURN(INT64, parseRuleValueSize[valueIdx]); default: ASSERT(type == cfgOptTypeTime); break; } FUNCTION_TEST_RETURN(INT64, parseRuleValueTime[valueIdx]); } /*********************************************************************************************************************************** Get default value from pack ***********************************************************************************************************************************/ static void cfgParseOptionValuePack( PackRead *const ruleData, const ConfigOptionType type, ConfigOptionValueType *value, const String **raw) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, ruleData); FUNCTION_TEST_PARAM(ENUM, type); FUNCTION_TEST_PARAM_P(VOID, value); FUNCTION_TEST_PARAM_PP(VOID, raw); FUNCTION_TEST_END(); switch (pckReadType(ruleData)) { case pckTypeBool: { value->boolean = pckReadBoolP(ruleData); if (raw != NULL) *raw = value->boolean ? Y_STR : N_STR; break; } default: { const unsigned int valueIdx = pckReadU32P(ruleData); switch (type) { case cfgOptTypeInteger: case cfgOptTypeSize: case cfgOptTypeTime: value->integer = cfgParseOptionValue(type, valueIdx); break; case cfgOptTypePath: case cfgOptTypeString: value->string = PARSE_RULE_VAL_STR_IDX(valueIdx); break; default: ASSERT(type == cfgOptTypeStringId); break; } if (raw != NULL) *raw = cfgParseOptionValueStr(type, valueIdx); } } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Find an optional rule ***********************************************************************************************************************************/ typedef struct CfgParseOptionalRuleState { PackRead *pack; unsigned int typeNext; bool done; // Valid const uint8_t *valid; size_t validSize; unsigned int matchValue; // Sequence bool sequence; // Allow range int64_t allowRangeMin; int64_t allowRangeMax; unsigned int allowRangeMinIdx; unsigned int allowRangeMaxIdx; bool allowRangeMatch; // Allow list const uint8_t *allowList; size_t allowListSize; // Default const String *const defaultDynamicBin; // Binary for dynamic default const String *defaultRaw; ConfigOptionValueType defaultValue; uint8_t defaultSequence; // Sequence for the default // Required bool required; } CfgParseOptionalRuleState; static bool cfgParseOptionalRule( CfgParseOptionalRuleState *const optionalRules, const ParseRuleOptionalType optionalRuleType, const ConfigCommand commandId, const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, optionalRules); FUNCTION_TEST_PARAM(ENUM, optionalRuleType); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(optionalRuleType != 0); ASSERT(commandId < CFG_COMMAND_TOTAL); ASSERT(optionId < CFG_OPTION_TOTAL); bool result = false; bool match = false; // Check for optional rules const ParseRuleOption *const ruleOption = &parseRuleOption[optionId]; if (!optionalRules->done && ruleOption->pack != NULL) { // Initialize optional rules if (optionalRules->pack == NULL) { MEM_CONTEXT_TEMP_BEGIN() { PackRead *const groupList = pckReadNewC(ruleOption->pack, ruleOption->packSize); // Search for a matching group do { // Get the group pack PackRead *group = pckReadPackReadConstP(groupList); // Process filters if any pckReadNext(group); if (pckReadType(group) == pckTypePack) { // Check for filter match bool match = false; PackRead *const filter = pckReadPackReadConstP(group); const ParseRuleFilterType filterType = (ParseRuleFilterType)pckReadU32P(filter); switch (filterType) { default: { ASSERT(filterType == parseRuleFilterTypeCommand); while (pckReadNext(filter)) { if ((ConfigCommand)pckReadU32P(filter) == commandId) { match = true; break; } } break; } } // Filter did not match if (!match) group = NULL; } // If the group matched if (group != NULL) { // Get first optional rule type. This is needed since pckReadNext() has already been called and it cannot // be called again until data is read. optionalRules->typeNext = pckReadU32P(group); ASSERT(optionalRules->typeNext != 0); // Move group to prior context and stop searching optionalRules->pack = pckReadMove(group, memContextPrior()); break; } } while (pckReadNext(groupList)); } MEM_CONTEXT_TEMP_END(); // If no group matched then done if (optionalRules->pack == NULL) { optionalRules->done = true; FUNCTION_TEST_RETURN(BOOL, false); } } // Search for the specified optional rule do { // Read the next optional rule type if it has not already been read if (optionalRules->typeNext == 0) { // If there are no more rules then done if (!pckReadNext(optionalRules->pack)) { optionalRules->done = true; FUNCTION_TEST_RETURN(BOOL, false); } optionalRules->typeNext = pckReadU32P(optionalRules->pack); ASSERT(optionalRules->typeNext != 0); } // If this is the requested optional rule if (optionalRules->typeNext == optionalRuleType) { // Optional rule was found and dependency was matched result = true; match = true; // Process optional rule switch (optionalRuleType) { case parseRuleOptionalTypeValid: { pckReadNext(optionalRules->pack); optionalRules->valid = pckReadBufPtr(optionalRules->pack); optionalRules->validSize = pckReadSize(optionalRules->pack); pckReadConsume(optionalRules->pack); break; } case parseRuleOptionalTypeSequence: optionalRules->sequence = !ruleOption->sequence; break; case parseRuleOptionalTypeAllowList: { pckReadNext(optionalRules->pack); optionalRules->allowList = pckReadBufPtr(optionalRules->pack); optionalRules->allowListSize = pckReadSize(optionalRules->pack); pckReadConsume(optionalRules->pack); break; } case parseRuleOptionalTypeAllowRange: { PackRead *const ruleData = pckReadPackReadConstP(optionalRules->pack); pckReadNext(ruleData); if (pckReadType(ruleData) == pckTypeArray) { pckReadArrayBeginP(ruleData); while (!pckReadNullP(ruleData)) // {uncovered_branch - no options with partial allow range maps} { const unsigned int map = pckReadU32P(ruleData); optionalRules->allowRangeMinIdx = pckReadU32P(ruleData); optionalRules->allowRangeMaxIdx = pckReadU32P(ruleData); if (map == optionalRules->matchValue) { optionalRules->allowRangeMatch = true; break; } } match = optionalRules->allowRangeMatch; } else { optionalRules->allowRangeMinIdx = pckReadU32P(ruleData); optionalRules->allowRangeMaxIdx = pckReadU32P(ruleData); } optionalRules->allowRangeMin = cfgParseOptionValue(ruleOption->type, optionalRules->allowRangeMinIdx); optionalRules->allowRangeMax = cfgParseOptionValue(ruleOption->type, optionalRules->allowRangeMaxIdx); break; } case parseRuleOptionalTypeDefault: { PackRead *const ruleData = pckReadPackReadConstP(optionalRules->pack); if (ruleOption->defaultType == cfgDefaultTypeDynamic) { ASSERT(ruleOption->type == cfgOptTypeString); ASSERT(!ruleOption->sequence); // No need to check the value until there is more than one pckReadU32P(ruleData); optionalRules->defaultValue.string = optionalRules->defaultDynamicBin; optionalRules->defaultRaw = optionalRules->defaultDynamicBin; } else { ASSERT(ruleOption->defaultType == cfgDefaultTypeStatic); pckReadNext(ruleData); if (pckReadType(ruleData) == pckTypeArray) { ASSERT(!ruleOption->sequence); pckReadArrayBeginP(ruleData); match = false; while (!pckReadNullP(ruleData)) { const unsigned int map = pckReadU32P(ruleData); cfgParseOptionValuePack( ruleData, ruleOption->type, &optionalRules->defaultValue, &optionalRules->defaultRaw); if (map == optionalRules->matchValue) { match = true; break; } } } else { cfgParseOptionValuePack( ruleData, ruleOption->type, &optionalRules->defaultValue, &optionalRules->defaultRaw); if (optionalRules->sequence) optionalRules->defaultSequence = (uint8_t)pckReadU32P(ruleData); } } break; } default: { ASSERT(optionalRuleType == parseRuleOptionalTypeRequired); optionalRules->required = !ruleOption->required; } } } // Else not the requested optional rule else { // If the optional rule type is greater than requested then return. The optional rule may be requested later. if (optionalRules->typeNext > optionalRuleType) FUNCTION_TEST_RETURN(BOOL, false); // Consume the unused optional rule pckReadConsume(optionalRules->pack); } // Type will need to be read again on next iteration optionalRules->typeNext = 0; } while (!result); } FUNCTION_TEST_RETURN(BOOL, result && match); } /*********************************************************************************************************************************** Resolve an option dependency ***********************************************************************************************************************************/ typedef struct CfgParseOptionalFilterDependResult { bool valid; // Is the dependency valid? bool defaultExists; // Is there a default of the dependency is not valid? ConfigOption dependId; // Option id for the dependency ConfigOptionValueType defaultValue; // Default value if dependency is not valid unsigned int matchValue; // Match value if the dependency is valid } CfgParseOptionalFilterDependResult; static CfgParseOptionalFilterDependResult cfgParseOptionalFilterDepend( PackRead *const filter, const ConfigOptionType optionType, const Config *const config, const unsigned int optionListIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PACK_READ, filter); FUNCTION_TEST_PARAM(ENUM, optionType); FUNCTION_TEST_PARAM_P(VOID, config); FUNCTION_TEST_PARAM(UINT, optionListIdx); FUNCTION_TEST_END(); CfgParseOptionalFilterDependResult result = {.valid = false}; // Default when the dependency is not resolved, if it exists pckReadNext(filter); if (pckReadType(filter) == pckTypeBool) { pckReadBoolP(filter); pckReadNext(filter); result.defaultExists = true; cfgParseOptionValuePack(filter, optionType, &result.defaultValue, NULL); } // Get the depend option value result.dependId = (ConfigOption)pckReadU32P(filter); ASSERT(config->option[result.dependId].index != NULL); const ConfigOptionValue *const dependValue = &config->option[result.dependId].index[parseRuleOption[result.dependId].group ? optionListIdx : 0]; // Is the dependency resolved? if (dependValue->set) { // If a depend list exists, make sure the value is in the list if (pckReadNext(filter)) { do { switch (cfgParseOptionDataType(result.dependId)) { case cfgOptDataTypeBoolean: result.valid = pckReadBoolP(filter) == dependValue->value.boolean; break; default: { ASSERT(cfgParseOptionDataType(result.dependId) == cfgOptDataTypeStringId); result.matchValue = pckReadU32P(filter); if (strEq(PARSE_RULE_VAL_STR_IDX(result.matchValue), dependValue->display)) result.valid = true; break; } } if (result.valid) break; } while (pckReadNext(filter)); } else result.valid = true; } FUNCTION_TEST_RETURN_TYPE(CfgParseOptionalFilterDependResult, result); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgParseOptionName(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN_CONST(STRINGZ, parseRuleOption[optionId].name); } /**********************************************************************************************************************************/ FN_EXTERN const char * cfgParseOptionKeyIdxName(const ConfigOption optionId, const unsigned int keyIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_PARAM(UINT, keyIdx); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); ASSERT((!parseRuleOption[optionId].group && keyIdx == 0) || parseRuleOption[optionId].group); // If the option is in a group then construct the name const ParseRuleOption *const ruleOption = &parseRuleOption[optionId]; if (ruleOption->group) { FUNCTION_TEST_RETURN_CONST( STRINGZ, zNewFmt( "%s%u%s", parseRuleOptionGroup[ruleOption->groupId].name, keyIdx + 1, ruleOption->name + strlen(parseRuleOptionGroup[ruleOption->groupId].name))); } // Else return the stored name FUNCTION_TEST_RETURN_CONST(STRINGZ, ruleOption->name); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgParseOptionSecure(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(BOOL, parseRuleOption[optionId].secure); } /**********************************************************************************************************************************/ FN_EXTERN ConfigOptionType cfgParseOptionType(const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(ENUM, parseRuleOption[optionId].type); } /**********************************************************************************************************************************/ FN_EXTERN bool cfgParseOptionValid(const ConfigCommand commandId, const ConfigCommandRole commandRoleId, const ConfigOption optionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, commandId); FUNCTION_TEST_PARAM(ENUM, commandRoleId); FUNCTION_TEST_PARAM(ENUM, optionId); FUNCTION_TEST_END(); ASSERT(commandId < CFG_COMMAND_TOTAL); ASSERT(optionId < CFG_OPTION_TOTAL); FUNCTION_TEST_RETURN(BOOL, parseRuleOption[optionId].commandRoleValid[commandRoleId] & ((uint32_t)1 << commandId)); } /*********************************************************************************************************************************** Load the configuration file(s) The parent mem context is used. Defaults are passed to make testing easier. Rules: - config and config-include-path are default. In this case, the config file will be loaded, if it exists, and *.conf files in the config-include-path will be appended, if they exist. A missing/empty dir will be ignored except that the original default for the config file will be attempted to be loaded if the current default is not found. - config only is specified. Only the specified config file will be loaded and is required. The default config-include-path will be ignored. - config and config-path are specified. The specified config file will be loaded and is required. The overridden default of the config-include-path (/conf.d) will be loaded if exists but is not required. - config-include-path only is specified. *.conf files in the config-include-path will be loaded and the path is required to exist. The default config will be be loaded if it exists. - config-include-path and config-path are specified. The *.conf files in the config-include-path will be loaded and the directory passed must exist. The overridden default of the config file path (/pgbackrest.conf) will be loaded if exists but is not required. - If the config and config-include-path are specified. The config file will be loaded and is expected to exist and *.conf files in the config-include-path will be appended and at least one is expected to exist. - If --no-config is specified and --config-include-path is specified then only *.conf files in the config-include-path will be loaded; the directory is required. - If --no-config is specified and --config-path is specified then only *.conf files in the overridden default config-include-path (/conf.d) will be loaded if exist but not required. - If --no-config is specified and neither --config-include-path nor --config-path are specified then no configs will be loaded. - If --config-path only, the defaults for config and config-include-path will be changed to use that as a base path but the files will not be required to exist since this is a default override. ***********************************************************************************************************************************/ static void cfgFileLoadPart(String **const config, const Buffer *const configPart) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(STRING, config); FUNCTION_LOG_PARAM(BUFFER, configPart); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); if (configPart != NULL) { // Validate the file by parsing it as an Ini object. If the file is not properly formed, an error will occur. if (bufUsed(configPart) > 0) { iniValid(iniNewP(ioBufferReadNew(configPart))); // Create the result config file if (*config == NULL) { *config = strNew(); } // Else add an LF in case the previous file did not end with one else strCat(*config, LF_STR); // Add the config part to the result config file strCat(*config, strNewBuf(configPart)); } } FUNCTION_LOG_RETURN_VOID(); } static String * cfgFileLoad( const Storage *const storage, // Storage to load configs const ParseOption *const optionList, // All options and their current settings // NOTE: Passing defaults to enable more complete test coverage const String *optConfigDefault, // Current default for --config option const String *optConfigIncludePathDefault, // Current default for --config-include-path option const String *const origConfigDefault) // Original --config option default (/etc/pgbackrest.conf) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM_P(PARSE_OPTION, optionList); FUNCTION_LOG_PARAM(STRING, optConfigDefault); FUNCTION_LOG_PARAM(STRING, optConfigIncludePathDefault); FUNCTION_LOG_PARAM(STRING, origConfigDefault); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(optionList != NULL); ASSERT(optConfigDefault != NULL); ASSERT(optConfigIncludePathDefault != NULL); ASSERT(origConfigDefault != NULL); bool loadConfig = true; bool loadConfigInclude = true; // If the option is specified on the command line, then found will be true meaning the file is required to exist, else it is // optional const bool configFound = optionList[cfgOptConfig].indexList != NULL && optionList[cfgOptConfig].indexList[0].found; bool configRequired = configFound; const bool configPathRequired = optionList[cfgOptConfigPath].indexList != NULL && optionList[cfgOptConfigPath].indexList[0].found; bool configIncludeRequired = optionList[cfgOptConfigIncludePath].indexList != NULL && optionList[cfgOptConfigIncludePath].indexList[0].found; // Save default for later determining if must check old original default config path const String *const optConfigDefaultCurrent = optConfigDefault; // If the config-path option is found on the command line, then its value will override the base path defaults for config and // config-include-path if (configPathRequired) { optConfigDefault = strNewFmt( "%s/%s", strZ(strLstGet(optionList[cfgOptConfigPath].indexList[0].valueList, 0)), strBaseZ(optConfigDefault)); optConfigIncludePathDefault = strNewFmt( "%s/%s", strZ(strLstGet(optionList[cfgOptConfigPath].indexList[0].valueList, 0)), PROJECT_CONFIG_INCLUDE_PATH); } // If the --no-config option was passed then do not load the config file if (optionList[cfgOptConfig].indexList != NULL && optionList[cfgOptConfig].indexList[0].negate) { loadConfig = false; configRequired = false; } // If --config option is specified on the command line but neither the --config-include-path nor the config-path are passed, // then do not attempt to load the include files if (configFound && !(configPathRequired || configIncludeRequired)) { loadConfigInclude = false; configIncludeRequired = false; } String *result = NULL; // Load the main config file if (loadConfig) { const String *configFileName = NULL; // Get the config file name from the command-line if it exists else default if (configRequired) configFileName = strLstGet(optionList[cfgOptConfig].indexList[0].valueList, 0); else configFileName = optConfigDefault; // Load the config file const Buffer *buffer = storageGetP(storageNewReadP(storage, configFileName, .ignoreMissing = !configRequired)); // Convert the contents of the file buffer to the config string object if (buffer != NULL) result = strCatBuf(strNew(), buffer); else if (strEq(configFileName, optConfigDefaultCurrent)) { // If config is current default and it was not found, attempt to load the config file from the old default location buffer = storageGetP(storageNewReadP(storage, origConfigDefault, .ignoreMissing = !configRequired)); if (buffer != NULL) result = strCatBuf(strNew(), buffer); } } // Load *.conf files from the include directory if (loadConfigInclude) { // Validate the file by parsing it as an Ini object. If the file is not properly formed, an error will occur. if (result != NULL) iniValid(iniNewP(ioBufferReadNew(BUFSTR(result)))); const String *configIncludePath = NULL; // Get the config include path from the command-line if it exists else default if (configIncludeRequired) configIncludePath = strLstGet(optionList[cfgOptConfigIncludePath].indexList[0].valueList, 0); else configIncludePath = optConfigIncludePathDefault; // Get a list of conf files from the specified path -error on missing directory if the option was passed on the command line StringList *const list = storageListP( storage, configIncludePath, .expression = STRDEF(".+\\.conf$"), .errorOnMissing = configIncludeRequired, .nullOnMissing = !configIncludeRequired); // If conf files are found, then add them to the config string if (list != NULL && !strLstEmpty(list)) { // Sort the list for reproducibility only -- order does not matter strLstSort(list, sortOrderAsc); for (unsigned int listIdx = 0; listIdx < strLstSize(list); listIdx++) { cfgFileLoadPart( &result, storageGetP( storageNewReadP( storage, strNewFmt("%s/%s", strZ(configIncludePath), strZ(strLstGet(list, listIdx))), .ignoreMissing = true))); } } } FUNCTION_LOG_RETURN(STRING, result); } /*********************************************************************************************************************************** ??? Add validation of section names and check all sections for invalid options in the check command. It's too expensive to add the logic to this critical path code. ***********************************************************************************************************************************/ // Helper to check that option values are valid for conditional builds. This is a bit tricky since the distros used for unit testing // have all possible features enabled, so this is split out to allow it to be tested independently. The loop variable is // intentionally integrated into this function to make it obvious if it is omitted from the caller. static bool cfgParseOptionValueCondition( bool more, PackRead *const allowList, const bool allowListFound, const unsigned int optionId, const unsigned int optionKeyIdx, const String *const valueAllow) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, more); FUNCTION_TEST_PARAM(PACK_READ, allowList); FUNCTION_TEST_PARAM(BOOL, allowListFound); FUNCTION_TEST_PARAM(UINT, optionId); FUNCTION_TEST_PARAM(UINT, optionKeyIdx); FUNCTION_TEST_PARAM(STRING, valueAllow); FUNCTION_TEST_END(); ASSERT(allowList != NULL); ASSERT(valueAllow != NULL); if (more && pckReadType(allowList) == pckTypeBool) { pckReadBoolP(allowList); if (allowListFound) { THROW_FMT( OptionInvalidValueError, PROJECT_NAME " not built with '%s=%s' support\n" "HINT: if " PROJECT_NAME " was installed from a package, does the package support this feature?\n" "HINT: if " PROJECT_NAME " was built from source, were the required development packages installed?", cfgParseOptionKeyIdxName(optionId, optionKeyIdx), strZ(valueAllow)); } more = pckReadNext(allowList); } FUNCTION_TEST_RETURN(BOOL, more); } FN_EXTERN void cfgParse(const Storage *const storage, const unsigned int argListSize, const char *argList[], const CfgParseParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(UINT, argListSize); FUNCTION_LOG_PARAM(CHARPY, argList); FUNCTION_LOG_PARAM(BOOL, param.noResetLogLevel); FUNCTION_LOG_PARAM(BOOL, param.noConfigLoad); FUNCTION_LOG_END(); // Initialize local mem context if (configParseLocal.memContext == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(ConfigParse, .childQty = MEM_CONTEXT_QTY_MAX) { configParseLocal.memContext = MEM_CONTEXT_NEW(); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_BEGIN() { // Create the config struct Config *config; OBJ_NEW_BASE_BEGIN(Config, .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { config = OBJ_NEW_ALLOC(); *config = (Config) { .memContext = MEM_CONTEXT_NEW(), .command = cfgCmdHelp, .bin = strNewZ(argList[0]), }; } OBJ_NEW_END(); // Phase 1: parse command line parameters // ------------------------------------------------------------------------------------------------------------------------- // List of parsed options ParseOption parseOptionList[CFG_OPTION_TOTAL] = {{0}}; // Only the first non-option parameter should be treated as a command so track if the command has been set bool commandSet = false; bool commandHelpSet = false; for (unsigned int argListIdx = 1; argListIdx < argListSize; argListIdx++) { const char *arg = argList[argListIdx]; // If an option if (arg[0] == '-') { // Options must start with -- if (arg[1] != '-') THROW_FMT(OptionInvalidError, "option '%s' must begin with --", arg); // Consume -- arg += 2; // Get the option name and the value when separated by = const String *optionName; const String *optionArg = NULL; const char *const equalPtr = strchr(arg, '='); if (equalPtr) { optionName = strNewZN(arg, (size_t)(equalPtr - arg)); optionArg = strNewZ(equalPtr + 1); } else optionName = strNewZ(arg); // Lookup the option name CfgParseOptionResult option = cfgParseOptionP(optionName, .prefixMatch = true); if (!option.found) THROW_FMT(OptionInvalidError, "invalid option '--%s'", arg); // If the option may have an argument (arguments are optional for boolean options) const ParseRuleOption *const ruleOption = &parseRuleOption[option.id]; if (!option.negate && !option.reset) { // Handle boolean (only y/n allowed as argument) if (ruleOption->type == cfgOptTypeBoolean) { // Validate argument/set negate when argument present if (optionArg != NULL) { if (strEqZ(optionArg, "n")) option.negate = true; else if (!strEqZ(optionArg, "y")) { THROW_FMT( OptionInvalidValueError, "boolean option '--%s' argument must be 'y' or 'n'", strZ(optionName)); } } } // If no argument was found with the option then try the next argument (unless bool-like) else if (optionArg == NULL) { // If bool-like then set arg to y if (ruleOption->boolLike) { optionArg = Y_STR; } // Else if there are no more arguments in the list else if (argListIdx == argListSize - 1) { THROW_FMT(OptionInvalidError, "option '--%s' requires an argument", strZ(optionName)); } // Else get arg else optionArg = strNewZ(argList[++argListIdx]); } } // Else error if an argument was found with the option else if (optionArg != NULL) THROW_FMT(OptionInvalidError, "option '%s' does not allow an argument", strZ(optionName)); // if negated and bool-like then set to n if (option.negate && ruleOption->boolLike) { option.negate = false; optionArg = N_STR; } // Error if this option is secure and cannot be passed on the command line if (cfgParseOptionSecure(option.id)) { THROW_FMT( OptionInvalidError, "option '%s' is not allowed on the command-line\n" "HINT: this option could expose secrets in the process list.\n" "HINT: specify the option in a configuration file or an environment variable instead.", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // If the option has not been found yet then set it ParseOptionValue *const optionValue = parseOptionIdxValue(parseOptionList, option.id, option.keyIdx); if (!optionValue->found) { *optionValue = (ParseOptionValue) { .found = true, .negate = option.negate, .reset = option.reset, .source = cfgSourceParam, }; // Only set the argument if the option has one if (optionArg != NULL) { optionValue->valueList = strLstNew(); strLstAdd(optionValue->valueList, optionArg); } } else { // Make sure option is not negated more than once. It probably wouldn't hurt anything to accept this case but // there's no point in allowing the user to be sloppy. if (optionValue->negate && option.negate) { THROW_FMT( OptionInvalidError, "option '%s' is negated multiple times", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Make sure option is not reset more than once. Same justification as negate. if (optionValue->reset && option.reset) { THROW_FMT( OptionInvalidError, "option '%s' is reset multiple times", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Don't allow an option to be both negated and reset if ((optionValue->reset && option.negate) || (optionValue->negate && option.reset)) { THROW_FMT( OptionInvalidError, "option '%s' cannot be negated and reset", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Don't allow an option to be both set and negated if (optionValue->negate != option.negate) { THROW_FMT( OptionInvalidError, "option '%s' cannot be set and negated", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Don't allow an option to be both set and reset if (optionValue->reset != option.reset) { THROW_FMT( OptionInvalidError, "option '%s' cannot be set and reset", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } // Add the argument if (optionArg != NULL && option.multi) { strLstAdd(optionValue->valueList, optionArg); } // Error if the option does not accept multiple arguments else { THROW_FMT( OptionInvalidError, "option '%s' cannot be set multiple times", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } } } // Else command or parameter else { // The first argument should be the command if (!commandSet) { // Try getting the command from the valid command list config->command = cfgParseCommandId(arg); // Set help if help command if (config->command == cfgCmdHelp && !commandHelpSet) { commandHelpSet = true; } // Else parse command role if appended else { const char *const colonPtr = strchr(arg, ':'); if (colonPtr != NULL) config->commandRole = cfgParseCommandRoleEnum(colonPtr + 1); else config->commandRole = cfgCmdRoleMain; // Error when role is not valid for the command if (!(parseRuleCommand[config->command].commandRoleValid & ((unsigned int)1 << config->commandRole))) THROW_FMT(CommandInvalidError, "invalid command/role combination '%s'", arg); // Error if help requested and role is not main if (commandHelpSet && config->commandRole != cfgCmdRoleMain) THROW_FMT(CommandInvalidError, "help is not available for the '%s' role", colonPtr + 1); config->help = commandHelpSet; commandSet = true; } } // Additional arguments are command arguments else { if (config->paramList == NULL) { MEM_CONTEXT_BEGIN(config->memContext) { config->paramList = strLstNew(); } MEM_CONTEXT_END(); } strLstAddZ(config->paramList, arg); } } } // Handle command not found if (!commandSet && !commandHelpSet) { ASSERT(config->command == cfgCmdHelp); for (unsigned int optionIdx = 0; optionIdx < CFG_OPTION_TOTAL; optionIdx++) { if (parseOptionList[optionIdx].indexListTotal != 0 && !cfgParseOptionValid(cfgCmdHelp, cfgCmdRoleMain, optionIdx)) THROW_FMT(CommandRequiredError, "no command found"); } } // Set command options const ParseRuleCommand *const ruleCommand = &parseRuleCommand[config->command]; config->lockRequired = ruleCommand->lockRequired; config->lockRemoteRequired = ruleCommand->lockRemoteRequired; config->lockType = (LockType)ruleCommand->lockType; config->logFile = ruleCommand->logFile; config->logLevelDefault = (LogLevel)ruleCommand->logLevelDefault; // Error when parameters found but the command does not allow parameters if (config->paramList != NULL && !config->help && !ruleCommand->parameterAllowed) THROW(ParamInvalidError, "command does not allow parameters"); // Enable logging for main role so config file warnings will be output if (!param.noResetLogLevel && config->commandRole == cfgCmdRoleMain) logInit(logLevelWarn, logLevelOff, logLevelOff, false, 0, 1, false); // Determine whether any options should be loaded from the environment or config files. The help and version commands do // not use any non command-line options so they can skip this step. This prevents basic help or version from failing if, // e.g. the process does not have access to load the config file or there is an invalid option in the environment. const bool optionLoad = config->command != cfgCmdHelp && config->command != cfgCmdVersion; // Phase 2: parse environment variables // ------------------------------------------------------------------------------------------------------------------------- if (optionLoad) { unsigned int environIdx = 0; // Loop through all environment variables and look for our env vars by matching the prefix while (environ[environIdx] != NULL) { const char *keyValue = environ[environIdx]; environIdx++; if (strstr(keyValue, PGBACKREST_ENV) == keyValue) { // Find the first = char const char *const equalPtr = strchr(keyValue, '='); ASSERT(equalPtr != NULL); // Get key and value const String *const key = strReplaceChr( strLower(strNewZN(keyValue + PGBACKREST_ENV_SIZE, (size_t)(equalPtr - (keyValue + PGBACKREST_ENV_SIZE)))), '_', '-'); const String *const value = STR(equalPtr + 1); // Find the option CfgParseOptionResult option = cfgParseOptionP(key); // Warn if the option not found if (!option.found) { LOG_WARN_FMT("environment contains invalid option '%s'", strZ(key)); continue; } // Warn if negate option found in env else if (option.negate) { LOG_WARN_FMT("environment contains invalid negate option '%s'", strZ(key)); continue; } // Warn if reset option found in env else if (option.reset) { LOG_WARN_FMT("environment contains invalid reset option '%s'", strZ(key)); continue; } // Continue if the option is not valid for this command if (!cfgParseOptionValid(config->command, config->commandRole, option.id)) continue; if (strSize(value) == 0) THROW_FMT(OptionInvalidValueError, "environment variable '%s' must have a value", strZ(key)); // Continue if the option has already been specified on the command line ParseOptionValue *optionValue = parseOptionIdxValue(parseOptionList, option.id, option.keyIdx); if (optionValue->found) continue; optionValue->found = true; optionValue->source = cfgSourceConfig; // Convert boolean to string if (cfgParseOptionType(option.id) == cfgOptTypeBoolean) { if (strEqZ(value, "n")) optionValue->negate = true; else if (!strEqZ(value, "y")) THROW_FMT(OptionInvalidValueError, "environment boolean option '%s' must be 'y' or 'n'", strZ(key)); } // Else split list/hash into separate values else if (option.multi) { optionValue->valueList = strLstNewSplitZ(value, ":"); } // Else add the string value else { optionValue->valueList = strLstNew(); strLstAdd(optionValue->valueList, value); } } } } // Phase 3: parse config file unless --no-config passed // ------------------------------------------------------------------------------------------------------------------------- // Load the configuration file(s) if (!param.noConfigLoad && optionLoad) { const String *const configString = cfgFileLoad( storage, parseOptionList, PARSE_RULE_VAL_STR_IDX(parseRuleValStrCFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_FILE), PARSE_RULE_VAL_STR_IDX(parseRuleValStrCFGOPTDEF_CONFIG_PATH_SP_QT_FS_QT_SP_PROJECT_CONFIG_INCLUDE_PATH), PGBACKREST_CONFIG_ORIG_PATH_FILE_STR); iniFree(configParseLocal.ini); configParseLocal.ini = NULL; if (configString != NULL) { MEM_CONTEXT_BEGIN(configParseLocal.memContext) { configParseLocal.ini = iniNewP(ioBufferReadNew(BUFSTR(configString)), .store = true); } MEM_CONTEXT_END(); } } if (configParseLocal.ini != NULL) { // Get the stanza name String *stanza = NULL; if (parseOptionList[cfgOptStanza].indexList != NULL) stanza = strLstGet(parseOptionList[cfgOptStanza].indexList[0].valueList, 0); // Build list of sections to search for options StringList *const sectionList = strLstNew(); if (stanza != NULL) { strLstAddFmt(sectionList, "%s:%s", strZ(stanza), cfgParseCommandName(config->command)); strLstAdd(sectionList, stanza); } strLstAddFmt(sectionList, CFGDEF_SECTION_GLOBAL ":%s", cfgParseCommandName(config->command)); strLstAddZ(sectionList, CFGDEF_SECTION_GLOBAL); // Loop through sections to search for options for (unsigned int sectionIdx = 0; sectionIdx < strLstSize(sectionList); sectionIdx++) { const String *const section = strLstGet(sectionList, sectionIdx); const StringList *const keyList = iniSectionKeyList(configParseLocal.ini, section); KeyValue *const optionFound = kvNew(); // Loop through keys to search for options for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { String *const key = strLstGet(keyList, keyIdx); // Find the optionName in the main list const CfgParseOptionResult option = cfgParseOptionP(key); const ParseRuleOption *const ruleOption = &parseRuleOption[option.id]; // Warn if the option not found if (!option.found) { LOG_WARN_FMT("configuration file contains invalid option '%s'", strZ(key)); continue; } // Warn if negate option found in config else if (option.negate) { LOG_WARN_FMT("configuration file contains negate option '%s'", strZ(key)); continue; } // Warn if reset option found in config else if (option.reset) { LOG_WARN_FMT("configuration file contains reset option '%s'", strZ(key)); continue; } // Warn if this option should be command-line only if (ruleOption->section == cfgSectionCommandLine) { LOG_WARN_FMT("configuration file contains command-line only option '%s'", strZ(key)); continue; } // Make sure this option does not appear in the same section with an alternate name const Variant *const optionFoundKey = VARUINT64(option.id * CFG_OPTION_KEY_MAX + option.keyIdx); const Variant *const optionFoundName = kvGet(optionFound, optionFoundKey); if (optionFoundName != NULL) { THROW_FMT( OptionInvalidError, "configuration file contains duplicate options ('%s', '%s') in section '[%s]'", strZ(key), strZ(varStr(optionFoundName)), strZ(section)); } else kvPut(optionFound, optionFoundKey, VARSTR(key)); // Continue if the option is not valid for this command if (!cfgParseOptionValid(config->command, config->commandRole, option.id)) { // Warn if it is in a command section if (sectionIdx % 2 == 0) { LOG_WARN_FMT( "configuration file contains option '%s' invalid for section '%s'", strZ(key), strZ(section)); } continue; } // Continue if stanza option is in a global section if (ruleOption->section == cfgSectionStanza && (strEqZ(section, CFGDEF_SECTION_GLOBAL) || strBeginsWithZ(section, CFGDEF_SECTION_GLOBAL ":"))) { LOG_WARN_FMT( "configuration file contains stanza-only option '%s' in global section '%s'", strZ(key), strZ(section)); continue; } // Continue if this option has already been found in another section or command-line/environment ParseOptionValue *const optionValue = parseOptionIdxValue(parseOptionList, option.id, option.keyIdx); if (optionValue->found) continue; optionValue->found = true; optionValue->source = cfgSourceConfig; // Process list if (iniSectionKeyIsList(configParseLocal.ini, section, key)) { // Error if the option cannot be specified multiple times if (!option.multi) { THROW_FMT( OptionInvalidError, "option '%s' cannot be set multiple times", cfgParseOptionKeyIdxName(option.id, option.keyIdx)); } optionValue->valueList = iniGetList(configParseLocal.ini, section, key); } else { // Get the option value const String *const value = iniGet(configParseLocal.ini, section, key); if (strSize(value) == 0) { THROW_FMT( OptionInvalidValueError, "section '%s', key '%s' must have a value", strZ(section), strZ(key)); } if (cfgParseOptionType(option.id) == cfgOptTypeBoolean) { if (strEqZ(value, "n")) optionValue->negate = true; else if (!strEqZ(value, "y")) THROW_FMT(OptionInvalidValueError, "boolean option '%s' must be 'y' or 'n'", strZ(key)); } // Else add the string value else { optionValue->valueList = strLstNew(); strLstAdd(optionValue->valueList, value); } } } } } // Phase 4: create the config and resolve indexed options for each group // ------------------------------------------------------------------------------------------------------------------------- // Determine how many indexes are used in each group bool groupIdxMap[CFG_OPTION_GROUP_TOTAL][CFG_OPTION_KEY_MAX] = {{0}}; for (unsigned int optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { // Always assign name since it may be needed for error messages const ParseRuleOption *const ruleOption = &parseRuleOption[optionId]; config->option[optionId].name = ruleOption->name; // Is the option valid for this command? if (cfgParseOptionValid(config->command, config->commandRole, optionId)) { config->option[optionId].valid = true; config->option[optionId].sequence = ruleOption->sequence; config->option[optionId].dataType = cfgParseOptionDataType(optionId); config->option[optionId].group = ruleOption->group; config->option[optionId].groupId = ruleOption->groupId; } else { // Error if the invalid option was explicitly set on the command-line if (parseOptionList[optionId].indexList != NULL) { THROW_FMT( OptionInvalidError, "option '%s' not valid for command '%s'", cfgParseOptionName(optionId), cfgParseCommandName(config->command)); } // Continue to the next option continue; } // If the option is in a group if (ruleOption->group) { const unsigned int groupId = ruleOption->groupId; config->optionGroup[groupId].valid = true; // Scan the option values to determine which indexes are in use. Store them in a map that will later be scanned to // create a list of just the used indexes. for (unsigned int optionKeyIdx = 0; optionKeyIdx < parseOptionList[optionId].indexListTotal; optionKeyIdx++) { if (parseOptionList[optionId].indexList[optionKeyIdx].found && !parseOptionList[optionId].indexList[optionKeyIdx].reset) { if (!groupIdxMap[groupId][optionKeyIdx]) { config->optionGroup[groupId].indexTotal++; groupIdxMap[groupId][optionKeyIdx] = true; } } } } } // Write the indexes into the group in order for (unsigned int groupId = 0; groupId < CFG_OPTION_GROUP_TOTAL; groupId++) { // Set group name config->optionGroup[groupId].name = parseRuleOptionGroup[groupId].name; // Skip the group if it is not valid if (!config->optionGroup[groupId].valid) continue; // Allocate memory for the index to key index map. The pg group always reserves index 0 for key 1 (see below) even when // key 1 was not configured, so allocate an extra slot in that case to avoid writing past the end of the map. unsigned int optionIdxTotal = config->optionGroup[groupId].indexTotal; if (groupId == cfgOptGrpPg && optionIdxTotal > 0 && !groupIdxMap[groupId][0]) optionIdxTotal++; MEM_CONTEXT_BEGIN(config->memContext) { config->optionGroup[groupId].indexMap = memNew(sizeof(unsigned int) * (optionIdxTotal == 0 ? 1 : optionIdxTotal)); } MEM_CONTEXT_END(); // If no values were found in any index then use index 0 since all valid groups must have at least one index. This may // lead to an error unless all options in the group have defaults but that will be resolved later. if (optionIdxTotal == 0) { config->optionGroup[groupId].indexTotal = 1; config->optionGroup[groupId].indexMap[0] = 0; } // Else write the key to index map for the group. This allows translation from keys to indexes and vice versa. else { unsigned int optionIdxMax = 0; unsigned int optionKeyIdx = 0; // ??? For the pg group, key 1 is required to maintain compatibility with older versions. Before removing this // constraint the pg group remap to key 1 for remotes will need to be dealt with in the protocol/helper module. if (groupId == cfgOptGrpPg) { optionKeyIdx = 1; optionIdxMax = 1; config->optionGroup[groupId].indexMap[0] = 0; } // Write keys into the index map for (; optionKeyIdx < CFG_OPTION_KEY_MAX; optionKeyIdx++) { if (groupIdxMap[groupId][optionKeyIdx]) { config->optionGroup[groupId].indexMap[optionIdxMax] = optionKeyIdx; optionIdxMax++; } } // Update the index total to include the reserved pg index 0 when key 1 was not configured config->optionGroup[groupId].indexTotal = optionIdxMax; } } // Phase 5: validate option definitions and load into configuration // ------------------------------------------------------------------------------------------------------------------------- // Determine whether a group index will be kept based on non-default values bool optionGroupIndexKeep[CFG_OPTION_GROUP_TOTAL][CFG_OPTION_KEY_MAX] = {{false}}; for (unsigned int optionOrderIdx = 0; optionOrderIdx < CFG_OPTION_TOTAL; optionOrderIdx++) { // Validate options based on the option resolve order. This allows resolving all options in a single pass. const ConfigOption optionId = optionResolveOrder[optionOrderIdx]; // Skip this option if it is not valid if (!config->option[optionId].valid) continue; // Determine the option index total. For options that are not indexed the index total is 1. const ParseRuleOption *const ruleOption = &parseRuleOption[optionId]; const bool optionGroup = ruleOption->group; const unsigned int optionGroupId = optionGroup ? ruleOption->groupId : UINT_MAX; const unsigned int optionListIndexTotal = optionGroup ? config->optionGroup[optionGroupId].indexTotal : 1; MEM_CONTEXT_BEGIN(config->memContext) { config->option[optionId].index = memNew(sizeof(ConfigOptionValue) * optionListIndexTotal); } MEM_CONTEXT_END(); // Loop through the option indexes const ConfigOptionType optionType = cfgParseOptionType(optionId); for (unsigned int optionListIdx = 0; optionListIdx < optionListIndexTotal; optionListIdx++) { // Get the key index by looking it up in the group or by defaulting to 0 for ungrouped options const unsigned optionKeyIdx = optionGroup ? config->optionGroup[optionGroupId].indexMap[optionListIdx] : 0; // Get the parsed value using the key index. Provide a default structure when the value was not found. const ParseOptionValue *const parseOptionValue = optionKeyIdx < parseOptionList[optionId].indexListTotal ? &parseOptionList[optionId].indexList[optionKeyIdx] : &(ParseOptionValue){0}; // Get the location where the value will be stored in the configuration ConfigOptionValue *const configOptionValue = &config->option[optionId].index[optionListIdx]; // Is the value set for this option? const bool optionSet = parseOptionValue->found && (optionType == cfgOptTypeBoolean || !parseOptionValue->negate) && !parseOptionValue->reset; // Initialize option value and set negate and reset flag *configOptionValue = (ConfigOptionValue){.negate = parseOptionValue->negate, .reset = parseOptionValue->reset}; // Is the option valid? CfgParseOptionalRuleState optionalRules = {.defaultDynamicBin = config->bin}; CfgParseOptionalFilterDependResult dependResult = {.valid = true}; const char *dependOptionName = NULL; if (cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeValid, config->command, optionId)) { PackRead *const filter = pckReadNewC(optionalRules.valid, optionalRules.validSize); dependResult = cfgParseOptionalFilterDepend(filter, optionType, config, optionListIdx); dependOptionName = cfgParseOptionKeyIdxName( dependResult.dependId, parseRuleOption[dependResult.dependId].group ? optionKeyIdx : 0); // If depend not resolved and option value is set on the command-line then error. It is OK to have unresolved // options in the config file because they may be there for another command. For instance, spool-path is only // loaded for the archive-push command when archive-async=y, and the presence of spool-path in the config file // should not cause an error here, it will just end up null. if (!dependResult.valid && optionSet && parseOptionValue->source == cfgSourceParam) { PackRead *const filter = pckReadNewC(optionalRules.valid, optionalRules.validSize); // If there is a boolean default value just consume it since it is not needed here pckReadNext(filter); if (pckReadType(filter) == pckTypeBool) { pckReadBoolP(filter); pckReadConsume(filter); } // Get depend option id and name const ConfigOption dependId = pckReadU32P(filter); // If depend value is not set ASSERT(config->option[dependId].index != NULL); if (!config->option[dependId].index[optionListIdx].set) { THROW_FMT( OptionInvalidError, "option '%s' not valid without option '%s'", cfgParseOptionKeyIdxName(optionId, optionKeyIdx), dependOptionName); } // Build type dependent error data const String *errorValue = EMPTY_STR; switch (parseRuleOption[dependId].type) { case cfgOptTypeBoolean: { if (!pckReadBoolP(filter)) dependOptionName = zNewFmt("no-%s", dependOptionName); break; } default: { String *const errorList = strNew(); unsigned int validSize = 0; while (pckReadNext(filter)) { strCatFmt( errorList, "%s'%s'", validSize != 0 ? ", " : "", strZ(cfgParseOptionValueStr(parseRuleOption[dependId].type, pckReadU32P(filter)))); validSize++; } ASSERT(validSize > 0); if (validSize == 1) errorValue = strNewFmt(" = %s", strZ(errorList)); else errorValue = strNewFmt(" in (%s)", strZ(errorList)); } } THROW_FMT( OptionInvalidError, "option '%s' not valid without option '%s'%s", cfgParseOptionKeyIdxName(optionId, optionKeyIdx), dependOptionName, strZ(errorValue)); } // Store the match value in case we need it to find the default, range, etc. optionalRules.matchValue = dependResult.matchValue; pckReadFree(filter); } // Does the command override sequence? if (cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeSequence, config->command, optionId)) config->option[optionId].sequence = optionalRules.sequence; else optionalRules.sequence = config->option[optionId].sequence; if (dependResult.valid) { // Is the option set? if (optionSet) { configOptionValue->set = true; configOptionValue->source = parseOptionValue->source; // Check beta status parseOptionBeta(optionId, optionKeyIdx, ruleOption->beta, &parseOptionList[cfgOptBeta]); if (optionType == cfgOptTypeBoolean) { configOptionValue->value.boolean = !parseOptionValue->negate; } else if (optionType == cfgOptTypeHash) { KeyValue *value = NULL; MEM_CONTEXT_BEGIN(config->memContext) { value = kvNew(); } MEM_CONTEXT_END(); for (unsigned int listIdx = 0; listIdx < strLstSize(parseOptionValue->valueList); listIdx++) { const char *const pair = strZ(strLstGet(parseOptionValue->valueList, listIdx)); const char *const equal = strchr(pair, '='); if (equal == NULL) { THROW_FMT( OptionInvalidError, "key/value '%s' not valid for '%s' option", strZ(strLstGet(parseOptionValue->valueList, listIdx)), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // Warn if a value will be overwritten const Variant *const key = VARSTR(strNewZN(pair, (size_t)(equal - pair))); const String *const old = varStr(kvGet(value, key)); const Variant *const new = VARSTRZ(equal + 1); if (old != NULL) { LOG_WARN_FMT( "key '%s' value '%s' is overwritten with '%s' for '%s' option", strZ(varStr(key)), strZ(old), strZ(varStr(new)), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // Put the value (overwriting if the key already exists) kvPut(value, key, new); } configOptionValue->value.keyValue = value; } else if (optionType == cfgOptTypeList) { MEM_CONTEXT_BEGIN(config->memContext) { configOptionValue->value.list = varLstNewStrLst(parseOptionValue->valueList); } MEM_CONTEXT_END(); } else { String *const value = strLstGet(parseOptionValue->valueList, 0); const String *const valueAllow = value; // Preserve original value to display MEM_CONTEXT_BEGIN(config->memContext) { configOptionValue->display = strDup(value); } MEM_CONTEXT_END(); // If a numeric type check that the value is valid if (optionType == cfgOptTypeInteger || optionType == cfgOptTypeSize || optionType == cfgOptTypeTime) { // Check that the value can be converted TRY_BEGIN() { switch (optionType) { case cfgOptTypeInteger: configOptionValue->value.integer = cvtZToInt64(strZ(value)); break; case cfgOptTypeSize: configOptionValue->value.integer = cfgParseSize(value); break; default: { ASSERT(optionType == cfgOptTypeTime); configOptionValue->value.integer = cfgParseTime(value); break; } } } CATCH_ANY() { THROW_FMT( OptionInvalidValueError, "'%s' is not valid for '%s' option", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } TRY_END(); if (cfgParseOptionalRule( &optionalRules, parseRuleOptionalTypeAllowRange, config->command, optionId) && (configOptionValue->value.integer < optionalRules.allowRangeMin || configOptionValue->value.integer > optionalRules.allowRangeMax)) { THROW_FMT( OptionInvalidValueError, "'%s' is out of range for '%s' option%s\n" "HINT: allowed range is %s to %s inclusive", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx), optionalRules.allowRangeMatch ? zNewFmt( " when '%s' option = '%s'", dependOptionName, strZ(PARSE_RULE_VAL_STR_IDX(optionalRules.matchValue))) : "", strZ(cfgParseOptionValueStr(optionType, optionalRules.allowRangeMinIdx)), strZ(cfgParseOptionValueStr(optionType, optionalRules.allowRangeMaxIdx))); } } // Else if string make sure it is valid else if (optionType != cfgOptTypeStringId) { ASSERT(optionType == cfgOptTypePath || optionType == cfgOptTypeString); // Set string value to display value configOptionValue->value.string = configOptionValue->display; // Empty strings are not valid if (strSize(value) == 0) { THROW_FMT( OptionInvalidValueError, "'%s' must be >= 1 character for '%s' option", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // If path make sure it is valid if (optionType == cfgOptTypePath) { // Make sure it starts with / if (!strBeginsWithZ(value, "/")) { THROW_FMT( OptionInvalidValueError, "'%s' must begin with / for '%s' option", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // Make sure there are no occurrences of // if (strstr(strZ(value), "//") != NULL) { THROW_FMT( OptionInvalidValueError, "'%s' cannot contain // for '%s' option", strZ(value), cfgParseOptionKeyIdxName(optionId, optionKeyIdx)); } // If the path ends with a / we'll strip it off (unless the value is just /) if (strEndsWithZ(value, "/") && strSize(value) != 1) { strTruncIdx(value, (int)strSize(value) - 1); // Reset string value since it was modified MEM_CONTEXT_BEGIN(config->memContext) { configOptionValue->value.string = strDup(value); } MEM_CONTEXT_END(); } } } // If the option has an allow list then check it if (cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeAllowList, config->command, optionId)) { PackRead *const allowList = pckReadNewC(optionalRules.allowList, optionalRules.allowListSize); bool allowListFound = false; unsigned int sequence = 0; pckReadNext(allowList); while (true) { // Compare based on option type const unsigned int valueIdx = pckReadU32P(allowList); switch (ruleOption->type) { case cfgOptTypeStringId: { if (strEq(PARSE_RULE_VAL_STR_IDX(valueIdx), configOptionValue->display)) { allowListFound = true; if (optionalRules.sequence) configOptionValue->sequence = (uint8_t)sequence; } sequence++; break; } default: allowListFound = cfgParseOptionValue(ruleOption->type, valueIdx) == configOptionValue->value.integer; break; } // Stop when allow list is exhausted or value is found if (!cfgParseOptionValueCondition( pckReadNext(allowList), allowList, allowListFound, optionId, optionKeyIdx, valueAllow) || allowListFound) { break; } } pckReadFree(allowList); if (!allowListFound) { PackRead *const allowList = pckReadNewC(optionalRules.allowList, optionalRules.allowListSize); String *const hintList = strNew(); while (!pckReadNullP(allowList)) { if (!strEmpty(hintList)) strCatZ(hintList, ", "); strCatFmt( hintList, "'%s'", strZ(cfgParseOptionValueStr(ruleOption->type, pckReadU32P(allowList)))); } THROW_FMT( OptionInvalidValueError, "'%s' is not allowed for '%s' option\n" "HINT: allowed values are %s", strZ(valueAllow), cfgParseOptionKeyIdxName(optionId, optionKeyIdx), strZ(hintList)); } } } } // Else set source when negated (value is already false) else if (parseOptionValue->negate) { configOptionValue->source = parseOptionValue->source; } if ((!configOptionValue->set && !parseOptionValue->negate) || config->help) { // If the option has a default if (cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeDefault, config->command, optionId)) { if (!configOptionValue->set) { configOptionValue->set = true; configOptionValue->value = optionalRules.defaultValue; configOptionValue->sequence = optionalRules.defaultSequence; configOptionValue->display = optionalRules.defaultRaw; } configOptionValue->defaultValue = optionalRules.defaultRaw; } // Else error if option is required and help was not requested else if (!config->help) { const bool required = cfgParseOptionalRule(&optionalRules, parseRuleOptionalTypeRequired, config->command, optionId) ? optionalRules.required : ruleOption->required; if (required) { THROW_FMT( OptionRequiredError, "%s command requires option: %s%s", cfgParseCommandName(config->command), cfgParseOptionKeyIdxName(optionId, optionKeyIdx), ruleOption->section == cfgSectionStanza ? "\nHINT: does this stanza exist?" : ""); } } } // If a non-default group option, keep the group index if (optionGroup && configOptionValue->source != cfgSourceDefault) optionGroupIndexKeep[optionGroupId][optionListIdx] = true; } // Else apply the default for the unresolved dependency, if it exists else if (dependResult.defaultExists) { // Fully reinitialize since it might have been left partially set if dependency was not resolved *configOptionValue = (ConfigOptionValue) { .set = true, .value = dependResult.defaultValue, .defaultValue = optionalRules.defaultRaw, .display = optionalRules.defaultRaw, }; } pckReadFree(optionalRules.pack); } } // Phase 6: Remove any group indexes that have all default values (unless there is only one) // // It is possible that a group index was created for an option that was later found to not meet dependencies. In this case // all values will be default leading to a phantom group, which can be quite confusing. Remove all group indexes that are // all default (except the final one) and make sure the key for the final all default group index is 1. // ------------------------------------------------------------------------------------------------------------------------- for (unsigned int optionGroupIdx = 0; optionGroupIdx < CFG_OPTION_GROUP_TOTAL; optionGroupIdx++) { ConfigOptionGroupData *const optionGroup = &config->optionGroup[optionGroupIdx]; // Iterate group indexes for (unsigned int keyIdx = optionGroup->indexTotal - 1; keyIdx + 1 > 0; keyIdx--) { // Break if there is only one index since each group must have at least one if (optionGroup->indexTotal == 1) break; // If the group index does not have a non-default value if (!optionGroupIndexKeep[optionGroupIdx][keyIdx]) { // Remove the value if it is not last if (keyIdx < optionGroup->indexTotal - 1) { // Remove index key memmove( optionGroup->indexMap + keyIdx, optionGroup->indexMap + (keyIdx + 1), sizeof(unsigned int) * (optionGroup->indexTotal - keyIdx - 1)); // Iterate all options for (unsigned int optionIdx = 0; optionIdx < CFG_OPTION_TOTAL; optionIdx++) { ConfigOptionData *const option = &config->option[optionIdx]; // Remove the value if in the correct group if (option->group && option->groupId == optionGroupIdx) { memmove( option->index + keyIdx, option->index + (keyIdx + 1), sizeof(ConfigOptionValue) * (optionGroup->indexTotal - keyIdx - 1)); } } } // Decrement index total optionGroup->indexTotal--; } } // If the remaining index contains all default values and is not key 1 then make it key 1. This prevents the key from // being determined by the key of an unused option. if (optionGroup->indexTotal == 1 && !optionGroupIndexKeep[optionGroupIdx][0] && optionGroup->indexMap[0] != 0) optionGroup->indexMap[0] = 0; } // Initialize config cfgInit(config); // Set option group default index. The first index in the group is automatically set unless the group option, e.g. pg, is // set. For now the group default options are hard-coded but they could be dynamic. An assert has been added to make sure // the code breaks if a new group is added. for (unsigned int groupId = 0; groupId < CFG_OPTION_GROUP_TOTAL; groupId++) { ASSERT(groupId == cfgOptGrpPg || groupId == cfgOptGrpRepo); // Get the group default option const unsigned int defaultOptionId = groupId == cfgOptGrpPg ? cfgOptPg : cfgOptRepo; // Does a default always exist? config->optionGroup[groupId].indexDefaultExists = // A default always exists for the pg group groupId == cfgOptGrpPg || // The repo group allows a default when the repo option is valid, i.e. either repo1 is the only key set or a repo // is specified cfgOptionValid(cfgOptRepo); // Does the group default option exist? if (cfgOptionTest(defaultOptionId)) { // Search for the key const unsigned int optionKeyIdx = cfgOptionUInt(defaultOptionId) - 1; unsigned int index = 0; for (; index < cfgOptionGroupIdxTotal(groupId); index++) { if (config->optionGroup[groupId].indexMap[index] == optionKeyIdx) break; } // Error if the key was not found if (index == cfgOptionGroupIdxTotal(groupId)) { THROW_FMT( OptionInvalidValueError, "key '%s' is not valid for '%s' option", strZ(cfgOptionDisplay(defaultOptionId)), cfgOptionName(defaultOptionId)); } // Set the default config->optionGroup[groupId].indexDefault = index; config->optionGroup[groupId].indexDefaultExists = true; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/parse.h0000664000175100017510000001644215240547630016741 0ustar00runnerrunner/*********************************************************************************************************************************** Parse Configuration ***********************************************************************************************************************************/ #ifndef CONFIG_PARSE_H #define CONFIG_PARSE_H #include "common/ini.h" #include "config/config.h" #include "storage/storage.h" /*********************************************************************************************************************************** Define global section name ***********************************************************************************************************************************/ #define CFGDEF_SECTION_GLOBAL "global" /*********************************************************************************************************************************** Prefix for environment variables ***********************************************************************************************************************************/ #define PGBACKREST_ENV "PGBACKREST_" #define PGBACKREST_ENV_SIZE (sizeof(PGBACKREST_ENV) - 1) /*********************************************************************************************************************************** In some environments this will not be extern'd ***********************************************************************************************************************************/ extern char **environ; /*********************************************************************************************************************************** Option type enum ***********************************************************************************************************************************/ typedef enum { cfgOptTypeInteger, // Signed 64-bit integer cfgOptTypeSize, // Size, e.g. 1m, 2gb cfgOptTypeTime, // Time in seconds, e.g. 23, 1.5 cfgOptTypeStringId, // StringId cfgOptTypeString, // String cfgOptTypePath, // Path string with validation cfgOptTypeBoolean, // Boolean cfgOptTypeHash, // Associative array, e.g. key1=val1,key2=val2 cfgOptTypeList, // String list, e.g. val1,val2 } ConfigOptionType; /*********************************************************************************************************************************** Underlying data type for an option ***********************************************************************************************************************************/ typedef enum { cfgOptDataTypeBoolean, // Boolean cfgOptDataTypeHash, // Hash cfgOptDataTypeInteger, // Signed 64-bit integer cfgOptDataTypeList, // List cfgOptDataTypeString, // String cfgOptDataTypeStringId, // StringId } ConfigOptionDataType; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Parse the command-line arguments and config file to produce final config data typedef struct CfgParseParam { VAR_PARAM_HEADER; bool noResetLogLevel; // Do not reset log level bool noConfigLoad; // Do not reload the config file const String *stanza; // Load config as stanza } CfgParseParam; #define cfgParseP(storage, argListSize, argList, ...) \ cfgParse(storage, argListSize, argList, (CfgParseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void cfgParse(const Storage *storage, unsigned int argListSize, const char *argList[], CfgParseParam param); // Get command name by id FN_EXTERN const char *cfgParseCommandName(ConfigCommand commandId); // Get command/role name with custom separator FN_EXTERN String *cfgParseCommandRoleName(const ConfigCommand commandId, const ConfigCommandRole commandRoleId); // Convert command role enum to String FN_EXTERN const String *cfgParseCommandRoleStr(ConfigCommandRole commandRole); // Return the parsed ini config FN_EXTERN const Ini *cfgParseIni(void); // Parse option name and return option info typedef struct CfgParseOptionParam { VAR_PARAM_HEADER; bool prefixMatch; // Allow prefix matches, e.g. 'stanz' for 'stanza' bool ignoreMissingIndex; // Help requires the base option name to be searchable } CfgParseOptionParam; typedef struct CfgParseOptionResult { bool found; // Was the option found? ConfigOption id; // Option ID unsigned int keyIdx; // Option key index, i.e. option key - 1 bool negate; // Was the option negated? bool reset; // Was the option reset? bool deprecated; // Is the option deprecated? bool beta; // Is the option in beta? bool multi; // Can the option be specified multiple times? } CfgParseOptionResult; #define cfgParseOptionP(optionName, ...) \ cfgParseOption(optionName, (CfgParseOptionParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN CfgParseOptionResult cfgParseOption(const String *const optionName, const CfgParseOptionParam param); // Option name from id FN_EXTERN const char *cfgParseOptionName(ConfigOption optionId); // Option name from id and key FN_EXTERN const char *cfgParseOptionKeyIdxName(ConfigOption optionId, unsigned int keyIdx); // Does the option need to be protected from showing up in logs, command lines, etc? FN_EXTERN bool cfgParseOptionSecure(ConfigOption optionId); // Option data type FN_EXTERN ConfigOptionType cfgParseOptionType(ConfigOption optionId); // Get the underlying data type for an option FN_EXTERN ConfigOptionDataType cfgParseOptionDataType(ConfigOption optionId); // Get list of stanzas in the configuration FN_EXTERN StringList *cfgParseStanzaList(void); // Is the option valid for the command? FN_EXTERN bool cfgParseOptionValid(ConfigCommand commandId, ConfigCommandRole commandRoleId, ConfigOption optionId); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/protocol.c0000664000175100017510000000511615240547630017457 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Protocol Handler ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/json.h" #include "config/config.intern.h" #include "config/parse.h" #include "config/protocol.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * configOptionProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { VariantList *const optionList = varLstNew(); while (pckReadNext(param)) { CfgParseOptionResult option = cfgParseOptionP(pckReadStrP(param)); CHECK(AssertError, option.found, "option not found"); varLstAdd(optionList, varDup(cfgOptionIdxVar(option.id, cfgOptionKeyToIdx(option.id, option.keyIdx + 1)))); } pckWriteStrP(protocolServerResultData(result), jsonFromVar(varNewVarLst(optionList))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN VariantList * configOptionRemote(ProtocolClient *const client, const VariantList *const paramList) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_PARAM(VARIANT_LIST, paramList); FUNCTION_LOG_END(); VariantList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const param = protocolPackNew(); for (unsigned int paramIdx = 0; paramIdx < varLstSize(paramList); paramIdx++) pckWriteStrP(param, varStr(varLstGet(paramList, paramIdx))); const VariantList *const list = varVarLst( jsonToVar(pckReadStrP(protocolClientRequestP(client, PROTOCOL_COMMAND_CONFIG_OPTION, .param = param)))); MEM_CONTEXT_PRIOR_BEGIN() { result = varLstDup(list); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(VARIANT_LIST, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/config/protocol.h0000664000175100017510000000314215240547630017461 0ustar00runnerrunner/*********************************************************************************************************************************** Configuration Protocol Handler ***********************************************************************************************************************************/ #ifndef CONFIG_PROTOCOL_H #define CONFIG_PROTOCOL_H #include "common/type/pack.h" #include "protocol/client.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process config protocol requests FN_EXTERN ProtocolServerResult *configOptionProtocol(PackRead *param); // Get option values from a remote process FN_EXTERN VariantList *configOptionRemote(ProtocolClient *client, const VariantList *paramList); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_CONFIG_OPTION STRID5("opt-g", 0x7dd20f0) #define PROTOCOL_SERVER_HANDLER_OPTION_LIST \ {.command = PROTOCOL_COMMAND_CONFIG_OPTION, .process = configOptionProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/db/0000775000175100017510000000000015240547630014567 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/db/db.c0000664000175100017510000007345515240547630015336 0ustar00runnerrunner/*********************************************************************************************************************************** Database Client ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/type/json.h" #include "common/wait.h" #include "config/config.h" #include "config/protocol.h" #include "db/db.h" #include "db/protocol.h" #include "postgres/interface.h" #include "postgres/version.h" #include "protocol/helper.h" #include "version.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define PG_BACKUP_ADVISORY_LOCK "12340078987004321" #define DB_PING_SEC 30 /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Db { DbPub pub; // Publicly accessible variables PgClient *client; // Local PostgreSQL client ProtocolClient *remoteClient; // Protocol client for remote db queries ProtocolClientSession *session; // Protocol session for requests const Storage *storage; // PostgreSQL storage const String *applicationName; // Used to identify this connection in PostgreSQL time_t pingTimeLast; // Last time cluster was pinged }; /*********************************************************************************************************************************** Close protocol connection. No need to close a locally created PgClient since it has its own destructor. ***********************************************************************************************************************************/ static void dbFreeResource(THIS_VOID) { THIS(Db); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); ASSERT(this != NULL); protocolClientSessionFree(this->session); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Db * dbNew(PgClient *const client, ProtocolClient *const remoteClient, const Storage *const storage, const String *const applicationName) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PG_CLIENT, client); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, remoteClient); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, applicationName); FUNCTION_LOG_END(); ASSERT((client != NULL && remoteClient == NULL) || (client == NULL && remoteClient != NULL)); ASSERT(storage != NULL); ASSERT(applicationName != NULL); OBJ_NEW_BEGIN(Db, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (Db) { .pub = { .memContext = memContextCurrent(), }, .remoteClient = remoteClient, .session = remoteClient != NULL ? protocolClientSessionNewP(remoteClient, PROTOCOL_COMMAND_DB) : NULL, .storage = storage, .applicationName = strDup(applicationName), }; this->client = pgClientMove(client, this->pub.memContext); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(DB, this); } /*********************************************************************************************************************************** Execute a query ***********************************************************************************************************************************/ static Pack * dbQuery(Db *const this, const PgClientQueryResult resultType, const String *const query) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING_ID, resultType); FUNCTION_LOG_PARAM(STRING, query); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(resultType != 0); ASSERT(query != NULL); Pack *result; // Query remotely if (this->remoteClient != NULL) { MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const param = protocolPackNew(); pckWriteStrIdP(param, resultType); pckWriteStrP(param, query); PackRead *const read = protocolClientSessionRequestP(this->session, .param = param); MEM_CONTEXT_PRIOR_BEGIN() { result = pckReadPackP(read); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } // Else locally else result = pgClientQuery(this->client, query, resultType); FUNCTION_LOG_RETURN(PACK, result); } /*********************************************************************************************************************************** Execute a command that expects no output ***********************************************************************************************************************************/ static void dbExec(Db *const this, const String *const command) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING, command); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(command != NULL); CHECK(AssertError, dbQuery(this, pgClientQueryResultNone, command) == NULL, "exec returned data"); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Execute a query that returns a single row and column ***********************************************************************************************************************************/ static PackRead * dbQueryColumn(Db *const this, const String *const query) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING, query); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(query != NULL); Pack *const pack = dbQuery(this, pgClientQueryResultColumn, query); PackRead *const result = pckReadNew(pack); pckMove(pack, objMemContext(result)); FUNCTION_LOG_RETURN(PACK_READ, result); } /*********************************************************************************************************************************** Execute a query that returns a single row ***********************************************************************************************************************************/ static PackRead * dbQueryRow(Db *const this, const String *const query) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING, query); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(query != NULL); Pack *const pack = dbQuery(this, pgClientQueryResultRow, query); PackRead *const result = pckReadNew(pack); pckMove(pack, objMemContext(result)); FUNCTION_LOG_RETURN(PACK_READ, result); } /*********************************************************************************************************************************** Is the cluster in recovery? ***********************************************************************************************************************************/ static bool dbIsInRecovery(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result; MEM_CONTEXT_TEMP_BEGIN() { result = pckReadBoolP(dbQueryColumn(this, STRDEF("select pg_catalog.pg_is_in_recovery()"))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN void dbOpen(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Open the connection if (this->remoteClient != NULL) { protocolClientSessionOpenP(this->session); // Set a callback to notify the remote when a connection is closed memContextCallbackSet(this->pub.memContext, dbFreeResource, this); // Get db-timeout from the remote since it might be different than the local value this->pub.dbTimeout = varUInt64Force( varLstGet(configOptionRemote(this->remoteClient, varLstAdd(varLstNew(), varNewStrZ(CFGOPT_DB_TIMEOUT))), 0)); } else { pgClientOpen(this->client); this->pub.dbTimeout = cfgOptionUInt64(cfgOptDbTimeout); } // Set search_path to prevent overrides of the functions we expect to call. All queries should also be schema-qualified, // but this is an extra level protection. dbExec(this, STRDEF("set search_path = 'pg_catalog'")); // Set client encoding to UTF8. This is the only encoding (other than ASCII) that we can safely work with. dbExec(this, STRDEF("set client_encoding = 'UTF8'")); // Query the version and data_directory. Be sure the update the total in the null check below when adding/removing columns. const Pack *const row = dbQuery( this, pgClientQueryResultRow, STRDEF( "select (select setting from pg_catalog.pg_settings where name = 'server_version_num')::int4," " (select setting from pg_catalog.pg_settings where name = 'data_directory')::text," " (select setting from pg_catalog.pg_settings where name = 'archive_mode')::text," " (select setting from pg_catalog.pg_settings where name = 'archive_command')::text," " (select setting from pg_catalog.pg_settings where name = 'checkpoint_timeout')::int4")); // Check that none of the return values are null, which indicates the user cannot select some rows in pg_settings PackRead *read = pckReadNew(row); for (unsigned int columnIdx = 0; columnIdx < 5; columnIdx++) { if (pckReadNullP(read, .id = columnIdx + 1)) { THROW( DbQueryError, "unable to select some rows from pg_settings\n" "HINT: is the backup running as the postgres user?\n" "HINT: is the pg_read_all_settings role assigned for " PG_NAME " >= " PG_VERSION_10_Z "?"); } } // Restart the read to get the data read = pckReadNew(row); // Strip the minor version off since we don't need it. In the future it might be a good idea to warn users when they are // running an old minor version. this->pub.pgVersion = (unsigned int)pckReadI32P(read) / 100 * 100; // Store the data directory that PostgreSQL is running in, the archive mode, and archive command. These can be compared to // the configured pgBackRest directory, and archive settings checked for validity, when validating the configuration. // Also store the checkpoint timeout to warn in case a backup is requested without using the start-fast option. MEM_CONTEXT_BEGIN(this->pub.memContext) { this->pub.pgDataPath = pckReadStrP(read); this->pub.archiveMode = pckReadStrP(read); this->pub.archiveCommand = pckReadStrP(read); this->pub.checkpointTimeout = (TimeMSec)pckReadI32P(read) * MSEC_PER_SEC; } MEM_CONTEXT_END(); // Set application name to help identify the backup session dbExec(this, strNewFmt("set application_name = '%s'", strZ(this->applicationName))); // There is no need to have parallelism enabled in a backup session. In particular, 9.6 marks pg_stop_backup() as // parallel-safe but an error will be thrown if pg_stop_backup() is run in a worker. dbExec(this, STRDEF("set max_parallel_workers_per_gather = 0")); // Is the cluster a standby? this->pub.standby = dbIsInRecovery(this); // Get control file this->pub.pgControl = pgControlFromFile(this->storage, cfgOptionStrNull(cfgOptPgVersionForce)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to build start backup query static String * dbBackupStartQuery(const unsigned int pgVersion, const bool startFast) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_PARAM(BOOL, startFast); FUNCTION_TEST_END(); // Build query to return start lsn and WAL segment name String *result = strCatFmt( strNew(), "select lsn::text as lsn,\n" " pg_catalog.pg_%sfile_name(lsn)::text as wal_segment_name\n" " from pg_catalog.pg_%s('" PROJECT_NAME " backup started at ' || current_timestamp", strZ(pgWalName(pgVersion)), pgVersion >= PG_VERSION_15 ? "backup_start" : "start_backup"); // Start backup after immediate checkpoint if (startFast) { strCatZ(result, ", " TRUE_Z); } // Else start backup at the next scheduled checkpoint else strCatZ(result, ", " FALSE_Z); // Use non-exclusive backup mode when available if (pgVersion <= PG_VERSION_14) strCatZ(result, ", " FALSE_Z); // Complete query strCatZ(result, ") as lsn"); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN DbBackupStartResult dbBackupStart(Db *const this, const bool startFast, const bool archiveCheck) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(BOOL, startFast); FUNCTION_LOG_PARAM(BOOL, archiveCheck); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); DbBackupStartResult result = {.lsn = NULL}; MEM_CONTEXT_TEMP_BEGIN() { // When the start-fast option is disabled and db-timeout is smaller than checkpoint_timeout, the command may timeout // before the backup actually starts if (!startFast && dbDbTimeout(this) <= dbCheckpointTimeout(this)) { LOG_WARN_FMT( CFGOPT_START_FAST " is disabled and " CFGOPT_DB_TIMEOUT " (%" PRIu64 "s) is smaller than the " PG_NAME " checkpoint_timeout (%" PRIu64 "s) - timeout may occur before the backup starts", dbDbTimeout(this) / MSEC_PER_SEC, dbCheckpointTimeout(this) / MSEC_PER_SEC); } // If archive check then get the current WAL segment const String *walSegmentCheck = NULL; if (archiveCheck) { walSegmentCheck = pckReadStrP( dbQueryColumn( this, strNewFmt( "select pg_catalog.pg_%sfile_name(pg_catalog.pg_current_%s_insert_%s())::text", strZ(pgWalName(dbPgVersion(this))), strZ(pgWalName(dbPgVersion(this))), strZ(pgLsnName(dbPgVersion(this)))))); } // Start backup PackRead *const read = dbQueryRow(this, dbBackupStartQuery(dbPgVersion(this), startFast)); // Make sure the backup start checkpoint was written to pg_control. This helps ensure that we have a consistent view of the // storage with PostgreSQL. const PgControl pgControl = pgControlFromFile(this->storage, cfgOptionStrNull(cfgOptPgVersionForce)); const String *const lsnStart = pckReadStrP(read); if (pgControl.checkpoint < pgLsnFromStr(lsnStart)) { THROW_FMT( DbMismatchError, "current checkpoint '%s' is less than backup start '%s'", strZ(pgLsnToStr(pgControl.checkpoint)), strZ(lsnStart)); } // If archive check then make sure WAL segment was switched on start backup. The WAL segment might not have been switched if // a switch occurred before the backup and nothing was written since then. In this case, force a switch so we can verify // that archiving is functional before starting the backup. const String *const walSegmentName = pckReadStrP(read); if (archiveCheck && strEq(walSegmentCheck, walSegmentName)) dbWalSwitch(this); // Check that the WAL timeline matches what is in pg_control if (pgTimelineFromWalSegment(walSegmentName) != dbPgControl(this).timeline) { THROW_FMT( DbMismatchError, "WAL timeline %u does not match " PG_FILE_PGCONTROL " timeline %u", pgTimelineFromWalSegment(walSegmentName), dbPgControl(this).timeline); } // Return results MEM_CONTEXT_PRIOR_BEGIN() { result.lsn = strDup(lsnStart); result.walSegmentName = strDup(walSegmentName); result.walSegmentCheck = strDup(walSegmentCheck); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ // Helper to build stop backup query static String * dbBackupStopQuery(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); // Build query to return start lsn and WAL segment name String *const result = strCatFmt( strNew(), "select lsn::text as lsn,\n" " pg_catalog.pg_%sfile_name(lsn)::text as wal_segment_name,\n" " labelfile::text as backuplabel_file,\n" " spcmapfile::text as tablespacemap_file\n" " from pg_catalog.pg_%s(", strZ(pgWalName(pgVersion)), pgVersion >= PG_VERSION_15 ? "backup_stop" : "stop_backup"); // Use non-exclusive backup mode when available if (pgVersion <= PG_VERSION_14) strCatZ(result, FALSE_Z); // Disable archive checking since we do this elsewhere if (pgVersion >= PG_VERSION_10) { if (pgVersion <= PG_VERSION_14) strCatZ(result, ", "); strCatZ(result, FALSE_Z); } // Complete query strCatZ(result, ")"); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN DbBackupStopResult dbBackupStop(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); DbBackupStopResult result = {.lsn = NULL}; MEM_CONTEXT_TEMP_BEGIN() { // Stop backup PackRead *const read = dbQueryRow(this, dbBackupStopQuery(dbPgVersion(this))); // Return results MEM_CONTEXT_PRIOR_BEGIN() { result.lsn = pckReadStrP(read); result.walSegmentName = pckReadStrP(read); result.backupLabel = pckReadStrP(read); // Return the tablespace map if it is not empty String *const tablespaceMap = pckReadStrP(read); String *const tablespaceMapTrim = strTrim(strDup(tablespaceMap)); if (!strEmpty(tablespaceMapTrim)) result.tablespaceMap = tablespaceMap; else strFree(tablespaceMap); strFree(tablespaceMapTrim); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN Pack * dbList(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN( PACK, dbQuery( this, pgClientQueryResultAny, STRDEF( "select oid::oid, datname::text, (select oid::oid from pg_catalog.pg_database where datname = 'template0')" " as datlastsysoid from pg_catalog.pg_database"))); } /**********************************************************************************************************************************/ FN_EXTERN void dbReplayWait(Db *const this, const String *const targetLsn, const uint32_t targetTimeline, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(STRING, targetLsn); FUNCTION_LOG_PARAM(UINT, targetTimeline); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(targetLsn != NULL); ASSERT(targetTimeline != 0); ASSERT(timeout > 0); MEM_CONTEXT_TEMP_BEGIN() { // Standby checkpoint before the backup started must be <= the target LSN. If not, it indicates that the standby was ahead // of the primary and cannot be following it. if (dbPgControl(this).checkpoint > pgLsnFromStr(targetLsn)) { THROW_FMT( DbMismatchError, "standby checkpoint '%s' is ahead of target '%s'", strZ(pgLsnToStr(dbPgControl(this).checkpoint)), strZ(targetLsn)); } // Loop until lsn has been reached or timeout Wait *wait = waitNew(timeout); bool targetReached = false; const char *const lsnName = strZ(pgLsnName(dbPgVersion(this))); const String *const replayLsnFunction = strNewFmt( "pg_catalog.pg_last_%s_replay_%s()", strZ(pgWalName(dbPgVersion(this))), lsnName); const String *replayLsn = NULL; do { // Build the query String *const query = strCatFmt( strNew(), "select replayLsn::text,\n" " (replayLsn > '%s')::bool as targetReached", strZ(targetLsn)); if (replayLsn != NULL) { strCatFmt( query, ",\n" " (replayLsn > '%s')::bool as replayProgress", strZ(replayLsn)); } strCatFmt( query, "\n" " from %s as replayLsn", strZ(replayLsnFunction)); // Execute the query and get replayLsn PackRead *const read = dbQueryRow(this, query); replayLsn = pckReadStrP(read); // Error when replayLsn is null which indicates that this is not a standby. This should have been sorted out before we // connected but it's possible that the standby was promoted in the meantime. if (replayLsn == NULL) { THROW_FMT( ArchiveTimeoutError, "unable to query replay lsn on the standby using '%s'\n" "HINT: Is this a standby?", strZ(replayLsnFunction)); } targetReached = pckReadBoolP(read); // If the target has not been reached but progress is being made then reset the timer if (!targetReached && pckReadBoolP(read, .defaultValue = true)) wait = waitNew(timeout); protocolKeepAlive(); } while (!targetReached && waitMore(wait)); // Error if a timeout occurred before the target lsn was reached if (!targetReached) { THROW_FMT( ArchiveTimeoutError, "timeout before standby replayed to %s - only reached %s\n" "HINT: is replication running and current on the standby?\n" "HINT: disable the 'backup-standby' option to backup directly from the primary.", strZ(targetLsn), strZ(replayLsn)); } // Perform a checkpoint dbExec(this, STRDEF("checkpoint")); // Loop until the checkpoint has reached the target lsn (or timeout) wait = waitNew(timeout); targetReached = false; const String *checkpointLsn = NULL; do { // Build the query const String *const query = strNewFmt( "select (checkpoint_%s >= '%s')::bool as targetReached,\n" " checkpoint_%s::text as checkpointLsn\n" " from pg_catalog.pg_control_checkpoint()", lsnName, strZ(targetLsn), lsnName); // Execute the query and get checkpointLsn PackRead *const read = dbQueryRow(this, query); targetReached = pckReadBoolP(read); checkpointLsn = pckReadStrP(read); protocolKeepAlive(); } while (!targetReached && waitMore(wait)); // Error if a timeout occurred before the target lsn was reached if (!targetReached) { THROW_FMT( ArchiveTimeoutError, "timeout before standby checkpoint lsn reached %s - only reached %s", strZ(targetLsn), strZ(checkpointLsn)); } // Reload pg_control in case timeline was updated by the checkpoint this->pub.pgControl = pgControlFromFile(this->storage, cfgOptionStrNull(cfgOptPgVersionForce)); // Check that the timeline matches the primary if (dbPgControl(this).timeline != targetTimeline) THROW_FMT(DbMismatchError, "standby is on timeline %u but expected %u", dbPgControl(this).timeline, targetTimeline); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void dbPing(Db *const this, const bool force) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_PARAM(BOOL, force); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Ping if forced or interval has elapsed const time_t timeNow = time(NULL); if (force || timeNow - this->pingTimeLast > DB_PING_SEC) { // Make sure recovery state has not changed if (dbIsInRecovery(this) != dbIsStandby(this)) { // If this is the standby then it must have been promoted if (dbIsStandby(this)) { THROW( DbMismatchError, "standby is no longer in recovery\n" "HINT: was the standby promoted during the backup?"); } // Else if a primary then something has gone seriously wrong else THROW(AssertError, "primary has switched to recovery"); } this->pingTimeLast = timeNow; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Pack * dbTablespaceList(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN( PACK, dbQuery(this, pgClientQueryResultAny, STRDEF("select oid::oid, spcname::text from pg_catalog.pg_tablespace"))); } /**********************************************************************************************************************************/ FN_EXTERN TimeMSec dbTimeMSec(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); TimeMSec result; MEM_CONTEXT_TEMP_BEGIN() { result = (TimeMSec)pckReadI64P( dbQueryColumn(this, STRDEF("select (extract(epoch from clock_timestamp()) * 1000)::bigint"))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(TIME_MSEC, result); } /**********************************************************************************************************************************/ FN_EXTERN String * dbWalSwitch(Db *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(DB, this); FUNCTION_LOG_END(); ASSERT(this != NULL); String *result; MEM_CONTEXT_TEMP_BEGIN() { // Create a restore point to ensure current WAL will be archived dbQueryColumn(this, STRDEF("select pg_catalog.pg_create_restore_point('" PROJECT_NAME " Archive Check')::text")); // Request a WAL segment switch const char *const walName = strZ(pgWalName(dbPgVersion(this))); const String *const walFileName = pckReadStrP( dbQueryColumn(this, strNewFmt("select pg_catalog.pg_%sfile_name(pg_catalog.pg_switch_%s())::text", walName, walName))); // Copy WAL segment name to the prior context MEM_CONTEXT_PRIOR_BEGIN() { result = strDup(walFileName); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void dbToLog(const Db *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{client: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT(this->client, pgClientToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", remoteClient: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT( this->remoteClient, protocolClientToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/db/db.h0000664000175100017510000001523115240547630015327 0ustar00runnerrunner/*********************************************************************************************************************************** Database Client Implements the required PostgreSQL queries and commands. Notice that there is no general purpose query function -- all queries are expected to be embedded in this object. ***********************************************************************************************************************************/ #ifndef DB_DB_H #define DB_DB_H #include "common/type/object.h" #include "postgres/client.h" #include "postgres/interface.h" #include "protocol/client.h" #include "storage/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Db Db; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Db *dbNew(PgClient *client, ProtocolClient *remoteClient, const Storage *storage, const String *applicationName); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct DbPub { MemContext *memContext; // Mem context PgControl pgControl; // Control info const String *archiveMode; // The archive_mode reported by the database const String *archiveCommand; // The archive_command reported by the database const String *pgDataPath; // Data directory reported by the database bool standby; // Is the cluster a standby? unsigned int pgVersion; // Version as reported by the database TimeMSec checkpointTimeout; // The checkpoint timeout reported by the database TimeMSec dbTimeout; // Database timeout for statements/queries from PG client } DbPub; // Archive mode loaded from the archive_mode GUC FN_INLINE_ALWAYS const String * dbArchiveMode(const Db *const this) { return THIS_PUB(Db)->archiveMode; } // Archive command loaded from the archive_command GUC FN_INLINE_ALWAYS const String * dbArchiveCommand(const Db *const this) { return THIS_PUB(Db)->archiveCommand; } // Control data FN_INLINE_ALWAYS PgControl dbPgControl(const Db *const this) { return THIS_PUB(Db)->pgControl; } // Data path loaded from the data_directory GUC FN_INLINE_ALWAYS const String * dbPgDataPath(const Db *const this) { return THIS_PUB(Db)->pgDataPath; } // Is the cluster a standby? FN_INLINE_ALWAYS bool dbIsStandby(const Db *const this) { return THIS_PUB(Db)->standby; } // Version loaded from the server_version_num GUC FN_INLINE_ALWAYS unsigned int dbPgVersion(const Db *const this) { return THIS_PUB(Db)->pgVersion; } // Checkpoint timeout loaded from the checkpoint_timeout GUC FN_INLINE_ALWAYS TimeMSec dbCheckpointTimeout(const Db *const this) { return THIS_PUB(Db)->checkpointTimeout; } // Database timeout from main/remote process FN_INLINE_ALWAYS TimeMSec dbDbTimeout(const Db *const this) { return THIS_PUB(Db)->dbTimeout; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open the db connection FN_EXTERN void dbOpen(Db *this); // Start backup and return starting lsn and wal segment name typedef struct DbBackupStartResult { String *lsn; String *walSegmentName; String *walSegmentCheck; // Segment used to check archiving, may be NULL } DbBackupStartResult; FN_EXTERN DbBackupStartResult dbBackupStart(Db *this, bool startFast, bool archiveCheck); // Stop backup and return starting lsn, wal segment name, backup label, and tablespace map typedef struct DbBackupStopResult { String *lsn; String *walSegmentName; String *backupLabel; String *tablespaceMap; } DbBackupStopResult; FN_EXTERN DbBackupStopResult dbBackupStop(Db *this); // Get list of databases in the cluster: select oid, datname, datlastsysoid from pg_database FN_EXTERN Pack *dbList(Db *this); // Waits for replay on the standby to equal the target LSN FN_EXTERN void dbReplayWait(Db *this, const String *targetLsn, uint32_t targetTimeline, TimeMSec timeout); // Check that the cluster is alive and correctly configured during the backup FN_EXTERN void dbPing(Db *this, bool force); // Epoch time on the PostgreSQL host in ms FN_EXTERN TimeMSec dbTimeMSec(Db *this); // Get list of tablespaces in the cluster: select oid, datname, datlastsysoid from pg_database FN_EXTERN Pack *dbTablespaceList(Db *this); // Switch the WAL segment and return the segment that should have been archived FN_EXTERN String *dbWalSwitch(Db *this); // Move to a new parent mem context FN_INLINE_ALWAYS Db * dbMove(Db *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void dbFree(Db *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void dbToLog(const Db *this, StringStatic *debugLog); #define FUNCTION_LOG_DB_TYPE \ Db * #define FUNCTION_LOG_DB_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, dbToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/db/helper.c0000664000175100017510000001164315240547630016217 0ustar00runnerrunner/*********************************************************************************************************************************** Database Helper ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "config/config.h" #include "db/helper.h" #include "postgres/interface.h" #include "protocol/helper.h" #include "storage/helper.h" #include "version.h" /**********************************************************************************************************************************/ // Helper to get a connection to the specified pg cluster static Db * dbGetIdx(const unsigned int pgIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgIdx); FUNCTION_LOG_END(); ASSERT(pgIdx < cfgOptionGroupIdxTotal(cfgOptGrpPg)); Db *result; MEM_CONTEXT_TEMP_BEGIN() { const String *applicationName = strNewFmt(PROJECT_NAME " [%s]", cfgCommandName()); if (pgIsLocal(pgIdx)) { result = dbNew( pgClientNew( cfgOptionIdxStrNull(cfgOptPgSocketPath, pgIdx), cfgOptionIdxUInt(cfgOptPgPort, pgIdx), cfgOptionIdxStr(cfgOptPgDatabase, pgIdx), cfgOptionIdxStrNull(cfgOptPgUser, pgIdx), cfgOptionUInt64(cfgOptDbTimeout)), NULL, storagePgIdx(pgIdx), applicationName); } else result = dbNew(NULL, protocolRemoteGet(protocolStorageTypePg, pgIdx, true), storagePgIdx(pgIdx), applicationName); dbMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(DB, result); } FN_EXTERN DbGetResult dbGet(const bool primaryOnly, const bool primaryRequired, const StringId standbyRequired) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BOOL, primaryOnly); FUNCTION_LOG_PARAM(BOOL, primaryRequired); FUNCTION_LOG_PARAM(STRING_ID, standbyRequired); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(!(primaryOnly && standbyRequired == CFGOPTVAL_BACKUP_STANDBY_Y)); DbGetResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Loop through to look for primary and standby (if required) for (unsigned int pgIdx = 0; pgIdx < cfgOptionGroupIdxTotal(cfgOptGrpPg); pgIdx++) { Db *db = NULL; bool standby = false; TRY_BEGIN() { db = dbGetIdx(pgIdx); // This needs to be nested because db can be reset to NULL on an error in the outer try but we need the pointer // to be able to free it. TRY_BEGIN() { dbOpen(db); standby = dbIsStandby(db); } CATCH_ANY() { dbFree(db); RETHROW(); } TRY_END(); } CATCH_ANY() { LOG_WARN_FMT( "unable to check %s: [%s] %s", cfgOptionGroupName(cfgOptGrpPg, pgIdx), errorTypeName(errorType()), errorMessage()); db = NULL; } TRY_END(); // Was the connection successful if (db != NULL) { // Is this cluster a standby if (standby) { // If a standby has not already been found then assign it if (result.standby == NULL && !primaryOnly) { result.standbyIdx = pgIdx; result.standby = db; } // Else close the connection since we don't need it else dbFree(db); } // Else is a primary else { // Error if more than one primary was found if (result.primary != NULL) THROW(DbConnectError, "more than one primary cluster found"); result.primaryIdx = pgIdx; result.primary = db; } } } // Error if no primary was found if (result.primary == NULL && primaryRequired) { THROW( DbConnectError, "unable to find primary cluster - cannot proceed\n" "HINT: are all available clusters in recovery?"); } // Error if no standby was found if (result.standby == NULL && standbyRequired == CFGOPTVAL_BACKUP_STANDBY_Y) THROW(DbConnectError, "unable to find standby cluster - cannot proceed"); dbMove(result.primary, memContextPrior()); dbMove(result.standby, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/db/helper.h0000664000175100017510000000363315240547630016224 0ustar00runnerrunner/*********************************************************************************************************************************** Database Helper Helper functions for getting connections to PostgreSQL. ***********************************************************************************************************************************/ #ifndef DB_HELPER_H #define DB_HELPER_H #include #include "db/db.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Get specified cluster(s) typedef struct DbGetResult { unsigned int primaryIdx; // cfgOptGrpPg index of the primary Db *primary; // Primary db object (NULL if none requested) unsigned int standbyIdx; // cfgOptGrpPg index of the standby Db *standby; // Standby db object (NULL if none requested) } DbGetResult; FN_EXTERN DbGetResult dbGet(bool primaryOnly, bool primaryRequired, StringId standbyRequired); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_DB_GET_RESULT_TYPE \ DbGetResult #define FUNCTION_LOG_DB_GET_RESULT_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "DbGetResult", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/db/protocol.c0000664000175100017510000000455715240547630016607 0ustar00runnerrunner/*********************************************************************************************************************************** Db Protocol Handler ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/json.h" #include "common/type/list.h" #include "config/config.h" #include "db/protocol.h" #include "postgres/client.h" #include "postgres/interface.h" /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * dbOpenProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param == NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { PgClient *const pgClient = pgClientNew( cfgOptionStrNull(cfgOptPgSocketPath), cfgOptionUInt(cfgOptPgPort), cfgOptionStr(cfgOptPgDatabase), cfgOptionStrNull(cfgOptPgUser), cfgOptionUInt64(cfgOptDbTimeout)); pgClientOpen(pgClient); // Set session data protocolServerResultSessionDataSet(result, pgClient); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * dbQueryProtocol(PackRead *const param, void *const pgClient) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_PARAM(PG_CLIENT, pgClient); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(param != NULL); ASSERT(pgClient != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { const PgClientQueryResult resultType = (PgClientQueryResult)pckReadStrIdP(param); const String *const query = pckReadStrP(param); pckWritePackP(protocolServerResultData(result), pgClientQuery(pgClient, query, resultType)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/db/protocol.h0000664000175100017510000000277015240547630016607 0ustar00runnerrunner/*********************************************************************************************************************************** Db Protocol Handler ***********************************************************************************************************************************/ #ifndef DB_PROTOCOL_H #define DB_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process db protocol requests FN_EXTERN ProtocolServerResult *dbOpenProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *dbQueryProtocol(PackRead *param, void *sessionData); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_DB STRID5("db", 0x440) #define PROTOCOL_SERVER_HANDLER_DB_LIST \ {.command = PROTOCOL_COMMAND_DB, .open = dbOpenProtocol, .processSession = dbQueryProtocol}, #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/0000775000175100017510000000000015240547630015135 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/info.c0000664000175100017510000005300515240547630016237 0ustar00runnerrunner/*********************************************************************************************************************************** Info Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/crypto/hash.h" #include "common/debug.h" #include "common/ini.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/convert.h" #include "common/type/json.h" #include "common/type/object.h" #include "info/info.h" #include "storage/helper.h" #include "version.h" /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ struct Info { InfoPub pub; // Publicly accessible variables }; struct InfoSave { MemContext *memContext; // Mem context IoWrite *write; // Write object IoFilter *checksum; // hash to generate file checksum String *sectionLast; // The last section seen }; /*********************************************************************************************************************************** Macros and buffer constants for checksum generation ***********************************************************************************************************************************/ #define INFO_CHECKSUM_BEGIN(checksum) \ do \ { \ ioFilterProcessIn(checksum, BRACEL_BUF); \ } \ while (0) BUFFER_STRDEF_STATIC(INFO_CHECKSUM_SECTION_END_BUF, "\":{"); #define INFO_CHECKSUM_SECTION(checksum, section) \ do \ { \ ioFilterProcessIn(checksum, QUOTED_BUF); \ ioFilterProcessIn(checksum, BUFSTR(section)); \ ioFilterProcessIn(checksum, INFO_CHECKSUM_SECTION_END_BUF); \ } \ while (0) BUFFER_STRDEF_STATIC(INFO_CHECKSUM_SECTION_NEXT_END_BUF, "},"); #define INFO_CHECKSUM_SECTION_NEXT(checksum) \ do \ { \ ioFilterProcessIn(checksum, INFO_CHECKSUM_SECTION_NEXT_END_BUF); \ } \ while (0) BUFFER_STRDEF_STATIC(INFO_CHECKSUM_KEY_VALUE_END_BUF, ":"); #define INFO_CHECKSUM_KEY_VALUE(checksum, key, value) \ do \ { \ ioFilterProcessIn(checksum, BUFSTR(jsonFromVar(VARSTR(key)))); \ ioFilterProcessIn(checksum, INFO_CHECKSUM_KEY_VALUE_END_BUF); \ ioFilterProcessIn(checksum, BUFSTR(value)); \ } \ while (0) #define INFO_CHECKSUM_KEY_VALUE_NEXT(checksum) \ do \ { \ ioFilterProcessIn(checksum, COMMA_BUF); \ } \ while (0) BUFFER_STRDEF_STATIC(INFO_CHECKSUM_END_BUF, "}}"); #define INFO_CHECKSUM_END(checksum) \ do \ { \ ioFilterProcessIn(checksum, INFO_CHECKSUM_END_BUF); \ } \ while (0) /**********************************************************************************************************************************/ FN_EXTERN Info * infoNew(const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_TEST_PARAM(STRING, cipherPass); // Use FUNCTION_TEST so cipher is not logged FUNCTION_LOG_END(); OBJ_NEW_BEGIN(Info, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Info){}; // Cipher used to encrypt/decrypt subsequent dependent files. Value may be NULL. infoCipherPassSet(this, cipherPass); this->pub.backrestVersion = STRDEF(PROJECT_VERSION); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO, this); } /**********************************************************************************************************************************/ #define INFO_SECTION_BACKREST "backrest" #define INFO_KEY_CHECKSUM "backrest-checksum" #define INFO_SECTION_CIPHER "cipher" #define INFO_KEY_CIPHER_PASS "cipher-pass" FN_EXTERN Info * infoNewLoad(IoRead *const read, InfoLoadNewCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(read != NULL); ASSERT(callbackFunction != NULL); ASSERT(callbackData != NULL); OBJ_NEW_BEGIN(Info, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Info){}; MEM_CONTEXT_TEMP_BEGIN() { String *const sectionLast = strNew(); // The last section seen during load IoFilter *const checksumActualFilter = cryptoHashNew(hashTypeSha1); // Checksum calculated from the file const String *checksumExpected = NULL; // Checksum found in ini file INFO_CHECKSUM_BEGIN(checksumActualFilter); TRY_BEGIN() { Ini *const ini = iniNewP(read, .strict = true); MEM_CONTEXT_TEMP_RESET_BEGIN() { const IniValue *value = iniValueNext(ini); while (value != NULL) { // Calculate checksum if (!(strEqZ(value->section, INFO_SECTION_BACKREST) && strEqZ(value->key, INFO_KEY_CHECKSUM))) { if (strEmpty(sectionLast) || !strEq(value->section, sectionLast)) { if (!strEmpty(sectionLast)) INFO_CHECKSUM_SECTION_NEXT(checksumActualFilter); INFO_CHECKSUM_SECTION(checksumActualFilter, value->section); strCat(strTrunc(sectionLast), value->section); } else INFO_CHECKSUM_KEY_VALUE_NEXT(checksumActualFilter); INFO_CHECKSUM_KEY_VALUE(checksumActualFilter, value->key, value->value); } // Process backrest section if (strEqZ(value->section, INFO_SECTION_BACKREST)) { // Validate format if (strEqZ(value->key, INFO_KEY_FORMAT)) { if (varUInt64(jsonToVar(value->value)) != REPOSITORY_FORMAT) { THROW_FMT( FormatError, "expected format %d but found %" PRIu64, REPOSITORY_FORMAT, varUInt64(jsonToVar(value->value))); } } // Store pgBackRest version else if (strEqZ(value->key, INFO_KEY_VERSION)) { MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.backrestVersion = varStr(jsonToVar(value->value)); } MEM_CONTEXT_END(); } // Store checksum to be validated later else if (strEqZ(value->key, INFO_KEY_CHECKSUM)) { MEM_CONTEXT_OBJ_BEGIN(this) { checksumExpected = varStr(jsonToVar(value->value)); } MEM_CONTEXT_END(); } } // Process cipher section else if (strEqZ(value->section, INFO_SECTION_CIPHER)) { // No validation needed for cipher-pass, just store it if (strEqZ(value->key, INFO_KEY_CIPHER_PASS)) { MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.cipherPass = varStr(jsonToVar(value->value)); } MEM_CONTEXT_END(); } } // Else pass to callback for processing else callbackFunction(callbackData, value->section, value->key, jsonReadNew(value->value)); value = iniValueNext(ini); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } CATCH(CryptoError) { THROW_FMT(CryptoError, "%s\nHINT: is or was the repo encrypted?", errorMessage()); } TRY_END(); INFO_CHECKSUM_END(checksumActualFilter); // Verify the checksum const String *const checksumActual = strNewEncode( encodingHex, pckReadBinP(pckReadNew(ioFilterResult(checksumActualFilter)))); if (checksumExpected == NULL) THROW_FMT(ChecksumError, "invalid checksum, actual '%s' but no checksum found", strZ(checksumActual)); else if (!strEq(checksumExpected, checksumActual)) { THROW_FMT( ChecksumError, "invalid checksum, actual '%s' but expected '%s'", strZ(checksumActual), strZ(checksumExpected)); } } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO, this); } /**********************************************************************************************************************************/ FN_EXTERN bool infoSaveSection(InfoSave *const infoSaveData, const char *const section, const String *const sectionNext) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_PARAM(STRINGZ, section); FUNCTION_TEST_PARAM(STRING, sectionNext); FUNCTION_TEST_END(); ASSERT(infoSaveData != NULL); ASSERT(section != NULL); FUNCTION_TEST_RETURN( BOOL, (infoSaveData->sectionLast == NULL || strCmpZ(infoSaveData->sectionLast, section) < 0) && (sectionNext == NULL || strCmpZ(sectionNext, section) > 0)); } /**********************************************************************************************************************************/ FN_EXTERN void infoSaveValue(InfoSave *const infoSaveData, const char *const section, const char *const key, const String *const jsonValue) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_PARAM(STRINGZ, section); FUNCTION_TEST_PARAM(STRINGZ, key); FUNCTION_TEST_PARAM(STRING, jsonValue); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(infoSaveData != NULL); ASSERT(section != NULL); ASSERT(key != NULL); ASSERT(jsonValue != NULL); ASSERT(strSize(jsonValue) != 0); // The JSON value must not be an array because this may be confused with a section in the ini file ASSERT(strZ(jsonValue)[0] != '['); // Save section if (infoSaveData->sectionLast == NULL || !strEqZ(infoSaveData->sectionLast, section)) { if (infoSaveData->sectionLast != NULL) { INFO_CHECKSUM_SECTION_NEXT(infoSaveData->checksum); ioWriteLine(infoSaveData->write, BUFSTRDEF("")); } INFO_CHECKSUM_SECTION(infoSaveData->checksum, STR(section)); ioWrite(infoSaveData->write, BRACKETL_BUF); ioWrite(infoSaveData->write, BUFSTRZ(section)); ioWriteLine(infoSaveData->write, BRACKETR_BUF); MEM_CONTEXT_BEGIN(infoSaveData->memContext) { infoSaveData->sectionLast = strNewZ(section); } MEM_CONTEXT_END(); } else INFO_CHECKSUM_KEY_VALUE_NEXT(infoSaveData->checksum); // Save key/value INFO_CHECKSUM_KEY_VALUE(infoSaveData->checksum, STR(key), jsonValue); ioWrite(infoSaveData->write, BUFSTRZ(key)); ioWrite(infoSaveData->write, EQ_BUF); ioWriteLine(infoSaveData->write, BUFSTR(jsonValue)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoSave(Info *const this, IoWrite *const write, InfoSaveCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); ASSERT(callbackFunction != NULL); ASSERT(callbackData != NULL); MEM_CONTEXT_TEMP_BEGIN() { InfoSave data = { .memContext = MEM_CONTEXT_TEMP(), .write = write, }; ioWriteOpen(data.write); // Begin checksum calculation data.checksum = cryptoHashNew(hashTypeSha1); INFO_CHECKSUM_BEGIN(data.checksum); // Add version and format callbackFunction(callbackData, STRDEF(INFO_SECTION_BACKREST), &data); infoSaveValue(&data, INFO_SECTION_BACKREST, INFO_KEY_FORMAT, jsonFromVar(VARUINT(REPOSITORY_FORMAT))); infoSaveValue(&data, INFO_SECTION_BACKREST, INFO_KEY_VERSION, jsonFromVar(VARSTRDEF(PROJECT_VERSION))); // Add cipher passphrase if defined if (infoCipherPass(this) != NULL) { callbackFunction(callbackData, STRDEF(INFO_SECTION_CIPHER), &data); infoSaveValue(&data, INFO_SECTION_CIPHER, INFO_KEY_CIPHER_PASS, jsonFromVar(VARSTR(infoCipherPass(this)))); } // Flush out any additional sections callbackFunction(callbackData, NULL, &data); // Add checksum (this must be set after all other values or it will not be valid) INFO_CHECKSUM_END(data.checksum); ioWrite(data.write, BUFSTRDEF("\n[" INFO_SECTION_BACKREST "]\n" INFO_KEY_CHECKSUM "=")); ioWriteLine( data.write, BUFSTR(jsonFromVar(VARSTR(strNewEncode(encodingHex, pckReadBinP(pckReadNew(ioFilterResult(data.checksum)))))))); // Close the file ioWriteClose(data.write); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ FN_EXTERN void infoCipherPassSet(Info *const this, const String *const cipherPass) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO, this); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_TEST_END(); FUNCTION_AUDIT_IF(memContextCurrent() != objMemContext(this)); // Do not audit calls from within the object ASSERT(this != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.cipherPass = strDup(cipherPass); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoLoad(const String *const error, InfoLoadCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, error); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(error != NULL); ASSERT(callbackFunction != NULL); ASSERT(callbackData != NULL); MEM_CONTEXT_TEMP_BEGIN() { unsigned int try = 0; volatile bool done = false; // Are all files tried? Must be preserved even on error. volatile bool loaded = false; // Was a file loaded? Must be preserved even on error. const ErrorType *loadErrorType = NULL; String *loadErrorMessage = NULL; do { // Attempt to load the file TRY_BEGIN() { loaded = callbackFunction(callbackData, try); done = true; CHECK(AssertError, loaded || try > 0, "file load must be attempted"); } CATCH_ANY() { // Set error type if none has been set if (loadErrorType == NULL) { loadErrorType = errorType(); loadErrorMessage = strCatFmt(strNew(), "%s:", strZ(error)); } // Else if the error type is different else if (loadErrorType != errorType()) { // Set type that is not file missing (which is likely the most common error) if (loadErrorType == &FileMissingError) { loadErrorType = errorType(); } // Else set a generic error else if (errorType() != &FileMissingError) loadErrorType = &FileOpenError; } // Append new error strCatFmt(loadErrorMessage, "\n%s: %s", errorTypeName(errorType()), errorMessage()); // Try again try++; } TRY_END(); } while (!done); // Error when no file was loaded if (!loaded) THROWP(loadErrorType, strZ(loadErrorMessage)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/info.h0000664000175100017510000001313715240547630016246 0ustar00runnerrunner/*********************************************************************************************************************************** Info Handler ***********************************************************************************************************************************/ #ifndef INFO_INFO_H #define INFO_INFO_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Info Info; typedef struct InfoSave InfoSave; #include "common/ini.h" #include "common/type/json.h" #include "storage/storage.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define INFO_COPY_EXT ".copy" #define INFO_KEY_FORMAT "backrest-format" #define INFO_KEY_VERSION "backrest-version" /*********************************************************************************************************************************** Function types for loading and saving ***********************************************************************************************************************************/ // The purpose of this callback is to attempt a load (from file or otherwise). Return true when the load is successful or throw an // error. Return false when there are no more loads to try, but always make at least one load attempt. The try parameter will start // at 0 and be incremented on each call. // {uncrustify_off - uncrustify unable to parse this statement} typedef bool InfoLoadCallback(void *data, unsigned int try); // {uncrustify_on} typedef void InfoLoadNewCallback(void *data, const String *section, const String *key, JsonRead *value); typedef void InfoSaveCallback(void *data, const String *sectionNext, InfoSave *infoSaveData); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Info *infoNew(const String *cipherPassSub); // Create new object and load contents from a file FN_EXTERN Info *infoNewLoad(IoRead *read, InfoLoadNewCallback *callbackFunction, void *callbackData); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct InfoPub { const String *backrestVersion; // pgBackRest version const String *cipherPass; // Cipher passphrase if set } InfoPub; // Cipher passphrase if set FN_INLINE_ALWAYS const String * infoCipherPass(const Info *const this) { return THIS_PUB(Info)->cipherPass; } FN_EXTERN void infoCipherPassSet(Info *this, const String *cipherPass); // pgBackRest version FN_INLINE_ALWAYS const String * infoBackrestVersion(const Info *const this) { return THIS_PUB(Info)->backrestVersion; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Save to file FN_EXTERN void infoSave(Info *this, IoWrite *write, InfoSaveCallback *callbackFunction, void *callbackData); // Check if the section should be saved FN_EXTERN bool infoSaveSection(InfoSave *infoSaveData, const char *section, const String *sectionNext); // Save a JSON formatted value and update checksum FN_EXTERN void infoSaveValue(InfoSave *infoSaveData, const char *section, const char *key, const String *jsonValue); /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Load info file(s) and throw error for each attempt if none are successful FN_EXTERN void infoLoad(const String *error, InfoLoadCallback *callbackFunction, void *callbackData); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_INFO_TYPE \ Info * #define FUNCTION_LOG_INFO_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Info", buffer, bufferSize) #define FUNCTION_LOG_INFO_SAVE_TYPE \ InfoSave * #define FUNCTION_LOG_INFO_SAVE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoSave", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/infoArchive.c0000664000175100017510000003003515240547630017537 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Info Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/ini.h" #include "common/io/bufferWrite.h" #include "common/io/io.h" #include "common/log.h" #include "info/infoArchive.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ STRING_EXTERN(INFO_ARCHIVE_PATH_FILE_STR, INFO_ARCHIVE_PATH_FILE); STRING_EXTERN(INFO_ARCHIVE_PATH_FILE_COPY_STR, INFO_ARCHIVE_PATH_FILE_COPY); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct InfoArchive { InfoArchivePub pub; // Publicly accessible variables }; /*********************************************************************************************************************************** Internal constructor ***********************************************************************************************************************************/ static InfoArchive * infoArchiveNewInternal(void) { FUNCTION_TEST_VOID(); InfoArchive *this = OBJ_NEW_ALLOC(); *this = (InfoArchive) { .pub = { .memContext = memContextCurrent(), }, }; FUNCTION_TEST_RETURN(INFO_ARCHIVE, this); } /**********************************************************************************************************************************/ FN_EXTERN InfoArchive * infoArchiveNew(const unsigned int pgVersion, const uint64_t pgSystemId, const String *const cipherPassSub) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_TEST_PARAM(STRING, cipherPassSub); FUNCTION_LOG_END(); ASSERT(pgVersion > 0 && pgSystemId > 0); InfoArchive *this; OBJ_NEW_BASE_BEGIN(InfoArchive, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoArchiveNewInternal(); // Initialize the pg data this->pub.infoPg = infoPgNew(infoPgArchive, cipherPassSub); infoArchivePgSet(this, pgVersion, pgSystemId); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_ARCHIVE, this); } /**********************************************************************************************************************************/ FN_EXTERN InfoArchive * infoArchiveNewLoad(IoRead *const read) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_END(); ASSERT(read != NULL); InfoArchive *this; OBJ_NEW_BASE_BEGIN(InfoArchive, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoArchiveNewInternal(); this->pub.infoPg = infoPgNewLoad(read, infoPgArchive, NULL, NULL); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_ARCHIVE, this); } /**********************************************************************************************************************************/ const String * infoArchiveIdHistoryMatch( const InfoArchive *const this, const unsigned int historyId, const unsigned int pgVersion, const uint64_t pgSystemId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_ARCHIVE, this); FUNCTION_LOG_PARAM(UINT, historyId); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_END(); ASSERT(this != NULL); String *archiveId = NULL; const InfoPg *const infoPg = infoArchivePg(this); // Search the history list, from newest to oldest for (unsigned int pgIdx = 0; pgIdx < infoPgDataTotal(infoPg); pgIdx++) { const InfoPgData pgDataArchive = infoPgData(infoPg, pgIdx); // If there is an exact match with the history, system and version then get the archiveId and stop if (historyId == pgDataArchive.id && pgSystemId == pgDataArchive.systemId && pgVersion == pgDataArchive.version) { archiveId = infoPgArchiveId(infoPg, pgIdx); break; } } // If there was not an exact match, then search for the first matching database system-id and version if (archiveId == NULL) { for (unsigned int pgIdx = 0; pgIdx < infoPgDataTotal(infoPg); pgIdx++) { const InfoPgData pgDataArchive = infoPgData(infoPg, pgIdx); if (pgSystemId == pgDataArchive.systemId && pgVersion == pgDataArchive.version) { archiveId = infoPgArchiveId(infoPg, pgIdx); break; } } } // If the archive id has not been found, then error if (archiveId == NULL) { THROW_FMT( ArchiveMismatchError, "unable to retrieve the archive id for database version '%s' and system-id '%" PRIu64 "'", strZ(pgVersionToStr(pgVersion)), pgSystemId); } FUNCTION_LOG_RETURN(STRING, archiveId); } /*********************************************************************************************************************************** Save to file ***********************************************************************************************************************************/ static void infoArchiveSave(InfoArchive *const this, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_ARCHIVE, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); MEM_CONTEXT_TEMP_BEGIN() { infoPgSave(infoArchivePg(this), write, NULL, NULL); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN InfoArchive * infoArchivePgSet(InfoArchive *const this, const unsigned int pgVersion, const uint64_t pgSystemId) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_ARCHIVE, this); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_END(); ASSERT(this != NULL); this->pub.infoPg = infoPgSet(infoArchivePg(this), infoPgArchive, pgVersion, pgSystemId, 0); FUNCTION_LOG_RETURN(INFO_ARCHIVE, this); } /**********************************************************************************************************************************/ typedef struct InfoArchiveLoadFileData { MemContext *memContext; // Mem context const Storage *storage; // Storage to load from const String *fileName; // Base filename CipherType cipherType; // Cipher type const String *cipherPass; // Cipher passphrase InfoArchive *infoArchive; // Loaded infoArchive object } InfoArchiveLoadFileData; static bool infoArchiveLoadFileCallback(void *const data, const unsigned int try) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, data); FUNCTION_LOG_PARAM(UINT, try); FUNCTION_LOG_END(); ASSERT(data != NULL); InfoArchiveLoadFileData *const loadData = data; bool result = false; if (try < 2) { MEM_CONTEXT_TEMP_BEGIN() { // Construct filename based on try const String *const fileName = try == 0 ? loadData->fileName : strNewFmt("%s" INFO_COPY_EXT, strZ(loadData->fileName)); // Attempt to load the file IoRead *const read = storageReadIo(storageNewReadP(loadData->storage, fileName)); cipherBlockFilterGroupAdd(ioReadFilterGroup(read), loadData->cipherType, cipherModeDecrypt, loadData->cipherPass); MEM_CONTEXT_BEGIN(loadData->memContext) { loadData->infoArchive = infoArchiveNewLoad(read); result = true; } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(BOOL, result); } FN_EXTERN InfoArchive * infoArchiveLoadFile( const Storage *const storage, const String *const fileName, const CipherType cipherType, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT((cipherType == cipherTypeNone && cipherPass == NULL) || (cipherType != cipherTypeNone && cipherPass != NULL)); InfoArchiveLoadFileData data = { .memContext = memContextCurrent(), .storage = storage, .fileName = fileName, .cipherType = cipherType, .cipherPass = cipherPass, }; MEM_CONTEXT_TEMP_BEGIN() { const char *const fileNamePath = strZ(storagePathP(storage, fileName)); TRY_BEGIN() { infoLoad( strNewFmt("unable to load info file '%s' or '%s" INFO_COPY_EXT "'", fileNamePath, fileNamePath), infoArchiveLoadFileCallback, &data); } CATCH_ANY() { THROWP_FMT( errorType(), "%s\n" "HINT: archive.info cannot be opened but is required to push/get WAL segments.\n" "HINT: is archive_command configured correctly in postgresql.conf?\n" "HINT: has a stanza-create been performed?\n" "HINT: use --no-archive-check to disable archive checks during backup if you have an alternate archiving scheme.", errorMessage()); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_ARCHIVE, data.infoArchive); } /**********************************************************************************************************************************/ FN_EXTERN void infoArchiveSaveFile( InfoArchive *const infoArchive, const Storage *const storage, const String *const fileName, const CipherType cipherType, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_ARCHIVE, infoArchive); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_LOG_END(); ASSERT(infoArchive != NULL); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT((cipherType == cipherTypeNone && cipherPass == NULL) || (cipherType != cipherTypeNone && cipherPass != NULL)); MEM_CONTEXT_TEMP_BEGIN() { // Write output into a buffer since it needs to be saved to storage twice Buffer *const buffer = bufNew(ioBufferSize()); IoWrite *const write = ioBufferWriteNew(buffer); cipherBlockFilterGroupAdd(ioWriteFilterGroup(write), cipherType, cipherModeEncrypt, cipherPass); infoArchiveSave(infoArchive, write); // Save the file and make a copy storagePutP(storageNewWriteP(storage, fileName), buffer); storagePutP(storageNewWriteP(storage, strNewFmt("%s" INFO_COPY_EXT, strZ(fileName))), buffer); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/infoArchive.h0000664000175100017510000001273115240547630017547 0ustar00runnerrunner/*********************************************************************************************************************************** Archive Info Handler ***********************************************************************************************************************************/ #ifndef INFO_INFOARCHIVE_H #define INFO_INFOARCHIVE_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct InfoArchive InfoArchive; #include "common/crypto/common.h" #include "common/type/object.h" #include "common/type/string.h" #include "info/infoPg.h" #include "storage/storage.h" /*********************************************************************************************************************************** Archive info filename ***********************************************************************************************************************************/ #define INFO_ARCHIVE_FILE "archive.info" #define REGEX_ARCHIVE_DIR_DB_VERSION "^[0-9]+(\\.[0-9]+)*-[0-9]+$" #define INFO_ARCHIVE_PATH_FILE STORAGE_REPO_ARCHIVE "/" INFO_ARCHIVE_FILE STRING_DECLARE(INFO_ARCHIVE_PATH_FILE_STR); #define INFO_ARCHIVE_PATH_FILE_COPY INFO_ARCHIVE_PATH_FILE INFO_COPY_EXT STRING_DECLARE(INFO_ARCHIVE_PATH_FILE_COPY_STR); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN InfoArchive *infoArchiveNew(const unsigned int pgVersion, const uint64_t pgSystemId, const String *cipherPassSub); // Create new object and load contents from IoRead FN_EXTERN InfoArchive *infoArchiveNewLoad(IoRead *read); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct InfoArchivePub { MemContext *memContext; // Mem context InfoPg *infoPg; // Contents of the DB data } InfoArchivePub; // PostgreSQL info FN_INLINE_ALWAYS InfoPg * infoArchivePg(const InfoArchive *const this) { return THIS_PUB(InfoArchive)->infoPg; } FN_EXTERN InfoArchive *infoArchivePgSet(InfoArchive *this, unsigned int pgVersion, uint64_t pgSystemId); // Current archive id FN_INLINE_ALWAYS const String * infoArchiveId(const InfoArchive *const this) { return infoPgArchiveId(infoArchivePg(this), infoPgDataCurrentId(infoArchivePg(this))); } // Cipher passphrase FN_INLINE_ALWAYS const String * infoArchiveCipherPass(const InfoArchive *const this) { return infoPgCipherPass(infoArchivePg(this)); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Given a backrest history id and postgres systemId and version, return the archiveId of the best match FN_EXTERN const String *infoArchiveIdHistoryMatch( const InfoArchive *this, const unsigned int historyId, const unsigned int pgVersion, const uint64_t pgSystemId); // Move to a new parent mem context FN_INLINE_ALWAYS InfoArchive * infoArchiveMove(InfoArchive *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void infoArchiveFree(InfoArchive *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Load archive info FN_EXTERN InfoArchive *infoArchiveLoadFile( const Storage *storage, const String *fileName, CipherType cipherType, const String *cipherPass); // Save archive info FN_EXTERN void infoArchiveSaveFile( InfoArchive *infoArchive, const Storage *storage, const String *fileName, CipherType cipherType, const String *cipherPass); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_INFO_ARCHIVE_TYPE \ InfoArchive * #define FUNCTION_LOG_INFO_ARCHIVE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoArchive", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/infoBackup.c0000664000175100017510000011711615240547630017371 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Info Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include "command/backup/common.h" #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/ini.h" #include "common/io/bufferWrite.h" #include "common/io/io.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/json.h" #include "common/type/list.h" #include "info/infoBackup.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" #include "version.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ STRING_EXTERN(INFO_BACKUP_PATH_FILE_STR, INFO_BACKUP_PATH_FILE); STRING_EXTERN(INFO_BACKUP_PATH_FILE_COPY_STR, INFO_BACKUP_PATH_FILE_COPY); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct InfoBackup { InfoBackupPub pub; // Publicly accessible variables }; /*********************************************************************************************************************************** Internal constructor ***********************************************************************************************************************************/ static InfoBackup * infoBackupNewInternal(void) { FUNCTION_TEST_VOID(); FUNCTION_AUDIT_HELPER(); InfoBackup *this = OBJ_NEW_ALLOC(); *this = (InfoBackup) { .pub = { .memContext = memContextCurrent(), .backup = lstNewP(sizeof(InfoBackupData), .comparator = lstComparatorStr), .updated = true, }, }; FUNCTION_TEST_RETURN(INFO_BACKUP, this); } /**********************************************************************************************************************************/ FN_EXTERN InfoBackup * infoBackupNew( const unsigned int pgVersion, const uint64_t pgSystemId, const unsigned int pgCatalogVersion, const String *const cipherPassSub) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(UINT, pgCatalogVersion); FUNCTION_TEST_PARAM(STRING, cipherPassSub); FUNCTION_LOG_END(); ASSERT(pgVersion > 0 && pgSystemId > 0 && pgCatalogVersion > 0); InfoBackup *this; OBJ_NEW_BASE_BEGIN(InfoBackup, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoBackupNewInternal(); // Initialize the pg data this->pub.infoPg = infoPgNew(infoPgBackup, cipherPassSub); infoBackupPgSet(this, pgVersion, pgSystemId, pgCatalogVersion); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, this); } /*********************************************************************************************************************************** Create new object and load contents from a file ***********************************************************************************************************************************/ #define INFO_BACKUP_SECTION "backup" #define INFO_BACKUP_SECTION_BACKUP_CURRENT INFO_BACKUP_SECTION ":current" #define INFO_BACKUP_KEY_BACKUP_ANNOTATION "backup-annotation" #define INFO_BACKUP_KEY_BACKUP_ARCHIVE_START "backup-archive-start" #define INFO_BACKUP_KEY_BACKUP_ARCHIVE_STOP "backup-archive-stop" #define INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE "backup-info-repo-size" #define INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_DELTA "backup-info-repo-size-delta" #define INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP "backup-info-repo-size-map" #define INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP_DELTA "backup-info-repo-size-map-delta" #define INFO_BACKUP_KEY_BACKUP_INFO_SIZE "backup-info-size" #define INFO_BACKUP_KEY_BACKUP_INFO_SIZE_DELTA "backup-info-size-delta" #define INFO_BACKUP_KEY_BACKUP_LSN_START "backup-lsn-start" #define INFO_BACKUP_KEY_BACKUP_LSN_STOP "backup-lsn-stop" #define INFO_BACKUP_KEY_BACKUP_PRIOR STRID5("backup-prior", 0x93d3286e1558c220) #define INFO_BACKUP_KEY_BACKUP_REFERENCE "backup-reference" #define INFO_BACKUP_KEY_BACKUP_TIMESTAMP_START "backup-timestamp-start" #define INFO_BACKUP_KEY_BACKUP_TIMESTAMP_STOP "backup-timestamp-stop" #define INFO_BACKUP_KEY_BACKUP_TYPE STRID5("backup-type", 0x1619a6e1558c220) #define INFO_BACKUP_KEY_BACKUP_ERROR STRID5("backup-error", 0x93e522ee1558c220) #define INFO_BACKUP_KEY_OPT_ARCHIVE_CHECK "option-archive-check" #define INFO_BACKUP_KEY_OPT_ARCHIVE_COPY "option-archive-copy" #define INFO_BACKUP_KEY_OPT_BACKUP_STANDBY "option-backup-standby" #define INFO_BACKUP_KEY_OPT_CHECKSUM_PAGE "option-checksum-page" #define INFO_BACKUP_KEY_OPT_COMPRESS "option-compress" #define INFO_BACKUP_KEY_OPT_HARDLINK "option-hardlink" #define INFO_BACKUP_KEY_OPT_ONLINE "option-online" static void infoBackupLoadCallback(void *const data, const String *const section, const String *const key, JsonRead *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(JSON_READ, json); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(data != NULL); ASSERT(section != NULL); ASSERT(key != NULL); ASSERT(json != NULL); InfoBackup *const infoBackup = (InfoBackup *)data; // Process current backup list if (strEqZ(section, INFO_BACKUP_SECTION_BACKUP_CURRENT)) { MEM_CONTEXT_BEGIN(lstMemContext(infoBackup->pub.backup)) { jsonReadObjectBegin(json); InfoBackupData info = { .backupLabel = strDup(key), }; // Format and version info.backrestFormat = jsonReadUInt(jsonReadKeyRequireZ(json, INFO_KEY_FORMAT)); info.backrestVersion = jsonReadStr(jsonReadKeyRequireZ(json, INFO_KEY_VERSION)); // Annotation if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_ANNOTATION)) info.backupAnnotation = jsonReadVar(json); // Archive start/stop if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_ARCHIVE_START)) info.backupArchiveStart = jsonReadStr(json); if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_ARCHIVE_STOP)) info.backupArchiveStop = jsonReadStr(json); // Report errors detected during the backup. The key may not exist in older versions. if (jsonReadKeyExpectStrId(json, INFO_BACKUP_KEY_BACKUP_ERROR)) info.backupError = varNewBool(jsonReadBool(json)); // Size info info.backupInfoRepoSize = jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE)); info.backupInfoRepoSizeDelta = jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_DELTA)); if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP)) { info.backupInfoRepoSizeMap = varNewUInt64(jsonReadUInt64(json)); info.backupInfoRepoSizeMapDelta = varNewUInt64( jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP_DELTA))); } info.backupInfoSize = jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_SIZE)); info.backupInfoSizeDelta = jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_INFO_SIZE_DELTA)); // Lsn start/stop if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_LSN_START)) info.backupLsnStart = jsonReadStr(json); if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_LSN_STOP)) info.backupLsnStop = jsonReadStr(json); // Prior backup if (jsonReadKeyExpectStrId(json, INFO_BACKUP_KEY_BACKUP_PRIOR)) info.backupPrior = jsonReadStr(json); // Reference if (jsonReadKeyExpectZ(json, INFO_BACKUP_KEY_BACKUP_REFERENCE)) info.backupReference = jsonReadStrLst(json); // Read timestamps as uint64 to ensure a positive value (guarantee no backups before 1970) info.backupTimestampStart = (time_t)jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_TIMESTAMP_START)); info.backupTimestampStop = (time_t)jsonReadUInt64(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_BACKUP_TIMESTAMP_STOP)); // backup type info.backupType = (BackupType)jsonReadStrId(jsonReadKeyRequireStrId(json, INFO_BACKUP_KEY_BACKUP_TYPE)); ASSERT(info.backupType == backupTypeFull || info.backupType == backupTypeDiff || info.backupType == backupTypeIncr); // Database id info.backupPgId = jsonReadUInt(jsonReadKeyRequireZ(json, INFO_KEY_DB_ID)); // Options info.optionArchiveCheck = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_ARCHIVE_CHECK)); info.optionArchiveCopy = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_ARCHIVE_COPY)); info.optionBackupStandby = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_BACKUP_STANDBY)); info.optionChecksumPage = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_CHECKSUM_PAGE)); info.optionCompress = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_COMPRESS)); info.optionHardlink = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_HARDLINK)); info.optionOnline = jsonReadBool(jsonReadKeyRequireZ(json, INFO_BACKUP_KEY_OPT_ONLINE)); // Add the backup data to the list lstAdd(infoBackup->pub.backup, &info); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN InfoBackup * infoBackupNewLoad(IoRead *const read) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_END(); ASSERT(read != NULL); InfoBackup *this; OBJ_NEW_BASE_BEGIN(InfoBackup, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoBackupNewInternal(); this->pub.infoPg = infoPgNewLoad(read, infoPgBackup, infoBackupLoadCallback, this); this->pub.updated = false; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, this); } /*********************************************************************************************************************************** Save to file ***********************************************************************************************************************************/ static void infoBackupSaveCallback(void *const data, const String *const sectionNext, InfoSave *const infoSaveData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(STRING, sectionNext); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(data != NULL); ASSERT(infoSaveData != NULL); InfoBackup *const infoBackup = (InfoBackup *)data; if (infoSaveSection(infoSaveData, INFO_BACKUP_SECTION_BACKUP_CURRENT, sectionNext)) { // Set the backup current section for (unsigned int backupIdx = 0; backupIdx < infoBackupDataTotal(infoBackup); backupIdx++) { const InfoBackupData backupData = infoBackupData(infoBackup, backupIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); jsonWriteUInt(jsonWriteKeyZ(json, INFO_KEY_FORMAT), backupData.backrestFormat); jsonWriteStr(jsonWriteKeyZ(json, INFO_KEY_VERSION), backupData.backrestVersion); if (backupData.backupAnnotation != NULL) jsonWriteVar(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_ANNOTATION), backupData.backupAnnotation); jsonWriteStr(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_ARCHIVE_START), backupData.backupArchiveStart); jsonWriteStr(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_ARCHIVE_STOP), backupData.backupArchiveStop); // Do not save backup-error if it was not loaded. This prevents backups that were added before the backup-error flag // was introduced from being saved with an incorrect value. if (backupData.backupError != NULL) jsonWriteBool(jsonWriteKeyStrId(json, INFO_BACKUP_KEY_BACKUP_ERROR), varBool(backupData.backupError)); jsonWriteUInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE), backupData.backupInfoRepoSize); jsonWriteUInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_DELTA), backupData.backupInfoRepoSizeDelta); ASSERT( (backupData.backupInfoRepoSizeMap != NULL && backupData.backupInfoRepoSizeMap != NULL) || (backupData.backupInfoRepoSizeMap == NULL && backupData.backupInfoRepoSizeMap == NULL)); if (backupData.backupInfoRepoSizeMap != NULL) { jsonWriteUInt64( jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP), varUInt64(backupData.backupInfoRepoSizeMap)); jsonWriteUInt64( jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_REPO_SIZE_MAP_DELTA), varUInt64(backupData.backupInfoRepoSizeMapDelta)); } jsonWriteUInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_SIZE), backupData.backupInfoSize); jsonWriteUInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_INFO_SIZE_DELTA), backupData.backupInfoSizeDelta); if (backupData.backupLsnStart != NULL) jsonWriteStr(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_LSN_START), backupData.backupLsnStart); if (backupData.backupLsnStop != NULL) jsonWriteStr(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_LSN_STOP), backupData.backupLsnStop); if (backupData.backupPrior != NULL) jsonWriteStr(jsonWriteKeyStrId(json, INFO_BACKUP_KEY_BACKUP_PRIOR), backupData.backupPrior); if (backupData.backupReference != NULL) jsonWriteStrLst(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_REFERENCE), backupData.backupReference); // When storing time_t treat as signed int to avoid casting jsonWriteInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_TIMESTAMP_START), backupData.backupTimestampStart); jsonWriteInt64(jsonWriteKeyZ(json, INFO_BACKUP_KEY_BACKUP_TIMESTAMP_STOP), backupData.backupTimestampStop); jsonWriteStrId(jsonWriteKeyStrId(json, INFO_BACKUP_KEY_BACKUP_TYPE), backupData.backupType); jsonWriteUInt(jsonWriteKeyZ(json, INFO_KEY_DB_ID), backupData.backupPgId); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_ARCHIVE_CHECK), backupData.optionArchiveCheck); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_ARCHIVE_COPY), backupData.optionArchiveCopy); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_BACKUP_STANDBY), backupData.optionBackupStandby); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_CHECKSUM_PAGE), backupData.optionChecksumPage); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_COMPRESS), backupData.optionCompress); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_HARDLINK), backupData.optionHardlink); jsonWriteBool(jsonWriteKeyZ(json, INFO_BACKUP_KEY_OPT_ONLINE), backupData.optionOnline); infoSaveValue( infoSaveData, INFO_BACKUP_SECTION_BACKUP_CURRENT, strZ(backupData.backupLabel), jsonWriteResult(jsonWriteObjectEnd(json))); } } FUNCTION_TEST_RETURN_VOID(); } static void infoBackupSave(InfoBackup *const this, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); MEM_CONTEXT_TEMP_BEGIN() { infoPgSave(infoBackupPg(this), write, infoBackupSaveCallback, this); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN InfoBackup * infoBackupPgSet( InfoBackup *const this, const unsigned int pgVersion, const uint64_t pgSystemId, const unsigned int pgCatalogVersion) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(UINT, pgCatalogVersion); FUNCTION_LOG_END(); this->pub.infoPg = infoPgSet(infoBackupPg(this), infoPgBackup, pgVersion, pgSystemId, pgCatalogVersion); this->pub.updated = true; FUNCTION_LOG_RETURN(INFO_BACKUP, this); } /**********************************************************************************************************************************/ FN_EXTERN InfoBackupData infoBackupData(const InfoBackup *const this, const unsigned int backupDataIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(UINT, backupDataIdx); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INFO_BACKUP_DATA, *(InfoBackupData *)lstGet(this->pub.backup, backupDataIdx)); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupDataAdd(InfoBackup *const this, const Manifest *const manifest) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(MANIFEST, manifest); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(manifest != NULL); MEM_CONTEXT_TEMP_BEGIN() { const ManifestData *const manData = manifestData(manifest); // Calculate backup sizes, references and report errors uint64_t backupSize = 0; uint64_t backupSizeDelta = 0; uint64_t backupRepoSize = 0; uint64_t backupRepoSizeDelta = 0; uint64_t backupRepoSizeMap = 0; uint64_t backupRepoSizeMapDelta = 0; bool backupError = false; for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); backupSize += file.size; backupRepoSize += file.sizeRepo > 0 ? file.sizeRepo : file.size; backupRepoSizeMap += file.blockIncrMapSize; // If a reference to a file exists, then it is in a previous backup and the delta calculation was already done if (file.reference == NULL) { backupSizeDelta += file.size; backupRepoSizeDelta += file.sizeRepo > 0 ? file.sizeRepo : file.size; backupRepoSizeMapDelta += file.blockIncrMapSize; } // Is there an error in the file? if (file.checksumPageError) backupError = true; } MEM_CONTEXT_BEGIN(lstMemContext(this->pub.backup)) { InfoBackupData infoBackupData = { .backupLabel = strDup(manData->backupLabel), .backrestFormat = REPOSITORY_FORMAT, .backrestVersion = strDup(manData->backrestVersion), .backupInfoRepoSize = backupRepoSize, .backupInfoRepoSizeDelta = backupRepoSizeDelta, .backupInfoSize = backupSize, .backupInfoSizeDelta = backupSizeDelta, .backupPgId = manData->pgId, .backupTimestampStart = manData->backupTimestampStart, .backupTimestampStop = manData->backupTimestampStop, .backupType = manData->backupType, .backupError = varNewBool(backupError), .backupAnnotation = varDup(manData->annotation), .backupArchiveStart = strDup(manData->archiveStart), .backupArchiveStop = strDup(manData->archiveStop), .backupLsnStart = strDup(manData->lsnStart), .backupLsnStop = strDup(manData->lsnStop), .optionArchiveCheck = manData->backupOptionArchiveCheck, .optionArchiveCopy = manData->backupOptionArchiveCopy, .optionBackupStandby = manData->backupOptionStandby != NULL ? varBool(manData->backupOptionStandby) : false, .optionChecksumPage = manData->backupOptionChecksumPage != NULL ? varBool(manData->backupOptionChecksumPage) : false, .optionCompress = manData->backupOptionCompressType != compressTypeNone, .optionHardlink = manData->backupOptionHardLink, .optionOnline = manData->backupOptionOnline, }; // Add map sizes when block incr if (manData->blockIncr) { infoBackupData.backupInfoRepoSizeMap = varNewUInt64(backupRepoSizeMap); infoBackupData.backupInfoRepoSizeMapDelta = varNewUInt64(backupRepoSizeMapDelta); } if (manData->backupType != backupTypeFull) { // This list may not be sorted for manifests created before the reference list was added. Remove the last reference // since it will always be the current backup. Technically the current backup is always referenced but this is not // useful information for the user. infoBackupData.backupReference = strLstSort(strLstDup(manifestReferenceList(manifest)), sortOrderAsc); ASSERT( strEq( strLstGet(infoBackupData.backupReference, strLstSize(infoBackupData.backupReference) - 1), infoBackupData.backupLabel)); strLstRemoveIdx(infoBackupData.backupReference, strLstSize(infoBackupData.backupReference) - 1); infoBackupData.backupPrior = strDup(manData->backupLabelPrior); } // Add the backup data to the current backup list lstAdd(this->pub.backup, &infoBackupData); // Ensure the list is sorted ascending by the backupLabel lstSort(this->pub.backup, sortOrderAsc); } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); this->pub.updated = true; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupDataAnnotationSet(InfoBackup *const this, const String *const backupLabel, const KeyValue *const newAnnotationKv) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INFO_BACKUP, this); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_PARAM(KEY_VALUE, newAnnotationKv); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(infoBackupLabelExists(this, backupLabel)); ASSERT(newAnnotationKv != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.backup)) { // Get data for specified backup InfoBackupData *const infoBackupData = lstFind(this->pub.backup, &backupLabel); // Create annotation if it does not exist if (infoBackupData->backupAnnotation == NULL) infoBackupData->backupAnnotation = varNewKv(kvNew()); // Update annotations KeyValue *const annotationKv = varKv(infoBackupData->backupAnnotation); const VariantList *const newAnnotationKeyList = kvKeyList(newAnnotationKv); for (unsigned int keyIdx = 0; keyIdx < varLstSize(newAnnotationKeyList); keyIdx++) { const Variant *const newKey = varLstGet(newAnnotationKeyList, keyIdx); const Variant *const newValue = kvGet(newAnnotationKv, newKey); // If value is empty remove the key if (strEmpty(varStr(newValue))) { kvRemove(annotationKv, newKey); } // Else put new key/value (this will overwrite an existing value) else kvPut(annotationKv, newKey, newValue); } // Clean field if there are no annotations left if (varLstSize(kvKeyList(annotationKv)) == 0) infoBackupData->backupAnnotation = NULL; } MEM_CONTEXT_END(); this->pub.updated = true; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupDataDelete(InfoBackup *const this, const String *const backupDeleteLabel) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(STRING, backupDeleteLabel); FUNCTION_LOG_END(); ASSERT(this != NULL); for (unsigned int idx = 0; idx < infoBackupDataTotal(this); idx++) { InfoBackupData backupData = infoBackupData(this, idx); if (strCmp(backupData.backupLabel, backupDeleteLabel) == 0) lstRemoveIdx(this->pub.backup, idx); } this->pub.updated = true; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN StringList * infoBackupDataLabelList(const InfoBackup *const this, const String *const expression) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_END(); ASSERT(this != NULL); // Return a 0 sized list if no current backups or none matching the filter StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Prepare regexp if an expression was passed RegExp *const regExp = (expression == NULL) ? NULL : regExpNew(expression); // For each backup label, compare it to the filter (if any) and sort it for return for (unsigned int backupLabelIdx = 0; backupLabelIdx < infoBackupDataTotal(this); backupLabelIdx++) { const InfoBackupData backupData = infoBackupData(this, backupLabelIdx); if (regExp == NULL || regExpMatch(regExp, backupData.backupLabel)) { strLstAdd(result, backupData.backupLabel); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN StringList * infoBackupDataDependentList(const InfoBackup *const this, const String *const backupLabel) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_BACKUP, this); FUNCTION_LOG_PARAM(STRING, backupLabel); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(backupLabel != NULL); // Return the given label as the only dependency or the given label and a list of labels that depend on it StringList *const result = strLstNew(); strLstAdd(result, backupLabel); MEM_CONTEXT_TEMP_BEGIN() { // For each backup label from oldest to newest in the current section, add each dependency to the list for (unsigned int backupLabelIdx = 0; backupLabelIdx < infoBackupDataTotal(this); backupLabelIdx++) { const InfoBackupData backupData = infoBackupData(this, backupLabelIdx); // If the backupPrior is in the dependency chain add the label to the list if (backupData.backupPrior != NULL && strLstExists(result, backupData.backupPrior)) strLstAdd(result, backupData.backupLabel); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ typedef struct InfoBackupLoadFileData { MemContext *memContext; // Mem context const Storage *storage; // Storage to load from const String *fileName; // Base filename CipherType cipherType; // Cipher type const String *cipherPass; // Cipher passphrase InfoBackup *infoBackup; // Loaded infoBackup object } InfoBackupLoadFileData; static bool infoBackupLoadFileCallback(void *const data, const unsigned int try) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, data); FUNCTION_LOG_PARAM(UINT, try); FUNCTION_LOG_END(); ASSERT(data != NULL); InfoBackupLoadFileData *const loadData = data; bool result = false; if (try < 2) { MEM_CONTEXT_TEMP_BEGIN() { // Construct filename based on try const String *const fileName = try == 0 ? loadData->fileName : strNewFmt("%s" INFO_COPY_EXT, strZ(loadData->fileName)); // Attempt to load the file IoRead *const read = storageReadIo(storageNewReadP(loadData->storage, fileName)); cipherBlockFilterGroupAdd(ioReadFilterGroup(read), loadData->cipherType, cipherModeDecrypt, loadData->cipherPass); MEM_CONTEXT_BEGIN(loadData->memContext) { loadData->infoBackup = infoBackupNewLoad(read); result = true; } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(BOOL, result); } FN_EXTERN InfoBackup * infoBackupLoadFile( const Storage *const storage, const String *const fileName, const CipherType cipherType, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT((cipherType == cipherTypeNone && cipherPass == NULL) || (cipherType != cipherTypeNone && cipherPass != NULL)); InfoBackupLoadFileData data = { .memContext = memContextCurrent(), .storage = storage, .fileName = fileName, .cipherType = cipherType, .cipherPass = cipherPass, }; MEM_CONTEXT_TEMP_BEGIN() { const char *const fileNamePath = strZ(storagePathP(storage, fileName)); TRY_BEGIN() { infoLoad( strNewFmt("unable to load info file '%s' or '%s" INFO_COPY_EXT "'", fileNamePath, fileNamePath), infoBackupLoadFileCallback, &data); } CATCH_ANY() { THROWP_FMT( errorType(), "%s\n" "HINT: backup.info cannot be opened and is required to perform a backup.\n" "HINT: has a stanza-create been performed?", errorMessage()); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, data.infoBackup); } /**********************************************************************************************************************************/ FN_EXTERN InfoBackup * infoBackupLoadFileReconstruct( const Storage *const storage, const String *const fileName, const CipherType cipherType, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT((cipherType == cipherTypeNone && cipherPass == NULL) || (cipherType != cipherTypeNone && cipherPass != NULL)); InfoBackup *const infoBackup = infoBackupLoadFile(storage, fileName, cipherType, cipherPass); MEM_CONTEXT_TEMP_BEGIN() { // Get a list of backups in the repo const StringList *const backupList = strLstSort( storageListP( storage, STORAGE_REPO_BACKUP_STR, .expression = backupRegExpP(.full = true, .differential = true, .incremental = true)), sortOrderAsc); // Get the list of current backups and remove backups from current that are no longer in the repository const StringList *backupCurrentList = strLstSort(infoBackupDataLabelList(infoBackup, NULL), sortOrderAsc); for (unsigned int backupCurrIdx = 0; backupCurrIdx < strLstSize(backupCurrentList); backupCurrIdx++) { const String *const backupLabel = strLstGet(backupCurrentList, backupCurrIdx); const String *const manifestFileName = strNewFmt(STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)); // If the manifest does not exist on disk and this backup has not already been deleted from the current list in the // infoBackup object, then remove it and its dependencies if (!storageExistsP(storage, manifestFileName) && infoBackupLabelExists(infoBackup, backupLabel)) { const StringList *const backupList = strLstSort( infoBackupDataDependentList(infoBackup, backupLabel), sortOrderDesc); for (unsigned int backupIdx = 0; backupIdx < strLstSize(backupList); backupIdx++) { const String *const removeBackup = strLstGet(backupList, backupIdx); LOG_WARN_FMT("backup '%s' missing manifest removed from " INFO_BACKUP_FILE, strZ(removeBackup)); infoBackupDataDelete(infoBackup, removeBackup); } } } // Get the updated current list of backups from backup.info backupCurrentList = strLstSort(infoBackupDataLabelList(infoBackup, NULL), sortOrderAsc); // For each backup in the repo, check if it exists in backup.info for (unsigned int backupIdx = 0; backupIdx < strLstSize(backupList); backupIdx++) { const String *const backupLabel = strLstGet(backupList, backupIdx); // If it does not exist in the list of current backups, then if it is valid, add it if (!strLstExists(backupCurrentList, backupLabel)) { const String *const manifestFileName = strNewFmt( STORAGE_REPO_BACKUP "/%s/" BACKUP_MANIFEST_FILE, strZ(backupLabel)); // Check if a completed backup exists (backup.manifest only - ignore .copy) if (storageExistsP(storage, manifestFileName)) { bool found = false; const Manifest *const manifest = manifestLoadFile( storage, manifestFileName, cipherType, infoPgCipherPass(infoBackupPg(infoBackup))); const ManifestData *const manData = manifestData(manifest); // If the pg data for the manifest exists in the history, then add it to current, but if something doesn't match // then warn that the backup is not valid for (unsigned int pgIdx = 0; pgIdx < infoPgDataTotal(infoBackupPg(infoBackup)); pgIdx++) { const InfoPgData pgHistory = infoPgData(infoBackupPg(infoBackup), pgIdx); // If there is an exact match with the history, system and version and there is no backup-prior dependency // or there is a backup-prior and it is in the list, then add this backup to the current backup list if (manData->pgId == pgHistory.id && manData->pgSystemId == pgHistory.systemId && manData->pgVersion == pgHistory.version && (manData->backupLabelPrior == NULL || infoBackupLabelExists(infoBackup, manData->backupLabelPrior))) { LOG_WARN_FMT("backup '%s' found in repository added to " INFO_BACKUP_FILE, strZ(backupLabel)); infoBackupDataAdd(infoBackup, manifest); found = true; break; } } if (!found) LOG_WARN_FMT("invalid backup '%s' cannot be added to current backups", strZ(manData->backupLabel)); } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_BACKUP, infoBackup); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupSaveFile( InfoBackup *const infoBackup, const Storage *const storage, const String *const fileName, const CipherType cipherType, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_BACKUP, infoBackup); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_LOG_END(); ASSERT(infoBackup != NULL); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT((cipherType == cipherTypeNone && cipherPass == NULL) || (cipherType != cipherTypeNone && cipherPass != NULL)); if (infoBackup->pub.updated) { MEM_CONTEXT_TEMP_BEGIN() { // Write output into a buffer since it needs to be saved to storage twice Buffer *const buffer = bufNew(ioBufferSize()); IoWrite *const write = ioBufferWriteNew(buffer); cipherBlockFilterGroupAdd(ioWriteFilterGroup(write), cipherType, cipherModeEncrypt, cipherPass); infoBackupSave(infoBackup, write); // Save the file and make a copy storagePutP(storageNewWriteP(storage, fileName), buffer); storagePutP(storageNewWriteP(storage, strNewFmt("%s" INFO_COPY_EXT, strZ(fileName))), buffer); } MEM_CONTEXT_TEMP_END(); infoBackup->pub.updated = false; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void infoBackupDataToLog(const InfoBackupData *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{label: %s, pgId: %u}", strZ(this->backupLabel), this->backupPgId); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/infoBackup.h0000664000175100017510000002277215240547630017401 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Info Handler ***********************************************************************************************************************************/ #ifndef INFO_INFOBACKUP_H #define INFO_INFOBACKUP_H #include "common/type/stringId.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct InfoBackup InfoBackup; /*********************************************************************************************************************************** Backup type enum ***********************************************************************************************************************************/ typedef enum { backupTypeFull = STRID5("full", 0x632a60), backupTypeDiff = STRID5("diff", 0x319240), backupTypeIncr = STRID5("incr", 0x90dc90), } BackupType; #include "common/type/object.h" #include "common/type/string.h" #include "common/type/stringList.h" #include "info/infoPg.h" #include "info/manifest/manifest.h" #include "storage/storage.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define INFO_BACKUP_FILE "backup.info" #define INFO_BACKUP_PATH_FILE STORAGE_REPO_BACKUP "/" INFO_BACKUP_FILE STRING_DECLARE(INFO_BACKUP_PATH_FILE_STR); #define INFO_BACKUP_PATH_FILE_COPY INFO_BACKUP_PATH_FILE INFO_COPY_EXT STRING_DECLARE(INFO_BACKUP_PATH_FILE_COPY_STR); /*********************************************************************************************************************************** Information about an existing backup ***********************************************************************************************************************************/ typedef struct InfoBackupData { const String *backupLabel; // backupLabel must be first to allow for built-in list sorting unsigned int backrestFormat; const String *backrestVersion; Variant *backupAnnotation; // Backup annotations, if present const String *backupArchiveStart; const String *backupArchiveStop; uint64_t backupInfoRepoSize; uint64_t backupInfoRepoSizeDelta; const Variant *backupInfoRepoSizeMap; const Variant *backupInfoRepoSizeMapDelta; uint64_t backupInfoSize; uint64_t backupInfoSizeDelta; const String *backupLsnStart; const String *backupLsnStop; unsigned int backupPgId; const String *backupPrior; StringList *backupReference; time_t backupTimestampStart; time_t backupTimestampStop; BackupType backupType; const Variant *backupError; // Were errors detected during the backup? bool optionArchiveCheck; bool optionArchiveCopy; bool optionBackupStandby; bool optionChecksumPage; bool optionCompress; bool optionHardlink; bool optionOnline; } InfoBackupData; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN InfoBackup *infoBackupNew( unsigned int pgVersion, uint64_t pgSystemId, unsigned int pgCatalogVersion, const String *cipherPassSub); // Create new object and load contents from IoRead FN_EXTERN InfoBackup *infoBackupNewLoad(IoRead *read); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct InfoBackupPub { MemContext *memContext; // Mem context InfoPg *infoPg; // Contents of the DB data List *backup; // List of current backups and their associated data bool updated; // Has the info been updated since the last save? } InfoBackupPub; // PostgreSQL info FN_INLINE_ALWAYS InfoPg * infoBackupPg(const InfoBackup *const this) { return THIS_PUB(InfoBackup)->infoPg; } FN_EXTERN InfoBackup *infoBackupPgSet(InfoBackup *this, unsigned int pgVersion, uint64_t pgSystemId, unsigned int pgCatalogVersion); // Cipher passphrase FN_INLINE_ALWAYS const String * infoBackupCipherPass(const InfoBackup *const this) { return infoPgCipherPass(infoBackupPg(this)); } // Return a structure of the backup data from a specific index FN_EXTERN InfoBackupData infoBackupData(const InfoBackup *this, unsigned int backupDataIdx); // Does the specified backup label exist? FN_INLINE_ALWAYS bool infoBackupLabelExists(const InfoBackup *const this, const String *const backupLabel) { ASSERT_INLINE(backupLabel != NULL); return lstFind(THIS_PUB(InfoBackup)->backup, &backupLabel) != NULL; } // Return a pointer to a structure from the current backup data given a label, else NULL FN_INLINE_ALWAYS InfoBackupData * infoBackupDataByLabel(const InfoBackup *const this, const String *const backupLabel) { ASSERT_INLINE(infoBackupLabelExists(this, backupLabel)); return lstFind(THIS_PUB(InfoBackup)->backup, &backupLabel); } // Get total current backups FN_INLINE_ALWAYS unsigned int infoBackupDataTotal(const InfoBackup *const this) { return lstSize(THIS_PUB(InfoBackup)->backup); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add backup to the current list FN_EXTERN void infoBackupDataAdd(InfoBackup *this, const Manifest *manifest); // Set Annotation in the backup data for a specific backup label FN_EXTERN void infoBackupDataAnnotationSet(InfoBackup *this, const String *const backupLabel, const KeyValue *annotationKv); // Delete backup from the current backup list FN_EXTERN void infoBackupDataDelete(InfoBackup *this, const String *backupDeleteLabel); // Given a backup label, get the dependency list FN_EXTERN StringList *infoBackupDataDependentList(const InfoBackup *this, const String *backupLabel); // Return a list of current backup labels, applying a regex expression if provided FN_EXTERN StringList *infoBackupDataLabelList(const InfoBackup *this, const String *expression); // Move to a new parent mem context FN_INLINE_ALWAYS InfoBackup * infoBackupMove(InfoBackup *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void infoBackupFree(InfoBackup *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Load backup info FN_EXTERN InfoBackup *infoBackupLoadFile( const Storage *storage, const String *fileName, CipherType cipherType, const String *cipherPass); // Load backup info and update it by adding valid backups from the repo or removing backups no longer in the repo FN_EXTERN InfoBackup *infoBackupLoadFileReconstruct( const Storage *storage, const String *fileName, CipherType cipherType, const String *cipherPass); // Save backup info FN_EXTERN void infoBackupSaveFile( InfoBackup *infoBackup, const Storage *storage, const String *fileName, CipherType cipherType, const String *cipherPass); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void infoBackupDataToLog(const InfoBackupData *this, StringStatic *debugLog); #define FUNCTION_LOG_INFO_BACKUP_TYPE \ InfoBackup * #define FUNCTION_LOG_INFO_BACKUP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoBackup", buffer, bufferSize) #define FUNCTION_LOG_INFO_BACKUP_DATA_TYPE \ InfoBackupData #define FUNCTION_LOG_INFO_BACKUP_DATA_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(&value, infoBackupDataToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/infoPg.c0000664000175100017510000004215515240547630016532 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Info Handler ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include "common/debug.h" #include "common/ini.h" #include "common/log.h" #include "common/type/json.h" #include "common/type/list.h" #include "common/type/object.h" #include "info/info.h" #include "info/infoPg.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct InfoPg { InfoPgPub pub; // Publicly accessible variables MemContext *memContext; // Mem context InfoPgType type; // Type of info file being loaded unsigned int historyCurrent; // Index of the current history item }; /*********************************************************************************************************************************** Internal constructor ***********************************************************************************************************************************/ static InfoPg * infoPgNewInternal(const InfoPgType type) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); InfoPg *this = OBJ_NEW_ALLOC(); *this = (InfoPg) { .pub = { .history = lstNewP(sizeof(InfoPgData)), }, .memContext = memContextCurrent(), .type = type, }; FUNCTION_TEST_RETURN(INFO_PG, this); } /**********************************************************************************************************************************/ FN_EXTERN InfoPg * infoPgNew(const InfoPgType type, const String *const cipherPassSub) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(STRING, cipherPassSub); FUNCTION_LOG_END(); InfoPg *this; OBJ_NEW_BASE_BEGIN(InfoPg, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoPgNewInternal(type); this->pub.info = infoNew(cipherPassSub); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_PG, this); } /**********************************************************************************************************************************/ #define INFO_SECTION_DB "db" #define INFO_SECTION_DB_HISTORY "db:history" #define INFO_KEY_DB_CATALOG_VERSION "db-catalog-version" #define INFO_KEY_DB_CONTROL_VERSION "db-control-version" #define INFO_KEY_DB_SYSTEM_ID "db-system-id" #define INFO_KEY_DB_VERSION "db-version" typedef struct InfoPgLoadData { InfoLoadNewCallback *callbackFunction; // Callback function for child object void *callbackData; // Callback data for child object InfoPg *infoPg; // Pg info unsigned int currentId; // Current database id } InfoPgLoadData; static void infoPgLoadCallback(void *const data, const String *const section, const String *const key, JsonRead *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(JSON_READ, json); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(data != NULL); ASSERT(section != NULL); ASSERT(key != NULL); ASSERT(json != NULL); InfoPgLoadData *const loadData = data; // Process db section if (strEqZ(section, INFO_SECTION_DB)) { if (strEqZ(key, INFO_KEY_DB_ID)) loadData->currentId = jsonReadUInt(json); } // Process db:history section else if (strEqZ(section, INFO_SECTION_DB_HISTORY)) { InfoPgData infoPgData = {.id = cvtZToUInt(strZ(key))}; jsonReadObjectBegin(json); // Catalog version if (loadData->infoPg->type == infoPgBackup) { infoPgData.catalogVersion = jsonReadUInt(jsonReadKeyRequireZ(json, INFO_KEY_DB_CATALOG_VERSION)); infoPgData.controlVersion = jsonReadUInt(jsonReadKeyRequireZ(json, INFO_KEY_DB_CONTROL_VERSION)); } // System id infoPgData.systemId = jsonReadUInt64( jsonReadKeyRequireZ(json, loadData->infoPg->type == infoPgArchive ? INFO_KEY_DB_ID : INFO_KEY_DB_SYSTEM_ID)); // PostgreSQL version infoPgData.version = pgVersionFromStr(jsonReadStr(jsonReadKeyRequireZ(json, INFO_KEY_DB_VERSION))); // Insert at beginning of list so the history is reverse ordered lstInsert(loadData->infoPg->pub.history, 0, &infoPgData); } // Callback if set else if (loadData->callbackFunction != NULL) loadData->callbackFunction(loadData->callbackData, section, key, json); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN InfoPg * infoPgNewLoad(IoRead *const read, const InfoPgType type, InfoLoadNewCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(read != NULL); ASSERT(type == infoPgBackup || type == infoPgArchive); ASSERT((callbackFunction == NULL && callbackData == NULL) || (callbackFunction != NULL && callbackData != NULL)); InfoPg *this; OBJ_NEW_BASE_BEGIN(InfoPg, .childQty = MEM_CONTEXT_QTY_MAX) { this = infoPgNewInternal(type); // Set historyCurrent to UINT_MAX so we can detect if it was loaded correctly this->historyCurrent = UINT_MAX; // Load InfoPgLoadData loadData = { .callbackFunction = callbackFunction, .callbackData = callbackData, .infoPg = this, }; this->pub.info = infoNewLoad(read, infoPgLoadCallback, &loadData); CHECK(FormatError, !lstEmpty(this->pub.history), "history is missing"); CHECK(FormatError, loadData.currentId > 0, "current id is missing"); // Find the current history item for (unsigned int historyIdx = 0; historyIdx < lstSize(this->pub.history); historyIdx++) { if (((InfoPgData *)lstGet(this->pub.history, historyIdx))->id == loadData.currentId) this->historyCurrent = historyIdx; } CHECK(FormatError, this->historyCurrent != UINT_MAX, "unable to find current id in history"); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(INFO_PG, this); } /**********************************************************************************************************************************/ FN_EXTERN void infoPgAdd(InfoPg *const this, const InfoPgData *const infoPgData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM_P(INFO_PG_DATA, infoPgData); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(infoPgData != NULL); lstInsert(this->pub.history, 0, infoPgData); this->historyCurrent = 0; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN InfoPg * infoPgSet( InfoPg *const this, const InfoPgType type, const unsigned int pgVersion, const uint64_t pgSystemId, const unsigned int pgCatalogVersion) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(UINT, pgCatalogVersion); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { unsigned int pgDataId = 1; // If there is some history, then get the historyId of the most current and increment it if (infoPgDataTotal(this) > 0) pgDataId = infoPgCurrentDataId(this) + 1; // Set db values that are common to all info files InfoPgData infoPgData = { .id = pgDataId, .version = pgVersion, // This is different in archive.info due to a typo that can't be fixed without a format version bump .systemId = pgSystemId, }; // Catalog/control version required for backup info to preserve the repo format if (this->type == infoPgBackup) { infoPgData.catalogVersion = pgCatalogVersion; infoPgData.controlVersion = pgControlVersion(pgVersion); } // Add the pg data to the history list infoPgAdd(this, &infoPgData); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(INFO_PG, this); } /**********************************************************************************************************************************/ typedef struct InfoPgSaveData { InfoSaveCallback *callbackFunction; // Callback function for child object void *callbackData; // Callback data for child object InfoPgType type; // Type of info file being loaded InfoPg *infoPg; // InfoPg object to be saved } InfoPgSaveData; static void infoPgSaveCallback(void *const data, const String *const sectionNext, InfoSave *const infoSaveData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(STRING, sectionNext); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(data != NULL); ASSERT(infoSaveData != NULL); InfoPgSaveData *const saveData = (InfoPgSaveData *)data; if (infoSaveSection(infoSaveData, INFO_SECTION_DB, sectionNext)) { if (saveData->callbackFunction != NULL) saveData->callbackFunction(saveData->callbackData, STRDEF(INFO_SECTION_DB), infoSaveData); const InfoPgData pgData = infoPgDataCurrent(saveData->infoPg); // These need to be saved because older pgBackRest versions expect them if (saveData->infoPg->type == infoPgBackup) { infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_CATALOG_VERSION, jsonFromVar(VARUINT(pgData.catalogVersion))); infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_CONTROL_VERSION, jsonFromVar(VARUINT(pgData.controlVersion))); } infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_ID, jsonFromVar(VARUINT(pgData.id))); infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_SYSTEM_ID, jsonFromVar(VARUINT64(pgData.systemId))); infoSaveValue(infoSaveData, INFO_SECTION_DB, INFO_KEY_DB_VERSION, jsonFromVar(VARSTR(pgVersionToStr(pgData.version)))); } if (infoSaveSection(infoSaveData, INFO_SECTION_DB_HISTORY, sectionNext)) { if (saveData->callbackFunction != NULL) saveData->callbackFunction(saveData->callbackData, STRDEF(INFO_SECTION_DB_HISTORY), infoSaveData); // Set the db history section in reverse so oldest history is first instead of last to be consistent with load for (unsigned int pgDataIdx = infoPgDataTotal(saveData->infoPg) - 1; (int)pgDataIdx >= 0; pgDataIdx--) { const InfoPgData pgData = infoPgData(saveData->infoPg, pgDataIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); // These need to be saved because older pgBackRest versions expect them if (saveData->infoPg->type == infoPgBackup) { jsonWriteUInt(jsonWriteKeyZ(json, INFO_KEY_DB_CATALOG_VERSION), pgData.catalogVersion); jsonWriteUInt(jsonWriteKeyZ(json, INFO_KEY_DB_CONTROL_VERSION), pgData.controlVersion); } if (saveData->infoPg->type == infoPgArchive) jsonWriteUInt64(jsonWriteKeyZ(json, INFO_KEY_DB_ID), pgData.systemId); if (saveData->infoPg->type == infoPgBackup) jsonWriteUInt64(jsonWriteKeyZ(json, INFO_KEY_DB_SYSTEM_ID), pgData.systemId); jsonWriteStr(jsonWriteKeyZ(json, INFO_KEY_DB_VERSION), pgVersionToStr(pgData.version)); infoSaveValue( infoSaveData, INFO_SECTION_DB_HISTORY, strZ(varStrForce(VARUINT(pgData.id))), jsonWriteResult(jsonWriteObjectEnd(json))); } } // Process the callback even if none of the sections above get executed if (saveData->callbackFunction != NULL) saveData->callbackFunction(saveData->callbackData, sectionNext, infoSaveData); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void infoPgSave(InfoPg *const this, IoWrite *const write, InfoSaveCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); ASSERT((callbackFunction == NULL && callbackData == NULL) || (callbackFunction != NULL && callbackData != NULL)); MEM_CONTEXT_TEMP_BEGIN() { InfoPgSaveData saveData = { .callbackFunction = callbackFunction, .callbackData = callbackData, .infoPg = this, }; infoSave(infoPgInfo(this), write, infoPgSaveCallback, &saveData); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * infoPgArchiveId(const InfoPg *const this, const unsigned int pgDataIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM(UINT, pgDataIdx); FUNCTION_LOG_END(); ASSERT(this != NULL); const InfoPgData pgData = infoPgData(this, pgDataIdx); String *const version = pgVersionToStr(pgData.version); String *const result = strNewFmt("%s-%u", strZ(version), pgData.id); strFree(version); FUNCTION_LOG_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN InfoPgData infoPgData(const InfoPg *const this, const unsigned int pgDataIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_PARAM(UINT, pgDataIdx); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INFO_PG_DATA, *(InfoPgData *)lstGet(this->pub.history, pgDataIdx)); } /**********************************************************************************************************************************/ FN_EXTERN InfoPgData infoPgDataCurrent(const InfoPg *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(INFO_PG_DATA, infoPgData(this, infoPgDataCurrentId(this))); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int infoPgDataCurrentId(const InfoPg *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_END(); ASSERT(this != NULL); FUNCTION_LOG_RETURN(UINT, this->historyCurrent); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int infoPgCurrentDataId(const InfoPg *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(INFO_PG, this); FUNCTION_LOG_END(); ASSERT(this != NULL); InfoPgData currentData = infoPgDataCurrent(this); FUNCTION_LOG_RETURN(UINT, currentData.id); } /**********************************************************************************************************************************/ FN_EXTERN void infoPgDataToLog(const InfoPgData *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{id: %u, version: %u, systemId: %" PRIu64 ", catalogVersion: %u}", this->id, this->version, this->systemId, this->catalogVersion); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/infoPg.h0000664000175100017510000001367515240547630016544 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Info Handler ***********************************************************************************************************************************/ #ifndef INFO_INFOPG_H #define INFO_INFOPG_H #include /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct InfoPg InfoPg; #include "common/crypto/common.h" #include "common/ini.h" #include "info/info.h" #include "storage/storage.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define INFO_KEY_DB_ID "db-id" /*********************************************************************************************************************************** Information about the PostgreSQL cluster ***********************************************************************************************************************************/ typedef struct InfoPgData { unsigned int id; uint64_t systemId; unsigned int catalogVersion; unsigned int version; // Control version is required to maintain the file format for older versions of pgBackRest but should not be used elsewhere unsigned int controlVersion; } InfoPgData; /*********************************************************************************************************************************** Info types for determining data in DB section ***********************************************************************************************************************************/ typedef enum { infoPgArchive = STRID5("archive", 0x16c940e410), // archive info file infoPgBackup = STRID5("backup", 0x21558c220), // backup info file } InfoPgType; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN InfoPg *infoPgNew(InfoPgType type, const String *cipherPassSub); // Create new object and load contents from a file FN_EXTERN InfoPg *infoPgNewLoad(IoRead *read, InfoPgType type, InfoLoadNewCallback *callbackFunction, void *callbackData); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct InfoPgPub { Info *info; // Info contents List *history; // A list of InfoPgData } InfoPgPub; // Archive id FN_EXTERN String *infoPgArchiveId(const InfoPg *this, unsigned int pgDataIdx); // Base info FN_INLINE_ALWAYS Info * infoPgInfo(const InfoPg *const this) { return THIS_PUB(InfoPg)->info; } // Return the cipher passphrase FN_INLINE_ALWAYS const String * infoPgCipherPass(const InfoPg *const this) { return infoCipherPass(infoPgInfo(this)); } // Return current pgId from the history FN_EXTERN unsigned int infoPgCurrentDataId(const InfoPg *this); // PostgreSQL info FN_EXTERN InfoPgData infoPgData(const InfoPg *this, unsigned int pgDataIdx); // Current PostgreSQL data FN_EXTERN InfoPgData infoPgDataCurrent(const InfoPg *this); // Current history index FN_EXTERN unsigned int infoPgDataCurrentId(const InfoPg *this); // Total PostgreSQL data in the history FN_INLINE_ALWAYS unsigned int infoPgDataTotal(const InfoPg *const this) { return lstSize(THIS_PUB(InfoPg)->history); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add Postgres data to the history list at position 0 to ensure the latest history is always first in the list FN_EXTERN void infoPgAdd(InfoPg *this, const InfoPgData *infoPgData); // Save to IO FN_EXTERN void infoPgSave(InfoPg *this, IoWrite *write, InfoSaveCallback *callbackFunction, void *callbackData); // Set the InfoPg object data based on values passed FN_EXTERN InfoPg *infoPgSet( InfoPg *this, InfoPgType type, const unsigned int pgVersion, const uint64_t pgSystemId, const unsigned int pgCatalogVersion); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void infoPgDataToLog(const InfoPgData *this, StringStatic *debugLog); #define FUNCTION_LOG_INFO_PG_TYPE \ InfoPg * #define FUNCTION_LOG_INFO_PG_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "InfoPg", buffer, bufferSize) #define FUNCTION_LOG_INFO_PG_DATA_TYPE \ InfoPgData #define FUNCTION_LOG_INFO_PG_DATA_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(&value, infoPgDataToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/manifest/0000775000175100017510000000000015240547630016743 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/manifest/build.c.inc0000664000175100017510000005673715240547630021000 0ustar00runnerrunner/*********************************************************************************************************************************** Build manifest from PGDATA ***********************************************************************************************************************************/ typedef struct ManifestBuildData { Manifest *manifest; // Manifest being build const Storage *storagePg; // PostgreSQL storage const String *tablespaceId; // Tablespace id if PostgreSQL version has one bool online; // Is this an online backup? bool checksumPage; // Are page checksums being checked? const String *manifestWalName; // Wal manifest name for this version of PostgreSQL RegExp *dbPathExp; // Identify paths containing relations RegExp *tempRelationExp; // Identify temp relations const Pack *tablespaceList; // List of tablespaces in the database ManifestLinkCheck *linkCheck; // List of links found during build (used for prefix check) StringList *excludeContent; // Exclude contents of directories StringList *excludeSingle; // Exclude a single file/link/path const ManifestBlockIncrMap *blockIncrMap; // Block incremental maps } ManifestBuildData; // Calculate block incremental size for a file. The block size is based on the size and age of the file. Larger files get larger // block sizes to reduce the cost of the map. Older files also get larger block sizes under the assumption that they are unlikely to // be modified if they have not been modified in a while. Very old and very small files skip block incremental entirely. // // Smaller blocks will be compressed/encrypted together in a larger super block for efficiency. static size_t manifestBuildBlockIncrSize(const ManifestBuildData *const buildData, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buildData); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); size_t result = 0; // Search size map for the appropriate block size for (unsigned int sizeIdx = 0; sizeIdx < lstSize(buildData->blockIncrMap->sizeMap); sizeIdx++) { const ManifestBlockIncrSizeMap *const sizeMap = lstGet(buildData->blockIncrMap->sizeMap, sizeIdx); if (file->size >= sizeMap->fileSize) { result = sizeMap->blockSize; break; } } // If block size > 0 then search age map for a multiplier if (result != 0) { const time_t fileAge = buildData->manifest->pub.data.backupTimestampStart - file->timestamp; for (unsigned int ageIdx = 0; ageIdx < lstSize(buildData->blockIncrMap->ageMap); ageIdx++) { const ManifestBlockIncrAgeMap *const ageMap = lstGet(buildData->blockIncrMap->ageMap, ageIdx); if (fileAge >= (time_t)ageMap->fileAge) { result *= ageMap->blockMultiplier; break; } } } FUNCTION_TEST_RETURN(SIZE, result); } // Get checksum size for a block size. Since these checksums are used to determine if a block has changed (not for corruption) they // should be set larger than usual. For example, it is common to use 32-bit checksums to check for corruption in blocks up to 4MiB // but here we used more bits to ensure block changes are correctly detected, since valid changes may look a lot different than // corruption. static size_t manifestBuildBlockIncrChecksumSize(const ManifestBuildData *const buildData, const size_t blockSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buildData); FUNCTION_TEST_PARAM(SIZE, blockSize); FUNCTION_TEST_END(); size_t result = BLOCK_INCR_CHECKSUM_SIZE_MIN; // Search checksum size map for larger value for (unsigned int sizeIdx = 0; sizeIdx < lstSize(buildData->blockIncrMap->checksumSizeMap); sizeIdx++) { const ManifestBlockIncrChecksumSizeMap *const sizeMap = lstGet(buildData->blockIncrMap->checksumSizeMap, sizeIdx); if (blockSize >= sizeMap->blockSize) { result = sizeMap->checksumSize; break; } } FUNCTION_TEST_RETURN(SIZE, result); } // Process files/links/paths and add them to the manifest static void manifestBuildInfo( ManifestBuildData *const buildData, const String *manifestParentName, const String *pgPath, const bool dbPath, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, buildData); FUNCTION_TEST_PARAM(STRING, manifestParentName); FUNCTION_TEST_PARAM(STRING, pgPath); FUNCTION_TEST_PARAM(BOOL, dbPath); FUNCTION_TEST_PARAM(STORAGE_INFO, *info); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(buildData != NULL); ASSERT(manifestParentName != NULL); ASSERT(pgPath != NULL); ASSERT(info != NULL); // Skip any path/file/link that begins with pgsql_tmp. The files are removed when the server is restarted and the directories // are recreated. if (strBeginsWithZ(info->name, PG_PREFIX_PGSQLTMP)) FUNCTION_TEST_RETURN_VOID(); // Get build data const unsigned int pgVersion = buildData->manifest->pub.data.pgVersion; // Construct the name used to identify this file/link/path in the manifest const String *manifestName = strNewFmt("%s/%s", strZ(manifestParentName), strZ(info->name)); // Skip excluded files/links/paths if (buildData->excludeSingle != NULL && strLstExists(buildData->excludeSingle, manifestName)) { LOG_INFO_FMT( "exclude '%s/%s' from backup using '%s' exclusion", strZ(pgPath), strZ(info->name), strZ(strSub(manifestName, sizeof(MANIFEST_TARGET_PGDATA)))); FUNCTION_TEST_RETURN_VOID(); } // Process storage types switch (info->type) { // Add paths // ------------------------------------------------------------------------------------------------------------------------- case storageTypePath: { // There should not be any paths in pg_tblspc if (strEqZ(manifestParentName, MANIFEST_TARGET_PGDATA "/" MANIFEST_TARGET_PGTBLSPC)) { THROW_FMT( LinkExpectedError, "'%s' is not a symlink - " MANIFEST_TARGET_PGTBLSPC " should contain only symlinks", strZ(manifestName)); } // Add path to manifest const ManifestPath path = { .name = manifestName, .mode = info->mode, .user = info->user, .group = info->group, }; manifestPathAdd(buildData->manifest, &path); // Skip excluded path content if (buildData->excludeContent != NULL && strLstExists(buildData->excludeContent, manifestName)) { LOG_INFO_FMT( "exclude contents of '%s/%s' from backup using '%s/' exclusion", strZ(pgPath), strZ(info->name), strZ(strSub(manifestName, sizeof(MANIFEST_TARGET_PGDATA)))); FUNCTION_TEST_RETURN_VOID(); } // Skip the contents of these paths if they exist in the base path since they won't be reused after recovery if (strEq(manifestParentName, MANIFEST_TARGET_PGDATA_STR)) { // Skip pg_dynshmem/* since these files cannot be reused on recovery if (strEqZ(info->name, PG_PATH_PGDYNSHMEM)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_notify/* since these files cannot be reused on recovery if (strEqZ(info->name, PG_PATH_PGNOTIFY)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_replslot/* since these files are generally not useful after a restore if (strEqZ(info->name, PG_PATH_PGREPLSLOT)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_serial/* since these files are reset if (strEqZ(info->name, PG_PATH_PGSERIAL)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_snapshots/* since these files cannot be reused on recovery if (strEqZ(info->name, PG_PATH_PGSNAPSHOTS)) FUNCTION_TEST_RETURN_VOID(); // Skip temporary statistics in pg_stat_tmp even when stats_temp_directory is set because PGSS_TEXT_FILE is always // created there in PostgreSQL < 15. PostgreSQL >= 15 no longer uses this directory, but it may be used by // extensions such as pg_stat_statements so it should still be excluded. if (strEqZ(info->name, PG_PATH_PGSTATTMP)) FUNCTION_TEST_RETURN_VOID(); // Skip pg_subtrans/* since these files are reset if (strEqZ(info->name, PG_PATH_PGSUBTRANS)) FUNCTION_TEST_RETURN_VOID(); } // Skip the contents of archive_status when online if (buildData->online && strEq(manifestParentName, buildData->manifestWalName) && strEqZ(info->name, PG_PATH_ARCHIVE_STATUS)) { FUNCTION_TEST_RETURN_VOID(); } // Recurse into the path const String *const pgPathSub = strNewFmt("%s/%s", strZ(pgPath), strZ(info->name)); const bool dbPathSub = regExpMatch(buildData->dbPathExp, manifestName); StorageIterator *const storageItr = storageNewItrP(buildData->storagePg, pgPathSub, .sortOrder = sortOrderAsc); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); manifestBuildInfo(buildData, manifestName, pgPathSub, dbPathSub, &info); // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); break; } // Add files // ------------------------------------------------------------------------------------------------------------------------- case storageTypeFile: { // There should not be any files in pg_tblspc if (strEqZ(manifestParentName, MANIFEST_TARGET_PGDATA "/" MANIFEST_TARGET_PGTBLSPC)) { THROW_FMT( LinkExpectedError, "'%s' is not a symlink - " MANIFEST_TARGET_PGTBLSPC " should contain only symlinks", strZ(manifestName)); } // Skip pg_internal.init since it is recreated on startup. It's also possible, (though unlikely) that a temp file with // the creating process id as the extension can exist so skip that as well. This seems to be a bug in PostgreSQL since // the temp file should be removed on startup. Use regExpMatchOne() here instead of preparing a regexp in advance since // the likelihood of needing the regexp should be very small. if (dbPath && strBeginsWithZ(info->name, PG_FILE_PGINTERNALINIT) && (strSize(info->name) == sizeof(PG_FILE_PGINTERNALINIT) - 1 || regExpMatchOne(STRDEF("\\.[0-9]+"), strSub(info->name, sizeof(PG_FILE_PGINTERNALINIT) - 1)))) { FUNCTION_TEST_RETURN_VOID(); } // Skip files in the root data path if (strEq(manifestParentName, MANIFEST_TARGET_PGDATA_STR)) { // Skip recovery files if (((strEqZ(info->name, PG_FILE_RECOVERYSIGNAL) || strEqZ(info->name, PG_FILE_STANDBYSIGNAL)) && pgVersion >= PG_VERSION_12) || ((strEqZ(info->name, PG_FILE_RECOVERYCONF) || strEqZ(info->name, PG_FILE_RECOVERYDONE)) && pgVersion < PG_VERSION_12) || // Skip temp file for safely writing postgresql.auto.conf (strEqZ(info->name, PG_FILE_POSTGRESQLAUTOCONFTMP)) || // Skip obsolete backup_label/backup_label.old strEqZ(info->name, PG_FILE_BACKUPLABEL) || strEqZ(info->name, PG_FILE_BACKUPLABELOLD) || // Skip backup_manifest/tmp in versions where it is created ((strEqZ(info->name, PG_FILE_BACKUPMANIFEST) || strEqZ(info->name, PG_FILE_BACKUPMANIFEST_TMP)) && pgVersion >= PG_VERSION_13) || // Skip running process options strEqZ(info->name, PG_FILE_POSTMTROPTS) || // Skip process id file to avoid confusing postgres after restore strEqZ(info->name, PG_FILE_POSTMTRPID)) { FUNCTION_TEST_RETURN_VOID(); } } // Skip the contents of the wal path when online. WAL will be restored from the archive or stored in the wal directory // at the end of the backup if the archive-copy option is set. if (buildData->online && strEq(manifestParentName, buildData->manifestWalName)) FUNCTION_TEST_RETURN_VOID(); // Skip temp relations in db paths if (dbPath && regExpMatch(buildData->tempRelationExp, info->name)) FUNCTION_TEST_RETURN_VOID(); // Add file to manifest ManifestFile file = { .name = manifestName, .copy = true, .mode = info->mode, .user = info->user, .group = info->group, .size = info->size, .sizeOriginal = info->size, .sizeRepo = info->size, .timestamp = info->timeModified, }; // When bundling zero-length files do not need to be copied if (info->size == 0 && buildData->manifest->pub.data.bundle) { file.copy = false; file.checksumSha1 = bufPtrConst(HASH_TYPE_SHA1_ZERO_BUF); } // Get block incremental size if (info->size != 0 && buildData->manifest->pub.data.blockIncr) { file.blockIncrSize = manifestBuildBlockIncrSize(buildData, &file); file.blockIncrChecksumSize = manifestBuildBlockIncrChecksumSize(buildData, file.blockIncrSize); } // Determine if this file should be page checksummed if (dbPath && buildData->checksumPage) { file.checksumPage = !strEndsWithZ(manifestName, "/" PG_FILE_PGFILENODEMAP) && !strEndsWithZ(manifestName, "/" PG_FILE_PGVERSION) && !strEqZ(manifestName, MANIFEST_TARGET_PGDATA "/" PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL); } manifestFileAdd(buildData->manifest, &file); break; } // Add links // ------------------------------------------------------------------------------------------------------------------------- case storageTypeLink: { // If the destination is another link then error. In the future we'll allow this by following the link chain to the // eventual destination but for now we are trying to maintain compatibility during the migration. To do this check we // need to read outside of the data directory but it is a read-only operation so is considered safe. const String *const linkDestinationAbsolute = strPathAbsolute(info->linkDestination, pgPath); const StorageInfo linkedCheck = storageInfoP( buildData->storagePg, linkDestinationAbsolute, .ignoreMissing = true, .noPathEnforce = true); if (linkedCheck.exists && linkedCheck.type == storageTypeLink) { THROW_FMT( LinkDestinationError, "link '%s/%s' cannot reference another link '%s'", strZ(pgPath), strZ(info->name), strZ(linkDestinationAbsolute)); } // Initialize link and target const ManifestLink link = { .name = manifestName, .user = info->user, .group = info->group, .destination = info->linkDestination, }; ManifestTarget target = { .name = manifestName, .type = manifestTargetTypeLink, }; // Make a copy of the link name because it will need to be modified when there are tablespace ids const String *linkName = info->name; // Is this a tablespace? if (strEq(manifestParentName, STRDEF(MANIFEST_TARGET_PGDATA "/" MANIFEST_TARGET_PGTBLSPC))) { // Strip pg_data off the manifest name so it begins with pg_tblspc instead. This reflects how the files are stored // in the backup directory. manifestName = strSub(manifestName, sizeof(MANIFEST_TARGET_PGDATA)); // Identify this target as a tablespace target.name = manifestName; target.tablespaceId = cvtZToUInt(strZ(info->name)); // Look for this tablespace in the provided list (list may be null for off-line backup) if (buildData->tablespaceList != NULL) { // Search list PackRead *const read = pckReadNew(buildData->tablespaceList); while (!pckReadNullP(read)) { pckReadArrayBeginP(read); if (target.tablespaceId == pckReadU32P(read)) target.tablespaceName = pckReadStrP(read); pckReadArrayEndP(read); } // Error if the tablespace could not be found. ??? This seems excessive, perhaps just warn here? if (target.tablespaceName == NULL) { THROW_FMT( AssertError, "tablespace with oid %u not found in tablespace map\n" "HINT: was a tablespace created or dropped during the backup?", target.tablespaceId); } } // If no tablespace name was found then create one if (target.tablespaceName == NULL) target.tablespaceName = strNewFmt("ts%s", strZ(info->name)); // Add a dummy pg_tblspc path entry if it does not already exist. This entry will be ignored by restore but it is // part of the original manifest format so we need to have it. lstSort(buildData->manifest->pub.pathList, sortOrderAsc); const ManifestPath *const pathBase = manifestPathFind(buildData->manifest, MANIFEST_TARGET_PGDATA_STR); if (manifestPathFindDefault(buildData->manifest, MANIFEST_TARGET_PGTBLSPC_STR, NULL) == NULL) { ManifestPath path = { .name = MANIFEST_TARGET_PGTBLSPC_STR, .mode = pathBase->mode, .user = pathBase->user, .group = pathBase->group, }; manifestPathAdd(buildData->manifest, &path); } // The tablespace link destination path is not the path where data will be stored so we can just store it as a dummy // path. This is because PostgreSQL creates a subpath with the version/catalog number so that multiple versions of // PostgreSQL can share a tablespace, which makes upgrades easier. const ManifestPath *const pathTblSpc = manifestPathFind( buildData->manifest, STRDEF(MANIFEST_TARGET_PGDATA "/" MANIFEST_TARGET_PGTBLSPC)); const ManifestPath path = { .name = manifestName, .mode = pathTblSpc->mode, .user = pathTblSpc->user, .group = pathTblSpc->group, }; manifestPathAdd(buildData->manifest, &path); // Update build structure to reflect the path added above and the tablespace id manifestParentName = manifestName; manifestName = strNewFmt("%s/%s", strZ(manifestName), strZ(buildData->tablespaceId)); pgPath = strNewFmt("%s/%s", strZ(pgPath), strZ(info->name)); linkName = buildData->tablespaceId; } // Add info about the linked file/path const String *const linkPgPath = strNewFmt("%s/%s", strZ(pgPath), strZ(linkName)); StorageInfo linkedInfo = storageInfoP( buildData->storagePg, linkPgPath, .followLink = true, .ignoreMissing = true); linkedInfo.name = linkName; // If the link destination exists then build the target if (linkedInfo.exists) { // If a path link then recurse if (linkedInfo.type == storageTypePath) { target.path = info->linkDestination; } // Else it must be a file or special (since we have already checked if it is a link) else { CHECK(FormatError, target.tablespaceId == 0, "tablespace links to a file"); // Identify target as a file target.path = strPath(info->linkDestination); target.file = strBase(info->linkDestination); } } // Else dummy up the target with a destination so manifestLinkCheck() can be run. This is so errors about links with // destinations in PGDATA will take precedence over missing a destination. We will probably simplify this once the // migration is done and it doesn't matter which error takes precedence. else target.path = info->linkDestination; // Add target and link manifestTargetAdd(buildData->manifest, &target); manifestLinkAdd(buildData->manifest, &link); // Make sure the link is valid manifestLinkCheckOne(buildData->manifest, buildData->linkCheck, manifestTargetTotal(buildData->manifest) - 1); // If the link check was successful but the destination does not exist then check it again to generate an error if (!linkedInfo.exists) storageInfoP(buildData->storagePg, linkPgPath, .followLink = true); // Recurse into the link destination manifestBuildInfo(buildData, manifestParentName, pgPath, dbPath, &linkedInfo); break; } // Skip special files // ------------------------------------------------------------------------------------------------------------------------- case storageTypeSpecial: LOG_WARN_FMT("exclude special file '%s/%s' from backup", strZ(pgPath), strZ(info->name)); break; } FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/manifest/link.c.inc0000664000175100017510000002265315240547630020624 0ustar00runnerrunner/*********************************************************************************************************************************** Ensure that symlinks do not point to the same file, directory, or subdirectory of another link There are two implementations: manifestLinkCheck(), which is externed, and manifestLinkCheckOne(), which is intended to be used internally during processing. manifestLinkCheck() works simply by calling manifestLinkCheckOne() for every link in the target list. manifestLinkCheckOne() is optimized to work quickly on a single link. ***********************************************************************************************************************************/ // Data needed in link list typedef struct ManifestLinkCheckItem { const String *path; // Link destination path terminated with / const String *file; // Link file if a file link unsigned int targetIdx; // Index of target used for error messages } ManifestLinkCheckItem; // Persistent data needed during processing of manifestLinkCheck/One() typedef struct ManifestLinkCheck { const String *basePath; // Base data path (initialized on first call) List *linkList; // Current list of link destination paths } ManifestLinkCheck; // Helper to initialize the link data static ManifestLinkCheck manifestLinkCheckInit(void) { FUNCTION_TEST_VOID(); FUNCTION_AUDIT_STRUCT(); FUNCTION_TEST_RETURN_TYPE( ManifestLinkCheck, (ManifestLinkCheck){.linkList = lstNewP(sizeof(ManifestLinkCheckItem), .comparator = lstComparatorStr)}); } // Helper to check a single link specified by targetIdx static void manifestLinkCheckOne(const Manifest *const this, ManifestLinkCheck *const linkCheck, const unsigned int targetIdx) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM_P(VOID, linkCheck); FUNCTION_LOG_PARAM(UINT, targetIdx); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(linkCheck != NULL); ASSERT(linkCheck->linkList != NULL); ASSERT(targetIdx < manifestTargetTotal(this)); MEM_CONTEXT_TEMP_BEGIN() { const ManifestTarget *const target1 = manifestTarget(this, targetIdx); // Only check link targets if (target1->type == manifestTargetTypeLink) { // Create link destination path for comparison with other paths. It must end in / so subpaths can be detected without // matching valid partial path names at the end of the path. const String *path = NULL; MEM_CONTEXT_BEGIN(lstMemContext(linkCheck->linkList)) { path = strNewFmt("%s/", strZ(manifestTargetPath(this, target1))); // Get base bath if (linkCheck->basePath == NULL) { linkCheck->basePath = strNewFmt( "%s/", strZ(manifestTargetPath(this, manifestTargetFind(this, MANIFEST_TARGET_PGDATA_STR)))); } } MEM_CONTEXT_END(); // Check that link destination is not in base data path if (strBeginsWith(path, linkCheck->basePath)) { THROW_FMT( LinkDestinationError, "link '%s' destination '%s' is in PGDATA", strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1))); } // Check if the link destination path already exists const ManifestLinkCheckItem *const link = lstFind(linkCheck->linkList, &path); if (link != NULL) { // If both links are files make sure they don't link to the same file if (target1->file != NULL && link->file != NULL) { if (strEq(target1->file, link->file)) { const ManifestTarget *const target2 = manifestTarget(this, link->targetIdx); THROW_FMT( LinkDestinationError, "link '%s' (%s/%s) destination is the same file as link '%s' (%s/%s)", strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1)), strZ(target1->file), strZ(manifestPathPg(target2->name)), strZ(manifestTargetPath(this, target2)), strZ(target2->file)); } } // Else error because one of the links is a path and cannot link to the same path as another file/path link else { const ManifestTarget *const target2 = manifestTarget(this, link->targetIdx); THROW_FMT( LinkDestinationError, "link '%s' (%s) destination is the same directory as link '%s' (%s)", strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1)), strZ(manifestPathPg(target2->name)), strZ(manifestTargetPath(this, target2))); } } // Else add to the link list and check against other links else { // Add the link destination path and sort lstAdd(linkCheck->linkList, &(ManifestLinkCheckItem){.path = path, .file = target1->file, .targetIdx = targetIdx}); lstSort(linkCheck->linkList, sortOrderAsc); // Find the path in the sorted list const unsigned int linkIdx = lstFindIdx(linkCheck->linkList, &path); ASSERT(linkIdx != LIST_NOT_FOUND); // Check path links against other links (file links have already been checked) if (target1->file == NULL) { // Check the link destination path to be sure it is not a subpath of a prior link destination path for (unsigned int priorLinkIdx = linkIdx - 1; (int)priorLinkIdx >= 0; priorLinkIdx--) { const ManifestLinkCheckItem *const priorLink = lstGet(linkCheck->linkList, priorLinkIdx); // Skip file links since they are allowed to be in the same path with each other and in the parent path of a // linked destination path. if (priorLink->file != NULL) continue; if (strBeginsWith(path, priorLink->path)) { const ManifestTarget *const target2 = manifestTarget(this, priorLink->targetIdx); THROW_FMT( LinkDestinationError, "link '%s' (%s) destination is a subdirectory of link '%s' (%s)", strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1)), strZ(manifestPathPg(target2->name)), strZ(manifestTargetPath(this, target2))); } // Stop once the first prior path link has been checked since it must be a parent (if there is one) break; } // Check the link destination path to be sure it is not a parent path of a subsequent link destination path for (unsigned int nextLinkIdx = linkIdx + 1; nextLinkIdx < lstSize(linkCheck->linkList); nextLinkIdx++) { const ManifestLinkCheckItem *const nextLink = lstGet(linkCheck->linkList, nextLinkIdx); // Skip file links since they are allowed to be in the same path with each other and in the parent path of a // linked destination path. if (nextLink->file != NULL) continue; if (strBeginsWith(nextLink->path, path)) { const ManifestTarget *const target2 = manifestTarget(this, nextLink->targetIdx); THROW_FMT( LinkDestinationError, "link '%s' (%s) destination is a subdirectory of link '%s' (%s)", strZ(manifestPathPg(target2->name)), strZ(manifestTargetPath(this, target2)), strZ(manifestPathPg(target1->name)), strZ(manifestTargetPath(this, target1))); } // Stop once the first next path link has been checked since it must be a subpath (if there is one) break; } } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } FN_EXTERN void manifestLinkCheck(const Manifest *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Check all links ManifestLinkCheck linkCheck = manifestLinkCheckInit(); for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(this); targetIdx++) manifestLinkCheckOne(this, &linkCheck, targetIdx); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/manifest/manifest.c0000664000175100017510000011275015240547630020723 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Manifest Handler ***********************************************************************************************************************************/ #include #include #include #include #include "common/crypto/cipherBlock.h" #include "common/debug.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/json.h" #include "common/type/list.h" #include "info/manifest/manifest.h" #include "postgres/interface.h" #include "postgres/version.h" #include "storage/storage.h" #include "version.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Manifest { ManifestPub pub; // Publicly accessible variables StringList *ownerList; // List of users/groups const String *fileUserDefault; // Default file user name const String *fileGroupDefault; // Default file group name mode_t fileModeDefault; // Default file mode }; // {uncrustify_off - order required for C includes} #include "store.c.inc" #include "link.c.inc" #include "build.c.inc" #include "serialize.c.inc" // {uncrustify_on} /*********************************************************************************************************************************** Internal constructor ***********************************************************************************************************************************/ static Manifest * manifestNewInternal(void) { FUNCTION_TEST_VOID(); FUNCTION_AUDIT_HELPER(); Manifest *this = OBJ_NEW_ALLOC(); *this = (Manifest) { .pub = { .memContext = memContextCurrent(), .dbList = lstNewP(sizeof(ManifestDb), .comparator = lstComparatorStr), .fileList = lstNewP(sizeof(ManifestFilePack *), .comparator = lstComparatorStr), .linkList = lstNewP(sizeof(ManifestLink), .comparator = lstComparatorStr), .pathList = lstNewP(sizeof(ManifestPath), .comparator = lstComparatorStr), .targetList = lstNewP(sizeof(ManifestTarget), .comparator = lstComparatorStr), .referenceList = strLstNew(), }, .ownerList = strLstNew(), }; FUNCTION_TEST_RETURN(MANIFEST, this); } // Regular expression constants #define RELATION_EXP "[0-9]+(_(fsm|vm)){0,1}(\\.[0-9]+){0,1}" #define DB_PATH_EXP \ "(" MANIFEST_TARGET_PGDATA "/(" PG_PATH_GLOBAL "|" PG_PATH_BASE "/[0-9]+)|" MANIFEST_TARGET_PGTBLSPC "/[0-9]+/%s/[0-9]+)" FN_EXTERN Manifest * manifestNewBuild( const Storage *const storagePg, const unsigned int pgVersion, const unsigned int pgCatalogVersion, const time_t timestampStart, const bool online, const bool checksumPage, const bool bundle, const bool blockIncr, const ManifestBlockIncrMap *blockIncrMap, const StringList *const excludeList, const Pack *const tablespaceList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storagePg); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(UINT, pgCatalogVersion); FUNCTION_LOG_PARAM(TIME, timestampStart); FUNCTION_LOG_PARAM(BOOL, online); FUNCTION_LOG_PARAM(BOOL, checksumPage); FUNCTION_LOG_PARAM(BOOL, bundle); FUNCTION_LOG_PARAM(BOOL, blockIncr); FUNCTION_LOG_PARAM(VOID, blockIncrMap); FUNCTION_LOG_PARAM(STRING_LIST, excludeList); FUNCTION_LOG_PARAM(PACK, tablespaceList); FUNCTION_LOG_END(); ASSERT(storagePg != NULL); ASSERT(pgVersion != 0); Manifest *this; OBJ_NEW_BASE_BEGIN(Manifest, .childQty = MEM_CONTEXT_QTY_MAX) { this = manifestNewInternal(); this->pub.info = infoNew(NULL); this->pub.data.backrestVersion = strNewZ(PROJECT_VERSION); this->pub.data.pgVersion = pgVersion; this->pub.data.pgCatalogVersion = pgCatalogVersion; this->pub.data.backupTimestampStart = timestampStart; this->pub.data.backupType = backupTypeFull; this->pub.data.backupOptionOnline = online; this->pub.data.backupOptionChecksumPage = varNewBool(checksumPage); this->pub.data.bundle = bundle; this->pub.data.bundleRaw = blockIncr; this->pub.data.blockIncr = blockIncr; MEM_CONTEXT_TEMP_BEGIN() { // Data needed to build the manifest ManifestLinkCheck linkCheck = manifestLinkCheckInit(); ManifestBuildData buildData = { .manifest = this, .storagePg = storagePg, .tablespaceId = pgTablespaceId(pgVersion, pgCatalogVersion), .online = online, .checksumPage = checksumPage, .tablespaceList = tablespaceList, .linkCheck = &linkCheck, .manifestWalName = strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(pgWalPath(pgVersion))), .blockIncrMap = blockIncrMap, }; // Build expressions to identify databases paths and temp relations // --------------------------------------------------------------------------------------------------------------------- ASSERT(buildData.tablespaceId != NULL); // Expression to identify database paths buildData.dbPathExp = regExpNew(strNewFmt("^" DB_PATH_EXP "$", strZ(buildData.tablespaceId))); // Expression to find temp relations buildData.tempRelationExp = regExpNew(STRDEF("^t[0-9]+_" RELATION_EXP "$")); // Build list of exclusions // --------------------------------------------------------------------------------------------------------------------- if (excludeList != NULL) { for (unsigned int excludeIdx = 0; excludeIdx < strLstSize(excludeList); excludeIdx++) { const String *const exclude = strNewFmt(MANIFEST_TARGET_PGDATA "/%s", strZ(strLstGet(excludeList, excludeIdx))); // If the exclusions refers to the contents of a path if (strEndsWithZ(exclude, "/")) { if (buildData.excludeContent == NULL) buildData.excludeContent = strLstNew(); strLstAddSub(buildData.excludeContent, exclude, strSize(exclude) - 1); } // Otherwise exclude a single file/link/path else { if (buildData.excludeSingle == NULL) buildData.excludeSingle = strLstNew(); strLstAdd(buildData.excludeSingle, exclude); } } } // Build manifest // --------------------------------------------------------------------------------------------------------------------- const String *const pgPath = storagePathP(storagePg, NULL); const StorageInfo info = storageInfoP(storagePg, pgPath, .followLink = true); const ManifestPath path = { .name = MANIFEST_TARGET_PGDATA_STR, .mode = info.mode, .user = info.user, .group = info.group, }; manifestPathAdd(this, &path); // Generate file defaults from base path MEM_CONTEXT_BEGIN(this->pub.memContext) { this->fileUserDefault = strDup(path.user); this->fileGroupDefault = strDup(path.group); this->fileModeDefault = path.mode & (S_IRUSR | S_IWUSR | S_IRGRP); } MEM_CONTEXT_END(); const ManifestTarget target = { .name = MANIFEST_TARGET_PGDATA_STR, .path = pgPath, .type = manifestTargetTypePath, }; manifestTargetAdd(this, &target); // Gather info for the rest of the files/links/paths StorageIterator *const storageItr = storageNewItrP( storagePg, pgPath, .errorOnMissing = true, .sortOrder = sortOrderAsc); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (storageItrMore(storageItr)) { const StorageInfo info = storageItrNext(storageItr); manifestBuildInfo(&buildData, MANIFEST_TARGET_PGDATA_STR, pgPath, false, &info); // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); // These may not be in order even if the incoming data was sorted lstSort(this->pub.fileList, sortOrderAsc); lstSort(this->pub.linkList, sortOrderAsc); lstSort(this->pub.pathList, sortOrderAsc); lstSort(this->pub.targetList, sortOrderAsc); // Remove unlogged relations from the manifest. This can't be done during the initial build because of the requirement // to check for _init files which will sort after the vast majority of the relation files. We could check storage for // each _init file but that would be expensive. // ------------------------------------------------------------------------------------------------------------------------- RegExp *relationExp = regExpNew(strNewFmt("^" DB_PATH_EXP "/" RELATION_EXP "$", strZ(buildData.tablespaceId))); unsigned int fileIdx = 0; char lastRelationFileId[21] = ""; // Large enough for a 64-bit unsigned integer bool lastRelationFileIdUnlogged = false; #ifdef DEBUG_MEM // Record the temp context size before the loop begins const size_t sizeBegin = memContextSize(memContextCurrent()); #endif while (fileIdx < manifestFileTotal(this)) { // If this file looks like a relation. Note that this never matches on _init forks. const String *const filePathName = manifestFileNameGet(this, fileIdx); if (regExpMatch(relationExp, filePathName)) { // Get the filename (without path) const char *const fileName = strBaseZ(filePathName); const size_t fileNameSize = strlen(fileName); // Strip off the numeric part of the relation char relationFileId[sizeof(lastRelationFileId)]; unsigned int nameIdx = 0; for (; nameIdx < fileNameSize; nameIdx++) { if (!isdigit(fileName[nameIdx])) break; relationFileId[nameIdx] = fileName[nameIdx]; } relationFileId[nameIdx] = '\0'; // The filename must have characters ASSERT(relationFileId[0] != '\0'); // Store the last relation so it does not need to be found every time if (strcmp(lastRelationFileId, relationFileId) != 0) { // Determine if the relation is unlogged String *const relationInit = strNewFmt( "%.*s%s_init", (int)(strSize(filePathName) - fileNameSize), strZ(filePathName), relationFileId); lastRelationFileIdUnlogged = manifestFileExists(this, relationInit); strFree(relationInit); // Save the file id so we don't need to do the lookup next time if it doesn't change memcpy(lastRelationFileId, relationFileId, nameIdx + 1); } // If relation is unlogged then remove it if (lastRelationFileIdUnlogged) { manifestFileRemove(this, filePathName); continue; } } fileIdx++; } #ifdef DEBUG_MEM // Make sure that the temp context did not grow too much during the loop ASSERT(memContextSize(memContextCurrent()) - sizeBegin < 256); #endif } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(MANIFEST, this); } /**********************************************************************************************************************************/ FN_EXTERN void manifestBuildValidate(Manifest *const this, const bool delta, const time_t copyStart, const CompressType compressType) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(BOOL, delta); FUNCTION_LOG_PARAM(TIME, copyStart); FUNCTION_LOG_PARAM(ENUM, compressType); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(copyStart > 0); MEM_CONTEXT_BEGIN(this->pub.memContext) { // Store the delta option. If true we can skip checks that automatically enable delta. this->pub.data.backupOptionDelta = varNewBool(delta); // If online then add one second to the copy start time to allow for database updates during the last second that the // manifest was being built. It's up to the caller to actually wait the remainder of the second, but for comparison purposes // we want the time when the waiting started. this->pub.data.backupTimestampCopyStart = copyStart + (this->pub.data.backupOptionOnline ? 1 : 0); // This value is not needed in this function, but it is needed for resumed manifests and this is last place to set it before // processing begins this->pub.data.backupOptionCompressType = compressType; } MEM_CONTEXT_END(); // Check the manifest for timestamp anomalies that require a delta backup (if delta is not already specified) if (!varBool(this->pub.data.backupOptionDelta)) { MEM_CONTEXT_TEMP_BEGIN() { for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(this); fileIdx++) { const ManifestFile file = manifestFile(this, fileIdx); // Check for timestamp in the future if (file.timestamp > copyStart) { LOG_WARN_FMT( "file '%s' has timestamp (%" PRId64 ") in the future (relative to copy start %" PRId64 "), enabling delta" " checksum", strZ(manifestPathPg(file.name)), (int64_t)file.timestamp, (int64_t)copyStart); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; break; } } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void manifestDeltaCheck(Manifest *const this, const Manifest *const manifestPrior) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(MANIFEST, manifestPrior); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Check for anomalies between manifests if delta is not already enabled if (!varBool(this->pub.data.backupOptionDelta)) { for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(this); fileIdx++) { const ManifestFile file = manifestFile(this, fileIdx); // If file was found in prior manifest then perform checks if (manifestFileExists(manifestPrior, file.name)) { const ManifestFile filePrior = manifestFileFind(manifestPrior, file.name); // Check for timestamp earlier than the prior backup if (file.timestamp < filePrior.timestamp) { LOG_WARN_FMT( "file '%s' has timestamp earlier than prior backup (prior %" PRId64 ", current %" PRId64 "), enabling" " delta checksum", strZ(manifestPathPg(file.name)), (int64_t)filePrior.timestamp, (int64_t)file.timestamp); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; break; } // Check for size change with no timestamp change if (file.sizeOriginal != filePrior.sizeOriginal && file.timestamp == filePrior.timestamp) { LOG_WARN_FMT( "file '%s' has same timestamp (%" PRId64 ") as prior but different size (prior %" PRIu64 ", current" " %" PRIu64 "), enabling delta checksum", strZ(manifestPathPg(file.name)), (int64_t)file.timestamp, filePrior.sizeOriginal, file.sizeOriginal); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; break; } } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void manifestBuildIncr( Manifest *const this, const Manifest *const manifestPrior, const BackupType type, const String *const archiveStart) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(MANIFEST, manifestPrior); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, archiveStart); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(manifestPrior != NULL); ASSERT(type == backupTypeDiff || type == backupTypeIncr); ASSERT(type != backupTypeDiff || manifestPrior->pub.data.backupType == backupTypeFull); ASSERT(archiveStart == NULL || strSize(archiveStart) == 24); MEM_CONTEXT_BEGIN(this->pub.memContext) { // Set prior backup label this->pub.data.backupLabelPrior = strDup(manifestPrior->pub.data.backupLabel); // Copy reference list this->pub.referenceList = strLstDup(manifestPrior->pub.referenceList); // Set diff/incr backup type this->pub.data.backupType = type; // Bundle raw must not change in a backup set this->pub.data.bundleRaw = manifestPrior->pub.data.bundleRaw; } MEM_CONTEXT_END(); MEM_CONTEXT_TEMP_BEGIN() { // Enable delta if timelines differ if (archiveStart != NULL && manifestData(manifestPrior)->archiveStop != NULL && !strEq(strSubN(archiveStart, 0, 8), strSubN(manifestData(manifestPrior)->archiveStop, 0, 8))) { LOG_WARN_FMT( "a timeline switch has occurred since the %s backup, enabling delta checksum\n" "HINT: this is normal after restoring from backup or promoting a standby.", strZ(manifestData(manifestPrior)->backupLabel)); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; } // Else enable delta if online differs else if (manifestData(manifestPrior)->backupOptionOnline != this->pub.data.backupOptionOnline) { LOG_WARN_FMT( "the online option has changed since the %s backup, enabling delta checksum", strZ(manifestData(manifestPrior)->backupLabel)); this->pub.data.backupOptionDelta = BOOL_TRUE_VAR; } // Enable delta if/when there are timestamp anomalies manifestDeltaCheck(this, manifestPrior); // Find files to (possibly) reference in the prior manifest const bool delta = varBool(this->pub.data.backupOptionDelta); for (unsigned int fileIdx = 0; fileIdx < lstSize(this->pub.fileList); fileIdx++) { ManifestFile file = manifestFile(this, fileIdx); // Check if a prior file exists for files that will be copied (i.e. not zero-length files when bundling). If a prior // file does exist it may be possible to reference it instead of copying the file. Use the "find or NULL" variant so we // do one binary search instead of the manifestFileExists()/manifestFileFind() pair, which would do two. const ManifestFilePack *const filePackPrior = file.copy ? manifestFilePackFindDefault(manifestPrior, file.name) : NULL; if (filePackPrior != NULL) { const ManifestFile filePrior = manifestFileUnpack(manifestPrior, filePackPrior); // If file size is equal to prior size then the file can be referenced instead of copied if it has not changed (this // must be determined during the backup). const bool fileSizeEqual = file.size == filePrior.size; // If prior file was stored with block incremental and file size is at least prior file block size then preserve // prior values. If file size is less than prior file block size then the entire file will need to be stored and the // map will be useless as long as the file stays at this size. It is not possible to change the block size in a // backup set and the map info will be required to compare against the prior block incremental. const bool fileBlockIncrPreserve = filePrior.blockIncrMapSize > 0 && file.size >= filePrior.blockIncrSize; // Perform delta if enabled and file size is equal to prior but not zero. Files of unequal length are always // different while zero-length files are always the same, so it wastes time to check them. It is possible for a file // to be truncated down to equal the prior file during backup, but the overhead of checking for such an unlikely // event does not seem worth the possible space saved. file.delta = delta && fileSizeEqual && file.size != 0; // Do not copy if size and prior size are both zero. Zero-length files are always equal so the file can simply be // referenced to the prior file. Note that this is only for the case where zero-length files are being explicitly // written to the repo. Bundled zero-length files disable copy at manifest build time and never reference the prior // file, even if it is also zero-length. if (file.size == 0 && filePrior.size == 0) file.copy = false; // If delta is disabled and size/timestamp are equal then the file is not copied if (!file.delta && fileSizeEqual && file.timestamp == filePrior.timestamp) file.copy = false; ASSERT(file.copy || !file.delta); ASSERT(file.copy || fileSizeEqual); ASSERT(!file.delta || fileSizeEqual); // Preserve values if the file will not be copied, is possibly equal to the prior file, or will be stored with block // incremental and the prior file is also stored with block incremental. If the file will not be copied then the // file size must be equal to the prior file so there is no need to check that condition separately. if (fileSizeEqual || fileBlockIncrPreserve) { file.size = filePrior.size; file.sizeRepo = filePrior.sizeRepo; file.checksumSha1 = filePrior.checksumSha1; file.checksumRepoSha1 = filePrior.checksumRepoSha1; file.reference = filePrior.reference != NULL ? filePrior.reference : manifestPrior->pub.data.backupLabel; file.checksumPage = filePrior.checksumPage; file.checksumPageError = filePrior.checksumPageError; file.checksumPageErrorList = filePrior.checksumPageErrorList; file.bundleId = filePrior.bundleId; file.bundleOffset = filePrior.bundleOffset; file.blockIncrSize = filePrior.blockIncrSize; file.blockIncrChecksumSize = filePrior.blockIncrChecksumSize; file.blockIncrMapSize = filePrior.blockIncrMapSize; ASSERT(file.checksumSha1 != NULL); ASSERT( (!file.checksumPage && !file.checksumPageError && file.checksumPageErrorList == NULL) || (file.checksumPage && !file.checksumPageError && file.checksumPageErrorList == NULL) || (file.checksumPage && file.checksumPageError)); ASSERT(file.size == 0 || file.sizeRepo != 0); ASSERT(file.reference != NULL); ASSERT(file.bundleId != 0 || file.bundleOffset == 0); ASSERT( (file.blockIncrSize == 0 && file.blockIncrChecksumSize == 0 && file.blockIncrMapSize == 0) || (file.blockIncrSize > 0 && file.blockIncrChecksumSize > 0 && file.blockIncrMapSize > 0)); } // Update by index since we already have it from the enclosing loop. Avoids by-name search of manifestFileUpdate(). manifestFileUpdateByIdx(this, fileIdx, &file); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void manifestBuildComplete( Manifest *const this, const String *const lsnStart, const String *const archiveStart, const time_t timestampStop, const String *const lsnStop, const String *const archiveStop, const unsigned int pgId, const uint64_t pgSystemId, const Pack *const dbList, const bool optionArchiveCheck, const bool optionArchiveCopy, const size_t optionBufferSize, const unsigned int optionCompressLevel, const unsigned int optionCompressLevelNetwork, const bool optionHardLink, const unsigned int optionProcessMax, const bool optionStandby, const KeyValue *const annotation) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(STRING, lsnStart); FUNCTION_LOG_PARAM(STRING, archiveStart); FUNCTION_LOG_PARAM(TIME, timestampStop); FUNCTION_LOG_PARAM(STRING, lsnStop); FUNCTION_LOG_PARAM(STRING, archiveStop); FUNCTION_LOG_PARAM(UINT, pgId); FUNCTION_LOG_PARAM(UINT64, pgSystemId); FUNCTION_LOG_PARAM(PACK, dbList); FUNCTION_LOG_PARAM(BOOL, optionArchiveCheck); FUNCTION_LOG_PARAM(BOOL, optionArchiveCopy); FUNCTION_LOG_PARAM(SIZE, optionBufferSize); FUNCTION_LOG_PARAM(UINT, optionCompressLevel); FUNCTION_LOG_PARAM(UINT, optionCompressLevelNetwork); FUNCTION_LOG_PARAM(BOOL, optionHardLink); FUNCTION_LOG_PARAM(UINT, optionProcessMax); FUNCTION_LOG_PARAM(BOOL, optionStandby); FUNCTION_LOG_PARAM(KEY_VALUE, annotation); FUNCTION_LOG_END(); MEM_CONTEXT_BEGIN(this->pub.memContext) { // Save info this->pub.data.lsnStart = strDup(lsnStart); this->pub.data.archiveStart = strDup(archiveStart); this->pub.data.backupTimestampStop = timestampStop; this->pub.data.lsnStop = strDup(lsnStop); this->pub.data.archiveStop = strDup(archiveStop); this->pub.data.pgId = pgId; this->pub.data.pgSystemId = pgSystemId; // Save db list if (dbList != NULL) { PackRead *const read = pckReadNew(dbList); while (!pckReadNullP(read)) { pckReadArrayBeginP(read); const unsigned int id = pckReadU32P(read); const String *const name = pckReadStrP(read); const unsigned int lastSystemId = pckReadU32P(read); pckReadArrayEndP(read); manifestDbAdd(this, &(ManifestDb){.id = id, .name = name, .lastSystemId = lastSystemId}); } lstSort(this->pub.dbList, sortOrderAsc); } // Save annotations if (annotation != NULL) { this->pub.data.annotation = varNewKv(kvNew()); KeyValue *const manifestAnnotationKv = varKv(this->pub.data.annotation); const VariantList *const annotationKeyList = kvKeyList(annotation); for (unsigned int keyIdx = 0; keyIdx < varLstSize(annotationKeyList); keyIdx++) { const Variant *const key = varLstGet(annotationKeyList, keyIdx); const Variant *const value = kvGet(annotation, key); // Skip empty values if (!strEmpty(varStr(value))) kvPut(manifestAnnotationKv, key, value); } // Clean field if there are no annotations to save if (varLstSize(kvKeyList(manifestAnnotationKv)) == 0) this->pub.data.annotation = NULL; } // Save options this->pub.data.backupOptionArchiveCheck = optionArchiveCheck; this->pub.data.backupOptionArchiveCopy = optionArchiveCopy; this->pub.data.backupOptionBufferSize = varNewUInt64(optionBufferSize); this->pub.data.backupOptionCompressLevel = varNewUInt(optionCompressLevel); this->pub.data.backupOptionCompressLevelNetwork = varNewUInt(optionCompressLevelNetwork); this->pub.data.backupOptionHardLink = optionHardLink; this->pub.data.backupOptionProcessMax = varNewUInt(optionProcessMax); this->pub.data.backupOptionStandby = varNewBool(optionStandby); } MEM_CONTEXT_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN Manifest * manifestNewLoad(IoRead *const read) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_END(); ASSERT(read != NULL); Manifest *this; OBJ_NEW_BASE_BEGIN(Manifest, .childQty = MEM_CONTEXT_QTY_MAX) { this = manifestNewInternal(); // Load the manifest ManifestLoadData loadData = { .memContext = memContextNewP("load", .childQty = MEM_CONTEXT_QTY_MAX), .manifest = this, }; // Set file defaults that will be updated when we know what the real defaults are. These need to be set to values that are // not valid for actual names or modes. this->fileUserDefault = STRDEF("@"); this->fileGroupDefault = this->fileUserDefault; this->fileModeDefault = (mode_t)-1; MEM_CONTEXT_BEGIN(loadData.memContext) { loadData.linkFoundList = lstNewP(sizeof(ManifestLoadFound)); loadData.pathFoundList = lstNewP(sizeof(ManifestLoadFound)); } MEM_CONTEXT_END(); this->pub.info = infoNewLoad(read, manifestLoadCallback, &loadData); this->pub.data.backrestVersion = infoBackrestVersion(this->pub.info); // Add the label to the reference list in case the manifest was created before 2.42 when the explicit reference list was // added. Most references are added when the file list is loaded but the current backup will never be referenced from a file // (the reference is assumed) so it must be added here. if (!loadData.referenceListFound) strLstAddIfMissing(this->pub.referenceList, this->pub.data.backupLabel); // Process link defaults for (unsigned int linkIdx = 0; linkIdx < manifestLinkTotal(this); linkIdx++) { ManifestLink *link = lstGet(this->pub.linkList, linkIdx); ManifestLoadFound *found = lstGet(loadData.linkFoundList, linkIdx); if (!found->group) link->group = manifestOwnerCache(this, manifestOwnerGet(loadData.linkGroupDefault)); if (!found->user) link->user = manifestOwnerCache(this, manifestOwnerGet(loadData.linkUserDefault)); } // Process path defaults for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(this); pathIdx++) { ManifestPath *const path = lstGet(this->pub.pathList, pathIdx); const ManifestLoadFound *const found = lstGet(loadData.pathFoundList, pathIdx); if (!found->group) path->group = manifestOwnerCache(this, manifestOwnerGet(loadData.pathGroupDefault)); if (!found->mode) path->mode = loadData.pathModeDefault; if (!found->user) path->user = manifestOwnerCache(this, manifestOwnerGet(loadData.pathUserDefault)); } // Sort the lists. They should already be sorted in the file but it is possible that this system has a different collation // that renders that sort useless. // // This must happen *after* the default processing because found lists are in natural file order and it is not worth writing // comparator routines for them. lstSort(this->pub.dbList, sortOrderAsc); lstSort(this->pub.fileList, sortOrderAsc); lstSort(this->pub.linkList, sortOrderAsc); lstSort(this->pub.pathList, sortOrderAsc); lstSort(this->pub.targetList, sortOrderAsc); // Make sure the base path exists manifestTargetBase(this); // Discard the context holding temporary load data memContextDiscard(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(MANIFEST, this); } /**********************************************************************************************************************************/ FN_EXTERN void manifestValidate(Manifest *const this, const bool strict) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(BOOL, strict); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { String *const error = strNew(); // Validate files for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(this); fileIdx++) { const ManifestFile file = manifestFile(this, fileIdx); // All files must have a checksum if (file.checksumSha1 == NULL) strCatFmt(error, "\nmissing checksum for file '%s'", strZ(file.name)); // These are strict checks to be performed only after a backup and before the final manifest save if (strict) { // Zero-length files must have a specific checksum if (file.size == 0 && !bufEq(HASH_TYPE_SHA1_ZERO_BUF, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE))) { strCatFmt( error, "\ninvalid checksum '%s' for zero size file '%s'", strZ(strNewEncode(encodingHex, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE))), strZ(file.name)); } // Non-zero size files must have non-zero repo size if (file.sizeRepo == 0 && file.size != 0) strCatFmt(error, "\nrepo size must be > 0 for file '%s'", strZ(file.name)); } } // Throw exception when there are errors if (strSize(error) > 0) THROW_FMT(FormatError, "manifest validation failed:%s", strZ(error)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN String * manifestPathPg(const String *const manifestPath) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, manifestPath); FUNCTION_TEST_END(); ASSERT(manifestPath != NULL); // If something in pg_data/ if (strBeginsWith(manifestPath, STRDEF(MANIFEST_TARGET_PGDATA "/"))) { FUNCTION_TEST_RETURN(STRING, strNewZ(strZ(manifestPath) + sizeof(MANIFEST_TARGET_PGDATA))); } // Else not pg_data (this is faster since the length of everything else will be different than pg_data) else if (!strEq(manifestPath, MANIFEST_TARGET_PGDATA_STR)) { // A tablespace target is the only valid option if not pg_data or pg_data/ ASSERT( strEq(manifestPath, MANIFEST_TARGET_PGTBLSPC_STR) || strBeginsWith(manifestPath, STRDEF(MANIFEST_TARGET_PGTBLSPC "/"))); FUNCTION_TEST_RETURN(STRING, strDup(manifestPath)); } FUNCTION_TEST_RETURN(STRING, NULL); } /**********************************************************************************************************************************/ FN_EXTERN void manifestBackupLabelSet(Manifest *const this, const String *const backupLabel) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, backupLabel); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->pub.data.backupLabel == NULL); MEM_CONTEXT_BEGIN(this->pub.memContext) { this->pub.data.backupLabel = strDup(backupLabel); strLstAdd(this->pub.referenceList, backupLabel); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/manifest/manifest.h0000664000175100017510000007165515240547630020740 0ustar00runnerrunner/*********************************************************************************************************************************** Backup Manifest Handler The backup manifest stores a complete list of all files, links, and paths in a backup along with metadata such as checksums, sizes, timestamps, etc. A list of databases is also included for selective restore. The purpose of the manifest is to allow the restore command to confidently reconstruct the PostgreSQL data directory and ensure that nothing is missing or corrupt. It is also useful for reporting, e.g. size of backup, backup time, etc. ***********************************************************************************************************************************/ #ifndef INFO_MANIFEST_MANIFEST_H #define INFO_MANIFEST_MANIFEST_H #include "common/type/string.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define BACKUP_MANIFEST_EXT ".manifest" #define BACKUP_MANIFEST_FILE "backup" BACKUP_MANIFEST_EXT STRING_DECLARE(BACKUP_MANIFEST_FILE_STR); #define MANIFEST_PATH_BUNDLE "bundle" STRING_DECLARE(MANIFEST_PATH_BUNDLE_STR); #define MANIFEST_TARGET_PGDATA "pg_data" STRING_DECLARE(MANIFEST_TARGET_PGDATA_STR); #define MANIFEST_TARGET_PGTBLSPC "pg_tblspc" STRING_DECLARE(MANIFEST_TARGET_PGTBLSPC_STR); // Minimum size for the block incremental checksum #define BLOCK_INCR_CHECKSUM_SIZE_MIN 6 /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Manifest Manifest; #include "command/backup/common.h" #include "common/compress/helper.h" #include "common/crypto/common.h" #include "common/crypto/hash.h" #include "common/type/object.h" #include "common/type/variant.h" #include "info/info.h" #include "info/infoBackup.h" #include "storage/storage.h" /*********************************************************************************************************************************** Manifest data ***********************************************************************************************************************************/ typedef struct ManifestData { const String *backrestVersion; // pgBackRest version const String *backupLabel; // Backup label (unique identifier for the backup) const String *backupLabelPrior; // Backup label for backup this diff/incr is based on time_t backupTimestampCopyStart; // When did the file copy start? time_t backupTimestampStart; // When did the backup start? time_t backupTimestampStop; // When did the backup stop? BackupType backupType; // Type of backup: full, diff, incr bool bundle; // Does the backup bundle files? bool bundleRaw; // Use raw compress/encrypt for bundling? bool blockIncr; // Does the backup perform block incremental? // ??? Note that these fields are redundant and verbose since storing the start/stop lsn as a uint64 would be sufficient. // However, we currently lack the functions to transform these values back and forth so this will do for now. const String *archiveStart; // First WAL file in the backup const String *archiveStop; // Last WAL file in the backup const String *lsnStart; // Start LSN for the backup const String *lsnStop; // Stop LSN for the backup unsigned int pgId; // PostgreSQL id in backup.info unsigned int pgVersion; // PostgreSQL version uint64_t pgSystemId; // PostgreSQL system identifier unsigned int pgCatalogVersion; // PostgreSQL catalog version const Variant *annotation; // Backup annotation(s) metadata bool backupOptionArchiveCheck; // Will WAL segments be checked at the end of the backup? bool backupOptionArchiveCopy; // Will WAL segments be copied to the backup? const Variant *backupOptionStandby; // Will the backup be performed from a standby? const Variant *backupOptionBufferSize; // Buffer size used for file/protocol operations const Variant *backupOptionChecksumPage; // Will page checksums be verified? CompressType backupOptionCompressType; // Compression type used for the backup const Variant *backupOptionCompressLevel; // Level used for compression (if type not none) const Variant *backupOptionCompressLevelNetwork; // Level used for network compression const Variant *backupOptionDelta; // Will a checksum delta be performed? bool backupOptionHardLink; // Will hardlinks be created in the backup? bool backupOptionOnline; // Will an online backup be performed? const Variant *backupOptionProcessMax; // How many processes will be used for backup? } ManifestData; /*********************************************************************************************************************************** Block incremental size maps ***********************************************************************************************************************************/ // Map file size to block size typedef struct ManifestBlockIncrSizeMap { unsigned int fileSize; // File size unsigned int blockSize; // Block size for files >= file size } ManifestBlockIncrSizeMap; // Map file age to block multiplier typedef struct ManifestBlockIncrAgeMap { uint32_t fileAge; // File age in seconds uint32_t blockMultiplier; // Block multiplier } ManifestBlockIncrAgeMap; // Map block size to checksum size typedef struct ManifestBlockIncrChecksumSizeMap { uint32_t blockSize; uint32_t checksumSize; } ManifestBlockIncrChecksumSizeMap; typedef struct ManifestBlockIncrMap { const List *sizeMap; // Block size map const List *ageMap; // File age map const List *checksumSizeMap; // Checksum size map } ManifestBlockIncrMap; /*********************************************************************************************************************************** Db type ***********************************************************************************************************************************/ typedef struct ManifestDb { const String *name; // Db name (must be first member in struct) unsigned int id; // Db oid unsigned int lastSystemId; // Highest oid used by system objects (deprecated - do not use) } ManifestDb; /*********************************************************************************************************************************** File type ***********************************************************************************************************************************/ typedef struct ManifestFile { const String *name; // File name (must be first member in struct) bool copy : 1; // Should the file be copied (backup only)? bool delta : 1; // Verify checksum in PGDATA before copying (backup only)? bool resume : 1; // Is the file being resumed (backup only)? bool checksumPage : 1; // Does this file have page checksums? bool checksumPageError : 1; // Is there an error in the page checksum? mode_t mode; // File mode const uint8_t *checksumSha1; // SHA1 checksum const uint8_t *checksumRepoSha1; // SHA1 checksum as stored in repo (including compression, etc.) const String *checksumPageErrorList; // List of page checksum errors if there are any const String *user; // User name const String *group; // Group name const String *reference; // Reference to a prior backup uint64_t bundleId; // Bundle id uint64_t bundleOffset; // Bundle offset size_t blockIncrSize; // Size of incremental blocks size_t blockIncrChecksumSize; // Size of incremental block checksum uint64_t blockIncrMapSize; // Block incremental map size // After manifest build size is either equal to sizeOriginal or it is copied from the prior file. After the file is backed up // the size will reflect the actual size found during backup. sizeOriginal is used to make sure all required bytes are copied // from a file even if it is referenced to a prior file. For example, an fsm file from a replica may be smaller than the // original on the primary, but during copy we need to be ready to read more bytes if in fact the size of the fsm has increased // on the replica. sizeRepo records the size of the file in the repository (even if it is part of a bundle). uint64_t size; // Final size (after copy) uint64_t sizeOriginal; // Original size (from manifest build) uint64_t sizeRepo; // Size in repo time_t timestamp; // Original timestamp } ManifestFile; /*********************************************************************************************************************************** Link type ***********************************************************************************************************************************/ typedef struct ManifestLink { const String *name; // Link name (must be first member in struct) const String *destination; // Link destination const String *user; // User name const String *group; // Group name } ManifestLink; /*********************************************************************************************************************************** Path type ***********************************************************************************************************************************/ typedef struct ManifestPath { const String *name; // Path name (must be first member in struct) mode_t mode; // Directory mode const String *user; // User name const String *group; // Group name } ManifestPath; /*********************************************************************************************************************************** Target type ***********************************************************************************************************************************/ typedef enum { manifestTargetTypePath, manifestTargetTypeLink, } ManifestTargetType; typedef struct ManifestTarget { const String *name; // Target name (must be first member in struct) ManifestTargetType type; // Target type const String *path; // Target path (if path or link) const String *file; // Target file (if file link) unsigned int tablespaceId; // Oid if this link is a tablespace const String *tablespaceName; // Name of the tablespace } ManifestTarget; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ // Build a new manifest for a PostgreSQL data directory FN_EXTERN Manifest *manifestNewBuild( const Storage *storagePg, unsigned int pgVersion, unsigned int pgCatalogVersion, time_t timestampStart, bool online, bool checksumPage, bool bundle, bool blockIncr, const ManifestBlockIncrMap *blockIncrMap, const StringList *excludeList, const Pack *tablespaceList); // Load a manifest from IO FN_EXTERN Manifest *manifestNewLoad(IoRead *read); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ManifestPub { MemContext *memContext; // Mem context Info *info; // Base info object ManifestData data; // Manifest data and options List *dbList; // List of databases List *fileList; // List of files List *linkList; // List of links List *pathList; // List of paths List *targetList; // List of targets StringList *referenceList; // List of file references } ManifestPub; // Get/set the cipher subpassphrase FN_INLINE_ALWAYS const String * manifestCipherSubPass(const Manifest *const this) { return infoCipherPass(THIS_PUB(Manifest)->info); } FN_INLINE_ALWAYS void manifestCipherSubPassSet(Manifest *const this, const String *const cipherSubPass) { infoCipherPassSet(THIS_PUB(Manifest)->info, cipherSubPass); } // Get manifest configuration and options FN_INLINE_ALWAYS const ManifestData * manifestData(const Manifest *const this) { return &(THIS_PUB(Manifest)->data); } // Get reference list FN_INLINE_ALWAYS const StringList * manifestReferenceList(const Manifest *const this) { return THIS_PUB(Manifest)->referenceList; } // Set backup label FN_EXTERN void manifestBackupLabelSet(Manifest *this, const String *backupLabel); /*********************************************************************************************************************************** Build functions ***********************************************************************************************************************************/ // Validate the timestamps in the manifest given a copy start time, i.e. all times should be <= the copy start time FN_EXTERN void manifestBuildValidate(Manifest *this, bool delta, time_t copyStart, CompressType compressType); // Create a diff/incr backup by comparing to a previous backup manifest FN_EXTERN void manifestBuildIncr(Manifest *this, const Manifest *prior, BackupType type, const String *archiveStart); // Set remaining values before the final save FN_EXTERN void manifestBuildComplete( Manifest *this, const String *lsnStart, const String *archiveStart, time_t timestampStop, const String *lsnStop, const String *archiveStop, unsigned int pgId, uint64_t pgSystemId, const Pack *dbList, bool optionArchiveCheck, bool optionArchiveCopy, size_t optionBufferSize, unsigned int optionCompressLevel, unsigned int optionCompressLevelNetwork, bool optionHardLink, unsigned int optionProcessMax, bool optionStandby, const KeyValue *annotation); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Ensure that symlinks do not point to the same file, directory, or subdirectory of another link FN_EXTERN void manifestLinkCheck(const Manifest *this); // Move to a new parent mem context FN_INLINE_ALWAYS Manifest * manifestMove(Manifest *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Manifest save FN_EXTERN void manifestSave(Manifest *this, IoWrite *write); // Validate a completed manifest. Use strict mode only when saving the manifest after a backup. FN_EXTERN void manifestValidate(Manifest *this, bool strict); /*********************************************************************************************************************************** Db functions and getters/setters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const ManifestDb * manifestDb(const Manifest *const this, const unsigned int dbIdx) { return lstGet(THIS_PUB(Manifest)->dbList, dbIdx); } // If the database requested is not found in the list, return the default passed rather than throw an error FN_INLINE_ALWAYS const ManifestDb * manifestDbFindDefault(const Manifest *const this, const String *const name, const ManifestDb *const dbDefault) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->dbList, &name, (const void *)dbDefault); } FN_INLINE_ALWAYS unsigned int manifestDbTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->dbList); } /*********************************************************************************************************************************** File functions and getters/setters ***********************************************************************************************************************************/ typedef struct ManifestFilePack ManifestFilePack; // Unpack file pack returned by manifestFilePackGet() FN_EXTERN ManifestFile manifestFileUnpack(const Manifest *manifest, const ManifestFilePack *filePack); // Get file in pack format by index FN_INLINE_ALWAYS const ManifestFilePack * manifestFilePackGet(const Manifest *const this, const unsigned int fileIdx) { return *(ManifestFilePack **)lstGet(THIS_PUB(Manifest)->fileList, fileIdx); } // Get file name FN_INLINE_ALWAYS const String * manifestFileNameGet(const Manifest *const this, const unsigned int fileIdx) { return (const String *)manifestFilePackGet(this, fileIdx); } // Get file by index FN_INLINE_ALWAYS ManifestFile manifestFile(const Manifest *const this, const unsigned int fileIdx) { return manifestFileUnpack(this, manifestFilePackGet(this, fileIdx)); } // Add a file FN_EXTERN void manifestFileAdd(Manifest *this, ManifestFile *file); // Find file in pack format by name FN_EXTERN const ManifestFilePack *manifestFilePackFind(const Manifest *this, const String *name); // Find file in pack format by name, returning NULL when missing. Cheaper than calling manifestFileExists() followed by // manifestFilePackFind() because it does only one binary search. FN_INLINE_ALWAYS const ManifestFilePack * manifestFilePackFindDefault(const Manifest *const this, const String *const name) { ASSERT_INLINE(name != NULL); const ManifestFilePack *const *const filePack = lstFindDefault(THIS_PUB(Manifest)->fileList, &name, NULL); return filePack == NULL ? NULL : *filePack; } // Find file by name FN_INLINE_ALWAYS ManifestFile manifestFileFind(const Manifest *const this, const String *const name) { ASSERT_INLINE(name != NULL); return manifestFileUnpack(this, manifestFilePackFind(this, name)); } // Does the file exist? FN_INLINE_ALWAYS bool manifestFileExists(const Manifest *const this, const String *const name) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->fileList, &name, NULL) != NULL; } FN_EXTERN void manifestFileRemove(const Manifest *this, const String *name); FN_INLINE_ALWAYS unsigned int manifestFileTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->fileList); } // Update a file with new data FN_EXTERN void manifestFileUpdate(Manifest *this, const ManifestFile *file); /*********************************************************************************************************************************** Link functions and getters/setters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const ManifestLink * manifestLink(const Manifest *const this, const unsigned int linkIdx) { return lstGet(THIS_PUB(Manifest)->linkList, linkIdx); } FN_EXTERN void manifestLinkAdd(Manifest *this, const ManifestLink *link); FN_EXTERN ManifestLink *manifestLinkFind(const Manifest *this, const String *name); // If the link requested is not found in the list, return the default passed rather than throw an error FN_INLINE_ALWAYS const ManifestLink * manifestLinkFindDefault(const Manifest *const this, const String *const name, const ManifestLink *const linkDefault) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->linkList, &name, (const void *)linkDefault); } FN_EXTERN void manifestLinkRemove(const Manifest *this, const String *name); FN_INLINE_ALWAYS unsigned int manifestLinkTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->linkList); } FN_EXTERN void manifestLinkUpdate(const Manifest *this, const String *name, const String *path); /*********************************************************************************************************************************** Path functions and getters/setters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const ManifestPath * manifestPath(const Manifest *const this, const unsigned int pathIdx) { return lstGet(THIS_PUB(Manifest)->pathList, pathIdx); } FN_EXTERN const ManifestPath *manifestPathFind(const Manifest *this, const String *name); // If the path requested is not found in the list, return the default passed rather than throw an error FN_INLINE_ALWAYS const ManifestPath * manifestPathFindDefault(const Manifest *const this, const String *const name, const ManifestPath *const pathDefault) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->pathList, &name, (const void *)pathDefault); } // Data directory relative path for any manifest file/link/path/target name FN_EXTERN String *manifestPathPg(const String *manifestPath); FN_INLINE_ALWAYS unsigned int manifestPathTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->pathList); } /*********************************************************************************************************************************** Target functions and getters/setters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const ManifestTarget * manifestTarget(const Manifest *const this, const unsigned int targetIdx) { return lstGet(THIS_PUB(Manifest)->targetList, targetIdx); } FN_EXTERN void manifestTargetAdd(Manifest *this, const ManifestTarget *target); FN_EXTERN ManifestTarget *manifestTargetFind(const Manifest *this, const String *name); // If the target requested is not found in the list, return the default passed rather than throw an error FN_INLINE_ALWAYS const ManifestTarget * manifestTargetFindDefault(const Manifest *const this, const String *const name, const ManifestTarget *const targetDefault) { ASSERT_INLINE(name != NULL); return lstFindDefault(THIS_PUB(Manifest)->targetList, &name, (const void *)targetDefault); } // Base target, i.e. the target that is the data directory FN_INLINE_ALWAYS const ManifestTarget * manifestTargetBase(const Manifest *const this) { return manifestTargetFind(this, MANIFEST_TARGET_PGDATA_STR); } // Absolute path to the target FN_EXTERN String *manifestTargetPath(const Manifest *this, const ManifestTarget *target); FN_EXTERN void manifestTargetRemove(const Manifest *this, const String *name); FN_INLINE_ALWAYS unsigned int manifestTargetTotal(const Manifest *const this) { return lstSize(THIS_PUB(Manifest)->targetList); } FN_EXTERN void manifestTargetUpdate(const Manifest *this, const String *name, const String *path, const String *file); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void manifestFree(Manifest *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Load backup manifest FN_EXTERN Manifest *manifestLoadFile( const Storage *storage, const String *fileName, CipherType cipherType, const String *cipherPass); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_MANIFEST_TYPE \ Manifest * #define FUNCTION_LOG_MANIFEST_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "Manifest", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_DB_TYPE \ ManifestDb * #define FUNCTION_LOG_MANIFEST_DB_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestDb", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_FILE_TYPE \ ManifestFile * #define FUNCTION_LOG_MANIFEST_FILE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestFile", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_LINK_TYPE \ ManifestLink * #define FUNCTION_LOG_MANIFEST_LINK_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestLink", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_PATH_TYPE \ ManifestPath * #define FUNCTION_LOG_MANIFEST_PATH_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestPath", buffer, bufferSize) #define FUNCTION_LOG_MANIFEST_TARGET_TYPE \ ManifestTarget * #define FUNCTION_LOG_MANIFEST_TARGET_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "ManifestTarget", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/manifest/serialize.c.inc0000664000175100017510000014751415240547630021662 0ustar00runnerrunner/**********************************************************************************************************************************/ #define MANIFEST_TARGET_TYPE_LINK "link" #define MANIFEST_TARGET_TYPE_PATH "path" #define MANIFEST_SECTION_BACKUP "backup" #define MANIFEST_SECTION_BACKUP_DB "backup:db" #define MANIFEST_SECTION_BACKUP_OPTION "backup:option" #define MANIFEST_SECTION_BACKUP_TARGET "backup:target" #define MANIFEST_SECTION_DB "db" #define MANIFEST_SECTION_METADATA "metadata" #define MANIFEST_SECTION_TARGET_FILE "target:file" #define MANIFEST_SECTION_TARGET_FILE_DEFAULT "target:file:default" #define MANIFEST_SECTION_TARGET_LINK "target:link" #define MANIFEST_SECTION_TARGET_LINK_DEFAULT "target:link:default" #define MANIFEST_SECTION_TARGET_PATH "target:path" #define MANIFEST_SECTION_TARGET_PATH_DEFAULT "target:path:default" #define MANIFEST_KEY_ANNOTATION "annotation" #define MANIFEST_KEY_BACKUP_ARCHIVE_START "backup-archive-start" #define MANIFEST_KEY_BACKUP_ARCHIVE_STOP "backup-archive-stop" #define MANIFEST_KEY_BACKUP_BLOCK_INCR "backup-block-incr" #define MANIFEST_KEY_BACKUP_BUNDLE "backup-bundle" #define MANIFEST_KEY_BACKUP_BUNDLE_RAW "backup-bundle-raw" #define MANIFEST_KEY_BACKUP_LABEL "backup-label" #define MANIFEST_KEY_BACKUP_LSN_START "backup-lsn-start" #define MANIFEST_KEY_BACKUP_LSN_STOP "backup-lsn-stop" #define MANIFEST_KEY_BACKUP_PRIOR "backup-prior" #define MANIFEST_KEY_BACKUP_REFERENCE "backup-reference" #define MANIFEST_KEY_BACKUP_TIMESTAMP_COPY_START "backup-timestamp-copy-start" #define MANIFEST_KEY_BACKUP_TIMESTAMP_START "backup-timestamp-start" #define MANIFEST_KEY_BACKUP_TIMESTAMP_STOP "backup-timestamp-stop" #define MANIFEST_KEY_BACKUP_TYPE "backup-type" #define MANIFEST_KEY_BLOCK_INCR STRID5("bi", 0x1220) #define MANIFEST_KEY_BLOCK_INCR_CHECKSUM STRID5("bic", 0xd220) #define MANIFEST_KEY_BLOCK_INCR_MAP STRID5("bim", 0x35220) #define MANIFEST_KEY_BUNDLE_ID STRID5("bni", 0x25c20) #define MANIFEST_KEY_BUNDLE_OFFSET STRID5("bno", 0x3dc20) #define MANIFEST_KEY_CHECKSUM STRID5("checksum", 0x6d66b195030) #define MANIFEST_KEY_CHECKSUM_REPO STRID5("rck", 0x2c720) #define MANIFEST_KEY_CHECKSUM_PAGE "checksum-page" #define MANIFEST_KEY_CHECKSUM_PAGE_ERROR "checksum-page-error" #define MANIFEST_KEY_DB_CATALOG_VERSION "db-catalog-version" #define MANIFEST_KEY_DB_ID "db-id" #define MANIFEST_KEY_DB_LAST_SYSTEM_ID "db-last-system-id" #define MANIFEST_KEY_DB_SYSTEM_ID "db-system-id" #define MANIFEST_KEY_DB_VERSION "db-version" #define MANIFEST_KEY_DESTINATION STRID5("destination", 0x39e9a05c9a4ca40) #define MANIFEST_KEY_FILE STRID5("file", 0x2b1260) #define MANIFEST_KEY_GROUP "group" #define MANIFEST_KEY_MODE "mode" #define MANIFEST_KEY_PATH STRID5("path", 0x450300) #define MANIFEST_KEY_REFERENCE STRID5("reference", 0x51b8b2298b20) #define MANIFEST_KEY_SIZE STRID5("size", 0x2e9330) #define MANIFEST_KEY_SIZE_ORIGINAL STRID5("szo", 0x3f530) #define MANIFEST_KEY_SIZE_REPO STRID5("repo-size", 0x5d267b7c0b20) #define MANIFEST_KEY_TABLESPACE_ID "tablespace-id" #define MANIFEST_KEY_TABLESPACE_NAME "tablespace-name" #define MANIFEST_KEY_TIMESTAMP STRID5("timestamp", 0x10686932b5340) #define MANIFEST_KEY_TYPE STRID5("type", 0x2c3340) #define MANIFEST_KEY_USER "user" #define MANIFEST_KEY_OPTION_ARCHIVE_CHECK "option-archive-check" #define MANIFEST_KEY_OPTION_ARCHIVE_COPY "option-archive-copy" #define MANIFEST_KEY_OPTION_BACKUP_STANDBY "option-backup-standby" #define MANIFEST_KEY_OPTION_BUFFER_SIZE "option-buffer-size" #define MANIFEST_KEY_OPTION_CHECKSUM_PAGE "option-checksum-page" #define MANIFEST_KEY_OPTION_COMPRESS "option-compress" #define MANIFEST_KEY_OPTION_COMPRESS_TYPE "option-compress-type" #define MANIFEST_KEY_OPTION_COMPRESS_LEVEL "option-compress-level" #define MANIFEST_KEY_OPTION_COMPRESS_LEVEL_NETWORK "option-compress-level-network" #define MANIFEST_KEY_OPTION_DELTA "option-delta" #define MANIFEST_KEY_OPTION_HARDLINK "option-hardlink" #define MANIFEST_KEY_OPTION_ONLINE "option-online" #define MANIFEST_KEY_OPTION_PROCESS_MAX "option-process-max" // Keep track of which values were found during load and which need to be loaded from defaults. There is no point in having // multiple structs since most of the fields are the same and the size shouldn't be more than 4/8 bytes. typedef struct ManifestLoadFound { bool group : 1; bool mode : 1; bool user : 1; } ManifestLoadFound; typedef struct ManifestLoadData { MemContext *memContext; // Mem context for data needed only during load Manifest *manifest; // Manifest info bool referenceListFound; // Was a reference list found? List *linkFoundList; // Values found in links const Variant *linkGroupDefault; // Link default group const Variant *linkUserDefault; // Link default user List *pathFoundList; // Values found in paths const Variant *pathGroupDefault; // Path default group mode_t pathModeDefault; // Path default mode const Variant *pathUserDefault; // Path default user } ManifestLoadData; // Helper to transform a variant that could be boolean or string into a string. If the boolean is false return NULL else return the // string. The boolean cannot be true. static const String * manifestOwnerGet(const Variant *const owner) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, owner); FUNCTION_TEST_END(); ASSERT(owner != NULL); // If bool then it should be false. This indicates that the owner could not be mapped to a name during the backup. if (varType(owner) == varTypeBool) { CHECK(FormatError, !varBool(owner), "owner bool must be false"); FUNCTION_TEST_RETURN_CONST(STRING, NULL); } FUNCTION_TEST_RETURN_CONST(STRING, varStr(owner)); } // Helper to check the variant type of owner and duplicate (call in the containing context) static const Variant * manifestOwnerDefaultGet(const Variant *const ownerDefault) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VARIANT, ownerDefault); FUNCTION_TEST_END(); ASSERT(ownerDefault != NULL); // Bool = false means the owner was not mapped to a name if (varType(ownerDefault) == varTypeBool) { // Value must be false CHECK(FormatError, !varBool(ownerDefault), "owner bool must be false"); FUNCTION_TEST_RETURN_CONST(VARIANT, BOOL_FALSE_VAR); } // Return a duplicate of the owner passed in FUNCTION_TEST_RETURN_CONST(VARIANT, varDup(ownerDefault)); } static void manifestLoadCallback(void *const callbackData, const String *const section, const String *const key, JsonRead *const json) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STRING, section); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_TEST_PARAM(JSON_READ, json); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(callbackData != NULL); ASSERT(section != NULL); ASSERT(key != NULL); ASSERT(json != NULL); ManifestLoadData *const loadData = (ManifestLoadData *)callbackData; Manifest *const manifest = loadData->manifest; // ----------------------------------------------------------------------------------------------------------------------------- if (strEqZ(section, MANIFEST_SECTION_TARGET_FILE)) { ManifestFile file = {.name = key}; jsonReadObjectBegin(json); // Block incremental info if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BLOCK_INCR)) { file.blockIncrSize = (size_t)jsonReadUInt64(json) * BLOCK_INCR_SIZE_FACTOR; if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BLOCK_INCR_CHECKSUM)) file.blockIncrChecksumSize = (size_t)jsonReadUInt64(json); else file.blockIncrChecksumSize = BLOCK_INCR_CHECKSUM_SIZE_MIN; if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BLOCK_INCR_MAP)) file.blockIncrMapSize = jsonReadUInt64(json); } // Bundle info if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BUNDLE_ID)) { file.bundleId = jsonReadUInt64(json); if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_BUNDLE_OFFSET)) file.bundleOffset = jsonReadUInt64(json); } // The checksum might not exist if this is a partial save that was done during the backup to preserve checksums for already // backed up files if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_CHECKSUM)) file.checksumSha1 = bufPtr(bufNewDecode(encodingHex, jsonReadStr(json))); // Page checksum errors if (jsonReadKeyExpectZ(json, MANIFEST_KEY_CHECKSUM_PAGE)) { file.checksumPage = true; file.checksumPageError = !jsonReadBool(json); if (jsonReadKeyExpectZ(json, MANIFEST_KEY_CHECKSUM_PAGE_ERROR)) file.checksumPageErrorList = jsonFromVar(jsonReadVar(json)); } // Group if (jsonReadKeyExpectZ(json, MANIFEST_KEY_GROUP)) file.group = manifestOwnerGet(jsonReadVar(json)); else file.group = manifest->fileGroupDefault; // Mode if (jsonReadKeyExpectZ(json, MANIFEST_KEY_MODE)) file.mode = cvtZToMode(strZ(jsonReadStr(json))); else file.mode = manifest->fileModeDefault; // The repo checksum might not exist if this is a partial save that was done during the backup to preserve checksums for // already backed up files or if this is an older manifest if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_CHECKSUM_REPO)) file.checksumRepoSha1 = bufPtr(bufNewDecode(encodingHex, jsonReadStr(json))); // Reference if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_REFERENCE)) { file.reference = jsonReadStr(json); if (!loadData->referenceListFound) file.reference = strLstAddIfMissing(manifest->pub.referenceList, file.reference); } // If "repo-size" is not present in the manifest file, then it is the same as size (i.e. uncompressed) - to save space, // the repo-size is only stored in the manifest file if it is different than size. const bool sizeRepoExists = jsonReadKeyExpectStrId(json, MANIFEST_KEY_SIZE_REPO); if (sizeRepoExists) file.sizeRepo = jsonReadUInt64(json); // Size is required so error if it is not present. Older versions removed the size before the backup to ensure that the // manifest was updated during the backup, so size can be missing in partial manifests. This error will prevent older // partials from being resumed. if (!jsonReadKeyExpectStrId(json, MANIFEST_KEY_SIZE)) THROW_FMT(FormatError, "missing size for file '%s'", strZ(key)); file.size = jsonReadUInt64(json); // If repo size did not exist then if (!sizeRepoExists) file.sizeRepo = file.size; // If file size is zero then assign the static zero hash if (file.size == 0) file.checksumSha1 = bufPtrConst(HASH_TYPE_SHA1_ZERO_BUF); // If original is not present in the manifest file then it is the same as size (i.e. the file did not change size during // copy) -- to save space the original size is only stored in the manifest file if it is different than size. if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_SIZE_ORIGINAL)) file.sizeOriginal = jsonReadUInt64(json); else file.sizeOriginal = file.size; // Timestamp is required so error if it is not present if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_TIMESTAMP)) file.timestamp = (time_t)jsonReadInt64(json); else THROW_FMT(FormatError, "missing timestamp for file '%s'", strZ(key)); // User if (jsonReadKeyExpectZ(json, MANIFEST_KEY_USER)) file.user = manifestOwnerGet(jsonReadVar(json)); else file.user = manifest->fileUserDefault; manifestFileAdd(manifest, &file); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_PATH)) { ManifestPath path = {.name = key}; ManifestLoadFound valueFound = {0}; jsonReadObjectBegin(json); if (jsonReadKeyExpectZ(json, MANIFEST_KEY_GROUP)) { valueFound.group = true; path.group = manifestOwnerGet(jsonReadVar(json)); } if (jsonReadKeyExpectZ(json, MANIFEST_KEY_MODE)) { valueFound.mode = true; path.mode = cvtZToMode(strZ(jsonReadStr(json))); } if (jsonReadKeyExpectZ(json, MANIFEST_KEY_USER)) { valueFound.user = true; path.user = manifestOwnerGet(jsonReadVar(json)); } lstAdd(loadData->pathFoundList, &valueFound); manifestPathAdd(manifest, &path); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_LINK)) { ManifestLink link = {.name = key}; ManifestLoadFound valueFound = {0}; jsonReadObjectBegin(json); // Link destination link.destination = jsonReadStr(jsonReadKeyRequireStrId(json, MANIFEST_KEY_DESTINATION)); // Group if (jsonReadKeyExpectZ(json, MANIFEST_KEY_GROUP)) { valueFound.group = true; link.group = manifestOwnerGet(jsonReadVar(json)); } // User if (jsonReadKeyExpectZ(json, MANIFEST_KEY_USER)) { valueFound.user = true; link.user = manifestOwnerGet(jsonReadVar(json)); } lstAdd(loadData->linkFoundList, &valueFound); manifestLinkAdd(manifest, &link); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_FILE_DEFAULT)) { if (strEqZ(key, MANIFEST_KEY_MODE)) manifest->fileModeDefault = cvtZToMode(strZ(jsonReadStr(json))); else { const Variant *const valueVar = jsonReadVar(json); MEM_CONTEXT_BEGIN(manifest->pub.memContext) { if (strEqZ(key, MANIFEST_KEY_GROUP)) manifest->fileGroupDefault = strDup(manifestOwnerGet(valueVar)); else if (strEqZ(key, MANIFEST_KEY_USER)) manifest->fileUserDefault = strDup(manifestOwnerGet(valueVar)); } MEM_CONTEXT_END(); } } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_PATH_DEFAULT)) { if (strEqZ(key, MANIFEST_KEY_MODE)) loadData->pathModeDefault = cvtZToMode(strZ(jsonReadStr(json))); else { const Variant *const valueVar = jsonReadVar(json); MEM_CONTEXT_BEGIN(loadData->memContext) { if (strEqZ(key, MANIFEST_KEY_GROUP)) loadData->pathGroupDefault = manifestOwnerDefaultGet(valueVar); else if (strEqZ(key, MANIFEST_KEY_USER)) loadData->pathUserDefault = manifestOwnerDefaultGet(valueVar); } MEM_CONTEXT_END(); } } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_TARGET_LINK_DEFAULT)) { const Variant *const valueVar = jsonReadVar(json); MEM_CONTEXT_BEGIN(loadData->memContext) { if (strEqZ(key, MANIFEST_KEY_GROUP)) loadData->linkGroupDefault = manifestOwnerDefaultGet(valueVar); else if (strEqZ(key, MANIFEST_KEY_USER)) loadData->linkUserDefault = manifestOwnerDefaultGet(valueVar); } MEM_CONTEXT_END(); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_BACKUP_TARGET)) { ManifestTarget target = {.name = key}; jsonReadObjectBegin(json); // File if (jsonReadKeyExpectStrId(json, MANIFEST_KEY_FILE)) target.file = jsonReadStr(json); // Path target.path = jsonReadStr(jsonReadKeyRequireStrId(json, MANIFEST_KEY_PATH)); // Tablespace oid if (jsonReadKeyExpectZ(json, MANIFEST_KEY_TABLESPACE_ID)) { target.tablespaceId = cvtZToUInt(strZ(jsonReadStr(json))); target.tablespaceName = jsonReadStr(jsonReadKeyRequireZ(json, MANIFEST_KEY_TABLESPACE_NAME)); } // Tablespace type const String *const targetType = jsonReadStr(jsonReadKeyRequireStrId(json, MANIFEST_KEY_TYPE)); ASSERT(strEqZ(targetType, MANIFEST_TARGET_TYPE_LINK) || strEqZ(targetType, MANIFEST_TARGET_TYPE_PATH)); target.type = strEqZ(targetType, MANIFEST_TARGET_TYPE_PATH) ? manifestTargetTypePath : manifestTargetTypeLink; manifestTargetAdd(manifest, &target); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_DB)) { ManifestDb db = {.name = key}; jsonReadObjectBegin(json); // Database oid db.id = jsonReadUInt(jsonReadKeyRequireZ(json, MANIFEST_KEY_DB_ID)); // Last system oid db.lastSystemId = jsonReadUInt(jsonReadKeyRequireZ(json, MANIFEST_KEY_DB_LAST_SYSTEM_ID)); manifestDbAdd(manifest, &db); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_METADATA)) { MEM_CONTEXT_BEGIN(manifest->pub.memContext) { if (strEqZ(key, MANIFEST_KEY_ANNOTATION)) manifest->pub.data.annotation = jsonReadVar(json); } MEM_CONTEXT_END(); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_BACKUP)) { if (strEqZ(key, MANIFEST_KEY_BACKUP_BLOCK_INCR)) manifest->pub.data.blockIncr = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_BUNDLE)) manifest->pub.data.bundle = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_BUNDLE_RAW)) manifest->pub.data.bundleRaw = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_TIMESTAMP_COPY_START)) manifest->pub.data.backupTimestampCopyStart = (time_t)jsonReadUInt64(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_TIMESTAMP_START)) manifest->pub.data.backupTimestampStart = (time_t)jsonReadUInt64(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_TIMESTAMP_STOP)) manifest->pub.data.backupTimestampStop = (time_t)jsonReadUInt64(json); else if (strEqZ(key, MANIFEST_KEY_BACKUP_TYPE)) { manifest->pub.data.backupType = (BackupType)jsonReadStrId(json); ASSERT( manifest->pub.data.backupType == backupTypeFull || manifest->pub.data.backupType == backupTypeDiff || manifest->pub.data.backupType == backupTypeIncr); } else { MEM_CONTEXT_BEGIN(manifest->pub.memContext) { String *const valueStr = jsonReadStr(json); if (strEqZ(key, MANIFEST_KEY_BACKUP_ARCHIVE_START)) manifest->pub.data.archiveStart = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_ARCHIVE_STOP)) manifest->pub.data.archiveStop = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_LABEL)) manifest->pub.data.backupLabel = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_LSN_START)) manifest->pub.data.lsnStart = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_LSN_STOP)) manifest->pub.data.lsnStop = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_PRIOR)) manifest->pub.data.backupLabelPrior = valueStr; else if (strEqZ(key, MANIFEST_KEY_BACKUP_REFERENCE)) { manifest->pub.referenceList = strLstNewSplitZ(valueStr, ","); loadData->referenceListFound = true; strFree(valueStr); } } MEM_CONTEXT_END(); } } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_BACKUP_DB)) { if (strEqZ(key, MANIFEST_KEY_DB_ID)) manifest->pub.data.pgId = jsonReadUInt(json); else if (strEqZ(key, MANIFEST_KEY_DB_SYSTEM_ID)) manifest->pub.data.pgSystemId = jsonReadUInt64(json); else if (strEqZ(key, MANIFEST_KEY_DB_CATALOG_VERSION)) manifest->pub.data.pgCatalogVersion = jsonReadUInt(json); else if (strEqZ(key, MANIFEST_KEY_DB_VERSION)) manifest->pub.data.pgVersion = pgVersionFromStr(jsonReadStr(json)); } // ----------------------------------------------------------------------------------------------------------------------------- else if (strEqZ(section, MANIFEST_SECTION_BACKUP_OPTION)) { // Required options if (strEqZ(key, MANIFEST_KEY_OPTION_ARCHIVE_CHECK)) manifest->pub.data.backupOptionArchiveCheck = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_OPTION_ARCHIVE_COPY)) manifest->pub.data.backupOptionArchiveCopy = jsonReadBool(json); // Historically this option meant to add gz compression else if (strEqZ(key, MANIFEST_KEY_OPTION_COMPRESS)) manifest->pub.data.backupOptionCompressType = jsonReadBool(json) ? compressTypeGz : compressTypeNone; // This new option allows any type of compression to be specified. It must be parsed after the option above so the value // does not get overwritten. Since options are stored in alpha order this should always be true. else if (strEqZ(key, MANIFEST_KEY_OPTION_COMPRESS_TYPE)) manifest->pub.data.backupOptionCompressType = compressTypeEnum(strStrId(jsonReadStr(json))); else if (strEqZ(key, MANIFEST_KEY_OPTION_HARDLINK)) manifest->pub.data.backupOptionHardLink = jsonReadBool(json); else if (strEqZ(key, MANIFEST_KEY_OPTION_ONLINE)) manifest->pub.data.backupOptionOnline = jsonReadBool(json); else { // Options that were added after v1.00 and may not be present in every manifest MEM_CONTEXT_BEGIN(manifest->pub.memContext) { if (strEqZ(key, MANIFEST_KEY_OPTION_BUFFER_SIZE)) manifest->pub.data.backupOptionBufferSize = varNewUInt(jsonReadUInt(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_COMPRESS_LEVEL)) manifest->pub.data.backupOptionCompressLevel = varNewUInt(jsonReadUInt(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_COMPRESS_LEVEL_NETWORK)) manifest->pub.data.backupOptionCompressLevelNetwork = varNewUInt(jsonReadUInt(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_PROCESS_MAX)) manifest->pub.data.backupOptionProcessMax = varNewUInt(jsonReadUInt(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_BACKUP_STANDBY)) manifest->pub.data.backupOptionStandby = varNewBool(jsonReadBool(json)); else if (strEqZ(key, MANIFEST_KEY_OPTION_CHECKSUM_PAGE)) manifest->pub.data.backupOptionChecksumPage = jsonReadVar(json); else if (strEqZ(key, MANIFEST_KEY_OPTION_DELTA)) manifest->pub.data.backupOptionDelta = jsonReadVar(json); } MEM_CONTEXT_END(); } } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ typedef struct ManifestSaveData { Manifest *manifest; // Manifest object to be saved const Variant *userDefault; // Default user const Variant *groupDefault; // Default group mode_t fileModeDefault; // File default mode mode_t pathModeDefault; // Path default mode } ManifestSaveData; // Helper to convert the owner MCV to a default. If the input is NULL boolean false should be returned, else the owner string. static const Variant * manifestOwnerVar(const String *const ownerDefault) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, ownerDefault); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(VARIANT, ownerDefault == NULL ? BOOL_FALSE_VAR : varNewStr(ownerDefault)); } static void manifestSaveCallback(void *const callbackData, const String *const sectionNext, InfoSave *const infoSaveData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STRING, sectionNext); FUNCTION_TEST_PARAM(INFO_SAVE, infoSaveData); FUNCTION_TEST_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(callbackData != NULL); ASSERT(infoSaveData != NULL); ManifestSaveData *const saveData = (ManifestSaveData *)callbackData; Manifest *const manifest = saveData->manifest; // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_BACKUP, sectionNext)) { if (manifest->pub.data.archiveStart != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_ARCHIVE_START, jsonFromVar(VARSTR(manifest->pub.data.archiveStart))); } if (manifest->pub.data.archiveStop != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_ARCHIVE_STOP, jsonFromVar(VARSTR(manifest->pub.data.archiveStop))); } if (manifest->pub.data.blockIncr) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_BLOCK_INCR, jsonFromVar(VARBOOL(manifest->pub.data.blockIncr))); } if (manifest->pub.data.bundle) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_BUNDLE, jsonFromVar(VARBOOL(manifest->pub.data.bundle))); } if (manifest->pub.data.bundleRaw) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_BUNDLE_RAW, jsonFromVar(VARBOOL(manifest->pub.data.bundleRaw))); } infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_LABEL, jsonFromVar(VARSTR(manifest->pub.data.backupLabel))); if (manifest->pub.data.lsnStart != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_LSN_START, jsonFromVar(VARSTR(manifest->pub.data.lsnStart))); } if (manifest->pub.data.lsnStop != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_LSN_STOP, jsonFromVar(VARSTR(manifest->pub.data.lsnStop))); } if (manifest->pub.data.backupLabelPrior != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_PRIOR, jsonFromVar(VARSTR(manifest->pub.data.backupLabelPrior))); } infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_REFERENCE, jsonFromVar(VARSTR(strLstJoin(manifest->pub.referenceList, ",")))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_TIMESTAMP_COPY_START, jsonFromVar(VARINT64(manifest->pub.data.backupTimestampCopyStart))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_TIMESTAMP_START, jsonFromVar(VARINT64(manifest->pub.data.backupTimestampStart))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_TIMESTAMP_STOP, jsonFromVar(VARINT64(manifest->pub.data.backupTimestampStop))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP, MANIFEST_KEY_BACKUP_TYPE, jsonFromVar(VARSTR(strNewStrId(manifest->pub.data.backupType)))); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_BACKUP_DB, sectionNext)) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_DB, MANIFEST_KEY_DB_CATALOG_VERSION, jsonFromVar(VARUINT(manifest->pub.data.pgCatalogVersion))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_DB, "db-control-version", jsonFromVar(VARUINT(pgControlVersion(manifest->pub.data.pgVersion)))); infoSaveValue(infoSaveData, MANIFEST_SECTION_BACKUP_DB, MANIFEST_KEY_DB_ID, jsonFromVar(VARUINT(manifest->pub.data.pgId))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_DB, MANIFEST_KEY_DB_SYSTEM_ID, jsonFromVar(VARUINT64(manifest->pub.data.pgSystemId))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_DB, MANIFEST_KEY_DB_VERSION, jsonFromVar(VARSTR(pgVersionToStr(manifest->pub.data.pgVersion)))); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, sectionNext)) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_ARCHIVE_CHECK, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionArchiveCheck))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_ARCHIVE_COPY, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionArchiveCopy))); if (manifest->pub.data.backupOptionStandby != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_BACKUP_STANDBY, jsonFromVar(manifest->pub.data.backupOptionStandby)); } if (manifest->pub.data.backupOptionBufferSize != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_BUFFER_SIZE, jsonFromVar(manifest->pub.data.backupOptionBufferSize)); } if (manifest->pub.data.backupOptionChecksumPage != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_CHECKSUM_PAGE, jsonFromVar(manifest->pub.data.backupOptionChecksumPage)); } // Set the option when compression is turned on. In older versions this also implied gz compression but in newer versions // the type option must also be set if compression is not gz. infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_COMPRESS, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionCompressType != compressTypeNone))); if (manifest->pub.data.backupOptionCompressLevel != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_COMPRESS_LEVEL, jsonFromVar(manifest->pub.data.backupOptionCompressLevel)); } if (manifest->pub.data.backupOptionCompressLevelNetwork != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_COMPRESS_LEVEL_NETWORK, jsonFromVar(manifest->pub.data.backupOptionCompressLevelNetwork)); } // Set the compression type. Older versions will ignore this and assume gz compression if the compress option is set. infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_COMPRESS_TYPE, jsonFromVar(VARSTR(compressTypeStr(manifest->pub.data.backupOptionCompressType)))); if (manifest->pub.data.backupOptionDelta != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_DELTA, jsonFromVar(manifest->pub.data.backupOptionDelta)); } infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_HARDLINK, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionHardLink))); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_ONLINE, jsonFromVar(VARBOOL(manifest->pub.data.backupOptionOnline))); if (manifest->pub.data.backupOptionProcessMax != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_OPTION, MANIFEST_KEY_OPTION_PROCESS_MAX, jsonFromVar(manifest->pub.data.backupOptionProcessMax)); } } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_BACKUP_TARGET, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int targetIdx = 0; targetIdx < manifestTargetTotal(manifest); targetIdx++) { const ManifestTarget *const target = manifestTarget(manifest, targetIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); if (target->file != NULL) jsonWriteStr(jsonWriteKeyStrId(json, MANIFEST_KEY_FILE), target->file); jsonWriteStr(jsonWriteKeyStrId(json, MANIFEST_KEY_PATH), target->path); if (target->tablespaceId != 0) jsonWriteStrFmt(jsonWriteKeyZ(json, MANIFEST_KEY_TABLESPACE_ID), "%u", target->tablespaceId); if (target->tablespaceName != NULL) jsonWriteStr(jsonWriteKeyZ(json, MANIFEST_KEY_TABLESPACE_NAME), target->tablespaceName); jsonWriteZ( jsonWriteKeyStrId(json, MANIFEST_KEY_TYPE), target->type == manifestTargetTypePath ? MANIFEST_TARGET_TYPE_PATH : MANIFEST_TARGET_TYPE_LINK); infoSaveValue( infoSaveData, MANIFEST_SECTION_BACKUP_TARGET, strZ(target->name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_DB, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int dbIdx = 0; dbIdx < manifestDbTotal(manifest); dbIdx++) { const ManifestDb *const db = manifestDb(manifest, dbIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); jsonWriteUInt(jsonWriteKeyZ(json, MANIFEST_KEY_DB_ID), db->id); jsonWriteUInt(jsonWriteKeyZ(json, MANIFEST_KEY_DB_LAST_SYSTEM_ID), db->lastSystemId); infoSaveValue(infoSaveData, MANIFEST_SECTION_DB, strZ(db->name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_METADATA, sectionNext)) { if (manifest->pub.data.annotation != NULL) { infoSaveValue( infoSaveData, MANIFEST_SECTION_METADATA, MANIFEST_KEY_ANNOTATION, jsonFromVar(manifest->pub.data.annotation)); } } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_FILE, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int fileIdx = 0; fileIdx < manifestFileTotal(manifest); fileIdx++) { const ManifestFile file = manifestFile(manifest, fileIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); // Block incremental info if (file.blockIncrSize != 0) { jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BLOCK_INCR), file.blockIncrSize / BLOCK_INCR_SIZE_FACTOR); if (file.blockIncrChecksumSize != BLOCK_INCR_CHECKSUM_SIZE_MIN) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BLOCK_INCR_CHECKSUM), file.blockIncrChecksumSize); if (file.blockIncrMapSize != 0) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BLOCK_INCR_MAP), file.blockIncrMapSize); } // Bundle info if (file.bundleId != 0) { jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BUNDLE_ID), file.bundleId); if (file.bundleOffset != 0) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_BUNDLE_OFFSET), file.bundleOffset); } // Save if the file size is not zero and the checksum exists. The checksum might not exist if this is a partial save // performed during a backup. if (file.size != 0 && file.checksumSha1 != NULL) { jsonWriteStr( jsonWriteKeyStrId(json, MANIFEST_KEY_CHECKSUM), strNewEncode(encodingHex, BUF(file.checksumSha1, HASH_TYPE_SHA1_SIZE))); } if (file.checksumPage) { jsonWriteBool(jsonWriteKeyZ(json, MANIFEST_KEY_CHECKSUM_PAGE), !file.checksumPageError); if (file.checksumPageErrorList != NULL) jsonWriteJson(jsonWriteKeyZ(json, MANIFEST_KEY_CHECKSUM_PAGE_ERROR), file.checksumPageErrorList); } if (!varEq(manifestOwnerVar(file.group), saveData->groupDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_GROUP), manifestOwnerVar(file.group)); if (file.mode != saveData->fileModeDefault) jsonWriteStrFmt(jsonWriteKeyZ(json, MANIFEST_KEY_MODE), "%04o", file.mode); // Save if the repo checksum is not null. The repo checksum for zero-length files may vary depending on compression // and encryption applied. if (file.checksumRepoSha1 != NULL) { jsonWriteStr( jsonWriteKeyStrId(json, MANIFEST_KEY_CHECKSUM_REPO), strNewEncode(encodingHex, BUF(file.checksumRepoSha1, HASH_TYPE_SHA1_SIZE))); } if (file.reference != NULL) jsonWriteStr(jsonWriteKeyStrId(json, MANIFEST_KEY_REFERENCE), file.reference); if (file.sizeRepo != file.size) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_SIZE_REPO), file.sizeRepo); jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_SIZE), file.size); if (file.sizeOriginal != file.size) jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_SIZE_ORIGINAL), file.sizeOriginal); jsonWriteUInt64(jsonWriteKeyStrId(json, MANIFEST_KEY_TIMESTAMP), (uint64_t)file.timestamp); if (!varEq(manifestOwnerVar(file.user), saveData->userDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_USER), manifestOwnerVar(file.user)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_FILE, strZ(file.name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_FILE_DEFAULT, sectionNext)) { infoSaveValue(infoSaveData, MANIFEST_SECTION_TARGET_FILE_DEFAULT, MANIFEST_KEY_GROUP, jsonFromVar(saveData->groupDefault)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_FILE_DEFAULT, MANIFEST_KEY_MODE, jsonFromVar(VARSTR(strNewFmt("%04o", saveData->fileModeDefault)))); infoSaveValue(infoSaveData, MANIFEST_SECTION_TARGET_FILE_DEFAULT, MANIFEST_KEY_USER, jsonFromVar(saveData->userDefault)); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_LINK, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int linkIdx = 0; linkIdx < manifestLinkTotal(manifest); linkIdx++) { const ManifestLink *const link = manifestLink(manifest, linkIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); jsonWriteStr(jsonWriteKeyStrId(json, MANIFEST_KEY_DESTINATION), link->destination); if (!varEq(manifestOwnerVar(link->group), saveData->groupDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_GROUP), manifestOwnerVar(link->group)); if (!varEq(manifestOwnerVar(link->user), saveData->userDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_USER), manifestOwnerVar(link->user)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_LINK, strZ(link->name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_LINK_DEFAULT, sectionNext)) { if (manifestLinkTotal(manifest) > 0) { infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_LINK_DEFAULT, MANIFEST_KEY_GROUP, jsonFromVar(saveData->groupDefault)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_LINK_DEFAULT, MANIFEST_KEY_USER, jsonFromVar(saveData->userDefault)); } } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_PATH, sectionNext)) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int pathIdx = 0; pathIdx < manifestPathTotal(manifest); pathIdx++) { const ManifestPath *const path = manifestPath(manifest, pathIdx); JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); if (!varEq(manifestOwnerVar(path->group), saveData->groupDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_GROUP), manifestOwnerVar(path->group)); if (path->mode != saveData->pathModeDefault) jsonWriteStrFmt(jsonWriteKeyZ(json, MANIFEST_KEY_MODE), "%04o", path->mode); if (!varEq(manifestOwnerVar(path->user), saveData->userDefault)) jsonWriteVar(jsonWriteKeyZ(json, MANIFEST_KEY_USER), manifestOwnerVar(path->user)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_PATH, strZ(path->name), jsonWriteResult(jsonWriteObjectEnd(json))); MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } // ----------------------------------------------------------------------------------------------------------------------------- if (infoSaveSection(infoSaveData, MANIFEST_SECTION_TARGET_PATH_DEFAULT, sectionNext)) { infoSaveValue(infoSaveData, MANIFEST_SECTION_TARGET_PATH_DEFAULT, MANIFEST_KEY_GROUP, jsonFromVar(saveData->groupDefault)); infoSaveValue( infoSaveData, MANIFEST_SECTION_TARGET_PATH_DEFAULT, MANIFEST_KEY_MODE, jsonFromVar(VARSTR(strNewFmt("%04o", saveData->pathModeDefault)))); infoSaveValue(infoSaveData, MANIFEST_SECTION_TARGET_PATH_DEFAULT, MANIFEST_KEY_USER, jsonFromVar(saveData->userDefault)); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ typedef struct ManifestLoadFileData { MemContext *memContext; // Mem context const Storage *storage; // Storage to load from const String *fileName; // Base filename CipherType cipherType; // Cipher type const String *cipherPass; // Cipher passphrase Manifest *manifest; // Loaded manifest object } ManifestLoadFileData; static bool manifestLoadFileCallback(void *const data, const unsigned int try) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, data); FUNCTION_LOG_PARAM(UINT, try); FUNCTION_LOG_END(); ASSERT(data != NULL); ManifestLoadFileData *const loadData = data; bool result = false; if (try < 2) { MEM_CONTEXT_TEMP_BEGIN() { // Construct filename based on try const String *const fileName = try == 0 ? loadData->fileName : strNewFmt("%s" INFO_COPY_EXT, strZ(loadData->fileName)); // Attempt to load the file IoRead *const read = storageReadIo(storageNewReadP(loadData->storage, fileName)); cipherBlockFilterGroupAdd(ioReadFilterGroup(read), loadData->cipherType, cipherModeDecrypt, loadData->cipherPass); MEM_CONTEXT_BEGIN(loadData->memContext) { loadData->manifest = manifestNewLoad(read); result = true; } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN(BOOL, result); } FN_EXTERN Manifest * manifestLoadFile( const Storage *const storage, const String *const fileName, const CipherType cipherType, const String *const cipherPass) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, fileName); FUNCTION_LOG_PARAM(STRING_ID, cipherType); FUNCTION_TEST_PARAM(STRING, cipherPass); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(fileName != NULL); ASSERT((cipherType == cipherTypeNone && cipherPass == NULL) || (cipherType != cipherTypeNone && cipherPass != NULL)); ManifestLoadFileData data = { .memContext = memContextCurrent(), .storage = storage, .fileName = fileName, .cipherType = cipherType, .cipherPass = cipherPass, }; MEM_CONTEXT_TEMP_BEGIN() { const char *const fileNamePath = strZ(storagePathP(storage, fileName)); infoLoad( strNewFmt("unable to load backup manifest file '%s' or '%s" INFO_COPY_EXT "'", fileNamePath, fileNamePath), manifestLoadFileCallback, &data); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(MANIFEST, data.manifest); } FN_EXTERN void manifestSave(Manifest *const this, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MANIFEST, this); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(write != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Files can be added from outside the manifest so make sure they are sorted lstSort(this->pub.fileList, sortOrderAsc); // Set default values based on the base path const ManifestPath *const pathBase = manifestPathFind(this, MANIFEST_TARGET_PGDATA_STR); ManifestSaveData saveData = { .manifest = this, .userDefault = manifestOwnerVar(pathBase->user), .groupDefault = manifestOwnerVar(pathBase->group), .fileModeDefault = pathBase->mode & (S_IRUSR | S_IWUSR | S_IRGRP), .pathModeDefault = pathBase->mode, }; // Save manifest infoSave(this->pub.info, write, manifestSaveCallback, &saveData); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/info/manifest/store.c.inc0000664000175100017510000006450015240547630021020 0ustar00runnerrunner/*********************************************************************************************************************************** Store manifest data in memory ***********************************************************************************************************************************/ /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ STRING_EXTERN(BACKUP_MANIFEST_FILE_STR, BACKUP_MANIFEST_FILE); STRING_EXTERN(MANIFEST_TARGET_PGDATA_STR, MANIFEST_TARGET_PGDATA); STRING_EXTERN(MANIFEST_TARGET_PGTBLSPC_STR, MANIFEST_TARGET_PGTBLSPC); // All block incremental sizes must be divisible by this factor #define BLOCK_INCR_SIZE_FACTOR 8192 /*********************************************************************************************************************************** // Add owner to the owner list if it is not there already and return the pointer. This saves a lot of space. ***********************************************************************************************************************************/ static const String * manifestOwnerCache(Manifest *const this, const String *const owner) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, owner); FUNCTION_TEST_END(); ASSERT(this != NULL); if (owner != NULL) FUNCTION_TEST_RETURN_CONST(STRING, strLstAddIfMissing(this->ownerList, owner)); FUNCTION_TEST_RETURN_CONST(STRING, NULL); } /*********************************************************************************************************************************** Db functions ***********************************************************************************************************************************/ static void manifestDbAdd(Manifest *const this, const ManifestDb *const db) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_DB, db); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(db != NULL); ASSERT(db->name != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.dbList)) { const ManifestDb dbAdd = { .id = db->id, .lastSystemId = db->lastSystemId, .name = strDup(db->name), }; lstAdd(this->pub.dbList, &dbAdd); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** File functions ***********************************************************************************************************************************/ // Base time used as a delta to reduce the size of packed timestamps. This will be set on the first call to manifestFilePack(). static time_t manifestPackBaseTime = -1; // Flags used to reduce the size of packed data. They should be ordered from most to least likely and can be reordered at will. typedef enum { manifestFilePackFlagChecksum, manifestFilePackFlagChecksumRepo, manifestFilePackFlagReference, manifestFilePackFlagBundle, manifestFilePackFlagBlockIncr, manifestFilePackFlagCopy, manifestFilePackFlagDelta, manifestFilePackFlagResume, manifestFilePackFlagChecksumPage, manifestFilePackFlagChecksumPageError, manifestFilePackFlagChecksumPageErrorList, manifestFilePackFlagSizeOriginal, manifestFilePackFlagMode, manifestFilePackFlagUser, manifestFilePackFlagUserNull, manifestFilePackFlagGroup, manifestFilePackFlagGroupNull, } ManifestFilePackFlag; // Pack file into a compact format to save memory static ManifestFilePack * manifestFilePack(const Manifest *const manifest, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(manifest != NULL); ASSERT(file != NULL); uint8_t buffer[512]; size_t bufferPos = 0; // Flags uint64_t flag = 0; if (file->checksumSha1 != NULL) flag |= 1 << manifestFilePackFlagChecksum; if (file->checksumRepoSha1 != NULL) flag |= 1 << manifestFilePackFlagChecksumRepo; if (file->copy) flag |= 1 << manifestFilePackFlagCopy; if (file->delta) flag |= 1 << manifestFilePackFlagDelta; if (file->resume) flag |= 1 << manifestFilePackFlagResume; if (file->checksumPage) flag |= 1 << manifestFilePackFlagChecksumPage; if (file->checksumPageError) flag |= 1 << manifestFilePackFlagChecksumPageError; if (file->checksumPageErrorList != NULL) flag |= 1 << manifestFilePackFlagChecksumPageErrorList; if (file->reference != NULL) flag |= 1 << manifestFilePackFlagReference; if (file->bundleId != 0) flag |= 1 << manifestFilePackFlagBundle; if (file->blockIncrSize != 0) flag |= 1 << manifestFilePackFlagBlockIncr; if (file->sizeOriginal != file->size) flag |= 1 << manifestFilePackFlagSizeOriginal; if (file->mode != manifest->fileModeDefault) flag |= 1 << manifestFilePackFlagMode; if (file->user == NULL) flag |= 1 << manifestFilePackFlagUserNull; else if (!strEq(file->user, manifest->fileUserDefault)) flag |= 1 << manifestFilePackFlagUser; if (file->group == NULL) flag |= 1 << manifestFilePackFlagGroupNull; else if (!strEq(file->group, manifest->fileGroupDefault)) flag |= 1 << manifestFilePackFlagGroup; cvtUInt64ToVarInt128(flag, buffer, &bufferPos, sizeof(buffer)); // Size cvtUInt64ToVarInt128(file->size, buffer, &bufferPos, sizeof(buffer)); // Original size if (flag & (1 << manifestFilePackFlagSizeOriginal)) cvtUInt64ToVarInt128(file->sizeOriginal, buffer, &bufferPos, sizeof(buffer)); // Use the first timestamp that appears as the base for all other timestamps. Ideally we would like a timestamp as close to the // middle as possible but it doesn't seem worth doing the calculation. if (manifestPackBaseTime == -1) manifestPackBaseTime = file->timestamp; // Timestamp cvtUInt64ToVarInt128(cvtInt64ToZigZag(manifestPackBaseTime - file->timestamp), buffer, &bufferPos, sizeof(buffer)); // SHA1 checksum if (file->checksumSha1 != NULL) { memcpy((uint8_t *)buffer + bufferPos, file->checksumSha1, HASH_TYPE_SHA1_SIZE); bufferPos += HASH_TYPE_SHA1_SIZE; } // SHA1 repo checksum if (file->checksumRepoSha1 != NULL) { memcpy((uint8_t *)buffer + bufferPos, file->checksumRepoSha1, HASH_TYPE_SHA1_SIZE); bufferPos += HASH_TYPE_SHA1_SIZE; } // Reference if (file->reference != NULL) { cvtUInt64ToVarInt128( strLstFindIdxP(manifest->pub.referenceList, file->reference, .required = true), buffer, &bufferPos, sizeof(buffer)); } // Mode if (flag & (1 << manifestFilePackFlagMode)) cvtUInt64ToVarInt128(file->mode, buffer, &bufferPos, sizeof(buffer)); // User/group if (flag & (1 << manifestFilePackFlagUser)) cvtUInt64ToVarInt128((uintptr_t)file->user, buffer, &bufferPos, sizeof(buffer)); if (flag & (1 << manifestFilePackFlagGroup)) cvtUInt64ToVarInt128((uintptr_t)file->group, buffer, &bufferPos, sizeof(buffer)); // Repo size cvtUInt64ToVarInt128(file->sizeRepo, buffer, &bufferPos, sizeof(buffer)); // Bundle if (flag & (1 << manifestFilePackFlagBundle)) { cvtUInt64ToVarInt128(file->bundleId, buffer, &bufferPos, sizeof(buffer)); cvtUInt64ToVarInt128(file->bundleOffset, buffer, &bufferPos, sizeof(buffer)); } // Block incremental if (flag & (1 << manifestFilePackFlagBlockIncr)) { ASSERT(file->blockIncrSize % BLOCK_INCR_SIZE_FACTOR == 0); cvtUInt64ToVarInt128(file->blockIncrSize / BLOCK_INCR_SIZE_FACTOR, buffer, &bufferPos, sizeof(buffer)); cvtUInt64ToVarInt128(file->blockIncrChecksumSize, buffer, &bufferPos, sizeof(buffer)); cvtUInt64ToVarInt128(file->blockIncrMapSize, buffer, &bufferPos, sizeof(buffer)); } // Allocate memory for the file pack const size_t nameSize = strSize(file->name) + 1; uint8_t *const result = memNew( sizeof(StringPub) + nameSize + bufferPos + (file->checksumPageErrorList != NULL ? ALIGN_OFFSET(StringPub, nameSize + bufferPos) + sizeof(StringPub) + strSize(file->checksumPageErrorList) + 1 : 0)); // Create string object for the file name *(StringPub *)result = (StringPub){.size = (unsigned int)strSize(file->name), .buffer = (char *)result + sizeof(StringPub)}; size_t resultPos = sizeof(StringPub); memcpy(result + resultPos, (const uint8_t *)strZ(file->name), nameSize); resultPos += nameSize; // Copy pack data memcpy(result + resultPos, buffer, bufferPos); // Create string object for the checksum error list if (file->checksumPageErrorList != NULL) { resultPos += bufferPos + ALIGN_OFFSET(StringPub, nameSize + bufferPos); *(StringPub *)(result + resultPos) = (StringPub) { .size = (unsigned int)strSize(file->checksumPageErrorList), .buffer = (char *)result + resultPos + sizeof(StringPub), }; resultPos += sizeof(StringPub); memcpy(result + resultPos, (const uint8_t *)strZ(file->checksumPageErrorList), strSize(file->checksumPageErrorList) + 1); } FUNCTION_TEST_RETURN_TYPE_P(ManifestFilePack, (ManifestFilePack *)result); } FN_EXTERN ManifestFile manifestFileUnpack(const Manifest *const manifest, const ManifestFilePack *const filePack) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, manifest); FUNCTION_TEST_PARAM_P(VOID, filePack); FUNCTION_TEST_END(); ASSERT(filePack != NULL); ASSERT(manifestPackBaseTime != -1); ManifestFile result = {0}; size_t bufferPos = 0; // Name result.name = (const String *)filePack; bufferPos += sizeof(StringPub) + strSize(result.name) + 1; // Flags const uint64_t flag = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); result.copy = (flag >> manifestFilePackFlagCopy) & 1; result.delta = (flag >> manifestFilePackFlagDelta) & 1; result.resume = (flag >> manifestFilePackFlagResume) & 1; // Size result.size = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); // Original size if (flag & (1 << manifestFilePackFlagSizeOriginal)) result.sizeOriginal = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); else result.sizeOriginal = result.size; // Timestamp result.timestamp = manifestPackBaseTime - (time_t)cvtInt64FromZigZag(cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX)); // Checksum page result.checksumPage = (flag >> manifestFilePackFlagChecksumPage) & 1; // SHA1 checksum if (flag & (1 << manifestFilePackFlagChecksum)) { result.checksumSha1 = (const uint8_t *)filePack + bufferPos; bufferPos += HASH_TYPE_SHA1_SIZE; } // SHA1 repo checksum if (flag & (1 << manifestFilePackFlagChecksumRepo)) { result.checksumRepoSha1 = (const uint8_t *)filePack + bufferPos; bufferPos += HASH_TYPE_SHA1_SIZE; } // Reference if (flag & (1 << manifestFilePackFlagReference)) { result.reference = strLstGet( manifest->pub.referenceList, (unsigned int)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX)); } // Mode if (flag & (1 << manifestFilePackFlagMode)) result.mode = (mode_t)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); else result.mode = manifest->fileModeDefault; // User/group if (flag & (1 << manifestFilePackFlagUser)) result.user = (const String *)(uintptr_t)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); else if (!(flag & (1 << manifestFilePackFlagUserNull))) result.user = manifest->fileUserDefault; if (flag & (1 << manifestFilePackFlagGroup)) result.group = (const String *)(uintptr_t)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); else if (!(flag & (1 << manifestFilePackFlagGroupNull))) result.group = manifest->fileGroupDefault; // Repo size result.sizeRepo = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); // Bundle if (flag & (1 << manifestFilePackFlagBundle)) { result.bundleId = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); result.bundleOffset = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); } // Block incremental if (flag & (1 << manifestFilePackFlagBlockIncr)) { result.blockIncrSize = (size_t)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX) * BLOCK_INCR_SIZE_FACTOR; result.blockIncrChecksumSize = (size_t)cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); result.blockIncrMapSize = cvtUInt64FromVarInt128((const uint8_t *)filePack, &bufferPos, UINT_MAX); } // Checksum page error result.checksumPageError = flag & (1 << manifestFilePackFlagChecksumPageError) ? true : false; if (flag & (1 << manifestFilePackFlagChecksumPageErrorList)) result.checksumPageErrorList = (const String *)((const uint8_t *)filePack + bufferPos + ALIGN_OFFSET(StringPub, bufferPos)); FUNCTION_TEST_RETURN_TYPE(ManifestFile, result); } FN_EXTERN void manifestFileAdd(Manifest *const this, ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(file->name != NULL); file->user = manifestOwnerCache(this, file->user); file->group = manifestOwnerCache(this, file->group); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.fileList)) { const ManifestFilePack *const filePack = manifestFilePack(this, file); lstAdd(this->pub.fileList, &filePack); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } // Update file pack by creating a new one and then freeing the old one static void manifestFilePackUpdate(Manifest *const this, ManifestFilePack **const filePack, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM_P(VOID, filePack); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(filePack != NULL); ASSERT(file != NULL); ASSERT( (!file->checksumPage && !file->checksumPageError && file->checksumPageErrorList == NULL) || (file->checksumPage && !file->checksumPageError && file->checksumPageErrorList == NULL) || (file->checksumPage && file->checksumPageError)); ASSERT(file->size != 0 || (file->bundleId == 0 && file->bundleOffset == 0)); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.fileList)) { ManifestFilePack *const filePackOld = *filePack; *filePack = manifestFilePack(this, file); memFree(filePackOld); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } static ManifestFilePack ** manifestFilePackFindInternal(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ManifestFilePack **const filePack = lstFind(this->pub.fileList, &name); if (filePack == NULL) THROW_FMT(AssertError, "unable to find '%s' in manifest file list", strZ(name)); FUNCTION_TEST_RETURN_TYPE_PP(ManifestFilePack, filePack); } const ManifestFilePack * manifestFilePackFind(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); FUNCTION_TEST_RETURN_TYPE_P(ManifestFilePack, *manifestFilePackFindInternal(this, name)); } FN_EXTERN void manifestFileRemove(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); if (!lstRemove(this->pub.fileList, &name)) THROW_FMT(AssertError, "unable to remove '%s' from manifest file list", strZ(name)); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void manifestFileUpdate(Manifest *const this, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(file != NULL); ManifestFilePack **const filePack = manifestFilePackFindInternal(this, file->name); manifestFilePackUpdate(this, filePack, file); FUNCTION_TEST_RETURN_VOID(); } // Update a file by index. Equivalent to manifestFileUpdate() but skips the by-name binary search since the caller already knows // the file's position in the list. Used by hot loops that iterate the file list (e.g. manifestBuildIncr). static void manifestFileUpdateByIdx(Manifest *const this, const unsigned int fileIdx, const ManifestFile *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(UINT, fileIdx); FUNCTION_TEST_PARAM(MANIFEST_FILE, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(fileIdx < lstSize(this->pub.fileList)); manifestFilePackUpdate(this, lstGet(this->pub.fileList, fileIdx), file); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Link functions ***********************************************************************************************************************************/ FN_EXTERN void manifestLinkAdd(Manifest *const this, const ManifestLink *const link) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_LINK, link); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(link != NULL); ASSERT(link->name != NULL); ASSERT(link->destination != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.linkList)) { const ManifestLink linkAdd = { .destination = strDup(link->destination), .name = strDup(link->name), .group = manifestOwnerCache(this, link->group), .user = manifestOwnerCache(this, link->user), }; lstAdd(this->pub.linkList, &linkAdd); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ManifestLink * manifestLinkFind(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ManifestLink *const result = lstFind(this->pub.linkList, &name); if (result == NULL) THROW_FMT(AssertError, "unable to find '%s' in manifest link list", strZ(name)); FUNCTION_TEST_RETURN(MANIFEST_LINK, result); } FN_EXTERN void manifestLinkRemove(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); if (!lstRemove(this->pub.linkList, &name)) THROW_FMT(AssertError, "unable to remove '%s' from manifest link list", strZ(name)); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void manifestLinkUpdate(const Manifest *const this, const String *const name, const String *const destination) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_PARAM(STRING, destination); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ASSERT(destination != NULL); ManifestLink *const link = (ManifestLink *const)manifestLinkFind(this, name); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.linkList)) { if (!strEq(link->destination, destination)) link->destination = strDup(destination); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Path functions ***********************************************************************************************************************************/ static void manifestPathAdd(Manifest *const this, const ManifestPath *const path) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_PATH, path); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(path != NULL); ASSERT(path->name != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.pathList)) { const ManifestPath pathAdd = { .mode = path->mode, .name = strDup(path->name), .group = manifestOwnerCache(this, path->group), .user = manifestOwnerCache(this, path->user), }; lstAdd(this->pub.pathList, &pathAdd); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN const ManifestPath * manifestPathFind(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); const ManifestPath *const result = lstFind(this->pub.pathList, &name); if (result == NULL) THROW_FMT(AssertError, "unable to find '%s' in manifest path list", strZ(name)); FUNCTION_TEST_RETURN_CONST(MANIFEST_PATH, result); } /*********************************************************************************************************************************** Manifest functions ***********************************************************************************************************************************/ FN_EXTERN void manifestTargetAdd(Manifest *const this, const ManifestTarget *const target) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_TARGET, target); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(target != NULL); ASSERT(target->path != NULL); ASSERT(target->name != NULL); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.targetList)) { const ManifestTarget targetAdd = { .file = strDup(target->file), .name = strDup(target->name), .path = strDup(target->path), .tablespaceId = target->tablespaceId, .tablespaceName = strDup(target->tablespaceName), .type = target->type, }; lstAdd(this->pub.targetList, &targetAdd); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ManifestTarget * manifestTargetFind(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ManifestTarget *const result = lstFind(this->pub.targetList, &name); if (result == NULL) THROW_FMT(AssertError, "unable to find '%s' in manifest target list", strZ(name)); FUNCTION_TEST_RETURN(MANIFEST_TARGET, result); } FN_EXTERN String * manifestTargetPath(const Manifest *const this, const ManifestTarget *const target) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(MANIFEST_TARGET, target); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(target != NULL); // If the target path is already absolute then just return it if (strBeginsWith(target->path, FSLASH_STR)) FUNCTION_TEST_RETURN(STRING, strDup(target->path)); // Construct it from the base pg path and a relative path String *result; MEM_CONTEXT_TEMP_BEGIN() { String *const pgPath = strCat(strNew(), strPath(manifestPathPg(target->name))); if (strSize(pgPath) != 0) strCatZ(pgPath, "/"); strCat(pgPath, target->path); MEM_CONTEXT_PRIOR_BEGIN() { result = strPathAbsolute(pgPath, manifestTargetBase(this)->path); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN void manifestTargetRemove(const Manifest *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); if (!lstRemove(this->pub.targetList, &name)) THROW_FMT(AssertError, "unable to remove '%s' from manifest target list", strZ(name)); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN void manifestTargetUpdate(const Manifest *const this, const String *const name, const String *const path, const String *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(MANIFEST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(STRING, file); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); ASSERT(path != NULL); ManifestTarget *const target = (ManifestTarget *)manifestTargetFind(this, name); ASSERT((target->file == NULL && file == NULL) || (target->file != NULL && file != NULL)); MEM_CONTEXT_BEGIN(lstMemContext(this->pub.targetList)) { if (!strEq(target->path, path)) target->path = strDup(path); if (!strEq(target->file, file)) target->file = strDup(file); } MEM_CONTEXT_END(); FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/main.c0000664000175100017510000002726615240547630015307 0ustar00runnerrunner/*********************************************************************************************************************************** Main ***********************************************************************************************************************************/ #include #include #include #include #include "command/annotate/annotate.h" #include "command/archive/get/get.h" #include "command/archive/push/push.h" #include "command/backup/backup.h" #include "command/check/check.h" #include "command/command.h" #include "command/control/start.h" #include "command/control/stop.h" #include "command/exit.h" #include "command/expire/expire.h" #include "command/help/help.h" #include "command/info/info.h" #include "command/local/local.h" #include "command/lock.h" #include "command/manifest/manifest.h" #include "command/remote/remote.h" #include "command/repo/get.h" #include "command/repo/ls.h" #include "command/repo/put.h" #include "command/repo/rm.h" #include "command/restore/restore.h" #include "command/server/ping.h" #include "command/server/server.h" #include "command/stanza/create.h" #include "command/stanza/delete.h" #include "command/stanza/upgrade.h" #include "command/verify/verify.h" #include "common/debug.h" #include "common/io/fdRead.h" #include "common/io/fdWrite.h" #include "common/stat.h" #include "config/config.h" #include "config/load.h" #include "postgres/interface.h" #include "protocol/helper.h" #include "storage/azure/helper.h" #include "storage/cifs/helper.h" #include "storage/gcs/helper.h" #include "storage/helper.h" #include "storage/s3/helper.h" #include "storage/sftp/helper.h" #include "version.h" /*********************************************************************************************************************************** Include automatically generated help data ***********************************************************************************************************************************/ #include "command/help/help.auto.c.inc" int main(int argListSize, const char *argList[]) { // Set stack trace and mem context error cleanup handlers static const ErrorHandlerFunction errorHandlerList[] = {stackTraceClean, memContextClean}; errorHandlerSet(errorHandlerList, LENGTH_OF(errorHandlerList)); // Set storage helpers static const StorageHelper storageHelperList[] = { STORAGE_AZURE_HELPER, STORAGE_CIFS_HELPER, STORAGE_GCS_HELPER, STORAGE_S3_HELPER, #ifdef HAVE_LIBSSH2 STORAGE_SFTP_HELPER, #endif STORAGE_END_HELPER }; storageHelperInit(storageHelperList); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(INT, argListSize); FUNCTION_LOG_PARAM(CHARPY, argList); FUNCTION_LOG_END(); // Initialize command with the start time cmdInit(); // Initialize statistics collector statInit(); // Initialize exit handler exitInit(); // Process commands volatile int result = 0; volatile bool error = false; TRY_BEGIN() { // Load the configuration // ------------------------------------------------------------------------------------------------------------------------- cfgLoad((unsigned int)argListSize, argList); const ConfigCommandRole commandRole = cfgCommandRole(); // Main/async commands // ------------------------------------------------------------------------------------------------------------------------- if (commandRole == cfgCmdRoleMain || commandRole == cfgCmdRoleAsync) { switch (cfgCommandHelp() ? cfgCmdHelp : cfgCommand()) { // Annotate command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdAnnotate: cmdAnnotate(); break; // Archive get command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdArchiveGet: { if (commandRole == cfgCmdRoleAsync) cmdArchiveGetAsync(); else result = cmdArchiveGet(); break; } // Archive push command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdArchivePush: { if (commandRole == cfgCmdRoleAsync) cmdArchivePushAsync(); else cmdArchivePush(); break; } // Backup command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdBackup: { // Run backup cmdBackup(); if (cfgOptionBool(cfgOptExpireAuto)) { // Switch to expire command cmdEnd(0, NULL); cfgCommandSet(cfgCmdExpire, cfgCmdRoleMain); cfgLoadLogFile(); cmdBegin(); // Null out any backup percent complete value in the backup lock file cmdLockWriteP(); // Run expire cmdExpire(); } break; } // Check command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdCheck: cmdCheck(); break; // Expire command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdExpire: cmdExpire(); break; // Info command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdInfo: cmdInfo(); break; // Manifest command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdManifest: cmdManifest(); break; // Repository get file command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdRepoGet: result = cmdStorageGet(); break; // Repository list paths/files command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdRepoLs: cmdStorageList(); break; // Repository put file command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdRepoPut: cmdStoragePut(); break; // Repository remove paths/files command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdRepoRm: cmdStorageRemove(); break; // Server command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdServer: cmdServer((unsigned int)argListSize, argList); break; // Server ping command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdServerPing: cmdServerPing(); break; // Restore command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdRestore: cmdRestore(); break; // Stanza create command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStanzaCreate: cmdStanzaCreate(); break; // Stanza delete command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStanzaDelete: cmdStanzaDelete(); break; // Stanza upgrade command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStanzaUpgrade: cmdStanzaUpgrade(); break; // Start command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStart: cmdStart(); break; // Stop command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdStop: cmdStop(); break; // Verify command // ----------------------------------------------------------------------------------------------------------------- case cfgCmdVerify: cmdVerify(); break; // Help/version commands // ----------------------------------------------------------------------------------------------------------------- case cfgCmdHelp: case cfgCmdVersion: cmdHelp(BUF(helpData, sizeof(helpData))); break; } } // Local/remote commands // ------------------------------------------------------------------------------------------------------------------------- else { ASSERT(commandRole == cfgCmdRoleLocal || commandRole == cfgCmdRoleRemote); const String *const service = commandRole == cfgCmdRoleLocal ? PROTOCOL_SERVICE_LOCAL_STR : PROTOCOL_SERVICE_REMOTE_STR; const String *const name = strNewFmt("%s-%s", strZ(service), strZ(cfgOptionDisplay(cfgOptProcess))); const TimeMSec timeout = cfgOptionUInt64(cfgOptProtocolTimeout); ProtocolServer *const server = protocolServerNew( name, service, ioFdReadNewOpen(name, STDIN_FILENO, timeout), ioFdWriteNewOpen(name, STDOUT_FILENO, timeout)); if (commandRole == cfgCmdRoleLocal) cmdLocal(server); else cmdRemote(server); } } CATCH_FATAL() { error = true; result = exitSafe(result, true, 0); } TRY_END(); // Free protocol objects protocolFree(); FUNCTION_LOG_RETURN(INT, error ? result : exitSafe(result, false, 0)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/meson.build0000664000175100017510000002242215240547630016346 0ustar00runnerrunner#################################################################################################################################### # Make externed functions/variables static when unity is enabled #################################################################################################################################### arg_unity = [] if get_option('unity') == 'on' configuration.set('FN_EXTERN', 'static', description: 'Function is static for unity build') configuration.set('VR_EXTERN_DECLARE', 'static', description: 'Variable declaration is static for unity build') configuration.set('VR_EXTERN_DEFINE', 'static', description: 'Variable definition is static for unity build') # Add args to builds to prevent non-pgbackrest binaries failing due to unused functions arg_unity += '-Wno-unused-function' else configuration.set( 'FN_EXTERN', 'extern __attribute__((visibility ("internal")))', description: 'Extern function required by other compilation units') configuration.set( 'VR_EXTERN_DECLARE', 'extern __attribute__((visibility ("internal")))', description: 'Extern variable declaration required by other compilation units') configuration.set('VR_EXTERN_DEFINE', '', description: 'Extern variable definition required by other compilation units') endif #################################################################################################################################### # Write configuration #################################################################################################################################### configure_file(output: 'build.auto.h', configuration: configuration) #################################################################################################################################### # Common source used by all targets #################################################################################################################################### src_common = files( 'common/debug.c', 'common/encode.c', 'common/error/error.c', 'common/io/filter/buffer.c', 'common/io/filter/filter.c', 'common/io/filter/group.c', 'common/io/filter/sink.c', 'common/io/bufferRead.c', 'common/io/bufferWrite.c', 'common/io/io.c', 'common/io/read.c', 'common/io/write.c', 'common/log.c', 'common/memContext.c', 'common/stackTrace.c', 'common/time.c', 'common/type/blob.c', 'common/type/buffer.c', 'common/type/convert.c', 'common/type/keyValue.c', 'common/type/list.c', 'common/type/object.c', 'common/type/pack.c', 'common/type/stringId.c', 'common/type/stringList.c', 'common/type/stringStatic.c', 'common/type/stringZ.c', 'common/type/variant.c', 'common/type/variantList.c', 'common/user.c', 'common/wait.c', 'config/common.c', 'storage/posix/read.c', 'storage/posix/storage.c', 'storage/posix/write.c', 'storage/iterator.c', 'storage/list.c', 'storage/read.c', 'storage/storage.c', 'storage/write.c', ) #################################################################################################################################### # Build code target #################################################################################################################################### src_build_code = [ 'build/common/regExp.c', 'build/common/render.c', 'build/common/string.c', 'build/common/xml.c', 'build/common/yaml.c', 'build/config/parse.c', 'build/config/render.c', 'build/error/parse.c', 'build/error/render.c', 'build/help/parse.c', 'build/help/render.c', 'build/postgres/parse.c', 'build/postgres/render.c', 'build/main.c', 'common/compress/common.c', 'common/compress/bz2/common.c', 'common/compress/bz2/compress.c', ] if not dist build_code = executable( 'build-code', src_common, src_build_code, dependencies: [ lib_backtrace, lib_bz2, lib_xml, lib_yaml ], c_args: [ arg_unity, ], ) endif # Build help.auto.c.inc (or use the copy shipped in the distribution tarball) subdir('command/help') # Build interface.auto.c.inc (or use the copy shipped in the distribution tarball) subdir('postgres') #################################################################################################################################### # pgBackRest target #################################################################################################################################### src_pgbackrest = [ 'command/annotate/annotate.c', 'command/archive/common.c', 'command/archive/find.c', 'command/archive/get/file.c', 'command/archive/get/get.c', 'command/archive/get/protocol.c', 'command/archive/push/file.c', 'command/archive/push/protocol.c', 'command/archive/push/push.c', 'command/backup/backup.c', 'command/backup/blockIncr.c', 'command/backup/blockMap.c', 'command/backup/common.c', 'command/backup/pageChecksum.c', 'command/backup/protocol.c', 'command/backup/file.c', 'command/check/check.c', 'command/check/common.c', 'command/check/report.c', 'command/exit.c', 'command/expire/expire.c', 'command/help/help.c', 'command/info/info.c', 'command/command.c', 'command/control/common.c', 'command/control/start.c', 'command/control/stop.c', 'command/local/local.c', 'command/lock.c', 'command/manifest/manifest.c', 'command/repo/common.c', 'command/repo/get.c', 'command/repo/ls.c', 'command/repo/put.c', 'command/repo/rm.c', 'command/restore/blockChecksum.c', 'command/restore/blockDelta.c', 'command/restore/file.c', 'command/restore/protocol.c', 'command/restore/restore.c', 'command/restore/timeline.c', 'command/remote/remote.c', 'command/server/ping.c', 'command/server/server.c', 'command/stanza/common.c', 'command/stanza/create.c', 'command/stanza/delete.c', 'command/stanza/upgrade.c', 'command/verify/file.c', 'command/verify/protocol.c', 'command/verify/verify.c', 'common/compress/common.c', 'common/compress/helper.c', 'common/compress/bz2/common.c', 'common/compress/bz2/compress.c', 'common/compress/bz2/decompress.c', 'common/compress/gz/common.c', 'common/compress/gz/compress.c', 'common/compress/gz/decompress.c', 'common/compress/lz4/common.c', 'common/compress/lz4/compress.c', 'common/compress/lz4/decompress.c', 'common/compress/zst/common.c', 'common/compress/zst/compress.c', 'common/compress/zst/decompress.c', 'common/crypto/cipherBlock.c', 'common/crypto/common.c', 'common/crypto/hash.c', 'common/crypto/xxhash.c', 'common/error/retry.c', 'common/exec.c', 'common/fork.c', 'common/ini.c', 'common/io/client.c', 'common/io/fd.c', 'common/io/fdRead.c', 'common/io/fdWrite.c', 'common/io/filter/size.c', 'common/io/http/client.c', 'common/io/http/common.c', 'common/io/http/header.c', 'common/io/http/query.c', 'common/io/http/request.c', 'common/io/http/response.c', 'common/io/http/session.c', 'common/io/http/url.c', 'common/io/limitRead.c', 'common/io/server.c', 'common/io/session.c', 'common/io/socket/address.c', 'common/io/socket/client.c', 'common/io/socket/common.c', 'common/io/socket/server.c', 'common/io/socket/session.c', 'common/io/tls/client.c', 'common/io/tls/common.c', 'common/io/tls/server.c', 'common/io/tls/session.c', 'common/lock.c', 'common/regExp.c', 'common/stat.c', 'common/type/json.c', 'common/type/string.c', 'common/type/xml.c', 'config/config.c', 'config/exec.c', 'config/load.c', 'config/parse.c', 'config/protocol.c', 'db/db.c', 'db/helper.c', 'db/protocol.c', 'info/info.c', 'info/infoArchive.c', 'info/infoBackup.c', 'info/infoPg.c', 'info/manifest/manifest.c', 'postgres/client.c', 'postgres/interface.c', 'postgres/interface/crc32.c', 'postgres/interface/page.c', 'protocol/client.c', 'protocol/helper.c', 'protocol/parallel.c', 'protocol/parallelJob.c', 'protocol/server.c', 'storage/azure/helper.c', 'storage/azure/read.c', 'storage/azure/storage.c', 'storage/azure/write.c', 'storage/cifs/helper.c', 'storage/cifs/storage.c', 'storage/gcs/helper.c', 'storage/gcs/read.c', 'storage/gcs/storage.c', 'storage/gcs/write.c', 'storage/helper.c', 'storage/remote/read.c', 'storage/remote/protocol.c', 'storage/remote/storage.c', 'storage/remote/write.c', 'storage/s3/helper.c', 'storage/s3/read.c', 'storage/s3/storage.c', 'storage/s3/write.c', 'storage/sftp/helper.c', 'storage/sftp/read.c', 'storage/sftp/storage.c', 'storage/sftp/write.c', 'main.c', ] pgbackrest_exe = executable( 'pgbackrest', src_common, src_pgbackrest, help_auto_c_inc, interface_auto_c_inc, dependencies: [ lib_backtrace, lib_bz2, lib_openssl, lib_lz4, lib_systemd, lib_pq, lib_ssh2, lib_xml, lib_z, lib_zstd, ], install: true, ) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7686205 pgbackrest-2.59.1/src/postgres/0000775000175100017510000000000015240547776016063 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/client.c0000664000175100017510000003613515240547630017502 0ustar00runnerrunner/*********************************************************************************************************************************** Postgres Client ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/fd.h" #include "common/log.h" #include "common/wait.h" #include "postgres/client.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct PgClient { PgClientPub pub; // Publicly accessible variables PGconn *connection; // Pg connection }; /*********************************************************************************************************************************** Close protocol connection ***********************************************************************************************************************************/ static void pgClientFreeResource(THIS_VOID) { THIS(PgClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PG_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); PQfinish(this->connection); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN PgClient * pgClientNew( const String *const host, const unsigned int port, const String *const database, const String *const user, const TimeMSec timeout) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(STRING, database); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_END(); ASSERT(port >= 1 && port <= 65535); ASSERT(database != NULL); OBJ_NEW_BEGIN(PgClient, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (PgClient) { .pub = { .host = strDup(host), .port = port, .database = strDup(database), .user = strDup(user), .timeout = timeout, }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PG_CLIENT, this); } /*********************************************************************************************************************************** Just ignore notices and warnings ***********************************************************************************************************************************/ static void pgClientNoticeProcessor(void *const arg, const char *const message) { (void)arg; (void)message; } /*********************************************************************************************************************************** Encode string to escape ' and \ ***********************************************************************************************************************************/ static String * pgClientEscape(const String *const string) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, string); FUNCTION_TEST_END(); ASSERT(string != NULL); String *const result = strCatZ(strNew(), "'"); // Iterate all characters in the string for (unsigned stringIdx = 0; stringIdx < strSize(string); stringIdx++) { char stringChar = strZ(string)[stringIdx]; // These characters are escaped if (stringChar == '\'' || stringChar == '\\') strCatChr(result, '\\'); strCatChr(result, stringChar); } strCatChr(result, '\''); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN PgClient * pgClientOpen(PgClient *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PG_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); CHECK(AssertError, this->connection == NULL, "invalid connection"); MEM_CONTEXT_TEMP_BEGIN() { // Base connection string String *const connInfo = strCatFmt( strNew(), "dbname=%s port=%u", strZ(pgClientEscape(pgClientDatabase(this))), pgClientPort(this)); // Add user if specified if (pgClientUser(this) != NULL) strCatFmt(connInfo, " user=%s", strZ(pgClientEscape(pgClientUser(this)))); // Add host if specified if (pgClientHost(this) != NULL) strCatFmt(connInfo, " host=%s", strZ(pgClientEscape(pgClientHost(this)))); // Make the connection this->connection = PQconnectdb(strZ(connInfo)); // Set a callback to shutdown the connection memContextCallbackSet(objMemContext(this), pgClientFreeResource, this); // Handle errors if (PQstatus(this->connection) != CONNECTION_OK) { THROW_FMT( DbConnectError, "unable to connect to '%s': %s", strZ(connInfo), strZ(strTrim(strNewZ(PQerrorMessage(this->connection))))); } // Set notice and warning processor PQsetNoticeProcessor(this->connection, pgClientNoticeProcessor, NULL); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PG_CLIENT, this); } /**********************************************************************************************************************************/ FN_EXTERN Pack * pgClientQuery(PgClient *const this, const String *const query, const PgClientQueryResult resultType) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PG_CLIENT, this); FUNCTION_LOG_PARAM(STRING, query); FUNCTION_LOG_PARAM(STRING_ID, resultType); FUNCTION_LOG_END(); ASSERT(this != NULL); CHECK(AssertError, this->connection != NULL, "invalid connection"); ASSERT(query != NULL); ASSERT(resultType != 0); Pack *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Send the query without waiting for results so we can timeout if needed if (!PQsendQuery(this->connection, strZ(query))) { THROW_FMT( DbQueryError, "unable to send query '%s': %s", strZ(query), strZ(strTrim(strNewZ(PQerrorMessage(this->connection))))); } // Wait for a result Wait *const wait = waitNew(pgClientTimeout(this)); bool busy = false; do { PQconsumeInput(this->connection); busy = PQisBusy(this->connection); } while (busy && fdReadyRead(PQsocket(this->connection), waitRemains(wait))); // If the query is still busy after the timeout attempt to cancel if (busy) { PGcancel *const cancel = PQgetCancel(this->connection); // If cancel is NULL then more than likely the server process crashed or disconnected if (cancel == NULL) THROW_FMT(DbQueryError, "unable to cancel query '%s': connection was lost", strZ(query)); TRY_BEGIN() { char error[256]; if (!PQcancel(cancel, error, sizeof(error))) THROW_FMT(DbQueryError, "unable to cancel query '%s': %s", strZ(query), strZ(strTrim(strNewZ(error)))); } FINALLY() { PQfreeCancel(cancel); } TRY_END(); } // Get the result (even if query was cancelled -- to prevent the connection being left in a bad state) PGresult *const pgResult = PQgetResult(this->connection); TRY_BEGIN() { // Throw timeout error if cancelled if (busy) THROW_FMT(DbQueryError, "query '%s' timed out after %" PRIu64 "ms", strZ(query), pgClientTimeout(this)); // If this was a command that returned no results then we are done const ExecStatusType resultStatus = PQresultStatus(pgResult); if (resultStatus == PGRES_COMMAND_OK) { if (resultType != pgClientQueryResultNone && resultType != pgClientQueryResultAny) THROW_FMT(DbQueryError, "result expected from '%s'", strZ(query)); } else { // If no tuples then the result is an error if (resultStatus != PGRES_TUPLES_OK) { THROW_FMT( DbQueryError, "unable to execute query '%s': %s", strZ(query), strZ(strTrim(strNewZ(PQresultErrorMessage(pgResult))))); } // Expect some rows to be returned if (resultType == pgClientQueryResultNone) THROW_FMT(DbQueryError, "no result expected from '%s'", strZ(query)); // Fetch row and column values PackWrite *const pack = pckWriteNewP(); const int rowTotal = PQntuples(pgResult); const int columnTotal = PQnfields(pgResult); if (resultType != pgClientQueryResultAny) { if (rowTotal != 1) THROW_FMT(DbQueryError, "expected one row from '%s'", strZ(query)); if (resultType == pgClientQueryResultColumn && columnTotal != 1) THROW_FMT(DbQueryError, "expected one column from '%s'", strZ(query)); } // Get column types Oid *const columnType = memNew(sizeof(int) * (size_t)columnTotal); for (int columnIdx = 0; columnIdx < columnTotal; columnIdx++) columnType[columnIdx] = PQftype(pgResult, columnIdx); // Get values for (int rowIdx = 0; rowIdx < rowTotal; rowIdx++) { if (resultType == pgClientQueryResultAny) pckWriteArrayBeginP(pack); for (int columnIdx = 0; columnIdx < columnTotal; columnIdx++) { const char *const value = PQgetvalue(pgResult, rowIdx, columnIdx); // If value is zero-length then check if it is null if (value[0] == '\0' && PQgetisnull(pgResult, rowIdx, columnIdx)) { pckWriteNullP(pack); } // Else convert the value to a variant else { // Convert column type. Not all PostgreSQL types are supported but these should suffice. switch (columnType[columnIdx]) { // Boolean type case 16: // bool pckWriteBoolP(pack, varBoolForce(varNewStrZ(value)), .defaultWrite = true); break; // Text/char types case 18: // char case 19: // name case 25: // text pckWriteStrP(pack, STR(value), .defaultWrite = true); break; // 64-bit integer type case 20: // int8 pckWriteI64P(pack, cvtZToInt64(value), .defaultWrite = true); break; // 32-bit integer type case 23: // int4 pckWriteI32P(pack, cvtZToInt(value), .defaultWrite = true); break; // 32-bit unsigned integer type case 26: // oid pckWriteU32P(pack, cvtZToUInt(value), .defaultWrite = true); break; default: THROW_FMT( FormatError, "unable to parse type %u in column %d for query '%s'", columnType[columnIdx], columnIdx, strZ(query)); } } } if (resultType == pgClientQueryResultAny) pckWriteArrayEndP(pack); } // End pack and return result pckWriteEndP(pack); result = pckMove(pckWriteResult(pack), memContextPrior()); } } FINALLY() { // Free the result PQclear(pgResult); // The command is only complete once a NULL result is returned. A non-NULL result here means an extra result was // received, which almost always means the connection was lost while the query was running. PGresult *const pgResultExtra = PQgetResult(this->connection); if (pgResultExtra != NULL) { // Attempt to retrieve the error so the actual cause is reported const String *const error = strTrim(strNewZ(PQresultErrorMessage(pgResultExtra))); PQclear(pgResultExtra); THROW_FMT( DbQueryError, "unable to complete query '%s'%s%s\n" "HINT: this is usually caused by the connection to PostgreSQL being lost while the query was running.\n" "HINT: check idle_session_timeout and idle_in_transaction_session_timeout for aggressive settings.\n" "HINT: check the PostgreSQL log and network for a crash, restart, or timeout.", strZ(query), strEmpty(error) ? "" : ": ", strZ(error)); } } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK, result); } /**********************************************************************************************************************************/ FN_EXTERN void pgClientToLog(const PgClient *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{host: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT(pgClientHost(this), strToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", database: "); strToLog(pgClientDatabase(this), debugLog); strStcCat(debugLog, ", user: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT(pgClientUser(this), strToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt( debugLog, ", port: %u, queryTimeout %" PRIu64 "}", pgClientPort(this), pgClientTimeout(this)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/client.h0000664000175100017510000001241215240547630017477 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Client Connect to a PostgreSQL database and run queries. This is not intended to be a general purpose client but is suitable for pgBackRest's limited needs. In particular, data type support is limited to text, int, and bool types so it may be necessary to add casts to queries to output one of these types. ***********************************************************************************************************************************/ #ifndef POSTGRES_QUERY_H #define POSTGRES_QUERY_H #include "common/time.h" #include "common/type/object.h" #include "common/type/pack.h" #include "common/type/string.h" /*********************************************************************************************************************************** Query result types ***********************************************************************************************************************************/ typedef enum { pgClientQueryResultAny = STRID5("any", 0x65c10), // Any number of rows/columns expected (even none) pgClientQueryResultRow = STRID5("row", 0x5df20), // One row expected pgClientQueryResultColumn = STRID5("column", 0x1cdab1e30), // One row/column expected pgClientQueryResultNone = STRID5("none", 0x2b9ee0), // No rows expected } PgClientQueryResult; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct PgClient PgClient; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN PgClient *pgClientNew( const String *host, const unsigned int port, const String *database, const String *user, const TimeMSec timeout); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct PgClientPub { const String *host; // Pg host unsigned int port; // Pg port const String *database; // Pg database const String *user; // Pg user TimeMSec timeout; // Timeout for statements/queries } PgClientPub; // Pg host FN_INLINE_ALWAYS const String * pgClientHost(const PgClient *const this) { return THIS_PUB(PgClient)->host; } // Pg port FN_INLINE_ALWAYS unsigned int pgClientPort(const PgClient *const this) { return THIS_PUB(PgClient)->port; } // Pg database FN_INLINE_ALWAYS const String * pgClientDatabase(const PgClient *const this) { return THIS_PUB(PgClient)->database; } // Pg user FN_INLINE_ALWAYS const String * pgClientUser(const PgClient *const this) { return THIS_PUB(PgClient)->user; } // Timeout for statements/queries FN_INLINE_ALWAYS TimeMSec pgClientTimeout(const PgClient *const this) { return THIS_PUB(PgClient)->timeout; } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Open connection to PostgreSQL FN_EXTERN PgClient *pgClientOpen(PgClient *this); // Move to a new parent mem context FN_INLINE_ALWAYS PgClient * pgClientMove(PgClient *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Execute a query and return results FN_EXTERN Pack *pgClientQuery(PgClient *this, const String *query, PgClientQueryResult resultType); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void pgClientFree(PgClient *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void pgClientToLog(const PgClient *this, StringStatic *debugLog); #define FUNCTION_LOG_PG_CLIENT_TYPE \ PgClient * #define FUNCTION_LOG_PG_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, pgClientToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/interface/0000775000175100017510000000000015240547630020010 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/interface/crc32.c0000664000175100017510000000756315240547630021103 0ustar00runnerrunner/*********************************************************************************************************************************** CRC-32 Calculation ***********************************************************************************************************************************/ #include #include "postgres/interface/crc32.h" /**********************************************************************************************************************************/ static const uint32_t crc32c_lookup[256] = { 0x00000000, 0xF26B8303, 0xE13B70F7, 0x1350F3F4, 0xC79A971F, 0x35F1141C, 0x26A1E7E8, 0xD4CA64EB, 0x8AD958CF, 0x78B2DBCC, 0x6BE22838, 0x9989AB3B, 0x4D43CFD0, 0xBF284CD3, 0xAC78BF27, 0x5E133C24, 0x105EC76F, 0xE235446C, 0xF165B798, 0x030E349B, 0xD7C45070, 0x25AFD373, 0x36FF2087, 0xC494A384, 0x9A879FA0, 0x68EC1CA3, 0x7BBCEF57, 0x89D76C54, 0x5D1D08BF, 0xAF768BBC, 0xBC267848, 0x4E4DFB4B, 0x20BD8EDE, 0xD2D60DDD, 0xC186FE29, 0x33ED7D2A, 0xE72719C1, 0x154C9AC2, 0x061C6936, 0xF477EA35, 0xAA64D611, 0x580F5512, 0x4B5FA6E6, 0xB93425E5, 0x6DFE410E, 0x9F95C20D, 0x8CC531F9, 0x7EAEB2FA, 0x30E349B1, 0xC288CAB2, 0xD1D83946, 0x23B3BA45, 0xF779DEAE, 0x05125DAD, 0x1642AE59, 0xE4292D5A, 0xBA3A117E, 0x4851927D, 0x5B016189, 0xA96AE28A, 0x7DA08661, 0x8FCB0562, 0x9C9BF696, 0x6EF07595, 0x417B1DBC, 0xB3109EBF, 0xA0406D4B, 0x522BEE48, 0x86E18AA3, 0x748A09A0, 0x67DAFA54, 0x95B17957, 0xCBA24573, 0x39C9C670, 0x2A993584, 0xD8F2B687, 0x0C38D26C, 0xFE53516F, 0xED03A29B, 0x1F682198, 0x5125DAD3, 0xA34E59D0, 0xB01EAA24, 0x42752927, 0x96BF4DCC, 0x64D4CECF, 0x77843D3B, 0x85EFBE38, 0xDBFC821C, 0x2997011F, 0x3AC7F2EB, 0xC8AC71E8, 0x1C661503, 0xEE0D9600, 0xFD5D65F4, 0x0F36E6F7, 0x61C69362, 0x93AD1061, 0x80FDE395, 0x72966096, 0xA65C047D, 0x5437877E, 0x4767748A, 0xB50CF789, 0xEB1FCBAD, 0x197448AE, 0x0A24BB5A, 0xF84F3859, 0x2C855CB2, 0xDEEEDFB1, 0xCDBE2C45, 0x3FD5AF46, 0x7198540D, 0x83F3D70E, 0x90A324FA, 0x62C8A7F9, 0xB602C312, 0x44694011, 0x5739B3E5, 0xA55230E6, 0xFB410CC2, 0x092A8FC1, 0x1A7A7C35, 0xE811FF36, 0x3CDB9BDD, 0xCEB018DE, 0xDDE0EB2A, 0x2F8B6829, 0x82F63B78, 0x709DB87B, 0x63CD4B8F, 0x91A6C88C, 0x456CAC67, 0xB7072F64, 0xA457DC90, 0x563C5F93, 0x082F63B7, 0xFA44E0B4, 0xE9141340, 0x1B7F9043, 0xCFB5F4A8, 0x3DDE77AB, 0x2E8E845F, 0xDCE5075C, 0x92A8FC17, 0x60C37F14, 0x73938CE0, 0x81F80FE3, 0x55326B08, 0xA759E80B, 0xB4091BFF, 0x466298FC, 0x1871A4D8, 0xEA1A27DB, 0xF94AD42F, 0x0B21572C, 0xDFEB33C7, 0x2D80B0C4, 0x3ED04330, 0xCCBBC033, 0xA24BB5A6, 0x502036A5, 0x4370C551, 0xB11B4652, 0x65D122B9, 0x97BAA1BA, 0x84EA524E, 0x7681D14D, 0x2892ED69, 0xDAF96E6A, 0xC9A99D9E, 0x3BC21E9D, 0xEF087A76, 0x1D63F975, 0x0E330A81, 0xFC588982, 0xB21572C9, 0x407EF1CA, 0x532E023E, 0xA145813D, 0x758FE5D6, 0x87E466D5, 0x94B49521, 0x66DF1622, 0x38CC2A06, 0xCAA7A905, 0xD9F75AF1, 0x2B9CD9F2, 0xFF56BD19, 0x0D3D3E1A, 0x1E6DCDEE, 0xEC064EED, 0xC38D26C4, 0x31E6A5C7, 0x22B65633, 0xD0DDD530, 0x0417B1DB, 0xF67C32D8, 0xE52CC12C, 0x1747422F, 0x49547E0B, 0xBB3FFD08, 0xA86F0EFC, 0x5A048DFF, 0x8ECEE914, 0x7CA56A17, 0x6FF599E3, 0x9D9E1AE0, 0xD3D3E1AB, 0x21B862A8, 0x32E8915C, 0xC083125F, 0x144976B4, 0xE622F5B7, 0xF5720643, 0x07198540, 0x590AB964, 0xAB613A67, 0xB831C993, 0x4A5A4A90, 0x9E902E7B, 0x6CFBAD78, 0x7FAB5E8C, 0x8DC0DD8F, 0xE330A81A, 0x115B2B19, 0x020BD8ED, 0xF0605BEE, 0x24AA3F05, 0xD6C1BC06, 0xC5914FF2, 0x37FACCF1, 0x69E9F0D5, 0x9B8273D6, 0x88D28022, 0x7AB90321, 0xAE7367CA, 0x5C18E4C9, 0x4F48173D, 0xBD23943E, 0xF36E6F75, 0x0105EC76, 0x12551F82, 0xE03E9C81, 0x34F4F86A, 0xC69F7B69, 0xD5CF889D, 0x27A40B9E, 0x79B737BA, 0x8BDCB4B9, 0x988C474D, 0x6AE7C44E, 0xBE2DA0A5, 0x4C4623A6, 0x5F16D052, 0xAD7D5351, }; FN_EXTERN uint32_t crc32cOne(const uint8_t *data, size_t size) { uint32_t result = 0xffffffff; while (size--) result = crc32c_lookup[(result ^ *data++) & 0xFF] ^ (result >> 8); return result ^ 0xffffffff; } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/interface/crc32.h0000664000175100017510000000153315240547630021077 0ustar00runnerrunner/*********************************************************************************************************************************** CRC-32 Calculation CRC-32 and CRC-32C calculations required to validate the integrity of pg_control. ***********************************************************************************************************************************/ #ifndef POSTGRES_INTERFACE_CRC32_H #define POSTGRES_INTERFACE_CRC32_H #include #include /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Generate CRC-32C checksum FN_EXTERN uint32_t crc32cOne(const uint8_t *data, size_t size); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/interface/page.c0000664000175100017510000001633715240547630021102 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Page Checksum Adapted from PostgreSQL src/include/storage/checksum_impl.h. ***********************************************************************************************************************************/ #include #include #include "postgres/interface/static.vendor.h" /*********************************************************************************************************************************** Page checksum calculation ***********************************************************************************************************************************/ // Number of checksums to calculate in parallel #define PARALLEL_SUM 32 // Prime multiplier of FNV-1a hash #define FNV_PRIME 16777619 // Calculate one round of the checksum #define CHECKSUM_ROUND(checksum, value) \ do \ { \ const uint32_t tmp = (checksum) ^ (value); \ checksum = tmp * FNV_PRIME ^ (tmp >> 17); \ } while (0) // Main calculation loop #define CHECKSUM_CASE(pageSize) \ case pageSize: \ for (uint32_t i = 0; i < (uint32) (pageSize / (sizeof(uint32) * PARALLEL_SUM)); i++) \ for (uint32_t j = 0; j < PARALLEL_SUM; j++) \ CHECKSUM_ROUND(sums[j], ((PgPageChecksum##pageSize *)page)->data[i][j]); \ \ break; /*********************************************************************************************************************************** Define unions that will make the code valid under strict aliasing for each page size ***********************************************************************************************************************************/ #define CHECKSUM_UNION(pageSize) \ typedef union \ { \ PageHeaderData phdr; \ uint32_t data[pageSize / (sizeof(uint32_t) * PARALLEL_SUM)][PARALLEL_SUM]; \ } PgPageChecksum##pageSize; CHECKSUM_UNION(pgPageSize1); CHECKSUM_UNION(pgPageSize2); CHECKSUM_UNION(pgPageSize4); CHECKSUM_UNION(pgPageSize8); CHECKSUM_UNION(pgPageSize16); CHECKSUM_UNION(pgPageSize32); /**********************************************************************************************************************************/ FN_EXTERN uint16_t pgPageChecksum(uint8_t *const page, const uint32_t blockNo, const PgPageSize pageSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(BYTEDATA, page); FUNCTION_TEST_PARAM(UINT, blockNo); FUNCTION_TEST_PARAM(ENUM, pageSize); FUNCTION_TEST_END(); // Save pd_checksum and temporarily set it to zero, so that the checksum calculation isn't affected by the old checksum stored // on the page. Restore it after, because actually updating the checksum is NOT part of the API of this function. const uint16_t checksumPrior = ((PageHeaderData *)page)->pd_checksum; ((PageHeaderData *)page)->pd_checksum = 0; // Initialize partial checksums to their corresponding offsets uint32_t sums[PARALLEL_SUM] = { 0x5b1f36e9, 0xb8525960, 0x02ab50aa, 0x1de66d2a, 0x79ff467a, 0x9bb9f8a3, 0x217e7cd2, 0x83e13d2c, 0xf8d4474f, 0xe39eb970, 0x42c6ae16, 0x993216fa, 0x7b093b5d, 0x98daff3c, 0xf718902a, 0x0b1c9cdb, 0xe58f764b, 0x187636bc, 0x5d7b3bb1, 0xe73de7de, 0x92bec979, 0xcca6c0b2, 0x304a0979, 0x85aa43d4, 0x783125bb, 0x6ca8eaa2, 0xe407eac6, 0x4b5cfc3e, 0x9fbf8c76, 0x15ca20be, 0xf2ca9fd3, 0x959bd756, }; // Main checksum calculation switch (pageSize) { CHECKSUM_CASE(pgPageSize8); // Default page size should be checked first CHECKSUM_CASE(pgPageSize1); CHECKSUM_CASE(pgPageSize2); CHECKSUM_CASE(pgPageSize4); CHECKSUM_CASE(pgPageSize16); CHECKSUM_CASE(pgPageSize32); default: pgPageSizeCheck(pageSize); } // Add in two rounds of zeroes for additional mixing for (uint32_t i = 0; i < 2; i++) for (uint32_t j = 0; j < PARALLEL_SUM; j++) CHECKSUM_ROUND(sums[j], 0); // Xor fold partial checksums together uint32 result = 0; for (uint32_t i = 0; i < PARALLEL_SUM; i++) result ^= sums[i]; // Restore prior checksum ((PageHeaderData *)page)->pd_checksum = checksumPrior; // Mix in the block number to detect transposed pages result ^= blockNo; // Reduce to a uint16 (to fit in the pd_checksum field) with an offset of one. That avoids checksums of zero, which seems like a // good idea. FUNCTION_TEST_RETURN(UINT16, (uint16_t)((result % 65535) + 1)); } /**********************************************************************************************************************************/ FN_EXTERN bool pgPageSizeValid(const PgPageSize pageSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, pageSize); FUNCTION_TEST_END(); switch (pageSize) { case pgPageSize1: case pgPageSize2: case pgPageSize4: case pgPageSize8: case pgPageSize16: case pgPageSize32: FUNCTION_TEST_RETURN(BOOL, true); } FUNCTION_TEST_RETURN(BOOL, false); } /**********************************************************************************************************************************/ FN_EXTERN void pgPageSizeCheck(const PgPageSize pageSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, pageSize); FUNCTION_TEST_END(); if (!pgPageSizeValid(pageSize)) { THROW_FMT( FormatError, "page size is %u but only %i, %i, %i, %i, %i, and %i are supported", pageSize, pgPageSize1, pgPageSize2, pgPageSize4, pgPageSize8, pgPageSize16, pgPageSize32); } FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/interface/static.vendor.h0000664000175100017510000003217315240547630022752 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Types That Do Not Vary By Version Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group Portions Copyright (c) 1994, Regents of the University of California For each supported release of PostgreSQL check the types in this file to see if they have changed. The easiest way to do this is to copy and paste in place and check git to see if there are any diffs. Tabs should be copied as is to make this process easy even though the pgBackRest project does not use tabs elsewhere. Comments should be copied with the types they apply to, even if the comment has not changed. This does get repetitive, but has no runtime cost and makes the rules a bit easier to follow. If a comment is changed then the newer comment should be copied. If the *type* has changed then it must be moved to version.auto.c which could have a large impact on dependencies. Hopefully that won't happen often. Note when adding new types it is safer to add them to version.auto.c unless they are needed for code that must be compatible across all versions of PostgreSQL supported by pgBackRest. ***********************************************************************************************************************************/ #ifndef POSTGRES_INTERFACE_STATICVENDOR_H #define POSTGRES_INTERFACE_STATICVENDOR_H #include "common/assert.h" #include "postgres/interface.h" /*********************************************************************************************************************************** Types from src/include/c.h ***********************************************************************************************************************************/ // uint16 type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint16_t uint16; // uint32 type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint32_t uint32; // uint64 type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint64_t uint64; // TransactionId type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint32 TransactionId; // FLEXIBLE_ARRAY_MEMBER macro // --------------------------------------------------------------------------------------------------------------------------------- /* * We require C99, hence the compiler should understand flexible array * members. However, for documentation purposes we still consider it to be * project style to write "field[FLEXIBLE_ARRAY_MEMBER]" not just "field[]". * When computing the size of such an object, use "offsetof(struct s, f)" * for portability. Don't use "offsetof(struct s, f[0])", as this doesn't * work with MSVC and with C++ compilers. */ #define FLEXIBLE_ARRAY_MEMBER /* empty */ /*********************************************************************************************************************************** Types from src/include/storage/checksum.h ***********************************************************************************************************************************/ // ChecksumStateType enum // --------------------------------------------------------------------------------------------------------------------------------- /* * Checksum state 0 is used for when data checksums are disabled (OFF). * PG_DATA_CHECKSUM_INPROGRESS_{ON|OFF} defines that data checksums are either * currently being enabled or disabled, and PG_DATA_CHECKSUM_VERSION defines * that data checksums are enabled. The ChecksumStateType is stored in * pg_control so changing requires a catversion bump, and the values cannot * be reordered. New states must be added at the end. */ typedef enum ChecksumStateType { PG_DATA_CHECKSUM_OFF = 0, PG_DATA_CHECKSUM_VERSION = 1, PG_DATA_CHECKSUM_INPROGRESS_OFF = 2, PG_DATA_CHECKSUM_INPROGRESS_ON = 3, } ChecksumStateType; /*********************************************************************************************************************************** Types from src/include/storage/itemid.h ***********************************************************************************************************************************/ // ItemIdData type // --------------------------------------------------------------------------------------------------------------------------------- /* * A line pointer on a buffer page. See buffer page definitions and comments * for an explanation of how line pointers are used. * * In some cases a line pointer is "in use" but does not have any associated * storage on the page. By convention, lp_len == 0 in every line pointer * that does not have storage, independently of its lp_flags state. */ typedef struct ItemIdData { unsigned lp_off:15, /* offset to tuple (from start of page) */ lp_flags:2, /* state of line pointer, see below */ lp_len:15; /* byte length of tuple */ } ItemIdData; /*********************************************************************************************************************************** Types from src/include/storage/bufpage.h ***********************************************************************************************************************************/ // LocationIndex type // --------------------------------------------------------------------------------------------------------------------------------- /* * location (byte offset) within a page. * * note that this is actually limited to 2^15 because we have limited * ItemIdData.lp_off and ItemIdData.lp_len to 15 bits (see itemid.h). */ typedef uint16 LocationIndex; // PageXLogRecPtr type // --------------------------------------------------------------------------------------------------------------------------------- /* * Store the LSN as a single 64-bit value, to allow atomic loads/stores. * * For historical reasons, the storage of 64-bit LSN values depends on CPU * endianness; PageXLogRecPtr used to be a struct consisting of two 32-bit * values. When reading (and writing) the pd_lsn field from page headers, the * caller must convert from (and convert to) the platform's native endianness. */ typedef struct { uint64 lsn; } PageXLogRecPtr; // PageHeaderData type // --------------------------------------------------------------------------------------------------------------------------------- /* * disk page organization * * space management information generic to any page * * pd_lsn - identifies xlog record for last change to this page. * pd_checksum - page checksum, if set. * pd_flags - flag bits. * pd_lower - offset to start of free space. * pd_upper - offset to end of free space. * pd_special - offset to start of special space. * pd_pagesize_version - size in bytes and page layout version number. * pd_prune_xid - oldest XID among potentially prunable tuples on page. * * The LSN is used by the buffer manager to enforce the basic rule of WAL: * "thou shalt write xlog before data". A dirty buffer cannot be dumped * to disk until xlog has been flushed at least as far as the page's LSN. * * pd_checksum stores the page checksum, if it has been set for this page; * zero is a valid value for a checksum. If a checksum is not in use then * we leave the field unset. This will typically mean the field is zero * though non-zero values may also be present if databases have been * pg_upgraded from releases prior to 9.3, when the same byte offset was * used to store the current timelineid when the page was last updated. * Note that there is no indication on a page as to whether the checksum * is valid or not, a deliberate design choice which avoids the problem * of relying on the page contents to decide whether to verify it. Hence * there are no flag bits relating to checksums. * * pd_prune_xid is a hint field that helps determine whether pruning will be * useful. It is currently unused in index pages. * * The page version number and page size are packed together into a single * uint16 field. This is for historical reasons: before PostgreSQL 7.3, * there was no concept of a page version number, and doing it this way * lets us pretend that pre-7.3 databases have page version number zero. * We constrain page sizes to be multiples of 256, leaving the low eight * bits available for a version number. * * Minimum possible page size is perhaps 64B to fit page header, opaque space * and a minimal tuple; of course, in reality you want it much bigger, so * the constraint on pagesize mod 256 is not an important restriction. * On the high end, we can only support pages up to 32KB because lp_off/lp_len * are 15 bits. */ typedef struct PageHeaderData { /* XXX LSN is member of *any* block, not only page-organized ones */ PageXLogRecPtr pd_lsn; /* LSN: next byte after last byte of xlog * record for last change to this page */ uint16 pd_checksum; /* checksum */ uint16 pd_flags; /* flag bits, see below */ LocationIndex pd_lower; /* offset to start of free space */ LocationIndex pd_upper; /* offset to end of free space */ LocationIndex pd_special; /* offset to start of special space */ uint16 pd_pagesize_version; TransactionId pd_prune_xid; /* oldest prunable XID, or zero if none */ ItemIdData pd_linp[FLEXIBLE_ARRAY_MEMBER]; /* line pointer array */ } PageHeaderData; // PageHeader type // --------------------------------------------------------------------------------------------------------------------------------- typedef PageHeaderData *PageHeader; /*********************************************************************************************************************************** Types from src/include/access/transam.h ***********************************************************************************************************************************/ // FirstNormalObjectId define // --------------------------------------------------------------------------------------------------------------------------------- /* * Object ID (OID) zero is InvalidOid. * * OIDs 1-9999 are reserved for manual assignment (see .dat files in * src/include/catalog/). Of these, 8000-9999 are reserved for * development purposes (such as in-progress patches and forks); * they should not appear in released versions. * * OIDs 10000-11999 are reserved for assignment by genbki.pl, for use * when the .dat files in src/include/catalog/ do not specify an OID * for a catalog entry that requires one. Note that genbki.pl assigns * these OIDs independently in each catalog, so they're not guaranteed * to be globally unique. Furthermore, the bootstrap backend and * initdb's post-bootstrap processing can also assign OIDs in this range. * The normal OID-generation logic takes care of any OID conflicts that * might arise from that. * * OIDs 12000-16383 are reserved for unpinned objects created by initdb's * post-bootstrap processing. initdb forces the OID generator up to * 12000 as soon as it's made the pinned objects it's responsible for. * * OIDs beginning at 16384 are assigned from the OID generator * during normal multiuser operation. (We force the generator up to * 16384 as soon as we are in normal operation.) * * The choices of 8000, 10000 and 12000 are completely arbitrary, and can be * moved if we run low on OIDs in any category. Changing the macros below, * and updating relevant documentation (see bki.sgml and RELEASE_CHANGES), * should be sufficient to do this. Moving the 16384 boundary between * initdb-assigned OIDs and user-defined objects would be substantially * more painful, however, since some user-defined OIDs will appear in * on-disk data; such a change would probably break pg_upgrade. * * NOTE: if the OID generator wraps around, we skip over OIDs 0-16383 * and resume with 16384. This minimizes the odds of OID conflict, by not * reassigning OIDs that might have been assigned during initdb. Critically, * it also ensures that no user-created object will be considered pinned. */ #define FirstNormalObjectId 16384 /*********************************************************************************************************************************** Types from src/include/access/xlog_internal.h ***********************************************************************************************************************************/ // WalSegMinSize/WalSegMinSize macros // --------------------------------------------------------------------------------------------------------------------------------- /* wal_segment_size can range from 1MB to 1GB */ #define WalSegMinSize 1024 * 1024 #define WalSegMaxSize 1024 * 1024 * 1024 // IsPowerOf2/IsValidWalSegSize macros // --------------------------------------------------------------------------------------------------------------------------------- /* check that the given size is a valid wal_segment_size */ #define IsPowerOf2(x) (x > 0 && ((x) & ((x)-1)) == 0) #define IsValidWalSegSize(size) \ (IsPowerOf2(size) && \ ((size) >= WalSegMinSize && (size) <= WalSegMaxSize)) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/interface/version.intern.h0000664000175100017510000003017715240547630023154 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Version Interface Macros for building version-specific functions that interface with the types in version.vendor.h. Due to the way PostgreSQL types evolve over time, this seems to be the easiest way to extract information from them. These macros should be kept as simple as possible, with most of the logic contained in postgres/interface.c. ***********************************************************************************************************************************/ #include "postgres/interface/version.vendor.h" #include "postgres/version.h" /*********************************************************************************************************************************** Determine if the supplied pg_control is for this version of PostgreSQL. When CATALOG_VERSION_NO_MAX is defined then the catalog will be accepted as a range that lasts until the end of the encoded year. This allows pgBackRest to work with PostgreSQL during the alpha/beta/rc period without needing to be updated, unless of course the actual interface changes. ***********************************************************************************************************************************/ #ifdef CATALOG_VERSION_NO_MAX #define PG_INTERFACE_CONTROL_IS(version) \ static bool \ pgInterfaceControlIs##version(const uint8_t *controlFile) \ { \ ASSERT(controlFile != NULL); \ \ return \ ((const ControlFileData *)controlFile)->pg_control_version == PG_CONTROL_VERSION && \ ((const ControlFileData *)controlFile)->catalog_version_no >= CATALOG_VERSION_NO && \ ((const ControlFileData *)controlFile)->catalog_version_no < (CATALOG_VERSION_NO / 100000 + 1) * 100000; \ } #else #define PG_INTERFACE_CONTROL_IS(version) \ static bool \ pgInterfaceControlIs##version(const uint8_t *controlFile) \ { \ ASSERT(controlFile != NULL); \ \ return \ ((const ControlFileData *)controlFile)->pg_control_version == PG_CONTROL_VERSION && \ ((const ControlFileData *)controlFile)->catalog_version_no == CATALOG_VERSION_NO; \ } #endif /*********************************************************************************************************************************** Read the version specific pg_control into a general data structure ***********************************************************************************************************************************/ #define PG_INTERFACE_CONTROL(version) \ static PgControl \ pgInterfaceControl##version(const uint8_t *controlFile) \ { \ ASSERT(controlFile != NULL); \ \ return (PgControl) \ { \ .systemId = ((const ControlFileData *)controlFile)->system_identifier, \ .catalogVersion = ((const ControlFileData *)controlFile)->catalog_version_no, \ .checkpoint = ((const ControlFileData *)controlFile)->checkPoint, \ .timeline = ((const ControlFileData *)controlFile)->checkPointCopy.ThisTimeLineID, \ .pageSize = ((const ControlFileData *)controlFile)->blcksz, \ .walSegmentSize = ((const ControlFileData *)controlFile)->xlog_seg_size, \ .pageChecksumVersion = ((const ControlFileData *)controlFile)->data_checksum_version, \ }; \ } /*********************************************************************************************************************************** Get control crc offset ***********************************************************************************************************************************/ #define PG_INTERFACE_CONTROL_CRC_OFFSET(version) \ static size_t \ pgInterfaceControlCrcOffset##version(void) \ { \ return offsetof(ControlFileData, crc); \ } /*********************************************************************************************************************************** Invalidate control checkpoint. PostgreSQL skips the first segment so any LSN in that segment is invalid. ***********************************************************************************************************************************/ #define PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(version) \ static void \ pgInterfaceControlCheckpointInvalidate##version(uint8_t *const controlFile) \ { \ ((ControlFileData *)controlFile)->checkPoint = PG_CONTROL_CHECKPOINT_INVALID; \ } /*********************************************************************************************************************************** Get the control version ***********************************************************************************************************************************/ #define PG_INTERFACE_CONTROL_VERSION(version) \ static uint32_t \ pgInterfaceControlVersion##version(void) \ { \ return PG_CONTROL_VERSION; \ } /*********************************************************************************************************************************** Determine if the supplied WAL is for this version of PostgreSQL ***********************************************************************************************************************************/ #define PG_INTERFACE_WAL_IS(version) \ static bool \ pgInterfaceWalIs##version(const uint8_t *walFile) \ { \ ASSERT(walFile != NULL); \ \ return ((const XLogPageHeaderData *)walFile)->xlp_magic == XLOG_PAGE_MAGIC; \ } /*********************************************************************************************************************************** Read the version specific WAL header into a general data structure ***********************************************************************************************************************************/ #define PG_INTERFACE_WAL(version) \ static PgWal \ pgInterfaceWal##version(const uint8_t *walFile) \ { \ ASSERT(walFile != NULL); \ \ return (PgWal) \ { \ .systemId = ((const XLogLongPageHeaderData *)walFile)->xlp_sysid, \ .size = ((const XLogLongPageHeaderData *)walFile)->xlp_seg_size, \ }; \ } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/interface/version.vendor.h0000664000175100017510000021123515240547630023146 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Types That Vary By Version Portions Copyright (c) 1996-2026, PostgreSQL Global Development Group Portions Copyright (c) 1994, Regents of the University of California For each supported release of PostgreSQL check the types in this file to see if they have changed. The easiest way to do this is to copy and paste in place and check git to see if there are any diffs. Tabs should be copied as is to make this process easy even though the pgBackRest project does not use tabs elsewhere. New versions should always be added to the top of each type's #if block, underneath `PG_VERSION > PG_VERSION_MAX` to cause as little churn as possible. This also ensures that new versions will not work until PG_VERSION_MAX and this file have been updated. New data structures do not need to add #elif branches for old versions. See FullTransactionId as an example. Comments should be copied with the types they apply to, even if the comment has not changed. This does get repetitive, but has no runtime cost and makes the rules a bit easier to follow. If a comment has syntax only changes, then the new version of the comment can be applied to older versions of the type. If a comment has changed in a way that implies a difference in the way the type is used, then a new version of the comment and type should be created. Unversioned types that are only used by versioned types are also included in this file. ***********************************************************************************************************************************/ #include "postgres/interface/static.vendor.h" #include "postgres/version.h" /*********************************************************************************************************************************** Types from src/include/c.h ***********************************************************************************************************************************/ // int64 type. The definition in c.h is more complicated but here we can rely on stdint.h for the correct type. // --------------------------------------------------------------------------------------------------------------------------------- typedef int64_t int64; // MultiXactId type // --------------------------------------------------------------------------------------------------------------------------------- /* MultiXactId must be equivalent to TransactionId, to fit in t_xmax */ typedef TransactionId MultiXactId; // MultiXactOffset // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 typedef uint64 MultiXactOffset; #elif PG_VERSION >= PG_VERSION_96 typedef uint32 MultiXactOffset; #endif /*********************************************************************************************************************************** Types from src/include/pgtime.h ***********************************************************************************************************************************/ // pg_time_t type // --------------------------------------------------------------------------------------------------------------------------------- /* * The API of this library is generally similar to the corresponding * C library functions, except that we use pg_time_t which (we hope) is * 64 bits wide, and which is most definitely signed not unsigned. */ typedef int64 pg_time_t; /*********************************************************************************************************************************** Types from src/include/postgres_ext.h ***********************************************************************************************************************************/ // Oid Type // --------------------------------------------------------------------------------------------------------------------------------- /* * Object ID is a fundamental type in Postgres. */ typedef unsigned int Oid; /*********************************************************************************************************************************** Types from src/include/port/pg_crc32.h ***********************************************************************************************************************************/ // pg_crc32/c type // --------------------------------------------------------------------------------------------------------------------------------- typedef uint32 pg_crc32c; /*********************************************************************************************************************************** Types from src/include/access/xlogdefs.h ***********************************************************************************************************************************/ // XLogRecPtr type // --------------------------------------------------------------------------------------------------------------------------------- /* * Pointer to a location in the XLOG. These pointers are 64 bits wide, * because we don't want them ever to overflow. */ typedef uint64 XLogRecPtr; // TimeLineID type // --------------------------------------------------------------------------------------------------------------------------------- /* * TimeLineID (TLI) - identifies different database histories to prevent * confusion after restoring a prior state of a database installation. * TLI does not change in a normal stop/restart of the database (including * crash-and-recover cases); but we must assign a new TLI after doing * a recovery to a prior state, a/k/a point-in-time recovery. This makes * the new WAL logfile sequence we generate distinguishable from the * sequence that was generated in the previous incarnation. */ typedef uint32 TimeLineID; /*********************************************************************************************************************************** Types from src/include/catalog/catversion.h ***********************************************************************************************************************************/ // CATALOG_VERSION_NO define // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202605131 #elif PG_VERSION >= PG_VERSION_18 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202506291 #elif PG_VERSION >= PG_VERSION_17 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202406281 #elif PG_VERSION >= PG_VERSION_16 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202307071 #elif PG_VERSION >= PG_VERSION_15 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202209061 #elif PG_VERSION >= PG_VERSION_14 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202107181 #elif PG_VERSION >= PG_VERSION_13 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 202007201 #elif PG_VERSION >= PG_VERSION_12 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 201909212 #elif PG_VERSION >= PG_VERSION_11 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 201809051 #elif PG_VERSION >= PG_VERSION_10 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 201707211 #elif PG_VERSION >= PG_VERSION_96 /* * We could use anything we wanted for version numbers, but I recommend * following the "YYYYMMDDN" style often used for DNS zone serial numbers. * YYYYMMDD are the date of the change, and N is the number of the change * on that day. (Hopefully we'll never commit ten independent sets of * catalog changes on the same day...) */ /* yyyymmddN */ #define CATALOG_VERSION_NO 201608131 #endif /*********************************************************************************************************************************** Types from src/include/access/transam.h ***********************************************************************************************************************************/ // FullTransactionId type // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_12 /* * A 64 bit value that contains an epoch and a TransactionId. This is * wrapped in a struct to prevent implicit conversion to/from TransactionId. * Not all values represent valid normal XIDs. */ typedef struct FullTransactionId { uint64 value; } FullTransactionId; #endif /*********************************************************************************************************************************** Types from src/include/catalog/pg_control.h ***********************************************************************************************************************************/ // PG_CONTROL_VERSION define // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1902 #elif PG_VERSION >= PG_VERSION_18 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1800 #elif PG_VERSION >= PG_VERSION_17 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1700 #elif PG_VERSION >= PG_VERSION_13 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1300 #elif PG_VERSION >= PG_VERSION_12 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1201 #elif PG_VERSION >= PG_VERSION_11 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1100 #elif PG_VERSION >= PG_VERSION_10 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 1002 #elif PG_VERSION >= PG_VERSION_96 /* Version identifier for this pg_control format */ #define PG_CONTROL_VERSION 960 #endif // MOCK_AUTH_NONCE_LEN define // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_10 /* Nonce key length, see below */ #define MOCK_AUTH_NONCE_LEN 32 #endif // CheckPoint Type // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ int wal_level; /* current wal_level */ bool logicalDecodingEnabled; /* current logical decoding status */ FullTransactionId nextXid; /* next free transaction ID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; /* data checksums state at the time of the checkpoint */ uint32 dataChecksumState; } CheckPoint; #elif PG_VERSION >= PG_VERSION_17 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ int wal_level; /* current wal_level */ FullTransactionId nextXid; /* next free transaction ID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; } CheckPoint; #elif PG_VERSION >= PG_VERSION_14 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ FullTransactionId nextXid; /* next free transaction ID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; } CheckPoint; #elif PG_VERSION >= PG_VERSION_12 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ FullTransactionId nextFullXid; /* next free full transaction ID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; } CheckPoint; #elif PG_VERSION >= PG_VERSION_96 /* * Body of CheckPoint XLOG records. This is declared here because we keep * a copy of the latest one in pg_control for possible disaster recovery. * Changing this struct requires a PG_CONTROL_VERSION bump. */ typedef struct CheckPoint { XLogRecPtr redo; /* next RecPtr available when we began to * create CheckPoint (i.e. REDO start point) */ TimeLineID ThisTimeLineID; /* current TLI */ TimeLineID PrevTimeLineID; /* previous TLI, if this record begins a new * timeline (equals ThisTimeLineID otherwise) */ bool fullPageWrites; /* current full_page_writes */ uint32 nextXidEpoch; /* higher-order bits of nextXid */ TransactionId nextXid; /* next free XID */ Oid nextOid; /* next free OID */ MultiXactId nextMulti; /* next free MultiXactId */ MultiXactOffset nextMultiOffset; /* next free MultiXact offset */ TransactionId oldestXid; /* cluster-wide minimum datfrozenxid */ Oid oldestXidDB; /* database with minimum datfrozenxid */ MultiXactId oldestMulti; /* cluster-wide minimum datminmxid */ Oid oldestMultiDB; /* database with minimum datminmxid */ pg_time_t time; /* time stamp of checkpoint */ TransactionId oldestCommitTsXid; /* oldest Xid with valid commit * timestamp */ TransactionId newestCommitTsXid; /* newest Xid with valid commit * timestamp */ /* * Oldest XID still running. This is only needed to initialize hot standby * mode from an online checkpoint, so we only bother calculating this for * online checkpoints and only when wal_level is replica. Otherwise it's * set to InvalidTransactionId. */ TransactionId oldestActiveXid; } CheckPoint; #endif // DBState enum // --------------------------------------------------------------------------------------------------------------------------------- /* * System status indicator. Note this is stored in pg_control; if you change * it, you must bump PG_CONTROL_VERSION */ typedef enum DBState { DB_STARTUP = 0, DB_SHUTDOWNED, DB_SHUTDOWNED_IN_RECOVERY, DB_SHUTDOWNING, DB_IN_CRASH_RECOVERY, DB_IN_ARCHIVE_RECOVERY, DB_IN_PRODUCTION, } DBState; // ControlFileData type // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 slru_pages_per_segment; /* size of each SLRU segment */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * True if the default signedness of char is "signed" on a platform where * the cluster is initialized. */ bool default_char_signedness; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_18 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * True if the default signedness of char is "signed" on a platform where * the cluster is initialized. */ bool default_char_signedness; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_15 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_13 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_12 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_wal_senders; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ /* flags indicating pass-by-value status of various types */ bool float4ByVal; /* float4 pass-by-value? */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_11 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ /* flags indicating pass-by-value status of various types */ bool float4ByVal; /* float4 pass-by-value? */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_10 /* * Contents of pg_control. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ XLogRecPtr prevCheckPoint; /* previous check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ /* flags indicating pass-by-value status of various types */ bool float4ByVal; /* float4 pass-by-value? */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* * Random nonce, used in authentication requests that need to proceed * based on values that are cluster-unique, like a SASL exchange that * failed at an early stage. */ char mock_authentication_nonce[MOCK_AUTH_NONCE_LEN]; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #elif PG_VERSION >= PG_VERSION_96 /* * Contents of pg_control. * * NOTE: try to keep this under 512 bytes so that it will fit on one physical * sector of typical disk drives. This reduces the odds of corruption due to * power failure midway through a write. */ typedef struct ControlFileData { /* * Unique system identifier --- to ensure we match up xlog files with the * installation that produced them. */ uint64 system_identifier; /* * Version identifier information. Keep these fields at the same offset, * especially pg_control_version; they won't be real useful if they move * around. (For historical reasons they must be 8 bytes into the file * rather than immediately at the front.) * * pg_control_version identifies the format of pg_control itself. * catalog_version_no identifies the format of the system catalogs. * * There are additional version identifiers in individual files; for * example, WAL logs contain per-page magic numbers that can serve as * version cues for the WAL log. */ uint32 pg_control_version; /* PG_CONTROL_VERSION */ uint32 catalog_version_no; /* see catversion.h */ /* * System status data */ DBState state; /* see enum above */ pg_time_t time; /* time stamp of last pg_control update */ XLogRecPtr checkPoint; /* last check point record ptr */ XLogRecPtr prevCheckPoint; /* previous check point record ptr */ CheckPoint checkPointCopy; /* copy of last check point record */ XLogRecPtr unloggedLSN; /* current fake LSN value, for unlogged rels */ /* * These two values determine the minimum point we must recover up to * before starting up: * * minRecoveryPoint is updated to the latest replayed LSN whenever we * flush a data change during archive recovery. That guards against * starting archive recovery, aborting it, and restarting with an earlier * stop location. If we've already flushed data changes from WAL record X * to disk, we mustn't start up until we reach X again. Zero when not * doing archive recovery. * * backupStartPoint is the redo pointer of the backup start checkpoint, if * we are recovering from an online backup and haven't reached the end of * backup yet. It is reset to zero when the end of backup is reached, and * we mustn't start up before that. A boolean would suffice otherwise, but * we use the redo pointer as a cross-check when we see an end-of-backup * record, to make sure the end-of-backup record corresponds the base * backup we're recovering from. * * backupEndPoint is the backup end location, if we are recovering from an * online backup which was taken from the standby and haven't reached the * end of backup yet. It is initialized to the minimum recovery point in * pg_control which was backed up last. It is reset to zero when the end * of backup is reached, and we mustn't start up before that. * * If backupEndRequired is true, we know for sure that we're restoring * from a backup, and must see a backup-end record before we can safely * start up. If it's false, but backupStartPoint is set, a backup_label * file was found at startup but it may have been a leftover from a stray * pg_start_backup() call, not accompanied by pg_stop_backup(). */ XLogRecPtr minRecoveryPoint; TimeLineID minRecoveryPointTLI; XLogRecPtr backupStartPoint; XLogRecPtr backupEndPoint; bool backupEndRequired; /* * Parameter settings that determine if the WAL can be used for archival * or hot standby. */ int wal_level; bool wal_log_hints; int MaxConnections; int max_worker_processes; int max_prepared_xacts; int max_locks_per_xact; bool track_commit_timestamp; /* * This data is used to check for hardware-architecture compatibility of * the database and the backend executable. We need not check endianness * explicitly, since the pg_control version will surely look wrong to a * machine of different endianness, but we do need to worry about MAXALIGN * and floating-point format. (Note: storage layout nominally also * depends on SHORTALIGN and INTALIGN, but in practice these are the same * on all architectures of interest.) * * Testing just one double value is not a very bulletproof test for * floating-point compatibility, but it will catch most cases. */ uint32 maxAlign; /* alignment requirement for tuples */ double floatFormat; /* constant 1234567.0 */ #define FLOATFORMAT_VALUE 1234567.0 /* * This data is used to make sure that configuration of this database is * compatible with the backend executable. */ uint32 blcksz; /* data block size for this DB */ uint32 relseg_size; /* blocks per segment of large relation */ uint32 xlog_blcksz; /* block size within WAL files */ uint32 xlog_seg_size; /* size of each WAL segment */ uint32 nameDataLen; /* catalog name field width */ uint32 indexMaxKeys; /* max number of columns in an index */ uint32 toast_max_chunk_size; /* chunk size in TOAST tables */ uint32 loblksize; /* chunk size in pg_largeobject */ /* flag indicating internal format of timestamp, interval, time */ bool enableIntTimes; /* int64 storage enabled? */ /* flags indicating pass-by-value status of various types */ bool float4ByVal; /* float4 pass-by-value? */ bool float8ByVal; /* float8, int8, etc pass-by-value? */ /* Are data pages protected by checksums? Zero if no checksum version */ uint32 data_checksum_version; /* CRC of all above ... MUST BE LAST! */ pg_crc32c crc; } ControlFileData; #endif /*********************************************************************************************************************************** Types from src/include/access/xlog_internal.h ***********************************************************************************************************************************/ // XLOG_PAGE_MAGIC define // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_19 #define XLOG_PAGE_MAGIC 0xD121 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_18 #define XLOG_PAGE_MAGIC 0xD118 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_17 #define XLOG_PAGE_MAGIC 0xD116 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_16 #define XLOG_PAGE_MAGIC 0xD113 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_15 #define XLOG_PAGE_MAGIC 0xD110 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_14 #define XLOG_PAGE_MAGIC 0xD10D /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_13 #define XLOG_PAGE_MAGIC 0xD106 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_12 #define XLOG_PAGE_MAGIC 0xD101 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_11 #define XLOG_PAGE_MAGIC 0xD098 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_10 #define XLOG_PAGE_MAGIC 0xD097 /* can be used as WAL version indicator */ #elif PG_VERSION >= PG_VERSION_96 #define XLOG_PAGE_MAGIC 0xD093 /* can be used as WAL version indicator */ #endif // XLogPageHeaderData type // --------------------------------------------------------------------------------------------------------------------------------- #if PG_VERSION > PG_VERSION_MAX #elif PG_VERSION >= PG_VERSION_14 /* * Each page of XLOG file has a header like this: */ typedef struct XLogPageHeaderData { uint16 xlp_magic; /* magic value for correctness checks */ uint16 xlp_info; /* flag bits, see below */ TimeLineID xlp_tli; /* TimeLineID of first record on page */ XLogRecPtr xlp_pageaddr; /* XLOG address of this page */ /* * When there is not enough space on current page for whole record, we * continue on the next page. xlp_rem_len is the number of bytes * remaining from a previous page; it tracks xl_tot_len in the initial * header. Note that the continuation data isn't necessarily aligned. */ uint32 xlp_rem_len; /* total len of remaining data for record */ } XLogPageHeaderData; #elif PG_VERSION >= PG_VERSION_96 /* * Each page of XLOG file has a header like this: */ typedef struct XLogPageHeaderData { uint16 xlp_magic; /* magic value for correctness checks */ uint16 xlp_info; /* flag bits, see below */ TimeLineID xlp_tli; /* TimeLineID of first record on page */ XLogRecPtr xlp_pageaddr; /* XLOG address of this page */ /* * When there is not enough space on current page for whole record, we * continue on the next page. xlp_rem_len is the number of bytes * remaining from a previous page. * * Note that xlp_rem_len includes backup-block data; that is, it tracks * xl_tot_len not xl_len in the initial header. Also note that the * continuation data isn't necessarily aligned. */ uint32 xlp_rem_len; /* total len of remaining data for record */ } XLogPageHeaderData; #endif // XLogLongPageHeaderData type // --------------------------------------------------------------------------------------------------------------------------------- /* * When the XLP_LONG_HEADER flag is set, we store additional fields in the * page header. (This is ordinarily done just in the first page of an * XLOG file.) The additional fields serve to identify the file accurately. */ typedef struct XLogLongPageHeaderData { XLogPageHeaderData std; /* standard header fields */ uint64 xlp_sysid; /* system identifier from pg_control */ uint32 xlp_seg_size; /* just as a cross-check */ uint32 xlp_xlog_blcksz; /* just as a cross-check */ } XLogLongPageHeaderData; // XLP_LONG_HEADER define // --------------------------------------------------------------------------------------------------------------------------------- /* This flag indicates a "long" page header */ #define XLP_LONG_HEADER 0x0002 ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7676206 pgbackrest-2.59.1/src/postgres/interface.auto.c.inc0000640000175100017510000012130515240547776021702 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Interface Automatically generated by 'build-code postgres' -- do not modify directly. ***********************************************************************************************************************************/ /*********************************************************************************************************************************** PostgreSQL 19 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_19 #define CheckPoint CheckPoint_19 #define ControlFileData ControlFileData_19 #define DBState DBState_19 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_19 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_19 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_19 #define DB_SHUTDOWNED DB_SHUTDOWNED_19 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_19 #define DB_SHUTDOWNING DB_SHUTDOWNING_19 #define DB_STARTUP DB_STARTUP_19 #define FullTransactionId FullTransactionId_19 #define MultiXactId MultiXactId_19 #define MultiXactOffset MultiXactOffset_19 #define Oid Oid_19 #define TimeLineID TimeLineID_19 #define XLogLongPageHeaderData XLogLongPageHeaderData_19 #define XLogPageHeaderData XLogPageHeaderData_19 #define XLogRecPtr XLogRecPtr_19 #define int64 int64_19 #define pg_crc32c pg_crc32c_19 #define pg_time_t pg_time_t_19 #define CATALOG_VERSION_NO_MAX #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(19); PG_INTERFACE_CONTROL(19); PG_INTERFACE_CONTROL_CRC_OFFSET(19); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(19); PG_INTERFACE_CONTROL_VERSION(19); PG_INTERFACE_WAL_IS(19); PG_INTERFACE_WAL(19); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 18 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_18 #define CheckPoint CheckPoint_18 #define ControlFileData ControlFileData_18 #define DBState DBState_18 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_18 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_18 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_18 #define DB_SHUTDOWNED DB_SHUTDOWNED_18 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_18 #define DB_SHUTDOWNING DB_SHUTDOWNING_18 #define DB_STARTUP DB_STARTUP_18 #define FullTransactionId FullTransactionId_18 #define MultiXactId MultiXactId_18 #define MultiXactOffset MultiXactOffset_18 #define Oid Oid_18 #define TimeLineID TimeLineID_18 #define XLogLongPageHeaderData XLogLongPageHeaderData_18 #define XLogPageHeaderData XLogPageHeaderData_18 #define XLogRecPtr XLogRecPtr_18 #define int64 int64_18 #define pg_crc32c pg_crc32c_18 #define pg_time_t pg_time_t_18 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(18); PG_INTERFACE_CONTROL(18); PG_INTERFACE_CONTROL_CRC_OFFSET(18); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(18); PG_INTERFACE_CONTROL_VERSION(18); PG_INTERFACE_WAL_IS(18); PG_INTERFACE_WAL(18); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 17 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_17 #define CheckPoint CheckPoint_17 #define ControlFileData ControlFileData_17 #define DBState DBState_17 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_17 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_17 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_17 #define DB_SHUTDOWNED DB_SHUTDOWNED_17 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_17 #define DB_SHUTDOWNING DB_SHUTDOWNING_17 #define DB_STARTUP DB_STARTUP_17 #define FullTransactionId FullTransactionId_17 #define MultiXactId MultiXactId_17 #define MultiXactOffset MultiXactOffset_17 #define Oid Oid_17 #define TimeLineID TimeLineID_17 #define XLogLongPageHeaderData XLogLongPageHeaderData_17 #define XLogPageHeaderData XLogPageHeaderData_17 #define XLogRecPtr XLogRecPtr_17 #define int64 int64_17 #define pg_crc32c pg_crc32c_17 #define pg_time_t pg_time_t_17 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(17); PG_INTERFACE_CONTROL(17); PG_INTERFACE_CONTROL_CRC_OFFSET(17); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(17); PG_INTERFACE_CONTROL_VERSION(17); PG_INTERFACE_WAL_IS(17); PG_INTERFACE_WAL(17); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 16 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_16 #define CheckPoint CheckPoint_16 #define ControlFileData ControlFileData_16 #define DBState DBState_16 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_16 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_16 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_16 #define DB_SHUTDOWNED DB_SHUTDOWNED_16 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_16 #define DB_SHUTDOWNING DB_SHUTDOWNING_16 #define DB_STARTUP DB_STARTUP_16 #define FullTransactionId FullTransactionId_16 #define MultiXactId MultiXactId_16 #define MultiXactOffset MultiXactOffset_16 #define Oid Oid_16 #define TimeLineID TimeLineID_16 #define XLogLongPageHeaderData XLogLongPageHeaderData_16 #define XLogPageHeaderData XLogPageHeaderData_16 #define XLogRecPtr XLogRecPtr_16 #define int64 int64_16 #define pg_crc32c pg_crc32c_16 #define pg_time_t pg_time_t_16 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(16); PG_INTERFACE_CONTROL(16); PG_INTERFACE_CONTROL_CRC_OFFSET(16); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(16); PG_INTERFACE_CONTROL_VERSION(16); PG_INTERFACE_WAL_IS(16); PG_INTERFACE_WAL(16); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 15 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_15 #define CheckPoint CheckPoint_15 #define ControlFileData ControlFileData_15 #define DBState DBState_15 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_15 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_15 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_15 #define DB_SHUTDOWNED DB_SHUTDOWNED_15 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_15 #define DB_SHUTDOWNING DB_SHUTDOWNING_15 #define DB_STARTUP DB_STARTUP_15 #define FullTransactionId FullTransactionId_15 #define MultiXactId MultiXactId_15 #define MultiXactOffset MultiXactOffset_15 #define Oid Oid_15 #define TimeLineID TimeLineID_15 #define XLogLongPageHeaderData XLogLongPageHeaderData_15 #define XLogPageHeaderData XLogPageHeaderData_15 #define XLogRecPtr XLogRecPtr_15 #define int64 int64_15 #define pg_crc32c pg_crc32c_15 #define pg_time_t pg_time_t_15 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(15); PG_INTERFACE_CONTROL(15); PG_INTERFACE_CONTROL_CRC_OFFSET(15); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(15); PG_INTERFACE_CONTROL_VERSION(15); PG_INTERFACE_WAL_IS(15); PG_INTERFACE_WAL(15); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 14 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_14 #define CheckPoint CheckPoint_14 #define ControlFileData ControlFileData_14 #define DBState DBState_14 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_14 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_14 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_14 #define DB_SHUTDOWNED DB_SHUTDOWNED_14 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_14 #define DB_SHUTDOWNING DB_SHUTDOWNING_14 #define DB_STARTUP DB_STARTUP_14 #define FullTransactionId FullTransactionId_14 #define MultiXactId MultiXactId_14 #define MultiXactOffset MultiXactOffset_14 #define Oid Oid_14 #define TimeLineID TimeLineID_14 #define XLogLongPageHeaderData XLogLongPageHeaderData_14 #define XLogPageHeaderData XLogPageHeaderData_14 #define XLogRecPtr XLogRecPtr_14 #define int64 int64_14 #define pg_crc32c pg_crc32c_14 #define pg_time_t pg_time_t_14 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(14); PG_INTERFACE_CONTROL(14); PG_INTERFACE_CONTROL_CRC_OFFSET(14); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(14); PG_INTERFACE_CONTROL_VERSION(14); PG_INTERFACE_WAL_IS(14); PG_INTERFACE_WAL(14); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 13 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_13 #define CheckPoint CheckPoint_13 #define ControlFileData ControlFileData_13 #define DBState DBState_13 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_13 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_13 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_13 #define DB_SHUTDOWNED DB_SHUTDOWNED_13 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_13 #define DB_SHUTDOWNING DB_SHUTDOWNING_13 #define DB_STARTUP DB_STARTUP_13 #define FullTransactionId FullTransactionId_13 #define MultiXactId MultiXactId_13 #define MultiXactOffset MultiXactOffset_13 #define Oid Oid_13 #define TimeLineID TimeLineID_13 #define XLogLongPageHeaderData XLogLongPageHeaderData_13 #define XLogPageHeaderData XLogPageHeaderData_13 #define XLogRecPtr XLogRecPtr_13 #define int64 int64_13 #define pg_crc32c pg_crc32c_13 #define pg_time_t pg_time_t_13 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(13); PG_INTERFACE_CONTROL(13); PG_INTERFACE_CONTROL_CRC_OFFSET(13); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(13); PG_INTERFACE_CONTROL_VERSION(13); PG_INTERFACE_WAL_IS(13); PG_INTERFACE_WAL(13); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 12 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_12 #define CheckPoint CheckPoint_12 #define ControlFileData ControlFileData_12 #define DBState DBState_12 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_12 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_12 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_12 #define DB_SHUTDOWNED DB_SHUTDOWNED_12 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_12 #define DB_SHUTDOWNING DB_SHUTDOWNING_12 #define DB_STARTUP DB_STARTUP_12 #define FullTransactionId FullTransactionId_12 #define MultiXactId MultiXactId_12 #define MultiXactOffset MultiXactOffset_12 #define Oid Oid_12 #define TimeLineID TimeLineID_12 #define XLogLongPageHeaderData XLogLongPageHeaderData_12 #define XLogPageHeaderData XLogPageHeaderData_12 #define XLogRecPtr XLogRecPtr_12 #define int64 int64_12 #define pg_crc32c pg_crc32c_12 #define pg_time_t pg_time_t_12 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(12); PG_INTERFACE_CONTROL(12); PG_INTERFACE_CONTROL_CRC_OFFSET(12); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(12); PG_INTERFACE_CONTROL_VERSION(12); PG_INTERFACE_WAL_IS(12); PG_INTERFACE_WAL(12); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 11 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_11 #define CheckPoint CheckPoint_11 #define ControlFileData ControlFileData_11 #define DBState DBState_11 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_11 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_11 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_11 #define DB_SHUTDOWNED DB_SHUTDOWNED_11 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_11 #define DB_SHUTDOWNING DB_SHUTDOWNING_11 #define DB_STARTUP DB_STARTUP_11 #define FullTransactionId FullTransactionId_11 #define MultiXactId MultiXactId_11 #define MultiXactOffset MultiXactOffset_11 #define Oid Oid_11 #define TimeLineID TimeLineID_11 #define XLogLongPageHeaderData XLogLongPageHeaderData_11 #define XLogPageHeaderData XLogPageHeaderData_11 #define XLogRecPtr XLogRecPtr_11 #define int64 int64_11 #define pg_crc32c pg_crc32c_11 #define pg_time_t pg_time_t_11 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(11); PG_INTERFACE_CONTROL(11); PG_INTERFACE_CONTROL_CRC_OFFSET(11); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(11); PG_INTERFACE_CONTROL_VERSION(11); PG_INTERFACE_WAL_IS(11); PG_INTERFACE_WAL(11); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 10 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_10 #define CheckPoint CheckPoint_10 #define ControlFileData ControlFileData_10 #define DBState DBState_10 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_10 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_10 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_10 #define DB_SHUTDOWNED DB_SHUTDOWNED_10 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_10 #define DB_SHUTDOWNING DB_SHUTDOWNING_10 #define DB_STARTUP DB_STARTUP_10 #define FullTransactionId FullTransactionId_10 #define MultiXactId MultiXactId_10 #define MultiXactOffset MultiXactOffset_10 #define Oid Oid_10 #define TimeLineID TimeLineID_10 #define XLogLongPageHeaderData XLogLongPageHeaderData_10 #define XLogPageHeaderData XLogPageHeaderData_10 #define XLogRecPtr XLogRecPtr_10 #define int64 int64_10 #define pg_crc32c pg_crc32c_10 #define pg_time_t pg_time_t_10 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(10); PG_INTERFACE_CONTROL(10); PG_INTERFACE_CONTROL_CRC_OFFSET(10); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(10); PG_INTERFACE_CONTROL_VERSION(10); PG_INTERFACE_WAL_IS(10); PG_INTERFACE_WAL(10); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL 9.6 interface ***********************************************************************************************************************************/ #define PG_VERSION PG_VERSION_96 #define CheckPoint CheckPoint_96 #define ControlFileData ControlFileData_96 #define DBState DBState_96 #define DB_IN_ARCHIVE_RECOVERY DB_IN_ARCHIVE_RECOVERY_96 #define DB_IN_CRASH_RECOVERY DB_IN_CRASH_RECOVERY_96 #define DB_IN_PRODUCTION DB_IN_PRODUCTION_96 #define DB_SHUTDOWNED DB_SHUTDOWNED_96 #define DB_SHUTDOWNED_IN_RECOVERY DB_SHUTDOWNED_IN_RECOVERY_96 #define DB_SHUTDOWNING DB_SHUTDOWNING_96 #define DB_STARTUP DB_STARTUP_96 #define FullTransactionId FullTransactionId_96 #define MultiXactId MultiXactId_96 #define MultiXactOffset MultiXactOffset_96 #define Oid Oid_96 #define TimeLineID TimeLineID_96 #define XLogLongPageHeaderData XLogLongPageHeaderData_96 #define XLogPageHeaderData XLogPageHeaderData_96 #define XLogRecPtr XLogRecPtr_96 #define int64 int64_96 #define pg_crc32c pg_crc32c_96 #define pg_time_t pg_time_t_96 #include "postgres/interface/version.intern.h" PG_INTERFACE_CONTROL_IS(96); PG_INTERFACE_CONTROL(96); PG_INTERFACE_CONTROL_CRC_OFFSET(96); PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE(96); PG_INTERFACE_CONTROL_VERSION(96); PG_INTERFACE_WAL_IS(96); PG_INTERFACE_WAL(96); #undef CheckPoint #undef ControlFileData #undef DBState #undef DB_IN_ARCHIVE_RECOVERY #undef DB_IN_CRASH_RECOVERY #undef DB_IN_PRODUCTION #undef DB_SHUTDOWNED #undef DB_SHUTDOWNED_IN_RECOVERY #undef DB_SHUTDOWNING #undef DB_STARTUP #undef FullTransactionId #undef MultiXactId #undef MultiXactOffset #undef Oid #undef TimeLineID #undef XLogLongPageHeaderData #undef XLogPageHeaderData #undef XLogRecPtr #undef int64 #undef pg_crc32c #undef pg_time_t #undef CATALOG_VERSION_NO #undef CATALOG_VERSION_NO_MAX #undef FLOATFORMAT_VALUE #undef MOCK_AUTH_NONCE_LEN #undef PG_CONTROL_VERSION #undef PG_VERSION #undef XLOG_PAGE_MAGIC #undef XLP_LONG_HEADER #undef PG_INTERFACE_CONTROL_IS #undef PG_INTERFACE_CONTROL #undef PG_INTERFACE_CONTROL_CRC_OFFSET #undef PG_INTERFACE_CONTROL_CHECKPOINT_INVALIDATE #undef PG_INTERFACE_CONTROL_VERSION #undef PG_INTERFACE_WAL_IS #undef PG_INTERFACE_WAL /*********************************************************************************************************************************** PostgreSQL interface struct ***********************************************************************************************************************************/ static const PgInterface pgInterface[] = { { .version = PG_VERSION_19, .controlIs = pgInterfaceControlIs19, .control = pgInterfaceControl19, .controlCrcOffset = pgInterfaceControlCrcOffset19, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate19, .controlVersion = pgInterfaceControlVersion19, .walIs = pgInterfaceWalIs19, .wal = pgInterfaceWal19, }, { .version = PG_VERSION_18, .controlIs = pgInterfaceControlIs18, .control = pgInterfaceControl18, .controlCrcOffset = pgInterfaceControlCrcOffset18, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate18, .controlVersion = pgInterfaceControlVersion18, .walIs = pgInterfaceWalIs18, .wal = pgInterfaceWal18, }, { .version = PG_VERSION_17, .controlIs = pgInterfaceControlIs17, .control = pgInterfaceControl17, .controlCrcOffset = pgInterfaceControlCrcOffset17, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate17, .controlVersion = pgInterfaceControlVersion17, .walIs = pgInterfaceWalIs17, .wal = pgInterfaceWal17, }, { .version = PG_VERSION_16, .controlIs = pgInterfaceControlIs16, .control = pgInterfaceControl16, .controlCrcOffset = pgInterfaceControlCrcOffset16, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate16, .controlVersion = pgInterfaceControlVersion16, .walIs = pgInterfaceWalIs16, .wal = pgInterfaceWal16, }, { .version = PG_VERSION_15, .controlIs = pgInterfaceControlIs15, .control = pgInterfaceControl15, .controlCrcOffset = pgInterfaceControlCrcOffset15, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate15, .controlVersion = pgInterfaceControlVersion15, .walIs = pgInterfaceWalIs15, .wal = pgInterfaceWal15, }, { .version = PG_VERSION_14, .controlIs = pgInterfaceControlIs14, .control = pgInterfaceControl14, .controlCrcOffset = pgInterfaceControlCrcOffset14, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate14, .controlVersion = pgInterfaceControlVersion14, .walIs = pgInterfaceWalIs14, .wal = pgInterfaceWal14, }, { .version = PG_VERSION_13, .controlIs = pgInterfaceControlIs13, .control = pgInterfaceControl13, .controlCrcOffset = pgInterfaceControlCrcOffset13, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate13, .controlVersion = pgInterfaceControlVersion13, .walIs = pgInterfaceWalIs13, .wal = pgInterfaceWal13, }, { .version = PG_VERSION_12, .controlIs = pgInterfaceControlIs12, .control = pgInterfaceControl12, .controlCrcOffset = pgInterfaceControlCrcOffset12, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate12, .controlVersion = pgInterfaceControlVersion12, .walIs = pgInterfaceWalIs12, .wal = pgInterfaceWal12, }, { .version = PG_VERSION_11, .controlIs = pgInterfaceControlIs11, .control = pgInterfaceControl11, .controlCrcOffset = pgInterfaceControlCrcOffset11, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate11, .controlVersion = pgInterfaceControlVersion11, .walIs = pgInterfaceWalIs11, .wal = pgInterfaceWal11, }, { .version = PG_VERSION_10, .controlIs = pgInterfaceControlIs10, .control = pgInterfaceControl10, .controlCrcOffset = pgInterfaceControlCrcOffset10, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate10, .controlVersion = pgInterfaceControlVersion10, .walIs = pgInterfaceWalIs10, .wal = pgInterfaceWal10, }, { .version = PG_VERSION_96, .controlIs = pgInterfaceControlIs96, .control = pgInterfaceControl96, .controlCrcOffset = pgInterfaceControlCrcOffset96, .controlCheckpointInvalidate = pgInterfaceControlCheckpointInvalidate96, .controlVersion = pgInterfaceControlVersion96, .walIs = pgInterfaceWalIs96, .wal = pgInterfaceWal96, }, }; ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/interface.c0000664000175100017510000007263415240547630020170 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Interface ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/time.h" #include "postgres/interface.h" #include "postgres/interface/crc32.h" #include "postgres/interface/static.vendor.h" #include "postgres/version.h" #include "storage/helper.h" /*********************************************************************************************************************************** Defines for various Postgres paths and files ***********************************************************************************************************************************/ STRING_EXTERN(PG_FILE_PGVERSION_STR, PG_FILE_PGVERSION); STRING_EXTERN(PG_FILE_POSTGRESQLAUTOCONF_STR, PG_FILE_POSTGRESQLAUTOCONF); STRING_EXTERN(PG_FILE_POSTMTRPID_STR, PG_FILE_POSTMTRPID); STRING_EXTERN(PG_FILE_RECOVERYCONF_STR, PG_FILE_RECOVERYCONF); STRING_EXTERN(PG_FILE_RECOVERYDONE_STR, PG_FILE_RECOVERYDONE); STRING_EXTERN(PG_FILE_RECOVERYSIGNAL_STR, PG_FILE_RECOVERYSIGNAL); STRING_EXTERN(PG_FILE_STANDBYSIGNAL_STR, PG_FILE_STANDBYSIGNAL); STRING_EXTERN(PG_PATH_GLOBAL_STR, PG_PATH_GLOBAL); STRING_EXTERN(PG_NAME_WAL_STR, PG_NAME_WAL); STRING_EXTERN(PG_NAME_XLOG_STR, PG_NAME_XLOG); // Wal path names depending on version STRING_STATIC(PG_PATH_PGWAL_STR, "pg_wal"); STRING_STATIC(PG_PATH_PGXLOG_STR, "pg_xlog"); // Transaction commit log path names depending on version STRING_STATIC(PG_PATH_PGCLOG_STR, "pg_clog"); STRING_STATIC(PG_PATH_PGXACT_STR, "pg_xact"); // Lsn name used in functions depending on version STRING_STATIC(PG_NAME_LSN_STR, "lsn"); STRING_STATIC(PG_NAME_LOCATION_STR, "location"); /*********************************************************************************************************************************** The control file is 8192 bytes but only the first 512 bytes are used to prevent torn pages even on really old storage with 512-byte sectors. This is true across all versions of PostgreSQL. Unfortunately, this is not sufficient to prevent torn pages during concurrent reads and writes so retries are required. ***********************************************************************************************************************************/ #define PG_CONTROL_SIZE 8192 #define PG_CONTROL_DATA_SIZE 512 /*********************************************************************************************************************************** PostgreSQL interface definitions The interface functions are documented here rather than the auto-generated file. ***********************************************************************************************************************************/ typedef struct PgInterface { // Version of PostgreSQL supported by this interface unsigned int version; // Does pg_control match this version of PostgreSQL? bool (*controlIs)(const uint8_t *); // Convert pg_control to a common data structure PgControl (*control)(const uint8_t *); // Get control crc offset size_t (*controlCrcOffset)(void); // Invalidate control checkpoint void (*controlCheckpointInvalidate)(uint8_t *); // Get the control version for this version of PostgreSQL uint32_t (*controlVersion)(void); // Does the WAL header match this version of PostgreSQL? bool (*walIs)(const uint8_t *); // Convert WAL header to a common data structure PgWal (*wal)(const uint8_t *); } PgInterface; // Include auto-generated interfaces #include "postgres/interface.auto.c.inc" /*********************************************************************************************************************************** These pg_control fields are common to all versions of PostgreSQL, so we can use them to generate error messages when the pg_control version cannot be found. ***********************************************************************************************************************************/ typedef struct PgControlCommon { uint64_t systemId; uint32_t controlVersion; uint32_t catalogVersion; } PgControlCommon; /*********************************************************************************************************************************** Get the interface for a PostgreSQL version ***********************************************************************************************************************************/ static const PgInterface * pgInterfaceVersion(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); const PgInterface *result = NULL; for (unsigned int interfaceIdx = 0; interfaceIdx < LENGTH_OF(pgInterface); interfaceIdx++) { if (pgInterface[interfaceIdx].version == pgVersion) { result = &pgInterface[interfaceIdx]; break; } } // If the version was not found then error if (result == NULL) { THROW_FMT( VersionNotSupportedError, "invalid " PG_NAME " version %s\n" "HINT: is this version of " PG_NAME " supported?\n" "HINT: is " PROJECT_NAME " up to date on all hosts?", strZ(pgVersionToStr(pgVersion))); } FUNCTION_TEST_RETURN_TYPE_CONST_P(PgInterface, result); } /**********************************************************************************************************************************/ FN_EXTERN bool pgDbIsTemplate(const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, strEqZ(name, "template0") || strEqZ(name, "template1")); } /**********************************************************************************************************************************/ FN_EXTERN bool pgDbIsSystem(const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, strEqZ(name, "postgres") || pgDbIsTemplate(name)); } /**********************************************************************************************************************************/ FN_EXTERN bool pgDbIsSystemId(const unsigned int id) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, id); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, id < FirstNormalObjectId); } /*********************************************************************************************************************************** Check expected WAL segment size for older PostgreSQL versions ***********************************************************************************************************************************/ static void pgWalSegmentSizeCheck(const unsigned int walSegmentSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, walSegmentSize); FUNCTION_TEST_END(); // Check that the WAL segment size is valid if (!IsValidWalSegSize(walSegmentSize)) { THROW_FMT( FormatError, "wal segment size is %u but must be a power of two between %d and %d inclusive", walSegmentSize, WalSegMinSize, WalSegMaxSize); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Validate the CRC in pg_control. If the version has been forced this may required scanning pg_control for the offset. Return the offset found so it can be used to update the CRC. ***********************************************************************************************************************************/ static size_t pgControlCrcValidate(const Buffer *const controlFile, const PgInterface *const interface, const bool pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, controlFile); FUNCTION_LOG_PARAM_P(VOID, interface); FUNCTION_LOG_PARAM(BOOL, pgVersionForce); FUNCTION_LOG_END(); ASSERT(controlFile != NULL); ASSERT(interface != NULL); size_t result = interface->controlCrcOffset(); do { // Calculate CRC and retrieve expected CRC const uint32_t crcCalculated = crc32cOne(bufPtrConst(controlFile), result); const uint32_t crcExpected = *((const uint32_t *)(bufPtrConst(controlFile) + result)); // If CRC does not match if (crcCalculated != crcExpected) { // If version is forced then the CRC might be later in the file (assuming the fork added extra fields to pg_control). // Increment the offset by CRC data size and continue to try again. if (pgVersionForce) { result += sizeof(uint32_t); if (result <= bufUsed(controlFile) - sizeof(uint32_t)) continue; } // If no retry then error THROW_FMT( ChecksumError, "calculated " PG_FILE_PGCONTROL " checksum does not match expected value\n" "HINT: calculated 0x%x but expected value is 0x%x\n" "%s" "HINT: is " PG_FILE_PGCONTROL " corrupt?\n" "HINT: does " PG_FILE_PGCONTROL " have a different layout than expected?", crcCalculated, crcExpected, pgVersionForce ? "HINT: checksum values may be misleading due to forced version scan\n" : ""); } // Do not retry if the CRC is valid break; } while (true); FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Update the CRC in pg_control ***********************************************************************************************************************************/ static void pgControlCrcUpdate(Buffer *const controlFile, const unsigned int pgVersion, const size_t crcOffset) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, controlFile); FUNCTION_LOG_PARAM(UINT, pgVersion); FUNCTION_LOG_PARAM(SIZE, crcOffset); FUNCTION_LOG_END(); ASSERT(controlFile != NULL); ASSERT(pgVersion != 0); ASSERT(crcOffset != 0); *((uint32_t *)(bufPtr(controlFile) + crcOffset)) = crc32cOne(bufPtrConst(controlFile), crcOffset); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static PgControl pgControlFromBuffer(const Buffer *const controlFile, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, controlFile); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(controlFile != NULL); // Search for the version of PostgreSQL that uses this control file const PgInterface *interface = NULL; if (pgVersionForce != NULL) interface = pgInterfaceVersion(pgVersionFromStr(pgVersionForce)); else { for (unsigned int interfaceIdx = 0; interfaceIdx < LENGTH_OF(pgInterface); interfaceIdx++) { if (pgInterface[interfaceIdx].controlIs(bufPtrConst(controlFile))) { interface = &pgInterface[interfaceIdx]; break; } } // If the version was not found then error with the control and catalog version that were found if (interface == NULL) { const PgControlCommon *controlCommon = (const PgControlCommon *)bufPtrConst(controlFile); THROW_FMT( VersionNotSupportedError, "unexpected control version = %u and catalog version = %u\n" "HINT: is this version of " PG_NAME " supported?\n" "HINT: is " PROJECT_NAME " up to date on all hosts?", controlCommon->controlVersion, controlCommon->catalogVersion); } } // Get info from the control file ASSERT(pgVersionForce != NULL || interface->controlIs(bufPtrConst(controlFile))); PgControl result = interface->control(bufPtrConst(controlFile)); result.version = interface->version; // Validate the CRC pgControlCrcValidate(controlFile, interface, pgVersionForce != NULL); // Check the segment size pgWalSegmentSizeCheck(result.walSegmentSize); // Check the page size pgPageSizeCheck(result.pageSize); // Check the checksum version if ((result.version <= PG_VERSION_18 && result.pageChecksumVersion > PG_DATA_CHECKSUM_VERSION) || (result.version > PG_VERSION_18 && result.pageChecksumVersion > PG_DATA_CHECKSUM_INPROGRESS_ON)) { THROW_FMT( FormatError, "page checksum version is %u but must be <= %d", result.pageChecksumVersion, result.version <= PG_VERSION_18 ? PG_DATA_CHECKSUM_VERSION : PG_DATA_CHECKSUM_INPROGRESS_ON); } // Clear checksum version values that represent clusters with incomplete checksums if (result.version > PG_VERSION_18 && result.pageChecksumVersion > PG_DATA_CHECKSUM_VERSION) result.pageChecksumVersion = 0; FUNCTION_LOG_RETURN(PG_CONTROL, result); } // Helper to compare control data to last read static bool pgControlBufferEq(const Buffer *const last, const Buffer *const current) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, last); FUNCTION_TEST_PARAM(BUFFER, current); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(BOOL, last != NULL && bufEq(last, current)); } FN_EXTERN Buffer * pgControlBufferFromFile(const Storage *const storage, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(storage != NULL); Buffer *result; MEM_CONTEXT_TEMP_BEGIN() { // On filesystems that do not implement atomicity of concurrent reads and writes, we might get garbage if the server is // writing the pg_control file at the same time as we try to read it. Keep trying until success or we read the same data // twice in a row. Do not use a timeout here because there are plenty of other errors that can happen and we don't want to // wait for them. Buffer *controlFileLast = NULL; bool done = false; do { // Read control file Buffer *const controlFile = storageGetP( storageNewReadP(storage, STRDEF(PG_PATH_GLOBAL "/" PG_FILE_PGCONTROL)), .exactSize = PG_CONTROL_DATA_SIZE); TRY_BEGIN() { // Check that control data is valid pgControlFromBuffer(controlFile, pgVersionForce); // Create a buffer of the correct size to hold pg_control and zero out the remaining portion result = bufNew(PG_CONTROL_SIZE); bufCat(result, controlFile); memset(bufPtr(result) + bufUsed(controlFile), 0, bufSize(result) - bufUsed(controlFile)); bufUsedSet(result, bufSize(result)); bufMove(result, memContextPrior()); done = true; } CATCH_ANY() { // If we get the same bad data twice in a row then error if (pgControlBufferEq(controlFileLast, controlFile)) RETHROW(); // Copy data to last bufFree(controlFileLast); controlFileLast = controlFile; // Sleep to let data stabilize sleepMSec(50); } TRY_END(); } while (!done); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BUFFER, result); } FN_EXTERN PgControl pgControlFromFile(const Storage *const storage, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(storage != NULL); Buffer *const buffer = pgControlBufferFromFile(storage, pgVersionForce); const PgControl result = pgControlFromBuffer(buffer, pgVersionForce); bufFree(buffer); FUNCTION_LOG_RETURN(PG_CONTROL, result); } /**********************************************************************************************************************************/ FN_EXTERN void pgControlCheckpointInvalidate(Buffer *const buffer, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(buffer != NULL); const PgControl pgControl = pgControlFromBuffer(buffer, pgVersionForce); const PgInterface *const interface = pgInterfaceVersion(pgControl.version); const size_t crcOffset = pgControlCrcValidate(buffer, interface, pgVersionForce); pgInterfaceVersion(pgControl.version)->controlCheckpointInvalidate(bufPtr(buffer)); pgControlCrcUpdate(buffer, interface->version, crcOffset); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN uint32_t pgControlVersion(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(UINT32, pgInterfaceVersion(pgVersion)->controlVersion()); } /*********************************************************************************************************************************** These WAL header fields are common to all versions of PostgreSQL, so we can use them to generate error messages when the WAL magic cannot be found. ***********************************************************************************************************************************/ typedef struct PgWalCommon { uint16_t magic; uint16_t flag; } PgWalCommon; #define PG_WAL_LONG_HEADER 0x0002 /**********************************************************************************************************************************/ FN_EXTERN PgWal pgWalFromBuffer(const Buffer *const walBuffer, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(BUFFER, walBuffer); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(walBuffer != NULL); // Check that this is a long format WAL header if (!(((const PgWalCommon *)bufPtrConst(walBuffer))->flag & PG_WAL_LONG_HEADER)) THROW_FMT(FormatError, "first page header in WAL file is expected to be in long format"); // Search for the version of PostgreSQL that uses this WAL magic const PgInterface *interface = NULL; if (pgVersionForce != NULL) interface = pgInterfaceVersion(pgVersionFromStr(pgVersionForce)); else { for (unsigned int interfaceIdx = 0; interfaceIdx < LENGTH_OF(pgInterface); interfaceIdx++) { if (pgInterface[interfaceIdx].walIs(bufPtrConst(walBuffer))) { interface = &pgInterface[interfaceIdx]; break; } } // If the version was not found then error with the magic that was found if (interface == NULL) { THROW_FMT( VersionNotSupportedError, "unexpected WAL magic %u\n" "HINT: is this version of " PG_NAME " supported?\n" "HINT: is " PROJECT_NAME " up to date on all hosts?", ((const PgWalCommon *)bufPtrConst(walBuffer))->magic); } } // Get info from the control file ASSERT(pgVersionForce != NULL || interface->walIs(bufPtrConst(walBuffer))); PgWal result = interface->wal(bufPtrConst(walBuffer)); result.version = interface->version; // Check the segment size pgWalSegmentSizeCheck(result.size); FUNCTION_LOG_RETURN(PG_WAL, result); } FN_EXTERN PgWal pgWalFromFile(const String *const walFile, const Storage *const storage, const String *const pgVersionForce) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, walFile); FUNCTION_LOG_PARAM(STRING, pgVersionForce); FUNCTION_LOG_END(); ASSERT(walFile != NULL); PgWal result; MEM_CONTEXT_TEMP_BEGIN() { // Read WAL segment header const Buffer *const walBuffer = storageGetP(storageNewReadP(storage, walFile), .exactSize = PG_WAL_HEADER_SIZE); result = pgWalFromBuffer(walBuffer, pgVersionForce); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PG_WAL, result); } /**********************************************************************************************************************************/ FN_EXTERN String * pgTablespaceId(const unsigned int pgVersion, const unsigned int pgCatalogVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_PARAM(UINT, pgCatalogVersion); FUNCTION_TEST_END(); String *result; MEM_CONTEXT_TEMP_BEGIN() { const String *const pgVersionStr = pgVersionToStr(pgVersion); MEM_CONTEXT_PRIOR_BEGIN() { result = strNewFmt("PG_%s_%u", strZ(pgVersionStr), pgCatalogVersion); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN uint64_t pgLsnFromStr(const String *lsn) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, lsn); FUNCTION_TEST_END(); uint64_t result = 0; MEM_CONTEXT_TEMP_BEGIN() { const StringList *const lsnPart = strLstNewSplit(lsn, FSLASH_STR); CHECK(FormatError, strLstSize(lsnPart) == 2, "lsn requires two parts"); result = (cvtZToUInt64Base(strZ(strLstGet(lsnPart, 0)), 16) << 32) + cvtZToUInt64Base(strZ(strLstGet(lsnPart, 1)), 16); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(UINT64, result); } FN_EXTERN String * pgLsnToStr(const uint64_t lsn) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, lsn); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN(STRING, strNewFmt("%x/%x", (unsigned int)(lsn >> 32), (unsigned int)(lsn & 0xFFFFFFFF))); } /**********************************************************************************************************************************/ FN_EXTERN String * pgLsnToWalSegment(const uint32_t timeline, const uint64_t lsn, const unsigned int walSegmentSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, timeline); FUNCTION_TEST_PARAM(UINT64, lsn); FUNCTION_TEST_PARAM(UINT, walSegmentSize); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( STRING, strNewFmt("%08X%08X%08X", timeline, (unsigned int)(lsn >> 32), (unsigned int)(lsn & 0xFFFFFFFF) / walSegmentSize)); } /**********************************************************************************************************************************/ FN_EXTERN uint32_t pgTimelineFromWalSegment(const String *const walSegment) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, walSegment); FUNCTION_TEST_END(); ASSERT(walSegment != NULL); ASSERT(strSize(walSegment) == 24); FUNCTION_TEST_RETURN(UINT32, cvtZSubNToUIntBase(strZ(walSegment), 0, 8, 16)); } /**********************************************************************************************************************************/ FN_EXTERN StringList * pgLsnRangeToWalSegmentList( const uint32_t timeline, const uint64_t lsnStart, const uint64_t lsnStop, const unsigned int walSegmentSize) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, timeline); FUNCTION_TEST_PARAM(UINT64, lsnStart); FUNCTION_TEST_PARAM(UINT64, lsnStop); FUNCTION_TEST_PARAM(UINT, walSegmentSize); FUNCTION_TEST_END(); ASSERT(timeline != 0); ASSERT(lsnStart <= lsnStop); ASSERT(walSegmentSize != 0); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Calculate the start and stop segments unsigned int startMajor = (unsigned int)(lsnStart >> 32); unsigned int startMinor = (unsigned int)(lsnStart & 0xFFFFFFFF) / walSegmentSize; const unsigned int stopMajor = (unsigned int)(lsnStop >> 32); const unsigned int stopMinor = (unsigned int)(lsnStop & 0xFFFFFFFF) / walSegmentSize; const unsigned int minorPerMajor = 0xFFFFFFFF / walSegmentSize; // Create list strLstAddFmt(result, "%08X%08X%08X", timeline, startMajor, startMinor); while (!(startMajor == stopMajor && startMinor == stopMinor)) { startMinor++; if (startMinor > minorPerMajor) { startMajor++; startMinor = 0; } strLstAddFmt(result, "%08X%08X%08X", timeline, startMajor, startMinor); } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN const String * pgLsnName(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, pgVersion >= PG_VERSION_WAL_RENAME ? PG_NAME_LSN_STR : PG_NAME_LOCATION_STR); } /*********************************************************************************************************************************** Get WAL name (wal/xlog) for a PostgreSQL version ***********************************************************************************************************************************/ FN_EXTERN const String * pgWalName(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, pgVersion >= PG_VERSION_WAL_RENAME ? PG_NAME_WAL_STR : PG_NAME_XLOG_STR); } /**********************************************************************************************************************************/ FN_EXTERN const String * pgWalPath(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, pgVersion >= PG_VERSION_WAL_RENAME ? PG_PATH_PGWAL_STR : PG_PATH_PGXLOG_STR); } /**********************************************************************************************************************************/ FN_EXTERN const String * pgXactPath(const unsigned int pgVersion) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgVersion); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN_CONST(STRING, pgVersion >= PG_VERSION_WAL_RENAME ? PG_PATH_PGXACT_STR : PG_PATH_PGCLOG_STR); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int pgVersionFromStr(const String *const version) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, version); FUNCTION_TEST_END(); ASSERT(version != NULL); // If format not number.number (9.6) or number only (10) then error. No check for valid/supported PG version is on purpose. if (!regExpMatchOne(STRDEF("^[0-9]+[.]*[0-9]+$"), version)) THROW_FMT(AssertError, "version %s format is invalid", strZ(version)); // If there is no dot then only the major version is needed const int idxStart = strChr(version, '.'); if (idxStart == -1) FUNCTION_TEST_RETURN(UINT, cvtZToUInt(strZ(version)) * 10000); // Major and minor version are needed FUNCTION_TEST_RETURN( UINT, cvtZSubNToUInt(strZ(version), 0, (size_t)idxStart) * 10000 + cvtZToUInt(strZ(version) + (size_t)idxStart + 1) * 100); } FN_EXTERN String * pgVersionToStr(const unsigned int version) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, version); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( STRING, version >= PG_VERSION_10 ? strNewFmt("%u", version / 10000) : strNewFmt("%u.%u", version / 10000, version % 10000 / 100)); } /**********************************************************************************************************************************/ FN_EXTERN void pgControlToLog(const PgControl *const pgControl, StringStatic *const debugLog) { strStcFmt( debugLog, "{version: %u, systemId: %" PRIu64 ", walSegmentSize: %u, pageChecksumVersion: %u}", pgControl->version, pgControl->systemId, pgControl->walSegmentSize, pgControl->pageChecksumVersion); } FN_EXTERN void pgWalToLog(const PgWal *const pgWal, StringStatic *const debugLog) { strStcFmt(debugLog, "{version: %u, systemId: %" PRIu64 "}", pgWal->version, pgWal->systemId); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/interface.h0000664000175100017510000002645515240547630020175 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Interface ***********************************************************************************************************************************/ #ifndef POSTGRES_INTERFACE_H #define POSTGRES_INTERFACE_H #include #include #include "common/debug.h" #include "common/type/string.h" #include "storage/storage.h" /*********************************************************************************************************************************** Defines for various Postgres paths and files ***********************************************************************************************************************************/ #define PG_FILE_BACKUPLABEL "backup_label" #define PG_FILE_BACKUPLABELOLD "backup_label.old" #define PG_FILE_BACKUPMANIFEST "backup_manifest" #define PG_FILE_BACKUPMANIFEST_TMP "backup_manifest.tmp" #define PG_FILE_PGCONTROL "pg_control" #define PG_FILE_PGFILENODEMAP "pg_filenode.map" #define PG_FILE_PGINTERNALINIT "pg_internal.init" #define PG_FILE_PGVERSION "PG_VERSION" STRING_DECLARE(PG_FILE_PGVERSION_STR); #define PG_FILE_POSTGRESQLAUTOCONF "postgresql.auto.conf" STRING_DECLARE(PG_FILE_POSTGRESQLAUTOCONF_STR); #define PG_FILE_POSTGRESQLAUTOCONFTMP "postgresql.auto.conf.tmp" #define PG_FILE_POSTMTROPTS "postmas""ter.opts" #define PG_FILE_POSTMTRPID "postmas""ter.pid" STRING_DECLARE(PG_FILE_POSTMTRPID_STR); #define PG_FILE_RECOVERYCONF "recovery.conf" STRING_DECLARE(PG_FILE_RECOVERYCONF_STR); #define PG_FILE_RECOVERYDONE "recovery.done" STRING_DECLARE(PG_FILE_RECOVERYDONE_STR); #define PG_FILE_RECOVERYSIGNAL "recovery.signal" STRING_DECLARE(PG_FILE_RECOVERYSIGNAL_STR); #define PG_FILE_STANDBYSIGNAL "standby.signal" STRING_DECLARE(PG_FILE_STANDBYSIGNAL_STR); #define PG_FILE_TABLESPACEMAP "tablespace_map" #define PG_PATH_ARCHIVE_STATUS "archive_status" #define PG_PATH_BASE "base" #define PG_PATH_GLOBAL "global" STRING_DECLARE(PG_PATH_GLOBAL_STR); #define PG_PATH_PGMULTIXACT "pg_multixact" #define PG_PATH_PGDYNSHMEM "pg_dynshmem" #define PG_PATH_PGNOTIFY "pg_notify" #define PG_PATH_PGREPLSLOT "pg_replslot" #define PG_PATH_PGSERIAL "pg_serial" #define PG_PATH_PGSNAPSHOTS "pg_snapshots" #define PG_PATH_PGSTATTMP "pg_stat_tmp" #define PG_PATH_PGSUBTRANS "pg_subtrans" #define PG_PATH_PGTBLSPC "pg_tblspc" #define PG_PREFIX_PGSQLTMP "pgsql_tmp" #define PG_NAME_WAL "wal" STRING_DECLARE(PG_NAME_WAL_STR); #define PG_NAME_XLOG "xlog" STRING_DECLARE(PG_NAME_XLOG_STR); /*********************************************************************************************************************************** Define allowed page sizes ***********************************************************************************************************************************/ typedef enum { pgPageSize1 = 1 * 1024, pgPageSize2 = 2 * 1024, pgPageSize4 = 4 * 1024, pgPageSize8 = 8 * 1024, pgPageSize16 = 16 * 1024, pgPageSize32 = 32 * 1024, } PgPageSize; /*********************************************************************************************************************************** Define default segment size and pages per segment Segment size can only be changed at compile time and is not known to be well-tested, so only the default segment size is supported. ***********************************************************************************************************************************/ #define PG_SEGMENT_SIZE_DEFAULT ((unsigned int)(1 * 1024 * 1024 * 1024)) /*********************************************************************************************************************************** WAL header size. It doesn't seem worth tracking the exact size of the WAL header across versions of PostgreSQL so just set it to something far larger needed but <= the minimum read size on just about any system. ***********************************************************************************************************************************/ #define PG_WAL_HEADER_SIZE ((unsigned int)(512)) /*********************************************************************************************************************************** Checkpoint written into pg_control on restore. This will prevent PostgreSQL from starting if backup_label is not present. ***********************************************************************************************************************************/ #define PG_CONTROL_CHECKPOINT_INVALID 0xDEAD /*********************************************************************************************************************************** PostgreSQL Control File Info ***********************************************************************************************************************************/ typedef struct PgControl { unsigned int version; uint64_t systemId; unsigned int catalogVersion; uint64_t checkpoint; // Last checkpoint LSN uint32_t timeline; // Current timeline PgPageSize pageSize; unsigned int walSegmentSize; unsigned int pageChecksumVersion; // Page checksum version (0 if no checksum, 1 if checksum) } PgControl; /*********************************************************************************************************************************** PostgreSQL WAL Info ***********************************************************************************************************************************/ typedef struct PgWal { unsigned int version; unsigned int size; uint64_t systemId; } PgWal; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Is this a template database? FN_EXTERN bool pgDbIsTemplate(const String *name); // Is this a system database, i.e. template or postgres? FN_EXTERN bool pgDbIsSystem(const String *name); // Does this database have a system id, i.e. less than the minimum assignable user id? FN_EXTERN bool pgDbIsSystemId(unsigned int id); // Get info from pg_control FN_EXTERN Buffer *pgControlBufferFromFile(const Storage *storage, const String *pgVersionForce); FN_EXTERN PgControl pgControlFromFile(const Storage *storage, const String *pgVersionForce); // Invalidate checkpoint record in pg_control FN_EXTERN void pgControlCheckpointInvalidate(Buffer *buffer, const String *pgVersionForce); // Get the control version for a PostgreSQL version FN_EXTERN uint32_t pgControlVersion(unsigned int pgVersion); // Convert version string to version number and vice versa FN_EXTERN unsigned int pgVersionFromStr(const String *version); FN_EXTERN String *pgVersionToStr(unsigned int version); // Get info from WAL header FN_EXTERN PgWal pgWalFromFile(const String *walFile, const Storage *storage, const String *pgVersionForce); FN_EXTERN PgWal pgWalFromBuffer(const Buffer *walBuffer, const String *pgVersionForce); // Get the tablespace identifier used to distinguish versions in a tablespace directory, e.g. PG_15_202209061 FN_EXTERN String *pgTablespaceId(unsigned int pgVersion, unsigned int pgCatalogVersion); // Convert a string to an lsn and vice versa FN_EXTERN uint64_t pgLsnFromStr(const String *lsn); FN_EXTERN String *pgLsnToStr(uint64_t lsn); // Convert a timeline and lsn to a wal segment and vice versa FN_EXTERN String *pgLsnToWalSegment(uint32_t timeline, uint64_t lsn, unsigned int walSegmentSize); // Get timeline from WAL segment name FN_EXTERN uint32_t pgTimelineFromWalSegment(const String *walSegment); // Convert a timeline and lsn range to a list of wal segments FN_EXTERN StringList *pgLsnRangeToWalSegmentList( uint32_t timeline, uint64_t lsnStart, uint64_t lsnStop, unsigned int walSegmentSize); // Get name used for lsn in functions (this was changed in PostgreSQL 10 for consistency since lots of names were changing) FN_EXTERN const String *pgLsnName(unsigned int pgVersion); // Calculate the checksum for a page. Page cannot be const because the page header is temporarily modified during processing. FN_EXTERN uint16_t pgPageChecksum(uint8_t *page, uint32_t blockNo, PgPageSize pageSize); // Returns true if page size is valid, false otherwise FN_EXTERN bool pgPageSizeValid(PgPageSize pageSize); // Throws an error if page size is not valid FN_EXTERN void pgPageSizeCheck(PgPageSize pageSize); FN_EXTERN const String *pgWalName(unsigned int pgVersion); // Get wal path (this was changed in PostgreSQL 10 to avoid including "log" in the name) FN_EXTERN const String *pgWalPath(unsigned int pgVersion); // Get transaction commit log path (this was changed in PostgreSQL 10 to avoid including "log" in the name) FN_EXTERN const String *pgXactPath(unsigned int pgVersion); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void pgControlToLog(const PgControl *pgControl, StringStatic *debugLog); FN_EXTERN void pgWalToLog(const PgWal *pgWal, StringStatic *debugLog); #define FUNCTION_LOG_PG_CONTROL_TYPE \ PgControl #define FUNCTION_LOG_PG_CONTROL_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(&value, pgControlToLog, buffer, bufferSize) #define FUNCTION_LOG_PG_WAL_TYPE \ PgWal #define FUNCTION_LOG_PG_WAL_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(&value, pgWalToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/meson.build0000664000175100017510000000156415240547630020220 0ustar00runnerrunner#################################################################################################################################### # Generate PostgreSQL interfaces #################################################################################################################################### # Generate interface.auto.c.inc, unless this is a distribution tarball that ships the file if not dist interface_auto_c_inc = custom_target( 'interface.auto.c.inc', output: 'interface.auto.c.inc', depend_files: [ '../build/postgres/postgres.yaml', ], command: [ build_code, 'postgres', '@CURRENT_SOURCE_DIR@/../..', ], ) else # The shipped file is included by interface.c via the implicit source include path, so it is not a compiled source interface_auto_c_inc = [] endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/version.auto.h0000664000175100017510000000533215240547630020660 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Version Automatically generated by 'build-code postgres-version' -- do not modify directly. ***********************************************************************************************************************************/ #ifndef POSTGRES_VERSION_AUTO_H #define POSTGRES_VERSION_AUTO_H /*********************************************************************************************************************************** PostgreSQL version constants ***********************************************************************************************************************************/ #define PG_VERSION_96 90600 #define PG_VERSION_10 100000 #define PG_VERSION_11 110000 #define PG_VERSION_12 120000 #define PG_VERSION_13 130000 #define PG_VERSION_14 140000 #define PG_VERSION_15 150000 #define PG_VERSION_16 160000 #define PG_VERSION_17 170000 #define PG_VERSION_18 180000 #define PG_VERSION_19 190000 #define PG_VERSION_MAX PG_VERSION_19 /*********************************************************************************************************************************** PostgreSQL version string constants for use in error messages ***********************************************************************************************************************************/ #define PG_VERSION_96_Z "9.6" #define PG_VERSION_10_Z "10" #define PG_VERSION_11_Z "11" #define PG_VERSION_12_Z "12" #define PG_VERSION_13_Z "13" #define PG_VERSION_14_Z "14" #define PG_VERSION_15_Z "15" #define PG_VERSION_16_Z "16" #define PG_VERSION_17_Z "17" #define PG_VERSION_18_Z "18" #define PG_VERSION_19_Z "19" #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/postgres/version.h0000664000175100017510000000251615240547630017712 0ustar00runnerrunner/*********************************************************************************************************************************** PostgreSQL Version Constants ***********************************************************************************************************************************/ #ifndef POSTGRES_VERSION_H #define POSTGRES_VERSION_H #include "postgres/version.auto.h" /*********************************************************************************************************************************** PostgreSQL name ***********************************************************************************************************************************/ #define PG_NAME "PostgreSQL" /*********************************************************************************************************************************** Version where various PostgreSQL capabilities were introduced ***********************************************************************************************************************************/ // xlog was renamed to wal #define PG_VERSION_WAL_RENAME PG_VERSION_10 // recovery settings are implemented as GUCs (recovery.conf is no longer valid) #define PG_VERSION_RECOVERY_GUC PG_VERSION_12 #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/0000775000175100017510000000000015240547630016043 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/client.c0000664000175100017510000005766115240547630017504 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Client ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/time.h" #include "common/type/json.h" #include "common/type/keyValue.h" #include "protocol/client.h" #include "protocol/server.h" #include "version.h" /*********************************************************************************************************************************** Client state enum ***********************************************************************************************************************************/ typedef enum { // Client is waiting for a response / ready to send a request protocolClientStateIdle = STRID5("idle", 0x2b0890), // Request is in progress protocolClientStateRequest = STRID5("request", 0x5265ac4b20), // Response is in progress protocolClientStateResponse = STRID5("response", 0x2cdcf84cb20), } ProtocolClientState; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct ProtocolClient { ProtocolClientPub pub; // Publicly accessible variables ProtocolClientState state; // Current client state IoWrite *write; // Write interface const String *name; // Name displayed in logging const String *errorPrefix; // Prefix used when throwing error TimeMSec keepAliveTime; // Last time data was put to the server uint64_t sessionTotal; // Total sessions (used to generate session ids) List *sessionList; // Session list }; struct ProtocolClientSession { ProtocolClientSessionPub pub; // Publicly accessible variables ProtocolClient *client; // Protocol client StringId command; // Command uint64_t sessionId; // Session id bool async; // Async requests allowed? // Stored message (read by another session and stored for later retrieval) bool stored; // Is a message currently stored? bool close; // Should the session be closed? ProtocolMessageType type; // Type of last message received PackRead *packRead; // Last message received (if any) }; /*********************************************************************************************************************************** Check protocol state ***********************************************************************************************************************************/ static void protocolClientStateExpectIdle(const ProtocolClient *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_CLIENT, this); FUNCTION_TEST_END(); if (this->state != protocolClientStateIdle) { THROW_FMT( ProtocolError, "client state is '%s' but expected '%s'", zNewStrId(this->state), zNewStrId(protocolClientStateIdle)); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Find a client session ***********************************************************************************************************************************/ static unsigned int protocolClientSessionFindIdx(const ProtocolClient *const this, const uint64_t sessionId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_CLIENT, this); FUNCTION_TEST_PARAM(UINT64, sessionId); FUNCTION_TEST_END(); unsigned int result = 0; for (; result < lstSize(this->sessionList); result++) { if ((*(ProtocolClientSession **)lstGet(this->sessionList, result))->sessionId == sessionId) break; } CHECK_FMT(ProtocolError, result < lstSize(this->sessionList), "unable to find protocol client session %" PRIu64, sessionId); FUNCTION_TEST_RETURN(UINT, result); } /**********************************************************************************************************************************/ static void protocolClientRequestInternal(ProtocolClientSession *const this, const ProtocolCommandType type, PackWrite *const param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(PACK_WRITE, param); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!protocolClientSessionQueued(this)); // Expect idle state before request protocolClientStateExpectIdle(this->client); MEM_CONTEXT_TEMP_BEGIN() { // Switch state to request this->client->state = protocolClientStateRequest; // Write request PackWrite *const request = pckWriteNewIo(this->client->write); pckWriteU32P(request, protocolMessageTypeRequest, .defaultWrite = true); pckWriteStrIdP(request, this->command); pckWriteStrIdP(request, type); pckWriteU64P(request, this->sessionId); pckWriteBoolP(request, this->pub.open); // Write request parameters if (param != NULL) { pckWriteEndP(param); pckWritePackP(request, pckWriteResult(param)); } pckWriteEndP(request); // Flush to send request immediately ioWriteFlush(this->client->write); this->pub.queued = true; // Switch state back to idle after successful request this->client->state = protocolClientStateIdle; } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Process errors ***********************************************************************************************************************************/ static void protocolClientError(ProtocolClient *const this, const ProtocolMessageType type, PackRead *const error) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, this); FUNCTION_LOG_PARAM(ENUM, type); FUNCTION_LOG_PARAM(PACK_READ, error); FUNCTION_LOG_END(); ASSERT(this != NULL); if (type == protocolMessageTypeError) { ASSERT(error != NULL); MEM_CONTEXT_TEMP_BEGIN() { const ErrorType *const type = errorTypeFromCode(pckReadI32P(error)); const String *const message = strNewFmt("%s: %s", strZ(this->errorPrefix), strZ(pckReadStrP(error))); const String *const stack = pckReadStrP(error); pckReadEndP(error); CHECK(FormatError, message != NULL && stack != NULL, "invalid error data"); errorInternalThrow(type, __FILE__, __func__, __LINE__, strZ(message), strZ(stack)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static PackRead * protocolClientResponseInternal(ProtocolClientSession *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Expect data-get state before data get protocolClientStateExpectIdle(this->client); PackRead *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Check the session for a stored response bool found = false; ProtocolMessageType type = protocolMessageTypeResponse; bool close = false; PackRead *packRead = NULL; if (this->stored) { found = true; type = this->type; close = this->close; packRead = pckReadMove(this->packRead, memContextCurrent()); this->stored = false; } // If not stored then read from protocol while (!found) { // Switch state to response this->client->state = protocolClientStateResponse; // Read response PackRead *const response = pckReadNewIo(this->client->pub.read); const uint64_t sessionId = pckReadU64P(response); type = (ProtocolMessageType)pckReadU32P(response); close = pckReadBoolP(response); packRead = pckReadPackReadP(response); pckReadEndP(response); // If this response is for another session then store it with that session. Session id 0 indicates a fatal error on the // server that should be reported by the first session that sees it. ASSERT(sessionId != 0 || type == protocolMessageTypeError); if (sessionId != 0 && sessionId != this->sessionId) { ProtocolClientSession *const sessionOther = *(ProtocolClientSession **)lstGet( this->client->sessionList, protocolClientSessionFindIdx(this->client, sessionId)); CHECK_FMT( ProtocolError, !sessionOther->stored, "protocol client session %" PRIu64 " already has a stored response", sessionOther->sessionId); sessionOther->stored = true; sessionOther->type = type; sessionOther->close = close; sessionOther->packRead = objMove(packRead, objMemContext(sessionOther)); } else found = true; // Switch state back to idle after successful response this->client->state = protocolClientStateIdle; } // The result is no longer queued -- either it will generate an error or be returned to the user this->pub.queued = false; // Close the session if requested if (close) this->pub.open = false; // Check if this result is an error and if so throw it protocolClientError(this->client, type, packRead); CHECK(FormatError, type == protocolMessageTypeResponse, "expected response message"); // Return result to the caller result = objMove(packRead, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK_READ, result); } /*********************************************************************************************************************************** Close protocol connection ***********************************************************************************************************************************/ static void protocolClientFreeResource(THIS_VOID) { THIS(ProtocolClient); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Stop client sessions from sending cancel requests for (unsigned int sessionIdx = 0; sessionIdx < lstSize(this->sessionList); sessionIdx++) memContextCallbackClear(objMemContext(*(ProtocolClientSession **)lstGet(this->sessionList, sessionIdx))); // Send an exit request but don't wait to see if it succeeds MEM_CONTEXT_TEMP_BEGIN() { // Switch state to idle so the request is sent no matter the current state this->state = protocolClientStateIdle; // Send exit request protocolClientRequestInternal(protocolClientSessionNewP(this, PROTOCOL_COMMAND_EXIT), protocolCommandTypeProcess, NULL); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolClient * protocolClientNew(const String *const name, const String *const service, IoRead *const read, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(STRING, service); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(name != NULL); ASSERT(read != NULL); ASSERT(write != NULL); OBJ_NEW_BEGIN(ProtocolClient, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (ProtocolClient) { .pub = { .read = read, }, .state = protocolClientStateIdle, .write = write, .name = strDup(name), .errorPrefix = strNewFmt("raised from %s", strZ(name)), .keepAliveTime = timeMSec(), .sessionList = lstNewP(sizeof(ProtocolClientSession)), }; // Read, parse, and check the protocol greeting MEM_CONTEXT_TEMP_BEGIN() { JsonRead *const greeting = jsonReadNew(ioReadLine(this->pub.read)); jsonReadObjectBegin(greeting); const struct { const StringId key; const char *const value; } expected[] = { {.key = PROTOCOL_GREETING_NAME, .value = PROJECT_NAME}, {.key = PROTOCOL_GREETING_SERVICE, .value = strZ(service)}, {.key = PROTOCOL_GREETING_VERSION, .value = PROJECT_VERSION}, }; for (unsigned int expectedIdx = 0; expectedIdx < LENGTH_OF(expected); expectedIdx++) { if (!jsonReadKeyExpectStrId(greeting, expected[expectedIdx].key)) THROW_FMT(ProtocolError, "unable to find greeting key '%s'", zNewStrId(expected[expectedIdx].key)); if (jsonReadTypeNext(greeting) != jsonTypeString) THROW_FMT(ProtocolError, "greeting key '%s' must be string type", zNewStrId(expected[expectedIdx].key)); const String *const actualValue = jsonReadStr(greeting); if (!strEqZ(actualValue, expected[expectedIdx].value)) { THROW_FMT( ProtocolError, "expected value '%s' for greeting key '%s' but got '%s'\n" "HINT: is the same version of " PROJECT_NAME " installed on the local and remote host?", expected[expectedIdx].value, zNewStrId(expected[expectedIdx].key), strZ(actualValue)); } } jsonReadObjectEnd(greeting); } MEM_CONTEXT_TEMP_END(); // Set a callback to shutdown the protocol memContextCallbackSet(objMemContext(this), protocolClientFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, this); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientRequest(ProtocolClient *const this, const StringId command, const ProtocolClientRequestParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, this); FUNCTION_LOG_PARAM(STRING_ID, command); FUNCTION_LOG_PARAM(PACK_WRITE, param.param); FUNCTION_LOG_END(); ASSERT(this != NULL); PackRead *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { ProtocolClientSession *const session = protocolClientSessionNewP(this, command); protocolClientRequestInternal(session, protocolCommandTypeProcess, param.param); MEM_CONTEXT_PRIOR_BEGIN() { result = protocolClientResponseInternal(session); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PACK_READ, result); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientNoOp(ProtocolClient *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, this); FUNCTION_LOG_END(); ASSERT(this != NULL); protocolClientRequestP(this, PROTOCOL_COMMAND_NOOP); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Free client session ***********************************************************************************************************************************/ static void protocolClientSessionFreeResource(THIS_VOID) { THIS(ProtocolClientSession); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If a result is queued then read it before sending cancel if (protocolClientSessionQueued(this)) { // Free stored result if (this->stored) { pckReadFree(this->packRead); } // Else read and free result else pckReadFree(protocolClientResponseInternal(this)); } // If open then cancel if (this->pub.open) { protocolClientRequestInternal(this, protocolCommandTypeCancel, NULL); protocolClientResponseInternal(this); ASSERT(!this->pub.open); } // Remove from session list lstRemoveIdx(this->client->sessionList, protocolClientSessionFindIdx(this->client, this->sessionId)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolClientSession * protocolClientSessionNew(ProtocolClient *const client, const StringId command, const ProtocolClientSessionNewParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_PARAM(STRING_ID, command); FUNCTION_LOG_PARAM(BOOL, param.async); FUNCTION_LOG_END(); ASSERT(client != NULL); OBJ_NEW_BEGIN(ProtocolClientSession, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (ProtocolClientSession) { .client = client, .command = command, .sessionId = ++client->sessionTotal, .async = param.async, }; } OBJ_NEW_END(); // If async then the session will need to be tracked in case another session gets a result for this session if (this->async) { lstAdd(this->client->sessionList, &this); // Set a callback to cleanup the session memContextCallbackSet(objMemContext(this), protocolClientSessionFreeResource, this); } FUNCTION_LOG_RETURN(PROTOCOL_CLIENT_SESSION, this); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientSessionOpen(ProtocolClientSession *const this, const ProtocolClientSessionOpenParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_PARAM(PACK_WRITE, param.param); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!this->pub.open); protocolClientRequestInternal(this, protocolCommandTypeOpen, param.param); this->pub.open = true; // Set a callback to cleanup the session if it was not already done for async if (!this->async) { lstAdd(this->client->sessionList, &this); memContextCallbackSet(objMemContext(this), protocolClientSessionFreeResource, this); } FUNCTION_LOG_RETURN(PACK_READ, protocolClientResponseInternal(this)); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientSessionRequest(ProtocolClientSession *const this, const ProtocolClientRequestParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_PARAM(PACK_WRITE, param.param); FUNCTION_LOG_END(); ASSERT(this != NULL); protocolClientRequestInternal(this, protocolCommandTypeProcess, param.param); FUNCTION_LOG_RETURN(PACK_READ, protocolClientResponseInternal(this)); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientSessionRequestAsync(ProtocolClientSession *const this, const ProtocolClientRequestParam param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_PARAM(PACK_WRITE, param.param); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->async); protocolClientRequestInternal(this, protocolCommandTypeProcess, param.param); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientSessionResponse(ProtocolClientSession *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->async); FUNCTION_LOG_RETURN(PACK_READ, protocolClientResponseInternal(this)); } /**********************************************************************************************************************************/ FN_EXTERN PackRead * protocolClientSessionClose(ProtocolClientSession *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.open); protocolClientRequestInternal(this, protocolCommandTypeClose, NULL); PackRead *const result = protocolClientResponseInternal(this); ASSERT(!this->pub.open); memContextCallbackClear(objMemContext(this)); protocolClientSessionFreeResource(this); FUNCTION_LOG_RETURN(PACK_READ, result); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientSessionCancel(ProtocolClientSession *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT_SESSION, this); FUNCTION_LOG_END(); ASSERT(this != NULL); memContextCallbackClear(objMemContext(this)); protocolClientSessionFreeResource(this); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientToLog(const ProtocolClient *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{name: %s, state: ", strZ(this->name)); strStcResultSizeInc(debugLog, strIdToLog(this->state, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCatChr(debugLog, '}'); } /**********************************************************************************************************************************/ FN_EXTERN void protocolClientSessionToLog(const ProtocolClientSession *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{client: "); protocolClientToLog(this->client, debugLog); strStcCat(debugLog, ", command: "); strStcResultSizeInc(debugLog, strIdToLog(this->command, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt( debugLog, ", sessionId: %" PRIu64 ", queued %s, stored: %s", this->sessionId, cvtBoolToConstZ(protocolClientSessionQueued(this)), cvtBoolToConstZ(this->stored)); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/client.h0000664000175100017510000003164215240547630017500 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Client There are three ways to make a request using the client: 1. Synchronous This is the most common way to make a request. Simply call protocolClientRequestP() and process the result. 2. Synchronous with Session In some cases it is useful to keep session state between requests. An example of this is queries against a database where it is more efficient to maintain the connection to the database between queries. The session is created with protocolClientSessionNewP(), opened with protocolClientSessionOpenP(), and closed with protocolClientSessionClose(). Requests are made with protocolClientSessionRequest(). 3. Asynchronous with Session Asynchronous requests provide better performance by allowing the client and server to process at the same time. An example of this is reading a file where a asynchronous request for the next block of data can be sent before the current block is being processed. The next block should be waiting when processing of the current block is complete. The same session functions are used for creation, open, and close, except that .async = true is passed to protocolClientSessionNewP(). Asynchronous requests are made with protocolClientSessionRequestAsync() and the responses are read with protocolClientSessionResponse(). Note that there can only be one outstanding asynchronous request, therefore protocolClientSessionResponse() must be called before protocolClientSessionRequestAsync() can be called again. ***********************************************************************************************************************************/ #ifndef PROTOCOL_CLIENT_H #define PROTOCOL_CLIENT_H /*********************************************************************************************************************************** Message types used by the protocol ***********************************************************************************************************************************/ typedef enum { // Data passed from server to client protocolMessageTypeResponse = 0, // Request sent from the client to the server protocolMessageTypeRequest = 1, // An error occurred on the server and the request ended abnormally. protocolMessageTypeResponse will not be sent to the client. protocolMessageTypeError = 2, } ProtocolMessageType; /*********************************************************************************************************************************** Command types ***********************************************************************************************************************************/ typedef enum { protocolCommandTypeOpen = STRID5("opn", 0x3a0f0), // Open command for processing protocolCommandTypeProcess = STRID5("prc", 0xe500), // Process command protocolCommandTypeClose = STRID5("cls", 0x4d830), // Close command protocolCommandTypeCancel = STRID5("cnc", 0xdc30), // Cancel command } ProtocolCommandType; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolClient ProtocolClient; typedef struct ProtocolClientSession ProtocolClientSession; #include "common/io/read.h" #include "common/io/write.h" #include "common/type/object.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define PROTOCOL_GREETING_NAME STRID5("name", 0x2b42e0) #define PROTOCOL_GREETING_SERVICE STRID5("service", 0x1469b48b30) #define PROTOCOL_GREETING_VERSION STRID5("version", 0x39e99c8b60) #define PROTOCOL_COMMAND_CONFIG STRID5("config", 0xe9339e30) #define PROTOCOL_COMMAND_EXIT STRID5("exit", 0xa27050) #define PROTOCOL_COMMAND_NOOP STRID5("noop", 0x83dee0) /*********************************************************************************************************************************** This size should be safe for most pack data without wasting a lot of space. If binary data is being transferred then this size can be added to the expected binary size to account for overhead. ***********************************************************************************************************************************/ #define PROTOCOL_PACK_DEFAULT_SIZE 1024 // Pack large enough for standard data. Note that the buffer will automatically resize when required. FN_INLINE_ALWAYS PackWrite * protocolPackNew(void) { return pckWriteNewP(.size = PROTOCOL_PACK_DEFAULT_SIZE); } /*********************************************************************************************************************************** Client Constructors ***********************************************************************************************************************************/ FN_EXTERN ProtocolClient *protocolClientNew(const String *name, const String *service, IoRead *read, IoWrite *write); /*********************************************************************************************************************************** Client Getters/Setters ***********************************************************************************************************************************/ typedef struct ProtocolClientPub { IoRead *read; // Read interface } ProtocolClientPub; // Read file descriptor FN_INLINE_ALWAYS int protocolClientIoReadFd(ProtocolClient *const this) { return ioReadFd(THIS_PUB(ProtocolClient)->read); } /*********************************************************************************************************************************** Client Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS ProtocolClient * protocolClientMove(ProtocolClient *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Do not send exit command to the server when the client is freed FN_INLINE_ALWAYS void protocolClientNoExit(ProtocolClient *const this) { memContextCallbackClear(objMemContext(this)); } // Send noop to test connection or keep it alive FN_EXTERN void protocolClientNoOp(ProtocolClient *this); // Simple request that does not require a session or async typedef struct ProtocolClientRequestParam { VAR_PARAM_HEADER; PackWrite *param; } ProtocolClientRequestParam; #define protocolClientRequestP(this, command, ...) \ protocolClientRequest(this, command, (ProtocolClientRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *protocolClientRequest(ProtocolClient *this, StringId command, ProtocolClientRequestParam param); /*********************************************************************************************************************************** Client Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolClientFree(ProtocolClient *const this) { objFree(this); } /*********************************************************************************************************************************** Session Constructors ***********************************************************************************************************************************/ // New session typedef struct ProtocolClientSessionNewParam { VAR_PARAM_HEADER; bool async; // Async requests allowed? } ProtocolClientSessionNewParam; #define protocolClientSessionNewP(client, command, ...) \ protocolClientSessionNew(client, command, (ProtocolClientSessionNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN ProtocolClientSession *protocolClientSessionNew( ProtocolClient *client, StringId command, ProtocolClientSessionNewParam param); /*********************************************************************************************************************************** Session Getters/Setters ***********************************************************************************************************************************/ typedef struct ProtocolClientSessionPub { bool open; // Is the session open? bool queued; // Is a response currently queued? } ProtocolClientSessionPub; // Is a response currently queued? FN_INLINE_ALWAYS bool protocolClientSessionQueued(const ProtocolClientSession *const this) { return THIS_PUB(ProtocolClientSession)->queued; } // Is the session closed? FN_INLINE_ALWAYS bool protocolClientSessionClosed(const ProtocolClientSession *const this) { return !THIS_PUB(ProtocolClientSession)->open; } /*********************************************************************************************************************************** Session Functions ***********************************************************************************************************************************/ // Session open typedef struct ProtocolClientSessionOpenParam { VAR_PARAM_HEADER; PackWrite *param; } ProtocolClientSessionOpenParam; #define protocolClientSessionOpenP(this, ...) \ protocolClientSessionOpen(this, (ProtocolClientSessionOpenParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *protocolClientSessionOpen(ProtocolClientSession *const this, ProtocolClientSessionOpenParam param); // Session request #define protocolClientSessionRequestP(this, ...) \ protocolClientSessionRequest(this, (ProtocolClientRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN PackRead *protocolClientSessionRequest(ProtocolClientSession *const this, ProtocolClientRequestParam param); // Session async request #define protocolClientSessionRequestAsyncP(this, ...) \ protocolClientSessionRequestAsync(this, (ProtocolClientRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void protocolClientSessionRequestAsync(ProtocolClientSession *const this, ProtocolClientRequestParam param); // Session response after a call to protocolClientSessionRequestAsyncP() FN_EXTERN PackRead *protocolClientSessionResponse(ProtocolClientSession *const this); // Session close FN_EXTERN PackRead *protocolClientSessionClose(ProtocolClientSession *const this); // Session cancel FN_EXTERN void protocolClientSessionCancel(ProtocolClientSession *const this); /*********************************************************************************************************************************** Session Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolClientSessionFree(ProtocolClientSession *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void protocolClientToLog(const ProtocolClient *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_CLIENT_TYPE \ ProtocolClient * #define FUNCTION_LOG_PROTOCOL_CLIENT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolClientToLog, buffer, bufferSize) FN_EXTERN void protocolClientSessionToLog(const ProtocolClientSession *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_CLIENT_SESSION_TYPE \ ProtocolClientSession * #define FUNCTION_LOG_PROTOCOL_CLIENT_SESSION_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolClientSessionToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/helper.c0000664000175100017510000010747115240547630017500 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Helper ***********************************************************************************************************************************/ #include #include #include "command/lock.h" #include "common/crypto/common.h" #include "common/debug.h" #include "common/exec.h" #include "common/io/client.h" #include "common/io/socket/client.h" #include "common/io/socket/server.h" #include "common/io/tls/client.h" #include "common/io/tls/server.h" #include "common/memContext.h" #include "config/config.intern.h" #include "config/exec.h" #include "config/load.h" #include "config/parse.h" #include "config/protocol.h" #include "postgres/version.h" #include "protocol/helper.h" #include "version.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ STRING_EXTERN(PROTOCOL_SERVICE_LOCAL_STR, PROTOCOL_SERVICE_LOCAL); STRING_EXTERN(PROTOCOL_SERVICE_REMOTE_STR, PROTOCOL_SERVICE_REMOTE); /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ typedef enum { protocolClientLocal = STRID5("local", 0xc08dec0), protocolClientRemote = STRID5("remote", 0xb47b4b20), } ProtocolClientType; typedef struct ProtocolHelperClient { ProtocolClientType type; // Client type ProtocolStorageType storageType; // Storage type unsigned int hostIdx; // Host index unsigned int processId; // Process id displayed in logs Exec *exec; // Executed client IoClient *ioClient; // Io client, e.g. TlsClient IoSession *ioSession; // Io session, e.g. TlsSession ProtocolClient *client; // Protocol client } ProtocolHelperClient; static struct { MemContext *memContext; // Mem context for protocol helper List *clientList; // Client List } protocolHelper; /*********************************************************************************************************************************** Init local mem context and data structure ***********************************************************************************************************************************/ static void protocolHelperInit(void) { // In the protocol helper has not been initialized if (protocolHelper.memContext == NULL) { // Create a mem context to store protocol objects MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN("ProtocolHelper", .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { protocolHelper.memContext = MEM_CONTEXT_NEW(); protocolHelper.clientList = lstNewP(sizeof(ProtocolHelperClient)); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } } /*********************************************************************************************************************************** Init local mem context and data structure ***********************************************************************************************************************************/ static ProtocolHelperClient * protocolHelperClientGet(ProtocolClientType type, ProtocolStorageType storageType, unsigned int hostIdx, unsigned int processId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING_ID, type); FUNCTION_TEST_PARAM(STRING_ID, storageType); FUNCTION_TEST_PARAM(UINT, hostIdx); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_END(); ASSERT(protocolHelper.clientList != NULL); for (unsigned int clientIdx = 0; clientIdx < lstSize(protocolHelper.clientList); clientIdx++) { ProtocolHelperClient *const match = lstGet(protocolHelper.clientList, clientIdx); if (match->type == type && match->storageType == storageType && match->hostIdx == hostIdx && match->processId == processId) FUNCTION_TEST_RETURN_P(VOID, match); } FUNCTION_TEST_RETURN_P(VOID, NULL); } /**********************************************************************************************************************************/ FN_EXTERN bool repoIsLocal(const unsigned int repoIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(BOOL, !cfgOptionIdxTest(cfgOptRepoHost, repoIdx)); } /**********************************************************************************************************************************/ FN_EXTERN bool pgIsLocal(const unsigned int pgIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, pgIdx); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN(BOOL, !cfgOptionIdxTest(cfgOptPgHost, pgIdx)); } /**********************************************************************************************************************************/ FN_EXTERN void pgIsLocalVerify(void) { FUNCTION_TEST_VOID(); if (!pgIsLocal(cfgOptionGroupIdxDefault(cfgOptGrpPg))) THROW_FMT(HostInvalidError, "%s command must be run on the " PG_NAME " host", cfgCommandName()); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Get the command line required for local protocol execution ***********************************************************************************************************************************/ static StringList * protocolLocalParam(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const unsigned int processId) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_PARAM(UINT, processId); FUNCTION_LOG_END(); StringList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Option replacements KeyValue *const optionReplace = kvNew(); // Add the process id -- used when more than one process will be called kvPut(optionReplace, VARSTRDEF(CFGOPT_PROCESS), VARUINT(processId)); // Add the pg default. Don't do this for repos because the repo default should come from the user or the local should // handle all the repos equally. Repos don't get special handling like pg primaries or standbys. if (protocolStorageType == protocolStorageTypePg) kvPut(optionReplace, VARSTRDEF(CFGOPT_PG), VARUINT(cfgOptionGroupIdxToKey(cfgOptGrpPg, hostIdx))); // Add the remote type kvPut(optionReplace, VARSTRDEF(CFGOPT_REMOTE_TYPE), VARSTR(strNewStrId(protocolStorageType))); // Only enable file logging on the local when requested kvPut( optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_FILE), VARUINT64( cfgOptionBool(cfgOptLogSubprocess) ? cfgOptionStrId(cfgOptLogLevelFile) : CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID)); // Always output errors on stderr for debugging purposes kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARUINT64(CFGOPTVAL_LOG_LEVEL_STDERR_ERROR_STRID)); // Disable output to stdout since it is used by the protocol kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARUINT64(CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID)); MEM_CONTEXT_PRIOR_BEGIN() { result = cfgExecParam(cfgCommand(), cfgCmdRoleLocal, optionReplace, true, false); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ // Helper to execute the local process. This is a separate function solely so that it can be shimmed during testing. static void protocolLocalExec( ProtocolHelperClient *const helper, const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const unsigned int processId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, helper); FUNCTION_TEST_PARAM(ENUM, protocolStorageType); FUNCTION_TEST_PARAM(UINT, hostIdx); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(helper != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Execute the protocol command const String *name = strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u process", processId); const StringList *const param = protocolLocalParam(protocolStorageType, hostIdx, processId); MEM_CONTEXT_PRIOR_BEGIN() { helper->exec = execNew(cfgBin(), param, name, cfgOptionUInt64(cfgOptProtocolTimeout)); } MEM_CONTEXT_PRIOR_END(); execOpen(helper->exec); // Create protocol object name = strNewFmt(PROTOCOL_SERVICE_LOCAL "-%u protocol", processId); MEM_CONTEXT_PRIOR_BEGIN() { helper->client = protocolClientNew( name, PROTOCOL_SERVICE_LOCAL_STR, execIoRead(helper->exec), execIoWrite(helper->exec)); } MEM_CONTEXT_PRIOR_END(); // Move client to exec context so they are freed together protocolClientMove(helper->client, execMemContext(helper->exec)); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ProtocolClient * protocolLocalGet(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const unsigned int processId) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_PARAM(UINT, processId); FUNCTION_LOG_END(); protocolHelperInit(); // Create protocol object ProtocolHelperClient *protocolHelperClient = protocolHelperClientGet( protocolClientLocal, protocolStorageType, hostIdx, processId); if (protocolHelperClient == NULL) { MEM_CONTEXT_BEGIN(protocolHelper.memContext) { ProtocolHelperClient protocolHelperClientAdd = { .type = protocolClientLocal, .storageType = protocolStorageType, .hostIdx = hostIdx, .processId = processId, }; protocolLocalExec(&protocolHelperClientAdd, protocolStorageType, hostIdx, processId); protocolHelperClient = lstAdd(protocolHelper.clientList, &protocolHelperClientAdd); } MEM_CONTEXT_END(); // Send noop to catch initialization errors protocolClientNoOp(protocolHelperClient->client); } FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient->client); } /*********************************************************************************************************************************** Free the protocol client and underlying exec'd process. Log any errors as warnings since it is not worth terminating the process while closing a local/remote that has already completed its work. The warning will be an indication that something is not right. ***********************************************************************************************************************************/ static void protocolHelperClientFree(ProtocolHelperClient *const protocolHelperClient) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM_P(VOID, protocolHelperClient); FUNCTION_LOG_END(); // Try to shutdown the protocol but only warn on error TRY_BEGIN() { protocolClientFree(protocolHelperClient->client); } CATCH_ANY() { LOG_WARN(errorMessage()); } TRY_END(); // Try to end the child process but only warn on error TRY_BEGIN() { execFree(protocolHelperClient->exec); } CATCH_ANY() { LOG_WARN(errorMessage()); } TRY_END(); // Free the io client/session (there should be no errors) ioSessionFree(protocolHelperClient->ioSession); ioClientFree(protocolHelperClient->ioClient); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper to check if client is authorized for a stanza static bool protocolServerAuthorize(const String *authListStr, const String *const stanza) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, authListStr); FUNCTION_TEST_PARAM(STRING, stanza); FUNCTION_TEST_END(); ASSERT(authListStr != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // Empty list is not valid. ??? It would be better if this were done during config parsing. authListStr = strTrim(strDup(authListStr)); if (strEmpty(authListStr)) THROW(OptionInvalidValueError, "'" CFGOPT_TLS_SERVER_AUTH "' option must have a value"); // If * then all stanzas are authorized if (strEqZ(authListStr, "*")) { result = true; } // Else check the stanza list for a match with the specified stanza. Each entry will need to be trimmed before comparing. else if (stanza != NULL) { const StringList *const authList = strLstNewSplitZ(authListStr, ","); for (unsigned int authListIdx = 0; authListIdx < strLstSize(authList); authListIdx++) { if (strEq(strTrim(strLstGet(authList, authListIdx)), stanza)) { result = true; break; } } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(BOOL, result); } FN_EXTERN ProtocolServer * protocolServer(IoServer *const tlsServer, IoSession *const socketSession) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(IO_SERVER, tlsServer); FUNCTION_LOG_PARAM(IO_SESSION, socketSession); FUNCTION_LOG_END(); ProtocolServer *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Start TLS IoSession *const tlsSession = ioServerAccept(tlsServer, socketSession); result = protocolServerNew( PROTOCOL_SERVICE_REMOTE_STR, PROTOCOL_SERVICE_REMOTE_STR, ioSessionIoReadP(tlsSession), ioSessionIoWrite(tlsSession)); // If session is authenticated if (ioSessionAuthenticated(tlsSession)) { TRY_BEGIN() { // Get parameter list from the client. This needs to happen even if we cannot authorize the client so that the // session id gets set for the error. const ProtocolServerRequestResult command = protocolServerRequest(result); CHECK(FormatError, command.id == PROTOCOL_COMMAND_CONFIG, "expected config command"); // Get list of authorized stanzas for this client CHECK(AssertError, cfgOptionTest(cfgOptTlsServerAuth), "missing auth data"); const String *const clientAuthList = strDup( varStr(kvGet(cfgOptionKv(cfgOptTlsServerAuth), VARSTR(ioSessionPeerName(tlsSession))))); // Error if the client is not authorized for anything if (clientAuthList == NULL) THROW(AccessError, "access denied"); // Load parameter list from the client StringList *const paramList = pckReadStrLstP(pckReadNew(command.param)); strLstInsert(paramList, 0, cfgBin()); cfgLoad(strLstSize(paramList), strLstPtr(paramList)); // Error if the client is not authorized for the requested stanza if (!protocolServerAuthorize(clientAuthList, cfgOptionStrNull(cfgOptStanza))) THROW(AccessError, "access denied"); } CATCH_ANY() { protocolServerError(result, errorCode(), STR(errorMessage()), STR(errorStackTrace())); RETHROW(); } TRY_END(); // Ack the config command protocolServerResponseP(result); // Move result to prior context and move session into result so there is only one return value protocolServerMove(result, memContextPrior()); ioSessionMove(tlsSession, objMemContext(result)); } // Else the client can only detect that the server is alive else { // A noop command should have been received const ProtocolServerRequestResult command = protocolServerRequest(result); CHECK(FormatError, command.id == PROTOCOL_COMMAND_NOOP, "expected config command"); // Send a data end message and return a NULL server protocolServerResponseP(result); // Set result to NULL so there is no server for the caller to use. The TLS session will be freed when the temp mem // context ends. result = NULL; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER, result); } /*********************************************************************************************************************************** Get the command line required for remote protocol execution ***********************************************************************************************************************************/ static StringList * protocolRemoteParam(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_END(); StringList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Is this a repo remote? const bool isRepo = protocolStorageType == protocolStorageTypeRepo; // Option replacements KeyValue *const optionReplace = kvNew(); // Replace config options with the host versions const unsigned int optConfig = isRepo ? cfgOptRepoHostConfig : cfgOptPgHostConfig; kvPut( optionReplace, VARSTRDEF(CFGOPT_CONFIG), cfgOptionIdxSource(optConfig, hostIdx) != cfgSourceDefault ? VARSTR(cfgOptionIdxStr(optConfig, hostIdx)) : NULL); const unsigned int optConfigIncludePath = isRepo ? cfgOptRepoHostConfigIncludePath : cfgOptPgHostConfigIncludePath; kvPut( optionReplace, VARSTRDEF(CFGOPT_CONFIG_INCLUDE_PATH), cfgOptionIdxSource(optConfigIncludePath, hostIdx) != cfgSourceDefault ? VARSTR(cfgOptionIdxStr(optConfigIncludePath, hostIdx)) : NULL); const unsigned int optConfigPath = isRepo ? cfgOptRepoHostConfigPath : cfgOptPgHostConfigPath; kvPut( optionReplace, VARSTRDEF(CFGOPT_CONFIG_PATH), cfgOptionIdxSource(optConfigPath, hostIdx) != cfgSourceDefault ? VARSTR(cfgOptionIdxStr(optConfigPath, hostIdx)) : NULL); // Update/remove repo/pg options that are sent to the remote for (ConfigOption optionId = 0; optionId < CFG_OPTION_TOTAL; optionId++) { // Skip options that are not part of a group if (!cfgOptionGroup(optionId)) continue; bool remove = false; bool skipHostZero = false; // Remove repo options that are not needed on the remote if (cfgOptionGroupId(optionId) == cfgOptGrpRepo) { // If remote type is pg then remove all repo options since they will not be used if (protocolStorageType == protocolStorageTypePg) { remove = true; } // Else remove repo options for indexes that are not being passed to the repo. This prevents the remote from getting // partial info about a repo, which could cause an error during validation. else { for (unsigned int optionIdx = 0; optionIdx < cfgOptionIdxTotal(optionId); optionIdx++) { if (cfgOptionIdxTest(optionId, optionIdx) && optionIdx != hostIdx) kvPut(optionReplace, VARSTRZ(cfgOptionIdxName(optionId, optionIdx)), NULL); } } } // Remove pg options that are not needed on the remote else { ASSERT(cfgOptionGroupId(optionId) == cfgOptGrpPg); // Move pg options to host index 0 (key 1) so they will be in the default index on the remote host if (protocolStorageType != protocolStorageTypeRepo) { if (hostIdx != 0) { kvPut( optionReplace, VARSTRZ(cfgOptionIdxName(optionId, 0)), cfgOptionIdxSource(optionId, hostIdx) != cfgSourceDefault ? cfgOptionIdxVar(optionId, hostIdx) : NULL); } remove = true; skipHostZero = true; } } // Remove options that have been marked for removal if they are not already null or invalid. This is more efficient // because cfgExecParam() won't have to search through as large a list looking for overrides. if (remove) { // Loop through option indexes for (unsigned int optionIdx = 0; optionIdx < cfgOptionIdxTotal(optionId); optionIdx++) { if (cfgOptionIdxTest(optionId, optionIdx) && !(skipHostZero && optionIdx == 0)) kvPut(optionReplace, VARSTRZ(cfgOptionIdxName(optionId, optionIdx)), NULL); } } } // Set repo default so the remote only operates on a single repo if (protocolStorageType == protocolStorageTypeRepo) { kvPut(optionReplace, VARSTRDEF(CFGOPT_REPO), VARUINT(cfgOptionGroupIdxToKey(cfgOptGrpRepo, hostIdx))); } // Else remove repo selection when the remote is pg else kvPut(optionReplace, VARSTRDEF(CFGOPT_REPO), NULL); // Add the process id if not set. This means that the remote is being started from the main process and should always get a // process id of 0. if (!cfgOptionTest(cfgOptProcess)) kvPut(optionReplace, VARSTRDEF(CFGOPT_PROCESS), VARINT(0)); // Don't pass log-path or lock-path since these are host specific kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_PATH), NULL); kvPut(optionReplace, VARSTRDEF(CFGOPT_LOCK_PATH), NULL); // Only enable file logging on the remote when requested kvPut( optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_FILE), VARUINT64( cfgOptionBool(cfgOptLogSubprocess) ? cfgOptionStrId(cfgOptLogLevelFile) : CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID)); // Always output errors on stderr for debugging purposes kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_STDERR), VARUINT64(CFGOPTVAL_LOG_LEVEL_STDERR_ERROR_STRID)); // Disable output to stdout since it is used by the protocol kvPut(optionReplace, VARSTRDEF(CFGOPT_LOG_LEVEL_CONSOLE), VARUINT64(CFGOPTVAL_LOG_LEVEL_FILE_OFF_STRID)); // Add the remote type kvPut(optionReplace, VARSTRDEF(CFGOPT_REMOTE_TYPE), VARSTR(strNewStrId(protocolStorageType))); // Add lock required on the remote if (cfgLockRemoteRequired()) kvPut(optionReplace, VARSTRDEF(CFGOPT_LOCK), varNewVarLst(varLstNewStrLst(cmdLockList()))); MEM_CONTEXT_PRIOR_BEGIN() { result = cfgExecParam(cfgCommand(), cfgCmdRoleRemote, optionReplace, false, true); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } // Helper to add SSH parameters when executing the remote via SSH static StringList * protocolRemoteParamSsh(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_END(); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { // Is this a repo remote? const bool isRepo = protocolStorageType == protocolStorageTypeRepo; // Fixed parameters for ssh command strLstAddZ(result, "-o"); strLstAddZ(result, "LogLevel=error"); strLstAddZ(result, "-o"); strLstAddZ(result, "Compression=no"); strLstAddZ(result, "-o"); strLstAddZ(result, "PasswordAuthentication=no"); // Append port if specified const ConfigOption optHostPort = isRepo ? cfgOptRepoHostPort : cfgOptPgHostPort; if (cfgOptionIdxTest(optHostPort, hostIdx)) { strLstAddZ(result, "-p"); strLstAddFmt(result, "%u", cfgOptionIdxUInt(optHostPort, hostIdx)); } // Append user/host strLstAddFmt( result, "%s@%s", strZ(cfgOptionIdxStr(isRepo ? cfgOptRepoHostUser : cfgOptPgHostUser, hostIdx)), strZ(cfgOptionIdxStr(isRepo ? cfgOptRepoHost : cfgOptPgHost, hostIdx))); // Add remote command and parameters StringList *const paramList = protocolRemoteParam(protocolStorageType, hostIdx); strLstInsert(paramList, 0, cfgOptionIdxStr(isRepo ? cfgOptRepoHostCmd : cfgOptPgHostCmd, hostIdx)); strLstAdd(result, strLstJoin(paramList, " ")); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ // Helper to execute the local process. This is a separate function solely so that it can be shimmed during testing. static void protocolRemoteExec( ProtocolHelperClient *const helper, const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const unsigned int processId) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, helper); FUNCTION_TEST_PARAM(ENUM, protocolStorageType); FUNCTION_TEST_PARAM(UINT, hostIdx); FUNCTION_TEST_PARAM(UINT, processId); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(helper != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Get remote info const bool isRepo = protocolStorageType == protocolStorageTypeRepo; const StringId remoteType = cfgOptionIdxStrId(isRepo ? cfgOptRepoHostType : cfgOptPgHostType, hostIdx); const String *const host = cfgOptionIdxStr(isRepo ? cfgOptRepoHost : cfgOptPgHost, hostIdx); // Handle remote types IoRead *read; IoWrite *write; switch (remoteType) { // SSH remote case CFGOPTVAL_REPO_HOST_TYPE_SSH: { // Exec SSH const String *const name = strNewFmt(PROTOCOL_SERVICE_REMOTE "-%u process on '%s'", processId, strZ(host)); const StringList *const param = protocolRemoteParamSsh(protocolStorageType, hostIdx); MEM_CONTEXT_PRIOR_BEGIN() { helper->exec = execNew(cfgOptionStr(cfgOptCmdSsh), param, name, cfgOptionUInt64(cfgOptProtocolTimeout)); } MEM_CONTEXT_PRIOR_END(); execOpen(helper->exec); read = execIoRead(helper->exec); write = execIoWrite(helper->exec); break; } // TLS remote default: { ASSERT(remoteType == CFGOPTVAL_REPO_HOST_TYPE_TLS); // Negotiate TLS MEM_CONTEXT_PRIOR_BEGIN() { helper->ioClient = tlsClientNewP( sckClientNew( host, cfgOptionIdxUInt(isRepo ? cfgOptRepoHostPort : cfgOptPgHostPort, hostIdx), cfgOptionUInt64(cfgOptIoTimeout), cfgOptionUInt64(cfgOptProtocolTimeout)), host, cfgOptionUInt64(cfgOptIoTimeout), cfgOptionUInt64(cfgOptProtocolTimeout), true, .caFile = cfgOptionIdxStrNull(isRepo ? cfgOptRepoHostCaFile : cfgOptPgHostCaFile, hostIdx), .caPath = cfgOptionIdxStrNull(isRepo ? cfgOptRepoHostCaPath : cfgOptPgHostCaPath, hostIdx), .certFile = cfgOptionIdxStr(isRepo ? cfgOptRepoHostCertFile : cfgOptPgHostCertFile, hostIdx), .keyFile = cfgOptionIdxStr(isRepo ? cfgOptRepoHostKeyFile : cfgOptPgHostKeyFile, hostIdx)); helper->ioSession = ioClientOpen(helper->ioClient); } MEM_CONTEXT_PRIOR_END(); read = ioSessionIoReadP(helper->ioSession); write = ioSessionIoWrite(helper->ioSession); break; } } // Create protocol object const String *const name = strNewFmt( PROTOCOL_SERVICE_REMOTE "-%u %s protocol on '%s'", processId, zNewStrId(remoteType), strZ(host)); MEM_CONTEXT_PRIOR_BEGIN() { helper->client = protocolClientNew(name, PROTOCOL_SERVICE_REMOTE_STR, read, write); } MEM_CONTEXT_PRIOR_END(); // Remote initialization switch (remoteType) { // SSH remote case CFGOPTVAL_REPO_HOST_TYPE_SSH: // Move client to exec context so they are freed together protocolClientMove(helper->client, execMemContext(helper->exec)); break; // TLS remote default: { ASSERT(remoteType == CFGOPTVAL_REPO_HOST_TYPE_TLS); TRY_BEGIN() { // Pass parameters to server PackWrite *const param = protocolPackNew(); pckWriteStrLstP(param, protocolRemoteParam(protocolStorageType, hostIdx)); protocolClientRequestP(helper->client, PROTOCOL_COMMAND_CONFIG, .param = param); } CATCH_ANY() { // Clear the callback so the client does not try to shut down the connection. Attempting to shut down the // connection will fail since the server has already disconnected and a new error will be thrown, masking the // original error. memContextCallbackClear(objMemContext(helper->client)); RETHROW(); } TRY_END(); break; } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ProtocolClient * protocolRemoteGet(const ProtocolStorageType protocolStorageType, const unsigned int hostIdx, const bool create) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, protocolStorageType); FUNCTION_LOG_PARAM(UINT, hostIdx); FUNCTION_LOG_PARAM(BOOL, create); FUNCTION_LOG_END(); // Is this a repo remote? const bool isRepo = protocolStorageType == protocolStorageTypeRepo; protocolHelperInit(); // Determine protocol id for the remote. If the process option is set then use that since we want the remote protocol id to // match the local protocol id. Otherwise set to 0 since the remote is being started from a main process and there should only // be one remote per host. const unsigned int processId = cfgOptionTest(cfgOptProcess) ? cfgOptionUInt(cfgOptProcess) : 0; // Create protocol object ProtocolHelperClient *protocolHelperClient = protocolHelperClientGet( protocolClientRemote, protocolStorageType, hostIdx, processId); if (protocolHelperClient == NULL && create) { MEM_CONTEXT_BEGIN(protocolHelper.memContext) { ProtocolHelperClient protocolHelperClientAdd = { .type = protocolClientRemote, .storageType = protocolStorageType, .hostIdx = hostIdx, .processId = processId, }; protocolRemoteExec(&protocolHelperClientAdd, protocolStorageType, hostIdx, processId); protocolHelperClient = lstAdd(protocolHelper.clientList, &protocolHelperClientAdd); // Send noop to catch initialization errors protocolClientNoOp(protocolHelperClient->client); // Get cipher options from the remote if none are locally configured if (isRepo && cfgOptionIdxStrId(cfgOptRepoCipherType, hostIdx) == cipherTypeNone) { // Options to query VariantList *const param = varLstNew(); varLstAdd(param, varNewStrZ(cfgOptionIdxName(cfgOptRepoCipherType, hostIdx))); varLstAdd(param, varNewStrZ(cfgOptionIdxName(cfgOptRepoCipherPass, hostIdx))); const VariantList *const optionList = configOptionRemote(protocolHelperClient->client, param); if (varUInt64(varLstGet(optionList, 0)) != cipherTypeNone) { cfgOptionIdxSet(cfgOptRepoCipherType, hostIdx, cfgSourceConfig, varLstGet(optionList, 0)); cfgOptionIdxSet(cfgOptRepoCipherPass, hostIdx, cfgSourceConfig, varLstGet(optionList, 1)); } } } MEM_CONTEXT_END(); } FUNCTION_LOG_RETURN(PROTOCOL_CLIENT, protocolHelperClient != NULL ? protocolHelperClient->client : NULL); } /**********************************************************************************************************************************/ FN_EXTERN void protocolHelperFree(ProtocolClient *const client) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_END(); if (client != NULL && protocolHelper.clientList != NULL) { for (unsigned int clientIdx = 0; clientIdx < lstSize(protocolHelper.clientList); clientIdx++) { ProtocolHelperClient *const match = lstGet(protocolHelper.clientList, clientIdx); if (match->client == client) { protocolHelperClientFree(match); lstRemoveIdx(protocolHelper.clientList, clientIdx); break; } } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolKeepAlive(void) { FUNCTION_LOG_VOID(logLevelTrace); if (protocolHelper.clientList) { for (unsigned int clientIdx = 0; clientIdx < lstSize(protocolHelper.clientList); clientIdx++) { ProtocolHelperClient *const match = lstGet(protocolHelper.clientList, clientIdx); if (match->type == protocolClientRemote) protocolClientNoOp(match->client); } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolFree(void) { FUNCTION_LOG_VOID(logLevelTrace); if (protocolHelper.clientList != NULL) { while (lstSize(protocolHelper.clientList) > 0) { protocolHelperClientFree(lstGet(protocolHelper.clientList, 0)); lstRemoveIdx(protocolHelper.clientList, 0); } } FUNCTION_LOG_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/helper.h0000664000175100017510000000604315240547630017476 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Helper ***********************************************************************************************************************************/ #ifndef PROTOCOL_HELPER_H #define PROTOCOL_HELPER_H #include "common/type/stringId.h" /*********************************************************************************************************************************** Protocol storage type enum ***********************************************************************************************************************************/ typedef enum { protocolStorageTypePg = STRID5("pg", 0xf00), protocolStorageTypeRepo = STRID5("repo", 0x7c0b20), } ProtocolStorageType; #include "common/io/server.h" #include "protocol/server.h" /*********************************************************************************************************************************** Constants ***********************************************************************************************************************************/ #define PROTOCOL_SERVICE_LOCAL "local" STRING_DECLARE(PROTOCOL_SERVICE_LOCAL_STR); #define PROTOCOL_SERVICE_REMOTE "remote" STRING_DECLARE(PROTOCOL_SERVICE_REMOTE_STR); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Is pg local? FN_EXTERN bool pgIsLocal(unsigned int pgIdx); // Error if PostgreSQL is not local, i.e. pg-host is set FN_EXTERN void pgIsLocalVerify(void); // Is the repository local? FN_EXTERN bool repoIsLocal(unsigned int repoIdx); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Send keepalives to all remotes FN_EXTERN void protocolKeepAlive(void); // Local protocol client FN_EXTERN ProtocolClient *protocolLocalGet(ProtocolStorageType protocolStorageType, unsigned int hostId, unsigned int protocolId); // Remote protocol client FN_EXTERN ProtocolClient *protocolRemoteGet(ProtocolStorageType protocolStorageType, unsigned int hostId, bool create); // Free (shutdown) a local/remote client FN_EXTERN void protocolHelperFree(ProtocolClient *client); // Initialize a server FN_EXTERN ProtocolServer *protocolServer(IoServer *const tlsServer, IoSession *const socketSession); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_EXTERN void protocolFree(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/parallel.c0000664000175100017510000002741315240547630020012 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Parallel Executor ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/log.h" #include "common/macro.h" #include "common/type/keyValue.h" #include "common/type/list.h" #include "protocol/helper.h" #include "protocol/parallel.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolParallelJobData { ProtocolParallelJob *job; // Job ProtocolClientSession *session; // Protocol session for the job } ProtocolParallelJobData; struct ProtocolParallel { TimeMSec timeout; // Max time to wait for jobs before returning ParallelJobCallback *callbackFunction; // Function to get new jobs void *callbackData; // Data to pass to callback function List *clientList; // List of clients to process jobs List *jobList; // List of jobs to be processed ProtocolParallelJobData *clientJobList; // Jobs being processing by each client ProtocolParallelJobState state; // Overall state of job processing }; /**********************************************************************************************************************************/ FN_EXTERN ProtocolParallel * protocolParallelNew(const TimeMSec timeout, ParallelJobCallback *const callbackFunction, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(UINT64, timeout); FUNCTION_LOG_PARAM(FUNCTIONP, callbackFunction); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); ASSERT(callbackFunction != NULL); ASSERT(callbackData != NULL); OBJ_NEW_BEGIN(ProtocolParallel, .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { *this = (ProtocolParallel) { .timeout = timeout, .callbackFunction = callbackFunction, .callbackData = callbackData, .clientList = lstNewP(sizeof(ProtocolClient *)), .jobList = lstNewP(sizeof(ProtocolParallelJob *)), .state = protocolParallelJobStatePending, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PROTOCOL_PARALLEL, this); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelClientAdd(ProtocolParallel *const this, ProtocolClient *const client) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(client != NULL); ASSERT(this->state == protocolParallelJobStatePending); if (protocolClientIoReadFd(client) == -1) THROW(AssertError, "client with read fd is required"); lstAdd(this->clientList, &client); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN unsigned int protocolParallelProcess(ProtocolParallel *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->state != protocolParallelJobStateDone); unsigned int result = 0; MEM_CONTEXT_TEMP_BEGIN() { // If called for the first time, initialize processing if (this->state == protocolParallelJobStatePending) { MEM_CONTEXT_OBJ_BEGIN(this) { this->clientJobList = memNew(lstSize(this->clientList) * sizeof(ProtocolParallelJobData)); for (unsigned int jobIdx = 0; jobIdx < lstSize(this->clientList); jobIdx++) this->clientJobList[jobIdx] = (ProtocolParallelJobData){0}; } MEM_CONTEXT_OBJ_END(); this->state = protocolParallelJobStateRunning; } // Initialize the file descriptor set used for select fd_set selectSet; FD_ZERO(&selectSet); int fdMax = -1; // Find clients that are running jobs unsigned int clientRunningTotal = 0; for (unsigned int clientIdx = 0; clientIdx < lstSize(this->clientList); clientIdx++) { if (this->clientJobList[clientIdx].job != NULL) { int fd = protocolClientIoReadFd(*(ProtocolClient **)lstGet(this->clientList, clientIdx)); FD_SET(fd, &selectSet); // Find the max file descriptor needed for select() MAX_ASSIGN(fdMax, fd); clientRunningTotal++; } } // If clients are running then wait for one to finish if (clientRunningTotal > 0) { // Initialize timeout struct used for select. Recreate this structure each time since Linux (at least) will modify it. struct timeval timeoutSelect; timeoutSelect.tv_sec = (time_t)(this->timeout / MSEC_PER_SEC); timeoutSelect.tv_usec = (suseconds_t)(this->timeout % MSEC_PER_SEC * 1000); // Determine if there is data to be read const int completed = select(fdMax + 1, &selectSet, NULL, NULL, &timeoutSelect); THROW_ON_SYS_ERROR(completed == -1, AssertError, "unable to select from parallel client(s)"); // If any jobs have completed then get the results if (completed > 0) { for (unsigned int clientIdx = 0; clientIdx < lstSize(this->clientList); clientIdx++) { ProtocolParallelJob *const job = this->clientJobList[clientIdx].job; if (job != NULL && FD_ISSET( protocolClientIoReadFd(*(ProtocolClient **)lstGet(this->clientList, clientIdx)), &selectSet)) { MEM_CONTEXT_TEMP_BEGIN() { TRY_BEGIN() { protocolParallelJobResultSet( job, protocolClientSessionResponse(this->clientJobList[clientIdx].session)); } CATCH_ANY() { protocolParallelJobErrorSet(job, errorCode(), STR(errorMessage())); } TRY_END(); protocolParallelJobStateSet(job, protocolParallelJobStateDone); this->clientJobList[clientIdx].job = NULL; protocolClientSessionFree(this->clientJobList[clientIdx].session); } MEM_CONTEXT_TEMP_END(); } } result = (unsigned int)completed; } } // Find new jobs to be run for (unsigned int clientIdx = 0; clientIdx < lstSize(this->clientList); clientIdx++) { ProtocolClient **const clientRef = lstGet(this->clientList, clientIdx); // If nothing is running for this client and it has not already been freed if (this->clientJobList[clientIdx].job == NULL && *clientRef != NULL) { MEM_CONTEXT_BEGIN(lstMemContext(this->jobList)) { // Get a new job ProtocolParallelJob *const job = this->callbackFunction(this->callbackData, clientIdx); ProtocolClient *const client = *clientRef; // If a new job was found if (job != NULL) { // Add to the job list lstAdd(this->jobList, &job); // Put command ProtocolClientSession *const session = protocolClientSessionNewP( client, protocolParallelJobCommand(job), .async = true); protocolClientSessionRequestAsyncP(session, .param = protocolParallelJobParam(job)); // Set client id and running state protocolParallelJobProcessIdSet(job, clientIdx + 1); protocolParallelJobStateSet(job, protocolParallelJobStateRunning); this->clientJobList[clientIdx].job = job; this->clientJobList[clientIdx].session = session; } // Else no more jobs for this client so free it. Clear the client reference so it is not freed again (and its // freed memory read while logging) on a subsequent call. else { protocolHelperFree(client); *clientRef = NULL; } } MEM_CONTEXT_END(); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(UINT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolParallelJob * protocolParallelResult(ProtocolParallel *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->state == protocolParallelJobStateRunning); ProtocolParallelJob *result = NULL; // Find the next completed job for (unsigned int jobIdx = 0; jobIdx < lstSize(this->jobList); jobIdx++) { ProtocolParallelJob *job = *(ProtocolParallelJob **)lstGet(this->jobList, jobIdx); if (protocolParallelJobState(job) == protocolParallelJobStateDone) { result = protocolParallelJobMove(job, memContextCurrent()); lstRemoveIdx(this->jobList, jobIdx); break; } } FUNCTION_LOG_RETURN(PROTOCOL_PARALLEL_JOB, result); } /**********************************************************************************************************************************/ FN_EXTERN bool protocolParallelDone(ProtocolParallel *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->state != protocolParallelJobStatePending); // If there are no jobs left then we are done if (this->state != protocolParallelJobStateDone && lstEmpty(this->jobList)) this->state = protocolParallelJobStateDone; FUNCTION_LOG_RETURN(BOOL, this->state == protocolParallelJobStateDone); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelToLog(const ProtocolParallel *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{state: "); strStcResultSizeInc(debugLog, strIdToLog(this->state, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt(debugLog, ", clientTotal: %u, jobTotal: %u}", lstSize(this->clientList), lstSize(this->jobList)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/parallel.h0000664000175100017510000000742715240547630020022 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Parallel Executor ***********************************************************************************************************************************/ #ifndef PROTOCOL_PARALLEL_H #define PROTOCOL_PARALLEL_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolParallel ProtocolParallel; #include "common/time.h" #include "common/type/object.h" #include "protocol/client.h" #include "protocol/parallelJob.h" /*********************************************************************************************************************************** Job request callback Called whenever a new job is required for processing. If no more jobs are available then NULL is returned. Note that NULL must be returned to each clientIdx in case job distribution varies by clientIdx. ***********************************************************************************************************************************/ typedef ProtocolParallelJob *ParallelJobCallback(void *data, unsigned int clientIdx); /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN ProtocolParallel *protocolParallelNew(TimeMSec timeout, ParallelJobCallback *callbackFunction, void *callbackData); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Are all jobs done? FN_EXTERN bool protocolParallelDone(ProtocolParallel *this); // Completed job result FN_EXTERN ProtocolParallelJob *protocolParallelResult(ProtocolParallel *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Add client FN_EXTERN void protocolParallelClientAdd(ProtocolParallel *this, ProtocolClient *client); // Process jobs FN_EXTERN unsigned int protocolParallelProcess(ProtocolParallel *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolParallelFree(ProtocolParallel *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void protocolParallelToLog(const ProtocolParallel *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_PARALLEL_TYPE \ ProtocolParallel * #define FUNCTION_LOG_PROTOCOL_PARALLEL_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolParallelToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/parallelJob.c0000664000175100017510000001304415240547630020440 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Parallel Job ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "protocol/parallelJob.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct ProtocolParallelJob { ProtocolParallelJobPub pub; // Publicly accessible variables }; /**********************************************************************************************************************************/ FN_EXTERN ProtocolParallelJob * protocolParallelJobNew(const Variant *const key, const StringId command, PackWrite *const param) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(VARIANT, key); FUNCTION_LOG_PARAM(STRING_ID, command); FUNCTION_LOG_PARAM(PACK_WRITE, param); FUNCTION_LOG_END(); OBJ_NEW_BEGIN(ProtocolParallelJob, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (ProtocolParallelJob) { .pub = { .state = protocolParallelJobStatePending, .key = varDup(key), .command = command, .param = pckWriteMove(param, objMemContext(this)), }, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PROTOCOL_PARALLEL_JOB, this); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobErrorSet(ProtocolParallelJob *const this, const int code, const String *const message) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this); FUNCTION_LOG_PARAM(INT, code); FUNCTION_LOG_PARAM(STRING, message); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(code != 0); ASSERT(message != NULL); MEM_CONTEXT_OBJ_BEGIN(this) { this->pub.code = code; this->pub.message = strDup(message); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobProcessIdSet(ProtocolParallelJob *const this, const unsigned int processId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this); FUNCTION_LOG_PARAM(UINT, processId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(processId > 0); this->pub.processId = processId; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobResultSet(ProtocolParallelJob *const this, PackRead *const result) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this); FUNCTION_LOG_PARAM(PACK_READ, result); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(protocolParallelJobErrorCode(this) == 0); this->pub.result = pckReadMove(result, objMemContext(this)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobStateSet(ProtocolParallelJob *const this, const ProtocolParallelJobState state) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_PARALLEL_JOB, this); FUNCTION_LOG_PARAM(STRING_ID, state); FUNCTION_LOG_END(); ASSERT(this != NULL); if (this->pub.state == protocolParallelJobStatePending && state == protocolParallelJobStateRunning) this->pub.state = protocolParallelJobStateRunning; else if (this->pub.state == protocolParallelJobStateRunning && state == protocolParallelJobStateDone) this->pub.state = protocolParallelJobStateDone; else { THROW_FMT( AssertError, "invalid state transition from '%s' to '%s'", zNewStrId(protocolParallelJobState(this)), zNewStrId(state)); } FUNCTION_LOG_RETURN_VOID(); } FN_EXTERN void protocolParallelJobToLog(const ProtocolParallelJob *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{state: "); strStcResultSizeInc(debugLog, strIdToLog(protocolParallelJobState(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", key: "); varToLog(protocolParallelJobKey(this), debugLog); strStcCat(debugLog, ", command: "); strStcResultSizeInc( debugLog, strIdToLog(protocolParallelJobCommand(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCat(debugLog, ", result: "); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT( protocolParallelJobResult(this), pckReadToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt(debugLog, ", code: %d, message: ", protocolParallelJobErrorCode(this)); strStcResultSizeInc( debugLog, FUNCTION_LOG_OBJECT_FORMAT( protocolParallelJobErrorMessage(this), strToLog, strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/parallelJob.h0000664000175100017510000001416615240547630020453 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Parallel Job ***********************************************************************************************************************************/ #ifndef PROTOCOL_PARALLEL_JOB_H #define PROTOCOL_PARALLEL_JOB_H #include "common/type/stringId.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolParallelJob ProtocolParallelJob; /*********************************************************************************************************************************** Job state enum ***********************************************************************************************************************************/ typedef enum { protocolParallelJobStatePending = STRID5("pending", 0x1dc9238b00), protocolParallelJobStateRunning = STRID5("running", 0x1dc973ab20), protocolParallelJobStateDone = STRID5("done", 0x2b9e40), } ProtocolParallelJobState; #include "common/time.h" #include "common/type/object.h" #include "common/type/pack.h" #include "protocol/client.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN ProtocolParallelJob *protocolParallelJobNew(const Variant *key, StringId command, PackWrite *param); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct ProtocolParallelJobPub { const Variant *key; // Unique key used to identify the job StringId command; // Command to be executed PackWrite *param; // Command parameters unsigned int processId; // Process that executed this job ProtocolParallelJobState state; // Current state of the job int code; // Non-zero result indicates an error String *message; // Message if there was a error PackRead *result; // Result if job was successful } ProtocolParallelJobPub; // Job command FN_INLINE_ALWAYS StringId protocolParallelJobCommand(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->command; } // Job command parameters FN_INLINE_ALWAYS PackWrite * protocolParallelJobParam(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->param; } // Job error FN_INLINE_ALWAYS int protocolParallelJobErrorCode(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->code; } FN_INLINE_ALWAYS const String * protocolParallelJobErrorMessage(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->message; } FN_EXTERN void protocolParallelJobErrorSet(ProtocolParallelJob *this, int code, const String *message); // Job key FN_INLINE_ALWAYS const Variant * protocolParallelJobKey(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->key; } // Process Id FN_INLINE_ALWAYS unsigned int protocolParallelJobProcessId(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->processId; } FN_EXTERN void protocolParallelJobProcessIdSet(ProtocolParallelJob *this, unsigned int processId); // Job result FN_INLINE_ALWAYS PackRead * protocolParallelJobResult(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->result; } FN_EXTERN void protocolParallelJobResultSet(ProtocolParallelJob *const this, PackRead *const result); // Job state FN_INLINE_ALWAYS ProtocolParallelJobState protocolParallelJobState(const ProtocolParallelJob *const this) { return THIS_PUB(ProtocolParallelJob)->state; } FN_EXTERN void protocolParallelJobStateSet(ProtocolParallelJob *this, ProtocolParallelJobState state); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to new parent mem context FN_INLINE_ALWAYS ProtocolParallelJob * protocolParallelJobMove(ProtocolParallelJob *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolParallelJobFree(ProtocolParallelJob *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void protocolParallelJobToLog(const ProtocolParallelJob *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_PARALLEL_JOB_TYPE \ ProtocolParallelJob * #define FUNCTION_LOG_PROTOCOL_PARALLEL_JOB_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolParallelJobToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/server.c0000664000175100017510000006165515240547630017532 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Server ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/error/retry.h" #include "common/log.h" #include "common/time.h" #include "common/type/json.h" #include "common/type/keyValue.h" #include "common/type/list.h" #include "protocol/helper.h" #include "protocol/server.h" #include "version.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct ProtocolServer { IoRead *read; // Read interface IoWrite *write; // Write interface const String *name; // Name displayed in logging List *sessionList; // List of active sessions uint64_t sessionId; // Current session being processed }; struct ProtocolServerResult { void *sessionData; // Session data PackWrite *data; // Result data size_t extra; // Extra bytes for initializing data bool close; // Close session? }; // Track server sessions typedef struct ProtocolServerSession { uint64_t id; // Session id void *data; // Data for the session } ProtocolServerSession; /**********************************************************************************************************************************/ FN_EXTERN ProtocolServer * protocolServerNew(const String *const name, const String *const service, IoRead *const read, IoWrite *const write) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(STRING, service); FUNCTION_LOG_PARAM(IO_READ, read); FUNCTION_LOG_PARAM(IO_WRITE, write); FUNCTION_LOG_END(); ASSERT(name != NULL); ASSERT(read != NULL); ASSERT(write != NULL); OBJ_NEW_BEGIN(ProtocolServer, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (ProtocolServer) { .read = read, .write = write, .name = strDup(name), .sessionList = lstNewP(sizeof(ProtocolServerSession)), }; // Send the protocol greeting MEM_CONTEXT_TEMP_BEGIN() { JsonWrite *const json = jsonWriteObjectBegin(jsonWriteNewP()); jsonWriteZ(jsonWriteKeyStrId(json, PROTOCOL_GREETING_NAME), PROJECT_NAME); jsonWriteStr(jsonWriteKeyStrId(json, PROTOCOL_GREETING_SERVICE), service); jsonWriteZ(jsonWriteKeyStrId(json, PROTOCOL_GREETING_VERSION), PROJECT_VERSION); ioWriteStrLine(this->write, jsonWriteResult(jsonWriteObjectEnd(json))); ioWriteFlush(this->write); } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER, this); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerResponse(ProtocolServer *const this, const ProtocolServerResponseParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_SERVER, this); FUNCTION_TEST_PARAM(ENUM, param.type); FUNCTION_TEST_PARAM(BOOL, param.close); FUNCTION_TEST_PARAM(PACK_WRITE, param.data); FUNCTION_TEST_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // End the pack if (param.data != NULL) pckWriteEndP(param.data); // Write the result PackWrite *const resultMessage = pckWriteNewIo(this->write); pckWriteU64P(resultMessage, this->sessionId); pckWriteU32P(resultMessage, param.type, .defaultWrite = true); pckWriteBoolP(resultMessage, param.close); pckWritePackP(resultMessage, pckWriteResult(param.data)); pckWriteEndP(resultMessage); ioWriteFlush(this->write); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerError(ProtocolServer *this, const int code, const String *const message, const String *const stack) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PROTOCOL_SERVER, this); FUNCTION_LOG_PARAM(INT, code); FUNCTION_LOG_PARAM(STRING, message); FUNCTION_LOG_PARAM(STRING, stack); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(code != 0); ASSERT(message != NULL); ASSERT(stack != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const packWrite = protocolPackNew(); pckWriteI32P(packWrite, code); pckWriteStrP(packWrite, message); pckWriteStrP(packWrite, stack); protocolServerResponseP(this, .type = protocolMessageTypeError, .data = packWrite); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerRequestResult protocolServerRequest(ProtocolServer *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(PROTOCOL_SERVER, this); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ProtocolServerRequestResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { PackRead *const request = pckReadNewIo(this->read); const ProtocolMessageType type = (ProtocolMessageType)pckReadU32P(request); CHECK(FormatError, type == protocolMessageTypeRequest, "expected request message"); MEM_CONTEXT_PRIOR_BEGIN() { result.id = pckReadStrIdP(request); result.type = (ProtocolCommandType)pckReadStrIdP(request); this->sessionId = pckReadU64P(request); result.sessionRequired = pckReadBoolP(request); result.param = pckReadPackP(request); ASSERT(this->sessionId != 0); } MEM_CONTEXT_PRIOR_END(); pckReadEndP(request); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerProcess(ProtocolServer *const this, const VariantList *const retryInterval, const List *const handlerList) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PROTOCOL_SERVER, this); FUNCTION_LOG_PARAM(VARIANT_LIST, retryInterval); FUNCTION_LOG_PARAM(LIST, handlerList); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(handlerList != NULL); ASSERT(!lstEmpty(handlerList)); // Loop until exit request is received bool exit = false; do { TRY_BEGIN() { MEM_CONTEXT_TEMP_BEGIN() { // Get request ProtocolServerRequestResult request = protocolServerRequest(this); // Find the handler const ProtocolServerHandler *handler = NULL; for (unsigned int handlerIdx = 0; handlerIdx < lstSize(handlerList); handlerIdx++) { const ProtocolServerHandler *const handlerMatch = lstGet(handlerList, handlerIdx); if (request.id == handlerMatch->command) { handler = handlerMatch; break; } } // If handler was found then process if (handler != NULL) { // Send the request to the handler MEM_CONTEXT_TEMP_BEGIN() { // Variables to store first error message and retry messages ErrorRetry *const errRetry = errRetryNew(); const String *errStackTrace = NULL; // Initialize retries in case of request failure bool retry = false; unsigned int retryRemaining = retryInterval != NULL ? varLstSize(retryInterval) : 0; TimeMSec retrySleepMs = 0; // Handler retry loop do { retry = false; TRY_BEGIN() { // Process request type switch (request.type) { // Open a protocol session case protocolCommandTypeOpen: { ASSERT(handler->open != NULL); // Call open handler ProtocolServerResult *const openResult = handler->open(pckReadNew(request.param)); ASSERT(openResult != NULL); ASSERT(!openResult->close); ProtocolServerSession session = {.id = this->sessionId}; // Send data protocolServerResponseP( this, .data = openResult->data, .close = openResult->sessionData == NULL); // Create session if open returned session data if (openResult->sessionData != NULL) { session.data = objMove(openResult->sessionData, objMemContext(this->sessionList)); lstAdd(this->sessionList, &session); } break; } // Process or close/cancel protocol session default: { // Find session data void *sessionData = NULL; unsigned int sessionListIdx = 0; if (request.sessionRequired) { ASSERT(handler->processSession != NULL); ASSERT(handler->process == NULL); for (; sessionListIdx < lstSize(this->sessionList); sessionListIdx++) { ProtocolServerSession *const session = lstGet(this->sessionList, sessionListIdx); if (session->id == this->sessionId) { sessionData = session->data; break; } } // Error when session not found if (sessionData == NULL && request.type != protocolCommandTypeCancel) { THROW_FMT( ProtocolError, "unable to find session id %" PRIu64 " for request %s:%s", this->sessionId, zNewStrId(request.id), zNewStrId(request.type)); } } // Process request type switch (request.type) { // Process request case protocolCommandTypeProcess: { // Process session ProtocolServerResult *processResult; if (handler->processSession != NULL) { ASSERT(handler->process == NULL); CHECK_FMT( ProtocolError, this->sessionId != 0, "no session id for request %s:%s", zNewStrId(request.id), zNewStrId(request.type)); processResult = handler->processSession(pckReadNew(request.param), sessionData); } // Standalone process else { ASSERT(handler->process != NULL); ASSERT(!request.sessionRequired); processResult = handler->process(pckReadNew(request.param)); } if (processResult == NULL) protocolServerResponseP(this); else { ASSERT(processResult->sessionData == NULL); ASSERT(handler->processSession != NULL || !processResult->close); protocolServerResponseP( this, .data = processResult->data, .close = processResult->close); // Free session when close is true. This optimization allows an explicit // close to be skipped. if (processResult->close) { objFree(sessionData); lstRemoveIdx(this->sessionList, sessionListIdx); } } break; } // Close protocol session case protocolCommandTypeClose: { // If there is a close handler then call it PackWrite *data = NULL; if (handler->close != NULL) { ProtocolServerResult *const closeResult = handler->close( pckReadNew(request.param), sessionData); ASSERT(closeResult != NULL); ASSERT(closeResult->sessionData == NULL); ASSERT(!closeResult->close); data = closeResult->data; } // Send data protocolServerResponseP(this, .data = data, .close = true); // Free the session objFree(sessionData); lstRemoveIdx(this->sessionList, sessionListIdx); break; } // Cancel protocol session default: { CHECK_FMT( ProtocolError, request.type == protocolCommandTypeCancel, "unknown request type '%s'", zNewStrId(request.type)); // Send NULL data protocolServerResponseP(this, .close = true); // Free the session if (sessionData != NULL) { objFree(sessionData); lstRemoveIdx(this->sessionList, sessionListIdx); } break; } } } } } CATCH_ANY() { // Add the error retry info errRetryAddP(errRetry); // On first error record the stack trace. Only the first error will contain a stack trace since // the first error is most likely to contain valuable information. if (errStackTrace == NULL) errStackTrace = strNewZ(errorStackTrace()); // Are there retries remaining? if (retryRemaining > 0) { // Get the sleep interval for this retry retrySleepMs = varUInt64(varLstGet(retryInterval, varLstSize(retryInterval) - retryRemaining)); // Log the retry LOG_DEBUG_FMT( "retry %s after %" PRIu64 "ms: %s", errorTypeName(errorType()), retrySleepMs, errorMessage()); // Sleep for interval sleepMSec(retrySleepMs); // Decrement retries remaining and retry retryRemaining--; retry = true; // Send keep-alive to remotes. A retry means the request is taking longer than usual so make // sure the remote does not timeout. protocolKeepAlive(); } // Else report error to the client else { protocolServerError( this, errorTypeCode(errRetryType(errRetry)), errRetryMessage(errRetry), errStackTrace); } } TRY_END(); } while (retry); } MEM_CONTEXT_TEMP_END(); } // Else check built-in requests else { switch (request.id) { case PROTOCOL_COMMAND_EXIT: exit = true; break; case PROTOCOL_COMMAND_NOOP: protocolServerResponseP(this); break; default: THROW_FMT( ProtocolError, "invalid request '%s' (0x%" PRIx64 ")", zNewStrId(request.id), request.id); } } // Send keep-alive to remotes. When a local process is doing work that does not involve the remote it is important // that the remote does not timeout. This will send a keep alive once per unit of work that is performed by the // local process. protocolKeepAlive(); } MEM_CONTEXT_TEMP_END(); } CATCH_FATAL() { // Zero session id so a fatal error will be handled by the first client that sees it this->sessionId = 0; // Report error to the client protocolServerError(this, errorCode(), STR(errorMessage()), STR(errorStackTrace())); // Rethrow so the process exits with an error RETHROW(); } TRY_END(); } while (!exit); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * protocolServerResultNew(const ProtocolServerResultNewParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, param.extra); FUNCTION_TEST_END(); OBJ_NEW_BEGIN(ProtocolServerResult, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (ProtocolServerResult) { .extra = param.extra, }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(PROTOCOL_SERVER_RESULT, this); } /**********************************************************************************************************************************/ FN_EXTERN PackWrite * protocolServerResultData(ProtocolServerResult *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_SERVER_RESULT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(this->data == NULL); MEM_CONTEXT_OBJ_BEGIN(this) { this->data = pckWriteNewP(.size = PROTOCOL_PACK_DEFAULT_SIZE + this->extra); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN(PACK_WRITE, this->data); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerResultSessionDataSet(ProtocolServerResult *const this, void *const sessionData) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_SERVER_RESULT, this); FUNCTION_TEST_PARAM_P(VOID, sessionData); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(sessionData != NULL); this->sessionData = objMove(sessionData, objMemContext(this)); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerResultCloseSet(ProtocolServerResult *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(PROTOCOL_SERVER_RESULT, this); FUNCTION_TEST_END(); ASSERT(this != NULL); this->close = true; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerToLog(const ProtocolServer *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{name: %s}", strZ(this->name)); } /**********************************************************************************************************************************/ FN_EXTERN void protocolServerResultToLog(const ProtocolServerResult *const this, StringStatic *const debugLog) { strStcFmt( debugLog, "{data: %s, sessionData: %s, close: %s}", cvtBoolToConstZ(this->data != NULL), cvtBoolToConstZ(this->sessionData != NULL), cvtBoolToConstZ(this->close)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/protocol/server.h0000664000175100017510000001763215240547630017533 0ustar00runnerrunner/*********************************************************************************************************************************** Protocol Server ***********************************************************************************************************************************/ #ifndef PROTOCOL_SERVER_H #define PROTOCOL_SERVER_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct ProtocolServer ProtocolServer; typedef struct ProtocolServerResult ProtocolServerResult; #include "common/io/read.h" #include "common/io/write.h" #include "common/type/object.h" #include "common/type/pack.h" #include "common/type/stringId.h" #include "protocol/client.h" /*********************************************************************************************************************************** Protocol command handler type and structure An array of this struct must be passed to protocolServerProcess() for the server to process commands. Each command handler should implement a single command, as defined by the command string. ***********************************************************************************************************************************/ typedef ProtocolServerResult *(*ProtocolServerCommandOpenHandler)(PackRead *param); typedef ProtocolServerResult *(*ProtocolServerCommandProcessHandler)(PackRead *param); typedef ProtocolServerResult *(*ProtocolServerCommandProcessSessionHandler)(PackRead *param, void *sessionData); typedef ProtocolServerResult *(*ProtocolServerCommandCloseHandler)(PackRead *param, void *sessionData); typedef struct ProtocolServerHandler { StringId command; // 5-bit StringId that identifies the protocol command ProtocolServerCommandOpenHandler open; // Function that opens the protocol session ProtocolServerCommandProcessHandler process; // Function that processes the protocol command ProtocolServerCommandProcessSessionHandler processSession; // Function that processes the protocol command for a session ProtocolServerCommandCloseHandler close; // Function that closes the protocol session } ProtocolServerHandler; /*********************************************************************************************************************************** Server Constructors ***********************************************************************************************************************************/ FN_EXTERN ProtocolServer *protocolServerNew(const String *name, const String *service, IoRead *read, IoWrite *write); /*********************************************************************************************************************************** Server Functions ***********************************************************************************************************************************/ // Get command from the client. Outside ProtocolServer, this is used when the first noop command needs to be processed before // running protocolServerProcess(), which allows an error to be returned to the client if initialization fails. typedef struct ProtocolServerRequestResult { StringId id; // Command identifier ProtocolCommandType type; // Command type bool sessionRequired; // Session with more than one command Pack *param; // Parameter pack } ProtocolServerRequestResult; FN_EXTERN ProtocolServerRequestResult protocolServerRequest(ProtocolServer *this); // Send response to client typedef struct ProtocolServerResponseParam { VAR_PARAM_HEADER; bool close; // Has the session been closed? ProtocolMessageType type; // Message type PackWrite *data; // Response data } ProtocolServerResponseParam; #define protocolServerResponseP(this, ...) \ protocolServerResponse(this, (ProtocolServerResponseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void protocolServerResponse(ProtocolServer *const this, ProtocolServerResponseParam param); // Send an error to client FN_EXTERN void protocolServerError(ProtocolServer *this, int code, const String *message, const String *stack); // Process requests FN_EXTERN void protocolServerProcess(ProtocolServer *this, const VariantList *retryInterval, const List *handlerList); // Move to a new parent mem context FN_INLINE_ALWAYS ProtocolServer * protocolServerMove(ProtocolServer *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Server Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void protocolServerFree(ProtocolServer *const this) { objFree(this); } /*********************************************************************************************************************************** Result Constructors ***********************************************************************************************************************************/ typedef struct ProtocolServerResultNewParam { VAR_PARAM_HEADER; size_t extra; } ProtocolServerResultNewParam; #define protocolServerResultNewP(...) \ protocolServerResultNew((ProtocolServerResultNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN ProtocolServerResult *protocolServerResultNew(ProtocolServerResultNewParam param); /*********************************************************************************************************************************** Result Getters/Setters ***********************************************************************************************************************************/ // Create PackWrite object required to send data to the client FN_EXTERN PackWrite *protocolServerResultData(ProtocolServerResult *this); // Set session data FN_EXTERN void protocolServerResultSessionDataSet(ProtocolServerResult *const this, void *const sessionData); // Close session FN_EXTERN void protocolServerResultCloseSet(ProtocolServerResult *const this); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void protocolServerToLog(const ProtocolServer *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_SERVER_TYPE \ ProtocolServer * #define FUNCTION_LOG_PROTOCOL_SERVER_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolServerToLog, buffer, bufferSize) FN_EXTERN void protocolServerResultToLog(const ProtocolServerResult *this, StringStatic *debugLog); #define FUNCTION_LOG_PROTOCOL_SERVER_RESULT_TYPE \ ProtocolServerResult * #define FUNCTION_LOG_PROTOCOL_SERVER_RESULT_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, protocolServerResultToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/0000775000175100017510000000000015240547630015646 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/0000775000175100017510000000000015240547630016774 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/helper.c0000664000175100017510000001061315240547630020420 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Helper ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/url.h" #include "common/io/io.h" #include "common/log.h" #include "config/config.h" #include "storage/azure/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageAzureHelper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_AZURE_TYPE); Storage *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Parse the endpoint url const HttpUrl *const url = httpUrlNewParseP( cfgOptionIdxStr(cfgOptRepoAzureEndpoint, repoIdx), .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); const String *endpoint = httpUrlHost(url); unsigned int port = httpUrlPort(url); HttpProtocolType protocolType = httpUrlProtocolType(url); StorageAzureUriStyle uriStyle = (StorageAzureUriStyle)cfgOptionIdxSeq(cfgOptRepoAzureUriStyle, repoIdx); // If the host is set then set it as the endpoint. The host option is used to set path-style URIs when working with Azurite. // This was ill-advised, so the uri-style option was added to allow the user to select the URI style used by the server. // Preserve the old behavior when uri-style is defaulted. if (cfgOptionIdxStrNull(cfgOptRepoStorageHost, repoIdx) != NULL) { const HttpUrl *const url = httpUrlNewParseP( cfgOptionIdxStr(cfgOptRepoStorageHost, repoIdx), .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); endpoint = httpUrlHost(url); port = httpUrlPort(url); protocolType = httpUrlProtocolType(url); if (cfgOptionIdxSource(cfgOptRepoAzureUriStyle, repoIdx) == cfgSourceDefault) uriStyle = storageAzureUriStylePath; } // Ensure the key is valid base64 when key type is shared const StorageAzureKeyType keyType = (StorageAzureKeyType)cfgOptionIdxSeq(cfgOptRepoAzureKeyType, repoIdx); const String *const key = cfgOptionIdxStrNull(cfgOptRepoAzureKey, repoIdx); if (keyType == storageAzureKeyTypeShared) { TRY_BEGIN() { bufNewDecode(encodingBase64, key); } CATCH(FormatError) { THROW_FMT( OptionInvalidValueError, "invalid value for '%s' option: %s\n" "HINT: value must be valid base64 when '%s = " CFGOPTVAL_REPO_AZURE_KEY_TYPE_SHARED_Z "'.", cfgOptionIdxName(cfgOptRepoAzureKey, repoIdx), errorMessage(), cfgOptionIdxName(cfgOptRepoAzureKeyType, repoIdx)); } TRY_END(); } // If port was specified, overwrite the parsed/default port if (cfgOptionIdxSource(cfgOptRepoStoragePort, repoIdx) != cfgSourceDefault) port = cfgOptionIdxUInt(cfgOptRepoStoragePort, repoIdx); MEM_CONTEXT_PRIOR_BEGIN() { result = storageAzureNew( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), write, storageRepoTargetTime(), pathExpressionCallback, cfgOptionIdxStr(cfgOptRepoAzureContainer, repoIdx), cfgOptionIdxStr(cfgOptRepoAzureAccount, repoIdx), keyType, key, (size_t)cfgOptionIdxUInt64(cfgOptRepoStorageUploadChunkSize, repoIdx), cfgOptionIdxKvNull(cfgOptRepoStorageTag, repoIdx), endpoint, uriStyle, port, ioTimeoutMs(), protocolType, cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaFile, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaPath, repoIdx)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/helper.h0000664000175100017510000000237415240547630020432 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_STORAGE_HELPER_H #define STORAGE_AZURE_STORAGE_HELPER_H #include "storage/azure/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageAzureHelper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_AZURE_HELPER {.type = STORAGE_AZURE_TYPE, .helper = storageAzureHelper} #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/read.c0000664000175100017510000001436115240547630020060 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Read ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/client.h" #include "common/log.h" #include "common/type/object.h" #include "storage/azure/read.h" #include "storage/read.h" /*********************************************************************************************************************************** Azure query tokens ***********************************************************************************************************************************/ STRING_STATIC(AZURE_QUERY_VERSION_ID_STR, "versionid"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageReadAzure { const StorageReadInterface *interface; // Interface StorageAzure *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) const String *versionId; // Version id (NULL for most recent) HttpResponse *httpResponse; // HTTP response }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadAzureOpen(THIS_VOID) { THIS(StorageReadAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse == NULL); // Request the file MEM_CONTEXT_OBJ_BEGIN(this) { this->httpResponse = storageAzureRequestP( this->storage, HTTP_VERB_GET_STR, .path = this->name, .query = this->versionId != NULL ? httpQueryPut(httpQueryNewP(), AZURE_QUERY_VERSION_ID_STR, this->versionId) : NULL, .header = httpHeaderPutRange(httpHeaderNew(NULL), this->offset, this->limit), .allowMissing = true, .contentIo = true); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(BOOL, httpResponseCodeOk(this->httpResponse)); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadAzure(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_AZURE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL && this->httpResponse != NULL); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); FUNCTION_LOG_RETURN(SIZE, ioRead(httpResponseIoRead(this->httpResponse), buffer)); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadAzureEof(THIS_VOID) { THIS(StorageReadAzure); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_AZURE, this); FUNCTION_TEST_END(); ASSERT(this != NULL && this->httpResponse != NULL); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); FUNCTION_TEST_RETURN(BOOL, ioReadEof(httpResponseIoRead(this->httpResponse))); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadAzureClose(THIS_VOID) { THIS(StorageReadAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse != NULL); httpResponseFree(this->httpResponse); this->httpResponse = NULL; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadAzureInterface = { .close = storageReadAzureClose, .eof = storageReadAzureEof, .open = storageReadAzureOpen, .read = storageReadAzure, }; FN_EXTERN StorageReadAzure * storageReadAzureNew( StorageAzure *const storage, const String *const name, const uint64_t offset, const Variant *const limit, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_AZURE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadAzure, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadAzure) { .interface = &storageReadAzureInterface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), .versionId = strDup(versionId), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_AZURE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/read.h0000664000175100017510000000345615240547630020070 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_READ_H #define STORAGE_AZURE_READ_H #include "storage/azure/storage.intern.h" #include "storage/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadAzure StorageReadAzure; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadAzure *storageReadAzureNew( StorageAzure *storage, const String *name, uint64_t offset, const Variant *limit, const String *versionId); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_AZURE_TYPE \ StorageReadAzure * #define FUNCTION_LOG_STORAGE_READ_AZURE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadAzure", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/storage.c0000664000175100017510000014105715240547630020614 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage ***********************************************************************************************************************************/ #include #include #include "common/crypto/common.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/http/client.h" #include "common/io/http/common.h" #include "common/io/socket/client.h" #include "common/io/tls/client.h" #include "common/log.h" #include "common/regExp.h" #include "common/stat.h" #include "common/type/json.h" #include "common/type/object.h" #include "common/type/xml.h" #include "storage/azure/read.h" #include "storage/azure/write.h" /*********************************************************************************************************************************** Defaults ***********************************************************************************************************************************/ #define STORAGE_AZURE_DELETE_MAX 256 /*********************************************************************************************************************************** Azure http headers ***********************************************************************************************************************************/ STRING_STATIC(AZURE_HEADER_TAGS, "x-ms-tags"); STRING_STATIC(AZURE_HEADER_VERSION_STR, "x-ms-version"); STRING_STATIC(AZURE_HEADER_VERSION_VALUE_STR, "2024-08-04"); /*********************************************************************************************************************************** Azure query tokens ***********************************************************************************************************************************/ STRING_STATIC(AZURE_QUERY_MARKER_STR, "marker"); STRING_STATIC(AZURE_QUERY_BATCH_STR, "batch"); STRING_EXTERN(AZURE_QUERY_COMP_STR, AZURE_QUERY_COMP); STRING_STATIC(AZURE_QUERY_DELIMITER_STR, "delimiter"); STRING_STATIC(AZURE_QUERY_INCLUDE_STR, "include"); STRING_STATIC(AZURE_QUERY_PREFIX_STR, "prefix"); STRING_EXTERN(AZURE_QUERY_RESTYPE_STR, AZURE_QUERY_RESTYPE); STRING_STATIC(AZURE_QUERY_SIG_STR, "sig"); STRING_STATIC(AZURE_QUERY_VALUE_LIST_STR, "list"); STRING_EXTERN(AZURE_QUERY_VALUE_CONTAINER_STR, AZURE_QUERY_VALUE_CONTAINER); STRING_STATIC(AZURE_QUERY_VALUE_VERSIONS_STR, "versions"); STRING_STATIC(AZURE_QUERY_API_VERSION, "api-version"); STRING_STATIC(AZURE_QUERY_RESOURCE, "resource"); /*********************************************************************************************************************************** XML tags ***********************************************************************************************************************************/ STRING_STATIC(AZURE_XML_TAG_BLOB_PREFIX_STR, "BlobPrefix"); STRING_STATIC(AZURE_XML_TAG_BLOB_STR, "Blob"); STRING_STATIC(AZURE_XML_TAG_BLOBS_STR, "Blobs"); STRING_STATIC(AZURE_XML_TAG_CONTENT_LENGTH_STR, "Content-Length"); STRING_STATIC(AZURE_XML_TAG_LAST_MODIFIED_STR, "Last-Modified"); STRING_STATIC(AZURE_XML_TAG_NEXT_MARKER_STR, "NextMarker"); STRING_STATIC(AZURE_XML_TAG_NAME_STR, "Name"); STRING_STATIC(AZURE_XML_TAG_PROPERTIES_STR, "Properties"); STRING_STATIC(AZURE_XML_TAG_VERSION_ID_STR, "VersionId"); /*********************************************************************************************************************************** Constants required for Azure managed identities Documentation for the response format is found at: https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-to-use-vm-token#get-a-token-using-curl ***********************************************************************************************************************************/ STRING_STATIC(AZURE_CREDENTIAL_HOST_STR, "169.254.169.254"); #define AZURE_CREDENTIAL_PORT 80 #define AZURE_CREDENTIAL_PATH "/metadata/identity/oauth2/token" #define AZURE_CREDENTIAL_API_VERSION "2018-02-01" VARIANT_STRDEF_STATIC(AZURE_JSON_TAG_ACCESS_TOKEN_VAR, "access_token"); VARIANT_STRDEF_STATIC(AZURE_JSON_TAG_EXPIRES_IN_VAR, "expires_in"); /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_STATIC(AZURE_STAT_REMOVE_STR, "azure.rm"); STRING_STATIC(AZURE_STAT_REMOVE_BATCH_STR, "azure.rm.batch"); STRING_STATIC(AZURE_STAT_REMOVE_BATCH_PART_STR, "azure.rm.batch.part"); STRING_STATIC(AZURE_STAT_REMOVE_BATCH_RETRY_STR, "azure.rm.batch.retry"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageAzure { STORAGE_COMMON_MEMBER; HttpClient *httpClient; // Http client to service requests StringList *headerRedactList; // List of headers to redact from logging StringList *queryRedactList; // List of query keys to redact from logging StorageAzureKeyType keyType; // Key type (e.g. storageAzureKeyTypeShared) const String *container; // Container to store data in const String *account; // Account const Buffer *sharedKey; // Shared key const HttpQuery *sasKey; // SAS key const String *host; // Host name size_t blockSize; // Block size for multi-block upload unsigned int deleteMax; // Maximum objects that can be deleted in one request const String *tag; // Tags to be applied to objects const String *pathPrefix; // Account/container prefix uint64_t fileId; // Id to used to make file block identifiers unique // For Azure managed identities authentication HttpClient *credHttpClient; // HTTP client to service credential requests const String *credHost; // Credentials host String *accessToken; // Access token time_t accessTokenExpirationTime; // Time the access token expires }; /*********************************************************************************************************************************** Generate authorization header and add it to the supplied header list Based on the documentation at https://docs.microsoft.com/en-us/rest/api/storageservices/authorize-with-shared-key ***********************************************************************************************************************************/ static void storageAzureAuth( StorageAzure *const this, const String *const verb, const String *const path, HttpQuery *const query, const String *const dateTime, HttpHeader *const httpHeader, const bool subRequest) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_AZURE, this); FUNCTION_TEST_PARAM(STRING, verb); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_PARAM(STRING, dateTime); FUNCTION_TEST_PARAM(KEY_VALUE, httpHeader); FUNCTION_TEST_PARAM(BOOL, subRequest); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(verb != NULL); ASSERT(path != NULL); ASSERT(dateTime != NULL); ASSERT(httpHeader != NULL); ASSERT(httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_LENGTH_STR) != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Host header is required for all authentication types but must not be set on batch sub-requests if (!subRequest) httpHeaderPut(httpHeader, HTTP_HEADER_HOST_STR, this->host); // Date header is required for shared key authentication (for signing) if (this->keyType == storageAzureKeyTypeShared) httpHeaderPut(httpHeader, HTTP_HEADER_DATE_STR, dateTime); // Set version header (required for shared key and auto auth types, not for SAS) but never on batch sub-requests, // which inherit the version from the top-level batch request if (this->keyType != storageAzureKeyTypeSas && !subRequest) httpHeaderPut(httpHeader, AZURE_HEADER_VERSION_STR, AZURE_HEADER_VERSION_VALUE_STR); // Shared key authentication if (this->keyType == storageAzureKeyTypeShared) { ASSERT(this->sharedKey != NULL); // Generate canonical headers String *const headerCanonical = strNew(); const StringList *const headerKeyList = httpHeaderList(httpHeader); for (unsigned int headerKeyIdx = 0; headerKeyIdx < strLstSize(headerKeyList); headerKeyIdx++) { const String *const headerKey = strLstGet(headerKeyList, headerKeyIdx); if (!strBeginsWithZ(headerKey, "x-ms-")) continue; strCatFmt(headerCanonical, "%s:%s\n", strZ(headerKey), strZ(httpHeaderGet(httpHeader, headerKey))); } // Generate canonical query String *const queryCanonical = strNew(); if (query != NULL) { const StringList *const queryKeyList = httpQueryList(query); ASSERT(!strLstEmpty(queryKeyList)); for (unsigned int queryKeyIdx = 0; queryKeyIdx < strLstSize(queryKeyList); queryKeyIdx++) { const String *const queryKey = strLstGet(queryKeyList, queryKeyIdx); strCatFmt(queryCanonical, "\n%s:%s", strZ(queryKey), strZ(httpQueryGet(query, queryKey))); } } // Generate string to sign const String *const contentLength = httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_LENGTH_STR); const String *const contentMd5 = httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_MD5_STR); const String *const contentType = httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_TYPE_STR); const String *const range = httpHeaderGet(httpHeader, HTTP_HEADER_RANGE_STR); const String *const stringToSign = strNewFmt( "%s\n" // verb "\n" // content-encoding "\n" // content-language "%s\n" // content-length "%s\n" // content-md5 "%s\n" // content-type "%s\n" // date "\n" // If-Modified-Since "\n" // If-Match "\n" // If-None-Match "\n" // If-Unmodified-Since "%s\n" // range "%s" // Canonicalized headers "/%s%s" // Canonicalized account/path "%s", // Canonicalized query strZ(verb), strEq(contentLength, ZERO_STR) ? "" : strZ(contentLength), contentMd5 == NULL ? "" : strZ(contentMd5), contentType == NULL ? "" : strZ(contentType), strZ(dateTime), range == NULL ? "" : strZ(range), strZ(headerCanonical), strZ(this->account), strZ(path), strZ(queryCanonical)); // Generate authorization header httpHeaderPut( httpHeader, HTTP_HEADER_AUTHORIZATION_STR, strNewFmt( "SharedKey %s:%s", strZ(this->account), strZ(strNewEncode(encodingBase64, cryptoHmacOne(hashTypeSha256, this->sharedKey, BUFSTR(stringToSign)))))); } // Auto authentication else if (this->keyType == storageAzureKeyTypeAuto) { const time_t timeBegin = time(NULL); if (timeBegin >= this->accessTokenExpirationTime) { // Retrieve the access token via the Managed Identities endpoint HttpHeader *const authHeader = httpHeaderNew(NULL); httpHeaderAdd(authHeader, STRDEF("Metadata"), TRUE_STR); httpHeaderAdd(authHeader, HTTP_HEADER_HOST_STR, this->credHost); httpHeaderAdd(authHeader, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); HttpQuery *const authQuery = httpQueryNewP(); httpQueryAdd(authQuery, AZURE_QUERY_API_VERSION, STRDEF(AZURE_CREDENTIAL_API_VERSION)); httpQueryAdd(authQuery, AZURE_QUERY_RESOURCE, strNewFmt("https://%s", strZ(this->host))); HttpRequest *const request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, STRDEF(AZURE_CREDENTIAL_PATH), .header = authHeader, .query = authQuery); HttpResponse *const response = httpRequestResponse(request, true); // Set the access_token on success and store an expiration time when we should re-fetch it if (httpResponseCodeOk(response)) { // Get credentials and expiration from the JSON response const KeyValue *const credential = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); const String *const accessToken = varStr(kvGet(credential, AZURE_JSON_TAG_ACCESS_TOKEN_VAR)); CHECK(FormatError, accessToken != NULL, "access token missing"); const Variant *const expiresInStr = kvGet(credential, AZURE_JSON_TAG_EXPIRES_IN_VAR); CHECK(FormatError, expiresInStr != NULL, "expiry missing"); MEM_CONTEXT_OBJ_BEGIN(this) { strCat(strTrunc(this->accessToken), accessToken); // Subtract http client timeout * 2 so the token does not expire in the middle of http retries const time_t clientTimeoutPeriod = ((time_t)(httpClientTimeout(this->httpClient) / MSEC_PER_SEC * 2)); const time_t expiresIn = (time_t)varInt64Force(expiresInStr); this->accessTokenExpirationTime = timeBegin + expiresIn - clientTimeoutPeriod; } MEM_CONTEXT_OBJ_END(); } else httpRequestError(request, response); } // Add authorization header httpHeaderPut(httpHeader, HTTP_HEADER_AUTHORIZATION_STR, strNewFmt("Bearer %s", strZ(this->accessToken))); } // SAS authentication else httpQueryMerge(query, this->sasKey); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Process Azure request ***********************************************************************************************************************************/ FN_EXTERN HttpRequest * storageAzureRequestAsync(StorageAzure *const this, const String *const verb, StorageAzureRequestAsyncParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, param.path); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(LIST, param.contentList); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(verb != NULL); HttpRequest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Prepend path prefix param.path = param.path == NULL ? this->pathPrefix : strNewFmt("%s%s", strZ(this->pathPrefix), strZ(param.path)); // Create header list HttpHeader *requestHeader = param.header == NULL ? httpHeaderNew(this->headerRedactList) : httpHeaderDup(param.header, this->headerRedactList); // Set content or construct multipart content const String *const date = httpDateFromTime(time(NULL)); const Buffer *content = param.content; if (param.contentList != NULL) { ASSERT(param.content == NULL); ASSERT(!lstEmpty(param.contentList)); HttpRequestMulti *const requestMulti = httpRequestMultiNew(); for (unsigned int contentIdx = 0; contentIdx < lstSize(param.contentList); contentIdx++) { const StorageAzureRequestPart *const requestPart = lstGet(param.contentList, contentIdx); HttpHeader *const partHeader = httpHeaderNew(this->headerRedactList); HttpQuery *const partQuery = this->sasKey != NULL ? httpQueryNewP(.redactList = this->queryRedactList) : NULL; const String *const path = strNewFmt("%s%s", strZ(this->pathPrefix), strZ(requestPart->path)); httpHeaderAdd(partHeader, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); storageAzureAuth(this, requestPart->verb, path, partQuery, date, partHeader, true); httpRequestMultiAddP( requestMulti, strNewFmt("%u", contentIdx), requestPart->verb, path, .header = partHeader, .query = partQuery); } httpRequestMultiHeaderAdd(requestMulti, requestHeader); content = httpRequestMultiContent(requestMulti); } // Set content length httpHeaderAdd( requestHeader, HTTP_HEADER_CONTENT_LENGTH_STR, content == NULL || bufEmpty(content) ? ZERO_STR : strNewFmt("%zu", bufUsed(content))); // Calculate content-md5 header if there is content if (content != NULL) { httpHeaderAdd( requestHeader, HTTP_HEADER_CONTENT_MD5_STR, strNewEncode(encodingBase64, cryptoHashOne(hashTypeMd5, content))); } // Set tags when requested and available if (param.tag && this->tag != NULL) httpHeaderPut(requestHeader, AZURE_HEADER_TAGS, this->tag); // Encode path const String *const path = httpUriEncode(param.path, true); // Make a copy of the query so it can be modified HttpQuery *const query = this->sasKey != NULL && param.query == NULL ? httpQueryNewP(.redactList = this->queryRedactList) : httpQueryDupP(param.query, .redactList = this->queryRedactList); // Generate authorization header storageAzureAuth(this, verb, path, query, date, requestHeader, false); // Send request MEM_CONTEXT_PRIOR_BEGIN() { result = httpRequestNewP( this->httpClient, verb, path, .query = query, .header = requestHeader, .content = content); } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_REQUEST, result); } FN_EXTERN HttpResponse * storageAzureResponse(HttpRequest *const request, const StorageAzureResponseParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, request); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_END(); ASSERT(request != NULL); HttpResponse *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get response result = httpRequestResponse(request, !param.contentIo); // Error if the request was not successful if (!httpResponseCodeOk(result) && (!param.allowMissing || httpResponseCode(result) != HTTP_RESPONSE_CODE_NOT_FOUND)) httpRequestError(request, result); // Move response to the prior context httpResponseMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } FN_EXTERN HttpResponse * storageAzureRequest(StorageAzure *const this, const String *const verb, const StorageAzureRequestParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, param.path); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(LIST, param.contentList); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_END(); HttpRequest *const request = storageAzureRequestAsyncP( this, verb, .path = param.path, .header = param.header, .query = param.query, .content = param.content, .contentList = param.contentList, .tag = param.tag); HttpResponse *const result = storageAzureResponseP(request, .allowMissing = param.allowMissing, .contentIo = param.contentIo); httpRequestFree(request); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /*********************************************************************************************************************************** General function for listing files to be used by other list routines Based on the documentation at https://learn.microsoft.com/en-us/rest/api/storageservices/list-blobs ***********************************************************************************************************************************/ static void storageAzureListInternal( StorageAzure *const this, const String *const path, const StorageInfoLevel level, const String *const expression, const bool recurse, const time_t targetTime, const StorageListCallback callback, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_PARAM(BOOL, recurse); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, callback); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build the base prefix by stripping off the initial / const String *const basePrefix = strSize(path) == 1 ? EMPTY_STR : strNewFmt("%s/", strZ(strSub(path, 1))); // Get the expression prefix when possible to limit initial results const String *const expressionPrefix = regExpPrefix(expression); // If there is an expression prefix then use it to build the query prefix, otherwise query prefix is base prefix const String *queryPrefix; if (expressionPrefix == NULL) queryPrefix = basePrefix; else { if (strEmpty(basePrefix)) queryPrefix = expressionPrefix; else queryPrefix = strNewFmt("%s%s", strZ(basePrefix), strZ(expressionPrefix)); } // Create query HttpQuery *const query = httpQueryNewP(); // Add the delimiter to not recurse if (!recurse) httpQueryAdd(query, AZURE_QUERY_DELIMITER_STR, FSLASH_STR); // Add resource type httpQueryAdd(query, AZURE_QUERY_RESTYPE_STR, AZURE_QUERY_VALUE_CONTAINER_STR); // Add list comp httpQueryAdd(query, AZURE_QUERY_COMP_STR, AZURE_QUERY_VALUE_LIST_STR); // Don't specify empty prefix because it is the default if (!strEmpty(queryPrefix)) httpQueryAdd(query, AZURE_QUERY_PREFIX_STR, queryPrefix); // Add versions if (targetTime != 0) httpQueryAdd(query, AZURE_QUERY_INCLUDE_STR, AZURE_QUERY_VALUE_VERSIONS_STR); // Store last info so it can be updated across requests for versioning String *const nameLast = strNew(); String *const versionIdLast = strNew(); StorageInfo infoLast = {.level = level, .name = nameLast}; if (targetTime != 0) infoLast.versionId = versionIdLast; // Loop as long as a continuation marker returned HttpRequest *request = NULL; do { // Use an inner mem context here because we could potentially be retrieving millions of files so it is a good idea to // free memory at regular intervals MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *response = NULL; // If there is an outstanding async request then wait for the response if (request != NULL) { response = storageAzureResponseP(request); httpRequestFree(request); request = NULL; } // Else get the response immediately from a sync request else response = storageAzureRequestP(this, HTTP_VERB_GET_STR, .query = query); const XmlNode *const xmlRoot = xmlDocumentRoot(xmlDocumentNewBuf(httpResponseContent(response))); // If a continuation marker exists then send an async request to get more data const String *const continuationMarker = xmlNodeContent( xmlNodeChild(xmlRoot, AZURE_XML_TAG_NEXT_MARKER_STR, false)); if (!strEq(continuationMarker, EMPTY_STR)) { httpQueryPut(query, AZURE_QUERY_MARKER_STR, continuationMarker); // Store request in the outer temp context MEM_CONTEXT_PRIOR_BEGIN() { request = storageAzureRequestAsyncP(this, HTTP_VERB_GET_STR, .query = query); } MEM_CONTEXT_PRIOR_END(); } // Get prefix list const XmlNode *const blobs = xmlNodeChild(xmlRoot, AZURE_XML_TAG_BLOBS_STR, true); const XmlNodeList *const blobPrefixList = xmlNodeChildList(blobs, AZURE_XML_TAG_BLOB_PREFIX_STR); for (unsigned int blobPrefixIdx = 0; blobPrefixIdx < xmlNodeLstSize(blobPrefixList); blobPrefixIdx++) { const XmlNode *const subPathNode = xmlNodeLstGet(blobPrefixList, blobPrefixIdx); // Get path name StorageInfo info = { .level = level, .name = xmlNodeContent(xmlNodeChild(subPathNode, AZURE_XML_TAG_NAME_STR, true)), .exists = true, }; // Strip off base prefix and final / info.name = strSubN(info.name, strSize(basePrefix), strSize(info.name) - strSize(basePrefix) - 1); // Add type info if requested if (level >= storageInfoLevelType) info.type = storageTypePath; // Callback with info callback(callbackData, &info); } // Get file list const XmlNodeList *const fileList = xmlNodeChildList(blobs, AZURE_XML_TAG_BLOB_STR); for (unsigned int fileIdx = 0; fileIdx < xmlNodeLstSize(fileList); fileIdx++) { const XmlNode *const fileNode = xmlNodeLstGet(fileList, fileIdx); const XmlNode *const property = xmlNodeChild(fileNode, AZURE_XML_TAG_PROPERTIES_STR, true); // Get file name and strip off the base prefix when present const String *name = xmlNodeContent(xmlNodeChild(fileNode, AZURE_XML_TAG_NAME_STR, true)); if (!strEmpty(basePrefix)) name = strSub(name, strSize(basePrefix)); // Return info for last file if new file if (infoLast.exists && !strEq(name, nameLast)) { callback(callbackData, &infoLast); infoLast.exists = false; } // If targeting by time exclude versions that are newer than targetTime. Note that the API does not provide // reliable delete markers so the filtering will also show files that have been deleted rather than replaced // with a new version. The problem with the delete markers is that Creation-Time/Last-Modified are set equal to // the times in the last version so we don't know when the file was deleted. It might be possible to use // VersionId for this purpose, since it appears to be a timestamp, but the field is described as "opaque" in the // documentation so it does not seem to be a good idea to use it. if (targetTime != 0) { infoLast.timeModified = httpDateToTime( xmlNodeContent(xmlNodeChild(property, AZURE_XML_TAG_LAST_MODIFIED_STR, true))); // Skip this version if it is newer than the time limit if (infoLast.timeModified > targetTime) continue; } // Update last name and set exists strCat(strTrunc(nameLast), name); infoLast.exists = true; // Add basic info if requested (no need to add type info since file is default type) if (level >= storageInfoLevelBasic) { infoLast.size = cvtZToUInt64( strZ(xmlNodeContent(xmlNodeChild(property, AZURE_XML_TAG_CONTENT_LENGTH_STR, true)))); if (targetTime == 0) { infoLast.timeModified = httpDateToTime( xmlNodeContent(xmlNodeChild(property, AZURE_XML_TAG_LAST_MODIFIED_STR, true))); } else { strCat( strTrunc(versionIdLast), xmlNodeContent(xmlNodeChild(fileNode, AZURE_XML_TAG_VERSION_ID_STR, true))); } } } } MEM_CONTEXT_TEMP_END(); } while (request != NULL); // Callback with last info if it exists if (infoLast.exists) callback(callbackData, &infoLast); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static StorageInfo storageAzureInfo(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); // Attempt to get file info HttpResponse *const httpResponse = storageAzureRequestP(this, HTTP_VERB_HEAD_STR, .path = file, .allowMissing = true); // Does the file exist? StorageInfo result = {.level = level, .exists = httpResponseCodeOk(httpResponse)}; // Add basic level info if requested and the file exists (no need to add type info since file is default type) if (result.level >= storageInfoLevelBasic && result.exists) { result.size = cvtZToUInt64(strZ(httpHeaderGet(httpResponseHeader(httpResponse), HTTP_HEADER_CONTENT_LENGTH_STR))); result.timeModified = httpDateToTime(httpHeaderGet(httpResponseHeader(httpResponse), HTTP_HEADER_LAST_MODIFIED_STR)); } httpResponseFree(httpResponse); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storageAzureListCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); storageLstAdd(callbackData, info); FUNCTION_TEST_RETURN_VOID(); } static StorageList * storageAzureList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_PARAM(TIME, param.targetTime); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); StorageList *const result = storageLstNew(level); storageAzureListInternal(this, path, level, param.expression, false, param.targetTime, storageAzureListCallback, result); FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void * storageAzureNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN(STORAGE_READ_AZURE, storageReadAzureNew(this, file, param.offset, param.limit, param.versionId)); } /**********************************************************************************************************************************/ static void * storageAzureNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.createPath); ASSERT(param.truncate); ASSERT(param.user == NULL); ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); FUNCTION_LOG_RETURN(STORAGE_WRITE_AZURE, storageWriteAzureNew(this, file, this->fileId++, this->blockSize)); } /**********************************************************************************************************************************/ typedef struct StorageAzurePathRemoveData { StorageAzure *this; // Storage object MemContext *memContext; // Mem context to create requests in HttpRequest *request; // Async remove request List *requestContentList; // Content list for async request List *contentList; // Content list currently being built const String *path; // Root path of remove } StorageAzurePathRemoveData; static void storageAzurePathRemoveInternal(StorageAzurePathRemoveData *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_END(); ASSERT(data != NULL); ASSERT(data->this != NULL); // Get response for async request if (data->request != NULL) { MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *const response = storageAzureResponseP(data->request); HttpResponseMulti *const responseMulti = httpResponseMultiNew( httpResponseContent(response), httpHeaderGet(httpResponseHeader(response), HTTP_HEADER_CONTENT_TYPE_STR)); // Loop through all response parts. Parts are mapped to the original request by ordinal position since the service // returns one response part per sub-request in request order. The echoed content-id is intentionally not used because // Azure omits it on some error responses, which would otherwise make a failed part impossible to retry. unsigned int partIdx = 0; HttpResponse *responsePart = httpResponseMultiNext(responseMulti); CHECK(FormatError, responsePart != NULL, "at least one response part is required"); do { // If not OK and not missing then retry if (!httpResponseCodeOk(responsePart) && httpResponseCode(responsePart) != HTTP_RESPONSE_CODE_NOT_FOUND) { // Get the original request for this part by position CHECK_FMT( FormatError, partIdx < lstSize(data->requestContentList), "response part %u is out of range", partIdx); const StorageAzureRequestPart *const content = lstGet(data->requestContentList, partIdx); // Retry remove statInc(AZURE_STAT_REMOVE_BATCH_RETRY_STR); httpResponseFree(storageAzureRequestP(data->this, content->verb, .path = content->path, .allowMissing = true)); } else statInc(AZURE_STAT_REMOVE_BATCH_PART_STR); httpResponseFree(responsePart); responsePart = httpResponseMultiNext(responseMulti); partIdx++; } while (responsePart != NULL); } MEM_CONTEXT_TEMP_END(); // Free request httpRequestFree(data->request); data->request = NULL; // Free content list lstFree(data->requestContentList); } // Send new async request if there is more to remove if (data->contentList != NULL) { statInc(AZURE_STAT_REMOVE_BATCH_STR); MEM_CONTEXT_BEGIN(data->memContext) { data->request = storageAzureRequestAsyncP( data->this, HTTP_VERB_POST_STR, .query = httpQueryAdd( httpQueryAdd(httpQueryNewP(), AZURE_QUERY_COMP_STR, AZURE_QUERY_BATCH_STR), AZURE_QUERY_RESTYPE_STR, AZURE_QUERY_VALUE_CONTAINER_STR), .contentList = data->contentList); } MEM_CONTEXT_END(); // Store the content list for use in error handling data->requestContentList = data->contentList; data->contentList = NULL; } FUNCTION_TEST_RETURN_VOID(); } static void storageAzurePathRemoveCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); // Only delete files since paths don't really exist if (info->type == storageTypeFile) { StorageAzurePathRemoveData *const data = callbackData; if (data->contentList == NULL) { MEM_CONTEXT_BEGIN(data->memContext) { data->contentList = lstNewP(sizeof(StorageAzureRequestPart)); } MEM_CONTEXT_END(); } MEM_CONTEXT_OBJ_BEGIN(data->contentList) { const StorageAzureRequestPart content = { .verb = HTTP_VERB_DELETE_STR, .path = strNewFmt("%s/%s", strZ(data->path), strZ(info->name)), }; lstAdd(data->contentList, &content); } MEM_CONTEXT_OBJ_END(); if (lstSize(data->contentList) == data->this->deleteMax) storageAzurePathRemoveInternal(data); } FUNCTION_TEST_RETURN_VOID(); } static bool storageAzurePathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, recurse); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { StorageAzurePathRemoveData data = { .this = this, .memContext = memContextCurrent(), .path = strEq(path, FSLASH_STR) ? EMPTY_STR : path, }; MEM_CONTEXT_TEMP_BEGIN() { storageAzureListInternal(this, path, storageInfoLevelType, NULL, true, 0, storageAzurePathRemoveCallback, &data); // Call if there is more to be removed if (data.contentList != NULL) storageAzurePathRemoveInternal(&data); // Check response on last async request storageAzurePathRemoveInternal(&data); } MEM_CONTEXT_TEMP_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, true); } /**********************************************************************************************************************************/ static void storageAzureRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageAzure); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_AZURE, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); statInc(AZURE_STAT_REMOVE_STR); httpResponseFree(storageAzureRequestP(this, HTTP_VERB_DELETE_STR, file, .allowMissing = true)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceAzure = { .feature = 1 << storageFeatureVersioning | 1 << storageFeatureReadRetry, .info = storageAzureInfo, .list = storageAzureList, .newRead = storageAzureNewRead, .newWrite = storageAzureNewWrite, .pathRemove = storageAzurePathRemove, .remove = storageAzureRemove, }; FN_EXTERN Storage * storageAzureNew( const String *const path, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *const container, const String *const account, const StorageAzureKeyType keyType, const String *const key, const size_t blockSize, const KeyValue *const tag, const String *const endpoint, const StorageAzureUriStyle uriStyle, const unsigned int port, const TimeMSec timeout, const HttpProtocolType protocolType, const bool verifyPeer, const String *const caFile, const String *const caPath) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(STRING, container); FUNCTION_TEST_PARAM(STRING, account); FUNCTION_LOG_PARAM(STRING_ID, keyType); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_LOG_PARAM(SIZE, blockSize); FUNCTION_LOG_PARAM(KEY_VALUE, tag); FUNCTION_LOG_PARAM(STRING, endpoint); FUNCTION_LOG_PARAM(ENUM, uriStyle); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_PARAM(ENUM, protocolType); FUNCTION_LOG_PARAM(BOOL, verifyPeer); FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, caPath); FUNCTION_LOG_END(); ASSERT(path != NULL); ASSERT(container != NULL); ASSERT(account != NULL); ASSERT(endpoint != NULL); ASSERT(keyType == storageAzureKeyTypeAuto || key != NULL); ASSERT(blockSize != 0); OBJ_NEW_BEGIN(StorageAzure, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageAzure) { .interface = storageInterfaceAzure, .container = strDup(container), .account = strDup(account), .blockSize = blockSize, .host = uriStyle == storageAzureUriStyleHost ? strNewFmt("%s.%s", strZ(account), strZ(endpoint)) : strDup(endpoint), .pathPrefix = uriStyle == storageAzureUriStyleHost ? strNewFmt("/%s", strZ(container)) : strNewFmt("/%s/%s", strZ(account), strZ(container)), .deleteMax = STORAGE_AZURE_DELETE_MAX, .keyType = keyType, }; // Create tag query string if (tag != NULL) { HttpQuery *const query = httpQueryNewP(.kv = tag); this->tag = httpQueryRenderP(query); httpQueryFree(query); } // Initialization by key type switch (keyType) { // Create authentication client case storageAzureKeyTypeAuto: this->accessToken = strNew(); this->credHost = AZURE_CREDENTIAL_HOST_STR; this->credHttpClient = httpClientNew( sckClientNew(this->credHost, AZURE_CREDENTIAL_PORT, timeout, timeout), timeout); break; // Store shared key case storageAzureKeyTypeShared: this->sharedKey = bufNewDecode(encodingBase64, key); break; // Parse sas query case storageAzureKeyTypeSas: this->sasKey = httpQueryNewStr(key); break; } // Create the http client used to service requests. Use plain socket for HTTP, TLS for HTTPS. IoClient *ioClient; if (protocolType == httpProtocolTypeHttp) ioClient = sckClientNew(this->host, port, timeout, timeout); else { ioClient = tlsClientNewP( sckClientNew(this->host, port, timeout, timeout), this->host, timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath); } this->httpClient = httpClientNew(ioClient, timeout); // Create list of redacted headers this->headerRedactList = strLstNew(); strLstAdd(this->headerRedactList, HTTP_HEADER_AUTHORIZATION_STR); strLstAdd(this->headerRedactList, HTTP_HEADER_DATE_STR); // Create list of redacted query keys this->queryRedactList = strLstNew(); strLstAdd(this->queryRedactList, AZURE_QUERY_SIG_STR); // Generate starting file id cryptoRandomBytes((uint8_t *)&this->fileId, sizeof(this->fileId)); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew(STORAGE_AZURE_TYPE, path, 0, 0, write, targetTime, pathExpressionFunction, this, this->interface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/storage.h0000664000175100017510000000442415240547630020615 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_STORAGE_H #define STORAGE_AZURE_STORAGE_H #include "common/io/http/url.h" #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_AZURE_TYPE STRID5("azure", 0x5957410) /*********************************************************************************************************************************** Key type ***********************************************************************************************************************************/ typedef enum { storageAzureKeyTypeShared, storageAzureKeyTypeSas, storageAzureKeyTypeAuto, } StorageAzureKeyType; /*********************************************************************************************************************************** URI style ***********************************************************************************************************************************/ typedef enum { storageAzureUriStyleHost, storageAzureUriStylePath, } StorageAzureUriStyle; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageAzureNew( const String *path, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *container, const String *account, StorageAzureKeyType keyType, const String *key, size_t blockSize, const KeyValue *tag, const String *endpoint, StorageAzureUriStyle uriStyle, unsigned int port, TimeMSec timeout, HttpProtocolType protocolType, bool verifyPeer, const String *caFile, const String *caPath); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/storage.intern.h0000664000175100017510000001327015240547630022112 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_STORAGE_INTERN_H #define STORAGE_AZURE_STORAGE_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageAzure StorageAzure; #include "common/io/http/request.h" #include "storage/azure/storage.h" /*********************************************************************************************************************************** Azure query tokens ***********************************************************************************************************************************/ #define AZURE_QUERY_COMP "comp" STRING_DECLARE(AZURE_QUERY_COMP_STR); #define AZURE_QUERY_RESTYPE "restype" STRING_DECLARE(AZURE_QUERY_RESTYPE_STR); #define AZURE_QUERY_VALUE_CONTAINER "container" STRING_DECLARE(AZURE_QUERY_VALUE_CONTAINER_STR); /*********************************************************************************************************************************** Multi-Part request data ***********************************************************************************************************************************/ typedef struct StorageAzureRequestPart { const String *path; // Request path const String *verb; // Verb (GET, PUT, etc) } StorageAzureRequestPart; /*********************************************************************************************************************************** Perform an Azure Request ***********************************************************************************************************************************/ // Perform async request typedef struct StorageAzureRequestAsyncParam { VAR_PARAM_HEADER; const String *path; // Request path const HttpHeader *header; // Request headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content const List *contentList; // Request content part list bool tag; // Add tags when available? } StorageAzureRequestAsyncParam; #define storageAzureRequestAsyncP(this, verb, ...) \ storageAzureRequestAsync(this, verb, (StorageAzureRequestAsyncParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpRequest *storageAzureRequestAsync(StorageAzure *this, const String *verb, StorageAzureRequestAsyncParam param); // Get async response typedef struct StorageAzureResponseParam { VAR_PARAM_HEADER; bool allowMissing; // Allow missing files (caller can check response code) bool contentIo; // Is IoRead interface required to read content? } StorageAzureResponseParam; #define storageAzureResponseP(request, ...) \ storageAzureResponse(request, (StorageAzureResponseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageAzureResponse(HttpRequest *request, StorageAzureResponseParam param); typedef struct StorageAzureRequestParam { VAR_PARAM_HEADER; const String *path; // Request path const HttpHeader *header; // Request headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content const List *contentList; // Request content part list bool allowMissing; // Allow missing files (caller can check response code) bool contentIo; // Is IoRead interface required to read content? bool tag; // Add tags when available? } StorageAzureRequestParam; #define storageAzureRequestP(this, verb, ...) \ storageAzureRequest(this, verb, (StorageAzureRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageAzureRequest(StorageAzure *this, const String *verb, StorageAzureRequestParam param); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_AZURE_TYPE \ StorageAzure * #define FUNCTION_LOG_STORAGE_AZURE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageAzure", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/write.c0000664000175100017510000002700415240547630020275 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage File Write ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/xml.h" #include "storage/azure/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Block defaults based on limits at https://learn.microsoft.com/en-us/rest/api/storageservices/put-blob ***********************************************************************************************************************************/ #define STORAGE_AZURE_SPLIT_DEFAULT 257 #define STORAGE_AZURE_SPLIT_MAX 9505 /*********************************************************************************************************************************** Azure HTTP headers ***********************************************************************************************************************************/ STRING_STATIC(AZURE_HEADER_BLOB_TYPE_STR, "x-ms-blob-type"); STRING_STATIC(AZURE_HEADER_VALUE_BLOCK_BLOB_STR, "BlockBlob"); /*********************************************************************************************************************************** Azure query tokens ***********************************************************************************************************************************/ STRING_STATIC(AZURE_QUERY_BLOCK_ID_STR, "blockid"); STRING_STATIC(AZURE_QUERY_VALUE_BLOCK_STR, "block"); STRING_STATIC(AZURE_QUERY_VALUE_BLOCK_LIST_STR, "blocklist"); /*********************************************************************************************************************************** XML tags ***********************************************************************************************************************************/ STRING_STATIC(AZURE_XML_TAG_BLOCK_LIST_STR, "BlockList"); STRING_STATIC(AZURE_XML_TAG_UNCOMMITTED_STR, "Uncommitted"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteAzure { const StorageWriteInterface *interface; // Interface StorageAzure *storage; // Storage that created this object const String *name; // File name HttpRequest *request; // Async block upload request uint64_t fileId; // Id to used to make file block identifiers unique size_t blockSize; // Size of blocks for multi-block upload Buffer *blockBuffer; // Block buffer (stores data until blockSize is reached) StringList *blockIdList; // List of uploaded block ids }; /*********************************************************************************************************************************** Flush bytes to upload block ***********************************************************************************************************************************/ static void storageWriteAzureBlock(StorageWriteAzure *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If there is an outstanding async request then wait for the response. Since the part id has already been stored there is // nothing to do except make sure the request did not error. if (this->request != NULL) { httpResponseFree(storageAzureResponseP(this->request)); httpRequestFree(this->request); this->request = NULL; } FUNCTION_LOG_RETURN_VOID(); } static void storageWriteAzureBlockAsync(StorageWriteAzure *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->blockBuffer != NULL); ASSERT(bufSize(this->blockBuffer) > 0); MEM_CONTEXT_TEMP_BEGIN() { // Complete prior async request, if any storageWriteAzureBlock(this); // Create the block id list if (this->blockIdList == NULL) { MEM_CONTEXT_OBJ_BEGIN(this) { this->blockIdList = strLstNew(); } MEM_CONTEXT_OBJ_END(); } // Generate block id. Combine the block number with the provided file id to create a (hopefully) unique block id that won't // overlap with any other process. This is to prevent another process from overwriting our blocks. If two processes are // writing against the same file then there may be problems anyway but we need to at least ensure the result is consistent, // i.e. we get all of one file or all of the other depending on who writes last. const String *const blockId = strNewFmt("%016" PRIX64 "x%07u", this->fileId, strLstSize(this->blockIdList)); // Upload the block and add to block list HttpQuery *const query = httpQueryNewP(); httpQueryAdd(query, AZURE_QUERY_COMP_STR, AZURE_QUERY_VALUE_BLOCK_STR); httpQueryAdd(query, AZURE_QUERY_BLOCK_ID_STR, blockId); MEM_CONTEXT_OBJ_BEGIN(this) { this->request = storageAzureRequestAsyncP( this->storage, HTTP_VERB_PUT_STR, .path = this->name, .query = query, .content = this->blockBuffer); } MEM_CONTEXT_OBJ_END(); strLstAdd(this->blockIdList, blockId); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to internal buffer ***********************************************************************************************************************************/ static void storageWriteAzure(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_AZURE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Continue until the write buffer has been exhausted size_t bytesTotal = 0; do { // Resize block buffer. This must be done in the loop because block size grows as blocks are written. storageWriteChunkBufferResize(buffer, bytesTotal, this->blockBuffer, this->blockSize); // Copy as many bytes as possible into the block buffer const size_t bytesNext = bufRemains(this->blockBuffer) > bufUsed(buffer) - bytesTotal ? bufUsed(buffer) - bytesTotal : bufRemains(this->blockBuffer); bufCatSub(this->blockBuffer, buffer, bytesTotal, bytesNext); bytesTotal += bytesNext; // If the block buffer is full then write it if (bufUsed(this->blockBuffer) == this->blockSize) { storageWriteAzureBlockAsync(this); bufUsedZero(this->blockBuffer); this->blockSize = storageWriteChunkSize( this->blockSize, STORAGE_AZURE_SPLIT_DEFAULT, STORAGE_AZURE_SPLIT_MAX, strLstSize(this->blockIdList)); } } while (bytesTotal != bufUsed(buffer)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteAzureClose(THIS_VOID) { THIS(StorageWriteAzure); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_AZURE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->blockBuffer != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // If a multi-block upload was started we need to finish that way if (this->blockIdList != NULL) { // If there is anything left in the block buffer then write it if (!bufEmpty(this->blockBuffer)) storageWriteAzureBlockAsync(this); // Complete prior async request, if any storageWriteAzureBlock(this); // Generate the xml block list XmlDocument *const blockXml = xmlDocumentNew(AZURE_XML_TAG_BLOCK_LIST_STR); for (unsigned int blockIdx = 0; blockIdx < strLstSize(this->blockIdList); blockIdx++) { xmlNodeContentSet( xmlNodeAdd(xmlDocumentRoot(blockXml), AZURE_XML_TAG_UNCOMMITTED_STR), strLstGet(this->blockIdList, blockIdx)); } // Finalize the multi-block upload storageAzureRequestP( this->storage, HTTP_VERB_PUT_STR, .path = this->name, .query = httpQueryAdd(httpQueryNewP(), AZURE_QUERY_COMP_STR, AZURE_QUERY_VALUE_BLOCK_LIST_STR), .content = xmlDocumentBuf(blockXml), .tag = true); } // Else upload all the data in a single block else { storageAzureRequestP( this->storage, HTTP_VERB_PUT_STR, .path = this->name, httpHeaderAdd(httpHeaderNew(NULL), AZURE_HEADER_BLOB_TYPE_STR, AZURE_HEADER_VALUE_BLOCK_BLOB_STR), .content = this->blockBuffer, .tag = true); } bufFree(this->blockBuffer); this->blockBuffer = NULL; } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteAzureInterface = { .close = storageWriteAzureClose, .write = storageWriteAzure, }; FN_EXTERN StorageWriteAzure * storageWriteAzureNew(StorageAzure *const storage, const String *const name, const uint64_t fileId, const size_t blockSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_AZURE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, fileId); FUNCTION_LOG_PARAM(UINT64, blockSize); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageWriteAzure, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWriteAzure) { .interface = &storageWriteAzureInterface, .storage = storage, .name = strDup(name), .fileId = fileId, .blockSize = blockSize, .blockBuffer = bufNew(0), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_AZURE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/azure/write.h0000664000175100017510000000343315240547630020302 0ustar00runnerrunner/*********************************************************************************************************************************** Azure Storage File Write ***********************************************************************************************************************************/ #ifndef STORAGE_AZURE_WRITE_H #define STORAGE_AZURE_WRITE_H #include "storage/azure/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteAzure StorageWriteAzure; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteAzure *storageWriteAzureNew(StorageAzure *storage, const String *name, uint64_t fileId, size_t blockSize); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_AZURE_TYPE \ StorageWriteAzure * #define FUNCTION_LOG_STORAGE_WRITE_AZURE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteAzure", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/cifs/0000775000175100017510000000000015240547630016572 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/cifs/helper.c0000664000175100017510000000227115240547630020217 0ustar00runnerrunner/*********************************************************************************************************************************** CIFS Storage Helper ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "config/config.h" #include "storage/cifs/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageCifsHelper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_CIFS_TYPE); Storage *const result = storageCifsNew( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, write, pathExpressionCallback); FUNCTION_LOG_RETURN(STORAGE, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/cifs/helper.h0000664000175100017510000000236515240547630020230 0ustar00runnerrunner/*********************************************************************************************************************************** CIFS Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_CIFS_STORAGE_HELPER_H #define STORAGE_CIFS_STORAGE_HELPER_H #include "storage/cifs/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageCifsHelper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_CIFS_HELPER {.type = STORAGE_CIFS_TYPE, .helper = storageCifsHelper} #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/cifs/storage.c0000664000175100017510000000232215240547630020401 0ustar00runnerrunner/*********************************************************************************************************************************** CIFS Storage ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/regExp.h" #include "storage/cifs/storage.h" #include "storage/posix/storage.intern.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageCifsNew( const String *const path, const mode_t modeFile, const mode_t modePath, const bool write, StoragePathExpressionCallback pathExpressionFunction) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN( STORAGE, storagePosixNewInternal(STORAGE_CIFS_TYPE, path, modeFile, modePath, write, pathExpressionFunction, false, false)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/cifs/storage.h0000664000175100017510000000225015240547630020406 0ustar00runnerrunner/*********************************************************************************************************************************** CIFS Storage ***********************************************************************************************************************************/ #ifndef STORAGE_CIFS_STORAGE_H #define STORAGE_CIFS_STORAGE_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_CIFS_TYPE STRID5("cifs", 0x999230) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageCifsNew( const String *path, mode_t modeFile, mode_t modePath, bool write, StoragePathExpressionCallback pathExpressionFunction); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/0000775000175100017510000000000015240547630016422 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/helper.c0000664000175100017510000000344615240547630020054 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Helper ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "config/config.h" #include "storage/gcs/helper.h" #include "storage/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageGcsHelper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_GCS_TYPE); Storage *const result = storageGcsNew( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), write, storageRepoTargetTime(), pathExpressionCallback, cfgOptionIdxStr(cfgOptRepoGcsBucket, repoIdx), (StorageGcsKeyType)cfgOptionIdxSeq(cfgOptRepoGcsKeyType, repoIdx), cfgOptionIdxStrNull(cfgOptRepoGcsKey, repoIdx), (size_t)cfgOptionIdxUInt64(cfgOptRepoStorageUploadChunkSize, repoIdx), cfgOptionIdxKvNull(cfgOptRepoStorageTag, repoIdx), cfgOptionIdxStr(cfgOptRepoGcsEndpoint, repoIdx), ioTimeoutMs(), cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaFile, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaPath, repoIdx), cfgOptionIdxStrNull(cfgOptRepoGcsUserProject, repoIdx)); FUNCTION_LOG_RETURN(STORAGE, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/helper.h0000664000175100017510000000235615240547630020060 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_STORAGE_HELPER_H #define STORAGE_GCS_STORAGE_HELPER_H #include "storage/gcs/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageGcsHelper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_GCS_HELPER {.type = STORAGE_GCS_TYPE, .helper = storageGcsHelper} #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/read.c0000664000175100017510000001451115240547630017503 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Read ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/client.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" #include "storage/gcs/read.h" #include "storage/read.h" /*********************************************************************************************************************************** GCS query tokens ***********************************************************************************************************************************/ STRING_STATIC(GCS_QUERY_ALT_STR, "alt"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageReadGcs { const StorageReadInterface *interface; // Interface StorageGcs *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) const String *versionId; // Version id (NULL for most recent) HttpResponse *httpResponse; // HTTP response }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadGcsOpen(THIS_VOID) { THIS(StorageReadGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_GCS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse == NULL); // Request the file MEM_CONTEXT_OBJ_BEGIN(this) { HttpQuery *const query = httpQueryAdd(httpQueryNewP(), GCS_QUERY_ALT_STR, GCS_QUERY_MEDIA_STR); if (this->versionId != NULL) httpQueryAdd(query, varStr(GCS_JSON_GENERATION_VAR), this->versionId); this->httpResponse = storageGcsRequestP( this->storage, HTTP_VERB_GET_STR, .object = this->name, .header = httpHeaderPutRange(httpHeaderNew(NULL), this->offset, this->limit), .allowMissing = true, .contentIo = true, .query = query); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(BOOL, httpResponseCodeOk(this->httpResponse)); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadGcs(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_GCS, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL && this->httpResponse != NULL); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); FUNCTION_LOG_RETURN(SIZE, ioRead(httpResponseIoRead(this->httpResponse), buffer)); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadGcsEof(THIS_VOID) { THIS(StorageReadGcs); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_GCS, this); FUNCTION_TEST_END(); ASSERT(this != NULL && this->httpResponse != NULL); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); FUNCTION_TEST_RETURN(BOOL, ioReadEof(httpResponseIoRead(this->httpResponse))); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadGcsClose(THIS_VOID) { THIS(StorageReadGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_GCS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse != NULL); httpResponseFree(this->httpResponse); this->httpResponse = NULL; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadGcsInterface = { .close = storageReadGcsClose, .eof = storageReadGcsEof, .open = storageReadGcsOpen, .read = storageReadGcs, }; FN_EXTERN StorageReadGcs * storageReadGcsNew( StorageGcs *const storage, const String *const name, const uint64_t offset, const Variant *const limit, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_GCS, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadGcs, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadGcs) { .interface = &storageReadGcsInterface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), .versionId = strDup(versionId), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_GCS, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/read.h0000664000175100017510000000343015240547630017506 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_READ_H #define STORAGE_GCS_READ_H #include "storage/gcs/storage.intern.h" #include "storage/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadGcs StorageReadGcs; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadGcs *storageReadGcsNew( StorageGcs *storage, const String *name, uint64_t offset, const Variant *limit, const String *versionId); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_GCS_TYPE \ StorageReadGcs * #define FUNCTION_LOG_STORAGE_READ_GCS_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadGcs", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/storage.c0000664000175100017510000015160515240547630020242 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/crypto/common.h" #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/http/client.h" #include "common/io/http/common.h" #include "common/io/http/url.h" #include "common/io/socket/client.h" #include "common/io/tls/client.h" #include "common/log.h" #include "common/regExp.h" #include "common/stat.h" #include "common/type/json.h" #include "common/type/object.h" #include "storage/gcs/read.h" #include "storage/gcs/write.h" #include "storage/posix/storage.h" /*********************************************************************************************************************************** Defaults ***********************************************************************************************************************************/ #define STORAGE_GCS_DELETE_MAX 100 /*********************************************************************************************************************************** HTTP headers ***********************************************************************************************************************************/ STRING_EXTERN(GCS_HEADER_UPLOAD_ID_STR, GCS_HEADER_UPLOAD_ID); STRING_STATIC(GCS_HEADER_METADATA_FLAVOR_STR, "metadata-flavor"); STRING_STATIC(GCS_HEADER_GOOGLE_STR, "Google"); /*********************************************************************************************************************************** Query tokens ***********************************************************************************************************************************/ STRING_STATIC(GCS_QUERY_DELIMITER_STR, "delimiter"); STRING_EXTERN(GCS_QUERY_FIELDS_STR, GCS_QUERY_FIELDS); STRING_EXTERN(GCS_QUERY_MEDIA_STR, GCS_QUERY_MEDIA); STRING_EXTERN(GCS_QUERY_NAME_STR, GCS_QUERY_NAME); STRING_STATIC(GCS_QUERY_PAGE_TOKEN_STR, "pageToken"); STRING_STATIC(GCS_QUERY_PREFIX_STR, "prefix"); STRING_EXTERN(GCS_QUERY_UPLOAD_ID_STR, GCS_QUERY_UPLOAD_ID); STRING_STATIC(GCS_QUERY_VERSIONS_STR, "versions"); STRING_STATIC(GCS_QUERY_USER_PROJECT_STR, "userProject"); /*********************************************************************************************************************************** JSON tokens ***********************************************************************************************************************************/ VARIANT_STRDEF_STATIC(GCS_JSON_ACCESS_TOKEN_VAR, "access_token"); VARIANT_STRDEF_STATIC(GCS_JSON_CLIENT_EMAIL_VAR, "client_email"); VARIANT_STRDEF_STATIC(GCS_JSON_ERROR_VAR, "error"); VARIANT_STRDEF_STATIC(GCS_JSON_ERROR_DESCRIPTION_VAR, "error_description"); VARIANT_STRDEF_STATIC(GCS_JSON_EXPIRES_IN_VAR, "expires_in"); #define GCS_JSON_ITEMS "items" VARIANT_STRDEF_STATIC(GCS_JSON_ITEMS_VAR, GCS_JSON_ITEMS); VARIANT_STRDEF_EXTERN(GCS_JSON_GENERATION_VAR, GCS_JSON_GENERATION); VARIANT_STRDEF_EXTERN(GCS_JSON_MD5_HASH_VAR, GCS_JSON_MD5_HASH); VARIANT_STRDEF_EXTERN(GCS_JSON_NAME_VAR, GCS_JSON_NAME); #define GCS_JSON_NEXT_PAGE_TOKEN "nextPageToken" VARIANT_STRDEF_STATIC(GCS_JSON_NEXT_PAGE_TOKEN_VAR, GCS_JSON_NEXT_PAGE_TOKEN); #define GCS_JSON_PREFIXES "prefixes" VARIANT_STRDEF_STATIC(GCS_JSON_PREFIXES_VAR, GCS_JSON_PREFIXES); VARIANT_STRDEF_STATIC(GCS_JSON_PRIVATE_KEY_VAR, "private_key"); VARIANT_STRDEF_EXTERN(GCS_JSON_SIZE_VAR, GCS_JSON_SIZE); VARIANT_STRDEF_STATIC(GCS_JSON_TOKEN_TYPE_VAR, "token_type"); VARIANT_STRDEF_STATIC(GCS_JSON_TOKEN_URI_VAR, "token_uri"); #define GCS_JSON_UPDATED "updated" VARIANT_STRDEF_STATIC(GCS_JSON_UPDATED_VAR, GCS_JSON_UPDATED); /*********************************************************************************************************************************** Statistics constants ***********************************************************************************************************************************/ STRING_STATIC(GCS_STAT_REMOVE_STR, "gcs.rm"); STRING_STATIC(GCS_STAT_REMOVE_BATCH_STR, "gcs.rm.batch"); STRING_STATIC(GCS_STAT_REMOVE_BATCH_PART_STR, "gcs.rm.batch.part"); STRING_STATIC(GCS_STAT_REMOVE_BATCH_RETRY_STR, "gcs.rm.batch.retry"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageGcs { STORAGE_COMMON_MEMBER; HttpClient *httpClient; // Http client to service requests StringList *headerRedactList; // List of headers to redact from logging StringList *queryRedactList; // List of query keys to redact from logging bool write; // Storage is writable const String *bucket; // Bucket to store data in const String *endpoint; // Endpoint size_t chunkSize; // Block size for resumable upload unsigned int deleteMax; // Maximum objects that can be deleted in one request const Buffer *tag; // Tags to be applied to objects const String *userProject; // Project ID StorageGcsKeyType keyType; // Auth key type const String *key; // Key (value depends on key type) String *token; // Token time_t tokenTimeExpire; // Token expiration time (if service auth) HttpUrl *authUrl; // URL for authentication server HttpClient *authClient; // Client to service auth requests }; /*********************************************************************************************************************************** Parse HTTP JSON response containing an authentication token and expiration Note that the function is intended to run directly in the caller's mem context and results will be placed in the caller's prior mem context. ***********************************************************************************************************************************/ typedef struct { String *tokenType; String *token; time_t timeExpire; } StorageGcsAuthTokenResult; static StorageGcsAuthTokenResult storageGcsAuthToken(HttpRequest *const request, const time_t timeBegin) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(HTTP_REQUEST, request); FUNCTION_TEST_PARAM(TIME, timeBegin); FUNCTION_TEST_END(); FUNCTION_AUDIT_STRUCT(); StorageGcsAuthTokenResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Get the response const KeyValue *const kvResponse = varKv(jsonToVar(strNewBuf(httpResponseContent(httpRequestResponse(request, true))))); // Check for an error const String *const error = varStr(kvGet(kvResponse, GCS_JSON_ERROR_VAR)); if (error != NULL) { THROW_FMT( ProtocolError, "unable to get authentication token: [%s] %s", strZ(error), strZNull(varStr(kvGet(kvResponse, GCS_JSON_ERROR_DESCRIPTION_VAR)))); } MEM_CONTEXT_PRIOR_BEGIN() { // Get token result.tokenType = strDup(varStr(kvGet(kvResponse, GCS_JSON_TOKEN_TYPE_VAR))); CHECK(FormatError, result.tokenType != NULL, "token type missing"); result.token = strDup(varStr(kvGet(kvResponse, GCS_JSON_ACCESS_TOKEN_VAR))); CHECK(FormatError, result.token != NULL, "access token missing"); // Get expiration const Variant *const expiresIn = kvGet(kvResponse, GCS_JSON_EXPIRES_IN_VAR); CHECK(FormatError, expiresIn != NULL, "expiry missing"); result.timeExpire = timeBegin + (time_t)varInt64Force(expiresIn); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_TYPE(StorageGcsAuthTokenResult, result); } /*********************************************************************************************************************************** Get authentication header for service keys Based on the documentation at https://developers.google.com/identity/protocols/oauth2/service-account#httprest ***********************************************************************************************************************************/ // Helper to construct a JSON Web Token static String * storageGcsAuthJwt(StorageGcs *const this, const time_t timeBegin) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(TIME, timeBegin); FUNCTION_TEST_END(); // Static header with dot delimiter String *const result = strCatZ(strNew(), "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9."); MEM_CONTEXT_TEMP_BEGIN() { // Load client email and private key const KeyValue *const kvKey = varKv( jsonToVar(strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), this->key))))); const String *const clientEmail = varStr(kvGet(kvKey, GCS_JSON_CLIENT_EMAIL_VAR)); const String *const privateKeyRaw = varStr(kvGet(kvKey, GCS_JSON_PRIVATE_KEY_VAR)); CHECK(FormatError, clientEmail != NULL && privateKeyRaw != NULL, "credentials missing"); // Add claim strCatEncode( result, encodingBase64Url, BUFSTR( strNewFmt( "{\"iss\":\"%s\",\"scope\":\"https://www.googleapis.com/auth/devstorage.read%s\",\"aud\":\"%s\"" ",\"exp\":%" PRIu64 ",\"iat\":%" PRIu64 "}", strZ(clientEmail), this->write ? "_write" : "_only", strZ(httpUrl(this->authUrl)), (uint64_t)timeBegin + 3600, (uint64_t)timeBegin))); // Sign with RSA key BIO *volatile bio = NULL; EVP_PKEY *volatile privateKey = NULL; EVP_MD_CTX *volatile sign = NULL; cryptoInit(); TRY_BEGIN() { // Load key bio = BIO_new(BIO_s_mem()); BIO_write(bio, strZ(privateKeyRaw), (int)strSize(privateKeyRaw)); privateKey = PEM_read_bio_PrivateKey(bio, NULL, NULL, NULL); cryptoError(privateKey == NULL, "unable to read PEM"); // Create signature sign = EVP_MD_CTX_create(); cryptoError(EVP_DigestSignInit(sign, NULL, EVP_sha256(), NULL, privateKey) <= 0, "unable to init"); cryptoError( EVP_DigestSignUpdate(sign, (const uint8_t *)strZ(result), (unsigned int)strSize(result)) <= 0, "unable to update"); size_t signatureLen = 0; cryptoError(EVP_DigestSignFinal(sign, NULL, &signatureLen) <= 0, "unable to get size"); Buffer *const signature = bufNew(signatureLen); bufUsedSet(signature, bufSize(signature)); cryptoError(EVP_DigestSignFinal(sign, bufPtr(signature), &signatureLen) <= 0, "unable to finalize"); // Add dot delimiter and signature strCatChr(result, '.'); strCatEncode(result, encodingBase64Url, signature); } FINALLY() { BIO_free(bio); EVP_MD_CTX_free(sign); EVP_PKEY_free(privateKey); } TRY_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } static StorageGcsAuthTokenResult storageGcsAuthService(StorageGcs *const this, const time_t timeBegin) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(TIME, timeBegin); FUNCTION_TEST_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(timeBegin > 0); StorageGcsAuthTokenResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { const String *const content = strNewFmt( "grant_type=urn%%3Aietf%%3Aparams%%3Aoauth%%3Agrant-type%%3Ajwt-bearer&assertion=%s", strZ(storageGcsAuthJwt(this, timeBegin))); HttpHeader *const header = httpHeaderNew(NULL); httpHeaderAdd(header, HTTP_HEADER_HOST_STR, httpUrlHost(this->authUrl)); httpHeaderAdd(header, HTTP_HEADER_CONTENT_TYPE_STR, HTTP_HEADER_CONTENT_TYPE_APP_FORM_URL_STR); httpHeaderAdd(header, HTTP_HEADER_CONTENT_LENGTH_STR, strNewFmt("%zu", strSize(content))); HttpRequest *const request = httpRequestNewP( this->authClient, HTTP_VERB_POST_STR, httpUrlPath(this->authUrl), NULL, .header = header, .content = BUFSTR(content)); MEM_CONTEXT_PRIOR_BEGIN() { result = storageGcsAuthToken(request, timeBegin); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_TYPE(StorageGcsAuthTokenResult, result); } /*********************************************************************************************************************************** Get authentication token automatically for instances running in GCE. Based on the documentation at https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#applications ***********************************************************************************************************************************/ static StorageGcsAuthTokenResult storageGcsAuthAuto(StorageGcs *const this, const time_t timeBegin) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(TIME, timeBegin); FUNCTION_TEST_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(timeBegin > 0); StorageGcsAuthTokenResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { HttpHeader *const header = httpHeaderNew(NULL); httpHeaderAdd(header, HTTP_HEADER_HOST_STR, httpUrlHost(this->authUrl)); httpHeaderAdd(header, GCS_HEADER_METADATA_FLAVOR_STR, GCS_HEADER_GOOGLE_STR); httpHeaderAdd(header, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); HttpRequest *const request = httpRequestNewP( this->authClient, HTTP_VERB_GET_STR, httpUrlPath(this->authUrl), NULL, .header = header); MEM_CONTEXT_PRIOR_BEGIN() { result = storageGcsAuthToken(request, timeBegin); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_TYPE(StorageGcsAuthTokenResult, result); } /*********************************************************************************************************************************** Generate authorization header and add it to the supplied header list ***********************************************************************************************************************************/ static void storageGcsAuth(StorageGcs *const this, HttpHeader *const httpHeader) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(HTTP_HEADER, httpHeader); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(httpHeader != NULL); ASSERT(httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_LENGTH_STR) != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Get the token if it was not supplied by the user if (this->keyType != storageGcsKeyTypeToken) { ASSERT(this->keyType == storageGcsKeyTypeAuto || this->keyType == storageGcsKeyTypeService); const time_t timeBegin = time(NULL); // If the current token has expired then request a new one if (timeBegin >= this->tokenTimeExpire) { const StorageGcsAuthTokenResult tokenResult = this->keyType == storageGcsKeyTypeAuto ? storageGcsAuthAuto(this, timeBegin) : storageGcsAuthService(this, timeBegin); MEM_CONTEXT_OBJ_BEGIN(this) { strFree(this->token); this->token = strNewFmt("%s %s", strZ(tokenResult.tokenType), strZ(tokenResult.token)); // Subtract http client timeout * 2 so the token does not expire in the middle of http retries this->tokenTimeExpire = tokenResult.timeExpire - ((time_t)(httpClientTimeout(this->httpClient) / MSEC_PER_SEC * 2)); } MEM_CONTEXT_OBJ_END(); } } // Add authorization header httpHeaderPut(httpHeader, HTTP_HEADER_AUTHORIZATION_STR, this->token); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Process Gcs request ***********************************************************************************************************************************/ // Helper to generate request path static String * storageGcsRequestPath(StorageGcs *const this, const String *const object, const bool bucket, const bool upload) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_GCS, this); FUNCTION_TEST_PARAM(STRING, object); FUNCTION_TEST_PARAM(BOOL, bucket); FUNCTION_TEST_PARAM(BOOL, upload); FUNCTION_TEST_END(); ASSERT(this != NULL); String *const result = strNew(); MEM_CONTEXT_TEMP_BEGIN() { strCatFmt(result, "%s/storage/v1/b", upload ? "/upload" : ""); if (bucket) strCatFmt(result, "/%s/o", strZ(this->bucket)); if (object != NULL) strCatFmt(result, "/%s", strZ(httpUriEncode(strSub(object, 1), false))); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } FN_EXTERN HttpRequest * storageGcsRequestAsync(StorageGcs *const this, const String *const verb, StorageGcsRequestAsyncParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(BOOL, param.noBucket); FUNCTION_LOG_PARAM(BOOL, param.upload); FUNCTION_LOG_PARAM(BOOL, param.noAuth); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_PARAM(STRING, param.path); FUNCTION_LOG_PARAM(STRING, param.object); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(LIST, param.contentList); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(verb != NULL); ASSERT(!param.noBucket || param.object == NULL); HttpRequest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Generate path const String *const path = param.path != NULL ? param.path : storageGcsRequestPath(this, param.object, !param.noBucket, param.upload); // Create header list HttpHeader *const requestHeader = param.header == NULL ? httpHeaderNew(this->headerRedactList) : httpHeaderDup(param.header, this->headerRedactList); // Add tags if (param.tag) { ASSERT(param.content == NULL); ASSERT(this->tag != NULL); httpHeaderPut(requestHeader, HTTP_HEADER_CONTENT_TYPE_STR, HTTP_HEADER_CONTENT_TYPE_JSON_STR); param.content = this->tag; } // Set host httpHeaderPut(requestHeader, HTTP_HEADER_HOST_STR, this->endpoint); // Set content or construct multipart content const Buffer *content = param.content; if (param.contentList != NULL) { ASSERT(param.content == NULL); HttpRequestMulti *const requestMulti = httpRequestMultiNew(); for (unsigned int contentIdx = 0; contentIdx < lstSize(param.contentList); contentIdx++) { const StorageGcsRequestPart *const requestPart = lstGet(param.contentList, contentIdx); HttpHeader *const partHeader = httpHeaderNew(this->headerRedactList); httpHeaderAdd(partHeader, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); httpRequestMultiAddP( requestMulti, strNewFmt("%u", contentIdx), requestPart->verb, storageGcsRequestPath(this, requestPart->object, true, false), .header = partHeader); } httpRequestMultiHeaderAdd(requestMulti, requestHeader); content = httpRequestMultiContent(requestMulti); } // Set content length httpHeaderPut( requestHeader, HTTP_HEADER_CONTENT_LENGTH_STR, content == NULL || bufEmpty(content) ? ZERO_STR : strNewFmt("%zu", bufUsed(content))); // Make a copy of the query so it can be modified HttpQuery *const query = this->userProject != NULL && param.query == NULL ? httpQueryNewP(.redactList = this->queryRedactList) : httpQueryDupP(param.query, .redactList = this->queryRedactList); // Add user project if (this->userProject != NULL) httpQueryAdd(query, GCS_QUERY_USER_PROJECT_STR, this->userProject); // Generate authorization header if (!param.noAuth) storageGcsAuth(this, requestHeader); // Send request MEM_CONTEXT_PRIOR_BEGIN() { result = httpRequestNewP( this->httpClient, verb, path, .query = query, .header = requestHeader, .content = content); } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_REQUEST, result); } FN_EXTERN HttpResponse * storageGcsResponse(HttpRequest *const request, const StorageGcsResponseParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, request); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.allowIncomplete); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_END(); ASSERT(request != NULL); ASSERT(!param.allowMissing || !param.allowIncomplete); HttpResponse *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get response result = httpRequestResponse(request, !param.contentIo); // Error if the request was not successful if (!httpResponseCodeOk(result) && (!param.allowMissing || httpResponseCode(result) != HTTP_RESPONSE_CODE_NOT_FOUND) && (!param.allowIncomplete || httpResponseCode(result) != HTTP_RESPONSE_CODE_PERMANENT_REDIRECT)) httpRequestError(request, result); // Move response to the prior context httpResponseMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } FN_EXTERN HttpResponse * storageGcsRequest(StorageGcs *const this, const String *const verb, const StorageGcsRequestParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(BOOL, param.noBucket); FUNCTION_LOG_PARAM(BOOL, param.upload); FUNCTION_LOG_PARAM(BOOL, param.noAuth); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_PARAM(STRING, param.path); FUNCTION_LOG_PARAM(STRING, param.object); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.allowIncomplete); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_END(); HttpRequest *const request = storageGcsRequestAsyncP( this, verb, .noBucket = param.noBucket, .upload = param.upload, .noAuth = param.noAuth, .tag = param.tag, .path = param.path, .object = param.object, .header = param.header, .query = param.query, .content = param.content, .contentList = param.contentList); HttpResponse *const result = storageGcsResponseP( request, .allowMissing = param.allowMissing, .allowIncomplete = param.allowIncomplete, .contentIo = param.contentIo); httpRequestFree(request); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /*********************************************************************************************************************************** General function for listing files to be used by other list routines Based on the documentation at https://cloud.google.com/storage/docs/json_api/v1/objects/list ***********************************************************************************************************************************/ // Helper to convert YYYY-MM-DDTHH:MM:SS.MSECZ format to time_t. This format is very nearly ISO-8601 except for the inclusion of // milliseconds, which are discarded here. static time_t storageGcsCvtTime(const String *const time) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, time); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( TIME, epochFromParts( cvtZSubNToInt(strZ(time), 0, 4), cvtZSubNToInt(strZ(time), 5, 2), cvtZSubNToInt(strZ(time), 8, 2), cvtZSubNToInt(strZ(time), 11, 2), cvtZSubNToInt(strZ(time), 14, 2), cvtZSubNToInt(strZ(time), 17, 2), 0)); } static void storageGcsInfoFile(StorageInfo *const info, const KeyValue *const file) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_PARAM(KEY_VALUE, file); FUNCTION_TEST_END(); info->size = cvtZToUInt64(strZ(varStr(kvGet(file, GCS_JSON_SIZE_VAR)))); info->timeModified = storageGcsCvtTime(varStr(kvGet(file, GCS_JSON_UPDATED_VAR))); FUNCTION_TEST_RETURN_VOID(); } static void storageGcsListInternal( StorageGcs *const this, const String *const path, const StorageInfoLevel level, const String *const expression, const bool recurse, const time_t targetTime, StorageListCallback callback, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_PARAM(BOOL, recurse); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, callback); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build the base prefix by stripping off the initial / const String *const basePrefix = strSize(path) == 1 ? EMPTY_STR : strNewFmt("%s/", strZ(strSub(path, 1))); // Get the expression prefix when possible to limit initial results const String *const expressionPrefix = regExpPrefix(expression); // If there is an expression prefix then use it to build the query prefix, otherwise query prefix is base prefix const String *queryPrefix; if (expressionPrefix == NULL) queryPrefix = basePrefix; else { if (strEmpty(basePrefix)) queryPrefix = expressionPrefix; else queryPrefix = strNewFmt("%s%s", strZ(basePrefix), strZ(expressionPrefix)); } // Create query HttpQuery *const query = httpQueryNewP(); // Add the delimiter to not recurse if (!recurse) httpQueryAdd(query, GCS_QUERY_DELIMITER_STR, FSLASH_STR); // Don't specify empty prefix because it is the default if (!strEmpty(queryPrefix)) httpQueryAdd(query, GCS_QUERY_PREFIX_STR, queryPrefix); // Specify fields to limit the amount of data returned String *const fields = strCatZ( strNew(), GCS_JSON_NEXT_PAGE_TOKEN "," GCS_JSON_PREFIXES "," GCS_JSON_ITEMS "(" GCS_JSON_NAME); if (level >= storageInfoLevelBasic || targetTime != 0) { if (level >= storageInfoLevelBasic) strCatZ(fields, "," GCS_JSON_SIZE); strCatZ(fields, "," GCS_JSON_UPDATED); if (targetTime != 0) strCatZ(fields, "," GCS_JSON_GENERATION); } strCatChr(fields, ')'); httpQueryAdd(query, GCS_QUERY_FIELDS_STR, fields); // Add versions if (targetTime != 0) httpQueryAdd(query, GCS_QUERY_VERSIONS_STR, TRUE_STR); // Store last info so it can be updated across requests for versioning String *const nameLast = strNew(); String *const versionIdLast = strNew(); StorageInfo infoLast = {.level = level, .name = nameLast}; if (targetTime != 0) infoLast.versionId = versionIdLast; // Loop as long as a continuation marker returned HttpRequest *request = NULL; do { // Use an inner mem context here because we could potentially be retrieving millions of files so it is a good idea to // free memory at regular intervals MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *response = NULL; // If there is an outstanding async request then wait for the response if (request != NULL) { response = storageGcsResponseP(request); httpRequestFree(request); request = NULL; } // Else get the response immediately from a sync request else response = storageGcsRequestP(this, HTTP_VERB_GET_STR, .query = query); const KeyValue *const content = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); // If next page token exists then send an async request to get more data const String *const nextPageToken = varStr(kvGet(content, GCS_JSON_NEXT_PAGE_TOKEN_VAR)); if (nextPageToken != NULL) { httpQueryPut(query, GCS_QUERY_PAGE_TOKEN_STR, nextPageToken); // Store request in the outer temp context MEM_CONTEXT_PRIOR_BEGIN() { request = storageGcsRequestAsyncP(this, HTTP_VERB_GET_STR, .query = query); } MEM_CONTEXT_PRIOR_END(); } // Get prefix list const VariantList *const prefixList = varVarLst(kvGet(content, GCS_JSON_PREFIXES_VAR)); if (prefixList != NULL) { for (unsigned int prefixIdx = 0; prefixIdx < varLstSize(prefixList); prefixIdx++) { // Get path name StorageInfo info = { .level = level, .name = varStr(varLstGet(prefixList, prefixIdx)), .exists = true, }; // Strip off base prefix and final / info.name = strSubN(info.name, strSize(basePrefix), strSize(info.name) - strSize(basePrefix) - 1); // Add type info if requested if (level >= storageInfoLevelType) info.type = storageTypePath; // Callback with info callback(callbackData, &info); } } // Get file list const VariantList *const fileList = varVarLst(kvGet(content, GCS_JSON_ITEMS_VAR)); if (fileList != NULL) { for (unsigned int fileIdx = 0; fileIdx < varLstSize(fileList); fileIdx++) { const KeyValue *const file = varKv(varLstGet(fileList, fileIdx)); CHECK(FormatError, file != NULL, "file missing"); // Get file name and strip off the base prefix when present const String *name = varStr(kvGet(file, GCS_JSON_NAME_VAR)); CHECK(FormatError, name != NULL, "file name missing"); if (!strEmpty(basePrefix)) name = strSub(name, strSize(basePrefix)); // Return info for last file if new file if (infoLast.exists && !strEq(name, nameLast)) { callback(callbackData, &infoLast); infoLast.exists = false; } // If targeting by time exclude versions that are newer than targetTime. Note that the API does not provide // delete markers so the filtering will also show files that have been deleted rather than replaced with a // new version. if (targetTime != 0) { infoLast.timeModified = storageGcsCvtTime(varStr(kvGet(file, GCS_JSON_UPDATED_VAR))); // Skip this version if it is newer than the time limit if (infoLast.timeModified > targetTime) continue; } // Update last name and set exists strCat(strTrunc(nameLast), name); infoLast.exists = true; // Add basic level info if requested if (level >= storageInfoLevelBasic) { storageGcsInfoFile(&infoLast, file); if (targetTime != 0) strCat(strTrunc(versionIdLast), varStr(kvGet(file, GCS_JSON_GENERATION_VAR))); } } } } MEM_CONTEXT_TEMP_END(); } while (request != NULL); // Callback with last info if it exists if (infoLast.exists) callback(callbackData, &infoLast); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static StorageInfo storageGcsInfo(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); StorageInfo result = {.level = level}; MEM_CONTEXT_TEMP_BEGIN() { // Attempt to get file info HttpResponse *const httpResponse = storageGcsRequestP( this, HTTP_VERB_GET_STR, .object = file, .allowMissing = true, .query = httpQueryAdd( httpQueryNewP(), GCS_QUERY_FIELDS_STR, level >= storageInfoLevelBasic ? STRDEF(GCS_JSON_SIZE "," GCS_JSON_UPDATED) : EMPTY_STR)); // Does the file exist? result.exists = httpResponseCodeOk(httpResponse); // Add basic level info if requested and the file exists if (result.level >= storageInfoLevelBasic && result.exists) { result.type = storageTypeFile; storageGcsInfoFile(&result, varKv(jsonToVar(strNewBuf(httpResponseContent(httpResponse))))); } httpResponseFree(httpResponse); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storageGcsListCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); storageLstAdd(callbackData, info); FUNCTION_TEST_RETURN_VOID(); } static StorageList * storageGcsList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_PARAM(TIME, param.targetTime); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); StorageList *const result = storageLstNew(level); storageGcsListInternal(this, path, level, param.expression, false, param.targetTime, storageGcsListCallback, result); FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void * storageGcsNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN(STORAGE_READ_GCS, storageReadGcsNew(this, file, param.offset, param.limit, param.versionId)); } /**********************************************************************************************************************************/ static void * storageGcsNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.createPath); ASSERT(param.truncate); ASSERT(param.user == NULL); ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); FUNCTION_LOG_RETURN(STORAGE_WRITE_GCS, storageWriteGcsNew(this, file, this->chunkSize, this->tag != NULL)); } /**********************************************************************************************************************************/ #define GCS_HEADER_CONTENTID_RESPONSE "response-" typedef struct StorageGcsPathRemoveData { StorageGcs *this; // Storage Object MemContext *memContext; // Mem context to create requests in HttpRequest *request; // Async remove request List *requestContentList; // Content list for async request List *contentList; // Content list currently being built const String *path; // Root path of remove } StorageGcsPathRemoveData; static void storageGcsPathRemoveInternal(StorageGcsPathRemoveData *const data) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_END(); ASSERT(data != NULL); ASSERT(data->this != NULL); // Get response for async request if (data->request != NULL) { MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *const response = storageGcsResponseP(data->request); HttpResponseMulti *const responseMulti = httpResponseMultiNew( httpResponseContent(response), httpHeaderGet(httpResponseHeader(response), HTTP_HEADER_CONTENT_TYPE_STR)); // Loop through all response parts HttpResponse *responsePart = httpResponseMultiNext(responseMulti); CHECK(FormatError, responsePart != NULL, "at least one response part is required"); do { // If not OK and not missing then retry if (!httpResponseCodeOk(responsePart) && httpResponseCode(responsePart) != HTTP_RESPONSE_CODE_NOT_FOUND) { // Extract and check content-id header const String *const contentId = httpHeaderGet(httpResponseHeader(responsePart), HTTP_HEADER_CONTENT_ID_STR); CHECK(FormatError, contentId != NULL, HTTP_HEADER_CONTENT_ID " header is not present"); CHECK_FMT( FormatError, strBeginsWithZ(contentId, GCS_HEADER_CONTENTID_RESPONSE), HTTP_HEADER_CONTENT_ID " header '%s' must begin with '" GCS_HEADER_CONTENTID_RESPONSE "'", strZ(contentId)); // Use content-id to get content const unsigned int contentIdx = cvtZToUInt(strZ(strSub(contentId, sizeof(GCS_HEADER_CONTENTID_RESPONSE) - 1))); const StorageGcsRequestPart *const content = lstGet(data->requestContentList, contentIdx); // Retry remove statInc(GCS_STAT_REMOVE_BATCH_RETRY_STR); storageGcsRequestP(data->this, content->verb, .object = content->object, .allowMissing = true); } else statInc(GCS_STAT_REMOVE_BATCH_PART_STR); httpResponseFree(responsePart); responsePart = httpResponseMultiNext(responseMulti); } while (responsePart != NULL); } MEM_CONTEXT_TEMP_END(); // Free request httpRequestFree(data->request); data->request = NULL; // Free content list lstFree(data->requestContentList); } // Send new async request if there is more to remove if (data->contentList != NULL) { statInc(GCS_STAT_REMOVE_BATCH_STR); MEM_CONTEXT_BEGIN(data->memContext) { data->request = storageGcsRequestAsyncP( data->this, HTTP_VERB_POST_STR, .path = STRDEF("/batch/storage/v1"), .contentList = data->contentList); } MEM_CONTEXT_END(); // Store the content list for use in error handling data->requestContentList = data->contentList; data->contentList = NULL; } FUNCTION_TEST_RETURN_VOID(); } static void storageGcsPathRemoveCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); // Only delete files since paths don't really exist if (info->type == storageTypeFile) { StorageGcsPathRemoveData *const data = callbackData; if (data->contentList == NULL) { MEM_CONTEXT_BEGIN(data->memContext) { data->contentList = lstNewP(sizeof(StorageGcsRequestPart)); } MEM_CONTEXT_END(); } MEM_CONTEXT_OBJ_BEGIN(data->contentList) { const StorageGcsRequestPart content = { .verb = HTTP_VERB_DELETE_STR, .object = strNewFmt("%s/%s", strZ(data->path), strZ(info->name)), }; lstAdd(data->contentList, &content); } MEM_CONTEXT_OBJ_END(); if (lstSize(data->contentList) == data->this->deleteMax) storageGcsPathRemoveInternal(data); } FUNCTION_TEST_RETURN_VOID(); } static bool storageGcsPathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, recurse); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { StorageGcsPathRemoveData data = { .this = this, .memContext = memContextCurrent(), .path = strEq(path, FSLASH_STR) ? EMPTY_STR : path, }; MEM_CONTEXT_TEMP_BEGIN() { storageGcsListInternal(this, path, storageInfoLevelType, NULL, true, 0, storageGcsPathRemoveCallback, &data); // Call if there is more to be removed if (data.contentList != NULL) storageGcsPathRemoveInternal(&data); // Check response on last async request storageGcsPathRemoveInternal(&data); } MEM_CONTEXT_TEMP_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, true); } /**********************************************************************************************************************************/ static void storageGcsRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageGcs); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_GCS, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); statInc(GCS_STAT_REMOVE_STR); httpResponseFree(storageGcsRequestP(this, HTTP_VERB_DELETE_STR, .object = file, .allowMissing = true)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceGcs = { .feature = 1 << storageFeatureVersioning | 1 << storageFeatureReadRetry, .info = storageGcsInfo, .list = storageGcsList, .newRead = storageGcsNewRead, .newWrite = storageGcsNewWrite, .pathRemove = storageGcsPathRemove, .remove = storageGcsRemove, }; FN_EXTERN Storage * storageGcsNew( const String *const path, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *const bucket, const StorageGcsKeyType keyType, const String *const key, const size_t chunkSize, const KeyValue *const tag, const String *const endpoint, const TimeMSec timeout, const bool verifyPeer, const String *const caFile, const String *const caPath, const String *const userProject) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(STRING, bucket); FUNCTION_LOG_PARAM(ENUM, keyType); FUNCTION_TEST_PARAM(STRING, key); FUNCTION_LOG_PARAM(SIZE, chunkSize); FUNCTION_LOG_PARAM(KEY_VALUE, tag); FUNCTION_LOG_PARAM(STRING, endpoint); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_PARAM(BOOL, verifyPeer); FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, caPath); FUNCTION_LOG_PARAM(STRING, userProject); FUNCTION_LOG_END(); ASSERT(path != NULL); ASSERT(bucket != NULL); ASSERT(keyType == storageGcsKeyTypeAuto || key != NULL); ASSERT(chunkSize != 0); OBJ_NEW_BEGIN(StorageGcs, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageGcs) { .interface = storageInterfaceGcs, .write = write, .bucket = strDup(bucket), .keyType = keyType, .chunkSize = chunkSize, .deleteMax = STORAGE_GCS_DELETE_MAX, .userProject = strDup(userProject), }; // Create tag JSON buffer if (write && tag != NULL) { MEM_CONTEXT_TEMP_BEGIN() { JsonWrite *const tagJson = jsonWriteObjectBegin( jsonWriteKeyStrId(jsonWriteObjectBegin(jsonWriteNewP()), STRID5("metadata", 0xd0240d0ad0))); const StringList *const keyList = strLstSort(strLstNewVarLst(kvKeyList(tag)), sortOrderAsc); for (unsigned int keyIdx = 0; keyIdx < strLstSize(keyList); keyIdx++) { const String *const key = strLstGet(keyList, keyIdx); jsonWriteStr(jsonWriteKey(tagJson, key), varStr(kvGet(tag, VARSTR(key)))); } const String *const tagStr = jsonWriteResult(jsonWriteObjectEnd(jsonWriteObjectEnd(tagJson))); MEM_CONTEXT_PRIOR_BEGIN() { this->tag = bufDup(BUFSTR(tagStr)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); } // Handle auth key types switch (keyType) { // Auto authentication for GCE instances case storageGcsKeyTypeAuto: { this->authUrl = httpUrlNewParseP( STRDEF("metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token"), .type = httpProtocolTypeHttp); this->authClient = httpClientNew( sckClientNew(httpUrlHost(this->authUrl), httpUrlPort(this->authUrl), timeout, timeout), timeout); break; } // Read data from file for service keys case storageGcsKeyTypeService: { const KeyValue *const kvKey = varKv( jsonToVar(strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), key))))); const String *const uri = varStr(kvGet(kvKey, GCS_JSON_TOKEN_URI_VAR)); CHECK(FormatError, uri != NULL, "uri missing"); this->key = strDup(key); this->authUrl = httpUrlNewParseP(uri, .type = httpProtocolTypeHttps); this->authClient = httpClientNew( tlsClientNewP( sckClientNew(httpUrlHost(this->authUrl), httpUrlPort(this->authUrl), timeout, timeout), httpUrlHost(this->authUrl), timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath), timeout); break; } // Store the authentication token case storageGcsKeyTypeToken: this->token = strDup(key); break; } // Parse the endpoint to extract the host, port, and protocol const HttpUrl *const url = httpUrlNewParseP(endpoint, .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); const HttpProtocolType protocolType = httpUrlProtocolType(url); this->endpoint = httpUrlHost(url); // Create the http client used to service requests. Use plain socket for HTTP, TLS for HTTPS. IoClient *ioClient; if (protocolType == httpProtocolTypeHttp) ioClient = sckClientNew(this->endpoint, httpUrlPort(url), timeout, timeout); else { ioClient = tlsClientNewP( sckClientNew(this->endpoint, httpUrlPort(url), timeout, timeout), this->endpoint, timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath); } this->httpClient = httpClientNew(ioClient, timeout); // Create list of redacted headers this->headerRedactList = strLstNew(); strLstAdd(this->headerRedactList, HTTP_HEADER_AUTHORIZATION_STR); strLstAdd(this->headerRedactList, GCS_HEADER_UPLOAD_ID_STR); // Create list of redacted query keys this->queryRedactList = strLstNew(); strLstAdd(this->queryRedactList, GCS_QUERY_UPLOAD_ID_STR); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew(STORAGE_GCS_TYPE, path, 0, 0, write, targetTime, pathExpressionFunction, this, this->interface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/storage.h0000664000175100017510000000342515240547630020243 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_STORAGE_H #define STORAGE_GCS_STORAGE_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_GCS_TYPE STRID5("gcs", 0x4c670) /*********************************************************************************************************************************** Key type ***********************************************************************************************************************************/ typedef enum { storageGcsKeyTypeService, storageGcsKeyTypeToken, storageGcsKeyTypeAuto, } StorageGcsKeyType; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageGcsNew( const String *path, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *bucket, StorageGcsKeyType keyType, const String *key, size_t blockSize, const KeyValue *tag, const String *endpoint, TimeMSec timeout, bool verifyPeer, const String *caFile, const String *caPath, const String *userProject); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/storage.intern.h0000664000175100017510000001761715240547630021551 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_STORAGE_INTERN_H #define STORAGE_GCS_STORAGE_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageGcs StorageGcs; #include "common/io/http/request.h" #include "storage/gcs/storage.h" /*********************************************************************************************************************************** HTTP headers ***********************************************************************************************************************************/ #define GCS_HEADER_UPLOAD_ID "x-guploader-uploadid" STRING_DECLARE(GCS_HEADER_UPLOAD_ID_STR); /*********************************************************************************************************************************** Query tokens ***********************************************************************************************************************************/ #define GCS_QUERY_FIELDS "fields" STRING_DECLARE(GCS_QUERY_FIELDS_STR); #define GCS_QUERY_MEDIA "media" STRING_DECLARE(GCS_QUERY_MEDIA_STR); #define GCS_QUERY_NAME "name" STRING_DECLARE(GCS_QUERY_NAME_STR); #define GCS_QUERY_UPLOAD_ID "upload_id" STRING_DECLARE(GCS_QUERY_UPLOAD_ID_STR); /*********************************************************************************************************************************** JSON tokens ***********************************************************************************************************************************/ #define GCS_JSON_GENERATION "generation" VARIANT_DECLARE(GCS_JSON_GENERATION_VAR); #define GCS_JSON_MD5_HASH "md5Hash" VARIANT_DECLARE(GCS_JSON_MD5_HASH_VAR); #define GCS_JSON_NAME "name" VARIANT_DECLARE(GCS_JSON_NAME_VAR); #define GCS_JSON_SIZE "size" VARIANT_DECLARE(GCS_JSON_SIZE_VAR); /*********************************************************************************************************************************** Multi-Part request data ***********************************************************************************************************************************/ typedef struct StorageGcsRequestPart { const String *object; // Object to include in URI const String *verb; // Verb (GET, PUT, etc) } StorageGcsRequestPart; /*********************************************************************************************************************************** Perform a GCS Request ***********************************************************************************************************************************/ // Perform async request typedef struct StorageGcsRequestAsyncParam { VAR_PARAM_HEADER; bool noBucket; // Exclude bucket from the URI? bool upload; // Is an object upload? bool noAuth; // Exclude authentication header? bool tag; // Add tags when available? const String *path; // URI path (this overrides object) const String *object; // Object to include in URI const HttpHeader *header; // Request headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content const List *contentList; // Request content part list } StorageGcsRequestAsyncParam; #define storageGcsRequestAsyncP(this, verb, ...) \ storageGcsRequestAsync(this, verb, (StorageGcsRequestAsyncParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpRequest *storageGcsRequestAsync(StorageGcs *this, const String *verb, StorageGcsRequestAsyncParam param); // Get async response typedef struct StorageGcsResponseParam { VAR_PARAM_HEADER; bool allowMissing; // Allow missing files (caller can check response code) bool allowIncomplete; // Allow incomplete resume (used for resumable upload) bool contentIo; // Is IoRead interface required to read content? } StorageGcsResponseParam; #define storageGcsResponseP(request, ...) \ storageGcsResponse(request, (StorageGcsResponseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageGcsResponse(HttpRequest *request, StorageGcsResponseParam param); typedef struct StorageGcsRequestParam { VAR_PARAM_HEADER; bool noBucket; // Exclude bucket from the URI? bool upload; // Is an object upload? bool noAuth; // Exclude authentication header? bool tag; // Add tags when available? const String *path; // URI path (this overrides object) const String *object; // Object to include in URI const HttpHeader *header; // Request headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content const List *contentList; // Request content part list bool allowMissing; // Allow missing files (caller can check response code) bool allowIncomplete; // Allow incomplete resume (used for resumable upload) bool contentIo; // Is IoRead interface required to read content? } StorageGcsRequestParam; #define storageGcsRequestP(this, verb, ...) \ storageGcsRequest(this, verb, (StorageGcsRequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageGcsRequest(StorageGcs *this, const String *verb, StorageGcsRequestParam param); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_GCS_TYPE \ StorageGcs * #define FUNCTION_LOG_STORAGE_GCS_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageGcs", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/write.c0000664000175100017510000003307215240547630017725 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage File Write ***********************************************************************************************************************************/ #include #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/filter/filter.h" #include "common/log.h" #include "common/type/json.h" #include "common/type/keyValue.h" #include "common/type/object.h" #include "storage/gcs/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Chunk defaults based on limits at https://cloud.google.com/storage/quotas#requests ***********************************************************************************************************************************/ #define STORAGE_GCS_SPLIT_DEFAULT 257 #define STORAGE_GCS_SPLIT_MAX 9505 /*********************************************************************************************************************************** GCS query tokens ***********************************************************************************************************************************/ STRING_STATIC(GCS_QUERY_UPLOAD_TYPE_STR, "uploadType"); STRING_STATIC(GCS_QUERY_RESUMABLE_STR, "resumable"); STRING_STATIC(GCS_QUERY_FIELDS_VALUE_STR, GCS_JSON_MD5_HASH "," GCS_JSON_SIZE); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteGcs { const StorageWriteInterface *interface; // Interface StorageGcs *storage; // Storage that created this object const String *name; // File name HttpRequest *request; // Async chunk upload request size_t chunkSize; // Size of chunks for resumable upload unsigned int chunkTotal; // Total chunks uploaded bool tag; // Are tags available? Buffer *chunkBuffer; // Block buffer (stores data until chunkSize is reached) const String *uploadId; // Id for resumable upload uint64_t uploadTotal; // Total bytes uploaded IoFilter *md5hash; // MD5 hash of file }; /*********************************************************************************************************************************** Verify upload ***********************************************************************************************************************************/ static void storageWriteGcsVerify(StorageWriteGcs *const this, HttpResponse *const response) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_PARAM(HTTP_RESPONSE, response); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { KeyValue *const content = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); // Check the md5 hash const String *const md5base64 = varStr(kvGet(content, GCS_JSON_MD5_HASH_VAR)); CHECK(FormatError, md5base64 != NULL, "MD5 missing"); const Buffer *const md5actual = bufNewDecode(encodingBase64, md5base64); const Buffer *const md5expected = pckReadBinP(pckReadNew(ioFilterResult(this->md5hash))); if (!bufEq(md5actual, md5expected)) { THROW_FMT( FormatError, "expected md5 '%s' for '%s' but actual is '%s'", strZ(strNewEncode(encodingHex, md5expected)), strZ(this->name), strZ(strNewEncode(encodingHex, md5actual))); } // Check the size when available const String *const sizeStr = varStr(kvGet(content, GCS_JSON_SIZE_VAR)); if (sizeStr != NULL) { const uint64_t size = cvtZToUInt64(strZ(sizeStr)); if (size != this->uploadTotal) { THROW_FMT( FormatError, "expected size %" PRIu64 " for '%s' but actual is %" PRIu64, size, strZ(this->name), this->uploadTotal); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Flush bytes to upload chunk ***********************************************************************************************************************************/ static void storageWriteGcsBlock(StorageWriteGcs *const this, const bool done) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_PARAM(BOOL, done); FUNCTION_LOG_END(); ASSERT(this != NULL); // If there is an outstanding async request then wait for the response to ensure the request did not error if (this->request != NULL) { HttpResponse *const response = storageGcsResponseP(this->request, .allowIncomplete = !done); // If done then verify the md5 checksum if (done) storageWriteGcsVerify(this, response); httpResponseFree(response); httpRequestFree(this->request); this->request = NULL; } FUNCTION_LOG_RETURN_VOID(); } static void storageWriteGcsBlockAsync(StorageWriteGcs *const this, const bool done) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_PARAM(BOOL, done); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->chunkBuffer != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Complete prior async request, if any storageWriteGcsBlock(this, false); // Build query HttpQuery *const query = httpQueryNewP(); httpQueryAdd(query, GCS_QUERY_NAME_STR, strSub(this->name, 1)); httpQueryAdd(query, GCS_QUERY_UPLOAD_TYPE_STR, GCS_QUERY_RESUMABLE_STR); // Get the upload id if (this->uploadId == NULL) { HttpResponse *response = storageGcsRequestP( this->storage, HTTP_VERB_POST_STR, .upload = true, .tag = this->tag, .query = query); MEM_CONTEXT_OBJ_BEGIN(this) { this->uploadId = strDup(httpHeaderGet(httpResponseHeader(response), GCS_HEADER_UPLOAD_ID_STR)); CHECK(FormatError, this->uploadId != NULL, "upload id missing"); } MEM_CONTEXT_OBJ_END(); } // Add data to md5 hash if (!bufEmpty(this->chunkBuffer)) ioFilterProcessIn(this->md5hash, this->chunkBuffer); // Upload the chunk. If this is the last chunk then add the total bytes in the file to the range rather than the * added to // prior chunks. This indicates that the resumable upload is complete. If the last chunk is zero-size, then the byte range // is * to indicate that there is no more data to upload. HttpHeader *const header = httpHeaderAdd( httpHeaderNew(NULL), HTTP_HEADER_CONTENT_RANGE_STR, strNewFmt( HTTP_HEADER_CONTENT_RANGE_BYTES " %s/%s", bufUsed(this->chunkBuffer) == 0 ? "*" : zNewFmt("%" PRIu64 "-%" PRIu64, this->uploadTotal, this->uploadTotal + bufUsed(this->chunkBuffer) - 1), done ? zNewFmt("%" PRIu64, this->uploadTotal + bufUsed(this->chunkBuffer)) : "*")); httpQueryAdd(query, GCS_QUERY_UPLOAD_ID_STR, this->uploadId); // Add fields needed to verify to upload if (done) httpQueryAdd(query, GCS_QUERY_FIELDS_STR, GCS_QUERY_FIELDS_VALUE_STR); MEM_CONTEXT_OBJ_BEGIN(this) { this->request = storageGcsRequestAsyncP( this->storage, HTTP_VERB_PUT_STR, .upload = true, .noAuth = true, .header = header, .query = query, .content = this->chunkBuffer); } MEM_CONTEXT_OBJ_END(); this->uploadTotal += bufUsed(this->chunkBuffer); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to internal buffer ***********************************************************************************************************************************/ static void storageWriteGcs(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Continue until the write buffer has been exhausted size_t bytesTotal = 0; do { // Resize chunk buffer. This must be done in the loop because chunk size grows as chunks are written. storageWriteChunkBufferResize(buffer, bytesTotal, this->chunkBuffer, this->chunkSize); // Copy as many bytes as possible into the chunk buffer const size_t bytesNext = bufRemains(this->chunkBuffer) > bufUsed(buffer) - bytesTotal ? bufUsed(buffer) - bytesTotal : bufRemains(this->chunkBuffer); bufCatSub(this->chunkBuffer, buffer, bytesTotal, bytesNext); bytesTotal += bytesNext; // If the chunk buffer is full then write it. It is possible that this is the last chunk and it would be better to wait, but // the chances of that are quite small so in general it is better to write now so there is less to write later. if (bufUsed(this->chunkBuffer) == this->chunkSize) { storageWriteGcsBlockAsync(this, false); bufUsedZero(this->chunkBuffer); this->chunkSize = storageWriteChunkSize(this->chunkSize, STORAGE_GCS_SPLIT_DEFAULT, STORAGE_GCS_SPLIT_MAX, ++this->chunkTotal); } } while (bytesTotal != bufUsed(buffer)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteGcsClose(THIS_VOID) { THIS(StorageWriteGcs); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_GCS, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->chunkBuffer != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // If a resumable upload was started then finish that way if (this->uploadId != NULL || this->tag) { // Write what is left in the chunk buffer storageWriteGcsBlockAsync(this, true); storageWriteGcsBlock(this, true); } // Else upload all the data in a single chunk else { // Add data to md5 hash if (bufUsed(this->chunkBuffer)) ioFilterProcessIn(this->md5hash, this->chunkBuffer); // Upload file HttpQuery *query = httpQueryNewP(); httpQueryAdd(query, GCS_QUERY_NAME_STR, strSub(this->name, 1)); httpQueryAdd(query, GCS_QUERY_UPLOAD_TYPE_STR, GCS_QUERY_MEDIA_STR); httpQueryAdd(query, GCS_QUERY_FIELDS_STR, GCS_QUERY_FIELDS_VALUE_STR); this->uploadTotal = bufUsed(this->chunkBuffer); storageWriteGcsVerify( this, storageGcsRequestP( this->storage, HTTP_VERB_POST_STR, .upload = true, .query = query, .content = this->chunkBuffer)); } bufFree(this->chunkBuffer); this->chunkBuffer = NULL; } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteGcsInterface = { .close = storageWriteGcsClose, .write = storageWriteGcs, }; FN_EXTERN StorageWriteGcs * storageWriteGcsNew(StorageGcs *const storage, const String *const name, const size_t chunkSize, const bool tag) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_GCS, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, chunkSize); FUNCTION_LOG_PARAM(BOOL, tag); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageWriteGcs, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWriteGcs) { .interface = &storageWriteGcsInterface, .storage = storage, .name = strDup(name), .chunkSize = chunkSize, .chunkBuffer = bufNew(0), .md5hash = cryptoHashNew(hashTypeMd5), .tag = tag, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_GCS, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/gcs/write.h0000664000175100017510000000337615240547630017736 0ustar00runnerrunner/*********************************************************************************************************************************** GCS Storage File Write ***********************************************************************************************************************************/ #ifndef STORAGE_GCS_WRITE_H #define STORAGE_GCS_WRITE_H #include "storage/gcs/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteGcs StorageWriteGcs; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteGcs *storageWriteGcsNew(StorageGcs *storage, const String *name, size_t chunkSize, bool tag); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_GCS_TYPE \ StorageWriteGcs * #define FUNCTION_LOG_STORAGE_WRITE_GCS_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteGcs", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/helper.c0000664000175100017510000005176615240547630017310 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Helper ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/memContext.h" #include "common/regExp.h" #include "config/config.h" #include "protocol/helper.h" #include "storage/helper.h" #include "storage/posix/storage.h" #include "storage/remote/storage.h" /*********************************************************************************************************************************** Storage path constants ***********************************************************************************************************************************/ STRING_EXTERN(STORAGE_SPOOL_ARCHIVE_STR, STORAGE_SPOOL_ARCHIVE); STRING_EXTERN(STORAGE_SPOOL_ARCHIVE_IN_STR, STORAGE_SPOOL_ARCHIVE_IN); STRING_EXTERN(STORAGE_SPOOL_ARCHIVE_OUT_STR, STORAGE_SPOOL_ARCHIVE_OUT); STRING_EXTERN(STORAGE_REPO_ARCHIVE_STR, STORAGE_REPO_ARCHIVE); STRING_EXTERN(STORAGE_REPO_BACKUP_STR, STORAGE_REPO_BACKUP); STRING_EXTERN(STORAGE_PATH_ARCHIVE_STR, STORAGE_PATH_ARCHIVE); STRING_EXTERN(STORAGE_PATH_BACKUP_STR, STORAGE_PATH_BACKUP); /*********************************************************************************************************************************** Error message when writable storage is requested in dry-run mode ***********************************************************************************************************************************/ #define WRITABLE_WHILE_DRYRUN \ "unable to get writable storage in dry-run mode or before dry-run is initialized" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct StorageHelperLocal { MemContext *memContext; // Mem context for storage helper const StorageHelper *helperList; // List of helpers to create storage Storage *storageLocal; // Local read-only storage Storage *storageLocalWrite; // Local write storage Storage **storagePg; // PostgreSQL read-only storage Storage **storagePgWrite; // PostgreSQL write storage Storage **storageRepo; // Repository read-only storage Storage **storageRepoWrite; // Repository write storage Storage *storageSpool; // Spool read-only storage Storage *storageSpoolWrite; // Spool write storage String *stanza; // Stanza for storage bool stanzaInit; // Has the stanza been initialized? bool dryRunInit; // Has dryRun been initialized? If not disallow writes. bool dryRun; // Disallow writes in dry-run mode. RegExp *walRegExp; // Regular expression for identifying wal files } storageHelper; /*********************************************************************************************************************************** Create the storage helper memory context ***********************************************************************************************************************************/ static void storageHelperContextInit(void) { FUNCTION_TEST_VOID(); if (storageHelper.memContext == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { MEM_CONTEXT_NEW_BEGIN(StorageHelper, .childQty = MEM_CONTEXT_QTY_MAX, .allocQty = MEM_CONTEXT_QTY_MAX) { storageHelper.memContext = MEM_CONTEXT_NEW(); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storageHelperInit(const StorageHelper *const helperList) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, helperList); FUNCTION_TEST_END(); storageHelper.helperList = helperList; FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storageHelperDryRunInit(const bool dryRun) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, dryRun); FUNCTION_TEST_END(); storageHelper.dryRunInit = true; storageHelper.dryRun = dryRun; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Initialize the stanza and error if it changes ***********************************************************************************************************************************/ static void storageHelperStanzaInit(const bool stanzaRequired) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BOOL, stanzaRequired); FUNCTION_TEST_END(); // If the stanza is NULL and the storage has not already been initialized then initialize the stanza if (!storageHelper.stanzaInit) { if (stanzaRequired && cfgOptionStrNull(cfgOptStanza) == NULL) THROW(AssertError, "stanza cannot be NULL for this storage object"); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.stanza = strDup(cfgOptionStrNull(cfgOptStanza)); storageHelper.stanzaInit = true; } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN const Storage * storageLocal(void) { FUNCTION_TEST_VOID(); if (storageHelper.storageLocal == NULL) { storageHelperContextInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageLocal = storagePosixNewP(FSLASH_STR); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageLocal); } FN_EXTERN const Storage * storageLocalWrite(void) { FUNCTION_TEST_VOID(); if (storageHelper.storageLocalWrite == NULL) { storageHelperContextInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageLocalWrite = storagePosixNewP(FSLASH_STR, .write = true); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageLocalWrite); } /*********************************************************************************************************************************** Get pg storage for the specified host id ***********************************************************************************************************************************/ static Storage * storagePgGet(const unsigned int pgIdx, const bool write) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgIdx); FUNCTION_TEST_PARAM(BOOL, write); FUNCTION_TEST_END(); Storage *result; // Use remote storage if (!pgIsLocal(pgIdx)) { result = storageRemoteNew( STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, write, 0, NULL, protocolRemoteGet(protocolStorageTypePg, pgIdx, true), cfgOptionUInt(cfgOptCompressLevelNetwork)); } // Use Posix storage else result = storagePosixNewP(cfgOptionIdxStr(cfgOptPgPath, pgIdx), .write = write); FUNCTION_TEST_RETURN(STORAGE, result); } /**********************************************************************************************************************************/ FN_EXTERN const Storage * storagePgIdx(const unsigned int pgIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgIdx); FUNCTION_TEST_END(); if (storageHelper.storagePg == NULL || storageHelper.storagePg[pgIdx] == NULL) { storageHelperContextInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { if (storageHelper.storagePg == NULL) storageHelper.storagePg = memNewPtrArray(cfgOptionGroupIdxTotal(cfgOptGrpPg)); storageHelper.storagePg[pgIdx] = storagePgGet(pgIdx, false); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storagePg[pgIdx]); } FN_EXTERN const Storage * storagePg(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STORAGE, storagePgIdx(cfgOptionGroupIdxDefault(cfgOptGrpPg))); } FN_EXTERN const Storage * storagePgIdxWrite(const unsigned int pgIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, pgIdx); FUNCTION_TEST_END(); // Writes not allowed in dry-run mode if (!storageHelper.dryRunInit || storageHelper.dryRun) THROW(AssertError, WRITABLE_WHILE_DRYRUN); if (storageHelper.storagePgWrite == NULL || storageHelper.storagePgWrite[pgIdx] == NULL) { storageHelperContextInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { if (storageHelper.storagePgWrite == NULL) storageHelper.storagePgWrite = memNewPtrArray(cfgOptionGroupIdxTotal(cfgOptGrpPg)); storageHelper.storagePgWrite[pgIdx] = storagePgGet(pgIdx, true); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storagePgWrite[pgIdx]); } FN_EXTERN const Storage * storagePgWrite(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STORAGE, storagePgIdxWrite(cfgOptionGroupIdxDefault(cfgOptGrpPg))); } /*********************************************************************************************************************************** Create the WAL regular expression ***********************************************************************************************************************************/ static void storageHelperRepoInit(void) { FUNCTION_TEST_VOID(); if (storageHelper.walRegExp == NULL) { MEM_CONTEXT_BEGIN(memContextTop()) { storageHelper.walRegExp = regExpNew(STRDEF("^[0-F]{24}")); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Construct a repo path from an expression and path ***********************************************************************************************************************************/ static String * storageRepoPathExpression(const String *const expression, const String *const path) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_END(); ASSERT(expression != NULL); String *const result = strNew(); if (strEq(expression, STORAGE_REPO_ARCHIVE_STR)) { // Construct the base path if (storageHelper.stanza != NULL) strCatFmt(result, STORAGE_PATH_ARCHIVE "/%s", strZ(storageHelper.stanza)); else strCat(result, STORAGE_PATH_ARCHIVE_STR); // If a subpath should be appended, determine if it is WAL path, else just append the subpath if (path != NULL) { StringList *const pathSplit = strLstNewSplitZ(path, "/"); const String *const file = strLstSize(pathSplit) == 2 ? strLstGet(pathSplit, 1) : NULL; if (file != NULL && regExpMatch(storageHelper.walRegExp, file)) strCatFmt(result, "/%s/%.16s/%s", strZ(strLstGet(pathSplit, 0)), strZ(file), strZ(file)); else strCatFmt(result, "/%s", strZ(path)); strLstFree(pathSplit); } } else if (strEq(expression, STORAGE_REPO_BACKUP_STR)) { // Construct the base path if (storageHelper.stanza != NULL) strCatFmt(result, STORAGE_PATH_BACKUP "/%s", strZ(storageHelper.stanza)); else strCatZ(result, STORAGE_PATH_BACKUP); // Append subpath if provided if (path != NULL) strCatFmt(result, "/%s", strZ(path)); } else THROW_FMT(AssertError, "invalid expression '%s'", strZ(expression)); ASSERT(result != NULL); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Get the repo storage ***********************************************************************************************************************************/ static Storage * storageRepoGet(const unsigned int repoIdx, const bool write) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_PARAM(BOOL, write); FUNCTION_TEST_END(); Storage *result = NULL; // Use remote storage if (!repoIsLocal(repoIdx)) { result = storageRemoteNew( STORAGE_MODE_FILE_DEFAULT, STORAGE_MODE_PATH_DEFAULT, write, storageRepoTargetTime(), storageRepoPathExpression, protocolRemoteGet(protocolStorageTypeRepo, repoIdx, true), cfgOptionUInt(cfgOptCompressLevelNetwork)); } // Use local storage else { // Search for the helper const StringId type = cfgOptionIdxStrId(cfgOptRepoType, repoIdx); if (storageHelper.helperList != NULL) { for (const StorageHelper *helper = storageHelper.helperList; helper->type != 0; helper++) { if (helper->type == type) { result = helper->helper(repoIdx, write, storageRepoPathExpression); break; } } } // If no helper was found then try Posix if (result == NULL) { CHECK(AssertError, type == STORAGE_POSIX_TYPE, "invalid storage type"); result = storagePosixNewP( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), .write = write, .pathExpressionFunction = storageRepoPathExpression, .noSymLink = cfgOptionValid(cfgOptRepoSymlink) ? !cfgOptionIdxBool(cfgOptRepoSymlink, repoIdx) : true); } } FUNCTION_TEST_RETURN(STORAGE, result); } /**********************************************************************************************************************************/ FN_EXTERN const Storage * storageRepoIdx(const unsigned int repoIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_END(); if (storageHelper.storageRepo == NULL) { storageHelperContextInit(); storageHelperStanzaInit(false); storageHelperRepoInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageRepo = memNewPtrArray(cfgOptionGroupIdxTotal(cfgOptGrpRepo)); } MEM_CONTEXT_END(); } if (storageHelper.storageRepo[repoIdx] == NULL) { MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageRepo[repoIdx] = storageRepoGet(repoIdx, false); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageRepo[repoIdx]); } FN_EXTERN const Storage * storageRepo(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STORAGE, storageRepoIdx(cfgOptionGroupIdxDefault(cfgOptGrpRepo))); } FN_EXTERN const Storage * storageRepoIdxWrite(const unsigned int repoIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT, repoIdx); FUNCTION_TEST_END(); // Writes not allowed in dry-run mode if (!storageHelper.dryRunInit || storageHelper.dryRun) THROW(AssertError, WRITABLE_WHILE_DRYRUN); if (storageHelper.storageRepoWrite == NULL) { storageHelperContextInit(); storageHelperStanzaInit(false); storageHelperRepoInit(); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageRepoWrite = memNewPtrArray(cfgOptionGroupIdxTotal(cfgOptGrpRepo)); } MEM_CONTEXT_END(); } if (storageHelper.storageRepoWrite[repoIdx] == NULL) { MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageRepoWrite[repoIdx] = storageRepoGet(repoIdx, true); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageRepoWrite[repoIdx]); } FN_EXTERN const Storage * storageRepoWrite(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN_CONST(STORAGE, storageRepoIdxWrite(cfgOptionGroupIdxDefault(cfgOptGrpRepo))); } /**********************************************************************************************************************************/ FN_EXTERN time_t storageRepoTargetTime(void) { FUNCTION_TEST_VOID(); FUNCTION_TEST_RETURN(TIME, cfgOptionTest(cfgOptRepoTargetTime) ? cvtZToTime(strZ(cfgOptionStr(cfgOptRepoTargetTime))) : 0); } /*********************************************************************************************************************************** Spool storage path expression ***********************************************************************************************************************************/ static String * storageSpoolPathExpression(const String *const expression, const String *const path) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, expression); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_END(); ASSERT(expression != NULL); ASSERT(storageHelper.stanza != NULL); String *result = NULL; if (strEqZ(expression, STORAGE_SPOOL_ARCHIVE)) { if (path == NULL) result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s", strZ(storageHelper.stanza)); else result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/%s", strZ(storageHelper.stanza), strZ(path)); } else if (strEqZ(expression, STORAGE_SPOOL_ARCHIVE_IN)) { if (path == NULL) result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/in", strZ(storageHelper.stanza)); else result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/in/%s", strZ(storageHelper.stanza), strZ(path)); } else if (strEqZ(expression, STORAGE_SPOOL_ARCHIVE_OUT)) { if (path == NULL) result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/out", strZ(storageHelper.stanza)); else result = strNewFmt(STORAGE_PATH_ARCHIVE "/%s/out/%s", strZ(storageHelper.stanza), strZ(path)); } else THROW_FMT(AssertError, "invalid expression '%s'", strZ(expression)); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN const Storage * storageSpool(void) { FUNCTION_TEST_VOID(); if (storageHelper.storageSpool == NULL) { storageHelperContextInit(); storageHelperStanzaInit(true); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageSpool = storagePosixNewP( cfgOptionStr(cfgOptSpoolPath), .pathExpressionFunction = storageSpoolPathExpression); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageSpool); } FN_EXTERN const Storage * storageSpoolWrite(void) { FUNCTION_TEST_VOID(); // Writes not allowed in dry-run mode if (!storageHelper.dryRunInit || storageHelper.dryRun) THROW(AssertError, WRITABLE_WHILE_DRYRUN); if (storageHelper.storageSpoolWrite == NULL) { storageHelperContextInit(); storageHelperStanzaInit(true); MEM_CONTEXT_BEGIN(storageHelper.memContext) { storageHelper.storageSpoolWrite = storagePosixNewP( cfgOptionStr(cfgOptSpoolPath), .write = true, .pathExpressionFunction = storageSpoolPathExpression); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_CONST(STORAGE, storageHelper.storageSpoolWrite); } /**********************************************************************************************************************************/ FN_EXTERN void storageHelperFree(void) { FUNCTION_TEST_VOID(); if (storageHelper.memContext != NULL) memContextFree(storageHelper.memContext); storageHelper = (struct StorageHelperLocal){.memContext = NULL, .helperList = storageHelper.helperList}; FUNCTION_TEST_RETURN_VOID(); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/helper.h0000664000175100017510000000711715240547630017304 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_HELPER_H #define STORAGE_HELPER_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage path constants ***********************************************************************************************************************************/ #define STORAGE_SPOOL_ARCHIVE "" STRING_DECLARE(STORAGE_SPOOL_ARCHIVE_STR); #define STORAGE_SPOOL_ARCHIVE_IN "" STRING_DECLARE(STORAGE_SPOOL_ARCHIVE_IN_STR); #define STORAGE_SPOOL_ARCHIVE_OUT "" STRING_DECLARE(STORAGE_SPOOL_ARCHIVE_OUT_STR); #define STORAGE_REPO_ARCHIVE "" STRING_DECLARE(STORAGE_REPO_ARCHIVE_STR); #define STORAGE_REPO_BACKUP "" STRING_DECLARE(STORAGE_REPO_BACKUP_STR); #define STORAGE_PATH_ARCHIVE "archive" STRING_DECLARE(STORAGE_PATH_ARCHIVE_STR); #define STORAGE_PATH_BACKUP "backup" STRING_DECLARE(STORAGE_PATH_BACKUP_STR); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Initialize dry-run for the current command. No writes are allowed until dry-run has been initialized and no writes are allowed // after initialization if dry-run is true. Note that storageLocalWrite() is exempt from this rule. The primary purpose is to // prevent damage to the repository from an error in dry-run coding in the individual commands. FN_EXTERN void storageHelperDryRunInit(bool dryRun); // Initialize helpers to create storage other than built-in Posix FN_EXTERN void storageHelperInit(const StorageHelper *helperList); // Local storage object. Writable local storage should be used very sparingly. If writes are not needed then always use // storageLocal() or a specific storage object instead. FN_EXTERN const Storage *storageLocal(void); FN_EXTERN const Storage *storageLocalWrite(void); // PostgreSQL storage by cfgOptGrpPg index FN_EXTERN const Storage *storagePgIdx(unsigned int pgIdx); FN_EXTERN const Storage *storagePgIdxWrite(unsigned int pgIdx); // PostgreSQL storage default (calculated from the pg option, when set, or the first cfgOptGrpPg index) FN_EXTERN const Storage *storagePg(void); FN_EXTERN const Storage *storagePgWrite(void); // Repository storage by cfgOptGrpRepo index FN_EXTERN const Storage *storageRepoIdx(unsigned int repoIdx); FN_EXTERN const Storage *storageRepoIdxWrite(unsigned int repoIdx); // Repository storage default (calculated from the repo option, when set, or the first cfgOptGrpPg index) FN_EXTERN const Storage *storageRepo(void); FN_EXTERN const Storage *storageRepoWrite(void); // Return target time for repository storage, if any FN_EXTERN time_t storageRepoTargetTime(void); // Spool storage FN_EXTERN const Storage *storageSpool(void); FN_EXTERN const Storage *storageSpoolWrite(void); // Free cached storage objects FN_EXTERN void storageHelperFree(void); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/info.h0000664000175100017510000001066615240547630016763 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Info ***********************************************************************************************************************************/ #ifndef STORAGE_INFO_H #define STORAGE_INFO_H #include /*********************************************************************************************************************************** Specify the level of information required when calling functions that return StorageInfo Each level includes the level below it, i.e. level storageInfoLevelBasic includes storageInfoLevelType which includes storageInfoLevelExists. ***********************************************************************************************************************************/ typedef enum { // The info type is determined by driver capabilities. This mimics the prior behavior where drivers would always return as much // information as they could. storageInfoLevelDefault = 0, // Only test for existence. All drivers must support this level. storageInfoLevelExists, // Include file type, e.g. storageTypeFile. All drivers must support this level. storageInfoLevelType, // Basic information. All drivers support this level. storageInfoLevelBasic, // Detailed information that is generally only available from filesystems such as Posix storageInfoLevelDetail, } StorageInfoLevel; /*********************************************************************************************************************************** Storage types. The values are used in the remote protocol so must not be changed. ***********************************************************************************************************************************/ typedef enum { storageTypeFile = 0, storageTypePath = 1, storageTypeLink = 2, storageTypeSpecial = 3, } StorageType; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageInfo { // Set when info type >= storageInfoLevelExists const String *name; // Name of path/file/link StorageInfoLevel level; // Level of information provided bool exists; // Does the path/file/link exist? // Set when info type >= storageInfoLevelType (undefined at lower levels) StorageType type; // Type file/path/link) // Set when info type >= storageInfoLevelBasic (undefined at lower levels) uint64_t size; // Size (path/link is 0) time_t timeModified; // Time file was last modified const String *versionId; // Version id when versioning enabled // Set when info type >= storageInfoLevelDetail (undefined at lower levels) mode_t mode; // Mode of path/file/link uid_t userId; // User that owns the file gid_t groupId; // Group that owns the file const String *user; // Name of user that owns the file const String *group; // Name of group that owns the file const String *linkDestination; // Destination if this is a link } StorageInfo; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_INFO_TYPE \ StorageInfo #define FUNCTION_LOG_STORAGE_INFO_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "StorageInfo", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/iterator.c0000664000175100017510000002640315240547630017650 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Iterator ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/list.h" #include "storage/iterator.h" #include "storage/list.h" #include "storage/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageIterator { void *driver; // Storage driver const String *path; // Path to iterate StorageInfoLevel level; // Info level bool recurse; // Recurse into paths SortOrder sortOrder; // Sort order time_t targetTime; // List max version <= time const String *expression; // Match expression RegExp *regExp; // Parsed match expression List *stack; // Stack of info lists bool returnedNext; // Next info was returned StorageInfo infoNext; // Info to be returned by next String *nameNext; // Name for next info }; // Path info list typedef struct StorageIteratorInfo { String *pathSub; // Subpath StorageList *list; // Storage info list unsigned int listIdx; // Current index in info list bool pathContentSkip; // Skip reading path content } StorageIteratorInfo; /*********************************************************************************************************************************** Check a path and add it to the stack if it exists and has content ***********************************************************************************************************************************/ typedef struct StorageItrPathAddResult { bool exists; // Does the path exist? bool content; // Does the path have content? } StorageItrPathAddResult; static StorageItrPathAddResult storageItrPathAdd(StorageIterator *const this, const String *const pathSub) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_ITERATOR, this); FUNCTION_LOG_PARAM(STRING, pathSub); FUNCTION_LOG_END(); ASSERT(this != NULL); StorageItrPathAddResult result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Get path content StorageList *const list = storageInterfaceListP( this->driver, pathSub == NULL ? this->path : strNewFmt("%s/%s", strZ(this->path), strZ(pathSub)), this->level, .expression = this->expression, .targetTime = this->targetTime); // If path exists if (list != NULL) { result.exists = true; // If the path has content if (!storageLstEmpty(list)) { result.content = true; // Sort if requested if (this->sortOrder != sortOrderNone) storageLstSort(list, this->sortOrder); // Add path to top of stack MEM_CONTEXT_OBJ_BEGIN(this->stack) { OBJ_NEW_BASE_BEGIN(StorageIteratorInfo, .childQty = MEM_CONTEXT_QTY_MAX) { StorageIteratorInfo *const listInfo = OBJ_NEW_ALLOC(); *listInfo = (StorageIteratorInfo) { .pathSub = strDup(pathSub), .list = storageLstMove(list, memContextCurrent()), }; lstAdd(this->stack, &listInfo); } OBJ_NEW_END(); } MEM_CONTEXT_OBJ_END(); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_STRUCT(result); } /**********************************************************************************************************************************/ FN_EXTERN StorageIterator * storageItrNew( void *const driver, const String *const path, const StorageInfoLevel level, const bool errorOnMissing, const bool nullOnMissing, const bool recurse, const SortOrder sortOrder, const time_t targetTime, const String *const expression) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(BOOL, errorOnMissing); FUNCTION_LOG_PARAM(BOOL, nullOnMissing); FUNCTION_LOG_PARAM(BOOL, recurse); FUNCTION_LOG_PARAM(ENUM, sortOrder); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_END(); ASSERT(driver != NULL); ASSERT(path != NULL); ASSERT(!recurse || level >= storageInfoLevelType); StorageIterator *this = NULL; MEM_CONTEXT_TEMP_BEGIN() { OBJ_NEW_BASE_BEGIN(StorageIterator, .childQty = MEM_CONTEXT_QTY_MAX) { this = OBJ_NEW_ALLOC(); *this = (StorageIterator) { .driver = driver, .path = strDup(path), .level = level, .recurse = recurse, .sortOrder = sortOrder, .targetTime = targetTime, .expression = strDup(expression), .stack = lstNewP(sizeof(StorageIteratorInfo *)), .nameNext = strNew(), .returnedNext = true, }; // Compile regular expression if (this->expression != NULL) this->regExp = regExpNew(this->expression); // If root path does not exist if (!storageItrPathAdd(this, NULL).exists) { // Throw an error when requested if (errorOnMissing) THROW_FMT(PathMissingError, STORAGE_ERROR_LIST_INFO_MISSING, strZ(this->path)); // Return NULL when requested if (nullOnMissing) this = NULL; } } OBJ_NEW_END(); storageItrMove(this, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_ITERATOR, this); } /**********************************************************************************************************************************/ FN_EXTERN bool storageItrMore(StorageIterator *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_ITERATOR, this); FUNCTION_TEST_END(); ASSERT(this != NULL); // If next has not been called then return true if (!this->returnedNext) FUNCTION_TEST_RETURN(BOOL, true); // Search stack for info while (lstSize(this->stack) != 0) { // Always pull from the top of the stack StorageIteratorInfo *const listInfo = *(StorageIteratorInfo **)lstGetLast(this->stack); // Search list for info for (; listInfo->listIdx < storageLstSize(listInfo->list); listInfo->listIdx++) { // Next info this->infoNext = storageLstGet(listInfo->list, listInfo->listIdx); // Update info name when in subpath if (listInfo->pathSub != NULL) { strCatFmt(strTrunc(this->nameNext), "%s/%s", strZ(listInfo->pathSub), strZ(this->infoNext.name)); this->infoNext.name = this->nameNext; } // Does the path have content? const bool pathContent = this->infoNext.type == storageTypePath && this->recurse && !listInfo->pathContentSkip && storageItrPathAdd(this, this->infoNext.name).content; // Clear path content skip flag if it was set on a previous iteration listInfo->pathContentSkip = false; // Skip info if it does match the provided expression if (this->regExp != NULL && !regExpMatch(this->regExp, this->infoNext.name)) { // Path content may match the expression even if the path does not. Break so the content on the top of the stack // will be checked and increment the index so that path will be skipped. if (pathContent) { listInfo->listIdx++; break; } // Skip info and continue continue; } // When sort order is descending the path will need to be output after the content. Break so that content gets checked // but set a flag so on the next iteration the path will be output but the content skipped. if (pathContent && this->sortOrder == sortOrderDesc) { listInfo->pathContentSkip = true; break; } // Return next info this->returnedNext = false; listInfo->listIdx++; FUNCTION_TEST_RETURN(BOOL, true); } // If no more info then free the list. This check is required because we may break out of the above loop early. // // Only free when the list is at the top of the stack. It is possible that this is the last entry in the current list but // it is a path that must be checked for entries. In that case the path entries will end up at the top of the stack so we'll // need to wait to free the list containing the path until the path entries have been processed. if (listInfo->listIdx >= storageLstSize(listInfo->list) && listInfo == *(StorageIteratorInfo **)lstGetLast(this->stack)) { objFree(listInfo); lstRemoveLast(this->stack); } } FUNCTION_TEST_RETURN(BOOL, false); } /**********************************************************************************************************************************/ FN_EXTERN StorageInfo storageItrNext(StorageIterator *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_ITERATOR, this); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(!this->returnedNext); this->returnedNext = true; FUNCTION_TEST_RETURN(STORAGE_INFO, this->infoNext); } /**********************************************************************************************************************************/ FN_EXTERN void storageItrToLog(const StorageIterator *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{stack: "); lstToLog(this->stack, debugLog); strStcCatChr(debugLog, '}'); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/iterator.h0000664000175100017510000000620115240547630017647 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Iterator ***********************************************************************************************************************************/ #ifndef STORAGE_ITERATOR_H #define STORAGE_ITERATOR_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageIterator StorageIterator; #include "common/type/string.h" #include "storage/info.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageIterator *storageItrNew( void *driver, const String *path, StorageInfoLevel level, bool errorOnMissing, bool nullOnMissing, bool recurse, SortOrder sortOrder, time_t targetTime, const String *expression); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Is there more info to be retrieved from the iterator? FN_EXTERN bool storageItrMore(StorageIterator *this); // Move to a new parent mem context FN_INLINE_ALWAYS StorageIterator * storageItrMove(StorageIterator *const this, MemContext *const parentNew) { return objMove(this, parentNew); } // Get next info. An error will be thrown if there is no more data so use storageItrMore() to check. Note that StorageInfo pointer // members (e.g. name) will be undefined after the next call to storageItrMore(). FN_EXTERN StorageInfo storageItrNext(StorageIterator *this); /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void storageItrFree(StorageIterator *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageItrToLog(const StorageIterator *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_ITERATOR_TYPE \ StorageIterator * #define FUNCTION_LOG_STORAGE_ITERATOR_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageItrToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/list.c0000664000175100017510000002364515240547630016777 0ustar00runnerrunner/*********************************************************************************************************************************** Storage List ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/type/blob.h" #include "common/type/list.h" #include "storage/list.h" #include "storage/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageList { StorageListPub pub; // Publicly accessible variables StringList *ownerList; // List of users/groups Blob *blob; // Blob of info data String *name; // Current info name String *linkDestination; // Current link destination String *versionId; // Current version id }; /*********************************************************************************************************************************** A more space efficient version of StorageInfo. Each level is contained in a struct to ensure alignment when only using part of the struct to store info. This also keeps the size calculations accurate if members are added to a level. ***********************************************************************************************************************************/ typedef struct StorageListInfo { // Set when info type >= storageInfoLevelExists struct { const char *name; // Name of path/file/link } exists; // Set when info type >= storageInfoLevelType (undefined at lower levels). Mode is only provided at higher detail levels but // included here to save space on 64-bit architectures struct { // Set when info type >= storageInfoLevelType (undefined at lower levels) StorageType type; // Type file/path/link) // Set when info type >= storageInfoLevelDetail (undefined at lower levels) mode_t mode; // Mode of path/file/link } type; // Set when info type >= storageInfoLevelBasic (undefined at lower levels) struct { uint64_t size; // Size (path/link is 0) time_t timeModified; // Time file was last modified const char *versionId; // Version id when versioning enabled } basic; // Set when info type >= storageInfoLevelDetail (undefined at lower levels) struct { const String *user; // Name of user that owns the file const String *group; // Name of group that owns the file uid_t userId; // User that owns the file uid_t groupId; // Group that owns the file const char *linkDestination; // Destination if this is a link } detail; } StorageListInfo; // Determine size of struct to store in the list. We only need to store the members that are used by the current level. static const uint8_t storageLstInfoSize[storageInfoLevelDetail + 1] = { 0, // storageInfoLevelDefault (should not be used) offsetof(StorageListInfo, type), // storageInfoLevelExists offsetof(StorageListInfo, basic), // storageInfoLevelType offsetof(StorageListInfo, detail), // storageInfoLevelBasic sizeof(StorageListInfo), // storageInfoLevelDetail }; /**********************************************************************************************************************************/ FN_EXTERN StorageList * storageLstNew(const StorageInfoLevel level) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(ENUM, level); FUNCTION_TEST_END(); ASSERT(level != storageInfoLevelDefault); OBJ_NEW_BEGIN(StorageList, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageList) { .pub = { .list = lstNewP(storageLstInfoSize[level], .comparator = lstComparatorZ), .level = level, }, .ownerList = strLstNew(), .blob = blbNew(), .name = strNew(), .linkDestination = strNew(), .versionId = strNew(), }; } OBJ_NEW_END(); FUNCTION_TEST_RETURN(STORAGE_LIST, this); } /**********************************************************************************************************************************/ FN_EXTERN void storageLstInsert(StorageList *const this, const unsigned int idx, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_LIST, this); FUNCTION_TEST_PARAM(UINT, idx); FUNCTION_TEST_PARAM_P(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(info != NULL); MEM_CONTEXT_OBJ_BEGIN(this->pub.list) { StorageListInfo listInfo = {.exists = {.name = blbAdd(this->blob, strZ(info->name), strSize(info->name) + 1)}}; switch (storageLstLevel(this)) { case storageInfoLevelDetail: { listInfo.type.mode = info->mode; listInfo.detail.user = strLstAddIfMissing(this->ownerList, info->user); listInfo.detail.group = strLstAddIfMissing(this->ownerList, info->group); listInfo.detail.userId = info->userId; listInfo.detail.groupId = info->groupId; if (info->linkDestination != NULL) { listInfo.detail.linkDestination = blbAdd( this->blob, strZ(info->linkDestination), strSize(info->linkDestination) + 1); } __attribute__((fallthrough)); } case storageInfoLevelBasic: { listInfo.basic.size = info->size; listInfo.basic.timeModified = info->timeModified; if (info->versionId != NULL) listInfo.basic.versionId = blbAdd(this->blob, strZ(info->versionId), strSize(info->versionId) + 1); __attribute__((fallthrough)); } case storageInfoLevelType: listInfo.type.type = info->type; __attribute__((fallthrough)); default: break; } lstInsert(this->pub.list, idx, &listInfo); } MEM_CONTEXT_OBJ_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN StorageInfo storageLstGet(const StorageList *const this, const unsigned int idx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_LIST, this); FUNCTION_TEST_PARAM(UINT, idx); FUNCTION_TEST_END(); ASSERT(this != NULL); const StorageListInfo *const listInfo = lstGet(this->pub.list, idx); StorageInfo result = { .name = strCatZ(strTrunc(this->name), listInfo->exists.name), .exists = true, .level = storageLstLevel(this) }; switch (result.level) { case storageInfoLevelDetail: { result.mode = listInfo->type.mode; result.user = listInfo->detail.user; result.group = listInfo->detail.group; result.userId = listInfo->detail.userId; result.groupId = listInfo->detail.groupId; if (listInfo->detail.linkDestination != NULL) result.linkDestination = strCatZ(strTrunc(this->linkDestination), listInfo->detail.linkDestination); __attribute__((fallthrough)); } case storageInfoLevelBasic: { result.size = listInfo->basic.size; result.timeModified = listInfo->basic.timeModified; if (listInfo->basic.versionId != NULL) result.versionId = strCatZ(strTrunc(this->versionId), listInfo->basic.versionId); __attribute__((fallthrough)); } case storageInfoLevelType: result.type = listInfo->type.type; __attribute__((fallthrough)); default: break; } FUNCTION_TEST_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ FN_EXTERN StorageInfo storageLstFind(const StorageList *const this, const String *const name) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_LIST, this); FUNCTION_TEST_PARAM(STRING, name); FUNCTION_TEST_END(); ASSERT(this != NULL); ASSERT(name != NULL); const char *const namePtr = strZ(name); const unsigned int listIdx = lstFindIdx(this->pub.list, &namePtr); if (listIdx == LIST_NOT_FOUND) FUNCTION_TEST_RETURN(STORAGE_INFO, (StorageInfo){.exists = false}); FUNCTION_TEST_RETURN(STORAGE_INFO, storageLstGet(this, listIdx)); } /**********************************************************************************************************************************/ FN_EXTERN void storageLstToLog(const StorageList *const this, StringStatic *const debugLog) { strStcFmt(debugLog, "{size: %u}", lstSize(this->pub.list)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/list.h0000664000175100017510000001047615240547630017002 0ustar00runnerrunner/*********************************************************************************************************************************** Storage List ***********************************************************************************************************************************/ #ifndef STORAGE_LIST_H #define STORAGE_LIST_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageList StorageList; #include "common/type/string.h" #include "storage/info.h" /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageList *storageLstNew(StorageInfoLevel level); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct StorageListPub { List *list; // Storage list StorageInfoLevel level; // Storage info level } StorageListPub; // Empty? FN_INLINE_ALWAYS bool storageLstEmpty(const StorageList *const this) { return lstEmpty(THIS_PUB(StorageList)->list); } // Storage info level FN_INLINE_ALWAYS StorageInfoLevel storageLstLevel(const StorageList *const this) { return THIS_PUB(StorageList)->level; } // List size FN_INLINE_ALWAYS unsigned int storageLstSize(const StorageList *const this) { return lstSize(THIS_PUB(StorageList)->list); } // List size FN_INLINE_ALWAYS StorageList * storageLstSort(StorageList *const this, const SortOrder sortOrder) { return (StorageList *)lstSort(THIS_PUB(StorageList)->list, sortOrder); } /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Insert info FN_EXTERN void storageLstInsert(StorageList *this, unsigned int idx, const StorageInfo *info); // Add info FN_INLINE_ALWAYS void storageLstAdd(StorageList *const this, const StorageInfo *const info) { storageLstInsert(this, storageLstSize(this), info); } // Get info. Note that StorageInfo pointer members (e.g. name) will be undefined after the next call to storageLstGet(). FN_EXTERN StorageInfo storageLstGet(const StorageList *this, unsigned int idx); // Find info. Note that StorageInfo pointer members (e.g. name) will be undefined after the next call to storageLstGet(). FN_EXTERN StorageInfo storageLstFind(const StorageList *this, const String *name); // Move to a new parent mem context FN_INLINE_ALWAYS StorageList * storageLstMove(StorageList *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void storageLstFree(StorageList *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageLstToLog(const StorageList *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_LIST_TYPE \ StorageList * #define FUNCTION_LOG_STORAGE_LIST_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageLstToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/posix/0000775000175100017510000000000015240547630017010 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/posix/read.c0000664000175100017510000002115515240547630020073 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage Read ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/object.h" #include "storage/posix/read.h" #include "storage/read.h" /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ struct StorageReadPosix { const StorageReadInterface *interface; // Interface StoragePosix *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) int fd; // File descriptor uint64_t current; // Current bytes read from file bool eof; }; /*********************************************************************************************************************************** Close file descriptor ***********************************************************************************************************************************/ static void storageReadPosixFreeResource(THIS_VOID) { THIS(StorageReadPosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); if (this->fd != -1) THROW_ON_SYS_ERROR_FMT(close(this->fd) == -1, FileCloseError, STORAGE_ERROR_READ_CLOSE, strZ(this->name)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadPosixOpen(THIS_VOID) { THIS(StorageReadPosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->fd == -1); // Open the file this->fd = open(strZ(this->name), O_RDONLY, 0); // Handle errors except missing, which is reported to the caller via the result if (this->fd == -1) { if (errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(FileOpenError, STORAGE_ERROR_READ_OPEN, strZ(this->name)); // {vm_covered} } // Else success else { // Set free callback to ensure the file descriptor is freed memContextCallbackSet(objMemContext(this), storageReadPosixFreeResource, this); // Seek to offset if (this->offset != 0) { THROW_ON_SYS_ERROR_FMT( lseek(this->fd, (off_t)this->offset, SEEK_SET) == -1, FileOpenError, STORAGE_ERROR_READ_SEEK, this->offset, strZ(this->name)); } } FUNCTION_LOG_RETURN(BOOL, this->fd != -1); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadPosix(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadPosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_POSIX, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL && this->fd != -1); ASSERT(buffer != NULL && !bufFull(buffer)); // Read if EOF has not been reached ssize_t actualBytes = 0; if (!this->eof) { // Rather than enable/disable limit checking just use a big number when there is no limit. We can feel pretty confident that // no files will be > UINT64_MAX in size. const uint64_t limit = this->limit == NULL ? UINT64_MAX : varUInt64(this->limit); // Determine expected bytes to read. If remaining size in the buffer would exceed the limit then reduce the expected read. size_t expectedBytes = bufRemains(buffer); if (this->current + expectedBytes > limit) expectedBytes = (size_t)(limit - this->current); // Read from file actualBytes = read(this->fd, bufRemainsPtr(buffer), expectedBytes); // Error occurred during read if (actualBytes == -1) THROW_SYS_ERROR_FMT(FileReadError, "unable to read '%s'", strZ(this->name)); // Update amount of buffer used bufUsedInc(buffer, (size_t)actualBytes); this->current += (uint64_t)actualBytes; // If less data than expected was read or the limit has been reached then EOF. The file may not actually be EOF but we are // not concerned with files that are growing. Just read up to the point where the file is being extended. if ((size_t)actualBytes != expectedBytes || this->current == limit) this->eof = true; } FUNCTION_LOG_RETURN(SIZE, (size_t)actualBytes); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadPosixClose(THIS_VOID) { THIS(StorageReadPosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); memContextCallbackClear(objMemContext(this)); storageReadPosixFreeResource(this); this->fd = -1; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadPosixEof(THIS_VOID) { THIS(StorageReadPosix); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_POSIX, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->eof); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int storageReadPosixFd(const THIS_VOID) { THIS(const StorageReadPosix); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_POSIX, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadPosixInterface = { .close = storageReadPosixClose, .eof = storageReadPosixEof, .fd = storageReadPosixFd, .open = storageReadPosixOpen, .read = storageReadPosix, }; FN_EXTERN StorageReadPosix * storageReadPosixNew(StoragePosix *const storage, const String *const name, const uint64_t offset, const Variant *const limit) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_END(); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadPosix, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageReadPosix) { .interface = &storageReadPosixInterface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), .fd = -1, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_POSIX, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/posix/read.h0000664000175100017510000000342015240547630020073 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_POSIX_READ_H #define STORAGE_POSIX_READ_H #include "storage/posix/storage.intern.h" #include "storage/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadPosix StorageReadPosix; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadPosix *storageReadPosixNew(StoragePosix *storage, const String *name, uint64_t offset, const Variant *limit); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_POSIX_TYPE \ StorageReadPosix * #define FUNCTION_LOG_STORAGE_READ_POSIX_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadPosix", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/posix/storage.c0000664000175100017510000006042415240547630020626 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage ***********************************************************************************************************************************/ #include #include #include #include #include #include #include #include #include #include "common/debug.h" #include "common/log.h" #include "common/regExp.h" #include "common/user.h" #include "storage/posix/read.h" #include "storage/posix/storage.intern.h" #include "storage/posix/write.h" /*********************************************************************************************************************************** Define PATH_MAX if it is not defined ***********************************************************************************************************************************/ #ifndef PATH_MAX #define PATH_MAX (4 * 1024) #endif /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StoragePosix { STORAGE_COMMON_MEMBER; }; /**********************************************************************************************************************************/ static StorageInfo storagePosixInfo(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(BOOL, param.followLink); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(file != NULL); StorageInfo result = {.level = level}; // Stat the file to check if it exists struct stat statFile; if ((param.followLink ? stat(strZ(file), &statFile) : lstat(strZ(file), &statFile)) == -1) { if (errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(FileOpenError, STORAGE_ERROR_INFO, strZ(file)); // {vm_covered} } // On success the file exists else { result.exists = true; // Add type info (no need set file type since it is the default) if (result.level >= storageInfoLevelType && !S_ISREG(statFile.st_mode)) { if (S_ISDIR(statFile.st_mode)) result.type = storageTypePath; else if (S_ISLNK(statFile.st_mode)) result.type = storageTypeLink; else result.type = storageTypeSpecial; } // Add basic level info if (result.level >= storageInfoLevelBasic) { result.timeModified = statFile.st_mtime; if (result.type == storageTypeFile) result.size = (uint64_t)statFile.st_size; } // Add detail level info if (result.level >= storageInfoLevelDetail) { result.groupId = statFile.st_gid; result.group = groupNameFromId(result.groupId); result.userId = statFile.st_uid; result.user = userNameFromId(result.userId); result.mode = statFile.st_mode & (S_IRWXU | S_IRWXG | S_IRWXO); if (result.type == storageTypeLink) { char linkDestination[PATH_MAX]; ssize_t linkDestinationSize = 0; THROW_ON_SYS_ERROR_FMT( (linkDestinationSize = readlink(strZ(file), linkDestination, sizeof(linkDestination) - 1)) == -1, FileReadError, "unable to get destination for link '%s'", strZ(file)); result.linkDestination = strNewZN(linkDestination, (size_t)linkDestinationSize); } } } FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storagePosixLinkCreate( THIS_VOID, const String *const target, const String *const linkPath, const StorageInterfaceLinkCreateParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, target); FUNCTION_LOG_PARAM(STRING, linkPath); FUNCTION_LOG_PARAM(ENUM, param.linkType); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(target != NULL); ASSERT(linkPath != NULL); // Create symlink if (param.linkType == storageLinkSym) { THROW_ON_SYS_ERROR_FMT( symlink(strZ(target), strZ(linkPath)) == -1, FileOpenError, "unable to create symlink '%s' to '%s'", strZ(linkPath), strZ(target)); } // Else create hardlink else { ASSERT(param.linkType == storageLinkHard); THROW_ON_SYS_ERROR_FMT( link(strZ(target), strZ(linkPath)) == -1, FileOpenError, "unable to create hardlink '%s' to '%s'", strZ(linkPath), strZ(target)); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ // Helper function to get info for a file if it exists. This logic can't live directly in storagePosixList() because there is a race // condition where a file might exist while listing the directory but it is gone before stat() can be called. In order to get // complete test coverage this function must be split out. static void storagePosixListEntry( StoragePosix *const this, StorageList *const list, const String *const path, const char *const name, const StorageInfoLevel level) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_POSIX, this); FUNCTION_TEST_PARAM(STORAGE_LIST, list); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(STRINGZ, name); FUNCTION_TEST_PARAM(ENUM, level); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(this != NULL); ASSERT(list != NULL); ASSERT(path != NULL); ASSERT(name != NULL); StorageInfo info = storageInterfaceInfoP(this, strNewFmt("%s/%s", strZ(path), name), level); if (info.exists) { info.name = STR(name); storageLstAdd(list, &info); } FUNCTION_TEST_RETURN_VOID(); } static StorageList * storagePosixList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); ASSERT(param.targetTime == 0); StorageList *result = NULL; // Open the directory for read DIR *const dir = opendir(strZ(path)); // If the directory could not be opened process errors and report missing directories if (dir == NULL) { if (errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(PathOpenError, STORAGE_ERROR_LIST_INFO, strZ(path)); // {vm_covered} } // Directory was found else { result = storageLstNew(level); TRY_BEGIN() { MEM_CONTEXT_TEMP_RESET_BEGIN() { // Read the directory entries const struct dirent *dirEntry = readdir(dir); while (dirEntry != NULL) { // Always skip . and .. if (!strEqZ(DOT_STR, dirEntry->d_name) && !strEqZ(DOTDOT_STR, dirEntry->d_name)) { // If only making a list of files that exist then no need to go get detailed info which requires calling // stat() and is therefore relatively slow if (level == storageInfoLevelExists) { const StorageInfo storageInfo = { .name = STR(dirEntry->d_name), .level = storageInfoLevelExists, .exists = true, }; storageLstAdd(result, &storageInfo); } // Else more info is required which requires a call to stat() else storagePosixListEntry(this, result, path, dirEntry->d_name, level); } // Get next entry dirEntry = readdir(dir); // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } FINALLY() { closedir(dir); } TRY_END(); } FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static bool storagePosixMove(THIS_VOID, StorageRead *const source, StorageWrite *const destination, const StorageInterfaceMoveParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STORAGE_READ, source); FUNCTION_LOG_PARAM(STORAGE_WRITE, destination); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(source != NULL); ASSERT(destination != NULL); bool result = true; MEM_CONTEXT_TEMP_BEGIN() { const String *const sourceFile = storageReadName(source); const String *const destinationFile = storageWriteName(destination); const String *const destinationPath = strPath(destinationFile); // Attempt to move the file if (rename(strZ(sourceFile), strZ(destinationFile)) == -1) { // Determine which file/path is missing if (errno == ENOENT) { // Check if the source is missing. Rename does not follow links so there is no need to set followLink. if (!storageInterfaceInfoP(this, sourceFile, storageInfoLevelExists).exists) THROW_SYS_ERROR_FMT(FileMissingError, "unable to move missing source '%s'", strZ(sourceFile)); if (!storageWriteCreatePath(destination)) { THROW_SYS_ERROR_FMT( PathMissingError, "unable to move '%s' to missing path '%s'", strZ(sourceFile), strZ(destinationPath)); } storageInterfacePathCreateP(this, destinationPath, false, false, storageWriteModePath(destination)); result = storageInterfaceMoveP(this, source, destination); } // Else the destination is on a different device so a copy will be needed else if (errno == EXDEV) // {vm_covered} { result = false; } else { THROW_SYS_ERROR_FMT( // {vm_covered} FileMoveError, "unable to move '%s' to '%s'", strZ(sourceFile), strZ(destinationFile)); // {vm_covered} } } // Sync paths on success else { // Sync source path if the destination path was synced and the paths are not equal if (storageWriteSyncPath(destination)) { const String *const sourcePath = strPath(sourceFile); if (!strEq(destinationPath, sourcePath)) storageInterfacePathSyncP(this, sourcePath); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ static void * storagePosixNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.versionId == NULL); FUNCTION_LOG_RETURN(STORAGE_READ_POSIX, storageReadPosixNew(this, file, param.offset, param.limit)); } /**********************************************************************************************************************************/ static void * storagePosixNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(STRING, param.user); FUNCTION_LOG_PARAM(STRING, param.group); FUNCTION_LOG_PARAM(TIME, param.timeModified); FUNCTION_LOG_PARAM(BOOL, param.createPath); FUNCTION_LOG_PARAM(BOOL, param.syncFile); FUNCTION_LOG_PARAM(BOOL, param.syncPath); FUNCTION_LOG_PARAM(BOOL, param.atomic); FUNCTION_LOG_PARAM(BOOL, param.truncate); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN( STORAGE_WRITE_POSIX, storageWritePosixNew( this, file, param.modeFile, param.modePath, param.user, param.group, param.timeModified, param.createPath, param.syncFile, this->interface.pathSync != NULL ? param.syncPath : false, param.atomic, param.truncate)); } /**********************************************************************************************************************************/ static void storagePosixPathCreate( THIS_VOID, const String *const path, const bool errorOnExists, const bool noParentCreate, const mode_t mode, const StorageInterfacePathCreateParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, errorOnExists); FUNCTION_LOG_PARAM(BOOL, noParentCreate); FUNCTION_LOG_PARAM(MODE, mode); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); // Attempt to create the directory if (mkdir(strZ(path), mode) == -1) { // If the parent path does not exist then create it if allowed if (errno == ENOENT && !noParentCreate) { String *const pathParent = strPath(path); storageInterfacePathCreateP(this, pathParent, errorOnExists, noParentCreate, mode); storageInterfacePathCreateP(this, path, errorOnExists, noParentCreate, mode); strFree(pathParent); } // Ignore path exists if allowed else if (errno != EEXIST || errorOnExists) THROW_SYS_ERROR_FMT(PathCreateError, "unable to create path '%s'", strZ(path)); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static bool storagePosixPathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, recurse); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); bool result = true; MEM_CONTEXT_TEMP_BEGIN() { // Recurse if requested if (recurse) { StorageList *const list = storageInterfaceListP(this, path, storageInfoLevelExists); if (list != NULL) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int listIdx = 0; listIdx < storageLstSize(list); listIdx++) { const String *const file = strNewFmt("%s/%s", strZ(path), strZ(storageLstGet(list, listIdx).name)); // Rather than stat the file to discover what type it is, just try to unlink it and see what happens if (unlink(strZ(file)) == -1) // {vm_covered} { // These errors indicate that the entry is actually a path so we'll try to delete it that way if (errno == EPERM || errno == EISDIR) // {uncovered_branch - no EPERM on tested systems} { storageInterfacePathRemoveP(this, file, true); } // Else error else THROW_SYS_ERROR_FMT(PathRemoveError, STORAGE_ERROR_FILE_REMOVE, strZ(file)); // {vm_covered} } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } } // Delete the path if (rmdir(strZ(path)) == -1) { if (errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(PathRemoveError, STORAGE_ERROR_PATH_REMOVE, strZ(path)); // {vm_covered} // Path does not exist result = false; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ static void storagePosixPathSync(THIS_VOID, const String *const path, const StorageInterfacePathSyncParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, path); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); // Open directory and handle errors const int fd = open(strZ(path), O_RDONLY, 0); // Handle errors if (fd == -1) { if (errno == ENOENT) // {vm_covered} THROW_FMT(PathMissingError, STORAGE_ERROR_PATH_SYNC_MISSING, strZ(path)); else THROW_SYS_ERROR_FMT(PathOpenError, STORAGE_ERROR_PATH_SYNC_OPEN, strZ(path)); // {vm_covered} } else { // Attempt to sync the directory if (fsync(fd) == -1) { const int errNo = errno; // Close the file descriptor to free resources but don't check for failure close(fd); THROW_SYS_ERROR_CODE_FMT(errNo, PathSyncError, STORAGE_ERROR_PATH_SYNC, strZ(path)); } THROW_ON_SYS_ERROR_FMT(close(fd) == -1, PathCloseError, STORAGE_ERROR_PATH_SYNC_CLOSE, strZ(path)); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void storagePosixRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StoragePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); // Attempt to unlink the file if (unlink(strZ(file)) == -1) { if (param.errorOnMissing || errno != ENOENT) // {vm_covered} THROW_SYS_ERROR_FMT(FileRemoveError, STORAGE_ERROR_FILE_REMOVE, strZ(file)); // {vm_covered} } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfacePosix = { .feature = 1 << storageFeaturePath, .info = storagePosixInfo, .linkCreate = storagePosixLinkCreate, .list = storagePosixList, .move = storagePosixMove, .newRead = storagePosixNewRead, .newWrite = storagePosixNewWrite, .pathCreate = storagePosixPathCreate, .pathRemove = storagePosixPathRemove, .pathSync = storagePosixPathSync, .remove = storagePosixRemove, }; FN_EXTERN Storage * storagePosixNewInternal( const StringId type, const String *const path, const mode_t modeFile, const mode_t modePath, const bool write, StoragePathExpressionCallback pathExpressionFunction, const bool pathSync, const bool symLink) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(BOOL, pathSync); FUNCTION_LOG_PARAM(BOOL, symLink); FUNCTION_LOG_END(); ASSERT(type != 0); ASSERT(path != NULL); ASSERT(modeFile != 0); ASSERT(modePath != 0); // Initialize user module userInit(); // Create the object OBJ_NEW_BEGIN(StoragePosix, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StoragePosix) { .interface = storageInterfacePosix, }; // Disable path sync when not supported if (!pathSync) this->interface.pathSync = NULL; // If this is a posix driver then add link features if (type == STORAGE_POSIX_TYPE) { this->interface.feature |= 1 << storageFeatureHardLink | (unsigned int)symLink << storageFeatureSymLink | 1 << storageFeaturePathSync | 1 << storageFeatureInfoDetail | 1 << storageFeatureFileRemoveMissing; } } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew(type, path, modeFile, modePath, write, 0, pathExpressionFunction, this, this->interface)); } FN_EXTERN Storage * storagePosixNew(const String *const path, const StoragePosixNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(BOOL, param.write); FUNCTION_LOG_PARAM(BOOL, param.noSymLink); FUNCTION_LOG_PARAM(FUNCTIONP, param.pathExpressionFunction); FUNCTION_LOG_END(); FUNCTION_LOG_RETURN( STORAGE, storagePosixNewInternal( STORAGE_POSIX_TYPE, path, param.modeFile == 0 ? STORAGE_MODE_FILE_DEFAULT : param.modeFile, param.modePath == 0 ? STORAGE_MODE_PATH_DEFAULT : param.modePath, param.write, param.pathExpressionFunction, true, !param.noSymLink)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/posix/storage.h0000664000175100017510000000315715240547630020633 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage ***********************************************************************************************************************************/ #ifndef STORAGE_POSIX_STORAGE_H #define STORAGE_POSIX_STORAGE_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_POSIX_TYPE STRID5("posix", 0x184cdf00) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct StoragePosixNewParam { VAR_PARAM_HEADER; bool write; bool noSymLink; // Do not create symlinks on this storage mode_t modeFile; mode_t modePath; StoragePathExpressionCallback *pathExpressionFunction; } StoragePosixNewParam; #define storagePosixNewP(path, ...) \ storagePosixNew(path, (StoragePosixNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Storage *storagePosixNew(const String *path, StoragePosixNewParam param); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/posix/storage.intern.h0000664000175100017510000000351715240547630022131 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_POSIX_STORAGE_INTERN_H #define STORAGE_POSIX_STORAGE_INTERN_H #include "storage/posix/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StoragePosix StoragePosix; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storagePosixNewInternal( StringId type, const String *path, mode_t modeFile, mode_t modePath, bool write, StoragePathExpressionCallback pathExpressionFunction, bool pathSync, bool symLink); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_POSIX_TYPE \ StoragePosix * #define FUNCTION_LOG_STORAGE_POSIX_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StoragePosix *", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/posix/write.c0000664000175100017510000003173415240547630020316 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage File write ***********************************************************************************************************************************/ #include #include #include #include #include #include "common/debug.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" #include "common/user.h" #include "storage/posix/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWritePosix { const StorageWriteInterface *interface; // Interface StoragePosix *storage; // Storage that created this object const String *name; // File name mode_t modeFile; // File mode mode_t modePath; // Path mode bool createPath; // Create path if missing bool atomic; // Atomic write bool truncate; // Truncate existing file bool syncPath; // Sync path after write bool syncFile; // Sync file after write const String *user; // User owner const String *group; // Group owner time_t timeModified; // Modified time const String *nameTmp; const String *path; int fd; // File descriptor }; /*********************************************************************************************************************************** File open constants Since open is called more than once use constants to make sure these parameters are always the same ***********************************************************************************************************************************/ #define FILE_OPEN_PURPOSE "write" /*********************************************************************************************************************************** Close file descriptor ***********************************************************************************************************************************/ static void storageWritePosixFreeResource(THIS_VOID) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); THROW_ON_SYS_ERROR_FMT(close(this->fd) == -1, FileCloseError, STORAGE_ERROR_WRITE_CLOSE, strZ(this->nameTmp)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static const char * storageWritePosixOpenOwner(const String *const owner, const char *const defaultOwner) { return owner == NULL ? defaultOwner : strZ(owner); } static const char * storageWritePosixOpenOwnerId(const String *const owner, const unsigned int ownerId) { return owner == NULL ? "" : zNewFmt("[%u]", ownerId); } static void storageWritePosixOpen(THIS_VOID) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->fd == -1); // Open the file const int flags = O_CREAT | O_WRONLY | (this->truncate ? O_TRUNC : 0); this->fd = open(strZ(this->nameTmp), flags, this->modeFile); // Attempt to create the path if it is missing if (this->fd == -1 && errno == ENOENT && this->createPath) // {vm_covered} { // Create the path storageInterfacePathCreateP(this->storage, this->path, false, false, this->modePath); // Open file again this->fd = open(strZ(this->nameTmp), flags, this->modeFile); } // Handle errors if (this->fd == -1) { if (errno == ENOENT) // {vm_covered} THROW_FMT(FileMissingError, STORAGE_ERROR_WRITE_MISSING, strZ(this->name)); else THROW_SYS_ERROR_FMT(FileOpenError, STORAGE_ERROR_WRITE_OPEN, strZ(this->name)); // {vm_covered} } // Set free callback to ensure the file descriptor is freed memContextCallbackSet(objMemContext(this), storageWritePosixFreeResource, this); // Update user/group owner if (this->user != NULL || this->group != NULL) { MEM_CONTEXT_TEMP_BEGIN() { const StorageInfo info = storageInterfaceInfoP( this->storage, this->nameTmp, storageInfoLevelDetail, .followLink = true); ASSERT(info.exists); uid_t updateUserId = userIdFromName(this->user); gid_t updateGroupId = groupIdFromName(this->group); if (updateUserId == (uid_t)-1) updateUserId = info.userId; if (updateGroupId == (gid_t)-1) updateGroupId = info.groupId; // Continue if one of the owners would be changed if (updateUserId != info.userId || updateGroupId != info.groupId) { THROW_ON_SYS_ERROR_FMT( chown(strZ(this->nameTmp), updateUserId, updateGroupId) == -1, FileOwnerError, "unable to set ownership for '%s' to %s%s:%s%s from %s[%u]:%s[%u]", strZ(this->nameTmp), storageWritePosixOpenOwner(this->user, "[none]"), storageWritePosixOpenOwnerId(this->user, updateUserId), storageWritePosixOpenOwner(this->group, "[none]"), storageWritePosixOpenOwnerId(this->group, updateGroupId), storageWritePosixOpenOwner(info.user, "[unknown]"), info.userId, storageWritePosixOpenOwner(info.group, "[unknown]"), info.groupId); } } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to the file ***********************************************************************************************************************************/ static void storageWritePosix(THIS_VOID, const Buffer *const buffer) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->fd != -1); // Write the data if (write(this->fd, bufPtrConst(buffer), bufUsed(buffer)) != (ssize_t)bufUsed(buffer)) THROW_SYS_ERROR_FMT(FileWriteError, "unable to write '%s'", strZ(this->nameTmp)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Seek to specified position relative to beginning of file ***********************************************************************************************************************************/ static void storageWritePosixSeek(THIS_VOID, const uint64_t position) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_PARAM(UINT64, position); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->fd != -1); THROW_ON_SYS_ERROR_FMT( lseek(this->fd, (off_t)position, SEEK_SET) == -1, FileWriteError, STORAGE_ERROR_WRITE_SEEK, position, strZ(this->nameTmp)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWritePosixClose(THIS_VOID) { THIS(StorageWritePosix); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->fd != -1) { // Sync the file if (this->syncFile) THROW_ON_SYS_ERROR_FMT(fsync(this->fd) == -1, FileSyncError, STORAGE_ERROR_WRITE_SYNC, strZ(this->nameTmp)); // Close the file memContextCallbackClear(objMemContext(this)); THROW_ON_SYS_ERROR_FMT(close(this->fd) == -1, FileCloseError, STORAGE_ERROR_WRITE_CLOSE, strZ(this->nameTmp)); this->fd = -1; // Update modified time if (this->timeModified != 0) { THROW_ON_SYS_ERROR_FMT( utime( strZ(this->nameTmp), &((struct utimbuf){.actime = this->timeModified, .modtime = this->timeModified})) == -1, FileInfoError, "unable to set time for '%s'", strZ(this->nameTmp)); } // Rename from temp file if (this->atomic) { if (rename(strZ(this->nameTmp), strZ(this->name)) == -1) THROW_SYS_ERROR_FMT(FileMoveError, "unable to move '%s' to '%s'", strZ(this->nameTmp), strZ(this->name)); } // Sync the path if (this->syncPath) storageInterfacePathSyncP(this->storage, this->path); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int storageWritePosixFd(const THIS_VOID) { THIS(const StorageWritePosix); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_WRITE_POSIX, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, this->fd); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWritePosixInterface = { .close = storageWritePosixClose, .fd = storageWritePosixFd, .open = storageWritePosixOpen, .seek = storageWritePosixSeek, .write = storageWritePosix, }; FN_EXTERN StorageWritePosix * storageWritePosixNew( StoragePosix *const storage, const String *const name, const mode_t modeFile, const mode_t modePath, const String *const user, const String *const group, const time_t timeModified, const bool createPath, const bool syncFile, const bool syncPath, const bool atomic, const bool truncate) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_POSIX, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(STRING, group); FUNCTION_LOG_PARAM(TIME, timeModified); FUNCTION_LOG_PARAM(BOOL, createPath); FUNCTION_LOG_PARAM(BOOL, syncFile); FUNCTION_LOG_PARAM(BOOL, syncPath); FUNCTION_LOG_PARAM(BOOL, atomic); FUNCTION_LOG_PARAM(BOOL, truncate); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); ASSERT(modeFile != 0); ASSERT(modePath != 0); OBJ_NEW_BEGIN(StorageWritePosix, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageWritePosix) { .interface = &storageWritePosixInterface, .storage = storage, .name = strDup(name), .modeFile = modeFile, .modePath = modePath, .createPath = createPath, .atomic = atomic, .truncate = truncate, .syncFile = syncFile, .syncPath = syncPath, .user = strDup(user), .group = strDup(group), .timeModified = timeModified, .nameTmp = atomic ? strNewFmt("%s." STORAGE_FILE_TEMP_EXT, strZ(name)) : strDup(name), .path = strPath(name), .fd = -1, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_POSIX, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/posix/write.h0000664000175100017510000000365415240547630020323 0ustar00runnerrunner/*********************************************************************************************************************************** Posix Storage File write ***********************************************************************************************************************************/ #ifndef STORAGE_POSIX_WRITE_H #define STORAGE_POSIX_WRITE_H #include "storage/posix/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWritePosix StorageWritePosix; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWritePosix *storageWritePosixNew( StoragePosix *storage, const String *name, mode_t modeFile, mode_t modePath, const String *user, const String *group, time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool truncate); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_POSIX_TYPE \ StorageWritePosix * #define FUNCTION_LOG_STORAGE_WRITE_POSIX_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWritePosix", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/read.c0000664000175100017510000002606415240547630016735 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Read Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/wait.h" #include "storage/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageRead { StorageReadPub pub; // Publicly accessible variables const Storage *storage; // Storage void *driver; // Driver uint64_t bytesRead; // Bytes that have been successfully read bool retry; // Are read retries allowed? bool compressible; // Is the read compressible? }; /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_INTERFACE_TYPE \ StorageReadInterface #define FUNCTION_LOG_STORAGE_READ_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "StorageReadInterface", buffer, bufferSize) /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadOpen(THIS_VOID) { THIS(StorageRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = false; // Open if not versioned or if versionId is not null if (!this->pub.version || this->pub.versionId != NULL) result = storageReadDriverInterface(this->driver)->open(this->driver); // Error when the file is missing and missing files are not ignored if (!result && !this->pub.ignoreMissing) THROW_FMT(FileMissingError, STORAGE_ERROR_READ_MISSING, strZ(storageReadName(this))); FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Read from a file and retry when there is a read failure ***********************************************************************************************************************************/ static size_t storageRead(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); const size_t bufUsedBegin = bufUsed(buffer); size_t result = 0; MEM_CONTEXT_TEMP_BEGIN() { // Use a specific number of tries here instead of a timeout because the underlying operations already have timeouts and // failures will generally happen after some delay, so it is not clear what timeout would be appropriate here. unsigned int try = this->retry ? 3 : 1; // While tries remaining while (try > 0) { TRY_BEGIN() { storageReadDriverInterface(this->driver)->read(this->driver, buffer, block); // Account for bytes that have been read result += bufUsed(buffer) - bufUsedBegin; this->bytesRead += bufUsed(buffer) - bufUsedBegin; // Set try to 1 to exit the loop try = 1; } CATCH_ANY() { // If there is another try remaining then close the file and reopen it to the new position, taking into account any // bytes that have already been read if (try > 1) { // Close the file storageReadDriverInterface(this->driver)->close(this->driver); // Ignore partial reads and restart from the last successful read bufUsedSet(buffer, bufUsedBegin); // Driver with new offset/limit MEM_CONTEXT_OBJ_BEGIN(this) { objFree(this->driver); this->driver = storageInterfaceNewReadP( storageDriver(this->storage), this->pub.name, .compressible = this->compressible, .offset = this->pub.offset + this->bytesRead, .limit = this->pub.limit != NULL ? varNewUInt64(varUInt64(this->pub.limit) - this->bytesRead) : NULL, .versionId = this->pub.versionId); // Error when the file is missing since it must have existed when the read began if (!storageReadDriverInterface(this->driver)->open(this->driver)) THROW_FMT(FileMissingError, STORAGE_ERROR_READ_MISSING, strZ(storageReadName(this))); } MEM_CONTEXT_OBJ_END(); } else RETHROW(); } TRY_END(); try--; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadClose(THIS_VOID) { THIS(StorageRead); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ, this); FUNCTION_LOG_END(); ASSERT(this != NULL); storageReadDriverInterface(this->driver)->close(this->driver); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadEof(THIS_VOID) { THIS(StorageRead); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, storageReadDriverInterface(this->driver)->eof(this->driver)); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int storageReadFd(const THIS_VOID) { THIS(const StorageRead); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, storageReadDriverInterface(this->driver)->fd(this->driver)); } /**********************************************************************************************************************************/ static const IoReadInterface storageIoReadInterface = { .open = storageReadOpen, .read = storageRead, .close = storageReadClose, .eof = storageReadEof, .fd = storageReadFd, }; FN_EXTERN StorageRead * storageReadNew( const Storage *const storage, const String *const name, const bool ignoreMissing, const bool compressible, const uint64_t offset, const Variant *const limit, const bool version, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(BOOL, ignoreMissing); FUNCTION_LOG_PARAM(BOOL, compressible); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(BOOL, version); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageRead, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageRead) { .storage = storage, .driver = storageInterfaceNewReadP( storageDriver(storage), name, .compressible = compressible, .offset = offset, .limit = limit, .versionId = versionId), .retry = storageFeature(storage, storageFeatureReadRetry), .compressible = compressible, .pub = { .type = storageType(storage), .name = strDup(name), .ignoreMissing = ignoreMissing, .offset = offset, .limit = varDup(limit), .version = version, .versionId = strDup(versionId), }, }; ASSERT(storageReadDriverInterface(this->driver)->eof != NULL); ASSERT(storageReadDriverInterface(this->driver)->close != NULL); ASSERT(storageReadDriverInterface(this->driver)->open != NULL); ASSERT(storageReadDriverInterface(this->driver)->read != NULL); // Remove fd method if it does not exist in the driver IoReadInterface storageIoReadInterfaceCopy = storageIoReadInterface; if (storageReadDriverInterface(this->driver)->fd == NULL) storageIoReadInterfaceCopy.fd = NULL; this->pub.io = ioReadNew(this, storageIoReadInterfaceCopy); // Set filter group when interface function exists if (storageReadDriverInterface(this->driver)->filterGroup != NULL) storageReadDriverInterface(this->driver)->filterGroup(this->driver, ioReadFilterGroup(storageReadIo(this))); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ, this); } /**********************************************************************************************************************************/ FN_EXTERN void storageReadToLog(const StorageRead *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(storageReadType(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt( debugLog, ", name: %s, ignoreMissing: %s}", strZ(storageReadName(this)), cvtBoolToConstZ(storageReadIgnoreMissing(this))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/read.h0000664000175100017510000001031615240547630016733 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Read Interface ***********************************************************************************************************************************/ #ifndef STORAGE_READ_H #define STORAGE_READ_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageRead StorageRead; #include "common/io/read.h" #include "common/type/object.h" #include "common/type/stringId.h" #include "storage/read.intern.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_INLINE_ALWAYS StorageRead * storageReadMove(StorageRead *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct StorageReadPub { StringId type; // Storage type const String *name; // File name uint64_t offset; // Where to start reading in the file Variant *limit; // Limit how many bytes are read (NULL for no limit) bool ignoreMissing; // Ignore missing file? bool version; // Read version const String *versionId; // File version to read IoRead *io; // Read interface } StorageReadPub; // Should a missing file be ignored? FN_INLINE_ALWAYS bool storageReadIgnoreMissing(const StorageRead *const this) { return THIS_PUB(StorageRead)->ignoreMissing; } // Read interface FN_INLINE_ALWAYS IoRead * storageReadIo(const StorageRead *const this) { return THIS_PUB(StorageRead)->io; } // Is there a read limit? NULL for no limit. FN_INLINE_ALWAYS const Variant * storageReadLimit(const StorageRead *const this) { return THIS_PUB(StorageRead)->limit; } // File name FN_INLINE_ALWAYS const String * storageReadName(const StorageRead *const this) { return THIS_PUB(StorageRead)->name; } // Is there a read limit? NULL for no limit. FN_INLINE_ALWAYS uint64_t storageReadOffset(const StorageRead *const this) { return THIS_PUB(StorageRead)->offset; } // Get file type FN_INLINE_ALWAYS StringId storageReadType(const StorageRead *const this) { return THIS_PUB(StorageRead)->type; } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void storageReadFree(StorageRead *const this) { objFree(this); } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageReadToLog(const StorageRead *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_READ_TYPE \ StorageRead * #define FUNCTION_LOG_STORAGE_READ_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageReadToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/read.intern.h0000664000175100017510000000425215240547630020233 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Read Interface Internal ***********************************************************************************************************************************/ #ifndef STORAGE_READ_INTERN_H #define STORAGE_READ_INTERN_H #include "common/io/read.h" #include "storage/storage.h" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct StorageReadInterface { void (*filterGroup)(void *driver, IoFilterGroup *filterGroup); // Set filter group (optional) bool (*open)(void *driver); // Open read size_t (*read)(void *driver, Buffer *buffer, bool block); // Read bytes bool (*eof)(void *driver); // Is read eof? int (*fd)(const void *driver); // Read file descriptor (optional) void (*close)(void *driver); // Close read } StorageReadInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageRead *storageReadNew( const Storage *storage, const String *name, bool ignoreMissing, bool compressible, uint64_t offset, const Variant *limit, bool version, const String *versionId); /*********************************************************************************************************************************** Getters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const StorageReadInterface * storageReadDriverInterface(const void *const driver) { return *(const StorageReadInterface *const *)driver; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/0000775000175100017510000000000015240547630017141 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/protocol.c0000664000175100017510000005704315240547630021157 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Protocol Handler ***********************************************************************************************************************************/ #include #include "common/compress/helper.h" #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/pack.h" #include "config/config.h" #include "protocol/helper.h" #include "storage/helper.h" #include "storage/remote/protocol.h" #include "storage/storage.intern.h" #include "storage/write.intern.h" /*********************************************************************************************************************************** Local variables ***********************************************************************************************************************************/ static struct { MemContext *memContext; // Mem context const Storage *storage; // Storage void *driver; // Storage driver used for requests const List *filterHandler; // Filter handler list } storageRemoteProtocolLocal; /*********************************************************************************************************************************** Set filter handlers ***********************************************************************************************************************************/ FN_EXTERN void storageRemoteFilterHandlerSet(const List *const filterHandler) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(VOID, filterHandler); FUNCTION_TEST_END(); ASSERT(filterHandler != NULL); ASSERT(!lstEmpty(filterHandler)); storageRemoteProtocolLocal.filterHandler = filterHandler; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Set filter group based on passed filters ***********************************************************************************************************************************/ static void storageRemoteFilterGroup(IoFilterGroup *const filterGroup, const Pack *const filterPack) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_TEST_PARAM(PACK, filterPack); FUNCTION_TEST_END(); ASSERT(filterGroup != NULL); ASSERT(filterPack != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackRead *const filterList = pckReadNew(filterPack); while (!pckReadNullP(filterList)) { const StringId filterKey = pckReadStrIdP(filterList); const Pack *const filterParam = pckReadPackP(filterList); // If a compression filter IoFilter *const filter = compressFilterPack(filterKey, filterParam); if (filter != NULL) { ioFilterGroupAdd(filterGroup, filter); } // Else a filter handler else { ASSERT(storageRemoteProtocolLocal.filterHandler != NULL); // Search for a filter handler unsigned int filterIdx = 0; for (; filterIdx < lstSize(storageRemoteProtocolLocal.filterHandler); filterIdx++) { const StorageRemoteFilterHandler *const filterHandler = lstGet( storageRemoteProtocolLocal.filterHandler, filterIdx); // If a match create the filter if (filterHandler->type == filterKey) { // Create a filter with parameters if (filterHandler->handlerParam != NULL) { ASSERT(filterParam != NULL); ioFilterGroupAdd(filterGroup, filterHandler->handlerParam(filterParam)); } // Else create a filter without parameters else { ASSERT(filterHandler->handlerNoParam != NULL); ASSERT(filterParam == NULL); ioFilterGroupAdd(filterGroup, filterHandler->handlerNoParam()); } // Break on filter match break; } } // Error when the filter was not found if (filterIdx == lstSize(storageRemoteProtocolLocal.filterHandler)) THROW_FMT(AssertError, "unable to add filter '%s'", zNewStrId(filterKey)); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteFeatureProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(param == NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Get storage based on remote type const Storage *const storage = cfgOptionStrId(cfgOptRemoteType) == protocolStorageTypeRepo ? storageRepoWrite() : storagePgWrite(); // Store local variables in the server context if (storageRemoteProtocolLocal.memContext == NULL) { MEM_CONTEXT_PRIOR_BEGIN() { MEM_CONTEXT_NEW_BEGIN(StorageRemoteProtocol, .childQty = MEM_CONTEXT_QTY_MAX) { storageRemoteProtocolLocal.memContext = memContextCurrent(); storageRemoteProtocolLocal.storage = storage; storageRemoteProtocolLocal.driver = storageDriver(storage); } MEM_CONTEXT_NEW_END(); } MEM_CONTEXT_PRIOR_END(); } // Return storage features PackWrite *const data = protocolServerResultData(result); pckWriteStrP(data, storagePathP(storage, NULL)); pckWriteU64P(data, storageInterface(storage).feature); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ typedef struct StorageRemoteInfoProtocolWriteData { time_t timeModifiedLast; // timeModified from last call mode_t modeLast; // mode from last call uid_t userIdLast; // userId from last call gid_t groupIdLast; // groupId from last call String *user; // user from last call String *group; // group from last call } StorageRemoteInfoProtocolWriteData; // Helper to write storage info into the protocol. This function is not called unless the info exists so no need to write exists or // check for level == storageInfoLevelExists. // // Fields that do not change from one call to the next are omitted to save bandwidth. static void storageRemoteInfoProtocolPut( StorageRemoteInfoProtocolWriteData *const data, PackWrite *const write, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(PACK_WRITE, write); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(data != NULL); ASSERT(write != NULL); ASSERT(info != NULL); // Write type and time pckWriteU32P(write, info->type); pckWriteTimeP(write, info->timeModified - data->timeModifiedLast); // Write size for files if (info->type == storageTypeFile) pckWriteU64P(write, info->size); // Write version pckWriteStrP(write, info->versionId); // Write fields needed for detail level if (info->level >= storageInfoLevelDetail) { // Write mode pckWriteModeP(write, info->mode, .defaultValue = data->modeLast); // Write user id/name pckWriteU32P(write, info->userId, .defaultValue = data->userIdLast); if (info->user == NULL) // {vm_covered} pckWriteBoolP(write, true); // {vm_covered} else { pckWriteBoolP(write, false); pckWriteStrP(write, info->user, .defaultValue = data->user); } // Write group id/name pckWriteU32P(write, info->groupId, .defaultValue = data->groupIdLast); if (info->group == NULL) // {vm_covered} pckWriteBoolP(write, true); // {vm_covered} else { pckWriteBoolP(write, false); pckWriteStrP(write, info->group, .defaultValue = data->group); } // Write link destination if (info->type == storageTypeLink) pckWriteStrP(write, info->linkDestination); } // Store defaults to use for the next call data->timeModifiedLast = info->timeModified; data->modeLast = info->mode; data->userIdLast = info->userId; data->groupIdLast = info->groupId; if (info->user != NULL && !strEq(info->user, data->user)) // {vm_covered} { strFree(data->user); data->user = strDup(info->user); } if (info->group != NULL && !strEq(info->group, data->group)) // {vm_covered} { strFree(data->group); data->group = strDup(info->group); } FUNCTION_TEST_RETURN_VOID(); } FN_EXTERN ProtocolServerResult * storageRemoteInfoProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Get file info const String *const file = pckReadStrP(param); const StorageInfoLevel level = (StorageInfoLevel)pckReadU32P(param); const bool followLink = pckReadBoolP(param); StorageInfo info = storageInterfaceInfoP(storageRemoteProtocolLocal.driver, file, level, .followLink = followLink); // Write file info to protocol PackWrite *const data = protocolServerResultData(result); pckWriteBoolP(data, info.exists, .defaultWrite = true); if (info.exists) storageRemoteInfoProtocolPut(&(StorageRemoteInfoProtocolWriteData){0}, data, &info); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteLinkCreateProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const target = pckReadStrP(param); const String *const linkPath = pckReadStrP(param); const StorageLinkType linkType = (StorageLinkType)pckReadU32P(param); storageInterfaceLinkCreateP(storageRemoteProtocolLocal.driver, target, linkPath, .linkType = linkType); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteListProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(.extra = ioBufferSize()); MEM_CONTEXT_TEMP_BEGIN() { const String *const path = pckReadStrP(param); const StorageInfoLevel level = (StorageInfoLevel)pckReadU32P(param); const time_t targetTime = pckReadTimeP(param); StorageRemoteInfoProtocolWriteData writeData = {0}; StorageList *const list = storageInterfaceListP(storageRemoteProtocolLocal.driver, path, level, .targetTime = targetTime); PackWrite *const data = protocolServerResultData(result); // Indicate whether or not the path was found pckWriteBoolP(data, list != NULL, .defaultWrite = true); // Put list if (list != NULL) { for (unsigned int listIdx = 0; listIdx < storageLstSize(list); listIdx++) { const StorageInfo info = storageLstGet(list, listIdx); pckWriteObjBeginP(data); pckWriteStrP(data, info.name); storageRemoteInfoProtocolPut(&writeData, data, &info); pckWriteObjEndP(data); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ static bool storageRemoteReadInternal(StorageRead *const fileRead, PackWrite *const packWrite) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_READ, fileRead); FUNCTION_LOG_PARAM(PACK_WRITE, packWrite); FUNCTION_LOG_END(); ASSERT(fileRead != NULL); ASSERT(packWrite != NULL); FUNCTION_AUDIT_HELPER(); // Read block and send to client Buffer *const buffer = bufNew(ioBufferSize()); ioRead(storageReadIo(fileRead), buffer); pckWriteBoolP(packWrite, bufEmpty(buffer)); if (!bufEmpty(buffer)) pckWriteBinP(packWrite, buffer); // On eof bool result = true; if (ioReadEof(storageReadIo(fileRead))) { // Close file (needed to get filter results) ioReadClose(storageReadIo(fileRead)); // Write filter results pckWritePackP(packWrite, ioFilterGroupResultAll(ioReadFilterGroup(storageReadIo(fileRead)))); // Let the server know to close the session result = false; } FUNCTION_LOG_RETURN(BOOL, result); } FN_EXTERN ProtocolServerResult * storageRemoteReadOpenProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(.extra = ioBufferSize()); MEM_CONTEXT_TEMP_BEGIN() { const String *file = pckReadStrP(param); const uint64_t offset = pckReadU64P(param); const Variant *const limit = pckReadNullP(param) ? NULL : VARUINT64(pckReadU64P(param)); const String *const versionId = pckReadStrP(param); const Pack *const filter = pckReadPackP(param); // Create the read object ignoring missing since the client decides whether a missing file is an error StorageRead *const fileRead = storageReadNew( storageRemoteProtocolLocal.storage, file, true, false, offset, limit, versionId != NULL, versionId); // Set filter group based on passed filters storageRemoteFilterGroup(ioReadFilterGroup(storageReadIo(fileRead)), filter); // Determine if file exists PackWrite *const data = protocolServerResultData(result); const bool exists = ioReadOpen(storageReadIo(fileRead)); pckWriteBoolP(data, exists, .defaultWrite = true); // If the file exists if (exists) { // If there is more to read then set session data if (storageRemoteReadInternal(fileRead, data)) protocolServerResultSessionDataSet(result, fileRead); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } FN_EXTERN ProtocolServerResult * storageRemoteReadProtocol(PackRead *const param, void *const fileRead) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_PARAM(STORAGE_READ, fileRead); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(param == NULL); ASSERT(fileRead != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(.extra = ioBufferSize()); if (!storageRemoteReadInternal(fileRead, protocolServerResultData(result))) protocolServerResultCloseSet(result); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteWriteOpenProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { // Create the write object const String *const file = pckReadStrP(param); const mode_t modeFile = pckReadModeP(param); const mode_t modePath = pckReadModeP(param); const String *const user = pckReadStrP(param); const String *const group = pckReadStrP(param); const time_t timeModified = pckReadTimeP(param); const bool createPath = pckReadBoolP(param); const bool syncFile = pckReadBoolP(param); const bool syncPath = pckReadBoolP(param); const bool atomic = pckReadBoolP(param); const Pack *const filter = pckReadPackP(param); StorageWrite *const fileWrite = storageWriteNew( storageRemoteProtocolLocal.storage, file, modeFile, modePath, user, group, timeModified, createPath, syncFile, syncPath, atomic, true, false); // Set filter group based on passed filters storageRemoteFilterGroup(ioWriteFilterGroup(storageWriteIo(fileWrite)), filter); // Open file ioWriteOpen(storageWriteIo(fileWrite)); // Set session data protocolServerResultSessionDataSet(result, fileWrite); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } FN_EXTERN ProtocolServerResult * storageRemoteWriteProtocol(PackRead *const param, void *const fileWrite) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_PARAM(STORAGE_WRITE, fileWrite); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(fileWrite != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { Buffer *const buffer = pckReadBinP(param); ioWrite(storageWriteIo(fileWrite), buffer); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } FN_EXTERN ProtocolServerResult * storageRemoteWriteCloseProtocol(PackRead *const param, void *const fileWrite) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_PARAM(STORAGE_WRITE, fileWrite); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(param == NULL); ASSERT(fileWrite != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { ioWriteClose(storageWriteIo(fileWrite)); // Send filter results pckWritePackP(protocolServerResultData(result), ioFilterGroupResultAll(ioWriteFilterGroup(storageWriteIo(fileWrite)))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemotePathCreateProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const path = pckReadStrP(param); const bool errorOnExists = pckReadBoolP(param); const bool noParentCreate = pckReadBoolP(param); const mode_t mode = pckReadModeP(param); storageInterfacePathCreateP(storageRemoteProtocolLocal.driver, path, errorOnExists, noParentCreate, mode); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemotePathRemoveProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); ProtocolServerResult *const result = protocolServerResultNewP(); MEM_CONTEXT_TEMP_BEGIN() { const String *const path = pckReadStrP(param); const bool recurse = pckReadBoolP(param); pckWriteBoolP( protocolServerResultData(result), storageInterfacePathRemoveP(storageRemoteProtocolLocal.driver, path, recurse), .defaultWrite = true); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, result); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemotePathSyncProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const path = pckReadStrP(param); storageInterfacePathSyncP(storageRemoteProtocolLocal.driver, path); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } /**********************************************************************************************************************************/ FN_EXTERN ProtocolServerResult * storageRemoteRemoveProtocol(PackRead *const param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(PACK_READ, param); FUNCTION_LOG_END(); ASSERT(param != NULL); ASSERT(storageRemoteProtocolLocal.driver != NULL); MEM_CONTEXT_TEMP_BEGIN() { const String *const file = pckReadStrP(param); const bool errorOnMissing = pckReadBoolP(param); storageInterfaceRemoveP(storageRemoteProtocolLocal.driver, file, .errorOnMissing = errorOnMissing); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(PROTOCOL_SERVER_RESULT, NULL); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/protocol.h0000664000175100017510000001265115240547630021160 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Protocol Handler ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_PROTOCOL_H #define STORAGE_REMOTE_PROTOCOL_H #include "common/type/pack.h" #include "protocol/server.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Process storage protocol requests FN_EXTERN ProtocolServerResult *storageRemoteFeatureProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteInfoProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteLinkCreateProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteListProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemotePathCreateProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemotePathRemoveProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemotePathSyncProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteReadOpenProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteReadProtocol(PackRead *param, void *fileRead); FN_EXTERN ProtocolServerResult *storageRemoteRemoveProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteWriteOpenProtocol(PackRead *param); FN_EXTERN ProtocolServerResult *storageRemoteWriteProtocol(PackRead *param, void *fileWrite); FN_EXTERN ProtocolServerResult *storageRemoteWriteCloseProtocol(PackRead *param, void *fileWrite); /*********************************************************************************************************************************** Protocol commands for ProtocolServerHandler arrays passed to protocolServerProcess() ***********************************************************************************************************************************/ #define PROTOCOL_COMMAND_STORAGE_FEATURE STRID5("s-f", 0x1b730) #define PROTOCOL_COMMAND_STORAGE_INFO STRID5("s-i", 0x27730) #define PROTOCOL_COMMAND_STORAGE_LINK_CREATE STRID5("s-lc", 0x1b3730) #define PROTOCOL_COMMAND_STORAGE_LIST STRID5("s-l", 0x33730) #define PROTOCOL_COMMAND_STORAGE_PATH_CREATE STRID5("s-pc", 0x1c3730) #define PROTOCOL_COMMAND_STORAGE_PATH_REMOVE STRID5("s-pr", 0x943730) #define PROTOCOL_COMMAND_STORAGE_PATH_SYNC STRID5("s-ps", 0x9c3730) #define PROTOCOL_COMMAND_STORAGE_READ STRID5("s-rd", 0x24b730) #define PROTOCOL_COMMAND_STORAGE_REMOVE STRID5("s-r", 0x4b730) #define PROTOCOL_COMMAND_STORAGE_WRITE STRID5("s-wr", 0x95f730) #define PROTOCOL_SERVER_HANDLER_STORAGE_REMOTE_LIST \ {.command = PROTOCOL_COMMAND_STORAGE_FEATURE, .process = storageRemoteFeatureProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_INFO, .process = storageRemoteInfoProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_LINK_CREATE, .process = storageRemoteLinkCreateProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_LIST, .process = storageRemoteListProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_PATH_CREATE, .process = storageRemotePathCreateProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_PATH_REMOVE, .process = storageRemotePathRemoveProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_PATH_SYNC, .process = storageRemotePathSyncProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_READ, .open = storageRemoteReadOpenProtocol, .processSession = storageRemoteReadProtocol},\ {.command = PROTOCOL_COMMAND_STORAGE_REMOVE, .process = storageRemoteRemoveProtocol}, \ {.command = PROTOCOL_COMMAND_STORAGE_WRITE, .open = storageRemoteWriteOpenProtocol, \ .processSession = storageRemoteWriteProtocol, .close = storageRemoteWriteCloseProtocol}, /*********************************************************************************************************************************** Filters that may be passed to a remote ***********************************************************************************************************************************/ typedef IoFilter *(*StorageRemoteFilterHandlerParam)(const Pack *paramList); typedef IoFilter *(*StorageRemoteFilterHandlerNoParam)(void); typedef struct StorageRemoteFilterHandler { StringId type; // Filter type StorageRemoteFilterHandlerParam handlerParam; // Handler for filter with parameters StorageRemoteFilterHandlerNoParam handlerNoParam; // Handler with no parameters } StorageRemoteFilterHandler; FN_EXTERN void storageRemoteFilterHandlerSet(const List *filterHandler); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/read.c0000664000175100017510000002650415240547630020227 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Read ***********************************************************************************************************************************/ #include #include #include #include "common/compress/helper.h" #include "common/debug.h" #include "common/io/io.h" #include "common/io/read.h" #include "common/log.h" #include "common/type/convert.h" #include "common/type/object.h" #include "storage/read.h" #include "storage/remote/protocol.h" #include "storage/remote/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageReadRemote { const StorageReadInterface *interface; // Interface StorageRemote *storage; // Storage that created this object IoFilterGroup *filterGroup; // Filter group const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) const String *versionId; // Version id (NULL for most recent) ProtocolClient *client; // Protocol client for requests ProtocolClientSession *session; // Protocol session for requests size_t remaining; // Bytes remaining to be read in block Buffer *block; // Block currently being read bool eof; // Has the file reached eof? bool eofFound; // Eof found but a block is remaining to be read bool compressible; // Is this file compressible? unsigned int compressLevel; // Level to use for compression #ifdef DEBUG uint64_t protocolReadBytes; // How many bytes were read from the protocol layer? #endif }; /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static void storageReadRemoteInternal(StorageReadRemote *const this, PackRead *const packRead) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_LOG_PARAM(PACK_READ, packRead); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(packRead != NULL); FUNCTION_AUDIT_HELPER(); // If not done then read the next block if (!pckReadBoolP(packRead)) { MEM_CONTEXT_OBJ_BEGIN(this) { this->block = pckReadBinP(packRead); this->remaining = bufUsed(this->block); } MEM_CONTEXT_OBJ_END(); } // If eof then get results if (protocolClientSessionClosed(this->session)) { ioFilterGroupResultAllSet(this->filterGroup, pckReadPackP(packRead)); this->eofFound = true; if (this->remaining == 0) this->eof = true; } #ifdef DEBUG this->protocolReadBytes += this->remaining; #endif FUNCTION_LOG_RETURN_VOID(); } static size_t storageReadRemote(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); size_t result = 0; // Read if eof has not been reached if (!this->eof) { do { // If no bytes remaining then read a new block if (this->remaining == 0) { MEM_CONTEXT_TEMP_BEGIN() { if (!protocolClientSessionQueued(this->session)) protocolClientSessionRequestAsyncP(this->session); storageReadRemoteInternal(this, protocolClientSessionResponse(this->session)); if (!this->eofFound) protocolClientSessionRequestAsyncP(this->session); } MEM_CONTEXT_TEMP_END(); } // Read if not eof if (!this->eof) { // Copy as much as possible into the output buffer const size_t remains = this->remaining < bufRemains(buffer) ? this->remaining : bufRemains(buffer); bufCatSub(buffer, this->block, bufUsed(this->block) - this->remaining, remains); result += remains; this->remaining -= remains; // If there is no more to copy from the block buffer then free it if (this->remaining == 0) { bufFree(this->block); this->block = NULL; if (this->eofFound) this->eof = true; } } } while (!this->eof && !bufFull(buffer)); } FUNCTION_LOG_RETURN(SIZE, result); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadRemoteEof(THIS_VOID) { THIS(StorageReadRemote); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->eof); } /*********************************************************************************************************************************** Set filter group ***********************************************************************************************************************************/ static void storageReadRemoteFilterGroup(THIS_VOID, IoFilterGroup *const filterGroup) { THIS(StorageReadRemote); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_TEST_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_TEST_END(); this->filterGroup = filterGroup; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadRemoteOpen(THIS_VOID) { THIS(StorageReadRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // If the file is compressible add compression filter on the remote if (this->compressible) ioFilterGroupAdd(this->filterGroup, compressFilterP(compressTypeLz4, (int)this->compressLevel, .raw = true)); PackWrite *const param = protocolPackNew(); pckWriteStrP(param, this->name); pckWriteU64P(param, this->offset); if (this->limit == NULL) pckWriteNullP(param); else pckWriteU64P(param, varUInt64(this->limit)); pckWriteStrP(param, this->versionId); pckWritePackP(param, ioFilterGroupParamAll(this->filterGroup)); // If the file exists PackRead *const packRead = protocolClientSessionOpenP(this->session, .param = param); result = pckReadBoolP(packRead); if (result) { // Clear filters since they will be run on the remote side ioFilterGroupClear(this->filterGroup); // If the file is compressible add decompression filter locally if (this->compressible) ioFilterGroupAdd(this->filterGroup, decompressFilterP(compressTypeLz4, .raw = true)); // Read the first block or eof storageReadRemoteInternal(this, packRead); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /*********************************************************************************************************************************** Close the file and read any remaining data. It is possible that all data has been read but if the amount of data is exactly divisible by the buffer size then the eof may not have been received. ***********************************************************************************************************************************/ static void storageReadRemoteClose(THIS_VOID) { THIS(StorageReadRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); protocolClientSessionCancel(this->session); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadRemoteInterface = { .close = storageReadRemoteClose, .eof = storageReadRemoteEof, .filterGroup = storageReadRemoteFilterGroup, .open = storageReadRemoteOpen, .read = storageReadRemote, }; FN_EXTERN StorageReadRemote * storageReadRemoteNew( StorageRemote *const storage, ProtocolClient *const client, const String *const name, const bool compressible, const unsigned int compressLevel, const uint64_t offset, const Variant *const limit, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, storage); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(BOOL, compressible); FUNCTION_LOG_PARAM(UINT, compressLevel); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(client != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadRemote, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageReadRemote) { .interface = &storageReadRemoteInterface, .storage = storage, .client = client, .session = protocolClientSessionNewP(client, PROTOCOL_COMMAND_STORAGE_READ, .async = true), .name = strDup(name), .compressible = compressible, .compressLevel = compressLevel, .offset = offset, .limit = varDup(limit), .versionId = strDup(versionId), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_REMOTE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/read.h0000664000175100017510000000364115240547630020231 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_READ_H #define STORAGE_REMOTE_READ_H #include "protocol/client.h" #include "storage/read.h" #include "storage/remote/storage.intern.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadRemote StorageReadRemote; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadRemote *storageReadRemoteNew( StorageRemote *storage, ProtocolClient *client, const String *name, bool compressible, unsigned int compressLevel, uint64_t offset, const Variant *limit, const String *versionId); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_REMOTE_TYPE \ StorageReadRemote * #define FUNCTION_LOG_STORAGE_READ_REMOTE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadRemote", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/storage.c0000664000175100017510000004321015240547630020751 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "common/type/object.h" #include "common/type/pack.h" #include "storage/remote/protocol.h" #include "storage/remote/read.h" #include "storage/remote/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageRemote { STORAGE_COMMON_MEMBER; ProtocolClient *client; // Protocol client unsigned int compressLevel; // Protocol compression level }; /**********************************************************************************************************************************/ typedef struct StorageRemoteInfoData { MemContext *memContext; // Mem context used to store values from last call time_t timeModifiedLast; // timeModified from last call mode_t modeLast; // mode from last call uid_t userIdLast; // userId from last call gid_t groupIdLast; // groupId from last call String *user; // user from last call String *group; // group from last call } StorageRemoteInfoData; // Helper to get storage info from the protocol output static void storageRemoteInfoGet(StorageRemoteInfoData *const data, PackRead *const read, StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, data); FUNCTION_TEST_PARAM(PACK_READ, read); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(data != NULL); ASSERT(read != NULL); ASSERT(info != NULL); // Read type and time modified info->type = pckReadU32P(read); info->timeModified = pckReadTimeP(read) + data->timeModifiedLast; // Read size for files if (info->type == storageTypeFile) info->size = pckReadU64P(read); // Read version info->versionId = pckReadStrP(read); // Read fields needed for detail level if (info->level >= storageInfoLevelDetail) { // Read mode info->mode = pckReadModeP(read, .defaultValue = data->modeLast); // Read user id/name info->userId = pckReadU32P(read, .defaultValue = data->userIdLast); if (pckReadBoolP(read)) // {vm_covered} info->user = NULL; // {vm_covered} else info->user = pckReadStrP(read, .defaultValue = data->user); // Read group id/name info->groupId = pckReadU32P(read, .defaultValue = data->groupIdLast); if (pckReadBoolP(read)) // {vm_covered} info->group = NULL; // {vm_covered} else info->group = pckReadStrP(read, .defaultValue = data->group); // Read link destination if (info->type == storageTypeLink) info->linkDestination = pckReadStrP(read); } // Store defaults to use for the next call data->timeModifiedLast = info->timeModified; data->modeLast = info->mode; data->userIdLast = info->userId; data->groupIdLast = info->groupId; if (info->user != NULL && !strEq(info->user, data->user)) // {vm_covered} { strFree(data->user); MEM_CONTEXT_BEGIN(data->memContext) { data->user = strDup(info->user); } MEM_CONTEXT_END(); } if (info->group != NULL && !strEq(info->group, data->group)) // {vm_covered} { strFree(data->group); MEM_CONTEXT_BEGIN(data->memContext) { data->group = strDup(info->group); } MEM_CONTEXT_END(); } FUNCTION_TEST_RETURN_VOID(); } static StorageInfo storageRemoteInfo(THIS_VOID, const String *file, StorageInfoLevel level, StorageInterfaceInfoParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(BOOL, param.followLink); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); StorageInfo result = {.level = level}; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, file); pckWriteU32P(commandParam, level); pckWriteBoolP(commandParam, param.followLink); // Read info from protocol PackRead *const read = protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_INFO, .param = commandParam); result.exists = pckReadBoolP(read); if (result.exists) { StorageRemoteInfoData parseData = {.memContext = memContextCurrent()}; // Read info from protocol into prior context MEM_CONTEXT_PRIOR_BEGIN() { storageRemoteInfoGet(&parseData, read, &result); } MEM_CONTEXT_PRIOR_END(); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storageRemoteLinkCreate( THIS_VOID, const String *const target, const String *const linkPath, const StorageInterfaceLinkCreateParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, target); FUNCTION_LOG_PARAM(STRING, linkPath); FUNCTION_LOG_PARAM(ENUM, param.linkType); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(target != NULL); ASSERT(linkPath != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, target); pckWriteStrP(commandParam, linkPath); pckWriteU32P(commandParam, param.linkType); protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_LINK_CREATE, .param = commandParam); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static StorageList * storageRemoteList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(TIME, param.targetTime); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); StorageList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, path); pckWriteU32P(commandParam, level); pckWriteTimeP(commandParam, param.targetTime); // Read list StorageRemoteInfoData parseData = {.memContext = memContextCurrent()}; PackRead *const read = protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_LIST, .param = commandParam); if (pckReadBoolP(read)) { result = storageLstNew(level); MEM_CONTEXT_TEMP_RESET_BEGIN() { while (pckReadNext(read)) { pckReadObjBeginP(read); StorageInfo info = {.exists = true, .level = level, .name = pckReadStrP(read)}; storageRemoteInfoGet(&parseData, read, &info); storageLstAdd(result, &info); pckReadObjEndP(read); // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); storageLstMove(result, memContextPrior()); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void * storageRemoteNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN( STORAGE_READ_REMOTE, storageReadRemoteNew( this, this->client, file, this->compressLevel > 0 ? param.compressible : false, this->compressLevel, param.offset, param.limit, param.versionId)); } /**********************************************************************************************************************************/ static void * storageRemoteNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(STRING, param.user); FUNCTION_LOG_PARAM(STRING, param.group); FUNCTION_LOG_PARAM(TIME, param.timeModified); FUNCTION_LOG_PARAM(BOOL, param.createPath); FUNCTION_LOG_PARAM(BOOL, param.syncFile); FUNCTION_LOG_PARAM(BOOL, param.syncPath); FUNCTION_LOG_PARAM(BOOL, param.atomic); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.truncate); FUNCTION_LOG_RETURN( STORAGE_WRITE_REMOTE, storageWriteRemoteNew( this, this->client, file, param.modeFile, param.modePath, param.user, param.group, param.timeModified, param.createPath, param.syncFile, param.syncPath, param.atomic, this->compressLevel > 0 ? param.compressible : false, this->compressLevel)); } /**********************************************************************************************************************************/ static void storageRemotePathCreate( THIS_VOID, const String *const path, const bool errorOnExists, const bool noParentCreate, const mode_t mode, const StorageInterfacePathCreateParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, errorOnExists); FUNCTION_LOG_PARAM(BOOL, noParentCreate); FUNCTION_LOG_PARAM(MODE, mode); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, path); pckWriteBoolP(commandParam, errorOnExists); pckWriteBoolP(commandParam, noParentCreate); pckWriteModeP(commandParam, mode); protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_PATH_CREATE, .param = commandParam); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static bool storageRemotePathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, recurse); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, path); pckWriteBoolP(commandParam, recurse); result = pckReadBoolP(protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_PATH_REMOVE, .param = commandParam)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ static void storageRemotePathSync(THIS_VOID, const String *const path, const StorageInterfacePathSyncParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, path); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, path); protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_PATH_SYNC, .param = commandParam); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void storageRemoteRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageRemote); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_REMOTE, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); MEM_CONTEXT_TEMP_BEGIN() { PackWrite *const commandParam = protocolPackNew(); pckWriteStrP(commandParam, file); pckWriteBoolP(commandParam, param.errorOnMissing); protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_REMOVE, .param = commandParam); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceRemote = { .info = storageRemoteInfo, .linkCreate = storageRemoteLinkCreate, .list = storageRemoteList, .newRead = storageRemoteNewRead, .newWrite = storageRemoteNewWrite, .pathCreate = storageRemotePathCreate, .pathRemove = storageRemotePathRemove, .pathSync = storageRemotePathSync, .remove = storageRemoteRemove, }; FN_EXTERN Storage * storageRemoteNew( const mode_t modeFile, const mode_t modePath, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, ProtocolClient *const client, const unsigned int compressLevel) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(PROTOCOL_CLIENT, client); FUNCTION_LOG_PARAM(UINT, compressLevel); FUNCTION_LOG_END(); ASSERT(modeFile != 0); ASSERT(modePath != 0); ASSERT(client != NULL); const String *path; OBJ_NEW_BEGIN(StorageRemote, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageRemote) { .client = client, .compressLevel = compressLevel, .interface = storageInterfaceRemote, }; // Get storage features from the remote MEM_CONTEXT_TEMP_BEGIN() { // Execute command and get result PackRead *const result = protocolClientRequestP(this->client, PROTOCOL_COMMAND_STORAGE_FEATURE); // Get path in parent context MEM_CONTEXT_PRIOR_BEGIN() { path = pckReadStrP(result); } MEM_CONTEXT_PRIOR_END(); this->interface.feature = pckReadU64P(result); } MEM_CONTEXT_TEMP_END(); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew( STORAGE_REMOTE_TYPE, path, modeFile, modePath, write, targetTime, pathExpressionFunction, this, this->interface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/storage.h0000664000175100017510000000241015240547630020753 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_STORAGE_H #define STORAGE_REMOTE_STORAGE_H #include "protocol/client.h" #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_REMOTE_TYPE STRID5("remote", 0xb47b4b20) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageRemoteNew( mode_t modeFile, mode_t modePath, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, ProtocolClient *client, unsigned int compressLevel); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/storage.intern.h0000664000175100017510000000254415240547630022261 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_STORAGE_INTERN_H #define STORAGE_REMOTE_STORAGE_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageRemote StorageRemote; #include "storage/remote/storage.h" /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_REMOTE_TYPE \ StorageRemote * #define FUNCTION_LOG_STORAGE_REMOTE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageRemote", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/write.c0000664000175100017510000002270215240547630020442 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage File write ***********************************************************************************************************************************/ #include #include "common/compress/helper.h" #include "common/debug.h" #include "common/io/io.h" #include "common/io/write.h" #include "common/log.h" #include "common/type/object.h" #include "storage/remote/protocol.h" #include "storage/remote/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteRemote { const StorageWriteInterface *interface; // Interface StorageRemote *storage; // Storage that created this object ProtocolClient *client; // Protocol client to make requests with ProtocolClientSession *session; // Protocol session for requests IoFilterGroup *filterGroup; // Filter group const String *name; // File name mode_t modeFile; // File mode mode_t modePath; // Path mode const String *user; // User name const String *group; // Group name time_t timeModified; // Time modified bool createPath; // Create path bool syncFile; // Sync file bool syncPath; // Sync path bool atomic; // Atomic write bool compressible; // Is this file compressible? unsigned int compressLevel; // Level to use for compression #ifdef DEBUG uint64_t protocolWriteBytes; // How many bytes were written to the protocol layer? #endif }; /*********************************************************************************************************************************** Set filter group ***********************************************************************************************************************************/ static void storageWriteRemoteFilterGroup(THIS_VOID, IoFilterGroup *const filterGroup) { THIS(StorageWriteRemote); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_WRITE_REMOTE, this); FUNCTION_TEST_PARAM(IO_FILTER_GROUP, filterGroup); FUNCTION_TEST_END(); this->filterGroup = filterGroup; FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static void storageWriteRemoteOpen(THIS_VOID) { THIS(StorageWriteRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); MEM_CONTEXT_TEMP_BEGIN() { // If the file is compressible add decompression filter on the remote if (this->compressible) { ioFilterGroupInsert( this->filterGroup, 0, decompressFilterP(compressTypeLz4, .raw = true)); } PackWrite *const param = protocolPackNew(); pckWriteStrP(param, this->name); pckWriteModeP(param, this->modeFile); pckWriteModeP(param, this->modePath); pckWriteStrP(param, this->user); pckWriteStrP(param, this->group); pckWriteTimeP(param, this->timeModified); pckWriteBoolP(param, this->createPath); pckWriteBoolP(param, this->syncFile); pckWriteBoolP(param, this->syncPath); pckWriteBoolP(param, this->atomic); pckWritePackP(param, ioFilterGroupParamAll(this->filterGroup)); protocolClientSessionOpenP(this->session, .param = param); // Clear filters since they will be run on the remote side ioFilterGroupClear(this->filterGroup); // If the file is compressible add compression filter locally if (this->compressible) { ioFilterGroupAdd( this->filterGroup, compressFilterP(compressTypeLz4, (int)this->compressLevel, .raw = true)); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to the file ***********************************************************************************************************************************/ static void storageWriteRemote(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_REMOTE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); MEM_CONTEXT_TEMP_BEGIN() { if (protocolClientSessionQueued(this->session)) protocolClientSessionResponse(this->session); PackWrite *const param = pckWriteNewP(.size = PROTOCOL_PACK_DEFAULT_SIZE + bufUsed(buffer)); pckWriteBinP(param, buffer); protocolClientSessionRequestAsyncP(this->session, .param = param); } MEM_CONTEXT_TEMP_END(); #ifdef DEBUG this->protocolWriteBytes += bufUsed(buffer); #endif FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteRemoteClose(THIS_VOID) { THIS(StorageWriteRemote); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_REMOTE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->client != NULL) { MEM_CONTEXT_TEMP_BEGIN() { if (protocolClientSessionQueued(this->session)) protocolClientSessionResponse(this->session); ioFilterGroupResultAllSet( this->filterGroup, pckReadPackP(protocolClientSessionClose(this->session))); } MEM_CONTEXT_TEMP_END(); this->client = NULL; } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteRemoteInterface = { .close = storageWriteRemoteClose, .filterGroup = storageWriteRemoteFilterGroup, .open = storageWriteRemoteOpen, .write = storageWriteRemote, }; FN_EXTERN StorageWriteRemote * storageWriteRemoteNew( StorageRemote *const storage, ProtocolClient *const client, const String *const name, const mode_t modeFile, const mode_t modePath, const String *const user, const String *const group, const time_t timeModified, const bool createPath, const bool syncFile, const bool syncPath, const bool atomic, const bool compressible, const unsigned int compressLevel) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_REMOTE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(STRING, group); FUNCTION_LOG_PARAM(TIME, timeModified); FUNCTION_LOG_PARAM(BOOL, createPath); FUNCTION_LOG_PARAM(BOOL, syncFile); FUNCTION_LOG_PARAM(BOOL, syncPath); FUNCTION_LOG_PARAM(BOOL, atomic); FUNCTION_LOG_PARAM(BOOL, compressible); FUNCTION_LOG_PARAM(UINT, compressLevel); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(client != NULL); ASSERT(name != NULL); ASSERT(modeFile != 0); ASSERT(modePath != 0); OBJ_NEW_BEGIN(StorageWriteRemote, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageWriteRemote) { .interface = &storageWriteRemoteInterface, .storage = storage, .client = client, .session = protocolClientSessionNewP(client, PROTOCOL_COMMAND_STORAGE_WRITE, .async = true), .name = strDup(name), .modeFile = modeFile, .modePath = modePath, .user = strDup(user), .group = strDup(group), .timeModified = timeModified, .createPath = createPath, .syncFile = syncFile, .syncPath = syncPath, .atomic = atomic, .compressible = compressible, .compressLevel = compressLevel, }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_REMOTE, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/remote/write.h0000664000175100017510000000402015240547630020440 0ustar00runnerrunner/*********************************************************************************************************************************** Remote Storage File write ***********************************************************************************************************************************/ #ifndef STORAGE_REMOTE_WRITE_H #define STORAGE_REMOTE_WRITE_H #include "protocol/client.h" #include "storage/remote/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteRemote StorageWriteRemote; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteRemote *storageWriteRemoteNew( StorageRemote *storage, ProtocolClient *client, const String *name, mode_t modeFile, mode_t modePath, const String *user, const String *group, time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool compressible, unsigned int compressLevel); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_REMOTE_TYPE \ StorageWriteRemote * #define FUNCTION_LOG_STORAGE_WRITE_REMOTE_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteRemote", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/0000775000175100017510000000000015240547630016173 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/helper.c0000664000175100017510000001476015240547630017626 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Helper ***********************************************************************************************************************************/ #include #include #include "common/debug.h" #include "common/io/http/url.h" #include "common/io/io.h" #include "common/log.h" #include "config/config.h" #include "storage/helper.h" #include "storage/posix/storage.h" #include "storage/s3/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageS3Helper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_S3_TYPE); Storage *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Parse the endpoint url const HttpUrl *const url = httpUrlNewParseP( cfgOptionIdxStr(cfgOptRepoS3Endpoint, repoIdx), .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); const String *const endPoint = httpUrlHost(url); unsigned int port = httpUrlPort(url); HttpProtocolType protocolType = httpUrlProtocolType(url); // If host was specified then use it const String *host = NULL; if (cfgOptionIdxSource(cfgOptRepoStorageHost, repoIdx) != cfgSourceDefault) { const HttpUrl *const url = httpUrlNewParseP( cfgOptionIdxStr(cfgOptRepoStorageHost, repoIdx), .type = httpProtocolTypeAny, .defaultType = httpProtocolTypeHttps); host = httpUrlHost(url); port = httpUrlPort(url); protocolType = httpUrlProtocolType(url); } // If port was specified, overwrite the parsed/default port if (cfgOptionIdxSource(cfgOptRepoStoragePort, repoIdx) != cfgSourceDefault) port = cfgOptionIdxUInt(cfgOptRepoStoragePort, repoIdx); // Get role and token const StorageS3KeyType keyType = (StorageS3KeyType)cfgOptionIdxSeq(cfgOptRepoS3KeyType, repoIdx); const String *role = cfgOptionIdxStrNull(cfgOptRepoS3Role, repoIdx); const String *tokenFile = NULL; const String *credUrl = NULL; // If web identity authentication then load the role and token filename from environment variables documented here: // https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html if (keyType == storageS3KeyTypeWebId) { #define S3_ENV_AWS_ROLE_ARN "AWS_ROLE_ARN" #define S3_ENV_AWS_WEB_IDENTITY_TOKEN_FILE "AWS_WEB_IDENTITY_TOKEN_FILE" const char *const roleZ = getenv(S3_ENV_AWS_ROLE_ARN); const char *const webIdTokenFileZ = getenv(S3_ENV_AWS_WEB_IDENTITY_TOKEN_FILE); if (roleZ == NULL || webIdTokenFileZ == NULL) { THROW_FMT( OptionInvalidError, "option '%s' is '" CFGOPTVAL_REPO_S3_KEY_TYPE_WEB_ID_Z "' but '" S3_ENV_AWS_ROLE_ARN "' and" " '" S3_ENV_AWS_WEB_IDENTITY_TOKEN_FILE "' are not set", cfgOptionIdxName(cfgOptRepoS3KeyType, repoIdx)); } role = strNewZ(roleZ); tokenFile = strNewZ(webIdTokenFileZ); } // If pod identity authentication then load the credentials url and token filename from environment variables documented // here: https://docs.aws.amazon.com/eks/latest/userguide/pod-id-how-it-works.html else if (keyType == storageS3KeyTypePodId) { #define S3_ENV_AWS_CONTAINER_CREDENTIALS_FULL_URI "AWS_CONTAINER_CREDENTIALS_FULL_URI" #define S3_ENV_AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE "AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE" const char *const credUrlZ = getenv(S3_ENV_AWS_CONTAINER_CREDENTIALS_FULL_URI); const char *const podIdTokenFileZ = getenv(S3_ENV_AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE); if (credUrlZ == NULL || podIdTokenFileZ == NULL) { THROW_FMT( OptionInvalidError, "option '%s' is '" CFGOPTVAL_REPO_S3_KEY_TYPE_POD_ID_Z "' but '" S3_ENV_AWS_CONTAINER_CREDENTIALS_FULL_URI "'" " and '" S3_ENV_AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE "' are not set", cfgOptionIdxName(cfgOptRepoS3KeyType, repoIdx)); } credUrl = strNewZ(credUrlZ); tokenFile = strNewZ(podIdTokenFileZ); } // If process credential authentication then get the command from config const StringList *const credCmd = keyType == storageS3KeyTypeProcess ? strLstNewVarLst(cfgOptionIdxLst(cfgOptRepoS3ProcessCmd, repoIdx)) : NULL; MEM_CONTEXT_PRIOR_BEGIN() { result = storageS3New( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), write, storageRepoTargetTime(), pathExpressionCallback, cfgOptionIdxStr(cfgOptRepoS3Bucket, repoIdx), endPoint, cfgOptionIdxStr(cfgOptRepoS3Region, repoIdx), cfgOptionIdxStr(cfgOptRepoS3Service, repoIdx), keyType, (StorageS3UriStyle)cfgOptionIdxSeq(cfgOptRepoS3UriStyle, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3Key, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3KeySecret, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3Token, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3KmsKeyId, repoIdx), cfgOptionIdxStrNull(cfgOptRepoS3SseCustomerKey, repoIdx), role, tokenFile, credUrl, credCmd, cfgOptionIdxStrNull(cfgOptRepoS3StsHost, repoIdx), (size_t)cfgOptionIdxUInt64(cfgOptRepoStorageUploadChunkSize, repoIdx), cfgOptionIdxKvNull(cfgOptRepoStorageTag, repoIdx), host, port, ioTimeoutMs(), protocolType, cfgOptionIdxBool(cfgOptRepoStorageVerifyTls, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaFile, repoIdx), cfgOptionIdxStrNull(cfgOptRepoStorageCaPath, repoIdx), cfgOptionIdxBool(cfgOptRepoS3RequesterPays, repoIdx)); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE, result); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/helper.h0000664000175100017510000000234715240547630017631 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_S3_STORAGE_HELPER_H #define STORAGE_S3_STORAGE_HELPER_H #include "storage/s3/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageS3Helper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_S3_HELPER {.type = STORAGE_S3_TYPE, .helper = storageS3Helper} #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/read.c0000664000175100017510000001415215240547630017255 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Read Based on the documentation at https://cloud.google.com/storage/docs/json_api/v1/objects/get ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/io/http/client.h" #include "common/log.h" #include "common/type/object.h" #include "storage/read.h" #include "storage/s3/read.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ #define STORAGE_READ_S3_TYPE StorageReadS3 #define STORAGE_READ_S3_PREFIX storageReadS3 struct StorageReadS3 { const StorageReadInterface *interface; // Interface StorageS3 *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) const String *versionId; // Version id (NULL for most recent) HttpResponse *httpResponse; // HTTP response }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadS3Open(THIS_VOID) { THIS(StorageReadS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse == NULL); // Request the file MEM_CONTEXT_OBJ_BEGIN(this) { this->httpResponse = storageS3RequestP( this->storage, HTTP_VERB_GET_STR, this->name, .header = httpHeaderPutRange(httpHeaderNew(NULL), this->offset, this->limit), .query = this->versionId != NULL ? httpQueryPut(httpQueryNewP(), STRDEF("versionId"), this->versionId) : NULL, .allowMissing = true, .contentIo = true, .sseC = true); } MEM_CONTEXT_OBJ_END(); FUNCTION_LOG_RETURN(BOOL, httpResponseCodeOk(this->httpResponse)); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadS3(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_S3, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL && this->httpResponse != NULL); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); FUNCTION_LOG_RETURN(SIZE, ioRead(httpResponseIoRead(this->httpResponse), buffer)); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadS3Eof(THIS_VOID) { THIS(StorageReadS3); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_S3, this); FUNCTION_TEST_END(); ASSERT(this != NULL && this->httpResponse != NULL); ASSERT(httpResponseIoRead(this->httpResponse) != NULL); FUNCTION_TEST_RETURN(BOOL, ioReadEof(httpResponseIoRead(this->httpResponse))); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadS3Close(THIS_VOID) { THIS(StorageReadS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->httpResponse != NULL); httpResponseFree(this->httpResponse); this->httpResponse = NULL; FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadS3Interface = { .close = storageReadS3Close, .eof = storageReadS3Eof, .open = storageReadS3Open, .read = storageReadS3, }; FN_EXTERN StorageReadS3 * storageReadS3New( StorageS3 *const storage, const String *const name, const uint64_t offset, const Variant *const limit, const String *const versionId) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_S3, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_PARAM(STRING, versionId); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); ASSERT(limit == NULL || varUInt64(limit) > 0); OBJ_NEW_BEGIN(StorageReadS3, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadS3) { .interface = &storageReadS3Interface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), .versionId = strDup(versionId), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_S3, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/read.h0000664000175100017510000000341515240547630017262 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_S3_READ_H #define STORAGE_S3_READ_H #include "storage/read.h" #include "storage/s3/storage.intern.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadS3 StorageReadS3; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadS3 *storageReadS3New( StorageS3 *storage, const String *name, uint64_t offset, const Variant *limit, const String *versionId); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_S3_TYPE \ StorageReadS3 * #define FUNCTION_LOG_STORAGE_READ_S3_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadS3", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/storage.c0000664000175100017510000020446615240547630020017 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage ***********************************************************************************************************************************/ #include #include #include "common/crypto/hash.h" #include "common/debug.h" #include "common/exec.h" #include "common/io/http/client.h" #include "common/io/http/common.h" #include "common/io/socket/client.h" #include "common/io/tls/client.h" #include "common/log.h" #include "common/regExp.h" #include "common/type/json.h" #include "common/type/object.h" #include "common/type/xml.h" #include "storage/posix/storage.h" #include "storage/s3/read.h" #include "storage/s3/write.h" /*********************************************************************************************************************************** Defaults ***********************************************************************************************************************************/ #define STORAGE_S3_DELETE_MAX 1000 /*********************************************************************************************************************************** S3 HTTP headers ***********************************************************************************************************************************/ STRING_STATIC(S3_HEADER_CONTENT_SHA256_STR, "x-amz-content-sha256"); STRING_STATIC(S3_HEADER_DATE_STR, "x-amz-date"); STRING_STATIC(S3_HEADER_TOKEN_STR, "x-amz-security-token"); STRING_STATIC(S3_HEADER_SRVSDENC_STR, "x-amz-server-side-encryption"); STRING_STATIC(S3_HEADER_SRVSDENC_KMS_STR, "aws:kms"); STRING_STATIC(S3_HEADER_SRVSDENC_KMSKEYID_STR, "x-amz-server-side-encryption-aws-kms-key-id"); STRING_STATIC(S3_HEADER_SSECUSTKEY_ALGO_STR, "x-amz-server-side-encryption-customer-algorithm"); STRING_STATIC(S3_HEADER_SSECUSTKEY_AES256_STR, "AES256"); STRING_STATIC(S3_HEADER_SSECUSTKEY_KEY_STR, "x-amz-server-side-encryption-customer-key"); STRING_STATIC(S3_HEADER_SSECUSTKEY_KEY_MD5_STR, "x-amz-server-side-encryption-customer-key-md5"); STRING_STATIC(S3_HEADER_TAGGING, "x-amz-tagging"); STRING_STATIC(S3_HEADER_REQUEST_PAYER, "x-amz-request-payer"); STRING_STATIC(S3_HEADER_REQUEST_PAYER_STR, "requester"); /*********************************************************************************************************************************** S3 query tokens ***********************************************************************************************************************************/ STRING_STATIC(S3_QUERY_CONTINUATION_TOKEN_STR, "continuation-token"); STRING_STATIC(S3_QUERY_DELETE_STR, "delete"); STRING_STATIC(S3_QUERY_DELIMITER_STR, "delimiter"); STRING_STATIC(S3_QUERY_LIST_TYPE_STR, "list-type"); STRING_STATIC(S3_QUERY_PREFIX_STR, "prefix"); STRING_STATIC(S3_QUERY_VERSIONS_STR, "versions"); STRING_STATIC(S3_QUERY_VALUE_LIST_TYPE_2_STR, "2"); /*********************************************************************************************************************************** XML tags ***********************************************************************************************************************************/ STRING_STATIC(S3_XML_TAG_COMMON_PREFIXES_STR, "CommonPrefixes"); STRING_STATIC(S3_XML_TAG_CONTENTS_STR, "Contents"); STRING_STATIC(S3_XML_TAG_DELETE_STR, "Delete"); STRING_STATIC(S3_XML_TAG_DELETE_MARKER_STR, "DeleteMarker"); STRING_STATIC(S3_XML_TAG_ERROR_STR, "Error"); STRING_STATIC(S3_XML_TAG_IS_TRUNCATED_STR, "IsTruncated"); STRING_STATIC(S3_XML_TAG_KEY_STR, "Key"); STRING_STATIC(S3_XML_TAG_LAST_MODIFIED_STR, "LastModified"); #define S3_XML_TAG_NEXT_CONTINUATION_TOKEN "NextContinuationToken" STRING_STATIC(S3_XML_TAG_NEXT_CONTINUATION_TOKEN_STR, S3_XML_TAG_NEXT_CONTINUATION_TOKEN); STRING_STATIC(S3_XML_TAG_OBJECT_STR, "Object"); STRING_STATIC(S3_XML_TAG_PREFIX_STR, "Prefix"); STRING_STATIC(S3_XML_TAG_QUIET_STR, "Quiet"); STRING_STATIC(S3_XML_TAG_SIZE_STR, "Size"); STRING_STATIC(S3_XML_TAG_VERSION_STR, "Version"); STRING_STATIC(S3_XML_TAG_VERSION_ID_STR, "VersionId"); /*********************************************************************************************************************************** AWS authentication v4 constants ***********************************************************************************************************************************/ #define AWS4 "AWS4" #define AWS4_REQUEST "aws4_request" BUFFER_STRDEF_STATIC(AWS4_REQUEST_BUF, AWS4_REQUEST); #define AWS4_HMAC_SHA256 "AWS4-HMAC-SHA256" /*********************************************************************************************************************************** Starting date for signing string so it will be regenerated on the first request ***********************************************************************************************************************************/ STRING_STATIC(YYYYMMDD_STR, "YYYYMMDD"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageS3 { STORAGE_COMMON_MEMBER; HttpClient *httpClient; // HTTP client to service requests StringList *headerRedactList; // List of headers to redact from logging const String *bucket; // Bucket to store data in const String *region; // e.g. us-east-1 const String *signingService; // SigV4 signing service StorageS3KeyType keyType; // Key type (e.g. storageS3KeyTypeShared) String *accessKey; // Access key String *secretAccessKey; // Secret access key String *securityToken; // Security token, if any const String *kmsKeyId; // Server-side encryption key const String *sseCustomerKey; // Base64 of SSE-C encryption key const String *sseCustomerKeyMd5; // Base64 of MD5 of SSE-C key size_t partSize; // Part size for multi-part upload const String *tag; // Tags to be applied to objects unsigned int deleteMax; // Maximum objects that can be deleted in one request StorageS3UriStyle uriStyle; // Path or host style URIs const String *bucketEndpoint; // Set to {bucket}.{endpoint} bool requesterPays; // Requester pays? // For retrieving temporary security credentials HttpClient *credHttpClient; // HTTP client to service credential requests const String *credHost; // Credentials host const String *credPath; // Credential url path const String *credRole; // Role to use for credential requests const String *tokenFile; // File with token to use for web-id/pod-id credential requests const StringList *credCmd; // Command and parameters for process credential requests time_t credExpirationTime; // Time the temporary credentials expire // Current signing key and date it is valid for const String *signingKeyDate; // Date of cached signing key (so we know when to regenerate) const Buffer *signingKey; // Cached signing key }; /*********************************************************************************************************************************** Expected ISO-8601 data/time size ***********************************************************************************************************************************/ #define ISO_8601_DATE_TIME_SIZE 16 /*********************************************************************************************************************************** Generate authorization header and add it to the supplied header list Based on the excellent documentation at http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html. ***********************************************************************************************************************************/ static void storageS3Auth( StorageS3 *const this, const String *const verb, const String *const path, const HttpQuery *const query, const String *const dateTime, HttpHeader *const httpHeader, const String *const payloadHash) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_S3, this); FUNCTION_TEST_PARAM(STRING, verb); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(HTTP_QUERY, query); FUNCTION_TEST_PARAM(STRING, dateTime); FUNCTION_TEST_PARAM(KEY_VALUE, httpHeader); FUNCTION_TEST_PARAM(STRING, payloadHash); FUNCTION_TEST_END(); ASSERT(verb != NULL); ASSERT(path != NULL); ASSERT(dateTime != NULL); ASSERT(httpHeader != NULL); ASSERT(payloadHash != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Get date from datetime const String *const date = strSubN(dateTime, 0, 8); // Set required headers httpHeaderPut(httpHeader, S3_HEADER_CONTENT_SHA256_STR, payloadHash); httpHeaderPut(httpHeader, S3_HEADER_DATE_STR, dateTime); httpHeaderPut(httpHeader, HTTP_HEADER_HOST_STR, this->bucketEndpoint); if (this->securityToken != NULL) httpHeaderPut(httpHeader, S3_HEADER_TOKEN_STR, this->securityToken); // Generate canonical request and signed headers const StringList *const headerList = strLstSort(strLstDup(httpHeaderList(httpHeader)), sortOrderAsc); String *signedHeaders = NULL; String *const canonicalRequest = strCatFmt( strNew(), "%s\n%s\n%s\n", strZ(verb), strZ(path), query == NULL ? "" : strZ(httpQueryRenderP(query))); for (unsigned int headerIdx = 0; headerIdx < strLstSize(headerList); headerIdx++) { const String *const headerKey = strLstGet(headerList, headerIdx); const String *const headerKeyLower = strLower(strDup(headerKey)); // Skip the authorization (exists on retry) and content-length headers since they do not need to be signed if (strEq(headerKeyLower, HTTP_HEADER_AUTHORIZATION_STR) || strEq(headerKeyLower, HTTP_HEADER_CONTENT_LENGTH_STR)) continue; strCatFmt(canonicalRequest, "%s:%s\n", strZ(headerKeyLower), strZ(httpHeaderGet(httpHeader, headerKey))); if (signedHeaders == NULL) signedHeaders = strCat(strNew(), headerKeyLower); else strCatFmt(signedHeaders, ";%s", strZ(headerKeyLower)); } strCatFmt(canonicalRequest, "\n%s\n%s", strZ(signedHeaders), strZ(payloadHash)); // Generate string to sign const String *const stringToSign = strNewFmt( AWS4_HMAC_SHA256 "\n%s\n%s/%s/%s/" AWS4_REQUEST "\n%s", strZ(dateTime), strZ(date), strZ(this->region), strZ(this->signingService), strZ(strNewEncode(encodingHex, cryptoHashOne(hashTypeSha256, BUFSTR(canonicalRequest))))); // Generate signing key. This key only needs to be regenerated every seven days but we'll do it once a day to keep the // logic simple. It's a relatively expensive operation so we'd rather not do it for every request. // If the cached signing key has expired (or has none been generated) then regenerate it if (!strEq(date, this->signingKeyDate)) { const Buffer *const dateKey = cryptoHmacOne( hashTypeSha256, BUFSTR(strNewFmt(AWS4 "%s", strZ(this->secretAccessKey))), BUFSTR(date)); const Buffer *const regionKey = cryptoHmacOne(hashTypeSha256, dateKey, BUFSTR(this->region)); const Buffer *const serviceKey = cryptoHmacOne(hashTypeSha256, regionKey, BUFSTR(this->signingService)); // Switch to the object context so signing key and date are not lost MEM_CONTEXT_OBJ_BEGIN(this) { this->signingKey = cryptoHmacOne(hashTypeSha256, serviceKey, AWS4_REQUEST_BUF); this->signingKeyDate = strDup(date); } MEM_CONTEXT_OBJ_END(); } // Generate authorization header const String *const authorization = strNewFmt( AWS4_HMAC_SHA256 " Credential=%s/%s/%s/%s/" AWS4_REQUEST ",SignedHeaders=%s,Signature=%s", strZ(this->accessKey), strZ(date), strZ(this->region), strZ(this->signingService), strZ(signedHeaders), strZ(strNewEncode(encodingHex, cryptoHmacOne(hashTypeSha256, this->signingKey, BUFSTR(stringToSign))))); httpHeaderPut(httpHeader, HTTP_HEADER_AUTHORIZATION_STR, authorization); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN_VOID(); } /*********************************************************************************************************************************** Convert YYYY-MM-DDTHH:MM:SS.MSECZ format to time_t. This format is very nearly ISO-8601 except for the inclusion of milliseconds, which are discarded here. ***********************************************************************************************************************************/ static time_t storageS3CvtTime(const String *const time) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, time); FUNCTION_TEST_END(); FUNCTION_TEST_RETURN( TIME, epochFromParts( cvtZSubNToInt(strZ(time), 0, 4), cvtZSubNToInt(strZ(time), 5, 2), cvtZSubNToInt(strZ(time), 8, 2), cvtZSubNToInt(strZ(time), 11, 2), cvtZSubNToInt(strZ(time), 14, 2), cvtZSubNToInt(strZ(time), 17, 2), 0)); } /*********************************************************************************************************************************** Automatically get credentials for an associated IAM role Documentation for the response format is found at: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html Documentation for IMDSv2 tokens: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html ***********************************************************************************************************************************/ STRING_STATIC(S3_CREDENTIAL_HOST_STR, "169.254.169.254"); #define S3_CREDENTIAL_PORT 80 #define S3_CREDENTIAL_PATH "/latest/meta-data/iam/security-credentials" #define S3_CREDENTIAL_RENEW_SEC (5 * 60) VARIANT_STRDEF_STATIC(S3_JSON_TAG_ACCESS_KEY_ID_VAR, "AccessKeyId"); VARIANT_STRDEF_STATIC(S3_JSON_TAG_CODE_VAR, "Code"); VARIANT_STRDEF_STATIC(S3_JSON_TAG_EXPIRATION_VAR, "Expiration"); VARIANT_STRDEF_STATIC(S3_JSON_TAG_SECRET_ACCESS_KEY_VAR, "SecretAccessKey"); VARIANT_STRDEF_STATIC(S3_JSON_TAG_TOKEN_VAR, "Token"); VARIANT_STRDEF_STATIC(S3_JSON_VALUE_SUCCESS_VAR, "Success"); static void storageS3AuthAuto(StorageS3 *const this, HttpHeader *const header) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(HTTP_HEADER, header); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Retrieve the IMDSv2 token. Do not store the token because it will likely be expired by the time the credentials expire // since credentials have a much longer default expiration. The TTL needs to be long enough to survive retries of the two // metadata queries. HttpHeader *const tokenHeader = httpHeaderDup(header, NULL); httpHeaderAdd( tokenHeader, STRDEF("x-aws-ec2-metadata-token-ttl-seconds"), strNewFmt("%" PRIu64, httpClientTimeout(this->credHttpClient) / 1000 * 3)); HttpRequest *request = httpRequestNewP( this->credHttpClient, HTTP_VERB_PUT_STR, STRDEF("/latest/api/token"), .header = tokenHeader); HttpResponse *response = httpRequestResponse(request, true); // Set IMDSv2 token on success. If the token request fails for any reason assume that IMDSv2 is not supported. If the // instance only supports IMDSv2 then subsequent metadata requests will fail because of the missing token. if (httpResponseCodeOk(response)) httpHeaderAdd(header, STRDEF("x-aws-ec2-metadata-token"), strNewBuf(httpResponseContent(response))); // If the role was not set explicitly or retrieved previously then retrieve it if (this->credRole == NULL) { // Request the role HttpRequest *const request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, STRDEF(S3_CREDENTIAL_PATH), .header = header); HttpResponse *const response = httpRequestResponse(request, true); // Not found likely means no role is associated with this instance if (httpResponseCode(response) == HTTP_RESPONSE_CODE_NOT_FOUND) { THROW( ProtocolError, "role to retrieve temporary credentials not found\n" "HINT: is a valid IAM role associated with this instance?"); } // Else an error that we can't handle else if (!httpResponseCodeOk(response)) httpRequestError(request, response); // Get role from the text response MEM_CONTEXT_OBJ_BEGIN(this) { this->credRole = strNewBuf(httpResponseContent(response)); } MEM_CONTEXT_OBJ_END(); } // Retrieve the temp credentials request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, strNewFmt(S3_CREDENTIAL_PATH "/%s", strZ(this->credRole)), .header = header); response = httpRequestResponse(request, true); // Not found likely means the role is not valid if (httpResponseCode(response) == HTTP_RESPONSE_CODE_NOT_FOUND) { THROW_FMT( ProtocolError, "role '%s' not found\n" "HINT: is '%s' a valid IAM role associated with this instance?", strZ(this->credRole), strZ(this->credRole)); } // Else an error that we can't handle else if (!httpResponseCodeOk(response)) httpRequestError(request, response); // Get credentials from the JSON response const KeyValue *const credential = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); MEM_CONTEXT_OBJ_BEGIN(this) { // Check the code field for errors const Variant *const code = kvGetDefault(credential, S3_JSON_TAG_CODE_VAR, VARSTRDEF("code field is missing")); CHECK(FormatError, code != NULL, "error code missing"); if (!varEq(code, S3_JSON_VALUE_SUCCESS_VAR)) THROW_FMT(FormatError, "unable to retrieve temporary credentials: %s", strZ(varStr(code))); // Make sure the required values are present CHECK(FormatError, kvGet(credential, S3_JSON_TAG_ACCESS_KEY_ID_VAR) != NULL, "access key missing"); CHECK(FormatError, kvGet(credential, S3_JSON_TAG_SECRET_ACCESS_KEY_VAR) != NULL, "secret access key missing"); CHECK(FormatError, kvGet(credential, S3_JSON_TAG_TOKEN_VAR) != NULL, "token missing"); // Copy credentials this->accessKey = strDup(varStr(kvGet(credential, S3_JSON_TAG_ACCESS_KEY_ID_VAR))); this->secretAccessKey = strDup(varStr(kvGet(credential, S3_JSON_TAG_SECRET_ACCESS_KEY_VAR))); this->securityToken = strDup(varStr(kvGet(credential, S3_JSON_TAG_TOKEN_VAR))); } MEM_CONTEXT_OBJ_END(); // Update expiration time CHECK(FormatError, kvGet(credential, S3_JSON_TAG_EXPIRATION_VAR) != NULL, "expiration missing"); this->credExpirationTime = storageS3CvtTime(varStr(kvGet(credential, S3_JSON_TAG_EXPIRATION_VAR))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Automatically get credentials for an associated web identity service account Documentation is found at: https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithWebIdentity.html ***********************************************************************************************************************************/ static void storageS3AuthWebId(StorageS3 *const this, const HttpHeader *const header) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(HTTP_HEADER, header); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Load the token from the given file for each request since the token may be updated during execution const String *const webIdToken = strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), this->tokenFile))); // Get credentials. Redact the web identity token so it is not exposed if the query is logged, e.g. in an error. StringList *const queryRedactList = strLstNew(); strLstAddZ(queryRedactList, "WebIdentityToken"); HttpQuery *const query = httpQueryNewP(.redactList = queryRedactList); httpQueryAdd(query, STRDEF("Action"), STRDEF("AssumeRoleWithWebIdentity")); httpQueryAdd(query, STRDEF("RoleArn"), this->credRole); httpQueryAdd(query, STRDEF("RoleSessionName"), STRDEF(PROJECT_NAME)); httpQueryAdd(query, STRDEF("Version"), STRDEF("2011-06-15")); httpQueryAdd(query, STRDEF("WebIdentityToken"), webIdToken); HttpRequest *const request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, FSLASH_STR, .header = header, .query = query); HttpResponse *const response = httpRequestResponse(request, true); // Error if the request was not successful, e.g. the web identity token is invalid. Without this the error would be a // confusing report that AssumeRoleWithWebIdentityResult is missing rather than the actual error returned by AWS. if (!httpResponseCodeOk(response)) httpRequestError(request, response); // Copy credentials const XmlNode *const xmlCred = xmlNodeChild( xmlNodeChild( xmlDocumentRoot(xmlDocumentNewBuf(httpResponseContent(response))), STRDEF("AssumeRoleWithWebIdentityResult"), true), STRDEF("Credentials"), true); const XmlNode *const accessKeyNode = xmlNodeChild(xmlCred, STRDEF("AccessKeyId"), true); const XmlNode *const secretAccessKeyNode = xmlNodeChild(xmlCred, STRDEF("SecretAccessKey"), true); const XmlNode *const securityTokenNode = xmlNodeChild(xmlCred, STRDEF("SessionToken"), true); MEM_CONTEXT_OBJ_BEGIN(this) { this->accessKey = xmlNodeContent(accessKeyNode); this->secretAccessKey = xmlNodeContent(secretAccessKeyNode); this->securityToken = xmlNodeContent(securityTokenNode); } MEM_CONTEXT_OBJ_END(); // Update expiration time this->credExpirationTime = storageS3CvtTime(xmlNodeContent(xmlNodeChild(xmlCred, STRDEF("Expiration"), true))); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Automatically get credentials for an associated pod identity The AWS documentation does not appear to provide a clear example of how to perform this authorization without using their SDK but this article proved helpful: https://securitylabs.datadoghq.com/articles/eks-pod-identity-deep-dive ***********************************************************************************************************************************/ static void storageS3AuthPodId(StorageS3 *const this, const HttpHeader *const header) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(HTTP_HEADER, header); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Load the token from the given file for each request since the token may be updated during execution const String *const podIdToken = strNewBuf(storageGetP(storageNewReadP(storagePosixNewP(FSLASH_STR), this->tokenFile))); // Get credentials. Redact the authorization header, which holds the token, so it is not exposed if the header is logged, // e.g. in an error. HttpRequest *const request = httpRequestNewP( this->credHttpClient, HTTP_VERB_GET_STR, this->credPath, .header = httpHeaderAdd(httpHeaderDup(header, this->headerRedactList), HTTP_HEADER_AUTHORIZATION_STR, podIdToken)); HttpResponse *const response = httpRequestResponse(request, true); // Error if the request was not successful, e.g. the pod identity token is invalid. Without this the error would be a // confusing report about missing JSON keys rather than the actual error returned by AWS. if (!httpResponseCodeOk(response)) httpRequestError(request, response); const KeyValue *const kvResponse = varKv(jsonToVar(strNewBuf(httpResponseContent(response)))); // Copy credentials MEM_CONTEXT_OBJ_BEGIN(this) { this->accessKey = strDup(varStr(kvGet(kvResponse, VARSTRDEF("AccessKeyId")))); CHECK(FormatError, this->accessKey != NULL, "access key missing"); this->secretAccessKey = strDup(varStr(kvGet(kvResponse, VARSTRDEF("SecretAccessKey")))); CHECK(FormatError, this->secretAccessKey != NULL, "secret access key missing"); this->securityToken = strDup(varStr(kvGet(kvResponse, VARSTRDEF("Token")))); CHECK(FormatError, this->securityToken != NULL, "token missing"); } MEM_CONTEXT_OBJ_END(); // Update expiration time const String *const credExpirationTime = varStr(kvGet(kvResponse, VARSTRDEF("Expiration"))); CHECK(FormatError, credExpirationTime != NULL, "expiration missing"); this->credExpirationTime = storageS3CvtTime(credExpirationTime); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get credentials from a process credential provider The process is executed and the JSON output is parsed for AccessKeyId, SecretAccessKey, SessionToken, and Expiration. Based on the "process credential provider" documentation at https://docs.aws.amazon.com/sdkref/latest/guide/feature-process-credentials.html ***********************************************************************************************************************************/ VARIANT_STRDEF_STATIC(S3_JSON_TAG_SESSION_TOKEN_VAR, "SessionToken"); static void storageS3AuthProcess(StorageS3 *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Execute the credential process and parse JSON output const String *const output = execOne(this->credCmd); const KeyValue *const kvResponse = varKv(jsonToVar(output)); // Copy credentials MEM_CONTEXT_OBJ_BEGIN(this) { this->accessKey = strDup(varStr(kvGet(kvResponse, S3_JSON_TAG_ACCESS_KEY_ID_VAR))); CHECK(FormatError, this->accessKey != NULL, "access key missing"); this->secretAccessKey = strDup(varStr(kvGet(kvResponse, S3_JSON_TAG_SECRET_ACCESS_KEY_VAR))); CHECK(FormatError, this->secretAccessKey != NULL, "secret access key missing"); this->securityToken = strDup(varStr(kvGet(kvResponse, S3_JSON_TAG_SESSION_TOKEN_VAR))); CHECK(FormatError, this->securityToken != NULL, "session token missing"); } MEM_CONTEXT_OBJ_END(); // Update expiration time const String *const credExpirationTime = varStr(kvGet(kvResponse, S3_JSON_TAG_EXPIRATION_VAR)); CHECK(FormatError, credExpirationTime != NULL, "expiration missing"); this->credExpirationTime = storageS3CvtTime(credExpirationTime); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Process S3 request ***********************************************************************************************************************************/ FN_EXTERN HttpRequest * storageS3RequestAsync(StorageS3 *const this, const String *const verb, const String *path, const StorageS3RequestAsyncParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(BOOL, param.sseKms); FUNCTION_LOG_PARAM(BOOL, param.sseC); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(verb != NULL); ASSERT(path != NULL); HttpRequest *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { HttpHeader *const requestHeader = param.header == NULL ? httpHeaderNew(this->headerRedactList) : httpHeaderDup(param.header, this->headerRedactList); // Set content length httpHeaderAdd( requestHeader, HTTP_HEADER_CONTENT_LENGTH_STR, param.content == NULL || bufEmpty(param.content) ? ZERO_STR : strNewFmt("%zu", bufUsed(param.content))); // Calculate content-md5 header if there is content if (param.content != NULL) { httpHeaderAdd( requestHeader, HTTP_HEADER_CONTENT_MD5_STR, strNewEncode(encodingBase64, cryptoHashOne(hashTypeMd5, param.content))); } // Set requester pays when requested if (this->requesterPays) httpHeaderPut(requestHeader, S3_HEADER_REQUEST_PAYER, S3_HEADER_REQUEST_PAYER_STR); // Set KMS headers when requested if (param.sseKms && this->kmsKeyId != NULL) { httpHeaderPut(requestHeader, S3_HEADER_SRVSDENC_STR, S3_HEADER_SRVSDENC_KMS_STR); httpHeaderPut(requestHeader, S3_HEADER_SRVSDENC_KMSKEYID_STR, this->kmsKeyId); } // Set SSE-C headers when requested if (param.sseC && this->sseCustomerKey != NULL) { httpHeaderPut(requestHeader, S3_HEADER_SSECUSTKEY_ALGO_STR, S3_HEADER_SSECUSTKEY_AES256_STR); httpHeaderPut(requestHeader, S3_HEADER_SSECUSTKEY_KEY_STR, this->sseCustomerKey); httpHeaderPut(requestHeader, S3_HEADER_SSECUSTKEY_KEY_MD5_STR, this->sseCustomerKeyMd5); } // Set tags when requested and available if (param.tag && this->tag != NULL) httpHeaderPut(requestHeader, S3_HEADER_TAGGING, this->tag); // When using path-style URIs the bucket name needs to be prepended if (this->uriStyle == storageS3UriStylePath) path = strNewFmt("/%s%s", strZ(this->bucket), strZ(path)); // If temp credentials will be expiring soon then renew them if (this->credExpirationTime != 0 && (this->credExpirationTime - time(NULL)) < S3_CREDENTIAL_RENEW_SEC) { // Free old credentials strFree(this->accessKey); strFree(this->secretAccessKey); strFree(this->securityToken); // Process credential authentication (no HTTP client needed) if (this->keyType == storageS3KeyTypeProcess) { storageS3AuthProcess(this); } // Else HTTP-based authentication methods else { // Set content-length and host headers HttpHeader *credHeader = httpHeaderNew(NULL); httpHeaderAdd(credHeader, HTTP_HEADER_CONTENT_LENGTH_STR, ZERO_STR); httpHeaderAdd(credHeader, HTTP_HEADER_HOST_STR, this->credHost); switch (this->keyType) { // Auto authentication case storageS3KeyTypeAuto: storageS3AuthAuto(this, credHeader); break; // Pod identity authentication case storageS3KeyTypePodId: storageS3AuthPodId(this, credHeader); break; // Web identity authentication default: { ASSERT(this->keyType == storageS3KeyTypeWebId); storageS3AuthWebId(this, credHeader); break; } } } // Reset the signing key date so the signing key gets regenerated this->signingKeyDate = YYYYMMDD_STR; } // Encode path path = httpUriEncode(path, true); // Generate authorization header storageS3Auth( this, verb, path, param.query, strNewTimeP("%Y%m%dT%H%M%SZ", time(NULL), .utc = true), requestHeader, strNewEncode( encodingHex, param.content == NULL || bufEmpty(param.content) ? HASH_TYPE_SHA256_ZERO_BUF : cryptoHashOne(hashTypeSha256, param.content))); // Send request MEM_CONTEXT_PRIOR_BEGIN() { result = httpRequestNewP( this->httpClient, verb, path, .query = param.query, .header = requestHeader, .content = param.content); } MEM_CONTEXT_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_REQUEST, result); } FN_EXTERN HttpResponse * storageS3Response(HttpRequest *const request, const StorageS3ResponseParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(HTTP_REQUEST, request); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_END(); ASSERT(request != NULL); HttpResponse *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Get response result = httpRequestResponse(request, !param.contentIo); // Error if the request was not successful if (!httpResponseCodeOk(result) && (!param.allowMissing || httpResponseCode(result) != HTTP_RESPONSE_CODE_NOT_FOUND)) httpRequestError(request, result); // Move response to the prior context httpResponseMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } FN_EXTERN HttpResponse * storageS3Request(StorageS3 *const this, const String *const verb, const String *const path, const StorageS3RequestParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, verb); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(HTTP_HEADER, param.header); FUNCTION_LOG_PARAM(HTTP_QUERY, param.query); FUNCTION_LOG_PARAM(BUFFER, param.content); FUNCTION_LOG_PARAM(BOOL, param.allowMissing); FUNCTION_LOG_PARAM(BOOL, param.contentIo); FUNCTION_LOG_PARAM(BOOL, param.sseKms); FUNCTION_LOG_PARAM(BOOL, param.sseC); FUNCTION_LOG_PARAM(BOOL, param.tag); FUNCTION_LOG_END(); HttpRequest *const request = storageS3RequestAsyncP( this, verb, path, .header = param.header, .query = param.query, .content = param.content, .sseKms = param.sseKms, .sseC = param.sseC, .tag = param.tag); HttpResponse *const result = storageS3ResponseP( request, .allowMissing = param.allowMissing, .contentIo = param.contentIo); httpRequestFree(request); FUNCTION_LOG_RETURN(HTTP_RESPONSE, result); } /*********************************************************************************************************************************** General function for listing files to be used by other list routines ***********************************************************************************************************************************/ static void storageS3ListInternal( StorageS3 *const this, const String *const path, const StorageInfoLevel level, const String *const expression, const bool recurse, const time_t targetTime, StorageListCallback callback, void *const callbackData) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, expression); FUNCTION_LOG_PARAM(BOOL, recurse); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, callback); FUNCTION_LOG_PARAM_P(VOID, callbackData); FUNCTION_LOG_END(); FUNCTION_AUDIT_CALLBACK(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { // Build the base prefix by stripping off the initial / const String *const basePrefix = strSize(path) == 1 ? EMPTY_STR : strNewFmt("%s/", strZ(strSub(path, 1))); // Get the expression prefix when possible to limit initial results const String *const expressionPrefix = regExpPrefix(expression); // If there is an expression prefix then use it to build the query prefix, otherwise query prefix is base prefix const String *queryPrefix; if (expressionPrefix == NULL) queryPrefix = basePrefix; else { if (strEmpty(basePrefix)) queryPrefix = expressionPrefix; else queryPrefix = strNewFmt("%s%s", strZ(basePrefix), strZ(expressionPrefix)); } // Create query HttpQuery *const query = httpQueryNewP(); // Add the delimiter to not recurse if (!recurse) httpQueryAdd(query, S3_QUERY_DELIMITER_STR, FSLASH_STR); // Use list type 2 or versions as specified if (targetTime != 0) httpQueryAdd(query, S3_QUERY_VERSIONS_STR, EMPTY_STR); else httpQueryAdd(query, S3_QUERY_LIST_TYPE_STR, S3_QUERY_VALUE_LIST_TYPE_2_STR); // Don't specify empty prefix because it is the default if (!strEmpty(queryPrefix)) httpQueryAdd(query, S3_QUERY_PREFIX_STR, queryPrefix); // Store last info so it can be updated across requests for versioning String *const nameLast = strNew(); String *const versionIdLast = strNew(); StorageInfo infoLast = {.level = level, .name = nameLast}; if (targetTime != 0) infoLast.versionId = versionIdLast; // Loop as long as a continuation token returned HttpRequest *request = NULL; do { // Use an inner mem context here because we could potentially be retrieving millions of files so it is a good idea to // free memory at regular intervals MEM_CONTEXT_TEMP_BEGIN() { HttpResponse *response = NULL; // If there is an outstanding async request then wait for the response if (request != NULL) { response = storageS3ResponseP(request); httpRequestFree(request); request = NULL; } // Else get the response immediately from a sync request else response = storageS3RequestP(this, HTTP_VERB_GET_STR, FSLASH_STR, .query = query); const XmlNode *const xmlRoot = xmlDocumentRoot(xmlDocumentNewBuf(httpResponseContent(response))); // If list is truncated then send an async request to get more data if (strEq(xmlNodeContent(xmlNodeChild(xmlRoot, S3_XML_TAG_IS_TRUNCATED_STR, true)), TRUE_STR)) { const String *const nextContinuationToken = xmlNodeContent( xmlNodeChild(xmlRoot, S3_XML_TAG_NEXT_CONTINUATION_TOKEN_STR, true)); CHECK(FormatError, !strEmpty(nextContinuationToken), S3_XML_TAG_NEXT_CONTINUATION_TOKEN " may not be empty"); httpQueryPut(query, S3_QUERY_CONTINUATION_TOKEN_STR, nextContinuationToken); // Store request in the outer temp context MEM_CONTEXT_PRIOR_BEGIN() { request = storageS3RequestAsyncP(this, HTTP_VERB_GET_STR, FSLASH_STR, .query = query); } MEM_CONTEXT_PRIOR_END(); } // Get prefix list const XmlNodeList *const subPathList = xmlNodeChildList(xmlRoot, S3_XML_TAG_COMMON_PREFIXES_STR); for (unsigned int subPathIdx = 0; subPathIdx < xmlNodeLstSize(subPathList); subPathIdx++) { const XmlNode *const subPathNode = xmlNodeLstGet(subPathList, subPathIdx); // Get path name StorageInfo info = { .level = level, .name = xmlNodeContent(xmlNodeChild(subPathNode, S3_XML_TAG_PREFIX_STR, true)), .exists = true, }; // Strip off base prefix and final / info.name = strSubN(info.name, strSize(basePrefix), strSize(info.name) - strSize(basePrefix) - 1); // Add type info if requested if (level >= storageInfoLevelType) info.type = storageTypePath; // Callback with info callback(callbackData, &info); } // Get file list const XmlNodeList *fileList; if (targetTime != 0) { StringList *const nameList = strLstNew(); strLstAdd(nameList, S3_XML_TAG_VERSION_STR); strLstAdd(nameList, S3_XML_TAG_DELETE_MARKER_STR); fileList = xmlNodeChildListMulti(xmlRoot, nameList); } else fileList = xmlNodeChildList(xmlRoot, S3_XML_TAG_CONTENTS_STR); for (unsigned int fileIdx = 0; fileIdx < xmlNodeLstSize(fileList); fileIdx++) { const XmlNode *const fileNode = xmlNodeLstGet(fileList, fileIdx); // Get file name and strip off the base prefix when present const String *name = xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_KEY_STR, true)); if (!strEmpty(basePrefix)) name = strSub(name, strSize(basePrefix)); // Return info for last file if new file if (infoLast.exists && !strEq(name, nameLast)) { callback(callbackData, &infoLast); infoLast.exists = false; } // If targeting by time if (targetTime != 0) { // Skip later versions infoLast.timeModified = storageS3CvtTime( xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_LAST_MODIFIED_STR, true))); if (infoLast.timeModified > targetTime) continue; // If a version has already been returned (or delete marker found) then skip this version if (strEq(infoLast.name, name)) continue; // If most recent version is a delete marker then the file will not be returned const bool deleteMarker = strEq(xmlNodeName(fileNode), S3_XML_TAG_DELETE_MARKER_STR); if (deleteMarker) { strCat(strTrunc(nameLast), name); infoLast.exists = false; continue; } } // Update last name and set exists strCat(strTrunc(nameLast), name); infoLast.exists = true; // Add basic info if requested (no need to add type info since file is default type) if (level >= storageInfoLevelBasic) { if (targetTime != 0) { strCat( strTrunc(versionIdLast), xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_VERSION_ID_STR, true))); } else { infoLast.timeModified = storageS3CvtTime( xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_LAST_MODIFIED_STR, true))); } infoLast.size = cvtZToUInt64(strZ(xmlNodeContent(xmlNodeChild(fileNode, S3_XML_TAG_SIZE_STR, true)))); } } } MEM_CONTEXT_TEMP_END(); } while (request != NULL); // Callback with last info if it exists if (infoLast.exists) callback(callbackData, &infoLast); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static StorageInfo storageS3Info(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); // Attempt to get file info HttpResponse *const httpResponse = storageS3RequestP(this, HTTP_VERB_HEAD_STR, file, .allowMissing = true, .sseC = true); // Does the file exist? StorageInfo result = {.level = level, .exists = httpResponseCodeOk(httpResponse)}; // Add basic level info if requested and the file exists (no need to add type info since file is default type) if (result.level >= storageInfoLevelBasic && result.exists) { const HttpHeader *const httpHeader = httpResponseHeader(httpResponse); const String *const contentLength = httpHeaderGet(httpHeader, HTTP_HEADER_CONTENT_LENGTH_STR); CHECK(FormatError, contentLength != NULL, "content length missing"); result.size = cvtZToUInt64(strZ(contentLength)); const String *const lastModified = httpHeaderGet(httpHeader, HTTP_HEADER_LAST_MODIFIED_STR); CHECK(FormatError, lastModified != NULL, "last modified missing"); result.timeModified = httpDateToTime(lastModified); } httpResponseFree(httpResponse); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ static void storageS3ListCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); storageLstAdd(callbackData, info); FUNCTION_TEST_RETURN_VOID(); } static StorageList * storageS3List(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_PARAM(TIME, param.targetTime); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); StorageList *const result = storageLstNew(level); storageS3ListInternal( this, path, level, param.expression, false, param.targetTime, storageS3ListCallback, result); FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void * storageS3NewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_PARAM(STRING, param.versionId); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); FUNCTION_LOG_RETURN(STORAGE_READ_S3, storageReadS3New(this, file, param.offset, param.limit, param.versionId)); } /**********************************************************************************************************************************/ static void * storageS3NewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.createPath); ASSERT(param.truncate); ASSERT(param.user == NULL); ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); FUNCTION_LOG_RETURN(STORAGE_WRITE_S3, storageWriteS3New(this, file, this->partSize)); } /**********************************************************************************************************************************/ typedef struct StorageS3PathRemoveData { StorageS3 *this; // Storage object MemContext *memContext; // Mem context to create xml document in unsigned int size; // Size of delete request HttpRequest *request; // Async delete request XmlDocument *xml; // Delete xml const String *path; // Root path of remove } StorageS3PathRemoveData; static HttpRequest * storageS3PathRemoveInternal(StorageS3 *const this, HttpRequest *const request, XmlDocument *const xml) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_S3, this); FUNCTION_TEST_PARAM(HTTP_REQUEST, request); FUNCTION_TEST_PARAM(XML_DOCUMENT, xml); FUNCTION_TEST_END(); ASSERT(this != NULL); // Get response for async request if (request != NULL) { MEM_CONTEXT_TEMP_BEGIN() { const Buffer *const response = httpResponseContent(storageS3ResponseP(request)); // Nothing is returned when there are no errors if (!bufEmpty(response)) { const XmlNodeList *const errorList = xmlNodeChildList( xmlDocumentRoot(xmlDocumentNewBuf(response)), S3_XML_TAG_ERROR_STR); // Attempt to remove errored files one at a time rather than retrying the batch for (unsigned int errorIdx = 0; errorIdx < xmlNodeLstSize(errorList); errorIdx++) { storageS3RequestP( this, HTTP_VERB_DELETE_STR, strNewFmt( "/%s", strZ(xmlNodeContent(xmlNodeChild(xmlNodeLstGet(errorList, errorIdx), S3_XML_TAG_KEY_STR, true))))); } } } MEM_CONTEXT_TEMP_END(); httpRequestFree(request); } // Send new async request if there is more to remove HttpRequest *result = NULL; if (xml != NULL) { HttpQuery *const query = httpQueryAdd(httpQueryNewP(), S3_QUERY_DELETE_STR, EMPTY_STR); Buffer *const content = xmlDocumentBuf(xml); result = storageS3RequestAsyncP(this, HTTP_VERB_POST_STR, FSLASH_STR, .query = query, .content = content); httpQueryFree(query); bufFree(content); } FUNCTION_TEST_RETURN(HTTP_REQUEST, result); } static void storageS3PathRemoveCallback(void *const callbackData, const StorageInfo *const info) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, callbackData); FUNCTION_TEST_PARAM(STORAGE_INFO, info); FUNCTION_TEST_END(); ASSERT(callbackData != NULL); ASSERT(info != NULL); // Only delete files since paths don't really exist if (info->type == storageTypeFile) { StorageS3PathRemoveData *const data = (StorageS3PathRemoveData *)callbackData; // If there is something to delete then create the request if (data->xml == NULL) { MEM_CONTEXT_BEGIN(data->memContext) { data->xml = xmlDocumentNew(S3_XML_TAG_DELETE_STR); xmlNodeContentSet(xmlNodeAdd(xmlDocumentRoot(data->xml), S3_XML_TAG_QUIET_STR), TRUE_STR); } MEM_CONTEXT_END(); } // Add to delete list MEM_CONTEXT_TEMP_BEGIN() { xmlNodeContentSet( xmlNodeAdd(xmlNodeAdd(xmlDocumentRoot(data->xml), S3_XML_TAG_OBJECT_STR), S3_XML_TAG_KEY_STR), strNewFmt("%s%s", strZ(data->path), strZ(info->name))); } MEM_CONTEXT_TEMP_END(); data->size++; // Delete list when it is full if (data->size == data->this->deleteMax) { MEM_CONTEXT_BEGIN(data->memContext) { data->request = storageS3PathRemoveInternal(data->this, data->request, data->xml); } MEM_CONTEXT_END(); xmlDocumentFree(data->xml); data->xml = NULL; data->size = 0; } } FUNCTION_TEST_RETURN_VOID(); } static bool storageS3PathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, recurse); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); MEM_CONTEXT_TEMP_BEGIN() { StorageS3PathRemoveData data = { .this = this, .memContext = memContextCurrent(), .path = strEq(path, FSLASH_STR) ? EMPTY_STR : strNewFmt("%s/", strZ(strSub(path, 1))), }; storageS3ListInternal(this, path, storageInfoLevelType, NULL, true, 0, storageS3PathRemoveCallback, &data); // Call if there is more to be removed if (data.xml != NULL) data.request = storageS3PathRemoveInternal(this, data.request, data.xml); // Check response on last async request storageS3PathRemoveInternal(this, data.request, NULL); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, true); } /**********************************************************************************************************************************/ static void storageS3Remove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageS3); FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_S3, this); FUNCTION_LOG_PARAM(STRING, file); (void)param; FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); httpResponseFree(storageS3RequestP(this, HTTP_VERB_DELETE_STR, file)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceS3 = { .feature = 1 << storageFeatureVersioning | 1 << storageFeatureReadRetry, .info = storageS3Info, .list = storageS3List, .newRead = storageS3NewRead, .newWrite = storageS3NewWrite, .pathRemove = storageS3PathRemove, .remove = storageS3Remove, }; FN_EXTERN Storage * storageS3New( const String *const path, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *const bucket, const String *const endPoint, const String *const region, const String *const service, const StorageS3KeyType keyType, const StorageS3UriStyle uriStyle, const String *const accessKey, const String *const secretAccessKey, const String *const securityToken, const String *const kmsKeyId, const String *sseCustomerKey, const String *const credRole, const String *const tokenFile, const String *const credUrl, const StringList *const credCmd, const String *const stsHost, const size_t partSize, const KeyValue *const tag, const String *host, const unsigned int port, const TimeMSec timeout, const HttpProtocolType protocolType, const bool verifyPeer, const String *const caFile, const String *const caPath, const bool requesterPays) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM(STRING, bucket); FUNCTION_LOG_PARAM(STRING, endPoint); FUNCTION_LOG_PARAM(STRING, region); FUNCTION_LOG_PARAM(STRING, service); FUNCTION_LOG_PARAM(ENUM, keyType); FUNCTION_LOG_PARAM(ENUM, uriStyle); FUNCTION_LOG_PARAM(STRING, stsHost); FUNCTION_TEST_PARAM(STRING, accessKey); FUNCTION_TEST_PARAM(STRING, secretAccessKey); FUNCTION_TEST_PARAM(STRING, securityToken); FUNCTION_TEST_PARAM(STRING, kmsKeyId); FUNCTION_TEST_PARAM(STRING, sseCustomerKey); FUNCTION_TEST_PARAM(STRING, credRole); FUNCTION_TEST_PARAM(STRING, tokenFile); FUNCTION_TEST_PARAM(STRING, credUrl); FUNCTION_TEST_PARAM(STRING_LIST, credCmd); FUNCTION_LOG_PARAM(SIZE, partSize); FUNCTION_LOG_PARAM(KEY_VALUE, tag); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_PARAM(ENUM, protocolType); FUNCTION_LOG_PARAM(BOOL, verifyPeer); FUNCTION_LOG_PARAM(STRING, caFile); FUNCTION_LOG_PARAM(STRING, caPath); FUNCTION_LOG_PARAM(BOOL, requesterPays); FUNCTION_LOG_END(); ASSERT(path != NULL); ASSERT(bucket != NULL); ASSERT(endPoint != NULL); ASSERT(region != NULL); ASSERT(service != NULL); ASSERT(partSize != 0); OBJ_NEW_BEGIN(StorageS3, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageS3) { .interface = storageInterfaceS3, .bucket = strDup(bucket), .region = strDup(region), .signingService = strDup(service), .keyType = keyType, .kmsKeyId = strDup(kmsKeyId), .requesterPays = requesterPays, .sseCustomerKey = strDup(sseCustomerKey), .partSize = partSize, .deleteMax = STORAGE_S3_DELETE_MAX, .uriStyle = uriStyle, .bucketEndpoint = uriStyle == storageS3UriStyleHost ? strNewFmt("%s.%s", strZ(bucket), strZ(endPoint)) : strDup(endPoint), // Force the signing key to be generated on the first run .signingKeyDate = YYYYMMDD_STR, }; // Create tag query string if (write && tag != NULL) { HttpQuery *const query = httpQueryNewP(.kv = tag); this->tag = httpQueryRenderP(query); httpQueryFree(query); } // Create the http client used to service requests. Use plain socket for HTTP, TLS for HTTPS. if (host == NULL) host = this->bucketEndpoint; IoClient *ioClient; if (protocolType == httpProtocolTypeHttp) ioClient = sckClientNew(host, port, timeout, timeout); else { ioClient = tlsClientNewP( sckClientNew(host, port, timeout, timeout), host, timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath); } this->httpClient = httpClientNew(ioClient, timeout); // Initialize authentication switch (this->keyType) { // Create the HTTP client used to retrieve temporary security credentials case storageS3KeyTypeAuto: { ASSERT(accessKey == NULL && secretAccessKey == NULL && securityToken == NULL); this->credRole = strDup(credRole); this->credHost = S3_CREDENTIAL_HOST_STR; this->credExpirationTime = time(NULL); this->credHttpClient = httpClientNew( sckClientNew(this->credHost, S3_CREDENTIAL_PORT, timeout, timeout), timeout); break; } // Create the HTTP client used to retrieve web identity security credentials case storageS3KeyTypeWebId: { ASSERT(accessKey == NULL && secretAccessKey == NULL && securityToken == NULL); ASSERT(credRole != NULL); ASSERT(tokenFile != NULL); const HttpUrl *const stsHostUrl = httpUrlNewParseP(stsHost, .type = httpProtocolTypeHttps); this->credRole = strDup(credRole); this->tokenFile = strDup(tokenFile); this->credHost = httpUrlHost(stsHostUrl); this->credExpirationTime = time(NULL); this->credHttpClient = httpClientNew( tlsClientNewP( sckClientNew(this->credHost, httpUrlPort(stsHostUrl), timeout, timeout), this->credHost, timeout, timeout, true, .caFile = caFile, .caPath = caPath), timeout); break; } // Create the HTTP client used to retrieve pod identity security credentials case storageS3KeyTypePodId: { ASSERT(accessKey == NULL && secretAccessKey == NULL && securityToken == NULL); ASSERT(credUrl != NULL); ASSERT(tokenFile != NULL); const HttpUrl *const url = httpUrlNewParseP(credUrl); this->tokenFile = strDup(tokenFile); this->credHost = httpUrlHost(url); this->credPath = strDup(httpUrlPath(url)); this->credExpirationTime = time(NULL); this->credHttpClient = httpClientNew(sckClientNew(this->credHost, httpUrlPort(url), timeout, timeout), timeout); break; } // Initialize process credential provider case storageS3KeyTypeProcess: { ASSERT(accessKey == NULL && secretAccessKey == NULL && securityToken == NULL); ASSERT(credCmd != NULL); this->credCmd = strLstDup(credCmd); this->credExpirationTime = time(NULL); break; } // Set shared key credentials default: { ASSERT(this->keyType == storageS3KeyTypeShared); ASSERT(accessKey != NULL && secretAccessKey != NULL); this->accessKey = strDup(accessKey); this->secretAccessKey = strDup(secretAccessKey); this->securityToken = strDup(securityToken); break; } } // Generate SSE customer key MD5 hash if (this->sseCustomerKey != NULL) { this->sseCustomerKeyMd5 = strNewEncode( encodingBase64, cryptoHashOne(hashTypeMd5, bufNewDecode(encodingBase64, this->sseCustomerKey))); } // Create list of redacted headers this->headerRedactList = strLstNew(); strLstAdd(this->headerRedactList, HTTP_HEADER_AUTHORIZATION_STR); strLstAdd(this->headerRedactList, S3_HEADER_DATE_STR); strLstAdd(this->headerRedactList, S3_HEADER_SSECUSTKEY_KEY_STR); strLstAdd(this->headerRedactList, S3_HEADER_SSECUSTKEY_KEY_MD5_STR); strLstAdd(this->headerRedactList, S3_HEADER_TOKEN_STR); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew(STORAGE_S3_TYPE, path, 0, 0, write, targetTime, pathExpressionFunction, this, this->interface)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/storage.h0000664000175100017510000000513115240547630020010 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage ***********************************************************************************************************************************/ #ifndef STORAGE_S3_STORAGE_H #define STORAGE_S3_STORAGE_H #include "common/io/http/url.h" #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_S3_TYPE STRID6("s3", 0x7d31) /*********************************************************************************************************************************** Key type ***********************************************************************************************************************************/ typedef enum { storageS3KeyTypeShared, storageS3KeyTypeAuto, storageS3KeyTypeWebId, storageS3KeyTypePodId, storageS3KeyTypeProcess, } StorageS3KeyType; /*********************************************************************************************************************************** URI style ***********************************************************************************************************************************/ typedef enum { storageS3UriStyleHost, storageS3UriStylePath, } StorageS3UriStyle; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN Storage *storageS3New( const String *path, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, const String *bucket, const String *endPoint, const String *region, const String *service, StorageS3KeyType keyType, StorageS3UriStyle uriStyle, const String *accessKey, const String *secretAccessKey, const String *securityToken, const String *kmsKeyId, const String *sseCustomerKey, const String *credRole, const String *tokenFile, const String *credUrl, const StringList *credCmd, const String *stsHost, size_t partSize, const KeyValue *tag, const String *host, unsigned int port, TimeMSec timeout, HttpProtocolType protocolType, bool verifyPeer, const String *caFile, const String *caPath, bool requesterPays); #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/storage.intern.h0000664000175100017510000001113115240547630021303 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_S3_STORAGE_INTERN_H #define STORAGE_S3_STORAGE_INTERN_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageS3 StorageS3; #include "common/io/http/request.h" #include "storage/s3/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Perform async request typedef struct StorageS3RequestAsyncParam { VAR_PARAM_HEADER; const HttpHeader *header; // Headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content bool sseKms; // Enable server-side encryption? bool sseC; // Enable server-side encryption with customer-provided keys? bool tag; // Add tags when available? } StorageS3RequestAsyncParam; #define storageS3RequestAsyncP(this, verb, path, ...) \ storageS3RequestAsync(this, verb, path, (StorageS3RequestAsyncParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpRequest *storageS3RequestAsync(StorageS3 *this, const String *verb, const String *path, StorageS3RequestAsyncParam param); // Get async response typedef struct StorageS3ResponseParam { VAR_PARAM_HEADER; bool allowMissing; // Allow missing files (caller can check response code) bool contentIo; // Is IoRead interface required to read content? } StorageS3ResponseParam; #define storageS3ResponseP(request, ...) \ storageS3Response(request, (StorageS3ResponseParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageS3Response(HttpRequest *request, StorageS3ResponseParam param); // Perform sync request typedef struct StorageS3RequestParam { VAR_PARAM_HEADER; const HttpHeader *header; // Headers const HttpQuery *query; // Query parameters const Buffer *content; // Request content bool allowMissing; // Allow missing files (caller can check response code) bool contentIo; // Is IoRead interface required to read content? bool sseKms; // Enable server-side encryption? bool sseC; // Enable server-side encryption with customer-provided keys? bool tag; // Add tags when available? } StorageS3RequestParam; #define storageS3RequestP(this, verb, path, ...) \ storageS3Request(this, verb, path, (StorageS3RequestParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN HttpResponse *storageS3Request(StorageS3 *this, const String *verb, const String *path, StorageS3RequestParam param); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_S3_TYPE \ StorageS3 * #define FUNCTION_LOG_STORAGE_S3_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageS3", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/write.c0000664000175100017510000002652715240547630017505 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage File Write ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/type/object.h" #include "common/type/xml.h" #include "storage/s3/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Part defaults based on limits at https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts.html ***********************************************************************************************************************************/ #define STORAGE_S3_SPLIT_DEFAULT 205 #define STORAGE_S3_SPLIT_MAX 9557 /*********************************************************************************************************************************** S3 query tokens ***********************************************************************************************************************************/ STRING_STATIC(S3_QUERY_PART_NUMBER_STR, "partNumber"); STRING_STATIC(S3_QUERY_UPLOADS_STR, "uploads"); STRING_STATIC(S3_QUERY_UPLOAD_ID_STR, "uploadId"); /*********************************************************************************************************************************** XML tags ***********************************************************************************************************************************/ STRING_STATIC(S3_XML_TAG_ETAG_STR, "ETag"); STRING_STATIC(S3_XML_TAG_UPLOAD_ID_STR, "UploadId"); STRING_STATIC(S3_XML_TAG_COMPLETE_MULTIPART_UPLOAD_STR, "CompleteMultipartUpload"); STRING_STATIC(S3_XML_TAG_PART_STR, "Part"); STRING_STATIC(S3_XML_TAG_PART_NUMBER_STR, "PartNumber"); /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteS3 { const StorageWriteInterface *interface; // Interface StorageS3 *storage; // Storage that created this object const String *name; // File name HttpRequest *request; // Async request size_t partSize; Buffer *partBuffer; const String *uploadId; StringList *uploadPartList; }; /*********************************************************************************************************************************** Flush bytes to upload part ***********************************************************************************************************************************/ static void storageWriteS3Part(StorageWriteS3 *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // If there is an outstanding async request then wait for the response and store the part id if (this->request != NULL) { HttpResponse *const response = storageS3ResponseP(this->request); strLstAdd(this->uploadPartList, httpHeaderGet(httpResponseHeader(response), HTTP_HEADER_ETAG_STR)); ASSERT(strLstGet(this->uploadPartList, strLstSize(this->uploadPartList) - 1) != NULL); httpResponseFree(response); httpRequestFree(this->request); this->request = NULL; } FUNCTION_LOG_RETURN_VOID(); } static void storageWriteS3PartAsync(StorageWriteS3 *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->partBuffer != NULL); ASSERT(bufSize(this->partBuffer) > 0); MEM_CONTEXT_TEMP_BEGIN() { // Complete prior async request, if any storageWriteS3Part(this); // Get the upload id if we have not already if (this->uploadId == NULL) { // Initiate mult-part upload const XmlNode *const xmlRoot = xmlDocumentRoot( xmlDocumentNewBuf( httpResponseContent( storageS3RequestP( this->storage, HTTP_VERB_POST_STR, this->name, .query = httpQueryAdd(httpQueryNewP(), S3_QUERY_UPLOADS_STR, EMPTY_STR), .sseKms = true, .sseC = true, .tag = true)))); // Store the upload id MEM_CONTEXT_OBJ_BEGIN(this) { this->uploadId = xmlNodeContent(xmlNodeChild(xmlRoot, S3_XML_TAG_UPLOAD_ID_STR, true)); this->uploadPartList = strLstNew(); } MEM_CONTEXT_OBJ_END(); } // Upload the part async HttpQuery *const query = httpQueryNewP(); httpQueryAdd(query, S3_QUERY_UPLOAD_ID_STR, this->uploadId); httpQueryAdd(query, S3_QUERY_PART_NUMBER_STR, strNewFmt("%u", strLstSize(this->uploadPartList) + 1)); MEM_CONTEXT_OBJ_BEGIN(this) { this->request = storageS3RequestAsyncP( this->storage, HTTP_VERB_PUT_STR, this->name, .query = query, .content = this->partBuffer, .sseC = true); } MEM_CONTEXT_OBJ_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to internal buffer ***********************************************************************************************************************************/ static void storageWriteS3(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_S3, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); // Continue until the write buffer has been exhausted size_t bytesTotal = 0; do { // Resize part buffer. This must be done in the loop because part size grows as parts are written. storageWriteChunkBufferResize(buffer, bytesTotal, this->partBuffer, this->partSize); // Copy as many bytes as possible into the part buffer const size_t bytesNext = bufRemains(this->partBuffer) > bufUsed(buffer) - bytesTotal ? bufUsed(buffer) - bytesTotal : bufRemains(this->partBuffer); bufCatSub(this->partBuffer, buffer, bytesTotal, bytesNext); bytesTotal += bytesNext; // If the part buffer is full then write it if (bufUsed(this->partBuffer) == this->partSize) { storageWriteS3PartAsync(this); bufUsedZero(this->partBuffer); this->partSize = storageWriteChunkSize( this->partSize, STORAGE_S3_SPLIT_DEFAULT, STORAGE_S3_SPLIT_MAX, strLstSize(this->uploadPartList)); } } while (bytesTotal != bufUsed(buffer)); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteS3Close(THIS_VOID) { THIS(StorageWriteS3); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_S3, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->partBuffer != NULL) { MEM_CONTEXT_TEMP_BEGIN() { // If a multi-part upload was started we need to finish that way if (this->uploadId != NULL) { // If there is anything left in the part buffer then write it if (!bufEmpty(this->partBuffer)) storageWriteS3PartAsync(this); // Complete prior async request, if any storageWriteS3Part(this); // Generate the xml part list XmlDocument *const partList = xmlDocumentNew(S3_XML_TAG_COMPLETE_MULTIPART_UPLOAD_STR); for (unsigned int partIdx = 0; partIdx < strLstSize(this->uploadPartList); partIdx++) { XmlNode *const partNode = xmlNodeAdd(xmlDocumentRoot(partList), S3_XML_TAG_PART_STR); xmlNodeContentSet(xmlNodeAdd(partNode, S3_XML_TAG_PART_NUMBER_STR), strNewFmt("%u", partIdx + 1)); xmlNodeContentSet(xmlNodeAdd(partNode, S3_XML_TAG_ETAG_STR), strLstGet(this->uploadPartList, partIdx)); } // Finalize the multi-part upload HttpRequest *const request = storageS3RequestAsyncP( this->storage, HTTP_VERB_POST_STR, this->name, .query = httpQueryAdd(httpQueryNewP(), S3_QUERY_UPLOAD_ID_STR, this->uploadId), .content = xmlDocumentBuf(partList)); HttpResponse *const response = storageS3ResponseP(request); // Error when no etag in the result. This indicates that the request did not succeed despite the success code. if (xmlNodeChild( xmlDocumentRoot(xmlDocumentNewBuf(httpResponseContent(response))), S3_XML_TAG_ETAG_STR, false) == NULL) { httpRequestError(request, response); } } // Else upload all the data in a single put else { storageS3RequestP( this->storage, HTTP_VERB_PUT_STR, this->name, .content = this->partBuffer, .sseKms = true, .sseC = true, .tag = true); } bufFree(this->partBuffer); this->partBuffer = NULL; } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteS3Interface = { .close = storageWriteS3Close, .write = storageWriteS3, }; FN_EXTERN StorageWriteS3 * storageWriteS3New(StorageS3 *const storage, const String *const name, const size_t partSize) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_S3, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(SIZE, partSize); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageWriteS3, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWriteS3) { .interface = &storageWriteS3Interface, .storage = storage, .name = strDup(name), .partSize = partSize, .partBuffer = bufNew(0), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_S3, this); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/s3/write.h0000664000175100017510000000335015240547630017477 0ustar00runnerrunner/*********************************************************************************************************************************** S3 Storage File Write ***********************************************************************************************************************************/ #ifndef STORAGE_S3_WRITE_H #define STORAGE_S3_WRITE_H #include "storage/s3/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteS3 StorageWriteS3; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteS3 *storageWriteS3New(StorageS3 *storage, const String *name, size_t partSize); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_S3_TYPE \ StorageWriteS3 * #define FUNCTION_LOG_STORAGE_WRITE_S3_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteS3", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/0000775000175100017510000000000015240547630016622 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/helper.c0000664000175100017510000000450015240547630020244 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Helper ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBSSH2 #include "common/debug.h" #include "common/log.h" #include "config/config.h" #include "storage/sftp/helper.h" /**********************************************************************************************************************************/ FN_EXTERN Storage * storageSftpHelper(const unsigned int repoIdx, const bool write, StoragePathExpressionCallback pathExpressionCallback) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(UINT, repoIdx); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM_P(VOID, pathExpressionCallback); FUNCTION_LOG_END(); ASSERT(cfgOptionIdxStrId(cfgOptRepoType, repoIdx) == STORAGE_SFTP_TYPE); Storage *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { const StringList *const knownHosts = strLstNewVarLst(cfgOptionIdxLst(cfgOptRepoSftpKnownHost, repoIdx)); MEM_CONTEXT_PRIOR_BEGIN() { result = storageSftpNewP( cfgOptionIdxStr(cfgOptRepoPath, repoIdx), cfgOptionIdxStr(cfgOptRepoSftpHost, repoIdx), cfgOptionIdxUInt(cfgOptRepoSftpHostPort, repoIdx), cfgOptionIdxStr(cfgOptRepoSftpHostUser, repoIdx), cfgOptionUInt64(cfgOptIoTimeout), cfgOptionIdxStr(cfgOptRepoSftpPrivateKeyFile, repoIdx), cfgOptionIdxStrId(cfgOptRepoSftpHostKeyHashType, repoIdx), .write = write, .pathExpressionFunction = pathExpressionCallback, .modeFile = STORAGE_MODE_FILE_DEFAULT, .modePath = STORAGE_MODE_PATH_DEFAULT, .keyPub = cfgOptionIdxStrNull(cfgOptRepoSftpPublicKeyFile, repoIdx), .keyPassphrase = cfgOptionIdxStrNull(cfgOptRepoSftpPrivateKeyPassphrase, repoIdx), .hostKeyCheckType = cfgOptionIdxStrId(cfgOptRepoSftpHostKeyCheckType, repoIdx), .hostFingerprint = cfgOptionIdxStrNull(cfgOptRepoSftpHostFingerprint, repoIdx), .knownHosts = knownHosts); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE, result); } #endif // HAVE_LIBSSH2 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/helper.h0000664000175100017510000000244215240547630020254 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Helper ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_STORAGE_HELPER_H #define STORAGE_SFTP_STORAGE_HELPER_H #ifdef HAVE_LIBSSH2 #include "storage/sftp/storage.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN Storage *storageSftpHelper(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); /*********************************************************************************************************************************** Storage helper for StorageHelper array passed to storageHelperInit() ***********************************************************************************************************************************/ #define STORAGE_SFTP_HELPER {.type = STORAGE_SFTP_TYPE, .helper = storageSftpHelper} #endif // HAVE_LIBSSH2 #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/read.c0000664000175100017510000002447315240547630017713 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Read ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBSSH2 #include "common/debug.h" #include "common/io/session.h" #include "common/log.h" #include "storage/read.h" #include "storage/sftp/read.h" /*********************************************************************************************************************************** Object types ***********************************************************************************************************************************/ struct StorageReadSftp { const StorageReadInterface *interface; // Interface StorageSftp *storage; // Storage that created this object const String *name; // File name uint64_t offset; // Read offset const Variant *limit; // Read limit (NULL for no limit) LIBSSH2_SFTP_HANDLE *sftpHandle; // LibSsh2 session sftp handle LIBSSH2_SFTP_ATTRIBUTES *attr; // LibSsh2 file attributes uint64_t current; // Current bytes read from file bool eof; // Did we reach end of file }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static bool storageReadSftpOpen(THIS_VOID) { THIS(StorageReadSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Retry the open once if the connection was lost (e.g. the server dropped an idle connection) and reopened successfully unsigned int retry = 0; int connErrno; do { do { this->sftpHandle = libssh2_sftp_open_ex( storageSftpSessionSftp(this->storage), strZ(this->name), (unsigned int)strSize(this->name), LIBSSH2_FXF_READ, 0, LIBSSH2_SFTP_OPENFILE); } while (this->sftpHandle == NULL && storageSftpWaitFd(this->storage, (connErrno = libssh2_session_last_errno(storageSftpSession(this->storage))))); } while (this->sftpHandle == NULL && retry++ == 0 && storageSftpReconnect(this->storage, connErrno)); if (this->sftpHandle == NULL) { const int rc = libssh2_session_last_errno(storageSftpSession(this->storage)); const uint64_t sftpErr = libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)); // Throw on any error except missing, which is reported to the caller via the result if (rc != LIBSSH2_ERROR_SFTP_PROTOCOL || sftpErr != LIBSSH2_FX_NO_SUCH_FILE) { storageSftpEvalLibSsh2Error( rc, sftpErr, &FileOpenError, strNewFmt(STORAGE_ERROR_READ_OPEN, strZ(this->name)), NULL); } } // Else success else { // Seek to offset, libssh2_sftp_seek64 returns void if (this->offset != 0) libssh2_sftp_seek64(this->sftpHandle, this->offset); } FUNCTION_LOG_RETURN(BOOL, this->sftpHandle != NULL); } /*********************************************************************************************************************************** Read from a file ***********************************************************************************************************************************/ static size_t storageReadSftp(THIS_VOID, Buffer *const buffer, const bool block) { THIS(StorageReadSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_SFTP, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_PARAM(BOOL, block); FUNCTION_LOG_END(); ASSERT(this != NULL && this->sftpHandle != NULL); ASSERT(buffer != NULL && !bufFull(buffer)); ssize_t actualBytes = 0; // Read if EOF has not been reached if (!this->eof) { // Rather than enable/disable limit checking just use a big number when there is no limit. We can feel pretty confident that // no files will be > UINT64_MAX in size. const uint64_t limit = this->limit == NULL ? UINT64_MAX : varUInt64(this->limit); // Determine expected bytes to read. If remaining size in the buffer would exceed the limit then reduce the expected read. size_t expectedBytes = bufRemains(buffer); if (this->current + expectedBytes > limit) expectedBytes = (size_t)(limit - this->current); bufLimitSet(buffer, expectedBytes); ssize_t rc = 0; // Read until EOF or buffer is full do { do { rc = libssh2_sftp_read(this->sftpHandle, (char *)bufRemainsPtr(buffer), bufRemains(buffer)); } while (storageSftpWaitFd(this->storage, rc)); // Break on EOF or error if (rc <= 0) break; // Account/shift for bytes read bufUsedInc(buffer, (size_t)rc); } while (!bufFull(buffer)); // Total bytes read into the buffer actualBytes = (ssize_t)bufUsed(buffer); // Error occurred during read if (rc < 0) { if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL) { uint64_t sftpErr = 0; // libssh2 sftp lseek seems to return LIBSSH2_FX_BAD_MESSAGE on a seek too far if ((sftpErr = libssh2_sftp_last_error(storageSftpSessionSftp(this->storage))) == LIBSSH2_FX_BAD_MESSAGE && this->offset > 0) { THROW_FMT(FileOpenError, STORAGE_ERROR_READ_SEEK, this->offset, strZ(this->name)); } else THROW_FMT(FileReadError, "unable to read '%s': sftp errno [%" PRIu64 "]", strZ(this->name), sftpErr); } else if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileReadError, "timeout reading '%s'", strZ(this->name)); else { storageSftpEvalLibSsh2Error( (int)rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileReadError, strNewFmt("unable to read '%s'", strZ(this->name)), NULL); } } // Update amount of buffer used this->current += (uint64_t)actualBytes; // If less data than expected was read or the limit has been reached then EOF. The file may not actually be EOF but we are // not concerned with files that are growing. Just read up to the point where the file is being extended. if ((size_t)actualBytes != expectedBytes || this->current == limit) this->eof = true; } FUNCTION_LOG_RETURN(SIZE, (size_t)actualBytes); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageReadSftpClose(THIS_VOID) { THIS(StorageReadSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_READ_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); if (this->sftpHandle != NULL) { int rc; // Close the file do { rc = libssh2_sftp_close(this->sftpHandle); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) THROW_FMT( FileCloseError, STORAGE_ERROR_READ_CLOSE ": libssh2 errno [%d]%s", strZ(this->name), rc, rc == LIBSSH2_ERROR_SFTP_PROTOCOL ? strZ(strNewFmt(": sftp errno [%lu]", libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)))) : ""); else { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileCloseError, strNewFmt("timeout closing file '%s'", strZ(this->name)), NULL); } } } this->sftpHandle = NULL; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Has file reached EOF? ***********************************************************************************************************************************/ static bool storageReadSftpEof(THIS_VOID) { THIS(StorageReadSftp); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_READ_SFTP, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(BOOL, this->eof); } /**********************************************************************************************************************************/ static const StorageReadInterface storageReadSftpInterface = { .close = storageReadSftpClose, .eof = storageReadSftpEof, .open = storageReadSftpOpen, .read = storageReadSftp, }; FN_EXTERN StorageReadSftp * storageReadSftpNew( StorageSftp *const storage, const String *const name, const uint64_t offset, const Variant *const limit) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(UINT64, offset); FUNCTION_LOG_PARAM(VARIANT, limit); FUNCTION_LOG_END(); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageReadSftp, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageReadSftp) { .interface = &storageReadSftpInterface, .storage = storage, .name = strDup(name), .offset = offset, .limit = varDup(limit), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_READ_SFTP, this); } #endif // HAVE_LIBSSH2 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/read.h0000664000175100017510000000340515240547630017710 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Read ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_READ_H #define STORAGE_SFTP_READ_H #include "storage/read.h" #include "storage/sftp/storage.intern.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageReadSftp StorageReadSftp; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageReadSftp *storageReadSftpNew(StorageSftp *storage, const String *name, uint64_t offset, const Variant *limit); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_READ_SFTP_TYPE \ StorageReadSftp * #define FUNCTION_LOG_STORAGE_READ_SFTP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageReadSftp", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/storage.c0000664000175100017510000017635615240547630020454 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBSSH2 #include "common/crypto/hash.h" #include "common/debug.h" #include "common/io/client.h" #include "common/io/fd.h" #include "common/io/session.h" #include "common/io/socket/client.h" #include "common/log.h" #include "common/regExp.h" #include "common/user.h" #include "storage/posix/storage.h" #include "storage/sftp/read.h" #include "storage/sftp/storage.intern.h" #include "storage/sftp/write.h" /*********************************************************************************************************************************** Define PATH_MAX if it is not defined ***********************************************************************************************************************************/ #ifndef PATH_MAX #define PATH_MAX (4 * 1024) #endif /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageSftp { STORAGE_COMMON_MEMBER; IoClient *ioClient; // IoClient (socket) used to (re)open sessions IoSession *ioSession; // IoSession (socket) connection to SFTP server LIBSSH2_SESSION *session; // LibSsh2 session LIBSSH2_SFTP *sftpSession; // LibSsh2 session sftp session TimeMSec timeout; // Session timeout // Connection parameters retained so the session can be reopened after the server drops an idle connection const String *host; // SFTP host unsigned int port; // SFTP port const String *user; // SFTP user const String *keyPriv; // Private key file path const String *keyPub; // Public key file path const String *keyPassphrase; // Private key passphrase StringId hostKeyHashType; // Host key hash type StringId hostKeyCheckType; // Host key check type const String *hostFingerprint; // Configured host fingerprint const StringList *knownHosts; // Known hosts file list }; /*********************************************************************************************************************************** Return known host key type based on host key type ***********************************************************************************************************************************/ static int storageSftpKnownHostKeyType(const int hostKeyType) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, hostKeyType); FUNCTION_TEST_END(); int result; switch (hostKeyType) { case LIBSSH2_HOSTKEY_TYPE_RSA: result = LIBSSH2_KNOWNHOST_KEY_SSHRSA; break; case LIBSSH2_HOSTKEY_TYPE_DSS: result = LIBSSH2_KNOWNHOST_KEY_SSHDSS; break; #ifdef LIBSSH2_HOSTKEY_TYPE_ECDSA_256 case LIBSSH2_HOSTKEY_TYPE_ECDSA_256: result = LIBSSH2_KNOWNHOST_KEY_ECDSA_256; break; #endif #ifdef LIBSSH2_HOSTKEY_TYPE_ECDSA_384 case LIBSSH2_HOSTKEY_TYPE_ECDSA_384: result = LIBSSH2_KNOWNHOST_KEY_ECDSA_384; break; #endif #ifdef LIBSSH2_HOSTKEY_TYPE_ECDSA_521 case LIBSSH2_HOSTKEY_TYPE_ECDSA_521: result = LIBSSH2_KNOWNHOST_KEY_ECDSA_521; break; #endif #ifdef LIBSSH2_HOSTKEY_TYPE_ED25519 case LIBSSH2_HOSTKEY_TYPE_ED25519: result = LIBSSH2_KNOWNHOST_KEY_ED25519; break; #endif default: result = 0; break; } FUNCTION_TEST_RETURN(INT, result); } /*********************************************************************************************************************************** Return error message based on error code ***********************************************************************************************************************************/ static const char * libssh2SftpErrorMsg(const uint64_t error) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(UINT64, error); FUNCTION_TEST_END(); const char *result; // SFTP error status codes (returned by libssh2_sftp_last_error() switch (error) { case LIBSSH2_FX_EOF: result = "eof"; break; case LIBSSH2_FX_NO_SUCH_FILE: result = "no such file"; break; case LIBSSH2_FX_PERMISSION_DENIED: result = "permission denied"; break; case LIBSSH2_FX_FAILURE: result = "failure"; break; case LIBSSH2_FX_BAD_MESSAGE: result = "bad message"; break; case LIBSSH2_FX_NO_CONNECTION: result = "no connection"; break; case LIBSSH2_FX_CONNECTION_LOST: result = "connection lost"; break; case LIBSSH2_FX_OP_UNSUPPORTED: result = "operation unsupported"; break; case LIBSSH2_FX_INVALID_HANDLE: result = "invalid handle"; break; case LIBSSH2_FX_NO_SUCH_PATH: result = "no such path"; break; case LIBSSH2_FX_FILE_ALREADY_EXISTS: result = "file already exists"; break; case LIBSSH2_FX_WRITE_PROTECT: result = "write protect"; break; case LIBSSH2_FX_NO_MEDIA: result = "no media"; break; case LIBSSH2_FX_NO_SPACE_ON_FILESYSTEM: result = "no space on filesystem"; break; case LIBSSH2_FX_QUOTA_EXCEEDED: result = "quota exceeded"; break; case LIBSSH2_FX_UNKNOWN_PRINCIPAL: result = "unknown principal"; break; case LIBSSH2_FX_LOCK_CONFLICT: result = "lock conflict"; break; case LIBSSH2_FX_DIR_NOT_EMPTY: result = "directory not empty"; break; case LIBSSH2_FX_NOT_A_DIRECTORY: result = "not a directory"; break; case LIBSSH2_FX_INVALID_FILENAME: result = "invalid filename"; break; case LIBSSH2_FX_LINK_LOOP: result = "link loop"; break; default: result = "unknown error"; break; } FUNCTION_TEST_RETURN_CONST(STRINGZ, result); } /*********************************************************************************************************************************** Return a match failed message based on known host check failure type ***********************************************************************************************************************************/ static const char * storageSftpKnownHostCheckpFailureMsg(const int rc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, rc); FUNCTION_TEST_END(); const char *result; switch (rc) { case LIBSSH2_KNOWNHOST_CHECK_FAILURE: result = "LIBSSH2_KNOWNHOST_CHECK_FAILURE"; break; case LIBSSH2_KNOWNHOST_CHECK_NOTFOUND: result = "not found in known hosts files: LIBSSH2_KNOWNHOST_CHECK_NOTFOUND"; break; case LIBSSH2_KNOWNHOST_CHECK_MISMATCH: result = "mismatch in known hosts files: LIBSSH2_KNOWNHOST_CHECK_MISMATCH"; break; default: result = "unknown failure"; break; } FUNCTION_TEST_RETURN_CONST(STRINGZ, result); } /**********************************************************************************************************************************/ static String * storageSftpLibSsh2SessionLastError(LIBSSH2_SESSION *const libSsh2Session) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM_P(VOID, libSsh2Session); FUNCTION_TEST_END(); String *result; char *libSsh2ErrMsg; int libSsh2ErrMsgLen; const int rc = libssh2_session_last_error(libSsh2Session, &libSsh2ErrMsg, &libSsh2ErrMsgLen, 0); if (libSsh2ErrMsgLen != 0) result = strNewZN(libSsh2ErrMsg, (size_t)libSsh2ErrMsgLen); else result = strNewFmt("libssh2 no session error message provided [%d]", rc); FUNCTION_TEST_RETURN(STRING, result); } /*********************************************************************************************************************************** Rewrite the user's known_hosts file with a new entry ***********************************************************************************************************************************/ static void storageSftpUpdateKnownHostsFile( StorageSftp *const this, const int hostKeyType, const String *const host, const char *const hostKey, const size_t hostKeyLen) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(INT, hostKeyType); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(STRINGZ, hostKey); FUNCTION_LOG_PARAM(SIZE, hostKeyLen); FUNCTION_LOG_END(); MEM_CONTEXT_TEMP_BEGIN() { // Init a known host collection for the user's known_hosts file const char *const userKnownHostsFile = strZ(strNewFmt("%s%s", strZ(userHome()), "/.ssh/known_hosts")); LIBSSH2_KNOWNHOSTS *const userKnownHostsList = libssh2_knownhost_init(this->session); LOG_WARN_FMT("host '%s' not found in known hosts files, attempting to add host to '%s'", strZ(host), userKnownHostsFile); if (userKnownHostsList == NULL) { // Get the libssh2 error message and emit warning const int rc = libssh2_session_last_errno(this->session); LOG_WARN_FMT( "libssh2_knownhost_init failed for '%s' for update: libssh2 errno [%d] %s", userKnownHostsFile, rc, strZ(storageSftpLibSsh2SessionLastError(this->session))); } else { // Read the user's known_hosts file entries into the collection. libssh2_knownhost_readfile() returns the number of // successfully loaded hosts or a negative value on error, an empty known hosts file will return 0. int rc; if ((rc = libssh2_knownhost_readfile(userKnownHostsList, userKnownHostsFile, LIBSSH2_KNOWNHOST_FILE_OPENSSH)) < 0) { // Missing known_hosts file will return LIBSSH2_ERROR_FILE. Possibly issues other than missing may return this. if (rc == LIBSSH2_ERROR_FILE) { // If user's known_hosts file is non-existent, create an empty one for libssh2 to operate on const Storage *const sshStorage = storagePosixNewP( strNewFmt("%s%s", strZ(userHome()), "/.ssh"), .modeFile = 0600, .modePath = 0700, .write = true); if (!storageExistsP(sshStorage, strNewFmt("%s", "known_hosts"))) storagePutP(storageNewWriteP(sshStorage, strNewFmt("%s", "known_hosts")), NULL); // Try to load the user's known_hosts file entries into the collection again rc = libssh2_knownhost_readfile(userKnownHostsList, userKnownHostsFile, LIBSSH2_KNOWNHOST_FILE_OPENSSH); } } // If the user's known_hosts file was read successfully, add the host to the collection and rewrite the file if (rc >= 0) { // Check for a supported known host key type const int knownHostKeyType = storageSftpKnownHostKeyType(hostKeyType); if (knownHostKeyType != 0) { // Add host to the internal list if (libssh2_knownhost_addc( userKnownHostsList, strZ(host), NULL, hostKey, hostKeyLen, strZ(strNewZ("Generated from " PROJECT_NAME)), strlen(strZ(strNewZ("Generated from " PROJECT_NAME))), LIBSSH2_KNOWNHOST_TYPE_PLAIN | LIBSSH2_KNOWNHOST_KEYENC_RAW | knownHostKeyType, NULL) == 0) { // Rewrite the updated known_hosts file rc = libssh2_knownhost_writefile(userKnownHostsList, userKnownHostsFile, LIBSSH2_KNOWNHOST_FILE_OPENSSH); if (rc != 0) LOG_WARN_FMT(PROJECT_NAME " unable to write '%s' for update", userKnownHostsFile); else LOG_WARN_FMT(PROJECT_NAME " added new host '%s' to '%s'", strZ(host), userKnownHostsFile); } else LOG_WARN_FMT(PROJECT_NAME " failed to add '%s' to known_hosts internal list", strZ(host)); } else LOG_WARN_FMT("unsupported key type [%d], unable to update knownhosts for '%s'", hostKeyType, strZ(host)); } else { // On readfile failure warn that we're unable to update the user's known_hosts file LOG_WARN_FMT( "libssh2 unable to read '%s' for update: libssh2 errno [%d] %s\n" "HINT: does '%s' exist with proper permissions?", userKnownHostsFile, rc, strZ(storageSftpLibSsh2SessionLastError(this->session)), userKnownHostsFile); } } // Free the user's known hosts list if (userKnownHostsList) libssh2_knownhost_free(userKnownHostsList); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Free libssh2 resources ***********************************************************************************************************************************/ static void storageSftpLibSsh2SessionFreeResource(THIS_VOID) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); int rc; if (this->sftpSession != NULL) { do { rc = libssh2_sftp_shutdown(this->sftpSession); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) THROW_FMT( ServiceError, "failed to shutdown sftpSession: libssh2 errno [%d]%s", rc, rc == LIBSSH2_ERROR_SFTP_PROTOCOL ? strZ(strNewFmt(": sftp errno [%lu] %s", libssh2_sftp_last_error(this->sftpSession), libssh2SftpErrorMsg(libssh2_sftp_last_error(this->sftpSession)))) : ""); else THROW_FMT( ServiceError, "timeout shutting down sftpSession: libssh2 errno [%d]", rc); } } if (this->session != NULL) { do { rc = libssh2_session_disconnect_ex(this->session, SSH_DISCONNECT_BY_APPLICATION, PROJECT_NAME " instance shutdown", ""); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "failed to disconnect libssh2 session: libssh2 errno [%d]", rc); else THROW_FMT(ServiceError, "timeout disconnecting libssh2 session: libssh2 errno [%d]", rc); } do { rc = libssh2_session_free(this->session); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "failed to free libssh2 session: libssh2 errno [%d]", rc); else THROW_FMT(ServiceError, "timeout freeing libssh2 session: libssh2 errno [%d]", rc); } } libssh2_exit(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN noreturn void storageSftpEvalLibSsh2Error( const int ssh2Errno, const uint64_t sftpErrno, const ErrorType *const errorType, const String *const message, const String *const hint) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(INT, ssh2Errno); FUNCTION_TEST_PARAM(UINT64, sftpErrno); FUNCTION_TEST_PARAM(ERROR_TYPE, errorType); FUNCTION_TEST_PARAM(STRING, message); FUNCTION_TEST_PARAM(STRING, hint); FUNCTION_TEST_END(); ASSERT(errorType != NULL); THROWP_FMT( errorType, "%s%s%s%s", message != NULL ? zNewFmt("%s%s", strZ(message), ssh2Errno == 0 ? "" : ": ") : "", ssh2Errno == 0 ? "" : zNewFmt("libssh2 error [%d]", ssh2Errno), ssh2Errno == LIBSSH2_ERROR_SFTP_PROTOCOL ? zNewFmt(": sftp error [%" PRIu64 "] %s", sftpErrno, libssh2SftpErrorMsg(sftpErrno)) : "", hint != NULL ? zNewFmt("\n%s", strZ(hint)) : ""); FUNCTION_TEST_NO_RETURN(); } /*********************************************************************************************************************************** Call in a loop whenever a libssh2 call might return LIBSSH2_ERROR_EAGAIN. We handle checking the rc from the libssh2 call here and will immediately exit out if it isn't LIBSSH2_ERROR_EAGAIN. Note that LIBSSH2_ERROR_EAGAIN can still be set after this call -- if that happens then there was a timeout while waiting for the fd to be ready. ***********************************************************************************************************************************/ FN_EXTERN bool storageSftpWaitFd(StorageSftp *const this, const int64_t rc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(INT64, rc); FUNCTION_TEST_END(); if (rc != LIBSSH2_ERROR_EAGAIN) FUNCTION_TEST_RETURN(BOOL, false); const int direction = libssh2_session_block_directions(this->session); const bool waitingRead = direction & LIBSSH2_SESSION_BLOCK_INBOUND; const bool waitingWrite = direction & LIBSSH2_SESSION_BLOCK_OUTBOUND; if (!waitingRead && !waitingWrite) FUNCTION_TEST_RETURN(BOOL, true); FUNCTION_TEST_RETURN(BOOL, fdReady(ioSessionFd(this->ioSession), waitingRead, waitingWrite, this->timeout)); } /**********************************************************************************************************************************/ static bool storageSftpLibSsh2FxNoSuchFile(THIS_VOID, const int rc) { THIS(StorageSftp); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(INT, rc); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN( BOOL, rc == LIBSSH2_ERROR_SFTP_PROTOCOL && libssh2_sftp_last_error(this->sftpSession) == LIBSSH2_FX_NO_SUCH_FILE); } /**********************************************************************************************************************************/ static StorageInfo storageSftpInfo(THIS_VOID, const String *const file, const StorageInfoLevel level, const StorageInterfaceInfoParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(ENUM, level); FUNCTION_LOG_PARAM(BOOL, param.followLink); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(file != NULL); StorageInfo result = {.level = level}; // Stat the file to check if it exists. Note that this function is also called per entry while a directory listing handle is // open (see storageSftpList()), so it must not reconnect on a lost connection -- doing so would free and invalidate the open // directory handle. A lost connection here is reported as an error and recovered by reconnecting at a higher-level operation. LIBSSH2_SFTP_ATTRIBUTES attr; int rc; do { rc = libssh2_sftp_stat_ex( this->sftpSession, strZ(file), (unsigned int)strSize(file), param.followLink ? LIBSSH2_SFTP_STAT : LIBSSH2_SFTP_LSTAT, &attr); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileOpenError, "timeout opening '%s'", strZ(file)); // Throw libssh2 on errors other than no such file if (!storageSftpLibSsh2FxNoSuchFile(this, rc)) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &FileOpenError, strNewFmt(STORAGE_ERROR_INFO, strZ(file)), NULL); } } // Else the file exists else { result.exists = true; // Add type info (no need set file type since it is the default) if (result.level >= storageInfoLevelType && !LIBSSH2_SFTP_S_ISREG(attr.permissions)) { if (LIBSSH2_SFTP_S_ISDIR(attr.permissions)) result.type = storageTypePath; else if (LIBSSH2_SFTP_S_ISLNK(attr.permissions)) result.type = storageTypeLink; else result.type = storageTypeSpecial; } // Add basic level info if (result.level >= storageInfoLevelBasic) { if ((attr.flags & LIBSSH2_SFTP_ATTR_ACMODTIME) != 0) result.timeModified = (time_t)attr.mtime; if (result.type == storageTypeFile) if ((attr.flags & LIBSSH2_SFTP_ATTR_SIZE) != 0) result.size = (uint64_t)attr.filesize; } // Add detail level info if (result.level >= storageInfoLevelDetail) { if ((attr.flags & LIBSSH2_SFTP_ATTR_UIDGID) != 0) { result.groupId = (unsigned int)attr.gid; result.userId = (unsigned int)attr.uid; } if ((attr.flags & LIBSSH2_SFTP_ATTR_PERMISSIONS) != 0) result.mode = attr.permissions & (LIBSSH2_SFTP_S_IRWXU | LIBSSH2_SFTP_S_IRWXG | LIBSSH2_SFTP_S_IRWXO); if (result.type == storageTypeLink) { char linkDestination[PATH_MAX] = {0}; ssize_t linkDestinationSize = 0; do { linkDestinationSize = libssh2_sftp_symlink_ex( this->sftpSession, strZ(file), (unsigned int)strSize(file), linkDestination, PATH_MAX - 1, LIBSSH2_SFTP_READLINK); } while (storageSftpWaitFd(this, linkDestinationSize)); if (linkDestinationSize == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileReadError, "timeout getting destination for link '%s'", strZ(file)); if (linkDestinationSize < 0) { storageSftpEvalLibSsh2Error( (int)linkDestinationSize, libssh2_sftp_last_error(this->sftpSession), &FileReadError, strNewFmt("unable to get destination for link '%s'", strZ(file)), NULL); } result.linkDestination = strNewZN(linkDestination, (size_t)linkDestinationSize); } } } FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /*********************************************************************************************************************************** Build known hosts file list. If knownHosts is empty build the default file list, otherwise build the list provided. knownHosts requires full path and/or leading tilde path entries. ***********************************************************************************************************************************/ static StringList * storageSftpKnownHostsFilesList(const StringList *const knownHosts) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING_LIST, knownHosts); FUNCTION_LOG_END(); StringList *const result = strLstNew(); MEM_CONTEXT_TEMP_BEGIN() { if (strLstEmpty(knownHosts)) { // Create default file list strLstAddFmt(result, "%s%s", strZ(userHome()), "/.ssh/known_hosts"); strLstAddFmt(result, "%s%s", strZ(userHome()), "/.ssh/known_hosts2"); strLstAddZ(result, "/etc/ssh/ssh_known_hosts"); strLstAddZ(result, "/etc/ssh/ssh_known_hosts2"); } else { // Process the known host list entries and add them to the result list for (unsigned int listIdx = 0; listIdx < strLstSize(knownHosts); listIdx++) { // Get the trimmed file path and add it to the result list const String *const filePath = strTrim(strLstGet(knownHosts, listIdx)); if (strBeginsWithZ(filePath, "~/")) { // Replace leading tilde with space, trim space, prepend user home path and add to the result list strLstAddFmt( result, "%s%s", strZ(userHome()), strZ(strTrim(strSub(filePath, (size_t)strChr(filePath, '~') + 1)))); } else strLstAdd(result, filePath); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ static String * storageSftpExpandTildePath(const String *const tildePath) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STRING, tildePath); FUNCTION_TEST_END(); String *const result = strNew(); // Append to user home directory path substring after the tilde MEM_CONTEXT_TEMP_BEGIN() { strCatFmt(result, "%s%s", strZ(userHome()), strZ(strSub(tildePath, (size_t)strChr(tildePath, '~') + 1))); } MEM_CONTEXT_TEMP_END(); FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ // Helper function to get info for a file if it exists. This logic can't live directly in storageSftpList() because there is a race // condition where a file might exist while listing the directory but it is gone before stat() can be called. In order to get // complete test coverage this function must be split out. static void storageSftpListEntry( StorageSftp *const this, StorageList *const list, const String *const path, const char *const name, const StorageInfoLevel level) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(STORAGE_LIST, list); FUNCTION_TEST_PARAM(STRING, path); FUNCTION_TEST_PARAM(STRINGZ, name); FUNCTION_TEST_PARAM(ENUM, level); FUNCTION_TEST_END(); FUNCTION_AUDIT_HELPER(); ASSERT(this != NULL); ASSERT(list != NULL); ASSERT(path != NULL); ASSERT(name != NULL); StorageInfo info = storageInterfaceInfoP(this, strNewFmt("%s/%s", strZ(path), name), level); if (info.exists) { info.name = STR(name); storageLstAdd(list, &info); } FUNCTION_TEST_RETURN_VOID(); } static StorageList * storageSftpList(THIS_VOID, const String *const path, const StorageInfoLevel level, const StorageInterfaceListParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(ENUM, level); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); ASSERT(param.targetTime == 0); StorageList *result = NULL; // Open the directory for read, retrying the open once if the connection was lost and reopened successfully LIBSSH2_SFTP_HANDLE *sftpHandle; unsigned int retry = 0; int connErrno; do { do { sftpHandle = libssh2_sftp_open_ex(this->sftpSession, strZ(path), (unsigned int)strSize(path), 0, 0, LIBSSH2_SFTP_OPENDIR); } while (sftpHandle == NULL && storageSftpWaitFd(this, (connErrno = libssh2_session_last_errno(this->session)))); } while (sftpHandle == NULL && retry++ == 0 && storageSftpReconnect(this, connErrno)); // If the directory could not be opened process errors and report missing directories if (sftpHandle == NULL) { const int rc = libssh2_session_last_errno(this->session); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileReadError, "timeout opening directory '%s'", strZ(path)); // If sftpHandle == NULL is due to LIBSSH2_FX_NO_SUCH_FILE, do not throw error here, return NULL result if (!storageSftpLibSsh2FxNoSuchFile(this, rc)) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathOpenError, strNewFmt(STORAGE_ERROR_LIST_INFO, strZ(path)), NULL); } } else { // Directory was found result = storageLstNew(level); TRY_BEGIN() { MEM_CONTEXT_TEMP_RESET_BEGIN() { LIBSSH2_SFTP_ATTRIBUTES attr; char filename[PATH_MAX] = {0}; int len; // Read the directory entries do { len = libssh2_sftp_readdir_ex(sftpHandle, filename, PATH_MAX - 1, NULL, 0, &attr); if (len > 0) { filename[len] = '\0'; // Always skip . and .. if (!strEqZ(DOT_STR, filename) && !strEqZ(DOTDOT_STR, filename)) { if (level == storageInfoLevelExists) { const StorageInfo storageInfo = { .name = STR(filename), .level = storageInfoLevelExists, .exists = true, }; storageLstAdd(result, &storageInfo); } else storageSftpListEntry(this, result, path, filename, level); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } while (len > 0 || storageSftpWaitFd(this, len)); } MEM_CONTEXT_TEMP_END(); } FINALLY() { int rc; do { rc = libssh2_sftp_closedir(sftpHandle); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc != LIBSSH2_ERROR_EAGAIN) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathCloseError, strNewFmt("unable to close path '%s' after listing", strZ(path)), NULL); } else THROW_FMT(PathCloseError, "timeout closing path '%s' after listing", strZ(path)); } sftpHandle = NULL; } TRY_END(); } FUNCTION_LOG_RETURN(STORAGE_LIST, result); } /**********************************************************************************************************************************/ static void storageSftpRemove(THIS_VOID, const String *const file, const StorageInterfaceRemoveParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(!param.errorOnMissing); // Attempt to unlink the file int rc; do { rc = libssh2_sftp_unlink_ex(this->sftpSession, strZ(file), (unsigned int)strSize(file)); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileRemoveError, "timeout removing '%s'", strZ(file)); if (!storageSftpLibSsh2FxNoSuchFile(this, rc)) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &FileRemoveError, strNewFmt(STORAGE_ERROR_FILE_REMOVE, strZ(file)), NULL); } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static void * storageSftpNewRead(THIS_VOID, const String *const file, const StorageInterfaceNewReadParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.versionId == NULL); FUNCTION_LOG_RETURN(STORAGE_READ_SFTP, storageReadSftpNew(this, file, param.offset, param.limit)); } /**********************************************************************************************************************************/ static void * storageSftpNewWrite(THIS_VOID, const String *const file, const StorageInterfaceNewWriteParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, file); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(STRING, param.user); FUNCTION_LOG_PARAM(STRING, param.group); FUNCTION_LOG_PARAM(TIME, param.timeModified); FUNCTION_LOG_PARAM(BOOL, param.createPath); FUNCTION_LOG_PARAM(BOOL, param.syncFile); FUNCTION_LOG_PARAM(BOOL, param.syncPath); FUNCTION_LOG_PARAM(BOOL, param.atomic); FUNCTION_LOG_PARAM(BOOL, param.truncate); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(file != NULL); ASSERT(param.createPath); ASSERT(param.truncate); ASSERT(param.user == NULL); ASSERT(param.group == NULL); ASSERT(param.timeModified == 0); FUNCTION_LOG_RETURN( STORAGE_WRITE_SFTP, storageWriteSftpNew( this, file, param.modeFile, param.modePath, param.user, param.group, param.timeModified, param.createPath, param.syncFile, param.atomic, param.truncate)); } /**********************************************************************************************************************************/ static void storageSftpPathCreate( THIS_VOID, const String *const path, const bool errorOnExists, const bool noParentCreate, const mode_t mode, const StorageInterfacePathCreateParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, errorOnExists); FUNCTION_LOG_PARAM(BOOL, noParentCreate); FUNCTION_LOG_PARAM(MODE, mode); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); int rc; // Attempt to create the directory do { rc = libssh2_sftp_mkdir_ex(this->sftpSession, strZ(path), (unsigned int)strSize(path), (int)mode); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(PathCreateError, "timeout creating path '%s'", strZ(path)); if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL) { const uint64_t sftpErrno = libssh2_sftp_last_error(this->sftpSession); // libssh2 may return LIBSSH2_FX_FAILURE if the directory already exists if (sftpErrno == LIBSSH2_FX_FAILURE || sftpErrno == LIBSSH2_FX_FILE_ALREADY_EXISTS) { // Check if the directory already exists LIBSSH2_SFTP_ATTRIBUTES attr; do { rc = libssh2_sftp_stat_ex( this->sftpSession, strZ(path), (unsigned int)strSize(path), LIBSSH2_SFTP_STAT, &attr); } while (storageSftpWaitFd(this, rc)); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(PathCreateError, "timeout stat'ing path '%s'", strZ(path)); // If rc = 0 then already exists if (rc == 0 && errorOnExists) { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathCreateError, strNewFmt("sftp error unable to create path '%s': path already exists", strZ(path)), NULL); } } // If the parent path does not exist then create it if allowed else if (sftpErrno == LIBSSH2_FX_NO_SUCH_FILE && !noParentCreate) { String *const pathParent = strPath(path); storageInterfacePathCreateP(this, pathParent, errorOnExists, noParentCreate, mode); storageInterfacePathCreateP(this, path, errorOnExists, noParentCreate, mode); strFree(pathParent); } else { storageSftpEvalLibSsh2Error( rc, sftpErrno, &PathCreateError, strNewFmt("sftp error unable to create path '%s'", strZ(path)), NULL); } } else { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathCreateError, strNewFmt("ssh2 error [%d] unable to create path '%s'", rc, strZ(path)), NULL); } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static bool storageSftpPathRemove(THIS_VOID, const String *const path, const bool recurse, const StorageInterfacePathRemoveParam param) { THIS(StorageSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(BOOL, recurse); (void)param; // No parameters are used FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(path != NULL); bool result = true; MEM_CONTEXT_TEMP_BEGIN() { // Recurse if requested if (recurse) { const StorageList *const list = storageInterfaceListP(this, path, storageInfoLevelExists); if (list != NULL) { MEM_CONTEXT_TEMP_RESET_BEGIN() { for (unsigned int listIdx = 0; listIdx < storageLstSize(list); listIdx++) { const String *const file = strNewFmt("%s/%s", strZ(path), strZ(storageLstGet(list, listIdx).name)); // Rather than stat the file to discover what type it is, just try to unlink it and see what happens int rc; do { rc = libssh2_sftp_unlink_ex(this->sftpSession, strZ(file), (unsigned int)strSize(file)); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(PathRemoveError, "timeout removing file '%s'", strZ(file)); // Attempting to unlink a directory appears to return LIBSSH2_FX_FAILURE or LIBSSH2_FX_PERMISSION_DENIED if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL) { const uint64_t sftpErrno = libssh2_sftp_last_error(this->sftpSession); if (sftpErrno == LIBSSH2_FX_FAILURE || sftpErrno == LIBSSH2_FX_PERMISSION_DENIED) storageInterfacePathRemoveP(this, file, true); else { THROW_FMT( PathRemoveError, STORAGE_ERROR_FILE_REMOVE ": libssh sftp [%" PRIu64 "] %s", strZ(file), sftpErrno, libssh2SftpErrorMsg(sftpErrno)); } } else THROW_FMT(PathRemoveError, STORAGE_ERROR_FILE_REMOVE ": libssh ssh [%d]", strZ(file), rc); } // Reset the memory context occasionally so we don't use too much memory or slow down processing MEM_CONTEXT_TEMP_RESET(1000); } } MEM_CONTEXT_TEMP_END(); } } // Delete the path int rc; do { rc = libssh2_sftp_rmdir_ex(this->sftpSession, strZ(path), (unsigned int)strSize(path)); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(PathRemoveError, "timeout removing path '%s'", strZ(path)); if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL) { const uint64_t sftpErrno = libssh2_sftp_last_error(this->sftpSession); if (sftpErrno != LIBSSH2_FX_NO_SUCH_FILE) { THROW_FMT( PathRemoveError, STORAGE_ERROR_PATH_REMOVE " sftp error [%" PRIu64 "] %s", strZ(path), sftpErrno, libssh2SftpErrorMsg(sftpErrno)); } // Path does not exist result = false; } else { // Path does not exist result = false; storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &PathRemoveError, strNewFmt(STORAGE_ERROR_PATH_REMOVE, strZ(path)), NULL); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN LIBSSH2_SESSION * storageSftpSession(StorageSftp *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_TYPE_P(LIBSSH2_SESSION, this->session); } FN_EXTERN LIBSSH2_SFTP * storageSftpSessionSftp(StorageSftp *const this) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN_TYPE_P(LIBSSH2_SFTP, this->sftpSession); } /*********************************************************************************************************************************** Is the libssh2 error a sign that the connection to the server has been lost? Socket-level errors always mean the connection is gone. LIBSSH2_ERROR_SOCKET_SEND is intentionally excluded since a failed send is ambiguous (and is used throughout the unit tests as a generic fatal error) -- the definitive "the peer is gone" signal is a failed receive. An sftp protocol error is ambiguous in general (e.g. mkdir returns LIBSSH2_FX_FAILURE when a directory already exists) but on a file or directory open a generic failure or explicit connection-lost status means the session is no longer usable, so treat those as a lost connection. This is only called from the open retry paths, so the ambiguous mkdir/rename cases do not reach here. ***********************************************************************************************************************************/ static bool storageSftpConnLost(StorageSftp *const this, const int rc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(INT, rc); FUNCTION_TEST_END(); ASSERT(this != NULL); switch (rc) { case LIBSSH2_ERROR_SOCKET_RECV: case LIBSSH2_ERROR_SOCKET_TIMEOUT: case LIBSSH2_ERROR_SOCKET_DISCONNECT: case LIBSSH2_ERROR_BAD_USE: FUNCTION_TEST_RETURN(BOOL, true); break; case LIBSSH2_ERROR_SFTP_PROTOCOL: { const uint64_t sftpErr = libssh2_sftp_last_error(this->sftpSession); FUNCTION_TEST_RETURN( BOOL, sftpErr == LIBSSH2_FX_FAILURE || sftpErr == LIBSSH2_FX_CONNECTION_LOST || sftpErr == LIBSSH2_FX_NO_CONNECTION); break; } default: break; } FUNCTION_TEST_RETURN(BOOL, false); } /*********************************************************************************************************************************** Free the current session without a graceful disconnect. Used before reopening since the connection is already known to be broken; errors are ignored because the only goal is to release resources before reconnecting. This is only ever called from storageSftpReopen() on a connection that has already been opened, so the session and socket are always set. ***********************************************************************************************************************************/ static void storageSftpConnFree(StorageSftp *const this) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->session != NULL); ASSERT(this->ioSession != NULL); // Free the libssh2 session int rc; do { rc = libssh2_session_free(this->session); } while (storageSftpWaitFd(this, rc)); this->session = NULL; this->sftpSession = NULL; // Free the socket session, which closes the file descriptor ioSessionFree(this->ioSession); this->ioSession = NULL; FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Open a socket session, perform the libssh2 handshake, host key verification, authentication, and init the sftp session. The connection parameters are read from the object so this can be called again by storageSftpReopen() after an idle connection is lost. ***********************************************************************************************************************************/ static void storageSftpOpen(StorageSftp *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // The socket/sftp session created here persists in the object context (not the caller's) so the single-return-object audit // does not apply. This also allows the session to be (re)opened from within another operation that is being audited. FUNCTION_AUDIT_HELPER(); // Open the socket session in the object context so it persists with the object MEM_CONTEXT_OBJ_BEGIN(this) { this->ioSession = ioClientOpen(this->ioClient); } MEM_CONTEXT_OBJ_END(); MEM_CONTEXT_TEMP_BEGIN() { this->session = libssh2_session_init(); if (this->session == NULL) THROW_FMT(ServiceError, "unable to init libssh2 session"); // Set session to non-blocking libssh2_session_set_blocking(this->session, 0); // Perform handshake int rc; do { rc = libssh2_session_handshake(this->session, ioSessionFd(this->ioSession)); } while (storageSftpWaitFd(this, rc)); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "timeout during libssh2 handshake [%d]", rc); if (rc != 0) { THROW_FMT( ServiceError, "libssh2 handshake failed [%d]: %s", rc, strZ(storageSftpLibSsh2SessionLastError(this->session))); } int hashType = LIBSSH2_HOSTKEY_HASH_SHA1; size_t hashSize = 0; // Verify that the fingerprint[N] buffer declared below is large enough when adding a new hashType switch (this->hostKeyHashType) { case hashTypeMd5: hashType = LIBSSH2_HOSTKEY_HASH_MD5; hashSize = HASH_TYPE_M5_SIZE; break; case hashTypeSha1: hashType = LIBSSH2_HOSTKEY_HASH_SHA1; hashSize = HASH_TYPE_SHA1_SIZE; break; #ifdef LIBSSH2_HOSTKEY_HASH_SHA256 case hashTypeSha256: hashType = LIBSSH2_HOSTKEY_HASH_SHA256; hashSize = HASH_TYPE_SHA256_SIZE; break; #endif // LIBSSH2_HOSTKEY_HASH_SHA256 default: THROW_FMT(ServiceError, "requested ssh2 hostkey hash type (%s) not available", zNewStrId(this->hostKeyHashType)); break; } // Compare fingerprint if provided else check known hosts files for a match if (this->hostKeyCheckType == SFTP_STRICT_HOSTKEY_CHECKING_FINGERPRINT) { const char *const binaryFingerprint = libssh2_hostkey_hash(this->session, hashType); if (binaryFingerprint == NULL) { THROW_FMT( ServiceError, "libssh2 hostkey hash failed: libssh2 errno [%d]", libssh2_session_last_errno(this->session)); } // 256 bytes is large enough to hold the hex representation of currently supported hash types. The hex encoded version // requires twice as much space (hashSize * 2) as the raw version. char fingerprint[256]; encodeToStr(encodingHex, (const uint8_t *)binaryFingerprint, hashSize, fingerprint); if (strcmp(fingerprint, strZ(this->hostFingerprint)) != 0) { THROW_FMT( ServiceError, "host [%s] and configured fingerprint (repo-sftp-host-fingerprint) [%s] do not match", fingerprint, strZ(this->hostFingerprint)); } } else if (this->hostKeyCheckType != SFTP_STRICT_HOSTKEY_CHECKING_NONE) { // Init the known host collection LIBSSH2_KNOWNHOSTS *const knownHostsList = libssh2_knownhost_init(this->session); if (knownHostsList == NULL) { const int rc = libssh2_session_last_errno(this->session); THROW_FMT( ServiceError, "failure during libssh2_knownhost_init: libssh2 errno [%d] %s", rc, strZ(storageSftpLibSsh2SessionLastError(this->session))); } // Get the list of known host files to search const StringList *const knownHostsPathList = storageSftpKnownHostsFilesList(this->knownHosts); // Loop through the list of known host files for (unsigned int listIdx = 0; listIdx < strLstSize(knownHostsPathList); listIdx++) { const char *const currentKnownHostFile = strZNull(strLstGet(knownHostsPathList, listIdx)); // Read the known hosts file entries into the collection, log message for readfile status. // libssh2_knownhost_readfile() returns the number of successfully loaded hosts or a negative value on error, an // empty known hosts file will return 0. if ((rc = libssh2_knownhost_readfile(knownHostsList, currentKnownHostFile, LIBSSH2_KNOWNHOST_FILE_OPENSSH)) <= 0) { if (rc == 0) LOG_DETAIL_FMT("libssh2 '%s' file is empty", currentKnownHostFile); else { LOG_DETAIL_FMT( "libssh2 read '%s' failed: libssh2 errno [%d] %s", currentKnownHostFile, rc, strZ(storageSftpLibSsh2SessionLastError(this->session))); } } else LOG_DETAIL_FMT("libssh2 read '%s' succeeded", currentKnownHostFile); } // Get the remote host key size_t hostKeyLen; int hostKeyType; const char *const hostKey = libssh2_session_hostkey(this->session, &hostKeyLen, &hostKeyType); // Check for a match in known hosts files else throw an error if no host key was retrieved if (hostKey != NULL) { rc = libssh2_knownhost_checkp( knownHostsList, strZ(this->host), (int)this->port, hostKey, hostKeyLen, LIBSSH2_KNOWNHOST_TYPE_PLAIN | LIBSSH2_KNOWNHOST_KEYENC_RAW, NULL); // Handle check success/failure if (rc == LIBSSH2_KNOWNHOST_CHECK_MATCH) LOG_DETAIL_FMT("known hosts match found for '%s'", strZ(this->host)); else { // Handle failure to match in a similar manner as ssh_config StrictHostKeyChecking. If this flag is set to // "strict", never automatically add host keys to the ~/.ssh/known_hosts file, and refuse to connect to hosts // whose host key has changed. This option forces the user to manually add all new hosts. If this flag is set to // "accept-new" then automatically add new host keys to the user known hosts files, but do not permit // connections to hosts with changed host keys. switch (this->hostKeyCheckType) { case SFTP_STRICT_HOSTKEY_CHECKING_STRICT: { // Throw an error when set to strict and we have any result other than match libssh2_knownhost_free(knownHostsList); THROW_FMT( ServiceError, "known hosts failure: '%s' %s [%d]: check type [%s]", strZ(this->host), storageSftpKnownHostCheckpFailureMsg(rc), rc, zNewStrId(this->hostKeyCheckType)); break; } default: { ASSERT(this->hostKeyCheckType == SFTP_STRICT_HOSTKEY_CHECKING_ACCEPT_NEW); // Throw an error when set to accept-new and match fails or mismatches else add the new host key to the // user's known_hosts file if (rc == LIBSSH2_KNOWNHOST_CHECK_MISMATCH || rc == LIBSSH2_KNOWNHOST_CHECK_FAILURE) { // Free the known hosts list libssh2_knownhost_free(knownHostsList); THROW_FMT( ServiceError, "known hosts failure: '%s': %s [%d]: check type [%s]", strZ(this->host), storageSftpKnownHostCheckpFailureMsg(rc), rc, zNewStrId(this->hostKeyCheckType)); } else storageSftpUpdateKnownHostsFile(this, hostKeyType, this->host, hostKey, hostKeyLen); break; } } } } else { THROW_FMT( ServiceError, "libssh2_session_hostkey failed to get hostkey: libssh2 error [%d]", libssh2_session_last_errno(this->session)); } // Free the known hosts list libssh2_knownhost_free(knownHostsList); } // Perform public key authorization, expand leading tilde key file paths if needed String *const privKeyPath = regExpMatchOne(STRDEF("^ *~"), this->keyPriv) ? storageSftpExpandTildePath(this->keyPriv) : strDup(this->keyPriv); String *const pubKeyPath = this->keyPub != NULL && regExpMatchOne(STRDEF("^ *~"), this->keyPub) ? storageSftpExpandTildePath(this->keyPub) : strDup(this->keyPub); do { rc = libssh2_userauth_publickey_fromfile( this->session, strZ(this->user), strZNull(pubKeyPath), strZ(privKeyPath), strZNull(this->keyPassphrase)); } while (storageSftpWaitFd(this, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "timeout during public key authentication"); storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &ServiceError, STRDEF("public key authentication failed"), STRDEF( "HINT: libssh2 compiled against non-openssl libraries requires --repo-sftp-private-key-file and" " --repo-sftp-public-key-file to be provided\n" "HINT: libssh2 versions before 1.9.0 expect a PEM format keypair, try ssh-keygen -m PEM -t rsa -P \"\" to" " generate the keypair\n" "HINT: check authorization log on the SFTP server")); } // Init the sftp session do { this->sftpSession = libssh2_sftp_init(this->session); } while (this->sftpSession == NULL && storageSftpWaitFd(this, libssh2_session_last_errno(this->session))); if (this->sftpSession == NULL) { if (libssh2_session_last_errno(this->session) == LIBSSH2_ERROR_EAGAIN) THROW_FMT(ServiceError, "timeout during init of libssh2_sftp session"); else { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(this->sftpSession), &ServiceError, strNewFmt("unable to init libssh2_sftp session"), NULL); } } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Reopen the session after the connection was lost ***********************************************************************************************************************************/ static void storageSftpReopen(StorageSftp *const this) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); LOG_DEBUG_FMT("reconnect to lost sftp connection '%s:%u'", strZ(this->host), this->port); storageSftpConnFree(this); storageSftpOpen(this); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool storageSftpReconnect(StorageSftp *const this, const int rc) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_SFTP, this); FUNCTION_TEST_PARAM(INT, rc); FUNCTION_TEST_END(); ASSERT(this != NULL); bool result = storageSftpConnLost(this, rc); if (result) storageSftpReopen(this); FUNCTION_TEST_RETURN(BOOL, result); } /**********************************************************************************************************************************/ static const StorageInterface storageInterfaceSftp = { .feature = 1 << storageFeaturePath | 1 << storageFeatureInfoDetail, .info = storageSftpInfo, .list = storageSftpList, .newRead = storageSftpNewRead, .newWrite = storageSftpNewWrite, .pathCreate = storageSftpPathCreate, .pathRemove = storageSftpPathRemove, .remove = storageSftpRemove, }; FN_EXTERN Storage * storageSftpNew( const String *const path, const String *const host, const unsigned int port, const String *const user, const TimeMSec timeout, const String *const keyPriv, const StringId hostKeyHashType, const StorageSftpNewParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(STRING, host); FUNCTION_LOG_PARAM(UINT, port); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(TIME_MSEC, timeout); FUNCTION_LOG_PARAM(STRING, keyPriv); FUNCTION_LOG_PARAM(STRING_ID, hostKeyHashType); FUNCTION_LOG_PARAM(STRING, param.keyPub); FUNCTION_TEST_PARAM(STRING, param.keyPassphrase); FUNCTION_LOG_PARAM(STRING_ID, param.hostKeyCheckType); FUNCTION_LOG_PARAM(STRING, param.hostFingerprint); FUNCTION_LOG_PARAM(STRING_LIST, param.knownHosts); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(BOOL, param.write); FUNCTION_LOG_PARAM(FUNCTIONP, param.pathExpressionFunction); FUNCTION_LOG_END(); ASSERT(path != NULL); ASSERT(host != NULL); ASSERT(port != 0); ASSERT(user != NULL); ASSERT(keyPriv != NULL); ASSERT(hostKeyHashType != 0); // Initialize user module userInit(); // Create the object OBJ_NEW_BEGIN(StorageSftp, .childQty = MEM_CONTEXT_QTY_MAX, .callbackQty = 1) { *this = (StorageSftp) { .interface = storageInterfaceSftp, .timeout = timeout, // Retain connection parameters so the session can be reopened after the server drops an idle connection .host = strDup(host), .port = port, .user = strDup(user), .keyPriv = strDup(keyPriv), .keyPub = strDup(param.keyPub), .keyPassphrase = strDup(param.keyPassphrase), .hostKeyHashType = hostKeyHashType, .hostKeyCheckType = param.hostKeyCheckType, .hostFingerprint = strDup(param.hostFingerprint), .knownHosts = strLstDup(param.knownHosts), }; // Init SFTP session if (libssh2_init(0) != 0) THROW_FMT(ServiceError, "unable to init libssh2"); this->ioClient = sckClientNew(host, port, timeout, timeout); // Open the connection (socket, handshake, host key verification, authentication, sftp session) storageSftpOpen(this); // Ensure libssh2/libssh2_sftp resources freed memContextCallbackSet(objMemContext(this), storageSftpLibSsh2SessionFreeResource, this); } OBJ_NEW_END(); FUNCTION_LOG_RETURN( STORAGE, storageNew( STORAGE_SFTP_TYPE, path, param.modeFile == 0 ? STORAGE_MODE_FILE_DEFAULT : param.modeFile, param.modePath == 0 ? STORAGE_MODE_PATH_DEFAULT : param.modePath, param.write, 0, param.pathExpressionFunction, this, this->interface)); } #endif // HAVE_LIBSSH2 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/storage.h0000664000175100017510000000510015240547630020433 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_STORAGE_H #define STORAGE_SFTP_STORAGE_H #include "storage/storage.h" /*********************************************************************************************************************************** Storage type ***********************************************************************************************************************************/ #define STORAGE_SFTP_TYPE STRID5("sftp", 0x850d30) #ifdef HAVE_LIBSSH2 /*********************************************************************************************************************************** SFTP StrictHostKeyChecking ***********************************************************************************************************************************/ #define SFTP_STRICT_HOSTKEY_CHECKING_ACCEPT_NEW STRID5("accept-new", 0x2e576e9028c610) #define SFTP_STRICT_HOSTKEY_CHECKING_FINGERPRINT STRID5("fingerprint", 0x51c9942453b9260) #define SFTP_STRICT_HOSTKEY_CHECKING_NONE STRID5("none", 0x2b9ee0) #define SFTP_STRICT_HOSTKEY_CHECKING_STRICT STRID5("strict", 0x2834ca930) /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct StorageSftpNewParam { VAR_PARAM_HEADER; bool write; mode_t modeFile; mode_t modePath; StoragePathExpressionCallback *pathExpressionFunction; const String *keyPub; const String *keyPassphrase; StringId hostKeyCheckType; const String *hostFingerprint; const StringList *knownHosts; } StorageSftpNewParam; #define storageSftpNewP(path, host, port, user, timeout, keyPriv, hostKeyHashType, ...) \ storageSftpNew(path, host, port, user, timeout, keyPriv, hostKeyHashType, (StorageSftpNewParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Storage *storageSftpNew( const String *path, const String *host, unsigned int port, const String *user, TimeMSec timeout, const String *keyPriv, StringId hostKeyHashType, const StorageSftpNewParam param); #endif // HAVE_LIBSSH2 #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/storage.intern.h0000664000175100017510000000532515240547630021742 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage Internal ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_STORAGE_INTERN_H #define STORAGE_SFTP_STORAGE_INTERN_H #include #include #include "storage/sftp/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageSftp StorageSftp; /*********************************************************************************************************************************** Getters ***********************************************************************************************************************************/ // Current libssh2 session/sftp session, used by the read/write objects FN_EXTERN LIBSSH2_SESSION *storageSftpSession(StorageSftp *this); FN_EXTERN LIBSSH2_SFTP *storageSftpSessionSftp(StorageSftp *this); /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ FN_EXTERN void storageSftpEvalLibSsh2Error( int ssh2Errno, uint64_t sftpErrno, const ErrorType *errorType, const String *msg, const String *hint); // Check which direction we are currently blocking on (reading, writing, or both) and wait for the fd to be ready accordingly FN_EXTERN bool storageSftpWaitFd(StorageSftp *this, int64_t rc); // If the error indicates the connection was lost then reopen the session so the operation can be retried. Returns true when the // session was reopened (the caller should retry the operation). FN_EXTERN bool storageSftpReconnect(StorageSftp *this, int rc); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_SFTP_TYPE \ StorageSftp * #define FUNCTION_LOG_STORAGE_SFTP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageSftp *", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/write.c0000664000175100017510000003460015240547630020123 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage File Write ***********************************************************************************************************************************/ #include #ifdef HAVE_LIBSSH2 #include "common/debug.h" #include "common/log.h" #include "common/user.h" #include "storage/sftp/write.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWriteSftp { const StorageWriteInterface *interface; // Interface const String *name; // File name mode_t modeFile; // File mode mode_t modePath; // Path mode bool createPath; // Create path if missing bool syncFile; // Sync file on close bool atomic; // Atomic write StorageSftp *storage; // Storage that created this object const String *nameTmp; // Temporary filename utilized for atomic ops const String *path; // Utilized for path operations LIBSSH2_SFTP_HANDLE *sftpHandle; // LibSsh2 session sftp handle }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static void storageWriteSftpOpen(THIS_VOID) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(storageSftpSessionSftp(this->storage) != NULL); const unsigned long int flags = LIBSSH2_FXF_CREAT | LIBSSH2_FXF_WRITE | LIBSSH2_FXF_TRUNC; // Retry the open once if the connection was lost (e.g. the server dropped an idle connection) and reopened successfully unsigned int retry = 0; int connErrno; do { // Open the file do { this->sftpHandle = libssh2_sftp_open_ex( storageSftpSessionSftp(this->storage), strZ(this->nameTmp), (unsigned int)strSize(this->nameTmp), flags, (int)this->modeFile, LIBSSH2_SFTP_OPENFILE); } while (this->sftpHandle == NULL && storageSftpWaitFd(this->storage, (connErrno = libssh2_session_last_errno(storageSftpSession(this->storage))))); // Attempt to create the path if it is missing if (this->sftpHandle == NULL && libssh2_session_last_errno(storageSftpSession(this->storage)) == LIBSSH2_ERROR_SFTP_PROTOCOL && libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)) == LIBSSH2_FX_NO_SUCH_FILE) { // Create the path storageInterfacePathCreateP(this->storage, this->path, false, false, this->modePath); // Open file again do { this->sftpHandle = libssh2_sftp_open_ex( storageSftpSessionSftp(this->storage), strZ(this->nameTmp), (unsigned int)strSize(this->nameTmp), flags, (int)this->modeFile, LIBSSH2_SFTP_OPENFILE); } while (this->sftpHandle == NULL && storageSftpWaitFd(this->storage, (connErrno = libssh2_session_last_errno(storageSftpSession(this->storage))))); } } while (this->sftpHandle == NULL && retry++ == 0 && storageSftpReconnect(this->storage, connErrno)); // Handle error if (this->sftpHandle == NULL) { const int rc = libssh2_session_last_errno(storageSftpSession(this->storage)); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileOpenError, "timeout while opening file '%s'", strZ(this->name)); const uint64_t sftpErr = libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)); if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL && sftpErr == LIBSSH2_FX_NO_SUCH_FILE) { THROW_FMT(FileMissingError, STORAGE_ERROR_WRITE_MISSING, strZ(this->name)); } else storageSftpEvalLibSsh2Error(rc, sftpErr, &FileOpenError, strNewFmt(STORAGE_ERROR_WRITE_OPEN, strZ(this->name)), NULL); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to the file ***********************************************************************************************************************************/ static void storageWriteSftp(THIS_VOID, const Buffer *const buffer) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(buffer != NULL); ASSERT(this->sftpHandle != NULL); ssize_t rc; size_t remains = bufUsed(buffer); // Amount left to write size_t offset = 0; // Offset into the buffer // Loop until all the data is written do { do { rc = libssh2_sftp_write(this->sftpHandle, (const char *)bufPtrConst(buffer) + offset, remains); } while (storageSftpWaitFd(this->storage, rc)); // Break on error. Error will be thrown below the loop. if (rc < 0) break; // Offset for next write start point offset += (size_t)rc; // Update amount left to write remains -= (size_t)rc; } while (remains); if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileWriteError, "timeout writing '%s'", strZ(this->nameTmp)); if (rc < 0) { storageSftpEvalLibSsh2Error( (int)rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileWriteError, strNewFmt("unable to write '%s'", strZ(this->nameTmp)), NULL); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Unlink already existing file ***********************************************************************************************************************************/ static void storageWriteSftpUnlinkExisting(THIS_VOID) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); int rc; do { rc = libssh2_sftp_unlink_ex(storageSftpSessionSftp(this->storage), strZ(this->name), (unsigned int)strSize(this->name)); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileWriteError, "timeout unlinking '%s'", strZ(this->name)); else storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileRemoveError, strNewFmt("unable to remove existing '%s'", strZ(this->name)), NULL); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Rename a file ***********************************************************************************************************************************/ static void storageWriteSftpRename(THIS_VOID) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); int rc; do { rc = libssh2_sftp_rename_ex( storageSftpSessionSftp(this->storage), strZ(this->nameTmp), (unsigned int)strSize(this->nameTmp), strZ(this->name), (unsigned int)strSize(this->name), LIBSSH2_SFTP_RENAME_OVERWRITE | LIBSSH2_SFTP_RENAME_ATOMIC | LIBSSH2_SFTP_RENAME_NATIVE); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileWriteError, "timeout moving '%s' to '%s'", strZ(this->nameTmp), strZ(this->name)); else storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileRemoveError, strNewFmt("unable to move '%s' to '%s'", strZ(this->nameTmp), strZ(this->name)), NULL); } FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteSftpClose(THIS_VOID) { THIS(StorageWriteSftp); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE_SFTP, this); FUNCTION_LOG_END(); ASSERT(this != NULL); // Close if the file has not already been closed if (this->sftpHandle != NULL) { int rc; if (this->syncFile) { do { rc = libssh2_sftp_fsync(this->sftpHandle); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileSyncError, "timeout syncing file '%s'", strZ(this->nameTmp)); else { storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileSyncError, strNewFmt(STORAGE_ERROR_WRITE_SYNC, strZ(this->nameTmp)), NULL); } } } // Close the file do { rc = libssh2_sftp_close(this->sftpHandle); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileCloseError, "timeout closing file '%s'", strZ(this->nameTmp)); storageSftpEvalLibSsh2Error( rc, libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)), &FileCloseError, strNewFmt(STORAGE_ERROR_WRITE_CLOSE, strZ(this->nameTmp)), NULL); } this->sftpHandle = NULL; // Rename from temp file if (this->atomic) { do { rc = libssh2_sftp_rename_ex( storageSftpSessionSftp(this->storage), strZ(this->nameTmp), (unsigned int)strSize(this->nameTmp), strZ(this->name), (unsigned int)strSize(this->name), LIBSSH2_SFTP_RENAME_OVERWRITE | LIBSSH2_SFTP_RENAME_ATOMIC | LIBSSH2_SFTP_RENAME_NATIVE); } while (storageSftpWaitFd(this->storage, rc)); if (rc != 0) { if (rc == LIBSSH2_ERROR_EAGAIN) THROW_FMT(FileCloseError, "timeout renaming file '%s'", strZ(this->nameTmp)); const uint64_t sftpErr = libssh2_sftp_last_error(storageSftpSessionSftp(this->storage)); // Some/most sftp servers will not rename over an existing file, in testing this returned LIBSSH2_FX_FAILURE if (rc == LIBSSH2_ERROR_SFTP_PROTOCOL && (sftpErr == LIBSSH2_FX_FAILURE || sftpErr == LIBSSH2_FX_FILE_ALREADY_EXISTS)) { // Remove the existing file and retry the rename storageWriteSftpUnlinkExisting(this); storageWriteSftpRename(this); } else { storageSftpEvalLibSsh2Error( rc, sftpErr, &FileCloseError, strNewFmt("unable to move '%s' to '%s'", strZ(this->nameTmp), strZ(this->name)), NULL); } } } } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const StorageWriteInterface storageWriteSftpInterface = { .close = storageWriteSftpClose, .open = storageWriteSftpOpen, .write = storageWriteSftp, }; FN_EXTERN StorageWriteSftp * storageWriteSftpNew( StorageSftp *const storage, const String *const name, const mode_t modeFile, const mode_t modePath, const String *const user, const String *const group, const time_t timeModified, const bool createPath, const bool syncFile, const bool atomic, const bool truncate) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_SFTP, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(STRING, group); FUNCTION_LOG_PARAM(TIME, timeModified); FUNCTION_LOG_PARAM(BOOL, createPath); FUNCTION_LOG_PARAM(BOOL, syncFile); FUNCTION_LOG_PARAM(BOOL, atomic); FUNCTION_LOG_PARAM(BOOL, truncate); FUNCTION_LOG_END(); ASSERT(storage != NULL); ASSERT(name != NULL); ASSERT(modeFile != 0); ASSERT(modePath != 0); OBJ_NEW_BEGIN(StorageWriteSftp, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWriteSftp) { .interface = &storageWriteSftpInterface, .name = strDup(name), .modeFile = modeFile, .modePath = modePath, .createPath = createPath, .syncFile = syncFile, .atomic = atomic, .storage = storage, .nameTmp = atomic ? strNewFmt("%s." STORAGE_FILE_TEMP_EXT, strZ(name)) : strDup(name), .path = strPath(name), }; } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE_SFTP, this); } #endif // HAVE_LIBSSH2 ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/sftp/write.h0000664000175100017510000000367015240547630020133 0ustar00runnerrunner/*********************************************************************************************************************************** SFTP Storage File Write ***********************************************************************************************************************************/ #ifndef STORAGE_SFTP_WRITE_H #define STORAGE_SFTP_WRITE_H #include "common/io/session.h" #include "storage/sftp/storage.h" #include "storage/sftp/storage.intern.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWriteSftp StorageWriteSftp; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWriteSftp *storageWriteSftpNew( StorageSftp *storage, const String *name, mode_t modeFile, mode_t modePath, const String *user, const String *group, time_t timeModified, bool createPath, bool syncFile, bool atomic, bool truncate); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_WRITE_SFTP_TYPE \ StorageWriteSftp * #define FUNCTION_LOG_STORAGE_WRITE_SFTP_FORMAT(value, buffer, bufferSize) \ objNameToLog(value, "StorageWriteSftp", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/storage.c0000664000175100017510000007552315240547630017472 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Interface ***********************************************************************************************************************************/ #include #include #include #include "common/debug.h" #include "common/io/io.h" #include "common/log.h" #include "common/memContext.h" #include "common/regExp.h" #include "common/type/list.h" #include "common/wait.h" #include "storage/storage.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct Storage { StoragePub pub; // Publicly accessible variables const String *path; mode_t modeFile; mode_t modePath; bool write; time_t targetTime; // Target repo reads by time List *cacheList; // Storage cache StoragePathExpressionCallback *pathExpressionFunction; }; // Storage list cache typedef struct StorageListCache { String *path; // Path StorageList *list; // List } StorageListCache; /**********************************************************************************************************************************/ FN_EXTERN Storage * storageNew( const StringId type, const String *const path, const mode_t modeFile, const mode_t modePath, const bool write, const time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, void *const driver, const StorageInterface interface) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STRING_ID, type); FUNCTION_LOG_PARAM(STRING, path); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(BOOL, write); FUNCTION_LOG_PARAM(TIME, targetTime); FUNCTION_LOG_PARAM(FUNCTIONP, pathExpressionFunction); FUNCTION_LOG_PARAM_P(VOID, driver); FUNCTION_LOG_PARAM(STORAGE_INTERFACE, interface); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(type != 0); ASSERT(strSize(path) >= 1 && strZ(path)[0] == '/'); ASSERT(driver != NULL); ASSERT(interface.info != NULL); ASSERT(interface.list != NULL); ASSERT(interface.newRead != NULL); ASSERT(interface.newWrite != NULL); ASSERT(interface.pathRemove != NULL); ASSERT(interface.remove != NULL); OBJ_NEW_BEGIN(Storage, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (Storage) { .pub = { .type = type, .driver = objMoveToInterface(driver, this, memContextPrior()), .interface = interface, }, .path = strDup(path), .modeFile = modeFile, .modePath = modePath, .write = write, .targetTime = targetTime, .pathExpressionFunction = pathExpressionFunction, }; // Create storage cache list if (this->targetTime != 0) this->cacheList = lstNewP(sizeof(StorageListCache), .comparator = lstComparatorStr); // If time limit is requested then versioning must be supported CHECK(AssertError, targetTime == 0 || storageFeature(this, storageFeatureVersioning), "versioning required for time limit"); // If time limit is requested then storage must be read-only CHECK(AssertError, targetTime == 0 || !write, "time limit requires read-only storage"); // If path sync feature is enabled then path feature must be enabled CHECK( AssertError, !storageFeature(this, storageFeaturePathSync) || storageFeature(this, storageFeaturePath), "path feature required"); // If hardlink feature is enabled then path feature must be enabled CHECK( AssertError, !storageFeature(this, storageFeatureHardLink) || storageFeature(this, storageFeaturePath), "path feature required"); // If symlink feature is enabled then path feature must be enabled CHECK( AssertError, !storageFeature(this, storageFeatureSymLink) || storageFeature(this, storageFeaturePath), "path feature required"); // If link features are enabled then linkCreate must be implemented CHECK( AssertError, (!storageFeature(this, storageFeatureSymLink) && !storageFeature(this, storageFeatureHardLink)) || interface.linkCreate != NULL, "linkCreate required"); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE, this); } /**********************************************************************************************************************************/ FN_EXTERN bool storageCopy(StorageRead *const source, StorageWrite *const destination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_READ, source); FUNCTION_LOG_PARAM(STORAGE_WRITE, destination); FUNCTION_LOG_END(); ASSERT(source != NULL); ASSERT(destination != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { // Open source file if (ioReadOpen(storageReadIo(source))) { // Open the destination file now that we know the source file exists and is readable ioWriteOpen(storageWriteIo(destination)); // Copy data from source to destination ioCopyP(storageReadIo(source), storageWriteIo(destination)); // Close the source and destination files ioReadClose(storageReadIo(source)); ioWriteClose(storageWriteIo(destination)); // Set result to indicate that the file was copied result = true; } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN bool storageExists(const Storage *const this, const String *const pathExp, const StorageExistsParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(TIMEMSEC, param.timeout); FUNCTION_LOG_END(); ASSERT(this != NULL); bool result = false; MEM_CONTEXT_TEMP_BEGIN() { Wait *const wait = waitNew(param.timeout); // Loop until file exists or timeout do { // storageInfoLevelBasic is required here because storageInfoLevelExists will not return the type and this function // specifically wants to test existence of a *file*, not just the existence of anything with the specified name. const StorageInfo info = storageInfoP( this, pathExp, .level = storageInfoLevelBasic, .ignoreMissing = true, .followLink = true); // Only exists if it is a file result = info.exists && info.type == storageTypeFile; } while (!result && waitMore(wait)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(BOOL, result); } /**********************************************************************************************************************************/ FN_EXTERN Buffer * storageGet(StorageRead *const file, const StorageGetParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_READ, file); FUNCTION_LOG_PARAM(SIZE, param.exactSize); FUNCTION_LOG_END(); ASSERT(file != NULL); Buffer *result = NULL; // If the file exists if (ioReadOpen(storageReadIo(file))) { MEM_CONTEXT_TEMP_BEGIN() { // If exact read if (param.exactSize > 0) { result = bufNew(param.exactSize); ioRead(storageReadIo(file), result); // If an exact read make sure the size is as expected if (bufUsed(result) != param.exactSize) THROW_FMT(FileReadError, "unable to read %zu byte(s) from '%s'", param.exactSize, strZ(storageReadName(file))); } // Else read entire file else { result = bufNew(0); Buffer *const read = bufNew(ioBufferSize()); do { // Read data ioRead(storageReadIo(file), read); // Add to result and free read buffer bufCat(result, read); bufUsedZero(read); } while (!ioReadEof(storageReadIo(file))); } // Move buffer to parent context on success bufMove(result, memContextPrior()); } MEM_CONTEXT_TEMP_END(); ioReadClose(storageReadIo(file)); } FUNCTION_LOG_RETURN(BUFFER, result); } /**********************************************************************************************************************************/ FN_EXTERN StorageInfo storageInfo(const Storage *const this, const String *const fileExp, StorageInfoParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, fileExp); FUNCTION_LOG_PARAM(ENUM, param.level); FUNCTION_LOG_PARAM(BOOL, param.ignoreMissing); FUNCTION_LOG_PARAM(BOOL, param.followLink); FUNCTION_LOG_PARAM(BOOL, param.noPathEnforce); FUNCTION_LOG_END(); FUNCTION_AUDIT_STRUCT(); ASSERT(this != NULL); ASSERT(this->pub.interface.info != NULL); StorageInfo result = {0}; MEM_CONTEXT_TEMP_BEGIN() { // Build the path const String *const file = storagePathP(this, fileExp, .noEnforce = param.noPathEnforce); // Set level when default if (param.level == storageInfoLevelDefault) param.level = storageFeature(this, storageFeatureInfoDetail) ? storageInfoLevelDetail : storageInfoLevelBasic; // If file is / then this is definitely a path so skip the call for drivers that do not support paths and do not provide // additional info to return. Also, some object stores (e.g. S3) behave strangely when getting info for /. if (strEq(file, FSLASH_STR) && !storageFeature(this, storageFeaturePath)) { result = (StorageInfo){.level = param.level}; } // Else call the driver else { // If targeting by time if (this->targetTime != 0) { // Find the version to read using the cache const String *const path = strPath(file); StorageListCache *cache = lstFind(this->cacheList, &path); if (cache == NULL) { MEM_CONTEXT_OBJ_BEGIN(this->cacheList) { StorageList *const list = storageInterfaceListP( storageDriver(this), path, storageInfoLevelBasic, .targetTime = this->targetTime); if (list != NULL) storageLstSort(list, sortOrderAsc); cache = memNew(sizeof(StorageListCache)); *cache = (StorageListCache){.path = strDup(path), .list = list}; lstAdd(this->cacheList, cache); } MEM_CONTEXT_OBJ_END(); lstSort(this->cacheList, sortOrderAsc); } if (cache->list != NULL) { result = storageLstFind(cache->list, strBase(file)); ASSERT(!result.exists || result.type != storageTypeFile || result.versionId != NULL); } } else result = storageInterfaceInfoP(storageDriver(this), file, param.level, .followLink = param.followLink); } // Error if the file missing and not ignoring if (!result.exists && !param.ignoreMissing) THROW_FMT(FileOpenError, STORAGE_ERROR_INFO_MISSING, strZ(file)); // Dup the strings into the prior context MEM_CONTEXT_PRIOR_BEGIN() { result.linkDestination = strDup(result.linkDestination); result.user = strDup(result.user); result.group = strDup(result.group); } MEM_CONTEXT_PRIOR_END(); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_INFO, result); } /**********************************************************************************************************************************/ FN_EXTERN StorageIterator * storageNewItr(const Storage *const this, const String *const pathExp, StorageNewItrParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(ENUM, param.level); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_PARAM(BOOL, param.recurse); FUNCTION_LOG_PARAM(BOOL, param.nullOnMissing); FUNCTION_LOG_PARAM(ENUM, param.sortOrder); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_PARAM(BOOL, param.recurse); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.interface.list != NULL); ASSERT(!param.errorOnMissing || storageFeature(this, storageFeaturePath)); StorageIterator *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { // Info level if (param.level == storageInfoLevelDefault) param.level = storageFeature(this, storageFeatureInfoDetail) ? storageInfoLevelDetail : storageInfoLevelBasic; result = storageItrMove( storageItrNew( storageDriver(this), storagePathP(this, pathExp), param.level, param.errorOnMissing, param.nullOnMissing, param.recurse, param.sortOrder, this->targetTime, param.expression), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_ITERATOR, result); } /**********************************************************************************************************************************/ FN_EXTERN void storageLinkCreate( const Storage *const this, const String *const target, const String *const linkPath, const StorageLinkCreateParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, target); FUNCTION_LOG_PARAM(STRING, linkPath); FUNCTION_LOG_PARAM(ENUM, param.linkType); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); ASSERT(target != NULL); ASSERT(linkPath != NULL); ASSERT(this->pub.interface.linkCreate != NULL); ASSERT( (param.linkType == storageLinkSym && storageFeature(this, storageFeatureSymLink)) || (param.linkType == storageLinkHard && storageFeature(this, storageFeatureHardLink))); MEM_CONTEXT_TEMP_BEGIN() { storageInterfaceLinkCreateP(storageDriver(this), target, linkPath, .linkType = param.linkType); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN StringList * storageList(const Storage *const this, const String *const pathExp, const StorageListParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_PARAM(BOOL, param.nullOnMissing); FUNCTION_LOG_PARAM(STRING, param.expression); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(!param.errorOnMissing || !param.nullOnMissing); StringList *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { StorageIterator *const storageItr = storageNewItrP( this, pathExp, .level = storageInfoLevelExists, .errorOnMissing = param.errorOnMissing, .nullOnMissing = param.nullOnMissing, .expression = param.expression); if (storageItr != NULL) { result = strLstNew(); while (storageItrMore(storageItr)) strLstAdd(result, storageItrNext(storageItr).name); strLstMove(result, memContextPrior()); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STRING_LIST, result); } /**********************************************************************************************************************************/ FN_EXTERN void storageMove(const Storage *const this, StorageRead *const source, StorageWrite *const destination) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_READ, source); FUNCTION_LOG_PARAM(STORAGE_WRITE, destination); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.interface.move != NULL); ASSERT(source != NULL); ASSERT(destination != NULL); ASSERT(!storageReadIgnoreMissing(source)); ASSERT(storageType(this) == storageReadType(source)); ASSERT(storageReadType(source) == storageWriteType(destination)); MEM_CONTEXT_TEMP_BEGIN() { // If the file can't be moved it will need to be copied if (!storageInterfaceMoveP(storageDriver(this), source, destination)) { // Perform the copy storageCopyP(source, destination); // Remove the source file storageInterfaceRemoveP(storageDriver(this), storageReadName(source)); // Sync source path if the destination path was synced. We know the source and destination paths are different because // the move did not succeed. This will need updating when drivers other than Posix/CIFS are implemented because there's // no way to get coverage on it now. if (storageWriteSyncPath(destination)) storageInterfacePathSyncP(storageDriver(this), strPath(storageReadName(source))); } } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN StorageRead * storageNewRead(const Storage *const this, const String *const fileExp, const StorageNewReadParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, fileExp); FUNCTION_LOG_PARAM(BOOL, param.ignoreMissing); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_PARAM(UINT64, param.offset); FUNCTION_LOG_PARAM(VARIANT, param.limit); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(param.limit == NULL || varType(param.limit) == varTypeUInt64); StorageRead *result = NULL; MEM_CONTEXT_TEMP_BEGIN() { const String *const path = storagePathP(this, fileExp); const String *versionId = NULL; // If targeting by time if (this->targetTime != 0) { versionId = storageInfoP(this, fileExp, .ignoreMissing = true).versionId; if (versionId == NULL && !param.ignoreMissing) THROW_FMT(FileMissingError, STORAGE_ERROR_READ_MISSING, strZ(path)); } result = storageReadMove( storageReadNew( this, path, param.ignoreMissing, param.compressible, param.offset, param.limit, this->targetTime != 0, versionId), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_READ, result); } /**********************************************************************************************************************************/ FN_EXTERN StorageWrite * storageNewWrite(const Storage *const this, const String *const fileExp, const StorageNewWriteParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, fileExp); FUNCTION_LOG_PARAM(MODE, param.modeFile); FUNCTION_LOG_PARAM(MODE, param.modePath); FUNCTION_LOG_PARAM(STRING, param.user); FUNCTION_LOG_PARAM(STRING, param.group); FUNCTION_LOG_PARAM(INT64, param.timeModified); FUNCTION_LOG_PARAM(BOOL, param.noCreatePath); FUNCTION_LOG_PARAM(BOOL, param.noSyncFile); FUNCTION_LOG_PARAM(BOOL, param.noSyncPath); FUNCTION_LOG_PARAM(BOOL, param.noAtomic); FUNCTION_LOG_PARAM(BOOL, param.noTruncate); FUNCTION_LOG_PARAM(BOOL, param.compressible); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); // noTruncate does not work with atomic writes because a new file is always created for atomic writes ASSERT(!param.noTruncate || param.noAtomic); StorageWrite *result; MEM_CONTEXT_TEMP_BEGIN() { result = storageWriteMove( storageWriteNew( this, storagePathP(this, fileExp), param.modeFile != 0 ? param.modeFile : this->modeFile, param.modePath != 0 ? param.modePath : this->modePath, param.user, param.group, param.timeModified, !param.noCreatePath, !param.noSyncFile, !param.noSyncPath, !param.noAtomic, !param.noTruncate, param.compressible), memContextPrior()); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE, result); } /**********************************************************************************************************************************/ FN_EXTERN String * storagePath(const Storage *const this, const String *pathExp, const StoragePathParam param) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE, this); FUNCTION_TEST_PARAM(STRING, pathExp); FUNCTION_TEST_PARAM(BOOL, param.noEnforce); FUNCTION_TEST_END(); ASSERT(this != NULL); String *result; // If there is no path expression then return the base storage path if (pathExp == NULL) { result = strDup(this->path); } else { // If the path expression is absolute then use it as is if ((strZ(pathExp))[0] == '/') { // Make sure the base storage path is contained within the path expression if (!strEqZ(this->path, "/")) { if (!param.noEnforce && (!strBeginsWith(pathExp, this->path) || !(strSize(pathExp) == strSize(this->path) || *(strZ(pathExp) + strSize(this->path)) == '/'))) { THROW_FMT(AssertError, "absolute path '%s' is not in base path '%s'", strZ(pathExp), strZ(this->path)); } } result = strDup(pathExp); } // Else path expression is relative so combine it with the base storage path else { // There may or may not be a path expression that needs to be evaluated String *pathEvaluated = NULL; // Check if there is a path expression that needs to be evaluated if ((strZ(pathExp))[0] == '<') { if (this->pathExpressionFunction == NULL) THROW_FMT(AssertError, "expression '%s' not valid without callback function", strZ(pathExp)); // Get position of the expression end const char *const end = strchr(strZ(pathExp), '>'); // Error if end is not found if (end == NULL) THROW_FMT(AssertError, "end > not found in path expression '%s'", strZ(pathExp)); // Create a string from the expression String *const expression = strNewZN(strZ(pathExp), (size_t)(end - strZ(pathExp) + 1)); // Create a string from the path if there is anything left after the expression String *path = NULL; if (strSize(expression) < strSize(pathExp)) { // Error if path separator is not found if (end[1] != '/') THROW_FMT(AssertError, "'/' should separate expression and path '%s'", strZ(pathExp)); // Only create path if there is something after the path separator if (end[2] == 0) THROW_FMT(AssertError, "path '%s' should not end in '/'", strZ(pathExp)); path = strNewZ(end + 2); } // Evaluate the path pathEvaluated = this->pathExpressionFunction(expression, path); // Evaluated path cannot be NULL if (pathEvaluated == NULL) THROW_FMT(AssertError, "evaluated path '%s' cannot be null", strZ(pathExp)); // Assign evaluated path to path pathExp = pathEvaluated; // Free temp vars strFree(expression); strFree(path); } if (strEqZ(this->path, "/")) result = strNewFmt("/%s", strZ(pathExp)); else result = strNewFmt("%s/%s", strZ(this->path), strZ(pathExp)); strFree(pathEvaluated); } } FUNCTION_TEST_RETURN(STRING, result); } /**********************************************************************************************************************************/ FN_EXTERN void storagePathCreate(const Storage *const this, const String *const pathExp, const StoragePathCreateParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(BOOL, param.errorOnExists); FUNCTION_LOG_PARAM(BOOL, param.noParentCreate); FUNCTION_LOG_PARAM(MODE, param.mode); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->pub.interface.pathCreate != NULL && storageFeature(this, storageFeaturePath)); ASSERT(this->write); MEM_CONTEXT_TEMP_BEGIN() { // Call driver function storageInterfacePathCreateP( storageDriver(this), storagePathP(this, pathExp), param.errorOnExists, param.noParentCreate, param.mode != 0 ? param.mode : this->modePath); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN bool storagePathExists(const Storage *const this, const String *const pathExp) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(storageFeature(this, storageFeaturePath)); // storageInfoLevelBasic is required here because storageInfoLevelExists will not return the type and this function specifically // wants to test existence of a *path*, not just the existence of anything with the specified name. const StorageInfo info = storageInfoP(this, pathExp, .level = storageInfoLevelBasic, .ignoreMissing = true, .followLink = true); FUNCTION_LOG_RETURN(BOOL, info.exists && info.type == storageTypePath); } /**********************************************************************************************************************************/ FN_EXTERN void storagePathRemove(const Storage *const this, const String *const pathExp, const StoragePathRemoveParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_PARAM(BOOL, param.recurse); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); ASSERT(!param.errorOnMissing || storageFeature(this, storageFeaturePath)); ASSERT(param.recurse || storageFeature(this, storageFeaturePath)); MEM_CONTEXT_TEMP_BEGIN() { // Build the path const String *const path = storagePathP(this, pathExp); // Call driver function if (!storageInterfacePathRemoveP(storageDriver(this), path, param.recurse) && param.errorOnMissing) THROW_FMT(PathRemoveError, STORAGE_ERROR_PATH_REMOVE_MISSING, strZ(path)); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storagePathSync(const Storage *const this, const String *const pathExp) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, pathExp); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); // Not all storage requires path sync so just do nothing if the function is not implemented if (this->pub.interface.pathSync != NULL) { MEM_CONTEXT_TEMP_BEGIN() { storageInterfacePathSyncP(storageDriver(this), storagePathP(this, pathExp)); } MEM_CONTEXT_TEMP_END(); } FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storagePut(StorageWrite *const file, const Buffer *const buffer) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE_WRITE, file); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(file != NULL); ioWriteOpen(storageWriteIo(file)); ioWrite(storageWriteIo(file), buffer); ioWriteClose(storageWriteIo(file)); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storageRemove(const Storage *const this, const String *const fileExp, const StorageRemoveParam param) { FUNCTION_LOG_BEGIN(logLevelDebug); FUNCTION_LOG_PARAM(STORAGE, this); FUNCTION_LOG_PARAM(STRING, fileExp); FUNCTION_LOG_PARAM(BOOL, param.errorOnMissing); FUNCTION_LOG_END(); ASSERT(this != NULL); ASSERT(this->write); ASSERT(!param.errorOnMissing || storageFeature(this, storageFeatureFileRemoveMissing)); MEM_CONTEXT_TEMP_BEGIN() { // Call driver function storageInterfaceRemoveP(storageDriver(this), storagePathP(this, fileExp), .errorOnMissing = param.errorOnMissing); } MEM_CONTEXT_TEMP_END(); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storageToLog(const Storage *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "), strStcResultSizeInc(debugLog, strIdToLog(storageType(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt(debugLog, ", path: %s, write: %s}", strZ(this->path), cvtBoolToConstZ(this->write)); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/storage.h0000664000175100017510000003066315240547630017473 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Interface ***********************************************************************************************************************************/ #ifndef STORAGE_STORAGE_H #define STORAGE_STORAGE_H #include /*********************************************************************************************************************************** Storage link type ***********************************************************************************************************************************/ typedef enum { // Symbolic (or soft) link storageLinkSym, // Hard link storageLinkHard, } StorageLinkType; /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct Storage Storage; #include "common/io/filter/group.h" #include "common/time.h" #include "common/type/buffer.h" #include "common/type/param.h" #include "common/type/stringList.h" #include "storage/info.h" #include "storage/iterator.h" #include "storage/read.h" #include "storage/storage.intern.h" #include "storage/write.h" /*********************************************************************************************************************************** Storage feature ***********************************************************************************************************************************/ typedef enum { // Does the storage support paths/directories as something that needs to be created and deleted? Object stores (e.g. S3) often // do not have paths/directories -- they are only inferred by the object name. Therefore it doesn't make sense to create or // remove directories since this implies something is happening on the storage and in the case of objects stores it would be a // noop. We'll error on any path operation (e.g. pathExists(), pathCreate(), non-recursive removes, error on missing paths, // etc.) for storage that does not support paths. storageFeaturePath, // Do paths need to be synced to ensure contents are durable? storageFeaturePath must also be enabled. storageFeaturePathSync, // Does the storage support hardlinks? Hardlinks allow the same file to be linked into multiple paths to save space. storageFeatureHardLink, // Does the storage support symlinks? Symlinks allow paths/files/links to be accessed from another path. storageFeatureSymLink, // Does the storage support detailed info, i.e. user, group, mode, link destination, etc. storageFeatureInfoDetail, // Does the storage support versioning? storageFeatureVersioning, // Can reads be retried? storageFeatureReadRetry, // Can files be detected as missing on remove? storageFeatureFileRemoveMissing, } StorageFeature; /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Copy a file #define storageCopyP(source, destination) \ storageCopy(source, destination) FN_EXTERN bool storageCopy(StorageRead *source, StorageWrite *destination); // Does a file exist? This function is only for files, not paths. typedef struct StorageExistsParam { VAR_PARAM_HEADER; TimeMSec timeout; } StorageExistsParam; #define storageExistsP(this, pathExp, ...) \ storageExists(this, pathExp, (StorageExistsParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN bool storageExists(const Storage *this, const String *pathExp, StorageExistsParam param); // Read from storage into a buffer typedef struct StorageGetParam { VAR_PARAM_HEADER; size_t exactSize; } StorageGetParam; #define storageGetP(file, ...) \ storageGet(file, (StorageGetParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN Buffer *storageGet(StorageRead *file, StorageGetParam param); // File/path info typedef struct StorageInfoParam { VAR_PARAM_HEADER; StorageInfoLevel level; bool ignoreMissing; bool followLink; bool noPathEnforce; } StorageInfoParam; #define storageInfoP(this, fileExp, ...) \ storageInfo(this, fileExp, (StorageInfoParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StorageInfo storageInfo(const Storage *this, const String *fileExp, StorageInfoParam param); // Iterator for all files/links/paths in a path which returns different info based on the value of the level parameter typedef struct StorageNewItrParam { VAR_PARAM_HEADER; StorageInfoLevel level; bool errorOnMissing; bool nullOnMissing; bool recurse; SortOrder sortOrder; const String *expression; } StorageNewItrParam; #define storageNewItrP(this, fileExp, ...) \ storageNewItr(this, fileExp, (StorageNewItrParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StorageIterator *storageNewItr(const Storage *this, const String *pathExp, StorageNewItrParam param); // Get a list of files from a directory typedef struct StorageListParam { VAR_PARAM_HEADER; bool errorOnMissing; bool nullOnMissing; const String *expression; } StorageListParam; #define storageListP(this, pathExp, ...) \ storageList(this, pathExp, (StorageListParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StringList *storageList(const Storage *this, const String *pathExp, StorageListParam param); // Move a file #define storageMoveP(this, source, destination) \ storageMove(this, source, destination) FN_EXTERN void storageMove(const Storage *this, StorageRead *source, StorageWrite *destination); // Open a file for reading typedef struct StorageNewReadParam { VAR_PARAM_HEADER; bool ignoreMissing; bool compressible; // Where to start reading in the file const uint64_t offset; // Limit bytes to read from the file (must be varTypeUInt64). NULL for no limit. const Variant *limit; } StorageNewReadParam; #define storageNewReadP(this, pathExp, ...) \ storageNewRead(this, pathExp, (StorageNewReadParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StorageRead *storageNewRead(const Storage *this, const String *fileExp, StorageNewReadParam param); // Open a file for writing typedef struct StorageNewWriteParam { VAR_PARAM_HEADER; bool noCreatePath; bool noSyncFile; bool noSyncPath; bool noAtomic; // Do not truncate file if it exists. Use this only in cases where the file will be manipulated directly through the file // handle, which should always be the exception and indicates functionality that should be added to the storage interface. bool noTruncate; bool compressible; mode_t modeFile; mode_t modePath; time_t timeModified; const String *user; const String *group; } StorageNewWriteParam; #define storageNewWriteP(this, pathExp, ...) \ storageNewWrite(this, pathExp, (StorageNewWriteParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN StorageWrite *storageNewWrite(const Storage *this, const String *fileExp, StorageNewWriteParam param); // Get absolute path in the storage typedef struct StoragePathParam { VAR_PARAM_HEADER; bool noEnforce; } StoragePathParam; #define storagePathP(this, pathExp, ...) \ storagePath(this, pathExp, (StoragePathParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN String *storagePath(const Storage *this, const String *pathExp, StoragePathParam param); // Create a path typedef struct StoragePathCreateParam { VAR_PARAM_HEADER; bool errorOnExists; bool noParentCreate; mode_t mode; } StoragePathCreateParam; #define storagePathCreateP(this, pathExp, ...) \ storagePathCreate(this, pathExp, (StoragePathCreateParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void storagePathCreate(const Storage *this, const String *pathExp, StoragePathCreateParam param); // Does a path exist? #define storagePathExistsP(this, pathExp) \ storagePathExists(this, pathExp) FN_EXTERN bool storagePathExists(const Storage *this, const String *pathExp); // Remove a path typedef struct StoragePathRemoveParam { VAR_PARAM_HEADER; bool errorOnMissing; bool recurse; } StoragePathRemoveParam; #define storagePathRemoveP(this, pathExp, ...) \ storagePathRemove(this, pathExp, (StoragePathRemoveParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void storagePathRemove(const Storage *this, const String *pathExp, StoragePathRemoveParam param); // Sync a path #define storagePathSyncP(this, pathExp) \ storagePathSync(this, pathExp) FN_EXTERN void storagePathSync(const Storage *this, const String *pathExp); // Write a buffer to storage #define storagePutP(file, buffer) \ storagePut(file, buffer) FN_EXTERN void storagePut(StorageWrite *file, const Buffer *buffer); // Remove a file typedef struct StorageRemoveParam { VAR_PARAM_HEADER; bool errorOnMissing; } StorageRemoveParam; #define storageRemoveP(this, fileExp, ...) \ storageRemove(this, fileExp, (StorageRemoveParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void storageRemove(const Storage *this, const String *fileExp, StorageRemoveParam param); // Create a hard or symbolic link typedef struct StorageLinkCreateParam { VAR_PARAM_HEADER; // Flag to create hard or symbolic link StorageLinkType linkType; } StorageLinkCreateParam; #define storageLinkCreateP(this, target, linkPath, ...) \ storageLinkCreate(this, target, linkPath, (StorageLinkCreateParam){VAR_PARAM_INIT, __VA_ARGS__}) FN_EXTERN void storageLinkCreate(const Storage *this, const String *target, const String *linkPath, StorageLinkCreateParam param); /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ // Is the feature supported by this storage? FN_INLINE_ALWAYS bool storageFeature(const Storage *const this, const StorageFeature feature) { return THIS_PUB(Storage)->interface.feature >> feature & 1; } // Storage type (posix, cifs, etc.) FN_INLINE_ALWAYS StringId storageType(const Storage *const this) { return THIS_PUB(Storage)->type; } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageToLog(const Storage *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_TYPE \ Storage * #define FUNCTION_LOG_STORAGE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/storage.intern.h0000664000175100017510000004756515240547630021002 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Interface Internal Storage drivers are implemented using this interface. The interface has required and optional functions. Currently the optional functions are only implemented by the Posix driver which can store either a repository or a PostgreSQL cluster. Drivers that are intended to store repositories only need to implement the required functions. The behavior of required functions is further modified by storage features defined by the StorageFeature enum. Details are included in the description of each function. ***********************************************************************************************************************************/ #ifndef STORAGE_STORAGE_INTERN_H #define STORAGE_STORAGE_INTERN_H #include "common/type/param.h" #include "storage/info.h" #include "storage/list.h" #include "storage/read.h" #include "storage/write.h" /*********************************************************************************************************************************** Default file and path modes ***********************************************************************************************************************************/ #define STORAGE_MODE_FILE_DEFAULT 0640 #define STORAGE_MODE_PATH_DEFAULT 0750 /*********************************************************************************************************************************** Error messages ***********************************************************************************************************************************/ #define STORAGE_ERROR_READ_CLOSE "unable to close file '%s' after read" #define STORAGE_ERROR_READ_OPEN "unable to open file '%s' for read" #define STORAGE_ERROR_READ_MISSING "unable to open missing file '%s' for read" #define STORAGE_ERROR_READ_SEEK "unable to seek to %" PRIu64 " in file '%s'" #define STORAGE_ERROR_INFO "unable to get info for path/file '%s'" #define STORAGE_ERROR_INFO_MISSING "unable to get info for missing path/file '%s'" #define STORAGE_ERROR_LIST_INFO "unable to list file info for path '%s'" #define STORAGE_ERROR_LIST_INFO_MISSING "unable to list file info for missing path '%s'" #define STORAGE_ERROR_FILE_REMOVE "unable to remove file '%s'" #define STORAGE_ERROR_PATH_REMOVE "unable to remove path '%s'" #define STORAGE_ERROR_PATH_REMOVE_MISSING "unable to remove missing path '%s'" #define STORAGE_ERROR_PATH_SYNC "unable to sync path '%s'" #define STORAGE_ERROR_PATH_SYNC_CLOSE "unable to close path '%s' after sync" #define STORAGE_ERROR_PATH_SYNC_OPEN "unable to open path '%s' for sync" #define STORAGE_ERROR_PATH_SYNC_MISSING "unable to sync missing path '%s'" #define STORAGE_ERROR_WRITE_CLOSE "unable to close file '%s' after write" #define STORAGE_ERROR_WRITE_OPEN "unable to open file '%s' for write" #define STORAGE_ERROR_WRITE_MISSING "unable to open file '%s' for write in missing path" #define STORAGE_ERROR_WRITE_SEEK "unable to seek to %" PRIu64 " in file '%s'" #define STORAGE_ERROR_WRITE_SYNC "unable to sync file '%s' after write" /*********************************************************************************************************************************** Path expression callback function type - used to modify paths based on expressions enclosed in <> ***********************************************************************************************************************************/ typedef String *StoragePathExpressionCallback(const String *expression, const String *path); /*********************************************************************************************************************************** Storage info callback function type - used to return storage info ***********************************************************************************************************************************/ typedef void (*StorageListCallback)(void *callbackData, const StorageInfo *info); /*********************************************************************************************************************************** Required interface functions ***********************************************************************************************************************************/ // Get information about a file/link/path // // The level parameter controls the amount of information that will be returned. See the StorageInfo type and StorageInfoLevel enum // for details about information that must be provided at each level. The driver should only return the amount of information // requested even if more is available. All drivers must implement the storageInfoLevelExists and storageInfoLevelBasic levels. Only // drivers with the storageFeatureInfoDetail feature must implement the storageInfoLevelDetail level. typedef struct StorageInterfaceInfoParam { VAR_PARAM_HEADER; // Should symlinks be followed? Only required on storage that supports symlinks. bool followLink; } StorageInterfaceInfoParam; typedef StorageInfo StorageInterfaceInfo( void *thisVoid, const String *file, StorageInfoLevel level, StorageInterfaceInfoParam param); #define storageInterfaceInfoP(thisVoid, file, level, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).info(thisVoid, file, level, (StorageInterfaceInfoParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Create a hard or symbolic link typedef struct StorageInterfaceLinkCreateParam { VAR_PARAM_HEADER; // Flag to create hard or symbolic link StorageLinkType linkType; } StorageInterfaceLinkCreateParam; typedef void StorageInterfaceLinkCreate( void *thisVoid, const String *target, const String *linkPath, StorageInterfaceLinkCreateParam param); #define storageInterfaceLinkCreateP(thisVoid, target, linkPath, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).linkCreate(thisVoid, target, linkPath, \ (StorageInterfaceLinkCreateParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Create a file read object. The file should not be opened immediately -- open() will be called on the IoRead interface when the // file needs to be opened. typedef struct StorageInterfaceNewReadParam { VAR_PARAM_HEADER; // Is the file compressible? This is used when the file must be moved across a network and temporary compression is helpful. bool compressible; // Where to start reading in the file const uint64_t offset; // Limit bytes read from the file. NULL for no limit. const Variant *limit; // Target a specific file version (NULL for current version) const String *versionId; } StorageInterfaceNewReadParam; typedef void *StorageInterfaceNewRead(void *thisVoid, const String *file, StorageInterfaceNewReadParam param); #define storageInterfaceNewReadP(thisVoid, file, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).newRead(thisVoid, file, (StorageInterfaceNewReadParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Create a file write object. The file should not be opened immediately -- open() will be called on the IoWrite interface when the // file needs to be opened. typedef struct StorageInterfaceNewWriteParam { VAR_PARAM_HEADER; // File/path mode for storage that supports Posix-style permissions. modePath is only used in conjunction with createPath. mode_t modeFile; mode_t modePath; // User/group name const String *user; const String *group; // Modified time time_t timeModified; // Will paths be created as needed? bool createPath; // Sync file/path when required by the storage bool syncFile; bool syncPath; // Ensure the file is written atomically. If this is false it's OK to write atomically if that's all the storage supports // (e.g. S3). Non-atomic writes are used in some places where there is a performance advantage and atomicity is not needed. bool atomic; // Truncate file if it exists. Disable this only in cases where the file will be manipulated directly through the file handle, // which should always be the exception and shows functionality that should be added to the storage interface. bool truncate; // Is the file compressible? This is used when the file must be moved across a network and temporary compression is helpful. bool compressible; } StorageInterfaceNewWriteParam; typedef void *StorageInterfaceNewWrite(void *thisVoid, const String *file, StorageInterfaceNewWriteParam param); #define storageInterfaceNewWriteP(thisVoid, file, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).newWrite(thisVoid, file, (StorageInterfaceNewWriteParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Get info for all files/links/paths in a path (does not recurse or return info about the path passed to the function) // // See storageInterfaceInfoP() for usage of the level parameter. typedef struct StorageInterfaceListParam { VAR_PARAM_HEADER; // Regular expression used to filter the results. The expression is always checked in the callback passed to // storageInterfaceIterP() so checking the expression in the driver is entirely optional. The driver should only use the // expression if it can improve performance or limit network transfer. // // Partial matching of the expression is fine as long as nothing that should match is excluded, e.g. it is OK to prefix match // using the prefix returned from regExpPrefix(). This may cause extra results to be sent to the callback but won't exclude // anything that matches the expression exactly. const String *expression; // Target max version <= the specified time time_t targetTime; } StorageInterfaceListParam; typedef StorageList *StorageInterfaceList( void *thisVoid, const String *path, StorageInfoLevel level, StorageInterfaceListParam param); #define storageInterfaceListP(thisVoid, path, level, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).list( \ thisVoid, path, level, (StorageInterfaceListParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Remove a path (and optionally recurse) typedef struct StorageInterfacePathRemoveParam { VAR_PARAM_HEADER; } StorageInterfacePathRemoveParam; typedef bool StorageInterfacePathRemove(void *thisVoid, const String *path, bool recurse, StorageInterfacePathRemoveParam param); #define storageInterfacePathRemoveP(thisVoid, path, recurse, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).pathRemove( \ thisVoid, path, recurse, (StorageInterfacePathRemoveParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Remove a file typedef struct StorageInterfaceRemoveParam { VAR_PARAM_HEADER; // Error when the file to delete is missing bool errorOnMissing; } StorageInterfaceRemoveParam; typedef void StorageInterfaceRemove(void *thisVoid, const String *file, StorageInterfaceRemoveParam param); #define storageInterfaceRemoveP(thisVoid, file, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).remove(thisVoid, file, (StorageInterfaceRemoveParam){VAR_PARAM_INIT, __VA_ARGS__}) /*********************************************************************************************************************************** Optional interface functions ***********************************************************************************************************************************/ // Move a path/file atomically typedef struct StorageInterfaceMoveParam { VAR_PARAM_HEADER; } StorageInterfaceMoveParam; typedef bool StorageInterfaceMove(void *thisVoid, StorageRead *source, StorageWrite *destination, StorageInterfaceMoveParam param); #define storageInterfaceMoveP(thisVoid, source, destination, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).move( \ thisVoid, source, destination, (StorageInterfaceMoveParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Create a path typedef struct StorageInterfacePathCreateParam { VAR_PARAM_HEADER; } StorageInterfacePathCreateParam; typedef void StorageInterfacePathCreate( void *thisVoid, const String *path, bool errorOnExists, bool noParentCreate, mode_t mode, StorageInterfacePathCreateParam param); #define storageInterfacePathCreateP(thisVoid, path, errorOnExists, noParentCreate, mode, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).pathCreate( \ thisVoid, path, errorOnExists, noParentCreate, mode, (StorageInterfacePathCreateParam){VAR_PARAM_INIT, __VA_ARGS__}) // --------------------------------------------------------------------------------------------------------------------------------- // Sync a path typedef struct StorageInterfacePathSyncParam { VAR_PARAM_HEADER; } StorageInterfacePathSyncParam; typedef void StorageInterfacePathSync(void *thisVoid, const String *path, StorageInterfacePathSyncParam param); #define storageInterfacePathSyncP(thisVoid, path, ...) \ STORAGE_COMMON_INTERFACE(thisVoid).pathSync(thisVoid, path, (StorageInterfacePathSyncParam){VAR_PARAM_INIT, __VA_ARGS__}) /*********************************************************************************************************************************** Storage type and helper function struct An array of this struct must be passed to storageHelperInit() to enable storage drivers other than built-in Posix. ***********************************************************************************************************************************/ typedef Storage *(*StorageHelperFunction)(unsigned int repoIdx, bool write, StoragePathExpressionCallback pathExpressionCallback); typedef struct StorageHelper { StringId type; // StringId that identifies the storage driver StorageHelperFunction helper; // Function that creates the storage } StorageHelper; #define STORAGE_END_HELPER {.type = 0} /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ typedef struct StorageInterface { // Features implemented by the storage driver uint64_t feature; // Required functions StorageInterfaceInfo *info; StorageInterfaceList *list; StorageInterfaceNewRead *newRead; StorageInterfaceNewWrite *newWrite; StorageInterfacePathRemove *pathRemove; StorageInterfaceRemove *remove; // Optional functions StorageInterfaceLinkCreate *linkCreate; StorageInterfaceMove *move; StorageInterfacePathCreate *pathCreate; StorageInterfacePathSync *pathSync; } StorageInterface; #define storageNewP(type, path, modeFile, modePath, write, pathExpressionFunction, driver, ...) \ storageNew(type, path, modeFile, modePath, write, targetTime, pathExpressionFunction, driver, (StorageInterface){__VA_ARGS__}) FN_EXTERN Storage *storageNew( StringId type, const String *path, mode_t modeFile, mode_t modePath, bool write, time_t targetTime, StoragePathExpressionCallback pathExpressionFunction, void *driver, StorageInterface interface); /*********************************************************************************************************************************** Common members to include in every storage driver and macros to extract the common elements ***********************************************************************************************************************************/ #define STORAGE_COMMON_MEMBER \ StorageInterface interface /* Storage interface */ typedef struct StorageCommon { STORAGE_COMMON_MEMBER; } StorageCommon; #define STORAGE_COMMON(thisVoid) \ ((const StorageCommon *)thisVoid) #define STORAGE_COMMON_INTERFACE(thisVoid) \ (STORAGE_COMMON(thisVoid)->interface) /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct StoragePub { StringId type; // Storage type void *driver; // Storage driver StorageInterface interface; // Storage interface } StoragePub; // Storage driver FN_INLINE_ALWAYS void * storageDriver(const Storage *const this) { return THIS_PUB(Storage)->driver; } // Storage interface FN_INLINE_ALWAYS StorageInterface storageInterface(const Storage *const this) { return THIS_PUB(Storage)->interface; } /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ #define FUNCTION_LOG_STORAGE_INTERFACE_TYPE \ StorageInterface #define FUNCTION_LOG_STORAGE_INTERFACE_FORMAT(value, buffer, bufferSize) \ objNameToLog(&value, "StorageInterface", buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/write.c0000664000175100017510000003074215240547630017152 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Write Interface ***********************************************************************************************************************************/ #include #include "common/debug.h" #include "common/log.h" #include "common/memContext.h" #include "storage/write.h" /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ struct StorageWrite { StorageWritePub pub; // Publicly accessible variables void *driver; // Driver }; /*********************************************************************************************************************************** Open the file ***********************************************************************************************************************************/ static void storageWriteOpen(THIS_VOID) { THIS(StorageWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); storageWriteDriverInterface(this->driver)->open(this->driver); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Write to a file ***********************************************************************************************************************************/ static void storageWrite(THIS_VOID, const Buffer *const buffer) { THIS(StorageWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE, this); FUNCTION_LOG_PARAM(BUFFER, buffer); FUNCTION_LOG_END(); ASSERT(this != NULL); storageWriteDriverInterface(this->driver)->write(this->driver, buffer); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Close the file ***********************************************************************************************************************************/ static void storageWriteClose(THIS_VOID) { THIS(StorageWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE, this); FUNCTION_LOG_END(); ASSERT(this != NULL); storageWriteDriverInterface(this->driver)->close(this->driver); FUNCTION_LOG_RETURN_VOID(); } /*********************************************************************************************************************************** Get file descriptor ***********************************************************************************************************************************/ static int storageWriteFd(const THIS_VOID) { THIS(const StorageWrite); FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(STORAGE_WRITE, this); FUNCTION_TEST_END(); ASSERT(this != NULL); FUNCTION_TEST_RETURN(INT, storageWriteDriverInterface(this->driver)->fd(this->driver)); } /*********************************************************************************************************************************** Seek in file ***********************************************************************************************************************************/ static void storageWriteSeek(THIS_VOID, const uint64_t position) { THIS(StorageWrite); FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE_WRITE, this); FUNCTION_LOG_PARAM(UINT64, position); FUNCTION_LOG_END(); ASSERT(this != NULL); storageWriteDriverInterface(this->driver)->seek(this->driver, position); FUNCTION_LOG_RETURN_VOID(); } /**********************************************************************************************************************************/ static const IoWriteInterface storageIoWriteInterface = { .close = storageWriteClose, .fd = storageWriteFd, .open = storageWriteOpen, .seek = storageWriteSeek, .write = storageWrite, }; FN_EXTERN StorageWrite * storageWriteNew( const Storage *const storage, const String *const name, const mode_t modeFile, const mode_t modePath, const String *const user, const String *const group, const time_t timeModified, const bool createPath, const bool syncFile, const bool syncPath, const bool atomic, const bool truncate, const bool compressible) { FUNCTION_LOG_BEGIN(logLevelTrace); FUNCTION_LOG_PARAM(STORAGE, storage); FUNCTION_LOG_PARAM(STRING, name); FUNCTION_LOG_PARAM(MODE, modeFile); FUNCTION_LOG_PARAM(MODE, modePath); FUNCTION_LOG_PARAM(STRING, user); FUNCTION_LOG_PARAM(STRING, group); FUNCTION_LOG_PARAM(TIME, timeModified); FUNCTION_LOG_PARAM(BOOL, createPath); FUNCTION_LOG_PARAM(BOOL, syncFile); FUNCTION_LOG_PARAM(BOOL, syncPath); FUNCTION_LOG_PARAM(BOOL, atomic); FUNCTION_LOG_PARAM(BOOL, truncate); FUNCTION_LOG_PARAM(BOOL, compressible); FUNCTION_LOG_END(); FUNCTION_AUDIT_HELPER(); ASSERT(storage != NULL); ASSERT(name != NULL); OBJ_NEW_BEGIN(StorageWrite, .childQty = MEM_CONTEXT_QTY_MAX) { *this = (StorageWrite) { .driver = storageInterfaceNewWriteP( storageDriver(storage), name, .modeFile = modeFile, .modePath = modePath, .user = user, .group = group, .timeModified = timeModified, .createPath = createPath, .syncFile = syncFile, .syncPath = syncPath, .atomic = atomic, .truncate = truncate, .compressible = compressible), .pub = { .type = storageType(storage), .name = strDup(name), .createPath = createPath, .atomic = atomic, .truncate = truncate, .syncPath = syncPath, .syncFile = syncFile, .user = strDup(user), .group = strDup(group), .modePath = modePath, .modeFile = modeFile, .timeModified = timeModified, }, }; ASSERT(storageWriteDriverInterface(this->driver)->close != NULL); ASSERT(storageWriteDriverInterface(this->driver)->write != NULL); // Remove fd method if it does not exist in the driver IoWriteInterface storageIoWriteInterfaceCopy = storageIoWriteInterface; if (storageWriteDriverInterface(this->driver)->fd == NULL) storageIoWriteInterfaceCopy.fd = NULL; // Remove open method if it does not exist in the driver (cloud drivers don't have open) if (storageWriteDriverInterface(this->driver)->open == NULL) storageIoWriteInterfaceCopy.open = NULL; // Remove seek method if it does not exist in the driver if (storageWriteDriverInterface(this->driver)->seek == NULL) storageIoWriteInterfaceCopy.seek = NULL; this->pub.io = ioWriteNew(this, storageIoWriteInterfaceCopy); // Set filter group when interface function exists if (storageWriteDriverInterface(this->driver)->filterGroup != NULL) storageWriteDriverInterface(this->driver)->filterGroup(this->driver, ioWriteFilterGroup(storageWriteIo(this))); } OBJ_NEW_END(); FUNCTION_LOG_RETURN(STORAGE_WRITE, this); } /**********************************************************************************************************************************/ #define STORAGE_CHUNK_SIZE_MAX (size_t)(1024 * 1024 * 1024) #define STORAGE_CHUNK_INCR (size_t)(8 * 1024 * 1024) FN_EXTERN size_t storageWriteChunkSize( const size_t chunkSizeDefault, const unsigned int splitDefault, const unsigned int splitMax, const unsigned int chunkIdx) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(SIZE, chunkSizeDefault); FUNCTION_TEST_PARAM(UINT, splitDefault); FUNCTION_TEST_PARAM(UINT, splitMax); FUNCTION_TEST_PARAM(UINT, chunkIdx); FUNCTION_TEST_END(); ASSERT(chunkSizeDefault > 0); ASSERT(STORAGE_CHUNK_SIZE_MAX >= chunkSizeDefault); ASSERT(STORAGE_CHUNK_INCR > 0); ASSERT(splitMax > splitDefault); // If below the default split then return default chunk size if (chunkIdx < splitDefault) { FUNCTION_TEST_RETURN(SIZE, chunkSizeDefault); } // Else if above max split then return max chunk size else if (chunkIdx > splitMax) FUNCTION_TEST_RETURN(SIZE, STORAGE_CHUNK_SIZE_MAX); // Calculate ascending chunk size uint64_t result = (uint64_t)(STORAGE_CHUNK_SIZE_MAX - STORAGE_CHUNK_INCR) * (chunkIdx - splitDefault + 1) / (splitMax - splitDefault); // If ascending chunk size is less than default then return default if (result <= chunkSizeDefault) { result = chunkSizeDefault; } // Else if ascending chunk size is not evenly divisible by chunk increment then round up else if (result % STORAGE_CHUNK_INCR != 0) result += STORAGE_CHUNK_INCR - (result % STORAGE_CHUNK_INCR); // Chunk size should never exceed SIZE_MAX (might happen on 32-bit platforms) CHECK(AssertError, result <= SIZE_MAX, "chunk size exceeds SIZE_MAX"); FUNCTION_TEST_RETURN(SIZE, (size_t)result); } /**********************************************************************************************************************************/ FN_EXTERN void storageWriteChunkBufferResize(const Buffer *const input, const size_t inputOffset, Buffer *const chunk, const size_t chunkSizeMax) { FUNCTION_TEST_BEGIN(); FUNCTION_TEST_PARAM(BUFFER, input); FUNCTION_TEST_PARAM(SIZE, inputOffset); FUNCTION_TEST_PARAM(BUFFER, chunk); FUNCTION_TEST_PARAM(SIZE, chunkSizeMax); FUNCTION_TEST_END(); ASSERT(input != NULL); ASSERT(inputOffset <= bufUsed(input)); ASSERT(chunk != NULL); ASSERT(chunkSizeMax > 0); // Size required for the chunk buffer to hold the input that has not been copied yet const size_t chunkSizeRequired = bufUsed(chunk) + (bufUsed(input) - inputOffset); // Resize chunk buffer if it cannot hold the remaining input and is less than max chunk size if (chunkSizeRequired > bufSize(chunk) && bufSize(chunk) < chunkSizeMax) { size_t chunkSize; // If the input buffer is full or the chunk buffer is not empty there is very likely more data coming, so increase the chunk // buffer size aggressively rather than resizing repeatedly as small amounts of data arrive if (bufFull(input) || !bufEmpty(chunk)) { // If this is the first write set chunk size equal to double the input buffer size if (bufSize(chunk) == 0) { chunkSize = bufSize(input) * 2; } // Else double chunk size so as not to resize the chunk buffer too many times since prior data must be copied else chunkSize = bufSize(chunk) * 2; // Chunk size must be large enough to hold the remaining input if (chunkSize < chunkSizeRequired) chunkSize = chunkSizeRequired; } // Else no more writes are expected so allocate only what is needed else chunkSize = chunkSizeRequired; // Chunk size cannot be larger than max chunk size if (chunkSize > chunkSizeMax) chunkSize = chunkSizeMax; // Resize the chunk buffer bufResize(chunk, chunkSize); } FUNCTION_TEST_RETURN_VOID(); } /**********************************************************************************************************************************/ FN_EXTERN void storageWriteToLog(const StorageWrite *const this, StringStatic *const debugLog) { strStcCat(debugLog, "{type: "); strStcResultSizeInc(debugLog, strIdToLog(storageWriteType(this), strStcRemains(debugLog), strStcRemainsSize(debugLog))); strStcFmt( debugLog, ", name: %s, modeFile: %04o, modePath: %04o, createPath: %s, syncFile: %s, syncPath: %s, atomic: %s}", strZ(storageWriteName(this)), storageWriteModeFile(this), storageWriteModePath(this), cvtBoolToConstZ(storageWriteCreatePath(this)), cvtBoolToConstZ(storageWriteSyncFile(this)), cvtBoolToConstZ(storageWriteSyncPath(this)), cvtBoolToConstZ(storageWriteAtomic(this))); } ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/write.h0000664000175100017510000001444715240547630017163 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Write Interface ***********************************************************************************************************************************/ #ifndef STORAGE_WRITE_H #define STORAGE_WRITE_H /*********************************************************************************************************************************** Object type ***********************************************************************************************************************************/ typedef struct StorageWrite StorageWrite; #include "common/io/write.h" #include "common/type/object.h" #include "common/type/string.h" #include "common/type/stringId.h" #include "storage/write.intern.h" /*********************************************************************************************************************************** Functions ***********************************************************************************************************************************/ // Move to a new parent mem context FN_INLINE_ALWAYS StorageWrite * storageWriteMove(StorageWrite *const this, MemContext *const parentNew) { return objMove(this, parentNew); } /*********************************************************************************************************************************** Getters/Setters ***********************************************************************************************************************************/ typedef struct StorageWritePub { StringId type; // Storage type const String *name; // File name bool createPath; // Will the path be created if required? bool atomic; // Are writes atomic? bool truncate; // Truncate file if it exists bool syncPath; // Will path be synced? bool syncFile; // Will file be synced? const String *user; // Set user ownership const String *group; // Set group ownership mode_t modePath; // Path mode mode_t modeFile; // File mode time_t timeModified; // Set modification time IoWrite *io; // Write interface } StorageWritePub; // Will the file be written atomically? Atomic writes means the file will be complete or be missing. Filesystems have different ways // to accomplish this. FN_INLINE_ALWAYS bool storageWriteAtomic(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->atomic; } // Will the path be created if required? FN_INLINE_ALWAYS bool storageWriteCreatePath(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->createPath; } // Write interface FN_INLINE_ALWAYS IoWrite * storageWriteIo(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->io; } // File mode FN_INLINE_ALWAYS mode_t storageWriteModeFile(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->modeFile; } // Path mode (if the destination path needs to be create) FN_INLINE_ALWAYS mode_t storageWriteModePath(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->modePath; } // File name FN_INLINE_ALWAYS const String * storageWriteName(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->name; } // Will the file be synced before it is closed? FN_INLINE_ALWAYS bool storageWriteSyncFile(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->syncFile; } // Will the path be synced after the file is closed? FN_INLINE_ALWAYS bool storageWriteSyncPath(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->syncPath; } // Will the file be truncated if it exists? FN_INLINE_ALWAYS bool storageWriteTruncate(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->truncate; } // File type FN_INLINE_ALWAYS StringId storageWriteType(const StorageWrite *const this) { return THIS_PUB(StorageWrite)->type; } /*********************************************************************************************************************************** Destructor ***********************************************************************************************************************************/ FN_INLINE_ALWAYS void storageWriteFree(StorageWrite *const this) { objFree(this); } /*********************************************************************************************************************************** Helper functions ***********************************************************************************************************************************/ // Calculate chunk size for multipart upload to an object store FN_EXTERN size_t storageWriteChunkSize( size_t chunkSizeDefault, unsigned int defaultSplit, unsigned int maxSplit, unsigned int chunkIdx); // Resize chunk buffer for multipart upload to an object store. After the resize the chunk buffer is able to hold the input from // inputOffset to the end of the input buffer, unless max chunk size has been reached. FN_EXTERN void storageWriteChunkBufferResize(const Buffer *input, size_t inputOffset, Buffer *const chunk, size_t chunkSizeMax); /*********************************************************************************************************************************** Macros for function logging ***********************************************************************************************************************************/ FN_EXTERN void storageWriteToLog(const StorageWrite *this, StringStatic *debugLog); #define FUNCTION_LOG_STORAGE_WRITE_TYPE \ StorageWrite * #define FUNCTION_LOG_STORAGE_WRITE_FORMAT(value, buffer, bufferSize) \ FUNCTION_LOG_OBJECT_FORMAT(value, storageWriteToLog, buffer, bufferSize) #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/storage/write.intern.h0000664000175100017510000000522515240547630020453 0ustar00runnerrunner/*********************************************************************************************************************************** Storage Write Interface Internal ***********************************************************************************************************************************/ #ifndef STORAGE_WRITE_INTERN_H #define STORAGE_WRITE_INTERN_H #include "common/io/write.h" #include "storage/storage.h" #include "version.h" /*********************************************************************************************************************************** Temporary file extension ***********************************************************************************************************************************/ #define STORAGE_FILE_TEMP_EXT PROJECT_BIN ".tmp" /*********************************************************************************************************************************** Interface ***********************************************************************************************************************************/ typedef struct StorageWriteInterface { void (*filterGroup)(void *driver, IoFilterGroup *filterGroup); // Set filter group (optional) void (*open)(void *driver); // Open write void (*write)(void *driver, const Buffer *buffer); // Write bytes void (*seek)(void *driver, uint64_t position); // Seek (optional) int (*fd)(const void *driver); // Write file descriptor (optional) void (*close)(void *driver); // Close write } StorageWriteInterface; /*********************************************************************************************************************************** Constructors ***********************************************************************************************************************************/ FN_EXTERN StorageWrite *storageWriteNew( const Storage *storage, const String *name, mode_t modeFile, mode_t modePath, const String *user, const String *group, time_t timeModified, bool createPath, bool syncFile, bool syncPath, bool atomic, bool truncate, bool compressible); /*********************************************************************************************************************************** Getters ***********************************************************************************************************************************/ FN_INLINE_ALWAYS const StorageWriteInterface * storageWriteDriverInterface(const void *const driver) { return *(const StorageWriteInterface *const *)driver; } #endif ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/src/version.h0000664000175100017510000000630515240547630016044 0ustar00runnerrunner/*********************************************************************************************************************************** Version Numbers and Names ***********************************************************************************************************************************/ #ifndef VERSION_H #define VERSION_H /*********************************************************************************************************************************** Official name of the project ***********************************************************************************************************************************/ #define PROJECT_NAME "pgBackRest" /*********************************************************************************************************************************** Standard binary name ***********************************************************************************************************************************/ #define PROJECT_BIN "pgbackrest" /*********************************************************************************************************************************** Config file name. The path will vary based on configuration. ***********************************************************************************************************************************/ #define PROJECT_CONFIG_FILE PROJECT_BIN ".conf" /*********************************************************************************************************************************** Config include path name. The parent path will vary based on configuration. ***********************************************************************************************************************************/ #define PROJECT_CONFIG_INCLUDE_PATH "conf.d" /*********************************************************************************************************************************** Format Number -- defines format for info and manifest files as well as on-disk structure. If this number changes then the repository will be invalid unless migration functions are written. ***********************************************************************************************************************************/ #define REPOSITORY_FORMAT 5 /*********************************************************************************************************************************** Project version components. PROJECT_VERSION and PROJECT_VERSION_NUM are automatically generated from the component parts. ***********************************************************************************************************************************/ #define PROJECT_VERSION_MAJOR 2 #define PROJECT_VERSION_MINOR 59 #define PROJECT_VERSION_PATCH 1 #define PROJECT_VERSION_SUFFIX "" #define PROJECT_VERSION "2.59.1" #define PROJECT_VERSION_NUM 2059001 #endif ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1786957821.7396202 pgbackrest-2.59.1/test/0000775000175100017510000000000015240547776014405 5ustar00runnerrunner././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/test/meson.build0000664000175100017510000000116715240547630016541 0ustar00runnerrunner#################################################################################################################################### # Smoke test #################################################################################################################################### # Test a built pgbackrest through a backup/restore cycle against every PostgreSQL found on the system. Shipped in the distribution # so packagers can verify a build. Fails when there is no PostgreSQL to test against. test( 'smoke', python, args: [files('smoke.py'), '--pgbackrest', pgbackrest_exe], suite: 'smoke', timeout: 1800, ) ././@PaxHeader0000000000000000000000000000002600000000000010213 xustar0022 mtime=1786957720.0 pgbackrest-2.59.1/test/smoke.py0000775000175100017510000005455415240547630016102 0ustar00runnerrunner#!/usr/bin/env python3 #################################################################################################################################### # pgBackRest Smoke Test # # A minimal, dependency-free (Python standard library only) sanity check that a built pgbackrest binary actually works. It finds # every PostgreSQL installation on the system and drives pgbackrest through a full backup/restore cycle against each version found: # # initdb -> configure -> stanza-create -> check -> full backup -> load data -> incr backup -> wipe -> restore -> verify # # The test uses a local POSIX (filesystem) repository and a private unix socket so it never collides with a running PostgreSQL or an # existing pgbackrest installation. It is a build-sanity check for packagers, not a functional test suite. # # Exit status: 0 = every supported version found was tested and passed, 1 = a version failed or nothing could be tested. Testing # nothing is a failure rather than a skip, since a test that silently does nothing is not a successful test. #################################################################################################################################### import argparse import glob import os import pwd import re import shutil import subprocess import sys import tempfile import time # Stanza name used throughout the test STANZA = "smoke" # PostgreSQL port. There is no TCP listener (listen_addresses is empty) so this only names the socket file in the private socket # directory and never collides with a system PostgreSQL. PG_PORT = 5432 # Per-command timeout (seconds). The clusters are tiny so every step is fast; this only guards against a hang. COMMAND_TIMEOUT = 300 class SmokeError(Exception): """A smoke test step failed.""" #################################################################################################################################### # Command runner that optionally drops privileges (when the script runs as root) #################################################################################################################################### class Runner: def __init__(self, verbose, uid=None, gid=None, user=None): self.verbose = verbose self.uid = uid # target uid when dropping privileges, else None self.gid = gid self.user = user # OS user commands run as (also the PostgreSQL superuser) # Return a preexec function that drops to the target user, or None when running as the current user def _preexec(self): if self.uid is None: return None uid, gid, user = self.uid, self.gid, self.user def demote(): os.setgid(gid) os.initgroups(user, gid) os.setuid(uid) return demote # Build a clean environment for child processes def _env(self, home, bin_path): env = { "HOME": home, "PATH": bin_path + os.pathsep + os.environ.get("PATH", "/usr/bin:/bin"), "LC_ALL": "C", "LANG": "C", } return env # Run a command, capturing combined output. Raise SmokeError on failure or timeout. def run(self, cmd, home, bin_path, input_text=None): if self.verbose: sys.stderr.write("+ " + " ".join(cmd) + "\n") sys.stderr.flush() try: result = subprocess.run( cmd, cwd=home, env=self._env(home, bin_path), preexec_fn=self._preexec(), input=input_text, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True, timeout=COMMAND_TIMEOUT, ) except subprocess.TimeoutExpired: raise SmokeError("timed out after %ds: %s" % (COMMAND_TIMEOUT, " ".join(cmd))) if self.verbose and result.stdout: sys.stderr.write(result.stdout) sys.stderr.flush() if result.returncode != 0: raise SmokeError("command failed (exit %d): %s\n%s" % (result.returncode, " ".join(cmd), result.stdout)) return result.stdout # Create a directory and hand ownership to the target user when dropping privileges def make_dir(self, path, mode=0o700): os.makedirs(path, mode=mode, exist_ok=True) if self.uid is not None: os.chown(path, self.uid, self.gid) # Recursively chown a tree to the target user (used after creating the work dir skeleton as root) def chown_tree(self, path): if self.uid is None: return os.chown(path, self.uid, self.gid) for root, dirs, files in os.walk(path): for name in dirs + files: os.chown(os.path.join(root, name), self.uid, self.gid) #################################################################################################################################### # Discover PostgreSQL installations #################################################################################################################################### # Candidate bin directories. This is a superset of the list in test/src/harness/host.c since the smoke test also runs where the # integration tests cannot, i.e. MacOS and FreeBSD. def pg_bin_candidates(extra_dirs): candidates = [] candidates += sorted(glob.glob("/usr/lib/postgresql/*/bin")) # Debian candidates += sorted(glob.glob("/usr/pgsql-*/bin")) # RHEL (PGDG) candidates += sorted(glob.glob("/usr/libexec/postgresql*")) # Alpine candidates += sorted(glob.glob("/opt/homebrew/opt/postgresql@*/bin")) # MacOS (Homebrew on ARM) candidates += sorted(glob.glob("/usr/local/opt/postgresql@*/bin")) # MacOS (Homebrew on Intel) candidates += ["/usr/local/bin"] # FreeBSD candidates += ["/usr/bin"] # Native (e.g. RHEL application stream) candidates += list(extra_dirs) return candidates # Read the major version from a bin directory, or None if it is not a usable PostgreSQL install def pg_version(bin_dir): for tool in ("initdb", "pg_ctl", "psql"): if not os.path.isfile(os.path.join(bin_dir, tool)) or not os.access(os.path.join(bin_dir, tool), os.X_OK): return None try: output = subprocess.check_output( [os.path.join(bin_dir, "pg_ctl"), "--version"], universal_newlines=True, stderr=subprocess.STDOUT ) except (subprocess.CalledProcessError, OSError): return None # e.g. "pg_ctl (PostgreSQL) 16.2" or "pg_ctl (PostgreSQL) 9.6.24". The version is encoded as major * 100 + minor so it sorts and # compares naturally: 9.6 -> 906, 10 -> 1000, 16 -> 1600. For version 10 and later the major is a single number and the trailing # component is only the patch release, so it is ignored. match = re.search(r"\)\s+(\d+)(?:\.(\d+))?", output) if match is None: return None major = int(match.group(1)) if major >= 10: return major * 100 # Versions before 10 have a two-part major version (e.g. 9.6) if match.group(2) is None: return None return major * 100 + int(match.group(2)) # Read the full version string from a bin directory, e.g. "pg_ctl (PostgreSQL) 19beta2". This is only called for an install # returned by find_postgres(), where pg_version() has already run pg_ctl successfully. def pg_version_full(bin_dir): output = subprocess.check_output( [os.path.join(bin_dir, "pg_ctl"), "--version"], universal_newlines=True, stderr=subprocess.STDOUT ) return output.strip() # Return a sorted list of (version, bin_dir) and a sorted list of versions skipped because pgBackRest does not support them. The # installs are deduplicated by the real path of initdb so a symlinked install is not run twice. def find_postgres(extra_dirs, only_versions, supported): found = {} skipped = set() for bin_dir in pg_bin_candidates(extra_dirs): if not os.path.isdir(bin_dir): continue version = pg_version(bin_dir) if version is None: continue # Skip versions that pgBackRest does not support, e.g. an old PostgreSQL that happens to be installed if supported is not None and version not in supported: skipped.add(version) continue if only_versions and version not in only_versions: continue key = os.path.realpath(os.path.join(bin_dir, "initdb")) if key not in found: found[key] = (version, bin_dir) return sorted(found.values()), sorted(skipped) # Parse a version string (e.g. "9.6" or "16") into the encoded form used internally def version_parse(version): if "." in version: major, minor = version.split(".", 1) return int(major) * 100 + int(minor) return int(version) * 100 # Read the PostgreSQL versions supported by pgBackRest from the generated version header. The header is generated from # src/build/postgres/postgres.yaml and, unlike that file, is shipped in the distribution, so this works from a source checkout or a # distribution tarball. Return None when the header cannot be found so that every version found is tested. def supported_versions(): path = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "src", "postgres", "version.auto.h") try: with open(path) as file: content = file.read() except OSError: return None result = {version_parse(match) for match in re.findall(r'^#define PG_VERSION_\w+_Z\s+"([0-9.]+)"', content, re.MULTILINE)} return result or None # Render a version number for display (e.g. 906 -> "9.6", 1600 -> "16") def version_str(version): major = version // 100 minor = version % 100 if major >= 10: return str(major) return "%d.%d" % (major, minor) #################################################################################################################################### # Run the smoke test for a single PostgreSQL version #################################################################################################################################### def smoke_one(pgbackrest, version, bin_dir, runner, keep): work = tempfile.mkdtemp(prefix="pgbackrest-smoke-", dir=temp_base()) data = os.path.join(work, "data") repo = os.path.join(work, "repo") log = os.path.join(work, "log") socket = os.path.join(work, "socket") lock = os.path.join(work, "lock") config = os.path.join(work, "pgbackrest.conf") pg_log = os.path.join(log, "postgresql.log") # Helpers scoped to this run def pg(cmd): return runner.run([os.path.join(bin_dir, cmd[0])] + cmd[1:], work, bin_dir) def pgbr(args): return runner.run([pgbackrest, "--config=" + config, "--stanza=" + STANZA] + args, work, bin_dir) def psql(sql): return runner.run( [ os.path.join(bin_dir, "psql"), "-h", socket, "-p", str(PG_PORT), "-U", runner.user, "-d", "postgres", "-v", "ON_ERROR_STOP=1", "-tAqc", sql, ], work, bin_dir, ) pg_started = False success = False try: # Pre-create the directories the tools do not create themselves and hand the whole tree to the run user for path in (socket, log, lock): os.makedirs(path, mode=0o700, exist_ok=True) runner.chown_tree(work) # Initialize the cluster (checksums on, no fsync since it is throwaway, trust auth for local connections) pg(["initdb", "-k", "-N", "--pgdata=" + data, "--auth=trust"]) # Append test configuration to postgresql.conf conf = ( "\n" "unix_socket_directories = '%s'\n" "listen_addresses = ''\n" "port = %d\n" "wal_level = replica\n" "archive_mode = on\n" "archive_command = '%s --config=%s --stanza=%s archive-push %%p'\n" "fsync = off\n" % (socket, PG_PORT, pgbackrest, config, STANZA) ) append_file(os.path.join(data, "postgresql.conf"), conf) # Write the pgbackrest configuration write_file( config, "[global]\n" "repo1-path=%s\n" "log-path=%s\n" "lock-path=%s\n" "log-level-console=detail\n" "log-level-file=off\n" "start-fast=y\n" "compress-type=lz4\n" "\n" "[%s]\n" "pg1-path=%s\n" "pg1-port=%d\n" "pg1-socket-path=%s\n" "pg1-user=%s\n" % (repo, log, lock, STANZA, data, PG_PORT, socket, runner.user), runner, ) # Start PostgreSQL pg(["pg_ctl", "start", "-D", data, "-l", pg_log, "-w"]) pg_started = True # Load data psql("CREATE TABLE smoke AS SELECT generate_series(1, 1000) AS id") # Create the stanza and confirm archiving works end to end pgbr(["stanza-create"]) pgbr(["check"]) # Full backup pgbr(["--type=full", "backup"]) # Add more data and force the WAL segment to archive psql("INSERT INTO smoke SELECT generate_series(1001, 2000)") psql("SELECT %s()" % ("pg_switch_wal" if version >= 1000 else "pg_switch_xlog")) # Incremental backup. The backup command waits for the required WAL to archive, so the restore below has no race. pgbr(["--type=incr", "backup"]) expected = 2000 # Stop PostgreSQL and remove the data directory to prove the restore rebuilds it from the repository pg(["pg_ctl", "stop", "-D", data, "-w", "-m", "fast"]) pg_started = False shutil.rmtree(data) runner.make_dir(data, mode=0o700) # Restore the latest backup (recovery config is written automatically) and start PostgreSQL pgbr(["restore"]) pg(["pg_ctl", "start", "-D", data, "-l", pg_log, "-w"]) pg_started = True # Wait for recovery to replay all WAL and promote (there is no standby.signal so PostgreSQL promotes at end of WAL) wait_for_promote(psql) # Verify the restored data matches what was backed up actual = int(psql("SELECT count(*) FROM smoke").strip()) if actual != expected: raise SmokeError("restored row count %d does not match expected %d" % (actual, expected)) # Show backup info (printed by the runner when verbose) pgbr(["info"]) pg(["pg_ctl", "stop", "-D", data, "-w", "-m", "fast"]) pg_started = False success = True finally: # Always try to stop PostgreSQL so no stray processes are left behind if pg_started: try: runner.run([os.path.join(bin_dir, "pg_ctl"), "stop", "-D", data, "-w", "-m", "immediate"], work, bin_dir) except SmokeError: pass # Remove the work directory on success, else report where it was kept so the failure can be investigated if success and not keep: shutil.rmtree(work, ignore_errors=True) else: sys.stderr.write("work directory kept: %s\n" % work) # Poll until the cluster has finished recovery and accepts connections def wait_for_promote(psql): deadline = time.time() + 60 while time.time() < deadline: try: if psql("SELECT pg_is_in_recovery()").strip() == "f": return except SmokeError: pass # server still starting or rejecting during recovery time.sleep(0.5) raise SmokeError("cluster did not finish recovery within 60s") #################################################################################################################################### # Small file and path helpers #################################################################################################################################### # Prefer /tmp for the work directory to keep unix socket paths short (macOS TMPDIR can exceed the socket path limit) def temp_base(): if os.path.isdir("/tmp") and os.access("/tmp", os.W_OK): return "/tmp" return tempfile.gettempdir() def write_file(path, contents, runner): with open(path, "w") as file: file.write(contents) if runner.uid is not None: os.chown(path, runner.uid, runner.gid) # No ownership change is needed here since the file being appended to was created by initdb as the run user def append_file(path, contents): with open(path, "a") as file: file.write(contents) #################################################################################################################################### # Resolve the user to run as, dropping privileges when running as root #################################################################################################################################### def resolve_runner(verbose, user_name): # Not root: run everything as the current user if os.geteuid() != 0: return Runner(verbose, user=pwd.getpwuid(os.geteuid()).pw_name) # Root: PostgreSQL refuses to run as root, so drop to an unprivileged user try: info = pwd.getpwnam(user_name) except KeyError: return None return Runner(verbose, uid=info.pw_uid, gid=info.pw_gid, user=info.pw_name) #################################################################################################################################### # Read the machine architecture and endianness from an ELF binary. This confirms which architecture is actually being exercised, # which matters when the binary is built for a foreign architecture and run under emulation. #################################################################################################################################### def binary_arch(path): # ELF e_machine values for the architectures pgBackRest is built on machines = {0x03: "i386", 0x3E: "x86_64", 0x15: "ppc64", 0x16: "s390x", 0xB7: "aarch64", 0x28: "arm"} with open(path, "rb") as file: header = file.read(20) if len(header) < 20 or header[:4] != b"\x7fELF": return "unknown (not an ELF binary)" little = header[5] == 1 machine = int.from_bytes(header[18:20], "little" if little else "big") return "%s (%s-bit %s-endian)" % ( machines.get(machine, "machine 0x%02X" % machine), "64" if header[4] == 2 else "32", "little" if little else "big", ) #################################################################################################################################### # Main #################################################################################################################################### def main(): parser = argparse.ArgumentParser(description="Run pgBackRest smoke tests against installed PostgreSQL versions.") parser.add_argument("--pgbackrest", required=True, help="path to the pgbackrest binary to test") parser.add_argument("--pg-bin-path", action="append", default=[], metavar="DIR", help="extra PostgreSQL bin directory") parser.add_argument("--version", action="append", default=[], metavar="MAJOR", help="only run this major version (repeatable)") parser.add_argument("--user", default="postgres", help="unprivileged user to drop to when run as root (default: postgres)") parser.add_argument("--keep", action="store_true", help="do not delete the work directory") parser.add_argument("-v", "--verbose", action="store_true", help="echo commands and their output") arg = parser.parse_args() pgbackrest = os.path.realpath(arg.pgbackrest) if not os.path.isfile(pgbackrest) or not os.access(pgbackrest, os.X_OK): sys.stderr.write("pgbackrest binary not found or not executable: %s\n" % pgbackrest) return 1 # Extra bin dirs from the command line and the environment (colon-separated) extra_dirs = list(arg.pg_bin_path) env_dirs = os.environ.get("PGBACKREST_SMOKE_PG_BIN_PATH", "") if env_dirs: extra_dirs += env_dirs.split(os.pathsep) # Parse the requested version filter (accepts "9.6" or "16") try: only_versions = {version_parse(item) for item in arg.version} except ValueError: sys.stderr.write("invalid --version: expected a major version such as 9.6 or 16\n") return 1 supported = supported_versions() installs, unsupported = find_postgres(extra_dirs, only_versions, supported) # Report any installs that were skipped because pgBackRest does not support them if unsupported: sys.stderr.write("skipped unsupported PostgreSQL: %s\n" % ", ".join(version_str(v) for v in unsupported)) # Testing nothing is a failure. If there is no PostgreSQL to test against then the build has not been verified. if not installs: sys.stderr.write("no supported PostgreSQL installation found to test against\n") return 1 runner = resolve_runner(arg.verbose, arg.user) if runner is None: sys.stderr.write("running as root but user '%s' does not exist so PostgreSQL cannot be run\n" % arg.user) return 1 sys.stdout.write( "pgbackrest smoke test: %s\nrunning as: %s\narchitecture: %s\nversions supported: %s\nversions found: %s\n\n" % ( pgbackrest, runner.user, binary_arch(pgbackrest), ", ".join(version_str(v) for v in sorted(supported)) if supported else "unknown", ", ".join(version_str(v) for v, _ in installs), ) ) passed = 0 failed = 0 for version, bin_dir in installs: label = "PostgreSQL %s (%s)" % (version_str(version), bin_dir) sys.stdout.write("=== %s ===\n" % label) # Log the version of PostgreSQL installed on the host. Only the major version is discovered above so this shows the minor # version, which is especially useful during the beta period. sys.stdout.write("%s\n" % pg_version_full(bin_dir)) sys.stdout.flush() start = time.time() try: smoke_one(pgbackrest, version, bin_dir, runner, arg.keep) passed += 1 sys.stdout.write("PASS %s (%.1fs)\n\n" % (label, time.time() - start)) except SmokeError as e: failed += 1 sys.stdout.write("FAIL %s (%.1fs)\n%s\n\n" % (label, time.time() - start, e)) sys.stdout.flush() sys.stdout.write("summary: %d passed, %d failed\n" % (passed, failed)) return 1 if failed else 0 if __name__ == "__main__": sys.exit(main())