pax_global_header00006660000000000000000000000064152622427530014522gustar00rootroot0000000000000052 comment=104198474662752f4f4845c79007100829aff442 httptools-0.9.0/000077500000000000000000000000001526224275300135705ustar00rootroot00000000000000httptools-0.9.0/.github/000077500000000000000000000000001526224275300151305ustar00rootroot00000000000000httptools-0.9.0/.github/RELEASING.rst000066400000000000000000000041721526224275300171770ustar00rootroot00000000000000Releasing httptools =================== The ``Release`` workflow builds the source distribution and builds and tests the Linux, macOS, and Windows wheel matrix. Unless it's a dry-run (see below), it also merges and tags the release PR, creates a GitHub release, and uploads the distributions to PyPI. Dry-run a release ----------------- Two ways of building and testing release wheels without permanent publishing: * Add the ``release-dry-run`` label to a PR targeting ``master``, ``ci``, or a release branch. The label stays in effect for later commits to that PR. It also works for PRs from forks and is useful to test changes to the release workflow itself or to test build targets that are not tested by regular PR CI. * On GitHub, open **Actions > Release > Run workflow**, then select a repository branch, including ``master``. Manual runs are always dry runs. GitHub shows the **Run workflow** button once this workflow is on the default branch. You can then find the ``dist`` artifact under **Artifacts** on the workflow run page. The wheels are built and tested by cibuildwheel. Publish a release ----------------- 1. Update ``__version__`` in ``httptools/_version.py`` and prepare the release changelog. ``.github/release_log.py`` can help gather changes since the previous release tag:: $ uv run --no-project --with requests \ .github/release_log.py 2. Open a PR to ``master`` with the version change and release notes in its body. Without the dry-run label, the ``Release`` workflow builds the distributions and tests the wheels. Check the build results, then have a Release Manager approve the pending ``pypi`` deployment using **Review deployments** on the workflow run. 3. After deployment approval, the workflow merges the PR, signs a tag on the merge commit, creates the GitHub release, and uploads the distributions to PyPI. Check these outputs after it finishes, then edit the GitHub release notes as needed. 4. Open ``master`` for development by updating ``httptools/_version.py`` to the next development version with a ``.dev0`` suffix. httptools-0.9.0/.github/release_log.py000077500000000000000000000024451526224275300177730ustar00rootroot00000000000000#!/usr/bin/env python3 # # Copyright (C) 2016-present the asyncpg authors and contributors # # # This module is part of asyncpg and is released under # the Apache 2.0 License: http://www.apache.org/licenses/LICENSE-2.0 import json import requests import re import sys BASE_URL = 'https://api.github.com/repos/magicstack/httptools/compare' def main(): if len(sys.argv) < 2: print('pass a sha1 hash as a first argument') sys.exit(1) from_hash = sys.argv[1] to_hash = 'master' if len(sys.argv) > 2: to_hash = sys.argv[2] r = requests.get(f'{BASE_URL}/{from_hash}...{to_hash}') data = json.loads(r.text) for commit in data['commits']: message = commit['commit']['message'] first_line = message.partition('\n\n')[0] if commit.get('author'): username = '@{}'.format(commit['author']['login']) else: username = commit['commit']['author']['name'] sha = commit["sha"][:8] m = re.search(r'\#(?P\d+)\b', message) if m: issue_num = m.group('num') else: issue_num = None print(f'* {first_line}') print(f' (by {username} in {sha}', end='') print(')') print() if __name__ == '__main__': main() httptools-0.9.0/.github/workflows/000077500000000000000000000000001526224275300171655ustar00rootroot00000000000000httptools-0.9.0/.github/workflows/release.yml000066400000000000000000000473631526224275300213450ustar00rootroot00000000000000name: Release on: workflow_dispatch: pull_request: types: [opened, reopened, synchronize, labeled, unlabeled] branches: - "master" - "ci" - "[0-9]+.[0-9]+.[0-9x]+*" permissions: contents: read # Serialize releases of the same PR; never cancel a publishing job. concurrency: group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }} cancel-in-progress: false jobs: classify-release: runs-on: ubuntu-24.04 permissions: pull-requests: read outputs: mode: ${{ steps.mode.outputs.mode }} steps: - name: Select release mode id: mode uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 with: script: | let mode = 'skip'; if (context.eventName === 'workflow_dispatch') { mode = 'dry_run'; } else { const { action, label, pull_request: pr } = context.payload; if (action === 'labeled' || action === 'unlabeled') { // Ignore unrelated label changes and do not release on removal. if (action === 'labeled' && label.name === 'release-dry-run') { mode = 'dry_run'; } } else if (pr.labels.some(({ name }) => name === 'release-dry-run')) { mode = 'dry_run'; } else { // Preserve the previous version-file filter for ordinary PRs. const files = await github.paginate(github.rest.pulls.listFiles, { ...context.repo, pull_number: pr.number, per_page: 100, }); if (files.some(({ filename }) => filename === 'httptools/_version.py')) { mode = 'release'; } } } core.setOutput('mode', mode); core.info(`Release mode: ${mode}`); build-sdist: needs: classify-release if: needs.classify-release.outputs.mode != 'skip' runs-on: ubuntu-24.04 env: PIP_DISABLE_PIP_VERSION_CHECK: 1 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 50 submodules: true persist-credentials: false - name: Set up Python uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 with: python-version: "3.14" activate-environment: true - name: Build source distribution run: | uv build --sdist - name: Store release version run: | python - <<'PY' import re from pathlib import Path source = Path('httptools/_version.py').read_text() match = re.search( r'''^__version__(?:\s*:\s*typing\.Final)?\s*=\s*['"]([^'"]+)['"]''', source, re.MULTILINE, ) if match is None: raise RuntimeError('Could not read the release version') Path('dist/VERSION').write_text(match.group(1) + '\n') PY - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: dist-sdist path: dist/ build-wheels-matrix: needs: classify-release if: needs.classify-release.outputs.mode != 'skip' runs-on: ubuntu-24.04 outputs: include: ${{ steps.set-matrix.outputs.include }} steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 with: python-version: "3.14" activate-environment: true - name: Install cibuildwheel run: uv pip install cibuildwheel==4.2.1 - name: Build cibuildwheel matrix id: set-matrix env: WHEEL_TARGETS: | [ {"platform": "linux", "archs": "x86_64", "os": "ubuntu-24.04"}, {"platform": "linux", "archs": "aarch64", "os": "ubuntu-24.04-arm"}, {"platform": "linux", "archs": "ppc64le", "os": "ubuntu-24.04"}, {"platform": "linux", "archs": "riscv64", "os": "ubuntu-24.04"}, {"platform": "macos", "archs": "x86_64", "os": "macos-26-intel"}, {"platform": "macos", "archs": "arm64", "os": "macos-26"}, {"platform": "windows", "archs": "x86,AMD64", "os": "windows-2025"}, {"platform": "windows", "archs": "ARM64", "os": "windows-11-arm"} ] run: | python - <<'PY' import json import os import subprocess include = [] for target in json.loads(os.environ['WHEEL_TARGETS']): identifiers = subprocess.check_output([ 'cibuildwheel', '--print-build-identifiers', '--platform', target['platform'], '--archs', target['archs'], ], text=True).splitlines() include.extend({'only': identifier, 'os': target['os']} for identifier in identifiers) print(f'Wheel matrix entries: {len(include)}') if not 0 < len(include) <= 256: raise RuntimeError('Wheel matrix must contain between 1 and 256 entries') if len({entry['only'] for entry in include}) != len(include): raise RuntimeError('Wheel matrix contains duplicate build identifiers') with open(os.environ['GITHUB_OUTPUT'], 'a') as output: output.write(f'include={json.dumps(include, separators=(",", ":"))}\n') PY build-wheels: needs: build-wheels-matrix if: ${{ !cancelled() && needs.build-wheels-matrix.result == 'success' }} runs-on: ${{ matrix.os }} name: Build ${{ matrix.only }} strategy: fail-fast: false matrix: include: ${{ fromJson(needs.build-wheels-matrix.outputs.include) }} defaults: run: shell: bash env: PIP_DISABLE_PIP_VERSION_CHECK: 1 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 50 submodules: true persist-credentials: false - name: Set up QEMU if: endsWith(matrix.only, '_ppc64le') || endsWith(matrix.only, '_riscv64') uses: docker/setup-qemu-action@99012661954931238ded8c8b007157a8430204e1 # v4.4.0 with: platforms: ppc64le,riscv64 - uses: pypa/cibuildwheel@e090b81e30c4d855ea63bf4b6e59204c09a101ae # v4.2.1 with: only: ${{ matrix.only }} env: CIBW_BUILD_VERBOSITY: 1 - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: dist-wheels-${{ matrix.only }} path: wheelhouse/*.whl merge-artifacts: runs-on: ubuntu-24.04 needs: [build-sdist, build-wheels] if: >- ${{ !cancelled() && needs.build-sdist.result == 'success' && needs.build-wheels.result == 'success' }} steps: - name: Merge Artifacts uses: actions/upload-artifact/merge@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: dist pattern: dist-* delete-merged: true dry-run-refs: needs: [classify-release, merge-artifacts] if: >- ${{ !cancelled() && needs.classify-release.outputs.mode == 'dry_run' && needs.merge-artifacts.result == 'success' }} runs-on: ubuntu-24.04 steps: - name: Checkout release commit uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 persist-credentials: false - name: Sign release tag locally uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 with: script: | const fs = require('node:fs'); const os = require('node:os'); const path = require('node:path'); const {execFileSync} = require('node:child_process'); const cwd = process.env.GITHUB_WORKSPACE; const pr = context.payload.pull_request; // PR workflows check out GitHub's merge commit. Its second parent // is the exact PR head selected for this dry run. const head = (await exec.getExecOutput('git', [ 'rev-parse', pr ? 'HEAD^2' : 'HEAD', ], { cwd, silent: true, })).stdout.trim(); if (pr && head !== pr.head.sha) { throw new Error(`Checked out ${head}, expected PR head ${pr.head.sha}`); } const source = (await exec.getExecOutput('git', [ 'show', `${head}:httptools/_version.py`, ], {cwd, silent: true})).stdout; const version = source.match(/^__version__(?:\s*:\s*typing\.Final)?\s*=\s*['"]([^'"]+)['"]/m)?.[1]; if (!version) throw new Error('Could not read the release version'); const tag = `v${version}`; const message = pr ? `${pr.title}\n\n${pr.body || ''}` : `Dry run for ${context.ref}`; // Live releases sign the merged commit with the release key. Generate // a disposable key here so fork PRs can exercise signing too. const gnupgHome = fs.mkdtempSync(path.join(os.tmpdir(), 'release-dry-run-gpg-')); fs.chmodSync(gnupgHome, 0o700); const originalGnupgHome = process.env.GNUPGHOME; try { const env = {...process.env, GNUPGHOME: gnupgHome}; execFileSync('gpg', [ '--batch', '--pinentry-mode', 'loopback', '--passphrase', '', '--quick-generate-key', 'Release dry run ', 'ed25519', 'sign', '0', ], {env}); const keys = execFileSync('gpg', ['--list-secret-keys', '--with-colons'], { env, encoding: 'utf8', }); const key = keys.split('\n').find(line => line.startsWith('fpr:'))?.split(':')[9]; if (!key) throw new Error('Could not identify the disposable signing key'); await exec.exec('git', ['config', 'user.name', 'Release dry run'], {cwd}); await exec.exec('git', ['config', 'user.email', 'release-dry-run@users.noreply.github.com'], {cwd}); await exec.exec('git', ['config', 'user.signingkey', key], {cwd}); process.env.GNUPGHOME = gnupgHome; await exec.exec('git', ['tag', '--sign', '--file=-', tag, head], { cwd, input: Buffer.from(message), }); await exec.exec('git', ['tag', '--verify', tag], {cwd}); await exec.exec('git', ['show', '--no-patch', tag], {cwd}); } finally { await exec.exec('gpgconf', ['--kill', 'gpg-agent'], { env: {...process.env, GNUPGHOME: gnupgHome}, ignoreReturnCode: true, }); if (originalGnupgHome === undefined) { delete process.env.GNUPGHOME; } else { process.env.GNUPGHOME = originalGnupgHome; } fs.rmSync(gnupgHome, {recursive: true, force: true}); } core.info(`Dry run: signed ${tag} at ${head} locally; no refs were pushed.`); publish: needs: [classify-release, merge-artifacts] if: >- ${{ !cancelled() && needs.classify-release.outputs.mode == 'release' && needs.merge-artifacts.result == 'success' }} runs-on: ubuntu-24.04 environment: name: pypi url: https://pypi.org/p/httptools permissions: id-token: write attestations: write contents: write deployments: write pull-requests: write steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: fetch-depth: 2 persist-credentials: false - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1 with: name: dist path: dist/ - name: Extract release version id: relver run: | echo "version=$(cat dist/VERSION)" >> "$GITHUB_OUTPUT" rm dist/VERSION - name: Merge release PR id: merge uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 with: github-token: ${{ secrets.RELEASE_BOT_GITHUB_TOKEN }} script: | const expected = context.payload.pull_request; const args = {...context.repo, pull_number: expected.number}; const {data: pr} = await github.rest.pulls.get(args); if (pr.head.sha !== expected.head.sha) { throw new Error('PR head changed since the distributions were built'); } let sha; if (pr.merged) { sha = pr.merge_commit_sha; core.info('Release PR is already merged.'); } else { const {data: result} = await github.rest.pulls.merge({ ...args, sha: expected.head.sha, merge_method: 'merge', }); if (!result.merged || !result.sha) { throw new Error('Could not merge release PR: ' + result.message); } sha = result.sha; } if (!sha) throw new Error('Could not identify the merge commit'); core.setOutput('sha', sha); - name: Sign and push release tag uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: RELEASE_VERSION: ${{ steps.relver.outputs.version }} RELEASE_SHA: ${{ steps.merge.outputs.sha }} RELEASE_BOT_GPG_KEY: ${{ secrets.RELEASE_BOT_GPG_KEY }} RELEASE_BOT_GITHUB_TOKEN: ${{ secrets.RELEASE_BOT_GITHUB_TOKEN }} with: github-token: ${{ secrets.RELEASE_BOT_GITHUB_TOKEN }} script: | const fs = require('node:fs'); const os = require('node:os'); const path = require('node:path'); const cwd = process.env.GITHUB_WORKSPACE; const tag = 'v' + process.env.RELEASE_VERSION; const sha = process.env.RELEASE_SHA; const keyId = '5C468778062D87BF'; const pr = context.payload.pull_request; if (!process.env.RELEASE_BOT_GPG_KEY) { throw new Error('Release signing key is not configured'); } await exec.exec('git', ['check-ref-format', 'refs/tags/' + tag], {cwd}); const gnupgHome = fs.mkdtempSync(path.join(os.tmpdir(), 'release-gpg-')); fs.chmodSync(gnupgHome, 0o700); const auth = Buffer.from('x-access-token:' + process.env.RELEASE_BOT_GITHUB_TOKEN) .toString('base64'); core.setSecret(auth); const env = { ...process.env, GNUPGHOME: gnupgHome, GIT_CONFIG_COUNT: '1', GIT_CONFIG_KEY_0: 'http.https://github.com/.extraheader', GIT_CONFIG_VALUE_0: 'AUTHORIZATION: basic ' + auth, }; const options = {cwd, env, silent: true}; try { await exec.exec('gpg', ['--batch', '--no-tty', '--import'], { ...options, input: Buffer.from(process.env.RELEASE_BOT_GPG_KEY), }); const keys = await exec.getExecOutput('gpg', [ '--batch', '--with-colons', '--list-secret-keys', keyId + '!', ], options); const fingerprint = keys.stdout.split('\n') .filter(line => line.startsWith('fpr:')) .map(line => line.split(':')[9]) .find(value => value.endsWith(keyId)); if (!fingerprint) throw new Error('Could not identify the release signing key'); const verify = async () => { const target = (await exec.getExecOutput('git', [ 'rev-parse', tag + '^{commit}', ], options)).stdout.trim(); if (target !== sha) throw new Error('Existing tag points to a different commit'); const signature = await exec.getExecOutput('git', [ '-c', 'gpg.format=openpgp', 'verify-tag', '--raw', tag, ], {...options, ignoreReturnCode: true}); const signer = signature.stderr.match(/^\[GNUPG:\] VALIDSIG ([0-9A-F]+)/m)?.[1]; if (signature.exitCode !== 0 || signer !== fingerprint) { throw new Error('Tag does not have a valid signature from the release key'); } }; const fetchTag = () => exec.exec('git', [ 'fetch', '--force', '--no-tags', 'origin', 'refs/tags/' + tag + ':refs/tags/' + tag, ], options); const tagExists = async () => { try { await github.rest.git.getRef({...context.repo, ref: 'tags/' + tag}); return true; } catch (error) { if (error.status !== 404) throw error; return false; } }; if (await tagExists()) { await fetchTag(); await verify(); core.info('Release tag already exists and has the expected commit and signature.'); return; } await exec.exec('git', ['fetch', '--no-tags', 'origin', sha], options); const {data: bot} = await github.rest.users.getAuthenticated(); const email = bot.email || bot.id + '+' + bot.login + '@users.noreply.github.com'; await exec.exec('git', [ '-c', 'user.name=' + (bot.name || bot.login), '-c', 'user.email=' + email, '-c', 'gpg.format=openpgp', 'tag', '--sign', '--local-user', fingerprint + '!', '--file=-', tag, sha, ], {...options, input: Buffer.from(pr.title + '\n\n' + (pr.body || ''))}); await verify(); const push = await exec.getExecOutput('git', [ 'push', 'origin', 'refs/tags/' + tag, ], {...options, ignoreReturnCode: true}); if (push.exitCode !== 0) { // Accept a concurrent push only when it created the same signed tag target. if (!(await tagExists())) throw new Error('Could not push release tag'); await fetchTag(); await verify(); } core.info('Signed and pushed ' + tag + ' at ' + sha); } finally { await exec.exec('gpgconf', ['--kill', 'gpg-agent'], { ...options, ignoreReturnCode: true, }); fs.rmSync(gnupgHome, {recursive: true, force: true}); } - name: Create GitHub release uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: RELEASE_VERSION: ${{ steps.relver.outputs.version }} RELEASE_SHA: ${{ steps.merge.outputs.sha }} RELEASE_BODY: ${{ github.event.pull_request.body }} with: script: | const tag = 'v' + process.env.RELEASE_VERSION; const args = {...context.repo, tag}; const existingRelease = async () => { try { return (await github.rest.repos.getReleaseByTag(args)).data; } catch (error) { if (error.status !== 404) throw error; return null; } }; if (await existingRelease()) { core.info('Release ' + tag + ' already exists, skipping.'); return; } try { await github.rest.repos.createRelease({ ...context.repo, tag_name: tag, name: tag, target_commitish: process.env.RELEASE_SHA, body: process.env.RELEASE_BODY, draft: false, prerelease: false, }); } catch (error) { if (error.status !== 422 || !(await existingRelease())) throw error; core.info('Release ' + tag + ' was created concurrently, skipping.'); } - name: Upload to PyPI uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2 with: attestations: true skip-existing: true httptools-0.9.0/.github/workflows/tests.yml000066400000000000000000000075721526224275300210650ustar00rootroot00000000000000name: Tests on: push: branches: [master, ci] pull_request: branches: [master] permissions: contents: read concurrency: group: ${{ github.workflow }}-${{ github.ref || github.run_id }} cancel-in-progress: ${{ github.event_name == 'pull_request' }} jobs: classify-tests: runs-on: ubuntu-24.04 permissions: pull-requests: read outputs: is_release: ${{ steps.version.outputs.is_release }} steps: - name: Check for a version PR id: version uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 with: script: | let isRelease = false; if (context.eventName === 'pull_request') { const files = await github.paginate(github.rest.pulls.listFiles, { ...context.repo, pull_number: context.payload.pull_request.number, per_page: 100, }); isRelease = files.some(({filename}) => filename === 'httptools/_version.py'); } core.setOutput('is_release', isRelease ? 'true' : 'false'); typecheck: runs-on: ubuntu-24.04 steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: submodules: true persist-credentials: false - name: Set up Python uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 with: python-version: "3.14" activate-environment: true - name: Install dependencies run: uv pip install -e . --group dev - name: Type check run: pyright - name: Audit GitHub Actions workflows uses: zizmorcore/zizmor-action@cc914d7f3750a2d13d75c7f184a1060aa0e9d482 # v0.6.4 env: GH_TOKEN: ${{ github.token }} with: token: ${{ github.token }} online-audits: true advanced-security: false annotations: true build: needs: classify-tests if: needs.classify-tests.outputs.is_release != 'true' runs-on: ${{ matrix.os }} strategy: matrix: python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14", "3.14t", "3.15", "3.15t"] os: [ubuntu-24.04, macos-26, windows-2025, windows-11-arm] exclude: - os: macos-26 python-version: "3.9" - os: windows-11-arm python-version: "3.9" - os: windows-11-arm python-version: "3.10" defaults: run: shell: bash steps: - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: submodules: true persist-credentials: false - name: Set up Python ${{ matrix.python-version }} uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 with: python-version: ${{ matrix.python-version }} activate-environment: true - name: Install dependencies run: uv pip install -e . --group dev - name: Test run: python -m unittest -v tests.suite # A stable required check for branch protection, including version PRs. regression-tests: name: Regression Tests needs: [classify-tests, typecheck, build] if: ${{ always() }} runs-on: ubuntu-24.04 permissions: {} steps: - name: Check required jobs uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: JOBS: ${{ toJSON(needs) }} with: script: | const jobs = JSON.parse(process.env.JOBS); const expected = { 'classify-tests': 'success', typecheck: 'success', build: jobs['classify-tests'].outputs.is_release === 'true' ? 'skipped' : 'success', }; for (const [job, result] of Object.entries(expected)) { if (jobs[job].result !== result) { core.setFailed(`${job}: expected ${result}, got ${jobs[job].result}`); } } httptools-0.9.0/.gitignore000066400000000000000000000004351526224275300155620ustar00rootroot00000000000000*._* *.pyc *.pyo *.ymlc *.ymlc~ *.scssc *.so *.pyd *~ .#* .DS_Store .project .pydevproject .settings .idea /.ropeproject \#*# /pub /test*.py /.local /perf.data* /config_local.yml /build __pycache__/ .d8_history /*.egg /*.egg-info /dist /.pytest_cache /.mypy_cache /.vscode .eggs .venv httptools-0.9.0/.gitmodules000066400000000000000000000003351526224275300157460ustar00rootroot00000000000000[submodule "vendor/http-parser"] path = vendor/http-parser url = https://github.com/nodejs/http-parser.git [submodule "vendor/llhttp"] path = vendor/llhttp url = https://github.com/nodejs/llhttp.git branch = release httptools-0.9.0/LICENSE000066400000000000000000000021051526224275300145730ustar00rootroot00000000000000The MIT License Copyright (c) 2015 MagicStack Inc. http://magic.io Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. httptools-0.9.0/MANIFEST.in000066400000000000000000000001661526224275300153310ustar00rootroot00000000000000recursive-include httptools *.pxd *.pyx recursive-include vendor *.c *.h LICENSE* README* include MANIFEST.in LICENSE httptools-0.9.0/Makefile000066400000000000000000000012541526224275300152320ustar00rootroot00000000000000.PHONY: compile release test distclean clean PYTHON ?= python3 ROOT = $(dir $(realpath $(firstword $(MAKEFILE_LIST)))) UV := $(shell command -v uv 2> /dev/null) ifdef UV PYTHON := uv run PIP := uv pip else PIP := pip endif compile: $(PIP) install -e . --group dev test: compile $(PYTHON) -m unittest -v typecheck: compile $(PYTHON) -m pyright clean: find $(ROOT)/httptools/parser -name '*.c' | xargs rm -f find $(ROOT)/httptools/parser -name '*.so' | xargs rm -f find $(ROOT)/httptools/parser -name '*.html' | xargs rm -f rm -rf build distclean: clean git --git-dir="$(ROOT)/vendor/http-parser/.git" clean -dfx git --git-dir="$(ROOT)/vendor/llhttp/.git" clean -dfx httptools-0.9.0/README.md000066400000000000000000000052701526224275300150530ustar00rootroot00000000000000![Tests](https://github.com/MagicStack/httptools/workflows/Tests/badge.svg) httptools is a Python binding for the nodejs HTTP parser. The package is available on PyPI: `pip install httptools`. # APIs httptools contains two classes `httptools.HttpRequestParser`, `httptools.HttpResponseParser` (fulfilled through [llhttp](https://github.com/nodejs/llhttp)) and a function for parsing URLs `httptools.parse_url` (through [http-parse](https://github.com/nodejs/http-parser) for now). See unittests for examples. ```python class HttpRequestParser: def __init__(self, protocol): """HttpRequestParser protocol -- a Python object with the following methods (all optional): - on_message_begin() - on_url(url: bytes) - on_header(name: bytes, value: bytes) - on_headers_complete() - on_body(body: bytes) - on_message_complete() - on_chunk_header() - on_chunk_complete() - on_status(status: bytes) """ def get_http_version(self) -> str: """Return an HTTP protocol version.""" def should_keep_alive(self) -> bool: """Return ``True`` if keep-alive mode is preferred.""" def should_upgrade(self) -> bool: """Return ``True`` if the parsed request is a valid Upgrade request. The method exposes a flag set just before on_headers_complete. Calling this method earlier will only yield `False`. """ def feed_data(self, data: bytes): """Feed data to the parser. Will eventually trigger callbacks on the ``protocol`` object. On HTTP upgrade, this method will raise an ``HttpParserUpgrade`` exception, with its sole argument set to the offset of the non-HTTP data in ``data``. """ def get_method(self) -> bytes: """Return HTTP request method (GET, HEAD, etc)""" class HttpResponseParser: """Has all methods except ``get_method()`` that HttpRequestParser has.""" def get_status_code(self) -> int: """Return the status code of the HTTP response""" def parse_url(url: bytes): """Parse URL strings into a structured Python object. Returns an instance of ``httptools.URL`` class with the following attributes: - schema: bytes - host: bytes - port: int - path: bytes - query: bytes - fragment: bytes - userinfo: bytes """ ``` # Development 1. Clone this repository with `git clone --recursive git@github.com:MagicStack/httptools.git` 2. Create a virtual environment with Python 3: `python3 -m venv envname` 3. Activate the environment with `source envname/bin/activate` 4. Run `make` and `make test`. # License MIT. httptools-0.9.0/httptools/000077500000000000000000000000001526224275300156305ustar00rootroot00000000000000httptools-0.9.0/httptools/__init__.py000066400000000000000000000013241526224275300177410ustar00rootroot00000000000000from . import parser from .parser import ( HTTPProtocol, HttpRequestParser, HttpResponseParser, HttpParserError, HttpParserCallbackError, HttpParserInvalidStatusError, HttpParserInvalidMethodError, HttpParserInvalidURLError, HttpParserUpgrade, parse_url, ) from ._version import __version__ __all__ = ( "parser", # protocol "HTTPProtocol", # parser "HttpRequestParser", "HttpResponseParser", # errors "HttpParserError", "HttpParserCallbackError", "HttpParserInvalidStatusError", "HttpParserInvalidMethodError", "HttpParserInvalidURLError", "HttpParserUpgrade", # url parser "parse_url", # version "__version__", ) httptools-0.9.0/httptools/_version.py000066400000000000000000000010771526224275300200330ustar00rootroot00000000000000# This file MUST NOT contain anything but the __version__ assignment. # # When making a release, change the value of __version__ # to an appropriate value, and open a pull request against # the correct branch (master if making a new feature release). # The commit message MUST contain a properly formatted release # log, and the commit must be signed. # # The release automation will: build and test the packages for the # supported platforms, publish the packages on PyPI, merge the PR # to the target branch, create a Git tag pointing to the commit. __version__ = '0.9.0' httptools-0.9.0/httptools/parser/000077500000000000000000000000001526224275300171245ustar00rootroot00000000000000httptools-0.9.0/httptools/parser/.gitignore000066400000000000000000000000201526224275300211040ustar00rootroot00000000000000*.so *.html *.c httptools-0.9.0/httptools/parser/__init__.py000066400000000000000000000012441526224275300212360ustar00rootroot00000000000000from .protocol import HTTPProtocol from .parser import HttpRequestParser, HttpResponseParser # NoQA from .errors import ( HttpParserError, HttpParserCallbackError, HttpParserInvalidStatusError, HttpParserInvalidMethodError, HttpParserInvalidURLError, HttpParserUpgrade, ) from .url_parser import parse_url __all__ = ( # protocol "HTTPProtocol", # parser "HttpRequestParser", "HttpResponseParser", # errors "HttpParserError", "HttpParserCallbackError", "HttpParserInvalidStatusError", "HttpParserInvalidMethodError", "HttpParserInvalidURLError", "HttpParserUpgrade", # url_parser "parse_url", ) httptools-0.9.0/httptools/parser/cparser.pxd000066400000000000000000000115611526224275300213040ustar00rootroot00000000000000from libc.stdint cimport int32_t, uint8_t, uint16_t, uint64_t cdef extern from "llhttp.h": struct llhttp__internal_s: int32_t _index void *_span_pos0 void *_span_cb0 int32_t error const char *reason const char *error_pos void *data void *_current uint64_t content_length uint8_t type uint8_t method uint8_t http_major uint8_t http_minor uint8_t header_state uint16_t flags uint8_t upgrade uint16_t status_code uint8_t finish void *settings ctypedef llhttp__internal_s llhttp__internal_t ctypedef llhttp__internal_t llhttp_t ctypedef int (*llhttp_data_cb) (llhttp_t*, const char *at, size_t length) except -1 ctypedef int (*llhttp_cb) (llhttp_t*) except -1 struct llhttp_settings_s: llhttp_cb on_message_begin llhttp_data_cb on_url llhttp_data_cb on_status llhttp_data_cb on_header_field llhttp_data_cb on_header_value llhttp_cb on_headers_complete llhttp_data_cb on_body llhttp_cb on_message_complete llhttp_cb on_chunk_header llhttp_cb on_chunk_complete ctypedef llhttp_settings_s llhttp_settings_t enum llhttp_type: HTTP_BOTH, HTTP_REQUEST, HTTP_RESPONSE ctypedef llhttp_type llhttp_type_t enum llhttp_errno: HPE_OK, HPE_INTERNAL, HPE_STRICT, HPE_LF_EXPECTED, HPE_UNEXPECTED_CONTENT_LENGTH, HPE_CLOSED_CONNECTION, HPE_INVALID_METHOD, HPE_INVALID_URL, HPE_INVALID_CONSTANT, HPE_INVALID_VERSION, HPE_INVALID_HEADER_TOKEN, HPE_INVALID_CONTENT_LENGTH, HPE_INVALID_CHUNK_SIZE, HPE_INVALID_STATUS, HPE_INVALID_EOF_STATE, HPE_INVALID_TRANSFER_ENCODING, HPE_CB_MESSAGE_BEGIN, HPE_CB_HEADERS_COMPLETE, HPE_CB_MESSAGE_COMPLETE, HPE_CB_CHUNK_HEADER, HPE_CB_CHUNK_COMPLETE, HPE_PAUSED, HPE_PAUSED_UPGRADE, HPE_USER ctypedef llhttp_errno llhttp_errno_t enum llhttp_flags: F_CONNECTION_KEEP_ALIVE, F_CONNECTION_CLOSE, F_CONNECTION_UPGRADE, F_CHUNKED, F_UPGRADE, F_CONTENT_LENGTH, F_SKIPBODY, F_TRAILING, F_LENIENT, F_TRANSFER_ENCODING ctypedef llhttp_flags llhttp_flags_t enum llhttp_method: HTTP_DELETE, HTTP_GET, HTTP_HEAD, HTTP_POST, HTTP_PUT, HTTP_CONNECT, HTTP_OPTIONS, HTTP_TRACE, HTTP_COPY, HTTP_LOCK, HTTP_MKCOL, HTTP_MOVE, HTTP_PROPFIND, HTTP_PROPPATCH, HTTP_SEARCH, HTTP_UNLOCK, HTTP_BIND, HTTP_REBIND, HTTP_UNBIND, HTTP_ACL, HTTP_REPORT, HTTP_MKACTIVITY, HTTP_CHECKOUT, HTTP_MERGE, HTTP_MSEARCH, HTTP_NOTIFY, HTTP_SUBSCRIBE, HTTP_UNSUBSCRIBE, HTTP_PATCH, HTTP_PURGE, HTTP_MKCALENDAR, HTTP_LINK, HTTP_UNLINK, HTTP_SOURCE, HTTP_PRI, HTTP_DESCRIBE, HTTP_ANNOUNCE, HTTP_SETUP, HTTP_PLAY, HTTP_PAUSE, HTTP_TEARDOWN, HTTP_GET_PARAMETER, HTTP_SET_PARAMETER, HTTP_REDIRECT, HTTP_RECORD, HTTP_FLUSH ctypedef llhttp_method llhttp_method_t void llhttp_init(llhttp_t* parser, llhttp_type_t type, const llhttp_settings_t* settings) void llhttp_settings_init(llhttp_settings_t* settings) llhttp_errno_t llhttp_execute(llhttp_t* parser, const char* data, size_t len) void llhttp_resume_after_upgrade(llhttp_t* parser) int llhttp_should_keep_alive(const llhttp_t* parser) const char* llhttp_get_error_pos(const llhttp_t* parser) const char* llhttp_get_error_reason(const llhttp_t* parser) const char* llhttp_method_name(llhttp_method_t method) void llhttp_set_error_reason(llhttp_t* parser, const char* reason); void llhttp_set_lenient_headers(llhttp_t* parser, bint enabled); void llhttp_set_lenient_chunked_length(llhttp_t* parser, bint enabled); void llhttp_set_lenient_keep_alive(llhttp_t* parser, bint enabled); void llhttp_set_lenient_transfer_encoding(llhttp_t* parser, bint enabled); void llhttp_set_lenient_version(llhttp_t* parser, bint enabled); void llhttp_set_lenient_data_after_close(llhttp_t* parser, bint enabled); void llhttp_set_lenient_optional_lf_after_cr(llhttp_t* parser, bint enabled); void llhttp_set_lenient_optional_cr_before_lf(llhttp_t* parser, bint enabled); void llhttp_set_lenient_optional_crlf_after_chunk(llhttp_t* parser, bint enabled); void llhttp_set_lenient_spaces_after_chunk_size(llhttp_t* parser, bint enabled); httptools-0.9.0/httptools/parser/errors.py000066400000000000000000000010661526224275300210150ustar00rootroot00000000000000__all__ = ('HttpParserError', 'HttpParserCallbackError', 'HttpParserInvalidStatusError', 'HttpParserInvalidMethodError', 'HttpParserInvalidURLError', 'HttpParserUpgrade') class HttpParserError(Exception): pass class HttpParserCallbackError(HttpParserError): pass class HttpParserInvalidStatusError(HttpParserError): pass class HttpParserInvalidMethodError(HttpParserError): pass class HttpParserInvalidURLError(HttpParserError): pass class HttpParserUpgrade(Exception): pass httptools-0.9.0/httptools/parser/parser.pyi000066400000000000000000000041551526224275300211500ustar00rootroot00000000000000from array import array from .protocol import HTTPProtocol class HttpParser: def __init__(self, protocol: HTTPProtocol | object) -> None: """The HTTP parser. Args: protocol (HTTPProtocol): Callback interface for the parser. """ def set_dangerous_leniencies( self, lenient_headers: bool | None = None, lenient_chunked_length: bool | None = None, lenient_keep_alive: bool | None = None, lenient_transfer_encoding: bool | None = None, lenient_version: bool | None = None, lenient_data_after_close: bool | None = None, lenient_optional_lf_after_cr: bool | None = None, lenient_optional_cr_before_lf: bool | None = None, lenient_optional_crlf_after_chunk: bool | None = None, lenient_spaces_after_chunk_size: bool | None = None, ) -> None: """Set dangerous leniencies for the parser.""" def get_http_version(self) -> str: """Retrieve the HTTP protocol version e.g. "1.1".""" def should_keep_alive(self) -> bool: """Return `True` if keep-alive mode is preferred.""" def should_upgrade(self) -> bool: """Return `True` if the parsed request is a valid Upgrade request. The method exposes a flag set just before on_headers_complete. Calling this method earlier will only yield `False`.""" def feed_data(self, data: bytes | bytearray | memoryview | array[int]) -> None: """Feed data to the parser. Will eventually trigger callbacks on the ``protocol`` object. On HTTP upgrade, this method will raise an ``HttpParserUpgrade`` exception, with its sole argument set to the offset of the non-HTTP data in ``data``. """ class HttpRequestParser(HttpParser): """Used for parsing http requests from the server side.""" def get_method(self) -> bytes: """Retrieve the HTTP method of the request.""" class HttpResponseParser(HttpParser): """Used for parsing http responses from the client side.""" def get_status_code(self) -> int: """Retrieve the status code of the HTTP response.""" httptools-0.9.0/httptools/parser/parser.pyx000066400000000000000000000354441526224275300211740ustar00rootroot00000000000000#cython: language_level=3 from __future__ import print_function from typing import Optional from cpython.mem cimport PyMem_Malloc, PyMem_Free from cpython cimport PyObject_GetBuffer, PyBuffer_Release, PyBUF_SIMPLE, \ Py_buffer, PyBytes_AsString from .python cimport PyMemoryView_Check, PyMemoryView_GET_BUFFER from .errors import (HttpParserError, HttpParserCallbackError, HttpParserInvalidStatusError, HttpParserInvalidMethodError, HttpParserInvalidURLError, HttpParserUpgrade) cimport cython from . cimport cparser __all__ = ('HttpRequestParser', 'HttpResponseParser') @cython.internal cdef class HttpParser: cdef: cparser.llhttp_t* _cparser cparser.llhttp_settings_t* _csettings bytes _current_header_name bytes _current_header_value _proto_on_url, _proto_on_status, _proto_on_body, \ _proto_on_header, _proto_on_headers_complete, \ _proto_on_message_complete, _proto_on_chunk_header, \ _proto_on_chunk_complete, _proto_on_message_begin object _last_error Py_buffer py_buf def __cinit__(self): self._cparser = \ PyMem_Malloc(sizeof(cparser.llhttp_t)) if self._cparser is NULL: raise MemoryError() self._csettings = \ PyMem_Malloc(sizeof(cparser.llhttp_settings_t)) if self._csettings is NULL: raise MemoryError() def __dealloc__(self): PyMem_Free(self._cparser) PyMem_Free(self._csettings) cdef _init(self, protocol, cparser.llhttp_type_t mode): cparser.llhttp_settings_init(self._csettings) cparser.llhttp_init(self._cparser, mode, self._csettings) self._cparser.data = self self._current_header_name = None self._current_header_value = None self._proto_on_header = getattr(protocol, 'on_header', None) if self._proto_on_header is not None: self._csettings.on_header_field = cb_on_header_field self._csettings.on_header_value = cb_on_header_value self._proto_on_headers_complete = getattr( protocol, 'on_headers_complete', None) self._csettings.on_headers_complete = cb_on_headers_complete self._proto_on_body = getattr(protocol, 'on_body', None) if self._proto_on_body is not None: self._csettings.on_body = cb_on_body self._proto_on_message_begin = getattr( protocol, 'on_message_begin', None) if self._proto_on_message_begin is not None: self._csettings.on_message_begin = cb_on_message_begin self._proto_on_message_complete = getattr( protocol, 'on_message_complete', None) if self._proto_on_message_complete is not None: self._csettings.on_message_complete = cb_on_message_complete self._proto_on_chunk_header = getattr( protocol, 'on_chunk_header', None) self._csettings.on_chunk_header = cb_on_chunk_header self._proto_on_chunk_complete = getattr( protocol, 'on_chunk_complete', None) self._csettings.on_chunk_complete = cb_on_chunk_complete self._last_error = None cdef _maybe_call_on_header(self): if self._current_header_value is not None: current_header_name = self._current_header_name current_header_value = self._current_header_value self._current_header_name = self._current_header_value = None if self._proto_on_header is not None: self._proto_on_header(current_header_name, current_header_value) cdef _on_header_field(self, bytes field): self._maybe_call_on_header() if self._current_header_name is None: self._current_header_name = field else: self._current_header_name += field cdef _on_header_value(self, bytes val): if self._current_header_value is None: self._current_header_value = val else: # This is unlikely, as mostly HTTP headers are one-line self._current_header_value += val cdef _on_headers_complete(self): self._maybe_call_on_header() if self._proto_on_headers_complete is not None: self._proto_on_headers_complete() cdef _on_chunk_header(self): if (self._current_header_value is not None or self._current_header_name is not None): raise HttpParserError('invalid headers state') if self._proto_on_chunk_header is not None: self._proto_on_chunk_header() cdef _on_chunk_complete(self): self._maybe_call_on_header() if self._proto_on_chunk_complete is not None: self._proto_on_chunk_complete() ### Public API ### def set_dangerous_leniencies( self, lenient_headers: Optional[bool] = None, lenient_chunked_length: Optional[bool] = None, lenient_keep_alive: Optional[bool] = None, lenient_transfer_encoding: Optional[bool] = None, lenient_version: Optional[bool] = None, lenient_data_after_close: Optional[bool] = None, lenient_optional_lf_after_cr: Optional[bool] = None, lenient_optional_cr_before_lf: Optional[bool] = None, lenient_optional_crlf_after_chunk: Optional[bool] = None, lenient_spaces_after_chunk_size: Optional[bool] = None, ): cdef cparser.llhttp_t* parser = self._cparser if lenient_headers is not None: cparser.llhttp_set_lenient_headers( parser, lenient_headers) if lenient_chunked_length is not None: cparser.llhttp_set_lenient_chunked_length( parser, lenient_chunked_length) if lenient_keep_alive is not None: cparser.llhttp_set_lenient_keep_alive( parser, lenient_keep_alive) if lenient_transfer_encoding is not None: cparser.llhttp_set_lenient_transfer_encoding( parser, lenient_transfer_encoding) if lenient_version is not None: cparser.llhttp_set_lenient_version( parser, lenient_version) if lenient_data_after_close is not None: cparser.llhttp_set_lenient_data_after_close( parser, lenient_data_after_close) if lenient_optional_lf_after_cr is not None: cparser.llhttp_set_lenient_optional_lf_after_cr( parser, lenient_optional_lf_after_cr) if lenient_optional_cr_before_lf is not None: cparser.llhttp_set_lenient_optional_cr_before_lf( parser, lenient_optional_cr_before_lf) if lenient_optional_crlf_after_chunk is not None: cparser.llhttp_set_lenient_optional_crlf_after_chunk( parser, lenient_optional_crlf_after_chunk) if lenient_spaces_after_chunk_size is not None: cparser.llhttp_set_lenient_spaces_after_chunk_size( parser, lenient_spaces_after_chunk_size) def get_http_version(self): cdef cparser.llhttp_t* parser = self._cparser return '{}.{}'.format(parser.http_major, parser.http_minor) def should_keep_alive(self): return bool(cparser.llhttp_should_keep_alive(self._cparser)) def should_upgrade(self): cdef cparser.llhttp_t* parser = self._cparser return bool(parser.upgrade) def feed_data(self, data): cdef: size_t data_len cparser.llhttp_errno_t err Py_buffer *buf bint owning_buf = False const char* err_pos if PyMemoryView_Check(data): buf = PyMemoryView_GET_BUFFER(data) data_len = buf.len err = cparser.llhttp_execute( self._cparser, buf.buf, data_len) else: buf = &self.py_buf PyObject_GetBuffer(data, buf, PyBUF_SIMPLE) owning_buf = True data_len = buf.len err = cparser.llhttp_execute( self._cparser, buf.buf, data_len) try: if self._cparser.upgrade == 1 and err == cparser.HPE_PAUSED_UPGRADE: err_pos = cparser.llhttp_get_error_pos(self._cparser) # Immediately free the parser from "error" state, simulating # http-parser behavior here because 1) we never had the API to # allow users manually "resume after upgrade", and 2) the use # case for resuming parsing is very rare. cparser.llhttp_resume_after_upgrade(self._cparser) # The err_pos here is specific for the input buf. So if we ever # switch to the llhttp behavior (re-raise HttpParserUpgrade for # successive calls to feed_data() until resume_after_upgrade is # called), we have to store the result and keep our own state. raise HttpParserUpgrade(err_pos - buf.buf) finally: if owning_buf: PyBuffer_Release(buf) if err != cparser.HPE_OK: ex = parser_error_from_errno( self._cparser, self._cparser.error) if isinstance(ex, HttpParserCallbackError): if self._last_error is not None: ex.__context__ = self._last_error self._last_error = None raise ex cdef class HttpRequestParser(HttpParser): def __init__(self, protocol): self._init(protocol, cparser.HTTP_REQUEST) self._proto_on_url = getattr(protocol, 'on_url', None) if self._proto_on_url is not None: self._csettings.on_url = cb_on_url def get_method(self): cdef cparser.llhttp_t* parser = self._cparser return cparser.llhttp_method_name( parser.method) cdef class HttpResponseParser(HttpParser): def __init__(self, protocol): self._init(protocol, cparser.HTTP_RESPONSE) self._proto_on_status = getattr(protocol, 'on_status', None) if self._proto_on_status is not None: self._csettings.on_status = cb_on_status def get_status_code(self): cdef cparser.llhttp_t* parser = self._cparser return parser.status_code cdef int cb_on_message_begin(cparser.llhttp_t* parser) except -1: cdef HttpParser pyparser = parser.data try: pyparser._proto_on_message_begin() except BaseException as ex: pyparser._last_error = ex return -1 else: return 0 cdef int cb_on_url(cparser.llhttp_t* parser, const char *at, size_t length) except -1: cdef HttpParser pyparser = parser.data try: pyparser._proto_on_url(at[:length]) except BaseException as ex: cparser.llhttp_set_error_reason(parser, "`on_url` callback error") pyparser._last_error = ex return cparser.HPE_USER else: return 0 cdef int cb_on_status(cparser.llhttp_t* parser, const char *at, size_t length) except -1: cdef HttpParser pyparser = parser.data try: pyparser._proto_on_status(at[:length]) except BaseException as ex: cparser.llhttp_set_error_reason(parser, "`on_status` callback error") pyparser._last_error = ex return cparser.HPE_USER else: return 0 cdef int cb_on_header_field(cparser.llhttp_t* parser, const char *at, size_t length) except -1: cdef HttpParser pyparser = parser.data try: pyparser._on_header_field(at[:length]) except BaseException as ex: cparser.llhttp_set_error_reason(parser, "`on_header_field` callback error") pyparser._last_error = ex return cparser.HPE_USER else: return 0 cdef int cb_on_header_value(cparser.llhttp_t* parser, const char *at, size_t length) except -1: cdef HttpParser pyparser = parser.data try: pyparser._on_header_value(at[:length]) except BaseException as ex: cparser.llhttp_set_error_reason(parser, "`on_header_value` callback error") pyparser._last_error = ex return cparser.HPE_USER else: return 0 cdef int cb_on_headers_complete(cparser.llhttp_t* parser) except -1: cdef HttpParser pyparser = parser.data try: pyparser._on_headers_complete() except BaseException as ex: pyparser._last_error = ex return -1 else: if pyparser._cparser.upgrade: return 1 else: return 0 cdef int cb_on_body(cparser.llhttp_t* parser, const char *at, size_t length) except -1: cdef HttpParser pyparser = parser.data try: pyparser._proto_on_body(at[:length]) except BaseException as ex: cparser.llhttp_set_error_reason(parser, "`on_body` callback error") pyparser._last_error = ex return cparser.HPE_USER else: return 0 cdef int cb_on_message_complete(cparser.llhttp_t* parser) except -1: cdef HttpParser pyparser = parser.data try: pyparser._proto_on_message_complete() except BaseException as ex: pyparser._last_error = ex return -1 else: return 0 cdef int cb_on_chunk_header(cparser.llhttp_t* parser) except -1: cdef HttpParser pyparser = parser.data try: pyparser._on_chunk_header() except BaseException as ex: pyparser._last_error = ex return -1 else: return 0 cdef int cb_on_chunk_complete(cparser.llhttp_t* parser) except -1: cdef HttpParser pyparser = parser.data try: pyparser._on_chunk_complete() except BaseException as ex: pyparser._last_error = ex return -1 else: return 0 cdef parser_error_from_errno(cparser.llhttp_t* parser, cparser.llhttp_errno_t errno): cdef bytes reason = cparser.llhttp_get_error_reason(parser) if errno in (cparser.HPE_CB_MESSAGE_BEGIN, cparser.HPE_CB_HEADERS_COMPLETE, cparser.HPE_CB_MESSAGE_COMPLETE, cparser.HPE_CB_CHUNK_HEADER, cparser.HPE_CB_CHUNK_COMPLETE, cparser.HPE_USER): cls = HttpParserCallbackError elif errno == cparser.HPE_INVALID_STATUS: cls = HttpParserInvalidStatusError elif errno == cparser.HPE_INVALID_METHOD: cls = HttpParserInvalidMethodError elif errno == cparser.HPE_INVALID_URL: cls = HttpParserInvalidURLError else: cls = HttpParserError return cls(reason.decode('latin-1')) httptools-0.9.0/httptools/parser/protocol.py000066400000000000000000000011121526224275300213320ustar00rootroot00000000000000from typing import Protocol class HTTPProtocol(Protocol): """Used for providing static type-checking when parsing through the http protocol""" def on_message_begin(self) -> None: ... def on_url(self, url: bytes) -> None: ... def on_header(self, name: bytes, value: bytes) -> None: ... def on_headers_complete(self) -> None: ... def on_body(self, body: bytes) -> None: ... def on_message_complete(self) -> None: ... def on_chunk_header(self) -> None: ... def on_chunk_complete(self) -> None: ... def on_status(self, status: bytes) -> None: ... httptools-0.9.0/httptools/parser/python.pxd000066400000000000000000000002121526224275300211550ustar00rootroot00000000000000cimport cpython cdef extern from "Python.h": cpython.Py_buffer* PyMemoryView_GET_BUFFER(object) bint PyMemoryView_Check(object) httptools-0.9.0/httptools/parser/url_cparser.pxd000066400000000000000000000014131526224275300221610ustar00rootroot00000000000000from libc.stdint cimport uint16_t cdef extern from "http_parser.h": # URL Parser enum http_parser_url_fields: UF_SCHEMA = 0, UF_HOST = 1, UF_PORT = 2, UF_PATH = 3, UF_QUERY = 4, UF_FRAGMENT = 5, UF_USERINFO = 6, UF_MAX = 7 struct http_parser_url_field_data: uint16_t off uint16_t len struct http_parser_url: uint16_t field_set uint16_t port http_parser_url_field_data[UF_MAX] field_data void http_parser_url_init(http_parser_url *u) int http_parser_parse_url(const char *buf, size_t buflen, int is_connect, http_parser_url *u) httptools-0.9.0/httptools/parser/url_parser.pyi000066400000000000000000000004441526224275300220270ustar00rootroot00000000000000from array import array class URL: schema: bytes host: bytes port: int path: bytes query: bytes fragment: bytes userinfo: bytes def parse_url(url: bytes | bytearray | memoryview | array[int]) -> URL: """Parse a URL string into a structured Python object.""" httptools-0.9.0/httptools/parser/url_parser.pyx000066400000000000000000000102131526224275300220410ustar00rootroot00000000000000#cython: language_level=3 from __future__ import print_function from cpython.mem cimport PyMem_Malloc, PyMem_Free from cpython cimport PyObject_GetBuffer, PyBuffer_Release, PyBUF_SIMPLE, \ Py_buffer from .errors import HttpParserInvalidURLError cimport cython from . cimport url_cparser as uparser __all__ = ('parse_url',) DEF MAX_URL_LENGTH = (1 << 16) - 1 @cython.freelist(250) cdef class URL: cdef readonly bytes schema cdef readonly bytes host cdef readonly object port cdef readonly bytes path cdef readonly bytes query cdef readonly bytes fragment cdef readonly bytes userinfo def __cinit__(self, bytes schema, bytes host, object port, bytes path, bytes query, bytes fragment, bytes userinfo): self.schema = schema self.host = host self.port = port self.path = path self.query = query self.fragment = fragment self.userinfo = userinfo def __repr__(self): return ('' .format(self.schema, self.host, self.port, self.path, self.query, self.fragment, self.userinfo)) def parse_url(url): cdef: Py_buffer py_buf char* buf_data uparser.http_parser_url* parsed int res bytes schema = None bytes host = None object port = None bytes path = None bytes query = None bytes fragment = None bytes userinfo = None object result = None int off int ln parsed = \ PyMem_Malloc(sizeof(uparser.http_parser_url)) uparser.http_parser_url_init(parsed) PyObject_GetBuffer(url, &py_buf, PyBUF_SIMPLE) try: if py_buf.len > MAX_URL_LENGTH: # http_parser stores URL field offsets/lengths as uint16_t, # so URLs longer than this will cause silent truncation. # See https://github.com/MagicStack/httptools/issues/142 raise HttpParserInvalidURLError( "url is too long: url length of {} bytes exceeds the " "maximum of {} bytes".format(py_buf.len, MAX_URL_LENGTH)) buf_data = py_buf.buf res = uparser.http_parser_parse_url(buf_data, py_buf.len, 0, parsed) if res == 0: if parsed.field_set & (1 << uparser.UF_SCHEMA): off = parsed.field_data[uparser.UF_SCHEMA].off ln = parsed.field_data[uparser.UF_SCHEMA].len schema = buf_data[off:off+ln] if parsed.field_set & (1 << uparser.UF_HOST): off = parsed.field_data[uparser.UF_HOST].off ln = parsed.field_data[uparser.UF_HOST].len host = buf_data[off:off+ln] if parsed.field_set & (1 << uparser.UF_PORT): port = parsed.port if parsed.field_set & (1 << uparser.UF_PATH): off = parsed.field_data[uparser.UF_PATH].off ln = parsed.field_data[uparser.UF_PATH].len path = buf_data[off:off+ln] if parsed.field_set & (1 << uparser.UF_QUERY): off = parsed.field_data[uparser.UF_QUERY].off ln = parsed.field_data[uparser.UF_QUERY].len query = buf_data[off:off+ln] if parsed.field_set & (1 << uparser.UF_FRAGMENT): off = parsed.field_data[uparser.UF_FRAGMENT].off ln = parsed.field_data[uparser.UF_FRAGMENT].len fragment = buf_data[off:off+ln] if parsed.field_set & (1 << uparser.UF_USERINFO): off = parsed.field_data[uparser.UF_USERINFO].off ln = parsed.field_data[uparser.UF_USERINFO].len userinfo = buf_data[off:off+ln] return URL(schema, host, port, path, query, fragment, userinfo) else: raise HttpParserInvalidURLError("invalid url {!r}".format(url)) finally: PyBuffer_Release(&py_buf) PyMem_Free(parsed) httptools-0.9.0/httptools/py.typed000066400000000000000000000000001526224275300173150ustar00rootroot00000000000000httptools-0.9.0/pyproject.toml000066400000000000000000000026631526224275300165130ustar00rootroot00000000000000[build-system] build-backend = "setuptools.build_meta" requires = ["setuptools>=80.9.0,<=82.0.1"] [project] name = "httptools" dynamic = ["version"] classifiers = [ "Intended Audience :: Developers", "Programming Language :: Python :: 3", "Operating System :: POSIX", "Operating System :: MacOS :: MacOS X", "Environment :: Web Environment", "Development Status :: 5 - Production/Stable", ] requires-python = ">=3.9" authors = [ {name = "Yury Selivanov", email="yury@magic.io"}, ] license = "MIT" license-files = [ "LICENSE", "vendor/http-parser/LICENSE-MIT", "vendor/llhttp/LICENSE", ] description = "A collection of framework independent HTTP protocol utils." readme = "README.md" [project.urls] Homepage = "https://github.com/MagicStack/httptools" [dependency-groups] dev = [ # type checker "pyright >= 1.1.406", # tests "pytest", # build "setuptools", "wheel" ] [tool.cibuildwheel] build-frontend = "build" skip = "pp*" test-command = "python {project}/tests/__init__.py" [tool.cibuildwheel.macos] environment = {MACOSX_DEPLOYMENT_TARGET = "11"} [tool.cibuildwheel.windows] test-command = "python {project}\\tests\\__init__.py" [tool.pyright] pythonVersion = "3.8" typeCheckingMode = "strict" reportMissingTypeStubs = false reportUnnecessaryIsInstance = false reportUnnecessaryTypeIgnoreComment = true reportMissingModuleSource = false include = ["httptools"] exclude = ["tests"] httptools-0.9.0/pytest.ini000066400000000000000000000001661526224275300156240ustar00rootroot00000000000000[pytest] addopts = --capture=no --assert=plain --strict-markers --tb=native --import-mode=importlib testpaths = tests httptools-0.9.0/setup.py000066400000000000000000000135471526224275300153140ustar00rootroot00000000000000import sys import os.path import pathlib from setuptools import setup, Extension from setuptools.command.build_ext import build_ext as build_ext CFLAGS = ['-O2', '-DCYTHON_FREETHREADING_COMPATIBLE=1'] ROOT = pathlib.Path(__file__).parent CYTHON_DEPENDENCY = 'Cython>=3.1.0' class httptools_build_ext(build_ext): user_options = build_ext.user_options + [ ('cython-always', None, 'run cythonize() even if .c files are present'), ('cython-annotate', None, 'Produce a colorized HTML version of the Cython source.'), ('cython-directives=', None, 'Cythion compiler directives'), ('use-system-llhttp', None, 'Use the system provided llhttp, instead of the bundled one'), ('use-system-http-parser', None, 'Use the system provided http-parser, instead of the bundled one'), ] boolean_options = build_ext.boolean_options + [ 'cython-always', 'cython-annotate', 'use-system-llhttp', 'use-system-http-parser', ] def initialize_options(self): # initialize_options() may be called multiple times on the # same command object, so make sure not to override previously # set options. if getattr(self, '_initialized', False): return super().initialize_options() self.use_system_llhttp = False self.use_system_http_parser = False self.cython_always = False self.cython_annotate = None self.cython_directives = None if 'editable_wheel' in sys.argv: self.inplace = True def finalize_options(self): # finalize_options() may be called multiple times on the # same command object, so make sure not to override previously # set options. if getattr(self, '_initialized', False): return need_cythonize = self.cython_always cfiles = {} for extension in self.distribution.ext_modules: for i, sfile in enumerate(extension.sources): if sfile.endswith('.pyx'): prefix, ext = os.path.splitext(sfile) cfile = prefix + '.c' if os.path.exists(cfile) and not self.cython_always: extension.sources[i] = cfile else: if os.path.exists(cfile): cfiles[cfile] = os.path.getmtime(cfile) else: cfiles[cfile] = 0 need_cythonize = True if need_cythonize: try: import Cython except ImportError: import setuptools.build_meta raise setuptools.build_meta.SetupRequirementsError([CYTHON_DEPENDENCY]) from Cython.Build import cythonize directives = {} if self.cython_directives: for directive in self.cython_directives.split(','): k, _, v = directive.partition('=') if v.lower() == 'false': v = False if v.lower() == 'true': v = True directives[k] = v self.distribution.ext_modules[:] = cythonize( self.distribution.ext_modules, compiler_directives=directives, annotate=self.cython_annotate) super().finalize_options() self._initialized = True def build_extensions(self): mod_parser, mod_url_parser = self.distribution.ext_modules if self.use_system_llhttp: mod_parser.libraries.append('llhttp') if sys.platform == 'darwin' and \ os.path.exists('/opt/local/include'): # Support macports on Mac OS X. mod_parser.include_dirs.append('/opt/local/include') else: mod_parser.include_dirs.append( str(ROOT / 'vendor' / 'llhttp' / 'include')) mod_parser.include_dirs.append( str(ROOT / 'vendor' / 'llhttp' / 'src')) mod_parser.sources.append('vendor/llhttp/src/api.c') mod_parser.sources.append('vendor/llhttp/src/http.c') mod_parser.sources.append('vendor/llhttp/src/llhttp.c') if self.use_system_http_parser: mod_url_parser.libraries.append('http_parser') if sys.platform == 'darwin' and \ os.path.exists('/opt/local/include'): # Support macports on Mac OS X. mod_url_parser.include_dirs.append('/opt/local/include') else: mod_url_parser.include_dirs.append( str(ROOT / 'vendor' / 'http-parser')) mod_url_parser.sources.append( 'vendor/http-parser/http_parser.c') super().build_extensions() with open(str(ROOT / 'httptools' / '_version.py')) as f: for line in f: if line.startswith('__version__ ='): _, _, version = line.partition('=') VERSION = version.strip(" \n'\"") break else: raise RuntimeError( 'unable to read the version from httptools/_version.py') setup( version=VERSION, platforms=['macOS', 'POSIX', 'Windows'], zip_safe=False, packages=['httptools', 'httptools.parser'], cmdclass={ 'build_ext': httptools_build_ext, }, ext_modules=[ Extension( "httptools.parser.parser", sources=[ "httptools/parser/parser.pyx", ], extra_compile_args=CFLAGS, ), Extension( "httptools.parser.url_parser", sources=[ "httptools/parser/url_parser.pyx", ], extra_compile_args=CFLAGS, ), ], include_package_data=True, exclude_package_data={"": ["*.c", "*.h"]}, ) httptools-0.9.0/tests/000077500000000000000000000000001526224275300147325ustar00rootroot00000000000000httptools-0.9.0/tests/__init__.py000066400000000000000000000006021526224275300170410ustar00rootroot00000000000000import os.path import sys import unittest import unittest.runner def suite(): test_loader = unittest.TestLoader() test_suite = test_loader.discover( os.path.dirname(__file__), pattern='test_*.py') return test_suite if __name__ == '__main__': runner = unittest.runner.TextTestRunner() result = runner.run(suite()) sys.exit(not result.wasSuccessful()) httptools-0.9.0/tests/test_parser.py000066400000000000000000000527341526224275300176520ustar00rootroot00000000000000import httptools import unittest from unittest import mock RESPONSE1_HEAD = b'''HTTP/1.1 200 OK Date: Mon, 23 May 2005 22:38:34 GMT Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux) Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT ETag: "3f80f-1b6-3e1cb03b" Content-Type: text/html; charset=UTF-8 Content-Length: 130 Accept-Ranges: bytes Connection: close '''.replace(b'\n', b'\r\n') RESPONSE1_SPACES_IN_HEAD = b'''HTTP/1.1 200 OK Date: Mon, 23 May 2005 22:38:34 GMT Server: Apache/1.3.3.7 (Unix) (Red-Hat/Linux) Last-Modified: Wed, 08 Jan 2003 23:11:55 GMT ETag: "3f80f-1b6-3e1cb03b" Content-Type: text/html; charset=UTF-8 Content-Length: 130 Accept-Ranges: bytes Connection: close '''.replace(b'\n', b'\r\n') RESPONSE1_BODY = b''' An Example Page Hello World, this is a very simple HTML document. ''' CHUNKED_REQUEST1_1 = b'''POST /test.php?a=b+c HTTP/1.1\r User-Agent: Fooo\r Host: bar\r Transfer-Encoding: chunked\r \r 5\r\nhello\r\n6\r\n world\r\n''' CHUNKED_REQUEST1_2 = b'''0\r\nVary: *\r\nUser-Agent: spam\r\n\r\n''' CHUNKED_REQUEST1_3 = b'''POST /test.php?a=b+c HTTP/1.1\r User-Agent: Fooo\r Host: bar\r Transfer-Encoding: chunked\r \r b\r\n+\xce\xcfM\xb5MI,I\x04\x00\r\n0\r\n\r\n''' UPGRADE_REQUEST1 = b'''GET /demo HTTP/1.1\r Host: example.com\r Connection: Upgrade\r Sec-WebSocket-Key2: 12998 5 Y3 1 .P00\r Sec-WebSocket-Protocol: sample\r Upgrade: WebSocket\r Sec-WebSocket-Key1: 4 @1 46546xW%0l 1 5\r Origin: http://example.com\r \r Hot diggity dogg''' UPGRADE_RESPONSE1 = b'''HTTP/1.1 101 Switching Protocols UPGRADE: websocket SEC-WEBSOCKET-ACCEPT: rVg+XakFNFOxk3ZH0lzrZBmg0aU= TRANSFER-ENCODING: chunked CONNECTION: upgrade DATE: Sat, 07 May 2016 23:44:32 GMT SERVER: Python/3.4 aiohttp/1.0.3 data'''.replace(b'\n', b'\r\n') class TestResponseParser(unittest.TestCase): def test_parser_response_1(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ p = httptools.HttpResponseParser(m) p.feed_data(memoryview(RESPONSE1_HEAD)) self.assertEqual(p.get_http_version(), '1.1') self.assertEqual(p.get_status_code(), 200) m.on_status.assert_called_once_with(b'OK') m.on_headers_complete.assert_called_once_with() self.assertEqual(m.on_header.call_count, 8) self.assertEqual(len(headers), 8) self.assertEqual(headers.get(b'Connection'), b'close') self.assertEqual(headers.get(b'Content-Type'), b'text/html; charset=UTF-8') self.assertFalse(m.on_body.called) p.feed_data(bytearray(RESPONSE1_BODY)) m.on_body.assert_called_once_with(RESPONSE1_BODY) m.on_message_complete.assert_called_once_with() self.assertFalse(m.on_url.called) self.assertFalse(m.on_chunk_header.called) self.assertFalse(m.on_chunk_complete.called) def test_parser_response_1b(self): p = httptools.HttpResponseParser(None) with self.assertRaisesRegex( httptools.HttpParserError, 'Expected HTTP/'): p.feed_data(b'12123123') def test_parser_response_leninent_headers_1(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ p = httptools.HttpResponseParser(m) with self.assertRaisesRegex( httptools.HttpParserError, "whitespace after header value", ): p.feed_data(memoryview(RESPONSE1_SPACES_IN_HEAD)) def test_parser_response_leninent_headers_2(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ p = httptools.HttpResponseParser(m) p.set_dangerous_leniencies(lenient_headers=True) p.feed_data(memoryview(RESPONSE1_SPACES_IN_HEAD)) self.assertEqual(p.get_http_version(), '1.1') self.assertEqual(p.get_status_code(), 200) m.on_status.assert_called_once_with(b'OK') m.on_headers_complete.assert_called_once_with() self.assertEqual(m.on_header.call_count, 8) self.assertEqual(len(headers), 8) self.assertEqual(headers.get(b'Connection'), b'close') self.assertEqual(headers.get(b'Content-Type'), b'text/html; charset=UTF-8') self.assertFalse(m.on_body.called) p.feed_data(bytearray(RESPONSE1_BODY)) m.on_body.assert_called_once_with(RESPONSE1_BODY) m.on_message_complete.assert_called_once_with() self.assertFalse(m.on_url.called) self.assertFalse(m.on_chunk_header.called) self.assertFalse(m.on_chunk_complete.called) def test_parser_response_2(self): with self.assertRaisesRegex(TypeError, 'a bytes-like object'): httptools.HttpResponseParser(None).feed_data('') def test_parser_response_3(self): callbacks = {'on_header', 'on_headers_complete', 'on_body', 'on_message_complete'} for cbname in callbacks: with self.subTest('{} callback fails correctly'.format(cbname)): with self.assertRaisesRegex(httptools.HttpParserCallbackError, 'callback error'): m = mock.Mock() getattr(m, cbname).side_effect = Exception() p = httptools.HttpResponseParser(m) p.feed_data(RESPONSE1_HEAD + RESPONSE1_BODY) def test_parser_response_4(self): p = httptools.HttpResponseParser(None) with self.assertRaises(httptools.HttpParserInvalidStatusError): p.feed_data(b'HTTP/1.1 1299 FOOSPAM\r\n') def test_parser_response_5(self): m = mock.Mock() m.on_status = None m.on_header = None m.on_body = None m.on_headers_complete = None m.on_chunk_header = None m.on_chunk_complete = None p = httptools.HttpResponseParser(m) p.feed_data(RESPONSE1_HEAD) p.feed_data(RESPONSE1_BODY) m.on_message_complete.assert_called_once_with() def test_parser_response_cb_on_status_1(self): class Error(Exception): pass m = mock.Mock() m.on_status.side_effect = Error() p = httptools.HttpResponseParser(m) try: p.feed_data(RESPONSE1_HEAD + RESPONSE1_BODY) except httptools.HttpParserCallbackError as ex: self.assertIsInstance(ex.__context__, Error) else: self.fail('HttpParserCallbackError was not raised') def test_parser_response_cb_on_body_1(self): class Error(Exception): pass m = mock.Mock() m.on_body.side_effect = Error() p = httptools.HttpResponseParser(m) try: p.feed_data(RESPONSE1_HEAD + RESPONSE1_BODY) except httptools.HttpParserCallbackError as ex: self.assertIsInstance(ex.__context__, Error) else: self.fail('HttpParserCallbackError was not raised') def test_parser_response_cb_on_message_complete_1(self): class Error(Exception): pass m = mock.Mock() m.on_message_complete.side_effect = Error() p = httptools.HttpResponseParser(m) try: p.feed_data(RESPONSE1_HEAD + RESPONSE1_BODY) except httptools.HttpParserCallbackError as ex: self.assertIsInstance(ex.__context__, Error) else: self.fail('HttpParserCallbackError was not raised') def test_parser_upgrade_response_1(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ p = httptools.HttpResponseParser(m) try: p.feed_data(UPGRADE_RESPONSE1) except httptools.HttpParserUpgrade as ex: offset = ex.args[0] else: self.fail('HttpParserUpgrade was not raised') self.assertEqual(UPGRADE_RESPONSE1[offset:], b'data') self.assertEqual(p.get_http_version(), '1.1') self.assertEqual(p.get_status_code(), 101) m.on_status.assert_called_once_with(b'Switching Protocols') m.on_headers_complete.assert_called_once_with() self.assertEqual(m.on_header.call_count, 6) self.assertEqual(len(headers), 6) m.on_message_complete.assert_called_once_with() class TestRequestParser(unittest.TestCase): def test_parser_request_chunked_1(self): m = mock.Mock() p = httptools.HttpRequestParser(m) p.feed_data(CHUNKED_REQUEST1_1) self.assertEqual(p.get_method(), b'POST') m.on_message_begin.assert_called_once_with() m.on_url.assert_called_once_with(b'/test.php?a=b+c') self.assertEqual(p.get_http_version(), '1.1') m.on_header.assert_called_with(b'Transfer-Encoding', b'chunked') m.on_chunk_header.assert_called_with() m.on_chunk_complete.assert_called_with() self.assertFalse(m.on_message_complete.called) m.on_message_begin.assert_called_once_with() m.reset_mock() p.feed_data(CHUNKED_REQUEST1_2) m.on_chunk_header.assert_called_with() m.on_chunk_complete.assert_called_with() m.on_header.assert_called_with(b'User-Agent', b'spam') self.assertEqual(m.on_header.call_count, 2) self.assertFalse(m.on_message_begin.called) m.on_message_complete.assert_called_once_with() def test_parser_request_chunked_2(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ m.on_url = None m.on_body = None m.on_headers_complete = None m.on_chunk_header = None m.on_chunk_complete = None p = httptools.HttpRequestParser(m) p.feed_data(CHUNKED_REQUEST1_1) p.feed_data(CHUNKED_REQUEST1_2) self.assertEqual( headers, {b'User-Agent': b'spam', b'Transfer-Encoding': b'chunked', b'Host': b'bar', b'Vary': b'*'}) def test_parser_request_chunked_cb_error_1(self): class Error(Exception): pass m = mock.Mock() m.on_chunk_header.side_effect = Error() p = httptools.HttpRequestParser(m) try: p.feed_data(CHUNKED_REQUEST1_1) except httptools.HttpParserCallbackError as ex: self.assertIsInstance(ex.__context__, Error) else: self.fail('HttpParserCallbackError was not raised') def test_parser_request_chunked_cb_error_2(self): class Error(Exception): pass m = mock.Mock() m.on_chunk_complete.side_effect = Error() p = httptools.HttpRequestParser(m) try: p.feed_data(CHUNKED_REQUEST1_1) except httptools.HttpParserCallbackError as ex: self.assertIsInstance(ex.__context__, Error) else: self.fail('HttpParserCallbackError was not raised') def test_parser_request_chunked_3(self): m = mock.Mock() p = httptools.HttpRequestParser(m) p.feed_data(CHUNKED_REQUEST1_3) self.assertEqual(p.get_method(), b'POST') m.on_url.assert_called_once_with(b'/test.php?a=b+c') self.assertEqual(p.get_http_version(), '1.1') m.on_header.assert_called_with(b'Transfer-Encoding', b'chunked') m.on_chunk_header.assert_called_with() m.on_chunk_complete.assert_called_with() self.assertTrue(m.on_message_complete.called) def test_parser_request_upgrade_1(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ p = httptools.HttpRequestParser(m) try: p.feed_data(UPGRADE_REQUEST1) except httptools.HttpParserUpgrade as ex: offset = ex.args[0] else: self.fail('HttpParserUpgrade was not raised') self.assertEqual(UPGRADE_REQUEST1[offset:], b'Hot diggity dogg') self.assertEqual(headers, { b'Sec-WebSocket-Key2': b'12998 5 Y3 1 .P00', b'Sec-WebSocket-Key1': b'4 @1 46546xW%0l 1 5', b'Connection': b'Upgrade', b'Origin': b'http://example.com', b'Sec-WebSocket-Protocol': b'sample', b'Host': b'example.com', b'Upgrade': b'WebSocket'}) # The parser can be used again for further parsing - this is a legacy # behavior from the time we were still using http-parser. p.feed_data(CHUNKED_REQUEST1_1) self.assertEqual(p.get_method(), b'POST') def test_parser_request_upgrade_flag(self): class Protocol: def __init__(self): self.parser = httptools.HttpRequestParser(self) def on_url(self, url): assert self.parser.should_upgrade() is False def on_headers_complete(self): assert self.parser.should_upgrade() is True def on_message_complete(self): assert self.parser.should_upgrade() is True protocol = Protocol() try: protocol.parser.feed_data(UPGRADE_REQUEST1) except httptools.HttpParserUpgrade: # Raise as usual. pass else: self.fail('HttpParserUpgrade was not raised') def test_parser_request_error_in_on_header(self): class Error(Exception): pass m = mock.Mock() m.on_header.side_effect = Error() p = httptools.HttpRequestParser(m) try: p.feed_data(UPGRADE_REQUEST1) except httptools.HttpParserCallbackError as ex: self.assertIsInstance(ex.__context__, Error) else: self.fail('HttpParserCallbackError was not raised') def test_parser_request_error_in_on_message_begin(self): class Error(Exception): pass m = mock.Mock() m.on_message_begin.side_effect = Error() p = httptools.HttpRequestParser(m) try: p.feed_data(UPGRADE_REQUEST1) except httptools.HttpParserCallbackError as ex: self.assertIsInstance(ex.__context__, Error) else: self.fail('HttpParserCallbackError was not raised') def test_parser_request_error_in_cb_on_url(self): class Error(Exception): pass m = mock.Mock() m.on_url.side_effect = Error() p = httptools.HttpRequestParser(m) try: p.feed_data(UPGRADE_REQUEST1) except httptools.HttpParserCallbackError as ex: self.assertIsInstance(ex.__context__, Error) else: self.fail('HttpParserCallbackError was not raised') def test_parser_request_error_in_cb_on_headers_complete(self): class Error(Exception): pass m = mock.Mock() m.on_headers_complete.side_effect = Error() p = httptools.HttpRequestParser(m) try: p.feed_data(UPGRADE_REQUEST1) except httptools.HttpParserCallbackError as ex: self.assertIsInstance(ex.__context__, Error) else: self.fail('HttpParserCallbackError was not raised') def test_parser_request_2(self): p = httptools.HttpRequestParser(None) with self.assertRaises(httptools.HttpParserInvalidMethodError): p.feed_data(b'SPAM /test.php?a=b+c HTTP/1.1') def test_parser_request_3(self): p = httptools.HttpRequestParser(None) with self.assertRaises(httptools.HttpParserInvalidURLError): p.feed_data(b'POST HTTP/1.1') def test_parser_request_4(self): p = httptools.HttpRequestParser(None) with self.assertRaisesRegex(TypeError, 'a bytes-like object'): p.feed_data('POST HTTP/1.1') def test_parser_request_fragmented(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ p = httptools.HttpRequestParser(m) REQUEST = ( b'PUT / HTTP/1.1\r\nHost: localhost:1234\r\nContent-Type: text/pl', b'ain; charset=utf-8\r\nX-Empty-Header: \r\nConnection: close\r\n', b'Content-Length: 10\r\n\r\n1234567890', ) p.feed_data(REQUEST[0]) m.on_message_begin.assert_called_once_with() m.on_url.assert_called_once_with(b'/') self.assertEqual(headers, {b'Host': b'localhost:1234'}) p.feed_data(REQUEST[1]) self.assertEqual( headers, {b'Host': b'localhost:1234', b'Content-Type': b'text/plain; charset=utf-8', b'X-Empty-Header': b''}) p.feed_data(REQUEST[2]) self.assertEqual( headers, {b'Host': b'localhost:1234', b'Content-Type': b'text/plain; charset=utf-8', b'X-Empty-Header': b'', b'Connection': b'close', b'Content-Length': b'10'}) m.on_message_complete.assert_called_once_with() def test_parser_request_fragmented_header(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ p = httptools.HttpRequestParser(m) REQUEST = ( b'PUT / HTTP/1.1\r\nHost: localhost:1234\r\nContent-', b'Type: text/plain; charset=utf-8\r\n\r\n', ) p.feed_data(REQUEST[0]) m.on_message_begin.assert_called_once_with() m.on_url.assert_called_once_with(b'/') self.assertEqual(headers, {b'Host': b'localhost:1234'}) p.feed_data(REQUEST[1]) self.assertEqual( headers, {b'Host': b'localhost:1234', b'Content-Type': b'text/plain; charset=utf-8'}) def test_parser_request_fragmented_value(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ p = httptools.HttpRequestParser(m) REQUEST = ( b'PUT / HTTP/1.1\r\nHost: localhost:1234\r\nContent-Type:', b' text/pla', b'in; chars', b'et=utf-8\r\n\r\n', ) p.feed_data(REQUEST[0]) m.on_message_begin.assert_called_once_with() m.on_url.assert_called_once_with(b'/') self.assertEqual(headers, {b'Host': b'localhost:1234'}) p.feed_data(REQUEST[1]) p.feed_data(REQUEST[2]) p.feed_data(REQUEST[3]) self.assertEqual( headers, {b'Host': b'localhost:1234', b'Content-Type': b'text/plain; charset=utf-8'}) def test_parser_request_fragmented_bytes(self): m = mock.Mock() headers = {} m.on_header.side_effect = headers.__setitem__ p = httptools.HttpRequestParser(m) REQUEST = \ b'PUT / HTTP/1.1\r\nHost: localhost:1234\r\nContent-' \ b'Type: text/plain; charset=utf-8\r\n\r\n' step = 1 for i in range(0, len(REQUEST), step): p.feed_data(REQUEST[i:i+step]) self.assertEqual( headers, {b'Host': b'localhost:1234', b'Content-Type': b'text/plain; charset=utf-8'}) class TestUrlParser(unittest.TestCase): def parse(self, url:bytes): parsed = httptools.parse_url(url) return (parsed.schema, parsed.host, parsed.port, parsed.path, parsed.query, parsed.fragment, parsed.userinfo) def test_parser_url_1(self): self.assertEqual( self.parse(b'dsf://aaa/b/c?aa#123'), (b'dsf', b'aaa', None, b'/b/c', b'aa', b'123', None)) self.assertEqual( self.parse(b'dsf://i:n@aaa:88/b/c?aa#123'), (b'dsf', b'aaa', 88, b'/b/c', b'aa', b'123', b'i:n')) self.assertEqual( self.parse(b'////'), (None, None, None, b'////', None, None, None)) self.assertEqual( self.parse(b'////1/1?a=b&c[]=d&c[]=z'), (None, None, None, b'////1/1', b'a=b&c[]=d&c[]=z', None, None)) self.assertEqual( self.parse(b'/////?#123'), (None, None, None, b'/////', None, b'123', None)) self.assertEqual( self.parse(b'/a/b/c?b=1&'), (None, None, None, b'/a/b/c', b'b=1&', None, None)) def test_parser_url_2(self): with self.assertRaises(httptools.HttpParserInvalidURLError): self.parse(b'') def test_parser_url_3(self): with self.assertRaises(httptools.HttpParserInvalidURLError): self.parse(b' ') def test_parser_url_4(self): with self.assertRaises(httptools.HttpParserInvalidURLError): self.parse(b':///1') def test_parser_url_5(self): self.assertEqual( self.parse(b'http://[1:2::3:4]:67/'), (b'http', b'1:2::3:4', 67, b'/', None, None, None)) def test_parser_url_6(self): self.assertEqual( self.parse(bytearray(b'/')), (None, None, None, b'/', None, None, None)) def test_parser_url_7(self): url = httptools.parse_url(b'/') with self.assertRaisesRegex(AttributeError, 'not writable'): url.port = 0 def test_parser_url_8(self): with self.assertRaises(TypeError): httptools.parse_url(None) def test_parser_url_9(self): with self.assertRaisesRegex(httptools.HttpParserInvalidURLError, r'a\\x00aa'): self.parse(b'dsf://a\x00aa') def test_parser_url_10(self): with self.assertRaisesRegex(TypeError, 'a bytes-like object'): self.parse('dsf://aaa') def test_parser_url_too_long(self): url = b'http://h/' + b'a' * 65535 with self.assertRaisesRegex(httptools.HttpParserInvalidURLError, 'url is too long'): self.parse(url) httptools-0.9.0/vendor/000077500000000000000000000000001526224275300150655ustar00rootroot00000000000000httptools-0.9.0/vendor/http-parser/000077500000000000000000000000001526224275300173365ustar00rootroot00000000000000httptools-0.9.0/vendor/llhttp/000077500000000000000000000000001526224275300163745ustar00rootroot00000000000000