pax_global_header00006660000000000000000000000064152426003560014515gustar00rootroot0000000000000052 comment=52625fd9e93eb8ba472e50a640b61675cd2448c5 benoitc-gunicorn_h1c-52625fd/000077500000000000000000000000001524260035600160545ustar00rootroot00000000000000benoitc-gunicorn_h1c-52625fd/.github/000077500000000000000000000000001524260035600174145ustar00rootroot00000000000000benoitc-gunicorn_h1c-52625fd/.github/workflows/000077500000000000000000000000001524260035600214515ustar00rootroot00000000000000benoitc-gunicorn_h1c-52625fd/.github/workflows/ci.yml000066400000000000000000000021201524260035600225620ustar00rootroot00000000000000name: CI on: push: branches: [main] pull_request: branches: [main] jobs: test: runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest] python-version: ["3.13", "3.14"] steps: - uses: actions/checkout@v4 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} allow-prereleases: true - name: Install dependencies run: | python -m pip install --upgrade pip pip install pytest pytest-benchmark pip install -e . - name: Run tests run: pytest tests/ -v lint: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up Python uses: actions/setup-python@v5 with: python-version: "3.12" - name: Install ruff run: pip install ruff - name: Check formatting run: ruff format --check . - name: Check linting run: ruff check . benoitc-gunicorn_h1c-52625fd/.gitignore000066400000000000000000000010731524260035600200450ustar00rootroot00000000000000# Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so *.dylib *.pyd # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ *.egg-info/ .installed.cfg *.egg # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover .hypothesis/ .pytest_cache/ # Environments .env .venv env/ venv/ ENV/ # IDE .idea/ .vscode/ *.swp *.swo # OS .DS_Store Thumbs.db benoitc-gunicorn_h1c-52625fd/CHANGELOG.md000066400000000000000000000241531524260035600176720ustar00rootroot00000000000000# Changelog All notable changes to this project will be documented in this file. ## [0.6.9] - 2026-08-23 ### Fixed - Parse the request body on messages that carry an `Upgrade` header. Any `Upgrade` value used to suppress body parsing: the message completed at the end of the headers, `on_body` never fired, and the payload was left in `remaining()` as raw bytes, still chunk-encoded when the request was chunked. An upgrade request is an ordinary HTTP/1.1 request, since the protocol switch happens only after the server answers 101 (RFC 9110 section 7.8) and an h2c client must send any payload in full before its first HTTP/2 frame (RFC 7540 section 3.2). On gunicorn's ASGI worker this lost the body silently, and made this parser disagree with the pure-Python one about the same request. ### Changed - `remaining()` no longer includes the body on those messages. A caller that worked around the old behaviour by taking the declared body off the front of the tail should drop that. - An upgrade request with an unterminated chunked body now stays incomplete instead of completing at the headers, exactly as a normal chunked request does. Messages carrying `Upgrade` that declare no body are unaffected, which covers the WebSocket handshake and a bare h2c upgrade. `CONNECT` is unaffected: it has no body framing, so everything after its headers stays in `remaining()`. ## [0.6.8] - 2026-08-20 ### Fixed - Bound the tail retained after a completed message. 0.6.7 kept everything fed to an already-complete parser, so a caller that feeds every socket read without checking `is_complete` let a client grow the parser buffer without limit between the end of a message and `reset()`. The tail now stops at `limit_remaining` (64 KB by default, `0` disables the cap). ### Added - `limit_remaining` constructor argument and the `remaining_truncated` property, true once the tail has hit the cap and bytes have been dropped. `feed()` never raises on overflow: it discards the excess, exactly as every version before 0.6.7 discarded the whole tail. Callers that ignore `remaining()` are unaffected apart from the memory bound. ## [0.6.7] - 2026-08-20 ### Added - `H1CProtocol.remaining()` returns the bytes fed after a completed message. Callers that feed whole socket reads can now recover what a client pipelined behind the request: the HTTP/2 preface after an `Upgrade: h2c` (which unblocks cleartext HTTP/2 on gunicorn's ASGI worker), a WebSocket frame sent straight after the handshake, or a second HTTP/1 request. - Returns `b""` until `is_complete` is true, since every byte fed to an in-progress message belongs to it. - Measured from the end of the message, so past the body or the terminating chunk and its trailers, not the end of the headers. - A tail split over several `feed()` calls is accumulated. - `reset()` drops the tail; read it first and feed it back to parse a pipelined request. ### Changed - Bytes fed to an already-complete parser are retained for `remaining()` instead of being discarded. Stop feeding a completed parser, or drain it with `remaining()` and `reset()`, rather than pumping a whole connection through `feed()`. ## [0.6.6] - 2026-08-17 ### Security - Reject repeated singleton header fields per RFC 9110 section 5.3. `Host` and `Content-Type` now raise `InvalidHeader` on a repeat, as `Content-Length` already did. A repeat cannot be merged into a list, so the message is ambiguous and gunicorn, a cache, and an upstream proxy may each resolve it differently. Duplicate `Host` is a routing- and cache-confusion vector. ### Changed - The singleton field set is now stated in one place in `pico_utils.h`, shared by `parse_request`, `parse_request_fast`, `parse_to_wsgi_environ`, `parse_to_asgi_scope`, and `H1CProtocol.feed()`. Response parsing is unaffected. - Repeated `Transfer-Encoding` remains accepted; it is not a singleton field and its repeats are handled by the framing checks. ### Compatibility - This turns requests that previously parsed into 400 Bad Request. That is intended, and it matches what the pure-Python parser does, but it is a behavior change for anyone sitting behind a proxy that duplicates `Host`. Such a proxy is itself non-conformant, and that ambiguity is the reason for the change. Duplicate list-valued fields (`Accept`, `Via`, `X-Forwarded-For`, `Set-Cookie`, and so on) are unaffected. ## [0.6.5] - 2026-04-20 ### Fixed - Reject Content-Length list form per RFC 9112 section 6.3 (e.g. `Content-Length: 42, 42`) ## [0.6.4] - 2026-04-19 ### Fixed - Reject forbidden trailer field-names per RFC 9110 section 6.5.1 - Enforce request-target form and method pairing per RFC 9112 - Reject control characters in header field-value per RFC 9110 section 5.5 ### Changed - Add H1CProtocol limit validation tests - Replace `assert False` with `pytest.fail()` to fix B011 lint error - Ruff format `tests/test_protocol.py` ## [0.6.3] - 2026-03-26 ### Added - `InvalidChunkExtension` exception for chunk extension validation errors - Validation to reject chunk extensions containing bare CR bytes (RFC 9112 compliance) ### Security - Prevents potential request smuggling via malformed chunk extensions ## [0.6.2] - 2026-03-26 ### Added - `asgi_headers` property on HttpRequest and H1CProtocol for ASGI-compliant lowercase header names - `pico_create_header_tuple_lowercase()` helper function in pico_utils.h ### Fixed - Starlette/FastAPI compatibility when using fast parser with ASGI worker (fixes #2) ## [0.6.1] - 2026-03-26 ### Added - GitHub Actions CI workflow (Python 3.13/3.14 on Linux and macOS) - Ruff configuration for linting and formatting ## [0.6.0] - 2026-03-26 ### Added - `finish()` method on H1CProtocol for EOF handling in chunked encoding - Header framing validation to prevent ambiguous requests - Reject duplicate Content-Length headers - Reject Content-Length with Transfer-Encoding (CL+TE conflict) - Reject chunked encoding in HTTP/1.0 - Reject stacked chunked encoding - Reject unknown Transfer-Encoding values - Strict chunk size parsing (reject whitespace, empty sizes) - `pico_validate_headers_full()` function for complete header validation - 30 new tests for header validation ### Changed - All parsers (parse_request, parse_request_fast, H1CProtocol) now perform framing validation ## [0.5.1] - 2026-03-22 ### Fixed - Header size calculation now includes CRLF to match Python parser behavior ## [0.5.0] - 2026-03-22 ### Fixed - Non-HTTP protocols (FTP/1.1, RTSP/1.0) now raise `InvalidHTTPVersion` instead of generic `ParseError` - Bare CR/LF in header values now detected and raise `InvalidHeader` in error analysis - Header size limit calculation now includes ": " separator (name + 2 + value) ## [0.4.1] - 2026-03-22 ### Improved - Specific exception analysis when `phr_parse_request()` returns -1 - Now analyzes the buffer to identify the exact cause of parse failure - Raises `InvalidRequestMethod` for invalid/lowercase method characters - Raises `InvalidHTTPVersion` for unsupported HTTP versions (e.g., HTTP/2.0, HTTP/0.9) - Raises `InvalidHeaderName` for invalid header name characters (e.g., space) - Raises `InvalidHeader` for NUL characters in header values - Falls back to generic `ParseError` only when specific cause cannot be determined ## [0.4.0] - 2026-03-22 ### Added - Limit enforcement matching gunicorn's Python parser behavior - `limit_request_line`: Maximum request line length (default 8190) - `limit_request_fields`: Maximum number of headers (default 100) - `limit_request_field_size`: Maximum header size (default 8190) - Gunicorn compatibility flags - `permit_unconventional_http_method`: Allow lowercase methods, short methods, and `#` character - `permit_unconventional_http_version`: Allow HTTP versions other than 1.0/1.1 - Specific exception types for validation errors (all inherit from `ParseError`) - `LimitRequestLine`: Request line exceeds limit - `LimitRequestHeaders`: Too many headers or header too large - `InvalidRequestMethod`: Invalid method characters or format - `InvalidHTTPVersion`: Invalid HTTP version - `InvalidHeaderName`: Invalid header name characters - `InvalidHeader`: Invalid header value (NUL, CR, LF) - Validation functions in `pico_utils.h` for RFC 9110 token character validation ### Changed - All parsing functions now validate requests by default - `parse_request`, `parse_request_fast`, `parse_to_wsgi_environ`, `parse_to_asgi_scope`, and `H1CProtocol` accept new limit/flag parameters ## [0.3.0] - 2026-03-21 ### Changed - Refactored shared utilities into `pico_utils.h` to reduce code duplication - Extracted common header parsing functions used across all three parser modules - Improved code maintainability with ~200 lines of duplicate code removed ## [0.2.0] - 2026-03-21 ### Added - `H1CProtocol` - Callback-based HTTP/1.1 parser for asyncio integration - Zero-copy, synchronous parsing in `data_received()` callbacks - Support for Content-Length and chunked transfer encoding body parsing - Callbacks: `on_message_begin`, `on_url`, `on_header`, `on_headers_complete`, `on_body`, `on_message_complete` - Properties: `method`, `path`, `http_version`, `headers`, `content_length`, `is_chunked`, `should_keep_alive`, `should_upgrade`, `is_complete` - `reset()` method for keepalive connection reuse - ~4.7M req/s when reusing parser (7% faster than pull-based API) - ~3x faster than pull-based API for incremental parsing ## [0.1.0] - 2026-03-21 ### Added - Initial release - `parse_request` - Basic HTTP request parsing returning dict - `parse_response` - HTTP response parsing returning dict - `parse_headers` - Header-only parsing - `parse_request_fast` - Zero-copy request parsing with lazy `HttpRequest` object - `parse_request_raw` - Ultra-fast parsing returning raw buffer offsets - `parse_to_wsgi_environ` - WSGI environ dict generation - `parse_to_asgi_scope` - ASGI scope dict generation - SIMD-optimized parsing via picohttpparser (SSE4.2 on x86, NEON on ARM) - Incremental parsing support with `last_len` parameter - Common header extraction (Content-Length, Transfer-Encoding, Connection) - Support for up to 256 headers per request - Python 3.9+ support benoitc-gunicorn_h1c-52625fd/LICENSE000066400000000000000000000045461524260035600170720ustar00rootroot00000000000000MIT License Copyright (c) 2026 Benoit Chesneau Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. --- The vendored picohttpparser sources in src/gunicorn_h1c/picohttpparser/ are dual-licensed under the MIT License or the Perl license by their authors. They are used here under the MIT License: Copyright (c) 2009-2014 Kazuho Oku, Tokuhiro Matsuno, Daisuke Murase, Shigeo Mitsunari Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. benoitc-gunicorn_h1c-52625fd/MANIFEST.in000066400000000000000000000002231524260035600176070ustar00rootroot00000000000000include LICENSE include README.md include pyproject.toml include setup.py recursive-include src/gunicorn_h1c *.c *.h recursive-include tests *.py benoitc-gunicorn_h1c-52625fd/README.md000066400000000000000000000371631524260035600173450ustar00rootroot00000000000000# gunicorn_h1c Fast HTTP/1.1 parser for Gunicorn using [picohttpparser](https://github.com/h2o/picohttpparser). ## Features - SIMD-optimized parsing (SSE4.2 on x86, NEON on ARM) - Zero-copy request parsing with lazy Python object creation - Callback-based parser for asyncio integration (H1CProtocol) - Common header extraction (Content-Length, Transfer-Encoding, Connection) - Incremental parsing support - Chunked transfer encoding support - WSGI environ and ASGI scope generation - Limit enforcement matching gunicorn's Python parser - Specific exception types for validation errors - Python 3.9+ ## Installation ```bash pip install gunicorn_h1c ``` ## Usage ### Basic Parsing ```python from gunicorn_h1c import parse_request data = b"GET /path?query=1 HTTP/1.1\r\nHost: localhost\r\nContent-Length: 0\r\n\r\n" result = parse_request(data) print(result["method"]) # b'GET' print(result["path"]) # b'/path?query=1' print(result["minor_version"]) # 1 (HTTP/1.1) print(result["headers"]) # [(b'Host', b'localhost'), (b'Content-Length', b'0')] print(result["consumed"]) # 67 (bytes consumed) ``` ### Fast Parsing (Zero-Copy) ```python from gunicorn_h1c import parse_request_fast data = ( b"POST /api HTTP/1.1\r\nContent-Length: 100\r\nTransfer-Encoding: chunked\r\n\r\n" ) req = parse_request_fast(data) # Properties are created lazily - only when accessed print(req.method) # b'POST' print(req.path) # b'/api' print(req.consumed) # bytes consumed # Common headers extracted during parse (no Python overhead) print(req.content_length) # 100 print(req.has_chunked) # True print(req.connection_close) # -1 (not set), 0 (keep-alive), 1 (close) # Header lookup (case-insensitive) print(req.get_header("content-length")) # b'100' ``` ### Response Parsing ```python from gunicorn_h1c import parse_response data = b"HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 13\r\n\r\n" result = parse_response(data) print(result["status"]) # 200 print(result["message"]) # b'OK' print(result["minor_version"]) # 1 print(result["headers"]) # [(b'Content-Type', b'text/html'), ...] print(result["consumed"]) # bytes consumed ``` ### Header-Only Parsing ```python from gunicorn_h1c import parse_headers data = b"Content-Type: text/html\r\nContent-Length: 100\r\n\r\n" headers = parse_headers(data) print(headers) # [(b'Content-Type', b'text/html'), (b'Content-Length', b'100')] ``` ### WSGI Environ Creation ```python from gunicorn_h1c import parse_to_wsgi_environ data = b"GET /path?foo=bar HTTP/1.1\r\nHost: example.com\r\nContent-Type: text/plain\r\n\r\n" environ = parse_to_wsgi_environ( data, server=("example.com", 80), client=("192.168.1.1", 54321), url_scheme="https" ) print(environ["REQUEST_METHOD"]) # 'GET' print(environ["PATH_INFO"]) # '/path' print(environ["QUERY_STRING"]) # 'foo=bar' print(environ["SERVER_NAME"]) # 'example.com' print(environ["SERVER_PORT"]) # '80' print(environ["REMOTE_ADDR"]) # '192.168.1.1' print(environ["HTTP_HOST"]) # 'example.com' print(environ["CONTENT_TYPE"]) # 'text/plain' print(environ["wsgi.url_scheme"]) # 'https' print(environ["_consumed"]) # bytes consumed ``` ### ASGI Scope Creation ```python from gunicorn_h1c import parse_to_asgi_scope data = b"POST /api HTTP/1.1\r\nHost: example.com\r\nContent-Length: 50\r\n\r\n" scope = parse_to_asgi_scope( data, server=("example.com", 443), client=("10.0.0.1", 12345), scheme="https", root_path="/v1", ) print(scope["type"]) # 'http' print(scope["asgi"]) # {'version': '3.0', 'spec_version': '2.4'} print(scope["http_version"]) # '1.1' print(scope["method"]) # 'POST' print(scope["scheme"]) # 'https' print(scope["path"]) # '/api' print(scope["raw_path"]) # b'/api' print(scope["query_string"]) # b'' print(scope["root_path"]) # '/v1' print(scope["headers"]) # [(b'host', b'example.com'), ...] print(scope["server"]) # ('example.com', 443) print(scope["client"]) # ('10.0.0.1', 12345) print(scope["_consumed"]) # bytes consumed ``` ### Callback-Based Protocol Parser (asyncio) For asyncio servers, `H1CProtocol` provides a callback-based API that enables zero-copy, synchronous parsing in `data_received()`: ```python import asyncio from gunicorn_h1c import H1CProtocol class MyProtocol(asyncio.Protocol): def connection_made(self, transport): self.transport = transport self.parser = H1CProtocol( on_headers_complete=self._on_headers, on_body=self._on_body, on_message_complete=self._on_complete, ) def data_received(self, data): try: self.parser.feed(data) except ParseError as e: self.transport.close() def _on_headers(self): # Build ASGI scope or process headers method = self.parser.method # b'GET' path = self.parser.path # b'/path' headers = self.parser.headers # [(b'Host', b'localhost'), ...] # Return True to skip body parsing (e.g., for HEAD requests) return self.parser.method == b"HEAD" def _on_body(self, chunk): # Process body chunk (zero-copy) pass def _on_complete(self): # Request complete, send response self.parser.reset() # Reuse for next request (keep-alive) ``` ### Limit Enforcement All parsing functions enforce limits matching gunicorn's Python parser: ```python from gunicorn_h1c import parse_request, LimitRequestLine, LimitRequestHeaders # Default limits: request_line=8190, fields=100, field_size=8190 try: result = parse_request(data) except LimitRequestLine: # Request line too long pass except LimitRequestHeaders: # Too many headers or header too large pass # Custom limits result = parse_request( data, limit_request_line=4096, # Max request line length limit_request_fields=50, # Max number of headers limit_request_field_size=4096, # Max header size (name + value) ) # Allow unconventional methods (lowercase, short, etc.) result = parse_request( b"get / HTTP/1.1\r\n\r\n", permit_unconventional_http_method=True ) ``` ### Incremental Parsing ```python from gunicorn_h1c import parse_request, IncompleteError buffer = b"GET / HTTP/1.1\r\n" last_len = 0 while True: try: result = parse_request(buffer, last_len=last_len) break # Complete request except IncompleteError: last_len = len(buffer) buffer += read_more_data() # Get more data ``` ### Raw Parsing (Maximum Speed) For scenarios requiring maximum performance, `parse_request_raw` returns offsets into the original buffer: ```python from gunicorn_h1c import parse_request_raw data = b"GET /path HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request_raw(data) # Returns: (method_offset, method_len, path_offset, path_len, # minor_version, header_count, consumed, header_data) ( method_offset, method_len, path_offset, path_len, version, header_count, consumed, header_data, ) = result method = data[method_offset : method_offset + method_len] # b'GET' path = data[path_offset : path_offset + path_len] # b'/path' ``` ## Performance Benchmarks on Apple M4 Pro (single thread): | Parser | Requests/sec | |--------|-------------| | gunicorn_h1c (fast) | ~2,500,000 | | gunicorn_h1c (H1CProtocol, reused) | ~4,700,000 | | httptools | ~2,200,000 | | Pure Python | ~150,000 | **H1CProtocol Performance:** - Simple GET: ~4.7M req/s (209ns/op) when reusing parser - Incremental parsing: ~3x faster than pull-based API with buffer + retry - Body parsing: ~3.0M req/s for chunked, ~3.7M req/s for Content-Length ## API Reference ### Request Parsing #### `parse_request(data, last_len=0, ...) -> dict` Parse HTTP request, returns dict with: - `method`: bytes - `path`: bytes - `minor_version`: int (0 or 1) - `headers`: list of (name, value) tuples - `consumed`: int (bytes consumed) **Optional parameters:** - `limit_request_line`: int (default 8190) - Maximum request line length - `limit_request_fields`: int (default 100) - Maximum number of headers - `limit_request_field_size`: int (default 8190) - Maximum header size - `permit_unconventional_http_method`: bool (default False) - Allow lowercase/short methods - `permit_unconventional_http_version`: bool (default False) - Allow non-1.0/1.1 versions #### `parse_request_fast(data, last_len=0, ...) -> HttpRequest` Parse HTTP request with zero-copy optimization, returns `HttpRequest` object with: - `method`: bytes (lazy) - `path`: bytes (lazy) - `minor_version`: int - `headers`: tuple of (name, value) tuples (lazy) - `consumed`: int - `header_count`: int - `content_length`: int (-1 if not set) - `has_chunked`: bool - `connection_close`: int (-1=unset, 0=keep-alive, 1=close) - `get_header(name)`: bytes or None (case-insensitive lookup) **Optional parameters:** Same as `parse_request()`. #### `parse_request_raw(data, last_len=0) -> tuple` Ultra-fast parsing returning raw offsets: - `method_offset`: int - `method_len`: int - `path_offset`: int - `path_len`: int - `minor_version`: int - `header_count`: int - `consumed`: int - `header_data`: bytes (packed header offsets) ### Callback-Based Protocol Parser #### `H1CProtocol` Callback-based HTTP/1.1 parser for asyncio integration. **Constructor:** ```python H1CProtocol( on_message_begin=None, # () -> None on_url=None, # (url: bytes) -> None on_header=None, # (name: bytes, value: bytes) -> None on_headers_complete=None, # () -> bool (return True to skip body) on_body=None, # (chunk: bytes) -> None on_message_complete=None, # () -> None limit_request_line=8190, # Maximum request line length limit_request_fields=100, # Maximum number of headers limit_request_field_size=8190, # Maximum header size permit_unconventional_http_method=False, permit_unconventional_http_version=False, limit_remaining=65536, # Max bytes retained after a message (0 = unlimited) ) ``` **Methods:** - `feed(data: bytes) -> None`: Feed data to parser. Callbacks fire synchronously. - `reset() -> None`: Reset parser for next request (keepalive). Drops any tail; see below. - `get_header(name: bytes) -> bytes | None`: Case-insensitive header lookup. - `remaining() -> bytes`: Bytes fed after the completed message. See below. **Properties (valid after on_headers_complete):** - `method`: bytes - HTTP method (GET, POST, etc.) - `path`: bytes - Request path including query string - `http_version`: tuple[int, int] - HTTP version as (major, minor) - `headers`: list[tuple[bytes, bytes]] - List of (name, value) tuples - `content_length`: int | None - Content-Length value or None - `is_chunked`: bool - True if Transfer-Encoding: chunked - `should_keep_alive`: bool - True if connection should be kept alive - `should_upgrade`: bool - True if Upgrade header present - `is_complete`: bool - True if message parsing is complete - `remaining_truncated`: bool - True if the tail hit `limit_remaining` #### Unconsumed bytes after a message When you feed a whole socket read, bytes belonging to whatever follows the request can arrive in the same call: the HTTP/2 preface after an `Upgrade: h2c`, a WebSocket frame sent straight after the handshake, or a pipelined second request. `remaining()` hands those back. ```python p = H1CProtocol() p.feed( b"GET / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Connection: Upgrade, HTTP2-Settings\r\n" b"HTTP2-Settings: AAMAAABkAAQAoAAAAAIAAAAA\r\n\r\n" b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n" ) p.is_complete # True p.should_upgrade # True p.remaining() # b'PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n' - hand to the HTTP/2 connection ``` Notes: - `remaining()` returns `b""` until `is_complete` is true. While a message is still being parsed every byte fed so far belongs to it, so there is nothing to report. - The tail is measured from the end of the message, not the end of the headers. For a body, that means past the last body byte or past the terminating chunk and its trailers. This holds for a request carrying an `Upgrade` header too: its body is parsed and delivered through `on_body` like any other, so the tail starts after it. `CONNECT` is different, since it carries no body and everything after its headers is tunnel traffic. - A tail split over several `feed()` calls is accumulated, so you can keep feeding until you are ready to read it. - `reset()` drops the tail. For pipelining, read it first and feed it back: ```python tail = p.remaining() p.reset() p.feed(tail) # parsed as the next request ``` - Once a message is complete, everything you feed is retained until you call `reset()`, up to `limit_remaining` (64 KB by default). Past that the excess is dropped and `remaining_truncated` becomes true. `feed()` never raises on overflow, so a caller that keeps feeding a completed parser sees bounded memory rather than an error. - Set `limit_remaining=0` for no cap. Only do that if you drain the tail promptly: a client that streams into a completed parser can then grow it without bound. ### Response Parsing #### `parse_response(data, last_len=0) -> dict` Parse HTTP response, returns dict with: - `status`: int (status code) - `message`: bytes (status message) - `minor_version`: int (0 or 1) - `headers`: list of (name, value) tuples - `consumed`: int (bytes consumed) ### Header Parsing #### `parse_headers(data, last_len=0) -> list` Parse HTTP headers only, returns list of (name, value) tuples. ### WSGI/ASGI Support #### `parse_to_wsgi_environ(data, server=None, client=None, url_scheme="http", ...) -> dict` Parse HTTP request and build WSGI environ dict. Parameters: - `data`: Raw HTTP request bytes - `server`: (host, port) tuple for SERVER_NAME/SERVER_PORT - `client`: (addr, port) tuple for REMOTE_ADDR/REMOTE_PORT - `url_scheme`: URL scheme (default "http") **Optional parameters:** Same limit/flag parameters as `parse_request()`. Returns dict with `REQUEST_METHOD`, `PATH_INFO`, `QUERY_STRING`, `SERVER_PROTOCOL`, `HTTP_*` headers, and `_consumed`. #### `parse_to_asgi_scope(data, server=None, client=None, scheme="http", root_path="", ...) -> dict` Parse HTTP request and build ASGI scope dict. Parameters: - `data`: Raw HTTP request bytes - `server`: (host, port) tuple - `client`: (addr, port) tuple - `scheme`: URL scheme (default "http") - `root_path`: ASGI root_path (default "") **Optional parameters:** Same limit/flag parameters as `parse_request()`. Returns dict with `type`, `asgi`, `http_version`, `method`, `scheme`, `path`, `raw_path`, `query_string`, `root_path`, `headers`, `server`, `client`, and `_consumed`. ### Exceptions **Base exceptions:** - `ParseError`: Base exception for parse errors (inherits from `ValueError`) - `IncompleteError`: Need more data (incremental parsing) **Validation exceptions (inherit from `ParseError`):** - `LimitRequestLine`: Request line exceeds `limit_request_line` - `LimitRequestHeaders`: Too many headers or header exceeds `limit_request_field_size` - `InvalidRequestMethod`: Invalid method characters or format (lowercase, too short, contains `#`) - `InvalidHTTPVersion`: HTTP version not 1.0 or 1.1 (e.g., HTTP/2.0, HTTP/0.9) - `InvalidHeaderName`: Invalid header name characters (not RFC 9110 token, e.g., space) - `InvalidHeader`: Invalid header value (contains NUL, CR, or LF) - `InvalidChunkExtension`: Chunk extension contains bare CR (RFC 9112 violation) When parsing fails, the parser analyzes the buffer to raise the most specific exception possible, helping identify the exact cause of malformed requests. ## License MIT License. See [LICENSE](LICENSE). The vendored [picohttpparser](https://github.com/h2o/picohttpparser) sources are also MIT licensed (dual-licensed MIT or Perl by their authors); their copyright notice is reproduced in `LICENSE`. ## Credits - [picohttpparser](https://github.com/h2o/picohttpparser) by Kazuho Oku et al. - Python bindings by Benoit Chesneau benoitc-gunicorn_h1c-52625fd/benchmarks/000077500000000000000000000000001524260035600201715ustar00rootroot00000000000000benoitc-gunicorn_h1c-52625fd/benchmarks/bench_protocol.py000066400000000000000000000161231524260035600235460ustar00rootroot00000000000000#!/usr/bin/env python3 """ Benchmark comparing callback-based H1CProtocol vs pull-based parse_request_fast. This benchmark measures: 1. Simple GET request parsing (headers only) 2. POST request with small body 3. POST request with chunked body 4. Incremental parsing (headers split across multiple feeds) """ import time from gunicorn_h1c import H1CProtocol, parse_request_fast # Test data SIMPLE_GET = ( b"GET / HTTP/1.1\r\nHost: localhost\r\nAccept: */*\r\nUser-Agent: bench/1.0\r\n\r\n" ) POST_WITH_BODY = ( b"POST /api/data HTTP/1.1\r\n" b"Host: localhost\r\n" b"Content-Type: application/json\r\n" b"Content-Length: 27\r\n" b"\r\n" b'{"key": "value", "num": 42}' ) CHUNKED_REQUEST = ( b"POST /api/upload HTTP/1.1\r\n" b"Host: localhost\r\n" b"Transfer-Encoding: chunked\r\n" b"\r\n" b"10\r\n" # 16 bytes b"0123456789abcdef\r\n" b"10\r\n" # 16 bytes b"fedcba9876543210\r\n" b"0\r\n\r\n" ) # Split headers for incremental test PARTIAL_HEADERS = [ b"GET /search?q=hello HTTP/1.1\r\n", b"Host: localhost\r\n", b"Accept: text/html\r\n", b"\r\n", ] def bench_pull_simple_get(iterations: int) -> float: """Benchmark pull-based API with simple GET.""" start = time.perf_counter() for _ in range(iterations): req = parse_request_fast(SIMPLE_GET) _ = req.method _ = req.path _ = req.headers end = time.perf_counter() return end - start def bench_callback_simple_get(iterations: int) -> float: """Benchmark callback-based API with simple GET.""" start = time.perf_counter() for _ in range(iterations): p = H1CProtocol() p.feed(SIMPLE_GET) _ = p.method _ = p.path _ = p.headers p.reset() end = time.perf_counter() return end - start def bench_callback_simple_get_reuse(iterations: int) -> float: """Benchmark callback-based API with parser reuse.""" p = H1CProtocol() start = time.perf_counter() for _ in range(iterations): p.feed(SIMPLE_GET) _ = p.method _ = p.path _ = p.headers p.reset() end = time.perf_counter() return end - start def bench_pull_post_body(iterations: int) -> float: """Benchmark pull-based API with POST body.""" start = time.perf_counter() for _ in range(iterations): req = parse_request_fast(POST_WITH_BODY) _ = req.method _ = req.content_length # Pull-based doesn't parse body end = time.perf_counter() return end - start def bench_callback_post_body(iterations: int) -> float: """Benchmark callback-based API with POST body.""" body_parts = [] def on_body(chunk): body_parts.append(chunk) p = H1CProtocol(on_body=on_body) start = time.perf_counter() for _ in range(iterations): body_parts.clear() p.feed(POST_WITH_BODY) _ = p.method _ = p.content_length p.reset() end = time.perf_counter() return end - start def bench_callback_chunked(iterations: int) -> float: """Benchmark callback-based API with chunked body.""" body_parts = [] def on_body(chunk): body_parts.append(chunk) p = H1CProtocol(on_body=on_body) start = time.perf_counter() for _ in range(iterations): body_parts.clear() p.feed(CHUNKED_REQUEST) _ = p.method _ = p.is_chunked p.reset() end = time.perf_counter() return end - start def bench_pull_incremental(iterations: int) -> float: """Benchmark pull-based API with incremental parsing.""" from gunicorn_h1c._parser_fast import IncompleteError start = time.perf_counter() for _ in range(iterations): buffer = bytearray() last_len = 0 for part in PARTIAL_HEADERS: buffer.extend(part) try: req = parse_request_fast(bytes(buffer), last_len) _ = req.method _ = req.path break except IncompleteError: last_len = len(buffer) end = time.perf_counter() return end - start def bench_callback_incremental(iterations: int) -> float: """Benchmark callback-based API with incremental parsing.""" p = H1CProtocol() start = time.perf_counter() for _ in range(iterations): for part in PARTIAL_HEADERS: p.feed(part) _ = p.method _ = p.path p.reset() end = time.perf_counter() return end - start def format_rate(elapsed: float, iterations: int) -> str: """Format rate as requests/sec.""" rate = iterations / elapsed if rate >= 1_000_000: return f"{rate / 1_000_000:.2f}M req/s" elif rate >= 1_000: return f"{rate / 1_000:.2f}K req/s" else: return f"{rate:.2f} req/s" def format_time(elapsed: float, iterations: int) -> str: """Format time per operation.""" ns_per_op = (elapsed / iterations) * 1_000_000_000 if ns_per_op >= 1_000_000: return f"{ns_per_op / 1_000_000:.2f}ms" elif ns_per_op >= 1_000: return f"{ns_per_op / 1_000:.2f}us" else: return f"{ns_per_op:.2f}ns" def run_benchmark(name: str, func, iterations: int) -> float: """Run a benchmark and print results.""" elapsed = func(iterations) rate = format_rate(elapsed, iterations) time_per_op = format_time(elapsed, iterations) print(f" {name:40} {rate:>15} ({time_per_op}/op)") return elapsed def main(): """Run all benchmarks.""" iterations = 500_000 print("=" * 75) print("H1CProtocol Callback vs Pull-based API Benchmark") print("=" * 75) print(f"Iterations: {iterations:,}") print() print("Simple GET (headers only):") print("-" * 75) pull_time = run_benchmark( "pull-based (parse_request_fast)", bench_pull_simple_get, iterations ) callback_time = run_benchmark( "callback (H1CProtocol, new each time)", bench_callback_simple_get, iterations ) reuse_time = run_benchmark( "callback (H1CProtocol, reused)", bench_callback_simple_get_reuse, iterations ) print() print("POST with body (Content-Length):") print("-" * 75) run_benchmark("pull-based (headers only)", bench_pull_post_body, iterations) run_benchmark("callback (headers + body)", bench_callback_post_body, iterations) print() print("Chunked transfer encoding:") print("-" * 75) run_benchmark( "callback (headers + chunked body)", bench_callback_chunked, iterations ) print() print("Incremental parsing (headers split across feeds):") print("-" * 75) run_benchmark("pull-based (buffer + retry)", bench_pull_incremental, iterations) run_benchmark("callback (multiple feeds)", bench_callback_incremental, iterations) print() print("=" * 75) print("Summary:") print("-" * 75) overhead = ((callback_time / pull_time) - 1) * 100 print(f" Callback overhead vs pull-based: {overhead:+.1f}%") reuse_overhead = ((reuse_time / pull_time) - 1) * 100 print(f" Callback overhead (reused parser): {reuse_overhead:+.1f}%") print() if __name__ == "__main__": main() benoitc-gunicorn_h1c-52625fd/pyproject.toml000066400000000000000000000033061524260035600207720ustar00rootroot00000000000000[build-system] requires = ["setuptools>=61.0", "wheel"] build-backend = "setuptools.build_meta" [project] name = "gunicorn_h1c" version = "0.6.9" description = "Fast HTTP/1.1 parser for Gunicorn using picohttpparser" readme = "README.md" license = {text = "MIT"} authors = [ {name = "Benoit Chesneau", email = "benoitc@gunicorn.org"} ] classifiers = [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: C", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", "Topic :: Internet :: WWW/HTTP", "Topic :: Internet :: WWW/HTTP :: HTTP Servers", ] requires-python = ">=3.9" [project.urls] Homepage = "https://github.com/benoitc/gunicorn_h1c" Repository = "https://github.com/benoitc/gunicorn_h1c" Issues = "https://github.com/benoitc/gunicorn_h1c/issues" [project.optional-dependencies] test = [ "pytest>=7.0", "pytest-benchmark", ] [tool.setuptools] packages = ["gunicorn_h1c"] package-dir = {"" = "src"} [tool.pytest.ini_options] testpaths = ["tests"] [tool.ruff] target-version = "py39" line-length = 88 [tool.ruff.lint] select = [ "E", # pycodestyle errors "W", # pycodestyle warnings "F", # pyflakes "I", # isort "B", # flake8-bugbear "UP", # pyupgrade ] ignore = [ "E501", # line too long (handled by formatter) ] [tool.ruff.format] quote-style = "double" benoitc-gunicorn_h1c-52625fd/setup.py000066400000000000000000000030251524260035600175660ustar00rootroot00000000000000""" Build script for gunicorn_h1c C extensions. """ import os import platform from setuptools import Extension, setup # Compiler flags for optimization extra_compile_args = ["-O3"] # Platform-specific SIMD flags if platform.machine() in ("x86_64", "AMD64", "i686", "i386"): # x86: Enable SSE4.2 for fast string operations extra_compile_args.append("-msse4.2") elif platform.machine() in ("arm64", "aarch64"): # ARM64: NEON is enabled by default on aarch64 pass # Base path for sources src_path = os.path.join("src", "gunicorn_h1c") pico_path = os.path.join(src_path, "picohttpparser") # Basic parser module pico_parser = Extension( "gunicorn_h1c._parser", sources=[ os.path.join(src_path, "pico_parser_module.c"), os.path.join(pico_path, "picohttpparser.c"), ], include_dirs=[pico_path], extra_compile_args=extra_compile_args, ) # Fast/optimized parser module pico_parser_fast = Extension( "gunicorn_h1c._parser_fast", sources=[ os.path.join(src_path, "pico_parser_fast.c"), os.path.join(pico_path, "picohttpparser.c"), ], include_dirs=[pico_path], extra_compile_args=extra_compile_args, ) # Callback-based protocol parser module pico_protocol = Extension( "gunicorn_h1c._protocol", sources=[ os.path.join(src_path, "pico_protocol.c"), os.path.join(pico_path, "picohttpparser.c"), ], include_dirs=[pico_path], extra_compile_args=extra_compile_args, ) setup( ext_modules=[pico_parser, pico_parser_fast, pico_protocol], ) benoitc-gunicorn_h1c-52625fd/src/000077500000000000000000000000001524260035600166435ustar00rootroot00000000000000benoitc-gunicorn_h1c-52625fd/src/gunicorn_h1c/000077500000000000000000000000001524260035600212225ustar00rootroot00000000000000benoitc-gunicorn_h1c-52625fd/src/gunicorn_h1c/__init__.py000066400000000000000000000057531524260035600233450ustar00rootroot00000000000000""" gunicorn_h1c - Fast HTTP/1.1 parser for Gunicorn SPDX-License-Identifier: MIT Copyright 2026 Benoit Chesneau This package provides high-performance HTTP/1.1 parsing using picohttpparser with SIMD optimizations (SSE4.2 on x86, NEON on ARM). Two parser implementations are available: 1. _parser - Basic parser returning dicts (simpler API) 2. _parser_fast - Optimized parser with zero-copy HttpRequest objects Usage: from gunicorn_h1c import parse_request, parse_request_fast, HttpRequest # Basic parsing (returns dict) result = parse_request(b"GET / HTTP/1.1\\r\\nHost: localhost\\r\\n\\r\\n") print(result['method']) # b'GET' print(result['path']) # b'/' print(result['headers']) # [(b'Host', b'localhost')] # Fast parsing (returns HttpRequest object with lazy evaluation) req = parse_request_fast(b"GET / HTTP/1.1\\r\\nHost: localhost\\r\\n\\r\\n") print(req.method) # b'GET' (created lazily) print(req.content_length) # -1 (extracted during parse) print(req.has_chunked) # False """ __version__ = "0.6.9" __all__ = [ # Basic parser "parse_request", "parse_response", "parse_headers", "parse_to_wsgi_environ", "parse_to_asgi_scope", # Fast parser "parse_request_fast", "parse_request_raw", "HttpRequest", # Callback-based protocol parser "H1CProtocol", # Base exceptions "ParseError", "IncompleteError", # Specific validation exceptions "LimitRequestLine", "LimitRequestHeaders", "InvalidRequestMethod", "InvalidHTTPVersion", "InvalidHeaderName", "InvalidHeader", "InvalidChunkExtension", ] # Import from C extensions try: from gunicorn_h1c._parser import ( IncompleteError, InvalidChunkExtension, InvalidHeader, InvalidHeaderName, InvalidHTTPVersion, InvalidRequestMethod, LimitRequestHeaders, LimitRequestLine, ParseError, parse_headers, parse_request, parse_response, parse_to_asgi_scope, parse_to_wsgi_environ, ) except ImportError as e: raise ImportError( "Failed to import gunicorn_h1c._parser C extension. " "Please ensure the package is properly installed with: " "pip install gunicorn_h1c" ) from e try: from gunicorn_h1c._parser_fast import ( HttpRequest, parse_request_raw, ) from gunicorn_h1c._parser_fast import ( parse_request as parse_request_fast, ) except ImportError as e: raise ImportError( "Failed to import gunicorn_h1c._parser_fast C extension. " "Please ensure the package is properly installed with: " "pip install gunicorn_h1c" ) from e try: from gunicorn_h1c._protocol import ( H1CProtocol, ) except ImportError as e: raise ImportError( "Failed to import gunicorn_h1c._protocol C extension. " "Please ensure the package is properly installed with: " "pip install gunicorn_h1c" ) from e benoitc-gunicorn_h1c-52625fd/src/gunicorn_h1c/pico_parser_fast.c000066400000000000000000000433331524260035600247170ustar00rootroot00000000000000/* * SPDX-License-Identifier: MIT * * Copyright 2026 Benoit Chesneau * Licensed under the MIT License (see LICENSE file). * * Optimized Python C extension for picohttpparser. * * Optimizations: * 1. Direct bytes: Returns PyBytes objects (avoids memoryview->bytes conversion) * 2. Lazy evaluation: Only creates Python objects when accessed * 3. Pre-allocated: Reuses header array * 4. Minimal allocations: Uses tuple instead of dict * 5. Common header extraction: content_length, has_chunked, connection_close */ #define PY_SSIZE_T_CLEAN #include #include "picohttpparser.h" #include "pico_utils.h" #define MAX_HEADERS 256 /* Exceptions - local copies */ static PyObject *ParseError; static PyObject *IncompleteError; /* Imported validation exceptions from _parser module */ static PyObject *LimitRequestLine; static PyObject *LimitRequestHeaders; static PyObject *InvalidRequestMethod; static PyObject *InvalidHTTPVersion; static PyObject *InvalidHeaderName; static PyObject *InvalidHeader; /* * HttpRequest type - holds parsed request with zero-copy access */ typedef struct { PyObject_HEAD /* Original buffer (kept alive) */ PyObject *buffer; Py_buffer view; /* Parsed pointers (into buffer) */ const char *method; size_t method_len; const char *path; size_t path_len; int minor_version; /* Headers */ struct phr_header headers[MAX_HEADERS]; size_t num_headers; /* Consumed bytes */ int consumed; /* Common headers (extracted during parse) */ Py_ssize_t content_length; /* -1 if not set */ int has_chunked; /* 1 if Transfer-Encoding: chunked */ int connection_close; /* 1=close, 0=keep-alive, -1=unset */ /* Cached Python objects (lazy) */ PyObject *py_method; PyObject *py_path; PyObject *py_headers; PyObject *py_asgi_headers; /* Headers with lowercase names for ASGI */ } HttpRequest; static void HttpRequest_dealloc(HttpRequest *self) { Py_XDECREF(self->py_method); Py_XDECREF(self->py_path); Py_XDECREF(self->py_headers); Py_XDECREF(self->py_asgi_headers); if (self->view.buf) { PyBuffer_Release(&self->view); } Py_XDECREF(self->buffer); Py_TYPE(self)->tp_free((PyObject *)self); } static PyObject * HttpRequest_get_method(HttpRequest *self, void *closure) { if (!self->py_method) { /* Return bytes directly - avoids memoryview->bytes conversion overhead */ self->py_method = PyBytes_FromStringAndSize(self->method, self->method_len); if (!self->py_method) return NULL; } Py_INCREF(self->py_method); return self->py_method; } static PyObject * HttpRequest_get_path(HttpRequest *self, void *closure) { if (!self->py_path) { /* Return bytes directly - avoids memoryview->bytes conversion overhead */ self->py_path = PyBytes_FromStringAndSize(self->path, self->path_len); if (!self->py_path) return NULL; } Py_INCREF(self->py_path); return self->py_path; } static PyObject * HttpRequest_get_version(HttpRequest *self, void *closure) { return PyLong_FromLong(self->minor_version); } static PyObject * HttpRequest_get_headers(HttpRequest *self, void *closure) { if (!self->py_headers) { self->py_headers = PyTuple_New(self->num_headers); if (!self->py_headers) return NULL; for (size_t i = 0; i < self->num_headers; i++) { PyObject *pair = pico_create_header_tuple( self->headers[i].name, self->headers[i].name_len, self->headers[i].value, self->headers[i].value_len); if (!pair) { Py_DECREF(self->py_headers); self->py_headers = NULL; return NULL; } PyTuple_SET_ITEM(self->py_headers, i, pair); } } Py_INCREF(self->py_headers); return self->py_headers; } static PyObject * HttpRequest_get_asgi_headers(HttpRequest *self, void *closure) { if (!self->py_asgi_headers) { self->py_asgi_headers = PyList_New(self->num_headers); if (!self->py_asgi_headers) return NULL; for (size_t i = 0; i < self->num_headers; i++) { PyObject *pair = pico_create_header_tuple_lowercase( self->headers[i].name, self->headers[i].name_len, self->headers[i].value, self->headers[i].value_len); if (!pair) { Py_DECREF(self->py_asgi_headers); self->py_asgi_headers = NULL; return NULL; } PyList_SET_ITEM(self->py_asgi_headers, i, pair); } } Py_INCREF(self->py_asgi_headers); return self->py_asgi_headers; } static PyObject * HttpRequest_get_consumed(HttpRequest *self, void *closure) { return PyLong_FromLong(self->consumed); } /* Fast header lookup by name - returns bytes */ static PyObject * HttpRequest_get_header(HttpRequest *self, PyObject *args) { const char *name; Py_ssize_t name_len; if (!PyArg_ParseTuple(args, "s#", &name, &name_len)) { return NULL; } return pico_find_header(self->headers, self->num_headers, name, (size_t)name_len); } /* Get header count without creating list */ static PyObject * HttpRequest_get_header_count(HttpRequest *self, void *closure) { return PyLong_FromSize_t(self->num_headers); } /* Getters for common headers (extracted during parse) */ static PyObject * HttpRequest_get_content_length(HttpRequest *self, void *closure) { return PyLong_FromSsize_t(self->content_length); } static PyObject * HttpRequest_get_has_chunked(HttpRequest *self, void *closure) { return PyBool_FromLong(self->has_chunked); } static PyObject * HttpRequest_get_connection_close(HttpRequest *self, void *closure) { return PyLong_FromLong(self->connection_close); } /* header_name_eq moved to pico_utils.h */ /* Extract common headers using shared utility */ static void extract_special_headers(HttpRequest *self) { PicoHeaderInfo info; pico_extract_headers(self->headers, self->num_headers, &info); self->content_length = info.content_length; self->has_chunked = info.has_chunked; self->connection_close = info.connection_close; } static PyGetSetDef HttpRequest_getset[] = { {"method", (getter)HttpRequest_get_method, NULL, "HTTP method", NULL}, {"path", (getter)HttpRequest_get_path, NULL, "Request path", NULL}, {"minor_version", (getter)HttpRequest_get_version, NULL, "HTTP minor version", NULL}, {"headers", (getter)HttpRequest_get_headers, NULL, "Request headers", NULL}, {"asgi_headers", (getter)HttpRequest_get_asgi_headers, NULL, "Headers with lowercase names for ASGI", NULL}, {"consumed", (getter)HttpRequest_get_consumed, NULL, "Bytes consumed", NULL}, {"header_count", (getter)HttpRequest_get_header_count, NULL, "Number of headers", NULL}, {"content_length", (getter)HttpRequest_get_content_length, NULL, "Content-Length (-1 if not set)", NULL}, {"has_chunked", (getter)HttpRequest_get_has_chunked, NULL, "True if Transfer-Encoding: chunked", NULL}, {"connection_close", (getter)HttpRequest_get_connection_close, NULL, "1=close, 0=keep-alive, -1=unset", NULL}, {NULL} }; static PyMethodDef HttpRequest_methods[] = { {"get_header", (PyCFunction)HttpRequest_get_header, METH_VARARGS, "Get header value by name (case-insensitive)"}, {NULL} }; static PyTypeObject HttpRequestType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "gunicorn_h1c._parser_fast.HttpRequest", .tp_doc = "Parsed HTTP request with zero-copy access", .tp_basicsize = sizeof(HttpRequest), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT, .tp_dealloc = (destructor)HttpRequest_dealloc, .tp_getset = HttpRequest_getset, .tp_methods = HttpRequest_methods, }; /* * parse_request(data: bytes, ...) -> HttpRequest * * Parse HTTP request with zero-copy optimization. * Returns HttpRequest object that references original buffer. */ static PyObject * pico_parse_request_fast(PyObject *self, PyObject *args, PyObject *kwargs) { static char *kwlist[] = { "data", "last_len", "limit_request_line", "limit_request_fields", "limit_request_field_size", "permit_unconventional_http_method", "permit_unconventional_http_version", NULL }; PyObject *data; Py_ssize_t last_len = 0; Py_ssize_t limit_request_line = PICO_DEFAULT_LIMIT_REQUEST_LINE; Py_ssize_t limit_request_fields = PICO_DEFAULT_LIMIT_REQUEST_FIELDS; Py_ssize_t limit_request_field_size = PICO_DEFAULT_LIMIT_REQUEST_FIELD_SIZE; int permit_unconventional_http_method = 0; int permit_unconventional_http_version = 0; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "O|nnnnpp", kwlist, &data, &last_len, &limit_request_line, &limit_request_fields, &limit_request_field_size, &permit_unconventional_http_method, &permit_unconventional_http_version)) { return NULL; } /* Create request object */ HttpRequest *req = PyObject_New(HttpRequest, &HttpRequestType); if (!req) return NULL; /* Initialize */ req->buffer = NULL; req->view.buf = NULL; req->py_method = NULL; req->py_path = NULL; req->py_headers = NULL; req->py_asgi_headers = NULL; /* Get buffer */ if (PyObject_GetBuffer(data, &req->view, PyBUF_SIMPLE) < 0) { Py_DECREF(req); return NULL; } /* Check request line length before parsing */ Py_ssize_t req_line_len = pico_find_request_line_length(req->view.buf, req->view.len); if (req_line_len >= 0 && req_line_len > limit_request_line) { Py_DECREF(req); PyErr_Format(LimitRequestLine, "Request line length (%zd) exceeds limit (%zd)", req_line_len, limit_request_line); return NULL; } /* Keep reference to original buffer */ Py_INCREF(data); req->buffer = data; /* Parse */ req->num_headers = MAX_HEADERS; int ret = phr_parse_request( req->view.buf, req->view.len, &req->method, &req->method_len, &req->path, &req->path_len, &req->minor_version, req->headers, &req->num_headers, (size_t)last_len ); if (ret == -2) { Py_DECREF(req); PyErr_SetString(IncompleteError, "Incomplete request"); return NULL; } else if (ret < 0) { pico_analyze_parse_error(req->view.buf, req->view.len, permit_unconventional_http_method, permit_unconventional_http_version, ParseError, InvalidRequestMethod, InvalidHTTPVersion, InvalidHeaderName, InvalidHeader); Py_DECREF(req); return NULL; } /* Parse successful - now validate */ /* Validate method */ if (pico_validate_method(req->method, req->method_len, permit_unconventional_http_method, InvalidRequestMethod) < 0) { Py_DECREF(req); return NULL; } /* Validate request-target form against method (RFC 9112) */ if (pico_validate_request_target(req->method, req->method_len, req->path, req->path_len, ParseError) < 0) { Py_DECREF(req); return NULL; } /* Validate HTTP version */ if (pico_validate_version(req->minor_version, permit_unconventional_http_version, InvalidHTTPVersion) < 0) { Py_DECREF(req); return NULL; } /* Validate headers */ PicoValidationConfig config = { .limit_request_line = limit_request_line, .limit_request_fields = limit_request_fields, .limit_request_field_size = limit_request_field_size, .permit_unconventional_http_method = permit_unconventional_http_method, .permit_unconventional_http_version = permit_unconventional_http_version }; if (pico_validate_headers_full(req->headers, req->num_headers, req->minor_version, &config, LimitRequestHeaders, InvalidHeaderName, InvalidHeader) < 0) { Py_DECREF(req); return NULL; } req->consumed = ret; extract_special_headers(req); return (PyObject *)req; } /* * parse_request_raw(data: bytes) -> tuple * * Ultra-fast parsing that returns raw tuple: * (method_offset, method_len, path_offset, path_len, version, * header_count, consumed, header_data) * * header_data is bytes containing packed header offsets/lengths */ static PyObject * pico_parse_request_raw(PyObject *self, PyObject *args, PyObject *kwargs) { static char *kwlist[] = {"data", "last_len", NULL}; Py_buffer buf; Py_ssize_t last_len = 0; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "y*|n", kwlist, &buf, &last_len)) { return NULL; } const char *method, *path; size_t method_len, path_len; int minor_version; struct phr_header headers[MAX_HEADERS]; size_t num_headers = MAX_HEADERS; int ret = phr_parse_request( buf.buf, buf.len, &method, &method_len, &path, &path_len, &minor_version, headers, &num_headers, (size_t)last_len ); if (ret > 0) { /* Calculate offsets relative to buffer start */ Py_ssize_t method_offset = method - (const char *)buf.buf; Py_ssize_t path_offset = path - (const char *)buf.buf; /* Pack header offsets into bytes */ /* Each header: 4 bytes name_offset, 2 bytes name_len, 4 bytes value_offset, 2 bytes value_len */ PyObject *header_data = PyBytes_FromStringAndSize(NULL, num_headers * 12); if (!header_data) { PyBuffer_Release(&buf); return NULL; } char *hdata = PyBytes_AS_STRING(header_data); for (size_t i = 0; i < num_headers; i++) { uint32_t name_off = (uint32_t)(headers[i].name - (const char *)buf.buf); uint16_t name_len = (uint16_t)headers[i].name_len; uint32_t val_off = (uint32_t)(headers[i].value - (const char *)buf.buf); uint16_t val_len = (uint16_t)headers[i].value_len; /* Little-endian pack */ memcpy(hdata + i * 12, &name_off, 4); memcpy(hdata + i * 12 + 4, &name_len, 2); memcpy(hdata + i * 12 + 6, &val_off, 4); memcpy(hdata + i * 12 + 10, &val_len, 2); } PyBuffer_Release(&buf); PyObject *result = Py_BuildValue("(nnnniiiO)", method_offset, (Py_ssize_t)method_len, path_offset, (Py_ssize_t)path_len, minor_version, (int)num_headers, ret, /* consumed */ header_data); Py_DECREF(header_data); return result; } PyBuffer_Release(&buf); if (ret == -2) { PyErr_SetString(IncompleteError, "Incomplete request"); } else { PyErr_SetString(ParseError, "Invalid HTTP request"); } return NULL; } /* Module methods */ static PyMethodDef pico_methods[] = { {"parse_request", (PyCFunction)pico_parse_request_fast, METH_VARARGS | METH_KEYWORDS, "Parse HTTP request (zero-copy, lazy evaluation)"}, {"parse_request_raw", (PyCFunction)pico_parse_request_raw, METH_VARARGS | METH_KEYWORDS, "Parse HTTP request (returns raw offsets for maximum speed)"}, {NULL, NULL, 0, NULL} }; static struct PyModuleDef pico_module = { PyModuleDef_HEAD_INIT, "gunicorn_h1c._parser_fast", "Ultra-fast HTTP parser with zero-copy optimization", -1, pico_methods }; PyMODINIT_FUNC PyInit__parser_fast(void) { PyObject *m; if (PyType_Ready(&HttpRequestType) < 0) return NULL; m = PyModule_Create(&pico_module); if (m == NULL) return NULL; Py_INCREF(&HttpRequestType); PyModule_AddObject(m, "HttpRequest", (PyObject *)&HttpRequestType); ParseError = PyErr_NewException("gunicorn_h1c._parser_fast.ParseError", PyExc_ValueError, NULL); if (!ParseError) return NULL; Py_INCREF(ParseError); PyModule_AddObject(m, "ParseError", ParseError); IncompleteError = PyErr_NewException("gunicorn_h1c._parser_fast.IncompleteError", PyExc_Exception, NULL); if (!IncompleteError) return NULL; Py_INCREF(IncompleteError); PyModule_AddObject(m, "IncompleteError", IncompleteError); /* Import validation exceptions from _parser module */ PyObject *parser_module = PyImport_ImportModule("gunicorn_h1c._parser"); if (!parser_module) return NULL; LimitRequestLine = PyObject_GetAttrString(parser_module, "LimitRequestLine"); if (!LimitRequestLine) { Py_DECREF(parser_module); return NULL; } LimitRequestHeaders = PyObject_GetAttrString(parser_module, "LimitRequestHeaders"); if (!LimitRequestHeaders) { Py_DECREF(parser_module); return NULL; } InvalidRequestMethod = PyObject_GetAttrString(parser_module, "InvalidRequestMethod"); if (!InvalidRequestMethod) { Py_DECREF(parser_module); return NULL; } InvalidHTTPVersion = PyObject_GetAttrString(parser_module, "InvalidHTTPVersion"); if (!InvalidHTTPVersion) { Py_DECREF(parser_module); return NULL; } InvalidHeaderName = PyObject_GetAttrString(parser_module, "InvalidHeaderName"); if (!InvalidHeaderName) { Py_DECREF(parser_module); return NULL; } InvalidHeader = PyObject_GetAttrString(parser_module, "InvalidHeader"); if (!InvalidHeader) { Py_DECREF(parser_module); return NULL; } Py_DECREF(parser_module); return m; } benoitc-gunicorn_h1c-52625fd/src/gunicorn_h1c/pico_parser_module.c000066400000000000000000001125171524260035600252500ustar00rootroot00000000000000/* * SPDX-License-Identifier: MIT * * Copyright 2026 Benoit Chesneau * Licensed under the MIT License (see LICENSE file). * * Python C extension for picohttpparser. * Provides fast HTTP request/response parsing using SIMD when available. */ #define PY_SSIZE_T_CLEAN #include #include "picohttpparser.h" #include "pico_utils.h" #define MAX_HEADERS 256 /* Forward declarations - Exception types */ static PyObject *PicoError; static PyObject *IncompleteError; /* Specific validation exception types */ static PyObject *LimitRequestLine; static PyObject *LimitRequestHeaders; static PyObject *InvalidRequestMethod; static PyObject *InvalidHTTPVersion; static PyObject *InvalidHeaderName; static PyObject *InvalidHeader; static PyObject *InvalidChunkExtension; /* * parse_request(data: bytes, last_len: int = 0, ...) -> dict * * Parse HTTP request and return dict with: * method, path, minor_version, headers, consumed * * Additional parameters for limit enforcement: * limit_request_line: max request line length (default 8190) * limit_request_fields: max number of headers (default 100) * limit_request_field_size: max header size (default 8190) * permit_unconventional_http_method: allow lowercase methods etc (default False) * permit_unconventional_http_version: allow non-1.0/1.1 versions (default False) */ static PyObject * pico_parse_request(PyObject *self, PyObject *args, PyObject *kwargs) { static char *kwlist[] = { "data", "last_len", "limit_request_line", "limit_request_fields", "limit_request_field_size", "permit_unconventional_http_method", "permit_unconventional_http_version", NULL }; Py_buffer buf; Py_ssize_t last_len = 0; Py_ssize_t limit_request_line = PICO_DEFAULT_LIMIT_REQUEST_LINE; Py_ssize_t limit_request_fields = PICO_DEFAULT_LIMIT_REQUEST_FIELDS; Py_ssize_t limit_request_field_size = PICO_DEFAULT_LIMIT_REQUEST_FIELD_SIZE; int permit_unconventional_http_method = 0; int permit_unconventional_http_version = 0; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "y*|nnnnpp", kwlist, &buf, &last_len, &limit_request_line, &limit_request_fields, &limit_request_field_size, &permit_unconventional_http_method, &permit_unconventional_http_version)) { return NULL; } /* Check request line length before parsing */ Py_ssize_t req_line_len = pico_find_request_line_length(buf.buf, buf.len); if (req_line_len >= 0 && req_line_len > limit_request_line) { PyBuffer_Release(&buf); PyErr_Format(LimitRequestLine, "Request line length (%zd) exceeds limit (%zd)", req_line_len, limit_request_line); return NULL; } const char *method; size_t method_len; const char *path; size_t path_len; int minor_version; struct phr_header headers[MAX_HEADERS]; size_t num_headers = MAX_HEADERS; int ret = phr_parse_request( buf.buf, buf.len, &method, &method_len, &path, &path_len, &minor_version, headers, &num_headers, (size_t)last_len ); if (ret == -2) { PyBuffer_Release(&buf); PyErr_SetString(IncompleteError, "Incomplete request, need more data"); return NULL; } else if (ret < 0) { pico_analyze_parse_error(buf.buf, buf.len, permit_unconventional_http_method, permit_unconventional_http_version, PicoError, InvalidRequestMethod, InvalidHTTPVersion, InvalidHeaderName, InvalidHeader); PyBuffer_Release(&buf); return NULL; } /* Parse successful - now validate */ /* Validate method */ if (pico_validate_method(method, method_len, permit_unconventional_http_method, InvalidRequestMethod) < 0) { PyBuffer_Release(&buf); return NULL; } /* Validate request-target form against method (RFC 9112) */ if (pico_validate_request_target(method, method_len, path, path_len, PicoError) < 0) { PyBuffer_Release(&buf); return NULL; } /* Validate HTTP version */ if (pico_validate_version(minor_version, permit_unconventional_http_version, InvalidHTTPVersion) < 0) { PyBuffer_Release(&buf); return NULL; } /* Validate headers */ PicoValidationConfig config = { .limit_request_line = limit_request_line, .limit_request_fields = limit_request_fields, .limit_request_field_size = limit_request_field_size, .permit_unconventional_http_method = permit_unconventional_http_method, .permit_unconventional_http_version = permit_unconventional_http_version }; if (pico_validate_headers_full(headers, num_headers, minor_version, &config, LimitRequestHeaders, InvalidHeaderName, InvalidHeader) < 0) { PyBuffer_Release(&buf); return NULL; } PyBuffer_Release(&buf); /* Success - build result dict */ PyObject *result = PyDict_New(); if (!result) return NULL; PyObject *py_method = PyBytes_FromStringAndSize(method, method_len); PyObject *py_path = PyBytes_FromStringAndSize(path, path_len); PyObject *py_version = PyLong_FromLong(minor_version); PyObject *py_consumed = PyLong_FromLong(ret); /* Build headers list */ PyObject *py_headers = PyList_New(num_headers); if (!py_headers) { Py_DECREF(result); Py_XDECREF(py_method); Py_XDECREF(py_path); Py_XDECREF(py_version); Py_XDECREF(py_consumed); return NULL; } for (size_t i = 0; i < num_headers; i++) { PyObject *tuple = pico_create_header_tuple( headers[i].name, headers[i].name_len, headers[i].value, headers[i].value_len); if (!tuple) { Py_DECREF(py_headers); Py_DECREF(result); Py_XDECREF(py_method); Py_XDECREF(py_path); Py_XDECREF(py_version); Py_XDECREF(py_consumed); return NULL; } PyList_SET_ITEM(py_headers, i, tuple); } PyDict_SetItemString(result, "method", py_method); PyDict_SetItemString(result, "path", py_path); PyDict_SetItemString(result, "minor_version", py_version); PyDict_SetItemString(result, "headers", py_headers); PyDict_SetItemString(result, "consumed", py_consumed); Py_DECREF(py_method); Py_DECREF(py_path); Py_DECREF(py_version); Py_DECREF(py_headers); Py_DECREF(py_consumed); return result; } /* * parse_response(data: bytes, last_len: int = 0) -> dict */ static PyObject * pico_parse_response(PyObject *self, PyObject *args, PyObject *kwargs) { static char *kwlist[] = {"data", "last_len", NULL}; Py_buffer buf; Py_ssize_t last_len = 0; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "y*|n", kwlist, &buf, &last_len)) { return NULL; } int minor_version; int status; const char *msg; size_t msg_len; struct phr_header headers[MAX_HEADERS]; size_t num_headers = MAX_HEADERS; int ret = phr_parse_response( buf.buf, buf.len, &minor_version, &status, &msg, &msg_len, headers, &num_headers, (size_t)last_len ); PyBuffer_Release(&buf); if (ret > 0) { PyObject *result = PyDict_New(); if (!result) return NULL; PyObject *py_status = PyLong_FromLong(status); PyObject *py_message = PyBytes_FromStringAndSize(msg, msg_len); PyObject *py_version = PyLong_FromLong(minor_version); PyObject *py_consumed = PyLong_FromLong(ret); PyObject *py_headers = PyList_New(num_headers); if (!py_headers) { Py_DECREF(result); Py_XDECREF(py_status); Py_XDECREF(py_message); Py_XDECREF(py_version); Py_XDECREF(py_consumed); return NULL; } for (size_t i = 0; i < num_headers; i++) { PyObject *tuple = pico_create_header_tuple( headers[i].name, headers[i].name_len, headers[i].value, headers[i].value_len); if (!tuple) { Py_DECREF(py_headers); Py_DECREF(result); Py_XDECREF(py_status); Py_XDECREF(py_message); Py_XDECREF(py_version); Py_XDECREF(py_consumed); return NULL; } PyList_SET_ITEM(py_headers, i, tuple); } PyDict_SetItemString(result, "status", py_status); PyDict_SetItemString(result, "message", py_message); PyDict_SetItemString(result, "minor_version", py_version); PyDict_SetItemString(result, "headers", py_headers); PyDict_SetItemString(result, "consumed", py_consumed); Py_DECREF(py_status); Py_DECREF(py_message); Py_DECREF(py_version); Py_DECREF(py_headers); Py_DECREF(py_consumed); return result; } else if (ret == -2) { PyErr_SetString(IncompleteError, "Incomplete response, need more data"); return NULL; } else { PyErr_SetString(PicoError, "Invalid HTTP response"); return NULL; } } /* * Helper: Split path into path_info and query_string * Returns pointer to '?' or NULL if not found */ static const char * find_query_string(const char *path, size_t path_len) { for (size_t i = 0; i < path_len; i++) { if (path[i] == '?') { return &path[i]; } } return NULL; } /* header_is_content_type and header_is_content_length moved to pico_utils.h as pico_header_name_eq */ /* * parse_to_wsgi_environ(data, server=None, client=None, url_scheme="http", ...) -> dict * * Parse HTTP request and return WSGI environ dict. */ static PyObject * pico_parse_to_wsgi_environ(PyObject *self, PyObject *args, PyObject *kwargs) { static char *kwlist[] = { "data", "server", "client", "url_scheme", "limit_request_line", "limit_request_fields", "limit_request_field_size", "permit_unconventional_http_method", "permit_unconventional_http_version", NULL }; Py_buffer buf; PyObject *server = NULL; PyObject *client = NULL; const char *url_scheme = "http"; Py_ssize_t url_scheme_len = 4; Py_ssize_t limit_request_line = PICO_DEFAULT_LIMIT_REQUEST_LINE; Py_ssize_t limit_request_fields = PICO_DEFAULT_LIMIT_REQUEST_FIELDS; Py_ssize_t limit_request_field_size = PICO_DEFAULT_LIMIT_REQUEST_FIELD_SIZE; int permit_unconventional_http_method = 0; int permit_unconventional_http_version = 0; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "y*|OOs#nnnpp", kwlist, &buf, &server, &client, &url_scheme, &url_scheme_len, &limit_request_line, &limit_request_fields, &limit_request_field_size, &permit_unconventional_http_method, &permit_unconventional_http_version)) { return NULL; } /* Check request line length before parsing */ Py_ssize_t req_line_len = pico_find_request_line_length(buf.buf, buf.len); if (req_line_len >= 0 && req_line_len > limit_request_line) { PyBuffer_Release(&buf); PyErr_Format(LimitRequestLine, "Request line length (%zd) exceeds limit (%zd)", req_line_len, limit_request_line); return NULL; } const char *method; size_t method_len; const char *path; size_t path_len; int minor_version; struct phr_header headers[MAX_HEADERS]; size_t num_headers = MAX_HEADERS; int ret = phr_parse_request( buf.buf, buf.len, &method, &method_len, &path, &path_len, &minor_version, headers, &num_headers, 0 ); if (ret == -2) { PyBuffer_Release(&buf); PyErr_SetString(IncompleteError, "Incomplete request, need more data"); return NULL; } else if (ret < 0) { pico_analyze_parse_error(buf.buf, buf.len, permit_unconventional_http_method, permit_unconventional_http_version, PicoError, InvalidRequestMethod, InvalidHTTPVersion, InvalidHeaderName, InvalidHeader); PyBuffer_Release(&buf); return NULL; } /* Validate method */ if (pico_validate_method(method, method_len, permit_unconventional_http_method, InvalidRequestMethod) < 0) { PyBuffer_Release(&buf); return NULL; } /* Validate request-target form against method (RFC 9112) */ if (pico_validate_request_target(method, method_len, path, path_len, PicoError) < 0) { PyBuffer_Release(&buf); return NULL; } /* Validate HTTP version */ if (pico_validate_version(minor_version, permit_unconventional_http_version, InvalidHTTPVersion) < 0) { PyBuffer_Release(&buf); return NULL; } /* Validate headers */ PicoValidationConfig config = { .limit_request_line = limit_request_line, .limit_request_fields = limit_request_fields, .limit_request_field_size = limit_request_field_size, .permit_unconventional_http_method = permit_unconventional_http_method, .permit_unconventional_http_version = permit_unconventional_http_version }; if (pico_validate_headers_full(headers, num_headers, minor_version, &config, LimitRequestHeaders, InvalidHeaderName, InvalidHeader) < 0) { PyBuffer_Release(&buf); return NULL; } PyBuffer_Release(&buf); /* Success - build WSGI environ dict */ PyObject *environ = PyDict_New(); if (!environ) return NULL; /* Split path into PATH_INFO and QUERY_STRING */ const char *query_start = find_query_string(path, path_len); size_t path_info_len; const char *query_string; size_t query_string_len; if (query_start) { path_info_len = query_start - path; query_string = query_start + 1; query_string_len = path_len - path_info_len - 1; } else { path_info_len = path_len; query_string = ""; query_string_len = 0; } /* REQUEST_METHOD (str, latin-1) */ PyObject *py_method = PyUnicode_DecodeLatin1(method, method_len, NULL); if (!py_method) goto error; PyDict_SetItemString(environ, "REQUEST_METHOD", py_method); Py_DECREF(py_method); /* PATH_INFO (str, latin-1) */ PyObject *py_path_info = PyUnicode_DecodeLatin1(path, path_info_len, NULL); if (!py_path_info) goto error; PyDict_SetItemString(environ, "PATH_INFO", py_path_info); Py_DECREF(py_path_info); /* QUERY_STRING (str) */ PyObject *py_query = PyUnicode_DecodeLatin1(query_string, query_string_len, NULL); if (!py_query) goto error; PyDict_SetItemString(environ, "QUERY_STRING", py_query); Py_DECREF(py_query); /* SERVER_PROTOCOL */ PyObject *py_protocol = minor_version == 0 ? PyUnicode_FromString("HTTP/1.0") : PyUnicode_FromString("HTTP/1.1"); if (!py_protocol) goto error; PyDict_SetItemString(environ, "SERVER_PROTOCOL", py_protocol); Py_DECREF(py_protocol); /* wsgi.url_scheme */ PyObject *py_scheme = PyUnicode_DecodeLatin1(url_scheme, url_scheme_len, NULL); if (!py_scheme) goto error; PyDict_SetItemString(environ, "wsgi.url_scheme", py_scheme); Py_DECREF(py_scheme); /* SERVER_NAME, SERVER_PORT from server tuple */ if (server && PyTuple_Check(server) && PyTuple_Size(server) >= 2) { PyObject *server_name = PyTuple_GetItem(server, 0); PyObject *server_port = PyTuple_GetItem(server, 1); PyDict_SetItemString(environ, "SERVER_NAME", server_name); /* Convert port to string */ if (PyLong_Check(server_port)) { PyObject *port_str = PyObject_Str(server_port); if (port_str) { PyDict_SetItemString(environ, "SERVER_PORT", port_str); Py_DECREF(port_str); } } else { PyDict_SetItemString(environ, "SERVER_PORT", server_port); } } /* REMOTE_ADDR, REMOTE_PORT from client tuple */ if (client && PyTuple_Check(client) && PyTuple_Size(client) >= 2) { PyObject *remote_addr = PyTuple_GetItem(client, 0); PyObject *remote_port = PyTuple_GetItem(client, 1); PyDict_SetItemString(environ, "REMOTE_ADDR", remote_addr); /* Convert port to string */ if (PyLong_Check(remote_port)) { PyObject *port_str = PyObject_Str(remote_port); if (port_str) { PyDict_SetItemString(environ, "REMOTE_PORT", port_str); Py_DECREF(port_str); } } else { PyDict_SetItemString(environ, "REMOTE_PORT", remote_port); } } /* Process headers - need to handle duplicates by joining with "," */ /* First, build a temporary dict to merge duplicate headers */ PyObject *header_dict = PyDict_New(); if (!header_dict) goto error; for (size_t i = 0; i < num_headers; i++) { const char *name = headers[i].name; size_t name_len = headers[i].name_len; const char *value = headers[i].value; size_t value_len = headers[i].value_len; /* Build the environ key name */ char key_buf[256]; size_t key_len = 0; int is_content_type = pico_header_name_eq(name, name_len, "content-type", 12); int is_content_length = pico_header_name_eq(name, name_len, "content-length", 14); if (is_content_type) { memcpy(key_buf, "CONTENT_TYPE", 12); key_len = 12; } else if (is_content_length) { memcpy(key_buf, "CONTENT_LENGTH", 14); key_len = 14; } else { /* HTTP_* prefix */ memcpy(key_buf, "HTTP_", 5); key_len = 5; /* Copy and transform header name */ for (size_t j = 0; j < name_len && key_len < 255; j++) { char c = name[j]; if (c >= 'a' && c <= 'z') { c -= 32; /* lowercase to uppercase */ } else if (c == '-') { c = '_'; /* dash to underscore */ } key_buf[key_len++] = c; } } key_buf[key_len] = '\0'; /* Decode value as latin-1 */ PyObject *py_value = PyUnicode_DecodeLatin1(value, value_len, NULL); if (!py_value) { Py_DECREF(header_dict); goto error; } /* Check if key already exists (duplicate header) */ PyObject *existing = PyDict_GetItemString(header_dict, key_buf); if (existing) { /* Join with comma */ PyObject *comma = PyUnicode_FromString(","); PyObject *joined = PyUnicode_Concat(existing, comma); Py_DECREF(comma); if (joined) { PyObject *final = PyUnicode_Concat(joined, py_value); Py_DECREF(joined); if (final) { PyDict_SetItemString(header_dict, key_buf, final); Py_DECREF(final); } } } else { PyDict_SetItemString(header_dict, key_buf, py_value); } Py_DECREF(py_value); } /* Merge header_dict into environ */ PyObject *key, *value; Py_ssize_t pos = 0; while (PyDict_Next(header_dict, &pos, &key, &value)) { PyDict_SetItem(environ, key, value); } Py_DECREF(header_dict); /* Add consumed bytes count */ PyObject *py_consumed = PyLong_FromLong(ret); if (!py_consumed) goto error; PyDict_SetItemString(environ, "_consumed", py_consumed); Py_DECREF(py_consumed); return environ; error: Py_DECREF(environ); return NULL; } /* * parse_to_asgi_scope(data, server=None, client=None, scheme="http", root_path="", ...) -> dict * * Parse HTTP request and return ASGI scope dict. */ static PyObject * pico_parse_to_asgi_scope(PyObject *self, PyObject *args, PyObject *kwargs) { static char *kwlist[] = { "data", "server", "client", "scheme", "root_path", "limit_request_line", "limit_request_fields", "limit_request_field_size", "permit_unconventional_http_method", "permit_unconventional_http_version", NULL }; Py_buffer buf; PyObject *server = NULL; PyObject *client = NULL; const char *scheme = "http"; Py_ssize_t scheme_len = 4; const char *root_path = ""; Py_ssize_t root_path_len = 0; Py_ssize_t limit_request_line = PICO_DEFAULT_LIMIT_REQUEST_LINE; Py_ssize_t limit_request_fields = PICO_DEFAULT_LIMIT_REQUEST_FIELDS; Py_ssize_t limit_request_field_size = PICO_DEFAULT_LIMIT_REQUEST_FIELD_SIZE; int permit_unconventional_http_method = 0; int permit_unconventional_http_version = 0; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "y*|OOs#s#nnnpp", kwlist, &buf, &server, &client, &scheme, &scheme_len, &root_path, &root_path_len, &limit_request_line, &limit_request_fields, &limit_request_field_size, &permit_unconventional_http_method, &permit_unconventional_http_version)) { return NULL; } /* Check request line length before parsing */ Py_ssize_t req_line_len = pico_find_request_line_length(buf.buf, buf.len); if (req_line_len >= 0 && req_line_len > limit_request_line) { PyBuffer_Release(&buf); PyErr_Format(LimitRequestLine, "Request line length (%zd) exceeds limit (%zd)", req_line_len, limit_request_line); return NULL; } const char *method; size_t method_len; const char *path; size_t path_len; int minor_version; struct phr_header headers[MAX_HEADERS]; size_t num_headers = MAX_HEADERS; int ret = phr_parse_request( buf.buf, buf.len, &method, &method_len, &path, &path_len, &minor_version, headers, &num_headers, 0 ); if (ret == -2) { PyBuffer_Release(&buf); PyErr_SetString(IncompleteError, "Incomplete request, need more data"); return NULL; } else if (ret < 0) { pico_analyze_parse_error(buf.buf, buf.len, permit_unconventional_http_method, permit_unconventional_http_version, PicoError, InvalidRequestMethod, InvalidHTTPVersion, InvalidHeaderName, InvalidHeader); PyBuffer_Release(&buf); return NULL; } /* Validate method */ if (pico_validate_method(method, method_len, permit_unconventional_http_method, InvalidRequestMethod) < 0) { PyBuffer_Release(&buf); return NULL; } /* Validate request-target form against method (RFC 9112) */ if (pico_validate_request_target(method, method_len, path, path_len, PicoError) < 0) { PyBuffer_Release(&buf); return NULL; } /* Validate HTTP version */ if (pico_validate_version(minor_version, permit_unconventional_http_version, InvalidHTTPVersion) < 0) { PyBuffer_Release(&buf); return NULL; } /* Validate headers */ PicoValidationConfig config = { .limit_request_line = limit_request_line, .limit_request_fields = limit_request_fields, .limit_request_field_size = limit_request_field_size, .permit_unconventional_http_method = permit_unconventional_http_method, .permit_unconventional_http_version = permit_unconventional_http_version }; if (pico_validate_headers_full(headers, num_headers, minor_version, &config, LimitRequestHeaders, InvalidHeaderName, InvalidHeader) < 0) { PyBuffer_Release(&buf); return NULL; } PyBuffer_Release(&buf); /* Success - build ASGI scope dict */ PyObject *scope = PyDict_New(); if (!scope) return NULL; /* type: "http" */ PyObject *py_type = PyUnicode_FromString("http"); if (!py_type) goto error; PyDict_SetItemString(scope, "type", py_type); Py_DECREF(py_type); /* asgi: {"version": "3.0", "spec_version": "2.4"} */ PyObject *asgi_dict = PyDict_New(); if (!asgi_dict) goto error; PyObject *asgi_version = PyUnicode_FromString("3.0"); PyObject *spec_version = PyUnicode_FromString("2.4"); if (!asgi_version || !spec_version) { Py_XDECREF(asgi_version); Py_XDECREF(spec_version); Py_DECREF(asgi_dict); goto error; } PyDict_SetItemString(asgi_dict, "version", asgi_version); PyDict_SetItemString(asgi_dict, "spec_version", spec_version); Py_DECREF(asgi_version); Py_DECREF(spec_version); PyDict_SetItemString(scope, "asgi", asgi_dict); Py_DECREF(asgi_dict); /* http_version: "1.0" or "1.1" */ PyObject *py_http_version = minor_version == 0 ? PyUnicode_FromString("1.0") : PyUnicode_FromString("1.1"); if (!py_http_version) goto error; PyDict_SetItemString(scope, "http_version", py_http_version); Py_DECREF(py_http_version); /* method: str */ PyObject *py_method = PyUnicode_DecodeLatin1(method, method_len, NULL); if (!py_method) goto error; PyDict_SetItemString(scope, "method", py_method); Py_DECREF(py_method); /* scheme: str */ PyObject *py_scheme = PyUnicode_DecodeLatin1(scheme, scheme_len, NULL); if (!py_scheme) goto error; PyDict_SetItemString(scope, "scheme", py_scheme); Py_DECREF(py_scheme); /* Split path into path and query_string */ const char *query_start = find_query_string(path, path_len); size_t path_only_len; const char *query_string; size_t query_string_len; if (query_start) { path_only_len = query_start - path; query_string = query_start + 1; query_string_len = path_len - path_only_len - 1; } else { path_only_len = path_len; query_string = ""; query_string_len = 0; } /* path: str (decoded) */ PyObject *py_path = PyUnicode_DecodeLatin1(path, path_only_len, NULL); if (!py_path) goto error; PyDict_SetItemString(scope, "path", py_path); Py_DECREF(py_path); /* raw_path: bytes (path without query string) */ PyObject *py_raw_path = PyBytes_FromStringAndSize(path, path_only_len); if (!py_raw_path) goto error; PyDict_SetItemString(scope, "raw_path", py_raw_path); Py_DECREF(py_raw_path); /* query_string: bytes */ PyObject *py_query = PyBytes_FromStringAndSize(query_string, query_string_len); if (!py_query) goto error; PyDict_SetItemString(scope, "query_string", py_query); Py_DECREF(py_query); /* root_path: str */ PyObject *py_root_path = PyUnicode_DecodeLatin1(root_path, root_path_len, NULL); if (!py_root_path) goto error; PyDict_SetItemString(scope, "root_path", py_root_path); Py_DECREF(py_root_path); /* server: tuple or None */ if (server && server != Py_None) { PyDict_SetItemString(scope, "server", server); } else { Py_INCREF(Py_None); PyDict_SetItemString(scope, "server", Py_None); Py_DECREF(Py_None); } /* client: tuple or None */ if (client && client != Py_None) { PyDict_SetItemString(scope, "client", client); } else { Py_INCREF(Py_None); PyDict_SetItemString(scope, "client", Py_None); Py_DECREF(Py_None); } /* headers: list of (bytes, bytes) tuples with lowercase names */ PyObject *py_headers = PyList_New(num_headers); if (!py_headers) goto error; for (size_t i = 0; i < num_headers; i++) { const char *name = headers[i].name; size_t name_len = headers[i].name_len; const char *value = headers[i].value; size_t value_len = headers[i].value_len; /* Create lowercase name */ char *lower_name = PyMem_Malloc(name_len); if (!lower_name) { Py_DECREF(py_headers); PyErr_NoMemory(); goto error; } for (size_t j = 0; j < name_len; j++) { char c = name[j]; if (c >= 'A' && c <= 'Z') { c += 32; /* uppercase to lowercase */ } lower_name[j] = c; } PyObject *py_name = PyBytes_FromStringAndSize(lower_name, name_len); PyMem_Free(lower_name); if (!py_name) { Py_DECREF(py_headers); goto error; } PyObject *py_value = PyBytes_FromStringAndSize(value, value_len); if (!py_value) { Py_DECREF(py_name); Py_DECREF(py_headers); goto error; } PyObject *tuple = PyTuple_Pack(2, py_name, py_value); Py_DECREF(py_name); Py_DECREF(py_value); if (!tuple) { Py_DECREF(py_headers); goto error; } PyList_SET_ITEM(py_headers, i, tuple); } PyDict_SetItemString(scope, "headers", py_headers); Py_DECREF(py_headers); /* Add consumed bytes count */ PyObject *py_consumed = PyLong_FromLong(ret); if (!py_consumed) goto error; PyDict_SetItemString(scope, "_consumed", py_consumed); Py_DECREF(py_consumed); return scope; error: Py_DECREF(scope); return NULL; } /* * parse_headers(data: bytes, last_len: int = 0) -> list */ static PyObject * pico_parse_headers(PyObject *self, PyObject *args, PyObject *kwargs) { static char *kwlist[] = {"data", "last_len", NULL}; Py_buffer buf; Py_ssize_t last_len = 0; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "y*|n", kwlist, &buf, &last_len)) { return NULL; } struct phr_header headers[MAX_HEADERS]; size_t num_headers = MAX_HEADERS; int ret = phr_parse_headers( buf.buf, buf.len, headers, &num_headers, (size_t)last_len ); PyBuffer_Release(&buf); if (ret > 0) { PyObject *py_headers = PyList_New(num_headers); if (!py_headers) return NULL; for (size_t i = 0; i < num_headers; i++) { PyObject *tuple = pico_create_header_tuple( headers[i].name, headers[i].name_len, headers[i].value, headers[i].value_len); if (!tuple) { Py_DECREF(py_headers); return NULL; } PyList_SET_ITEM(py_headers, i, tuple); } return py_headers; } else if (ret == -2) { PyErr_SetString(IncompleteError, "Incomplete headers"); return NULL; } else { PyErr_SetString(PicoError, "Invalid headers"); return NULL; } } /* Module method table */ static PyMethodDef pico_methods[] = { {"parse_request", (PyCFunction)pico_parse_request, METH_VARARGS | METH_KEYWORDS, "Parse HTTP request.\n\n" "Args:\n" " data: Raw HTTP request bytes\n" " last_len: Previously parsed length for incremental parsing\n" " limit_request_line: Max request line length (default 8190)\n" " limit_request_fields: Max number of headers (default 100)\n" " limit_request_field_size: Max header size (default 8190)\n" " permit_unconventional_http_method: Allow lowercase methods (default False)\n" " permit_unconventional_http_version: Allow non-1.0/1.1 (default False)\n\n" "Returns:\n" " dict with method, path, minor_version, headers, consumed\n\n" "Raises:\n" " LimitRequestLine: Request line exceeds limit\n" " LimitRequestHeaders: Too many headers or header too large\n" " InvalidRequestMethod: Bad method characters\n" " InvalidHTTPVersion: Not HTTP/1.0 or HTTP/1.1\n" " InvalidHeaderName: Bad header name characters\n" " InvalidHeader: Bad header value (NUL, CR, LF)"}, {"parse_response", (PyCFunction)pico_parse_response, METH_VARARGS | METH_KEYWORDS, "Parse HTTP response.\n\n" "Args:\n" " data: Raw HTTP response bytes\n" " last_len: Previously parsed length for incremental parsing\n\n" "Returns:\n" " dict with status, message, minor_version, headers, consumed"}, {"parse_headers", (PyCFunction)pico_parse_headers, METH_VARARGS | METH_KEYWORDS, "Parse HTTP headers only.\n\n" "Args:\n" " data: Raw header bytes\n" " last_len: Previously parsed length\n\n" "Returns:\n" " list of (name, value) tuples"}, {"parse_to_wsgi_environ", (PyCFunction)pico_parse_to_wsgi_environ, METH_VARARGS | METH_KEYWORDS, "Parse HTTP request and build WSGI environ dict.\n\n" "Args:\n" " data: Raw HTTP request bytes\n" " server: (host, port) tuple for SERVER_NAME/SERVER_PORT\n" " client: (addr, port) tuple for REMOTE_ADDR/REMOTE_PORT\n" " url_scheme: URL scheme (default 'http')\n\n" "Returns:\n" " WSGI environ dict"}, {"parse_to_asgi_scope", (PyCFunction)pico_parse_to_asgi_scope, METH_VARARGS | METH_KEYWORDS, "Parse HTTP request and build ASGI scope dict.\n\n" "Args:\n" " data: Raw HTTP request bytes\n" " server: (host, port) tuple\n" " client: (addr, port) tuple\n" " scheme: URL scheme (default 'http')\n" " root_path: ASGI root_path (default '')\n\n" "Returns:\n" " ASGI scope dict"}, {NULL, NULL, 0, NULL} }; /* Module definition */ static struct PyModuleDef pico_module = { PyModuleDef_HEAD_INIT, "gunicorn_h1c._parser", "Fast HTTP parser using picohttpparser.\n\n" "This module provides high-performance HTTP parsing using the\n" "picohttpparser library with SIMD optimizations.", -1, pico_methods }; /* Module initialization */ PyMODINIT_FUNC PyInit__parser(void) { PyObject *m = PyModule_Create(&pico_module); if (m == NULL) return NULL; /* Create base exception types */ PicoError = PyErr_NewException("gunicorn_h1c._parser.ParseError", PyExc_ValueError, NULL); if (!PicoError) return NULL; Py_INCREF(PicoError); PyModule_AddObject(m, "ParseError", PicoError); IncompleteError = PyErr_NewException("gunicorn_h1c._parser.IncompleteError", PyExc_Exception, NULL); if (!IncompleteError) return NULL; Py_INCREF(IncompleteError); PyModule_AddObject(m, "IncompleteError", IncompleteError); /* Create specific validation exception types (inherit from ParseError) */ LimitRequestLine = PyErr_NewException("gunicorn_h1c._parser.LimitRequestLine", PicoError, NULL); if (!LimitRequestLine) return NULL; Py_INCREF(LimitRequestLine); PyModule_AddObject(m, "LimitRequestLine", LimitRequestLine); LimitRequestHeaders = PyErr_NewException("gunicorn_h1c._parser.LimitRequestHeaders", PicoError, NULL); if (!LimitRequestHeaders) return NULL; Py_INCREF(LimitRequestHeaders); PyModule_AddObject(m, "LimitRequestHeaders", LimitRequestHeaders); InvalidRequestMethod = PyErr_NewException("gunicorn_h1c._parser.InvalidRequestMethod", PicoError, NULL); if (!InvalidRequestMethod) return NULL; Py_INCREF(InvalidRequestMethod); PyModule_AddObject(m, "InvalidRequestMethod", InvalidRequestMethod); InvalidHTTPVersion = PyErr_NewException("gunicorn_h1c._parser.InvalidHTTPVersion", PicoError, NULL); if (!InvalidHTTPVersion) return NULL; Py_INCREF(InvalidHTTPVersion); PyModule_AddObject(m, "InvalidHTTPVersion", InvalidHTTPVersion); InvalidHeaderName = PyErr_NewException("gunicorn_h1c._parser.InvalidHeaderName", PicoError, NULL); if (!InvalidHeaderName) return NULL; Py_INCREF(InvalidHeaderName); PyModule_AddObject(m, "InvalidHeaderName", InvalidHeaderName); InvalidHeader = PyErr_NewException("gunicorn_h1c._parser.InvalidHeader", PicoError, NULL); if (!InvalidHeader) return NULL; Py_INCREF(InvalidHeader); PyModule_AddObject(m, "InvalidHeader", InvalidHeader); InvalidChunkExtension = PyErr_NewException("gunicorn_h1c._parser.InvalidChunkExtension", PicoError, NULL); if (!InvalidChunkExtension) return NULL; Py_INCREF(InvalidChunkExtension); PyModule_AddObject(m, "InvalidChunkExtension", InvalidChunkExtension); return m; } benoitc-gunicorn_h1c-52625fd/src/gunicorn_h1c/pico_protocol.c000066400000000000000000001211771524260035600242520ustar00rootroot00000000000000/* * SPDX-License-Identifier: MIT * * Copyright 2026 Benoit Chesneau * Licensed under the MIT License (see LICENSE file). * * Callback-based HTTP/1.1 parser for asyncio integration. * * This module provides H1CProtocol, a zero-copy callback parser * optimized for asyncio's data_received() pattern. */ #define PY_SSIZE_T_CLEAN #include #include "picohttpparser.h" #include "pico_utils.h" #define MAX_HEADERS 256 #define INITIAL_BUFFER_SIZE 4096 /* Cap on bytes retained after a completed message (0 = unlimited). * Bounds memory when a caller keeps feeding a parser that is already * complete, which would otherwise grow without limit. */ #define PICO_DEFAULT_LIMIT_REMAINING 65536 /* Parser states */ #define STATE_IDLE 0 #define STATE_HEADERS 1 #define STATE_BODY 2 #define STATE_BODY_CHUNKED_SIZE 3 #define STATE_BODY_CHUNKED_DATA 4 #define STATE_BODY_CHUNKED_CRLF 5 #define STATE_BODY_CHUNKED_TRAILER 6 #define STATE_COMPLETE 7 /* ParseError exception */ static PyObject *ParseError; /* Imported validation exceptions from _parser module */ static PyObject *LimitRequestLine; static PyObject *LimitRequestHeaders; static PyObject *InvalidRequestMethod; static PyObject *InvalidHTTPVersion; static PyObject *InvalidHeaderName; static PyObject *InvalidHeader; static PyObject *InvalidChunkExtension; /* * H1CProtocol type */ typedef struct { PyObject_HEAD /* Callbacks */ PyObject *on_message_begin; PyObject *on_url; PyObject *on_header; PyObject *on_headers_complete; PyObject *on_body; PyObject *on_message_complete; /* Validation configuration */ Py_ssize_t limit_request_line; Py_ssize_t limit_request_fields; Py_ssize_t limit_request_field_size; Py_ssize_t limit_remaining; int permit_unconventional_http_method; int permit_unconventional_http_version; int remaining_truncated; /* Parser state */ int state; char *buffer; size_t buffer_size; size_t buffer_len; size_t last_len; /* Request data (valid after headers complete) */ PyObject *py_method; PyObject *py_path; PyObject *py_headers; /* list of (name, value) tuples */ PyObject *py_asgi_headers; /* list with lowercase names for ASGI */ int minor_version; Py_ssize_t content_length; /* -1 if not set */ int is_chunked; int connection_close; /* -1=unset, 0=keep-alive, 1=close */ int should_upgrade; /* Body parsing state */ Py_ssize_t body_remaining; size_t chunk_size; size_t chunk_remaining; int skip_body; } H1CProtocol; /* Forward declarations */ static int H1CProtocol_feed_headers(H1CProtocol *self); static int H1CProtocol_feed_body_content_length(H1CProtocol *self, const char *data, size_t len, int data_is_buffer); static int H1CProtocol_feed_body_chunked(H1CProtocol *self); static int H1CProtocol_keep_tail(H1CProtocol *self, const char *data, size_t len, size_t consumed, int data_is_buffer); static void H1CProtocol_clamp_tail(H1CProtocol *self); /* header_name_eq moved to pico_utils.h as pico_header_name_eq */ /* Ensure buffer can hold additional bytes */ static int ensure_buffer_capacity(H1CProtocol *self, size_t additional) { size_t needed = self->buffer_len + additional; if (needed <= self->buffer_size) { return 0; } /* Grow buffer */ size_t new_size = self->buffer_size * 2; while (new_size < needed) { new_size *= 2; } char *new_buffer = PyMem_Realloc(self->buffer, new_size); if (!new_buffer) { PyErr_NoMemory(); return -1; } self->buffer = new_buffer; self->buffer_size = new_size; return 0; } /* * Keep the bytes that follow a finished message so remaining() can hand them * back: a pipelined request, a WebSocket frame sent straight after the * handshake, or the HTTP/2 preface after an Upgrade: h2c request. * * `data` is the block just parsed and `consumed` how much of it the message * used. data_is_buffer says whether `data` points into self->buffer (body fed * on from the header buffer) or at caller-owned memory, since the first case * must shift in place rather than copy over itself. */ /* * Bound the retained tail. Overflow is never an error here: feed() must not * start raising on input it used to discard, or a caller that keeps feeding a * completed parser would see its connections fail. The excess is dropped and * the loss is reported through remaining_truncated, which only a caller that * asks for the tail will look at. */ static void H1CProtocol_clamp_tail(H1CProtocol *self) { if (self->limit_remaining == 0) { return; /* unlimited */ } if (self->buffer_len > (size_t)self->limit_remaining) { self->buffer_len = (size_t)self->limit_remaining; self->remaining_truncated = 1; } } static int H1CProtocol_keep_tail(H1CProtocol *self, const char *data, size_t len, size_t consumed, int data_is_buffer) { size_t tail_len = len - consumed; if (tail_len == 0) { self->buffer_len = 0; return 0; } if (data_is_buffer) { memmove(self->buffer, data + consumed, tail_len); } else { self->buffer_len = 0; if (ensure_buffer_capacity(self, tail_len) < 0) { return -1; } memcpy(self->buffer, data + consumed, tail_len); } self->buffer_len = tail_len; H1CProtocol_clamp_tail(self); return 0; } static void H1CProtocol_dealloc(H1CProtocol *self) { Py_XDECREF(self->on_message_begin); Py_XDECREF(self->on_url); Py_XDECREF(self->on_header); Py_XDECREF(self->on_headers_complete); Py_XDECREF(self->on_body); Py_XDECREF(self->on_message_complete); Py_XDECREF(self->py_method); Py_XDECREF(self->py_path); Py_XDECREF(self->py_headers); Py_XDECREF(self->py_asgi_headers); if (self->buffer) { PyMem_Free(self->buffer); } Py_TYPE(self)->tp_free((PyObject *)self); } static int H1CProtocol_init(H1CProtocol *self, PyObject *args, PyObject *kwargs) { static char *kwlist[] = { "on_message_begin", "on_url", "on_header", "on_headers_complete", "on_body", "on_message_complete", "limit_request_line", "limit_request_fields", "limit_request_field_size", "permit_unconventional_http_method", "permit_unconventional_http_version", "limit_remaining", NULL }; PyObject *on_message_begin = NULL; PyObject *on_url = NULL; PyObject *on_header = NULL; PyObject *on_headers_complete = NULL; PyObject *on_body = NULL; PyObject *on_message_complete = NULL; Py_ssize_t limit_request_line = PICO_DEFAULT_LIMIT_REQUEST_LINE; Py_ssize_t limit_request_fields = PICO_DEFAULT_LIMIT_REQUEST_FIELDS; Py_ssize_t limit_request_field_size = PICO_DEFAULT_LIMIT_REQUEST_FIELD_SIZE; int permit_unconventional_http_method = 0; int permit_unconventional_http_version = 0; Py_ssize_t limit_remaining = PICO_DEFAULT_LIMIT_REMAINING; if (!PyArg_ParseTupleAndKeywords(args, kwargs, "|OOOOOOnnnppn", kwlist, &on_message_begin, &on_url, &on_header, &on_headers_complete, &on_body, &on_message_complete, &limit_request_line, &limit_request_fields, &limit_request_field_size, &permit_unconventional_http_method, &permit_unconventional_http_version, &limit_remaining)) { return -1; } if (limit_remaining < 0) { PyErr_SetString(PyExc_ValueError, "limit_remaining must not be negative"); return -1; } /* Store callbacks (allow None to mean no callback) */ if (on_message_begin && on_message_begin != Py_None) { Py_INCREF(on_message_begin); self->on_message_begin = on_message_begin; } if (on_url && on_url != Py_None) { Py_INCREF(on_url); self->on_url = on_url; } if (on_header && on_header != Py_None) { Py_INCREF(on_header); self->on_header = on_header; } if (on_headers_complete && on_headers_complete != Py_None) { Py_INCREF(on_headers_complete); self->on_headers_complete = on_headers_complete; } if (on_body && on_body != Py_None) { Py_INCREF(on_body); self->on_body = on_body; } if (on_message_complete && on_message_complete != Py_None) { Py_INCREF(on_message_complete); self->on_message_complete = on_message_complete; } /* Store validation config */ self->limit_request_line = limit_request_line; self->limit_request_fields = limit_request_fields; self->limit_request_field_size = limit_request_field_size; self->limit_remaining = limit_remaining; self->remaining_truncated = 0; self->permit_unconventional_http_method = permit_unconventional_http_method; self->permit_unconventional_http_version = permit_unconventional_http_version; /* Initialize state */ self->state = STATE_IDLE; self->buffer = PyMem_Malloc(INITIAL_BUFFER_SIZE); if (!self->buffer) { PyErr_NoMemory(); return -1; } self->buffer_size = INITIAL_BUFFER_SIZE; self->buffer_len = 0; self->last_len = 0; /* Request data */ self->py_method = NULL; self->py_path = NULL; self->py_headers = NULL; self->py_asgi_headers = NULL; self->minor_version = 1; self->content_length = -1; self->is_chunked = 0; self->connection_close = -1; self->should_upgrade = 0; /* Body state */ self->body_remaining = 0; self->chunk_size = 0; self->chunk_remaining = 0; self->skip_body = 0; return 0; } /* * Feed data to the parser. Callbacks fire synchronously. */ static PyObject * H1CProtocol_feed(H1CProtocol *self, PyObject *args) { Py_buffer buf; if (!PyArg_ParseTuple(args, "y*", &buf)) { return NULL; } if (buf.len == 0) { PyBuffer_Release(&buf); Py_RETURN_NONE; } /* Handle state transitions */ if (self->state == STATE_IDLE) { self->state = STATE_HEADERS; /* Fire on_message_begin */ if (self->on_message_begin) { PyObject *result = PyObject_CallNoArgs(self->on_message_begin); if (!result) { PyBuffer_Release(&buf); return NULL; } Py_DECREF(result); } } if (self->state == STATE_HEADERS) { /* Buffer data for header parsing */ if (ensure_buffer_capacity(self, buf.len) < 0) { PyBuffer_Release(&buf); return NULL; } memcpy(self->buffer + self->buffer_len, buf.buf, buf.len); self->buffer_len += buf.len; PyBuffer_Release(&buf); if (H1CProtocol_feed_headers(self) < 0) { return NULL; } } else if (self->state == STATE_BODY) { int ret = H1CProtocol_feed_body_content_length(self, buf.buf, buf.len, 0); PyBuffer_Release(&buf); if (ret < 0) { return NULL; } } else if (self->state >= STATE_BODY_CHUNKED_SIZE && self->state <= STATE_BODY_CHUNKED_TRAILER) { /* Buffer data for chunked parsing */ if (ensure_buffer_capacity(self, buf.len) < 0) { PyBuffer_Release(&buf); return NULL; } memcpy(self->buffer + self->buffer_len, buf.buf, buf.len); self->buffer_len += buf.len; PyBuffer_Release(&buf); if (H1CProtocol_feed_body_chunked(self) < 0) { return NULL; } } else if (self->state == STATE_COMPLETE) { /* The message is already finished, so every byte here belongs to * whatever follows it. Accumulate so a tail split across several * feed() calls still arrives whole, but never past limit_remaining: * a caller that keeps feeding a completed parser must not be able to * grow this buffer without bound. */ size_t take = (size_t)buf.len; if (self->limit_remaining > 0) { size_t cap = (size_t)self->limit_remaining; size_t room = (self->buffer_len >= cap) ? 0 : cap - self->buffer_len; if (take > room) { take = room; self->remaining_truncated = 1; } } if (take > 0) { if (ensure_buffer_capacity(self, take) < 0) { PyBuffer_Release(&buf); return NULL; } memcpy(self->buffer + self->buffer_len, buf.buf, take); self->buffer_len += take; } PyBuffer_Release(&buf); } else { PyBuffer_Release(&buf); } Py_RETURN_NONE; } /* * Parse headers from buffer. * Returns 0 on success (or need more data), -1 on error. */ static int H1CProtocol_feed_headers(H1CProtocol *self) { /* Check request line length before parsing */ Py_ssize_t req_line_len = pico_find_request_line_length(self->buffer, self->buffer_len); if (req_line_len >= 0 && req_line_len > self->limit_request_line) { PyErr_Format(LimitRequestLine, "Request line length (%zd) exceeds limit (%zd)", req_line_len, self->limit_request_line); return -1; } const char *method, *path; size_t method_len, path_len; int minor_version; struct phr_header headers[MAX_HEADERS]; size_t num_headers = MAX_HEADERS; int ret = phr_parse_request( self->buffer, self->buffer_len, &method, &method_len, &path, &path_len, &minor_version, headers, &num_headers, self->last_len ); if (ret == -2) { /* Need more data */ self->last_len = self->buffer_len; return 0; } if (ret == -1) { pico_analyze_parse_error(self->buffer, self->buffer_len, self->permit_unconventional_http_method, self->permit_unconventional_http_version, ParseError, InvalidRequestMethod, InvalidHTTPVersion, InvalidHeaderName, InvalidHeader); return -1; } /* Validate method */ if (pico_validate_method(method, method_len, self->permit_unconventional_http_method, InvalidRequestMethod) < 0) { return -1; } /* Validate request-target form against method (RFC 9112) */ if (pico_validate_request_target(method, method_len, path, path_len, ParseError) < 0) { return -1; } /* Validate HTTP version */ if (pico_validate_version(minor_version, self->permit_unconventional_http_version, InvalidHTTPVersion) < 0) { return -1; } /* Validate headers */ PicoValidationConfig config = { .limit_request_line = self->limit_request_line, .limit_request_fields = self->limit_request_fields, .limit_request_field_size = self->limit_request_field_size, .permit_unconventional_http_method = self->permit_unconventional_http_method, .permit_unconventional_http_version = self->permit_unconventional_http_version }; if (pico_validate_headers(headers, num_headers, &config, LimitRequestHeaders, InvalidHeaderName, InvalidHeader) < 0) { return -1; } /* Headers complete - store data */ self->minor_version = minor_version; /* Create method bytes */ Py_XDECREF(self->py_method); self->py_method = PyBytes_FromStringAndSize(method, method_len); if (!self->py_method) return -1; /* Create path bytes */ Py_XDECREF(self->py_path); self->py_path = PyBytes_FromStringAndSize(path, path_len); if (!self->py_path) return -1; /* Fire on_url callback */ if (self->on_url) { PyObject *result = PyObject_CallOneArg(self->on_url, self->py_path); if (!result) return -1; Py_DECREF(result); } /* Process headers */ Py_XDECREF(self->py_headers); self->py_headers = PyList_New(num_headers); if (!self->py_headers) return -1; /* Clear cached ASGI headers */ Py_XDECREF(self->py_asgi_headers); self->py_asgi_headers = NULL; /* Extract special headers using shared utility */ PicoHeaderInfo hdr_info; pico_extract_headers(headers, num_headers, &hdr_info); /* Validate header framing for request smuggling prevention */ if (pico_validate_header_framing(&hdr_info, minor_version, InvalidHeader) < 0) { return -1; } self->content_length = hdr_info.content_length; self->is_chunked = hdr_info.has_chunked; self->connection_close = hdr_info.connection_close; self->should_upgrade = hdr_info.should_upgrade; for (size_t i = 0; i < num_headers; i++) { const char *name = headers[i].name; size_t name_len = headers[i].name_len; const char *value = headers[i].value; size_t value_len = headers[i].value_len; /* Create header tuple using shared utility */ PyObject *tuple = pico_create_header_tuple(name, name_len, value, value_len); if (!tuple) return -1; PyList_SET_ITEM(self->py_headers, i, tuple); /* Fire on_header callback */ if (self->on_header) { PyObject *name_obj = PyBytes_FromStringAndSize(name, name_len); PyObject *value_obj = PyBytes_FromStringAndSize(value, value_len); if (!name_obj || !value_obj) { Py_XDECREF(name_obj); Py_XDECREF(value_obj); return -1; } PyObject *result = PyObject_CallFunctionObjArgs( self->on_header, name_obj, value_obj, NULL); Py_DECREF(name_obj); Py_DECREF(value_obj); if (!result) return -1; Py_DECREF(result); } } /* Fire on_headers_complete callback */ int skip_body = 0; if (self->on_headers_complete) { PyObject *result = PyObject_CallNoArgs(self->on_headers_complete); if (!result) return -1; if (result == Py_True) { skip_body = 1; } Py_DECREF(result); } self->skip_body = skip_body; /* Keep unconsumed data */ size_t consumed = ret; size_t remaining = self->buffer_len - consumed; if (remaining > 0) { memmove(self->buffer, self->buffer + consumed, remaining); } self->buffer_len = remaining; self->last_len = 0; /* Determine body parsing mode. * * An Upgrade header does not suppress the body. The protocol switch only * happens once the server answers 101 (RFC 9110 section 7.8), which is * after the request has been read in full, and RFC 7540 section 3.2 * requires an h2c client to send any payload in its entirety before its * first HTTP/2 frame. So the bytes after the headers are the request * body, framed by Content-Length or Transfer-Encoding as usual, and * remaining() starts at the end of it. * * CONNECT needs no case of its own: it carries no framing, so it lands in * the no-body branch below and its tunnel bytes stay in remaining(). */ if (skip_body) { self->state = STATE_COMPLETE; H1CProtocol_clamp_tail(self); if (self->on_message_complete) { PyObject *result = PyObject_CallNoArgs(self->on_message_complete); if (!result) return -1; Py_DECREF(result); } } else if (self->is_chunked) { self->state = STATE_BODY_CHUNKED_SIZE; self->chunk_size = 0; self->chunk_remaining = 0; if (remaining > 0) { return H1CProtocol_feed_body_chunked(self); } } else if (self->content_length > 0) { self->body_remaining = self->content_length; self->state = STATE_BODY; if (remaining > 0) { return H1CProtocol_feed_body_content_length( self, self->buffer, remaining, 1); } } else { /* No body */ self->state = STATE_COMPLETE; H1CProtocol_clamp_tail(self); if (self->on_message_complete) { PyObject *result = PyObject_CallNoArgs(self->on_message_complete); if (!result) return -1; Py_DECREF(result); } } return 0; } /* * Parse body with Content-Length. */ static int H1CProtocol_feed_body_content_length(H1CProtocol *self, const char *data, size_t len, int data_is_buffer) { if (self->skip_body) { if ((Py_ssize_t)len >= self->body_remaining) { size_t consumed = (size_t)self->body_remaining; self->body_remaining = 0; self->state = STATE_COMPLETE; if (H1CProtocol_keep_tail(self, data, len, consumed, data_is_buffer) < 0) { return -1; } if (self->on_message_complete) { PyObject *result = PyObject_CallNoArgs(self->on_message_complete); if (!result) return -1; Py_DECREF(result); } } else { self->body_remaining -= len; /* Clear buffer since we're not using buffered data for body */ self->buffer_len = 0; } return 0; } size_t to_consume; if ((Py_ssize_t)len <= self->body_remaining) { to_consume = len; } else { to_consume = (size_t)self->body_remaining; } /* Fire on_body callback */ if (self->on_body && to_consume > 0) { PyObject *chunk = PyBytes_FromStringAndSize(data, to_consume); if (!chunk) return -1; PyObject *result = PyObject_CallOneArg(self->on_body, chunk); Py_DECREF(chunk); if (!result) return -1; Py_DECREF(result); } self->body_remaining -= to_consume; if (self->body_remaining > 0) { self->buffer_len = 0; /* Clear buffer */ } if (self->body_remaining == 0) { self->state = STATE_COMPLETE; /* Anything past the body belongs to the next message */ if (H1CProtocol_keep_tail(self, data, len, to_consume, data_is_buffer) < 0) { return -1; } if (self->on_message_complete) { PyObject *result = PyObject_CallNoArgs(self->on_message_complete); if (!result) return -1; Py_DECREF(result); } } return 0; } /* * Parse chunked transfer encoding body. */ static int H1CProtocol_feed_body_chunked(H1CProtocol *self) { while (1) { if (self->state == STATE_BODY_CHUNKED_SIZE) { /* Look for chunk size line */ char *crlf = memmem(self->buffer, self->buffer_len, "\r\n", 2); if (!crlf) { return 0; /* Need more data */ } size_t line_len = crlf - self->buffer; /* Parse chunk size (may have extensions after ;) */ char *semicolon = memchr(self->buffer, ';', line_len); size_t size_len = semicolon ? (size_t)(semicolon - self->buffer) : line_len; /* RFC 9112: chunk-ext must not contain bare CR */ if (semicolon) { size_t ext_len = line_len - size_len - 1; if (memchr(semicolon + 1, '\r', ext_len)) { PyErr_SetString(InvalidChunkExtension, "Invalid chunk extension: bare CR not allowed"); return -1; } } /* Reject empty chunk size */ if (size_len == 0) { PyErr_SetString(ParseError, "Empty chunk size"); return -1; } /* Reject leading/trailing whitespace (request smuggling vector) */ if (self->buffer[0] == ' ' || self->buffer[0] == '\t') { PyErr_SetString(ParseError, "Whitespace in chunk size"); return -1; } if (size_len > 0 && (self->buffer[size_len-1] == ' ' || self->buffer[size_len-1] == '\t')) { PyErr_SetString(ParseError, "Whitespace in chunk size"); return -1; } /* Parse hex number - strict validation, no whitespace allowed */ size_t chunk_size = 0; for (size_t i = 0; i < size_len; i++) { char c = self->buffer[i]; size_t digit; if (c >= '0' && c <= '9') { digit = c - '0'; } else if (c >= 'a' && c <= 'f') { digit = c - 'a' + 10; } else if (c >= 'A' && c <= 'F') { digit = c - 'A' + 10; } else { /* Reject any non-hex character including whitespace */ PyErr_SetString(ParseError, "Invalid character in chunk size"); return -1; } chunk_size = chunk_size * 16 + digit; } self->chunk_size = chunk_size; self->chunk_remaining = chunk_size; /* Remove size line from buffer */ size_t consumed = line_len + 2; memmove(self->buffer, self->buffer + consumed, self->buffer_len - consumed); self->buffer_len -= consumed; if (chunk_size == 0) { /* Last chunk - parse trailers */ self->state = STATE_BODY_CHUNKED_TRAILER; } else { self->state = STATE_BODY_CHUNKED_DATA; } } else if (self->state == STATE_BODY_CHUNKED_DATA) { if (self->buffer_len == 0) { return 0; /* Need more data */ } size_t available = self->buffer_len; if (available > self->chunk_remaining) { available = self->chunk_remaining; } /* Fire on_body callback */ if (!self->skip_body && self->on_body && available > 0) { PyObject *chunk = PyBytes_FromStringAndSize(self->buffer, available); if (!chunk) return -1; PyObject *result = PyObject_CallOneArg(self->on_body, chunk); Py_DECREF(chunk); if (!result) return -1; Py_DECREF(result); } /* Remove consumed data from buffer */ memmove(self->buffer, self->buffer + available, self->buffer_len - available); self->buffer_len -= available; self->chunk_remaining -= available; if (self->chunk_remaining == 0) { self->state = STATE_BODY_CHUNKED_CRLF; } } else if (self->state == STATE_BODY_CHUNKED_CRLF) { /* Expect CRLF after chunk data */ if (self->buffer_len < 2) { return 0; /* Need more data */ } if (self->buffer[0] != '\r' || self->buffer[1] != '\n') { PyErr_SetString(ParseError, "Missing CRLF after chunk"); return -1; } memmove(self->buffer, self->buffer + 2, self->buffer_len - 2); self->buffer_len -= 2; self->state = STATE_BODY_CHUNKED_SIZE; } else if (self->state == STATE_BODY_CHUNKED_TRAILER) { /* Parse trailers (we just skip them) */ char *crlf = memmem(self->buffer, self->buffer_len, "\r\n", 2); if (!crlf) { return 0; /* Need more data */ } size_t line_len = crlf - self->buffer; if (line_len == 0) { /* Empty line - end of trailers */ memmove(self->buffer, self->buffer + 2, self->buffer_len - 2); self->buffer_len -= 2; self->state = STATE_COMPLETE; H1CProtocol_clamp_tail(self); if (self->on_message_complete) { PyObject *result = PyObject_CallNoArgs(self->on_message_complete); if (!result) return -1; Py_DECREF(result); } return 0; } /* RFC 9110 section 6.5.1: forbid routing/framing/auth fields * in a trailer. Find the colon that ends the field-name. */ char *colon = memchr(self->buffer, ':', line_len); if (colon) { char *name_start = self->buffer; size_t name_len = (size_t)(colon - name_start); while (name_len > 0 && (name_start[name_len - 1] == ' ' || name_start[name_len - 1] == '\t')) { name_len--; } if (pico_is_forbidden_trailer_name(name_start, name_len)) { PyErr_Format(InvalidHeaderName, "Header field not allowed in trailer: %.*s", (int)name_len, name_start); return -1; } } /* Skip trailer line */ size_t consumed = line_len + 2; memmove(self->buffer, self->buffer + consumed, self->buffer_len - consumed); self->buffer_len -= consumed; } else { break; } } return 0; } /* * Return the bytes fed to the parser that follow the finished message. */ static PyObject * H1CProtocol_remaining(H1CProtocol *self, PyObject *Py_UNUSED(ignored)) { /* Only meaningful once the message is complete. While one is still being * parsed, every byte fed so far belongs to it, so report nothing. */ if (self->state != STATE_COMPLETE) { return PyBytes_FromStringAndSize(NULL, 0); } return PyBytes_FromStringAndSize(self->buffer, (Py_ssize_t)self->buffer_len); } /* * True if the tail overflowed limit_remaining and bytes were dropped. */ static PyObject * H1CProtocol_get_remaining_truncated(H1CProtocol *self, void *closure) { return PyBool_FromLong(self->remaining_truncated); } /* * Mark parsing complete for EOF handling. * Call when no more data will be received. Handles edge cases like * chunked encoding without final trailer CRLF. */ static PyObject * H1CProtocol_finish(H1CProtocol *self, PyObject *Py_UNUSED(ignored)) { if (self->state == STATE_BODY_CHUNKED_TRAILER) { /* All body data received, just missing final CRLF */ self->state = STATE_COMPLETE; /* EOF: nothing follows, so any half-parsed trailer is not a tail */ self->buffer_len = 0; if (self->on_message_complete) { PyObject *result = PyObject_CallNoArgs(self->on_message_complete); if (!result) return NULL; Py_DECREF(result); } } Py_RETURN_NONE; } /* * Reset the parser for the next request (keepalive). */ static PyObject * H1CProtocol_reset(H1CProtocol *self, PyObject *Py_UNUSED(ignored)) { self->state = STATE_IDLE; /* Drops any tail: capture remaining() first, then feed it back after * reset() so it is parsed as the next message. */ self->buffer_len = 0; self->remaining_truncated = 0; self->last_len = 0; Py_XDECREF(self->py_method); self->py_method = NULL; Py_XDECREF(self->py_path); self->py_path = NULL; Py_XDECREF(self->py_headers); self->py_headers = NULL; Py_XDECREF(self->py_asgi_headers); self->py_asgi_headers = NULL; self->minor_version = 1; self->content_length = -1; self->is_chunked = 0; self->connection_close = -1; self->should_upgrade = 0; self->body_remaining = 0; self->chunk_size = 0; self->chunk_remaining = 0; self->skip_body = 0; Py_RETURN_NONE; } /* Property getters */ static PyObject * H1CProtocol_get_method(H1CProtocol *self, void *closure) { if (self->py_method) { Py_INCREF(self->py_method); return self->py_method; } return PyBytes_FromString(""); } static PyObject * H1CProtocol_get_path(H1CProtocol *self, void *closure) { if (self->py_path) { Py_INCREF(self->py_path); return self->py_path; } return PyBytes_FromString(""); } static PyObject * H1CProtocol_get_http_version(H1CProtocol *self, void *closure) { return Py_BuildValue("(ii)", 1, self->minor_version); } static PyObject * H1CProtocol_get_headers(H1CProtocol *self, void *closure) { if (self->py_headers) { Py_INCREF(self->py_headers); return self->py_headers; } return PyList_New(0); } static PyObject * H1CProtocol_get_asgi_headers(H1CProtocol *self, void *closure) { if (!self->py_headers) { return PyList_New(0); } if (!self->py_asgi_headers) { Py_ssize_t n = PyList_GET_SIZE(self->py_headers); self->py_asgi_headers = PyList_New(n); if (!self->py_asgi_headers) return NULL; for (Py_ssize_t i = 0; i < n; i++) { PyObject *tuple = PyList_GET_ITEM(self->py_headers, i); PyObject *name = PyTuple_GET_ITEM(tuple, 0); PyObject *value = PyTuple_GET_ITEM(tuple, 1); const char *name_str = PyBytes_AS_STRING(name); Py_ssize_t name_len = PyBytes_GET_SIZE(name); PyObject *pair = pico_create_header_tuple_lowercase( name_str, name_len, PyBytes_AS_STRING(value), PyBytes_GET_SIZE(value)); if (!pair) { Py_DECREF(self->py_asgi_headers); self->py_asgi_headers = NULL; return NULL; } PyList_SET_ITEM(self->py_asgi_headers, i, pair); } } Py_INCREF(self->py_asgi_headers); return self->py_asgi_headers; } static PyObject * H1CProtocol_get_content_length(H1CProtocol *self, void *closure) { if (self->content_length < 0) { Py_RETURN_NONE; } return PyLong_FromSsize_t(self->content_length); } static PyObject * H1CProtocol_get_is_chunked(H1CProtocol *self, void *closure) { return PyBool_FromLong(self->is_chunked); } static PyObject * H1CProtocol_get_should_keep_alive(H1CProtocol *self, void *closure) { if (self->connection_close == 1) { Py_RETURN_FALSE; } if (self->connection_close == 0) { Py_RETURN_TRUE; } /* Default based on HTTP version */ if (self->minor_version >= 1) { Py_RETURN_TRUE; /* HTTP/1.1+ defaults to keep-alive */ } Py_RETURN_FALSE; /* HTTP/1.0 defaults to close */ } static PyObject * H1CProtocol_get_should_upgrade(H1CProtocol *self, void *closure) { return PyBool_FromLong(self->should_upgrade); } static PyObject * H1CProtocol_get_is_complete(H1CProtocol *self, void *closure) { return PyBool_FromLong(self->state == STATE_COMPLETE); } /* get_header method */ static PyObject * H1CProtocol_get_header(H1CProtocol *self, PyObject *args) { Py_buffer name_buf; if (!PyArg_ParseTuple(args, "y*", &name_buf)) { return NULL; } if (!self->py_headers) { PyBuffer_Release(&name_buf); Py_RETURN_NONE; } Py_ssize_t n = PyList_GET_SIZE(self->py_headers); for (Py_ssize_t i = 0; i < n; i++) { PyObject *tuple = PyList_GET_ITEM(self->py_headers, i); PyObject *hname = PyTuple_GET_ITEM(tuple, 0); PyObject *hvalue = PyTuple_GET_ITEM(tuple, 1); /* Case-insensitive compare */ const char *hname_str = PyBytes_AS_STRING(hname); Py_ssize_t hname_len = PyBytes_GET_SIZE(hname); if (hname_len == (Py_ssize_t)name_buf.len) { int match = 1; for (Py_ssize_t j = 0; j < hname_len; j++) { char c1 = hname_str[j]; char c2 = ((char *)name_buf.buf)[j]; if (c1 >= 'A' && c1 <= 'Z') c1 += 32; if (c2 >= 'A' && c2 <= 'Z') c2 += 32; if (c1 != c2) { match = 0; break; } } if (match) { PyBuffer_Release(&name_buf); Py_INCREF(hvalue); return hvalue; } } } PyBuffer_Release(&name_buf); Py_RETURN_NONE; } static PyGetSetDef H1CProtocol_getset[] = { {"method", (getter)H1CProtocol_get_method, NULL, "HTTP method (GET, POST, etc.)", NULL}, {"path", (getter)H1CProtocol_get_path, NULL, "Request path including query string", NULL}, {"http_version", (getter)H1CProtocol_get_http_version, NULL, "HTTP version as (major, minor) tuple", NULL}, {"headers", (getter)H1CProtocol_get_headers, NULL, "List of (name, value) header tuples", NULL}, {"asgi_headers", (getter)H1CProtocol_get_asgi_headers, NULL, "Headers with lowercase names for ASGI", NULL}, {"content_length", (getter)H1CProtocol_get_content_length, NULL, "Content-Length header value, or None if not set", NULL}, {"is_chunked", (getter)H1CProtocol_get_is_chunked, NULL, "True if Transfer-Encoding: chunked", NULL}, {"should_keep_alive", (getter)H1CProtocol_get_should_keep_alive, NULL, "True if the connection should be kept alive", NULL}, {"should_upgrade", (getter)H1CProtocol_get_should_upgrade, NULL, "True if the Upgrade header is present", NULL}, {"is_complete", (getter)H1CProtocol_get_is_complete, NULL, "True if the message parsing is complete", NULL}, {"remaining_truncated", (getter)H1CProtocol_get_remaining_truncated, NULL, "True if the tail exceeded limit_remaining and bytes were dropped", NULL}, {NULL} }; static PyMethodDef H1CProtocol_methods[] = { {"feed", (PyCFunction)H1CProtocol_feed, METH_VARARGS, "Feed data to the parser. Callbacks fire synchronously."}, {"finish", (PyCFunction)H1CProtocol_finish, METH_NOARGS, "Mark parsing complete for EOF handling."}, {"reset", (PyCFunction)H1CProtocol_reset, METH_NOARGS, "Reset the parser for the next request (keepalive)."}, {"get_header", (PyCFunction)H1CProtocol_get_header, METH_VARARGS, "Get header value by name (case-insensitive)."}, {"remaining", (PyCFunction)H1CProtocol_remaining, METH_NOARGS, "Bytes fed after the completed message (b'' if none or not complete)."}, {NULL} }; static PyTypeObject H1CProtocolType = { PyVarObject_HEAD_INIT(NULL, 0) .tp_name = "gunicorn_h1c._protocol.H1CProtocol", .tp_doc = "Callback-based HTTP/1.1 parser for asyncio integration", .tp_basicsize = sizeof(H1CProtocol), .tp_itemsize = 0, .tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, .tp_new = PyType_GenericNew, .tp_init = (initproc)H1CProtocol_init, .tp_dealloc = (destructor)H1CProtocol_dealloc, .tp_getset = H1CProtocol_getset, .tp_methods = H1CProtocol_methods, }; /* Module definition */ static PyMethodDef module_methods[] = { {NULL, NULL, 0, NULL} }; static struct PyModuleDef protocol_module = { PyModuleDef_HEAD_INIT, "gunicorn_h1c._protocol", "Callback-based HTTP/1.1 parser for asyncio integration", -1, module_methods }; PyMODINIT_FUNC PyInit__protocol(void) { PyObject *m; if (PyType_Ready(&H1CProtocolType) < 0) return NULL; m = PyModule_Create(&protocol_module); if (m == NULL) return NULL; Py_INCREF(&H1CProtocolType); PyModule_AddObject(m, "H1CProtocol", (PyObject *)&H1CProtocolType); ParseError = PyErr_NewException( "gunicorn_h1c._protocol.ParseError", PyExc_ValueError, NULL); if (!ParseError) return NULL; Py_INCREF(ParseError); PyModule_AddObject(m, "ParseError", ParseError); /* Import validation exceptions from _parser module */ PyObject *parser_module = PyImport_ImportModule("gunicorn_h1c._parser"); if (!parser_module) return NULL; LimitRequestLine = PyObject_GetAttrString(parser_module, "LimitRequestLine"); if (!LimitRequestLine) { Py_DECREF(parser_module); return NULL; } LimitRequestHeaders = PyObject_GetAttrString(parser_module, "LimitRequestHeaders"); if (!LimitRequestHeaders) { Py_DECREF(parser_module); return NULL; } InvalidRequestMethod = PyObject_GetAttrString(parser_module, "InvalidRequestMethod"); if (!InvalidRequestMethod) { Py_DECREF(parser_module); return NULL; } InvalidHTTPVersion = PyObject_GetAttrString(parser_module, "InvalidHTTPVersion"); if (!InvalidHTTPVersion) { Py_DECREF(parser_module); return NULL; } InvalidHeaderName = PyObject_GetAttrString(parser_module, "InvalidHeaderName"); if (!InvalidHeaderName) { Py_DECREF(parser_module); return NULL; } InvalidHeader = PyObject_GetAttrString(parser_module, "InvalidHeader"); if (!InvalidHeader) { Py_DECREF(parser_module); return NULL; } InvalidChunkExtension = PyObject_GetAttrString(parser_module, "InvalidChunkExtension"); if (!InvalidChunkExtension) { Py_DECREF(parser_module); return NULL; } Py_DECREF(parser_module); return m; } benoitc-gunicorn_h1c-52625fd/src/gunicorn_h1c/pico_utils.h000066400000000000000000001074101524260035600235500ustar00rootroot00000000000000/* * SPDX-License-Identifier: MIT * * Copyright 2026 Benoit Chesneau * Licensed under the MIT License (see LICENSE file). * * Shared utilities for gunicorn_h1c parsers. * Contains common header parsing functions to reduce code duplication. */ #ifndef PICO_UTILS_H #define PICO_UTILS_H #include #include "picohttpparser.h" /* * RFC 9110 section 5.3: fields whose grammar admits only a single member. * A repeat cannot be merged into a comma-separated list, so the message is * ambiguous and gunicorn, a cache, or an upstream proxy may each resolve it * differently. Duplicate Host in particular is a routing- and * cache-confusion vector, so all of these are rejected outright. * * Transfer-Encoding is deliberately NOT listed: repeats there are legal and * are handled by the framing checks below. * * This is the only place the singleton set is stated. `name` must be * lowercase (matching pico_header_name_eq); `display` is the canonical * spelling used in the error message. */ static const struct { const char *name; size_t name_len; const char *display; } pico_singleton_fields[] = { {"host", 4, "Host"}, {"content-length", 14, "Content-Length"}, {"content-type", 12, "Content-Type"}, }; #define PICO_SINGLETON_FIELD_COUNT \ (sizeof(pico_singleton_fields) / sizeof(pico_singleton_fields[0])) /* Shared header extraction info */ typedef struct { Py_ssize_t content_length; /* -1 if not set */ int has_chunked; /* 1 if Transfer-Encoding: chunked */ int connection_close; /* -1=unset, 0=keep-alive, 1=close */ int should_upgrade; /* 1 if Upgrade header present */ int has_content_length; /* 1 if Content-Length header present */ int chunked_count; /* Number of "chunked" values seen */ int has_te_after_chunked; /* 1 if T-E value after chunked */ int has_unknown_te; /* 1 if unknown Transfer-Encoding value */ unsigned int singleton_seen; /* Bitmask of pico_singleton_fields seen */ int duplicate_singleton; /* Index of first repeated singleton, -1 if none */ } PicoHeaderInfo; /* * Case-insensitive header name comparison. * Target must be lowercase. */ static inline int pico_header_name_eq(const char *name, size_t name_len, const char *target, size_t target_len) { if (name_len != target_len) return 0; for (size_t i = 0; i < name_len; i++) { char c = name[i]; if (c >= 'A' && c <= 'Z') c += 32; if (c != target[i]) return 0; } return 1; } /* * Parse Content-Length value from header value string. * * RFC 9112 section 6.2: Content-Length = 1*DIGIT. The list form * ("5, 5") is also a classic smuggling vector even when values match, * so we reject anything that is not purely digits. * * Returns parsed value or -1 on error. */ static inline Py_ssize_t pico_parse_content_length(const char *value, size_t value_len) { if (value_len == 0) return -1; Py_ssize_t cl = 0; for (size_t i = 0; i < value_len; i++) { char c = value[i]; if (c >= '0' && c <= '9') { cl = cl * 10 + (c - '0'); } else { return -1; } } return cl; } /* * Check if value contains "chunked" (case-insensitive). */ static inline int pico_contains_chunked(const char *value, size_t value_len) { for (size_t j = 0; j + 7 <= value_len; j++) { char c0 = value[j]; if (c0 >= 'A' && c0 <= 'Z') c0 += 32; char c1 = value[j+1]; if (c1 >= 'A' && c1 <= 'Z') c1 += 32; char c2 = value[j+2]; if (c2 >= 'A' && c2 <= 'Z') c2 += 32; char c3 = value[j+3]; if (c3 >= 'A' && c3 <= 'Z') c3 += 32; char c4 = value[j+4]; if (c4 >= 'A' && c4 <= 'Z') c4 += 32; char c5 = value[j+5]; if (c5 >= 'A' && c5 <= 'Z') c5 += 32; char c6 = value[j+6]; if (c6 >= 'A' && c6 <= 'Z') c6 += 32; if (c0 == 'c' && c1 == 'h' && c2 == 'u' && c3 == 'n' && c4 == 'k' && c5 == 'e' && c6 == 'd') { return 1; } } return 0; } /* * Check if value contains "close" (case-insensitive). */ static inline int pico_contains_close(const char *value, size_t value_len) { for (size_t j = 0; j + 5 <= value_len; j++) { char c0 = value[j]; if (c0 >= 'A' && c0 <= 'Z') c0 += 32; char c1 = value[j+1]; if (c1 >= 'A' && c1 <= 'Z') c1 += 32; char c2 = value[j+2]; if (c2 >= 'A' && c2 <= 'Z') c2 += 32; char c3 = value[j+3]; if (c3 >= 'A' && c3 <= 'Z') c3 += 32; char c4 = value[j+4]; if (c4 >= 'A' && c4 <= 'Z') c4 += 32; if (c0 == 'c' && c1 == 'l' && c2 == 'o' && c3 == 's' && c4 == 'e') { return 1; } } return 0; } /* * Check if value contains "keep-alive" (case-insensitive). */ static inline int pico_contains_keepalive(const char *value, size_t value_len) { for (size_t j = 0; j + 10 <= value_len; j++) { char c0 = value[j]; if (c0 >= 'A' && c0 <= 'Z') c0 += 32; char c1 = value[j+1]; if (c1 >= 'A' && c1 <= 'Z') c1 += 32; char c2 = value[j+2]; if (c2 >= 'A' && c2 <= 'Z') c2 += 32; char c3 = value[j+3]; if (c3 >= 'A' && c3 <= 'Z') c3 += 32; char c4 = value[j+4]; /* '-' */ char c5 = value[j+5]; if (c5 >= 'A' && c5 <= 'Z') c5 += 32; char c6 = value[j+6]; if (c6 >= 'A' && c6 <= 'Z') c6 += 32; char c7 = value[j+7]; if (c7 >= 'A' && c7 <= 'Z') c7 += 32; char c8 = value[j+8]; if (c8 >= 'A' && c8 <= 'Z') c8 += 32; char c9 = value[j+9]; if (c9 >= 'A' && c9 <= 'Z') c9 += 32; if (c0 == 'k' && c1 == 'e' && c2 == 'e' && c3 == 'p' && c4 == '-' && c5 == 'a' && c6 == 'l' && c7 == 'i' && c8 == 'v' && c9 == 'e') { return 1; } } return 0; } /* * Check if a Transfer-Encoding token matches a known value. * Returns: 1=chunked, 2=identity, 3=gzip/deflate/compress, 0=unknown */ static inline int pico_check_te_token(const char *token, size_t token_len) { /* Skip leading/trailing whitespace */ while (token_len > 0 && (*token == ' ' || *token == '\t')) { token++; token_len--; } while (token_len > 0 && (token[token_len-1] == ' ' || token[token_len-1] == '\t')) { token_len--; } if (token_len == 0) return 0; /* Check for known values (case-insensitive) */ if (token_len == 7) { /* "chunked" */ char buf[7]; for (size_t i = 0; i < 7; i++) { buf[i] = token[i]; if (buf[i] >= 'A' && buf[i] <= 'Z') buf[i] += 32; } if (buf[0] == 'c' && buf[1] == 'h' && buf[2] == 'u' && buf[3] == 'n' && buf[4] == 'k' && buf[5] == 'e' && buf[6] == 'd') { return 1; /* chunked */ } } else if (token_len == 8) { /* "identity" */ char buf[8]; for (size_t i = 0; i < 8; i++) { buf[i] = token[i]; if (buf[i] >= 'A' && buf[i] <= 'Z') buf[i] += 32; } if (buf[0] == 'i' && buf[1] == 'd' && buf[2] == 'e' && buf[3] == 'n' && buf[4] == 't' && buf[5] == 'i' && buf[6] == 't' && buf[7] == 'y') { return 2; /* identity */ } /* "compress" */ if (buf[0] == 'c' && buf[1] == 'o' && buf[2] == 'm' && buf[3] == 'p' && buf[4] == 'r' && buf[5] == 'e' && buf[6] == 's' && buf[7] == 's') { return 3; /* compress */ } } else if (token_len == 4) { /* "gzip" */ char buf[4]; for (size_t i = 0; i < 4; i++) { buf[i] = token[i]; if (buf[i] >= 'A' && buf[i] <= 'Z') buf[i] += 32; } if (buf[0] == 'g' && buf[1] == 'z' && buf[2] == 'i' && buf[3] == 'p') { return 3; /* gzip */ } } else if (token_len == 7) { /* "deflate" */ char buf[7]; for (size_t i = 0; i < 7; i++) { buf[i] = token[i]; if (buf[i] >= 'A' && buf[i] <= 'Z') buf[i] += 32; } if (buf[0] == 'd' && buf[1] == 'e' && buf[2] == 'f' && buf[3] == 'l' && buf[4] == 'a' && buf[5] == 't' && buf[6] == 'e') { return 3; /* deflate */ } } return 0; /* unknown */ } /* * Parse and validate Transfer-Encoding header value. * Updates info with validation flags. */ static inline void pico_parse_transfer_encoding(const char *value, size_t value_len, PicoHeaderInfo *info) { const char *p = value; const char *end = value + value_len; while (p < end) { /* Find end of token (comma or end) */ const char *token_start = p; const char *token_end = p; while (token_end < end && *token_end != ',') { token_end++; } size_t token_len = token_end - token_start; int te_type = pico_check_te_token(token_start, token_len); if (te_type == 1) { /* chunked */ if (info->has_chunked) { info->chunked_count++; /* Stacked chunked */ } info->has_chunked = 1; } else if (te_type == 2 || te_type == 3) { /* identity, gzip, deflate, compress */ if (info->has_chunked) { info->has_te_after_chunked = 1; /* T-E value after chunked */ } } else if (te_type == 0 && token_len > 0) { /* Unknown transfer encoding */ /* Skip empty tokens */ const char *t = token_start; size_t tlen = token_len; while (tlen > 0 && (*t == ' ' || *t == '\t')) { t++; tlen--; } while (tlen > 0 && (t[tlen-1] == ' ' || t[tlen-1] == '\t')) { tlen--; } if (tlen > 0) { info->has_unknown_te = 1; } } /* Move past comma */ p = token_end; if (p < end && *p == ',') p++; } } /* * Extract special headers into PicoHeaderInfo. * Processes Content-Length, Transfer-Encoding, Connection, and Upgrade headers. * Also records repeats of the singleton fields and performs validation for * request smuggling prevention. */ static inline void pico_extract_headers(struct phr_header *headers, size_t num_headers, PicoHeaderInfo *info) { info->content_length = -1; info->has_chunked = 0; info->connection_close = -1; info->should_upgrade = 0; info->has_content_length = 0; info->chunked_count = 0; info->has_te_after_chunked = 0; info->has_unknown_te = 0; info->singleton_seen = 0; info->duplicate_singleton = -1; for (size_t i = 0; i < num_headers; i++) { const char *name = headers[i].name; size_t name_len = headers[i].name_len; const char *value = headers[i].value; size_t value_len = headers[i].value_len; /* Singleton field repeat detection (case-insensitive) */ for (size_t s = 0; s < PICO_SINGLETON_FIELD_COUNT; s++) { if (!pico_header_name_eq(name, name_len, pico_singleton_fields[s].name, pico_singleton_fields[s].name_len)) { continue; } unsigned int bit = 1u << s; if (info->singleton_seen & bit) { if (info->duplicate_singleton < 0) { info->duplicate_singleton = (int)s; } } else { info->singleton_seen |= bit; } break; } if (pico_header_name_eq(name, name_len, "content-length", 14)) { /* Keep the first value; a repeat is rejected by the framing check */ if (!info->has_content_length) { info->content_length = pico_parse_content_length(value, value_len); info->has_content_length = 1; } } else if (pico_header_name_eq(name, name_len, "transfer-encoding", 17)) { /* Parse and validate Transfer-Encoding value */ pico_parse_transfer_encoding(value, value_len, info); } else if (pico_header_name_eq(name, name_len, "connection", 10)) { if (pico_contains_close(value, value_len)) { info->connection_close = 1; } else if (pico_contains_keepalive(value, value_len)) { info->connection_close = 0; } } else if (pico_header_name_eq(name, name_len, "upgrade", 7)) { info->should_upgrade = 1; } } } /* * Validate headers for request smuggling prevention. * Returns 0 on success, -1 on error (sets Python exception). * * Checks: * - Repeated singleton fields (Host, Content-Length, Content-Type) * - Content-Length with Transfer-Encoding (CL+TE conflict) * - Chunked encoding in HTTP/1.0 * - Stacked chunked encoding * - Unknown Transfer-Encoding values * - Transfer-Encoding values after chunked */ static inline int pico_validate_header_framing(PicoHeaderInfo *info, int minor_version, PyObject *InvalidHeader) { /* Check for a repeated singleton field (RFC 9110 section 5.3) */ if (info->duplicate_singleton >= 0) { PyErr_Format(InvalidHeader, "Duplicate %s header", pico_singleton_fields[info->duplicate_singleton].display); return -1; } /* Check for negative Content-Length (invalid) */ if (info->has_content_length && info->content_length < 0) { PyErr_SetString(InvalidHeader, "Invalid Content-Length value"); return -1; } /* Check for unknown Transfer-Encoding */ if (info->has_unknown_te) { PyErr_SetString(InvalidHeader, "Unsupported Transfer-Encoding"); return -1; } /* Check for stacked chunked encoding */ if (info->chunked_count > 0) { PyErr_SetString(InvalidHeader, "Stacked chunked encoding"); return -1; } /* Check for Transfer-Encoding value after chunked */ if (info->has_te_after_chunked) { PyErr_SetString(InvalidHeader, "Invalid Transfer-Encoding after chunked"); return -1; } /* If chunked encoding is present */ if (info->has_chunked) { /* Reject chunked in HTTP/1.0 (RFC 9112 Section 6.1) */ if (minor_version == 0) { PyErr_SetString(InvalidHeader, "Chunked encoding not allowed in HTTP/1.0"); return -1; } /* Reject Content-Length with Transfer-Encoding (request smuggling vector) */ if (info->has_content_length) { PyErr_SetString(InvalidHeader, "Content-Length with Transfer-Encoding"); return -1; } } return 0; } /* * Create (name, value) bytes tuple for header. * Returns new reference or NULL on error. */ static inline PyObject * pico_create_header_tuple(const char *name, size_t name_len, const char *value, size_t value_len) { PyObject *py_name = PyBytes_FromStringAndSize(name, name_len); PyObject *py_value = PyBytes_FromStringAndSize(value, value_len); if (!py_name || !py_value) { Py_XDECREF(py_name); Py_XDECREF(py_value); return NULL; } PyObject *tuple = PyTuple_Pack(2, py_name, py_value); Py_DECREF(py_name); Py_DECREF(py_value); return tuple; } /* * Create (name, value) bytes tuple for header with lowercase name. * Used for ASGI compliance where header names should be lowercase. * Returns new reference or NULL on error. */ static inline PyObject * pico_create_header_tuple_lowercase(const char *name, size_t name_len, const char *value, size_t value_len) { char *lower_name = (char *)PyMem_Malloc(name_len); if (!lower_name) { PyErr_NoMemory(); return NULL; } for (size_t i = 0; i < name_len; i++) { char c = name[i]; if (c >= 'A' && c <= 'Z') { c += 32; } lower_name[i] = c; } PyObject *py_name = PyBytes_FromStringAndSize(lower_name, name_len); PyMem_Free(lower_name); PyObject *py_value = PyBytes_FromStringAndSize(value, value_len); if (!py_name || !py_value) { Py_XDECREF(py_name); Py_XDECREF(py_value); return NULL; } PyObject *tuple = PyTuple_Pack(2, py_name, py_value); Py_DECREF(py_name); Py_DECREF(py_value); return tuple; } /* * Find header value by name (case-insensitive). * Returns new PyBytes reference or Py_None (new ref) if not found. * Returns NULL on error. */ static inline PyObject * pico_find_header(struct phr_header *headers, size_t num_headers, const char *name, size_t name_len) { for (size_t i = 0; i < num_headers; i++) { if (headers[i].name_len == name_len) { int match = 1; for (size_t j = 0; j < name_len; j++) { char c1 = headers[i].name[j]; char c2 = name[j]; if (c1 >= 'A' && c1 <= 'Z') c1 += 32; if (c2 >= 'A' && c2 <= 'Z') c2 += 32; if (c1 != c2) { match = 0; break; } } if (match) { return PyBytes_FromStringAndSize( headers[i].value, headers[i].value_len); } } } Py_RETURN_NONE; } /* * Validation configuration for limit enforcement. */ typedef struct { Py_ssize_t limit_request_line; Py_ssize_t limit_request_fields; Py_ssize_t limit_request_field_size; int permit_unconventional_http_method; int permit_unconventional_http_version; } PicoValidationConfig; /* Default validation config values matching gunicorn */ #define PICO_DEFAULT_LIMIT_REQUEST_LINE 8190 #define PICO_DEFAULT_LIMIT_REQUEST_FIELDS 100 #define PICO_DEFAULT_LIMIT_REQUEST_FIELD_SIZE 8190 /* * RFC 9110 token character table. * token = 1*tchar * tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" * / "+" / "-" / "." / "0"-"9" / "A"-"Z" / "^" / "_" * / "`" / "a"-"z" / "|" / "~" */ static const unsigned char pico_token_chars[256] = { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0x00-0x0F */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0x10-0x1F */ 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, /* 0x20-0x2F: ! # $ % & ' * + - . */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, /* 0x30-0x3F: 0-9 */ 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, /* 0x40-0x4F: A-O */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, /* 0x50-0x5F: P-Z ^ _ */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, /* 0x60-0x6F: ` a-o */ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 0, /* 0x70-0x7F: p-z | ~ */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0x80-0x8F */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0x90-0x9F */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0xA0-0xAF */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0xB0-0xBF */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0xC0-0xCF */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0xD0-0xDF */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, /* 0xE0-0xEF */ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 /* 0xF0-0xFF */ }; /* * Check if character is valid RFC 9110 token character. */ static inline int pico_is_token_char(unsigned char c) { return pico_token_chars[c]; } /* * Validate HTTP method. * Returns 0 on success, -1 on error (sets Python exception). * * Rules: * - All characters must be RFC 9110 token chars * - Unless permit_unconventional: no lowercase, no '#', length 3-20 */ static inline int pico_validate_method(const char *method, size_t method_len, int permit_unconventional, PyObject *InvalidRequestMethod) { /* Check token characters */ for (size_t i = 0; i < method_len; i++) { unsigned char c = (unsigned char)method[i]; if (!pico_is_token_char(c)) { PyErr_Format(InvalidRequestMethod, "Invalid character '\\x%02x' in HTTP method", c); return -1; } } if (!permit_unconventional) { /* Check length (gunicorn requires 3-20) */ if (method_len < 3 || method_len > 20) { PyErr_Format(InvalidRequestMethod, "HTTP method length %zu out of range (3-20)", method_len); return -1; } /* Check for lowercase and '#' */ for (size_t i = 0; i < method_len; i++) { unsigned char c = (unsigned char)method[i]; if (c >= 'a' && c <= 'z') { PyErr_SetString(InvalidRequestMethod, "Lowercase letters not allowed in HTTP method"); return -1; } if (c == '#') { PyErr_SetString(InvalidRequestMethod, "'#' not allowed in HTTP method"); return -1; } } } return 0; } /* * Case-insensitive equality check against a lowercase ASCII literal. */ static inline int pico_name_ieq(const char *name, size_t name_len, const char *lower, size_t lower_len) { if (name_len != lower_len) return 0; for (size_t i = 0; i < name_len; i++) { char c = name[i]; if (c >= 'A' && c <= 'Z') c += 32; if (c != lower[i]) return 0; } return 1; } /* * RFC 9110 section 6.5.1: reject header fields that must not appear in a * trailer section because they alter routing, framing, or authentication. * Returns 1 if name is forbidden, 0 otherwise. */ static inline int pico_is_forbidden_trailer_name(const char *name, size_t name_len) { return pico_name_ieq(name, name_len, "host", 4) || pico_name_ieq(name, name_len, "content-length", 14) || pico_name_ieq(name, name_len, "transfer-encoding", 17) || pico_name_ieq(name, name_len, "trailer", 7) || pico_name_ieq(name, name_len, "authorization", 13) || pico_name_ieq(name, name_len, "te", 2); } /* * Validate request-target form against method (RFC 9112 sections 3.2.3 and 3.2.4). * Returns 0 on success, -1 on error (sets Python exception). * * Enforces: * - asterisk-form ("*") requires OPTIONS. * - authority-form (host:port with no scheme, no leading "/") requires CONNECT. * - relative-reference is rejected (falls out of authority-form check). */ static inline int pico_validate_request_target(const char *method, size_t method_len, const char *path, size_t path_len, PyObject *ParseError) { if (path_len == 0) { PyErr_SetString(ParseError, "Empty request-target"); return -1; } int is_connect = (method_len == 7 && method[0] == 'C' && method[1] == 'O' && method[2] == 'N' && method[3] == 'N' && method[4] == 'E' && method[5] == 'C' && method[6] == 'T'); int is_options = (method_len == 7 && method[0] == 'O' && method[1] == 'P' && method[2] == 'T' && method[3] == 'I' && method[4] == 'O' && method[5] == 'N' && method[6] == 'S'); /* asterisk-form */ if (path_len == 1 && path[0] == '*') { if (!is_options) { PyErr_SetString(ParseError, "asterisk-form request-target requires OPTIONS method"); return -1; } return 0; } /* origin-form */ if (path[0] == '/') { return 0; } /* absolute-form: contains "://" */ for (size_t i = 0; i + 2 < path_len; i++) { if (path[i] == ':' && path[i + 1] == '/' && path[i + 2] == '/') { return 0; } } /* Remaining shape is authority-form or a relative reference; both * are only legal in a CONNECT request. */ if (!is_connect) { PyErr_SetString(ParseError, "authority-form or relative request-target requires CONNECT method"); return -1; } return 0; } /* * Validate HTTP version. * Returns 0 on success, -1 on error (sets Python exception). * * Unless permit_unconventional, only HTTP/1.0 and HTTP/1.1 allowed. */ static inline int pico_validate_version(int minor_version, int permit_unconventional, PyObject *InvalidHTTPVersion) { if (!permit_unconventional) { if (minor_version != 0 && minor_version != 1) { PyErr_Format(InvalidHTTPVersion, "Invalid HTTP version: HTTP/1.%d (only 1.0 and 1.1 allowed)", minor_version); return -1; } } return 0; } /* * Validate header name. * Returns 0 on success, -1 on error (sets Python exception). * * All characters must be RFC 9110 token chars. */ static inline int pico_validate_header_name(const char *name, size_t name_len, PyObject *InvalidHeaderName) { for (size_t i = 0; i < name_len; i++) { unsigned char c = (unsigned char)name[i]; if (!pico_is_token_char(c)) { PyErr_Format(InvalidHeaderName, "Invalid character '\\x%02x' in header name", c); return -1; } } return 0; } /* * Validate header value. * Returns 0 on success, -1 on error (sets Python exception). * * RFC 9110 section 5.5: field-vchar = VCHAR / obs-text; plus SP / HTAB. * Any other control byte (0x00-0x08, 0x0A-0x1F, 0x7F) is invalid. */ static inline int pico_validate_header_value(const char *value, size_t value_len, PyObject *InvalidHeader) { for (size_t i = 0; i < value_len; i++) { unsigned char c = (unsigned char)value[i]; if (c <= 0x08 || (c >= 0x0A && c <= 0x1F) || c == 0x7F) { PyErr_Format(InvalidHeader, "Invalid control character '\\x%02x' in header value", c); return -1; } } return 0; } /* * Calculate request line length by finding first CRLF. * Returns length or -1 if not found. */ static inline Py_ssize_t pico_find_request_line_length(const char *buf, size_t buf_len) { for (size_t i = 0; i + 1 < buf_len; i++) { if (buf[i] == '\r' && buf[i + 1] == '\n') { return (Py_ssize_t)i; } } return -1; } /* * Analyze buffer when phr_parse_request returns -1 and set specific exception. * Returns -1 (always sets an exception). * * Analysis order: * 1. Find method and validate characters * 2. Find HTTP version and validate * 3. Scan headers for invalid characters * 4. Fallback to generic ParseError */ static inline int pico_analyze_parse_error(const char *buf, size_t buf_len, int permit_unconventional_method, int permit_unconventional_version, PyObject *ParseError, PyObject *InvalidRequestMethod, PyObject *InvalidHTTPVersion, PyObject *InvalidHeaderName, PyObject *InvalidHeader) { /* 1. Find and validate method (before first space) */ size_t method_end = 0; for (size_t i = 0; i < buf_len; i++) { if (buf[i] == ' ') { method_end = i; break; } } if (method_end > 0) { /* Check method for invalid chars */ for (size_t i = 0; i < method_end; i++) { unsigned char c = (unsigned char)buf[i]; if (!pico_is_token_char(c)) { PyErr_Format(InvalidRequestMethod, "Invalid character '\\x%02x' in HTTP method", c); return -1; } if (!permit_unconventional_method && c >= 'a' && c <= 'z') { PyErr_SetString(InvalidRequestMethod, "Lowercase letters not allowed in HTTP method"); return -1; } } } /* 2. Find and validate HTTP version (after path, look for "HTTP/") */ const char *http_marker = NULL; for (size_t i = 0; i + 5 <= buf_len; i++) { if (buf[i] == 'H' && buf[i+1] == 'T' && buf[i+2] == 'T' && buf[i+3] == 'P' && buf[i+4] == '/') { http_marker = &buf[i]; break; } } if (http_marker && !permit_unconventional_version) { /* Check if it's HTTP/1.0 or HTTP/1.1 */ size_t remaining = buf_len - (http_marker - buf); if (remaining >= 8) { /* HTTP/X.Y followed by \r or end */ char major = http_marker[5]; char dot = http_marker[6]; char minor = http_marker[7]; if (major != '1' || dot != '.' || (minor != '0' && minor != '1')) { PyErr_SetString(InvalidHTTPVersion, "Invalid HTTP version (only HTTP/1.0 and HTTP/1.1 allowed)"); return -1; } } } /* Check for non-HTTP protocol markers (e.g., FTP/1.1) */ if (!http_marker && !permit_unconventional_version) { /* Find second space (after path) to locate version field */ size_t second_space = 0; int space_count = 0; for (size_t i = 0; i < buf_len && i < 8192; i++) { if (buf[i] == ' ') { space_count++; if (space_count == 2) { second_space = i; break; } } if (buf[i] == '\r' || buf[i] == '\n') break; } if (second_space > 0 && second_space + 1 < buf_len) { /* Check if there's something after second space that looks like a version */ const char *ver_start = &buf[second_space + 1]; size_t ver_remaining = buf_len - second_space - 1; /* Look for X/X.X pattern but not HTTP/ */ if (ver_remaining >= 3) { /* If it starts with letters and has a '/', it's a protocol version */ int has_slash = 0; for (size_t i = 0; i < ver_remaining && i < 10; i++) { if (ver_start[i] == '/') { has_slash = 1; break; } if (ver_start[i] == '\r' || ver_start[i] == '\n') break; } if (has_slash) { PyErr_SetString(InvalidHTTPVersion, "Invalid HTTP version (only HTTP/1.0 and HTTP/1.1 allowed)"); return -1; } } } } /* 3. Find and scan headers (after first \r\n) */ const char *headers_start = NULL; for (size_t i = 0; i + 1 < buf_len; i++) { if (buf[i] == '\r' && buf[i+1] == '\n') { headers_start = &buf[i + 2]; break; } } if (headers_start) { size_t headers_len = buf_len - (headers_start - buf); const char *p = headers_start; const char *end = headers_start + headers_len; while (p < end) { /* Find colon (end of header name) */ const char *colon = NULL; const char *line_end = NULL; for (const char *q = p; q < end; q++) { if (*q == ':' && !colon) colon = q; if (q + 1 < end && *q == '\r' && *(q+1) == '\n') { line_end = q; break; } } if (!line_end) break; /* Incomplete */ /* Empty line = end of headers */ if (p == line_end) break; /* Validate header name */ if (colon && colon > p) { for (const char *q = p; q < colon; q++) { unsigned char c = (unsigned char)*q; if (!pico_is_token_char(c)) { PyErr_Format(InvalidHeaderName, "Invalid character '\\x%02x' in header name", c); return -1; } } } /* Validate header value: RFC 9110 section 5.5 field-vchar. */ if (colon && colon + 1 < line_end) { const char *value_start = colon + 1; while (value_start < line_end && (*value_start == ' ' || *value_start == '\t')) { value_start++; } for (const char *q = value_start; q < line_end; q++) { unsigned char c = (unsigned char)*q; if (c <= 0x08 || (c >= 0x0A && c <= 0x1F) || c == 0x7F) { PyErr_Format(InvalidHeader, "Invalid control character '\\x%02x' in header value", c); return -1; } } } p = line_end + 2; /* Skip \r\n */ } } /* Fallback to generic error */ PyErr_SetString(ParseError, "Invalid HTTP request"); return -1; } /* * Validate all headers against limits, character rules, and framing rules. * Returns 0 on success, -1 on error (sets Python exception). */ static inline int pico_validate_headers_full(struct phr_header *headers, size_t num_headers, int minor_version, const PicoValidationConfig *config, PyObject *LimitRequestHeaders, PyObject *InvalidHeaderName, PyObject *InvalidHeader) { /* Check header count */ if ((Py_ssize_t)num_headers > config->limit_request_fields) { PyErr_Format(LimitRequestHeaders, "Number of headers (%zu) exceeds limit (%zd)", num_headers, config->limit_request_fields); return -1; } for (size_t i = 0; i < num_headers; i++) { /* Check header line size (name + ": " + value + CRLF) */ size_t header_size = headers[i].name_len + 2 + headers[i].value_len + 2; if ((Py_ssize_t)header_size > config->limit_request_field_size) { PyErr_Format(LimitRequestHeaders, "Header size (%zu) exceeds limit (%zd)", header_size, config->limit_request_field_size); return -1; } /* Validate header name */ if (pico_validate_header_name(headers[i].name, headers[i].name_len, InvalidHeaderName) < 0) { return -1; } /* Validate header value */ if (pico_validate_header_value(headers[i].value, headers[i].value_len, InvalidHeader) < 0) { return -1; } } /* Extract header info and validate framing for request smuggling prevention */ PicoHeaderInfo info; pico_extract_headers(headers, num_headers, &info); if (pico_validate_header_framing(&info, minor_version, InvalidHeader) < 0) { return -1; } return 0; } /* * Validate all headers against limits and character rules (legacy, no framing check). * Returns 0 on success, -1 on error (sets Python exception). */ static inline int pico_validate_headers(struct phr_header *headers, size_t num_headers, const PicoValidationConfig *config, PyObject *LimitRequestHeaders, PyObject *InvalidHeaderName, PyObject *InvalidHeader) { /* Check header count */ if ((Py_ssize_t)num_headers > config->limit_request_fields) { PyErr_Format(LimitRequestHeaders, "Number of headers (%zu) exceeds limit (%zd)", num_headers, config->limit_request_fields); return -1; } for (size_t i = 0; i < num_headers; i++) { /* Check header line size (name + ": " + value + CRLF) */ size_t header_size = headers[i].name_len + 2 + headers[i].value_len + 2; if ((Py_ssize_t)header_size > config->limit_request_field_size) { PyErr_Format(LimitRequestHeaders, "Header size (%zu) exceeds limit (%zd)", header_size, config->limit_request_field_size); return -1; } /* Validate header name */ if (pico_validate_header_name(headers[i].name, headers[i].name_len, InvalidHeaderName) < 0) { return -1; } /* Validate header value */ if (pico_validate_header_value(headers[i].value, headers[i].value_len, InvalidHeader) < 0) { return -1; } } return 0; } #endif /* PICO_UTILS_H */ benoitc-gunicorn_h1c-52625fd/src/gunicorn_h1c/picohttpparser/000077500000000000000000000000001524260035600242715ustar00rootroot00000000000000benoitc-gunicorn_h1c-52625fd/src/gunicorn_h1c/picohttpparser/picohttpparser.c000066400000000000000000000667211524260035600275200ustar00rootroot00000000000000/* * Copyright (c) 2009-2014 Kazuho Oku, Tokuhiro Matsuno, Daisuke Murase, * Shigeo Mitsunari * * The software is licensed under either the MIT License (below) or the Perl * license. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to * deal in the Software without restriction, including without limitation the * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or * sell copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS * IN THE SOFTWARE. */ #include #include #include #ifdef __SSE4_2__ #ifdef _MSC_VER #include #else #include #endif #endif #include "picohttpparser.h" #if __GNUC__ >= 3 #define likely(x) __builtin_expect(!!(x), 1) #define unlikely(x) __builtin_expect(!!(x), 0) #else #define likely(x) (x) #define unlikely(x) (x) #endif #ifdef _MSC_VER #define ALIGNED(n) _declspec(align(n)) #else #define ALIGNED(n) __attribute__((aligned(n))) #endif #define IS_PRINTABLE_ASCII(c) ((unsigned char)(c)-040u < 0137u) #define CHECK_EOF() \ if (buf == buf_end) { \ *ret = -2; \ return NULL; \ } #define EXPECT_CHAR_NO_CHECK(ch) \ if (*buf++ != ch) { \ *ret = -1; \ return NULL; \ } #define EXPECT_CHAR(ch) \ CHECK_EOF(); \ EXPECT_CHAR_NO_CHECK(ch); #define ADVANCE_TOKEN(tok, toklen) \ do { \ const char *tok_start = buf; \ static const char ALIGNED(16) ranges2[16] = "\000\040\177\177"; \ int found2; \ buf = findchar_fast(buf, buf_end, ranges2, 4, &found2); \ if (!found2) { \ CHECK_EOF(); \ } \ while (1) { \ if (*buf == ' ') { \ break; \ } else if (unlikely(!IS_PRINTABLE_ASCII(*buf))) { \ if ((unsigned char)*buf < '\040' || *buf == '\177') { \ *ret = -1; \ return NULL; \ } \ } \ ++buf; \ CHECK_EOF(); \ } \ tok = tok_start; \ toklen = buf - tok_start; \ } while (0) static const char *token_char_map = "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" "\0\1\0\1\1\1\1\1\0\0\1\1\0\1\1\0\1\1\1\1\1\1\1\1\1\1\0\0\0\0\0\0" "\0\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\0\0\0\1\1" "\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\0\1\0\1\0" "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0" "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"; static const char *findchar_fast(const char *buf, const char *buf_end, const char *ranges, size_t ranges_size, int *found) { *found = 0; #if __SSE4_2__ if (likely(buf_end - buf >= 16)) { __m128i ranges16 = _mm_loadu_si128((const __m128i *)ranges); size_t left = (buf_end - buf) & ~15; do { __m128i b16 = _mm_loadu_si128((const __m128i *)buf); int r = _mm_cmpestri(ranges16, ranges_size, b16, 16, _SIDD_LEAST_SIGNIFICANT | _SIDD_CMP_RANGES | _SIDD_UBYTE_OPS); if (unlikely(r != 16)) { buf += r; *found = 1; break; } buf += 16; left -= 16; } while (likely(left != 0)); } #else /* suppress unused parameter warning */ (void)buf_end; (void)ranges; (void)ranges_size; #endif return buf; } static const char *get_token_to_eol(const char *buf, const char *buf_end, const char **token, size_t *token_len, int *ret) { const char *token_start = buf; #ifdef __SSE4_2__ static const char ALIGNED(16) ranges1[16] = "\0\010" /* allow HT */ "\012\037" /* allow SP and up to but not including DEL */ "\177\177"; /* allow chars w. MSB set */ int found; buf = findchar_fast(buf, buf_end, ranges1, 6, &found); if (found) goto FOUND_CTL; #else /* find non-printable char within the next 8 bytes, this is the hottest code; manually inlined */ while (likely(buf_end - buf >= 8)) { #define DOIT() \ do { \ if (unlikely(!IS_PRINTABLE_ASCII(*buf))) \ goto NonPrintable; \ ++buf; \ } while (0) DOIT(); DOIT(); DOIT(); DOIT(); DOIT(); DOIT(); DOIT(); DOIT(); #undef DOIT continue; NonPrintable: if ((likely((unsigned char)*buf < '\040') && likely(*buf != '\011')) || unlikely(*buf == '\177')) { goto FOUND_CTL; } ++buf; } #endif for (;; ++buf) { CHECK_EOF(); if (unlikely(!IS_PRINTABLE_ASCII(*buf))) { if ((likely((unsigned char)*buf < '\040') && likely(*buf != '\011')) || unlikely(*buf == '\177')) { goto FOUND_CTL; } } } FOUND_CTL: if (likely(*buf == '\015')) { ++buf; EXPECT_CHAR('\012'); *token_len = buf - 2 - token_start; } else if (*buf == '\012') { *token_len = buf - token_start; ++buf; } else { *ret = -1; return NULL; } *token = token_start; return buf; } static const char *is_complete(const char *buf, const char *buf_end, size_t last_len, int *ret) { int ret_cnt = 0; buf = last_len < 3 ? buf : buf + last_len - 3; while (1) { CHECK_EOF(); if (*buf == '\015') { ++buf; CHECK_EOF(); EXPECT_CHAR('\012'); ++ret_cnt; } else if (*buf == '\012') { ++buf; ++ret_cnt; } else { ++buf; ret_cnt = 0; } if (ret_cnt == 2) { return buf; } } *ret = -2; return NULL; } #define PARSE_INT(valp_, mul_) \ if (*buf < '0' || '9' < *buf) { \ buf++; \ *ret = -1; \ return NULL; \ } \ *(valp_) = (mul_) * (*buf++ - '0'); #define PARSE_INT_3(valp_) \ do { \ int res_ = 0; \ PARSE_INT(&res_, 100) \ *valp_ = res_; \ PARSE_INT(&res_, 10) \ *valp_ += res_; \ PARSE_INT(&res_, 1) \ *valp_ += res_; \ } while (0) /* returned pointer is always within [buf, buf_end), or null */ static const char *parse_token(const char *buf, const char *buf_end, const char **token, size_t *token_len, char next_char, int *ret) { /* We use pcmpestri to detect non-token characters. This instruction can take no more than eight character ranges (8*2*8=128 * bits that is the size of a SSE register). Due to this restriction, characters `|` and `~` are handled in the slow loop. */ static const char ALIGNED(16) ranges[] = "\x00 " /* control chars and up to SP */ "\"\"" /* 0x22 */ "()" /* 0x28,0x29 */ ",," /* 0x2c */ "//" /* 0x2f */ ":@" /* 0x3a-0x40 */ "[]" /* 0x5b-0x5d */ "{\xff"; /* 0x7b-0xff */ const char *buf_start = buf; int found; buf = findchar_fast(buf, buf_end, ranges, sizeof(ranges) - 1, &found); if (!found) { CHECK_EOF(); } while (1) { if (*buf == next_char) { break; } else if (!token_char_map[(unsigned char)*buf]) { *ret = -1; return NULL; } ++buf; CHECK_EOF(); } *token = buf_start; *token_len = buf - buf_start; return buf; } /* returned pointer is always within [buf, buf_end), or null */ static const char *parse_http_version(const char *buf, const char *buf_end, int *minor_version, int *ret) { /* we want at least [HTTP/1.] to try to parse */ if (buf_end - buf < 9) { *ret = -2; return NULL; } EXPECT_CHAR_NO_CHECK('H'); EXPECT_CHAR_NO_CHECK('T'); EXPECT_CHAR_NO_CHECK('T'); EXPECT_CHAR_NO_CHECK('P'); EXPECT_CHAR_NO_CHECK('/'); EXPECT_CHAR_NO_CHECK('1'); EXPECT_CHAR_NO_CHECK('.'); PARSE_INT(minor_version, 1); return buf; } static const char *parse_headers(const char *buf, const char *buf_end, struct phr_header *headers, size_t *num_headers, size_t max_headers, int *ret) { for (;; ++*num_headers) { CHECK_EOF(); if (*buf == '\015') { ++buf; EXPECT_CHAR('\012'); break; } else if (*buf == '\012') { ++buf; break; } if (*num_headers == max_headers) { *ret = -1; return NULL; } if (!(*num_headers != 0 && (*buf == ' ' || *buf == '\t'))) { /* parsing name, but do not discard SP before colon, see * http://www.mozilla.org/security/announce/2006/mfsa2006-33.html */ if ((buf = parse_token(buf, buf_end, &headers[*num_headers].name, &headers[*num_headers].name_len, ':', ret)) == NULL) { return NULL; } if (headers[*num_headers].name_len == 0) { *ret = -1; return NULL; } ++buf; for (;; ++buf) { CHECK_EOF(); if (!(*buf == ' ' || *buf == '\t')) { break; } } } else { headers[*num_headers].name = NULL; headers[*num_headers].name_len = 0; } const char *value; size_t value_len; if ((buf = get_token_to_eol(buf, buf_end, &value, &value_len, ret)) == NULL) { return NULL; } /* remove trailing SPs and HTABs */ const char *value_end = value + value_len; for (; value_end != value; --value_end) { const char c = *(value_end - 1); if (!(c == ' ' || c == '\t')) { break; } } headers[*num_headers].value = value; headers[*num_headers].value_len = value_end - value; } return buf; } static const char *parse_request(const char *buf, const char *buf_end, const char **method, size_t *method_len, const char **path, size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers, size_t max_headers, int *ret) { /* skip first empty line (some clients add CRLF after POST content) */ CHECK_EOF(); if (*buf == '\015') { ++buf; EXPECT_CHAR('\012'); } else if (*buf == '\012') { ++buf; } /* parse request line */ if ((buf = parse_token(buf, buf_end, method, method_len, ' ', ret)) == NULL) { return NULL; } do { ++buf; CHECK_EOF(); } while (*buf == ' '); ADVANCE_TOKEN(*path, *path_len); do { ++buf; CHECK_EOF(); } while (*buf == ' '); if (*method_len == 0 || *path_len == 0) { *ret = -1; return NULL; } if ((buf = parse_http_version(buf, buf_end, minor_version, ret)) == NULL) { return NULL; } if (*buf == '\015') { ++buf; EXPECT_CHAR('\012'); } else if (*buf == '\012') { ++buf; } else { *ret = -1; return NULL; } return parse_headers(buf, buf_end, headers, num_headers, max_headers, ret); } int phr_parse_request(const char *buf_start, size_t len, const char **method, size_t *method_len, const char **path, size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers, size_t last_len) { const char *buf = buf_start, *buf_end = buf_start + len; size_t max_headers = *num_headers; int r; *method = NULL; *method_len = 0; *path = NULL; *path_len = 0; *minor_version = -1; *num_headers = 0; /* if last_len != 0, check if the request is complete (a fast countermeasure againt slowloris */ if (last_len != 0 && is_complete(buf, buf_end, last_len, &r) == NULL) { return r; } if ((buf = parse_request(buf, buf_end, method, method_len, path, path_len, minor_version, headers, num_headers, max_headers, &r)) == NULL) { return r; } return (int)(buf - buf_start); } static const char *parse_response(const char *buf, const char *buf_end, int *minor_version, int *status, const char **msg, size_t *msg_len, struct phr_header *headers, size_t *num_headers, size_t max_headers, int *ret) { /* parse "HTTP/1.x" */ if ((buf = parse_http_version(buf, buf_end, minor_version, ret)) == NULL) { return NULL; } /* skip space */ if (*buf != ' ') { *ret = -1; return NULL; } do { ++buf; CHECK_EOF(); } while (*buf == ' '); /* parse status code, we want at least [:digit:][:digit:][:digit:] to try to parse */ if (buf_end - buf < 4) { *ret = -2; return NULL; } PARSE_INT_3(status); /* get message including preceding space */ if ((buf = get_token_to_eol(buf, buf_end, msg, msg_len, ret)) == NULL) { return NULL; } if (*msg_len == 0) { /* ok */ } else if (**msg == ' ') { /* Remove preceding space. Successful return from `get_token_to_eol` guarantees that we would hit something other than SP * before running past the end of the given buffer. */ do { ++*msg; --*msg_len; } while (**msg == ' '); } else { /* garbage found after status code */ *ret = -1; return NULL; } return parse_headers(buf, buf_end, headers, num_headers, max_headers, ret); } int phr_parse_response(const char *buf_start, size_t len, int *minor_version, int *status, const char **msg, size_t *msg_len, struct phr_header *headers, size_t *num_headers, size_t last_len) { const char *buf = buf_start, *buf_end = buf + len; size_t max_headers = *num_headers; int r; *minor_version = -1; *status = 0; *msg = NULL; *msg_len = 0; *num_headers = 0; /* if last_len != 0, check if the response is complete (a fast countermeasure against slowloris */ if (last_len != 0 && is_complete(buf, buf_end, last_len, &r) == NULL) { return r; } if ((buf = parse_response(buf, buf_end, minor_version, status, msg, msg_len, headers, num_headers, max_headers, &r)) == NULL) { return r; } return (int)(buf - buf_start); } int phr_parse_headers(const char *buf_start, size_t len, struct phr_header *headers, size_t *num_headers, size_t last_len) { const char *buf = buf_start, *buf_end = buf + len; size_t max_headers = *num_headers; int r; *num_headers = 0; /* if last_len != 0, check if the response is complete (a fast countermeasure against slowloris */ if (last_len != 0 && is_complete(buf, buf_end, last_len, &r) == NULL) { return r; } if ((buf = parse_headers(buf, buf_end, headers, num_headers, max_headers, &r)) == NULL) { return r; } return (int)(buf - buf_start); } enum { CHUNKED_IN_CHUNK_SIZE, CHUNKED_IN_CHUNK_EXT, CHUNKED_IN_CHUNK_HEADER_EXPECT_LF, CHUNKED_IN_CHUNK_DATA, CHUNKED_IN_CHUNK_DATA_EXPECT_CR, CHUNKED_IN_CHUNK_DATA_EXPECT_LF, CHUNKED_IN_TRAILERS_LINE_HEAD, CHUNKED_IN_TRAILERS_LINE_MIDDLE }; static int decode_hex(int ch) { if ('0' <= ch && ch <= '9') { return ch - '0'; } else if ('A' <= ch && ch <= 'F') { return ch - 'A' + 0xa; } else if ('a' <= ch && ch <= 'f') { return ch - 'a' + 0xa; } else { return -1; } } ssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *buf, size_t *_bufsz) { size_t dst = 0, src = 0, bufsz = *_bufsz; ssize_t ret = -2; /* incomplete */ decoder->_total_read += bufsz; while (1) { switch (decoder->_state) { case CHUNKED_IN_CHUNK_SIZE: for (;; ++src) { int v; if (src == bufsz) goto Exit; if ((v = decode_hex(buf[src])) == -1) { if (decoder->_hex_count == 0) { ret = -1; goto Exit; } /* the only characters that may appear after the chunk size are BWS, semicolon, or CRLF */ switch (buf[src]) { case ' ': case '\011': case ';': case '\012': case '\015': break; default: ret = -1; goto Exit; } break; } if (decoder->_hex_count == sizeof(size_t) * 2) { ret = -1; goto Exit; } decoder->bytes_left_in_chunk = decoder->bytes_left_in_chunk * 16 + v; ++decoder->_hex_count; } decoder->_hex_count = 0; decoder->_state = CHUNKED_IN_CHUNK_EXT; /* fallthru */ case CHUNKED_IN_CHUNK_EXT: /* RFC 7230 A.2 "Line folding in chunk extensions is disallowed" */ for (;; ++src) { if (src == bufsz) goto Exit; if (buf[src] == '\015') { break; } else if (buf[src] == '\012') { ret = -1; goto Exit; } } ++src; decoder->_state = CHUNKED_IN_CHUNK_HEADER_EXPECT_LF; /* fallthru */ case CHUNKED_IN_CHUNK_HEADER_EXPECT_LF: if (src == bufsz) goto Exit; if (buf[src] != '\012') { ret = -1; goto Exit; } ++src; if (decoder->bytes_left_in_chunk == 0) { if (decoder->consume_trailer) { decoder->_state = CHUNKED_IN_TRAILERS_LINE_HEAD; break; } else { goto Complete; } } decoder->_state = CHUNKED_IN_CHUNK_DATA; /* fallthru */ case CHUNKED_IN_CHUNK_DATA: { size_t avail = bufsz - src; if (avail < decoder->bytes_left_in_chunk) { if (dst != src) memmove(buf + dst, buf + src, avail); src += avail; dst += avail; decoder->bytes_left_in_chunk -= avail; goto Exit; } if (dst != src) memmove(buf + dst, buf + src, decoder->bytes_left_in_chunk); src += decoder->bytes_left_in_chunk; dst += decoder->bytes_left_in_chunk; decoder->bytes_left_in_chunk = 0; decoder->_state = CHUNKED_IN_CHUNK_DATA_EXPECT_CR; } /* fallthru */ case CHUNKED_IN_CHUNK_DATA_EXPECT_CR: if (src == bufsz) goto Exit; if (buf[src] != '\015') { ret = -1; goto Exit; } ++src; decoder->_state = CHUNKED_IN_CHUNK_DATA_EXPECT_LF; /* fallthru */ case CHUNKED_IN_CHUNK_DATA_EXPECT_LF: if (src == bufsz) goto Exit; if (buf[src] != '\012') { ret = -1; goto Exit; } ++src; decoder->_state = CHUNKED_IN_CHUNK_SIZE; break; case CHUNKED_IN_TRAILERS_LINE_HEAD: for (;; ++src) { if (src == bufsz) goto Exit; if (buf[src] != '\015') break; } if (buf[src++] == '\012') goto Complete; decoder->_state = CHUNKED_IN_TRAILERS_LINE_MIDDLE; /* fallthru */ case CHUNKED_IN_TRAILERS_LINE_MIDDLE: for (;; ++src) { if (src == bufsz) goto Exit; if (buf[src] == '\012') break; } ++src; decoder->_state = CHUNKED_IN_TRAILERS_LINE_HEAD; break; default: assert(!"decoder is corrupt"); } } Complete: ret = bufsz - src; Exit: if (dst != src) memmove(buf + dst, buf + src, bufsz - src); *_bufsz = dst; /* if incomplete but the overhead of the chunked encoding is >=100KB and >80%, signal an error */ if (ret == -2) { decoder->_total_overhead += bufsz - dst; if (decoder->_total_overhead >= 100 * 1024 && decoder->_total_read - decoder->_total_overhead < decoder->_total_read / 4) ret = -1; } return ret; } int phr_decode_chunked_is_in_data(struct phr_chunked_decoder *decoder) { return decoder->_state == CHUNKED_IN_CHUNK_DATA; } #undef CHECK_EOF #undef EXPECT_CHAR #undef ADVANCE_TOKEN benoitc-gunicorn_h1c-52625fd/src/gunicorn_h1c/picohttpparser/picohttpparser.h000066400000000000000000000067211524260035600275170ustar00rootroot00000000000000/* * Copyright (c) 2009-2014 Kazuho Oku, Tokuhiro Matsuno, Daisuke Murase, * Shigeo Mitsunari * * The software is licensed under either the MIT License (below) or the Perl * license. * * Permission is hereby granted, free of charge, to any person obtaining a copy * of this software and associated documentation files (the "Software"), to * deal in the Software without restriction, including without limitation the * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or * sell copies of the Software, and to permit persons to whom the Software is * furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS * IN THE SOFTWARE. */ #ifndef picohttpparser_h #define picohttpparser_h #include #include #ifdef _MSC_VER #define ssize_t intptr_t #endif #ifdef __cplusplus extern "C" { #endif /* contains name and value of a header (name == NULL if is a continuing line * of a multiline header */ struct phr_header { const char *name; size_t name_len; const char *value; size_t value_len; }; /* returns number of bytes consumed if successful, -2 if request is partial, * -1 if failed */ int phr_parse_request(const char *buf, size_t len, const char **method, size_t *method_len, const char **path, size_t *path_len, int *minor_version, struct phr_header *headers, size_t *num_headers, size_t last_len); /* ditto */ int phr_parse_response(const char *_buf, size_t len, int *minor_version, int *status, const char **msg, size_t *msg_len, struct phr_header *headers, size_t *num_headers, size_t last_len); /* ditto */ int phr_parse_headers(const char *buf, size_t len, struct phr_header *headers, size_t *num_headers, size_t last_len); /* should be zero-filled before start */ struct phr_chunked_decoder { size_t bytes_left_in_chunk; /* number of bytes left in current chunk */ char consume_trailer; /* if trailing headers should be consumed */ char _hex_count; char _state; uint64_t _total_read; uint64_t _total_overhead; }; /* the function rewrites the buffer given as (buf, bufsz) removing the chunked- * encoding headers. When the function returns without an error, bufsz is * updated to the length of the decoded data available. Applications should * repeatedly call the function while it returns -2 (incomplete) every time * supplying newly arrived data. If the end of the chunked-encoded data is * found, the function returns a non-negative number indicating the number of * octets left undecoded, that starts from the offset returned by `*bufsz`. * Returns -1 on error. */ ssize_t phr_decode_chunked(struct phr_chunked_decoder *decoder, char *buf, size_t *bufsz); /* returns if the chunked decoder is in middle of chunked data */ int phr_decode_chunked_is_in_data(struct phr_chunked_decoder *decoder); #ifdef __cplusplus } #endif #endif benoitc-gunicorn_h1c-52625fd/tests/000077500000000000000000000000001524260035600172165ustar00rootroot00000000000000benoitc-gunicorn_h1c-52625fd/tests/test_header_validation.py000066400000000000000000000240551524260035600242770ustar00rootroot00000000000000"""Tests for header validation against request smuggling attacks. These tests verify that the parsers properly reject requests with ambiguous or conflicting framing indicators. """ import pytest from gunicorn_h1c._protocol import ParseError as ProtocolParseError from gunicorn_h1c import ( H1CProtocol, InvalidHeader, parse_request, parse_request_fast, ) class TestDuplicateContentLength: """Duplicate Content-Length headers must be rejected.""" def test_basic_parser_duplicate_cl(self): """Basic parser rejects duplicate Content-Length.""" data = b"POST / HTTP/1.1\r\nContent-Length: 5\r\nContent-Length: 10\r\n\r\n" with pytest.raises(InvalidHeader, match="Duplicate Content-Length"): parse_request(data) def test_fast_parser_duplicate_cl(self): """Fast parser rejects duplicate Content-Length.""" data = b"POST / HTTP/1.1\r\nContent-Length: 5\r\nContent-Length: 10\r\n\r\n" with pytest.raises(InvalidHeader, match="Duplicate Content-Length"): parse_request_fast(data) def test_protocol_duplicate_cl(self): """H1CProtocol rejects duplicate Content-Length.""" data = b"POST / HTTP/1.1\r\nContent-Length: 5\r\nContent-Length: 10\r\n\r\n" p = H1CProtocol() with pytest.raises(InvalidHeader, match="Duplicate Content-Length"): p.feed(data) def test_duplicate_cl_same_value(self): """Even identical duplicate Content-Length values are rejected.""" data = b"POST / HTTP/1.1\r\nContent-Length: 5\r\nContent-Length: 5\r\n\r\n" with pytest.raises(InvalidHeader, match="Duplicate Content-Length"): parse_request(data) def test_duplicate_cl_same_value_fast(self): """Fast parser also rejects identical duplicate Content-Length.""" data = b"POST / HTTP/1.1\r\nContent-Length: 5\r\nContent-Length: 5\r\n\r\n" with pytest.raises(InvalidHeader, match="Duplicate Content-Length"): parse_request_fast(data) class TestContentLengthWithTransferEncoding: """CL+TE conflict must be rejected (request smuggling vector).""" def test_basic_parser_cl_te_conflict(self): """Basic parser rejects CL with Transfer-Encoding.""" data = b"POST / HTTP/1.1\r\nContent-Length: 5\r\nTransfer-Encoding: chunked\r\n\r\n" with pytest.raises( InvalidHeader, match="Content-Length with Transfer-Encoding" ): parse_request(data) def test_fast_parser_cl_te_conflict(self): """Fast parser rejects CL with Transfer-Encoding.""" data = b"POST / HTTP/1.1\r\nContent-Length: 5\r\nTransfer-Encoding: chunked\r\n\r\n" with pytest.raises( InvalidHeader, match="Content-Length with Transfer-Encoding" ): parse_request_fast(data) def test_protocol_cl_te_conflict(self): """H1CProtocol rejects CL with Transfer-Encoding.""" data = b"POST / HTTP/1.1\r\nContent-Length: 5\r\nTransfer-Encoding: chunked\r\n\r\n" p = H1CProtocol() with pytest.raises( InvalidHeader, match="Content-Length with Transfer-Encoding" ): p.feed(data) def test_te_before_cl(self): """Order doesn't matter - TE before CL is also rejected.""" data = b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\nContent-Length: 5\r\n\r\n" with pytest.raises( InvalidHeader, match="Content-Length with Transfer-Encoding" ): parse_request(data) def test_te_before_cl_fast(self): """Fast parser: TE before CL also rejected.""" data = b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\nContent-Length: 5\r\n\r\n" with pytest.raises( InvalidHeader, match="Content-Length with Transfer-Encoding" ): parse_request_fast(data) class TestChunkedInHTTP10: """Chunked Transfer-Encoding in HTTP/1.0 must be rejected.""" def test_basic_parser_chunked_http10(self): """Basic parser rejects chunked in HTTP/1.0.""" data = b"POST / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n\r\n" with pytest.raises(InvalidHeader, match="[Cc]hunked.*HTTP/1.0"): parse_request(data) def test_fast_parser_chunked_http10(self): """Fast parser rejects chunked in HTTP/1.0.""" data = b"POST / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n\r\n" with pytest.raises(InvalidHeader, match="[Cc]hunked.*HTTP/1.0"): parse_request_fast(data) def test_protocol_chunked_http10(self): """H1CProtocol rejects chunked in HTTP/1.0.""" data = b"POST / HTTP/1.0\r\nTransfer-Encoding: chunked\r\n\r\n" p = H1CProtocol() with pytest.raises(InvalidHeader, match="[Cc]hunked.*HTTP/1.0"): p.feed(data) class TestStackedChunkedEncoding: """Stacked chunked encoding must be rejected.""" def test_basic_parser_stacked_chunked(self): """Basic parser rejects stacked chunked encoding.""" data = b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked, chunked\r\n\r\n" with pytest.raises(InvalidHeader, match="[Ss]tacked"): parse_request(data) def test_fast_parser_stacked_chunked(self): """Fast parser rejects stacked chunked encoding.""" data = b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked, chunked\r\n\r\n" with pytest.raises(InvalidHeader, match="[Ss]tacked"): parse_request_fast(data) def test_protocol_stacked_chunked(self): """H1CProtocol rejects stacked chunked encoding.""" data = b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked, chunked\r\n\r\n" p = H1CProtocol() with pytest.raises(InvalidHeader, match="[Ss]tacked"): p.feed(data) def test_stacked_chunked_multiple_headers(self): """Multiple T-E headers with chunked is also stacked.""" data = b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\nTransfer-Encoding: chunked\r\n\r\n" with pytest.raises(InvalidHeader, match="[Ss]tacked"): parse_request(data) def test_stacked_chunked_multiple_headers_fast(self): """Fast parser: multiple T-E headers with chunked is stacked.""" data = b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\nTransfer-Encoding: chunked\r\n\r\n" with pytest.raises(InvalidHeader, match="[Ss]tacked"): parse_request_fast(data) class TestValidRequests: """Valid requests should still be accepted.""" def test_single_content_length(self): """Single Content-Length header is valid.""" data = b"POST / HTTP/1.1\r\nHost: localhost\r\nContent-Length: 5\r\n\r\nHello" result = parse_request(data) assert result["method"] == b"POST" def test_single_content_length_fast(self): """Fast parser: single Content-Length is valid.""" data = b"POST / HTTP/1.1\r\nHost: localhost\r\nContent-Length: 5\r\n\r\nHello" req = parse_request_fast(data) assert req.method == b"POST" assert req.content_length == 5 def test_chunked_http11(self): """Chunked encoding in HTTP/1.1 is valid.""" data = ( b"POST / HTTP/1.1\r\nHost: localhost\r\nTransfer-Encoding: chunked\r\n\r\n" ) result = parse_request(data) assert result["method"] == b"POST" def test_chunked_http11_fast(self): """Fast parser: chunked in HTTP/1.1 is valid.""" data = ( b"POST / HTTP/1.1\r\nHost: localhost\r\nTransfer-Encoding: chunked\r\n\r\n" ) req = parse_request_fast(data) assert req.method == b"POST" assert req.has_chunked def test_gzip_chunked(self): """gzip, chunked Transfer-Encoding is valid.""" data = b"POST / HTTP/1.1\r\nHost: localhost\r\nTransfer-Encoding: gzip, chunked\r\n\r\n" result = parse_request(data) assert result["method"] == b"POST" def test_gzip_chunked_fast(self): """Fast parser: gzip, chunked is valid.""" data = b"POST / HTTP/1.1\r\nHost: localhost\r\nTransfer-Encoding: gzip, chunked\r\n\r\n" req = parse_request_fast(data) assert req.method == b"POST" assert req.has_chunked def test_protocol_valid_chunked(self): """H1CProtocol: chunked in HTTP/1.1 is valid.""" data = ( b"POST / HTTP/1.1\r\nHost: localhost\r\nTransfer-Encoding: chunked\r\n\r\n" ) p = H1CProtocol() p.feed(data) assert p.method == b"POST" assert p.is_chunked def test_protocol_full_chunked_request(self): """H1CProtocol: complete chunked request works.""" chunks = [] p = H1CProtocol(on_body=lambda c: chunks.append(c)) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5\r\nHello\r\n") p.feed(b"0\r\n\r\n") assert chunks == [b"Hello"] assert p.is_complete class TestChunkSizeValidation: """Chunk size must be strictly validated.""" def test_protocol_invalid_chunk_hex(self): """Invalid hex in chunk size is rejected.""" p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") with pytest.raises(ProtocolParseError): p.feed(b"GG\r\n") def test_protocol_empty_chunk_size(self): """Empty chunk size line is rejected.""" p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") with pytest.raises(ProtocolParseError): p.feed(b"\r\n") def test_protocol_whitespace_in_chunk_size(self): """Leading whitespace in chunk size is rejected.""" p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") with pytest.raises(ProtocolParseError): p.feed(b" 5\r\n") def test_protocol_valid_chunk_extension(self): """Chunk extension after semicolon is allowed.""" chunks = [] p = H1CProtocol(on_body=lambda c: chunks.append(c)) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5;ext=value\r\nHello\r\n") p.feed(b"0\r\n\r\n") assert chunks == [b"Hello"] benoitc-gunicorn_h1c-52625fd/tests/test_parser.py000066400000000000000000000715161524260035600221350ustar00rootroot00000000000000"""Tests for gunicorn_h1c HTTP parser.""" import pytest from gunicorn_h1c import ( InvalidHeader, InvalidHeaderName, InvalidHTTPVersion, InvalidRequestMethod, LimitRequestHeaders, LimitRequestLine, ParseError, parse_request, parse_request_fast, ) class TestBasicParser: """Tests for the basic parser (_parser module).""" def test_parse_simple_get(self): from gunicorn_h1c import parse_request data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request(data) assert result["method"] == b"GET" assert result["path"] == b"/" assert result["minor_version"] == 1 assert len(result["headers"]) == 1 assert result["headers"][0] == (b"Host", b"localhost") assert result["consumed"] == len(data) def test_parse_post_with_headers(self): from gunicorn_h1c import parse_request data = b"POST /api/users HTTP/1.1\r\nHost: api.example.com\r\nContent-Type: application/json\r\nContent-Length: 42\r\n\r\n" result = parse_request(data) assert result["method"] == b"POST" assert result["path"] == b"/api/users" assert len(result["headers"]) == 3 def test_parse_with_query_string(self): from gunicorn_h1c import parse_request data = b"GET /search?q=hello&page=1 HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request(data) assert result["path"] == b"/search?q=hello&page=1" def test_http_10(self): from gunicorn_h1c import parse_request data = b"GET / HTTP/1.0\r\nHost: localhost\r\n\r\n" result = parse_request(data) assert result["minor_version"] == 0 def test_incomplete_request(self): from gunicorn_h1c import IncompleteError, parse_request data = b"GET / HTTP/1.1\r\nHost: local" with pytest.raises(IncompleteError): parse_request(data) def test_invalid_request(self): from gunicorn_h1c import ParseError, parse_request data = b"INVALID REQUEST\r\n\r\n" with pytest.raises(ParseError): parse_request(data) def test_incremental_parsing(self): from gunicorn_h1c import IncompleteError, parse_request # Start with partial data buffer = b"GET / HTTP/1.1\r\n" with pytest.raises(IncompleteError): parse_request(buffer, last_len=0) # Add more data buffer += b"Host: localhost\r\n\r\n" result = parse_request(buffer, last_len=16) assert result["method"] == b"GET" class TestFastParser: """Tests for the fast parser (_parser_fast module).""" def test_parse_simple_get(self): from gunicorn_h1c import parse_request_fast data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" req = parse_request_fast(data) assert req.method == b"GET" assert req.path == b"/" assert req.minor_version == 1 assert len(req.headers) == 1 assert req.consumed == len(data) def test_content_length_extraction(self): from gunicorn_h1c import parse_request_fast data = b"POST /api HTTP/1.1\r\nContent-Length: 100\r\n\r\n" req = parse_request_fast(data) assert req.content_length == 100 assert not req.has_chunked def test_chunked_detection(self): from gunicorn_h1c import parse_request_fast data = b"POST /api HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n" req = parse_request_fast(data) assert req.has_chunked assert req.content_length == -1 def test_connection_close(self): from gunicorn_h1c import parse_request_fast data = b"GET / HTTP/1.1\r\nConnection: close\r\n\r\n" req = parse_request_fast(data) assert req.connection_close == 1 def test_connection_keepalive(self): from gunicorn_h1c import parse_request_fast data = b"GET / HTTP/1.1\r\nConnection: keep-alive\r\n\r\n" req = parse_request_fast(data) assert req.connection_close == 0 def test_get_header(self): from gunicorn_h1c import parse_request_fast data = b"GET / HTTP/1.1\r\nHost: localhost\r\nX-Custom: value\r\n\r\n" req = parse_request_fast(data) assert req.get_header("Host") == b"localhost" assert req.get_header("host") == b"localhost" # case-insensitive assert req.get_header("X-Custom") == b"value" assert req.get_header("X-Missing") is None def test_header_count(self): from gunicorn_h1c import parse_request_fast data = b"GET / HTTP/1.1\r\nA: 1\r\nB: 2\r\nC: 3\r\n\r\n" req = parse_request_fast(data) assert req.header_count == 3 def test_asgi_headers_lowercase(self): """asgi_headers should return headers with lowercase names.""" from gunicorn_h1c import parse_request_fast data = b"GET / HTTP/1.1\r\nHost: localhost\r\nContent-Type: text/plain\r\n\r\n" req = parse_request_fast(data) # Regular headers preserve original case assert req.headers[0] == (b"Host", b"localhost") assert req.headers[1] == (b"Content-Type", b"text/plain") # ASGI headers have lowercase names asgi_headers = req.asgi_headers assert asgi_headers[0] == (b"host", b"localhost") assert asgi_headers[1] == (b"content-type", b"text/plain") def test_asgi_headers_is_list(self): """asgi_headers should return a list (not tuple) per ASGI spec.""" from gunicorn_h1c import parse_request_fast data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" req = parse_request_fast(data) assert isinstance(req.asgi_headers, list) def test_incremental_parsing_fast(self): from gunicorn_h1c._parser_fast import IncompleteError from gunicorn_h1c import parse_request_fast # Partial request buffer = b"GET / HTTP/1.1\r\n" with pytest.raises(IncompleteError): parse_request_fast(buffer, last_len=0) # Complete request with last_len optimization buffer += b"Host: localhost\r\n\r\n" req = parse_request_fast(buffer, last_len=16) assert req.method == b"GET" def test_many_headers(self): from gunicorn_h1c import parse_request_fast # Build request with 200 headers (must increase limit from default 100) headers = b"".join(f"X-Header-{i}: value{i}\r\n".encode() for i in range(200)) data = b"GET / HTTP/1.1\r\n" + headers + b"\r\n" req = parse_request_fast(data, limit_request_fields=256) assert req.header_count == 200 def test_incremental_parsing_raw(self): from gunicorn_h1c._parser_fast import IncompleteError, parse_request_raw buffer = b"GET / HTTP/1.1\r\n" with pytest.raises(IncompleteError): parse_request_raw(buffer, last_len=0) buffer += b"Host: localhost\r\n\r\n" result = parse_request_raw(buffer, last_len=16) assert result[6] == len(buffer) # consumed == full length class TestParseResponse: """Tests for response parsing.""" def test_parse_simple_response(self): from gunicorn_h1c import parse_response data = b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\n" result = parse_response(data) assert result["status"] == 200 assert result["message"] == b"OK" assert result["minor_version"] == 1 assert result["consumed"] == len(data) def test_parse_404_response(self): from gunicorn_h1c import parse_response data = b"HTTP/1.1 404 Not Found\r\n\r\n" result = parse_response(data) assert result["status"] == 404 assert result["message"] == b"Not Found" class TestParseHeaders: """Tests for header-only parsing.""" def test_parse_headers(self): from gunicorn_h1c import parse_headers data = b"Host: localhost\r\nContent-Type: text/plain\r\n\r\n" headers = parse_headers(data) assert len(headers) == 2 assert headers[0] == (b"Host", b"localhost") assert headers[1] == (b"Content-Type", b"text/plain") class TestParseToWsgiEnviron: """Tests for WSGI environ generation.""" def test_simple_get(self): from gunicorn_h1c import parse_to_wsgi_environ data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" environ = parse_to_wsgi_environ(data) assert environ["REQUEST_METHOD"] == "GET" assert environ["PATH_INFO"] == "/" assert environ["QUERY_STRING"] == "" assert environ["SERVER_PROTOCOL"] == "HTTP/1.1" assert environ["HTTP_HOST"] == "localhost" def test_with_query_string(self): from gunicorn_h1c import parse_to_wsgi_environ data = b"GET /search?q=hello&page=1 HTTP/1.1\r\nHost: localhost\r\n\r\n" environ = parse_to_wsgi_environ(data) assert environ["PATH_INFO"] == "/search" assert environ["QUERY_STRING"] == "q=hello&page=1" def test_server_and_client(self): from gunicorn_h1c import parse_to_wsgi_environ data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" environ = parse_to_wsgi_environ( data, server=("127.0.0.1", 8000), client=("10.0.0.1", 54321) ) assert environ["SERVER_NAME"] == "127.0.0.1" assert environ["SERVER_PORT"] == "8000" assert environ["REMOTE_ADDR"] == "10.0.0.1" assert environ["REMOTE_PORT"] == "54321" def test_url_scheme(self): from gunicorn_h1c import parse_to_wsgi_environ data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" environ = parse_to_wsgi_environ(data, url_scheme="https") assert environ["wsgi.url_scheme"] == "https" def test_content_type_header(self): from gunicorn_h1c import parse_to_wsgi_environ data = b"POST /api HTTP/1.1\r\nContent-Type: application/json\r\n\r\n" environ = parse_to_wsgi_environ(data) assert environ["CONTENT_TYPE"] == "application/json" assert "HTTP_CONTENT_TYPE" not in environ def test_content_length_header(self): from gunicorn_h1c import parse_to_wsgi_environ data = b"POST /api HTTP/1.1\r\nContent-Length: 42\r\n\r\n" environ = parse_to_wsgi_environ(data) assert environ["CONTENT_LENGTH"] == "42" assert "HTTP_CONTENT_LENGTH" not in environ def test_header_transformation(self): from gunicorn_h1c import parse_to_wsgi_environ data = b"GET / HTTP/1.1\r\nAccept-Language: en-US\r\nX-Custom-Header: value\r\n\r\n" environ = parse_to_wsgi_environ(data) assert environ["HTTP_ACCEPT_LANGUAGE"] == "en-US" assert environ["HTTP_X_CUSTOM_HEADER"] == "value" def test_duplicate_headers(self): from gunicorn_h1c import parse_to_wsgi_environ data = ( b"GET / HTTP/1.1\r\nAccept: text/html\r\nAccept: application/json\r\n\r\n" ) environ = parse_to_wsgi_environ(data) assert environ["HTTP_ACCEPT"] == "text/html,application/json" def test_http_10(self): from gunicorn_h1c import parse_to_wsgi_environ data = b"GET / HTTP/1.0\r\nHost: localhost\r\n\r\n" environ = parse_to_wsgi_environ(data) assert environ["SERVER_PROTOCOL"] == "HTTP/1.0" class TestParseToAsgiScope: """Tests for ASGI scope generation.""" def test_simple_get(self): from gunicorn_h1c import parse_to_asgi_scope data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" scope = parse_to_asgi_scope(data) assert scope["type"] == "http" assert scope["asgi"] == {"version": "3.0", "spec_version": "2.4"} assert scope["http_version"] == "1.1" assert scope["method"] == "GET" assert scope["path"] == "/" assert scope["raw_path"] == b"/" assert scope["query_string"] == b"" assert scope["root_path"] == "" def test_with_query_string(self): from gunicorn_h1c import parse_to_asgi_scope data = b"GET /search?q=hello&page=1 HTTP/1.1\r\nHost: localhost\r\n\r\n" scope = parse_to_asgi_scope(data) assert scope["path"] == "/search" assert scope["raw_path"] == b"/search" assert scope["query_string"] == b"q=hello&page=1" def test_server_and_client(self): from gunicorn_h1c import parse_to_asgi_scope data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" scope = parse_to_asgi_scope( data, server=("127.0.0.1", 8000), client=("10.0.0.1", 54321) ) assert scope["server"] == ("127.0.0.1", 8000) assert scope["client"] == ("10.0.0.1", 54321) def test_scheme_and_root_path(self): from gunicorn_h1c import parse_to_asgi_scope data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" scope = parse_to_asgi_scope(data, scheme="https", root_path="/app") assert scope["scheme"] == "https" assert scope["root_path"] == "/app" def test_headers_lowercase(self): from gunicorn_h1c import parse_to_asgi_scope data = b"GET / HTTP/1.1\r\nHost: localhost\r\nContent-Type: text/plain\r\n\r\n" scope = parse_to_asgi_scope(data) headers = scope["headers"] assert len(headers) == 2 assert headers[0] == (b"host", b"localhost") assert headers[1] == (b"content-type", b"text/plain") def test_http_10(self): from gunicorn_h1c import parse_to_asgi_scope data = b"GET / HTTP/1.0\r\nHost: localhost\r\n\r\n" scope = parse_to_asgi_scope(data) assert scope["http_version"] == "1.0" def test_server_none_default(self): from gunicorn_h1c import parse_to_asgi_scope data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" scope = parse_to_asgi_scope(data) assert scope["server"] is None assert scope["client"] is None class TestValidationExceptions: """Tests for validation exception hierarchy.""" def test_exception_hierarchy(self): """All validation exceptions should inherit from ParseError.""" assert issubclass(LimitRequestLine, ParseError) assert issubclass(LimitRequestHeaders, ParseError) assert issubclass(InvalidRequestMethod, ParseError) assert issubclass(InvalidHTTPVersion, ParseError) assert issubclass(InvalidHeaderName, ParseError) assert issubclass(InvalidHeader, ParseError) def test_parse_error_inherits_value_error(self): """ParseError should inherit from ValueError.""" assert issubclass(ParseError, ValueError) class TestLimitRequestLine: """Tests for request line limit enforcement.""" def test_request_line_exceeds_limit(self): """Should raise LimitRequestLine when request line is too long.""" long_path = b"/" + b"x" * 200 data = b"GET " + long_path + b" HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(LimitRequestLine): parse_request(data, limit_request_line=100) def test_request_line_within_limit(self): """Should succeed when request line is within limit.""" data = b"GET /short HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request(data, limit_request_line=100) assert result["method"] == b"GET" def test_request_line_at_limit(self): """Should succeed when request line is exactly at limit.""" # "GET / HTTP/1.1" is 14 chars data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request(data, limit_request_line=14) assert result["method"] == b"GET" def test_fast_parser_limit_request_line(self): """Fast parser should also enforce request line limits.""" long_path = b"/" + b"x" * 200 data = b"GET " + long_path + b" HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(LimitRequestLine): parse_request_fast(data, limit_request_line=100) class TestLimitRequestHeaders: """Tests for header limit enforcement.""" def test_too_many_headers(self): """Should raise LimitRequestHeaders when too many headers.""" headers = b"".join(f"X-Header-{i}: value\r\n".encode() for i in range(50)) data = b"GET / HTTP/1.1\r\n" + headers + b"\r\n" with pytest.raises(LimitRequestHeaders): parse_request(data, limit_request_fields=10) def test_header_count_within_limit(self): """Should succeed when header count is within limit.""" headers = b"".join(f"X-Header-{i}: value\r\n".encode() for i in range(5)) data = b"GET / HTTP/1.1\r\n" + headers + b"\r\n" result = parse_request(data, limit_request_fields=10) assert result["method"] == b"GET" def test_header_size_exceeds_limit(self): """Should raise LimitRequestHeaders when header is too large.""" long_value = b"x" * 1000 data = b"GET / HTTP/1.1\r\nX-Long: " + long_value + b"\r\n\r\n" with pytest.raises(LimitRequestHeaders): parse_request(data, limit_request_field_size=100) def test_header_size_within_limit(self): """Should succeed when header size is within limit.""" data = b"GET / HTTP/1.1\r\nX-Short: value\r\n\r\n" result = parse_request(data, limit_request_field_size=100) assert result["method"] == b"GET" def test_header_size_includes_crlf(self): """Header size calculation must include CRLF to match Python parser. Header line: "X-Test: value\\r\\n" Size = name(6) + ": "(2) + value(5) + CRLF(2) = 15 bytes """ # Exactly at limit (15 bytes) - should pass data = b"GET / HTTP/1.1\r\nX-Test: value\r\n\r\n" result = parse_request(data, limit_request_field_size=15) assert result["method"] == b"GET" # One byte over limit - should fail with pytest.raises(LimitRequestHeaders): parse_request(data, limit_request_field_size=14) def test_fast_parser_header_limits(self): """Fast parser should also enforce header limits.""" headers = b"".join(f"X-Header-{i}: value\r\n".encode() for i in range(50)) data = b"GET / HTTP/1.1\r\n" + headers + b"\r\n" with pytest.raises(LimitRequestHeaders): parse_request_fast(data, limit_request_fields=10) class TestInvalidRequestMethod: """Tests for method validation.""" def test_lowercase_method_rejected(self): """Lowercase methods should be rejected by default.""" data = b"get / HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(InvalidRequestMethod): parse_request(data) def test_lowercase_method_with_permit(self): """Lowercase methods should be allowed with permit flag.""" data = b"get / HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request(data, permit_unconventional_http_method=True) assert result["method"] == b"get" def test_mixed_case_method_rejected(self): """Mixed case methods should be rejected by default.""" data = b"GeT / HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(InvalidRequestMethod): parse_request(data) def test_short_method_rejected(self): """Methods shorter than 3 chars should be rejected.""" data = b"GO / HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(InvalidRequestMethod): parse_request(data) def test_short_method_with_permit(self): """Short methods should be allowed with permit flag.""" data = b"GO / HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request(data, permit_unconventional_http_method=True) assert result["method"] == b"GO" def test_hash_in_method_rejected(self): """Methods with # should be rejected by default.""" data = b"GET# / HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(InvalidRequestMethod): parse_request(data) def test_standard_methods_accepted(self): """Standard HTTP methods should be accepted.""" for method in [ b"GET", b"POST", b"PUT", b"DELETE", b"PATCH", b"HEAD", b"OPTIONS", ]: data = method + b" / HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request(data) assert result["method"] == method def test_fast_parser_method_validation(self): """Fast parser should also validate methods.""" data = b"get / HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(InvalidRequestMethod): parse_request_fast(data) class TestInvalidHTTPVersion: """Tests for HTTP version validation.""" def test_http_10_accepted(self): """HTTP/1.0 should be accepted.""" data = b"GET / HTTP/1.0\r\nHost: localhost\r\n\r\n" result = parse_request(data) assert result["minor_version"] == 0 def test_http_11_accepted(self): """HTTP/1.1 should be accepted.""" data = b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request(data) assert result["minor_version"] == 1 def test_fast_parser_version_validation(self): """Fast parser should also validate HTTP version.""" data = b"GET / HTTP/1.0\r\nHost: localhost\r\n\r\n" req = parse_request_fast(data) assert req.minor_version == 0 class TestHeaderValidation: """Tests for header name and value validation.""" def test_valid_header_names(self): """Valid header names should be accepted.""" data = b"GET / HTTP/1.1\r\nX-Custom-Header: value\r\nContent-Type: text/plain\r\n\r\n" result = parse_request(data) assert len(result["headers"]) == 2 def test_fast_parser_header_validation(self): """Fast parser should also validate headers.""" data = b"GET / HTTP/1.1\r\nX-Valid: value\r\n\r\n" req = parse_request_fast(data) assert req.header_count == 1 class TestDefaultLimits: """Tests for default limit values.""" def test_default_limits_allow_reasonable_requests(self): """Default limits should allow reasonable sized requests.""" # Request line under 8190 path = b"/" + b"x" * 1000 headers = b"".join(f"X-Header-{i}: value\r\n".encode() for i in range(50)) data = b"GET " + path + b" HTTP/1.1\r\n" + headers + b"\r\n" result = parse_request(data) assert result["method"] == b"GET" def test_default_limit_request_line(self): """Default request line limit should be 8190.""" # Create a request line that exceeds 8190 long_path = b"/" + b"x" * 9000 data = b"GET " + long_path + b" HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(LimitRequestLine): parse_request(data) def test_default_limit_request_fields(self): """Default header count limit should be 100.""" headers = b"".join(f"X-Header-{i}: value\r\n".encode() for i in range(150)) data = b"GET / HTTP/1.1\r\n" + headers + b"\r\n" with pytest.raises(LimitRequestHeaders): parse_request(data) class TestParseErrorSpecificExceptions: """Tests for specific exceptions when phr_parse_request returns -1.""" def test_parse_error_lowercase_method(self): """Lowercase method should raise InvalidRequestMethod.""" with pytest.raises(InvalidRequestMethod): parse_request(b"get / HTTP/1.1\r\n\r\n") def test_parse_error_non_token_char_in_method(self): """Non-token char in method should raise InvalidRequestMethod.""" # Using a control character (0x01) inside the method with pytest.raises(InvalidRequestMethod): parse_request(b"GE\x01T / HTTP/1.1\r\n\r\n") def test_parse_error_invalid_http_version(self): """HTTP/2.0 should raise InvalidHTTPVersion.""" with pytest.raises(InvalidHTTPVersion): parse_request(b"GET / HTTP/2.0\r\n\r\n") def test_parse_error_http_09(self): """HTTP/0.9 should raise InvalidHTTPVersion.""" with pytest.raises(InvalidHTTPVersion): parse_request(b"GET / HTTP/0.9\r\n\r\n") def test_parse_error_space_in_header_name(self): """Space in header name should raise InvalidHeaderName.""" with pytest.raises(InvalidHeaderName): parse_request(b"GET / HTTP/1.1\r\nBad Header: value\r\n\r\n") def test_parse_error_nul_in_header_value(self): """NUL in header value should raise InvalidHeader.""" with pytest.raises(InvalidHeader): parse_request(b"GET / HTTP/1.1\r\nX-Test: val\x00ue\r\n\r\n") def test_parse_error_bare_lf_in_header_value(self): """Bare LF in header value should raise InvalidHeader.""" with pytest.raises(InvalidHeader): parse_request(b"GET / HTTP/1.1\r\nX-Test: value\nmore\r\n\r\n") def test_parse_error_bare_cr_in_header_value(self): """Bare CR in header value should raise InvalidHeader.""" with pytest.raises(InvalidHeader): parse_request(b"GET / HTTP/1.1\r\nX-Test: value\rmore\r\n\r\n") def test_parse_error_ftp_protocol(self): """FTP/1.1 should raise InvalidHTTPVersion.""" with pytest.raises(InvalidHTTPVersion): parse_request(b"GET / FTP/1.1\r\n\r\n") def test_parse_error_rtsp_protocol(self): """RTSP/1.0 should raise InvalidHTTPVersion.""" with pytest.raises(InvalidHTTPVersion): parse_request(b"GET / RTSP/1.0\r\n\r\n") def test_fast_parser_parse_error_lowercase_method(self): """Fast parser: lowercase method should raise InvalidRequestMethod.""" with pytest.raises(InvalidRequestMethod): parse_request_fast(b"get / HTTP/1.1\r\n\r\n") def test_fast_parser_parse_error_invalid_http_version(self): """Fast parser: HTTP/2.0 should raise InvalidHTTPVersion.""" with pytest.raises(InvalidHTTPVersion): parse_request_fast(b"GET / HTTP/2.0\r\n\r\n") def test_fast_parser_parse_error_space_in_header_name(self): """Fast parser: space in header name should raise InvalidHeaderName.""" with pytest.raises(InvalidHeaderName): parse_request_fast(b"GET / HTTP/1.1\r\nBad Header: value\r\n\r\n") def test_fast_parser_parse_error_nul_in_header_value(self): """Fast parser: NUL in header value should raise InvalidHeader.""" with pytest.raises(InvalidHeader): parse_request_fast(b"GET / HTTP/1.1\r\nX-Test: val\x00ue\r\n\r\n") def test_fast_parser_parse_error_bare_lf_in_header_value(self): """Fast parser: bare LF in header value should raise InvalidHeader.""" with pytest.raises(InvalidHeader): parse_request_fast(b"GET / HTTP/1.1\r\nX-Test: value\nmore\r\n\r\n") def test_fast_parser_parse_error_ftp_protocol(self): """Fast parser: FTP/1.1 should raise InvalidHTTPVersion.""" with pytest.raises(InvalidHTTPVersion): parse_request_fast(b"GET / FTP/1.1\r\n\r\n") class TestProtocolParseErrorSpecificExceptions: """Tests for specific exceptions in H1CProtocol when phr_parse_request returns -1.""" def test_protocol_parse_error_lowercase_method(self): """H1CProtocol: lowercase method should raise InvalidRequestMethod.""" import gunicorn_h1c protocol = gunicorn_h1c.H1CProtocol(on_headers_complete=lambda: None) with pytest.raises(InvalidRequestMethod): protocol.feed(b"get / HTTP/1.1\r\n\r\n") def test_protocol_parse_error_invalid_http_version(self): """H1CProtocol: HTTP/2.0 should raise InvalidHTTPVersion.""" import gunicorn_h1c protocol = gunicorn_h1c.H1CProtocol(on_headers_complete=lambda: None) with pytest.raises(InvalidHTTPVersion): protocol.feed(b"GET / HTTP/2.0\r\n\r\n") def test_protocol_parse_error_space_in_header_name(self): """H1CProtocol: space in header name should raise InvalidHeaderName.""" import gunicorn_h1c protocol = gunicorn_h1c.H1CProtocol(on_headers_complete=lambda: None) with pytest.raises(InvalidHeaderName): protocol.feed(b"GET / HTTP/1.1\r\nBad Header: value\r\n\r\n") def test_protocol_parse_error_nul_in_header_value(self): """H1CProtocol: NUL in header value should raise InvalidHeader.""" import gunicorn_h1c protocol = gunicorn_h1c.H1CProtocol(on_headers_complete=lambda: None) with pytest.raises(InvalidHeader): protocol.feed(b"GET / HTTP/1.1\r\nX-Test: val\x00ue\r\n\r\n") def test_protocol_parse_error_bare_lf_in_header_value(self): """H1CProtocol: bare LF in header value should raise InvalidHeader.""" import gunicorn_h1c protocol = gunicorn_h1c.H1CProtocol(on_headers_complete=lambda: None) with pytest.raises(InvalidHeader): protocol.feed(b"GET / HTTP/1.1\r\nX-Test: value\nmore\r\n\r\n") def test_protocol_parse_error_ftp_protocol(self): """H1CProtocol: FTP/1.1 should raise InvalidHTTPVersion.""" import gunicorn_h1c protocol = gunicorn_h1c.H1CProtocol(on_headers_complete=lambda: None) with pytest.raises(InvalidHTTPVersion): protocol.feed(b"GET / FTP/1.1\r\n\r\n") benoitc-gunicorn_h1c-52625fd/tests/test_protocol.py000066400000000000000000001122761524260035600225010ustar00rootroot00000000000000"""Tests for H1CProtocol callback-based parser.""" import pytest from gunicorn_h1c._protocol import ParseError as ProtocolParseError from gunicorn_h1c import ( H1CProtocol, InvalidChunkExtension, LimitRequestHeaders, LimitRequestLine, ParseError, # From _parser module ) class TestH1CProtocolBasic: """Basic tests for callback-based parser.""" def test_simple_get(self): """Test parsing a simple GET request.""" headers_complete = False message_complete = False def on_headers(): nonlocal headers_complete headers_complete = True def on_complete(): nonlocal message_complete message_complete = True p = H1CProtocol( on_headers_complete=on_headers, on_message_complete=on_complete, ) p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n") assert headers_complete assert message_complete assert p.method == b"GET" assert p.path == b"/" assert p.http_version == (1, 1) assert p.is_complete def test_post_with_body(self): """Test parsing POST with Content-Length body.""" body_chunks = [] complete = False def on_body(chunk): body_chunks.append(chunk) def on_complete(): nonlocal complete complete = True p = H1CProtocol( on_body=on_body, on_message_complete=on_complete, ) p.feed(b"POST /api HTTP/1.1\r\nContent-Length: 5\r\n\r\nHello") assert complete assert body_chunks == [b"Hello"] assert p.method == b"POST" assert p.content_length == 5 def test_chunked_body(self): """Test parsing chunked transfer encoding.""" chunks = [] complete = False def on_body(chunk): chunks.append(chunk) def on_complete(): nonlocal complete complete = True p = H1CProtocol( on_body=on_body, on_message_complete=on_complete, ) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5\r\nHello\r\n") p.feed(b"0\r\n\r\n") assert complete assert chunks == [b"Hello"] assert p.is_chunked def test_partial_headers(self): """Headers split across multiple feeds.""" complete = False def on_headers(): nonlocal complete complete = True p = H1CProtocol(on_headers_complete=on_headers) p.feed(b"GET / HTTP/1.1\r\n") assert not complete p.feed(b"Host: local") assert not complete p.feed(b"host\r\n\r\n") assert complete assert p.method == b"GET" def test_http_10(self): """Test HTTP/1.0 parsing.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.0\r\nHost: localhost\r\n\r\n") assert p.http_version == (1, 0) assert not p.should_keep_alive # HTTP/1.0 defaults to close def test_http_11_keep_alive(self): """Test HTTP/1.1 default keep-alive.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n") assert p.http_version == (1, 1) assert p.should_keep_alive # HTTP/1.1 defaults to keep-alive def test_connection_close(self): """Test Connection: close header.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nConnection: close\r\n\r\n") assert not p.should_keep_alive def test_connection_keep_alive_http10(self): """Test Connection: keep-alive with HTTP/1.0.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.0\r\nConnection: keep-alive\r\n\r\n") assert p.should_keep_alive class TestH1CProtocolCallbacks: """Test callback invocation order and data.""" def test_callback_order(self): """Test callbacks fire in correct order.""" events = [] p = H1CProtocol( on_message_begin=lambda: events.append("begin"), on_url=lambda url: events.append(f"url:{url}"), on_header=lambda n, v: events.append(f"header:{n}:{v}"), on_headers_complete=lambda: events.append("headers_complete"), on_body=lambda c: events.append(f"body:{c}"), on_message_complete=lambda: events.append("message_complete"), ) p.feed(b"POST / HTTP/1.1\r\nHost: localhost\r\nContent-Length: 5\r\n\r\nHello") # Callback order: begin, url, headers, headers_complete, body, message_complete assert events == [ "begin", "url:b'/'", "header:b'Host':b'localhost'", "header:b'Content-Length':b'5'", "headers_complete", "body:b'Hello'", "message_complete", ] def test_skip_body(self): """Test returning True from on_headers_complete skips body.""" body_received = False def on_headers(): return True # Skip body def on_body(chunk): nonlocal body_received body_received = True p = H1CProtocol( on_headers_complete=on_headers, on_body=on_body, ) p.feed(b"POST / HTTP/1.1\r\nContent-Length: 5\r\n\r\nHello") assert not body_received assert p.is_complete def test_multiple_headers(self): """Test multiple headers are captured.""" headers = [] def on_header(name, value): headers.append((name, value)) p = H1CProtocol(on_header=on_header) p.feed( b"GET / HTTP/1.1\r\nHost: localhost\r\nAccept: */*\r\nUser-Agent: test\r\n\r\n" ) assert len(headers) == 3 assert headers[0] == (b"Host", b"localhost") assert headers[1] == (b"Accept", b"*/*") assert headers[2] == (b"User-Agent", b"test") class TestH1CProtocolChunked: """Test chunked transfer encoding parsing.""" def test_multiple_chunks(self): """Test parsing multiple chunks.""" chunks = [] p = H1CProtocol(on_body=lambda c: chunks.append(c)) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5\r\nHello\r\n") p.feed(b"6\r\n World\r\n") p.feed(b"0\r\n\r\n") assert chunks == [b"Hello", b" World"] def test_chunk_split_across_feeds(self): """Test chunk data split across multiple feeds.""" chunks = [] p = H1CProtocol(on_body=lambda c: chunks.append(c)) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"b\r\n") # 11 bytes = "Hello World" p.feed(b"Hello") p.feed(b" Worl") p.feed(b"d\r\n") # last byte of chunk + CRLF p.feed(b"0\r\n\r\n") assert b"".join(chunks) == b"Hello World" def test_chunk_size_with_extension(self): """Test chunk size with extension (ignored).""" chunks = [] p = H1CProtocol(on_body=lambda c: chunks.append(c)) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5;ext=value\r\nHello\r\n") p.feed(b"0\r\n\r\n") assert chunks == [b"Hello"] def test_chunk_with_trailers(self): """Test chunked encoding with trailers (ignored).""" complete = False def on_complete(): nonlocal complete complete = True p = H1CProtocol(on_message_complete=on_complete) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5\r\nHello\r\n") p.feed(b"0\r\nX-Trailer: value\r\n\r\n") assert complete class TestH1CProtocolReset: """Test parser reset for keepalive.""" def test_reset(self): """Test reset clears state for next request.""" p = H1CProtocol() p.feed(b"GET /first HTTP/1.1\r\nHost: localhost\r\n\r\n") assert p.path == b"/first" assert p.is_complete p.reset() assert not p.is_complete assert p.method == b"" assert p.path == b"" p.feed(b"POST /second HTTP/1.1\r\nHost: localhost\r\n\r\n") assert p.method == b"POST" assert p.path == b"/second" assert p.is_complete def test_keepalive_sequence(self): """Test multiple requests on same parser.""" methods = [] def on_headers(): methods.append(p.method) p = H1CProtocol(on_headers_complete=on_headers) # First request p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n") assert p.is_complete p.reset() # Second request p.feed(b"POST /api HTTP/1.1\r\nHost: localhost\r\nContent-Length: 0\r\n\r\n") assert p.is_complete p.reset() # Third request p.feed(b"DELETE /resource HTTP/1.1\r\nHost: localhost\r\n\r\n") assert p.is_complete assert methods == [b"GET", b"POST", b"DELETE"] class TestH1CProtocolFinish: """Test finish() for EOF handling.""" def test_finish_chunked_in_trailer_state(self): """Test finish() completes request when in chunked trailer state.""" chunks = [] complete = False def on_body(chunk): chunks.append(chunk) def on_complete(): nonlocal complete complete = True p = H1CProtocol( on_body=on_body, on_message_complete=on_complete, ) # Send chunked request without final CRLF after zero-chunk p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5\r\nHello\r\n") p.feed(b"0\r\n") # Zero chunk but no final CRLF assert not complete # Not complete yet - waiting for final CRLF assert not p.is_complete p.finish() # Signal EOF assert complete assert p.is_complete assert chunks == [b"Hello"] def test_finish_no_effect_when_complete(self): """Test finish() is no-op when already complete.""" complete_count = 0 def on_complete(): nonlocal complete_count complete_count += 1 p = H1CProtocol(on_message_complete=on_complete) p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n") assert complete_count == 1 assert p.is_complete p.finish() # Should be no-op assert complete_count == 1 # Not called again def test_finish_no_effect_during_headers(self): """Test finish() is no-op during header parsing.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\n") # Incomplete headers assert not p.is_complete p.finish() assert not p.is_complete # Still incomplete class TestH1CProtocolProperties: """Test property accessors.""" def test_headers_list(self): """Test headers property returns list of tuples.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\nAccept: */*\r\n\r\n") headers = p.headers assert isinstance(headers, list) assert len(headers) == 2 assert headers[0] == (b"Host", b"localhost") assert headers[1] == (b"Accept", b"*/*") def test_get_header(self): """Test case-insensitive header lookup.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\nContent-Type: text/plain\r\n\r\n") assert p.get_header(b"Host") == b"localhost" assert p.get_header(b"host") == b"localhost" assert p.get_header(b"HOST") == b"localhost" assert p.get_header(b"Content-Type") == b"text/plain" assert p.get_header(b"content-type") == b"text/plain" assert p.get_header(b"X-Missing") is None def test_asgi_headers_lowercase(self): """Test asgi_headers returns headers with lowercase names.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\nContent-Type: text/plain\r\n\r\n") # Regular headers preserve original case assert p.headers[0] == (b"Host", b"localhost") assert p.headers[1] == (b"Content-Type", b"text/plain") # ASGI headers have lowercase names asgi_headers = p.asgi_headers assert asgi_headers[0] == (b"host", b"localhost") assert asgi_headers[1] == (b"content-type", b"text/plain") assert isinstance(asgi_headers, list) def test_content_length(self): """Test content_length property.""" p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nContent-Length: 100\r\n\r\n") assert p.content_length == 100 def test_content_length_none(self): """Test content_length is None when not set.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n") assert p.content_length is None def test_should_upgrade(self): """Test should_upgrade property.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nUpgrade: websocket\r\n\r\n") assert p.should_upgrade class TestH1CProtocolErrors: """Test error handling.""" def test_invalid_request(self): """Test invalid HTTP request raises ParseError.""" p = H1CProtocol() with pytest.raises(ProtocolParseError): p.feed(b"INVALID\r\n\r\n") def test_invalid_chunk_size(self): """Test invalid chunk size raises ParseError.""" p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") with pytest.raises(ProtocolParseError): p.feed(b"ZZZZ\r\n") # Invalid hex def test_parse_error_has_status_code(self): """Test ParseError has status_code attribute.""" p = H1CProtocol() with pytest.raises(ProtocolParseError): p.feed(b"INVALID\r\n\r\n") class TestH1CProtocolBodyParsing: """Test body parsing edge cases.""" def test_zero_content_length(self): """Test Content-Length: 0 completes immediately.""" complete = False def on_complete(): nonlocal complete complete = True p = H1CProtocol(on_message_complete=on_complete) p.feed(b"POST / HTTP/1.1\r\nContent-Length: 0\r\n\r\n") assert complete def test_body_split_across_feeds(self): """Test body data split across multiple feeds.""" chunks = [] p = H1CProtocol(on_body=lambda c: chunks.append(c)) p.feed(b"POST / HTTP/1.1\r\nContent-Length: 10\r\n\r\nHello") p.feed(b"World") assert b"".join(chunks) == b"HelloWorld" def test_body_received_after_headers(self): """Test body arrives after headers in separate feed.""" headers_complete = False body_chunks = [] def on_headers(): nonlocal headers_complete headers_complete = True p = H1CProtocol( on_headers_complete=on_headers, on_body=lambda c: body_chunks.append(c), ) p.feed(b"POST / HTTP/1.1\r\nContent-Length: 5\r\n\r\n") assert headers_complete assert body_chunks == [] p.feed(b"Hello") assert body_chunks == [b"Hello"] class TestH1CProtocolQueryString: """Test path with query string.""" def test_path_includes_query(self): """Test path includes query string.""" p = H1CProtocol() p.feed(b"GET /search?q=hello&page=1 HTTP/1.1\r\nHost: localhost\r\n\r\n") assert p.path == b"/search?q=hello&page=1" def test_on_url_receives_full_path(self): """Test on_url callback receives path with query.""" url_received = None def on_url(url): nonlocal url_received url_received = url p = H1CProtocol(on_url=on_url) p.feed(b"GET /api?key=value HTTP/1.1\r\nHost: localhost\r\n\r\n") assert url_received == b"/api?key=value" class TestH1CProtocolChunkExtensionValidation: """Test chunk extension validation per RFC 9112.""" def test_bare_cr_in_chunk_extension_rejected(self): """Test chunk extension with bare CR is rejected.""" p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") with pytest.raises(InvalidChunkExtension) as exc_info: p.feed(b"1;\r\r\na\r\n0\r\n\r\n") assert "bare CR" in str(exc_info.value) def test_bare_cr_in_chunk_extension_value_rejected(self): """Test chunk extension value with bare CR is rejected.""" p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") with pytest.raises(InvalidChunkExtension) as exc_info: p.feed(b"1;ext=val\rue\r\na\r\n0\r\n\r\n") assert "bare CR" in str(exc_info.value) def test_valid_chunk_extension_accepted(self): """Test valid chunk extension is accepted.""" chunks = [] p = H1CProtocol(on_body=lambda c: chunks.append(c)) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5;ext=value\r\nHello\r\n0\r\n\r\n") assert chunks == [b"Hello"] assert p.is_complete def test_multiple_chunk_extensions_accepted(self): """Test multiple chunk extensions are accepted.""" chunks = [] p = H1CProtocol(on_body=lambda c: chunks.append(c)) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5;ext1=a;ext2=b\r\nHello\r\n0\r\n\r\n") assert chunks == [b"Hello"] assert p.is_complete def test_chunk_without_extension_accepted(self): """Test chunk without extension works normally.""" chunks = [] p = H1CProtocol(on_body=lambda c: chunks.append(c)) p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") p.feed(b"5\r\nHello\r\n0\r\n\r\n") assert chunks == [b"Hello"] assert p.is_complete def test_invalid_chunk_extension_is_parse_error(self): """Test InvalidChunkExtension inherits from ParseError.""" assert issubclass(InvalidChunkExtension, Exception) # It should inherit from ParseError (which inherits from ValueError) p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nTransfer-Encoding: chunked\r\n\r\n") try: p.feed(b"1;\r\r\na\r\n0\r\n\r\n") pytest.fail("Should have raised InvalidChunkExtension") except InvalidChunkExtension: pass # Expected except Exception as e: pytest.fail(f"Wrong exception type: {type(e).__name__}") class TestH1CProtocolLimitValidation: """Test request limit validation.""" def test_limit_request_line_exceeded(self): """Test LimitRequestLine raised when request line is too long.""" p = H1CProtocol(limit_request_line=20) with pytest.raises(LimitRequestLine): # Request line "GET /very/long/path HTTP/1.1" exceeds 20 bytes p.feed(b"GET /very/long/path HTTP/1.1\r\nHost: localhost\r\n\r\n") def test_limit_request_line_at_boundary(self): """Test request line at exactly the limit is accepted.""" # "GET / HTTP/1.1" is 14 bytes p = H1CProtocol(limit_request_line=14) p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n") assert p.is_complete assert p.method == b"GET" def test_limit_request_fields_exceeded(self): """Test LimitRequestHeaders raised when too many headers.""" p = H1CProtocol(limit_request_fields=2) with pytest.raises(LimitRequestHeaders): p.feed( b"GET / HTTP/1.1\r\n" b"Host: localhost\r\n" b"Accept: */*\r\n" b"User-Agent: test\r\n" # 3rd header exceeds limit of 2 b"\r\n" ) def test_limit_request_fields_at_boundary(self): """Test number of headers at exactly the limit is accepted.""" p = H1CProtocol(limit_request_fields=2) p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\nAccept: */*\r\n\r\n") assert p.is_complete assert len(p.headers) == 2 def test_limit_request_field_size_exceeded(self): """Test LimitRequestHeaders raised when header value is too large.""" p = H1CProtocol(limit_request_field_size=20) with pytest.raises(LimitRequestHeaders): p.feed( b"GET / HTTP/1.1\r\n" b"Host: this-is-a-very-long-hostname.example.com\r\n" b"\r\n" ) def test_limit_request_field_size_at_boundary(self): """Test header field at exactly the limit is accepted.""" # "Host" (4) + ": " (2) + "localhost" (9) = 15, plus name "Host" = 4 # Total header line without name: ": localhost" = 11 p = H1CProtocol(limit_request_field_size=20) p.feed(b"GET / HTTP/1.1\r\nHost: localhost\r\n\r\n") assert p.is_complete def test_limit_exceptions_inherit_from_parse_error(self): """Test limit exceptions inherit from ParseError for proper handling.""" # This ensures gunicorn can catch these with a single ParseError handler assert issubclass(LimitRequestLine, ParseError) assert issubclass(LimitRequestHeaders, ParseError) class TestH1CProtocolRemaining: """remaining() exposes bytes fed after a completed message. gunicorn's ASGI worker feeds whole data_received() buffers into feed(), so anything a client pipelines behind the request lands in the same call: the HTTP/2 preface after Upgrade: h2c, a WebSocket frame sent straight after the handshake, or a second HTTP/1 request. """ H2C_UPGRADE = ( b"GET / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Connection: Upgrade, HTTP2-Settings\r\n" b"HTTP2-Settings: AAMAAABkAAQAoAAAAAIAAAAA\r\n\r\n" ) H2C_PREFACE = b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n" def test_h2c_preface_recovered(self): """The blocking case: HTTP/2 preface pipelined behind the upgrade.""" p = H1CProtocol() p.feed(self.H2C_UPGRADE + self.H2C_PREFACE) assert p.is_complete assert p.should_upgrade assert p.remaining() == self.H2C_PREFACE def test_websocket_frame_after_handshake(self): """A client frame sent immediately after the handshake survives.""" frame = b"\x81\x85\x37\xfa\x21\x3d\x7f\x9f\x4d\x51\x58" p = H1CProtocol() p.feed( b"GET /ws HTTP/1.1\r\nHost: a\r\n" b"Upgrade: websocket\r\nConnection: Upgrade\r\n\r\n" + frame ) assert p.is_complete assert p.remaining() == frame def test_pipelined_request(self): """A second HTTP/1 request in the same segment is recoverable.""" second = b"GET /2 HTTP/1.1\r\nHost: a\r\n\r\n" p = H1CProtocol() p.feed(b"GET /1 HTTP/1.1\r\nHost: a\r\n\r\n" + second) assert p.path == b"/1" assert p.remaining() == second def test_no_tail_is_empty_bytes(self): p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nHost: a\r\n\r\n") assert p.is_complete assert p.remaining() == b"" def test_empty_while_incomplete(self): """While a message is still parsing, every byte fed belongs to it.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nHost: ") assert not p.is_complete assert p.remaining() == b"" def test_empty_mid_body(self): p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nHost: a\r\nContent-Length: 5\r\n\r\nhel") assert not p.is_complete assert p.remaining() == b"" def test_content_length_body_tail(self): """Tail is measured from the end of the body, not the headers.""" p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nHost: a\r\nContent-Length: 5\r\n\r\nhelloTAIL") assert p.is_complete assert p.remaining() == b"TAIL" def test_chunked_body_tail(self): """Same for a chunked body: measured past the terminating chunk.""" p = H1CProtocol() p.feed( b"POST / HTTP/1.1\r\nHost: a\r\nTransfer-Encoding: chunked\r\n\r\n" b"5\r\nhello\r\n0\r\n\r\nAFTER" ) assert p.is_complete assert p.remaining() == b"AFTER" def test_chunked_with_trailer_tail(self): p = H1CProtocol() p.feed( b"POST / HTTP/1.1\r\nHost: a\r\nTransfer-Encoding: chunked\r\n\r\n" b"5\r\nhello\r\n0\r\nX-Trace-Id: 1\r\n\r\nAFTER" ) assert p.is_complete assert p.remaining() == b"AFTER" def test_tail_split_across_feeds(self): """A tail arriving in pieces is accumulated, not truncated.""" p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nHost: a\r\n\r\nPRI * HTTP") p.feed(b"/2.0\r\n\r\nSM\r\n\r\n") assert p.remaining() == self.H2C_PREFACE def test_tail_split_byte_by_byte(self): p = H1CProtocol() p.feed(b"GET / HTTP/1.1\r\nHost: a\r\n\r\n") for byte in b"abcdef": p.feed(bytes([byte])) assert p.remaining() == b"abcdef" def test_message_split_before_tail(self): """Headers split across feeds, tail arriving with the body.""" p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nHost: a\r\nContent-Len") p.feed(b"gth: 5\r\n\r\nhelloTAIL") assert p.is_complete assert p.remaining() == b"TAIL" def test_body_split_before_tail(self): p = H1CProtocol() p.feed(b"POST / HTTP/1.1\r\nHost: a\r\nContent-Length: 5\r\n\r\nhel") p.feed(b"loTAIL") assert p.is_complete assert p.remaining() == b"TAIL" def test_chunked_split_before_tail(self): p = H1CProtocol() p.feed( b"POST / HTTP/1.1\r\nHost: a\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhel" ) p.feed(b"lo\r\n0\r\n") assert not p.is_complete p.feed(b"\r\nAFTER") assert p.is_complete assert p.remaining() == b"AFTER" def test_reset_drops_the_tail(self): """reset() must not leak the tail into the next message.""" p = H1CProtocol() p.feed(b"GET /1 HTTP/1.1\r\nHost: a\r\n\r\nGET /2 HTTP/1.1\r\nHost: b\r\n\r\n") assert p.remaining() != b"" p.reset() assert p.remaining() == b"" def test_reset_then_feed_tail_parses_pipeline(self): """The keepalive loop: capture, reset, feed back.""" p = H1CProtocol() p.feed( b"GET /1 HTTP/1.1\r\nHost: a\r\n\r\n" b"GET /2 HTTP/1.1\r\nHost: b\r\n\r\n" b"GET /3 HTTP/1.1\r\nHost: c\r\n\r\n" ) paths = [p.path] tail = p.remaining() while tail: p.reset() p.feed(tail) paths.append(p.path) tail = p.remaining() assert paths == [b"/1", b"/2", b"/3"] def test_pipelined_bodies_survive_reset(self): """Bodies too, not just header-only requests.""" bodies = [] p = H1CProtocol(on_body=bodies.append) p.feed( b"POST /1 HTTP/1.1\r\nHost: a\r\nContent-Length: 3\r\n\r\nabc" b"POST /2 HTTP/1.1\r\nHost: a\r\nContent-Length: 3\r\n\r\ndef" ) assert bodies == [b"abc"] tail = p.remaining() p.reset() p.feed(tail) assert p.path == b"/2" assert bodies == [b"abc", b"def"] assert p.remaining() == b"" def test_finish_reports_no_tail(self): """At EOF nothing follows, so a half-parsed trailer is not a tail.""" p = H1CProtocol() p.feed( b"POST / HTTP/1.1\r\nHost: a\r\nTransfer-Encoding: chunked\r\n\r\n" b"5\r\nhello\r\n0\r\n" ) p.finish() assert p.is_complete assert p.remaining() == b"" class TestH1CProtocolRemainingLimit: """The retained tail is bounded so a completed parser cannot grow forever. Callers that feed every socket read without checking is_complete would otherwise let a client stream unbounded data into the parser between the end of a message and reset(). """ def complete_with_tail(self, tail, **kwargs): p = H1CProtocol(**kwargs) p.feed(b"GET / HTTP/1.1\r\nHost: a\r\n\r\n" + tail) return p def test_tail_capped_at_limit(self): p = self.complete_with_tail(b"0123456789EXTRA", limit_remaining=10) assert p.remaining() == b"0123456789" assert p.remaining_truncated def test_not_truncated_within_limit(self): p = self.complete_with_tail(b"0123456789", limit_remaining=10) assert p.remaining() == b"0123456789" assert not p.remaining_truncated def test_default_limit_bounds_a_flood(self): """Feeding a completed parser must not grow the buffer without bound.""" p = self.complete_with_tail(b"") for _ in range(64): p.feed(b"x" * 65536) # 4 MB total assert len(p.remaining()) == 65536 assert p.remaining_truncated def test_overflow_never_raises_from_feed(self): """feed() must not start failing on input it used to discard.""" p = self.complete_with_tail(b"", limit_remaining=4) for _ in range(100): p.feed(b"flood") # no exception assert p.remaining() == b"floo" def test_limit_applies_across_split_feeds(self): p = H1CProtocol(limit_remaining=10) p.feed(b"GET / HTTP/1.1\r\nHost: a\r\n\r\n01234") p.feed(b"56789EXTRA") assert p.remaining() == b"0123456789" assert p.remaining_truncated @pytest.mark.parametrize( "request_bytes", [ b"GET / HTTP/1.1\r\nHost: a\r\n\r\n", b"GET / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\nConnection: Upgrade\r\n\r\n", b"POST / HTTP/1.1\r\nHost: a\r\nContent-Length: 2\r\n\r\nhi", b"POST / HTTP/1.1\r\nHost: a\r\nTransfer-Encoding: chunked\r\n\r\n0\r\n\r\n", ], ids=["no-body", "upgrade", "content-length", "chunked"], ) def test_every_completion_path_clamps(self, request_bytes): p = H1CProtocol(limit_remaining=10) p.feed(request_bytes + b"0123456789EXTRA") assert p.remaining() == b"0123456789" assert p.remaining_truncated def test_zero_means_unlimited(self): p = self.complete_with_tail(b"", limit_remaining=0) p.feed(b"y" * 200000) assert len(p.remaining()) == 200000 assert not p.remaining_truncated def test_reset_clears_truncated_flag(self): p = self.complete_with_tail(b"0123456789EXTRA", limit_remaining=10) assert p.remaining_truncated p.reset() assert not p.remaining_truncated def test_negative_limit_rejected(self): with pytest.raises(ValueError): H1CProtocol(limit_remaining=-1) def test_h2c_preface_fits_default_limit(self): """The case this all exists for stays well inside the default.""" p = H1CProtocol() p.feed( b"GET / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Connection: Upgrade, HTTP2-Settings\r\n" b"HTTP2-Settings: AAMAAABkAAQAoAAAAAIAAAAA\r\n\r\n" b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n" ) assert p.remaining() == b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n" assert not p.remaining_truncated class TestH1CProtocolUpgradeBody: """An upgrade request is an ordinary request and its body is parsed. The protocol switch happens only after the server answers 101 (RFC 9110 section 7.8), and RFC 7540 section 3.2 requires an h2c client to send any payload in full before its first HTTP/2 frame. So the bytes after the headers are the request body, framed as on any other request. """ def probe(self, raw, **kwargs): """Feed raw and return (events, parser).""" events = [] p = H1CProtocol( on_headers_complete=lambda: events.append("headers"), on_body=lambda chunk: events.append(("body", chunk)), on_message_complete=lambda: events.append("complete"), **kwargs, ) p.feed(raw) return events, p def body_of(self, events): return b"".join(c for e, c in (x for x in events if isinstance(x, tuple))) def test_content_length_body_is_delivered(self): events, p = self.probe( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Content-Length: 11\r\n\r\nhello=world" ) assert self.body_of(events) == b"hello=world" assert events[-1] == "complete" assert p.is_complete assert p.should_upgrade assert p.remaining() == b"" def test_body_then_pipelined_tail(self): """The tail starts after the body, not after the headers.""" preface = b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n" events, p = self.probe( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Content-Length: 11\r\n\r\nhello=world" + preface ) assert self.body_of(events) == b"hello=world" assert p.remaining() == preface def test_chunked_body_is_dechunked(self): events, p = self.probe( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Transfer-Encoding: chunked\r\n\r\n" b"b\r\nhello=world\r\n0\r\n\r\n" ) assert self.body_of(events) == b"hello=world" assert p.is_complete assert p.remaining() == b"" def test_chunked_body_then_tail(self): events, p = self.probe( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Transfer-Encoding: chunked\r\n\r\n" b"5\r\nhello\r\n6\r\n=world\r\n0\r\n\r\nAFTER" ) assert self.body_of(events) == b"hello=world" assert p.remaining() == b"AFTER" def test_body_split_across_feeds(self): events = [] p = H1CProtocol(on_body=lambda c: events.append(c)) p.feed( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\nContent-Length: 11\r\n\r\nhel" ) assert not p.is_complete p.feed(b"lo=worldAFTER") assert b"".join(events) == b"hello=world" assert p.is_complete assert p.remaining() == b"AFTER" def test_chunked_body_split_mid_chunk(self): events = [] p = H1CProtocol(on_body=lambda c: events.append(c)) p.feed( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Transfer-Encoding: chunked\r\n\r\nb\r\nhel" ) assert not p.is_complete p.feed(b"lo=world\r\n0\r\n\r\nAFTER") assert b"".join(events) == b"hello=world" assert p.is_complete assert p.remaining() == b"AFTER" def test_unterminated_chunked_stays_incomplete(self): """No longer completes at the headers, as any chunked request.""" _, p = self.probe( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Transfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n" ) assert not p.is_complete def test_no_body_is_unchanged(self): """The WebSocket handshake and a bare h2c upgrade keep working.""" preface = b"PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n" events, p = self.probe( b"GET / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n\r\n" + preface ) assert events == ["headers", "complete"] assert p.should_upgrade assert p.remaining() == preface def test_content_length_zero_is_unchanged(self): events, p = self.probe( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\nContent-Length: 0\r\n\r\nAFTER" ) assert events == ["headers", "complete"] assert p.remaining() == b"AFTER" @pytest.mark.parametrize( "value", [b"h2c", b"websocket", b"bogus"], ids=["h2c", "websocket", "bogus"] ) def test_upgrade_value_is_not_special_cased(self, value): events, p = self.probe( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: " + value + b"\r\n" b"Content-Length: 11\r\n\r\nhello=world" ) assert self.body_of(events) == b"hello=world" assert p.remaining() == b"" def test_connection_upgrade_without_upgrade_header(self): events, p = self.probe( b"POST / HTTP/1.1\r\nHost: a\r\nConnection: upgrade\r\n" b"Content-Length: 11\r\n\r\nhello=world" ) assert self.body_of(events) == b"hello=world" assert not p.should_upgrade assert p.remaining() == b"" def test_connect_tunnel_bytes_stay_in_remaining(self): """CONNECT is not an upgrade: everything after the headers is tunnel.""" events, p = self.probe( b"CONNECT a:443 HTTP/1.1\r\nHost: a:443\r\n\r\nTUNNELBYTES" ) assert events == ["headers", "complete"] assert p.remaining() == b"TUNNELBYTES" def test_reset_between_pipelined_upgrade_requests(self): events, p = self.probe( b"POST /1 HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\n" b"Content-Length: 5\r\n\r\nfirst" b"POST /2 HTTP/1.1\r\nHost: a\r\nContent-Length: 6\r\n\r\nsecond" ) assert self.body_of(events) == b"first" assert p.path == b"/1" tail = p.remaining() p.reset() p.feed(tail) assert p.path == b"/2" assert p.is_complete assert p.remaining() == b"" def test_large_body_does_not_hit_the_tail_cap(self): """The body is consumed, so only the real tail counts against it.""" payload = b"x" * 200000 events, p = self.probe( b"POST / HTTP/1.1\r\nHost: a\r\nUpgrade: h2c\r\nContent-Length: 200000\r\n\r\n" + payload ) assert self.body_of(events) == payload assert not p.remaining_truncated assert p.remaining() == b"" benoitc-gunicorn_h1c-52625fd/tests/test_rfc9110_compliance.py000066400000000000000000000243451524260035600241160ustar00rootroot00000000000000"""RFC 9110 / RFC 9112 compliance tests. Each test pins a specific RFC rule so parser regressions surface as a named failure. New rules added here must land with the C parser change that enforces them (TDD workflow). """ import pytest from gunicorn_h1c._parser_fast import ParseError as FastParseError from gunicorn_h1c._protocol import ParseError as ProtocolParseError from gunicorn_h1c import ( H1CProtocol, InvalidHeader, InvalidHeaderName, ParseError, parse_request, parse_request_fast, ) FORBIDDEN_TRAILERS = [ b"Host", b"Content-Length", b"Transfer-Encoding", b"Trailer", b"Authorization", b"TE", ] CONTROL_BYTE_IDS = ["BEL-0x07", "FF-0x0c", "VT-0x0b", "DEL-0x7f"] CONTROL_BYTES = [b"\x07", b"\x0c", b"\x0b", b"\x7f"] # (request, field named in the error) for fields whose grammar admits a single # member. Transfer-Encoding is absent on purpose: repeats there are legal. SINGLETON_REPEAT_IDS = ["Host", "Content-Type", "host-vs-HOST", "content-TYPE"] SINGLETON_REPEATS = [ ( b"GET /p HTTP/1.1\r\nHost: a.example\r\nHost: b.example\r\n\r\n", "Duplicate Host header", ), ( b"POST /p HTTP/1.1\r\nHost: localhost\r\n" b"Content-Type: text/plain\r\nContent-Type: application/json\r\n\r\n", "Duplicate Content-Type header", ), ( b"GET /p HTTP/1.1\r\nhost: a.example\r\nHOST: b.example\r\n\r\n", "Duplicate Host header", ), ( b"POST /p HTTP/1.1\r\nHost: localhost\r\n" b"content-type: text/plain\r\nContent-TYPE: application/json\r\n\r\n", "Duplicate Content-Type header", ), ] class TestFieldValueControlChars: """RFC 9110 section 5.5: only VCHAR, SP, HTAB, obs-text allowed in value.""" @pytest.mark.parametrize("ctl", CONTROL_BYTES, ids=CONTROL_BYTE_IDS) def test_basic_parser_rejects(self, ctl): data = ( b"GET / HTTP/1.1\r\nHost: localhost\r\nX-Value: bad" + ctl + b"char\r\n\r\n" ) with pytest.raises(InvalidHeader): parse_request(data) @pytest.mark.parametrize("ctl", CONTROL_BYTES, ids=CONTROL_BYTE_IDS) def test_fast_parser_rejects(self, ctl): data = ( b"GET / HTTP/1.1\r\nHost: localhost\r\nX-Value: bad" + ctl + b"char\r\n\r\n" ) with pytest.raises(InvalidHeader): parse_request_fast(data) @pytest.mark.parametrize("ctl", CONTROL_BYTES, ids=CONTROL_BYTE_IDS) def test_protocol_rejects(self, ctl): data = ( b"GET / HTTP/1.1\r\nHost: localhost\r\nX-Value: bad" + ctl + b"char\r\n\r\n" ) p = H1CProtocol() with pytest.raises(InvalidHeader): p.feed(data) class TestRequestTargetFormMethodPairing: """RFC 9112 sections 3.2.3 & 3.2.4: form must match the method.""" def test_basic_parser_rejects_asterisk_with_get(self): data = b"GET * HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(ParseError): parse_request(data) def test_fast_parser_rejects_asterisk_with_get(self): data = b"GET * HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(FastParseError): parse_request_fast(data) def test_protocol_rejects_asterisk_with_get(self): data = b"GET * HTTP/1.1\r\nHost: localhost\r\n\r\n" p = H1CProtocol() with pytest.raises(ProtocolParseError): p.feed(data) def test_basic_parser_rejects_authority_with_get(self): data = b"GET example.com:443 HTTP/1.1\r\nHost: example.com:443\r\n\r\n" with pytest.raises(ParseError): parse_request(data) def test_fast_parser_rejects_authority_with_get(self): data = b"GET example.com:443 HTTP/1.1\r\nHost: example.com:443\r\n\r\n" with pytest.raises(FastParseError): parse_request_fast(data) def test_protocol_rejects_authority_with_get(self): data = b"GET example.com:443 HTTP/1.1\r\nHost: example.com:443\r\n\r\n" p = H1CProtocol() with pytest.raises(ProtocolParseError): p.feed(data) def test_basic_parser_rejects_relative_target(self): data = b"GET foo/bar HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(ParseError): parse_request(data) def test_fast_parser_rejects_relative_target(self): data = b"GET foo/bar HTTP/1.1\r\nHost: localhost\r\n\r\n" with pytest.raises(FastParseError): parse_request_fast(data) def test_protocol_rejects_relative_target(self): data = b"GET foo/bar HTTP/1.1\r\nHost: localhost\r\n\r\n" p = H1CProtocol() with pytest.raises(ProtocolParseError): p.feed(data) def test_options_asterisk_still_valid(self): data = b"OPTIONS * HTTP/1.1\r\nHost: localhost\r\n\r\n" result = parse_request(data) assert result["method"] == b"OPTIONS" assert result["path"] == b"*" def test_connect_authority_still_valid(self): data = b"CONNECT example.com:443 HTTP/1.1\r\nHost: example.com:443\r\n\r\n" result = parse_request(data) assert result["method"] == b"CONNECT" assert result["path"] == b"example.com:443" class TestForbiddenTrailerFields: """RFC 9110 section 6.5.1: certain headers must not appear in trailers.""" @pytest.mark.parametrize("name", FORBIDDEN_TRAILERS) def test_protocol_rejects(self, name): data = ( b"POST /p HTTP/1.1\r\nHost: localhost\r\nTransfer-Encoding: chunked\r\n\r\n" b"5\r\nhello\r\n" b"0\r\n" + name + b": evil\r\n\r\n" ) p = H1CProtocol() with pytest.raises(InvalidHeaderName): p.feed(data) def test_protocol_accepts_benign_trailer(self): data = ( b"POST /p HTTP/1.1\r\nHost: localhost\r\nTransfer-Encoding: chunked\r\n\r\n" b"5\r\nhello\r\n" b"0\r\nX-Trace-Id: 123\r\n\r\n" ) p = H1CProtocol() p.feed(data) # no exception class TestContentLengthListForm: """RFC 9112 section 6.3: reject list-form Content-Length (smuggling risk).""" def test_basic_parser_rejects(self): data = ( b"POST /p HTTP/1.1\r\nHost: localhost\r\nContent-Length: 5, 5\r\n\r\nhello" ) with pytest.raises(InvalidHeader): parse_request(data) def test_fast_parser_rejects(self): data = ( b"POST /p HTTP/1.1\r\nHost: localhost\r\nContent-Length: 5, 5\r\n\r\nhello" ) with pytest.raises(InvalidHeader): parse_request_fast(data) def test_protocol_rejects(self): data = ( b"POST /p HTTP/1.1\r\nHost: localhost\r\nContent-Length: 5, 5\r\n\r\nhello" ) p = H1CProtocol() with pytest.raises(InvalidHeader): p.feed(data) def test_basic_parser_rejects_mismatch(self): data = ( b"POST /p HTTP/1.1\r\nHost: localhost\r\nContent-Length: 5, 6\r\n\r\nhello" ) with pytest.raises(InvalidHeader): parse_request(data) class TestSingletonFieldRepeats: """RFC 9110 section 5.3: singleton fields must not be repeated. Host, Content-Length, and Content-Type each have a grammar admitting only one member, so a repeat cannot be merged into a list and leaves the message ambiguous. Duplicate Host is a routing- and cache-confusion vector. """ @pytest.mark.parametrize( ("data", "message"), SINGLETON_REPEATS, ids=SINGLETON_REPEAT_IDS ) def test_basic_parser_rejects(self, data, message): with pytest.raises(InvalidHeader, match=message): parse_request(data) @pytest.mark.parametrize( ("data", "message"), SINGLETON_REPEATS, ids=SINGLETON_REPEAT_IDS ) def test_fast_parser_rejects(self, data, message): with pytest.raises(InvalidHeader, match=message): parse_request_fast(data) @pytest.mark.parametrize( ("data", "message"), SINGLETON_REPEATS, ids=SINGLETON_REPEAT_IDS ) def test_protocol_rejects(self, data, message): p = H1CProtocol() with pytest.raises(InvalidHeader, match=message): p.feed(data) def test_single_host_accepted(self): data = b"GET /p HTTP/1.1\r\nHost: a.example\r\n\r\n" result = parse_request(data) assert (b"Host", b"a.example") in result["headers"] def test_single_content_type_accepted(self): data = ( b"POST /p HTTP/1.1\r\nHost: localhost\r\nContent-Type: text/plain\r\n" b"Content-Length: 5\r\n\r\nhello" ) req = parse_request_fast(data) assert req.get_header("content-type") == b"text/plain" assert req.content_length == 5 def test_protocol_single_occurrence_accepted(self): data = ( b"POST /p HTTP/1.1\r\nHost: a.example\r\nContent-Type: text/plain\r\n" b"Content-Length: 5\r\n\r\nhello" ) p = H1CProtocol() p.feed(data) assert (b"Host", b"a.example") in p.headers assert (b"Content-Type", b"text/plain") in p.headers def test_repeated_transfer_encoding_accepted(self): """Transfer-Encoding is not a singleton; framing logic owns it.""" data = ( b"POST /p HTTP/1.1\r\nHost: localhost\r\n" b"Transfer-Encoding: gzip\r\nTransfer-Encoding: chunked\r\n\r\n" ) result = parse_request(data) assert result["method"] == b"POST" def test_repeated_transfer_encoding_accepted_protocol(self): data = ( b"POST /p HTTP/1.1\r\nHost: localhost\r\n" b"Transfer-Encoding: gzip\r\nTransfer-Encoding: chunked\r\n\r\n" b"0\r\n\r\n" ) p = H1CProtocol() p.feed(data) # no exception def test_repeated_list_valued_field_accepted(self): """Ordinary list-valued fields repeat legitimately and must pass.""" data = ( b"GET /p HTTP/1.1\r\nHost: localhost\r\n" b"Accept: text/html\r\nAccept: application/json\r\n" b"Via: 1.1 alpha\r\nVia: 1.1 beta\r\n\r\n" ) result = parse_request(data) assert (b"Accept", b"text/html") in result["headers"] assert (b"Accept", b"application/json") in result["headers"] assert (b"Via", b"1.1 alpha") in result["headers"] assert (b"Via", b"1.1 beta") in result["headers"]