././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1700382465.8931386 funcparserlib-2.0.0a0/LICENSE0000644000000000000000000000205414526343402012530 0ustar00Copyright © 2009/2023 Andrey Vlasovskikh Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1700382465.8931386 funcparserlib-2.0.0a0/README.md0000644000000000000000000001316014526343402013002 0ustar00Funcparserlib ============= Recursive descent parsing library for Python based on functional combinators. [![PyPI](https://img.shields.io/pypi/v/funcparserlib)](https://pypi.org/project/funcparserlib/) [![PyPI - Downloads](https://img.shields.io/pypi/dm/funcparserlib)](https://pypi.org/project/funcparserlib/) Description ----------- The primary focus of `funcparserlib` is **parsing little languages** or **external DSLs** (domain specific languages). Parsers made with `funcparserlib` are pure-Python LL(\*) parsers. It means that it's **very easy to write parsers** without thinking about lookaheads and other hardcore parsing stuff. However, recursive descent parsing is a rather slow method compared to LL(k) or LR(k) algorithms. Still, parsing with `funcparserlib` is **at least twice faster than PyParsing**, a very popular library for Python. The source code of `funcparserlib` is only 1.2K lines of code, with lots of comments. Its API is fully type hinted. It features the longest parsed prefix error reporting, as well as a tiny lexer generator for token position tracking. The idea of parser combinators used in `funcparserlib` comes from the [Introduction to Functional Programming](https://www.cl.cam.ac.uk/teaching/Lectures/funprog-jrh-1996/) course. We have converted it from ML into Python. Installation ------------ You can install `funcparserlib` from [PyPI](https://pypi.org/project/funcparserlib/): ```shell $ pip install funcparserlib ``` There are no dependencies on other libraries. Documentation ------------- * [Getting Started](https://funcparserlib.pirx.ru/getting-started/) * Your **starting point** with `funcparserlib` * [API Reference](https://funcparserlib.pirx.ru/api/) * Learn the details of the API There are several examples available in the `tests/` directory: * [GraphViz DOT parser](https://github.com/vlasovskikh/funcparserlib/blob/master/tests/dot.py) * [JSON parser](https://github.com/vlasovskikh/funcparserlib/blob/master/tests/json.py) See also [the changelog](https://funcparserlib.pirx.ru/changes/). Example ------- Let's consider a little language of **numeric expressions** with a syntax similar to Python expressions. Here are some expression strings in this language: ``` 0 1 + 2 + 3 -1 + 2 ** 32 3.1415926 * (2 + 7.18281828e-1) * 42 ``` Here is **the complete source code** of the tokenizer and the parser for this language written using `funcparserlib`: ```python from typing import List, Tuple, Union from dataclasses import dataclass from funcparserlib.lexer import make_tokenizer, TokenSpec, Token from funcparserlib.parser import tok, Parser, many, forward_decl, finished @dataclass class BinaryExpr: op: str left: "Expr" right: "Expr" Expr = Union[BinaryExpr, int, float] def tokenize(s: str) -> List[Token]: specs = [ TokenSpec("whitespace", r"\s+"), TokenSpec("float", r"[+\-]?\d+\.\d*([Ee][+\-]?\d+)*"), TokenSpec("int", r"[+\-]?\d+"), TokenSpec("op", r"(\*\*)|[+\-*/()]"), ] tokenizer = make_tokenizer(specs) return [t for t in tokenizer(s) if t.type != "whitespace"] def parse(tokens: List[Token]) -> Expr: int_num = tok("int") >> int float_num = tok("float") >> float number = int_num | float_num expr: Parser[Token, Expr] = forward_decl() parenthesized = -op("(") + expr + -op(")") primary = number | parenthesized power = primary + many(op("**") + primary) >> to_expr term = power + many((op("*") | op("/")) + power) >> to_expr sum = term + many((op("+") | op("-")) + term) >> to_expr expr.define(sum) document = expr + -finished return document.parse(tokens) def op(name: str) -> Parser[Token, str]: return tok("op", name) def to_expr(args: Tuple[Expr, List[Tuple[str, Expr]]]) -> Expr: first, rest = args result = first for op, expr in rest: result = BinaryExpr(op, result, expr) return result ``` Now, consider this numeric expression: `3.1415926 * (2 + 7.18281828e-1) * 42`. Let's `tokenize()` it using the tokenizer we've created with `funcparserlib.lexer`: ``` [ Token('float', '3.1415926'), Token('op', '*'), Token('op', '('), Token('int', '2'), Token('op', '+'), Token('float', '7.18281828e-1'), Token('op', ')'), Token('op', '*'), Token('int', '42'), ] ``` Let's `parse()` these tokens into an expression tree using our parser created with `funcparserlib.parser`: ``` BinaryExpr( op='*', left=BinaryExpr( op='*', left=3.1415926, right=BinaryExpr(op='+', left=2, right=0.718281828), ), right=42, ) ``` Learn how to write this parser using `funcparserlib` in the [Getting Started](https://funcparserlib.pirx.ru/getting-started/) guide! Used By ------- Some open-source projects that use `funcparserlib` as an explicit dependency: * [Hy](https://github.com/hylang/hy), a Lisp dialect that's embedded in Python * 4.7K stars, version `~=1.0`, Python 3.8+ * [Splash](https://github.com/scrapinghub/splash), a JavaScript rendering service with HTTP API, by Scrapinghub * 3.9K stars, version `*`. Python 3 in Docker * [graphite-beacon](https://github.com/klen/graphite-beacon), a simple alerting system for Graphite metrics * 453 stars, version `==0.3.6`, Python 2 and 3 * [blockdiag](https://github.com/blockdiag/blockdiag), generates block-diagram image file from spec-text file * 194 stars, version `>= 1.0.0a0`, Python 3.7+ * [kll](https://github.com/kiibohd/kll), Keyboard Layout Language (KLL) compiler * 113 stars, copied source code, Python 3.5+ Next ---- Read the [Getting Started](https://funcparserlib.pirx.ru/getting-started/) guide to start learning `funcparserlib`. ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1700382465.8971386 funcparserlib-2.0.0a0/funcparserlib/__init__.py0000644000000000000000000000000014526343402016460 0ustar00././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1700382465.8971386 funcparserlib-2.0.0a0/funcparserlib/lexer.py0000644000000000000000000001703114526343402016054 0ustar00# Copyright © 2009/2023 Andrey Vlasovskikh # # Permission is hereby granted, free of charge, to any person obtaining a copy of this # software and associated documentation files (the "Software"), to deal in the Software # without restriction, including without limitation the rights to use, copy, modify, # merge, publish, distribute, sublicense, and/or sell copies of the Software, and to # permit persons to whom the Software is furnished to do so, subject to the following # conditions: # # The above copyright notice and this permission notice shall be included in all copies # or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A # PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF # CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE # OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. __all__ = ["make_tokenizer", "TokenSpec", "Token", "LexerError"] import re from typing import Callable, Iterable, List, Tuple, Optional, Sequence, Pattern, Union _Place = Tuple[int, int] _Spec = Tuple[str, Tuple] class LexerError(Exception): def __init__(self, place: _Place, msg: str) -> None: self.place = place self.msg = msg def __str__(self) -> str: s = "cannot tokenize data" line, pos = self.place return '%s: %d,%d: "%s"' % (s, line, pos, self.msg) class TokenSpec: """A token specification for generating a lexer via `make_tokenizer()`.""" def __init__(self, type: str, pattern: str, flags: int = 0) -> None: """Initialize a `TokenSpec` object. Parameters: type (str): User-defined type of the token (e.g. `"name"`, `"number"`, `"operator"`) pattern (str): Regexp for matching this token type flags (int, optional): Regexp flags, the second argument of `re.compile()` """ self.type = type self.pattern = pattern self.flags = flags def __repr__(self) -> str: return "TokenSpec(%r, %r, %r)" % (self.type, self.pattern, self.flags) class Token: """A token object that represents a substring of certain type in your text. You can compare tokens for equality using the `==` operator. Tokens also define custom `repr()` and `str()`. Attributes: type (str): User-defined type of the token (e.g. `"name"`, `"number"`, `"operator"`) value (str): Text value of the token start (Optional[Tuple[int, int]]): Start position (_line_, _column_) end (Optional[Tuple[int, int]]): End position (_line_, _column_) """ def __init__( self, type: str, value: str, start: Optional[_Place] = None, end: Optional[_Place] = None, ) -> None: """Initialize a `Token` object.""" self.type = type self.value = value self.start = start self.end = end def __repr__(self) -> str: return "Token(%r, %r)" % (self.type, self.value) def __eq__(self, other: object) -> bool: # FIXME: Case sensitivity is assumed here if not isinstance(other, Token): return False else: return self.type == other.type and self.value == other.value def _pos_str(self) -> str: if self.start is None or self.end is None: return "" else: sl, sp = self.start el, ep = self.end return "%d,%d-%d,%d:" % (sl, sp, el, ep) def __str__(self) -> str: s = "%s %s '%s'" % (self._pos_str(), self.type, self.value) return s.strip() @property def name(self) -> str: return self.value def pformat(self) -> str: return "%s %s '%s'" % ( self._pos_str().ljust(20), # noqa self.type.ljust(14), self.value, ) def make_tokenizer( specs: Sequence[Union[TokenSpec, _Spec]], ) -> Callable[[str], Iterable[Token]]: # noinspection GrazieInspection """Make a function that tokenizes text based on the regexp specs. Type: `(Sequence[TokenSpec | Tuple]) -> Callable[[str], Iterable[Token]]` A token spec is `TokenSpec` instance. !!! Note For legacy reasons, a token spec may also be a tuple of (_type_, _args_), where _type_ sets the value of `Token.type` for the token, and _args_ are the positional arguments for `re.compile()`: either just (_pattern_,) or (_pattern_, _flags_). It returns a tokenizer function that takes a string and returns an iterable of `Token` objects, or raises `LexerError` if it cannot tokenize the string according to its token specs. Examples: ```pycon >>> tokenize = make_tokenizer([ ... TokenSpec("space", r"\\s+"), ... TokenSpec("id", r"\\w+"), ... TokenSpec("op", r"[,!]"), ... ]) >>> text = "Hello, World!" >>> [t for t in tokenize(text) if t.type != "space"] # noqa [Token('id', 'Hello'), Token('op', ','), Token('id', 'World'), Token('op', '!')] >>> text = "Bye?" >>> list(tokenize(text)) Traceback (most recent call last): ... lexer.LexerError: cannot tokenize data: 1,4: "Bye?" ``` """ compiled: List[Tuple[str, Pattern[str]]] = [] for spec in specs: if isinstance(spec, TokenSpec): c = spec.type, re.compile(spec.pattern, spec.flags) else: name, args = spec c = name, re.compile(*args) compiled.append(c) def match_specs(s: str, i: int, position: Tuple[int, int]) -> Token: line, pos = position for type, regexp in compiled: m = regexp.match(s, i) if m is not None: value = m.group() nls = value.count("\n") n_line = line + nls if nls == 0: n_pos = pos + len(value) else: n_pos = len(value) - value.rfind("\n") - 1 return Token(type, value, (line, pos + 1), (n_line, n_pos)) else: err_line = s.splitlines()[line - 1] raise LexerError((line, pos + 1), err_line) def f(s: str) -> Iterable[Token]: length = len(s) line, pos = 1, 0 i = 0 while i < length: t = match_specs(s, i, (line, pos)) yield t if t.end is None: raise ValueError("Token %r has no end specified", (t,)) line, pos = t.end i += len(t.value) return f # This is an example of token specs. See also [this article][1] for a # discussion of searching for multiline comments using regexps (including `*?`). # # [1]: http://ostermiller.org/findcomment.html _example_token_specs = [ TokenSpec("COMMENT", r"\(\*(.|[\r\n])*?\*\)", re.MULTILINE), TokenSpec("COMMENT", r"\{(.|[\r\n])*?\}", re.MULTILINE), TokenSpec("COMMENT", r"//.*"), TokenSpec("NL", r"[\r\n]+"), TokenSpec("SPACE", r"[ \t\r\n]+"), TokenSpec("NAME", r"[A-Za-z_][A-Za-z_0-9]*"), TokenSpec("REAL", r"[0-9]+\.[0-9]*([Ee][+\-]?[0-9]+)*"), TokenSpec("INT", r"[0-9]+"), TokenSpec("INT", r"\$[0-9A-Fa-f]+"), TokenSpec("OP", r"(\.\.)|(<>)|(<=)|(>=)|(:=)|[;,=\(\):\[\]\.+\-<>\*/@\^]"), TokenSpec("STRING", r"'([^']|(''))*'"), TokenSpec("CHAR", r"#[0-9]+"), TokenSpec("CHAR", r"#\$[0-9A-Fa-f]+"), ] # tokenize = make_tokenizer(_example_token_specs) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1700382465.8971386 funcparserlib-2.0.0a0/funcparserlib/parser.py0000644000000000000000000007053314526343402016237 0ustar00# Copyright © 2009/2023 Andrey Vlasovskikh # # Permission is hereby granted, free of charge, to any person obtaining a copy of this # software and associated documentation files (the "Software"), to deal in the Software # without restriction, including without limitation the rights to use, copy, modify, # merge, publish, distribute, sublicense, and/or sell copies of the Software, and to # permit persons to whom the Software is furnished to do so, subject to the following # conditions: # # The above copyright notice and this permission notice shall be included in all copies # or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A # PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF # CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE # OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. """Functional parsing combinators. Parsing combinators define an internal domain-specific language (DSL) for describing the parsing rules of a grammar. The DSL allows you to start with a few primitive parsers, then combine your parsers to get more complex ones, and finally cover the whole grammar you want to parse. The structure of the language: * Class `Parser` * All the primitives and combinators of the language return `Parser` objects * It defines the main `Parser.parse(tokens)` method * Primitive parsers * `tok(type, value)`, `a(value)`, `some(pred)`, `forward_decl()`, `finished` * Parser combinators * `p1 + p2`, `p1 | p2`, `p >> f`, `-p`, `maybe(p)`, `many(p)`, `oneplus(p)`, `skip(p)` * Abstraction * Use regular Python variables `p = ... # Expression of type Parser` to define new rules (non-terminals) of your grammar Every time you apply one of the combinators, you get a new `Parser` object. In other words, the set of `Parser` objects is closed under the means of combination. !!! Note We took the parsing combinators language from the book [Introduction to Functional Programming][1] and translated it from ML into Python. [1]: https://www.cl.cam.ac.uk/teaching/Lectures/funprog-jrh-1996/ """ __all__ = [ "some", "a", "tok", "many", "pure", "finished", "maybe", "skip", "oneplus", "forward_decl", "NoParseError", "Parser", ] import logging import warnings from typing import ( Any, Callable, Generic, List, Optional, Sequence, Tuple, TypeVar, Union, cast, overload, ) from funcparserlib.lexer import Token log = logging.getLogger("funcparserlib") debug = False _A = TypeVar("_A") _B = TypeVar("_B") _C = TypeVar("_C") class Parser(Generic[_A, _B]): """A parser object that can parse a sequence of tokens or can be combined with other parsers using `+`, `|`, `>>`, `many()`, and other parsing combinators. Type: `Parser[A, B]` The generic variables in the type are: `A` — the type of the tokens in the sequence to parse,`B` — the type of the parsed value. In order to define a parser for your grammar: 1. You start with primitive parsers by calling `a(value)`, `some(pred)`, `forward_decl()`, `finished` 2. You use parsing combinators `p1 + p2`, `p1 | p2`, `p >> f`, `many(p)`, and others to combine parsers into a more complex parser 3. You can assign complex parsers to variables to define names that correspond to the rules of your grammar !!! Note The constructor `Parser.__init__()` is considered **internal** and may be changed in future versions. Use primitive parsers and parsing combinators to construct new parsers. """ def __init__( self, p: Union[ "Parser[_A, _B]", Callable[[Sequence[_A], "State"], Tuple[_B, "State"]], ], ) -> None: """Wrap the parser function `p` into a `Parser` object.""" self.name = "" self.define(p) def named(self, name: str) -> "Parser[_A, _B]": # noinspection GrazieInspection """Specify the name of the parser for easier debugging. Type: `(str) -> Parser[A, B]` This name is used in the debug-level parsing log. You can also get it via the `Parser.name` attribute. Examples: ```pycon >>> expr = (a("x") + a("y")).named("expr") >>> expr.name 'expr' ``` ```pycon >>> expr = a("x") + a("y") >>> expr.name "('x', 'y')" ``` !!! Note You can enable the parsing log this way: ```python import logging logging.basicConfig(level=logging.DEBUG) import funcparserlib.parser funcparserlib.parser.debug = True ``` The way to enable the parsing log may be changed in future versions. """ self.name = name return self def define( self, p: Union[ "Parser[_A, _B]", Callable[[Sequence[_A], "State"], Tuple[_B, "State"]], ], ) -> None: """Define the parser created earlier as a forward declaration. Type: `(Parser[A, B]) -> None` Use `p = forward_decl()` in combination with `p.define(...)` to define recursive parsers. See the examples in the docs for `forward_decl()`. """ f = getattr(p, "run", p) if debug: setattr(self, "_run", f) else: setattr(self, "run", f) name = getattr(p, "name", p.__doc__) if name is not None: self.named(name) def run(self, tokens: Sequence[_A], s: "State") -> Tuple[_B, "State"]: """Run the parser against the tokens with the specified parsing state. Type: `(Sequence[A], State) -> Tuple[B, State]` The parsing state includes the current position in the sequence being parsed, and the position of the rightmost token that has been consumed while parsing for better error messages. If the parser fails to parse the tokens, it raises `NoParseError`. !!! Warning This is method is **internal** and may be changed in future versions. Use `Parser.parse(tokens)` instead and let the parser object take care of updating the parsing state. """ if debug: log.debug("trying %s" % self.name) return self._run(tokens, s) def _run(self, tokens: Sequence[_A], s: "State") -> Tuple[_B, "State"]: raise NotImplementedError("you must define() a parser") def parse(self, tokens: Sequence[_A]) -> _B: """Parse the sequence of tokens and return the parsed value. Type: `(Sequence[A]) -> B` It takes a sequence of tokens of arbitrary type `A` and returns the parsed value of arbitrary type `B`. If the parser fails to parse the tokens, it raises `NoParseError`. !!! Note Although `Parser.parse()` can parse sequences of any objects (including `str` which is a sequence of `str` chars), **the recommended way** is parsing sequences of `Token` objects. You **should** use a regexp-based tokenizer `make_tokenizer()` defined in `funcparserlib.lexer` to convert your text into a sequence of `Token` objects before parsing it. You will get more readable parsing error messages (as `Token` objects contain their position in the source file) and good separation of the lexical and syntactic levels of the grammar. """ try: (tree, _) = self.run(tokens, State(0, 0, None)) return tree except NoParseError as e: max = e.state.max if len(tokens) > max: t = tokens[max] if isinstance(t, Token): if t.start is None or t.end is None: loc = "" else: s_line, s_pos = t.start e_line, e_pos = t.end loc = "%d,%d-%d,%d: " % (s_line, s_pos, e_line, e_pos) msg = "%s%s: %r" % (loc, e.msg, t.value) elif isinstance(t, str): msg = "%s: %r" % (e.msg, t) else: msg = "%s: %s" % (e.msg, t) else: msg = "got unexpected end of input" e_parser = e.state.parser if isinstance(e_parser, Parser): msg = "%s, expected: %s" % (msg, e_parser.name) e.msg = msg raise @overload def __add__( # type: ignore[misc] self, other: "_IgnoredParser[_A]" ) -> "Parser[_A, _B]": pass @overload def __add__(self, other: "Parser[_A, _C]") -> "_TupleParser[_A, Tuple[_B, _C]]": pass def __add__( self, other: Union["_IgnoredParser[_A]", "Parser[_A, _C]"], ) -> Union["Parser[_A, _B]", "_TupleParser[_A, Tuple[_B, _C]]"]: """Sequential combination of parsers. It runs this parser, then the other parser. The return value of the resulting parser is a tuple of each parsed value in the sum of parsers. We merge all parsing results of `p1 + p2 + ... + pN` into a single tuple. It means that the parsing result may be a 2-tuple, a 3-tuple, a 4-tuple, etc. of parsed values. You avoid this by transforming the parsed pair into a new value using the `>>` combinator. You can also skip some parsing results in the resulting parsers by using `-p` or `skip(p)` for some parsers in your sum of parsers. It means that the parsing result might be a single value, not a tuple of parsed values. See the docs for `Parser.__neg__()` for more examples. Overloaded types (lots of them to provide stricter checking for the quite dynamic return type of this method): * `(self: Parser[A, B], _IgnoredParser[A]) -> Parser[A, B]` * `(self: Parser[A, B], Parser[A, C]) -> _TupleParser[A, Tuple[B, C]]` * `(self: _TupleParser[A, B], _IgnoredParser[A]) -> _TupleParser[A, B]` * `(self: _TupleParser[A, B], Parser[A, Any]) -> Parser[A, Any]` * `(self: _IgnoredParser[A], _IgnoredParser[A]) -> _IgnoredParser[A]` * `(self: _IgnoredParser[A], Parser[A, C]) -> Parser[A, C]` Examples: ```pycon >>> expr = a("x") + a("y") >>> expr.parse("xy") ('x', 'y') ``` ```pycon >>> expr = a("x") + a("y") + a("z") >>> expr.parse("xyz") ('x', 'y', 'z') ``` ```pycon >>> expr = a("x") + a("y") >>> expr.parse("xz") Traceback (most recent call last): ... parser.NoParseError: got unexpected token: 'z', expected: 'y' ``` """ def magic(v1: Any, v2: Any) -> _Tuple: if isinstance(v1, _Tuple): return _Tuple(v1 + (v2,)) else: return _Tuple((v1, v2)) @_TupleParser def _add(tokens: Sequence[_A], s: State) -> Tuple[Tuple[_B, _C], State]: (v1, s2) = self.run(tokens, s) (v2, s3) = other.run(tokens, s2) return cast(Tuple[_B, _C], magic(v1, v2)), s3 @Parser def ignored_right(tokens: Sequence[_A], s: State) -> Tuple[_B, State]: v, s2 = self.run(tokens, s) _, s3 = other.run(tokens, s2) return v, s3 name = "(%s, %s)" % (self.name, other.name) if isinstance(other, _IgnoredParser): return ignored_right.named(name) else: _add.name = name return _add def __or__(self, other: "Parser[_A, _C]") -> "Parser[_A, Union[_B, _C]]": """Choice combination of parsers. It runs this parser and returns its result. If the parser fails, it runs the other parser. Examples: ```pycon >>> expr = a("x") | a("y") >>> expr.parse("x") 'x' >>> expr.parse("y") 'y' >>> expr.parse("z") Traceback (most recent call last): ... parser.NoParseError: got unexpected token: 'z', expected: 'x' or 'y' ``` """ @Parser def _or(tokens: Sequence[_A], s: State) -> Tuple[Union[_B, _C], State]: try: return self.run(tokens, s) except NoParseError as e: state = e.state try: return other.run(tokens, State(s.pos, state.max, state.parser)) except NoParseError as e: if s.pos == e.state.max: e.state = State(e.state.pos, e.state.max, _or) raise _or.name = "%s or %s" % (self.name, other.name) return _or def __rshift__(self, f: Callable[[_B], _C]) -> "Parser[_A, _C]": """Transform the parsing result by applying the specified function. Type: `(Callable[[B], C]) -> Parser[A, C]` You can use it for transforming the parsed value into another value before including it into the parse tree (the AST). Examples: ```pycon >>> def make_canonical_name(s): ... return s.lower() >>> expr = (a("D") | a("d")) >> make_canonical_name >>> expr.parse("D") 'd' >>> expr.parse("d") 'd' ``` """ @Parser def _shift(tokens: Sequence[_A], s: State) -> Tuple[_C, State]: (v, s2) = self.run(tokens, s) return f(v), s2 return _shift.named(self.name) def bind(self, f: Callable[[_B], "Parser[_A, _C]"]) -> "Parser[_A, _C]": """Bind the parser to a monadic function that returns a new parser. Type: `(Callable[[B], Parser[A, C]]) -> Parser[A, C]` Also known as `>>=` in Haskell. !!! Note You can parse any context-free grammar without resorting to `bind`. Due to its poor performance please use it only when you really need it. """ @Parser def _bind(tokens: Sequence[_A], s: State) -> Tuple[_C, State]: (v, s2) = self.run(tokens, s) return f(v).run(tokens, s2) _bind.name = "(%s >>=)" % (self.name,) return _bind def __neg__(self) -> "_IgnoredParser[_A]": """Return a parser that parses the same tokens, but its parsing result is ignored by the sequential `+` combinator. Type: `(Parser[A, B]) -> _IgnoredParser[A]` You can use it for throwing away elements of concrete syntax (e.g. `","`, `";"`). Examples: ```pycon >>> expr = -a("x") + a("y") >>> expr.parse("xy") 'y' ``` ```pycon >>> expr = a("x") + -a("y") >>> expr.parse("xy") 'x' ``` ```pycon >>> expr = a("x") + -a("y") + a("z") >>> expr.parse("xyz") ('x', 'z') ``` ```pycon >>> expr = -a("x") + a("y") + -a("z") >>> expr.parse("xyz") 'y' ``` ```pycon >>> expr = -a("x") + a("y") >>> expr.parse("yz") Traceback (most recent call last): ... parser.NoParseError: got unexpected token: 'y', expected: 'x' ``` ```pycon >>> expr = a("x") + -a("y") >>> expr.parse("xz") Traceback (most recent call last): ... parser.NoParseError: got unexpected token: 'z', expected: 'y' ``` !!! Note You **should not** pass the resulting parser to any combinators other than `+`. You **should** have at least one non-skipped value in your `p1 + p2 + ... + pN`. The parsed value of `-p` is an **internal** `_Ignored` object, not intended for actual use. """ return _IgnoredParser(self) class State: """Parsing state that is maintained basically for error reporting. It consists of the current position `pos` in the sequence being parsed, and the position `max` of the rightmost token that has been consumed while parsing. """ def __init__( self, pos: int, max: int, parser: Union[ Parser, Callable[[Any, "State"], Tuple[Any, "State"]], None, ] = None, ) -> None: self.pos = pos self.max = max self.parser = parser def __str__(self) -> str: return str((self.pos, self.max)) def __repr__(self) -> str: return "State(%r, %r)" % (self.pos, self.max) class NoParseError(Exception): def __init__(self, msg: str, state: State) -> None: self.msg = msg self.state = state def __str__(self) -> str: return self.msg class _Tuple(tuple): pass class _TupleParser(Parser[_A, _B], Generic[_A, _B]): @overload # type: ignore[override] def __add__(self, other: "_IgnoredParser[_A]") -> "_TupleParser[_A, _B]": pass @overload def __add__(self, other: Parser[_A, Any]) -> Parser[_A, Any]: pass def __add__( self, other: Union["_IgnoredParser[_A]", Parser[_A, Any]] ) -> Union["_TupleParser[_A, _B]", Parser[_A, Any]]: return super().__add__(other) class _Ignored: def __init__(self, value: Any) -> None: self.value = value def __repr__(self) -> str: return "_Ignored(%s)" % repr(self.value) def __eq__(self, other: object) -> bool: return isinstance(other, _Ignored) and self.value == other.value @Parser def finished(tokens: Sequence[Any], s: State) -> Tuple[None, State]: """A parser that throws an exception if there are any unparsed tokens left in the sequence.""" if s.pos >= len(tokens): return None, s else: s2 = State(s.pos, s.max, finished if s.pos == s.max else s.parser) raise NoParseError("got unexpected token", s2) finished.name = "end of input" def many(p: Parser[_A, _B]) -> Parser[_A, List[_B]]: """Return a parser that applies the parser `p` as many times as it succeeds at parsing the tokens. Return a parser that infinitely applies the parser `p` to the input sequence of tokens as long as it successfully parses them. The parsed value is a list of the sequentially parsed values. Examples: ```pycon >>> expr = many(a("x")) >>> expr.parse("x") ['x'] >>> expr.parse("xx") ['x', 'x'] >>> expr.parse("xxxy") # noqa ['x', 'x', 'x'] >>> expr.parse("y") [] ``` """ @Parser def _many(tokens: Sequence[_A], s: State) -> Tuple[List[_B], State]: res = [] try: while True: (v, s) = p.run(tokens, s) res.append(v) except NoParseError as e: s2 = State(s.pos, e.state.max, e.state.parser) if debug: log.debug( "*matched* %d instances of %s, new state = %s" % (len(res), _many.name, s2) ) return res, s2 _many.name = "{ %s }" % p.name return _many def some(pred: Callable[[_A], bool]) -> Parser[_A, _A]: """Return a parser that parses a token if it satisfies the predicate `pred`. Type: `(Callable[[A], bool]) -> Parser[A, A]` Examples: ```pycon >>> expr = some(lambda s: s.isalpha()).named('alpha') >>> expr.parse("x") 'x' >>> expr.parse("y") 'y' >>> expr.parse("1") Traceback (most recent call last): ... parser.NoParseError: got unexpected token: '1', expected: alpha ``` !!! Warning The `some()` combinator is quite slow and may be changed or removed in future versions. If you need a parser for a token by its type (e.g. any identifier) and maybe its value, use `tok(type[, value])` instead. You should use `make_tokenizer()` from `funcparserlib.lexer` to tokenize your text first. """ @Parser def _some(tokens: Sequence[_A], s: State) -> Tuple[_A, State]: if s.pos >= len(tokens): s2 = State(s.pos, s.max, _some if s.pos == s.max else s.parser) raise NoParseError("got unexpected end of input", s2) else: t = tokens[s.pos] if pred(t): pos = s.pos + 1 s2 = State(pos, max(pos, s.max), s.parser) if debug: log.debug("*matched* %r, new state = %s" % (t, s2)) return t, s2 else: s2 = State(s.pos, s.max, _some if s.pos == s.max else s.parser) if debug and isinstance(s2.parser, Parser): log.debug( "failed %r, state = %s, expected = %s" % (t, s2, s2.parser.name) ) raise NoParseError("got unexpected token", s2) _some.name = "some(...)" return _some def a(value: _A) -> Parser[_A, _A]: """Return a parser that parses a token if it's equal to `value`. Type: `(A) -> Parser[A, A]` Examples: ```pycon >>> expr = a("x") >>> expr.parse("x") 'x' >>> expr.parse("y") Traceback (most recent call last): ... parser.NoParseError: got unexpected token: 'y', expected: 'x' ``` !!! Note Although `Parser.parse()` can parse sequences of any objects (including `str` which is a sequence of `str` chars), **the recommended way** is parsing sequences of `Token` objects. You **should** use a regexp-based tokenizer `make_tokenizer()` defined in `funcparserlib.lexer` to convert your text into a sequence of `Token` objects before parsing it. You will get more readable parsing error messages (as `Token` objects contain their position in the source file) and good separation of the lexical and syntactic levels of the grammar. """ name = getattr(value, "name", value) def eq_value(t: _A) -> bool: return t == value return some(eq_value).named(repr(name)) def tok(type: str, value: Optional[str] = None) -> Parser[Token, str]: """Return a parser that parses a `Token` and returns the string value of the token. Type: `(str, Optional[str]) -> Parser[Token, str]` You can match any token of the specified `type` or you can match a specific token by its `type` and `value`. Examples: ```pycon >>> expr = tok("expr") >>> expr.parse([Token("expr", "foo")]) 'foo' >>> expr.parse([Token("expr", "bar")]) 'bar' >>> expr.parse([Token("op", "=")]) Traceback (most recent call last): ... parser.NoParseError: got unexpected token: '=', expected: expr ``` ```pycon >>> expr = tok("op", "=") >>> expr.parse([Token("op", "=")]) '=' >>> expr.parse([Token("op", "+")]) Traceback (most recent call last): ... parser.NoParseError: got unexpected token: '+', expected: '=' ``` !!! Note In order to convert your text to parse into a sequence of `Token` objects, use a regexp-based tokenizer `make_tokenizer()` defined in `funcparserlib.lexer`. You will get more readable parsing error messages (as `Token` objects contain their position in the source file) and good separation of the lexical and syntactic levels of the grammar. """ def eq_type(t: Token) -> bool: return t.type == type if value is not None: p = a(Token(type, value)) else: p = some(eq_type).named(type) return (p >> (lambda t: t.value)).named(p.name) def pure(x: _A) -> Parser[Any, _A]: """Wrap any object into a parser. Type: `(A) -> Parser[A, A]` A pure parser doesn't touch the tokens sequence, it just returns its pure `x` value. Also known as `return` in Haskell. """ @Parser def _pure(_: Sequence[Any], s: State) -> Tuple[_A, State]: return x, s _pure.name = "(pure %r)" % (x,) return _pure def maybe(p: Parser[_A, _B]) -> Parser[_A, Optional[_B]]: """Return a parser that returns `None` if the parser `p` fails. Examples: ```pycon >>> expr = maybe(a("x")) >>> expr.parse("x") 'x' >>> expr.parse("y") is None True ``` """ return (p | pure(None)).named("[ %s ]" % (p.name,)) def skip(p: Parser[_A, Any]) -> "_IgnoredParser[_A]": """An alias for `-p`. See also the docs for `Parser.__neg__()`. """ return -p class _IgnoredParser(Parser[_A, Any]): def __init__( self, p: Union[ Parser[_A, Any], Callable[[Sequence[_A], "State"], Tuple[Any, "State"]], ], ) -> None: super(_IgnoredParser, self).__init__(p) run = self._run if debug else self.run def ignored(tokens: Sequence[_A], s: State) -> Tuple[Any, State]: v, s2 = run(tokens, s) return v if isinstance(v, _Ignored) else _Ignored(v), s2 self.define(ignored) name = getattr(p, "name", p.__doc__) if name is not None: self.name = name @overload # type: ignore[override] def __add__(self, other: "_IgnoredParser[_A]") -> "_IgnoredParser[_A]": pass @overload def __add__(self, other: Parser[_A, _C]) -> Parser[_A, _C]: pass def __add__( self, other: Union["_IgnoredParser[_A]", Parser[_A, _C]] ) -> Union["_IgnoredParser[_A]", Parser[_A, _C]]: if isinstance(other, _IgnoredParser): @_IgnoredParser def ip(tokens: Sequence[_A], s: State) -> Tuple[Any, State]: _, s2 = self.run(tokens, s) v, s3 = other.run(tokens, s2) return v, s3 ip.name = "(%s, %s)" % (self.name, other.name) return ip else: @Parser def p(tokens: Sequence[_A], s: State) -> Tuple[_C, State]: _, s2 = self.run(tokens, s) v, s3 = other.run(tokens, s2) return v, s3 p.name = "(%s, %s)" % (self.name, other.name) return p def oneplus(p: Parser[_A, _B]) -> Parser[_A, List[_B]]: """Return a parser that applies the parser `p` one or more times. A similar parser combinator `many(p)` means apply `p` zero or more times, whereas `oneplus(p)` means apply `p` one or more times. Examples: ```pycon >>> expr = oneplus(a("x")) >>> expr.parse("x") ['x'] >>> expr.parse("xx") ['x', 'x'] >>> expr.parse("y") Traceback (most recent call last): ... parser.NoParseError: got unexpected token: 'y', expected: 'x' ``` """ @Parser def _oneplus(tokens: Sequence[_A], s: State) -> Tuple[List[_B], State]: (v1, s2) = p.run(tokens, s) (v2, s3) = many(p).run(tokens, s2) return [v1] + v2, s3 _oneplus.name = "(%s, { %s })" % (p.name, p.name) return _oneplus def with_forward_decls(suspension: Callable[[], Parser[_A, _B]]) -> Parser[_A, _B]: warnings.warn( "Use forward_decl() instead:\n" "\n" " p = forward_decl()\n" " ...\n" " p.define(parser_value)\n", DeprecationWarning, ) @Parser def f(tokens: Sequence[_A], s: State) -> Tuple[_B, State]: return suspension().run(tokens, s) return f def forward_decl() -> Parser[Any, Any]: """Return an undefined parser that can be used as a forward declaration. Type: `Parser[Any, Any]` Use `p = forward_decl()` in combination with `p.define(...)` to define recursive parsers. Examples: ```pycon >>> expr = forward_decl() >>> expr.define(a("x") + maybe(expr) + a("y")) >>> expr.parse("xxyy") # noqa ('x', ('x', None, 'y'), 'y') >>> expr.parse("xxy") Traceback (most recent call last): ... parser.NoParseError: got unexpected end of input, expected: 'y' ``` !!! Note If you care about static types, you should add a type hint for your forward declaration, so that your type checker can check types in `p.define(...)` later: ```python p: Parser[str, int] = forward_decl() p.define(a("x")) # Type checker error p.define(a("1") >> int) # OK ``` """ @Parser def f(_tokens: Any, _s: Any) -> Any: raise NotImplementedError("you must define() a forward_decl somewhere") f.name = "forward_decl()" return f if __name__ == "__main__": import doctest doctest.testmod() ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1700382465.8971386 funcparserlib-2.0.0a0/funcparserlib/py.typed0000644000000000000000000000000014526343402016046 0ustar00././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1700382465.8971386 funcparserlib-2.0.0a0/funcparserlib/util.py0000644000000000000000000000530014526343402015706 0ustar00# Copyright © 2009/2023 Andrey Vlasovskikh # # Permission is hereby granted, free of charge, to any person obtaining a copy of this # software and associated documentation files (the "Software"), to deal in the Software # without restriction, including without limitation the rights to use, copy, modify, # merge, publish, distribute, sublicense, and/or sell copies of the Software, and to # permit persons to whom the Software is furnished to do so, subject to the following # conditions: # # The above copyright notice and this permission notice shall be included in all copies # or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A # PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF # CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE # OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. from typing import TypeVar, Callable, Sequence _A = TypeVar("_A") def pretty_tree( x: _A, kids: Callable[[_A], Sequence[_A]], show: Callable[[_A], str], ) -> str: """Return a pseudo-graphic tree representation of the object `x` similar to the `tree` command in Unix. Type: `(T, Callable[[T], List[T]], Callable[[T], str]) -> str` It applies the parameter `show` (which is a function of type `(T) -> str`) to get a textual representation of the objects to show. It applies the parameter `kids` (which is a function of type `(T) -> List[T]`) to list the children of the object to show. Examples: ```pycon >>> print(pretty_tree( ... ["foo", ["bar", "baz"], "quux"], ... lambda obj: obj if isinstance(obj, list) else [], ... lambda obj: "[]" if isinstance(obj, list) else str(obj), ... )) [] |-- foo |-- [] | |-- bar | `-- baz `-- quux ``` """ (MID, END, CONT, LAST, ROOT) = ("|-- ", "`-- ", "| ", " ", "") def rec(obj: _A, indent: str, sym: str) -> str: line = indent + sym + show(obj) obj_kids = kids(obj) if len(obj_kids) == 0: return line else: if sym == MID: next_indent = indent + CONT elif sym == ROOT: next_indent = indent + ROOT else: next_indent = indent + LAST chars = [MID] * (len(obj_kids) - 1) + [END] lines = [rec(kid, next_indent, sym) for kid, sym in zip(obj_kids, chars)] return "\n".join([line] + lines) return rec(x, "", ROOT) ././@PaxHeader0000000000000000000000000000003400000000000010212 xustar0028 mtime=1700382465.8971386 funcparserlib-2.0.0a0/pyproject.toml0000644000000000000000000000214314526343402014436 0ustar00[tool.poetry] name = "funcparserlib" version = "2.0.0a0" description = "Recursive descent parsing library based on functional combinators" authors = ["Andrey Vlasovskikh "] license = "MIT" readme = "README.md" homepage = "https://funcparserlib.pirx.ru" repository = "https://github.com/vlasovskikh/funcparserlib" classifiers = [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", ] [tool.poetry.dependencies] python = "^3.8" [tool.poetry.dev-dependencies] pre-commit = {version = "^3.5.0"} tox = {version = "^4.4.6"} mkdocs = {version = "^1.4.2"} mkdocs-material = {version = "^9.1.1"} mkdocstrings = {extras = ["python"], version = "^0.24.0"} [build-system] requires = ["poetry-core>=1.5.1"] build-backend = "poetry.core.masonry.api" funcparserlib-2.0.0a0/PKG-INFO0000644000000000000000000001476100000000000012566 0ustar00Metadata-Version: 2.1 Name: funcparserlib Version: 2.0.0a0 Summary: Recursive descent parsing library based on functional combinators Home-page: https://funcparserlib.pirx.ru License: MIT Author: Andrey Vlasovskikh Author-email: andrey.vlasovskikh@gmail.com Requires-Python: >=3.8,<4.0 Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: MIT License Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.8 Classifier: Programming Language :: Python :: 3.9 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Project-URL: Repository, https://github.com/vlasovskikh/funcparserlib Description-Content-Type: text/markdown Funcparserlib ============= Recursive descent parsing library for Python based on functional combinators. [![PyPI](https://img.shields.io/pypi/v/funcparserlib)](https://pypi.org/project/funcparserlib/) [![PyPI - Downloads](https://img.shields.io/pypi/dm/funcparserlib)](https://pypi.org/project/funcparserlib/) Description ----------- The primary focus of `funcparserlib` is **parsing little languages** or **external DSLs** (domain specific languages). Parsers made with `funcparserlib` are pure-Python LL(\*) parsers. It means that it's **very easy to write parsers** without thinking about lookaheads and other hardcore parsing stuff. However, recursive descent parsing is a rather slow method compared to LL(k) or LR(k) algorithms. Still, parsing with `funcparserlib` is **at least twice faster than PyParsing**, a very popular library for Python. The source code of `funcparserlib` is only 1.2K lines of code, with lots of comments. Its API is fully type hinted. It features the longest parsed prefix error reporting, as well as a tiny lexer generator for token position tracking. The idea of parser combinators used in `funcparserlib` comes from the [Introduction to Functional Programming](https://www.cl.cam.ac.uk/teaching/Lectures/funprog-jrh-1996/) course. We have converted it from ML into Python. Installation ------------ You can install `funcparserlib` from [PyPI](https://pypi.org/project/funcparserlib/): ```shell $ pip install funcparserlib ``` There are no dependencies on other libraries. Documentation ------------- * [Getting Started](https://funcparserlib.pirx.ru/getting-started/) * Your **starting point** with `funcparserlib` * [API Reference](https://funcparserlib.pirx.ru/api/) * Learn the details of the API There are several examples available in the `tests/` directory: * [GraphViz DOT parser](https://github.com/vlasovskikh/funcparserlib/blob/master/tests/dot.py) * [JSON parser](https://github.com/vlasovskikh/funcparserlib/blob/master/tests/json.py) See also [the changelog](https://funcparserlib.pirx.ru/changes/). Example ------- Let's consider a little language of **numeric expressions** with a syntax similar to Python expressions. Here are some expression strings in this language: ``` 0 1 + 2 + 3 -1 + 2 ** 32 3.1415926 * (2 + 7.18281828e-1) * 42 ``` Here is **the complete source code** of the tokenizer and the parser for this language written using `funcparserlib`: ```python from typing import List, Tuple, Union from dataclasses import dataclass from funcparserlib.lexer import make_tokenizer, TokenSpec, Token from funcparserlib.parser import tok, Parser, many, forward_decl, finished @dataclass class BinaryExpr: op: str left: "Expr" right: "Expr" Expr = Union[BinaryExpr, int, float] def tokenize(s: str) -> List[Token]: specs = [ TokenSpec("whitespace", r"\s+"), TokenSpec("float", r"[+\-]?\d+\.\d*([Ee][+\-]?\d+)*"), TokenSpec("int", r"[+\-]?\d+"), TokenSpec("op", r"(\*\*)|[+\-*/()]"), ] tokenizer = make_tokenizer(specs) return [t for t in tokenizer(s) if t.type != "whitespace"] def parse(tokens: List[Token]) -> Expr: int_num = tok("int") >> int float_num = tok("float") >> float number = int_num | float_num expr: Parser[Token, Expr] = forward_decl() parenthesized = -op("(") + expr + -op(")") primary = number | parenthesized power = primary + many(op("**") + primary) >> to_expr term = power + many((op("*") | op("/")) + power) >> to_expr sum = term + many((op("+") | op("-")) + term) >> to_expr expr.define(sum) document = expr + -finished return document.parse(tokens) def op(name: str) -> Parser[Token, str]: return tok("op", name) def to_expr(args: Tuple[Expr, List[Tuple[str, Expr]]]) -> Expr: first, rest = args result = first for op, expr in rest: result = BinaryExpr(op, result, expr) return result ``` Now, consider this numeric expression: `3.1415926 * (2 + 7.18281828e-1) * 42`. Let's `tokenize()` it using the tokenizer we've created with `funcparserlib.lexer`: ``` [ Token('float', '3.1415926'), Token('op', '*'), Token('op', '('), Token('int', '2'), Token('op', '+'), Token('float', '7.18281828e-1'), Token('op', ')'), Token('op', '*'), Token('int', '42'), ] ``` Let's `parse()` these tokens into an expression tree using our parser created with `funcparserlib.parser`: ``` BinaryExpr( op='*', left=BinaryExpr( op='*', left=3.1415926, right=BinaryExpr(op='+', left=2, right=0.718281828), ), right=42, ) ``` Learn how to write this parser using `funcparserlib` in the [Getting Started](https://funcparserlib.pirx.ru/getting-started/) guide! Used By ------- Some open-source projects that use `funcparserlib` as an explicit dependency: * [Hy](https://github.com/hylang/hy), a Lisp dialect that's embedded in Python * 4.7K stars, version `~=1.0`, Python 3.8+ * [Splash](https://github.com/scrapinghub/splash), a JavaScript rendering service with HTTP API, by Scrapinghub * 3.9K stars, version `*`. Python 3 in Docker * [graphite-beacon](https://github.com/klen/graphite-beacon), a simple alerting system for Graphite metrics * 453 stars, version `==0.3.6`, Python 2 and 3 * [blockdiag](https://github.com/blockdiag/blockdiag), generates block-diagram image file from spec-text file * 194 stars, version `>= 1.0.0a0`, Python 3.7+ * [kll](https://github.com/kiibohd/kll), Keyboard Layout Language (KLL) compiler * 113 stars, copied source code, Python 3.5+ Next ---- Read the [Getting Started](https://funcparserlib.pirx.ru/getting-started/) guide to start learning `funcparserlib`.