Some random product page
$ 100.12
''' l = ItemLoader(selector=Selector(html_data)) l.add_xpath('name', '//div[@class="product_name"]/text()') l.add_xpath('name', '//div[@class="product_title"]/text()') l.add_css('price', '#price::text') l.add_value('last_updated', 'today') # you can also use literal values item = l.load_item() item # {'name': ['Some random product page'], 'price': ['$ 100.12'], 'last_updated': ['today']} By quickly looking at that code, we can see the ``name`` field is being extracted from two different XPath locations in the page: 1. ``//div[@class="product_name"]`` 2. ``//div[@class="product_title"]`` In other words, data is being collected by extracting it from two XPath locations, using the :meth:`~ItemLoader.add_xpath` method. This is the data that will be assigned to the ``name`` field later. Instead of one call per location, you can pass an iterable of XPath expressions:: l.add_xpath('name', ['//div[@class="product_name"]/text()', '//div[@class="product_title"]/text()']) Afterwards, similar calls are used for ``price`` field using a CSS selector with the :meth:`~ItemLoader.add_css` method, and finally the ``last_update`` field is populated directly with a literal value (``today``) using a different method: :meth:`~ItemLoader.add_value`. Finally, when all data is collected, the :meth:`ItemLoader.load_item` method is called which actually returns the item populated with the data previously extracted and collected with the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css`, and :meth:`~ItemLoader.add_value` calls. Contents -------- .. toctree:: declaring-loaders processors loaders-context nested-loaders extending-loaders rule-usage built-in-processors api-reference release-notes scrapy-itemloaders-9303326/docs/loaders-context.rst 0000664 0000000 0000000 00000003336 15256713457 0022335 0 ustar 00root root 0000000 0000000 .. currentmodule:: itemloaders .. _loaders-context: Item Loader Context =================== The Item Loader Context is a mechanism that allows to change the input/ouput processors behavior. It's just a ``dict`` of arbitrary key/values which is shared among all processors. By default, the context contains the ``selector`` and any other `keyword arguments` sent to the Loader's ``__init__``. The context can be passed when declaring, instantiating or using Item Loader. For example, suppose you have a function ``parse_length`` which receives a text value and extracts a length from it:: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a ``loader_context`` argument the function is explicitly telling the Item Loader that it's able to receive an Item Loader context, so the Item Loader passes the currently active context when calling it, and the processor function (``parse_length`` in this case) can thus use them. There are several ways to modify Item Loader context values: 1. By modifying the currently active Item Loader context (:attr:`~ItemLoader.context` attribute):: loader = ItemLoader(product) loader.context['unit'] = 'cm' 2. On Item Loader instantiation (the keyword arguments of Item Loader ``__init__`` method are stored in the Item Loader context):: loader = ItemLoader(product, unit='cm') 3. On Item Loader declaration, for those input/output processors that support instantiating them with an Item Loader context. :class:`~processors.MapCompose` is one of them:: class ProductLoader(ItemLoader): length_out = MapCompose(parse_length, unit='cm') scrapy-itemloaders-9303326/docs/nested-loaders.rst 0000664 0000000 0000000 00000003320 15256713457 0022124 0 ustar 00root root 0000000 0000000 .. _nested-loaders: Nested Loaders ============== When parsing related values from a subsection of a document, it can be useful to create nested loaders. Imagine you're extracting details from a footer of a page that looks something like: Example:: Without nested loaders, you need to specify the full xpath (or css) for each value that you wish to extract. Example:: loader = ItemLoader() # load stuff not in the footer loader.add_xpath('social', '//footer/a[@class = "social"]/@href') loader.add_xpath('email', '//footer/a[@class = "email"]/@href') loader.load_item() Instead, you can create a nested loader with the footer selector and add values relative to the footer. The functionality is the same but you avoid repeating the footer selector. Example:: loader = ItemLoader() # load stuff not in the footer footer_loader = loader.nested_xpath('//footer') footer_loader.add_xpath('social', 'a[@class = "social"]/@href') footer_loader.add_xpath('email', 'a[@class = "email"]/@href') # no need to call footer_loader.load_item() loader.load_item() You can nest loaders arbitrarily and they work with either xpath or css selectors. As a general guideline, use nested loaders when they make your code simpler but do not go overboard with nesting or your parser can become difficult to read. .. versionchanged:: VERSION Nested loaders inherit the parent's processor context, with explicit keyword arguments taking precedence. scrapy-itemloaders-9303326/docs/processors.rst 0000664 0000000 0000000 00000010420 15256713457 0021414 0 ustar 00root root 0000000 0000000 .. currentmodule:: itemloaders .. _processors: Input and Output processors =========================== An Item Loader contains one input processor and one output processor for each (item) field. The input processor processes the extracted data as soon as it's received (through the :meth:`~ItemLoader.add_xpath`, :meth:`~ItemLoader.add_css`, :meth:`~ItemLoader.add_jmes` or :meth:`~ItemLoader.add_value` methods) and the result of the input processor is collected and kept inside the ItemLoader. After collecting all data, the :meth:`ItemLoader.load_item` method is called to populate and get the populated item object. That's when the output processor is called with the data previously collected (and processed using the input processor). The result of the output processor is the final value that gets assigned to the item. Let's see an example to illustrate how the input and output processors are called for a particular field (the same applies for any other field):: l = ItemLoader(selector=some_selector) l.add_xpath('name', xpath1) # (1) l.add_xpath('name', xpath2) # (2) l.add_css('name', css) # (3) l.add_value('name', 'test') # (4) return l.load_item() # (5) So what happens is: 1. Data from ``xpath1`` is extracted, and passed through the *input processor* of the ``name`` field. The result of the input processor is collected and kept in the Item Loader (but not yet assigned to the item). 2. Data from ``xpath2`` is extracted, and passed through the same *input processor* used in (1). The result of the input processor is appended to the data collected in (1) (if any). 3. This case is similar to the previous ones, except that the data is extracted from the ``css`` CSS selector, and passed through the same *input processor* used in (1) and (2). The result of the input processor is appended to the data collected in (1) and (2) (if any). 4. This case is also similar to the previous ones, except that the value to be collected is assigned directly, instead of being extracted from a XPath expression or a CSS selector. However, the value is still passed through the input processors. In this case, since the value is not iterable it is converted to an iterable of a single element before passing it to the input processor, because input processor always receive iterables. 5. The data collected in steps (1), (2), (3) and (4) is passed through the *output processor* of the ``name`` field. The result of the output processor is the value assigned to the ``name`` field in the item. It's worth noticing that processors are just callable objects, which are called with the data to be parsed, and return a parsed value. So you can use any function as input or output processor. The only requirement is that they must accept one (and only one) positional argument, which will be an iterable. .. note:: Both input and output processors must receive an iterable as their first argument. The output of those functions can be anything. The result of input processors will be appended to an internal list (in the Loader) containing the collected values (for that field). The result of the output processors is the value that will be finally assigned to the item. The other thing you need to keep in mind is that the values returned by input processors are collected internally (in lists) and then passed to output processors to populate the fields. Last, but not least, ``itemloaders`` comes with some :ref:`commonly used processorsColor TV
loader.add_xpath('name', '//p[@class="product-name"]') # HTML snippet:the price is $1200
loader.add_xpath('price', '//p[@id="price"]', re='the price is (.*)') """ values = self._get_xpathvalues(xpath, field_name, **kw) return self.add_value(field_name, values, *processors, re=re, **kw) def replace_xpath( self, field_name: str | None, xpath: str | Iterable[str], *processors: Callable[..., Any], re: str | Pattern[str] | None = None, **kw: Any, ) -> Self: """ Similar to :meth:`add_xpath` but replaces collected data instead of adding it. :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader """ values = self._get_xpathvalues(xpath, field_name, **kw) return self.replace_value(field_name, values, *processors, re=re, **kw) def get_xpath( self, xpath: str | Iterable[str], *processors: Callable[..., Any], re: str | Pattern[str] | None = None, **kw: Any, ) -> Any: """ Similar to :meth:`ItemLoader.get_value` but receives an XPath instead of a value, which is used to extract a list of strings from the selector associated with this :class:`ItemLoader`. *xpath* may also be an iterable of XPath expressions, in which case the strings extracted by all of them are collected. *re* is a regular expression, as a string or a compiled pattern, used to further extract data from the selected XPath region. Examples:: # HTML snippet:Color TV
loader.get_xpath('//p[@class="product-name"]') # HTML snippet:the price is $1200
loader.get_xpath('//p[@id="price"]', TakeFirst(), re='the price is (.*)') """ values = self._get_xpathvalues(xpath, **kw) return self.get_value(values, *processors, re=re, **kw) def _get_xpathvalues( self, xpaths: str | Iterable[str], field_name: str | None = None, **kw: Any, ) -> list[Any]: self._check_selector_method() assert self.selector is not None xpaths = arg_to_iter(xpaths) return flatten( self._track_rule( field_name, "xpath", xpath, self.selector.xpath(xpath, **kw).getall() ) for xpath in xpaths ) def add_css( self, field_name: str | None, css: str | Iterable[str], *processors: Callable[..., Any], re: str | Pattern[str] | None = None, **kw: Any, ) -> Self: """ Similar to :meth:`ItemLoader.add_value` but receives a CSS selector instead of a value, which is used to extract a list of strings from the selector associated with this :class:`ItemLoader`. *css* may also be an iterable of CSS selectors, in which case the strings extracted by all of them are collected. See :meth:`get_css` for ``kwargs``. :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader Examples:: # HTML snippet:Color TV
loader.add_css('name', 'p.product-name') # HTML snippet:the price is $1200
loader.add_css('price', 'p#price', re='the price is (.*)') """ values = self._get_cssvalues(css, field_name) return self.add_value(field_name, values, *processors, re=re, **kw) def replace_css( self, field_name: str | None, css: str | Iterable[str], *processors: Callable[..., Any], re: str | Pattern[str] | None = None, **kw: Any, ) -> Self: """ Similar to :meth:`add_css` but replaces collected data instead of adding it. :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader """ values = self._get_cssvalues(css, field_name) return self.replace_value(field_name, values, *processors, re=re, **kw) def get_css( self, css: str | Iterable[str], *processors: Callable[..., Any], re: str | Pattern[str] | None = None, **kw: Any, ) -> Any: """ Similar to :meth:`ItemLoader.get_value` but receives a CSS selector instead of a value, which is used to extract a list of strings from the selector associated with this :class:`ItemLoader`. *css* may also be an iterable of CSS selectors, in which case the strings extracted by all of them are collected. *re* is a regular expression, as a string or a compiled pattern, used to further extract data from the selected CSS region. Examples:: # HTML snippet:Color TV
loader.get_css('p.product-name') # HTML snippet:the price is $1200
loader.get_css('p#price', TakeFirst(), re='the price is (.*)') """ values = self._get_cssvalues(css) return self.get_value(values, *processors, re=re, **kw) def _get_cssvalues( self, csss: str | Iterable[str], field_name: str | None = None ) -> list[Any]: self._check_selector_method() assert self.selector is not None csss = arg_to_iter(csss) return flatten( self._track_rule(field_name, "css", css, self.selector.css(css).getall()) for css in csss ) def add_jmes( self, field_name: str | None, jmes: str | Iterable[str], *processors: Callable[..., Any], re: str | Pattern[str] | None = None, **kw: Any, ) -> Self: """ Similar to :meth:`ItemLoader.add_value` but receives a JMESPath selector instead of a value, which is used to extract a list of strings from the selector associated with this :class:`ItemLoader`. *jmes* may also be an iterable of JMESPath selectors, in which case the strings extracted by all of them are collected. See :meth:`get_jmes` for ``kwargs``. :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader Examples:: # JSON snippet: {"name": "Color TV"} loader.add_jmes('name', 'name') # JSON snippet: {"price": "the price is $1200"} loader.add_jmes('price', 'price', TakeFirst(), re='the price is (.*)') """ values = self._get_jmesvalues(jmes, field_name) return self.add_value(field_name, values, *processors, re=re, **kw) def replace_jmes( self, field_name: str | None, jmes: str | Iterable[str], *processors: Callable[..., Any], re: str | Pattern[str] | None = None, **kw: Any, ) -> Self: """ Similar to :meth:`add_jmes` but replaces collected data instead of adding it. :returns: The current ItemLoader instance for method chaining. :rtype: ItemLoader """ values = self._get_jmesvalues(jmes, field_name) return self.replace_value(field_name, values, *processors, re=re, **kw) def get_jmes( self, jmes: str | Iterable[str], *processors: Callable[..., Any], re: str | Pattern[str] | None = None, **kw: Any, ) -> Any: """ Similar to :meth:`ItemLoader.get_value` but receives a JMESPath selector instead of a value, which is used to extract a list of strings from the selector associated with this :class:`ItemLoader`. *jmes* may also be an iterable of JMESPath selectors, in which case the strings extracted by all of them are collected. *re* is a regular expression, as a string or a compiled pattern, used to further extract data from the selected JMESPath. Examples:: # JSON snippet: {"name": "Color TV"} loader.get_jmes('name') # JSON snippet: {"price": "the price is $1200"} loader.get_jmes('price', TakeFirst(), re='the price is (.*)') """ values = self._get_jmesvalues(jmes) return self.get_value(values, *processors, re=re, **kw) def _get_jmesvalues( self, jmess: str | Iterable[str], field_name: str | None = None ) -> list[Any]: self._check_selector_method() assert self.selector is not None jmess = arg_to_iter(jmess) if not hasattr(self.selector, "jmespath"): raise AttributeError( "Please install parsel >= 1.8.1 to get JMESPath support" ) return flatten( self._track_rule( field_name, "jmes", jmes, self.selector.jmespath(jmes).getall() ) for jmes in jmess ) scrapy-itemloaders-9303326/itemloaders/common.py 0000664 0000000 0000000 00000001755 15256713457 0021715 0 ustar 00root root 0000000 0000000 """Common functions used in Item Loaders code""" from __future__ import annotations from functools import lru_cache, partial from typing import TYPE_CHECKING, Any from itemloaders.utils import get_func_args if TYPE_CHECKING: from collections.abc import Callable, MutableMapping @lru_cache(maxsize=1024) def _takes_loader_context(function: Callable[..., Any]) -> bool: return "loader_context" in get_func_args(function) def wrap_loader_context( function: Callable[..., Any], context: MutableMapping[str, Any] ) -> Callable[..., Any]: """Wrap functions that receive loader_context to contain the context "pre-loaded" and expose an interface that receives only one argument """ try: takes_loader_context = _takes_loader_context(function) except TypeError: # unhashable function takes_loader_context = "loader_context" in get_func_args(function) if takes_loader_context: return partial(function, loader_context=context) return function scrapy-itemloaders-9303326/itemloaders/processors.py 0000664 0000000 0000000 00000021641 15256713457 0022623 0 ustar 00root root 0000000 0000000 """ This module provides some commonly used processors for Item Loaders. See documentation in :ref:`declaring-loaders`. """ from __future__ import annotations from collections import ChainMap from typing import TYPE_CHECKING, Any from itemloaders.common import wrap_loader_context from itemloaders.utils import arg_to_iter if TYPE_CHECKING: from collections.abc import Callable, Iterable, MutableMapping class MapCompose: """ A processor which is constructed from the composition of the given functions, similar to the :class:`Compose` processor. The difference with this processor is the way internal results are passed among functions, which is as follows: The input value of this processor is *iterated* and the first function is applied to each element. The results of these function calls (one for each element) are concatenated to construct a new iterable, which is then used to apply the second function, and so on, until the last function is applied to each value of the list of values collected so far. The output values of the last function are concatenated together to produce the output of this processor. Each particular function can return a value or a list of values, which is flattened with the list of values returned by the same function applied to the other input values. The functions can also return ``None`` in which case the output of that function is ignored for further processing over the chain. This processor provides a convenient way to compose functions that only work with single values (instead of iterables). For this reason the :class:`MapCompose` processor is typically used as input processor, since data is often extracted using the :meth:`~parsel.selector.Selector.extract` method of `parsel selectors`_, which returns a list of strings. The example below should clarify how it works: >>> def filter_world(x): ... return None if x == 'world' else x ... >>> from itemloaders.processors import MapCompose >>> proc = MapCompose(filter_world, str.upper) >>> proc(['hello', 'world', 'this', 'is', 'something']) ['HELLO', 'THIS', 'IS', 'SOMETHING'] As with the Compose processor, functions can receive Loader contexts, and ``__init__`` method keyword arguments are used as default context values. See :class:`Compose` processor for more info. .. _`parsel selectors`: https://parsel.readthedocs.io/en/latest/parsel.html#parsel.selector.Selector.extract """ def __init__(self, *functions: Callable[..., Any], **default_loader_context: Any): self.functions = functions self.default_loader_context = default_loader_context def __call__( self, value: Any, loader_context: MutableMapping[str, Any] | None = None ) -> Iterable[Any]: values = arg_to_iter(value) context: MutableMapping[str, Any] if loader_context: context = ChainMap(loader_context, self.default_loader_context) else: context = self.default_loader_context wrapped_funcs = [wrap_loader_context(f, context) for f in self.functions] for func in wrapped_funcs: next_values: list[Any] = [] for v in values: try: next_values += arg_to_iter(func(v)) except Exception as e: raise ValueError( f"Error in MapCompose with " f"{func!s} value={value!r} error='{type(e).__name__}: {e!s}'" ) from e values = next_values return values class Compose: """ A processor which is constructed from the composition of the given functions. This means that each input value of this processor is passed to the first function, and the result of that function is passed to the second function, and so on, until the last function returns the output value of this processor. By default, stop process on ``None`` value. This behaviour can be changed by passing keyword argument ``stop_on_none=False``. Example: >>> from itemloaders.processors import Compose >>> proc = Compose(lambda v: v[0], str.upper) >>> proc(['hello', 'world']) 'HELLO' Each function can optionally receive a ``loader_context`` parameter. For those which do, this processor will pass the currently active :ref:`Loader context') >>> proc(['one', 'two', 'three']) 'one
two
three' """ def __init__(self, separator: str = " "): self.separator = separator def __call__(self, values: Any) -> str: return self.separator.join(values) scrapy-itemloaders-9303326/itemloaders/py.typed 0000664 0000000 0000000 00000000000 15256713457 0021530 0 ustar 00root root 0000000 0000000 scrapy-itemloaders-9303326/itemloaders/utils.py 0000664 0000000 0000000 00000006035 15256713457 0021561 0 ustar 00root root 0000000 0000000 """ Copy/paste from scrapy source at the moment, to ensure tests are working. Refactoring to come later """ from __future__ import annotations import inspect from abc import get_cache_token from collections.abc import Callable, Iterable, Iterator from collections.abc import Set as AbstractSet from functools import partial from typing import Any # Whether each class seen so far is one that arg_to_iter returns as is. Testing # a class against an abstract base class costs an order of magnitude more than # a dictionary lookup, so each class is only tested once. _iterable_classes: dict[type, bool] = {} _abc_cache_token = get_cache_token() # Reaching this many entries means classes are being created dynamically, and # caching a result per class would grow the cache without bound. _MAX_ITERABLE_CLASSES = 1000 def _is_iterable_class(cls: type) -> bool: """Tell whether :func:`arg_to_iter` should return instances of *cls* as is, caching the result for later calls. Only call this for classes that are not already cached as iterable. """ global _abc_cache_token # noqa: PLW0603 # pylint: disable=global-statement # Registering a class against an abstract base class turns a negative # result into a positive one, so only negative results can go stale, and # only until the next registration bumps the token. token = get_cache_token() if token != _abc_cache_token: _iterable_classes.clear() _abc_cache_token = token elif cls in _iterable_classes: return False if len(_iterable_classes) >= _MAX_ITERABLE_CLASSES: _iterable_classes.clear() result = issubclass(cls, (list, tuple, Iterator, AbstractSet)) _iterable_classes[cls] = result return result def arg_to_iter(arg: Any) -> Iterable[Any]: """Return an iterable based on *arg*. If *arg* is a :class:`list`, a :class:`tuple`, a :class:`~collections.abc.Set` or an :class:`~collections.abc.Iterator`, it will be returned as is. If *arg* is ``None``, an empty list will be returned. If *arg* is anything else, a list will be returned with *arg* as its only item, i.e. ``[arg]``. """ if arg is None: return [] cls = arg.__class__ if _iterable_classes.get(cls) or _is_iterable_class(cls): return arg # type: ignore[no-any-return] return [arg] def get_func_args(func: Callable[..., Any], stripself: bool = False) -> list[str]: """Return the argument name list of a callable object""" if not callable(func): raise TypeError(f"func must be callable, got {type(func).__name__!r}") args: list[str] = [] try: sig = inspect.signature(func) except ValueError: return args if isinstance(func, partial): partial_kw = func.keywords for name, param in sig.parameters.items(): if partial_kw and param.name in partial_kw: continue args.append(name) else: args = list(sig.parameters) if stripself and args and args[0] == "self": args = args[1:] return args scrapy-itemloaders-9303326/pyproject.toml 0000664 0000000 0000000 00000013114 15256713457 0020447 0 ustar 00root root 0000000 0000000 [build-system] requires = ["hatchling>=1.27.0"] build-backend = "hatchling.build" [project] name = "itemloaders" version = "1.5.0" description = "Base library for scrapy's ItemLoader" authors = [ { name = "Zyte", email = "opensource@zyte.com" }, ] readme = "README.rst" license = "BSD-3-Clause" license-files = ["LICENSE"] dependencies = [ # before updating these versions, be sure they are not higher than # scrapy's requirements "itemadapter>=0.1.0", "jmespath>=0.9.5", "parsel>=1.5.0", ] requires-python = ">=3.10" classifiers = [ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.11", "Programming Language :: Python :: 3.12", "Programming Language :: Python :: 3.13", "Programming Language :: Python :: 3.14", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Operating System :: OS Independent", "Framework :: Scrapy", "Topic :: Internet :: WWW/HTTP", "Topic :: Software Development :: Libraries :: Python Modules", ] [project.urls] Homepage = "https://github.com/scrapy/itemloaders" Documentation = "https://itemloaders.readthedocs.io/" Source = "https://github.com/scrapy/itemloaders" Tracker = "https://github.com/scrapy/itemloaders/issues" "Release notes" = "https://itemloaders.readthedocs.io/en/latest/release-notes.html" [tool.hatch.build.targets.sdist] include = [ "/docs", "/itemloaders", "/tests", "/tox.ini", ] [tool.bumpversion] commit = true tag = true [[tool.bumpversion.files]] filename = "docs/release-notes.rst" search = "\\(unreleased\\)$" replace = "({now:%Y-%m-%d})" regex = true [tool.coverage.run] branch = true [tool.mypy] strict = true [[tool.mypy.overrides]] # Allow test functions to be untyped module = "tests.*" allow_untyped_defs = true [tool.pylint.MASTER] persistent = "no" [tool.pylint."MESSAGES CONTROL"] enable = [ "useless-suppression", ] disable = [ "broad-exception-caught", "disallowed-name", "duplicate-code", "fixme", "import-error", "import-outside-toplevel", "invalid-name", "line-too-long", "missing-class-docstring", "missing-function-docstring", "missing-module-docstring", "not-callable", "protected-access", "redefined-builtin", "redefined-outer-name", "too-few-public-methods", "too-many-arguments", "too-many-lines", "too-many-positional-arguments", "too-many-public-methods", "unused-argument", "use-implicit-booleaness-not-comparison", ] [tool.ruff.lint] extend-select = [ # flake8-builtins "A", # flake8-async "ASYNC", # flake8-bugbear "B", # flake8-comprehensions "C4", # flake8-commas "COM", # pydocstyle "D", # flake8-future-annotations "FA", # flynt "FLY", # refurb "FURB", # isort "I", # flake8-implicit-str-concat "ISC", # flake8-logging "LOG", # Perflint "PERF", # pygrep-hooks "PGH", # flake8-pie "PIE", # pylint "PL", # flake8-pytest-style "PT", # flake8-use-pathlib "PTH", # flake8-pyi "PYI", # flake8-quotes "Q", # flake8-return "RET", # flake8-raise "RSE", # Ruff-specific rules "RUF", # flake8-bandit "S", # flake8-simplify "SIM", # flake8-slots "SLOT", # flake8-debugger "T10", # flake8-type-checking "TC", # pyupgrade "UP", # pycodestyle warnings "W", # flake8-2020 "YTT", ] ignore = [ # Do not catch blind exception "BLE001", # Trailing comma missing "COM812", # Missing docstring in public module "D100", # Missing docstring in public class "D101", # Missing docstring in public method "D102", # Missing docstring in public function "D103", # Missing docstring in public package "D104", # Missing docstring in magic method "D105", # Missing docstring in public nested class "D106", # Missing docstring in __init__ "D107", # One-line docstring should fit on one line with quotes "D200", # No blank lines allowed after function docstring "D202", # 1 blank line required between summary line and description "D205", # Multi-line docstring closing quotes should be on a separate line "D209", # First line should end with a period "D400", # First line should be in imperative mood; try rephrasing "D401", # First line should not be the function's "signature" "D402", # First word of the first line should be properly capitalized "D403", # No blank lines allowed between a section header and its content "D412", # `try`-`except` within a loop incurs performance overhead "PERF203", # Too many return statements "PLR0911", # Too many branches "PLR0912", # Too many arguments in function definition "PLR0913", # Too many statements "PLR0915", # Too many positional arguments "PLR0917", # Magic value used in comparison "PLR2004", # String contains ambiguous {}. "RUF001", # Docstring contains ambiguous {}. "RUF002", # Comment contains ambiguous {}. "RUF003", # Mutable class attributes should be annotated with `typing.ClassVar` "RUF012", # Use of `assert` detected "S101", ] [tool.ruff.lint.isort] split-on-trailing-comma = false [tool.ruff.lint.pydocstyle] convention = "pep257" scrapy-itemloaders-9303326/tests/ 0000775 0000000 0000000 00000000000 15256713457 0016675 5 ustar 00root root 0000000 0000000 scrapy-itemloaders-9303326/tests/__init__.py 0000664 0000000 0000000 00000000000 15256713457 0020774 0 ustar 00root root 0000000 0000000 scrapy-itemloaders-9303326/tests/benchmarks/ 0000775 0000000 0000000 00000000000 15256713457 0021012 5 ustar 00root root 0000000 0000000 scrapy-itemloaders-9303326/tests/benchmarks/__init__.py 0000664 0000000 0000000 00000000150 15256713457 0023117 0 ustar 00root root 0000000 0000000 import pytest pytest.importorskip("pytest_codspeed", reason="Benchmark tests require pytest-codspeed") scrapy-itemloaders-9303326/tests/benchmarks/test_benchmark_common.py 0000664 0000000 0000000 00000002626 15256713457 0025733 0 ustar 00root root 0000000 0000000 from __future__ import annotations from functools import partial from typing import TYPE_CHECKING, Any import pytest from itemloaders.common import wrap_loader_context if TYPE_CHECKING: from pytest_codspeed import BenchmarkFixture # Each round calls the function this many times, so that the measurement # reflects the function itself rather than the benchmark call overhead. CALLS_PER_ROUND = 100 CONTEXT: dict[str, Any] = {"encoding": "utf-8"} def _function(value: str, chars: str = " ", loader_context: Any = None) -> str: return value.strip(chars) class _Processor: def __call__(self, value: str, loader_context: Any = None) -> str: return value.strip() def method(self, value: str) -> str: return value.strip() class _UnhashableProcessor(_Processor): __hash__ = None # type: ignore[assignment] _processor = _Processor() CASES: dict[str, Any] = { "function": _function, "partial": partial(_function, chars="-"), "method": _processor.method, "processor": _processor, "unhashable_processor": _UnhashableProcessor(), "type": float, "method_descriptor": str.strip, } @pytest.mark.parametrize("case", CASES) def test_wrap_loader_context(benchmark: BenchmarkFixture, case: str) -> None: function = CASES[case] @benchmark def factory() -> None: for _ in range(CALLS_PER_ROUND): wrap_loader_context(function, CONTEXT) scrapy-itemloaders-9303326/tests/benchmarks/test_benchmark_loader.py 0000664 0000000 0000000 00000007674 15256713457 0025721 0 ustar 00root root 0000000 0000000 from __future__ import annotations from dataclasses import dataclass from typing import TYPE_CHECKING, Any from parsel import Selector from itemloaders import ItemLoader from itemloaders.processors import Identity, Join, MapCompose, TakeFirst if TYPE_CHECKING: from pytest_codspeed import BenchmarkFixture SELECTOR = Selector( text="""
a product name
First paragraph.
Second paragraph.
") il = TakeFirstItemLoader2() il.add_value("name", ["mar", "ta"]) assert il.get_output_value("name") == "Mar
Ta" def test_default_output_processor(self): il = CustomItemLoader() il.add_value("name", ["mar", "ta"]) assert il.get_output_value("name") == ["Mar", "Ta"] class LalaItemLoader(CustomItemLoader): default_output_processor = Identity() il = LalaItemLoader() il.add_value("name", ["mar", "ta"]) assert il.get_output_value("name") == ["Mar", "Ta"] def test_loader_context_on_declaration(self): class ChildItemLoader(CustomItemLoader): url_in = MapCompose(processor_with_args, key="val") il = ChildItemLoader() il.add_value("url", "text") assert il.get_output_value("url") == ["val"] il.replace_value("url", "text2") assert il.get_output_value("url") == ["val"] def test_loader_context_on_instantiation(self): class ChildItemLoader(CustomItemLoader): url_in = MapCompose(processor_with_args) il = ChildItemLoader(key="val") il.add_value("url", "text") assert il.get_output_value("url") == ["val"] il.replace_value("url", "text2") assert il.get_output_value("url") == ["val"] def test_loader_context_on_assign(self): class ChildItemLoader(CustomItemLoader): url_in = MapCompose(processor_with_args) il = ChildItemLoader() il.context["key"] = "val" il.add_value("url", "text") assert il.get_output_value("url") == ["val"] il.replace_value("url", "text2") assert il.get_output_value("url") == ["val"] def test_item_passed_to_input_processor_functions(self): def processor(value, loader_context): return loader_context["item"]["name"] class ChildItemLoader(CustomItemLoader): url_in = MapCompose(processor) it = {"name": "marta"} il = ChildItemLoader(item=it) il.add_value("url", "text") assert il.get_output_value("url") == ["marta"] il.replace_value("url", "text2") assert il.get_output_value("url") == ["marta"] # def test_add_value_on_unknown_field(self): # il = CustomItemLoader() # self.assertRaises(KeyError, il.add_value, 'wrong_field', ['lala', 'lolo']) def test_compose_processor(self): class CustomItemLoader(ItemLoader): name_out = Compose(lambda v: v[0], lambda v: v.title(), lambda v: v[:-1]) il = CustomItemLoader() il.add_value("name", ["marta", "other"]) assert il.get_output_value("name") == "Mart" item = il.load_item() assert item["name"] == "Mart" def test_partial_processor(self): def join(values, sep=None, loader_context=None, ignored=None): if sep is not None: return sep.join(values) if loader_context and "sep" in loader_context: return loader_context["sep"].join(values) return "".join(values) class CustomItemLoader(ItemLoader): name_out = Compose(partial(join, sep="+")) url_out = Compose(partial(join, loader_context={"sep": "."})) summary_out = Compose(partial(join, ignored="foo")) il = CustomItemLoader() il.add_value("name", ["rabbit", "hole"]) il.add_value("url", ["rabbit", "hole"]) il.add_value("summary", ["rabbit", "hole"]) item = il.load_item() assert item["name"] == "rabbit+hole" assert item["url"] == "rabbit.hole" assert item["summary"] == "rabbithole" def test_error_input_processor(self): class CustomItemLoader(ItemLoader): name_in = MapCompose(float) il = CustomItemLoader() with pytest.raises( ValueError, match=r"Error with input processor MapCompose: .* " r"error='ValueError: Error in MapCompose .* " r"error='ValueError: could not convert", ): il.add_value("name", ["marta", "other"]) def test_error_output_processor(self): class CustomItemLoader(ItemLoader): name_out = Compose(Join(), float) il = CustomItemLoader() il.add_value("name", "marta") with pytest.raises( ValueError, match=r"Error with output processor: .* " r"error='ValueError: Error in Compose .* " r"error='ValueError: could not convert", ): il.load_item() def test_error_processor_as_argument(self): il = CustomItemLoader() with pytest.raises( ValueError, match=r"Error with processor Compose .* " r"error='ValueError: Error in Compose .* " r"error='TypeError: float\(\) argument", ): il.add_value("name", ["marta", "other"], Compose(float)) def test_get_unset_value(self): loader = ItemLoader() assert loader.load_item() == {} assert loader.get_output_value("foo") == [] assert loader.load_item() == {} class BaseNoInputReprocessingLoader(ItemLoader): title_in = MapCompose(str.upper) title_out = TakeFirst() class NoInputReprocessingDictLoader(BaseNoInputReprocessingLoader): default_item_class = dict class TestNoInputReprocessingFromDict: """ Loaders initialized from loaded items must not reprocess fields (dict instances) """ def test_avoid_reprocessing_with_initial_values_single(self): il = NoInputReprocessingDictLoader(item={"title": "foo"}) il_loaded = il.load_item() assert il_loaded == {"title": "foo"} assert NoInputReprocessingDictLoader(item=il_loaded).load_item() == { "title": "foo" } def test_avoid_reprocessing_with_initial_values_list(self): il = NoInputReprocessingDictLoader(item={"title": ["foo", "bar"]}) il_loaded = il.load_item() assert il_loaded == {"title": "foo"} assert NoInputReprocessingDictLoader(item=il_loaded).load_item() == { "title": "foo" } def test_avoid_reprocessing_without_initial_values_single(self): il = NoInputReprocessingDictLoader() il.add_value("title", "foo") il_loaded = il.load_item() assert il_loaded == {"title": "FOO"} assert NoInputReprocessingDictLoader(item=il_loaded).load_item() == { "title": "FOO" } def test_avoid_reprocessing_without_initial_values_list(self): il = NoInputReprocessingDictLoader() il.add_value("title", ["foo", "bar"]) il_loaded = il.load_item() assert il_loaded == {"title": "FOO"} assert NoInputReprocessingDictLoader(item=il_loaded).load_item() == { "title": "FOO" } scrapy-itemloaders-9303326/tests/test_common.py 0000664 0000000 0000000 00000000616 15256713457 0021601 0 ustar 00root root 0000000 0000000 from typing import Any from itemloaders.common import wrap_loader_context CONTEXT: dict[str, Any] = {"encoding": "utf-8"} class Processor: __hash__ = None # type: ignore[assignment] def __call__(self, value: str, loader_context: Any = None) -> Any: return loader_context def test_unhashable_function(): assert wrap_loader_context(Processor(), CONTEXT)("foo") == CONTEXT scrapy-itemloaders-9303326/tests/test_loader_initialization.py 0000664 0000000 0000000 00000011125 15256713457 0024663 0 ustar 00root root 0000000 0000000 from __future__ import annotations from abc import ABC, abstractmethod from dataclasses import dataclass from typing import Any from itemloaders import ItemLoader class TestInitializationBase(ABC): @property @abstractmethod def item_class(self) -> type[Any]: raise NotImplementedError def test_keep_single_value(self) -> None: """Loaded item should contain values from the initial item""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) loaded_item = il.load_item() assert isinstance(loaded_item, self.item_class) assert dict(loaded_item) == {"name": ["foo"]} def test_keep_list(self) -> None: """Loaded item should contain values from the initial item""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) loaded_item = il.load_item() assert isinstance(loaded_item, self.item_class) assert dict(loaded_item) == {"name": ["foo", "bar"]} def test_add_value_singlevalue_singlevalue( self, ) -> None: """Values added after initialization should be appended""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) il.add_value("name", "bar") loaded_item = il.load_item() assert isinstance(loaded_item, self.item_class) assert dict(loaded_item) == {"name": ["foo", "bar"]} def test_add_value_singlevalue_list(self) -> None: """Values added after initialization should be appended""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) il.add_value("name", ["item", "loader"]) loaded_item = il.load_item() assert isinstance(loaded_item, self.item_class) assert dict(loaded_item) == {"name": ["foo", "item", "loader"]} def test_add_value_list_singlevalue(self) -> None: """Values added after initialization should be appended""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) il.add_value("name", "qwerty") loaded_item = il.load_item() assert isinstance(loaded_item, self.item_class) assert dict(loaded_item) == {"name": ["foo", "bar", "qwerty"]} def test_add_value_list_list(self) -> None: """Values added after initialization should be appended""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) il.add_value("name", ["item", "loader"]) loaded_item = il.load_item() assert isinstance(loaded_item, self.item_class) assert dict(loaded_item) == {"name": ["foo", "bar", "item", "loader"]} def test_get_output_value_singlevalue(self) -> None: """Getting output value must not remove value from item""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) assert il.get_output_value("name") == ["foo"] loaded_item = il.load_item() assert isinstance(loaded_item, self.item_class) assert loaded_item == {"name": ["foo"]} def test_get_output_value_list(self) -> None: """Getting output value must not remove value from item""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) assert il.get_output_value("name") == ["foo", "bar"] loaded_item = il.load_item() assert isinstance(loaded_item, self.item_class) assert loaded_item == {"name": ["foo", "bar"]} def test_values_single(self) -> None: """Values from initial item must be added to loader._values""" input_item = self.item_class(name="foo") il = ItemLoader(item=input_item) assert il._values.get("name") == ["foo"] def test_values_list(self) -> None: """Values from initial item must be added to loader._values""" input_item = self.item_class(name=["foo", "bar"]) il = ItemLoader(item=input_item) assert il._values.get("name") == ["foo", "bar"] class TestInitializationFromDict(TestInitializationBase): item_class = dict @dataclass class DefaultItem: name: Any = "default" def test_ignore_field_defaults_of_the_built_item() -> None: class DefaultItemLoader(ItemLoader): default_item_class = DefaultItem il = DefaultItemLoader() il.add_value("name", "foo") assert il.load_item() == DefaultItem(name=["foo"]) def test_keep_field_defaults_of_the_given_item() -> None: il = ItemLoader(item=DefaultItem()) il.add_value("name", "foo") assert il.load_item() == DefaultItem(name=["default", "foo"]) scrapy-itemloaders-9303326/tests/test_nested_items.py 0000664 0000000 0000000 00000002045 15256713457 0022772 0 ustar 00root root 0000000 0000000 from __future__ import annotations from typing import Any import pytest from itemloaders import ItemLoader def _test_item(item: Any) -> None: il = ItemLoader() il.add_value("item_list", item) assert il.load_item() == {"item_list": [item]} def test_attrs(): try: import attr # noqa: PLC0415 except ImportError: pytest.skip("Cannot import attr") @attr.s class TestItem: foo = attr.ib() _test_item(TestItem(foo="bar")) def test_dataclass(): try: from dataclasses import dataclass # noqa: PLC0415 except ImportError: pytest.skip("Cannot import dataclasses.dataclass") @dataclass class TestItem: foo: str _test_item(TestItem(foo="bar")) def test_dict(): _test_item({"foo": "bar"}) def test_scrapy_item(): try: from scrapy import Field, Item # noqa: PLC0415 except ImportError: pytest.skip("Cannot import Field or Item from scrapy") class TestItem(Item): foo = Field() _test_item(TestItem(foo="bar")) scrapy-itemloaders-9303326/tests/test_nested_loader.py 0000664 0000000 0000000 00000014302 15256713457 0023116 0 ustar 00root root 0000000 0000000 import pytest from parsel import Selector from itemloaders import ItemLoader from itemloaders.processors import MapCompose class TestSubselectorLoader: selector = Selector( text="""
marta
paragraph
marta
']
assert loader.get_output_value("name_value") == ["marta"]
assert loader.get_output_value("name") == nl.get_output_value("name")
assert loader.get_output_value("name_div") == nl.get_output_value("name_div")
assert loader.get_output_value("name_value") == nl.get_output_value(
"name_value"
)
def test_nested_css(self):
loader = ItemLoader(selector=self.selector)
nl = loader.nested_css("header")
nl.add_xpath("name", "div/text()")
nl.add_css("name_div", "#id")
assert nl.selector
nl.add_value("name_value", nl.selector.xpath('div[@id = "id"]/text()').getall())
assert loader.get_output_value("name") == ["marta"]
assert loader.get_output_value("name_div") == ['marta
']
assert loader.get_output_value("name_value") == ["marta"]
assert loader.get_output_value("name") == nl.get_output_value("name")
assert loader.get_output_value("name_div") == nl.get_output_value("name_div")
assert loader.get_output_value("name_value") == nl.get_output_value(
"name_value"
)
def test_nested_replace(self):
loader = ItemLoader(selector=self.selector)
nl1 = loader.nested_xpath("//footer")
nl2 = nl1.nested_xpath("a")
loader.add_xpath("url", "//footer/a/@href")
assert loader.get_output_value("url") == ["http://www.scrapy.org"]
nl1.replace_xpath("url", "img/@src")
assert loader.get_output_value("url") == ["/images/logo.png"]
nl2.replace_xpath("url", "@href")
assert loader.get_output_value("url") == ["http://www.scrapy.org"]
def test_nested_ordering(self):
loader = ItemLoader(selector=self.selector)
nl1 = loader.nested_xpath("//footer")
nl2 = nl1.nested_xpath("a")
nl1.add_xpath("url", "img/@src")
loader.add_xpath("url", "//footer/a/@href")
nl2.add_xpath("url", "text()")
loader.add_xpath("url", "//footer/a/@href")
assert loader.get_output_value("url") == [
"/images/logo.png",
"http://www.scrapy.org",
"homepage",
"http://www.scrapy.org",
]
def test_nested_load_item(self):
loader = ItemLoader(selector=self.selector)
nl1 = loader.nested_xpath("//footer")
nl2 = nl1.nested_xpath("img")
loader.add_xpath("name", "//header/div/text()")
nl1.add_xpath("url", "a/@href")
nl2.add_xpath("image", "@src")
item = loader.load_item()
assert item is loader.item
assert item is nl1.item
assert item is nl2.item
assert item["name"] == ["marta"]
assert item["url"] == ["http://www.scrapy.org"]
assert item["image"] == ["/images/logo.png"]
def test_nested_empty_selector(self):
loader = ItemLoader(selector=self.selector)
nested_xpath = loader.nested_xpath("//bar")
assert isinstance(nested_xpath, ItemLoader)
nested_xpath.add_xpath("foo", "./foo")
nested_css = loader.nested_css("bar")
assert isinstance(nested_css, ItemLoader)
nested_css.add_css("foo", "foo")
@pytest.mark.parametrize(
("method", "expression"), [("nested_xpath", "//footer"), ("nested_css", "footer")]
)
def test_nested_context_inheritance(method: str, expression: str) -> None:
marker = object()
loader = ItemLoader(
selector=TestSubselectorLoader.selector, marker=marker, prefix="parent"
)
original_context = dict(loader.context)
child = getattr(loader, method)(expression, prefix="child")
sibling = getattr(loader, method)(expression)
grandchild = child.nested_css("a")
for nested in (child, sibling, grandchild):
assert nested.context["marker"] is marker
assert nested.context["item"] is loader.item
assert nested.context["selector"] is nested.selector
assert nested.context is not loader.context
assert child.context["prefix"] == grandchild.context["prefix"] == "child"
assert sibling.context["prefix"] == "parent"
assert loader.context == original_context
child.context["new"] = True
assert "new" not in loader.context
assert "new" not in sibling.context
grandchild.default_input_processor = MapCompose(
lambda value, loader_context: loader_context["prefix"] + ":" + value
)
grandchild.add_css("name", "::text")
assert loader.load_item()["name"] == ["child:homepage"]
@pytest.mark.parametrize(
("method", "expression"), [("nested_xpath", "//div"), ("nested_css", "div")]
)
def test_nested_scrapy_response_context(method: str, expression: str) -> None:
scrapy_loader = pytest.importorskip("scrapy.loader")
scrapy_http = pytest.importorskip("scrapy.http")
response = scrapy_http.HtmlResponse(
url="https://example.com/", body=b''
)
loader = scrapy_loader.ItemLoader(item={}, response=response)
nested = getattr(loader, method)(expression)
nested.default_input_processor = MapCompose(
lambda value, loader_context: loader_context["response"].urljoin(value)
)
nested.add_css(method, "a::attr(href)")
assert nested.context["response"] is response
assert loader.load_item()[method] == ["https://example.com/target"]
scrapy-itemloaders-9303326/tests/test_output_processor.py 0000664 0000000 0000000 00000002254 15256713457 0023750 0 ustar 00root root 0000000 0000000 from __future__ import annotations
from typing import Any
from itemloaders import ItemLoader
from itemloaders.processors import Compose, Identity, TakeFirst
class TestOutputProcessorDict:
def test_output_processor(self):
class TempDict(dict[str, Any]):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.setdefault("temp", 0.3)
class TempLoader(ItemLoader):
default_item_class = TempDict
default_input_processor = Identity()
default_output_processor = Compose(TakeFirst())
loader = TempLoader()
item = loader.load_item()
assert isinstance(item, TempDict)
assert dict(item) == {"temp": 0.3}
class TestOutputProcessorItem:
def test_output_processor(self):
class TempLoader(ItemLoader):
default_input_processor = Identity()
default_output_processor = Compose(TakeFirst())
item: dict[str, Any] = {}
item.setdefault("temp", 0.3)
loader = TempLoader(item=item)
item = loader.load_item()
assert isinstance(item, dict)
assert dict(item) == {"temp": 0.3}
scrapy-itemloaders-9303326/tests/test_processors.py 0000664 0000000 0000000 00000005100 15256713457 0022504 0 ustar 00root root 0000000 0000000 from functools import partial
import pytest
from itemloaders.processors import Compose, Identity, Join, MapCompose, TakeFirst
def test_take_first():
proc = TakeFirst()
assert proc([None, "", "hello", "world"]) == "hello"
assert proc([None, "", 0, "hello", "world"]) == 0
def test_identity():
proc = Identity()
assert proc([None, "", "hello", "world"]) == [None, "", "hello", "world"]
def test_join():
proc = Join()
with pytest.raises(TypeError):
proc([None, "", "hello", "world"])
assert proc(["", "hello", "world"]) == " hello world"
assert proc(["hello", "world"]) == "hello world"
assert isinstance(proc(["hello", "world"]), str)
def test_compose():
proc = Compose(lambda v: v[0], str.upper)
assert proc(["hello", "world"]) == "HELLO"
proc = Compose(str.upper)
assert proc(None) is None
proc = Compose(str.upper, stop_on_none=False)
with pytest.raises(
ValueError,
match=r"Error in Compose with .* error='TypeError: (descriptor 'upper'|'str' object expected)",
):
proc(None)
proc = Compose(str.upper, lambda x: x + 1)
with pytest.raises(
ValueError,
match=r"Error in Compose with .* error='TypeError: (can only|unsupported operand)",
):
proc("hello")
def test_mapcompose():
def filter_world(x):
return None if x == "world" else x
proc = MapCompose(filter_world, str.upper)
assert proc(["hello", "world", "this", "is", "scrapy"]) == [
"HELLO",
"THIS",
"IS",
"SCRAPY",
]
proc = MapCompose(filter_world, str.upper)
assert proc(None) == []
proc = MapCompose(filter_world, str.upper)
with pytest.raises(
ValueError,
match=r"Error in MapCompose with .* error='TypeError: (descriptor 'upper'|'str' object expected)",
):
proc([1])
proc = MapCompose(filter_world, lambda x: x + 1)
with pytest.raises(
ValueError,
match=r"Error in MapCompose with .* error='TypeError: (can only|unsupported operand)",
):
proc("hello")
@pytest.mark.parametrize("processor", [Compose, MapCompose])
def test_partial_processor_keeps_context_with_matching_value(processor):
def prefix_text(prefix, value, loader_context):
return prefix + value + loader_context["suffix"]
proc = processor(partial(prefix_text, "loader_context"))
value = ["text"] if processor is MapCompose else "text"
expected = (
["loader_contexttext!"] if processor is MapCompose else "loader_contexttext!"
)
assert proc(value, {"suffix": "!"}) == expected
scrapy-itemloaders-9303326/tests/test_rule_usage.py 0000664 0000000 0000000 00000003726 15256713457 0022451 0 ustar 00root root 0000000 0000000 from collections import Counter
import pytest
from parsel import Selector
from itemloaders import ItemLoader
from tests.test_selector_loader import needs_parsel_jmespath
class Stats:
def __init__(self) -> None:
self.values: Counter[str] = Counter()
def inc_value(self, key: str, count: int = 1) -> None:
self.values[key] += count
@pytest.fixture
def stats():
return Stats()
@pytest.fixture
def loader(stats):
selector = Selector(
text='Color TV
$1200
' ) return ItemLoader(selector=selector, stats=stats) def test_css(loader, stats): loader.add_css("name", ["h1::text", 'meta[name="title"]::attr(content)']) assert stats.values == { "parser/name/css/h1::text": 1, 'parser/name/css/meta[name="title"]::attr(content)': 0, } def test_xpath(loader, stats): loader.replace_xpath("name", "//h1/text()") assert stats.values == {"parser/name/xpath///h1/text()": 1} @needs_parsel_jmespath def test_jmes(stats): loader = ItemLoader(selector=Selector(text='{"name": "Color TV"}'), stats=stats) loader.add_jmes("name", "name") loader.add_jmes("name", "title") assert stats.values == { "parser/name/jmes/name": 1, "parser/name/jmes/title": 0, } def test_no_field_name(loader, stats): assert loader.get_css("h1::text") == ["Color TV"] assert stats.values == {"parser/Color TV
")) loader.add_css("name", "h1::text") assert loader.get_output_value("name") == ["Color TV"] def test_nested_loader(loader, stats): loader.nested_css("header").add_css("name", "h1::text") assert stats.values == {"parser/name/css/h1::text": 1} def test_repeated_rule(loader, stats): loader.add_css("name", "h1::text") loader.add_css("name", "h1::text") assert stats.values == {"parser/name/css/h1::text": 2} scrapy-itemloaders-9303326/tests/test_select_jmes.py 0000664 0000000 0000000 00000001223 15256713457 0022601 0 ustar 00root root 0000000 0000000 from itemloaders.processors import SelectJmes test_list_equals = { "simple": ("foo.bar", {"foo": {"bar": "baz"}}, "baz"), "invalid": ("foo.bar.baz", {"foo": {"bar": "baz"}}, None), "top_level": ("foo", {"foo": {"bar": "baz"}}, {"bar": "baz"}), "dict": ( "foo.bar[*].name", {"foo": {"bar": [{"name": "one"}, {"name": "two"}]}}, ["one", "two"], ), "list": ("[1]", [1, 2], 2), } def test_output(): for key, value in test_list_equals.items(): expr, test_list, expected = value test = SelectJmes(expr)(test_list) assert test == expected, f"test {key!r} got {test} expected {expected}" scrapy-itemloaders-9303326/tests/test_selector_loader.py 0000664 0000000 0000000 00000030251 15256713457 0023455 0 ustar 00root root 0000000 0000000 import re from unittest.mock import MagicMock import pytest from parsel import Selector from itemloaders import ItemLoader from itemloaders.processors import MapCompose, TakeFirst needs_parsel_jmespath = pytest.mark.skipif( not hasattr(Selector, "jmespath"), reason="parsel < 1.8.1" ) class CustomItemLoader(ItemLoader): name_in = MapCompose(lambda v: v.title()) class TestSelectorItemLoader: selector = Selector( text="""marta
paragraph
homepage
"""
)
jmes_selector = Selector(
text="""
{
"name": "marta",
"description": "paragraph",
"website": {
"url": "http://www.scrapy.org",
"name": "homepage"
},
"logo": "/images/logo.png"
}
"""
)
def test_init_method(self):
loader = CustomItemLoader()
assert loader.selector is None
def test_init_method_errors(self):
loader = CustomItemLoader()
with pytest.raises(RuntimeError):
loader.add_xpath("url", "//a/@href")
with pytest.raises(RuntimeError):
loader.replace_xpath("url", "//a/@href")
with pytest.raises(RuntimeError):
loader.get_xpath("//a/@href")
with pytest.raises(RuntimeError):
loader.add_css("name", "#name::text")
with pytest.raises(RuntimeError):
loader.replace_css("name", "#name::text")
with pytest.raises(RuntimeError):
loader.get_css("#name::text")
with pytest.raises(RuntimeError):
loader.add_jmes("name", "name")
with pytest.raises(RuntimeError):
loader.replace_jmes("name", "name")
with pytest.raises(RuntimeError):
loader.get_jmes("name")
def test_init_method_with_selector(self):
loader = CustomItemLoader(selector=self.selector)
assert loader.selector
loader.add_xpath("name", "//div/text()")
assert loader.get_output_value("name") == ["Marta"]
def test_init_method_with_selector_css(self):
loader = CustomItemLoader(selector=self.selector)
assert loader.selector
loader.add_css("name", "div::text")
assert loader.get_output_value("name") == ["Marta"]
loader.add_css("url", "a::attr(href)")
assert loader.get_output_value("url") == ["http://www.scrapy.org"]
# combining/accumulating CSS selectors and XPath expressions
loader.add_xpath("name", "//div/text()")
assert loader.get_output_value("name") == ["Marta", "Marta"]
loader.add_xpath("url", "//img/@src")
assert loader.get_output_value("url") == [
"http://www.scrapy.org",
"/images/logo.png",
]
def test_add_xpath_re(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath("name", "//div/text()", re="ma")
assert loader.get_output_value("name") == ["Ma"]
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath("name", "//div/text()", re=re.compile("ma"))
assert loader.get_output_value("name") == ["Ma"]
def test_add_xpath_variables(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath("name", "id($id)/text()", id="id")
assert loader.get_output_value("name") == ["Marta"]
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath("name", "id($id)/text()", id="id2")
assert loader.get_output_value("name") == []
def test_replace_xpath(self):
loader = CustomItemLoader(selector=self.selector)
assert loader.selector
loader.add_xpath("name", "//div/text()")
assert loader.get_output_value("name") == ["Marta"]
loader.replace_xpath("name", "//p/text()")
assert loader.get_output_value("name") == ["Paragraph"]
loader.replace_xpath("name", ["//p/text()", "//div/text()"])
assert loader.get_output_value("name") == ["Paragraph", "Marta"]
def test_get_xpath(self):
loader = CustomItemLoader(selector=self.selector)
assert loader.get_xpath("//p/text()") == ["paragraph"]
assert loader.get_xpath("//p/text()", TakeFirst()) == "paragraph"
assert loader.get_xpath("//p/text()", TakeFirst(), re="pa") == "pa"
assert loader.get_xpath(["//p/text()", "//div/text()"]) == [
"paragraph",
"marta",
]
def test_replace_xpath_multi_fields(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_xpath(None, "//div/text()", TakeFirst(), lambda x: {"name": x})
assert loader.get_output_value("name") == ["Marta"]
loader.replace_xpath(None, "//p/text()", TakeFirst(), lambda x: {"name": x})
assert loader.get_output_value("name") == ["Paragraph"]
def test_replace_xpath_re(self):
loader = CustomItemLoader(selector=self.selector)
assert loader.selector
loader.add_xpath("name", "//div/text()")
assert loader.get_output_value("name") == ["Marta"]
loader.replace_xpath("name", "//div/text()", re="ma")
assert loader.get_output_value("name") == ["Ma"]
def test_add_css_re(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_css("name", "div::text", re="ma")
assert loader.get_output_value("name") == ["Ma"]
loader.add_css("url", "a::attr(href)", re="http://(.+)")
assert loader.get_output_value("url") == ["www.scrapy.org"]
loader = CustomItemLoader(selector=self.selector)
loader.add_css("name", "div::text", re=re.compile("ma"))
assert loader.get_output_value("name") == ["Ma"]
loader.add_css("url", "a::attr(href)", re=re.compile("http://(.+)"))
assert loader.get_output_value("url") == ["www.scrapy.org"]
def test_replace_css(self):
loader = CustomItemLoader(selector=self.selector)
assert loader.selector
loader.add_css("name", "div::text")
assert loader.get_output_value("name") == ["Marta"]
loader.replace_css("name", "p::text")
assert loader.get_output_value("name") == ["Paragraph"]
loader.replace_css("name", ["p::text", "div::text"])
assert loader.get_output_value("name") == ["Paragraph", "Marta"]
loader.add_css("url", "a::attr(href)", re="http://(.+)")
assert loader.get_output_value("url") == ["www.scrapy.org"]
loader.replace_css("url", "img::attr(src)")
assert loader.get_output_value("url") == ["/images/logo.png"]
def test_get_css(self):
loader = CustomItemLoader(selector=self.selector)
assert loader.get_css("p::text") == ["paragraph"]
assert loader.get_css("p::text", TakeFirst()) == "paragraph"
assert loader.get_css("p::text", TakeFirst(), re="pa") == "pa"
assert loader.get_css(["p::text", "div::text"]) == ["paragraph", "marta"]
assert loader.get_css(["a::attr(href)", "img::attr(src)"]) == [
"http://www.scrapy.org",
"/images/logo.png",
]
def test_replace_css_multi_fields(self):
loader = CustomItemLoader(selector=self.selector)
loader.add_css(None, "div::text", TakeFirst(), lambda x: {"name": x})
assert loader.get_output_value("name") == ["Marta"]
loader.replace_css(None, "p::text", TakeFirst(), lambda x: {"name": x})
assert loader.get_output_value("name") == ["Paragraph"]
loader.add_css(None, "a::attr(href)", TakeFirst(), lambda x: {"url": x})
assert loader.get_output_value("url") == ["http://www.scrapy.org"]
loader.replace_css(None, "img::attr(src)", TakeFirst(), lambda x: {"url": x})
assert loader.get_output_value("url") == ["/images/logo.png"]
def test_replace_css_re(self):
loader = CustomItemLoader(selector=self.selector)
assert loader.selector
loader.add_css("url", "a::attr(href)")
assert loader.get_output_value("url") == ["http://www.scrapy.org"]
loader.replace_css("url", "a::attr(href)", re=r"http://www\.(.+)")
assert loader.get_output_value("url") == ["scrapy.org"]
def test_jmes_not_installed(self):
selector = MagicMock(spec=Selector)
del selector.jmespath
loader = CustomItemLoader(selector=selector)
with pytest.raises(AttributeError) as err:
loader.add_jmes("name", "name", re="ma")
assert (
str(err.value) == "Please install parsel >= 1.8.1 to get JMESPath support"
)
@needs_parsel_jmespath
def test_add_jmes_re(self):
loader = CustomItemLoader(selector=self.jmes_selector)
loader.add_jmes("name", "name", re="ma")
assert loader.get_output_value("name") == ["Ma"]
loader.add_jmes("url", "website.url", re="http://(.+)")
assert loader.get_output_value("url") == ["www.scrapy.org"]
loader = CustomItemLoader(selector=self.jmes_selector)
loader.add_jmes("name", "name", re=re.compile("ma"))
assert loader.get_output_value("name") == ["Ma"]
loader.add_jmes("url", "website.url", re=re.compile("http://(.+)"))
assert loader.get_output_value("url") == ["www.scrapy.org"]
@needs_parsel_jmespath
def test_get_jmes(self):
loader = CustomItemLoader(selector=self.jmes_selector)
assert loader.get_jmes("description") == ["paragraph"]
assert loader.get_jmes("description", TakeFirst()) == "paragraph"
assert loader.get_jmes("description", TakeFirst(), re="pa") == "pa"
assert loader.get_jmes(["description", "name"]) == ["paragraph", "marta"]
assert loader.get_jmes(["website.url", "logo"]) == [
"http://www.scrapy.org",
"/images/logo.png",
]
@needs_parsel_jmespath
def test_replace_jmes(self):
loader = CustomItemLoader(selector=self.jmes_selector)
assert loader.selector
loader.add_jmes("name", "name")
assert loader.get_output_value("name") == ["Marta"]
loader.replace_jmes("name", "description")
assert loader.get_output_value("name") == ["Paragraph"]
loader.replace_jmes("name", ["description", "name"])
assert loader.get_output_value("name") == ["Paragraph", "Marta"]
loader.add_jmes("url", "website.url", re="http://(.+)")
assert loader.get_output_value("url") == ["www.scrapy.org"]
loader.replace_jmes("url", "logo")
assert loader.get_output_value("url") == ["/images/logo.png"]
@needs_parsel_jmespath
def test_replace_jmes_multi_fields(self):
loader = CustomItemLoader(selector=self.jmes_selector)
loader.add_jmes(None, "name", TakeFirst(), lambda x: {"name": x})
assert loader.get_output_value("name") == ["Marta"]
loader.replace_jmes(None, "description", TakeFirst(), lambda x: {"name": x})
assert loader.get_output_value("name") == ["Paragraph"]
loader.add_jmes(None, "website.url", TakeFirst(), lambda x: {"url": x})
assert loader.get_output_value("url") == ["http://www.scrapy.org"]
loader.replace_jmes(None, "logo", TakeFirst(), lambda x: {"url": x})
assert loader.get_output_value("url") == ["/images/logo.png"]
@needs_parsel_jmespath
def test_replace_jmes_re(self):
loader = CustomItemLoader(selector=self.jmes_selector)
assert loader.selector
loader.add_jmes("url", "website.url")
assert loader.get_output_value("url") == ["http://www.scrapy.org"]
loader.replace_jmes("url", "website.url", re=r"http://www\.(.+)")
assert loader.get_output_value("url") == ["scrapy.org"]
def test_fluent_interface(self):
loader = ItemLoader(selector=self.selector)
item = (
loader.add_xpath("name", "//body/text()")
.replace_xpath("name", "//div/text()")
.add_css("description", "div::text")
.replace_css("description", "p::text")
.add_value("url", "http://example.com")
.replace_value("url", "http://foo")
.load_item()
)
assert item == {
"name": ["marta"],
"description": ["paragraph"],
"url": ["http://foo"],
}
scrapy-itemloaders-9303326/tests/test_utils_misc.py 0000664 0000000 0000000 00000004206 15256713457 0022463 0 ustar 00root root 0000000 0000000 from collections.abc import Set as AbstractSet
from itemloaders.utils import _MAX_ITERABLE_CLASSES, _iterable_classes, arg_to_iter
def test_arg_to_iter():
assert hasattr(arg_to_iter(None), "__iter__")
assert hasattr(arg_to_iter(100), "__iter__")
assert hasattr(arg_to_iter("lala"), "__iter__")
assert hasattr(arg_to_iter([1, 2, 3]), "__iter__")
assert hasattr(arg_to_iter(letter for letter in "abcd"), "__iter__")
assert list(arg_to_iter(None)) == []
assert list(arg_to_iter("lala")) == ["lala"]
assert list(arg_to_iter(100)) == [100]
assert list(arg_to_iter(letter for letter in "abc")) == ["a", "b", "c"]
assert list(arg_to_iter([1, 2, 3])) == [1, 2, 3]
assert list(arg_to_iter({"a": 1})) == [{"a": 1}]
assert sorted(arg_to_iter({1, 2, 3})) == [1, 2, 3]
assert sorted(arg_to_iter(frozenset({1, 2, 3}))) == [1, 2, 3]
assert list(arg_to_iter({"a": 1}.keys())) == ["a"]
assert list(arg_to_iter(map(str, [1, 2]))) == ["1", "2"]
assert list(arg_to_iter(iter([1, 2]))) == [1, 2]
def test_arg_to_iter_custom_iterator():
class Countdown:
def __init__(self, start: int):
self.value = start
def __iter__(self):
return self
def __next__(self):
if not self.value:
raise StopIteration
self.value -= 1
return self.value
assert list(arg_to_iter(Countdown(2))) == [1, 0]
def test_arg_to_iter_late_registration():
class Pair:
def __init__(self, first: int, second: int):
self._items = (first, second)
def __iter__(self):
return iter(self._items)
def __len__(self):
return 2
def __contains__(self, value):
return value in self._items
pair = Pair(1, 2)
assert list(arg_to_iter(pair)) == [pair]
AbstractSet.register(Pair)
assert list(arg_to_iter(pair)) == [1, 2]
def test_arg_to_iter_dynamic_classes():
for index in range(_MAX_ITERABLE_CLASSES + 10):
obj = type(f"Dynamic{index}", (), {})()
assert list(arg_to_iter(obj)) == [obj]
assert len(_iterable_classes) <= _MAX_ITERABLE_CLASSES
scrapy-itemloaders-9303326/tests/test_utils_python.py 0000664 0000000 0000000 00000005062 15256713457 0023052 0 ustar 00root root 0000000 0000000 from __future__ import annotations
import functools
import operator
import platform
import sys
from typing import Any
import pytest
from itemloaders.utils import get_func_args
def test_get_func_args():
def f1(a, b, c):
pass
def f2(a, b=None, c=None):
pass
def f3(a, b=None, *, c=None):
pass
class A:
def __init__(self, a: Any, b: Any, c: Any):
pass
def method(self, a, b, c):
pass
class Callable:
def __call__(self, a, b, c):
pass
a = A(1, 2, 3)
cal = Callable()
partial_f1 = functools.partial(f1, None)
partial_f2 = functools.partial(f1, b=None)
partial_f3 = functools.partial(partial_f2, None)
assert get_func_args(f1) == ["a", "b", "c"]
assert get_func_args(f2) == ["a", "b", "c"]
assert get_func_args(f3) == ["a", "b", "c"]
assert get_func_args(A) == ["a", "b", "c"]
assert get_func_args(a.method) == ["a", "b", "c"]
assert get_func_args(partial_f1) == ["b", "c"]
assert get_func_args(partial_f2) == ["a", "c"]
assert get_func_args(partial_f3) == ["c"]
assert get_func_args(cal) == ["a", "b", "c"]
assert get_func_args(object) == []
assert get_func_args(str.split, stripself=True) == ["sep", "maxsplit"]
assert get_func_args(" ".join, stripself=True) == ["iterable"]
if sys.version_info >= (3, 13) or platform.python_implementation() == "PyPy":
# the correct and correctly extracted signature
assert get_func_args(operator.itemgetter(2), stripself=True) == ["obj"]
elif platform.python_implementation() == "CPython":
# ["args", "kwargs"] is a correct result for the pre-3.13 incorrect function signature
# [] is an incorrect result on even older CPython (https://github.com/python/cpython/issues/86951)
assert get_func_args(operator.itemgetter(2), stripself=True) in [
[],
["args", "kwargs"],
]
def test_partial_argument_values_are_not_parameter_names():
def function(value, loader_context):
return value, loader_context
assert get_func_args(functools.partial(function, "loader_context")) == [
"loader_context"
]
def test_get_func_args_not_callable():
with pytest.raises(TypeError):
get_func_args("not a callable") # type: ignore[arg-type]
def test_get_func_args_without_signature():
class Unintrospectable:
@property
def __signature__(self):
raise ValueError
def __call__(self, a):
pass
assert get_func_args(Unintrospectable()) == []
scrapy-itemloaders-9303326/tox.ini 0000664 0000000 0000000 00000003303 15256713457 0017045 0 ustar 00root root 0000000 0000000 [tox]
envlist = py310,py311,py312,py313,py314
[testenv]
deps =
pytest
pytest-cov >= 7.0.0
commands =
pytest \
--cov-report=term --cov-report=html --cov-report= --cov=itemloaders \
--doctest-modules \
{posargs:itemloaders tests}
[testenv:min]
basepython = python3.10
deps =
{[testenv]deps}
itemadapter==0.1.0
jmespath==0.9.5
parsel==1.5.0
# parsel 1.5.0 uses cssselect._unicode_safe_getattr, removed in cssselect 1.2.0
cssselect==1.1.0
[testenv:benchmark]
basepython = python3.14
deps =
{[testenv:extra-deps]deps}
pytest-codspeed
passenv =
*codspeed*
*ci*
commands =
pytest --no-cov --codspeed --codspeed-mode=simulation tests/benchmarks/
[testenv:extra-deps]
deps =
{[testenv]deps}
attrs
scrapy
[testenv:pypy3]
basepython = pypy3
[docs]
changedir = docs
deps =
-rdocs/requirements.txt
setenv =
READTHEDOCS_PROJECT=itemloaders
READTHEDOCS_VERSION=master
[testenv:docs]
basepython = python3
changedir = {[docs]changedir}
deps = {[docs]deps}
setenv = {[docs]setenv}
commands =
sphinx-build -W -b html . {envtmpdir}/html
[testenv:twinecheck]
basepython = python3
deps =
twine==7.0.0
build==1.5.0
commands =
python -m build --sdist
twine check dist/*
[testenv:typing]
basepython = python3
deps =
mypy==2.3.0
attrs==26.1.0
pytest==9.1.1
pytest-codspeed==5.0.3
Scrapy==2.17.0
types-jmespath==1.1.0.20260724
commands =
mypy {posargs:itemloaders tests}
[testenv:pre-commit]
basepython = python3
deps =
pre-commit
commands =
pre-commit run {posargs:--all-files}
[testenv:pylint]
deps =
{[testenv]deps}
pylint==4.0.7
commands =
pylint docs itemloaders tests