pax_global_header 0000666 0000000 0000000 00000000064 15250510124 0014505 g ustar 00root root 0000000 0000000 52 comment=8026deeaac371a5d9a3edbe4886d58f61139d464
scrapy-scrapy-5569de3/ 0000775 0000000 0000000 00000000000 15250510124 0014653 5 ustar 00root root 0000000 0000000 scrapy-scrapy-5569de3/.git-blame-ignore-revs 0000664 0000000 0000000 00000000502 15250510124 0020750 0 ustar 00root root 0000000 0000000 # .git-blame-ignore-revs
# adding black formatter to all the code
e211ec0aa26ecae0da8ae55d064ea60e1efe4d0d
# reapplying black to the code with default line length
303f0a70fcf8067adf0a909c2096a5009162383a
# reapplying black again and removing line length on pre-commit black config
c5cdd0d30ceb68ccba04af0e71d1b8e6678e2962
scrapy-scrapy-5569de3/.gitattributes 0000664 0000000 0000000 00000000034 15250510124 0017543 0 ustar 00root root 0000000 0000000 tests/sample_data/** binary
scrapy-scrapy-5569de3/.github/ 0000775 0000000 0000000 00000000000 15250510124 0016213 5 ustar 00root root 0000000 0000000 scrapy-scrapy-5569de3/.github/ISSUE_TEMPLATE/ 0000775 0000000 0000000 00000000000 15250510124 0020376 5 ustar 00root root 0000000 0000000 scrapy-scrapy-5569de3/.github/ISSUE_TEMPLATE/bug_report.md 0000664 0000000 0000000 00000002763 15250510124 0023100 0 ustar 00root root 0000000 0000000 ---
name: Bug report
about: Report a problem to help us improve
---
### Description
[Description of the issue]
### Steps to Reproduce
1. [First Step]
2. [Second Step]
3. [and so on...]
**Expected behavior:** [What you expect to happen]
**Actual behavior:** [What actually happens]
**Reproduces how often:** [What percentage of the time does it reproduce?]
### Versions
Please paste here the output of executing `scrapy version --verbose` in the command line.
### Additional context
Any additional information, configuration, data or output from commands that might be necessary to reproduce or understand the issue. Please try not to include screenshots of code or the command line, paste the contents as text instead. You can use [GitHub Flavored Markdown](https://help.github.com/en/articles/creating-and-highlighting-code-blocks) to make the text look better.
scrapy-scrapy-5569de3/.github/ISSUE_TEMPLATE/feature_request.md 0000664 0000000 0000000 00000002370 15250510124 0024125 0 ustar 00root root 0000000 0000000 ---
name: Feature request
about: Suggest an idea for an enhancement or new feature
---
## Summary
One paragraph explanation of the feature.
## Motivation
Why are we doing this? What use cases does it support? What is the expected outcome?
## Describe alternatives you've considered
A clear and concise description of the alternative solutions you've considered. Be sure to explain why Scrapy's existing customizability isn't suitable for this feature.
## Additional context
Any additional information about the feature request here.
scrapy-scrapy-5569de3/.github/ISSUE_TEMPLATE/question.md 0000664 0000000 0000000 00000000644 15250510124 0022573 0 ustar 00root root 0000000 0000000 ---
name: Question / Help
about: Ask a question about Scrapy or ask for help with your Scrapy code.
---
Thanks for taking an interest in Scrapy!
The Scrapy GitHub issue tracker is not meant for questions or help. Please ask
for help in the [Scrapy community resources](https://scrapy.org/community/)
instead.
The GitHub issue tracker's purpose is to deal with bug reports and feature
requests for the project itself.
scrapy-scrapy-5569de3/.github/dependabot.yml 0000664 0000000 0000000 00000000333 15250510124 0021042 0 ustar 00root root 0000000 0000000 version: 2
updates:
- package-ecosystem: github-actions
directory: "/"
schedule:
interval: monthly
groups:
github-actions:
patterns:
- "*"
cooldown:
default-days: 7
scrapy-scrapy-5569de3/.github/workflows/ 0000775 0000000 0000000 00000000000 15250510124 0020250 5 ustar 00root root 0000000 0000000 scrapy-scrapy-5569de3/.github/workflows/checks.yml 0000664 0000000 0000000 00000004130 15250510124 0022231 0 ustar 00root root 0000000 0000000 name: Checks
permissions:
contents: read
on:
push:
branches:
- master
- '[0-9]+.[0-9]+'
pull_request:
types: [opened, synchronize, reopened, closed]
concurrency:
group: ${{github.workflow}}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
jobs:
checks:
if: github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
env:
# Make uv use the interpreter that actions/setup-python installed instead
# of downloading one of its own.
UV_PYTHON_PREFERENCE: only-system
strategy:
fail-fast: false
matrix:
include:
- python-version: "3.14"
env:
TOXENV: pylint
- python-version: "3.10"
env:
TOXENV: mypy
- python-version: "3.10"
env:
TOXENV: mypy-tests
# Keep in sync with pyproject.toml tool.sphinx-scrapy.python-version.
- python-version: "3.14"
env:
TOXENV: docs
- python-version: "3.13"
env:
TOXENV: docs-tests
- python-version: "3.14"
env:
TOXENV: twinecheck
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
docs/requirements.txt
pyproject.toml
tox.ini
- name: Run check
env: ${{ matrix.env }}
run: uvx --with tox-uv tox
pre-commit:
if: github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: pre-commit/action@2c7b3805fd2a0fd8c1884dcaebf91fc102a13ecd # v3.0.1
scrapy-scrapy-5569de3/.github/workflows/codspeed.yml 0000664 0000000 0000000 00000007117 15250510124 0022567 0 ustar 00root root 0000000 0000000 ---
name: codspeed
on:
push:
branches:
- master
pull_request:
types: [opened, synchronize, reopened, closed]
paths:
- scrapy/**
- tests/benchmarks/**
- .github/workflows/codspeed.yml
- tox.ini
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
permissions: {}
jobs:
benchmark:
if: github.event.action != 'closed'
runs-on: ubuntu-latest
timeout-minutes: 25
env:
# Make uv use the interpreter that actions/setup-python installed
# instead of downloading one of its own.
UV_PYTHON_PREFERENCE: only-system
permissions:
contents: read
id-token: write # OIDC authentication with CodSpeed
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python 3.14
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.14'
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
tox.ini
- name: Install dependencies
# tox must stay on PATH for the CodSpeed action to invoke it.
run: |
uv tool install --with tox-uv tox
tox -n -e benchmark
- name: Run benchmarks
uses: CodSpeedHQ/action@373d6868929f444bc08d901fd0eb0ad52a8875ea # v5.2.1
with:
mode: simulation
run: tox -e benchmark
benchmark-walltime:
# Only benchmarks marked "walltime" (ones where a real concurrency
# win can't be shown by instruction counting) run here, on
# CodSpeed's dedicated, minutes-limited macro runner; everything
# else runs above, on a regular GitHub-hosted runner.
if: github.event.action != 'closed'
runs-on: codspeed-macro
env:
UV_PYTHON_PREFERENCE: only-system
permissions:
contents: read
id-token: write # OIDC authentication with CodSpeed
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python 3.14
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.14'
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
tox.ini
- name: Install dependencies
run: |
uv tool install --with tox-uv tox
tox -n -e benchmark
- name: Run benchmarks
uses: CodSpeedHQ/action@373d6868929f444bc08d901fd0eb0ad52a8875ea # v5.2.1
with:
mode: walltime
# No benchmark is walltime-marked on some branches (e.g.
# this one), which would otherwise fail the step.
allow-empty: true
run: tox -e benchmark -- tests/benchmarks -m walltime
scrapy-scrapy-5569de3/.github/workflows/flag-prs-for-triage.yml 0000664 0000000 0000000 00000033100 15250510124 0024540 0 ustar 00root root 0000000 0000000 name: Flag PRs for triage
# Labels pull requests whose author's public activity suggests that an LLM is
# writing them without supervision, and records the evidence in the workflow
# run summary so that triaging one does not require reading a user profile.
#
# Four independent signals, any of which is enough to label. Each one abstains
# when the data it needs is unavailable, so a missing signal never counts
# against an author:
#
# - Rejection burst: pull requests of theirs closed unmerged elsewhere within
# the last month. Volume of rejections in absolute terms separates spraying
# from ordinary contribution far better than a merge ratio does, since
# ratios reward authors who accumulate merges in trivial repositories.
# - Spray breadth: unrelated repositories they open pull requests against
# within one week. Breadth catches an agent on its first day, before any of
# its pull requests have been closed, and it comes from the event feed, so it
# also covers authors that the search API refuses to return.
# - Assistant voice: their recent comments across GitHub read as assistant
# output rather than as a developer talking, by section headings, bullet
# lists, em dash density or stock acknowledgement phrases.
# - Agent branch: the branch name carries an agent prefix.
#
# Authors that the organisations behind this repository already trust are left
# alone before any of that runs: public members of those organisations, and
# authors with a track record of pull requests merged into their repositories.
# Trust from a merge record rather than from a list of names keeps the exemption
# in step with who is actually contributing.
#
# Deliberately not used: account age, fork age, follower count, total pull
# request count and cross-repository merge ratio. All of them were measured
# against hand-labelled pull requests and either failed to separate or, in the
# case of the merge ratio, inverted on held-out data.
#
# The label is advisory, and it says the author's history is worth a look
# before reviewing in depth; it does not say the pull request is bad.
#
# The workflow only reads pull request and public activity metadata through the
# API, it never checks out or runs pull request code, so pull_request_target is
# safe here.
on: # zizmor: ignore[dangerous-triggers]
pull_request_target:
types: [opened]
permissions:
contents: read
pull-requests: write
jobs:
flag-pr-for-triage:
name: Label PR if the author's activity suggests unsupervised LLM use
runs-on: ubuntu-latest
steps:
- name: Score the author and label the PR
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const LABEL = 'needs triage';
const RETRIES = 5;
const RETRY_WAIT_MS = 60000;
const REJECTION_WINDOW_DAYS = 30;
const MIN_REJECTIONS = 1;
const MIN_COMMENTS = 2;
const MAX_REPOS_PER_WEEK = 2;
const VOICE = { structure: 0.10, emDashPerKChar: 0.30, acknowledgement: 0.40 };
const EVENT_PAGES = 3;
const TRUSTED_ORGS = ['scrapy', 'scrapy-plugins', 'scrapinghub', 'zytedata'];
const MIN_TRUSTED_MERGES = 10;
const AGENT_BRANCH = /^(agent|codex|claude|cursor|devin|copilot|jules|bot)[\/_-]/i;
const { owner, repo } = context.repo;
const pr = context.payload.pull_request;
const author = pr.user.login;
if (pr.user.type === 'Bot'
|| ['MEMBER', 'OWNER', 'COLLABORATOR'].includes(pr.author_association)) {
core.info(`Skipping PR #${pr.number} by ${author} (${pr.user.type}, ${pr.author_association}).`);
return;
}
// Rate and abuse limits reset on the order of a minute, so waiting
// is enough; other errors are not worth retrying.
const retriable = new Set([403, 429, 500, 502, 503, 504]);
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
async function withRetries(description, call) {
for (let attempt = 1; ; attempt++) {
try {
return await call();
} catch (error) {
if (!retriable.has(error.status) || attempt > RETRIES) throw error;
const reset = Number(error.response?.headers?.['x-ratelimit-reset']) * 1000 - Date.now();
const after = Number(error.response?.headers?.['retry-after']) * 1000;
const wait = Math.min(Math.max(after || reset || RETRY_WAIT_MS, RETRY_WAIT_MS), 15 * RETRY_WAIT_MS);
core.info(`${description} failed with ${error.status}, retrying in ${Math.round(wait / 1000)}s (attempt ${attempt}/${RETRIES}).`);
await sleep(wait);
}
}
}
// Accounts excluded from search, deleted users and the like leave a
// signal unmeasurable rather than negative.
const orNull = promise => promise.catch(error => {
if ([404, 410, 422].includes(error.status)) return null;
throw error;
});
// author_association only reports membership of the organisation
// that owns this repository, and only when it is public, so trust
// in the author is established here instead.
const trustedOrg = (await Promise.all(TRUSTED_ORGS.map(org =>
orNull(withRetries(`Checking public membership of ${org}`, () =>
github.rest.orgs.checkPublicMembershipForUser({ org, username: author }),
)).then(response => response && org),
))).find(Boolean);
if (trustedOrg) {
core.info(`Skipping PR #${pr.number} by ${author} (public member of ${trustedOrg}).`);
return;
}
// Repeating a qualifier narrows the search instead of widening it,
// hence the explicit disjunction.
const trustedMerges = await orNull(withRetries('Counting merged PRs in trusted organisations', () =>
github.rest.search.issuesAndPullRequests({
q: `author:${author} type:pr is:merged`
+ ` (${TRUSTED_ORGS.map(org => `org:${org}`).join(' OR ')})`,
advanced_search: 'true', per_page: 1,
}).then(response => response.data.total_count),
));
if (trustedMerges >= MIN_TRUSTED_MERGES) {
core.info(`Skipping PR #${pr.number} by ${author}`
+ ` (${trustedMerges} PR(s) merged into ${TRUSTED_ORGS.join(', ')}).`);
return;
}
const opened = new Date(pr.created_at);
const daysBefore = date => (opened - new Date(date)) / 86400000;
// Signal 1: pull requests closed unmerged elsewhere, recently.
const search = await orNull(withRetries('Searching for PRs by the author', () =>
github.rest.search.issuesAndPullRequests({
q: `author:${author} type:pr`, advanced_search: 'true',
sort: 'created', order: 'desc', per_page: 100,
}).then(response => response.data),
));
let rejections = null;
if (search) {
rejections = search.items.filter(item => {
const itemOwner = item.repository_url.split('/repos/')[1].split('/')[0].toLowerCase();
return itemOwner !== author.toLowerCase()
&& item.state === 'closed' && !item.pull_request?.merged_at
&& daysBefore(item.created_at) >= 0
&& daysBefore(item.created_at) <= REJECTION_WINDOW_DAYS;
}).map(item => item.html_url);
}
// Signal 2: how their recent comments across GitHub read.
const events = [];
for (let page = 1; page <= EVENT_PAGES; page++) {
const batch = await orNull(withRetries(`Reading public events page ${page}`, () =>
github.rest.activity.listPublicEventsForUser({
username: author, per_page: 100, page,
}).then(response => response.data),
));
if (!batch?.length) break;
events.push(...batch);
if (batch.length < 100) break;
}
const comments = events
.filter(event => ['IssueCommentEvent', 'PullRequestReviewCommentEvent'].includes(event.type))
.map(event => event.payload?.comment?.body)
.filter(Boolean);
// Signal 3: how many unrelated projects they open pull requests
// against in a single week. Breadth rather than volume: a focused
// contributor sends many pull requests to few repositories, while
// an unattended agent sprays a few across many. Taken from the
// event feed, which unlike search covers authors that search
// refuses to return.
const weeks = {};
for (const event of events) {
if (event.type !== 'PullRequestEvent' || event.payload?.action !== 'opened') continue;
const name = event.repo?.name;
if (!name || name.toLowerCase().startsWith(`${author.toLowerCase()}/`)) continue;
const week = Math.floor(new Date(event.created_at) / (7 * 86400000));
(weeks[week] ??= new Set()).add(name);
}
const breadth = events.length
? Math.max(0, ...Object.values(weeks).map(repos => repos.size))
: null;
const STRUCTURE = [/^\s*#{2,3}\s/m, /^\s*[-*]\s.+\n\s*[-*]\s/m, /\*\*[^*]+\*\*/, /```/];
const ACKNOWLEDGEMENT = [
/thanks for (the )?(review|feedback|pointing|catching|flagging|clarif)/i,
/you'?re (absolutely )?right/i, /great catch/i, /that makes sense/i,
/i'?ll (continue|investigate|update|submit|look into|make sure)/i,
/let me know (if|whether)/i, /happy to (update|adjust|revise|change)/i,
/i understand that/i, /thanks for your time/i, /just following up/i,
/hope (this|that) helps/i, /please let me know/i, /i'?ve (updated|addressed|fixed)/i,
];
let voice = null;
if (comments.length >= MIN_COMMENTS) {
const chars = comments.reduce((total, body) => total + body.length, 0);
const rate = patterns => comments.filter(body => patterns.some(re => re.test(body))).length / comments.length;
voice = {
comments: comments.length,
structure: rate(STRUCTURE),
acknowledgement: rate(ACKNOWLEDGEMENT),
emDashPerKChar: 1000 * comments.reduce((total, body) => total + (body.match(/—/g) || []).length, 0) / chars,
};
}
const reasons = [];
if (rejections && rejections.length >= MIN_REJECTIONS) {
reasons.push(`${rejections.length} PR(s) of theirs closed unmerged elsewhere in the last`
+ ` ${REJECTION_WINDOW_DAYS} days: ${rejections.slice(0, 10).join(' ')}`);
}
if (voice && (voice.structure > VOICE.structure
|| voice.emDashPerKChar > VOICE.emDashPerKChar
|| voice.acknowledgement > VOICE.acknowledgement)) {
reasons.push(`comment style over ${voice.comments} recent comments:`
+ ` ${(100 * voice.structure).toFixed(0)}% structured,`
+ ` ${(100 * voice.acknowledgement).toFixed(0)}% stock acknowledgements,`
+ ` ${voice.emDashPerKChar.toFixed(2)} em dashes per 1000 characters`);
}
if (breadth !== null && breadth > MAX_REPOS_PER_WEEK) {
reasons.push(`opened pull requests against ${breadth} unrelated repositories within a week`);
}
if (AGENT_BRANCH.test(pr.head?.ref || '')) {
reasons.push(`branch name carries an agent prefix: ${pr.head.ref}`);
}
await core.summary
.addHeading(`PR #${pr.number} by ${author}`, 3)
.addList([
rejections === null
? 'recent rejections elsewhere: unmeasurable, the author cannot be searched'
: `recent rejections elsewhere: ${rejections.length}`,
voice === null
? `comment style: unmeasurable, fewer than ${MIN_COMMENTS} recent comments found`
: `comment style: ${(100 * voice.structure).toFixed(0)}% structured,`
+ ` ${(100 * voice.acknowledgement).toFixed(0)}% stock acknowledgements,`
+ ` ${voice.emDashPerKChar.toFixed(2)} em dashes per 1000 characters`
+ ` over ${voice.comments} comments`,
breadth === null
? 'repositories per week: unmeasurable, no public events found'
: `repositories per week, at most: ${breadth}`,
`branch: ${pr.head?.ref ?? 'unknown'}`,
`verdict: ${reasons.length ? `labelled "${LABEL}"` : 'not labelled'}`,
])
.addRaw(reasons.length ? `\n${reasons.map(reason => `- ${reason}`).join('\n')}\n` : '')
.write();
if (!reasons.length) {
core.info(`Not labelling PR #${pr.number}.`);
return;
}
await withRetries('Adding the label', () =>
github.rest.issues.addLabels({ owner, repo, issue_number: pr.number, labels: [LABEL] }),
);
core.info(`Labelled PR #${pr.number}: ${reasons.join(' | ')}`);
scrapy-scrapy-5569de3/.github/workflows/publish.yml 0000664 0000000 0000000 00000002431 15250510124 0022441 0 ustar 00root root 0000000 0000000 name: Publish
permissions:
contents: read
on:
push:
tags:
- '[0-9]+.[0-9]+.[0-9]+'
concurrency:
group: ${{github.workflow}}-${{ github.ref }}
cancel-in-progress: true
jobs:
build:
name: Build distribution
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.14"
- run: |
python -m pip install --upgrade build
python -m build
- uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: python-package-distributions
path: dist/
publish:
name: Upload release to PyPI
needs:
- build
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/p/Scrapy
permissions:
id-token: write
steps:
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: python-package-distributions
path: dist/
- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
scrapy-scrapy-5569de3/.github/workflows/tests-macos.yml 0000664 0000000 0000000 00000004626 15250510124 0023245 0 ustar 00root root 0000000 0000000 name: macOS
permissions:
contents: read
on:
push:
branches:
- master
- '[0-9]+.[0-9]+'
pull_request:
types: [opened, synchronize, reopened, closed]
concurrency:
group: ${{github.workflow}}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
jobs:
tests:
if: github.event.action != 'closed'
name: tests (${{ matrix.python-version }}, ${{ matrix.env.TOXENV }})
runs-on: macos-latest
timeout-minutes: 15
env:
PYTEST_ADDOPTS: ${{ matrix.coverage && '-n logical' || '-n logical --no-cov' }}
# Make uv use the interpreter that actions/setup-python installed instead
# of downloading one of its own.
UV_PYTHON_PREFERENCE: only-system
strategy:
fail-fast: false
matrix:
include:
- python-version: "3.10"
env:
TOXENV: py
- python-version: "3.14"
env:
TOXENV: py
coverage: true
- python-version: "3.14"
env:
TOXENV: no-reactor
# 3.15 pre-release
- python-version: "3.15.0-rc.1"
env:
TOXENV: py
- python-version: "3.15.0-rc.1"
env:
TOXENV: no-reactor
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
tox.ini
- name: Install mitmproxy
env:
# mitmproxy needs a newer Python than the oldest matrix entries, so let
# uv download one where no system interpreter is new enough.
UV_PYTHON_PREFERENCE: system
run: uv tool install mitmproxy
- name: Run tests
env: ${{ matrix.env }}
run: uvx --with tox-uv tox
- name: Upload coverage report
if: ${{ matrix.coverage }}
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
- name: Upload test results
if: ${{ !cancelled() }}
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
report_type: test_results
scrapy-scrapy-5569de3/.github/workflows/tests-ubuntu.yml 0000664 0000000 0000000 00000010400 15250510124 0023450 0 ustar 00root root 0000000 0000000 name: Ubuntu
permissions:
contents: read
on:
push:
branches:
- master
- '[0-9]+.[0-9]+'
pull_request:
types: [opened, synchronize, reopened, closed]
concurrency:
group: ${{github.workflow}}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
jobs:
tests:
if: github.event.action != 'closed'
name: tests (${{ matrix.python-version }}, ${{ matrix.env.TOXENV }})
runs-on: ubuntu-latest
timeout-minutes: 20
env:
PYTEST_ADDOPTS: ${{ matrix.coverage && '-n logical' || '-n logical --no-cov' }}
# Make uv use the interpreter that actions/setup-python installed instead
# of downloading one of its own.
UV_PYTHON_PREFERENCE: only-system
strategy:
fail-fast: false
matrix:
include:
- python-version: "3.10"
env:
TOXENV: py
- python-version: "3.11"
env:
TOXENV: py
- python-version: "3.12"
env:
TOXENV: py
- python-version: "3.13"
env:
TOXENV: py
- python-version: "3.14"
env:
TOXENV: py
coverage: true
- python-version: "3.14"
env:
TOXENV: default-reactor
coverage: true
- python-version: "3.14"
env:
TOXENV: no-reactor
coverage: true
# min deps
- python-version: "3.10.19"
env:
TOXENV: min
coverage: true
- python-version: "3.10.19"
env:
TOXENV: min-default-reactor
coverage: true
# pinned due to https://github.com/pypy/pypy/issues/5388
- python-version: pypy3.11-7.3.20
env:
TOXENV: min-pypy3
- python-version: "3.10.19"
env:
TOXENV: min-extra-deps
coverage: true
- python-version: "3.10.19"
env:
TOXENV: min-botocore
coverage: true
- python-version: "3.14"
env:
TOXENV: extra-deps
coverage: true
- python-version: "3.14"
env:
TOXENV: no-reactor-extra-deps
coverage: true
# pinned due to https://github.com/pypy/pypy/issues/5388
- python-version: pypy3.11-7.3.20
env:
TOXENV: pypy3-extra-deps
coverage: true
- python-version: "3.14"
env:
TOXENV: botocore
coverage: true
# 3.15 pre-release
- python-version: "3.15.0-rc.1"
env:
TOXENV: py
- python-version: "3.15.0-rc.1"
env:
TOXENV: default-reactor
- python-version: "3.15.0-rc.1"
env:
TOXENV: no-reactor
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
- name: Install system libraries
if: contains(matrix.python-version, 'pypy') || contains(matrix.env.TOXENV, 'min')
run: |
sudo apt-get update
sudo apt-get install libxml2-dev libxslt-dev
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
tox.ini
- name: Install mitmproxy
env:
# mitmproxy needs a newer Python than the oldest matrix entries, so let
# uv download one where no system interpreter is new enough.
UV_PYTHON_PREFERENCE: system
# mitmproxy has no PyPy wheels, so run it on CPython regardless of the
# interpreter under test.
run: uv tool install --python cpython mitmproxy
- name: Run tests
env: ${{ matrix.env }}
run: uvx --with tox-uv tox
- name: Upload coverage report
if: ${{ matrix.coverage }}
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
- name: Upload test results
if: ${{ !cancelled() }}
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
report_type: test_results
scrapy-scrapy-5569de3/.github/workflows/tests-vcs-deps.yml 0000664 0000000 0000000 00000003131 15250510124 0023655 0 ustar 00root root 0000000 0000000 name: VCS dependencies
permissions:
contents: read
on:
schedule:
- cron: '0 4 * * *'
workflow_dispatch:
concurrency:
group: ${{github.workflow}}-${{ github.ref }}
cancel-in-progress: true
jobs:
tests:
name: tests
runs-on: ubuntu-latest
timeout-minutes: 30
env:
# A development branch of a dependency can make a test hang forever, so
# tests get a time limit here that they do not need elsewhere.
PYTEST_ADDOPTS: -n logical --no-cov --timeout=120
TOXENV: vcs-deps
UV_PYTHON_PREFERENCE: only-system
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.15"
allow-prereleases: true
# Dependencies that ship wheels on PyPI are built from source here, so
# their build dependencies are needed: libxml2 and libxslt for lxml,
# libjpeg and zlib for Pillow, and autotools for the libuv bundled in
# uvloop.
- name: Install system libraries
run: |
sudo apt-get update
sudo apt-get install automake libjpeg-dev libtool libxml2-dev libxslt-dev zlib1g-dev
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
tox.ini
- name: Install mitmproxy
run: uv tool install --python cpython mitmproxy
- name: Run tests
run: uvx --with tox-uv tox
scrapy-scrapy-5569de3/.github/workflows/tests-windows.yml 0000664 0000000 0000000 00000005370 15250510124 0023632 0 ustar 00root root 0000000 0000000 name: Windows
permissions:
contents: read
on:
push:
branches:
- master
- '[0-9]+.[0-9]+'
pull_request:
types: [opened, synchronize, reopened, closed]
concurrency:
group: ${{github.workflow}}-${{ github.head_ref || github.ref }}
cancel-in-progress: true
jobs:
tests:
if: github.event.action != 'closed'
name: tests (${{ matrix.python-version }}, ${{ matrix.env.TOXENV }})
runs-on: windows-latest
timeout-minutes: 20
env:
PYTEST_ADDOPTS: ${{ matrix.coverage && '-n logical' || '-n logical --no-cov' }}
# Make uv use the interpreter that actions/setup-python installed instead
# of downloading one of its own.
UV_PYTHON_PREFERENCE: only-system
strategy:
fail-fast: false
matrix:
include:
- python-version: "3.10"
env:
TOXENV: py
- python-version: "3.14"
env:
TOXENV: py
coverage: true
- python-version: "3.14"
env:
TOXENV: no-reactor
# min deps
- python-version: "3.10.11"
env:
TOXENV: min
- python-version: "3.10.11"
env:
TOXENV: min-extra-deps
- python-version: "3.14"
env:
TOXENV: extra-deps
# 3.15 pre-release
- python-version: "3.15.0-rc.1"
env:
TOXENV: py
- python-version: "3.15.0-rc.1"
env:
TOXENV: default-reactor
- python-version: "3.15.0-rc.1"
env:
TOXENV: no-reactor
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}
- name: Set up uv
uses: astral-sh/setup-uv@20cfd1bf945f4377ade1205e4dbc17946fc9a30d # v10.0.1
with:
cache-dependency-glob: |
pyproject.toml
tox.ini
- name: Install mitmproxy
env:
# mitmproxy needs a newer Python than the oldest matrix entries, so let
# uv download one where no system interpreter is new enough.
UV_PYTHON_PREFERENCE: system
run: uv tool install mitmproxy
- name: Run tests
env: ${{ matrix.env }}
run: uvx --with tox-uv tox
- name: Upload coverage report
if: ${{ matrix.coverage }}
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
- name: Upload test results
if: ${{ !cancelled() }}
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
report_type: test_results
scrapy-scrapy-5569de3/.gitignore 0000664 0000000 0000000 00000000514 15250510124 0016643 0 ustar 00root root 0000000 0000000 /.vagrant
/scrapy.iml
*.pyc
_trial_temp*
dropin.cache
docs/_build
*egg-info
.tox/
venv/
.venv/
build/
dist/
.idea/
.vscode/
htmlcov/
.pytest_cache/
.coverage
.coverage.*
coverage.*
*.junit.xml
test-output.*
.cache/
.mypy_cache/
/tests/keys/localhost.crt
/tests/keys/localhost.key
# Windows
Thumbs.db
# OSX miscellaneous
.DS_Store
scrapy-scrapy-5569de3/.pre-commit-config.yaml 0000664 0000000 0000000 00000001510 15250510124 0021131 0 ustar 00root root 0000000 0000000 exclude: |
(?x)(
^docs/_static|
^docs/_tests|
^tests/sample_data
)
repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.15.20
hooks:
- id: ruff-check
args: [ --fix ]
- id: ruff-format
- repo: https://github.com/adamchainz/blacken-docs
rev: 1.20.0
hooks:
- id: blacken-docs
additional_dependencies:
- black==26.5.1
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
hooks:
- id: end-of-file-fixer
- id: trailing-whitespace
- repo: https://github.com/sphinx-contrib/sphinx-lint
rev: v1.0.2
hooks:
- id: sphinx-lint
- repo: https://github.com/scrapy/sphinx-scrapy
rev: 0.9.0
hooks:
- id: sphinx-scrapy
- repo: https://github.com/zizmorcore/zizmor-pre-commit
rev: v1.28.0
hooks:
- id: zizmor
args: [--no-progress, --fix]
scrapy-scrapy-5569de3/.readthedocs.yml 0000664 0000000 0000000 00000000325 15250510124 0017741 0 ustar 00root root 0000000 0000000 version: 2
build:
os: ubuntu-24.04
tools:
python: "3.14"
commands:
- pip install tox
- tox -e docs
- mkdir -p $READTHEDOCS_OUTPUT/html
- cp -a docs/_build/all/. $READTHEDOCS_OUTPUT/html/
scrapy-scrapy-5569de3/AUTHORS 0000664 0000000 0000000 00000002404 15250510124 0015723 0 ustar 00root root 0000000 0000000 Scrapy was brought to life by Shane Evans while hacking a scraping framework
prototype for Mydeco (mydeco.com). It soon became maintained, extended and
improved by Insophia (insophia.com), with the initial sponsorship of Mydeco to
bootstrap the project. In mid-2011, Scrapinghub (now Zyte) became the new
official maintainer.
Here is the list of the primary authors & contributors:
* Pablo Hoffman
* Daniel Graña
* Martin Olveyra
* Gabriel García
* Michael Cetrulo
* Artem Bogomyagkov
* Damian Canabal
* Andres Moreira
* Ismael Carnales
* Matías Aguirre
* German Hoffmann
* Anibal Pacheco
* Bruno Deferrari
* Shane Evans
* Ezequiel Rivero
* Patrick Mezard
* Rolando Espinoza
* Ping Yin
* Lucian Ursu
* Shuaib Khan
* Didier Deshommes
* Vikas Dhiman
* Jochen Maes
* Darian Moody
* Jordi Lonch
* Zuhao Wan
* Steven Almeroth
* Tom Mortimer-Jones
* Chris Tilden
* Alexandr N Zamaraev
* Emanuel Schorsch
* Michal Danilak
* Natan Lao
* Hasnain Lakhani
* Pedro Faustino
* Alex Cepoi
* Ilya Baryshev
* Libor Nenadál
* Jae-Myoung Yu
* Vladislav Poluhin
* Marc Abramowitz
* Valentin-Costel Hăloiu
* Jason Yeo
* Сергей Прохоров
* Simon Ratne
* Julien Duponchelle
* Jochen Maes
* Vikas Dhiman
* Juan Picca
* Nicolás Ramírez
scrapy-scrapy-5569de3/CITATION.cff 0000664 0000000 0000000 00000000253 15250510124 0016545 0 ustar 00root root 0000000 0000000 cff-version: 1.2.0
message: If you use Scrapy in published research, please cite it as below.
title: Scrapy
authors:
- name: Scrapy contributors
url: https://scrapy.org
scrapy-scrapy-5569de3/CODE_OF_CONDUCT.md 0000664 0000000 0000000 00000012555 15250510124 0017462 0 ustar 00root root 0000000 0000000
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, caste, color, religion, or sexual
identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the overall
community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or advances of
any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email address,
without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
opensource@zyte.com.
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series of
actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or permanent
ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within the
community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.1, available at
[https://www.contributor-covenant.org/version/2/1/code_of_conduct.html][v2.1].
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available at
[https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.1]: https://www.contributor-covenant.org/version/2/1/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/diversity
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
scrapy-scrapy-5569de3/CONTRIBUTING.md 0000664 0000000 0000000 00000000457 15250510124 0017112 0 ustar 00root root 0000000 0000000 The guidelines for contributing are available here:
https://docs.scrapy.org/en/master/contributing.html
Please do not abuse the issue tracker for support questions.
If your issue topic can be rephrased to "How to ...?", please use the
support channels to get it answered: https://scrapy.org/community/
scrapy-scrapy-5569de3/INSTALL.md 0000664 0000000 0000000 00000000233 15250510124 0016301 0 ustar 00root root 0000000 0000000 For information about installing Scrapy see:
* [Local docs](docs/intro/install.rst)
* [Online docs](https://docs.scrapy.org/en/latest/intro/install.html)
scrapy-scrapy-5569de3/LICENSE 0000664 0000000 0000000 00000002755 15250510124 0015671 0 ustar 00root root 0000000 0000000 Copyright (c) Scrapy developers.
All rights reserved.
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions, and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions, and the following disclaimer in the
documentation and/or other materials provided with the distribution.
3. Neither the name of Scrapy nor the names of its contributors may be used
to endorse or promote products derived from this software without
specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
scrapy-scrapy-5569de3/NEWS 0000664 0000000 0000000 00000000022 15250510124 0015344 0 ustar 00root root 0000000 0000000 See docs/news.rst
scrapy-scrapy-5569de3/README.rst 0000664 0000000 0000000 00000003401 15250510124 0016340 0 ustar 00root root 0000000 0000000 |logo|
.. |logo| image:: https://raw.githubusercontent.com/scrapy/scrapy/master/docs/_static/logo.svg
:target: https://scrapy.org
:alt: Scrapy
:width: 480px
|version| |python_version| |tests| |coverage| |conda| |deepwiki|
.. |version| image:: https://img.shields.io/pypi/v/Scrapy.svg
:target: https://pypi.org/pypi/Scrapy
:alt: PyPI Version
.. |python_version| image:: https://img.shields.io/pypi/pyversions/Scrapy.svg
:target: https://pypi.org/pypi/Scrapy
:alt: Supported Python Versions
.. |tests| image:: https://img.shields.io/github/check-runs/scrapy/scrapy/master?label=tests
:target: https://github.com/scrapy/scrapy/actions?query=branch%3Amaster
:alt: Tests
.. |coverage| image:: https://img.shields.io/codecov/c/github/scrapy/scrapy/master.svg
:target: https://codecov.io/github/scrapy/scrapy?branch=master
:alt: Coverage report
.. |conda| image:: https://anaconda.org/conda-forge/scrapy/badges/version.svg
:target: https://anaconda.org/conda-forge/scrapy
:alt: Conda Version
.. |deepwiki| image:: https://deepwiki.com/badge.svg
:target: https://deepwiki.com/scrapy/scrapy
:alt: Ask DeepWiki
Scrapy_ is a web scraping framework to extract structured data from websites.
It is cross-platform, and requires Python 3.10+. It is maintained by Zyte_
(formerly Scrapinghub) and `many other contributors`_.
.. _many other contributors: https://github.com/scrapy/scrapy/graphs/contributors
.. _Scrapy: https://scrapy.org/
.. _Zyte: https://www.zyte.com/
Install with:
.. code:: bash
pip install scrapy
And follow the documentation_ to learn how to use it.
.. _documentation: https://docs.scrapy.org/en/latest/
If you wish to contribute, see Contributing_.
.. _Contributing: https://docs.scrapy.org/en/master/contributing.html
scrapy-scrapy-5569de3/SECURITY.md 0000664 0000000 0000000 00000000463 15250510124 0016447 0 ustar 00root root 0000000 0000000 # Security Policy
## Supported Versions
| Version | Supported |
| ------- | ------------------ |
| 2.19.x | :white_check_mark: |
| < 2.19.x | :x: |
## Reporting a Vulnerability
Please report the vulnerability using https://github.com/scrapy/scrapy/security/advisories/new.
scrapy-scrapy-5569de3/codecov.yml 0000664 0000000 0000000 00000000120 15250510124 0017011 0 ustar 00root root 0000000 0000000 comment:
layout: "header, diff, tree"
coverage:
status:
project: false
scrapy-scrapy-5569de3/conftest.py 0000664 0000000 0000000 00000017542 15250510124 0017063 0 ustar 00root root 0000000 0000000 from __future__ import annotations
import os
from importlib.util import find_spec
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from twisted.web.http import H2_ENABLED
from scrapy.core.engine import ExecutionEngine
from scrapy.utils.reactor import set_asyncio_event_loop_policy
from scrapy.utils.reactorless import install_reactor_import_hook
from tests.keys import generate_keys
from tests.mockserver.http import MockServer
from tests.mockserver.mitm_proxy import MitmProxy, mitmdump_cmd
if TYPE_CHECKING:
from collections.abc import Generator
def _py_files(folder):
return (str(p) for p in Path(folder).rglob("*.py"))
collect_ignore = [
# may need extra deps
"docs/_ext",
# contains scripts to be run by tests/test_crawler_subprocess.py::AsyncCrawlerProcessSubprocess
*_py_files("tests/AsyncCrawlerProcess"),
# contains scripts to be run by tests/test_crawler_subprocess.py::AsyncCrawlerRunnerSubprocess
*_py_files("tests/AsyncCrawlerRunner"),
# contains scripts to be run by tests/test_crawler_subprocess.py::CrawlerProcessSubprocess
*_py_files("tests/CrawlerProcess"),
# contains scripts to be run by tests/test_crawler_subprocess.py::CrawlerRunnerSubprocess
*_py_files("tests/CrawlerRunner"),
]
base_dir = Path(__file__).parent
ignore_file_path = base_dir / "tests" / "ignores.txt"
with ignore_file_path.open(encoding="utf-8") as reader:
for line in reader:
file_path = line.strip()
if file_path and file_path[0] != "#":
collect_ignore.append(file_path)
if not H2_ENABLED:
collect_ignore.extend(
(
"scrapy/core/downloader/handlers/http2.py",
*_py_files("scrapy/core/_http2"),
)
)
if find_spec("httpx2") is None and find_spec("httpx") is None:
collect_ignore.append("scrapy/core/downloader/handlers/_httpx.py")
if find_spec("pytest_codspeed") is None:
collect_ignore.append("tests/benchmarks")
def pytest_addoption(parser, pluginmanager):
if pluginmanager.hasplugin("twisted"):
return
# add the full choice set so that pytest doesn't complain about invalid choices in some cases
parser.addoption(
"--reactor",
default="none",
choices=["asyncio", "default", "none"],
)
@pytest.fixture(autouse=True)
def fast_engine_heartbeat(monkeypatch: pytest.MonkeyPatch) -> None:
"""Shorten the interval at which the engine checks for work while idle.
Whenever nothing else wakes the engine up, e.g. while a component sleeps,
it stays idle until its next heartbeat, so tests that go through that wait
pay the whole interval.
"""
monkeypatch.setattr(ExecutionEngine, "_SLOT_HEARTBEAT_INTERVAL", 0.1)
@pytest.fixture(scope="session")
def mockserver() -> Generator[MockServer]:
with MockServer() as mockserver:
yield mockserver
@pytest.fixture(scope="session")
def _mitm_proxies() -> Generator[dict[str, tuple[MitmProxy, str]]]:
proxies: dict[str, tuple[MitmProxy, str]] = {}
try:
yield proxies
finally:
for proxy, _url in proxies.values():
proxy.stop()
@pytest.fixture # function scope because it modifies os.environ
def proxy_server(
request: pytest.FixtureRequest,
monkeypatch: pytest.MonkeyPatch,
_mitm_proxies: dict[str, tuple[MitmProxy, str]],
) -> str:
kind: str = request.param
if kind not in _mitm_proxies:
proxy = MitmProxy(mode="socks5" if kind == "socks5" else None)
_mitm_proxies[kind] = (proxy, proxy.start())
_, url = _mitm_proxies[kind]
if kind == "https":
url = url.replace("http://", "https://")
monkeypatch.setenv("http_proxy", url)
monkeypatch.setenv("https_proxy", url)
return kind
@pytest.fixture(scope="session")
def reactor_pytest(request) -> str:
return request.config.getoption("--reactor")
def pytest_configure(config):
if config.getoption("--reactor") in {"asyncio", "none"}:
# Needed on Windows to switch from proactor to selector, which supports
# add_reader/add_writer (required by the Twisted asyncio reactor, and by
# tests that register their own readers) and is what Twisted expects.
set_asyncio_event_loop_policy()
if config.getoption("--reactor") == "none":
install_reactor_import_hook()
# Test modules made up mostly of tests that spawn a subprocess per test (see
# tests/utils/cmdline.py). Collected in file order, they form one contiguous
# block. pytest-xdist's initial scheduling (LoadScheduling.schedule() in
# xdist/scheduler/load.py) hands each worker one contiguous slice of the
# collection, so whichever worker's slice overlaps the block ends up running
# most of it while workers with lighter slices drain them and sit idle for
# the rest of the run. Spreading these tests evenly through the whole
# collection instead means every slice gets a proportional share.
_SUBPROCESS_HEAVY_SUFFIXES = (
"tests/test_cmdline/__init__.py",
"tests/test_cmdline_crawl_with_pipeline/__init__.py",
"tests/test_command_check.py",
"tests/test_command_crawl.py",
"tests/test_command_fetch.py",
"tests/test_command_genspider.py",
"tests/test_command_parse.py",
"tests/test_command_runspider.py",
"tests/test_command_shell.py",
"tests/test_command_startproject.py",
"tests/test_command_version.py",
"tests/test_commands.py",
"tests/test_crawler_subprocess.py",
)
def _is_subprocess_heavy(item):
return item.path.as_posix().endswith(_SUBPROCESS_HEAVY_SUFFIXES)
def _interleave_evenly(items, is_heavy):
"""Merge *light* and *heavy* items so heavy ones land at roughly even
intervals throughout the result, instead of clustered together."""
light = [item for item in items if not is_heavy(item)]
heavy = [item for item in items if is_heavy(item)]
result = []
light_i = heavy_i = 0
while light_i < len(light) or heavy_i < len(heavy):
heavy_progress = heavy_i / len(heavy) if heavy else 1.0
light_progress = light_i / len(light) if light else 1.0
if heavy_i < len(heavy) and heavy_progress <= light_progress:
result.append(heavy[heavy_i])
heavy_i += 1
else:
result.append(light[light_i])
light_i += 1
return result
def pytest_collection_modifyitems(items):
items[:] = _interleave_evenly(items, _is_subprocess_heavy)
for item in items:
if item.get_closest_marker("requires_internet"):
# Requests to real websites fail every now and then in CI for
# reasons unrelated to the code under test.
item.add_marker(pytest.mark.flaky(reruns=2, reruns_delay=5))
def pytest_runtest_setup(item):
# Skip tests based on reactor markers
reactor = item.config.getoption("--reactor")
if item.get_closest_marker("requires_reactor") and reactor == "none":
pytest.skip('This test is only run when the --reactor value is not "none"')
if item.get_closest_marker("only_asyncio") and reactor not in {"asyncio", "none"}:
pytest.skip(
'This test is only run when the --reactor value is "asyncio" (default) or "none"'
)
if item.get_closest_marker("only_not_asyncio") and reactor in {"asyncio", "none"}:
pytest.skip(
'This test is only run when the --reactor value is not "asyncio" (default) or "none"'
)
# Skip tests requiring optional dependencies
optional_deps = [
"uvloop",
"botocore",
"boto3",
]
for module in optional_deps:
if item.get_closest_marker(f"requires_{module}") and find_spec(module) is None:
pytest.skip(f"{module} is not installed")
if item.get_closest_marker("requires_mitmproxy") and mitmdump_cmd() is None:
pytest.skip("mitmdump is not available")
# Generate localhost certificate files, needed by some tests (but only once if xdist is used)
if "PYTEST_XDIST_WORKER" not in os.environ:
generate_keys()
scrapy-scrapy-5569de3/docs/ 0000775 0000000 0000000 00000000000 15250510124 0015603 5 ustar 00root root 0000000 0000000 scrapy-scrapy-5569de3/docs/Makefile 0000664 0000000 0000000 00000001171 15250510124 0017243 0 ustar 00root root 0000000 0000000 # Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = build
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
scrapy-scrapy-5569de3/docs/_ext/ 0000775 0000000 0000000 00000000000 15250510124 0016542 5 ustar 00root root 0000000 0000000 scrapy-scrapy-5569de3/docs/_ext/scrapydocs.py 0000664 0000000 0000000 00000012450 15250510124 0021270 0 ustar 00root root 0000000 0000000 # pylint: disable=import-error
from collections.abc import Sequence
from operator import itemgetter
from typing import Any, TypedDict
from docutils import nodes
from docutils.nodes import Element, General, Node, document
from docutils.parsers.rst import Directive
from sphinx.application import Sphinx
from sphinx.util.nodes import make_refnode
class SettingData(TypedDict):
docname: str
setting_name: str
refid: str
class SettingslistNode(General, Element):
pass
class SettingsListDirective(Directive):
def run(self) -> Sequence[Node]:
return [SettingslistNode()]
def is_setting_index(node: Node) -> bool:
if node.tagname == "index" and node["entries"]: # type: ignore[index,attr-defined]
# index entries for setting directives look like:
# [('pair', 'SETTING_NAME; setting', 'std:setting-SETTING_NAME', '')]
entry_type, info, _ = node["entries"][0][:3] # type: ignore[index]
return entry_type == "pair" and info.endswith("; setting")
return False
def get_setting_name_and_refid(node: Node) -> tuple[str, str]:
"""Extract setting name from directive index node"""
_, info, refid = node["entries"][0][:3] # type: ignore[index]
return info.replace("; setting", ""), refid
def collect_scrapy_settings_refs(app: Sphinx, doctree: document) -> None:
env = app.builder.env
if not hasattr(env, "scrapy_all_settings"):
emptyList: list[SettingData] = []
env.scrapy_all_settings = emptyList # type: ignore[attr-defined]
for node in doctree.findall(is_setting_index):
setting_name, refid = get_setting_name_and_refid(node)
env.scrapy_all_settings.append( # type: ignore[attr-defined]
SettingData(
docname=env.docname,
setting_name=setting_name,
refid=refid,
)
)
def make_setting_element(
setting_data: SettingData, app: Sphinx, fromdocname: str
) -> Any:
refnode = make_refnode(
app.builder,
fromdocname,
todocname=setting_data["docname"],
targetid=setting_data["refid"],
child=nodes.Text(setting_data["setting_name"]),
)
p = nodes.paragraph()
p += refnode
item = nodes.list_item()
item += p
return item
def make_setting_markdown_item(
setting_data: SettingData, app: Sphinx, fromdocname: str
) -> str:
uri = app.builder.get_relative_uri(fromdocname, setting_data["docname"])
if uri.startswith("#"):
target = f"#{setting_data['refid']}"
else:
target = f"{uri}#{setting_data['refid']}"
return f"* [{setting_data['setting_name']}]({target})"
def _iter_sorted_settings(env: Any, fromdocname: str) -> list[SettingData]:
return [
d
for d in sorted(env.scrapy_all_settings, key=itemgetter("setting_name")) # type: ignore[attr-defined]
if fromdocname != d["docname"]
]
def replace_settingslist_nodes(
app: Sphinx, doctree: document, fromdocname: str
) -> None:
env = app.builder.env
for node in doctree.findall(SettingslistNode):
settings_list = nodes.bullet_list()
settings_list.extend(
[
make_setting_element(d, app, fromdocname)
for d in _iter_sorted_settings(env, fromdocname)
]
)
node.replace_self(settings_list)
def visit_settingslist_node_markdown(translator: Any, _node: Node) -> None:
builder = translator.builder
env = builder.env
fromdocname = getattr(builder, "current_doc_name", env.docname)
lines = [
make_setting_markdown_item(setting_data, builder.app, fromdocname)
for setting_data in _iter_sorted_settings(env, fromdocname)
]
if lines:
translator.add("\n".join(lines), prefix_eol=2, suffix_eol=2)
raise nodes.SkipNode
def depart_settingslist_node_markdown(_translator: Any, _node: Node) -> None:
return None
def source_role(
name, rawtext, text: str, lineno, inliner, options=None, content=None
) -> tuple[list[Any], list[Any]]:
ref = "https://github.com/scrapy/scrapy/blob/master/" + text
node = nodes.reference(rawtext, text, refuri=ref, **options)
return [node], []
def commit_role(
name, rawtext, text: str, lineno, inliner, options=None, content=None
) -> tuple[list[Any], list[Any]]:
ref = "https://github.com/scrapy/scrapy/commit/" + text
node = nodes.reference(rawtext, "commit " + text, refuri=ref)
return [node], []
def rev_role(
name, rawtext, text: str, lineno, inliner, options=None, content=None
) -> tuple[list[Any], list[Any]]:
ref = "http://hg.scrapy.org/scrapy/changeset/" + text
node = nodes.reference(rawtext, "r" + text, refuri=ref)
return [node], []
def setup(app: Sphinx) -> dict[str, Any]:
app.add_role("source", source_role)
app.add_role("commit", commit_role)
app.add_role("rev", rev_role)
app.add_node(
SettingslistNode,
markdown=(visit_settingslist_node_markdown, depart_settingslist_node_markdown),
singlemarkdown=(
visit_settingslist_node_markdown,
depart_settingslist_node_markdown,
),
)
app.add_directive("settingslist", SettingsListDirective)
app.connect("doctree-read", collect_scrapy_settings_refs)
app.connect("doctree-resolved", replace_settingslist_nodes)
return {"parallel_read_safe": True}
scrapy-scrapy-5569de3/docs/_ext/scrapyfixautodoc.py 0000664 0000000 0000000 00000001207 15250510124 0022503 0 ustar 00root root 0000000 0000000 """
Must be included after 'sphinx.ext.autodoc'. Fixes unwanted 'alias of' behavior.
https://github.com/sphinx-doc/sphinx/issues/4422
"""
from typing import Any
# pylint: disable=import-error
from sphinx.application import Sphinx
def maybe_skip_member(app: Sphinx, what, name: str, obj, skip: bool, options) -> bool:
if not skip:
# autodoc was generating the text "alias of" for the following members
return name in {"default_item_class", "default_selector_class"}
return skip
def setup(app: Sphinx) -> dict[str, Any]:
app.connect("autodoc-skip-member", maybe_skip_member)
return {"parallel_read_safe": True}
scrapy-scrapy-5569de3/docs/_static/ 0000775 0000000 0000000 00000000000 15250510124 0017231 5 ustar 00root root 0000000 0000000 scrapy-scrapy-5569de3/docs/_static/custom.css 0000664 0000000 0000000 00000003227 15250510124 0021261 0 ustar 00root root 0000000 0000000 /* Move lists closer to their introducing paragraph */
.rst-content .section ol p, .rst-content .section ul p {
margin-bottom: 0px;
}
.rst-content p + ol, .rst-content p + ul {
margin-top: -18px; /* Compensates margin-top: 24px of p */
}
.rst-content dl p + ol, .rst-content dl p + ul {
margin-top: -6px; /* Compensates margin-top: 12px of p */
}
/*override some styles in
sphinx-rtd-dark-mode/static/dark_mode_css/general.css*/
.theme-switcher {
right: 0.4em !important;
top: 0.6em !important;
-webkit-box-shadow: 0px 3px 14px 4px rgba(0, 0, 0, 0.30) !important;
box-shadow: 0px 3px 14px 4px rgba(0, 0, 0, 0.30) !important;
height: 2em !important;
width: 2em !important;
}
/*place the toggle button for dark mode
at the bottom right corner on small screens*/
@media (max-width: 768px) {
.theme-switcher {
right: 0.4em !important;
bottom: 2.6em !important;
top: auto !important;
}
}
/*persist blue color at the top left used in
default rtd theme*/
html[data-theme="dark"] .wy-side-nav-search,
html[data-theme="dark"] .wy-nav-top {
background-color: #1d577d !important;
}
/*all the styles below used to present
API objects nicely in dark mode*/
html[data-theme="dark"] .sig.sig-object {
border-left-color: #3e4446 !important;
background-color: #202325 !important
}
html[data-theme="dark"] .sig-name,
html[data-theme="dark"] .sig-prename,
html[data-theme="dark"] .property,
html[data-theme="dark"] .sig-param,
html[data-theme="dark"] .sig-paren,
html[data-theme="dark"] .sig-return-icon,
html[data-theme="dark"] .sig-return-typehint,
html[data-theme="dark"] .optional {
color: #e8e6e3 !important
}
scrapy-scrapy-5569de3/docs/_static/logo.svg 0000664 0000000 0000000 00000016737 15250510124 0020730 0 ustar 00root root 0000000 0000000
scrapy-scrapy-5569de3/docs/_static/selectors-sample1.html 0000664 0000000 0000000 00000001233 15250510124 0023461 0 ustar 00root root 0000000 0000000
Example website
scrapy-scrapy-5569de3/docs/_templates/ 0000775 0000000 0000000 00000000000 15250510124 0017740 5 ustar 00root root 0000000 0000000 scrapy-scrapy-5569de3/docs/_templates/layout.html 0000664 0000000 0000000 00000001523 15250510124 0022144 0 ustar 00root root 0000000 0000000 {% extends "!layout.html" %}
{# Overridden to include a link to scrapy.org, not just to the docs root #}
{%- block sidebartitle %}
{# the logo helper function was removed in Sphinx 6 and deprecated since Sphinx 4 #}
{# the master_doc variable was renamed to root_doc in Sphinx 4 (master_doc still exists in later Sphinx versions) #}
{%- set _logo_url = logo_url|default(pathto('_static/' + (logo or ""), 1)) %}
{%- set _root_doc = root_doc|default(master_doc) %}
scrapy.org / docs
{%- if READTHEDOCS or DEBUG %}
{%- if theme_version_selector or theme_language_selector %}
“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein
(about)
“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”by Albert Einstein
(about)
scrapy-scrapy-5569de3/docs/conf.py 0000664 0000000 0000000 00000015611 15250510124 0017106 0 ustar 00root root 0000000 0000000 # Configuration file for the Sphinx documentation builder.
#
# For the full list of built-in configuration values, see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
import os
import sys
from collections.abc import Sequence
from pathlib import Path
# If your extensions are in another directory, add it here. If the directory
# is relative to the documentation root, use Path.absolute to make it absolute.
sys.path.append(str(Path(__file__).parent / "_ext"))
sys.path.insert(0, str(Path(__file__).parent.parent))
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
project = "Scrapy"
project_copyright = "Scrapy developers"
author = "Scrapy developers"
# -- General configuration ---------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
extensions = [
"notfound.extension",
"scrapydocs",
"sphinx_scrapy",
"scrapyfixautodoc", # Must be after "sphinx.ext.autodoc"
"sphinx.ext.coverage",
"sphinx_reredirects",
"sphinx_rtd_dark_mode",
]
redirects = {
"topics/broad-crawls": "optimize.html#broad-crawls",
}
templates_path = ["_templates"]
exclude_patterns = ["build", "Thumbs.db", ".DS_Store"]
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
try:
import scrapy
version = ".".join(map(str, scrapy.version_info[:2]))
release = scrapy.__version__
except ImportError:
version = ""
release = ""
suppress_warnings = ["epub.unknown_project_files"]
# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
html_theme = "sphinx_rtd_theme"
html_static_path = ["_static"]
html_last_updated_fmt = "%b %d, %Y"
html_css_files = [
"custom.css",
]
html_context = {
"display_github": True,
"github_user": "scrapy",
"github_repo": "scrapy",
"github_version": "master",
"conf_py_path": "/docs/",
}
# Set canonical URL from the Read the Docs Domain
html_baseurl = os.environ.get("READTHEDOCS_CANONICAL_URL", "")
# -- Options for LaTeX output ------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-latex-output
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, document class [howto/manual]).
latex_documents = [
("index", "Scrapy.tex", "Scrapy Documentation", "Scrapy developers", "manual"),
]
# -- Options for the linkcheck builder ---------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-the-linkcheck-builder
linkcheck_ignore = [
r"http://localhost:\d+",
"http://hg.scrapy.org",
r"https://github.com/scrapy/scrapy/commit/\w+",
r"https://github.com/scrapy/scrapy/issues/\d+",
]
linkcheck_anchors_ignore_for_url = ["https://github.com/pyca/cryptography/issues/2692"]
# -- Options for the Coverage extension --------------------------------------
# https://www.sphinx-doc.org/en/master/usage/extensions/coverage.html#configuration
coverage_ignore_pyobjects = [
# Contract’s add_pre_hook and add_post_hook are not documented because
# they should be transparent to contract developers, for whom pre_hook and
# post_hook should be the actual concern.
r"\bContract\.add_(pre|post)_hook$",
# ContractsManager is an internal class, developers are not expected to
# interact with it directly in any way.
r"\bContractsManager\b$",
# For default contracts we only want to document their general purpose in
# their __init__ method, the methods they reimplement to achieve that purpose
# should be irrelevant to developers using those contracts.
r"\w+Contract\.(adjust_request_args|(pre|post)_process)$",
# Methods of downloader middlewares are not documented, only the classes
# themselves, since downloader middlewares are controlled through Scrapy
# settings.
r"^scrapy\.downloadermiddlewares\.\w*?\.(\w*?Middleware|DownloaderStats)\.",
# Base classes of downloader middlewares are implementation details that
# are not meant for users.
r"^scrapy\.downloadermiddlewares\.\w*?\.Base\w*?Middleware",
# The interface methods of duplicate request filtering classes are already
# covered in the interface documentation part of the DUPEFILTER_CLASS
# setting documentation.
r"^scrapy\.dupefilters\.[A-Z]\w*?\.(from_crawler|request_seen|open|close|log)$",
# Private exception used by the command-line interface implementation.
r"^scrapy\.exceptions\.UsageError",
# Methods of BaseItemExporter subclasses are only documented in
# BaseItemExporter.
r"^scrapy\.exporters\.(?!BaseItemExporter\b)\w*?\.",
# Extension behavior is only modified through settings. Methods of
# extension classes, as well as helper functions, are implementation
# details that are not documented.
r"^scrapy\.extensions\.[a-z]\w*?\.[A-Z]\w*?\.", # methods
r"^scrapy\.extensions\.[a-z]\w*?\.[a-z]", # helper functions
# Never documented before, and deprecated now.
r"^scrapy\.linkextractors\.FilteringLinkExtractor$",
# Implementation detail of LxmlLinkExtractor
r"^scrapy\.linkextractors\.lxmlhtml\.LxmlParserLinkExtractor",
]
# -- Options for the autodoc extension ----------------------------------------
autodoc_member_order = "bysource"
# -- Options for the InterSphinx extension -----------------------------------
# https://www.sphinx-doc.org/en/master/usage/extensions/intersphinx.html#configuration
intersphinx_disabled_reftypes: Sequence[str] = []
# sphinx-scrapy ---------------------------------------------------------------
scrapy_intersphinx_enable = [
"attrs",
"coverage",
"cryptography",
"cssselect",
"form2request",
"itemloaders",
"parsel",
"platformdirs",
"pytest",
"pypug",
"scrapy-lint",
"sphinx",
"tox",
"twisted",
"twistedapi",
"w3lib",
]
# sphinx_llms_txt -------------------------------------------------------------
llms_txt_exclude = [
# Changelog, not useful for an LLM answering "how do I use Scrapy"
# questions, and the largest single contributor to llms-full.txt size.
"news.rst",
"contributing.rst",
]
def _exclude_llms_txt_docs_from_llms_full_txt(app):
# llms-full.txt is the output of the singlemarkdown builder.
if app.builder.name != "singlemarkdown":
return
app.config.exclude_patterns.extend(llms_txt_exclude)
def setup(app):
app.connect("builder-inited", _exclude_llms_txt_docs_from_llms_full_txt)
# -- Other options ------------------------------------------------------------
default_dark_mode = False
scrapy-scrapy-5569de3/docs/conftest.py 0000664 0000000 0000000 00000001631 15250510124 0020003 0 ustar 00root root 0000000 0000000 from doctest import ELLIPSIS, NORMALIZE_WHITESPACE
from pathlib import Path
from sybil import Sybil
from sybil.parsers.doctest import DocTestParser
from sybil.parsers.skip import skip
try:
# >2.0.1
from sybil.parsers.codeblock import PythonCodeBlockParser
except ImportError:
from sybil.parsers.codeblock import CodeBlockParser as PythonCodeBlockParser
from scrapy.http.response.html import HtmlResponse
def load_response(url: str, filename: str) -> HtmlResponse:
input_path = Path(__file__).parent / "_tests" / filename
return HtmlResponse(url, body=input_path.read_bytes())
def setup(namespace):
namespace["load_response"] = load_response
pytest_collect_file = Sybil(
parsers=[
DocTestParser(optionflags=ELLIPSIS | NORMALIZE_WHITESPACE),
PythonCodeBlockParser(future_imports=["print_function"]),
skip,
],
pattern="*.rst",
setup=setup,
).pytest()
scrapy-scrapy-5569de3/docs/contributing.rst 0000664 0000000 0000000 00000041624 15250510124 0021053 0 ustar 00root root 0000000 0000000 .. _topics-contributing:
======================
Contributing to Scrapy
======================
.. important::
Double check that you are reading the most recent version of this document
at https://docs.scrapy.org/en/master/contributing.html
By participating in this project you agree to abide by the terms of our
`Code of Conduct
`_. Please
report unacceptable behavior to opensource@zyte.com.
There are many ways to contribute to Scrapy. Here are some of them:
* Report bugs and request features in the `issue tracker`_, trying to follow
the guidelines detailed in `Reporting bugs`_ below.
* Submit patches for new functionalities and/or bug fixes. Please read
:ref:`writing-patches` and `Submitting patches`_ below for details on how to
write and submit a patch.
* Blog about Scrapy. Tell the world how you're using Scrapy. This will help
newcomers with more examples and will help the Scrapy project to increase its
visibility.
* Join the `Scrapy subreddit`_ and share your ideas on how to
improve Scrapy. We're always open to suggestions.
* Answer Scrapy questions at
`Stack Overflow `__.
Reporting bugs
==============
.. note::
Please report security issues **only** to
scrapy-security@googlegroups.com. This is a private list only open to
trusted Scrapy developers, and its archives are not public.
Well-written bug reports are very helpful, so keep in mind the following
guidelines when you're going to report a new bug.
* check the :ref:`FAQ ` first to see if your issue is addressed in a
well-known question
* if you have a general question about Scrapy usage, please ask it at
`Stack Overflow `__
(use "scrapy" tag).
* check the `open issues`_ to see if the issue has already been reported. If it
has, don't dismiss the report, but check the ticket history and comments. If
you have additional useful information, please leave a comment, or consider
:ref:`sending a pull request ` with a fix.
* search the `scrapy-users`_ list and `Scrapy subreddit`_ to see if it has
been discussed there, or if you're not sure if what you're seeing is a bug.
You can also ask in the ``#scrapy`` IRC channel.
* write **complete, reproducible, specific bug reports**. The smaller the test
case, the better. Remember that other developers won't have your project to
reproduce the bug, so please include all relevant files required to reproduce
it. See for example StackOverflow's guide on creating a
`Minimal, Complete, and Verifiable example`_ exhibiting the issue.
* the most awesome way to provide a complete reproducible example is to
send a pull request which adds a failing test case to the
Scrapy testing suite (see :ref:`submitting-patches`).
This is helpful even if you don't have an intention to
fix the issue yourselves.
* include the output of ``scrapy version -v`` so developers working on your bug
know exactly which version and platform it occurred on, which is often very
helpful for reproducing it, or knowing if it was already fixed.
.. _Minimal, Complete, and Verifiable example: https://stackoverflow.com/help/mcve
.. _find-work:
Finding work
============
If you have decided to make a contribution to Scrapy, but you do not know what
to contribute, you have a few options to find pending work:
- Check out the `contribution GitHub page`_, which lists open issues tagged
as **good first issue**.
.. _contribution GitHub page: https://github.com/scrapy/scrapy/contribute
There are also `help wanted issues`_ but mind that some may require
familiarity with the Scrapy code base. You can also target any other issue
provided it is not tagged as **discuss**.
- If you enjoy writing documentation, there are `documentation issues`_ as
well, but mind that some may require familiarity with the Scrapy code base
as well.
.. _documentation issues: https://github.com/scrapy/scrapy/issues?q=is%3Aissue+is%3Aopen+label%3Adocs+
- If you enjoy :ref:`writing automated tests `, you can work on
increasing our `test coverage`_.
- If you enjoy code cleanup, we welcome fixes for issues detected by our
static analysis tools. See ``pyproject.toml`` for silenced issues that may
need addressing.
Mind that some issues we do not aim to address at all, and usually include
a comment on them explaining the reason; not to confuse with comments that
state what the issue is about, for non-descriptive issue codes.
If you have found an issue, make sure you read the entire issue thread before
you ask questions. That includes related issues and pull requests that show up
in the issue thread when the issue is mentioned elsewhere.
We do not assign issues, and you do not need to announce that you are going to
start working on an issue either. If you want to work on an issue, just go
ahead and :ref:`write a patch for it `.
Do not discard an issue simply because there is an open pull request for it.
Check if open pull requests are active first. And even if some are active, if
you think you can build a better implementation, feel free to create a pull
request with your approach.
If you decide to work on something without an open issue, please:
- Do not create an issue to work on code coverage or code cleanup, create a
pull request directly.
- Do not create both an issue and a pull request right away. Either open an
issue first to get feedback on whether or not the issue is worth
addressing, and create a pull request later only if the feedback from the
team is positive, or create only a pull request, if you think a discussion
will be easier over your code.
- Do not add docstrings for the sake of adding docstrings, or only to address
silenced Ruff issues. We expect docstrings to exist only when they add
something significant to readers, such as explaining something that is not
easier to understand from reading the corresponding code, summarizing a
long, hard-to-read implementation, providing context about calling code, or
indicating purposely uncaught exceptions from called code.
- Do not add tests that use as much mocking as possible just to touch a given
line of code and hence improve line coverage. While we do aim to maximize
test coverage, tests should be written for real scenarios, with minimum
mocking. We usually prefer end-to-end tests.
.. _writing-patches:
Writing patches
===============
The better a patch is written, the higher the chances that it'll get accepted and the sooner it will be merged.
Well-written patches should:
* contain the minimum amount of code required for the specific change. Small
patches are easier to review and merge. So, if you're doing more than one
change (or bug fix), please consider submitting one patch per change. Do not
collapse multiple changes into a single patch. For big changes consider using
a patch queue.
* pass all unit-tests. See `Running tests`_ below.
* include one (or more) test cases that check the bug fixed or the new
functionality added. See `Writing tests`_ below.
* if you're adding or changing a public (documented) API, please include
the documentation changes in the same patch. See `Documentation policies`_
below.
* if you're adding a private API, please add a regular expression to the
``coverage_ignore_pyobjects`` variable of ``docs/conf.py`` to exclude the new
private API from documentation coverage checks.
To see if your private API is skipped properly, generate a documentation
coverage report as follows::
tox -e docs-coverage
* if you are removing deprecated code, first make sure that at least 1 year
(12 months) has passed since the release that introduced the deprecation.
See :ref:`deprecation-policy`.
.. _submitting-patches:
Submitting patches
==================
The best way to submit a patch is to issue a `pull request`_ on GitHub,
optionally creating a new issue first.
Remember to explain what was fixed or the new functionality (what it is, why
it's needed, etc). The more info you include, the easier will be for core
developers to understand and accept your patch.
If your pull request aims to resolve an open issue, `link it accordingly
`__,
e.g.:
.. code-block:: none
Resolves #123
You can also discuss the new functionality (or bug fix) before creating the
patch, but it's always good to have a patch ready to illustrate your arguments
and show that you have put some additional thought into the subject. A good
starting point is to send a pull request on GitHub. It can be simple enough to
illustrate your idea, and leave documentation/tests for later, after the idea
has been validated and proven useful. Alternatively, you can start a
conversation in the `Scrapy subreddit`_ to discuss your idea first.
Sometimes there is an existing pull request for the problem you'd like to
solve, which is stalled for some reason. Often the pull request is in a
right direction, but changes are requested by Scrapy maintainers, and the
original pull request author hasn't had time to address them.
In this case consider picking up this pull request: open
a new pull request with all commits from the original pull request, as well as
additional changes to address the raised issues. Doing so helps a lot; it is
not considered rude as long as the original author is acknowledged by keeping
his/her commits.
You can pull an existing pull request to a local branch
by running ``git fetch upstream pull/$PR_NUMBER/head:$BRANCH_NAME_TO_CREATE``
(replace 'upstream' with a remote name for scrapy repository,
``$PR_NUMBER`` with an ID of the pull request, and ``$BRANCH_NAME_TO_CREATE``
with a name of the branch you want to create locally).
See also: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally#modifying-an-inactive-pull-request-locally.
When writing GitHub pull requests, try to keep titles short but descriptive.
E.g. For bug #411: "Scrapy hangs if an exception raises in start_requests"
prefer "Fix hanging when exception occurs in start_requests (#411)"
instead of "Fix for #411". Complete titles make it easy to skim through
the issue tracker.
Finally, try to keep aesthetic changes (:pep:`8` compliance, unused imports
removal, etc) in separate commits from functional changes. This will make pull
requests easier to review and more likely to get merged.
.. _coding-style:
Coding style
============
Please follow these coding conventions when writing code for inclusion in
Scrapy:
* We use `Ruff `_ for code formatting.
There is a hook in the pre-commit config
that will automatically format your code before every commit. You can also
run Ruff manually with ``tox -e pre-commit``.
* Don't put your name in the code you contribute; git provides enough
metadata to identify author of the code.
See https://docs.github.com/en/get-started/git-basics/setting-your-username-in-git
for setup instructions.
.. _scrapy-pre-commit:
Pre-commit
==========
We use `pre-commit`_ to automatically address simple code issues before every
commit.
.. _pre-commit: https://pre-commit.com/
After your create a local clone of your fork of the Scrapy repository:
#. `Install pre-commit `_.
#. On the root of your local clone of the Scrapy repository, run the following
command:
.. code-block:: bash
pre-commit install
Now pre-commit will check your changes every time you create a Git commit. Upon
finding issues, pre-commit aborts your commit, and either fixes those issues
automatically, or only reports them to you. If it fixes those issues
automatically, creating your commit again should succeed. Otherwise, you may
need to address the corresponding issues manually first.
.. _documentation-policies:
Documentation policies
======================
For reference documentation of API members (classes, methods, etc.) use
docstrings and make sure that the Sphinx documentation uses the
:mod:`~sphinx.ext.autodoc` extension to pull the docstrings. API reference
documentation should follow docstring conventions (`PEP 257`_) and be
IDE-friendly: short, to the point, and it may provide short examples.
Other types of documentation, such as tutorials or topics, should be covered in
files within the ``docs/`` directory. This includes documentation that is
specific to an API member, but goes beyond API reference documentation.
In any case, if something is covered in a docstring, use the
:mod:`~sphinx.ext.autodoc` extension to pull the docstring into the
documentation instead of duplicating the docstring in files within the
``docs/`` directory.
Documentation updates that cover new or modified features must use Sphinx’s
:rst:dir:`versionadded` and :rst:dir:`versionchanged` directives. Use
``VERSION`` as version, we will replace it with the actual version right before
the corresponding release. When we release a new major or minor version of
Scrapy, we remove these directives if they are older than 3 years.
Documentation about deprecated features must be removed as those features are
deprecated, so that new readers do not run into it. New deprecations and
deprecation removals are documented in the :ref:`release notes `.
.. _write-tests:
Tests
=====
Tests are implemented using pytest_. Running tests requires :doc:`tox
`.
.. _pytest: https://pytest.org
.. _running-tests:
Running tests
-------------
To run all tests::
tox
To run a specific test (say ``tests/test_loader.py``) use:
``tox -- tests/test_loader.py``
To run the tests on a specific :doc:`tox ` environment, use
``-e `` with an environment name from ``tox.ini``. For example, to run
the tests with Python 3.10 use::
tox -e py310
You can also specify a comma-separated list of environments, and use :ref:`tox’s
parallel mode ` to run the tests on multiple environments in
parallel::
tox -e py39,py310 -p auto
To pass command-line options to :doc:`pytest `, add them after
``--`` in your call to :doc:`tox `. Using ``--`` overrides the
default positional arguments defined in ``tox.ini``, so you must include those
default positional arguments (``scrapy tests``) after ``--`` as well::
tox -- scrapy tests -x # stop after first failure
You can also use the `pytest-xdist`_ plugin. For example, to run all tests on
the Python 3.10 :doc:`tox ` environment using all your CPU cores::
tox -e py310 -- scrapy tests -n auto
To see coverage report install :doc:`coverage `
(``pip install coverage``) and run:
``coverage report``
see output of ``coverage --help`` for more options like html or xml report.
Some tests need a ``mitmdump`` executable (from mitmproxy_) to test against a
fully featured proxy server; they are skipped when one cannot be found
(``mitmproxy`` is intentionally not a test dependency that would be installed
into test venvs, as that sometimes leads to various dependency conflicts).
To run these tests, make ``mitmdump`` available in one of these ways:
* install ``mitmproxy`` so that ``mitmdump`` is on your ``PATH``, e.g. with
pipx_ (``pipx install mitmproxy``) or uv_ (``uv tool install mitmproxy``);
* have uv_ installed, in which case the tests will run
``uvx --from mitmproxy mitmdump``;
* set the ``MITMDUMP`` environment variable to the path of a ``mitmdump``
executable.
Writing tests
-------------
All functionality (including new features and bug fixes) must include a test
case to check that it works as expected, so please include tests for your
patches if you want them to get accepted sooner.
Scrapy uses unit-tests, which are located in the `tests/`_ directory.
Their module name typically resembles the full path of the module they're
testing. For example, the item loaders code is in::
scrapy.loader
And their unit-tests are in::
tests/test_loader.py
.. _issue tracker: https://github.com/scrapy/scrapy/issues
.. _scrapy-users: https://groups.google.com/forum/#!forum/scrapy-users
.. _Scrapy subreddit: https://www.reddit.com/r/scrapy/
.. _tests/: https://github.com/scrapy/scrapy/tree/master/tests
.. _open issues: https://github.com/scrapy/scrapy/issues
.. _PEP 257: https://peps.python.org/pep-0257/
.. _pull request: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request
.. _pytest-xdist: https://github.com/pytest-dev/pytest-xdist
.. _help wanted issues: https://github.com/scrapy/scrapy/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22
.. _test coverage: https://app.codecov.io/gh/scrapy/scrapy
.. _mitmproxy: https://mitmproxy.org/
.. _pipx: https://pipx.pypa.io/
.. _uv: https://docs.astral.sh/uv/
scrapy-scrapy-5569de3/docs/faq.rst 0000664 0000000 0000000 00000037314 15250510124 0017114 0 ustar 00root root 0000000 0000000 .. _faq:
Frequently Asked Questions
==========================
.. _faq-scrapy-bs-cmp:
How does Scrapy compare to BeautifulSoup or lxml?
-------------------------------------------------
`BeautifulSoup`_ and `lxml`_ are libraries for parsing HTML and XML. Scrapy is
an application framework for writing web spiders that crawl web sites and
extract data from them.
Scrapy provides a built-in mechanism for extracting data (called
:ref:`selectors `) but you can easily use `BeautifulSoup`_
(or `lxml`_) instead, if you feel more comfortable working with them. After
all, they're just parsing libraries which can be imported and used from any
Python code.
In other words, comparing `BeautifulSoup`_ (or `lxml`_) to Scrapy is like
comparing `jinja2`_ to `Django`_.
.. _BeautifulSoup: https://www.crummy.com/software/BeautifulSoup/
.. _lxml: https://lxml.de/
.. _jinja2: https://palletsprojects.com/projects/jinja/
.. _Django: https://www.djangoproject.com/
Can I use Scrapy with BeautifulSoup?
------------------------------------
Yes, you can.
As mentioned :ref:`above `, `BeautifulSoup`_ can be used
for parsing HTML responses in Scrapy callbacks.
You just have to feed the response's body into a ``BeautifulSoup`` object
and extract whatever data you need from it.
Here's an example spider using BeautifulSoup API, with ``lxml`` as the HTML parser:
.. skip: next
.. code-block:: python
from bs4 import BeautifulSoup
import scrapy
class ExampleSpider(scrapy.Spider):
name = "example"
allowed_domains = ["example.com"]
start_urls = ("http://www.example.com/",)
def parse(self, response):
# use lxml to get decent HTML parsing speed
soup = BeautifulSoup(response.text, "lxml")
yield {"url": response.url, "title": soup.h1.string}
.. note::
``BeautifulSoup`` supports several HTML/XML parsers.
See `BeautifulSoup's official documentation`_ on which ones are available.
.. _BeautifulSoup's official documentation: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#specifying-the-parser-to-use
Did Scrapy "steal" X from Django?
---------------------------------
Probably, but we don't like that word. We think Django_ is a great open source
project and an example to follow, so we've used it as an inspiration for
Scrapy.
We believe that, if something is already done well, there's no need to reinvent
it. This concept, besides being one of the foundations for open source and free
software, not only applies to software but also to documentation, procedures,
policies, etc. So, instead of going through each problem ourselves, we choose
to copy ideas from those projects that have already solved them properly, and
focus on the real problems we need to solve.
We'd be proud if Scrapy serves as an inspiration for other projects. Feel free
to steal from us!
Does Scrapy work with HTTP proxies?
-----------------------------------
Yes. Support for HTTP proxies is provided through the HTTP Proxy downloader
middleware. See
:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware`.
Does Scrapy work with SOCKS proxies?
------------------------------------
Yes, when using
:class:`~scrapy.core.downloader.handlers._httpx.HttpxDownloadHandler`. See
:class:`~scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware` and the
handler documentation.
How can I scrape an item with attributes in different pages?
------------------------------------------------------------
See :ref:`callback-data`.
How can I simulate a user login in my spider?
---------------------------------------------
See :ref:`topics-request-response-ref-request-userlogin`.
.. _faq-bfo-dfo:
Does Scrapy crawl in breadth-first or depth-first order?
--------------------------------------------------------
:ref:`DFO by default, but other orders are possible `.
My Scrapy crawler has memory leaks. What can I do?
--------------------------------------------------
See :ref:`topics-leaks`.
Also, Python has a builtin memory leak issue which is described in
:ref:`topics-leaks-without-leaks`.
How can I make Scrapy consume less memory?
------------------------------------------
See previous question.
How can I prevent memory errors due to many allowed domains?
------------------------------------------------------------
If you have a spider with a long list of :attr:`~scrapy.Spider.allowed_domains`
(e.g. 50,000+), consider replacing the default
:class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` downloader
middleware with a :ref:`custom downloader middleware
` that requires less memory. For example:
- If your domain names are similar enough, use your own regular expression
instead of joining the strings in :attr:`~scrapy.Spider.allowed_domains` into
a complex regular expression.
- If you can meet the installation requirements, use pyre2_ instead of
Python’s re_ to compile your URL-filtering regular expression. See
:gh:`1908`.
See also `other suggestions at StackOverflow
`__.
.. note:: Remember to disable
:class:`scrapy.downloadermiddlewares.offsite.OffsiteMiddleware` when you
enable your custom implementation:
.. code-block:: python
DOWNLOADER_MIDDLEWARES = {
"scrapy.downloadermiddlewares.offsite.OffsiteMiddleware": None,
"myproject.middlewares.CustomOffsiteMiddleware": 50,
}
.. _pyre2: https://github.com/andreasvc/pyre2
.. _re: https://docs.python.org/3/library/re.html
Can I use Basic HTTP Authentication in my spiders?
--------------------------------------------------
Yes, see :class:`~scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware`.
Why does Scrapy download pages in English instead of my native language?
------------------------------------------------------------------------
Try changing the default `Accept-Language`_ request header by overriding the
:setting:`DEFAULT_REQUEST_HEADERS` setting.
.. _Accept-Language: https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.4
Where can I find some example Scrapy projects?
----------------------------------------------
See :ref:`intro-examples`.
Can I run a spider without creating a project?
----------------------------------------------
Yes. You can use the :command:`runspider` command. For example, if you have a
spider written in a ``my_spider.py`` file you can run it with::
scrapy runspider my_spider.py
See :command:`runspider` command for more info.
I get "Filtered offsite request" messages. How can I fix them?
--------------------------------------------------------------
Those messages (logged with ``DEBUG`` level) don't necessarily mean there is a
problem, so you may not need to fix them.
Those messages are thrown by
:class:`~scrapy.downloadermiddlewares.offsite.OffsiteMiddleware`, which is a
downloader middleware (enabled by default) whose purpose is to filter out
requests to domains outside the ones covered by the spider.
What is the recommended way to deploy a Scrapy crawler in production?
---------------------------------------------------------------------
See :ref:`topics-deploy`.
Can I use JSON for large exports?
---------------------------------
It'll depend on how large your output is. See :ref:`this warning
` in :class:`~scrapy.exporters.JsonItemExporter`
documentation.
Can I return (Twisted) deferreds from signal handlers?
------------------------------------------------------
Some signals support returning deferreds from their handlers, others don't. See
the :ref:`topics-signals-ref` to know which ones.
What does the response status code 999 mean?
--------------------------------------------
999 is a custom response status code used by some sites to throttle requests.
Try slowing down the crawling speed by using a download delay of ``2`` (or
higher) for the affected domains, with the :setting:`DOWNLOAD_SLOTS` setting:
.. code-block:: python
DOWNLOAD_SLOTS = {
"example.com": {"delay": 2},
}
Or by setting a global download delay in your project with the
:setting:`DOWNLOAD_DELAY` setting.
Can I call ``pdb.set_trace()`` from my spiders to debug them?
-------------------------------------------------------------
Yes, but you can also use the Scrapy shell which allows you to quickly analyze
(and even modify) the response being processed by your spider, which is, quite
often, more useful than plain old ``pdb.set_trace()``.
For more info see :ref:`topics-shell-inspect-response`.
You can also :ref:`connect to live crawls `, which is the
best way to debug running spiders without adding breakpoints to their code in
advance.
Simplest way to dump all my scraped items into a JSON/CSV/XML file?
-------------------------------------------------------------------
To dump into a JSON file::
scrapy crawl myspider -O items.json
To dump into a CSV file::
scrapy crawl myspider -O items.csv
To dump into an XML file::
scrapy crawl myspider -O items.xml
For more information see :ref:`topics-feed-exports`
What's this huge cryptic ``__VIEWSTATE`` parameter used in some forms?
----------------------------------------------------------------------
The ``__VIEWSTATE`` parameter is used in sites built with ASP.NET/VB.NET. For
more info on how it works see `this page`_. Also, here's an `example spider`_
which scrapes one of these sites.
.. _this page: https://metacpan.org/release/ECARROLL/HTML-TreeBuilderX-ASP_NET-0.09/view/lib/HTML/TreeBuilderX/ASP_NET.pm
.. _example spider: https://github.com/AmbientLighter/rpn-fas/blob/master/fas/spiders/rnp.py
What's the best way to parse big XML/CSV data feeds?
----------------------------------------------------
Parsing big feeds with XPath selectors can be problematic since they need to
build the DOM of the entire feed in memory, and this can be quite slow and
consume a lot of memory.
In order to avoid parsing all the entire feed at once in memory, you can use
the :func:`~scrapy.utils.iterators.xmliter_lxml` and
:func:`~scrapy.utils.iterators.csviter` functions. In fact, this is what
:class:`~scrapy.spiders.XMLFeedSpider` and
:class:`~scrapy.spiders.CSVFeedSpider` use.
.. autofunction:: scrapy.utils.iterators.xmliter_lxml
.. autofunction:: scrapy.utils.iterators.csviter
Does Scrapy manage cookies automatically?
-----------------------------------------
Yes, Scrapy receives and keeps track of cookies sent by servers, and sends them
back on subsequent requests, like any regular web browser does.
For more info see :ref:`cookies`.
How can I see the cookies being sent and received from Scrapy?
--------------------------------------------------------------
Enable the :setting:`COOKIES_DEBUG` setting.
How can I instruct a spider to stop itself?
-------------------------------------------
Raise the :exc:`~scrapy.exceptions.CloseSpider` exception from a callback. For
more info see: :exc:`~scrapy.exceptions.CloseSpider`.
How can I prevent my Scrapy bot from getting banned?
----------------------------------------------------
See :ref:`bans`.
Should I use spider arguments or settings to configure my spider?
-----------------------------------------------------------------
Both :ref:`spider arguments ` and :ref:`settings `
can be used to configure your spider. There is no strict rule that mandates to
use one or the other, but settings are more suited for parameters that, once
set, don't change much, while spider arguments are meant to change more often,
even on each spider run and sometimes are required for the spider to run at all
(for example, to set the start url of a spider).
To illustrate with an example, assuming you have a spider that needs to log
into a site to scrape data, and you only want to scrape data from a certain
section of the site (which varies each time). In that case, the credentials to
log in would be settings, while the url of the section to scrape would be a
spider argument.
I'm scraping an XML document and my XPath selector doesn't return any items
---------------------------------------------------------------------------
You may need to remove namespaces. See :ref:`removing-namespaces`.
.. _faq-split-item:
How to split an item into multiple items in an item pipeline?
-------------------------------------------------------------
:ref:`Item pipelines ` cannot yield multiple items per
input item. :ref:`Create a spider middleware `
instead, and use its
:meth:`~scrapy.spidermiddlewares.SpiderMiddleware.process_spider_output`
method for this purpose. For example:
.. code-block:: python
from copy import deepcopy
from itemadapter import ItemAdapter
from scrapy import Request
class MultiplyItemsMiddleware:
def process_spider_output(self, response, result):
for item_or_request in result:
if isinstance(item_or_request, Request):
yield item_or_request
continue
adapter = ItemAdapter(item_or_request)
for _ in range(adapter["multiply_by"]):
yield deepcopy(item_or_request)
Does Scrapy support IPv6 addresses?
-----------------------------------
Yes, but when using
:class:`~scrapy.core.downloader.handlers.http11.HTTP11DownloadHandler` or
:class:`~scrapy.core.downloader.handlers.http2.H2DownloadHandler` you need to
set :setting:`TWISTED_DNS_RESOLVER` to ``scrapy.resolver.CachingHostnameResolver``.
Note that by doing so, you lose the ability to set a specific timeout for DNS requests
(the value of the :setting:`DNS_TIMEOUT` setting is ignored).
.. _faq-specific-reactor:
How to deal with ``: filedescriptor out of range in select()`` exceptions?
----------------------------------------------------------------------------------------------
This issue `has been reported`_ to appear when running broad crawls in macOS, where the default
Twisted reactor was :class:`twisted.internet.selectreactor.SelectReactor` at that time.
If you have switched to this reactor using the :setting:`TWISTED_REACTOR` setting you can switch
to a different one in the same way.
.. _faq-stop-response-download:
How can I cancel the download of a given response?
--------------------------------------------------
In some situations, it might be useful to stop the download of a certain response.
For instance, sometimes you can determine whether or not you need the full contents
of a response by inspecting its headers or the first bytes of its body. In that case,
you could save resources by attaching a handler to the :class:`~scrapy.signals.bytes_received`
or :class:`~scrapy.signals.headers_received` signals and raising a
:exc:`~scrapy.exceptions.StopDownload` exception. Please refer to the
:ref:`topics-stop-response-download` topic for additional information and examples.
.. _faq-blank-request:
How can I make a blank request?
-------------------------------
.. code-block:: python
from scrapy import Request
blank_request = Request("data:,")
In this case, the URL is set to a data URI scheme. Data URLs allow you to include data
inline within web pages, similar to external resources. The "data:" scheme with an empty
content (",") essentially creates a request to a data URL without any specific content.
Running ``runspider`` I get ``error: No spider found in file: ``
--------------------------------------------------------------------------
This may happen if your Scrapy project has a spider module with a name that
conflicts with the name of one of the `Python standard library modules`_, such
as ``csv.py`` or ``os.py``, or any `Python package`_ that you have installed.
See :gh:`2680`.
.. _has been reported: https://github.com/scrapy/scrapy/issues/2905
.. _Python standard library modules: https://docs.python.org/3/py-modindex.html
.. _Python package: https://pypi.org/
scrapy-scrapy-5569de3/docs/index.rst 0000664 0000000 0000000 00000016263 15250510124 0017454 0 ustar 00root root 0000000 0000000 .. _topics-index:
==============================
Scrapy |version| documentation
==============================
Scrapy is a fast high-level `web crawling`_ and `web scraping`_ framework, used
to crawl websites and extract structured data from their pages. It can be used
for a wide range of purposes, from data mining to monitoring and automated
testing.
.. _web crawling: https://en.wikipedia.org/wiki/Web_crawler
.. _web scraping: https://en.wikipedia.org/wiki/Web_scraping
.. _getting-help:
Getting help
============
Having trouble? We'd like to help!
* Try the :doc:`FAQ ` -- it's got answers to some common questions.
* Looking for specific information? Try the :ref:`genindex` or :ref:`modindex`.
* Ask or search questions in `StackOverflow using the scrapy tag`_.
* Ask or search questions in the `Scrapy subreddit`_.
* Search for questions on the archives of the `scrapy-users mailing list`_.
* Ask a question in the `#scrapy IRC channel`_.
* Report bugs with Scrapy in our `issue tracker`_.
* Join the Discord community `Scrapy Discord`_.
.. _scrapy-users mailing list: https://groups.google.com/forum/#!forum/scrapy-users
.. _Scrapy subreddit: https://www.reddit.com/r/scrapy/
.. _StackOverflow using the scrapy tag: https://stackoverflow.com/tags/scrapy
.. _#scrapy IRC channel: irc://irc.freenode.net/scrapy
.. _issue tracker: https://github.com/scrapy/scrapy/issues
.. _Scrapy Discord: https://discord.com/invite/mv3yErfpvq
First steps
===========
.. toctree::
:caption: First steps
:hidden:
intro/overview
intro/install
intro/tutorial
intro/examples
:doc:`intro/overview`
Understand what Scrapy is and how it can help you.
:doc:`intro/install`
Get Scrapy installed on your computer.
:doc:`intro/tutorial`
Write your first Scrapy project.
:doc:`intro/examples`
Learn more by playing with a pre-made Scrapy project.
.. _section-basics:
Basic concepts
==============
.. toctree::
:caption: Basic concepts
:hidden:
topics/concepts
topics/commands
topics/spiders
topics/selectors
topics/items
topics/loaders
topics/shell
topics/item-pipeline
topics/feed-exports
topics/request-response
topics/cookies
topics/link-extractors
topics/settings
topics/exceptions
:doc:`topics/concepts`
Get a quick map of Scrapy's main concepts and when to use each.
:doc:`topics/commands`
Learn about the command-line tool used to manage your Scrapy project.
:doc:`topics/spiders`
Write the rules to crawl your websites.
:doc:`topics/selectors`
Extract the data from web pages using XPath.
:doc:`topics/items`
Define the data you want to scrape.
:doc:`topics/loaders`
Populate your items with the extracted data.
:doc:`topics/shell`
Test your extraction code in an interactive environment.
:doc:`topics/item-pipeline`
Post-process and store your scraped data.
:doc:`topics/feed-exports`
Output your scraped data using different formats and storages.
:doc:`topics/request-response`
Understand the classes used to represent HTTP requests and responses.
:doc:`topics/cookies`
Send and receive cookies.
:doc:`topics/link-extractors`
Convenient classes to extract links to follow from pages.
:doc:`topics/settings`
Learn how to configure Scrapy and see all :ref:`available settings `.
:doc:`topics/exceptions`
See all available exceptions and their meaning.
Built-in services
=================
.. toctree::
:caption: Built-in services
:hidden:
topics/logging
topics/stats
topics/telnetconsole
:doc:`topics/logging`
Learn how to use Python's built-in logging on Scrapy.
:doc:`topics/stats`
Collect statistics about your scraping crawler.
:doc:`topics/telnetconsole`
Inspect a running crawler using a built-in Python console.
Solving specific problems
=========================
.. toctree::
:caption: Solving specific problems
:hidden:
faq
topics/agents
topics/debug
topics/contracts
topics/practices
topics/security
topics/optimize
topics/developer-tools
topics/dynamic-content
topics/leaks
topics/media-pipeline
topics/deploy
topics/autothrottle
topics/benchmarking
topics/jobs
topics/coroutines
topics/asyncio
:doc:`faq`
Get answers to most frequently asked questions.
:doc:`topics/agents`
Learn how to use Scrapy with coding agents efficiently.
:doc:`topics/debug`
Learn how to debug common problems of your Scrapy spider.
:doc:`topics/contracts`
Learn how to use contracts for testing your spiders.
:doc:`topics/practices`
Get familiar with some Scrapy common practices.
:doc:`topics/security`
Understand the security implications of Scrapy defaults and how to harden
them.
:doc:`topics/optimize`
Find the bottleneck of your crawls and learn how to address it.
:doc:`topics/developer-tools`
Learn how to scrape with your browser's developer tools.
:doc:`topics/dynamic-content`
Read webpage data that is loaded dynamically.
:doc:`topics/leaks`
Learn how to find and get rid of memory leaks in your crawler.
:doc:`topics/media-pipeline`
Download files and/or images associated with your scraped items.
:doc:`topics/deploy`
Deploying your Scrapy spiders and run them in a remote server.
:doc:`topics/autothrottle`
Adjust crawl rate dynamically based on load.
:doc:`topics/benchmarking`
Check how Scrapy performs on your hardware.
:doc:`topics/jobs`
Learn how to pause and resume crawls for large spiders.
:doc:`topics/coroutines`
Use the :ref:`coroutine syntax `.
:doc:`topics/asyncio`
Use :mod:`asyncio` and :mod:`asyncio`-powered libraries.
.. _extending-scrapy:
Extending Scrapy
================
.. toctree::
:caption: Extending Scrapy
:hidden:
topics/architecture
topics/addons
topics/downloader-middleware
topics/spider-middleware
topics/extensions
topics/signals
topics/scheduler
topics/exporters
topics/download-handlers
topics/components
topics/api
:doc:`topics/architecture`
Understand the Scrapy architecture.
:doc:`topics/addons`
Enable and configure third-party extensions.
:doc:`topics/downloader-middleware`
Customize how pages get requested and downloaded.
:doc:`topics/spider-middleware`
Customize the input and output of your spiders.
:doc:`topics/extensions`
Extend Scrapy with your custom functionality
:doc:`topics/signals`
See all available signals and how to work with them.
:doc:`topics/scheduler`
Understand the scheduler component.
:doc:`topics/exporters`
Quickly export your scraped items to a file (XML, CSV, etc).
:doc:`topics/download-handlers`
Customize how requests are downloaded or add support for new URL schemes.
:doc:`topics/components`
Learn the common API and some good practices when building custom Scrapy
components.
:doc:`topics/api`
Use it on extensions and middlewares to extend Scrapy functionality.
All the rest
============
.. toctree::
:caption: All the rest
:hidden:
news
contributing
versioning
:doc:`news`
See what has changed in recent Scrapy versions.
:doc:`contributing`
Learn how to contribute to the Scrapy project.
:doc:`versioning`
Understand Scrapy versioning and API stability.
scrapy-scrapy-5569de3/docs/intro/ 0000775 0000000 0000000 00000000000 15250510124 0016736 5 ustar 00root root 0000000 0000000 scrapy-scrapy-5569de3/docs/intro/examples.rst 0000664 0000000 0000000 00000001351 15250510124 0021306 0 ustar 00root root 0000000 0000000 .. _intro-examples:
========
Examples
========
The best way to learn is with examples, and Scrapy is no exception. For this
reason, there is an example Scrapy project named quotesbot_, that you can use to
play and learn more about Scrapy. It contains two spiders for
https://quotes.toscrape.com, one using CSS selectors and another one using XPath
expressions.
The quotesbot_ project is available at: https://github.com/scrapy/quotesbot.
You can find more information about it in the project's README.
If you're familiar with git, you can checkout the code. Otherwise you can
download the project as a zip file by clicking
`here `_.
.. _quotesbot: https://github.com/scrapy/quotesbot
scrapy-scrapy-5569de3/docs/intro/install.rst 0000664 0000000 0000000 00000026236 15250510124 0021147 0 ustar 00root root 0000000 0000000 .. _intro-install:
==================
Installation guide
==================
.. _faq-python-versions:
Supported Python versions
=========================
Scrapy requires Python 3.10+, either the CPython implementation (default) or
the PyPy implementation (see :ref:`python:implementations`).
.. _intro-install-scrapy:
Installing Scrapy
=================
If you're using `Anaconda`_ or `Miniconda`_, you can install the package from
the `conda-forge`_ channel, which has up-to-date packages for Linux, Windows
and macOS.
To install Scrapy using ``conda``, run::
conda install -c conda-forge scrapy
Alternatively, if you’re already familiar with installation of Python packages,
you can install Scrapy and its dependencies from PyPI with::
pip install Scrapy
We strongly recommend that you install Scrapy in :ref:`a dedicated virtualenv `,
to avoid conflicting with your system packages.
Note that sometimes this may require solving compilation issues for some Scrapy
dependencies depending on your operating system, so be sure to check the
:ref:`intro-install-platform-notes`.
For more detailed and platform-specific instructions, as well as
troubleshooting information, read on.
Things that are good to know
----------------------------
Scrapy is written in pure Python and depends on a few key Python packages (among others):
* `lxml`_, an efficient XML and HTML parser
* `parsel`_, an HTML/XML data extraction library written on top of lxml,
* `w3lib`_, a multi-purpose helper for dealing with URLs and web page encodings
* `twisted`_, an asynchronous networking framework
* `cryptography`_ and `pyOpenSSL`_, to deal with various network-level security needs
Some of these packages themselves depend on non-Python packages
that might require additional installation steps depending on your platform.
Please check :ref:`platform-specific guides below `.
In case of any trouble related to these dependencies,
please refer to their respective installation instructions:
* `lxml installation`_
* :doc:`cryptography installation `
.. _lxml installation: https://lxml.de/installation.html
.. _intro-using-virtualenv:
Using a virtual environment (recommended)
-----------------------------------------
TL;DR: We recommend installing Scrapy inside a virtual environment
on all platforms.
Python packages can be installed either globally (a.k.a system wide),
or in user-space. We do not recommend installing Scrapy system wide.
Instead, we recommend that you install Scrapy within a so-called
"virtual environment" (:mod:`venv`).
Virtual environments allow you to not conflict with already-installed Python
system packages (which could break some of your system tools and scripts),
and still install packages normally with ``pip`` (without ``sudo`` and the likes).
See :ref:`tut-venv` on how to create your virtual environment.
Once you have created a virtual environment, you can install Scrapy inside it with ``pip``,
just like any other Python package.
(See :ref:`platform-specific guides `
below for non-Python dependencies that you may need to install beforehand).
.. _extras:
Optional extras
===============
Scrapy provides optional :ref:`extras `
that install additional dependencies to enable specific features. To install
Scrapy with one or more extras, list them in square brackets:
.. code-block:: console
pip install scrapy[s3,images]
The following extras are available:
.. list-table::
:header-rows: 1
* - Extra
- Provides
* - ``bpython``
- :ref:`bpython shell `
* - ``color``
- :setting:`LOG_COLOR`
* - ``gcs``
- :ref:`Google Cloud Storage ` for
:ref:`feed exports ` and
:ref:`media pipelines `
* - ``httpx``
- :ref:`httpx-handler`, including its HTTP/2 and SOCKS proxy support
* - ``images``
- :ref:`Images pipeline `
* - ``ipython``
- :ref:`IPython shell `
* - ``ptpython``
- :ref:`ptpython shell `
* - ``robotparser``
- :ref:`Robotexclusionrulesparser robots.txt parsing `
* - ``s3``
- :ref:`Amazon S3 ` storage for
:ref:`feed exports `,
:ref:`media pipelines `, and
:ref:`S3 downloads `
* - ``twisted-http2``
- :ref:`twisted-http2-handler`
* - ``uvloop``
- `uvloop `_ event loop
.. _intro-install-platform-notes:
Platform specific installation notes
====================================
.. _intro-install-windows:
Windows
-------
Though it's possible to install Scrapy on Windows using pip, we recommend you
install `Anaconda`_ or `Miniconda`_ and use the package from the
`conda-forge`_ channel, which will avoid most installation issues.
Once you've installed `Anaconda`_ or `Miniconda`_, install Scrapy with::
conda install -c conda-forge scrapy
To install Scrapy on Windows using ``pip``:
.. warning::
This installation method requires “Microsoft Visual C++” for installing some
Scrapy dependencies, which demands significantly more disk space than Anaconda.
#. Download and execute `Microsoft C++ Build Tools`_ to install the Visual Studio Installer.
#. Run the Visual Studio Installer.
#. Under the Workloads section, select **C++ build tools**.
#. Check the installation details and make sure following packages are selected as optional components:
* **MSVC** (e.g MSVC v142 - VS 2019 C++ x64/x86 build tools (v14.23) )
* **Windows SDK** (e.g Windows 10 SDK (10.0.18362.0))
#. Install the Visual Studio Build Tools.
Now, you should be able to :ref:`install Scrapy ` using ``pip``.
.. _intro-install-ubuntu:
Ubuntu 14.04 or above
---------------------
Scrapy is currently tested with recent-enough versions of lxml,
twisted and pyOpenSSL, and is compatible with recent Ubuntu distributions.
But it should support older versions of Ubuntu too, like Ubuntu 14.04,
albeit with potential issues with TLS connections.
**Don't** use the ``python-scrapy`` package provided by Ubuntu, they are
typically too old and slow to catch up with the latest Scrapy release.
To install Scrapy on Ubuntu (or Ubuntu-based) systems, you need to install
these dependencies::
sudo apt-get install python3 python3-dev python3-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev
- ``python3-dev``, ``zlib1g-dev``, ``libxml2-dev`` and ``libxslt1-dev``
are required for ``lxml``
- ``libssl-dev`` and ``libffi-dev`` are required for ``cryptography``
Inside a :ref:`virtualenv `,
you can install Scrapy with ``pip`` after that::
pip install scrapy
.. note::
The same non-Python dependencies can be used to install Scrapy in Debian
Jessie (8.0) and above.
.. _intro-install-macos:
macOS
-----
Building Scrapy's dependencies requires the presence of a C compiler and
development headers. On macOS this is typically provided by Apple’s Xcode
development tools. To install the Xcode command-line tools, open a terminal
window and run::
xcode-select --install
There's a `known issue `_ that
prevents ``pip`` from updating system packages. This has to be addressed to
successfully install Scrapy and its dependencies. Here are some proposed
solutions:
* *(Recommended)* **Don't** use system Python. Install a new, updated version
that doesn't conflict with the rest of your system. Here's how to do it using
the `homebrew`_ package manager:
* Install `homebrew`_ following the instructions in https://brew.sh/
* Update your ``PATH`` variable to state that homebrew packages should be
used before system packages (Change ``.bashrc`` to ``.zshrc`` accordingly
if you're using `zsh`_ as default shell)::
echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc
* Reload ``.bashrc`` to ensure the changes have taken place::
source ~/.bashrc
* Install python::
brew install python
* *(Optional)* :ref:`Install Scrapy inside a Python virtual environment
`.
This method is a workaround for the above macOS issue, but it's an overall
good practice for managing dependencies and can complement the first method.
After any of these workarounds you should be able to install Scrapy::
pip install Scrapy
PyPy
----
We recommend using the latest PyPy version.
For PyPy3, only Linux installation was tested.
Most Scrapy dependencies now have binary wheels for CPython, but not for PyPy.
This means that these dependencies will be built during installation.
On macOS, you are likely to face an issue with building the Cryptography
dependency. The solution to this problem is described
`here `_,
that is to ``brew install openssl`` and then export the flags that this command
recommends (only needed when installing Scrapy). Installing on Linux has no special
issues besides installing build dependencies.
Installing Scrapy with PyPy on Windows is not tested.
You can check that Scrapy is installed correctly by running ``scrapy bench``.
If this command gives errors such as
``TypeError: ... got 2 unexpected keyword arguments``, this means
that the ``PyPyDispatcher`` dependency wasn't installed. To fix this issue, run
``pip install 'PyPyDispatcher>=2.1.0'``.
.. _intro-install-troubleshooting:
Troubleshooting
===============
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'
----------------------------------------------------------------
After you install or upgrade Scrapy, Twisted or pyOpenSSL, you may get an
exception with the following traceback::
[…]
File "[…]/site-packages/twisted/protocols/tls.py", line 63, in
from twisted.internet._sslverify import _setAcceptableProtocols
File "[…]/site-packages/twisted/internet/_sslverify.py", line 38, in
TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'
The reason you get this exception is that your system or virtual environment
has a version of pyOpenSSL that your version of Twisted does not support.
To install a version of pyOpenSSL that your version of Twisted supports,
reinstall Twisted with the :code:`tls` extra option::
pip install twisted[tls]
For details, see `Issue #2473 `_.
.. _Python: https://www.python.org/
.. _lxml: https://lxml.de/index.html
.. _parsel: https://pypi.org/project/parsel/
.. _w3lib: https://pypi.org/project/w3lib/
.. _twisted: https://twisted.org/
.. _cryptography: https://cryptography.io/en/latest/
.. _pyOpenSSL: https://pypi.org/project/pyOpenSSL/
.. _setuptools: https://pypi.org/pypi/setuptools
.. _homebrew: https://brew.sh/
.. _zsh: https://www.zsh.org/
.. _Anaconda: https://www.anaconda.com/docs/main
.. _Miniconda: https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html
.. _Microsoft C++ Build Tools: https://visualstudio.microsoft.com/visual-cpp-build-tools/
.. _conda-forge: https://conda-forge.org/
scrapy-scrapy-5569de3/docs/intro/overview.rst 0000664 0000000 0000000 00000015054 15250510124 0021343 0 ustar 00root root 0000000 0000000 .. _intro-overview:
==================
Scrapy at a glance
==================
Scrapy (/ˈskreɪpaɪ/) is an application framework for crawling web sites and extracting
structured data which can be used for a wide range of useful applications, like
data mining, information processing or historical archival.
Even though Scrapy was originally designed for `web scraping`_, it can also be
used to extract data using APIs (such as `Amazon Associates Web Services`_) or
as a general purpose web crawler.
Walk-through of an example spider
=================================
In order to show you what Scrapy brings to the table, we'll walk you through an
example of a Scrapy Spider using the simplest way to run a spider.
Here's the code for a spider that scrapes famous quotes from website
https://quotes.toscrape.com, following the pagination:
.. code-block:: python
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
"https://quotes.toscrape.com/tag/humor/",
]
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"author": quote.xpath("span/small/text()").get(),
"text": quote.css("span.text::text").get(),
}
next_page = response.css('li.next a::attr("href")').get()
if next_page is not None:
yield response.follow(next_page, self.parse)
Put this in a text file, name it something like ``quotes_spider.py``
and run the spider using the :command:`runspider` command::
scrapy runspider quotes_spider.py -o quotes.jsonl
When this finishes you will have in the ``quotes.jsonl`` file a list of the
quotes in JSON Lines format, containing the text and author, which will look like this::
{"author": "Jane Austen", "text": "\u201cThe person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.\u201d"}
{"author": "Steve Martin", "text": "\u201cA day without sunshine is like, you know, night.\u201d"}
{"author": "Garrison Keillor", "text": "\u201cAnyone who thinks sitting in church can make you a Christian must also think that sitting in a garage can make you a car.\u201d"}
...
What just happened?
-------------------
When you ran the command ``scrapy runspider quotes_spider.py``, Scrapy looked for a
Spider definition inside it and ran it through its crawler engine.
The crawl started by making requests to the URLs defined in the ``start_urls``
attribute (in this case, only the URL for quotes in the *humor* category)
and called the default callback method ``parse``, passing the response object as
an argument. In the ``parse`` callback, we loop through the quote elements
using a CSS Selector, yield a Python dict with the extracted quote text and author,
look for a link to the next page and schedule another request using the same
``parse`` method as callback.
Here you will notice one of the main advantages of Scrapy: requests are
:ref:`scheduled and processed asynchronously `. This
means that Scrapy doesn't need to wait for a request to be finished and
processed, it can send another request or do other things in the meantime. This
also means that other requests can keep going even if a request fails or an
error happens while handling it.
While this enables you to do very fast crawls (sending multiple concurrent
requests at the same time, in a fault-tolerant way) Scrapy also gives you
control over the politeness of the crawl through :ref:`a few settings
`. You can do things like setting a download delay between
each request, limiting the amount of concurrent requests per domain, and
even :ref:`using an auto-throttling extension ` that tries
to figure these settings out automatically.
.. note::
This is using :ref:`feed exports ` to generate the
JSON Lines file, you can easily change the export format (XML or CSV, for
example) or the storage backend (FTP or `Amazon S3`_, for example). You can
also write an :ref:`item pipeline ` to store the
items in a database.
.. _topics-whatelse:
What else?
==========
You've seen how to extract and store items from a website using Scrapy, but
this is just the surface. Scrapy provides a lot of powerful features for making
scraping easy and efficient, such as:
* Built-in support for :ref:`selecting and extracting ` data
from HTML/XML sources using extended CSS selectors and XPath expressions,
with helper methods for extraction using regular expressions.
* An :ref:`interactive shell console ` (IPython aware) for trying
out the CSS and XPath expressions to scrape data, which is very useful when writing or
debugging your spiders.
* Built-in support for :ref:`generating feed exports ` in
multiple formats (JSON, CSV, XML) and storing them in multiple backends (FTP,
S3, local filesystem)
* Robust encoding support and auto-detection, for dealing with foreign,
non-standard and broken encoding declarations.
* :ref:`Strong extensibility support `, allowing you to plug
in your own functionality using :ref:`signals ` and a
well-defined API (middlewares, :ref:`extensions `, and
:ref:`pipelines `).
* A wide range of built-in extensions and middlewares for handling:
- cookies and session handling
- HTTP features like compression, authentication, caching
- user-agent spoofing
- robots.txt
- crawl depth restriction
- and more
* Several ways to :ref:`execute code inside a Scrapy process
`, to introspect and debug it.
* An official :ref:`plugin for coding agents `.
* Plus other goodies like reusable spiders to crawl sites from `Sitemaps`_ and
XML/CSV feeds, a media pipeline for :ref:`automatically downloading images
` (or any other media) associated with the scraped
items, a caching DNS resolver, and much more!
What's next?
============
The next steps for you are to :ref:`install Scrapy `,
:ref:`follow through the tutorial ` to learn how to create
a full-blown Scrapy project and `join the community`_. Thanks for your
interest!
.. _join the community: https://www.scrapy.org/community
.. _web scraping: https://en.wikipedia.org/wiki/Web_scraping
.. _Amazon Associates Web Services: https://affiliate-program.amazon.com/welcome/ecs
.. _Amazon S3: https://aws.amazon.com/s3/
.. _Sitemaps: https://www.sitemaps.org/index.html
scrapy-scrapy-5569de3/docs/intro/tutorial.rst 0000664 0000000 0000000 00000074254 15250510124 0021347 0 ustar 00root root 0000000 0000000 .. _intro-tutorial:
===============
Scrapy Tutorial
===============
In this tutorial, we'll assume that Scrapy is already installed on your system.
If that's not the case, see :ref:`intro-install`.
We are going to scrape `quotes.toscrape.com `_, a website
that lists quotes from famous authors.
This tutorial will walk you through these tasks:
1. Creating a new Scrapy project
2. Writing a :ref:`spider ` to crawl a site and extract data
3. Exporting the scraped data using the command line
4. Changing spider to recursively follow links
5. Using spider arguments
Scrapy is written in Python_. The more you learn about Python, the more you
can get out of Scrapy.
If you're already familiar with other languages and want to learn Python quickly, the
`Python Tutorial`_ is a good resource.
If you're new to programming and want to start with Python, the following books
may be useful to you:
* `Automate the Boring Stuff With Python`_
* `How To Think Like a Computer Scientist`_
* `Learn Python 3 The Hard Way`_
You can also take a look at `this list of Python resources for non-programmers`_,
as well as the `suggested resources in the learnpython-subreddit`_.
.. _Python: https://www.python.org/
.. _this list of Python resources for non-programmers: https://wiki.python.org/moin/BeginnersGuide/NonProgrammers
.. _Python Tutorial: https://docs.python.org/3/tutorial
.. _Automate the Boring Stuff With Python: https://automatetheboringstuff.com/
.. _How To Think Like a Computer Scientist: http://openbookproject.net/thinkcs/python/english3e/
.. _Learn Python 3 The Hard Way: https://learnpythonthehardway.org/python3/
.. _suggested resources in the learnpython-subreddit: https://www.reddit.com/r/learnpython/wiki/index#wiki_new_to_python.3F
Creating a project
==================
Before you start scraping, you will have to set up a new Scrapy project. Enter a
directory where you'd like to store your code and run::
scrapy startproject tutorial
This will create a ``tutorial`` directory with the following contents::
tutorial/
scrapy.cfg # deploy configuration file
tutorial/ # project's Python module, you'll import your code from here
__init__.py
items.py # project items definition file
middlewares.py # project middlewares file
pipelines.py # project pipelines file
settings.py # project settings file
spiders/ # a directory where you'll later put your spiders
__init__.py
Before crawling anything, open ``settings.py`` and uncomment the
:setting:`USER_AGENT` line to identify yourself, e.g. a project name plus a URL
or an email address. Website owners who take issue with your crawler can then
ask you to adjust it, rather than block it.
Our first Spider
================
Spiders are classes that you define and that Scrapy uses to scrape information from a website
(or a group of websites). They must subclass :class:`~scrapy.Spider` and define the initial
requests to be made, and optionally, how to follow links in pages and parse the downloaded
page content to extract data.
This is the code for our first Spider. Save it in a file named
``quotes_spider.py`` under the ``tutorial/spiders`` directory in your project:
.. code-block:: python
from pathlib import Path
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
async def start(self):
urls = [
"https://quotes.toscrape.com/page/1/",
"https://quotes.toscrape.com/page/2/",
]
for url in urls:
yield scrapy.Request(url=url, callback=self.parse)
def parse(self, response):
page = response.url.split("/")[-2]
filename = f"quotes-{page}.html"
Path(filename).write_bytes(response.body)
self.log(f"Saved file {filename}")
As you can see, our Spider subclasses :class:`scrapy.Spider `
and defines some attributes and methods:
* :attr:`~scrapy.Spider.name`: identifies the Spider. It must be
unique within a project, that is, you can't set the same name for different
Spiders.
* :meth:`~scrapy.Spider.start`: must be an asynchronous generator that
yields requests (and, optionally, items) for the spider to start crawling.
Subsequent requests will be generated successively from these initial
requests.
* :meth:`~scrapy.Spider.parse`: a method that will be called to handle
the response downloaded for each of the requests made. The response parameter
is an instance of :class:`~scrapy.http.TextResponse` that holds
the page content and has further helpful methods to handle it.
The :meth:`~scrapy.Spider.parse` method usually parses the response, extracting
the scraped data as dicts and also finding new URLs to
follow and creating new requests (:class:`~scrapy.Request`) from them.
How to run our spider
---------------------
To put our spider to work, go to the project's top level directory and run::
scrapy crawl quotes
This command runs the spider named ``quotes`` that we've just added, that
will send some requests for the ``quotes.toscrape.com`` domain. You will get an output
similar to this::
... (omitted for brevity)
2016-12-16 21:24:05 [scrapy.core.engine] INFO: Spider opened
2016-12-16 21:24:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:24:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) (referer: None)
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None)
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None)
2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-1.html
2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-2.html
2016-12-16 21:24:05 [scrapy.core.engine] INFO: Closing spider (finished)
...
Now, check the files in the current directory. You should notice that two new
files have been created: *quotes-1.html* and *quotes-2.html*, with the content
for the respective URLs, as our ``parse`` method instructs.
.. note:: If you are wondering why we haven't parsed the HTML yet, hold
on, we will cover that soon.
What just happened under the hood?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Scrapy sends the first :class:`scrapy.Request ` objects yielded
by the :meth:`~scrapy.Spider.start` spider method. Upon receiving a
response for each one, Scrapy calls the callback method associated with the
request (in this case, the ``parse`` method) with a
:class:`~scrapy.http.Response` object.
A shortcut to the ``start`` method
----------------------------------
Instead of implementing a :meth:`~scrapy.Spider.start` method that yields
:class:`~scrapy.Request` objects from URLs, you can define a
:attr:`~scrapy.Spider.start_urls` class attribute with a list of URLs. This
list will then be used by the default implementation of
:meth:`~scrapy.Spider.start` to create the initial requests for your
spider.
.. code-block:: python
from pathlib import Path
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
"https://quotes.toscrape.com/page/1/",
"https://quotes.toscrape.com/page/2/",
]
def parse(self, response):
page = response.url.split("/")[-2]
filename = f"quotes-{page}.html"
Path(filename).write_bytes(response.body)
The :meth:`~scrapy.Spider.parse` method will be called to handle each
of the requests for those URLs, even though we haven't explicitly told Scrapy
to do so. This happens because :meth:`~scrapy.Spider.parse` is Scrapy's
default callback method, which is called for requests without an explicitly
assigned callback.
Extracting data
---------------
The best way to learn how to extract data with Scrapy is trying selectors
using the :ref:`Scrapy shell `. Run::
scrapy shell 'https://quotes.toscrape.com/page/1/'
.. note::
Remember to always enclose URLs in quotes when running Scrapy shell from the
command line, otherwise URLs containing arguments (i.e. ``&`` character)
will not work.
On Windows, use double quotes instead::
scrapy shell "https://quotes.toscrape.com/page/1/"
You will see something like::
[ ... Scrapy log here ... ]
2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) (referer: None)
[s] Available Scrapy objects:
[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)
[s] crawler
[s] item {}
[s] request
[s] response <200 https://quotes.toscrape.com/page/1/>
[s] settings
[s] spider
[s] Useful shortcuts:
[s] shelp() Shell help (print this help)
[s] fetch(req_or_url) Fetch request (or URL) and update local objects
[s] view(response) View response in a browser
Using the shell, you can try selecting elements using `CSS`_ with the response
object:
.. invisible-code-block: python
response = load_response('https://quotes.toscrape.com/page/1/', 'quotes1.html')
.. code-block:: pycon
>>> response.css("title")
[]
The result of running ``response.css('title')`` is a list-like object called
:class:`~scrapy.selector.SelectorList`, which represents a list of
:class:`~scrapy.Selector` objects that wrap around XML/HTML elements
and allow you to run further queries to refine the selection or extract the
data.
To extract the text from the title above, you can do:
.. code-block:: pycon
>>> response.css("title::text").getall()
['Quotes to Scrape']
There are two things to note here: one is that we've added ``::text`` to the
CSS query, to mean we want to select only the text elements directly inside
```` element. If we don't specify ``::text``, we'd get the full title
element, including its tags:
.. code-block:: pycon
>>> response.css("title").getall()
['<title>Quotes to Scrape']
The other thing is that the result of calling ``.getall()`` is a list: it is
possible that a selector returns more than one result, so we extract them all.
When you know you just want the first result, as in this case, you can do:
.. code-block:: pycon
>>> response.css("title::text").get()
'Quotes to Scrape'
As an alternative, you could've written:
.. code-block:: pycon
>>> response.css("title::text")[0].get()
'Quotes to Scrape'
Accessing an index on a :class:`~scrapy.selector.SelectorList` instance will
raise an :exc:`IndexError` exception if there are no results:
.. code-block:: pycon
>>> response.css("noelement")[0].get()
Traceback (most recent call last):
...
IndexError: list index out of range
You might want to use ``.get()`` directly on the
:class:`~scrapy.selector.SelectorList` instance instead, which returns ``None``
if there are no results:
.. code-block:: pycon
>>> response.css("noelement").get()
There's a lesson here: for most scraping code, you want it to be resilient to
errors due to things not being found on a page, so that even if some parts fail
to be scraped, you can at least get **some** data.
Besides the :meth:`~scrapy.selector.SelectorList.getall` and
:meth:`~scrapy.selector.SelectorList.get` methods, you can also use
the :meth:`~scrapy.selector.SelectorList.re` method to extract using
:doc:`regular expressions `:
.. code-block:: pycon
>>> response.css("title::text").re(r"Quotes.*")
['Quotes to Scrape']
>>> response.css("title::text").re(r"Q\w+")
['Quotes']
>>> response.css("title::text").re(r"(\w+) to (\w+)")
['Quotes', 'Scrape']
In order to find the proper CSS selectors to use, you might find it useful to open
the response page from the shell in your web browser using ``view(response)``.
You can use your browser's developer tools to inspect the HTML and come up
with a selector (see :ref:`topics-developer-tools`).
`Selector Gadget`_ is also a nice tool to quickly find CSS selector for
visually selected elements, which works in many browsers.
.. _Selector Gadget: https://selectorgadget.com/
XPath: a brief intro
^^^^^^^^^^^^^^^^^^^^
Besides `CSS`_, Scrapy selectors also support using `XPath`_ expressions:
.. code-block:: pycon
>>> response.xpath("//title")
[]
>>> response.xpath("//title/text()").get()
'Quotes to Scrape'
XPath expressions are very powerful, and are the foundation of Scrapy
Selectors. In fact, CSS selectors are converted to XPath under-the-hood. You
can see that if you read the text representation of the selector
objects in the shell closely.
While perhaps not as popular as CSS selectors, XPath expressions offer more
power because besides navigating the structure, it can also look at the
content. Using XPath, you're able to select things like: *the link
that contains the text "Next Page"*. This makes XPath very fitting to the task
of scraping, and we encourage you to learn XPath even if you already know how to
construct CSS selectors, it will make scraping much easier.
We won't cover much of XPath here, but you can read more about :ref:`using XPath
with Scrapy Selectors here `. To learn more about XPath, we
recommend `this tutorial to learn XPath through examples
`_, and `this tutorial to learn "how
to think in XPath" `_.
.. _XPath: https://www.w3.org/TR/xpath-10/
.. _CSS: https://www.w3.org/TR/selectors
Extracting quotes and authors
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Now that you know a bit about selection and extraction, let's complete our
spider by writing the code to extract the quotes from the web page.
Each quote in https://quotes.toscrape.com is represented by HTML elements that look
like this:
.. code-block:: html
Name: My image 2 
Name: My image 3 
Name: My image 4 
Name: My image 5 