Command Line Interface#

Installing the package puts a scraper command on your PATH. The same parser is reachable three ways, all equivalent:

scraper --url <URL> --patterns .csv .zip --dir data --max-files 10
python -m scraper_lib --url <URL> --patterns .csv .zip --dir data --max-files 10
python -m scraper_lib.cli --url <URL> --patterns .csv .zip --dir data --max-files 10

Options#

The list below is generated by Sphinx from scraper_lib.ScraperLib.build_arg_parser(), the parser the command actually uses, so it cannot drift from the code.

Generic parallel file downloader

usage: scraper [-h] --url URL --patterns PATTERNS [PATTERNS ...] [--dir DIR]
               [--incremental] [--max-files MAX_FILES]
               [--max-concurrent MAX_CONCURRENT] [--state-file STATE_FILE]
               [--log-file LOG_FILE] [--report-prefix REPORT_PREFIX]
               [--headers HEADERS] [--user-agents USER_AGENTS]
               [--disable-logging] [--disable-terminal-logging]
               [--dataset-name DATASET_NAME] [--disable-progress-bar]
               [--output-dir OUTPUT_DIR] [--max-old-logs [MAX_OLD_LOGS]]
               [--max-old-runs [MAX_OLD_RUNS]] [--chunk-size CHUNK_SIZE]
               [--initial-delay INITIAL_DELAY] [--max-delay MAX_DELAY]
               [--max-retries MAX_RETRIES]

Named Arguments#

--url

Base URL to scrape for files

--patterns

List of file patterns to match (e.g. .csv .zip)

--dir

Download directory

Default: 'data'

--incremental

Enable incremental download state

Default: False

--max-files

Limit number of files to download

--max-concurrent

Max parallel downloads

--state-file

Path for download state file

Default: 'download_state.json'

--log-file

Path for main log file

Default: 'process_log.log'

--report-prefix

Prefix for report files

Default: 'download_report'

--headers

Path to JSON file with custom headers

--user-agents

Path to text file with custom user agents (one per line)

--disable-logging

Disable all logging for production pipelines

Default: False

--disable-terminal-logging

Disable terminal logging

Default: False

--dataset-name

Dataset name for banner

--disable-progress-bar

Disable tqdm progress bar

Default: False

--output-dir

Directory for report PNGs and JSON

Default: '.'

--max-old-logs

Max old log files to keep (default: 25, None disables rotation)

Default: 25

--max-old-runs

Max old report/png runs to keep (default: 25, None disables rotation)

Default: 25

--chunk-size

Chunk size for downloads (e.g. 1gb, 10mb, 8 bytes)

Default: '5mb'

--initial-delay

Initial delay between retries (seconds)

Default: 1.0

--max-delay

Maximum delay between retries (seconds)

Default: 60.0

--max-retries

Maximum number of download retries

Default: 5

Examples#

Download every CSV published on a page:

scraper --url https://data.example.com --patterns .csv

Limit the run to 10 files and put them in data/:

scraper --url https://data.example.com --patterns .csv --dir data --max-files 10

Resume an interrupted run – --incremental reads the state file and skips what is already on disk:

scraper --url https://data.example.com --patterns .csv .zip \
    --dir data --incremental --state-file state/download_state.json

Run 16 downloads in parallel with 10 MB chunks and quiet the progress bar, which is what you want inside a scheduled pipeline:

scraper --url https://data.example.com --patterns .parquet \
    --dir data --max-concurrent 16 --chunk-size 10mb \
    --disable-progress-bar --disable-terminal-logging
On this page

This Page