ScraperLib

class scraper_lib.ScraperLib.ScraperLib(base_url: str, file_patterns: List[str], download_dir: str = 'downloads', state_file: str = 'state/download_state.json', log_file: str = 'logs/scraper_log.log', output_dir: str = 'output', incremental: bool = True, max_files: int | None = None, max_concurrent: int | None = None, headers: Dict[str, str] | None = None, user_agents: List[str] | None = None, report_prefix: str = 'download_report', disable_logging: bool = False, disable_terminal_logging: bool = False, dataset_name: str | None = None, disable_progress_bar: bool = False, max_old_logs: int = 10, max_old_runs: int = 10, ray_instance: Any | None = None, chunk_size: str | int = '5MB', initial_delay: float = 1.0, max_delay: float = 60.0, max_retries: int = 5)

Bases: object

Library for parallel file extraction and download, report generation, and log/result rotation.

static build_arg_parser() → ArgumentParser

Build the argument parser used by every entry point.

Exposed separately so that docs/cli.rst can render the flags with the .. argparse:: directive straight from this definition. The page used to be maintained by hand and documented --output, --max-workers and --resume, none of which ever existed here.

Returns:

Parser with every documented flag.

Return type:

argparse.ArgumentParser

The flag list is deliberately NOT repeated in prose here or in the parser epilog: add_argument below is the single source of truth and docs/cli.rst renders it directly.

static cli(argv: List[str] | None = None) → None

Command-line interface to run ScraperLib.

Parses arguments, instantiates ScraperLib and runs the process. Reached by scraper, python -m scraper_lib and python -m scraper_lib.cli – see scraper_lib.cli.

Parameters:

argv (Optional[List[str]]) – Argument list; defaults to sys.argv[1:].

Usage example:

python -m scraper_lib –url <URL> –patterns .csv .zip –dir data –max-files 10

run() → None

Runs the main scraping, download, and report generation flow.

This method initializes the logger, prepares directories, collects links, performs parallel downloads, and generates reports and charts at the end of the process.