DownloadState#

class scraper_lib.ScraperLib.DownloadState(state_file: str = 'download_state.json', incremental: bool = True)[source]#

Bases: object

Manages the persistent state of downloads, including completed, failed, and delay statistics.

This class provides atomic file operations for safe concurrent access, tracks download progress, and maintains statistics for reporting and incremental downloads.

Parameters:

  • state_file (str) – Path to the state file (JSON).

  • incremental (bool) – If True, loads existing state; otherwise, starts fresh.

add_completed(url: str, filepath: str, size: int) → None[source]#

Mark a file as successfully downloaded.

Parameters:

  • url (str) – File URL.

  • filepath (str) – Local file path.

  • size (int) – File size in bytes.

add_delay(delay: float, success: bool = True) → None[source]#

Record a new delay and update statistics.

Parameters:

  • delay (float) – Delay value in seconds.

  • success (bool) – True if the delay was for a successful download.

add_failed(url: str, error: Any) → None[source]#

Mark a file as failed to download.

Parameters:

  • url (str) – File URL.

  • error (str) – Error message.

generate() → Dict[str, Any][source]#

Initialize a fresh state structure and salva no disco.

Returns:

The initialized state dictionary.

Return type:

dict

get_file_id(url: str) → str[source]#

Generate a unique file ID for a given URL.

Parameters:

url (str) – File URL.

Returns:

MD5 hash of the URL.

Return type:

str

is_completed(url: str, data_dir: str | None = None) → bool[source]#

Check if a file has already been downloaded and exists on disk.

Parameters:

  • url (str) – File URL.

  • data_dir (str, optional) – Directory where files are stored.

Returns:

True if completed and file exists, False otherwise.

Return type:

bool

load_state() → None[source]#

Load state from disk into memory.

Reads the JSON from the configured state_file path and populates the internal cache. If the file does not exist or contains invalid JSON, a fresh state is generated and saved.

Returns:

None

Raises:

IOError – If the state file cannot be opened or read.

save_state() → None[source]#

Save the current state to disk.

Updates the ‘last_update’ timestamp and writes the in-memory state cache to the state file using atomic (locked) file operations for safety.

Returns:

None

Raises:

IOError – If the state file cannot be written due to I/O errors.

property state: Dict[str, Any]#

Get the current state dictionary.

Returns:

The current state.

Return type:

dict