paper_firehose.core.http_client

Shared HTTP client with retry logic and rate limiting.

Classes

RetryableHTTPClient([rps, max_retries, timeout])

HTTP client with exponential backoff retry logic and rate limiting.

class paper_firehose.core.http_client.RetryableHTTPClient(rps=1.0, max_retries=3, timeout=15)

Bases: object

HTTP client with exponential backoff retry logic and rate limiting.

Handles common failure scenarios (429, 500, 502, 503, 504) with exponential backoff, respects Retry-After headers, and enforces rate limiting.

Parameters:

• rps (float) – Maximum requests per second (default: 1.0)

• max_retries (int) – Maximum number of retry attempts (default: 3)

• timeout (int) – Request timeout in seconds (default: 15)

close()

Close the underlying session.

get_with_retry(url, headers=None, params=None, timeout=None, return_none_on_404=True)

Make GET request with exponential backoff retry logic.

Parameters:

• url (str) – URL to fetch

• headers (Optional[Dict[str, str]]) – Optional request headers

• params (Optional[Dict[str, Any]]) – Optional query parameters

• timeout (Optional[int]) – Optional timeout override (uses instance default if None)

• return_none_on_404 (bool) – If True, return None on 404; if False, let it raise

Return type:

Optional[Response]

Returns:

Response object on success, None on 404 (if return_none_on_404=True)

Raises:

• requests.HTTPError – On non-retryable HTTP errors

• requests.RequestException – On network errors after retries exhausted