paper_firehose.commands.pqa_summary

PDF full-text summaries

This command selects entries from papers.db for a given topic with rank_score >= the configured threshold and downloads arXiv PDFs for them, adhering to arXiv API Terms of Use (polite rate limiting and descriptive User-Agent with contact email).

Workflow

  1. Load configuration and identify ranked entries above threshold

  2. Download arXiv PDFs (respecting rate limits, reusing archived copies)

  3. With summarize=True (CLI --summarize): extract each PDF’s text and summarize it with one OpenAI call (processors/pdf_summarizer.py)

  4. Write {"summary", "methods"} JSON into the paper_qa_summary column of papers.db and matched_entries_history.db

The command and the paperqa: topic section keep their historical names; paper-qa itself is no longer used.

Functions

run(config_path[, topic, rps, limit, arxiv, ...])

Download arXiv PDFs for ranked entries and optionally summarize them.

paper_firehose.commands.pqa_summary.run(config_path, topic=None, *, rps=None, limit=None, arxiv=None, entry_ids=None, use_history=False, history_date=None, history_feed_like=None, summarize=False)[source]

Download arXiv PDFs for ranked entries and optionally summarize them.

Return type:

None

Parameters:
  • config_path (str)

  • topic (str | None)

  • rps (float | None)

  • limit (int | None)

  • arxiv (List[str] | None)

  • entry_ids (List[str] | None)

  • use_history (bool)

  • history_date (str | None)

  • history_feed_like (str | None)

  • summarize (bool)

Workflow overview

  • Load configuration/database state and prepare download/archive folders.

  • Determine targets either from ranked topic entries (respecting the download rank threshold and optional limit) or from the explicit arxiv/entry_ids arguments, optionally pulling metadata from the history database when use_history is enabled.

  • Resolve arXiv IDs, reuse archived PDFs when possible, download missing PDFs under the configured rate limit, and archive successful downloads.

  • When summarize is True, summarize each PDF with one OpenAI call using the topic’s paperqa.llm/prompt and write the JSON result back to both papers.db and matched_entries_history.db when an entry_id is available.

  • rps overrides the per-topic (or default) arXiv request rate.