paper_firehose.commands.pqa_summary¶
PDF full-text summaries¶
This command selects entries from papers.db for a given topic with
rank_score >= the configured threshold and downloads arXiv PDFs for them,
adhering to arXiv API Terms of Use (polite rate limiting and descriptive
User-Agent with contact email).
Workflow¶
Load configuration and identify ranked entries above threshold
Download arXiv PDFs (respecting rate limits, reusing archived copies)
With
summarize=True(CLI--summarize): extract each PDF’s text and summarize it with one OpenAI call (processors/pdf_summarizer.py)Write
{"summary", "methods"}JSON into thepaper_qa_summarycolumn of papers.db and matched_entries_history.db
The command and the paperqa: topic section keep their historical names;
paper-qa itself is no longer used.
Functions
|
Download arXiv PDFs for ranked entries and optionally summarize them. |
- paper_firehose.commands.pqa_summary.run(config_path, topic=None, *, rps=None, limit=None, arxiv=None, entry_ids=None, use_history=False, history_date=None, history_feed_like=None, summarize=False)[source]¶
Download arXiv PDFs for ranked entries and optionally summarize them.
- Return type:
- Parameters:
Workflow overview¶
Load configuration/database state and prepare download/archive folders.
Determine targets either from ranked topic entries (respecting the download rank threshold and optional
limit) or from the explicitarxiv/entry_idsarguments, optionally pulling metadata from the history database whenuse_historyis enabled.Resolve arXiv IDs, reuse archived PDFs when possible, download missing PDFs under the configured rate limit, and archive successful downloads.
When
summarizeis True, summarize each PDF with one OpenAI call using the topic’spaperqa.llm/promptand write the JSON result back to bothpapers.dbandmatched_entries_history.dbwhen anentry_idis available.rpsoverrides the per-topic (or default) arXiv request rate.