Structured Extraction from a PDF¶
This notebook demonstrates structured_output() applied to a real-world task:
extracting structured metadata from a research paper that has been converted
to markdown.
We fetch the ReAct: Synergizing Reasoning and Acting in Language Models
paper (Yao et al., 2022), parse its first three pages to markdown using
pymupdf4llm, then ask the LLM to fill in a Pydantic model with the paper's
key fields.
Chapter 3 concept:
structured_output()accepts any text prompt and a Pydantic model class, and returns a validated instance of that model. Here we use it to turn unstructured PDF text into a typed Python object in a single call.
Setting up the LLM¶
I recommend running these notebooks on Ollama Cloud. It serves much larger models than most laptops can run. The default cloud model is picked by benchmarking candidates on these notebooks, and an automated run checks every notebook against it each week. The notebook status page shows the current model and results.
- Create an API key at https://ollama.com/settings/keys.
- Export it before launching Jupyter:
export OLLAMA_API_KEY=<your key>. - That's all. When a notebook builds its LLM, it detects the key and uses Ollama Cloud.
Prefer to run locally? Install Ollama from https://ollama.com/download
and pull the model with ollama pull qwen3:14b. If no key is set, the
notebook starts the Ollama service for you, so there is no need to run
ollama serve. Local models are smaller, so your outputs will differ from
the recorded ones.
Prefer OpenAI or Anthropic?
- Install the extra:
uv sync --extra openaioruv sync --extra anthropic. - Export
OPENAI_API_KEYorANTHROPIC_API_KEYbefore launching Jupyter. If you forget, you will be prompted for it, which keeps the key out of the saved notebook. - Set
LLM_PROVIDER=openai(oranthropic), or passprovider="openai"tomake_llm(). A key on its own never switches providers, so one exported for unrelated work cannot reroute you.
The switch applies wherever a notebook builds its LLM with make_llm().
A notebook that constructs OllamaLLM directly stays on Ollama. Output on
gpt-5 or Claude will differ from the recorded outputs, and prompt-sensitive
examples (the ch09 evaluator pattern, ch08 supervised trajectories) may
behave noticeably differently.
In [1]:
Copied!
# Uncomment the line below to install `llm-agents-from-scratch` from PyPI
# !pip install llm-agents-from-scratch
# Uncomment the line below to install `llm-agents-from-scratch` from PyPI
# !pip install llm-agents-from-scratch
In [2]:
Copied!
import urllib.request
from llm_agents_from_scratch.notebook_utils import make_llm
import urllib.request
from llm_agents_from_scratch.notebook_utils import make_llm
Installing the PDF Parser¶
We use pymupdf4llm
to convert PDF pages to markdown. It is not part of the core
llm-agents-from-scratch dependencies, so we install it here.
In [3]:
Copied!
!uv pip install pymupdf4llm
!uv pip install pymupdf4llm
Using Python 3.13.5 environment at: /home/nerdai/Projects/llm-agents-from-scratch/.venv ⠋ Resolving dependencies... ⠋ Resolving dependencies... ⠙ Resolving dependencies... ⠙ pymupdf4llm==1.28.2 ⠙ pymupdf==1.28.2 ⠙ pymupdf-layout==1.28.2 ⠙ tabulate==0.10.0 ⠙ psutil==7.2.2 ⠙ pyyaml==6.0.3 ⠙ numpy==2.4.4 ⠙ onnxruntime==1.26.0 ⠙ networkx==3.7 ⠙ flatbuffers==25.12.19 ⠙ packaging==26.2 ⠙ protobuf==6.33.6 ⠙ Resolved 12 packages in 130ms ⠋ Preparing packages... (0/0) ⠋ Preparing packages... (0/4) ⠙ Preparing packages... (0/4) ⠙ Preparing packages... (0/4) networkx ------------------------------ 0 B/2.04 MiB ⠙ Preparing packages... (0/4) networkx ------------------------------ 0 B/2.04 MiB ⠙ Preparing packages... (0/4) networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 0 B/172.27 KiB networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 16.00 KiB/172.27 KiB networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 32.00 KiB/172.27 KiB networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 48.00 KiB/172.27 KiB networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 63.17 KiB/172.27 KiB networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 79.17 KiB/172.27 KiB networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 95.17 KiB/172.27 KiB networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 111.17 KiB/172.27 KiB networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 127.17 KiB/172.27 KiB networkx ------------------------------ 0 B/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 127.17 KiB/172.27 KiB networkx ------------------------------ 16.00 KiB/2.04 MiB pymupdf-layout ------------------------------ 0 B/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 127.17 KiB/172.27 KiB networkx ------------------------------ 16.00 KiB/2.04 MiB pymupdf-layout ------------------------------ 16.00 KiB/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf4llm ------------------------------ 127.17 KiB/172.27 KiB networkx ------------------------------ 16.00 KiB/2.04 MiB pymupdf-layout ------------------------------ 16.00 KiB/40.95 MiB ⠙ Preparing packages... (0/4) networkx ------------------------------ 94.44 KiB/2.04 MiB pymupdf ------------------------------ 96.00 KiB/24.57 MiB pymupdf-layout ------------------------------ 92.85 KiB/40.95 MiB ⠙ Preparing packages... (0/4) networkx ------------------------------ 526.44 KiB/2.04 MiB pymupdf ------------------------------ 954.89 KiB/24.57 MiB pymupdf-layout ------------------------------ 796.85 KiB/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf ------------------------------ 2.39 MiB/24.57 MiB pymupdf-layout ------------------------------ 1.98 MiB/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf ------------------------------ 3.39 MiB/24.57 MiB pymupdf-layout ------------------------------ 2.93 MiB/40.95 MiB ⠙ Preparing packages... (0/4) pymupdf ------------------------------ 6.03 MiB/24.57 MiB pymupdf-layout ------------------------------ 5.84 MiB/40.95 MiB ⠹ Preparing packages... (2/4) pymupdf ------------------------------ 8.68 MiB/24.57 MiB pymupdf-layout ------------------------------ 7.53 MiB/40.95 MiB ⠹ Preparing packages... (2/4) pymupdf ------------------------------ 12.11 MiB/24.57 MiB pymupdf-layout ------------------------------ 10.50 MiB/40.95 MiB ⠹ Preparing packages... (2/4) pymupdf ------------------------------ 14.86 MiB/24.57 MiB pymupdf-layout ------------------------------ 13.42 MiB/40.95 MiB ⠹ Preparing packages... (2/4) pymupdf ------------------------------ 17.62 MiB/24.57 MiB pymupdf-layout ------------------------------ 16.33 MiB/40.95 MiB ⠸ Preparing packages... (2/4) pymupdf ------------------------------ 20.26 MiB/24.57 MiB pymupdf-layout ------------------------------ 19.05 MiB/40.95 MiB ⠸ Preparing packages... (2/4) pymupdf ------------------------------ 23.12 MiB/24.57 MiB pymupdf-layout ------------------------------ 22.01 MiB/40.95 MiB ⠸ Preparing packages... (2/4) pymupdf-layout ------------------------------ 24.03 MiB/40.95 MiB ⠸ Preparing packages... (2/4) pymupdf-layout ------------------------------ 26.15 MiB/40.95 MiB ⠸ Preparing packages... (2/4) pymupdf-layout ------------------------------ 31.57 MiB/40.95 MiB ⠼ Preparing packages... (3/4) pymupdf-layout ------------------------------ 36.71 MiB/40.95 MiB ⠼ Preparing packages... (3/4) Prepared 4 packages in 638ms ░░░░░░░░░░░░░░░░░░░░ [0/0] Installing wheels... ░░░░░░░░░░░░░░░░░░░░ [0/5] Installing wheels... ░░░░░░░░░░░░░░░░░░░░ [0/5] pymupdf-layout==1.28.2 ████░░░░░░░░░░░░░░░░ [1/5] pymupdf-layout==1.28.2 ████░░░░░░░░░░░░░░░░ [1/5] pymupdf==1.28.2 ████████░░░░░░░░░░░░ [2/5] pymupdf==1.28.2 ████████░░░░░░░░░░░░ [2/5] tabulate==0.10.0 ████████████░░░░░░░░ [3/5] tabulate==0.10.0 ████████████░░░░░░░░ [3/5] pymupdf4llm==1.28.2 ████████████████░░░░ [4/5] pymupdf4llm==1.28.2 ████████████████░░░░ [4/5] networkx==3.7 ████████████████████ [5/5] networkx==3.7 Installed 5 packages in 7ms + networkx==3.7 + pymupdf==1.28.2 + pymupdf-layout==1.28.2 + pymupdf4llm==1.28.2 + tabulate==0.10.0
Fetching the Paper¶
We download the first three pages of the ReAct paper directly from arXiv. These pages cover the title, authors, abstract, and opening sections — enough context for the LLM to fill in all extraction fields.
In [4]:
Copied!
import tempfile
from pathlib import Path
import pymupdf4llm
PDF_URL = "https://arxiv.org/pdf/2210.03629"
PDF_PAGES = [0, 1, 2] # title, abstract, intro
req = urllib.request.Request(
PDF_URL,
headers={"User-Agent": "llm-agents-from-scratch/1.0"},
)
with urllib.request.urlopen(req) as resp:
pdf_bytes = resp.read()
print(f"Downloaded {len(pdf_bytes):,} bytes")
import tempfile
from pathlib import Path
import pymupdf4llm
PDF_URL = "https://arxiv.org/pdf/2210.03629"
PDF_PAGES = [0, 1, 2] # title, abstract, intro
req = urllib.request.Request(
PDF_URL,
headers={"User-Agent": "llm-agents-from-scratch/1.0"},
)
with urllib.request.urlopen(req) as resp:
pdf_bytes = resp.read()
print(f"Downloaded {len(pdf_bytes):,} bytes")
Downloaded 633,805 bytes
Parsing PDF to Markdown¶
In [5]:
Copied!
with tempfile.NamedTemporaryFile(
suffix=".pdf",
delete=False,
) as tmp:
tmp.write(pdf_bytes)
tmp_path = Path(tmp.name)
try:
md_text = pymupdf4llm.to_markdown(str(tmp_path), pages=PDF_PAGES)
finally:
tmp_path.unlink(missing_ok=True)
print(f"Extracted {len(md_text):,} characters of markdown")
print("--- preview (first 500 chars) ---")
print(md_text[:500])
with tempfile.NamedTemporaryFile(
suffix=".pdf",
delete=False,
) as tmp:
tmp.write(pdf_bytes)
tmp_path = Path(tmp.name)
try:
md_text = pymupdf4llm.to_markdown(str(tmp_path), pages=PDF_PAGES)
finally:
tmp_path.unlink(missing_ok=True)
print(f"Extracted {len(md_text):,} characters of markdown")
print("--- preview (first 500 chars) ---")
print(md_text[:500])
Extracted 15,008 characters of markdown
--- preview (first 500 chars) ---
Published as a conference paper at ICLR 2023
# REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS
Shunyu Yao<sup>_∗_*,1</sup> , Jeffrey Zhao<sup>2</sup> , Dian Yu<sup>2</sup> , Nan Du<sup>2</sup> , Izhak Shafran<sup>2</sup> , Karthik Narasimhan<sup>1</sup> , Yuan Cao<sup>2</sup>
1Department of Computer Science, Princeton University
2Google Research, Brain team
1{shunyuy,karthikn}@princeton.edu
2{jeffreyzhao,dianyu,dunan,izhak,yuancao}@google.com
## ABSTRACT
While large la
Defining the Extraction Model¶
We define a Pydantic model that captures the key metadata fields we want
to pull from the paper. The LLM will populate every field from the markdown
text in a single structured_output() call.
In [6]:
Copied!
from pydantic import BaseModel, Field
class PaperSummary(BaseModel):
"""Structured metadata extracted from a research paper."""
title: str = Field(description="Full title of the paper.")
authors: list[str] = Field(
description="List of author names as they appear on the paper.",
)
year: int = Field(
description="Year the paper was published or submitted.",
)
abstract: str = Field(
description="The paper's abstract, faithfully reproduced.",
)
key_contributions: list[str] = Field(
description=(
"Three to five concise bullet points summarising "
"the paper's main contributions."
),
)
primary_topic: str = Field(
description=(
"One short phrase describing the paper's primary research topic "
"(e.g. 'LLM reasoning', 'tool use', 'multi-agent systems')."
),
)
from pydantic import BaseModel, Field
class PaperSummary(BaseModel):
"""Structured metadata extracted from a research paper."""
title: str = Field(description="Full title of the paper.")
authors: list[str] = Field(
description="List of author names as they appear on the paper.",
)
year: int = Field(
description="Year the paper was published or submitted.",
)
abstract: str = Field(
description="The paper's abstract, faithfully reproduced.",
)
key_contributions: list[str] = Field(
description=(
"Three to five concise bullet points summarising "
"the paper's main contributions."
),
)
primary_topic: str = Field(
description=(
"One short phrase describing the paper's primary research topic "
"(e.g. 'LLM reasoning', 'tool use', 'multi-agent systems')."
),
)
Extracting Structured Data¶
We pass the markdown text as the prompt and PaperSummary as the target
model. structured_output() returns a fully validated PaperSummary
instance — no parsing or post-processing needed.
In [7]:
Copied!
llm = make_llm()
prompt = (
"Extract structured metadata from the following research paper.\n\n"
f"{md_text}"
)
summary = await llm.structured_output(prompt=prompt, mdl=PaperSummary)
print(type(summary), "\n")
print(summary.model_dump())
llm = make_llm()
prompt = (
"Extract structured metadata from the following research paper.\n\n"
f"{md_text}"
)
summary = await llm.structured_output(prompt=prompt, mdl=PaperSummary)
print(type(summary), "\n")
print(summary.model_dump())
✓ Using Ollama Cloud (kimi-k2.7-code:cloud)
<class '__main__.PaperSummary'>
{'title': 'REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS', 'authors': ['Shunyu Yao', 'Jeffrey Zhao', 'Dian Yu', 'Nan Du', 'Izhak Shafran', 'Karthik Narasimhan', 'Yuan Cao'], 'year': 2023, 'abstract': 'While large language models (LLMs) have demonstrated impressive performance across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics. In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with and gather additional information from external sources such as knowledge bases or environments. We apply our approach, named ReAct, to a diverse set of language and decision making tasks and demonstrate its effectiveness over state-of-the-art baselines in addition to improved human interpretability and trustworthiness. Concretely, on question answering (HotpotQA) and fact verification (Fever), ReAct overcomes prevalent issues of hallucination and error propagation in chain-of-thought reasoning by interacting with a simple Wikipedia API, and generating human-like task-solving trajectories that are more interpretable than baselines without reasoning traces. Furthermore, on two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples.', 'key_contributions': ['Introduce ReAct, a novel prompt-based paradigm that interleaves verbal reasoning traces with task-specific actions in LLMs.', 'Demonstrate strong empirical performance on diverse benchmarks: HotpotQA, Fever, ALFWorld, and WebShop.', 'Show that combining reasoning and acting reduces hallucination and error propagation in reasoning tasks and improves decision making in interactive environments.', 'Provide systematic ablations analyzing the importance of acting for reasoning and reasoning for acting.', 'Perform initial fine-tuning experiments and discuss future directions for scaling ReAct with additional training and reinforcement learning.'], 'primary_topic': 'LLM reasoning and acting'}
Result¶
In [8]:
Copied!
print(f"Title: {summary.title}")
print(f"Year: {summary.year}")
print(f"Authors: {', '.join(summary.authors)}")
print(f"Primary topic: {summary.primary_topic}")
print()
print("Abstract:")
print(summary.abstract)
print()
print("Key contributions:")
for point in summary.key_contributions:
print(f" • {point}")
print(f"Title: {summary.title}")
print(f"Year: {summary.year}")
print(f"Authors: {', '.join(summary.authors)}")
print(f"Primary topic: {summary.primary_topic}")
print()
print("Abstract:")
print(summary.abstract)
print()
print("Key contributions:")
for point in summary.key_contributions:
print(f" • {point}")
Title: REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS Year: 2023 Authors: Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao Primary topic: LLM reasoning and acting Abstract: While large language models (LLMs) have demonstrated impressive performance across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics. In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with and gather additional information from external sources such as knowledge bases or environments. We apply our approach, named ReAct, to a diverse set of language and decision making tasks and demonstrate its effectiveness over state-of-the-art baselines in addition to improved human interpretability and trustworthiness. Concretely, on question answering (HotpotQA) and fact verification (Fever), ReAct overcomes prevalent issues of hallucination and error propagation in chain-of-thought reasoning by interacting with a simple Wikipedia API, and generating human-like task-solving trajectories that are more interpretable than baselines without reasoning traces. Furthermore, on two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples. Key contributions: • Introduce ReAct, a novel prompt-based paradigm that interleaves verbal reasoning traces with task-specific actions in LLMs. • Demonstrate strong empirical performance on diverse benchmarks: HotpotQA, Fever, ALFWorld, and WebShop. • Show that combining reasoning and acting reduces hallucination and error propagation in reasoning tasks and improves decision making in interactive environments. • Provide systematic ablations analyzing the importance of acting for reasoning and reasoning for acting. • Perform initial fine-tuning experiments and discuss future directions for scaling ReAct with additional training and reinforcement learning.