Notebook¶

Prerequisites: This capstone assumes you've read Chapters 2–4.

Setup Instructions¶

To ensure you have the required dependencies to run this notebook, you'll need to have our llm-agents-from-scratch framework installed on the running Jupyter kernel. To do this, you can launch this notebook with the following command while within the project's root directory:

# we need the notebook-utils and openai extras for this notebook
uv sync --extra notebook-utils --extra openai

# to launch the notebook
uv run --with jupyter jupyter lab

Alternatively, if you just want to use the published version of llm-agents-from-scratch without local development, you can install it from PyPI by uncommenting the cell below.

Setting up the LLM¶

I recommend running these notebooks on Ollama Cloud. It serves much larger models than most laptops can run. The default cloud model is picked by benchmarking candidates on these notebooks, and an automated run checks every notebook against it each week. The notebook status page shows the current model and results.

1. Create an API key at https://ollama.com/settings/keys.
2. Export it before launching Jupyter: export OLLAMA_API_KEY=<your key>.
3. That's all. When a notebook builds its LLM, it detects the key and uses Ollama Cloud.

Prefer to run locally? Install Ollama from https://ollama.com/download and pull the model with ollama pull qwen3:14b. If no key is set, the notebook starts the Ollama service for you, so there is no need to run ollama serve. Local models are smaller, so your outputs will differ from the recorded ones.

Prefer OpenAI or Anthropic?

1. Install the extra: uv sync --extra openai or uv sync --extra anthropic.
2. Export OPENAI_API_KEY or ANTHROPIC_API_KEY before launching Jupyter. If you forget, you will be prompted for it, which keeps the key out of the saved notebook.
3. Set LLM_PROVIDER=openai (or anthropic), or pass provider="openai" to make_llm(). A key on its own never switches providers, so one exported for unrelated work cannot reroute you.

The switch applies wherever a notebook builds its LLM with make_llm(). A notebook that constructs OllamaLLM directly stays on Ollama. Output on gpt-5 or Claude will differ from the recorded outputs, and prompt-sensitive examples (the ch09 evaluator pattern, ch08 supervised trajectories) may behave noticeably differently.

In [1]:

# Uncomment the line below to install `llm-agents-from-scratch` from PyPI
# !pip install 'llm-agents-from-scratch[notebook-utils,openai]'

# Uncomment the line below to install `llm-agents-from-scratch` from PyPI # !pip install 'llm-agents-from-scratch[notebook-utils,openai]'

Setup¶

In [2]:

import logging
import os

from llm_agents_from_scratch.logger import enable_console_logging

import logging import os from llm_agents_from_scratch.logger import enable_console_logging

Constants¶

In [3]:

IS_ON_RUNPOD = "RUNPOD_POD_ID" in os.environ
LOGGING_ENABLED = True
LOGGING_LEVEL = logging.INFO

# for task execution
MAX_STEPS = 20
NUM_REPLICATIONS = 10

IS_ON_RUNPOD = "RUNPOD_POD_ID" in os.environ LOGGING_ENABLED = True LOGGING_LEVEL = logging.INFO # for task execution MAX_STEPS = 20 NUM_REPLICATIONS = 10

In [4]:

# Install additional dependencies for notebook
if IS_ON_RUNPOD:
    !uv pip install numpy pandas --system
else:
    !uv pip install numpy pandas

# Install additional dependencies for notebook if IS_ON_RUNPOD: !uv pip install numpy pandas --system else: !uv pip install numpy pandas

Using Python 3.13.5 environment at: /home/nerdai/Projects/llm-agents-from-scratch/.venv
Audited 2 packages in 1ms

In [5]:

# maybe enable logging
if LOGGING_ENABLED:
    enable_console_logging(LOGGING_LEVEL)

# maybe enable logging if LOGGING_ENABLED: enable_console_logging(LOGGING_LEVEL)

LLMs¶

In [6]:

from llm_agents_from_scratch.notebook_utils import make_llm

from llm_agents_from_scratch.notebook_utils import make_llm

Build Tools¶

(Listing 1) Tool: generate_random_sample()¶

In [7]:

import uuid

import numpy as np
from pydantic import BaseModel, ConfigDict, Field, computed_field

from llm_agents_from_scratch.tools import PydanticFunctionTool

# Global registry of samples. A Monte Carlo estimate only needs to know how
# many points landed inside the circle, so each sample keeps two counts, not
# its points: sample_id -> (points inside, total points)
SAMPLE_REGISTRY: dict[str, tuple[int, int]] = {}

# points are drawn this many at a time, so memory stays small for any n
CHUNK_SIZE = 1_000_000


def count_inside(n: int) -> int:
    """Draw n random points; return how many land inside the circle."""
    inside = 0
    for start in range(0, n, CHUNK_SIZE):
        size = min(CHUNK_SIZE, n - start)
        x, y = np.random.uniform(size=(2, size))
        inside += int(np.count_nonzero(x**2 + y**2 < 1))
    return inside


class RandomSampleParams(BaseModel):
    """Params for generate_random_sample."""

    model_config = ConfigDict(extra="forbid")
    n: int = Field(description="The number of random points to generate")


class RandomSample(BaseModel):
    """Result from generate_random_sample."""

    sample_id: str = Field(
        description="Pass this sample_id to monte_carlo_estimate",
    )

    @computed_field
    @property
    def sample_size(
        self,
    ) -> int:
        """Look up n in SAMPLE_REGISTRY."""
        return SAMPLE_REGISTRY[self.sample_id][1]

    def __str__(self) -> str:
        """String representation of RandomSample."""
        return self.model_dump_json()


def generate_random_sample(params: RandomSampleParams) -> RandomSample:
    """Generate n random points in [0, 1] × [0, 1].

    Returns a sample_id. Pass this sample_id directly to monte_carlo_estimate.
    """
    sample_id = str(uuid.uuid4())
    SAMPLE_REGISTRY[sample_id] = (count_inside(params.n), params.n)

    return RandomSample(sample_id=sample_id)


# generate random sample tool
random_sample_tool = PydanticFunctionTool(generate_random_sample)

import uuid import numpy as np from pydantic import BaseModel, ConfigDict, Field, computed_field from llm_agents_from_scratch.tools import PydanticFunctionTool # Global registry of samples. A Monte Carlo estimate only needs to know how # many points landed inside the circle, so each sample keeps two counts, not # its points: sample_id -> (points inside, total points) SAMPLE_REGISTRY: dict[str, tuple[int, int]] = {} # points are drawn this many at a time, so memory stays small for any n CHUNK_SIZE = 1_000_000 def count_inside(n: int) -> int: """Draw n random points; return how many land inside the circle.""" inside = 0 for start in range(0, n, CHUNK_SIZE): size = min(CHUNK_SIZE, n - start) x, y = np.random.uniform(size=(2, size)) inside += int(np.count_nonzero(x**2 + y**2 < 1)) return inside class RandomSampleParams(BaseModel): """Params for generate_random_sample.""" model_config = ConfigDict(extra="forbid") n: int = Field(description="The number of random points to generate") class RandomSample(BaseModel): """Result from generate_random_sample.""" sample_id: str = Field( description="Pass this sample_id to monte_carlo_estimate", ) @computed_field @property def sample_size( self, ) -> int: """Look up n in SAMPLE_REGISTRY.""" return SAMPLE_REGISTRY[self.sample_id][1] def __str__(self) -> str: """String representation of RandomSample.""" return self.model_dump_json() def generate_random_sample(params: RandomSampleParams) -> RandomSample: """Generate n random points in [0, 1] × [0, 1]. Returns a sample_id. Pass this sample_id directly to monte_carlo_estimate. """ sample_id = str(uuid.uuid4()) SAMPLE_REGISTRY[sample_id] = (count_inside(params.n), params.n) return RandomSample(sample_id=sample_id) # generate random sample tool random_sample_tool = PydanticFunctionTool(generate_random_sample)

Demonstration¶

In [8]:

from llm_agents_from_scratch.data_structures import ToolCall

rs_tool_call = ToolCall(
    tool_name=random_sample_tool.name,
    arguments={"n": 5000},
)
rs_tool_call_result = random_sample_tool(rs_tool_call)
rs_tool_call_result

from llm_agents_from_scratch.data_structures import ToolCall rs_tool_call = ToolCall( tool_name=random_sample_tool.name, arguments={"n": 5000}, ) rs_tool_call_result = random_sample_tool(rs_tool_call) rs_tool_call_result

Out[8]:

ToolCallResult(tool_call_id='b070865b-6975-4f33-8641-f0b6c5da725e', content='{"sample_id":"49e8a9ee-8281-4a7e-9367-487fe38ef764","sample_size":5000}', error=False)

(Listing 2) Tool: add_more_points()¶

In [9]:

class AddPointsParams(BaseModel):
    """Params for add_more_points_to_sample."""

    model_config = ConfigDict(extra="forbid")
    sample_id: str = Field(
        description="The sample_id of the sample to augment",
    )
    n: int = Field(description="The number of random points to generate")


def add_more_points_to_sample(params: AddPointsParams) -> RandomSample:
    """Add n more random points to an existing random sample.

    Returns a sample_id and the total number of points.
    """
    # augment sample: count the new points into its totals
    inside, total = SAMPLE_REGISTRY[params.sample_id]
    SAMPLE_REGISTRY[params.sample_id] = (
        inside + count_inside(params.n),
        total + params.n,
    )

    return RandomSample(sample_id=params.sample_id)


# create tool
add_more_points_tool = PydanticFunctionTool(add_more_points_to_sample)

class AddPointsParams(BaseModel): """Params for add_more_points_to_sample.""" model_config = ConfigDict(extra="forbid") sample_id: str = Field( description="The sample_id of the sample to augment", ) n: int = Field(description="The number of random points to generate") def add_more_points_to_sample(params: AddPointsParams) -> RandomSample: """Add n more random points to an existing random sample. Returns a sample_id and the total number of points. """ # augment sample: count the new points into its totals inside, total = SAMPLE_REGISTRY[params.sample_id] SAMPLE_REGISTRY[params.sample_id] = ( inside + count_inside(params.n), total + params.n, ) return RandomSample(sample_id=params.sample_id) # create tool add_more_points_tool = PydanticFunctionTool(add_more_points_to_sample)

Demonstration¶

In [10]:

# get the sample ID of the previous random_sample_tool() invocation
random_sample = RandomSample.model_validate_json(rs_tool_call_result.content)

# build tool call for add more points
add_pts_tool_call = ToolCall(
    tool_name=add_more_points_tool.name,
    arguments={
        "sample_id": random_sample.sample_id,
        "n": 500,
    },
)
add_pts_tool_call_result = add_more_points_tool(add_pts_tool_call)
add_pts_tool_call_result

# get the sample ID of the previous random_sample_tool() invocation random_sample = RandomSample.model_validate_json(rs_tool_call_result.content) # build tool call for add more points add_pts_tool_call = ToolCall( tool_name=add_more_points_tool.name, arguments={ "sample_id": random_sample.sample_id, "n": 500, }, ) add_pts_tool_call_result = add_more_points_tool(add_pts_tool_call) add_pts_tool_call_result

Out[10]:

ToolCallResult(tool_call_id='5f7cc977-bb66-47d5-b3d7-226240474092', content='{"sample_id":"49e8a9ee-8281-4a7e-9367-487fe38ef764","sample_size":5500}', error=False)

(Listing 3) Tool: monte_carlo_estimate()¶

In [11]:

class MonteCarloEstimateParams(BaseModel):
    """Params for monte_carlo_estimate."""

    model_config = ConfigDict(extra="forbid")
    sample_id: str = Field(
        description="The sample_id returned by generate_random_sample",
    )


class MonteCarloEstimateResult(BaseModel):
    """Results for monte_carlo_estimate."""

    sample_id: str
    sample_size: int
    estimate: float

    def __str__(self) -> str:
        """String representation of MonteCarloEstimateResult."""
        return self.model_dump_json()


def monte_carlo_estimate(
    params: MonteCarloEstimateParams,
) -> MonteCarloEstimateResult:
    """Estimate pi using Monte Carlo method.

    Args:
        params: Contains sample_id from generate_random_sample.

    Returns:
        Estimate of pi (float).
    """
    inside, n = SAMPLE_REGISTRY[params.sample_id]
    return MonteCarloEstimateResult(
        estimate=(inside / n) * 4,
        sample_id=params.sample_id,
        sample_size=n,
    )


# create tool
monte_carlo_estimate_tool = PydanticFunctionTool(monte_carlo_estimate)

class MonteCarloEstimateParams(BaseModel): """Params for monte_carlo_estimate.""" model_config = ConfigDict(extra="forbid") sample_id: str = Field( description="The sample_id returned by generate_random_sample", ) class MonteCarloEstimateResult(BaseModel): """Results for monte_carlo_estimate.""" sample_id: str sample_size: int estimate: float def __str__(self) -> str: """String representation of MonteCarloEstimateResult.""" return self.model_dump_json() def monte_carlo_estimate( params: MonteCarloEstimateParams, ) -> MonteCarloEstimateResult: """Estimate pi using Monte Carlo method. Args: params: Contains sample_id from generate_random_sample. Returns: Estimate of pi (float). """ inside, n = SAMPLE_REGISTRY[params.sample_id] return MonteCarloEstimateResult( estimate=(inside / n) * 4, sample_id=params.sample_id, sample_size=n, ) # create tool monte_carlo_estimate_tool = PydanticFunctionTool(monte_carlo_estimate)

Demonstration¶

In [12]:

# build tool call for estimating Pi
mc_estimate_tool_call = ToolCall(
    tool_name=monte_carlo_estimate_tool.name,
    arguments={
        "sample_id": random_sample.sample_id,
    },
)
mc_estimate_tool_call_result = monte_carlo_estimate_tool(mc_estimate_tool_call)
mc_estimate_tool_call_result

# build tool call for estimating Pi mc_estimate_tool_call = ToolCall( tool_name=monte_carlo_estimate_tool.name, arguments={ "sample_id": random_sample.sample_id, }, ) mc_estimate_tool_call_result = monte_carlo_estimate_tool(mc_estimate_tool_call) mc_estimate_tool_call_result

Out[12]:

ToolCallResult(tool_call_id='bc3cc752-99e2-4a6a-82f3-3fa3c80f7442', content='{"sample_id":"49e8a9ee-8281-4a7e-9367-487fe38ef764","sample_size":5500,"estimate":3.152}', error=False)

Define the Task¶

(Listing 4) Writing the task instruction¶

In [13]:

instruction = """
You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the estimate falls in the range [3.1415, 3.1425).
Any value from 3.1415 up to (but not including) 3.1425 is a success.

Examples:
- 3.14159 ✓ (within range)
- 3.14200 ✓ (within range)
- 3.14149 ✗ (too low)
- 3.14250 ✗ (too high)

<algorithm>
1. Call generate_random_sample(1000000) to start with 1M points
2. Call monte_carlo_estimate(sample_id) to get estimate
3. Check: is the estimate between 3.1415 and 3.1425?
   - YES → Report success and STOP
   - NO → Continue to step 4
4. Call add_more_points_to_sample, doubling the points each time:
   - First add: 1 million
   - Second add: 2 million
   - Third add: 4 million
   - And so on, doubling each iteration
5. After adding points, go back to step 2

Exponential growth ensures faster convergence while demonstrating adaptive
sampling.
</algorithm>

<critical_rules>
- If the task is not complete, your response MUST contain a tool call
- Do not just describe what you plan to do—actually call the tool
- Do not stop until the estimate falls within the target range
- Keep track of your iteration to calculate the correct doubling amount
- NEVER fabricate tool results-only use actual tool responses
- NEVER invent a sample_id
</critical_rules>

<final_output>
When the estimate reaches the target precision, respond with this exact JSON
structure and nothing else:

{"sample_id": "<the-actual-sample-id-from-tool-response>"}

No explanation, no markdown formatting, no code blocks—just the raw JSON.
</final_output>

Begin by calling generate_random_sample(1000000).
""".strip()

instruction = """ You are tasked with estimating pi using Monte Carlo methods. TARGET: Get an estimate accurate to 3 decimal places. Success means the estimate falls in the range [3.1415, 3.1425). Any value from 3.1415 up to (but not including) 3.1425 is a success. Examples: - 3.14159 ✓ (within range) - 3.14200 ✓ (within range) - 3.14149 ✗ (too low) - 3.14250 ✗ (too high) <algorithm> 1. Call generate_random_sample(1000000) to start with 1M points 2. Call monte_carlo_estimate(sample_id) to get estimate 3. Check: is the estimate between 3.1415 and 3.1425? - YES → Report success and STOP - NO → Continue to step 4 4. Call add_more_points_to_sample, doubling the points each time: - First add: 1 million - Second add: 2 million - Third add: 4 million - And so on, doubling each iteration 5. After adding points, go back to step 2 Exponential growth ensures faster convergence while demonstrating adaptive sampling. </algorithm> <critical_rules> - If the task is not complete, your response MUST contain a tool call - Do not just describe what you plan to do—actually call the tool - Do not stop until the estimate falls within the target range - Keep track of your iteration to calculate the correct doubling amount - NEVER fabricate tool results-only use actual tool responses - NEVER invent a sample_id </critical_rules> <final_output> When the estimate reaches the target precision, respond with this exact JSON structure and nothing else: {"sample_id": "<the-actual-sample-id-from-tool-response>"} No explanation, no markdown formatting, no code blocks—just the raw JSON. </final_output> Begin by calling generate_random_sample(1000000). """.strip()

(Listing 5) The Task¶

In [14]:

from llm_agents_from_scratch.data_structures import Task

task = Task(
    instruction=instruction,
)

from llm_agents_from_scratch.data_structures import Task task = Task( instruction=instruction, )

(Listing 6) Creating our LLMAgent¶

In [15]:

from llm_agents_from_scratch import LLMAgent

llm = make_llm()
llm_agent = LLMAgent(
    llm=llm,
    tools=[
        random_sample_tool,
        add_more_points_tool,
        monte_carlo_estimate_tool,
    ],
)

from llm_agents_from_scratch import LLMAgent llm = make_llm() llm_agent = LLMAgent( llm=llm, tools=[ random_sample_tool, add_more_points_tool, monte_carlo_estimate_tool, ], )

✓ Using Ollama Cloud (kimi-k2.7-code:cloud)

Perform the Task¶

In [16]:

handler = llm_agent.run(task, max_steps=MAX_STEPS)

handler = llm_agent.run(task, max_steps=MAX_STEPS)

INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]

In [17]:

# if need to cancel uncomment code below
# handler.cancel()  # noqa: ERA001

# if need to cancel uncomment code below # handler.cancel() # noqa: ERA001

In [18]:

handler.done()

handler.done()

Out[18]:

False

In [19]:

if handler.done():
    # check if there was an error
    handler.exception()

if handler.done(): # check if there was an error handler.exception()

In [20]:

print(handler.rollout)

print(handler.rollout)

In [21]:

await:handler

result = handler.exception() or handler.result()
result

result = handler.exception() or handler.result() result

Out[21]:

TaskResult(task_id='fb858816-7e72-4963-84b4-30bf446afd3a', content='The returned estimate is 3.141918 with a sample size of 2,000,000. This falls within the target range [3.1415, 3.1425), so the task is complete.\n\n{"sample_id": "215b6b8e-1932-4f24-80cf-65987464bd06"}')

Evaluation¶

(Listing 7) Evaluating Task Success¶

In [22]:

import json
from json import JSONDecodeError

from pydantic import ValidationError


def estimate_has_target_precision(estimate: MonteCarloEstimateResult) -> bool:
    """Checks if the estimate achieved the desired precision.

    Target precision is 3 decimal places (3.142), meaning the estimate
    should be between 3.1415 and 3.1425.
    """
    upper_bound = 3.1425
    lower_bound = 3.1415
    return lower_bound <= estimate.estimate < upper_bound


def is_task_success(
    handler: LLMAgent.TaskHandler,
    verbose: bool = False,
) -> bool:
    """Determines task success.

    Args:
        handler (LLMAgent.TaskHandler): The handler containing the
            result or exception of the task execution
        verbose (bool): Whether to print out details of the
            determination. Defaults to False.

    Returns:
        bool: True if task was successful. False, otherwise.
    """
    if handler.exception():
        if verbose:
            print(handler.exception())
        return False

    result = handler.result()
    try:
        output_data = json.loads(result.content)
        sample_id = output_data["sample_id"]
        params = MonteCarloEstimateParams(
            sample_id=sample_id,
        )
        estimate = monte_carlo_estimate(params)
        if verbose:
            print(
                f"Estimate: {estimate}",
            )
        return estimate_has_target_precision(estimate)
    except (ValidationError, KeyError, JSONDecodeError) as e:
        # invalid sample_id provided by LLM agent—unsuccessful task
        if verbose:
            print(f"The LLM agent returned an invalid output: {str(e)}.")
        return False

import json from json import JSONDecodeError from pydantic import ValidationError def estimate_has_target_precision(estimate: MonteCarloEstimateResult) -> bool: """Checks if the estimate achieved the desired precision. Target precision is 3 decimal places (3.142), meaning the estimate should be between 3.1415 and 3.1425. """ upper_bound = 3.1425 lower_bound = 3.1415 return lower_bound <= estimate.estimate < upper_bound def is_task_success( handler: LLMAgent.TaskHandler, verbose: bool = False, ) -> bool: """Determines task success. Args: handler (LLMAgent.TaskHandler): The handler containing the result or exception of the task execution verbose (bool): Whether to print out details of the determination. Defaults to False. Returns: bool: True if task was successful. False, otherwise. """ if handler.exception(): if verbose: print(handler.exception()) return False result = handler.result() try: output_data = json.loads(result.content) sample_id = output_data["sample_id"] params = MonteCarloEstimateParams( sample_id=sample_id, ) estimate = monte_carlo_estimate(params) if verbose: print( f"Estimate: {estimate}", ) return estimate_has_target_precision(estimate) except (ValidationError, KeyError, JSONDecodeError) as e: # invalid sample_id provided by LLM agent—unsuccessful task if verbose: print(f"The LLM agent returned an invalid output: {str(e)}.") return False

In [23]:

is_task_success(handler, verbose=True)

is_task_success(handler, verbose=True)

The LLM agent returned an invalid output: Expecting value: line 1 column 1 (char 0).

Out[23]:

False

Trajectory Analysis¶

In [24]:

trajectory_judge = make_llm(role="judge")

trajectory_judge = make_llm(role="judge")

✓ Using Ollama Cloud (kimi-k2.7-code:cloud)

(Listing 8) Rubric for LLM judge¶

In [25]:

class TrajectoryEvalRubric(BaseModel):
    """Rubric for evaluating an execution trajectory."""

    reached_target_precision: bool = Field(
        description="True if agent achieved estimate that rounds to 3.142",
    )

    completed_without_max_steps: bool = Field(
        description=(
            "True if agent completed task without hitting max steps limit"
        ),
    )

    always_added_points_before_reestimating: bool = Field(
        description=(
            "False if agent called monte_carlo_estimate consecutively more "
            "than once before adding points"
        ),
    )

    reused_sample: bool = Field(
        description=(
            "True if agent used add_more_points_to_sample to grow the sample "
            "instead of creating new samples"
        ),
    )

    no_false_completion: bool = Field(
        description=(
            "True if agent only claimed success when the actual tool result "
            "showed 3.142. False if agent claimed convergence based on a "
            "fabricated or misread estimate."
        ),
    )

    no_missed_completion: bool = Field(
        description=(
            "True if agent stopped when estimate reached 3.142. False if "
            "agent continued adding points after already achieving target."
        ),
    )

    followed_output_format: bool = Field(
        description=(
            "True if agent's final response contained only the sample_id "
            "as instructed, with no additional text or explanation."
        ),
    )

    largest_sample_size: int | None = Field(
        description=(
            "The largest sample size achieved during the trajectory, "
            "or None if not determinable from tool outputs"
        ),
    )

    summary: str = Field(
        description="One sentence summary of trajectory quality",
    )

class TrajectoryEvalRubric(BaseModel): """Rubric for evaluating an execution trajectory.""" reached_target_precision: bool = Field( description="True if agent achieved estimate that rounds to 3.142", ) completed_without_max_steps: bool = Field( description=( "True if agent completed task without hitting max steps limit" ), ) always_added_points_before_reestimating: bool = Field( description=( "False if agent called monte_carlo_estimate consecutively more " "than once before adding points" ), ) reused_sample: bool = Field( description=( "True if agent used add_more_points_to_sample to grow the sample " "instead of creating new samples" ), ) no_false_completion: bool = Field( description=( "True if agent only claimed success when the actual tool result " "showed 3.142. False if agent claimed convergence based on a " "fabricated or misread estimate." ), ) no_missed_completion: bool = Field( description=( "True if agent stopped when estimate reached 3.142. False if " "agent continued adding points after already achieving target." ), ) followed_output_format: bool = Field( description=( "True if agent's final response contained only the sample_id " "as instructed, with no additional text or explanation." ), ) largest_sample_size: int | None = Field( description=( "The largest sample size achieved during the trajectory, " "or None if not determinable from tool outputs" ), ) summary: str = Field( description="One sentence summary of trajectory quality", )

(Listing 9) LLM judge instruction prompt¶

In [26]:

judge_prompt_template = """Evaluate this Monte Carlo pi estimation trajectory.

The agent had three tools:
- `generate_random_sample(n)` - Creates NEW sample
- `add_more_points_to_sample(sample_id, n)` - Adds points to EXISTING sample
- `monte_carlo_estimate(sample_id)` - Returns pi estimate

Correct behavior:
1. Create sample once
2. Estimate → if not between 3.1415 and 3.1425,
   add points → re-estimate → repeat
3. When target reached, respond with ONLY the sample_id (no other text)

Note: If final_response is "Max steps error", the agent failed to complete
the task within the allowed number of steps.

HALLUCINATION MARKER: If you see "💬 assistant: 🔧 tool:" in the trajectory,
the agent fabricated a tool response instead of waiting for the actual result.
This is a critical failure—set no_false_completion to False.

<final_response>
{result}
</final_response>

<trajectory>
{trajectory}
</trajectory>

Evaluate and submit your judgment.""".strip()

judge_prompt_template = """Evaluate this Monte Carlo pi estimation trajectory. The agent had three tools: - `generate_random_sample(n)` - Creates NEW sample - `add_more_points_to_sample(sample_id, n)` - Adds points to EXISTING sample - `monte_carlo_estimate(sample_id)` - Returns pi estimate Correct behavior: 1. Create sample once 2. Estimate → if not between 3.1415 and 3.1425, add points → re-estimate → repeat 3. When target reached, respond with ONLY the sample_id (no other text) Note: If final_response is "Max steps error", the agent failed to complete the task within the allowed number of steps. HALLUCINATION MARKER: If you see "💬 assistant: 🔧 tool:" in the trajectory, the agent fabricated a tool response instead of waiting for the actual result. This is a critical failure—set no_false_completion to False. <final_response> {result} </final_response> <trajectory> {trajectory} </trajectory> Evaluate and submit your judgment.""".strip()

In [27]:

trajectory_eval = await trajectory_judge.structured_output(
    prompt=judge_prompt_template.format(
        result=str(result),
        trajectory=handler.rollout,
    ),
    mdl=TrajectoryEvalRubric,
)

trajectory_eval = await trajectory_judge.structured_output( prompt=judge_prompt_template.format( result=str(result), trajectory=handler.rollout, ), mdl=TrajectoryEvalRubric, )

In [28]:

print(trajectory_eval.model_dump_json(indent=4))

print(trajectory_eval.model_dump_json(indent=4))

{
    "reached_target_precision": true,
    "completed_without_max_steps": true,
    "always_added_points_before_reestimating": true,
    "reused_sample": true,
    "no_false_completion": true,
    "no_missed_completion": true,
    "followed_output_format": false,
    "largest_sample_size": 2000000,
    "summary": "The agent reached the target estimate using the correct tools and valid tool responses, but its final answer included explanatory text before the JSON instead of only the raw JSON as required."
}

Replications for a more reliable evaluation¶

In this section, we'll repeat the task multiple times to get a more robust evaluation of our LLM agent's performance.

(Listing 10) Repeated task executions with our LLM agent¶

In [29]:

handlers = []
for _ in range(NUM_REPLICATIONS):
    h = llm_agent.run(task, max_steps=MAX_STEPS)
    handlers.append(h)

handlers = [] for _ in range(NUM_REPLICATIONS): h = llm_agent.run(task, max_steps=MAX_STEPS) handlers.append(h)

INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]
INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]
INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]
INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]
INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]
INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]
INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]
INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]
INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]
INFO (llm_agents_fs.LLMAgent) :      🚀 Starting task: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means the...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) :      ⚙️ Processing Step: You are tasked with estimating pi using Monte Carlo methods.

TARGET: Get an estimate accurate to 3 decimal places.
Success means ...[TRUNCATED]

In [30]:

# can execute this repeatedly until all handlers are done
[str(h.exception() or h.result()) if h.done() else "Not Done" for h in handlers]

# can execute this repeatedly until all handlers are done [str(h.exception() or h.result()) if h.done() else "Not Done" for h in handlers]

Out[30]:

['Not Done',
 'Not Done',
 'Not Done',
 'Not Done',
 'Not Done',
 'Not Done',
 'Not Done',
 'Not Done',
 'Not Done',
 'Not Done']

(Listing 11) Task Success and Trajectory Evaluations of Individual Runs¶

In [31]:

await:handlers

import asyncio

task_success_evals = []
trajectory_eval_coros = []
for handler in handlers:
    # task success evaluation
    task_success_evals.append(is_task_success(handler))

    # trajectory evaluation coroutines
    coro = trajectory_judge.structured_output(
        prompt=judge_prompt_template.format(
            result=str(handler.exception() or handler.result()),
            trajectory=handler.rollout,
        ),
        mdl=TrajectoryEvalRubric,
    )
    trajectory_eval_coros.append(coro)

trajectory_evals = await asyncio.gather(*trajectory_eval_coros)

import asyncio task_success_evals = [] trajectory_eval_coros = [] for handler in handlers: # task success evaluation task_success_evals.append(is_task_success(handler)) # trajectory evaluation coroutines coro = trajectory_judge.structured_output( prompt=judge_prompt_template.format( result=str(handler.exception() or handler.result()), trajectory=handler.rollout, ), mdl=TrajectoryEvalRubric, ) trajectory_eval_coros.append(coro) trajectory_evals = await asyncio.gather(*trajectory_eval_coros)

In [32]:

print(task_success_evals)
print(trajectory_evals)

print(task_success_evals) print(trajectory_evals)

[False, False, False, False, False, False, False, False, False, False]
[TrajectoryEvalRubric(reached_target_precision=True, completed_without_max_steps=True, always_added_points_before_reestimating=False, reused_sample=True, no_false_completion=True, no_missed_completion=True, followed_output_format=False, largest_sample_size=2000000, summary='Agent achieved target precision and reused the sample, but had a duplicate consecutive monte_carlo_estimate call before adding points and final response included explanatory text before the JSON.'), TrajectoryEvalRubric(reached_target_precision=True, completed_without_max_steps=True, always_added_points_before_reestimating=False, reused_sample=True, no_false_completion=True, no_missed_completion=True, followed_output_format=False, largest_sample_size=4000000, summary='Agent reached the target precision and reused the same sample, but violated the final output format by including explanatory text and made consecutive monte_carlo_estimate calls without adding points between the first and second estimates.'), TrajectoryEvalRubric(reached_target_precision=False, completed_without_max_steps=True, always_added_points_before_reestimating=True, reused_sample=True, no_false_completion=False, no_missed_completion=True, followed_output_format=False, largest_sample_size=2000000, summary='Agent falsely claimed success with estimate 3.142486, which is outside the required [3.1415, 3.1425) range, and also violated the final-output format by adding explanatory text.'), TrajectoryEvalRubric(reached_target_precision=True, completed_without_max_steps=True, always_added_points_before_reestimating=True, reused_sample=True, no_false_completion=True, no_missed_completion=True, followed_output_format=False, largest_sample_size=30000000, summary='Agent correctly reused one sample, doubled points each iteration, and reached 3.14152, but added explanatory text before the required JSON-only final output.'), TrajectoryEvalRubric(reached_target_precision=False, completed_without_max_steps=False, always_added_points_before_reestimating=False, reused_sample=True, no_false_completion=True, no_missed_completion=False, followed_output_format=False, largest_sample_size=512000000, summary='Agent hit Max steps error after only reaching 3.14143240625 at 128M points, failing to achieve target precision and produce final output.'), TrajectoryEvalRubric(reached_target_precision=True, completed_without_max_steps=True, always_added_points_before_reestimating=False, reused_sample=True, no_false_completion=True, no_missed_completion=True, followed_output_format=False, largest_sample_size=2000000, summary='Agent reached the target estimate 3.141746 by reusing and growing the same sample, but violated the output-format rule by adding explanatory text to the final response and made a redundant second monte_carlo_estimate call at 1M points without adding points between.'), TrajectoryEvalRubric(reached_target_precision=True, completed_without_max_steps=True, always_added_points_before_reestimating=False, reused_sample=True, no_false_completion=True, no_missed_completion=True, followed_output_format=False, largest_sample_size=24000000, summary='Agent reached target with a single reused sample, but doubled incorrectly (8M then 12M then 12M instead of 16M) and produced non-JSON final text.'), TrajectoryEvalRubric(reached_target_precision=True, completed_without_max_steps=True, always_added_points_before_reestimating=False, reused_sample=True, no_false_completion=True, no_missed_completion=True, followed_output_format=False, largest_sample_size=8000000, summary='Agent eventually reached the target and used the correct sample_id, but it called add_more_points_to_sample multiple times with duplicate IDs in a row without checking tool results, then failed to follow the final output format by adding explanatory text before the JSON.'), TrajectoryEvalRubric(reached_target_precision=True, completed_without_max_steps=True, always_added_points_before_reestimating=False, reused_sample=True, no_false_completion=True, no_missed_completion=True, followed_output_format=False, largest_sample_size=61000000, summary='The agent ultimately reached the target estimate with a 61M-point sample but violated the required final-output format by prepending explanatory text before the JSON, and made a duplicate consecutive monte_carlo_estimate call without adding points in between.'), TrajectoryEvalRubric(reached_target_precision=True, completed_without_max_steps=True, always_added_points_before_reestimating=False, reused_sample=True, no_false_completion=True, no_missed_completion=True, followed_output_format=False, largest_sample_size=3000000, summary='Agent reached target precision with reused sample but called monte_carlo_estimate consecutively without adding points twice, and final response included explanatory text before the JSON.')]

Evaluation Summary¶

In [33]:

import pandas as pd

from llm_agents_from_scratch.notebook_utils import set_dataframe_display_options

# sets display options for pd.DataFrame in notebooks
set_dataframe_display_options()

import pandas as pd from llm_agents_from_scratch.notebook_utils import set_dataframe_display_options # sets display options for pd.DataFrame in notebooks set_dataframe_display_options()

In [34]:

# shape eval results into a pd.DataFrame
evals_df = pd.DataFrame(
    data=[e.model_dump() for e in trajectory_evals],
)

# add task_success column
evals_df.insert(0, "task_success", task_success_evals)

# separate summary column
summary_df = evals_df[["summary"]].copy()
evals_df = evals_df.drop(columns=["summary"])

# compute aggregations: TOTAL and AVG rows
total_row = {}
avg_row = {}

for col, dtype in evals_df.dtypes.items():
    if dtype == "bool" or pd.api.types.is_numeric_dtype(dtype):
        total_row[col] = evals_df[col].sum()
        avg_row[col] = evals_df[col].mean()
    else:
        total_row[col] = "TOTAL"
        avg_row[col] = "AVG"

# merge evaluations and aggregations dataframes
evals_df = pd.concat(
    [
        evals_df,
        pd.DataFrame([total_row, avg_row], index=["TOTAL", "AVG"]),
    ],
)

# style
evals_df.style.apply(
    lambda r: ["border-top: 2px solid #444"] * len(r)
    if r.name == "TOTAL"
    else [""] * len(r),
    axis=1,
)

# shape eval results into a pd.DataFrame evals_df = pd.DataFrame( data=[e.model_dump() for e in trajectory_evals], ) # add task_success column evals_df.insert(0, "task_success", task_success_evals) # separate summary column summary_df = evals_df[["summary"]].copy() evals_df = evals_df.drop(columns=["summary"]) # compute aggregations: TOTAL and AVG rows total_row = {} avg_row = {} for col, dtype in evals_df.dtypes.items(): if dtype == "bool" or pd.api.types.is_numeric_dtype(dtype): total_row[col] = evals_df[col].sum() avg_row[col] = evals_df[col].mean() else: total_row[col] = "TOTAL" avg_row[col] = "AVG" # merge evaluations and aggregations dataframes evals_df = pd.concat( [ evals_df, pd.DataFrame([total_row, avg_row], index=["TOTAL", "AVG"]), ], ) # style evals_df.style.apply( lambda r: ["border-top: 2px solid #444"] * len(r) if r.name == "TOTAL" else [""] * len(r), axis=1, )

Out[34]:

	task_success	reached_target_precision	completed_without_max_steps	always_added_points_before_reestimating	reused_sample	no_false_completion	no_missed_completion	followed_output_format	largest_sample_size
0	False	True	True	False	True	True	True	False	2000000.000000
1	False	True	True	False	True	True	True	False	4000000.000000
2	False	False	True	True	True	False	True	False	2000000.000000
3	False	True	True	True	True	True	True	False	30000000.000000
4	False	False	False	False	True	True	False	False	512000000.000000
5	False	True	True	False	True	True	True	False	2000000.000000
6	False	True	True	False	True	True	True	False	24000000.000000
7	False	True	True	False	True	True	True	False	8000000.000000
8	False	True	True	False	True	True	True	False	61000000.000000
9	False	True	True	False	True	True	True	False	3000000.000000
TOTAL	0.000000	8.000000	9.000000	2.000000	10.000000	9.000000	9.000000	0.000000	648000000.000000
AVG	0.000000	0.800000	0.900000	0.200000	1.000000	0.900000	0.900000	0.000000	64800000.000000

In [35]:

summary_df

summary_df

Out[35]:

	summary
0	Agent achieved target precision and reused the sample, but had a duplicate consecutive monte_carlo_estimate call before adding points and final response included explanatory text before the JSON.
1	Agent reached the target precision and reused the same sample, but violated the final output format by including explanatory text and made consecutive monte_carlo_estimate calls without adding points between the first and second estimates.
2	Agent falsely claimed success with estimate 3.142486, which is outside the required [3.1415, 3.1425) range, and also violated the final-output format by adding explanatory text.
3	Agent correctly reused one sample, doubled points each iteration, and reached 3.14152, but added explanatory text before the required JSON-only final output.
4	Agent hit Max steps error after only reaching 3.14143240625 at 128M points, failing to achieve target precision and produce final output.
5	Agent reached the target estimate 3.141746 by reusing and growing the same sample, but violated the output-format rule by adding explanatory text to the final response and made a redundant second monte_carlo_estimate call at 1M points without adding points between.
6	Agent reached target with a single reused sample, but doubled incorrectly (8M then 12M then 12M instead of 16M) and produced non-JSON final text.
7	Agent eventually reached the target and used the correct sample_id, but it called add_more_points_to_sample multiple times with duplicate IDs in a row without checking tool results, then failed to follow the final output format by adding explanatory text before the JSON.
8	The agent ultimately reached the target estimate with a 61M-point sample but violated the required final-output format by prepending explanatory text before the JSON, and made a duplicate consecutive monte_carlo_estimate call without adding points in between.
9	Agent reached target precision with reused sample but called monte_carlo_estimate consecutively without adding points twice, and final response included explanatory text before the JSON.

In [36]:

# write results to json
evals_df.to_json("evals_df.json")
summary_df.to_json("summary_df.json")

# write results to json evals_df.to_json("evals_df.json") summary_df.to_json("summary_df.json")