Chapter 8 — Human in the Loop¶
Setup Instructions¶
To ensure you have the required dependencies to run this notebook, you'll need to have our llm-agents-from-scratch framework installed on the running Jupyter kernel. To do this, you can launch this notebook with the following command while within the project's root directory:
uv run --with jupyter jupyter lab
Alternatively, if you just want to use the published version of llm-agents-from-scratch without local development, you can install it from PyPi by uncommenting the cell below.
# Uncomment the line below to install `llm-agents-from-scratch` from PyPi
# !pip install llm-agents-from-scratch
Running an Ollama service¶
To execute the code provided in this notebook, you'll need to have Ollama installed on your local machine and have its LLM hosting service running. To download Ollama, follow the instructions found on this page: https://ollama.com/download. After downloading and installing Ollama, you can start a service by opening a terminal and running the command ollama serve.
import os, shutil, subprocess, time, urllib.request, urllib.error
def ensure_ollama(host="http://localhost:11434", timeout=15):
"""Start Ollama if not already running and wait until responsive."""
def _up():
try:
urllib.request.urlopen(f"{host}/api/tags", timeout=1)
return True
except (urllib.error.URLError, ConnectionError, TimeoutError):
return False
if _up():
return print(f"✓ Ollama already running at {host}")
# Lightning persistent path first, then standard locations
ollama_path = shutil.which("ollama")
if ollama_path is None:
for candidate in [
"/teamspace/studios/this_studio/.local/bin/ollama",
"/usr/local/bin/ollama",
"/usr/bin/ollama",
]:
if os.path.exists(candidate):
ollama_path = candidate
break
if ollama_path is None:
raise RuntimeError(
"Could not find the ollama binary. Install with: "
"curl -fsSL https://ollama.com/install.sh | sh"
)
print(f"Starting Ollama server ({ollama_path})...")
subprocess.Popen(
[ollama_path, "serve"],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
deadline = time.time() + timeout
while time.time() < deadline:
if _up():
return print(f"✓ Ollama up and running at {host}")
time.sleep(0.5)
raise RuntimeError(f"Ollama did not start within {timeout}s")
ensure_ollama()
✓ Ollama already running at http://localhost:11434
Examples¶
Example 1: Agent-Initiated Pause with HumanInputTool¶
import logging
from llm_agents_from_scratch import LLMAgent
from llm_agents_from_scratch.data_structures import Task
from llm_agents_from_scratch.logger import enable_console_logging
from llm_agents_from_scratch.llms import OllamaLLM
from llm_agents_from_scratch.tools import SimpleFunctionTool
from llm_agents_from_scratch.tools.default import HumanInputTool
enable_console_logging(logging.INFO)
def next_number(x: int) -> int:
if x % 2 == 0:
return x // 2
return 3 * x + 1
next_number_tool = SimpleFunctionTool(func=next_number)
human_input_tool = HumanInputTool()
llm = OllamaLLM(model="qwen3:14b", think=False)
agent = LLMAgent(llm=llm, tools=[next_number_tool, human_input_tool])
task = Task(
instruction=(
"Compute the full Hailstone sequence step by step using next_number. "
"The starting number must come from the human operator — ask them "
"before beginning."
)
)
result = await agent.run(task)
print(result.content)
INFO (llm_agents_fs.LLMAgent) : 🚀 Starting task: Compute the full Hailstone sequence step by step using next_number. The starting number must come from the human operator — ask them ...[TRUNCATED] INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Compute the full Hailstone sequence step by step using next_number. The starting number must come from the human operator — ask th...[TRUNCATED] INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I need to ask the human operator for the starting number to compute the Hailstone sequence. INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Ask the human operator for the starting number to compute the Hailstone sequence. INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Ask the human operator for the starting number to compute the Hailstone sequence. INFO (llm_agents_fs.TaskHandler) : 🛠️ Executing Tool Call: human_input
╭────────────────────────────────────────────────── Human Input ──────────────────────────────────────────────────╮ │ Please provide the starting number for the Hailstone sequence. │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
>:
INFO (llm_agents_fs.TaskHandler) : ✅ Successful Tool Call: 2 INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I have received the starting number, which is 2. I will now compute the Hailstone sequence step by step using the next_number tool unti...[TRUNCATED] INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Compute the next number in the Hailstone sequence using the starting number 2. INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Compute the next number in the Hailstone sequence using the starting number 2. INFO (llm_agents_fs.TaskHandler) : 🛠️ Executing Tool Call: next_number INFO (llm_agents_fs.TaskHandler) : ✅ Successful Tool Call: 1 INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: The next number in the Hailstone sequence after 2 is 1. Since we have reached 1, the sequence is complete. The full sequence is [2, 1]. INFO (llm_agents_fs.TaskHandler) : No new step required. INFO (llm_agents_fs.LLMAgent) : 🏁 Task completed: The next number in the Hailstone sequence after 2 is 1. Since we have reached 1, the sequence is complete. The full sequence is [2, ...[TRUNCATED] The next number in the Hailstone sequence after 2 is 1. Since we have reached 1, the sequence is complete. The full sequence is [2, 1].
Example 2: End-of-Loop Approval Gate¶
import logging
from llm_agents_from_scratch import LLMAgent
from llm_agents_from_scratch.data_structures import Task
from llm_agents_from_scratch.logger import enable_console_logging
from llm_agents_from_scratch.llms import OllamaLLM
from llm_agents_from_scratch.tools import SimpleFunctionTool
enable_console_logging(logging.INFO)
def next_number(x: int) -> int:
if x % 2 == 0:
return x // 2
return 3 * x + 1
next_number_tool = SimpleFunctionTool(func=next_number)
llm = OllamaLLM(model="qwen3:14b", think=False)
agent = LLMAgent(llm=llm, tools=[next_number_tool])
task = Task(
instruction=(
"Compute the full Hailstone sequence starting from 5, "
"step by step using next_number, until you reach 1."
)
)
result = await agent.run(task, with_approval=True)
print(result.content)
INFO (llm_agents_fs.LLMAgent) : 🚀 Starting task: Compute the full Hailstone sequence starting from 5, step by step using next_number, until you reach 1.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Compute the full Hailstone sequence starting from 5, step by step using next_number, until you reach 1.
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I need to call the stop-at-one skill with the starting number 5.
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the stop-at-one skill with the starting number 5.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the stop-at-one skill with the starting number 5.
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result:
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the stop-at-one skill with the starting number 5.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the stop-at-one skill with the starting number 5.
INFO (llm_agents_fs.TaskHandler) : 🛠️ Executing Tool Call: from_scratch__use_skill
INFO (llm_agents_fs.TaskHandler) : ✅ Successful Tool Call: <skill_content name="stop-at-one">
# Stop At One
Compute a full sequence from a starting number down to 1
using the `next_num...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I need to call the next_number tool with the starting number 5.
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the starting number 5.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the starting number 5.
INFO (llm_agents_fs.TaskHandler) : 🛠️ Executing Tool Call: next_number
INFO (llm_agents_fs.TaskHandler) : ✅ Successful Tool Call: 16
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: The next number in the sequence after 5 is 16. I will continue the process by calling the next_number tool with the current number 16.
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 16.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 16.
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I need to make the following tool call(s):
{
"id_": "12345678-90ab-cdef-1234-567890abcdef",
"tool_name": "next_number",
"a...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 16.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 16.
INFO (llm_agents_fs.TaskHandler) : 🛠️ Executing Tool Call: next_number
INFO (llm_agents_fs.TaskHandler) : ✅ Successful Tool Call: 8
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: The next number in the sequence after 16 is 8. I will continue the process by calling the next_number tool with the current number 8.
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 8.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 8.
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I need to make the following tool call(s):
{
"id_": "12345678-90ab-cdef-1234-567890abcdef",
"tool_name": "next_number",
"a...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 8.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 8.
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I need to make the following tool call(s):
{
"id_": "12345678-90ab-cdef-1234-567890abcdef",
"tool_name": "next_number",
"a...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 8.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 8.
INFO (llm_agents_fs.TaskHandler) : 🛠️ Executing Tool Call: next_number
INFO (llm_agents_fs.TaskHandler) : ✅ Successful Tool Call: 4
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: The next number in the sequence after 8 is 4. I will continue the process by calling the next_number tool with the current number 4.
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 4.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 4.
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I need to make the following tool call(s):
{
"id_": "12345678-90ab-cdef-1234-567890abcdef",
"tool_name": "next_number",
"a...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 4.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 4.
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I need to make the following tool call(s):
{
"id_": "12345678-90ab-cdef-1234-567890abcdef",
"tool_name": "next_number",
"a...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 4.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 4.
INFO (llm_agents_fs.TaskHandler) : 🛠️ Executing Tool Call: next_number
INFO (llm_agents_fs.TaskHandler) : ✅ Successful Tool Call: 2
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: The next number in the sequence after 4 is 2. I will continue the process by calling the next_number tool with the current number 2.
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 2.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 2.
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: I need to make the following tool call(s):
{
"id_": "12345678-90ab-cdef-1234-567890abcdef",
"tool_name": "next_number",
"a...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) : 🧠 New Step: Call the next_number tool with the current number 2.
INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: Call the next_number tool with the current number 2.
INFO (llm_agents_fs.TaskHandler) : 🛠️ Executing Tool Call: next_number
INFO (llm_agents_fs.TaskHandler) : ✅ Successful Tool Call: 1
INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: The next number in the sequence after 2 is 1. Now that we have reached 1, the sequence is complete. Let me summarize the full Hailstone...[TRUNCATED]
INFO (llm_agents_fs.TaskHandler) : No new step required.
╭───────────────────────────────────────────── Proposed Task Result ──────────────────────────────────────────────╮ │ The next number in the sequence after 2 is 1. Now that we have reached 1, the sequence is complete. Let me │ │ summarize the full Hailstone sequence starting from 5: │ │ │ │ **Sequence:** 5 → 16 → 8 → 4 → 2 → 1 │ │ **Starting number:** 5 │ │ **Total steps taken:** 5 │ │ **Maximum value reached:** 16 │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Approve this result? [y/n]:
Provide your correction rationale for the agent to address:
INFO (llm_agents_fs.LLMAgent) : 🔁 Task result rejected; re-entering loop with feedback. INFO (llm_agents_fs.TaskHandler) : 🧠 New Step (rejection): Everything looks good, but just omit the first sentence in your response. INFO (llm_agents_fs.TaskHandler) : ⚙️ Processing Step: The human operator REJECTED your proposed task result. Revise your approach and try again. <proposed-result> The next number in t...[TRUNCATED] INFO (llm_agents_fs.TaskHandler) : ✅ Step Result: Let me summarize the full Hailstone sequence starting from 5: **Sequence:** 5 → 16 → 8 → 4 → 2 → 1 **Starting number:** 5 **Total ...[TRUNCATED] INFO (llm_agents_fs.TaskHandler) : No new step required.
╭───────────────────────────────────────────── Proposed Task Result ──────────────────────────────────────────────╮ │ Let me summarize the full Hailstone sequence starting from 5: │ │ │ │ **Sequence:** 5 → 16 → 8 → 4 → 2 → 1 │ │ **Starting number:** 5 │ │ **Total steps taken:** 5 │ │ **Maximum value reached:** 16 │ ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Approve this result? [y/n]:
INFO (llm_agents_fs.LLMAgent) : 🏁 Task completed: Let me summarize the full Hailstone sequence starting from 5: **Sequence:** 5 → 16 → 8 → 4 → 2 → 1 **Starting number:** 5 **Tot...[TRUNCATED] Let me summarize the full Hailstone sequence starting from 5: **Sequence:** 5 → 16 → 8 → 4 → 2 → 1 **Starting number:** 5 **Total steps taken:** 5 **Maximum value reached:** 16
Example 3: Caller-Driven Stepwise Execution with run_supervised()¶
import logging
from llm_agents_from_scratch import LLMAgent
from llm_agents_from_scratch.data_structures import Task
from llm_agents_from_scratch.logger import enable_console_logging
from llm_agents_from_scratch.llms import OllamaLLM
from llm_agents_from_scratch.tools import SimpleFunctionTool
enable_console_logging(logging.INFO)
def next_number(x: int) -> int:
if x % 2 == 0:
return x // 2
return 3 * x + 1
next_number_tool = SimpleFunctionTool(func=next_number)
llm = OllamaLLM(model="qwen3:14b", think=False)
agent = LLMAgent(llm=llm, tools=[next_number_tool])
task = Task(
instruction=(
"Compute the full Hailstone sequence starting from 4, "
"step by step using next_number, until you reach 1."
)
)
task_handler = await agent.run_supervised(task, explicit_only_skills=["stop-at-one"])
step = await task_handler.get_next_step(None)
step
TaskStep(id_='2d14732d-e307-4706-aaac-944ebfdf0c69', task_id='fb5b7d5f-de29-4bec-b569-a88897230cf6', instruction='Compute the full Hailstone sequence starting from 4, step by step using next_number, until you reach 1.')
step_result = await task_handler.run_step(step)
step_result
INFO (llm_agents_fs.SupervisedTaskHandler) : ⚙️ Processing Step: Compute the full Hailstone sequence starting from 4, step by step using next_number, until you reach 1. INFO (llm_agents_fs.SupervisedTaskHandler) : ✅ Step Result: I need to call the next_number tool with x=4.
TaskStepResult(task_step_id='2d14732d-e307-4706-aaac-944ebfdf0c69', content='I need to call the next_number tool with x=4.')
step = await task_handler.get_next_step(step_result)
step
INFO (llm_agents_fs.SupervisedTaskHandler) : 🧠 New Step: Call the next_number tool with x=4.
TaskStep(id_='690b7f82-7f4f-45fa-ade0-af13029d85d2', task_id='fb5b7d5f-de29-4bec-b569-a88897230cf6', instruction='Call the next_number tool with x=4.')
step_result = await task_handler.run_step(step)
step_result
INFO (llm_agents_fs.SupervisedTaskHandler) : ⚙️ Processing Step: Call the next_number tool with x=4. INFO (llm_agents_fs.SupervisedTaskHandler) : 🛠️ Executing Tool Call: next_number INFO (llm_agents_fs.SupervisedTaskHandler) : ✅ Successful Tool Call: 2 INFO (llm_agents_fs.SupervisedTaskHandler) : ✅ Step Result: The next number in the Hailstone sequence starting from 4 is 2. I will now use the next_number tool again with x=2 to continue the sequ...[TRUNCATED]
TaskStepResult(task_step_id='690b7f82-7f4f-45fa-ade0-af13029d85d2', content='The next number in the Hailstone sequence starting from 4 is 2. I will now use the next_number tool again with x=2 to continue the sequence.')
step = await task_handler.get_next_step(step_result)
step
INFO (llm_agents_fs.SupervisedTaskHandler) : 🧠 New Step: Call the next_number tool with x=2.
TaskStep(id_='a145a42f-89b1-4be0-8213-94eb3e9dd7fb', task_id='fb5b7d5f-de29-4bec-b569-a88897230cf6', instruction='Call the next_number tool with x=2.')
step_result = await task_handler.run_step(step)
step_result
INFO (llm_agents_fs.SupervisedTaskHandler) : ⚙️ Processing Step: Call the next_number tool with x=2. INFO (llm_agents_fs.SupervisedTaskHandler) : 🛠️ Executing Tool Call: next_number INFO (llm_agents_fs.SupervisedTaskHandler) : ✅ Successful Tool Call: 1 INFO (llm_agents_fs.SupervisedTaskHandler) : ✅ Step Result: The next number in the Hailstone sequence starting from 2 is 1. Since we have reached 1, the sequence is complete. The full Hailstone s...[TRUNCATED]
TaskStepResult(task_step_id='a145a42f-89b1-4be0-8213-94eb3e9dd7fb', content='The next number in the Hailstone sequence starting from 2 is 1. Since we have reached 1, the sequence is complete. The full Hailstone sequence starting from 4 is: 4, 2, 1.')
result = await task_handler.get_next_step(step_result)
result
INFO (llm_agents_fs.SupervisedTaskHandler) : No new step required.
TaskResult(task_id='fb5b7d5f-de29-4bec-b569-a88897230cf6', content='The next number in the Hailstone sequence starting from 2 is 1. Since we have reached 1, the sequence is complete. The full Hailstone sequence starting from 4 is: 4, 2, 1.')
# since result is a TaskResult, we can call complete()
await task_handler.complete(result)
print(result.content)
The next number in the Hailstone sequence starting from 2 is 1. Since we have reached 1, the sequence is complete. The full Hailstone sequence starting from 4 is: 4, 2, 1.