Build a web research multi-agent system with AG2 and Zenrows
Build a web research multi-agent system with AG2 and Zenrows
This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/web-research-multi-agent-ag2-zenrows
Why the fetch layer determines everything downstream
AG2's group chat architecture is built around a shared history. Every message, including every tool response, enters a shared context window that all agents read on their turn. This is what makes multi-agent pipelines powerful: each agent builds on what the agents before it produced.
It's also what makes a broken fetch expensive. When the researcher can't reach a URL, that failure enters the shared history as a message. The analyst reads it, has nothing to work with, and says so. The critic reads both, validates that neither produced anything, and terminates. Every agent in the pipeline spent tokens on content that never existed.
The fix is architectural. Register a fetch tool at the researcher level that actually works, one that handles JavaScript-rendered and anti-bot-protected pages, so clean content enters the history from the first turn. This guide does that with Zenrows Fetch, compares token usage across both configurations, and explains the cost dynamics in a multi-agent context.
AG2 is the actively maintained community fork of Microsoft's AutoGen, with native MCP client support added in v0.12. This guide uses the OpenAI client.
All the code is on GitHub.
Architecture overview
The pipeline has four agents in one group chat.
user_proxy is the execution agent. It runs any registered function the LLM requests and terminates the chat when an agent message contains TERMINATE.
researcher holds the fetch tool. When given a URL, it calls fetch_page_content, receives clean Markdown from Zenrows, and passes a summary to the analyst.
analyst reads the researcher's summary and extracts structured fields — product names, prices, or whatever the task specifies. It doesn't call any tools.
critic validates the analyst's output against the researcher's original content, flags gaps or inconsistencies, and terminates the pipeline when the output checks out.
The key design decision is where the tool is registered. register_for_llm(researcher) exposes the schema to the researcher so the model knows to call it. register_for_execution(user_proxy) tells AG2 which agent runs the function — in a group chat, that's always the UserProxyAgent. This split is required; registering both on the same agent doesn't work.
Prerequisites
Python 3.10 or later
AG2 v0.12.2
Install AG2 with its OpenAI client:
pip install "ag2[openai]==0.12.2"
Two things before you run anything. The package name and the import name differ: install as ag2, import as autogen. Using import ag2 raises a ModuleNotFoundError in v0.12.2; AG2 is deprecating the classic framework at v1.0, at which point import ag2 becomes correct. AG2 also doesn't bundle the OpenAI client, which is why [openai] is required.
Load keys from .env:
import os
from dotenv import load_dotenv
import autogen
from autogen import AssistantAgent, GroupChat, GroupChatManager, UserProxyAgent
load_dotenv()
llm_config = {
"config_list": [
{
"model": "gpt-4o-mini",
"api_key": os.getenv("OPENAI_API_KEY"),
}
],
# temperature 0 for deterministic output across runs
"temperature": 0,
}
Set up an AG2 group chat
Define four agents, wire them into a group chat, and run a baseline to see the pipeline with no fetch tool:
# user_proxy executes tools and terminates on TERMINATE
user_proxy = UserProxyAgent(
name="user_proxy",
human_input_mode="NEVER",
code_execution_config=False,
is_termination_msg=lambda msg: "TERMINATE" in (msg.get("content") or ""),
)
# researcher fetches and summarizes — no tool registered yet
researcher = AssistantAgent(
name="researcher",
system_message=(
"You are a research agent. When given a URL, fetch its web content and "
"summarize what you find. Be factual and note anything that looks "
"incomplete or missing from the page. Pass your summary to the analyst."
),
llm_config=llm_config,
)
# analyst extracts structured fields from researcher output
analyst = AssistantAgent(
name="analyst",
system_message=(
"You are a data analyst. Extract structured fields, such as plan names "
"and prices, from the content the researcher provides. Present the "
"result as a clean list. If the researcher's content has no usable "
"data, say so explicitly instead of guessing."
),
llm_config=llm_config,
)
# critic validates and terminates
critic = AssistantAgent(
name="critic",
system_message=(
"You are a critic. Validate the analyst's structured output against "
"the researcher's original content. Flag any gaps, missing fields, or "
"inconsistencies. When the output checks out or no further progress is "
"possible, say so and end your message with TERMINATE."
),
llm_config=llm_config,
)
group_chat = GroupChat(
agents=[user_proxy, researcher, analyst, critic],
messages=[],
max_round=6,
)
manager = GroupChatManager(groupchat=group_chat, llm_config=llm_config)
chat_result = user_proxy.initiate_chat(
manager,
message=(
"Fetch the product listing from this page and extract the "
"product name and price: https://www.walmart.com/ip/AirPods-Pro-3/17835006350"
),
)
For accurate token counts, call autogen.gather_usage_summary() on the full agent list. chat_result.cost always reports zero for group chats:
# gather_usage_summary is the only reliable source for group chat token counts
usage_summary = autogen.gather_usage_summary(
[user_proxy, researcher, analyst, critic, manager]
)
print(usage_summary)
The baseline output shows the failure propagating:
researcher: I'm unable to access external URLs directly to fetch content.
analyst: I cannot access external URLs or fetch content from them.
critic: Both parties acknowledge the inability to access external URLs. TERMINATE
Total tokens: 990 | Cost: $0.000252
| Run | Products extracted | Total tokens | Cost |
|---|---|---|---|
| Baseline (no fetch tool) | No | 990 | $0.000252 |
Register Zenrows Fetch as a typed AG2 tool
AG2 builds the tool schema from type hints. Annotated[type, description] on every input parameter is required. Without it, AG2 can't build the schema, and registration fails silently or raises a validation error.
import requests
from typing import Annotated
from autogen.tools import Tool
ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")
ZENROWS_ENDPOINT = "https://api.zenrows.com/v1/"
def fetch_page_content(
url: Annotated[str, "The target URL to fetch through Zenrows using adaptive stealth mode."],
) -> str:
"""Fetch a URL through Zenrows Fetch and return clean Markdown."""
params = {
"url": url,
"apikey": ZENROWS_API_KEY,
# mode=auto activates Adaptive Stealth Mode:
# Zenrows evaluates the target and enables JS rendering
# or premium proxies only when the site requires them.
# This keeps credit spend proportional to what each request needs.
"mode": "auto",
# response_type=markdown returns clean text rather than raw HTML,
# keeping token count proportional to content rather than markup.
"response_type": "markdown",
}
response = requests.get(ZENROWS_ENDPOINT, params=params, timeout=60)
response.raise_for_status()
return response.text
Why this approach
mode=auto is Adaptive Stealth Mode. Rather than always enabling JavaScript rendering and premium proxies, which adds cost on pages that don't need them. Zenrows evaluates each target individually. A basic static page costs 1 credit. JavaScript rendering costs 5, premium proxies cost 10, and a heavily protected site needing both costs 25. You're billed once, for the configuration that succeeds.
response_type=markdown matters specifically in a multi-agent context. Raw HTML inflates the shared history, which inflates every subsequent agent turn that reads it. Markdown gives the model what it needs without the markup overhead.
Wrap the function and register it on the correct agents:
fetch_tool = Tool(
name="fetch_page_content",
description=(
"Fetch a URL through Zenrows and return clean Markdown content from "
"JavaScript-rendered and anti-bot-protected pages."
),
func_or_tool=fetch_page_content,
)
# register_for_llm exposes the schema to the researcher
fetch_tool.register_for_llm(researcher)
# register_for_execution tells AG2 which agent runs the function
# — always user_proxy in a group chat
fetch_tool.register_for_execution(user_proxy)
Update the researcher's system message to name the tool:
researcher = AssistantAgent(
name="researcher",
system_message=(
"You are a research agent. When given a URL, use the fetch_page_content "
"tool to retrieve its web content, then summarize what you find. Be "
"factual and note anything that looks incomplete or missing from the "
"page. Pass your summary to the analyst."
),
llm_config=llm_config,
)
Run the pipeline and compare token spend
With Zenrows registered, the researcher calls fetch_page_content. Zenrows retrieves the page through Adaptive Stealth Mode. Clean Markdown enters the shared history. The analyst extracts product data. The critic validates and terminates.
researcher: ***** Suggested tool call: fetch_page_content *****
Arguments: {"url":"https://www.walmart.com/ip/AirPods-Pro-3/17835006350"}
>>>>>>>> EXECUTING FUNCTION fetch_page_content...
[Markdown from Walmart page enters shared history]
analyst:
- Product Name: Apple AirPods Pro 3
- Price: $189.99 (was $249.00, you save $59.01)
critic: The analyst's output is consistent with the researcher's content.
No gaps or missing fields. TERMINATE
Total tokens: 7,236 | Cost: $0.001147
| Run | Products extracted | Total tokens | Cost |
|---|---|---|---|
| Baseline (no fetch tool) | No | 990 | $0.000252 |
| Fetch with mode=auto | Yes | 7,236 | $0.001147 |
The working run spends more tokens because the Markdown from a fully fetched page is far larger than the researcher's brief refusal in the baseline. That token spend is proportional to content. The baseline was cheaper because it retrieved nothing.
Both runs terminate in approximately the same number of turns, so the round count isn't the cost variable. What changes is whether there's real content in the history.
Keeping costs predictable in production
Content volume in the shared history is the primary cost driver, not the number of turns. Every agent turn re-reads and forwards the full history, so page size multiplies across the pipeline.
max_round sets a ceiling on your worst case. Start conservative and raise it only when the pipeline consistently needs more turns to finish naturally. Every failed or empty fetch that enters the history pushes the round count higher without producing anything useful, so reliable retrieval is the most direct way to keep max_round from becoming your main cost control.
For pipelines that fetch the same URL more than once per session, caching the Zenrows response after the first call cuts both token spend and credit usage. For pipelines that research several URLs in one run, Zenrows Batch takes the full list as one managed job and handles concurrency and retries.
Conclusion
AG2's shared history is the architectural fact that makes the fetch layer critical. A failed fetch doesn't stay isolated. It enters the history and every agent after reads and forwards it, spending tokens on turns that had nothing real to work with.
Registering Zenrows as a typed fetch tool with mode=auto addresses that at the source. Zenrows decides what each target needs, whether that's plain rendering, JavaScript execution, or premium proxies, without you configuring it per site. Clean Markdown enters the shared history on the first call, so the analyst and critic get real content and the pipeline terminates naturally.
What to build next
Zenrows MCP server: AG2 v0.12 has native MCP client support if you prefer ready-made tools over a registered function
OpenAI Agents SDK + Zenrows: the same Zenrows fetch pattern in a single-agent loop with
@function_toolZenrows smolagents guide: the same wrapper under a
@tooldecorator in a different frameworkZenrows Batch: scheduled or high-volume retrieval across many URLs without managing fetch queues
Adaptive Stealth Mode pricing table: full breakdown of credit costs by configuration
