Build a web-aware agent with OpenAI Agents SDK and Zenrows
Build a web-aware agent with OpenAI Agents SDK and Zenrows
This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/web-aware-agent-openai-agents-sdk-zenrows
The retrieval gap in OpenAI's built-in web search
The OpenAI Agents SDK ships WebSearchTool as a hosted tool that handles discovery well. It falls short in two structural ways, both of which matter for production agents.
First, it returns indexed snapshots, not live pages. When a site allows crawling, the built-in search returns a snippet of OpenAI's cached copy. For anything where freshness matters: a price, a stock level, a rate, that copy lags the live page by however long the refresh cycle takes.
Second, JavaScript-rendered content isn't in the index at all. Many pages send an HTML shell and populate prices, inventory, and structured data via JavaScript at runtime. The crawler stores the shell. The data you want was never captured, and crawling more often doesn't change that.
The architectural response is not to replace built-in search but to add a second tool alongside it, one that retrieves the full, live contents of a specific URL when discovery alone isn't enough. This guide shows how to register Zenrows Fetch as a @function_tool in the same agent, so the model selects between them each turn from their descriptions alone.
All the code is on GitHub.
Architecture overview
The agent holds two tools in one tools list. The model reads each tool's docstring every turn and selects whichever one fits the task.
WebSearchTool runs on OpenAI's infrastructure. It takes a query, not a URL, and returns indexed content from public pages. It covers open-ended discovery, finding relevant documentation, and answering questions from public indexed content.
fetch_page_content is a Python function decorated with @function_tool. It takes a specific URL, calls the Zenrows Fetch API with JavaScript rendering and adaptive proxy handling enabled, and returns the current page content as Markdown. It covers the cases built-in search can't reach: known URLs, protected pages, and JavaScript-rendered content.
The routing between them requires no conditional logic and no orchestration code on your side. It lives in the docstrings.
Prerequisites
Python 3.9 or later
An OpenAI API key from your OpenAI developer dashboard
A Zenrows API key from your Zenrows dashboard
Install dependencies:
python3 -m pip install openai-agents python-dotenv
Add both keys to a .env file:
OPENAI_API_KEY=your_openai_api_key_here
ZENROWS_API_KEY=your_zenrows_api_key_here
Add .env to .gitignore. Then load in your script:
from dotenv import load_dotenv
load_dotenv()
Set up a baseline agent with the built-in search
WebSearchTool is a hosted tool that runs on OpenAI's infrastructure through the Responses API. Enabling it requires one line.
import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool
load_dotenv()
agent = Agent(
name="Product Researcher",
instructions=(
"You research products and web pages. "
"Report exactly what you find, and state plainly when "
"specific information is missing from your results."
),
tools=[WebSearchTool()],
)
async def main():
result = await Runner.run(
agent,
"Which laptops did Apple release most recently, and where can I buy them?",
)
print(result.final_output)
asyncio.run(main())
On broad discovery queries, this works. The problem appears on live or protected targets.
Where the built-in search stops
Asking the agent to fetch a live price from a JavaScript-heavy retailer illustrates both gaps:
# This demonstrates the retrieval gap on a JavaScript-rendered page
result = await Runner.run(
agent,
"What is the current price and stock status of "
"https://www.amazon.com/dp/B0GR1JKMBV/ref=fs_a_mbt2_us1?th=1",
)
print(result.final_output)
The agent finds the page, returns the product title, and then states clearly what it couldn't access:
I visited the Amazon product page for ASIN B0GR1JKMBV...
The page displays: "To see product details, add this item to your cart."
Therefore I could not retrieve the current price or availability.
No error. The data simply wasn't in the index. This is the structural gap Zenrows fills.
Register Zenrows Fetch as a @function_tool
import os
import requests
from agents import function_tool
@function_tool
def fetch_page_content(url: str) -> str:
"""Fetch the full, current content of a specific web page as Markdown.
Use this when you need the complete content of a known URL rather than
a search result summary, such as live prices, stock levels, or data
that loads via JavaScript after the page opens.
Args:
url: The full URL of the page to retrieve.
"""
response = requests.get(
"https://api.zenrows.com/v1/",
params={
"url": url,
"apikey": os.getenv("ZENROWS_API_KEY"),
# mode=auto activates Adaptive Stealth Mode:
# JavaScript rendering and premium proxy escalation
# are enabled only when the target site requires them.
# This keeps costs proportional to what each request needs.
"mode": "auto",
# response_type=markdown strips HTML markup
# so the model reads clean text rather than raw HTML.
"response_type": "markdown",
},
timeout=90,
)
# Return the error as a string so the agent can react to it
# rather than receiving an unhandled exception.
if response.status_code != 200:
return f"Zenrows returned {response.status_code} for {url}"
return response.text
Why this approach
The @function_tool decorator builds the tool schema from the function signature and type hints automatically. You don't write a JSON schema. The model uses the schema to construct valid calls.
mode=auto is the important parameter choice. It activates Adaptive Stealth Mode, which evaluates each target individually and enables JavaScript rendering and premium proxies only when the site requires them. An always-on premium proxy would work, but mode=auto keeps cost proportional to what each request actually needs.
The docstring is the routing mechanism. The model reads it every turn to decide whether the tool applies. The phrase "rather than a search result summary" is what distinguishes this tool from WebSearchTool in the model's decision-making. Without that phrase, the model treats both tools as interchangeable for any web-related task.
Run both tools in a single agent loop
import asyncio
import os
import requests
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool, function_tool
load_dotenv()
@function_tool
def fetch_page_content(url: str) -> str:
"""Fetch the full, current content of a specific web page as Markdown.
Use this when you need the complete content of a known URL rather than
a search result summary, such as live prices, stock levels, or data
that loads via JavaScript after the page opens.
Args:
url: The full URL of the page to retrieve.
"""
response = requests.get(
"https://api.zenrows.com/v1/",
params={
"url": url,
"apikey": os.getenv("ZENROWS_API_KEY"),
"mode": "auto",
"response_type": "markdown",
},
timeout=90,
)
if response.status_code != 200:
return f"Zenrows returned {response.status_code} for {url}"
return response.text
# Both tools are registered in the same list.
# The model selects between them from their descriptions each turn.
agent = Agent(
name="Web-Aware Researcher",
instructions=(
"You research products on the web. "
"Use web search to find the product page URL. "
"Then call fetch_page_content on that URL and report the price "
"and stock status from the fetched page content. "
"State which tool each figure came from."
),
tools=[WebSearchTool(), fetch_page_content],
)
def print_trace(items):
# Inspect result.new_items to confirm which tools the agent called
for item in items:
raw = getattr(item, "raw_item", None)
name = getattr(raw, "name", None) or getattr(raw, "type", "")
print(f"[{item.type}] {name}")
if item.type == "tool_call_item" and name == "fetch_page_content":
print(" args:", getattr(raw, "arguments", ""))
if item.type == "tool_call_output_item":
out = str(getattr(item, "output", ""))
print(f" output: {len(out)} chars")
print(f" body: {out[:500]}")
async def main():
result = await Runner.run(
agent,
"Find the PriceOye page for the iPhone 17 Pro "
"and report its current price and stock status.",
)
print_trace(result.new_items)
print("\n---\n")
print(result.final_output)
asyncio.run(main())
The trace shows the two-step pattern. Turn one is a web_search_call, discovery with no URL specified. Turn two calls fetch_page_content with the URL it found:
[tool_call_item] web_search_call
[message_output_item] message
[tool_call_item] fetch_page_content
args: {"url":"https://priceoye.pk/mobiles/apple/apple-iphone-17-pro"}
[tool_call_output_item]
---
Here are the details for the iPhone 17 Pro from its PriceOye product page:
Current Price: Rs 471,999
Stock Status: Only 1 left in stock
Price and stock status are directly extracted from the fetched page content
(functions.fetch_page_content tool).
The routing decision the model made: web_search_call for discovery and fetch_page_content for retrieval matches the docstrings almost exactly. There's no conditional logic anywhere in the agent definition.
When to use each tool
WebSearchTool covers discovery: finding relevant pages, answering questions from indexed public content, identifying URLs for follow-up retrieval.
Zenrows covers retrieval: full contents of a specific URL, protected pages, JavaScript-rendered content, anything where freshness matters.
If you need specific fields rather than full-page Markdown, Zenrows Extract returns structured data so the model never has to parse Markdown for a price. For recurring high-volume retrieval across a catalog, Zenrows Batch runs those jobs without you managing concurrency.
The same @function_tool wrapper carries over to other agent frameworks. The guide on building a web research multi-agent system with AG2 and Zenrows shows how the pattern extends to multi-agent setups. Zenrows smolagents uses the same function under a @tool decorator.
Conclusion
The agent now reaches the web two ways. WebSearchTool handles discovery. Zenrows reads the full live page when you have a URL and need the current contents, including JavaScript-rendered data and anti-bot-defended targets. The model routes between them from the tool descriptions alone, with no orchestration code.
What to build next
Zenrows MCP server: ready-made tools via the Responses API
mcptool type, with no Python wrapper to maintainOpenAI Agents SDK integration guide: full reference for MCP and
@function_toolpatterns in the SDKZenrows Batch: scheduled retrieval across many URLs without managing concurrency
Zenrows Extract: structured field extraction so the model doesn't parse raw Markdown
