Integrating Zenrows into smolagents for production web access

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/zenrows-smolagents

Architecture overview

smolagents' VisitWebpageTool is built on Python's requests library. That means every fetch call carries the HTTP client fingerprint of a headless Python process. On protected or JavaScript-rendered pages, the target site identifies the request as automated traffic at the TLS handshake level and returns a verification challenge instead of content.

The problem is not that the tool fails visibly. It fails silently. The agent receives the challenge page and cannot distinguish it from real data, so it reasons over bad input and still returns output. In production workflows, that means research, summarization, and classification tasks all run correctly against the wrong content.

This guide replaces VisitWebpageTool with a Zenrows fetch tool registered via the smolagents @tool decorator. Zenrows sits between your agent and the target site, handles JavaScript rendering and bot detection transparently using Adaptive Stealth Mode, and returns clean Markdown. Your CodeAgent architecture stays unchanged.

What you'll implement:

• A fetch_page tool built on Zenrows Fetch with mode=auto

• A docstring structured to drive correct tool selection at runtime

• A CodeAgent with fetch_page registered in its tools list

• Concurrent Hugging Face Spaces handling via Zenrows routing

Full project code: GitHub repository

Why VisitWebpageTool breaks in production

Here is what VisitWebpageTool returned when run against a protected Walmart product page:

Robot or human?
===============

Activate and hold the button to confirm that you're human. Thank You


(c) Walmart Stores, Inc.

The TLS handshake exposed the Python HTTP client, and Walmart returned a verification page. The agent received that, had no mechanism to validate it, and would have reasoned from it.

max_output_length on VisitWebpageTool controls truncation, not retrieval. Raising it does not change how the tool fetches pages. The fix is at the retrieval layer.

Prerequisites

• Python 3.10 or later

• A Zenrows account for your API key

• A Hugging Face account for the model access token

Step 1: Install packages

pip install smolagents requests python-dotenv

Step 2: Configure credentials

# .env
ZENROWS_API_KEY=your_zenrows_api_key_here
HF_TOKEN=your_hugging_face_token_here

Step 3: Build the Zenrows fetch tool

Create zenrows_tool.py. The @tool decorator is the registration mechanism. It is what makes fetch_page visible to CodeAgent during planning. mode=auto delegates retrieval strategy selection to Zenrows' Adaptive Stealth Mode, which chooses between direct fetch, headless browser, and proxy routing based on how the target responds.

import os

import requests
from dotenv import load_dotenv
from smolagents import tool

load_dotenv()

# Load the API key once at module level, not inside the function,
# so it does not trigger a file read on every tool call.
ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

@tool
def fetch_page(url: str) -> str:
    """
    Fetches webpage content and returns it as clean Markdown, including
    JavaScript-rendered and protected pages.

    Use this tool whenever you need to read a specific URL and retrieve
    webpage content for research, summarization, or analysis.

    Args:
        url: The webpage URL to fetch.
    """
    try:
        response = requests.get(
            "https://api.zenrows.com/v1/",
            params={
                "url": url,
                "apikey": ZENROWS_API_KEY,
                "mode": "auto",           # Adaptive Stealth Mode — picks retrieval strategy per target
                "response_type": "markdown",  # Return clean Markdown, not raw HTML
            },
            timeout=30,
        )
        response.raise_for_status()
        return response.text

    except requests.RequestException as exc:
        raise RuntimeError(f"Failed to retrieve content from {url}") from exc

Why this approach: the docstring drives tool selection

smolagents constructs the tool description the model sees at inference time from the function name, type hints, and docstring. The model consults that description when deciding which tool to call and how to call it. A vague docstring produces incorrect behavior. The agent skips the tool, calls it with the wrong argument type, or falls back to writing its own fetch code.

The docstring in zenrows_tool.py specifies three things explicitly:

1. What the tool returns: clean Markdown, including from JS-rendered and protected pages

2. When to use it: any time the agent needs to read a specific URL

3. What kinds of pages it handles: JavaScript-rendered and protected targets

Without that specificity, the model cannot reliably route retrieval tasks to fetch_page.

Step 4: Test the fetch tool

Verify the tool returns real content before wiring it to an agent:

# Add to the bottom of zenrows_tool.py
if __name__ == "__main__":
    # Test against the same protected Walmart page that failed with VisitWebpageTool
    result = fetch_page("https://www.walmart.com/ip/AirPods-Pro-3/17835006350")
    # Print a mid-page slice to confirm product content, not a bot check
    print(result[1500:2500])

python zenrows_tool.py

The output should contain Markdown from the product page, not a verification challenge.

Step 5: Register the tool and run the CodeAgent

Create agent.py. The CodeAgent receives fetch_page in its tools list. During a run, the model plans in Python, calls fetch_page when it needs page content, and structures its output from what Zenrows returns.

import os

from dotenv import load_dotenv
from smolagents import CodeAgent, InferenceClientModel

from zenrows_tool import fetch_page

load_dotenv()

# Initialize the model via Hugging Face Inference Providers.
# Swap in any compatible model_id.
model = InferenceClientModel(
    model_id="Qwen/Qwen2.5-7B-Instruct",
    token=os.getenv("HF_TOKEN"),
)

# Register fetch_page as the only retrieval tool.
# The agent will call it for any URL it needs to read.
agent = CodeAgent(
    tools=[fetch_page],
    model=model,
)

response = agent.run(
    """
    Go to https://techcrunch.com/2026/07/23/amd-takes-on-nvidia-with-its-helios-ai-rack-scale-system/

    Read the article and identify:
    - the company involved,
    - the main announcement,
    - the news category,
    - why it matters.
    """,
    max_steps=8,  # Cap steps to avoid unnecessary token spend. Default is 20.
)

print(response)

python agent.py

The run finished in three steps. The agent called fetch_page once, received valid content, and structured its output:

{'company': 'AMD',
 'main_announcement': 'AMD has launched the Helios AI rack-scale system, challenging Nvidia in the AI rack-scale system market.',
 'news_category': 'AI',
 'why_it_matters': 'AMD is challenging Nvidia in the AI rack-scale system market, which is a significant move as Nvidia has historically dominated this market.'}

Handling concurrent Hugging Face Spaces deployments

Shared Hugging Face Spaces route all outbound requests through a shared pool of managed IP addresses. Sites that enforce per-IP rate limits see all Spaces traffic as one source. Under concurrent usage, that produces slow responses, temporary blocks, or verification challenges at scale.

fetch_page routes requests through Zenrows rather than directly from your Space to the target. mode=auto is already set in the params block, so the agent retrieves live content under concurrent load without proxy configuration of your own.

params={
    "url": url,
    "apikey": ZENROWS_API_KEY,
    "mode": "auto",            
    "response_type": "markdown",
}

What to build next

The @tool pattern generalises across agent frameworks:

• OpenAI Agents SDK: uses @function_tool in place of @tool, same retrieval pattern

• AG2 multi-agent system: typed Tool registered on two agents in a group chat

• LlamaIndex RAG pipeline: indexes and queries live web content using the same Zenrows Fetch layer

If you prefer not to manage the tool wrapper yourself, the Zenrows MCP server exposes the same capabilities and smolagents supports loading tools from an MCP server directly.

Full project: GitHub repository