This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/zenrows-smolagents

Architecture overview

smolagents' VisitWebpageTool is built on Python's requests library. That means every fetch call carries the HTTP client fingerprint of a headless Python process. On protected or JavaScript-rendered pages, the target site identifies the request as automated traffic at the TLS handshake level and returns a verification challenge instead of content.

The problem is not that the tool fails visibly. It fails silently. The agent receives the challenge page and cannot distinguish it from real data, so it reasons over bad input and still returns output. In production workflows, that means research, summarization, and classification tasks all run correctly against the wrong content.

This guide replaces VisitWebpageTool with a Zenrows fetch tool registered via the smolagents @tool decorator. Zenrows sits between your agent and the target site, handles JavaScript rendering and bot detection transparently using Adaptive Stealth Mode, and returns clean Markdown. Your CodeAgent architecture stays unchanged.

What you'll implement:

• A fetch_page tool built on Zenrows Fetch with mode=auto

• A docstring structured to drive correct tool selection at runtime

• A CodeAgent with fetch_page registered in its tools list

• Concurrent Hugging Face Spaces handling via Zenrows routing

Full project code: GitHub repository

Why VisitWebpageTool breaks in production

Here is what VisitWebpageTool returned when run against a protected Walmart product page:

Robot or human?
===============

Activate and hold the button to confirm that you're human. Thank You


(c) Walmart Stores, Inc.

The TLS handshake exposed the Python HTTP client, and Walmart returned a verification page. The agent received that, had no mechanism to validate it, and would have reasoned from it.

max_output_length on VisitWebpageTool controls truncation, not retrieval. Raising it does not change how the tool fetches pages. The fix is at the retrieval layer.

Prerequisites

• Python 3.10 or later

• A Zenrows account for your API key

• A Hugging Face account for the model access token

Step 1: Install packages

pip install smolagents requests python-dotenv

Step 2: Configure credentials

# .env
ZENROWS_API_KEY=your_zenrows_api_key_here
HF_TOKEN=your_hugging_face_token_here

Step 3: Build the Zenrows fetch tool

Create zenrows_tool.py. The @tool decorator is the registration mechanism. It is what makes fetch_page visible to CodeAgent during planning. mode=auto delegates retrieval strategy selection to Zenrows' Adaptive Stealth Mode, which chooses between direct fetch, headless browser, and proxy routing based on how the target responds.

import os

import requests
from dotenv import load_dotenv
from smolagents import tool

load_dotenv()

# Load the API key once at module level, not inside the function,
# so it does not trigger a file read on every tool call.
ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

@tool
def fetch_page(url: str) -> str:
    """
    Fetches webpage content and returns it as clean Markdown, including
    JavaScript-rendered and protected pages.

    Use this tool whenever you need to read a specific URL and retrieve
    webpage content for research, summarization, or analysis.

    Args:
        url: The webpage URL to fetch.
    """
    try:
        response = requests.get(
            "https://api.zenrows.com/v1/",
            params={
                "url": url,
                "apikey": ZENROWS_API_KEY,
                "mode": "auto",           # Adaptive Stealth Mode — picks retrieval strategy per target
                "response_type": "markdown",  # Return clean Markdown, not raw HTML
            },
            timeout=30,
        )
        response.raise_for_status()
        return response.text

    except requests.RequestException as exc:
        raise RuntimeError(f"Failed to retrieve content from {url}") from exc

Why this approach: the docstring drives tool selection

smolagents constructs the tool description the model sees at inference time from the function name, type hints, and docstring. The model consults that description when deciding which tool to call and how to call it. A vague docstring produces incorrect behavior. The agent skips the tool, calls it with the wrong argument type, or falls back to writing its own fetch code.

The docstring in zenrows_tool.py specifies three things explicitly:

1. What the tool returns: clean Markdown, including from JS-rendered and protected pages

2. When to use it: any time the agent needs to read a specific URL

3. What kinds of pages it handles: JavaScript-rendered and protected targets

Without that specificity, the model cannot reliably route retrieval tasks to fetch_page.

Step 4: Test the fetch tool

Verify the tool returns real content before wiring it to an agent:

# Add to the bottom of zenrows_tool.py
if __name__ == "__main__":
    # Test against the same protected Walmart page that failed with VisitWebpageTool
    result = fetch_page("https://www.walmart.com/ip/AirPods-Pro-3/17835006350")
    # Print a mid-page slice to confirm product content, not a bot check
    print(result[1500:2500])

python zenrows_tool.py

The output should contain Markdown from the product page, not a verification challenge.

Step 5: Register the tool and run the CodeAgent

Create agent.py. The CodeAgent receives fetch_page in its tools list. During a run, the model plans in Python, calls fetch_page when it needs page content, and structures its output from what Zenrows returns.

import os

from dotenv import load_dotenv
from smolagents import CodeAgent, InferenceClientModel

from zenrows_tool import fetch_page

load_dotenv()

# Initialize the model via Hugging Face Inference Providers.
# Swap in any compatible model_id.
model = InferenceClientModel(
    model_id="Qwen/Qwen2.5-7B-Instruct",
    token=os.getenv("HF_TOKEN"),
)

# Register fetch_page as the only retrieval tool.
# The agent will call it for any URL it needs to read.
agent = CodeAgent(
    tools=[fetch_page],
    model=model,
)

response = agent.run(
    """
    Go to https://techcrunch.com/2026/07/23/amd-takes-on-nvidia-with-its-helios-ai-rack-scale-system/

    Read the article and identify:
    - the company involved,
    - the main announcement,
    - the news category,
    - why it matters.
    """,
    max_steps=8,  # Cap steps to avoid unnecessary token spend. Default is 20.
)

print(response)

python agent.py

The run finished in three steps. The agent called fetch_page once, received valid content, and structured its output:

{'company': 'AMD',
 'main_announcement': 'AMD has launched the Helios AI rack-scale system, challenging Nvidia in the AI rack-scale system market.',
 'news_category': 'AI',
 'why_it_matters': 'AMD is challenging Nvidia in the AI rack-scale system market, which is a significant move as Nvidia has historically dominated this market.'}

Handling concurrent Hugging Face Spaces deployments

Shared Hugging Face Spaces route all outbound requests through a shared pool of managed IP addresses. Sites that enforce per-IP rate limits see all Spaces traffic as one source. Under concurrent usage, that produces slow responses, temporary blocks, or verification challenges at scale.

fetch_page routes requests through Zenrows rather than directly from your Space to the target. mode=auto is already set in the params block, so the agent retrieves live content under concurrent load without proxy configuration of your own.

params={
    "url": url,
    "apikey": ZENROWS_API_KEY,
    "mode": "auto",            
    "response_type": "markdown",
}

What to build next

The @tool pattern generalises across agent frameworks:

• OpenAI Agents SDK: uses @function_tool in place of @tool, same retrieval pattern

• AG2 multi-agent system: typed Tool registered on two agents in a group chat

• LlamaIndex RAG pipeline: indexes and queries live web content using the same Zenrows Fetch layer

If you prefer not to manage the tool wrapper yourself, the Zenrows MCP server exposes the same capabilities and smolagents supports loading tools from an MCP server directly.

Full project: GitHub repository

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/web-data-llm-fine-tuning

Architecture overview

LLM fine-tuning pipelines that rely entirely on synthetic data are structurally unstable. The generator produces examples using the teacher model's patterns; the judge filters against those same patterns. Each retrain cycle amplifies the biases from the previous one. By the third or fourth cycle, the model has learned to reproduce domain text rather than to reflect domain knowledge.

The intervention is a real-data seed, a small, verified set of examples collected from authoritative sources before any synthetic expansion begins. The seed does two things. It gives the generator concrete domain examples to expand from rather than its own patterns. It gives the judge something to filter against that is not its own output.

Fine-tuning amplifies signal differently than pretraining. A single mislabeled extraction in a dataset of a few hundred rows represents a measurable percentage of the training signal, and the model takes it as a rule. This guide covers collecting that seed from sources that block standard crawlers, formatting it correctly, validating each row, and measuring whether the fine-tune held.

What you'll implement:

• Protected web data collection using Zenrows Fetch with mode=auto

• PDF retrieval and structured chunking

• Provenance tracking per source document

• Concurrent batch collection with rate limit management

• Three fine-tuning data formats: instruction-output pairs, preference pairs, domain continuation

• Seed validation with length, schema, and deduplication checks

• Holdout split and accuracy measurement

All code: GitHub repository

Why authoritative sources are the hard part

Authoritative domain sources like regulatory filings, clinical guidance, technical documentation, expert-reviewed publications, are almost always behind access controls. They block standard crawlers at the TLS handshake level. A plain requests.get() exposes the Python HTTP client fingerprint, the target identifies it as automated traffic, and the response is a challenge page or an empty shell.

This is not a scraping problem that can be solved with headers or user-agent rotation alone. JavaScript-rendered pages require a headless browser context. Protected sources require IP rotation and fingerprint masking at the session level. The collection layer has to handle both, per request, without per-site configuration.

Prerequisites

• Python 3.9 or later

• Zenrows account

• pypdf for PDF text extraction

python3 -m pip install requests python-dotenv pypdf jsonschema

# .env — add to .gitignore before committing
ZENROWS_API_KEY=your_zenrows_api_key_here

Collecting from protected sources

Zenrows Fetch handles JavaScript rendering, access control, and Markdown conversion in one API call. mode=auto is Adaptive Stealth Mode. Zenrows selects the retrieval configuration per target based on how that target responds, alternating between direct fetch, headless browser, and proxy escalation as needed.

import os
import requests
from dotenv import load_dotenv

load_dotenv()

# Load at module level — one environment read, not one per request
apikey = os.getenv("ZENROWS_API_KEY")

params = {
    "url": "https://www.scrapingcourse.com/ecommerce/",
    "apikey": apikey,
    "mode": "auto",              # Adaptive Stealth Mode picks retrieval strategy per target
    "response_type": "markdown", # Returns clean Markdown, not raw HTML
}

response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)

The Markdown output strips most navigation, footer, and boilerplate. That reduced noise is what makes it a better training input than raw HTML.

Handling PDFs and non-HTML documents

The most authoritative domain sources are often PDFs. response_type=pdf returns raw bytes; js_render=true is required and must be included explicitly.

params = {
    "url": pdf_url,
    "apikey": apikey,
    "js_render": "true",    # Required for PDF response type
    "response_type": "pdf", # Returns raw bytes — extract text separately
}

response = requests.get("https://api.zenrows.com/v1/", params=params)

with open("source-document.pdf", "wb") as f:
    f.write(response.content)

Why this approach: chunk on structure, not character count

A regulatory filing can run to tens of thousands of tokens. Fixed-character chunking cuts mid-sentence, mid-argument, and mid-table. Chunking on the document's own page boundaries preserves context. Each chunk stays internally coherent, which is what makes it a usable training example.

from pypdf import PdfReader
import io

reader = PdfReader(io.BytesIO(response.content))

def chunk_by_page(reader, max_chars=6000):
    chunks = []
    for page in reader.pages:
        text = page.extract_text()
        if not text or not text.strip():
            continue  # skip blank or image-only pages
        if len(text) <= max_chars:
            chunks.append(text)
        else:
            # only split within a page when the page itself exceeds the limit
            for i in range(0, len(text), max_chars):
                chunks.append(text[i:i + max_chars])
    return chunks

chunks = chunk_by_page(reader)
print(f"Produced {len(chunks)} chunks from {len(reader.pages)} pages")

Provenance tracking

Every row in an authoritative seed needs a traceable source. A metadata record per document keeps the dataset auditable, covers legal requirements for commercial use, and makes it possible to re-retrieve and re-verify specific rows later.

from datetime import date

# Fetch the license from the landing page using Zenrows CSS extractor
license_params = {
    "url": landing_page_url,
    "apikey": apikey,
    "js_render": "true",
    "css_extractor": '{"license_url": "a[href*=\\"creativecommons.org\\"] @href"}',
}
license_resp = requests.get("https://api.zenrows.com/v1/", params=license_params)
license_data = license_resp.json()

source = {
    "url": pdf_url,
    "retrieved": date.today().isoformat(),
    "license": license_data.get("license_url", "unknown"),
}

Batch collection with rate limit management

Match concurrency to your plan limit. Spreading requests across workers with a short pause keeps the pipeline inside your quota without throttling from the target side.

import time
from concurrent.futures import ThreadPoolExecutor

MAX_CONCURRENT = 50  # Verify against your Zenrows plan

def fetch(url):
    params = {
        "url": url,
        "apikey": apikey,
        "response_type": "markdown",
        "mode": "auto",
        "wait": 3000,  # Wait for dynamic content to settle before returning
    }
    response = requests.get("https://api.zenrows.com/v1/", params=params)
    time.sleep(0.5)  # Pace requests within each worker thread

    if response.status_code != 200:
        return {"url": url, "status": response.status_code, "content": None}
    return {"url": url, "status": response.status_code, "content": response.text}

with ThreadPoolExecutor(max_workers=MAX_CONCURRENT) as executor:
    results = list(executor.map(fetch, urls))

For larger runs, Zenrows Batch manages concurrency, retries, and scheduling across large URL lists without a local orchestration layer.

Three fine-tuning data formats

The right format depends on the training method, not on what is easiest to produce.

Instruction-output pairs

Standard for supervised fine-tuning. Each example pairs the page Markdown (input) with the correct structured extraction (output). The model learns what to extract and how to format it.

import json

def to_training_example(instruction, markdown_content, extracted_json):
    return {
        "messages": [
            {"role": "system", "content": instruction},
            # The Markdown you pulled from the protected source
            {"role": "user", "content": markdown_content},
            # The verified structured output for this input
            {"role": "assistant", "content": json.dumps(extracted_json)},
        ]
    }

instruction = "Extract the product name, price, and availability as JSON."

with open("training_data.jsonl", "w") as f:
    for markdown_content, extracted_json in seed_examples:
        example = to_training_example(instruction, markdown_content, extracted_json)
        f.write(json.dumps(example) + "\n")  # One JSON object per line — JSONL format

Preference pairs

Used in DPO and RLHF. Two candidate outputs for the same input, one marked preferred. Correct when the model already produces reasonable outputs and the goal is distributional refinement rather than teaching the task from scratch.

def to_preference_pair(prompt, chosen, rejected):
    return {
        "prompt": prompt,
        "chosen": chosen,   # The preferred output
        "rejected": rejected, # The not-preferred output to the same prompt
    }

Domain text continuation

Raw domain text with no input-output structure. Useful for domain vocabulary and style familiarization, weak for extraction tasks because the model is never shown what should be extracted.

def to_continuation_example(markdown_content):
    # One field — the model learns by predicting what comes next
    return {"text": markdown_content}

For extraction tasks, instruction-output pairs are the correct choice. Continuation is a pretraining-style objective applied at fine-tuning scale; it improves fluency, not accuracy.

Seed validation

A row that fails any of these three checks does not enter the training set.

import hashlib
from jsonschema import validate, ValidationError

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string", "minLength": 1},
        "price": {"type": "string", "minLength": 1},
    },
    "required": ["name", "price"],
}

seen = set()

def keep(markdown, extracted):
    # Length: pages that rendered to almost nothing carry no useful signal
    if len(markdown.strip()) < 200:
        return False
    try:
        # Schema: catches wrong types, empty values, missing required fields
        validate(instance=extracted, schema=schema)
    except ValidationError:
        return False
    # Deduplication: normalise whitespace before hashing to catch formatting-only duplicates
    h = hashlib.sha256(" ".join(markdown.lower().split()).encode()).hexdigest()
    if h in seen:
        return False
    seen.add(h)
    return True

clean_examples = [(url, md, js) for url, md, js in seed_examples if keep(md, js)]

Measuring whether the seed held

Hold out 10 to 20 percent of the real seed before any synthetic expansion and keep those examples permanently out of training and out of the synthetic generator.

import random

random.seed(42)       # Reproducible split across runs
random.shuffle(seed_examples)

split = int(len(seed_examples) * 0.85)
train_examples = seed_examples[:split]
holdout_examples = seed_examples[split:]  # Never enters training

After fine-tuning, run the model against the holdout and record task accuracy. Record the score after each retrain. A stable or improving score means the seed is anchoring correctly. A declining score is an early signal of drift before it compounds across cycles.

What to build next

• How to extract web data for AI training: broader training data collection patterns

• Zenrows Batch: managed concurrency and retries for large-scale collection

• Zenrows MCP server: web access for agents using the same retrieval layer

• Zenrows Fetch documentation: full parameter reference for mode, wait_for, css_extractor, and PDF handling

The retrieval gap in OpenAI's built-in web search

The OpenAI Agents SDK ships WebSearchTool as a hosted tool that handles discovery well. It falls short in two structural ways, both of which matter for production agents.

First, it returns indexed snapshots, not live pages. When a site allows crawling, the built-in search returns a snippet of OpenAI's cached copy. For anything where freshness matters: a price, a stock level, a rate, that copy lags the live page by however long the refresh cycle takes.

Second, JavaScript-rendered content isn't in the index at all. Many pages send an HTML shell and populate prices, inventory, and structured data via JavaScript at runtime. The crawler stores the shell. The data you want was never captured, and crawling more often doesn't change that.

The architectural response is not to replace built-in search but to add a second tool alongside it, one that retrieves the full, live contents of a specific URL when discovery alone isn't enough. This guide shows how to register Zenrows Fetch as a @function_tool in the same agent, so the model selects between them each turn from their descriptions alone.

All the code is on GitHub.

Architecture overview

The agent holds two tools in one tools list. The model reads each tool's docstring every turn and selects whichever one fits the task.

WebSearchTool runs on OpenAI's infrastructure. It takes a query, not a URL, and returns indexed content from public pages. It covers open-ended discovery, finding relevant documentation, and answering questions from public indexed content.

fetch_page_content is a Python function decorated with @function_tool. It takes a specific URL, calls the Zenrows Fetch API with JavaScript rendering and adaptive proxy handling enabled, and returns the current page content as Markdown. It covers the cases built-in search can't reach: known URLs, protected pages, and JavaScript-rendered content.

The routing between them requires no conditional logic and no orchestration code on your side. It lives in the docstrings.

Prerequisites

• Python 3.9 or later

• An OpenAI API key from your OpenAI developer dashboard

• A Zenrows API key from your Zenrows dashboard

Install dependencies:

python3 -m pip install openai-agents python-dotenv

Add both keys to a .env file:

OPENAI_API_KEY=your_openai_api_key_here
ZENROWS_API_KEY=your_zenrows_api_key_here

Add .env to .gitignore. Then load in your script:

from dotenv import load_dotenv

load_dotenv()

Set up a baseline agent with the built-in search

WebSearchTool is a hosted tool that runs on OpenAI's infrastructure through the Responses API. Enabling it requires one line.

import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool

load_dotenv()

agent = Agent(
    name="Product Researcher",
    instructions=(
        "You research products and web pages. "
        "Report exactly what you find, and state plainly when "
        "specific information is missing from your results."
    ),
    tools=[WebSearchTool()],
)

async def main():
    result = await Runner.run(
        agent,
        "Which laptops did Apple release most recently, and where can I buy them?",
    )
    print(result.final_output)

asyncio.run(main())

On broad discovery queries, this works. The problem appears on live or protected targets.

Where the built-in search stops

Asking the agent to fetch a live price from a JavaScript-heavy retailer illustrates both gaps:

# This demonstrates the retrieval gap on a JavaScript-rendered page
result = await Runner.run(
    agent,
    "What is the current price and stock status of "
    "https://www.amazon.com/dp/B0GR1JKMBV/ref=fs_a_mbt2_us1?th=1",
)
print(result.final_output)

The agent finds the page, returns the product title, and then states clearly what it couldn't access:

I visited the Amazon product page for ASIN B0GR1JKMBV...
The page displays: "To see product details, add this item to your cart."
Therefore I could not retrieve the current price or availability.

No error. The data simply wasn't in the index. This is the structural gap Zenrows fills.

Register Zenrows Fetch as a @function_tool

import os
import requests
from agents import function_tool

@function_tool
def fetch_page_content(url: str) -> str:
    """Fetch the full, current content of a specific web page as Markdown.

    Use this when you need the complete content of a known URL rather than
    a search result summary, such as live prices, stock levels, or data
    that loads via JavaScript after the page opens.

    Args:
        url: The full URL of the page to retrieve.
    """
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "url": url,
            "apikey": os.getenv("ZENROWS_API_KEY"),
            # mode=auto activates Adaptive Stealth Mode:
            # JavaScript rendering and premium proxy escalation
            # are enabled only when the target site requires them.
            # This keeps costs proportional to what each request needs.
            "mode": "auto",
            # response_type=markdown strips HTML markup
            # so the model reads clean text rather than raw HTML.
            "response_type": "markdown",
        },
        timeout=90,
    )

    # Return the error as a string so the agent can react to it
    # rather than receiving an unhandled exception.
    if response.status_code != 200:
        return f"Zenrows returned {response.status_code} for {url}"

    return response.text

Why this approach

The @function_tool decorator builds the tool schema from the function signature and type hints automatically. You don't write a JSON schema. The model uses the schema to construct valid calls.

mode=auto is the important parameter choice. It activates Adaptive Stealth Mode, which evaluates each target individually and enables JavaScript rendering and premium proxies only when the site requires them. An always-on premium proxy would work, but mode=auto keeps cost proportional to what each request actually needs.

The docstring is the routing mechanism. The model reads it every turn to decide whether the tool applies. The phrase "rather than a search result summary" is what distinguishes this tool from WebSearchTool in the model's decision-making. Without that phrase, the model treats both tools as interchangeable for any web-related task.

Run both tools in a single agent loop

import asyncio
import os
import requests
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool, function_tool

load_dotenv()

@function_tool
def fetch_page_content(url: str) -> str:
    """Fetch the full, current content of a specific web page as Markdown.

    Use this when you need the complete content of a known URL rather than
    a search result summary, such as live prices, stock levels, or data
    that loads via JavaScript after the page opens.

    Args:
        url: The full URL of the page to retrieve.
    """
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "url": url,
            "apikey": os.getenv("ZENROWS_API_KEY"),
            "mode": "auto",
            "response_type": "markdown",
        },
        timeout=90,
    )

    if response.status_code != 200:
        return f"Zenrows returned {response.status_code} for {url}"

    return response.text

# Both tools are registered in the same list.
# The model selects between them from their descriptions each turn.
agent = Agent(
    name="Web-Aware Researcher",
    instructions=(
        "You research products on the web. "
        "Use web search to find the product page URL. "
        "Then call fetch_page_content on that URL and report the price "
        "and stock status from the fetched page content. "
        "State which tool each figure came from."
    ),
    tools=[WebSearchTool(), fetch_page_content],
)

def print_trace(items):
    # Inspect result.new_items to confirm which tools the agent called
    for item in items:
        raw = getattr(item, "raw_item", None)
        name = getattr(raw, "name", None) or getattr(raw, "type", "")
        print(f"[{item.type}] {name}")

        if item.type == "tool_call_item" and name == "fetch_page_content":
            print("  args:", getattr(raw, "arguments", ""))

        if item.type == "tool_call_output_item":
            out = str(getattr(item, "output", ""))
            print(f"  output: {len(out)} chars")
            print(f"  body: {out[:500]}")

async def main():
    result = await Runner.run(
        agent,
        "Find the PriceOye page for the iPhone 17 Pro "
        "and report its current price and stock status.",
    )

    print_trace(result.new_items)
    print("\n---\n")
    print(result.final_output)

asyncio.run(main())

The trace shows the two-step pattern. Turn one is a web_search_call, discovery with no URL specified. Turn two calls fetch_page_content with the URL it found:

[tool_call_item] web_search_call
[message_output_item] message
[tool_call_item] fetch_page_content
  args: {"url":"https://priceoye.pk/mobiles/apple/apple-iphone-17-pro"}
[tool_call_output_item]

---

Here are the details for the iPhone 17 Pro from its PriceOye product page:

Current Price: Rs 471,999
Stock Status: Only 1 left in stock

Price and stock status are directly extracted from the fetched page content
(functions.fetch_page_content tool).

The routing decision the model made: web_search_call for discovery and fetch_page_content for retrieval matches the docstrings almost exactly. There's no conditional logic anywhere in the agent definition.

When to use each tool

WebSearchTool covers discovery: finding relevant pages, answering questions from indexed public content, identifying URLs for follow-up retrieval.

Zenrows covers retrieval: full contents of a specific URL, protected pages, JavaScript-rendered content, anything where freshness matters.

If you need specific fields rather than full-page Markdown, Zenrows Extract returns structured data so the model never has to parse Markdown for a price. For recurring high-volume retrieval across a catalog, Zenrows Batch runs those jobs without you managing concurrency.

The same @function_tool wrapper carries over to other agent frameworks. The guide on building a web research multi-agent system with AG2 and Zenrows shows how the pattern extends to multi-agent setups. Zenrows smolagents uses the same function under a @tool decorator.

Conclusion

The agent now reaches the web two ways. WebSearchTool handles discovery. Zenrows reads the full live page when you have a URL and need the current contents, including JavaScript-rendered data and anti-bot-defended targets. The model routes between them from the tool descriptions alone, with no orchestration code.

What to build next

• Zenrows MCP server: ready-made tools via the Responses API mcp tool type, with no Python wrapper to maintain

• OpenAI Agents SDK integration guide: full reference for MCP and @function_tool patterns in the SDK

• Zenrows Batch: scheduled retrieval across many URLs without managing concurrency

• Zenrows Extract: structured field extraction so the model doesn't parse raw Markdown

This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/scrape-2026-fifa-world-cup-data-python

Architecture overview

Building a reliable sports data pipeline means designing around the access pattern of each source. Every source behaves differently. The 2026 World Cup pipeline pulled from Sofascore for player statistics, FIFA for team metrics, and Wikipedia for tournament metadata. Each delivered its data through a different mechanism, and each needed a different extraction strategy.

The three access patterns broke down like this.

• A JavaScript-rendered single-page application exposing a JSON endpoint (Sofascore)

• A backend API protected by a per-session JWT issued during the browser session (FIFA)

• Server-rendered HTML with tabular data (Wikipedia)

Standard requests handles only the third reliably. The first returns an empty application shell. The second returns a 401 without the correct session token. This guide covers how we handled all three, how we normalized the outputs into a single dataset, and what the data revealed.

Full source: GitHub repository

Prerequisites

• Python 3.10 or later

• Zenrows account

• playwright, requests, beautifulsoup4, pandas, python-dotenv

pip install requests playwright beautifulsoup4 pandas python-dotenv
playwright install chromium

Retrieving Sofascore data from a JavaScript endpoint

Why this approach

Sofascore is a single-page application. The browser loads a JavaScript bundle, which calls an internal statistics API and renders the result into the DOM. A bare HTTP request retrieves the application shell with no data inside it.

Zenrows Fetch with mode=auto intercepts the API call through a rendered browser context and returns the JSON response directly. Because the target is a JSON endpoint rather than an HTML page, no parsing step is needed on our side.

import os
import time
from urllib.parse import urlencode

import requests

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")

class SofascoreClient:
    BASE_URL = "https://www.sofascore.com/api/v1"

    def _get(self, endpoint: str, params: dict | None = None):
        base_url = f"{self.BASE_URL}/{endpoint.lstrip('/')}"
        target_url = f"{base_url}?{urlencode(params)}" if params else base_url

        retries = 3
        for attempt in range(retries):
            try:
                response = requests.get(
                    "https://api.zenrows.com/v1/",
                    params={
                        "apikey": ZENROWS_API_KEY,
                        "url": target_url,
                        # Adaptive Stealth Mode: picks JS rendering or proxy escalation per request
                        "mode": "auto",
                    },
                    timeout=120,
                )
                response.raise_for_status()
                return response.json()
            except (
                requests.exceptions.ReadTimeout,
                requests.exceptions.ConnectionError,
                requests.exceptions.HTTPError,
            ) as e:
                print(f"Attempt {attempt + 1}/{retries} failed: {e}")
                if attempt == retries - 1:
                    raise
                wait = 2 ** attempt  # exponential backoff
                time.sleep(wait)

    def fetch_statistics(
        self, group, order, page=1, limit=20, accumulation="total"
    ):
        endpoint = f"unique-tournament/{TOURNAMENT_ID}/season/{SEASON_ID}/statistics"
        params = {
            "group": group,
            "order": order,
            "page": page,
            "limit": limit,
            "accumulation": accumulation,
        }
        return self._get(endpoint, params=params)

Retrieving FIFA data from an authenticated API

Why this approach

FIFA's statistics page calls a backend API with an authorization header carrying a JWT bearer token. The token is issued per browsing session. Every visitor gets a temporary credential and the front end attaches it to all subsequent API requests. Querying the API endpoint directly without that token returns a 401.

Fetch alone cannot acquire a session token because that requires executing the full browser session authentication flow. Zenrows Browser Sessions with Playwright follows the same flow a normal browser does, maintains the token across requests, and lets us intercept API responses as the page navigates through each statistics tab.

import os
from playwright.async_api import async_playwright

class FIFAClient:
    def __init__(self):
        self.api_key = os.getenv("ZENROWS_API_KEY")
        # Connect to a Zenrows-managed browser over CDP
        self.connection_url = f"wss://browser.zenrows.com?apikey={self.api_key}"
        self.responses = {}

    async def _handle_response(self, response):
        # Filter to only the FIFA statistics API responses
        if "gameday-prod.fifa.mangodev.co.uk" not in response.url:
            return
        # Store the JSON response keyed by URL for later normalization
        self.responses[response.url] = await response.json()

    async def fetch_all(self):
        async with async_playwright() as p:
            browser = await p.chromium.connect_over_cdp(self.connection_url)
            context = browser.contexts[0]
            page = await context.new_page()

            # Intercept all network responses before navigating
            page.on("response", self._handle_response)
            await page.goto(self.URL)
            await self._remove_cookie_banner(page)

            # Click through each statistics tab to trigger the API calls
            for tab in self.TABS:
                await self._click_tab(page, tab)

            self._save_files()

Retrieving Wikipedia tournament metadata

Why this approach

Wikipedia is server-rendered. All content is in the HTML the server returns. Fetch retrieved it without additional capabilities. BeautifulSoup located the correct table by checking for the presence of specific column headers rather than relying on a fixed table index, which breaks when Wikipedia's page structure changes.

import re
from bs4 import BeautifulSoup

def normalize(self):
    with open(self.INPUT_FILE, encoding="utf-8") as f:
        soup = BeautifulSoup(f, "html.parser")

    target_table = None

    # Locate by headers rather than by table index — more resilient to page edits
    for table in soup.find_all("table", class_="wikitable"):
        headers = [th.get_text(" ", strip=True) for th in table.find_all("th")]
        if (
            "Year" in headers
            and "Host" in headers
            and "Cities" in headers
            and "Stadiums" in headers
        ):
            target_table = table
            break

    if target_table is None:
        raise ValueError("Could not locate the World Cup stadium table.")

    records = []
    for row in target_table.find("tbody").find_all("tr")[1:]:
        cells = row.find_all(["th", "td"])
        if len(cells) < 4:
            continue
        year_match = re.search(r"\d{4}", cells[0].get_text())
        if not year_match:
            continue
        records.append({
            "year": int(year_match.group()),
            "hosts": [t.strip() for t in cells[1].stripped_strings if t.strip()],
            "cities": int(re.search(r"\d+", cells[2].get_text()).group()),
            "stadiums": int(re.search(r"\d+", cells[3].get_text()).group()),
        })

Normalization

Each source's raw output went through its own normalization layer before merging into a single dataset. Sofascore returns the same player across multiple statistic groups, so we deduplicated by player ID rather than by name.

import json

class SofascoreNormalizer:
    def normalize_player(self, record):
        player = record.get("player", {})
        team = record.get("team", {})
        # Flatten stats into a single dict, excluding the top-level player and team keys
        stats = {k: v for k, v in record.items() if k not in {"player", "team", "rating"}}
        return {
            "player_id": player.get("id"),
            "player": player.get("name"),
            "player_slug": player.get("slug"),
            "team_id": team.get("id"),
            "team": team.get("name"),
            "rating": record.get("rating"),
            "stats": stats,
        }

    def normalize_dataset(self, dataset):
        with open(f"data/raw/sofascore/{dataset}.json") as f:
            data = json.load(f)

        players = {}
        for record in data:
            normalized = self.normalize_player(record)
            player_id = normalized["player_id"]
            if player_id is None:
                continue
            # Last write wins per player ID — keeps the most recently processed stat group
            players[player_id] = normalized

        return list(players.values())

We loaded each normalized dataset into a Pandas DataFrame for a consistent interface across sources before calculating derived metrics and producing charts.

What the data revealed

Fifteen findings came out of the combined dataset. Several challenged what we expected going in. The ones that stood out most were these.

• Ousmane Dembélé scored six goals from just over two expected goals, close to three times his xG, the most efficient conversion among the top strikers

• The Golden Glove went to Unai Simón, who doesn't appear in the top ten goalkeepers by rating in our dataset. Orlando Gill, whose Paraguay went out in the Round of 16, tops our list and led all goalkeepers in saves

• Jude Bellingham, Rodri, and Aurélien Tchouaméni ranked above all centre-backs in combined defensive actions

• Japan recorded the highest shot conversion rate in the tournament while taking the lowest share of shots from inside the box. Only 50% of their shots came from inside the box compared to over 70% for most other top-converting teams

• Michael Olise set a new World Cup record with seven assists and led the tournament in big chances created while holding one of the highest passing accuracies

• France, England, and Argentina applied the most defensive pressure and had among the slowest ball recovery times. Türkiye applied far fewer pressures and recovered the ball fastest

• Argentina reached the final with the most bookings in the tournament and no VAR interventions against them

• Spain recorded the most defensive line breaks while holding around 90% passing accuracy. Most teams trade accuracy for line-breaking volume. Spain didn't.

• The 2026 tournament was hosted by three countries and used 16 stadiums, fewer than Spain used alone in 1982

The full findings with visualizations are in the original article.

What to build next

• Zenrows Browser Sessions for authenticated browser workflows

• Zenrows Batch for managed concurrency across large URL lists

• Web data for LLM fine-tuning for taking scraped data into training pipelines

• Zenrows Fetch documentation covering mode, wait_for, and JSON response handling

Why the fetch layer determines everything downstream

AG2's group chat architecture is built around a shared history. Every message, including every tool response, enters a shared context window that all agents read on their turn. This is what makes multi-agent pipelines powerful: each agent builds on what the agents before it produced.

It's also what makes a broken fetch expensive. When the researcher can't reach a URL, that failure enters the shared history as a message. The analyst reads it, has nothing to work with, and says so. The critic reads both, validates that neither produced anything, and terminates. Every agent in the pipeline spent tokens on content that never existed.

The fix is architectural. Register a fetch tool at the researcher level that actually works, one that handles JavaScript-rendered and anti-bot-protected pages, so clean content enters the history from the first turn. This guide does that with Zenrows Fetch, compares token usage across both configurations, and explains the cost dynamics in a multi-agent context.

AG2 is the actively maintained community fork of Microsoft's AutoGen, with native MCP client support added in v0.12. This guide uses the OpenAI client.

All the code is on GitHub.

Architecture overview

The pipeline has four agents in one group chat.

user_proxy is the execution agent. It runs any registered function the LLM requests and terminates the chat when an agent message contains TERMINATE.

researcher holds the fetch tool. When given a URL, it calls fetch_page_content, receives clean Markdown from Zenrows, and passes a summary to the analyst.

analyst reads the researcher's summary and extracts structured fields — product names, prices, or whatever the task specifies. It doesn't call any tools.

critic validates the analyst's output against the researcher's original content, flags gaps or inconsistencies, and terminates the pipeline when the output checks out.

The key design decision is where the tool is registered. register_for_llm(researcher) exposes the schema to the researcher so the model knows to call it. register_for_execution(user_proxy) tells AG2 which agent runs the function — in a group chat, that's always the UserProxyAgent. This split is required; registering both on the same agent doesn't work.

Prerequisites

• Python 3.10 or later

• AG2 v0.12.2

• An OpenAI API key

• A Zenrows API key

Install AG2 with its OpenAI client:

pip install "ag2[openai]==0.12.2"

Two things before you run anything. The package name and the import name differ: install as ag2, import as autogen. Using import ag2 raises a ModuleNotFoundError in v0.12.2; AG2 is deprecating the classic framework at v1.0, at which point import ag2 becomes correct. AG2 also doesn't bundle the OpenAI client, which is why [openai] is required.

Load keys from .env:

import os
from dotenv import load_dotenv

import autogen
from autogen import AssistantAgent, GroupChat, GroupChatManager, UserProxyAgent

load_dotenv()

llm_config = {
    "config_list": [
        {
            "model": "gpt-4o-mini",
            "api_key": os.getenv("OPENAI_API_KEY"),
        }
    ],
    # temperature 0 for deterministic output across runs
    "temperature": 0,
}

Set up an AG2 group chat

Define four agents, wire them into a group chat, and run a baseline to see the pipeline with no fetch tool:

# user_proxy executes tools and terminates on TERMINATE
user_proxy = UserProxyAgent(
    name="user_proxy",
    human_input_mode="NEVER",
    code_execution_config=False,
    is_termination_msg=lambda msg: "TERMINATE" in (msg.get("content") or ""),
)

# researcher fetches and summarizes — no tool registered yet
researcher = AssistantAgent(
    name="researcher",
    system_message=(
        "You are a research agent. When given a URL, fetch its web content and "
        "summarize what you find. Be factual and note anything that looks "
        "incomplete or missing from the page. Pass your summary to the analyst."
    ),
    llm_config=llm_config,
)

# analyst extracts structured fields from researcher output
analyst = AssistantAgent(
    name="analyst",
    system_message=(
        "You are a data analyst. Extract structured fields, such as plan names "
        "and prices, from the content the researcher provides. Present the "
        "result as a clean list. If the researcher's content has no usable "
        "data, say so explicitly instead of guessing."
    ),
    llm_config=llm_config,
)

# critic validates and terminates
critic = AssistantAgent(
    name="critic",
    system_message=(
        "You are a critic. Validate the analyst's structured output against "
        "the researcher's original content. Flag any gaps, missing fields, or "
        "inconsistencies. When the output checks out or no further progress is "
        "possible, say so and end your message with TERMINATE."
    ),
    llm_config=llm_config,
)

group_chat = GroupChat(
    agents=[user_proxy, researcher, analyst, critic],
    messages=[],
    max_round=6,
)

manager = GroupChatManager(groupchat=group_chat, llm_config=llm_config)

chat_result = user_proxy.initiate_chat(
    manager,
    message=(
        "Fetch the product listing from this page and extract the "
        "product name and price: https://www.walmart.com/ip/AirPods-Pro-3/17835006350"
    ),
)

For accurate token counts, call autogen.gather_usage_summary() on the full agent list. chat_result.cost always reports zero for group chats:

# gather_usage_summary is the only reliable source for group chat token counts
usage_summary = autogen.gather_usage_summary(
    [user_proxy, researcher, analyst, critic, manager]
)
print(usage_summary)

The baseline output shows the failure propagating:

researcher: I'm unable to access external URLs directly to fetch content.
analyst: I cannot access external URLs or fetch content from them.
critic: Both parties acknowledge the inability to access external URLs. TERMINATE

Total tokens: 990 | Cost: $0.000252

Register Zenrows Fetch as a typed AG2 tool

AG2 builds the tool schema from type hints. Annotated[type, description] on every input parameter is required. Without it, AG2 can't build the schema, and registration fails silently or raises a validation error.

import requests
from typing import Annotated
from autogen.tools import Tool

ZENROWS_API_KEY = os.getenv("ZENROWS_API_KEY")
ZENROWS_ENDPOINT = "https://api.zenrows.com/v1/"

def fetch_page_content(
    url: Annotated[str, "The target URL to fetch through Zenrows using adaptive stealth mode."],
) -> str:
    """Fetch a URL through Zenrows Fetch and return clean Markdown."""
    params = {
        "url": url,
        "apikey": ZENROWS_API_KEY,
        # mode=auto activates Adaptive Stealth Mode:
        # Zenrows evaluates the target and enables JS rendering
        # or premium proxies only when the site requires them.
        # This keeps credit spend proportional to what each request needs.
        "mode": "auto",
        # response_type=markdown returns clean text rather than raw HTML,
        # keeping token count proportional to content rather than markup.
        "response_type": "markdown",
    }
    response = requests.get(ZENROWS_ENDPOINT, params=params, timeout=60)
    response.raise_for_status()
    return response.text

Why this approach

mode=auto is Adaptive Stealth Mode. Rather than always enabling JavaScript rendering and premium proxies, which adds cost on pages that don't need them. Zenrows evaluates each target individually. A basic static page costs 1 credit. JavaScript rendering costs 5, premium proxies cost 10, and a heavily protected site needing both costs 25. You're billed once, for the configuration that succeeds.

response_type=markdown matters specifically in a multi-agent context. Raw HTML inflates the shared history, which inflates every subsequent agent turn that reads it. Markdown gives the model what it needs without the markup overhead.

Wrap the function and register it on the correct agents:

fetch_tool = Tool(
    name="fetch_page_content",
    description=(
        "Fetch a URL through Zenrows and return clean Markdown content from "
        "JavaScript-rendered and anti-bot-protected pages."
    ),
    func_or_tool=fetch_page_content,
)

# register_for_llm exposes the schema to the researcher
fetch_tool.register_for_llm(researcher)
# register_for_execution tells AG2 which agent runs the function
# — always user_proxy in a group chat
fetch_tool.register_for_execution(user_proxy)

Update the researcher's system message to name the tool:

researcher = AssistantAgent(
    name="researcher",
    system_message=(
        "You are a research agent. When given a URL, use the fetch_page_content "
        "tool to retrieve its web content, then summarize what you find. Be "
        "factual and note anything that looks incomplete or missing from the "
        "page. Pass your summary to the analyst."
    ),
    llm_config=llm_config,
)

Run the pipeline and compare token spend

With Zenrows registered, the researcher calls fetch_page_content. Zenrows retrieves the page through Adaptive Stealth Mode. Clean Markdown enters the shared history. The analyst extracts product data. The critic validates and terminates.

researcher: ***** Suggested tool call: fetch_page_content *****
Arguments: {"url":"https://www.walmart.com/ip/AirPods-Pro-3/17835006350"}
>>>>>>>> EXECUTING FUNCTION fetch_page_content...

[Markdown from Walmart page enters shared history]

analyst:
- Product Name: Apple AirPods Pro 3
- Price: $189.99 (was $249.00, you save $59.01)

critic: The analyst's output is consistent with the researcher's content.
No gaps or missing fields. TERMINATE

Total tokens: 7,236 | Cost: $0.001147

The working run spends more tokens because the Markdown from a fully fetched page is far larger than the researcher's brief refusal in the baseline. That token spend is proportional to content. The baseline was cheaper because it retrieved nothing.

Both runs terminate in approximately the same number of turns, so the round count isn't the cost variable. What changes is whether there's real content in the history.

Keeping costs predictable in production

Content volume in the shared history is the primary cost driver, not the number of turns. Every agent turn re-reads and forwards the full history, so page size multiplies across the pipeline.

max_round sets a ceiling on your worst case. Start conservative and raise it only when the pipeline consistently needs more turns to finish naturally. Every failed or empty fetch that enters the history pushes the round count higher without producing anything useful, so reliable retrieval is the most direct way to keep max_round from becoming your main cost control.

For pipelines that fetch the same URL more than once per session, caching the Zenrows response after the first call cuts both token spend and credit usage. For pipelines that research several URLs in one run, Zenrows Batch takes the full list as one managed job and handles concurrency and retries.

Conclusion

AG2's shared history is the architectural fact that makes the fetch layer critical. A failed fetch doesn't stay isolated. It enters the history and every agent after reads and forwards it, spending tokens on turns that had nothing real to work with.

Registering Zenrows as a typed fetch tool with mode=auto addresses that at the source. Zenrows decides what each target needs, whether that's plain rendering, JavaScript execution, or premium proxies, without you configuring it per site. Clean Markdown enters the shared history on the first call, so the analyst and critic get real content and the pipeline terminates naturally.

What to build next

• Zenrows MCP server: AG2 v0.12 has native MCP client support if you prefer ready-made tools over a registered function

• OpenAI Agents SDK + Zenrows: the same Zenrows fetch pattern in a single-agent loop with @function_tool

• Zenrows smolagents guide: the same wrapper under a @tool decorator in a different framework

• Zenrows Batch: scheduled or high-volume retrieval across many URLs without managing fetch queues

• Adaptive Stealth Mode pricing table: full breakdown of credit costs by configuration

Why Cursor's built-in web access breaks on the pages that matter most

Cursor's built-in web access reads from cached or pre-rendered snapshots. That approach works on stable, public, text-heavy pages. It breaks in three specific situations that agent workflows hit constantly.

JavaScript-rendered pages. Many pages ship an HTML shell and populate their actual content like product grids, pricing tables, and API responses via JavaScript at runtime. Cursor fetches the shell. The data was never in it.

Bot-protected sites. Pages behind Cloudflare, DataDome, or similar anti-bot systems detect the request, return a challenge interstitial, and block the data behind it. No error is surfaced in the agent response. Your agent returns the interstitial, not the page.

Recently updated content. Documentation, pricing, and API references can change faster than a cache refreshes. The snapshot Cursor reads may be hours or days old with nothing marking it stale.

The failure mode is the same in all three cases: your agent returns something that looks like a valid response, just not what the page actually contains. In an agent workflow, that silent wrong answer propagates downstream.

Zenrows MCP closes that gap at the configuration level. You add one JSON file, restart Cursor, and the agent routes web requests through Zenrows' web data infrastructure with a 99.93% success rate across supported targets, including JavaScript-rendered and anti-bot-protected pages.

Architecture overview

The Zenrows MCP server runs as a local process spawned by Cursor via npx. When the agent decides to fetch a URL, it calls the Zenrows scrape tool instead of Cursor's built-in browsing. The tool sends the request to Zenrows, which evaluates what the target page needs, whether that's a plain HTTP request, JavaScript rendering, premium proxy routing, or a combination, and returns clean Markdown, JSON, or HTML depending on what you ask for.

The mcp.json config is the only integration surface. No code changes or SDK setup. The agent prompt tells Cursor which tool to use.

Prerequisites

• Node.js v18 or later. Run node --version to confirm

• A Zenrows API key from app.zenrows.com

• Cursor installed

Add Zenrows MCP to Cursor

Step 1: Create the config file

The mcp.json path depends on your operating system:

On macOS and Linux, create the file in one command:

mkdir -p ~/.cursor && cat > ~/.cursor/mcp.json << 'EOF'
{
  "mcpServers": {
    "zenrows": {
      "command": "npx",
      "args": ["-y", "@zenrows/mcp"],
      "env": {
        "ZENROWS_API_KEY": "your_api_key_here"
      }
    }
  }
}
EOF

On Windows, open %USERPROFILE%\.cursor\ in File Explorer, create mcp.json, and paste the same JSON.

Step 2: Validate the JSON before opening Cursor

# A trailing comma or missing bracket stops the config loading silently.
# Run this before opening Cursor so syntax errors surface early.
node -e "JSON.parse(require('fs').readFileSync(require('os').homedir()+'/.cursor/mcp.json','utf8')); console.log('JSON valid')"

JSON valid means the file parsed correctly. Any output other than that is a syntax error in the config.

Step 3: Confirm the connection in Cursor

Open Cursor → gear icon (bottom left) → search "MCP" → Tools & MCPs → Home MCP Servers. The zenrows entry should show a green status dot with 37 tools and 3 prompts enabled.

On macOS, use ⌘Q rather than closing the window when restarting. The MCP server process reads your API key on spawn. Closing the window doesn't trigger a respawn, so the key isn't re-read.

Workflow 1: Fetch live data from a protected site

The goal is clean Markdown from a page that sits behind an anti-bot challenge which is the category Cursor's built-in browsing can't reach.

Paste this into Cursor's agent:

Use the Zenrows MCP scrape tool to fetch https://www.scrapingcourse.com/antibot-challenge
and return the content as clean Markdown.

Do not use your built-in web browsing. Only use the Zenrows scrape tool.

Cursor's built-in browsing returns the challenge interstitial for this URL. Zenrows identifies the challenge, activates the appropriate bypass configuration, and returns the page behind it as Markdown.

Why this approach

The scrape tool prompt tells the agent to use Zenrows explicitly and not to fall back to built-in browsing. Without that instruction, Cursor may try its built-in access first and return the challenge page as a valid response. The explicit tool name in the prompt is what enforces the routing.

Workflow 2: Extract structured data from a JavaScript-rendered page

Use the Zenrows MCP scrape tool to fetch https://www.scrapingcourse.com/javascript-rendering
and extract a structured JSON object containing the first 5 products with their name and price.
Do not use your built-in web browsing.

The products on this page load dynamically after JavaScript runs. A plain HTTP request returns the HTML shell with an empty product grid. Zenrows renders the page in a full browser environment before returning it, so the #product-grid container is populated when the agent parses it.

The result is structured JSON the agent can pass directly into a downstream task. Workflow 3 does exactly that.

Workflow 3: Generate TypeScript from live protected data

This workflow demonstrates the two-level fetch pattern that most real scraping jobs require: a category listing that carries some fields, and individual product pages that carry the rest.

On Home Depot, the category listing returns product names and prices. SKU and model number exist only on each product's own page. The agent fetches the listing, then fetches each product page, then combines both data levels into a unified structure, and generates a TypeScript function that replicates that workflow programmatically.

Use the Zenrows MCP scrape tool to fetch
https://www.homedepot.com/b/Tools-Woodworking-Tools/N-5yc1vZc2gv?catStyle=ShowProducts

and return the first 5 products as structured JSON with name and price. Then fetch each
product's individual page and add its SKU and model number.
Set proxy country to US. Do not use your built-in web browsing.

Then write a TypeScript function that fetches and displays product details for a given
product name.

proxy_country: US routes the request through a US IP address. A US retailer returns its normal catalogue and pricing to US IPs. Without this, the response varies by origin country, and product availability or pricing may differ from what US-based users see.

The agent combines both page levels into this structure:

{
  "source_url": "https://www.homedepot.com/b/Tools-Woodworking-Tools/N-5yc1vZc2gv?catStyle=ShowProducts",
  "products": [
    {
      "name": "Gorilla 4 fl. oz. Wood Glue",
      "price": "$3.98",
      "sku": "1003827526",
      "model_number": "62020"
    },
    {
      "name": "DEWALT 20V MAX XR Cordless Brushless Fixed Base Compact Router (Tool Only)",
      "price": "$249.00",
      "sku": "1004095707",
      "model_number": "DCW600B"
    },
    {
      "name": "Titebond 8 oz. Original Wood Glue",
      "price": "$3.69",
      "sku": "676828",
      "model_number": "5063"
    }
  ]
}

Why this approach

Each SKU and model number required a separate Zenrows fetch. That means five individual product page requests in addition to the category listing and six Zenrows calls total. Each returned real content, which is what let the agent map the structure correctly and generate code that handles the two-level case rather than assuming all fields come from one page.

The generated TypeScript function has this signature:

export async function fetchAndDisplayProductDetails(
  productName: string,
  apiKey = process.env.ZENROWS_API_KEY ?? ""
): Promise<HomeDepotProductDetails | null>

It scrapes the category listing, finds a product by partial name match, fetches that product's page for SKU and model number, and prints the combined result. Run it with:

export ZENROWS_API_KEY="your_api_key_here"
npm run build
npm run homedepot -- "Gorilla 4 fl. oz. Wood Glue"

Results are saved to data/homedepot-products.json.

Zenrows vs Firecrawl in Cursor

Both integrate with Cursor. Firecrawl is a fast, lightweight option for public pages where discovery and clean Markdown matter most. Zenrows is the better fit when the pages are protected, JavaScript-rendered, or part of a production workflow that runs on a schedule.

Full comparison: Zenrows vs Firecrawl

Debugging

Zenrows MCP isn't appearing after I added the config

Validate your mcp.json with the command in Step 2, then fully restart Cursor with ⌘Q on macOS. The server process only reads the config on spawn — closing the window doesn't trigger a respawn.

401 error in the agent response

The server is connected but the API key it's passing is invalid. Check that ZENROWS_API_KEY in mcp.json matches a live key in your Zenrows account and isn't still the your_api_key_here placeholder. Save, quit Cursor fully, and reopen.

EACCES error on macOS

Restore npm cache ownership, then restart Cursor:

sudo chown -R $(id -u):$(id -g) ~/.npm

Prefer the hosted MCP server over running npx locally?

Use https://mcp.zenrows.com/mcp with your API key as a Bearer token in Cursor's remote MCP configuration.

What to build next

• Zenrows MCP overview: full tool and prompt reference

• Cursor integration docs: reference configuration for the integration

• Zenrows MCP setup for Claude Desktop and other agents: the same config pattern across MCP-compatible clients

• OpenAI Agents SDK + Zenrows: register Zenrows as a @function_tool in code instead of running an MCP server

• AG2 + Zenrows multi-agent pipeline: the same Zenrows fetch layer in a researcher–analyst–critic group chat