Web data for LLM fine-tuning: a practical 2026 guide
Web data for LLM fine-tuning, a practical 2026 guide
This article was originally published on the Zenrows blog. Read the original here: https://www.zenrows.com/blog/web-data-llm-fine-tuning
Architecture overview
LLM fine-tuning pipelines that rely entirely on synthetic data are structurally unstable. The generator produces examples using the teacher model's patterns; the judge filters against those same patterns. Each retrain cycle amplifies the biases from the previous one. By the third or fourth cycle, the model has learned to reproduce domain text rather than to reflect domain knowledge.
The intervention is a real-data seed, a small, verified set of examples collected from authoritative sources before any synthetic expansion begins. The seed does two things. It gives the generator concrete domain examples to expand from rather than its own patterns. It gives the judge something to filter against that is not its own output.
Fine-tuning amplifies signal differently than pretraining. A single mislabeled extraction in a dataset of a few hundred rows represents a measurable percentage of the training signal, and the model takes it as a rule. This guide covers collecting that seed from sources that block standard crawlers, formatting it correctly, validating each row, and measuring whether the fine-tune held.
What you'll implement:
Protected web data collection using Zenrows Fetch with
mode=autoPDF retrieval and structured chunking
Provenance tracking per source document
Concurrent batch collection with rate limit management
Three fine-tuning data formats: instruction-output pairs, preference pairs, domain continuation
Seed validation with length, schema, and deduplication checks
Holdout split and accuracy measurement
All code: GitHub repository
Why authoritative sources are the hard part
Authoritative domain sources like regulatory filings, clinical guidance, technical documentation, expert-reviewed publications, are almost always behind access controls. They block standard crawlers at the TLS handshake level. A plain requests.get() exposes the Python HTTP client fingerprint, the target identifies it as automated traffic, and the response is a challenge page or an empty shell.
This is not a scraping problem that can be solved with headers or user-agent rotation alone. JavaScript-rendered pages require a headless browser context. Protected sources require IP rotation and fingerprint masking at the session level. The collection layer has to handle both, per request, without per-site configuration.
Prerequisites
Python 3.9 or later
pypdffor PDF text extraction
python3 -m pip install requests python-dotenv pypdf jsonschema
# .env — add to .gitignore before committing
ZENROWS_API_KEY=your_zenrows_api_key_here
Collecting from protected sources
Zenrows Fetch handles JavaScript rendering, access control, and Markdown conversion in one API call. mode=auto is Adaptive Stealth Mode. Zenrows selects the retrieval configuration per target based on how that target responds, alternating between direct fetch, headless browser, and proxy escalation as needed.
import os
import requests
from dotenv import load_dotenv
load_dotenv()
# Load at module level — one environment read, not one per request
apikey = os.getenv("ZENROWS_API_KEY")
params = {
"url": "https://www.scrapingcourse.com/ecommerce/",
"apikey": apikey,
"mode": "auto", # Adaptive Stealth Mode picks retrieval strategy per target
"response_type": "markdown", # Returns clean Markdown, not raw HTML
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)
The Markdown output strips most navigation, footer, and boilerplate. That reduced noise is what makes it a better training input than raw HTML.
Handling PDFs and non-HTML documents
The most authoritative domain sources are often PDFs. response_type=pdf returns raw bytes; js_render=true is required and must be included explicitly.
params = {
"url": pdf_url,
"apikey": apikey,
"js_render": "true", # Required for PDF response type
"response_type": "pdf", # Returns raw bytes — extract text separately
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
with open("source-document.pdf", "wb") as f:
f.write(response.content)
Why this approach: chunk on structure, not character count
A regulatory filing can run to tens of thousands of tokens. Fixed-character chunking cuts mid-sentence, mid-argument, and mid-table. Chunking on the document's own page boundaries preserves context. Each chunk stays internally coherent, which is what makes it a usable training example.
from pypdf import PdfReader
import io
reader = PdfReader(io.BytesIO(response.content))
def chunk_by_page(reader, max_chars=6000):
chunks = []
for page in reader.pages:
text = page.extract_text()
if not text or not text.strip():
continue # skip blank or image-only pages
if len(text) <= max_chars:
chunks.append(text)
else:
# only split within a page when the page itself exceeds the limit
for i in range(0, len(text), max_chars):
chunks.append(text[i:i + max_chars])
return chunks
chunks = chunk_by_page(reader)
print(f"Produced {len(chunks)} chunks from {len(reader.pages)} pages")
Provenance tracking
Every row in an authoritative seed needs a traceable source. A metadata record per document keeps the dataset auditable, covers legal requirements for commercial use, and makes it possible to re-retrieve and re-verify specific rows later.
from datetime import date
# Fetch the license from the landing page using Zenrows CSS extractor
license_params = {
"url": landing_page_url,
"apikey": apikey,
"js_render": "true",
"css_extractor": '{"license_url": "a[href*=\\"creativecommons.org\\"] @href"}',
}
license_resp = requests.get("https://api.zenrows.com/v1/", params=license_params)
license_data = license_resp.json()
source = {
"url": pdf_url,
"retrieved": date.today().isoformat(),
"license": license_data.get("license_url", "unknown"),
}
Batch collection with rate limit management
Match concurrency to your plan limit. Spreading requests across workers with a short pause keeps the pipeline inside your quota without throttling from the target side.
import time
from concurrent.futures import ThreadPoolExecutor
MAX_CONCURRENT = 50 # Verify against your Zenrows plan
def fetch(url):
params = {
"url": url,
"apikey": apikey,
"response_type": "markdown",
"mode": "auto",
"wait": 3000, # Wait for dynamic content to settle before returning
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
time.sleep(0.5) # Pace requests within each worker thread
if response.status_code != 200:
return {"url": url, "status": response.status_code, "content": None}
return {"url": url, "status": response.status_code, "content": response.text}
with ThreadPoolExecutor(max_workers=MAX_CONCURRENT) as executor:
results = list(executor.map(fetch, urls))
For larger runs, Zenrows Batch manages concurrency, retries, and scheduling across large URL lists without a local orchestration layer.
Three fine-tuning data formats
The right format depends on the training method, not on what is easiest to produce.
Instruction-output pairs
Standard for supervised fine-tuning. Each example pairs the page Markdown (input) with the correct structured extraction (output). The model learns what to extract and how to format it.
import json
def to_training_example(instruction, markdown_content, extracted_json):
return {
"messages": [
{"role": "system", "content": instruction},
# The Markdown you pulled from the protected source
{"role": "user", "content": markdown_content},
# The verified structured output for this input
{"role": "assistant", "content": json.dumps(extracted_json)},
]
}
instruction = "Extract the product name, price, and availability as JSON."
with open("training_data.jsonl", "w") as f:
for markdown_content, extracted_json in seed_examples:
example = to_training_example(instruction, markdown_content, extracted_json)
f.write(json.dumps(example) + "\n") # One JSON object per line — JSONL format
Preference pairs
Used in DPO and RLHF. Two candidate outputs for the same input, one marked preferred. Correct when the model already produces reasonable outputs and the goal is distributional refinement rather than teaching the task from scratch.
def to_preference_pair(prompt, chosen, rejected):
return {
"prompt": prompt,
"chosen": chosen, # The preferred output
"rejected": rejected, # The not-preferred output to the same prompt
}
Domain text continuation
Raw domain text with no input-output structure. Useful for domain vocabulary and style familiarization, weak for extraction tasks because the model is never shown what should be extracted.
def to_continuation_example(markdown_content):
# One field — the model learns by predicting what comes next
return {"text": markdown_content}
For extraction tasks, instruction-output pairs are the correct choice. Continuation is a pretraining-style objective applied at fine-tuning scale; it improves fluency, not accuracy.
Seed validation
A row that fails any of these three checks does not enter the training set.
import hashlib
from jsonschema import validate, ValidationError
schema = {
"type": "object",
"properties": {
"name": {"type": "string", "minLength": 1},
"price": {"type": "string", "minLength": 1},
},
"required": ["name", "price"],
}
seen = set()
def keep(markdown, extracted):
# Length: pages that rendered to almost nothing carry no useful signal
if len(markdown.strip()) < 200:
return False
try:
# Schema: catches wrong types, empty values, missing required fields
validate(instance=extracted, schema=schema)
except ValidationError:
return False
# Deduplication: normalise whitespace before hashing to catch formatting-only duplicates
h = hashlib.sha256(" ".join(markdown.lower().split()).encode()).hexdigest()
if h in seen:
return False
seen.add(h)
return True
clean_examples = [(url, md, js) for url, md, js in seed_examples if keep(md, js)]
Measuring whether the seed held
Hold out 10 to 20 percent of the real seed before any synthetic expansion and keep those examples permanently out of training and out of the synthetic generator.
import random
random.seed(42) # Reproducible split across runs
random.shuffle(seed_examples)
split = int(len(seed_examples) * 0.85)
train_examples = seed_examples[:split]
holdout_examples = seed_examples[split:] # Never enters training
After fine-tuning, run the model against the holdout and record task accuracy. Record the score after each retrain. A stable or improving score means the seed is anchoring correctly. A declining score is an early signal of drift before it compounds across cycles.
What to build next
How to extract web data for AI training: broader training data collection patterns
Zenrows Batch: managed concurrency and retries for large-scale collection
Zenrows MCP server: web access for agents using the same retrieval layer
Zenrows Fetch documentation: full parameter reference for
mode,wait_for,css_extractor, and PDF handling
