Filtering Web Crawls for AI Training Data with URL Categories
Raw web crawls are full of adult pages, scams, parked domains and spam. Removing them page by page is slow, costly and never quite complete. Domain categories let AI data teams drop whole unwanted hosts in one cheap step, tag the rest by subject, and spend heavy filters where they matter.
Why filter crawl data at the domain level first
Most quality and safety filters work on single documents. They are necessary, but running them on everything is expensive. A domain lookup is one join on a table, and it settles the fate of thousands of pages at once.
Cheap
One lookup per host rather than one model call per page. Billions of URLs reduce to millions of domains.
Thorough
Removes every page of a bad host, including pages your page-level filter would have missed.
Consistent
The same rule applies to every crawl snapshot, so datasets built months apart stay comparable.
Explainable
"We removed these 40 categories" is easy to document, review and report in a data card.
Which categories to remove, down-weight or tag
There is no universal rule set; it depends on what the model is for, who will use it and in which markets. The table gives a sensible default for a general-purpose text model, using the web filtering categories in our data.
| Category | Why it matters for training | Default action |
|---|---|---|
| Adult | Explicit content, often with low-quality template text and spam | Remove |
| Malware / Phishing / Suspicious | Deceptive text, scam scripts and malicious links | Remove |
| Parked / For-Sale Domains | Identical template pages repeated across many domains | Remove |
| Hate/Discrimination | Hateful material the model should not learn to reproduce | Remove, or keep a reviewed slice for safety classifiers |
| Illegal Drugs | Sales and promotion pages; harm-reduction text lives elsewhere | Remove |
| File Sharing/P2P | Pirated material and pages with unclear rights | Remove |
| Gambling | Promotional and affiliate text with little general value | Down-weight |
| Advertising/Marketing | Sales copy and landing pages that skew style toward ads | Down-weight |
| URL Shorteners | Redirect pages with no content of their own | Remove |
| Violence; Weapons | Mix of news, history and harmful material | Tag and filter by page |
| Dating; Lingerie/Swimsuits/Intimate Apparel | Often thin or suggestive, sometimes borderline adult | Down-weight |
| Forums / Message Boards; Community Sites | Valuable dialogue, but personal data and toxicity risk | Tag and apply page filters |
| Education/Reference; Science | Dense, factual, well-edited text | Tag and consider up-weighting |
Because a domain can carry several ranked categories, write rules on the main label and on any label you never want, for example "remove if any label is Adult". The full category taxonomy lists all 59 web filtering categories and the IAB tree.
Low-quality web content: what categories catch and what they miss
"Low quality" covers several different problems. Domain data is strong for some of them and only a hint for others, so it pays to know which is which before you rely on it.
Parked and for-sale pages
A whole domain shows a holding page. Its category settles it with no page analysis needed.
Scam and malware hosts
Phishing kits and fake shops sit on dedicated domains that are labeled as suspicious.
Adult and piracy networks
Large families of related domains, easy to remove by category in one pass.
Made-for-advertising sites
Sites built to farm ad revenue. Categories plus popularity help; see our made-for-advertising sites guide.
Thin affiliate sites
Shopping and review domains with little original text. Down-weight by category, then filter by page quality.
Generated spam on good hosts
Spam pages on blog platforms or hacked sites. Only page-level checks catch these reliably.
Where domain filtering sits in a training data pipeline
Put the domain step early, right after URLs are extracted and normalised. Everything downstream then runs on a smaller, cleaner set, and expensive model-based filters only see pages worth scoring.
Extract and normalise
Pull URLs from the crawl, reduce hosts to registrable domains and lower-case them.
Join categories
Join every domain to the category file. Attach main label, all labels, IAB path and popularity group.
Apply domain rules
Remove, down-weight or tag by category. Send uncategorized domains to a stricter path.
Page-level filters
Language identification, deduplication, quality scoring, toxicity and personal data scrubbing.
Rights and opt-out checks
Honour machine-readable opt-outs, site terms and your licensing policy. Keep records.
Mix and sample
Balance subjects using the category tags, then build the final training and evaluation sets.
What the category join adds to every document
| Field | Example value | Used for |
|---|---|---|
| Main web filtering label | Education/Reference | Remove, down-weight or protect rules |
| All labels, ranked | Health & Medicine, Science | "Never any of these" safety rules |
| IAB path | Medical Health > Diseases and Conditions | Fine-grained subject mixing |
| Country | United Kingdom | Language and market balance |
| Popularity group | Global top 10,000 | Trust floors and allow lists |
A licensed file runs entirely inside your cluster, which matters when crawl data cannot leave your environment. Compare the options in URL categorization API vs database.
Using categories to balance the training mix
Once every document carries its domain's categories, you can measure and steer the subject mix of the corpus instead of accepting whatever the crawler happened to collect. These are the sizes of some useful groups in our full database, counted as domains carrying the label.
| Mixing goal | Category handle | Typical choice |
|---|---|---|
| More factual knowledge | Education/Reference, Science, Government & Legal | Up-weight, with deduplication of mirrored reference pages |
| Better technical answers | Developer Tools / Code Hosting, Computers & Technology | Up-weight documentation hosts, filter forum noise by page |
| Current events awareness | News/Media | Keep, with care around paywalled and opted-out publishers |
| Less sales tone | Shopping, Advertising/Marketing, Marketplaces | Down-weight to stop marketing language dominating |
| Safer medical answers | Health & Medicine | Prefer high popularity and institutional hosts, filter by page |
| Language and country balance | Country field and country popularity | Sample per country to avoid a single-market corpus |
Writing filter rules: worked examples
Rules are easiest to review when they read like policy. These examples show the pattern; adapt the thresholds to your own model and risk appetite.
| Rule in plain words | Logic on the category join | Effect |
|---|---|---|
| Drop anything adult or malicious | If any label is Adult or Malware / Phishing / Suspicious, remove the domain | Hard removal, no exceptions |
| Drop holding pages | If the main label is Parked / For-Sale Domains, remove | Cuts template duplication |
| Limit sales copy | If the main label is Shopping or Advertising/Marketing, keep one document in N | Down-weights without losing product vocabulary |
| Protect reference text | If the main label is Education/Reference or Science, skip the aggressive quality filter | Keeps dense factual text that heuristics sometimes misjudge |
| Extra care for forums | If any label is Forums / Message Boards, run personal data scrubbing at strict settings | Lowers privacy risk in dialogue data |
| Unknown hosts | If the domain has no category, send to the strict page-filter path | New and obscure hosts get more scrutiny |
Main label vs any label: big multi-purpose sites carry many labels. Removing on "any label" for broad categories such as Shopping would drop huge, useful sites, so reserve "any label" rules for categories you never want.
Building a high-trust allow list
Some projects need the opposite of a block list: a small, trusted set of hosts for instruction tuning, evaluation or a domain-specific model. Categories and popularity make a strong first cut.
Pick subjects
Choose the IAB categories and web filtering labels that match the model's purpose, for example Medical Health and Science.
Set a popularity floor
Keep hosts in the higher popularity groups, where established, maintained sites cluster.
Exclude risky secondary labels
Drop hosts that also carry labels such as Advertising/Marketing or Gambling.
Review by hand
Read the top few hundred hosts, confirm rights and opt-out status, and record the decision for each.
Licensing and legal cautions
Filtering by category improves quality and safety. It does not answer whether you may use the content. Treat these as separate questions, keep separate records for each, and involve your legal team early, before the corpus is fixed.
A category is not a licence
A domain labeled Education/Reference may still hold copyrighted material that its owner has reserved from text and data mining. Categories describe content, never rights.
Copyright and text and data mining
Rules differ by country. In the EU, rights holders can reserve their works from commercial text and data mining with a machine-readable opt-out.
Machine-readable opt-outs
Check robots.txt rules, including those aimed at AI crawlers, and other opt-out signals at collection time and keep a record of what you saw.
EU AI Act duties
Providers of general-purpose AI models in the EU must have a copyright policy that respects opt-outs and must publish a summary of training content. Check the current guidance and templates.
Site terms and paywalls
Terms of service may forbid scraping or reuse. Paywalled and login-only content needs its own review.
Personal data
Forums, social and community pages contain personal data. Data protection law applies to training sets too.
Your category data licence
Our licence covers using the category data inside your own systems. It does not grant rights to any website's content, and the raw category data may not be resold.
This section is general information, not legal advice. For data protection basics around categorization, see privacy compliance and GDPR.
Auditing your filters
Filters that nobody checks drift into removing good data or keeping bad data. A light audit routine catches both, and it gives you the evidence you need when someone asks why a source is or is not in the corpus.
- Count documents and tokens removed per category for every crawl snapshot
- Read a random set of removed documents per rule to check for false removals
- Read a random set of kept documents from risky categories to check for leaks
- Track the share of uncategorized domains and route them to stricter filters
- Re-run the join when you refresh category data and diff the outcome
- Record the rule set and category data date in the dataset card
- Check that high-value domains are not removed by a secondary label
- Review rules when the model's purpose changes, for example from chat to code
Mistakes that weaken crawl filtering
Blocking on any label
Large platforms carry many labels. Blocking broad categories on any label throws away some of the best text on the web.
Stale category data
Using a year-old file on a fresh crawl leaves new scam and parked domains uncaught.
Ignoring subdomains
Matching on full hostnames without reducing to registrable domains lowers match rates sharply.
Treating filtering as rights clearance
Clean data is not licensed data. Keep the two reviews separate and documented.
No record of rules
Without the rule set and data date in the dataset card, nobody can reproduce or explain the corpus later.
One rule set for every model
A safety classifier needs examples of harmful text that a chat model should never see. Rules follow purpose.
Freshness matters: crawls contain many young domains. About 300,000 newly registered domains are categorized daily, and all others are refreshed monthly. See newly registered domains and coverage and freshness.
Beyond pre-training: other AI uses of category data
The same domain join is useful well after the corpus is built. Wherever a model reads from or acts on the open web, categories give a fast, explainable control point.
Retrieval and search grounding
Restrict retrieval to allowed categories and block malware, adult and parked hosts before results reach a model or a user.
Fine-tuning sets
Build domain-specific sets, for example only Health & Medicine hosts with high popularity, as a first cut before human review.
Evaluation sets
Sample test documents evenly across categories so evaluation reflects the whole web, not just its largest subjects.
Agent browsing guardrails
Let AI agents visit only allowed categories and stop them at suspicious or adult domains.
Classifier training
Use labeled domains to train or test your own content classifiers. See the website classification dataset.
Tracking AI tools
Monitor the growing set of AI product domains, covered on our AI domains page.
Filtering crawl data with URL categories: FAQ
Why filter web crawl data by domain category?
It removes whole hosts of unwanted content in one cheap step before heavier page-level filters run, and it lets you tag and balance the mix by subject.
Is domain filtering enough for LLM training data?
No. It is a strong first pass. Page-level filters for language, duplication, quality, toxicity and personal data are still needed.
Does a category label mean I can use the content for training?
No. A category describes the subject, not the rights. Check opt-outs, terms and law separately, ideally with legal advice.
Should parked domains be removed from training data?
Yes, in almost every case. Their template pages add duplication and noise without useful language.
Can URL categories help balance a training mix?
Yes. Tag every document with its domain's categories, measure the subject mix and up-weight or down-weight groups as needed.
How do you handle new domains that are not categorized yet?
Treat them as a separate bucket with stricter page filters, and refresh your category data, since new domains are added daily.
Should harmful categories be removed completely?
For a general chat or writing model, usually yes. Teams building safety classifiers or red-team evaluation sets often keep a small, reviewed and access-controlled slice, because a classifier cannot learn to spot what it has never seen.
Does domain filtering reduce personal data in a corpus?
It helps a little, by letting you treat forums, community and social sites with stricter scrubbing. It does not remove personal data on its own, so a dedicated personal data filter is still needed.
How much of a crawl will category rules remove?
It depends on the crawl and the rule set. Measure it on your own data: count pages and tokens removed per rule, and check that the most valuable hosts are untouched before you commit.
Which file format suits large crawl joins?
Parquet is usually best for distributed joins. CSV and JSON are also available; see download formats.
Category data for cleaner training corpora
Web filtering categories, IAB categories, country and popularity for over 120 million domains, delivered as files that run inside your own environment, with daily updates for new domains.
See licence plans How to evaluate category dataIAB is a trademark of the Interactive Advertising Bureau. URL Categorization Database is not affiliated with, sponsored or endorsed by IAB.