Website Classification Dataset for Machine Learning
A labeled dataset of over 120 million domains, each tagged with IAB Content Taxonomy tiers, web filtering categories, personas, country and popularity bands. Built for teams that train or evaluate website and text classifiers.
domain: "example-bakery.com"
iab_v3_tier1: "Food & Drink"
iab_v3_tier2: "Desserts and Baking"
web_filtering: ["Food & Drink", "Shopping"]
personas: ["Baking Enthusiast", "Food Blogger"]
country: "United Kingdom"
popularity_global: "500001-1000000"
# illustrative row, not a real record
What a labeled website dataset looks like
A website classification dataset pairs an identifier, usually a domain, with labels from one or more fixed taxonomies. The value sits in label coverage, label depth and how consistent the labels are across millions of rows.
Columns in the dataset
Every file has the same column layout, so you can switch between a research extract and the full release without changing your loaders. The full field guide is in the data dictionary.
| Column | Type | What it holds | ML role |
|---|---|---|---|
Domain | string | Registrable domain, lower case | Join key, input for URL based models |
IAB v2 Tier 1 to Tier 4 | string | Path through the IAB v2 hierarchy | Targets for legacy ad systems |
IAB v3 Tier 1 to Tier 4 | string | Path through the 703 category IAB v3 hierarchy | Hierarchical classification targets |
Web Filtering Category | list, pipe separated | One or more of 59 filtering labels | Multi-label targets, safety classifiers |
Personas | list, semicolon separated | Audience types the site attracts | Audience and intent modelling |
Country | string | Country association, where one can be assigned | Stratification, regional models |
Global Popularity Rank Group | band | Popularity band among real users worldwide | Sample weighting, head versus tail splits |
Country Level Popularity Rank Group | band | Popularity band inside the site's country | Regional weighting |
Label coverage in numbers
These figures come from the current release. They show how many rows carry each kind of label, which matters when you decide which column to use as a training target.
Base: the full database. Country and popularity bands apply only where a domain has a clear country association or enough visitors to be banded.
Deeper IAB tiers are filled only where the site is specific enough to support them. A general news portal stops at Tier 1 or Tier 2, while a site about one dog breed can reach Tier 4.
How many labels does a domain carry?
The web filtering field is multi-label. Most domains describe one main subject, but large platforms cover many. The distribution below counts labels per domain in the current release.
| Filtering labels per domain | Domains | Share |
|---|---|---|
| 1 | 94,116,673 | 91.1% |
| 2 | 6,723,672 | 6.5% |
| 3 | 2,128,694 | 2.1% |
| 4 | 324,742 | 0.3% |
| 5 | 57,323 | 0.1% |
| 6 | 10,916 | 0.0% |
| 7 | 3,329 | 0.0% |
| 8 | 1,296 | 0.0% |
| 9 | 521 | 0.0% |
| 10 or more | 335 | 0.0% |
The first label is the one that describes most of the site. Use it as the target for single-label models, and the full list for multi-label models with a ranking loss.
Class balance across the top categories
Real web data is imbalanced. Business and technology sites are common, while categories such as weapons or tobacco are rare. Plan sampling and loss weighting with these counts in mind.
| Web filtering category | Domains | Share of rows | Class size |
|---|---|---|---|
| Computers & Technology | 18,883,407 | 18.27% | Common |
| Business | 16,195,218 | 15.67% | Common |
| Arts & Entertainment | 13,615,861 | 13.17% | Common |
| Shopping | 8,117,477 | 7.85% | Common |
| Health & Medicine | 7,431,659 | 7.19% | Common |
| Home & Garden | 6,299,837 | 6.09% | Common |
| Education/Reference | 3,557,549 | 3.44% | Common |
| Financial Services | 3,282,275 | 3.18% | Common |
| Automotive & Transportation | 3,096,363 | 3.00% | Common |
| Sports & Recreation | 2,872,482 | 2.78% | Common |
| Advertising/Marketing | 2,856,394 | 2.76% | Common |
| Travel | 2,818,617 | 2.73% | Common |
| Forums / Message Boards | 44,122 | 0.043% | Rare |
| Cloud Storage / File Hosting | 41,886 | 0.041% | Rare |
| Tobacco | 31,386 | 0.030% | Rare |
| VPN / Proxy / Anonymizers | 15,865 | 0.015% | Rare |
| File Sharing/P2P | 15,388 | 0.015% | Rare |
| URL Shorteners | 10,208 | 0.010% | Rare |
| Hate/Discrimination | 5,279 | 0.005% | Rare |
| Remote Access Tools | 2,342 | 0.002% | Rare |
The largest class is about 8,062 times the size of the smallest. Rare classes such as Hate/Discrimination still hold thousands of examples, enough to train and test on.
Taxonomies inside the dataset
Using more than one taxonomy lets the same dataset serve advertising, safety and research projects. Each taxonomy answers a different question about a site.
IAB Content Taxonomy v3
703 categories in a four level tree, the standard language of contextual advertising. Read the full list on our taxonomy page.
IAB Content Taxonomy v2
The previous edition, still used by many ad servers and data platforms. Keeping both lets you train on one and evaluate on the other.
59 web filtering categories
Safety and policy oriented labels such as Adult, Gambling, Malware / Phishing / Suspicious and AI / LLM Tools. See the filtering categories.
Why both together
The same site gets a subject label and a policy label. A wine shop is Food & Drink in IAB terms and Alcohol in filtering terms, which lets one model serve advertising and safety teams.
Personas
Audience types such as Software Developer or Home Baker. Useful for audience modelling and for testing whether a model understands who a site is for.
Machine learning use cases
Teams license the dataset as training data, as a benchmark, or as a feature table joined to their own logs. These are the most common projects.
Text classification
Pair our labels with page text you collect yourself and train a classifier for site topics. The hierarchy supports both flat and tree aware models.
Contextual AI for advertising
Train models that place ads by page context instead of personal data. See contextual targeting for how buyers apply the labels.
URL only classifiers
Predict a category from the domain string alone, for traffic where no page content is available, such as DNS logs or encrypted connections.
Safety and policy models
Use the filtering labels to train detectors for adult, gambling or phishing content, with rare classes boosted by sampling.
Evaluation benchmarks
Hold out a stratified slice to measure how well a general model labels websites, by category, country and popularity band.
Feature enrichment
Join categories to clickstream, CRM or security logs as features for churn, fraud or lead scoring models. See data enrichment.
Preparing the data for training
Website data has a few traps that inflate offline scores. This checklist keeps your evaluation honest.
- Split by registrable domain, never by URL, so the same site never lands in train and test
- Group sister domains of one brand into the same split to avoid leakage
- Stratify by Tier 1 category so rare classes appear in every split
- Keep a popularity stratified test set, because head sites and tail sites behave differently
- Report hierarchical metrics: a wrong Tier 3 under the right Tier 1 is a smaller error
- Drop parked and for-sale domains unless your task is to detect them
- Track country, since language changes what a model sees
- Version every extract and record the release month next to your results
Example tasks and how to evaluate them
The same file supports very different models. Pick the target column and metric that match the decision your model will make.
| Task | Target column | Suggested metric | Watch out for |
|---|---|---|---|
| Topic classifier for ad placement | IAB v3 Tier 1, then Tier 2 | Macro F1 per tier | Large classes hiding weak rare classes |
| Hierarchical classifier | IAB v3 Tier 1 to Tier 4 | Hierarchical precision and recall | Empty deeper tiers, which are not errors |
| Safety or policy detector | Web Filtering Category, one label vs rest | Precision at a fixed recall | Cost of false blocks versus missed sites |
| Multi-label site tagger | Web Filtering Category, all labels | Micro F1 and ranking metrics | Platforms with many labels dominating the loss |
| Audience model | Personas | Top-k accuracy | Overlapping persona names |
| Domain name only model | Any label, domain string as input | Accuracy by popularity band | Brand names that reveal nothing about the topic |
Label quality and known limits
Every label is categorized from site content, then reviewed and validated. Regular manual validations on random samples measure label accuracy. A good dataset card is honest about the limits that remain.
What is validated
Random samples are checked by people against the live site, by category and by tier. Licence holders can ask for the current validation figures for the categories they use.
Sites change
A label describes the site at the time it was categorized. Monthly refreshes keep most rows current, but a small share of sites always changes between releases.
Language mix
The web is not evenly spread across languages. Check the country field before you assume a model trained on this data works equally well in every market.
Domain, not page
Labels describe a whole domain. A news site's sports section still carries the site level labels, so page level models need extra care.
Common extract sizes
Most projects do not need every row. These are the extract sizes research and product teams ask for most often.
Balanced training set
Equal rows per category- A fixed number of domains per Tier 1 or filtering category
- Good for first models and fair evaluation
Popular head
The most visited domains- The domains people visit most, with all labels
- Good for benchmarks and production checks
Full release
Every categorized domain- All rows, refreshed monthly
- Good for long tail coverage and lookups at scale
Licensing for research and commercial use
The dataset is licensed, not given away. Each licence states who may use the data, for what, and whether trained outputs may ship to your customers.
Research
Universities, institutes, non-commercial labs- Training and evaluation inside the research group
- Publication of results, metrics and aggregate statistics
- No publication of the raw labeled rows
Commercial internal
Companies using the data inside their own systems- Training models used inside your company
- Joining labels to your own logs and records
- Monthly refreshes for the licence term
Product and OEM
Companies whose product carries the labels or a trained model- Labels or derived models inside a product you sell
- Group licences for several products
- Terms set per deployment
No licence allows resale or publication of the raw data. Read the general terms on the licensing page and see plans on the pricing page.
How to request the dataset
Requests are handled by a person, not a download form, so the extract fits your project from the start.
Most requests are answered with a written licence proposal that names the extract, the refresh schedule and the permitted uses.
Want to see labels for a few specific sites first? Use the domain category checker on any domain you know.
Related lists and guides
The same data powers our ranked lists and industry guides. They are a quick way to see how the labels behave on real sites.
Top websites ranking
The most popular domains among real users, with their categories.
Top 100 websites by category
Ten leading domains in each of ten major categories.
Top-level domains list
Counts and top categories for the main TLDs.
Website to industry
How categories become industry labels for company records.
Frequently asked questions
What is a website classification dataset?
It is a table of domains or URLs where each row carries one or more category labels from a defined taxonomy. Machine learning teams use it to train and evaluate models that predict what a website is about.
Which taxonomies are included?
Each domain carries IAB Content Taxonomy labels in version 2 and version 3, up to four tiers deep, plus up to several of 59 web filtering categories, audience personas, a country and popularity bands.
Can the dataset be used for commercial models?
Yes, under a commercial licence. Licences cover research, internal commercial use, and use inside a product. Publishing or reselling the raw labeled data is not allowed under any licence.
Is it multi-label?
Yes. The web filtering field can hold several categories per domain, ranked by how much of the site they describe, and the IAB fields form a hierarchy from Tier 1 to Tier 4.
Does the dataset include page text?
The dataset holds domain level labels and context fields. Most teams pair it with page text or features they collect under their own policies, which keeps the licence simple.
How large is the dataset?
The current release holds over 120 million categorized domains. Almost every row carries IAB v3 Tier 1 and web filtering labels, while deeper IAB tiers, country and popularity bands are filled where the site supports them.
Can I get a balanced dataset?
Yes. Many teams ask for a fixed number of domains per category so that rare classes are well represented. Tell us the categories and the rows per class you need.
Which file formats are available?
CSV, Parquet and JSON. Parquet is the usual choice for large training jobs, because it loads quickly and keeps column types.
Is the dataset suitable for language models?
It can be used to fine-tune or evaluate models that label websites, or as a benchmark for how well a general model describes sites. Tell us whether the trained model will ship to customers so the licence fits.
How do I request the dataset?
Contact us with your use case, the volume of domains you need, the taxonomies you care about and whether the result will be research, internal or part of a product. We reply with licence options.
Train on labels that cover the real web
Tell us about your model and we will propose an extract and a licence that fit it.
Request the dataset View pricingIAB and the IAB Content Taxonomy are trademarks of the Interactive Advertising Bureau. They are used here only to identify the taxonomy discussed. URL Categorization Database is not affiliated with, sponsored or endorsed by the Interactive Advertising Bureau.