Website categories for web measurement research
Web measurement studies crawl or observe thousands of websites to learn how the web behaves: who tracks visitors, which sites adopt a security feature, how cookie banners look across sectors.
Category labels turn those measurements into findings about kinds of sites. This guide covers how to use them well, and how to report and cite them.
Use categories to stratify your sample and to explain differences in your results. Fix one data snapshot for the whole study, state the taxonomy and counting rule, check a sample of labels by hand, and cite the dataset with its date and licence.
1.Research questions where categories help
Most measurement papers ask how common something is and where it is more common, and categories answer the second part. These are typical study themes where category labels do real work.
Third-party tracking by sector
Do news, health or shopping sites embed more third-party scripts than government or education sites?
Consent banner practice
Do consent banners differ in design or compliance between categories and countries?
Security feature adoption
Which kinds of sites lag in HTTPS settings, security headers or modern protocols?
Accessibility by sector
Are public service and education sites more accessible than commercial ones?
Crawler and AI access rules
Which categories of sites restrict automated crawlers in their robots rules, and how is that changing?
Internet filtering and blocking
When sites are unreachable in a network or country, which categories are affected most?
Advertising and page weight
How do ad density, page size and load time vary between kinds of publishers?
Privacy policies
Do privacy policies in some sectors disclose more, or less, than others?
Hosting and infrastructure
Do some categories concentrate on a few hosting or delivery networks, and what does that mean for resilience when one of them fails?
Web change over time
How fast do new sites appear in each category, and which kinds of sites disappear or turn into parked pages within a year?
Language and regional webs
How does the mix of site categories differ between language communities and regions, and what is missing from smaller webs?
2.Three ways to use categories in a study design
Categories can enter a study at three points. Many papers use more than one, and it helps to say clearly which.
Stratified sampling
- Draw a fixed number of sites per category, not only the most popular.
- Prevents a few dominant categories from swamping the sample.
- Lets you report results per category with similar confidence.
- Weight back to population shares when reporting overall rates.
Explanatory variable
- Add category as a column next to your measurements.
- Compare rates between categories with suitable statistics.
- Control for popularity and country, which often correlate with category.
- Report effect sizes, not only significance.
Filtering the sample
- Remove parked, for-sale and inactive domains before measuring.
- Remove adult or malicious sites if your ethics review requires it.
- Keep a log of every removal and its reason.
- Report how many domains each filter removed.
Why not only popular sites? Studies of the most visited sites describe where attention goes, while studies that sample the long tail describe what the web is. Both are valid, so say which question you are asking.
3.What a category record gives a researcher
Each domain row carries several fields. Knowing what each represents, and what it does not, prevents overclaiming in the results section.
| Field | What it describes | Read it as |
|---|---|---|
| Web filtering category | One of 59 broad categories | The main type of site, suited to top level comparisons |
| IAB content categories | Topics in up to four tiers | Finer subject detail for niche analysis |
| Ranked multiple categories | Several labels ordered by how much of the site they describe | Evidence that a site spans topics |
| Country | Country associated with the domain | A best attribution, not a legal fact about the owner |
| Popularity bands | Global and country bands based on popularity among real users | A coarse ordering for weighting, not visitor counts |
| Personas | Likely audience roles | An aid for interpretation, rarely a primary variable |
Exact column definitions are in the data dictionary. The taxonomies are listed on the category taxonomy page.
4.Pitfalls and how to handle them
Reviewers in measurement venues look for these problems. Addressing them in the methods section makes a paper much easier to accept and to reuse.
Domain versus page
Mixed sites
Popularity confound
Country comparisons
Time drift
Label error
False precision
5.Validating labels in your own sample
A short validation step makes your results far more credible. It also tells you which categories are reliable enough for your specific question.
| Step | What to do | Report in the paper |
|---|---|---|
| 1. Draw | Random sample of 200 to 500 domains from your study set, stratified by category | Sample size and method |
| 2. Label | Two people assign categories by hand, without seeing the dataset label | Labelling guidelines |
| 3. Agree | Measure agreement between the two people, resolve disagreements | Inter-annotator agreement |
| 4. Compare | Compare resolved labels with dataset labels | Agreement per category |
| 5. Decide | Merge or drop categories that are too unreliable for your question | Any merges and why |
Spot checks while planning: the domain category checker shows the category of any single domain. It is handy for exploring a handful of sites before you design the full study.
6.Reproducibility checklist
Measurement results are only useful if others can understand and repeat them. These items belong in the methods section or the appendix.
Snapshot date
The date of the category data used, and the date of your crawl or observation.
Taxonomy and level
Web filtering categories or IAB categories, which tier, and the counting rule.
Filters applied
Every removal step, such as parked or inactive domains, with counts removed.
Sampling procedure
How domains were drawn, with strata sizes and any random seed.
Validation result
Agreement between hand labels and dataset labels on your sample.
Licence statement
The licence type, and how other researchers can obtain the same data.
7.How to cite category data
Datasets deserve a proper citation, the same as papers. The 2014 Joint Declaration of Data Citation Principles makes the case that data should be cited as a first class research output, in the reference list.
Mention it in the text too: a sentence in the methods section, such as "domains were labelled with web filtering categories from the URL Categorization Database (snapshot of [month year])", helps readers who never reach the reference list.
8.What you can publish
Research licences are written for publication. The line runs between your findings, which you publish freely, and the raw labelled rows, which stay inside the research group.
Results and aggregates
- Counts and shares per category.
- Rates of the behaviour you measured, per category.
- Charts, tables and statistical tests.
- Examples of a few well known sites to illustrate a point.
The raw labelled rows
- Publishing the domain to category table.
- Uploading it to a public repository as supplementary data.
- Sharing it with groups outside the licence.
- Reselling or redistributing it in any form.
Sharing your own artefacts
- Publish your crawl code and your own measurements.
- Publish domain lists without category labels where your licence allows.
- Explain how others can license the labels.
- Keep a private copy of the exact snapshot for reviewers' questions.
General licence terms are on the licensing page. For machine learning work, the website classification dataset page describes research, commercial and product licences.
9.Ethics in web measurement
Category data itself describes websites, not people. Measurement work still affects site operators and sometimes users, and ethics boards expect you to have thought about it.
Respect for persons
Avoid collecting personal data during crawls. If pages contain it, minimize, protect and delete it.
Beneficence
Keep crawl load light, honour site rules where your design allows, and weigh harms against benefits.
Justice
Think about who carries the costs of your measurement, such as small sites with limited hosting.
Respect for law and public interest
Check legal limits on crawling and data use in your jurisdiction, and be transparent about methods.
A useful framework: the four headings above follow the Menlo Report, which adapted classic research ethics principles to information and communication technology research. Many measurement researchers use it when writing ethics sections.
10.A worked example study
An invented thesis project shows the pieces together. A master's student wants to know whether consent banners differ between categories of sites in three European countries.
| Stage | Decision |
|---|---|
| Question | Do consent banners offer a reject option as often on news sites as on shopping sites? |
| Population | Domains attributed to three countries, from one category snapshot |
| Cleaning | Remove parked, for-sale, adult and malicious domains, log counts removed |
| Sampling | 500 domains per category per country, split across popularity bands |
| Validation | Hand check 300 labels, report agreement per category |
| Measurement | Crawl each homepage once, record banner presence and options |
| Analysis | Compare rates by category within each country, control for popularity band |
| Reporting | Rates with confidence intervals, snapshot date, licence and citation |
What the student publishes: the code, the banner measurements and aggregate tables per category. The labelled domain list stays inside the research group under the licence.
11.Joining labels to your crawl data
Most lost hours in category based studies come from the join between crawl output and labels. A few rules keep it clean and keep the join rate high.
Make keys match
- Lowercase hostnames and strip any trailing dot.
- Reduce hostnames to the registered domain with a public suffix list.
- Use one form for internationalized names on both sides.
- Keep the original hostname in a separate column.
Decide which site you mean
- Crawls often land on a different domain after redirects.
- Decide whether to label the starting domain or the final one.
- Record both and report your choice.
- Check how many sites changed domain on the way.
Measure the join
- Report the share of crawled domains that received a label.
- Describe the unlabelled group: new, inactive or rare domains.
- Do not silently drop unlabelled sites.
- Show results with and without them where it matters.
A methods sentence that covers it: "Hostnames were reduced to registered domains and joined to category labels from a snapshot of [month year]; [share] of crawled domains received a label, and results exclude [n] unlabelled domains."
12.Getting access for a study
The data is licensed for research, as for any other use. Planning access early keeps it from becoming the bottleneck of a project.
Students
Talk to your supervisor first. Licences are usually arranged by the research group or department.
Research groups
Describe the study, the countries and the number of domains you need, and the publication plans.
Grant proposals
Budget for data access in the proposal, with the licence term matched to the project.
Contact
Write to [email protected] with a short description, or see plans and pricing.
Research using counts rather than measurements? Category data for market research covers sizing and trend methods, and filtering web crawls for AI training data covers corpus work.
Frequently asked questions
Why do web measurement studies need website categories?
They let you ask whether a measured behaviour differs by type of site, and they help build balanced samples instead of measuring only the most popular sites.
Which taxonomy should I use in a study?
Broad web filtering categories for top level comparisons, IAB categories for finer topics. Report the taxonomy, level and counting rule.
How should I cite category data in a paper?
Name the dataset and provider, snapshot date, taxonomy, sample size and licence type, in the reference list and in the methods section.
Can I publish the category labels with my paper?
You can publish results and aggregate statistics. The raw labelled rows may not be published, so share aggregates and explain how others can obtain the data.
Is there a free dataset for students?
No, the data is licensed for research as for other uses. Talk to your supervisor about a research licence, and use the domain category checker for small spot checks.
What are the main pitfalls of category based analysis?
Treating domain labels as page labels, ignoring mixed sites, comparing raw counts across countries, mixing snapshots and false precision.
Does the data include personal information?
No. Each row describes a website, not a person.
How large should a category stratified sample be?
It depends on how small a difference you want to detect between categories. Many studies use a few hundred to a few thousand sites per category, and a power calculation before crawling tells you what your question needs.
Can I use one snapshot for a study that runs for a year?
Yes, and it is usually the right choice. Record the snapshot date, and if your crawls span many months, check a sample of sites for changes between the snapshot and the crawl.
Category data for your next study
Over 120 million categorized domains with web filtering and IAB categories, country and popularity bands, licensed for research with publication of results and aggregates.
IAB is a trademark of the Interactive Advertising Bureau, named here only to identify the content taxonomy discussed. URL Categorization Database is not affiliated with, sponsored or endorsed by the Interactive Advertising Bureau.