How to evaluate a URL categorization database
Buying category data is a long term decision. The data ends up inside products, policies and reports, and changing source later is expensive.
This guide gives procurement and technical teams an RFP structure, a question bank and a fair way to test accuracy on your own domains.
The evaluation in four phases
A structured process keeps the comparison fair and the timeline short. Most teams finish in a few weeks when the test sample is ready before the RFP goes out.
Define needs
Write down the use case, the decisions the data will drive and the categories that matter most.
Build the test set
Draw domains from your own traffic or records and label a sample by hand.
Send the RFP
Use the same question bank for every provider and ask for written answers.
Score and decide
Score answers and test results with agreed weights, then check licence terms with legal.
Start with the decision, not the data: "block phishing for 40,000 small office users" and "size the pet supply market in five countries" need very different things from the same database.
Writing the needs statement
The first page of the RFP tells providers what you are trying to do. A clear needs statement gets you specific answers instead of brochure text.
Include
- The decision the data will drive, in one sentence.
- Where the data will run: product, internal system, research environment.
- Rough lookup volume and how many distinct domains you see per month.
- Countries and languages that matter most.
- The categories you must have, and those that are nice to have.
- How often you can load updates.
Leave out
- Your test sample itself, which you keep for scoring.
- Internal budget figures, which can anchor answers.
- A preferred provider's terms copied into the requirements.
- Requirements nobody on the team can explain.
- Long legal clauses better handled after shortlisting.
Support and working relationship
Category data is a long relationship, not a one off download. A few questions about how the provider works with customers round out the picture.
Who answers questions?
Ask how technical and data quality questions are handled, and through which channel.
How are disputes handled?
Ask how a domain you believe is wrong is reported and reviewed.
What notice for changes?
Ask how changes to columns, formats or categories are announced in advance.
Who are similar customers?
Ask for descriptions of customers with a similar use, since names are often confidential.
How is onboarding run?
Ask what help is available during the first load, mapping and automation of updates, and how long a typical start takes.
The RFP question bank
Copy the questions that fit your use. Each area ends with a note on why it matters, so reviewers know what a good answer looks like.
1. Coverage
Does the data answer for the domains you actually see?- How many domains carry at least one category, and how many are active sites rather than parked or empty?
- What share of the most visited domains is covered?
- How is the long tail covered, including small regional sites?
- Which countries and languages are covered well, and which less so?
- What happens when we look up a domain you do not have?
Why it matters: headline totals are hard to compare. Your own hit rate on real traffic, measured in the test, is the number that predicts performance.
2. Freshness
How quickly does the data follow the web?- How many newly registered domains are reviewed each day?
- Are new domains checked again after launch, and when?
- How often are established domains refreshed?
- How do updates arrive: full files, delta files, or both, and how often?
- How can we get a current answer for a domain between updates?
Why it matters: security uses need new domains within a day, while research needs stable snapshots. Ask for the cadence that fits your decision.
3. Taxonomy fit
Do the categories match the decisions you make?- Which taxonomies are delivered, and how many categories does each have?
- Can a domain carry several categories, and are they ranked?
- Is there a published list of categories with definitions?
- How are sensitive categories defined, such as adult, gambling or weapons?
- How often does the taxonomy change, and how are changes communicated?
Why it matters: a category you cannot map to your policy or report is a category you cannot use. Try mapping your own list before you score.
4. Accuracy
Are the labels right on your kind of domains?- How is accuracy measured, how often, and on what kind of sample?
- Can we see results for the categories that matter to us?
- How are parked, for-sale and inactive domains handled?
- How can we report a domain we believe is wrong?
Why it matters: accuracy figures from a provider describe their own sample. Your own test set is the only like for like comparison.
5. Delivery and integration
Will the data fit your systems?- Which file formats are available, and is there a data dictionary?
- Can the data run fully inside our own systems, including offline?
- How are files delivered and verified?
- Is an API included, and with what limits?
- Are ready made exports available for our tools, such as DNS or firewall list formats?
Why it matters: integration effort is a real cost. Local files remove per lookup latency and keep your queries private.
6. Licence terms
Can you use the data the way you plan to?- Is the licence for internal use, embedding in our product, or both?
- Which of our products, subsidiaries and customers are covered?
- Are updates included for the whole term?
- What happens to the data and derived outputs at the end of the term?
- How is the renewal price set?
Why it matters: a licence that does not cover your real deployment creates risk later. Fix scope in writing before technical work starts.
Building your own test sample
The test sample is the heart of a fair evaluation. Draw it from your world, not from a provider's demo list, so the result predicts what you will see in production.
Draw from real data
Take domains from your proxy logs, DNS logs, bid requests or CRM, depending on the use case.
Stratify by popularity
Include popular sites, mid tail and long tail in roughly the proportions your traffic has.
Add new domains
Include a slice of domains registered in the last few weeks to test freshness.
Over sample what matters
If gambling or phishing decisions are critical, include extra domains from those areas.
Label blind
Have two people label each domain by hand without seeing any provider's answer.
Settle disagreements
Where labellers disagree, a third person decides. Record unclear cases separately.
| Sample size | What it gives you | Good for |
|---|---|---|
| A few hundred domains | A first view of hit rate and obvious errors | Shortlisting |
| One to two thousand domains | Firmer numbers for the main categories | Final decision |
| Larger unlabelled log sample | Hit rate only, no accuracy | Coverage on real traffic |
Spot checks first: while the full sample is being labelled, try a few domains you know well in the domain category checker to get a feel for how categories are assigned.
Measuring the test results
Agree on the measures before results arrive, so nobody is tempted to pick the numbers that favour a preferred option.
Hit rate
Share of your domains that receive any category. Measure on the large log sample.
Primary match
Share where the provider's main category matches your hand label.
Any match
Share where your label appears anywhere in the ranked categories.
Critical misses
Share of truly risky domains given a harmless category. Usually the costliest error.
False alarms
Share of harmless domains given a risky category. Drives user complaints.
New domain result
All of the above on the recently registered slice only, to test freshness.
Map before you measure: providers use different category names. Map each taxonomy to your own labels once, write the map down and apply it to everyone the same way.
A weighting model you can adapt
Weights should reflect your use case. The table shows starting weights for three common buyers; adjust them before scoring begins.
| Area | Security product | Ad platform | Research team |
|---|---|---|---|
| Coverage on your traffic | 20% | 25% | 20% |
| Freshness | 25% | 10% | 5% |
| Taxonomy fit | 10% | 25% | 25% |
| Accuracy on your sample | 25% | 25% | 30% |
| Delivery and integration | 10% | 10% | 5% |
| Licence fit | 10% | 5% | 15% |
Scoring each area: give each provider a score from 1 to 5 per area, using written evidence only. For coverage and accuracy, convert your measured results to the scale with bands agreed in advance, such as fixed hit rate ranges.
Two reviewers per area: have two people score each area on their own, then compare. Large gaps between them usually reveal an unclear question rather than a weak provider.
Price is usually scored separately once the quality ranking is known, so it does not hide a weak result. Plans and tiers for this database are on the pricing page.
Taxonomy fit: a quick mapping test
Before scoring accuracy, check whether the categories can express your decisions at all. A one hour exercise reveals most gaps.
How to run it
- List the 20 to 40 categories your product or report uses today.
- For each, find the matching category or categories in the provider's list.
- Mark each as exact match, combination, partial or missing.
- Count the missing and partial ones that matter for your decisions.
What to look for
- Separate categories for things you treat differently, such as News/Media and Political.
- Depth where you need detail, such as IAB tier 3 and 4 topics for advertising.
- Ranked multiple categories, so mixed sites are not forced into one box.
- Clear definitions for sensitive categories.
Our category taxonomy lists both systems delivered here, and the IAB taxonomy mapping guide shows a mapping method step by step.
Licence terms to get in writing
Legal review goes faster when the technical team has already listed how the data will be used. These are the points that most often need clarification.
| Term | Question to settle |
|---|---|
| Scope of use | Internal only, inside a product you sell, or both |
| Entities covered | Which companies, subsidiaries and products may use the data |
| Derived outputs | Whether block lists, scores and reports built from the data can reach your customers |
| Raw data | Confirmation that raw rows are not resold or published |
| Updates | Which update files are included, and for how long |
| Deployment | Cloud, on-premise, air-gapped, and in which regions |
| End of term | What must be deleted, and what derived outputs may be kept |
| Renewal | How the renewal price and scope are set |
Our own terms are summarized on the licensing page, and the technical side in URL database specification and delivery.
Answers that need a follow up question
Some answers sound complete but leave the key point open. These patterns are worth a polite second question, whichever provider gives them.
Vague on numbers
- "Millions of domains" without saying how many are active sites.
- "Highly accurate" without saying how it was measured or on what sample.
- "Regular updates" without a cadence for new and established domains.
Vague on terms
- "Flexible licensing" without saying what your planned use requires.
- "Custom pricing" without the factors that set it.
- "API available" without limits, or whether it is included.
Fair to everyone: send the same follow up to every provider. The goal is comparable answers, not catching anyone out.
Use case notes
The six areas apply to everyone, but each type of buyer leans on different ones. Read the note for your use before setting weights.
Web and DNS filtering
Weight freshness and critical misses, and test new domains and risky categories heavily. See web content filtering.
Brand safety and targeting
Weight taxonomy depth and false alarms on news. Test on bid request domains.
Security analytics
Weight coverage of rare domains and parked detection. Test on SOC alert domains.
CRM enrichment
Weight coverage of company domains and country data. Test on your account list.
Research and AI data
Weight licence terms for publication and stable snapshots. Test on your corpus domains.
OEM products
Weight licence scope, delivery formats and update automation. Test on customer traffic samples.
For quality dimensions in more depth, see the category data quality checklist. For delivery choices, see URL categorization API vs database.
After the decision
The evaluation work keeps paying off after signature. The same test set becomes your quality check for the life of the contract.
Reuse the map
The taxonomy map built for testing becomes your production mapping.
Rerun the test
Score the same sample against the current data to see how quality holds.
Log disagreements
Record domains your users dispute and report them to the provider.
Decide on evidence
Use a year of test results and hit rates when the renewal comes up.
Frequently asked questions
What should an RFP for a URL categorization database cover?
Coverage on your domains, freshness, taxonomy fit, accuracy on your own sample, delivery and licence terms. Ask for written answers to compare side by side.
How do I test accuracy fairly?
Label a sample of your own domains by hand without seeing any answers, then compare each provider against your labels with the same rules.
How big should my test sample be?
A few hundred labelled domains for a first view, one to two thousand for firmer numbers. Mix popular, long tail and new domains like your traffic.
Why is coverage on my own domains more important than total size?
Totals say little about the domains your users visit. Your own hit rate is the coverage figure that predicts performance.
What licence terms matter most?
Scope of use, entities covered, derived outputs, included updates, renewal and what happens at the end of the term.
How long does an evaluation take?
Usually a few weeks: preparation, provider answers, then scoring and reference checks.
Can I check how a domain is categorized before buying?
Yes. The domain category checker shows the category for any domain you enter.
Put our data through your evaluation
Over 120 million categorized domains with web filtering and IAB categories, country, personas and popularity bands. Send us your RFP and we answer every question in writing.
IAB is a trademark of the Interactive Advertising Bureau, named here only to identify the content taxonomy discussed. URL Categorization Database is not affiliated with, sponsored or endorsed by the Interactive Advertising Bureau.