URL database specification and delivery
Procurement asks what the licence allows, while engineering asks what arrives, in which format and how big it is. This page answers both in one place.
It gives the overview, and links to the documentation pages where each detail is defined in full.
- Coverage
- Over 120 million categorized domains
- Taxonomies
- 59 web filtering categories plus IAB content categories, tiers 1 to 4
- Extra fields
- Country, personas, popularity bands
- Formats
- CSV, Parquet, JSON
- Delivery
- HTTPS download with checksums, lookup API included
- Updates
- Weekly delta files, daily on request, monthly refresh
- Licences
- Internal use, OEM, group
What one record contains
The database is a table with one row per registered domain. Columns fall into six groups, and every licence tier carries the same columns: tiers differ by how many domains they cover.
Identity
The domain itself, lowercase, without protocol or path. It is the key you join on: domain.
Web filtering category
One of 59 categories built for policy decisions, from News/Media to Malware / Phishing / Suspicious. Never empty.
IAB content categories
Topic categories in up to four tiers, from broad tier 1 subjects to narrow tier 4 niches, in the IAB v2 and v3 versions.
Ranked multi-labels
A domain can carry several categories, ordered by how much of the site each one describes.
Context fields
Country associated with the domain, and personas that describe the likely audience in plain roles.
Popularity bands
Global and country popularity bands based on popularity among real users, available for the most visited domains.
The row above is an illustration of the layout, not a real record. The exact column names, types and empty value rules are defined in the data dictionary.
Taxonomy references
Two category systems sit side by side in every row. Each has its own reference page, so this guide only summarizes what each is for.
| Taxonomy | Built for | Depth | Reference |
|---|---|---|---|
| Web filtering categories | Policy decisions: allow, block, warn, report | 59 flat categories | Web filtering categories |
| IAB content categories | Advertising, content analysis, audience work | Up to four tiers | IAB category reference |
| Both together | Mixed use: targeting with a safety layer | One row, both systems | Category taxonomy overview |
Mapping your own categories: if your product already has its own category list, map it once to the delivered categories and keep the map in version control. The mapping usually takes a few days of work, not weeks.
Choosing your coverage
The first specification decision is how much of the web you need. More domains cost more storage and a larger licence, so match coverage to the traffic you actually see.
The most visited head
The 20 million most visited domains cover 99.99% of web traffic and usage. This suits filters, gateways and ad platforms that act on real browsing.
The full long tail
Over 120 million domains, including small and rarely visited sites. This suits security research, market sizing and training data selection.
Country or category cuts
Only the domains attributed to chosen countries, or only chosen categories. This suits regional products and focused research.
Column cuts
All rows but fewer columns, such as domain and web filtering category only. This suits appliances with tight memory.
How to decide: run a week of your own traffic or bid logs against a candidate coverage level and measure the share of requests it answers. The database category coverage page shows how categories are spread.
Domain matching rules
Categories are attached to registered domains. Your systems see full hostnames and URLs, so a few matching rules decide how often you find an answer.
| Input you see | Recommended handling |
|---|---|
| Full URL with path and query | Extract the hostname first, then look it up |
| Hostname with www | Strip www and look up the registered domain |
| Deeper subdomain | Try the full hostname, then fall back to the registered domain |
| Multi-part endings such as .co.uk | Use a public suffix list so the registered domain is cut correctly |
| Internationalized domain names | Normalize to one form, either Unicode or punycode, on both sides of the join |
| Uppercase or trailing dot | Lowercase and remove the trailing dot before lookup |
Not found locally? Send the domain to the included lookup API, or treat it as uncategorized under your own policy, since very new domains are the usual reason and arrive in the next delta file.
A typical onboarding timeline
Most teams go from signed licence to production use in a short project. The phases below are typical, and small teams often move faster.
First download
Fetch the snapshot, verify checksums, load into a test environment.
Map and match
Map categories to your product, settle domain matching rules.
Test on real traffic
Replay logs, measure hit rate, review a sample of decisions.
Automate updates
Schedule delta and monthly loads with staging and swap.
Production
Go live and monitor hit rates and category mix over time.
Questions engineers usually ask
These come up in almost every technical kickoff call. Short answers here save a meeting.
Is the domain unique?
Yes, one row per domain, so it works as a primary key.
Can a category be empty?
The web filtering category is always filled. Deeper IAB tiers can be empty where a site only fits a broad topic.
Do column names change?
Columns are documented in the data dictionary. Plan your loader to fail loudly if a header differs, as good practice.
How are deletions handled?
Delta files add and change rows. A periodic full reload from the monthly refresh also removes domains that have left the database.
Can we load only some columns?
Yes. Parquet makes this cheap, and CSV loaders can skip columns on import.
What about very high lookup rates?
Local files have no rate limits. Speed depends on your own hardware and index.
File formats at a glance
The same data ships in three main formats, plus ready made exports for security tools. The domain list download formats page covers each one in depth, including import pitfalls.
| Format | Best for | Notes |
|---|---|---|
| CSV | Databases, spreadsheets for small cuts, simple scripts | UTF-8, header row, comma separated. Largest on disk. |
| Parquet | Data warehouses, data lakes, analytics notebooks | Columnar and compressed. Much smaller and faster to query. |
| JSON | Applications and pipelines that expect nested records | Line delimited, one domain per line, easy to stream. |
| Security exports | Firewalls, DNS resolvers, proxies, home filters | Hosts files, DNS RPZ, AdGuard and Pi-hole style lists, external dynamic lists, PAC files. |
File sizes in general terms
Size depends on two things: how many domains your licence covers and which columns you take. The figures below are orders of magnitude for planning storage and memory, not exact numbers.
Planning tip: appliances and DNS filters often need only the domain and one category column. Dropping unused columns can shrink the in-memory footprint several times over.
How delivery works
Delivery is deliberately simple. Files are pulled over HTTPS by you or your automation, checked, and loaded into your own systems.
Licence confirmed
Coverage, columns, formats and update cadence are fixed in writing.
Download link
You receive secure HTTPS links for the full snapshot.
Verify
Compare each file with its published checksum before loading.
Load and index
Import into your database or store, indexed on the domain.
Apply updates
Delta files and monthly refreshes keep the copy current.
Secure download
Files arrive over HTTPS. Links can be fetched by scripts, so loading can run on a schedule without manual steps.
Checksums
Each delivery comes with checksums. A mismatch means the download was incomplete and should be repeated.
Lookup API included
Licences include an API for single domain checks, handy for spot checks and for domains not yet in your local copy. See the API overview.
Import patterns for relational databases, key value stores and memory mapped files are in the integration guide. Service and delivery terms are on the service level and delivery page.
Snapshots and update files
You start with a full snapshot and keep it current with smaller files. The schedule itself is explained in update schedule and freshness, so this section covers only the file side.
| File type | Contents | How to apply it |
|---|---|---|
| Full snapshot | Every domain in your licence with all licensed columns | Initial load, or a clean reload to reset drift |
| Weekly delta | Newly registered domains categorized that week, plus domains changed by their week and month rechecks | Upsert by domain |
| Daily delta, on request | The same content at daily granularity | Upsert by domain, suits security use |
| Monthly refresh | Refreshed categories for the rest of the database | Replace the table or upsert the changed rows |
Keep it atomic: load each update into a staging table, check row counts, then swap it in. Your product never queries a half loaded table.
Licensing basics
The data is licensed, not sold outright, and the licence states who may use it, inside which systems and for which customers. Full terms are on the licensing page.
Internal use
- Use inside your own organization and systems.
- Security, analytics, enrichment, ad operations.
- Derived outputs such as block lists for your own campaigns.
- No redistribution to third parties.
OEM and embedded
- Data inside a product or service you sell.
- Firewalls, DNS filters, gateways, parental controls, ad platforms.
- Your customers use the product, not the raw files.
- Terms set per deployment.
Group licences
- For companies with several products or business units.
- One agreement, several deployments.
- Simplifies renewals and updates.
- Scope agreed in writing.
Always excluded: no licence allows resale or publication of the raw data. Aggregated statistics and derived decisions, such as blocking a domain, are normal outputs.
Privacy and security review
Security and legal teams usually ask the same handful of questions. The short answers are below, and they make a vendor review quick.
Personal data
None. The rows describe websites: domain, categories and domain level metadata.
Data flows
Once loaded, lookups run in your own systems. Domains you check locally are not sent anywhere.
Offline operation
The files work in isolated and air-gapped networks, because nothing calls home.
Transfer security
Downloads use HTTPS, and checksums confirm the files arrived intact.
Agreements
NDAs for evaluation and data processing agreements can be discussed when your process needs them.
API use
The included API receives only the domain you ask about. Use local files where even that must stay inside.
Technical requirements for loading
The database needs no special software. These are the practical points engineers check before the first load, and none of them should block a typical team.
| Area | What to plan |
|---|---|
| Storage | Room for the snapshot, one update in staging and a backup copy |
| Memory | For in-memory lookups, enough RAM for the columns you keep, not the full file |
| Index | A unique index on the domain column; optional indexes on category columns for analysis |
| Domain normalization | Lowercase, strip protocol and path, decide how subdomains fall back to the registered domain |
| Text encoding | UTF-8 throughout, including internationalized domain names |
| Automation | A scheduled job that downloads, verifies, stages and swaps each update |
Common loading problems and quick fixes
Most first load problems come from a handful of causes. Each one has a quick fix that takes minutes once you know what to look for.
| Symptom | Likely cause | Fix |
|---|---|---|
| Row count lower than expected | Download cut short, or quotes inside a field broke the CSV parser | Verify the checksum, use a CSV parser that respects quoting |
| Garbled characters in some domains | File read with the wrong text encoding | Read as UTF-8 everywhere in the pipeline |
| Few lookups find a match | Lookups use full hostnames or URLs | Apply the domain matching rules above |
| Lookups slow after loading | No index on the domain column | Add a unique index, or load into a key value store |
| Personas column splits into extra columns | Delimiter handling in a spreadsheet tool | Import with explicit quoting, or use Parquet |
| Updates seem to undo each other | Delta files applied out of order | Apply in date order, after the latest full snapshot |
Procurement checklist
Use this list to collect the answers your purchasing process needs. Each item maps to a section above or to a documentation page.
- Number of domains needed: most visited head, full long tail, or a country cut
- Taxonomies needed: web filtering, IAB, or both
- Extra columns: country, personas, popularity bands
- File formats, including any security export formats
- Update cadence: weekly deltas, daily on request, monthly refresh
- Delivery: who downloads, where files land, checksum checks
- Licence type: internal, OEM or group, and which products are covered
- Where the data runs: cloud, on-premise or air-gapped
- API use for single lookups alongside the files
- Contract term and renewal, from the pricing page
Comparing several data sources at once? Our guide on how to evaluate a URL categorization database turns these points into RFP questions.
Frequently asked questions
What fields does each record contain?
One domain per row with its web filtering category, IAB categories in up to four tiers, country, personas and popularity bands where available. See the data dictionary.
Which file formats are available?
CSV, Parquet and JSON, plus ready made exports for security tools such as hosts files, DNS RPZ, external dynamic lists and PAC files.
How large are the files?
A few million domains are a gigabyte or two as uncompressed CSV; the full database is tens of gigabytes. Parquet and compressed archives are much smaller.
How are files delivered?
Through secure HTTPS download links with checksums. A lookup API is included for single domain checks.
How do updates arrive?
Weekly delta files for newly registered domains, daily on request, and a monthly refresh of the rest of the database.
Can I use the data inside my product?
Yes, with an OEM licence, while internal use licences cover your own systems. No licence allows resale or publication of the raw data.
Does the data include personal information?
No. It describes websites, not people: domains, categories and domain level metadata only.
Get the specification for your deployment
Tell us how many domains, which columns and which formats you need, and where the data will run. We reply with a written proposal covering coverage, delivery and licence terms.
IAB is a trademark of the Interactive Advertising Bureau, and AdGuard and Pi-hole are trademarks of their respective owners, named here only to identify the taxonomy and list formats discussed. URL Categorization Database is not affiliated with, sponsored or endorsed by any of them.