URL Categorization API vs Database: Which One Fits Your Use Case?
You can get website categories two ways: ask a provider's API one domain at a time, or license the whole categorized database and run it yourself. The right answer depends on volume, speed, privacy and how much engineering you want to own, and for many teams it turns out to be a mix of both. This guide compares both honestly.
Live API
Send a domain, get categories back over the network.
- Quick to start
- Pay per call
- Provider runs everything
Licensed database
Load the full file into your own systems and look up locally.
- Fastest lookups
- Fixed licence cost
- Data never leaves your network
How each option works
Categorization API
Every lookup is a request to the provider. You hold an API key, send a domain or URL and receive categories as JSON. Usage is metered, usually per call or per monthly quota.
Categorization database
You download the categorized domains as CSV, JSON or Parquet, load them into a key-value store, a database table or memory, and apply update files on a schedule.
What one lookup returns, either way
| Field | Example for coursera.org | Typical use |
|---|---|---|
| IAB v3 tiers | Education > Online Education | Contextual targeting, reporting |
| IAB v2 tiers | Education > Online Education | Partners still on the older version |
| Web filtering labels | Education/Reference, Webinars/E-learning | Filtering, safety, data cleaning |
| Personas | Lifelong Learner, Adult Learner, Online Learner | Audience-style planning |
| Country and popularity | United States; global group 1,001 to 5,000 | Market splits, reach estimates |
Our licences include both: the full file for your own systems and a lookup API for ad hoc checks. What the file contains is described in the database of websites by category.
Head-to-head comparison
The table compares the two on the points buyers ask about most. A tag marks the option that usually comes out ahead, though your own volume and set-up can change the picture.
| Factor | Live API | Licensed database |
|---|---|---|
| Lookup speed | Network round trip per call, plus timeouts and retries | Local, often microseconds from memoryusually wins |
| Throughput | Limited by rate limits and quotas | Limited only by your hardwareusually wins |
| Cost model | Per call or quota; grows with volume | Fixed licence; cost per lookup falls with volume |
| Time to first result | Minutes: get a key and call itusually wins | Hours to days: load and integrate the file |
| Coverage | Whatever the provider has, sometimes with on-demand checks for unknown domains | Everything in the file, over 120 million domains in ours |
| Freshness | Reflects the provider's latest state | As fresh as your last update: monthly refresh plus daily new domains |
| Privacy | Provider sees every domain you look up | No lookup leaves your networkusually wins |
| Offline and air-gapped | Not possible | Fully supportedusually wins |
| Bulk analysis | Slow and costly for millions of rows | One join in your warehouseusually wins |
| Engineering effort | Low: one HTTP callusually wins | Moderate: storage, loading and updates |
| Dependence on provider uptime | Every lookup depends on it | Only updates depend on itusually wins |
| Reproducibility | Answers can change between calls | Same file, same answer; easy to auditusually wins |
volume is low or unpredictable, lookups are interactive, a short delay is acceptable and you want to start today without storing anything.
lookups sit in a real-time path, volume runs into millions, data must stay inside your network, or you need to analyse large lists in bulk.
production systems need speed and privacy while analysts, support staff and small tools need quick one-off answers.
your volume grows tenfold, a new product needs inline decisions, or a customer asks where lookups are processed.
Latency: where milliseconds decide
For many tasks a few hundred milliseconds do not matter. For some, they decide whether the lookup can happen at all. Ask where your lookup sits: in the path of a user request, or in a background job.
Real-time bidding
An auction often allows around 100 milliseconds end to end for everything. A remote call per bid request is rarely affordable, so bidders keep categories in memory.
DNS and web filtering
Every page load waits for the decision. Filters keep category data local and treat remote checks only as a fallback.
CRM and form enrichment
A lead can wait a second for its industry to appear. An API call at form submission works well here.
For adtech set-ups, see programmatic advertising and RTB; for filtering, see DNS filtering and security.
Cost: how each option scales with volume
APIs are cheap to start and grow linearly. A licence is a fixed cost that does not care how many lookups you make. Somewhere along the volume axis the lines cross.
Illustration only, not to scale. Actual break-even depends on your volume and plan.
- Count lookups honestly: every bid request, DNS query or page view that needs a category counts
- Remember repeated lookups of the same popular domains still cost money on a metered API
- Add the engineering cost of storing and updating a file, which is real but usually one-off
- Include the cost of a slow or failed lookup, such as a lost bid or a blocked page
- Bulk backfills, such as enriching a CRM or a crawl, can use a whole year's API budget in a day
Plans and volumes are listed on our pricing page.
Coverage and freshness: what to really compare
Coverage and freshness are where API and file differ less than people expect, as long as the file is updated properly. The real differences sit in the details below, so ask about each one before you sign.
Long-tail coverage
A file can only answer for domains in it. Check how deep it goes beyond the most popular sites, since logs and crawls are full of small domains.
New domains
Fresh domains are where scams, parked pages and new brands appear. Daily delta files close most of the gap with an API.
Re-checks
New sites often add content weeks after launch. Ask whether new domains are checked again later. Ours are re-checked after a week and after a month.
Refresh cycle
Established domains change subject slowly. A monthly full refresh keeps them current for almost every use.
Unknown lookups
Whatever you choose, some lookups will find nothing. Decide upfront how your system treats an unknown domain.
Same taxonomy everywhere
Make sure API answers and file rows use the same category names and IDs, so mixed set-ups stay consistent.
Our approach is described in coverage and freshness, and the daily feed in newly registered domains list.
Decision guide by use case
| Use case | Best fit | Why |
|---|---|---|
| Real-time bidding and ad serving | Database | Sub-millisecond lookups inside the auction, no per-call cost at bid volume |
| DNS filter, firewall, secure gateway | Database with daily updates | Inline decisions, privacy of user browsing, works if the provider is unreachable |
| Bulk CRM or warehouse enrichment | Database | One join instead of millions of calls |
| Enrichment at form submission | API | Low volume, simple integration, a second of delay is fine |
| Analyst checks and support tools | API | Occasional single lookups from a browser or script |
| Filtering crawls for AI training | Database | Billions of URLs, data must stay in your cluster |
| Security investigations | Both | File for bulk log scans, API for one-off checks during triage |
| Prototypes and proofs of concept | API first | Fast to start; move to the file once volume is clear |
Related guides: domain enrichment for CRM and filtering web crawls for AI training data.
The hybrid pattern most teams end up with
In practice the question is rarely "API or database". It is which lookups go where. The pattern below keeps every time-critical and high-volume lookup local, while still giving people a quick way to check a single domain.
Load the file
Keep all categorized domains in a fast local store.
Look up locally
Every real-time and bulk lookup hits the local store first.
Handle misses
Unknown domains get a default policy and are logged.
Use the API sparingly
Interactive tools and analysts use the API for one-off checks.
Apply updates
Daily deltas and monthly refreshes keep the store current.
Implementation tips for running the database yourself
Pick the right store
A hash map in memory for bidders and filters, a key-value store for shared services, a warehouse table for analytics.
Normalise once
Lower-case, strip "www." and reduce to the registrable domain before every lookup, using the same rules as the file.
Keep only what you need
A filter may need one label per domain; an ad platform may need IAB tiers and personas. Load only those columns.
Swap atomically
Load updates into a new table or map, then switch over in one step so lookups never see half a file.
Choose the format
Parquet for warehouses, CSV for simple loaders, JSON for document stores. See download formats.
Watch your miss rate
A rising share of unknown lookups usually means a normalisation bug or a missed update, not missing data.
Common myths about APIs and database files
Both options carry some outdated assumptions, often formed years ago when files were updated quarterly and APIs were the only way to get fresh answers. These are the ones we hear most from buyers.
A file that applies daily delta files is at most a day behind for new domains, and established domains change slowly.
Accuracy comes from how the data is built and validated, not from how it is delivered. The same data can sit behind both.
Over 120 million rows sounds large, but with only the columns you need it fits comfortably in a modern warehouse or key-value store.
Production use still needs caching, retries, rate-limit handling, timeouts and a fallback when the API is unreachable.
Most mature set-ups use the file for volume and the API for interactive checks.
A cache only holds domains you have already seen. A file also answers for the first visit to any domain it contains.
Security, compliance and procurement angles
Beyond speed and cost, the choice often comes down to what your security and legal teams will approve. Security products, banks, public bodies and health organisations in particular tend to prefer data that never leaves their own environment, and some cannot use a remote lookup at all.
| Concern | Live API | Licensed database |
|---|---|---|
| Data sent to a third party | Every looked-up domain, which can reveal user browsing or customer lists | None during lookups |
| Vendor security review | Needed for a system in your request path | Lighter: a file delivery, not a live dependency |
| Data residency | Depends on where the provider runs | Wherever you store the file |
| Outage impact | Lookups fail or slow down | Lookups continue; only updates pause |
| Audit trail | Answers may change between calls | Versioned file dates make results reproducible |
Moving from an API to a file
Measure
Count monthly lookups and distinct domains from your API logs.
Map fields
Match API response fields to file columns and category IDs.
Run in parallel
Look up locally and via API for a week and compare answers.
Switch
Point production lookups at the local store with a default for misses.
Keep the API
Leave it in place for analysts and one-off checks.
Questions to ask any category data provider
Mixed set-ups are common, so the same data in both forms saves a second contract.
Ask about daily additions and whether new domains are checked again later.
IAB versions 2 and 3 and a security-oriented set in one row make integration simpler.
Essential for some security, government and regulated buyers.
Internal use, embedding in a product and group-wide use are different licences.
Ask how validation is done and whether results are shared with licence holders.
For a full evaluation framework, read our category data quality checklist, or test individual domains with the domain checker.
API vs database: frequently asked questions
What is the difference between a URL categorization API and a database?
An API answers one lookup at a time over the network. A database is a licensed file you load into your own systems and query locally.
Which is faster, an API or a local database?
A local database. In-memory lookups take microseconds, while an API adds a network round trip and timeout risk.
Which is cheaper?
It depends on volume. APIs suit low or unpredictable volume; a licence becomes cheaper per lookup as volume grows.
Is a database file as fresh as an API?
It can be close. Monthly full refreshes plus about 300,000 new domains a day as delta files keep a file current for most uses.
Can I use both?
Yes. The file covers bulk and real-time lookups, and the included lookup API covers occasional checks.
Which option is better for privacy?
A local database, because no lookup leaves your network and no third party sees which domains are visited.
How much engineering does a database file need?
For most teams, a loader that reads the file into a table or key-value store, a small job that applies daily updates, and a normalisation function shared with the lookup code. Once built, it rarely needs attention.
Can a browser extension or mobile app use a database?
Usually not the full file, because of size. Such products tend to ship a small list of the most relevant domains and call an API or their own backend, which holds the full file, for everything else.
What happens when a domain is not in the file?
Your system applies the default you choose, such as allow, block or "review later", and logs the miss. Daily updates add new domains, so many misses resolve within a day or two.
Is the API included with the database licence?
Yes. Our licences include a lookup API alongside the downloadable files, so you can use both without a second contract.
Get the file and the API in one licence
Over 120 million categorized domains as CSV, JSON or Parquet, daily new-domain updates, monthly refreshes and a lookup API included.
IAB is a trademark of the Interactive Advertising Bureau. URL Categorization Database is not affiliated with, sponsored or endorsed by IAB.