URL database update schedule and freshness
A category database is only as useful as it is current. New sites go live every day, young sites change after launch and older sites drift, get parked or change hands.
This page explains the rhythm that keeps our data current, and how to plan your own updates around it.
The life of a new domain
Every newly registered domain passes through three looks before it settles into the monthly cycle. The rechecks exist because a domain on day one rarely shows what it will become.
First review
The new domain is reviewed and categorized from what it shows. Many show a placeholder, a parking page or nothing at all at this stage.
First recheck
A week later the domain is checked again. Shops, blogs and company sites often have real content by now, and the category is updated.
Second recheck
After a month, slow launches and sites that changed purpose are caught. Some domains also turn risky once they start receiving traffic.
Monthly cycle
The domain joins the monthly refresh with the rest of the database, so later changes in topic or status are picked up.
Why this matters to you: a database that looks at new domains only once will label many of them parked or empty forever. Rechecks turn those early guesses into the category the site actually grew into.
What changes, and how often
Different parts of the web change at very different speeds. The update rhythm follows that, putting effort where change is fastest.
| Part of the web | How fast it changes | How it is kept current |
|---|---|---|
| Newly registered domains | Very fast: content appears, changes or vanishes within days | Daily review, rechecks after a week and a month |
| Short lived risky domains | Very fast: phishing and scam sites often live for days | Caught in the daily review and rechecks, shipped in delta files |
| Small and long tail sites | Moderate: lapses, parking and owner changes | Monthly refresh |
| Established popular sites | Slow: the main subject rarely changes | Monthly refresh, catches pivots and new sections |
| Expiring domains | Event driven: a domain lapses and may be re-registered for a new purpose | Monthly refresh, and treated as new when re-registered |
How updates reach you
Customers start with a full snapshot and keep it current with delta files. A typical month looks like the calendar below, with the monthly refresh on top.
Weekly delta files
- Standard for every licence that includes updates.
- Contain newly categorized domains and domains changed by rechecks.
- Small enough to apply in minutes.
- Suit advertising, analytics, enrichment and research.
Daily delta files, on request
- Same content at daily granularity.
- Suit phishing and malware protection, DNS filtering and security analytics.
- Shorten the window where a new risky domain is unknown to you.
- Need an automated loading job on your side.
File layouts, formats and delivery steps are described in URL database specification and delivery, and the import side in the integration guide.
Which cadence fits your use
Not every team needs daily files. The right cadence depends on how much harm a few days of staleness causes in your product.
Why young domains need a second and third look
The rechecks are the part of the schedule buyers ask about most. These are the patterns they exist to catch.
Slow launches
A shop or company site often goes live weeks after the domain is registered.
Parking that ends
A domain parked on day one may become a real site, or stay parked for years.
Late turning scams
Some sites show harmless content first and switch to phishing pages later.
Changed plans
Owners pivot: a blog becomes a shop, or a project domain becomes a landing page.
Sites that vanish
Many new domains never get content and stop resolving. Rechecks keep them from lingering as live sites.
Growing scope
A site that starts with one topic often adds more. Rechecks add the extra ranked categories.
Seasonal sites
Event, holiday and campaign sites appear quickly and often go quiet after a few weeks. Rechecks follow both the launch and the fade.
Language and country
A new site may launch in one language and add others later. Rechecks update the country attribution when the evidence changes.
Fast moving fields
Fields such as AI tools see many launches every week. Rechecks sort real products from placeholder pages in those busy categories.
If new domains are your main interest, the newly registered domains list describes that dataset on its own.
The monthly refresh
Beyond new domains, the whole database is refreshed every month. This is what keeps established sites accurate as the web slowly shifts.
| Change caught | Example | Effect in your copy |
|---|---|---|
| Topic shift | A technology blog adds an AI tools section that becomes its main content | Ranked categories reordered or extended |
| Became parked | A small business closes and the domain shows a parking page | Moves to Parked / For-Sale Domains |
| Ownership change | An expired domain is bought and relaunched with new content | Category replaced to match the new site |
| Stopped resolving | The domain lapses and returns nothing | Removed or flagged as inactive |
| Popularity movement | A site gains or loses many visitors | Popularity bands updated |
| Country attribution | A site adds a local language version and audience | Country field updated where the evidence changed |
Applying updates without surprises
A few habits keep your local copy in step with ours. Most of them are one-time setup work for your data team.
Automate the pull
Schedule a job that downloads each delta, verifies its checksum and loads it.
Apply in order
Apply deltas by date on top of the latest snapshot, so newer rows win.
Upsert by domain
Insert new domains and replace changed ones using the domain as the key.
Stage and swap
Load into a staging table and swap, so lookups never see half an update.
Watch the numbers
Track rows added per update. A sudden drop usually means a failed download.
Reload now and then
A periodic full reload clears any drift built up from missed or partial updates.
When a domain is not in your copy yet
Even with daily files, you will meet domains that were registered hours ago. Decide in advance how your system treats them.
Ask the API
- The lookup API included with your licence returns the current category for single domains.
- Good for low volume gaps and for spot checks.
- Cache answers locally until the next delta arrives.
Apply a default policy
- Security tools often treat uncategorized domains with caution or a warning page.
- Ad platforms often skip them, since they carry little traffic yet.
- Analytics teams label them "new" and revisit after the next update.
Tip: log the domains that miss locally. If the same domains keep missing after several deltas, send them to [email protected] so they can be reviewed.
Three scenarios: freshness in practice
Schedules are easier to judge with real situations in mind. These invented examples show how the rhythm plays out for three kinds of customers.
A DNS filter for small offices
- Loads daily delta files every morning through an automated job.
- A phishing domain registered on Monday appears in Tuesday's file once categorized.
- Until then, the filter shows a caution page for uncategorized domains.
- The week and month rechecks catch domains that turn malicious after a harmless start.
An ad platform's brand safety team
- Loads weekly delta files every Monday and the monthly refresh at month start.
- New risky domains rarely carry ad traffic in their first week, so weekly is enough.
- Parked domains that turn into real publishers become eligible after their rechecks.
- Block lists are rebuilt monthly from the refreshed data.
A university research group
- Takes one full snapshot at the start of the study and keeps it frozen.
- Records the snapshot date and licence terms in the paper's methods section.
- For a follow up on change over time, takes a second snapshot months later and compares both with the same rules.
- Never mixes delta files into the frozen snapshot, so results can be repeated.
Freshness and accuracy go together
A fast update that carries wrong labels does not help anyone. The schedule is paired with checks that keep the labels trustworthy as they change.
Manual validations
Regular manual validations measure accuracy on samples of categorized domains, including new ones.
Content based
Domains are categorized from site content, then reviewed and validated, rather than guessed from their names.
Ranked labels
When a site covers several subjects, categories are ranked by how much of the site each one describes.
Customer reports
Customers can report a domain they believe is wrong by email, and it is reviewed.
Update terms in plain words
Contracts and documentation use a few terms that are easy to mix up. Here is what each one means on this site.
| Term | Meaning |
|---|---|
| Snapshot | A complete copy of every domain in your licence at one point in time |
| Delta file | Only the rows added or changed since the previous file |
| Newly registered domain | A domain registered recently, reviewed in the daily cycle |
| Recheck | A repeat review of a new domain after one week and after one month |
| Monthly refresh | The regular review of all other domains in the database |
| Uncategorized | A domain your copy does not contain yet, or one with no content to judge |
| Upsert | Insert a row if the domain is new, replace it if it already exists |
Freshness questions to ask any data source
Update claims are easy to make and hard to compare. These questions turn them into something you can check, and they work for any provider.
| Question | Why it matters | Our answer |
|---|---|---|
| How many new domains are reviewed per day? | Shows whether the new web is covered or sampled | About 300,000 |
| Are new domains checked again? | Without rechecks, early placeholder labels stick | After one week and one month |
| How often do established domains refresh? | Drift builds slowly but steadily | Monthly |
| How do updates arrive? | Full reloads only are heavy and slow | Weekly delta files, daily on request |
| How is accuracy measured over time? | Freshness without accuracy is not useful | Regular manual validations |
| Can I see a domain's current category on demand? | Covers the gap between files | Lookup API included with licences |
For a full buyer checklist beyond freshness, see how to evaluate a URL categorization database and the category data quality checklist. Coverage by popularity tier is described on coverage and freshness.
Freshness for research and fixed studies
Not every user wants the newest data every week. Researchers and analysts often need the opposite: a fixed snapshot that does not move while a study runs.
Freeze a snapshot
Keep the snapshot you started with, and record its date in your methods section.
Compare snapshots
For trend work, compare monthly snapshots built with the same rules.
Separate new domains
Treat recently registered domains as their own group, since their categories settle after the rechecks.
More on this in website categories for web measurement research.
Signs your copy has gone stale
A local copy can fall behind quietly when a job fails or a licence lapses. These symptoms usually show up before anyone checks the load logs.
| Symptom | What it usually means |
|---|---|
| The share of uncategorized lookups rises week after week | Delta files are not being applied |
| Users report blocks on sites that clearly changed topic | Monthly refreshes are missing |
| Parked domains keep appearing in ad placement reports | The copy predates the refresh that marked them |
| Row counts have not changed in weeks | The update job stopped or loads empty files |
| Lookup API answers differ from local answers for many domains | Local data is several updates behind |
Setting up your update job: a checklist
Your side of freshness is a small, reliable job. Teams that tick these boxes once rarely think about updates again.
Pick the cadence
Weekly for most uses, daily for security. Match the job schedule to the files you receive.
Store access safely
Keep download links and API keys in your secrets store, not in scripts.
Verify every file
Compare checksums before loading. Stop the job on any mismatch.
Alert on failure
Send an alert if a file is missing, late or much smaller than usual.
Keep the last good copy
Hold the previous version so you can roll back if a load goes wrong.
Name an owner
One person or team owns the job and reads its alerts.
What freshness cannot do: no schedule makes a domain known before it has content to judge. A clear policy for uncategorized domains is always part of a sound setup.
Frequently asked questions
How often is the database updated?
About 300,000 newly registered domains are reviewed and categorized every day, delivered in weekly delta files or daily on request. The rest of the database is refreshed monthly.
Why are new domains checked again after a week and a month?
Many sites launch as a placeholder and add content later. The rechecks make sure the category reflects what the site became.
What is in a delta file?
Domains added or changed since the previous file: newly categorized domains plus domains changed by their rechecks. Apply it by updating rows by domain.
Do I need daily updates?
Daily files suit security uses such as phishing protection. Weekly files suit most advertising, analytics and research work.
What happens between monthly refreshes?
Established sites change slowly, so their categories stay valid for months. The monthly refresh catches pivots, parking and lapses.
How do I handle a domain not yet in my copy?
Use the included lookup API, or apply a default policy for uncategorized domains. It usually appears in a following delta file.
How is accuracy kept up over time?
Regular manual validations measure accuracy on samples. Rechecks and monthly refreshes keep categories in line with what sites show today.
Data that keeps up with the web
Over 120 million categorized domains, about 300,000 new domains reviewed daily with week and month rechecks, weekly delta files and a monthly refresh.