Domain List Download Formats
A domain list is only useful once it is loaded where you work. The right format makes that a five-minute job; the wrong one turns it into a weekend of fixing broken imports.
This guide compares the three formats we deliver, explains column choices and file sizes, and shows how buyers load lists into spreadsheets, databases and data platforms.
CSV, JSON and Parquet at a glance
All three carry exactly the same data. They differ in how easy they are to open, how they store lists of categories, and how much space they take. None of them is better in general; each is better for a particular kind of tool.
Comma-separated values
Plain text, one domain per line. Opens in anything, from a text editor to a database import tool.
- Most compatible format
- Easy to inspect by eye
- Several categories share one field
- Largest on disk
JSON Lines
One JSON object per line. Categories and personas are real arrays, so nothing needs splitting.
- Natural fit for code and APIs
- Arrays for multi-value fields
- Streams line by line
- Bigger than CSV, field names repeat
Apache Parquet
A compressed, columnar binary format. Tools read only the columns a query needs.
- Smallest files
- Fastest for analytics
- Typed columns and real lists
- Needs a data tool to open
The examples above use fictional domains and a shortened set of columns. The full field list is on the database of websites by category page.
The schema behind every format
Whatever the format, the columns and their meaning stay the same. This is how each field is typed, which matters most for Parquet and for database tables.
| Field | Type in CSV | Type in JSON and Parquet | Example value |
|---|---|---|---|
| Domain | Text | String | example-travel.es |
| Web filtering categories | Text with separator | List of strings | Travel, Business |
| IAB v2 Tier 1 to 4 | Text, one column per tier | String per tier | Travel > Travel Type |
| IAB v3 Tier 1 to 4 | Text, one column per tier | String per tier | Travel > Travel Locations |
| Personas | Text with separator | List of strings | Frequent Flyer, Adventure Seeker |
| Country | Text, may be empty | String or null | Spain |
| Global popularity group | Text label, may be empty | String or null | 1000001-5000000 |
| Country popularity group | Text label, may be empty | String or null | 100000 |
Which format should you choose?
Choose by the tool that will read the file first, not by habit. This table covers the common cases.
| You will load the list into | CSV | JSON | Parquet |
|---|---|---|---|
| A spreadsheet (small lists only) | Best | Poor | Not supported |
| An ad platform or DSP site list upload | Best | Rarely | Rarely |
| A relational database | Best | Good | Good with tools |
| A cloud data warehouse or lake | Good | Good | Best |
| A notebook for analysis | Good | Good | Best |
| Your own application or service | Good | Best | Good |
| A key-value store for real-time lookups | Good | Best | Good |
Ad operations teams
Usually CSV. Site lists go straight into campaign tools, and the files per segment stay small enough to check by eye.
Data and analytics teams
Usually Parquet. The full file sits next to logs and bid data in the warehouse, and queries read only the columns they need.
Product engineers
Usually JSON Lines or Parquet. Records feed a service that answers lookups inside their own product.
Security teams
Usually CSV plus a ready export for their filter or resolver, refreshed with daily delta files.
Choosing your columns
You do not have to take every field. Fewer columns mean smaller files, faster loads and simpler tables. Fields fall into four groups.
Core
- Domain
- Web filtering categories
Enough for filtering, blocklists and simple segmentation.
Advertising taxonomy
- IAB v2 Tier 1 to 4
- IAB v3 Tier 1 to 4
Needed for contextual targeting and ad platform integration.
Audience
- Personas
Useful for audience planning and segment building.
Market and size
- Country
- Global popularity group
- Country popularity group
For territory work, prioritising and cutting the long tail.
| Use case | Suggested columns |
|---|---|
| Network filtering or DNS policy | Domain, web filtering categories |
| Contextual advertising | Domain, IAB v2 or v3 tiers, country, popularity groups |
| Brand safety exclusion list | Domain, web filtering categories, IAB v3 Tier 1 and 2 |
| B2B lead generation | Domain, web filtering categories, IAB v3 tiers, country, popularity groups |
| Audience research | Domain, IAB v3 tiers, personas, country |
| Model training data | Domain, all category fields |
File sizes in general terms
Size depends on three things: how many rows, how many columns, and which format. These bars show the relative size of the same full file in each format, with all columns.
Bars are indicative, not exact; real ratios depend on the columns you pick. Category fields compress very well because the same values repeat across millions of rows, which is why Parquet and gzip shrink these files so much.
Full database
Over 120 million rows. Tens of gigabytes as uncompressed CSV with all columns, and a fraction of that as Parquet.
One country
From well under 100,000 rows for small markets to several million for the largest ones. Usually megabytes to a few gigabytes.
One category
From tens of thousands to millions of rows, depending on the category. See counts on websites by category.
Core columns only
Domain plus web filtering categories is a small share of the full width, so even the full row count stays manageable.
How buyers load domain lists
Most loads follow one of these paths. Pick the one that matches your stack. Whichever you choose, load into a fresh table first and swap it in once checks pass, so users never query a half-loaded list.
Spreadsheets
Fine for lists up to a few hundred thousand rows. Most spreadsheet programs stop at about one million rows per sheet, so split or filter larger files first.
Relational databases
Use the database's bulk import command for CSV rather than row-by-row inserts. Add an index on the domain column right after the load.
Cloud warehouses
Upload Parquet to object storage and create an external or native table. Columns and types come straight from the file.
Notebooks and scripts
Data frame libraries and embedded analytical databases read Parquet directly and query it without a full load.
Real-time services
Load domain and the fields you need into an in-memory key-value store. Lookups then take a fraction of a millisecond.
Ad platforms
Export a single-column list of domains per segment from your copy, then upload it as an inclusion or exclusion list.
| Tool type | Typical load method | Tip |
|---|---|---|
| PostgreSQL | COPY from CSV | Create the table first with text columns, then index the domain |
| MySQL or MariaDB | LOAD DATA INFILE from CSV | Set the file encoding to UTF-8 explicitly |
| DuckDB | Query the Parquet file in place | No import step needed for analysis |
| Apache Spark | Read Parquet into a data frame | Partition by country for faster country filters |
| Python data frames | Read Parquet or CSV | Load only needed columns to save memory |
Details that break imports
The same handful of problems cause most failed loads. All of them are easy to avoid once you know them.
Character encoding
Files are UTF-8. Opening them as another encoding garbles category names with accents and symbols such as "&".
Multi-value fields
In CSV, several categories share one field with a separator. Split on that character, not on commas, which already separate columns.
Internationalised domains
Domains in non-Latin scripts appear in their ASCII form starting with "xn--". Keep them as text; do not convert them during import.
Empty fields
Empty Tier 3, country or popularity fields are expected. Import them as empty or null, never as zero or "unknown".
Labels, not numbers
Popularity groups are labels such as "1-1000", not numbers. Import them as text so spreadsheet tools do not turn them into dates.
Line endings
Some older tools expect Windows line endings. Most modern importers handle both, but check if rows appear merged.
Compression and splitting
Big files travel better compressed and in parts. These options keep downloads quick and loads predictable.
Gzip or zip
Text formats compress to a small share of their size. Many import tools read gzip directly, with no unpacking step.
Split by size
Very large files can arrive in numbered parts, so one failed download does not mean starting again.
Split by key
Files per country or category let each team load only its own slice, and let warehouses skip whole files in queries.
Full snapshots and delta files
Data that changes needs a delivery rhythm. Two patterns cover nearly every buyer, and a third combines them. The choice depends on how quickly your system has to reflect a new or changed domain.
Full file each refresh
Replace your table with the new file at each regular refresh. Simple, safe and easy to audit.
- Best for analytics and research
- Easy rollback: keep the last file
Only new and changed rows
Upsert the daily changes into your table, keyed on the domain. Small files and quick loads.
- Best for live filtering and ad systems
- Pairs with the newly registered domains list
Snapshot plus deltas
Load a full snapshot now and then, apply deltas in between. The most common setup for production systems.
- Fresh data every day
- Clean baseline at each refresh
Ready-to-use export formats for security tools
Licensed customers who feed filters and resolvers can also receive category lists shaped for those tools, so no conversion script is needed.
| Export | What it is | Typical consumer |
|---|---|---|
| Hosts file | Domains mapped to a non-routable address | Endpoints and simple local blocking |
| DNS RPZ | Response policy zone for resolvers | DNS servers and protective resolvers |
| Ad-blocker style lists | One rule per domain in a common blocklist syntax | Home and small office DNS filters |
| External dynamic lists | Plain domain lists hosted for a firewall to fetch | Firewalls and secure gateways |
| PAC file | Proxy auto-config script that routes by domain | Browsers in managed environments |
How teams use these exports is covered in DNS filtering security and enterprise network security.
Joining a domain list to your own data
Most buyers do not use the list on its own. They join it to logs, CRM records or bid data. The format matters less than getting the join key right.
Normalise your side
Lowercase hosts, remove "www.", ports and paths. The list is already normalised, so only your data needs the treatment.
Exact match first
Join on the full host. Many subdomains of large platforms are listed on their own because they act as separate sites.
Root fallback
For hosts with no exact match, join again on the registrable domain. Keep a flag showing which join produced the match.
Index the key
An index or sort on the domain column makes joins against hundreds of millions of log lines practical.
Single domains can be checked any time with a website category lookup, which is handy for testing a join on a few known examples.
Load checklist
Run through this list on the first delivery. After that, most of it can be automated as a short script that runs after every download and stops the swap if a check fails.
| Check | Why it matters |
|---|---|
| Row count matches the delivery note | Catches truncated downloads and broken imports |
| Checksum matches | Confirms the file arrived intact |
| Accented category names display correctly | Confirms UTF-8 was used on import |
| Multi-value fields split into the right number of values | Confirms the separator was handled |
| Empty fields are null, not zero | Prevents wrong counts and filters later |
| Domain column is unique and indexed | Keeps joins fast and correct |
| A handful of known domains look right | A quick sense check before production use |
Delivery and handling
Download
Files are made available for secure download. Large files come compressed, and can be split into parts.
Your systems only
Once downloaded, the data runs entirely inside your environment, including closed networks without outside access.
Integrity checks
Compare row counts and file checksums after each download before swapping a new file into production.
Licence terms
Use is internal or inside your own product. Resale or publication of the raw files is not allowed. See pricing.
Domain list formats: questions
Which formats are domain lists delivered in?
CSV, JSON and Parquet. CSV suits spreadsheets and simple imports, JSON suits applications, and Parquet suits data warehouses and large analytical jobs.
How big is a full domain list file?
The full database of over 120 million domains runs to tens of gigabytes as uncompressed CSV. Parquet files are a fraction of that size. Subsets by category or country are much smaller.
Can I open a domain list in a spreadsheet?
Small lists, yes. Most spreadsheet programs stop at about one million rows per sheet, so larger lists belong in a database or data tool.
Can I choose which columns I get?
Yes. You can take only the domain and categories, or add IAB tiers, personas, country and popularity groups as needed.
How are multiple categories stored in one row?
In CSV they share one field with a separator character. In JSON and Parquet they can be stored as real lists, which avoids splitting text when you load the file.
Do you deliver updates as full files or changes?
Both are possible. Full snapshots are simplest to reload, and delta files with only new and changed domains suit daily updates such as newly registered domains.
Which compression do you use?
Large CSV and JSON files are delivered compressed, commonly as gzip or zip. Parquet files are compressed internally, so they need no extra step.
Can I get one file per country or category?
Yes. Files can be split by country, category or popularity group, which helps when different teams need different slices.
Can I get a different separator for multi-value fields?
For CSV, the separator for categories and personas can be agreed with you, so it never clashes with your own tools.
Do domains include "www." or "https://"?
No. Domains are delivered bare, such as example.com, so they join cleanly to normalised data on your side.
Can I check a few domains before choosing a format?
Yes. Use a website category lookup or explore the database to see the fields on real examples.
Get your domain list in the format you need
Tell us the categories, columns and format that fit your stack. We will size the file and send the options.
Request a domain list Browse categoriesPostgreSQL, MySQL, MariaDB, DuckDB, Apache Spark and Apache Parquet are trademarks of their respective owners. They are named here only to describe compatible tools. URL Categorization Database is not affiliated with, sponsored or endorsed by any of them.