Brotli Shared Dictionary Compression Results

The test fetched each of the URLs using an anonymous connection with no cookies and the requested user agent string:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36

Then, for each HTML response, it generated an HTML dictionary using the responses from all of the other URLs (excluding the URL being evaluated) to see how well a dictionary generated from the other pages would compress the page being tested.

Summary

Brotli supports a compression scale from 1 to 11 and Zstandard uses a scale from 1 to 22.

Brotli starts using external dictionaries for compression at level 5. Below those levels, dictionary-based compression is equivalent to not using dictionaries. The bulk of the savings for dictionary-based compression with brotli is already realized at level 5.

Zstandard supports external dictionaries starting at level 1. The bulk of the savings for using dictionary-based compression with Zstandard is usually seen by level 3 with incremental gains beyond that.

Using a custom 1024 KB compression dictionary with brotli for the requested pages resulted in HTML that was 7% to 58% smaller than brotli alone (31% average).

A dictionary was created using all of the provided URLs for future use. You can download it here. It is 886,129 bytes (87,714 compressed with brotli 11).

Run a new test

Details

https://news.ycombinator.com/news

Zstandard level 10 with a dictionary is 59% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	36,162	6,080	17%	6,080	17%	100% (0% smaller)
Brotli 5	36,162	4,982	14%	2,096	6%	42% (58% smaller)
Brotli 11	36,162	4,421	12%	1,867	5%	42% (58% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	36,162	5,781	16%	4,744	13%	82% (18% smaller)
Zstandard 2	36,162	5,699	16%	2,646	7%	46% (54% smaller)
Zstandard 3	36,162	5,575	15%	2,482	7%	45% (55% smaller)
Zstandard 10	36,162	5,359	15%	2,188	6%	41% (59% smaller)
Zstandard 19	36,162	5,156	14%	1,961	5%	38% (62% smaller)
Zstandard 22	36,162	5,156	14%	1,961	5%	38% (62% smaller)

https://news.ycombinator.com/newest

Zstandard level 10 with a dictionary is 25% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	43,472	7,081	16%	7,081	16%	100% (0% smaller)
Brotli 5	43,472	5,887	14%	4,683	11%	80% (20% smaller)
Brotli 11	43,472	5,141	12%	4,088	9%	80% (20% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	43,472	6,836	16%	5,883	14%	86% (14% smaller)
Zstandard 2	43,472	6,735	15%	5,299	12%	79% (21% smaller)
Zstandard 3	43,472	6,680	15%	5,322	12%	80% (20% smaller)
Zstandard 10	43,472	6,349	15%	4,778	11%	75% (25% smaller)
Zstandard 19	43,472	6,095	14%	4,477	10%	73% (27% smaller)
Zstandard 22	43,472	6,095	14%	4,477	10%	73% (27% smaller)

https://news.ycombinator.com/front

Zstandard level 10 with a dictionary is 49% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	35,813	6,277	18%	6,277	18%	100% (0% smaller)
Brotli 5	35,813	5,163	14%	2,685	7%	52% (48% smaller)
Brotli 11	35,813	4,583	13%	2,414	7%	53% (47% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	35,813	6,016	17%	5,071	14%	84% (16% smaller)
Zstandard 2	35,813	5,874	16%	4,188	12%	71% (29% smaller)
Zstandard 3	35,813	5,803	16%	3,148	9%	54% (46% smaller)
Zstandard 10	35,813	5,555	16%	2,825	8%	51% (49% smaller)
Zstandard 19	35,813	5,369	15%	2,532	7%	47% (53% smaller)
Zstandard 22	35,813	5,369	15%	2,532	7%	47% (53% smaller)

https://news.ycombinator.com/newcomments

Zstandard level 10 with a dictionary is 30% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	44,532	9,046	20%	9,046	20%	100% (0% smaller)
Brotli 5	44,532	7,794	18%	5,773	13%	74% (26% smaller)
Brotli 11	44,532	6,513	15%	5,122	12%	79% (21% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	44,532	8,785	20%	7,349	17%	84% (16% smaller)
Zstandard 2	44,532	8,683	19%	6,740	15%	78% (22% smaller)
Zstandard 3	44,532	8,561	19%	6,405	14%	75% (25% smaller)
Zstandard 10	44,532	8,316	19%	5,780	13%	70% (30% smaller)
Zstandard 19	44,532	8,130	18%	5,515	12%	68% (32% smaller)
Zstandard 22	44,532	8,130	18%	5,515	12%	68% (32% smaller)

https://news.ycombinator.com/ask

Zstandard level 10 with a dictionary is 36% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	30,381	4,788	16%	4,788	16%	100% (0% smaller)
Brotli 5	30,381	3,802	13%	2,538	8%	67% (33% smaller)
Brotli 11	30,381	3,284	11%	2,257	7%	69% (31% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	30,381	4,434	15%	3,548	12%	80% (20% smaller)
Zstandard 2	30,381	4,329	14%	3,039	10%	70% (30% smaller)
Zstandard 3	30,381	4,271	14%	2,962	10%	69% (31% smaller)
Zstandard 10	30,381	4,158	14%	2,643	9%	64% (36% smaller)
Zstandard 19	30,381	3,996	13%	2,416	8%	60% (40% smaller)
Zstandard 22	30,381	3,996	13%	2,416	8%	60% (40% smaller)

https://news.ycombinator.com/show

Zstandard level 10 with a dictionary is 36% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	35,012	5,740	16%	5,740	16%	100% (0% smaller)
Brotli 5	35,012	4,662	13%	3,118	9%	67% (33% smaller)
Brotli 11	35,012	4,122	12%	2,820	8%	68% (32% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	35,012	5,468	16%	4,495	13%	82% (18% smaller)
Zstandard 2	35,012	5,354	15%	3,880	11%	72% (28% smaller)
Zstandard 3	35,012	5,251	15%	3,534	10%	67% (33% smaller)
Zstandard 10	35,012	5,005	14%	3,218	9%	64% (36% smaller)
Zstandard 19	35,012	4,864	14%	2,964	8%	61% (39% smaller)
Zstandard 22	35,012	4,864	14%	2,964	8%	61% (39% smaller)

https://news.ycombinator.com/jobs

Zstandard level 10 with a dictionary is 35% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	24,285	4,151	17%	4,151	17%	100% (0% smaller)
Brotli 5	24,285	3,352	14%	2,336	10%	70% (30% smaller)
Brotli 11	24,285	2,981	12%	2,117	9%	71% (29% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	24,285	4,050	17%	3,200	13%	79% (21% smaller)
Zstandard 2	24,285	3,989	16%	2,864	12%	72% (28% smaller)
Zstandard 3	24,285	3,950	16%	2,729	11%	69% (31% smaller)
Zstandard 10	24,285	3,643	15%	2,379	10%	65% (35% smaller)
Zstandard 19	24,285	3,568	15%	2,207	9%	62% (38% smaller)
Zstandard 22	24,285	3,568	15%	2,207	9%	62% (38% smaller)

https://news.ycombinator.com/item?id=41542409

Zstandard level 10 with a dictionary is 57% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	16,736	4,058	24%	4,058	24%	100% (0% smaller)
Brotli 5	16,736	3,206	19%	1,510	9%	47% (53% smaller)
Brotli 11	16,736	2,752	16%	1,366	8%	50% (50% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	16,736	3,852	23%	2,186	13%	57% (43% smaller)
Zstandard 2	16,736	3,780	23%	1,911	11%	51% (49% smaller)
Zstandard 3	16,736	3,729	22%	1,723	10%	46% (54% smaller)
Zstandard 10	16,736	3,580	21%	1,535	9%	43% (57% smaller)
Zstandard 19	16,736	3,490	21%	1,433	9%	41% (59% smaller)
Zstandard 22	16,736	3,490	21%	1,433	9%	41% (59% smaller)

https://news.ycombinator.com/item?id=41550364

Zstandard level 10 with a dictionary is 16% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	157,491	24,289	15%	24,289	15%	100% (0% smaller)
Brotli 5	157,491	20,237	13%	17,537	11%	87% (13% smaller)
Brotli 11	157,491	17,015	11%	15,375	10%	90% (10% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	157,491	23,071	15%	21,341	14%	93% (7% smaller)
Zstandard 2	157,491	22,552	14%	20,367	13%	90% (10% smaller)
Zstandard 3	157,491	22,425	14%	19,688	13%	88% (12% smaller)
Zstandard 10	157,491	20,873	13%	17,499	11%	84% (16% smaller)
Zstandard 19	157,491	19,894	13%	16,224	10%	82% (18% smaller)
Zstandard 22	157,491	19,894	13%	16,224	10%	82% (18% smaller)

https://news.ycombinator.com/item?id=41519229

Zstandard level 10 with a dictionary is 25% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	90,266	14,506	16%	14,506	16%	100% (0% smaller)
Brotli 5	90,266	12,018	13%	9,516	11%	79% (21% smaller)
Brotli 11	90,266	10,202	11%	8,447	9%	83% (17% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	90,266	13,880	15%	13,158	15%	95% (5% smaller)
Zstandard 2	90,266	13,625	15%	11,223	12%	82% (18% smaller)
Zstandard 3	90,266	13,365	15%	10,726	12%	80% (20% smaller)
Zstandard 10	90,266	12,563	14%	9,485	11%	75% (25% smaller)
Zstandard 19	90,266	11,985	13%	8,818	10%	74% (26% smaller)
Zstandard 22	90,266	11,985	13%	8,818	10%	74% (26% smaller)

https://news.ycombinator.com/item?id=41547395

Zstandard level 10 with a dictionary is 9% smaller than without a dictionary.

Brotli Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Brotli 1	461,863	58,142	13%	58,142	13%	100% (0% smaller)
Brotli 5	461,863	47,567	10%	44,301	10%	93% (7% smaller)
Brotli 11	461,863	39,962	9%	38,194	8%	96% (4% smaller)

Zstandard Compression

Compression Level	Original Size	Compressed	Relative Size	With Dictionary	Relative to Original	Relative to Compressed
Zstandard 1	461,863	55,540	12%	53,162	12%	96% (4% smaller)
Zstandard 2	461,863	53,916	12%	50,903	11%	94% (6% smaller)
Zstandard 3	461,863	52,520	11%	49,849	11%	95% (5% smaller)
Zstandard 10	461,863	47,967	10%	43,835	9%	91% (9% smaller)
Zstandard 19	461,863	45,229	10%	40,954	9%	91% (9% smaller)
Zstandard 22	461,863	45,229	10%	40,954	9%	91% (9% smaller)