Real-World Comparisons
The only convincing way to demonstrate the compression benefits of one
image format over another is to do an actual comparison of the two on
a set of real images. The problem is choosing the set of
images--what works for one person may not work for another. What
I've done here is to gather together results from a number of
real-world tests performed over the past few years. Readers can
expect to achieve similar results on similar sets of images, but keep
in mind that one can always choose a particular set of images
for which the results will be dramatically different. I'll explain
that remark after we see a few cases.
For starters, let's look at a small, very unscientifically chosen set
of images: seven nonanimated GIF images that happened to be the only
ones readily available on my machine one fine day in June 1998.
Table 9-2.
Seven Non-Animated, Non-Scientifically Selected GIF Images
|
Name |
GIF Size |
linux-penguins |
38,280 |
linux-tinypeng |
1,249 |
linux_bigcrash |
298,529 |
linux_lgeorges |
20,224 |
linux_rasterman |
4,584 |
sun-tinylogo |
1,226 |
techweb-scsi-compare |
27,660 |
TOTAL |
391,752 |
The images ranged in size from just over a kilobyte to nearly 300
kilobytes (the exact byte sizes are given in Table 9-2) and
in dimension from 32 × 32 to 800 × 600. All but the first
and last were interlaced. When converted to PNG with the gif2png
utility (Chapter 5, "Applications: Image Converters"), preserving
interlacing manually and introducing no new text annotations, things
improved somewhat; Table 9-3 shows the
preliminary results.
Table 9-3.
Same Seven GIF Images After Conversion to PNG
|
Name |
PNG Size |
Change |
linux-penguins |
35,224 |
-8.0% |
linux-tinypeng |
722 |
-42.2% |
linux_bigcrash |
283,839 |
-4.9% |
linux_lgeorges |
20,476 |
+1.2% |
linux_rasterman |
4,812 |
+5.0% |
sun-tinylogo |
566 |
-53.8% |
techweb-scsi-compare |
20,704 |
-25.1% |
TOTAL |
366,343 |
-6.5% |
Five of the images shrank when converted to PNG--three of them quite
significantly--while two grew. Overall, the set achieved a 6.5% improvement
in byte size. Since gif2png uses the standard settings of the
libpng reference code,[73]
its results may be considered typical of ``good'' PNG encoders. But the owner
of a web site will often be willing to spend a little more time up front on
compression in return for additional bandwidth savings in the long run. That's
where pngcrush (also discussed in
Chapter 5, "Applications: Image Converters")
comes in. Table 9-4 shows its results;
the percentages are again relative to the original GIF file sizes.
Table 9-4.
Same Seven GIF Images After PNG Conversion and Optimization
|
Name |
Optimized PNG Size |
Change |
linux-penguins |
34,546 |
-9.8% |
linux-tinypeng |
710 |
-43.2% |
linux_bigcrash |
282,948 |
-5.2% |
linux_lgeorges |
19,898 |
-1.6% |
linux_rasterman |
4,731 |
+3.2% |
sun-tinylogo |
550 |
-55.1% |
techweb-scsi-compare |
19,155 |
-30.7% |
TOTAL |
362,538 |
-7.5% |
So we see that the current state-of-the-art PNG encoder ekes out
another percentage point in the total size, with all but one of the
images now smaller than the original. That lone holdout is worth a
closer look in this case. I already noted that
linux_rasterman.gif was one of the interlaced images; suppose it
had not been? The noninterlaced GIF version is 4,568 bytes, only 16
bytes smaller than the original. But the noninterlaced PNG version is
either 4,067 bytes (gif2png) or 4,000 bytes (pngcrush)--a savings of
11.0% or 12.4% over the noninterlaced GIF. In other words, PNG's
two-dimensional interlacing scheme can have a significant negative
impact on compression, particularly for small images. This is an
important point to consider when creating images: is interlacing
really needed for a 152 × 96 image (as in this case) when the
penalty is more than 18% of the file size?
This example may have been instructive, but seven images do not constitute a
statistically valid sample.[74]
So let's consider a few more data sets. One real-life example comes from
the course entitled ``Authoring Compelling and Efficient VRML 2.0 Worlds''
at the VRML98 conference in Monterey, California. Though the content of the
course was otherwise outstanding, one slide comparing image formats for 3D
textures was rather alarming from a PNG perspective. It showed the information
displayed in Table 9-5, together with the textures themselves (which are omitted here):
Table 9-5.
Original PNG, GIF, and JPEG Comparison from VRML98 Course
|
Name |
Dimensions |
Type |
JPEG Size |
GIF Size |
PNG Size |
linoleum1 |
128 × 128 |
grayscale |
10,956 |
7,055 |
16,008 |
doggie |
128 × 256 |
color |
9,897 |
24,605 |
89,022 |
fog |
128 × 128 |
grayscale + alpha |
-- |
-- |
26,732 |
circlefade |
128 × 128 |
grayscale + alpha |
-- |
-- |
15,735 |
buttfly |
128 × 128 |
color + transparency |
-- |
4,367 |
-- |
Even with no more details than are shown here, at least one problem is
apparent: in row 2, the JPEG image is 24 bits deep, while the GIF is only
8 bits. Judging by the size of the corresponding PNG, one might assume
(correctly) that the PNG is also 24 bits. So on the one hand, PNG is being
compared with an image of the same depth that uses lossy compression, while
on the other it is being compared with an image only one-third as deep, which
also amounts to lossy compression. That hurts.
As it turned out, there were other problems: the PNG images were created
with an image editor not known for its compression capabilities, and some
of the PNGs were interlaced even though their GIF counterparts were not.
(And since this was a VRML course, I should note that no VRML browser in
existence actually uses interlacing to render textures progressively, so
there is generally no point in creating such images.) The upshot is that
all of these factors--JPEG's lossy compression, mixing 24-bit and 8-bit
images, mixing interlaced and noninterlaced images, and using a particularly
poor encoder to compress the PNGs--worked against our favorite image format.
After evening the playing field by using the GIFs as the source images for
the PNGs, turning off interlacing, and using a combination of conversion and
encoding tools (including pngcrush), the results were considerably
better for PNG, as shown in Table 9-6.
Table 9-6.
Updated PNG, GIF, and JPEG Comparison for VRML98 Course Images
|
Name |
JPEG Size |
GIF Size |
Original PNG Size |
Optimized PNG Size |
PNG Change |
linoleum1 |
10,956 |
7,055 |
16,008 |
6,753 |
-57.8% |
doggie |
9,897 |
24,605 |
89,022 |
22,555 |
-74.7% |
fog |
-- |
-- |
26,732 |
16,221 |
-39.3% |
circlefade |
-- |
-- |
15,735 |
6,638 |
-57.8% |
buttfly |
-- |
4,367 |
-- |
3,965 |
-- |
Here, I've marked the smallest version of each image
in boldface type; the only one that isn't a PNG is the color JPEG, which is
hardly surprising. What is interesting is that the grayscale JPEG
(linoleum1.jpg) is
larger than both the GIF and optimized PNG versions, despite the presumed
benefits of lossy compression. There are at least three reasons for this.
First, GIF and PNG both get an automatic factor-of-three savings from the fact
that each pixel is only 1 byte deep instead of 3 bytes. Second, JPEG
is at a relative disadvantage when dealing with grayscale images, because most
of its compression benefits arise from how it treats the color components
of an image. Third, this particular image is more artificial than natural,
with quite a few relatively sharp features, which makes it particularly
ill suited to JPEG-style compression.
But perhaps the most striking feature of Table 9-6 is just how poorly the original
encoder did on its PNG
images. Realizable savings of 40% to 75% are unusual, but thanks to
poor encoding software, they are not as unusual as one might hope.
As another real example (but one that is perhaps more representative
of what a typical web site might expect), the owner of
http://www.feynman.com/ found that
when he converted 54
nonanimated GIFs to PNGs, the collection grew in size from 270,431
bytes to 327,590 bytes. Insofar as all of the original images had
depths of 8 bits or less--and even the worst PNG encoder will, on
average, do as well or better than GIF on colormapped PNG
images--the most likely explanation for the 21% increase in size is
that the conversion utility produced 24-bit RGB PNGs. Indeed, the
owner indicated that he had used ImageMagick's convert utility, older
versions of which reportedly had the unfortunate habit of creating
24-bit PNGs unless explicitly given the -depth 8
option. (This problem seems to have been fixed in more recent versions,
but even current versions include 160 bytes' worth of text and background
chunks per image.) When the original GIFs were converted to PNG with gif2png
instead, the total size dropped to 215,668 bytes, for a 20% overall
savings over GIF. Individually, the GIFs were smaller in 15 of the 54
cases, but never by more than 340 bytes. Of the 39 images in which
the PNG version was smaller, one-third of them differed by more than a
kilobyte, and one was 14 KB smaller.
For the last GIF comparison, I downloaded the World Wide Web
Consortium's icon collection, consisting of 448 noncorrupted GIF
images. Of these, 43 had embedded text comments and 39 were
interlaced. Most of the images were icon-sized (as would be
expected), at 64 × 64 or smaller, but there were a handful of
larger images, too. The total size of the files was 1,810,239 bytes.
Conversion to PNG via gif2png, handling the interlaced and
noninterlaced images separately in order to preserve their status,
resulted in a total PNG size of 1,587,337 bytes, or a 12.3% reduction.
Additional compression via pngcrush resulted in a total of 1,554,965
bytes, or a 14.1% reduction (relative to the GIF size). Out of the
448 images, PNG won the size comparison in 285 cases, lost in 161
cases, and tied in 2 cases. As in the previous test, however, the
magnitude of the differences was the critical factor: GIF won by more
than a kilobyte in only 1 case, while PNG won by that amount in 37
cases--4 of which were greater than 10 KB, 1 more than 64 KB. The
average difference for the 285 cases in which PNG was smaller was 940
bytes; for the 161 GIF cases, it was a mere 78 bytes.
Finally, I've mentioned an upcoming JPEG standard for lossless compression
a couple of times; it's worth a quick look, too. JPEG-LS, as the standard
will be known,[75]
is based on Hewlett-Packard's LOCO-I algorithm. As this is written, it
is implemented in version 0.90 of HP's locoe encoder, available only
in binary form for the HP-UX, Solaris, IRIX, and 32-bit Windows
platforms. (An independent implementation is available as C source
code from the University of British Columbia.) In a comparison
performed by Adam Costello, the HP encoder was tested against pnmtopng
and pngcrush on the eight standard color images in the Waterloo
BragZone's ColorSet. pnmtopng is of interest only for speed reasons;
even though it is moderately fast, locoe was considerably faster. I
have omitted its size results from the comparison since, as expected,
pngcrush outperformed it in all cases, though at a considerable cost
in speed.
The results were fascinating. In the five test images categorized by the
University of Waterloo as ``natural,'' JPEG-LS beat PNG by between 5% and
11%--not a huge difference, but certainly significant. However, in the
three images marked ``artistic,'' PNG proved superior by wide margins, with
one image more than three times smaller than the corresponding JPEG-LS version.
These results are summarized in Table 9-7; once again, the byte size of the
winning format for each image is highlighted in boldface type.
Table 9-7.
PNG and JPEG-LS Comparison for Waterloo BragZone Color Images
|
Classification |
Name |
Total Pixels |
JPEG-LS Size |
PNG IDAT Size |
Relative Difference |
``natural'' |
lena |
262,144 |
445,799 |
475,430 |
+6.6% |
monarch |
393,216 |
555,012 |
615,260 |
+10.9% |
peppers |
262,144 |
385,047 |
425,560 |
+10.5% |
sail |
393,216 |
767,374 |
808,606 |
+5.4% |
tulips |
393,216 |
616,536 |
680,881 |
+10.4% |
``artistic'' |
clegg |
716,320 |
653,299 |
484,589 |
-25.8% |
frymire |
1,235,390 |
935,285 |
251,865 |
-73.1% |
serrano |
499,426 |
293,532 |
106,765 |
-63.6% |
Note that in the final column I used the JPEG-LS size as the
reference, which effectively works against PNG--had I used PNG
instead, the frymire image, for example, would show JPEG-LS as
271.3% larger, which looks much more impressive! Also note that I
used the size of the PNG IDAT data for comparison rather than the
actual PNG file size; this was done because locoe appears to encode
raw JPEG data, with none of the overhead of standard JPEG file formats
like JFIF and SPIFF.
In any case, the results are only slightly more statistically valid than
the first comparison of GIF images was. Eight samples, even if they are a
carefully chosen set of standard research images, cannot tell the full
story. And results as intriguing as these certainly deserve more extensive
testing, which will no doubt happen in due course.
|