[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Of interest: File services via web Major Net Cache Vendors CompeteHead-to-Head in Performance Test; Results Released at Fourth InternationalWeb Caching Workshop (fwd)
Subject: MEDIA ALERT: Major Net Cache Vendors Compete Head-to-Head in
Performance Test; Results Released at Fourth International Web Caching
Workshop
SUMMARY:
Six major sources of Web Cache systems participated in a head-to-head
competition sponsored by NLANR (a National Science Foundation networking
research and support organization) -- IBM, InfoLibria, Network Appliance,
Novell (in an OEM agreement with Dell), the University of Wisconsin, and
NLANR itself. The competition was open to any company or organization with
a Web cache product. Detailed performance results are available on the Web.
FOR IMMEDIATE RELEASE
April 5, 1999
For more information, contact:
Duane Wessels, NLANR, 303-497-1822, <wessels@ircache.net>
Web Cache Competition: http://bakeoff.ircache.net/
Web Caching Workshop: http://workshop.ircache.net/
Report: http://bakeoff.ircache.net/bakeoff-01/
NLANR: http://www.nlanr.net/
Major Net Cache Vendors Compete Head-to-Head in Performance Test:
Results Released at Fourth International Web Caching Workshop
UNIVERSITY OF CALIFORNIA, SAN DIEGO -- The results of the first Web Cache
"Bakeoff" competition were released at the Fourth International Web Caching
Workshop in San Diego on April 2. The competition was organized by the
IRCACHE team of the National Laboratory for Applied Network Research
(NLANR), an independent research and support organization for
high-performance networking funded by the National Science Foundation.
Six major sources of Web Cache systems -- IBM, InfoLibria, Network
Appliance, Novell (in an OEM agreement with Dell), the University of
Wisconsin, and NLANR itself -- participated in the competition, which was
open to any company or organization with a Web cache product. Several other
vendors, while initially interested, declined to participate in a
head-to-head comparison.
"A 'bakeoff' is a test of several similar products that perform similar
functions," said NLANR's Duane Wessels, organizer of the competition. "In
this case, we evaluated the performance of systems that speed up
information retrieval over the Internet -- Web cache servers available from
commercial vendors and non-profit organizations."
More than half of the traffic on the Internet backbone is related to the
World Wide Web. In the basic client-server transaction model, each Web
browser client connects directly to each server, so a single file often
would be transmitted many times over the same network paths when requested
by different clients. This mode of operation can cause severe congestion on
many of the Internet's wide-area links.
Caching is a way to reduce client-server traffic and improve response time.
Instead of connecting directly to servers, clients are configured to
connect to an "HTTP proxy server" at the Internet Service Provider (ISP),
which requests Web objects from their source servers (or from other caches)
and then saves these files for use in response to future requests. Popular
objects collect in the caches and might be used many times without
reloading from remote sites.
NLANR operates the IRCACHE Web Caching project under funding from the
National Science Foundation's Directorate for Computer and Information
Sciences and Engineering. NLANR created and maintains nine high-level Web
caches located throughout the United States. These caches are directly
connected to approximately 400 other caches, and indirectly to 1100
worldwide. Collectively, the NLANR caches receive approximately 7,000,000
requests per day from the others. The operational system of intermeshed Web
caches has been credited with significantly reducing Internet congestion.
NLANR also developed and distributes the Squid open source software
package, which is widely used in the ISP community.
The Bakeoff Competition
=======================
Several Web cache products have entered the marketplace within the past
couple of years. Unfortunately, competing performance claims hare been
difficult to verify, and haven't meant quite the same thing from vendor to
vendor.
Wessels and his colleagues Alex Rousskov and Glenn Chisholm have developed
a freely available software package called Web Polygraph. Polygraph
simulates Web clients and servers and is becoming a de facto benchmarking
standard for the Web caching industry. The package is designed to give Web
*servers* a workout -- it generates about 1000 requests per second on a
100baseT network between a client-server pair, and can specify such
important workload parameters as hit ratio, response sizes, and server-side
delays.
"To ensure a valid comparison, every product was tested under identical
conditions within a short period of time and at a single location," Wessels
explained. The Web Cache Bakeoff competition was held March 15 through 17
in Redwood City, CA, in an industrial space donated for the occasion by
Paul Vixie of Vixie Enterprises (http://www.vix.com/vix/).
Seemingly minor variations in workload parameters or system configuration
can markedly affect performance. "In our report, we specify as much detail
as possible about our benchmarking environment so the results of our tests
can be reliably reproduced by others," Wessels said.
The bake-off took place over three days. The first day was used for testing
the network and computer systems. The next two days were dedicated to
running the benchmark. Theoretically, all benchmarking could have been
finished by the end day two, so the third day was a saftey net.
Participants also had the option of repeating some runs if necessary.
Each vendor was allowed to bring more than one product to the bake-off;
each tested product was considered an independant participant, with a
separate benchmarking harness (bench) for every participant. More than 80
Compaq Pentium II computers were rented for use as Polygraph clients and
servers in the bakeoff.
IBM, InfoLibria (two entries), Network Appliance (two entries), Novell (in
OEM agreement with Dell, two entries), the University of Wisconsin, and
NLANR participated in the competition, which was open to any company or
organization with a Web cache product. The precise parameters of the test
were arrived at by discussion and mutual agreement among the competitors
and researchers.
"The bakeoff set a high standard both for design and execution, and for the
cache robustness required for completion," said Abdelsalam A. Heddaya,
InfoLibria's VP of Research and Architecture. "Because it was the first
truly independent benchmark of network caches, we believe it will be of
tremendous value to the industry."
CacheFlow, Cisco, Entera, and Inktomi had expressed strong interest in the
bakeoff, but eventually decided not to participate. In addition, IBM and
Network Appliance decided not to disclose the results of their trials
after the bakeoff, a "bail-out" option previously agreed upon by the
competitors and testers.
"Certainly we are disappointed by their choice," Wessels said. "We feel
that benchmarking results are more useful when there are more results to
compare. At the same time, we take it as a compliment that our benchmark
was taken very seriously -- by those who competed, and by those who didn't."
The Competitors
===============
* IBM -- IBM brought a 34C00 Web Cache Manager system to the bakeoff.
* InfoLibria (Large configuration) -- InfoLibria's "large" bakeoff entry
was a cluster of four DynaCache IL-100-7 servers
(http://www.infolibria.com/products/f-dyna.htm).
* InfoLibria (Small configuration) -- InfoLibria's "small" entry was a
single DynaCache IL-200X-14 (http://www.infolibria.com/products/f-dyna.htm).
* Network Appliance (Large configuration) -- Network Appliance entered a
cluster of three C720S NetCache Appliances as their "large" solution.
* Network Appliance (Small configuration) -- Network Appliance's "small"
solution was a single C720S NetCache Appliance.
* Novell/Dell (Large configuration) -- Novell/Dell's "large" entry was the
Novell Internet Caching System (Beta version), running on a Dell PowerEdge
6350 (http://www.novell.com/products/nics/).
* Novell/Dell (Small configuration) -- Novell/Dell's "small" entry was the
Novell Internet Caching System (Beta version), running on a Dell PowerEdge
2300 (http://www.novell.com/products/nics/).
* Peregrine v 1.0 -- The University of Wisconsin brought their Peregrine
software package, running on a Dell PowerEdge 2300
(http://www.cs.wisc.edu/~cao/).
* NLANR Squid -- NLANR brought their Squid software package, running on a
generic Pentium-II system (http://squid.nlanr.net/).
The Results
===========
The results of the Web Cache Bakeoff were released on April 2 at the Fourth
International Web Caching Workshop in San Diego. The conference was
organized by NLANR and CAIDA (the Cooperative Association for Internet Data
Analysis). Detailed information about the competition and the formal report
of its results can be found at http://bakeoff.ircache.net/bakeoff-01/.
There is no single absolute measure of performance for all situations --
some customers will place the highest value on throughput, while others
emphasize bandwidth or response time savings. Maximizing cache hit ratio is
essential at many Web cache installations. For some sites price is
important, for others it's the price/performance ratio.
The following table summarizes the results.
Maximum Mean Achieved
Sustained Response Hit
Throughput Time Ratio
Cache (req/sec) (seconds) (%)
InfoLibria (large) 1680 1.4-4.2 14-55
InfoLibria (small) 690 1.3-3.7 9-55
Novell/Dell (large) 1500 1.4-1.7 47-55
Novell/Dell (small) 400 1.6-1.7 47-51
Peregrine 620 1.4-3.0 55
Squid 96 1.4-6.0 39-55
IBM and Network Appliance declined to make their results public.
Detailed results of each test, with graphs and analyses, are contained in
the report (at http://bakeoff.ircache.net/bakeoff-01/).
"We strongly caution against drawing hasty conclusions from these
benchmarking results," Wessels said. "Since the tested caches differ a
lot, it is tempting to draw conclusions about participants based on a
single performance graph or pricing table. We believe such conclusions will
virtually always be wrong."
"Our report contains a lot of performance numbers and configuration
information; take advantage of it," he continued. "Compare several
performance factors: throughput, response time, and hit ratio, and weigh
each factor based on your preferences. Don't overlook pricing information
and price/performance analysis. And always read the Polyteam and
Participant Comments sections in the Appendices.
"Our benchmark addresses only the performance aspects of Web cache
products. Any given cache will have numerous features that are not
addressed here. For example, we think that manageability, reliability, and
correctness are very important attributes that should be considered in any
buying decisions."
Most customers also have to consider price in their decision making
process. The report summarizes the pricing of the participating products
and gives detailed product configurations. Note that these costs represent
list prices of the equipment only. In reality, there are many additional
costs of owning and operating a Web cache. These may include
software/technical support, power and cooling requirements, rack/floor
space, etc. A more thorough cost analysis might try to determine something
like a two-year cost of ownership figure.
The Competitors' Views
======================
In the interest of fairness, all of the competitors were invited to comment
on the results. Participant comments contain forward-looking statements
concerning, among other things, future performance results. All such
forward-looking statements are, by necessity, only estimates of future
results and actual results achieved by the Participant may differ
substantially from these statements.
We have received the following information from the participating vendors:
* InfoLibria
(Press release at http://www.infolibria.com/press/f-pressrel.htm)
The testing showed Infolibria's DynaCache to be an extremely robust and
high-performance network caching solution. DynaCache achieved: (1)
Sustained peak throughput that [...] is many times faster than any
previously documented cache performance, and amply meets the most punishing
demands of leading national ISPs. (2) Response time that is assured to
remain low throughout DynaCache's entire performance range, via a
patent-pending dynamic connection pass-through mechanism. (3) Expanded
bandwidth delivered to Web clients by 22-24% (with a payback, based on
bandwidth savings alone, of five to nine months).
These results show DynaCache addressing the rigorous demands of the ISP
marketplace. But a caching product must also exhibit a number of equally
key characteristics that were not measured in the bakeoff for it to meet
the intense demands of the Internet environment. An Internet-grade cache
must maintain the integrity of content, be fail-safe and easily manageable,
and not impede ecommerce transactions or the collection of visitor data.
DynaCache is the only network cache available with all of these attributes.
* Novell/Dell
(Press release at http://www.novell.com/press/archive/1999/03/pr99029.html)
The Novell Internet Caching System (NICS) used in these tests will soon be
available from Dell and other Novell OEMs. This new product is a headless
appliance that blends high-performance and scalability with web-based
remote management and a rack-mount form factor. These results were produced
with Dell's beta version of "Novell Internet Caching System Powered By
Dell."
"NLANR's Web Cache Bakeoff results substantiate Novell's leadership postion
in the Web caching market," commented Drew Major, vice president and chief
scientist, Novell. "The Web Polygraph benchmark, which is representative of
real-world Internet useage scenarios, was very challenging for Novell.
Since the bakeoff we have already begun making improvements to the Novell
Internet Caching System and our customers can expect even better
performance when it ships through Compaq and Dell channels."
"It is of great industry benefit to have the NLANR Web Polygraph tool as an
unbiased standard measurement that customers can use to determine which
cache solution provides the best performance," commented Ron Lee, manager,
Advanced Development Performance Labs, Novell. "We're particularly proud of
our mean response time results. While our competitors' systems showed
significant response time degradation under increased loads, Novell's
Internet Caching System response times remained almost unchanged from the
lowest request rates to the heaviest peak loads. These results suggest that
customers can rely on Novell caching solutions for predictable and reliable
Web cache perfomance."
* University of Wisconsin Peregrine version 1.0
The Peregrine proxy is built as a part of the WisWeb research project
funded by the National Science Foundation. The proxy is currently under
evaluation by the University of Wisconsin-Madison for campus wide Web
caching. The system will be available by September 1999. Though the system
used in the bakeoff employs an Alteon ACEdirector, the same performance can
be achieved with a Netgear FS516 Fast Ethernet switch.
Performance of Peregrine systems can be scaled linearly via clustering. For
example, four Pentium-based Peregrine systems connected with a
load-balancing switch such as Alteon's CacheDirector or Foundry Networks'
FastIron can offer potential throughput up to four times the throughput
reported here. The designers of Peregrine avoided the "healing mode" in
order to maximize the bandwidth savings of the cache, but the mode can be
turned on via a configuration parameter. During the bakeoff, the system was
able to handle experiments at request rate of 630-650 req/sec, though at
such rates, about 2% of client connections were refused due to overload.
The version of Peregrine tested during the bakeoff does not preserve cache
contents between proxy restarts; adding this essential feature may affect
its performance.
A note on Peregrine v 1.0: At this time, the developers of Peregrine have
requested permission from the University of Wisconsin to distribute their
software at no cost. There is a chance, however, that Peregrine will not be
free, and price/performance evaluations should take this possibility into
account.
* NLANR Squid
NLANR's Squid Internet Object Cache is freely available software (available
under the GNU General Public License) that runs on all modern versions of
Unix. Duane Wessels is building and integrating the code as a worldwide
collaboration, and puts it into public trust. Squid has been constantly
evolving since 1996, and includes many features not found in commercial
caching products. An estimated 5-10,000 installations use Squid --
commercial ISPs, corporations, organizations, universities, and K-12
schools.
"We were not surprised by Squid's performance at the bakeoff," Wessels
said. "We have known for quite some time that the Unix filesystem is our
major bottleneck. This is an example of the price that Squid pays for
being highly portable. However, we are actively working on a filesystem
API for Squid that will allow us to experiment with new types of storage
for Squid (e.g. "SquidFS"), while still remaining compatible with the Unix
filesystem."
* Others
Several of the competitors who declined to participate or disclose their
results nevertheless were quite positive about the value of the
competition, and indicated a willingness to participate in the next
bakeoff, tentatively scheduled for this Fall.
"IBM is delighted to begin working with the IRCACHE team in shaping an
industry standard web caching benchmark. With the exponential growth of
World Wide Web traffic, Web caching will be an important component of
industrial strength Web infrastructures," said Nancy Ann Coon, worldwide
marketing manager for the IBM Web Cache Manager. "We look forward to future
participation in IRCACHE sponsored web caching benchmarks."
CacheFlow has supplied this statement: "CacheFlow declined to participate
in this first round of tests but will likely participate in future tests as
the Polygraph tool evolves. In order to produce benchmark information for
CacheFlow's products that is very meaningful to customers, CacheFlow chose
to wait for a later version of Polygraph that more completely models the
variable, realistic nature of the Internet.
"Polygraph is a good start toward a benchmark that will accurately
represent how an Internet accelerator performs in a livenetwork
environment. CacheFlow continues to actively participate in discussions
with the NLANR team and other vendors to help evolve the tool further."
Conclusions
===========
The Web caching industry's thirst for a benchmarking standard led to the
creation of the Web Polygraph suite and the launch of a series of IRCACHE
bakeoffs. We consider the first bakeoff to be a success. Despite the
absence of several big players in the industry, the IRCACHE team collected
a representative set of interesting performance data, and prepared the
first industry document that provides a fair performance comparison of a
variety of caching proxies. We hope the performance numbers and our
analysis will be used by buyers and developers of caching products.
The IRCACHE team applauds the vendors who came to the bakeoff and disclosed
their results. We regret that other cache vendors did not show their
leadership. We certainly hope that more companies will participate in
future benchmarking events and will have the courage to disclose their
results.
We expect discussions of the bakeoff and its results to appear, possibly
including attempts to denounce bakeoffs in general. We believe that, while
not perfect, this first bakeoff's rules and workload give knowledgeable
customers a lot of valid, useful, and unique performance data. Future
bakeoffs will further improve the quality and variety of our tests. We do
not know of a better substitute for a fair same-rules, same-time
competition.
Finally, we expect some companies will try to mimic bake-off experiments in
private labs, and we certainly welcome such activities. We trust the reader
will be able to separate unsubstantiated speculations and semi-correct
bake-off clones from true performance analysis. If unsure about the
validity of vendor tests, consult this report and Polyteam members
directly.
The second Web Cache Bakeoff has been tentatively scheduled for six months
from now.
---
The National Laboratory for Applied Network Research (NLANR) has as its
primary goal to provide technical, engineering, and traffic analysis
support for NSF High-Performance Connections sites and high-performance
network service providers such such as Internet 2, Next Generation
Internet, the NSF/MCI vBNS, and STAR TAP. Founded by the National Science
Foundation's Computer and Information Science and Engineering Directorate
in 1995, NLANR is a "distributed national laboratory" with researchers and
engineers at the San Diego Supercomputer Center, the National Center for
Supercomputing Applications, the Pittsburgh Supercomputing Center, and the
National Center for Atmospheric Research, among other sites. See
http://www.nlanr.net/ for more information.
The Cooperative Association for Internet Data Analysis is a collaborative
undertaking among government, industry, and the research community to
promote greater cooperation in the engineering and maintenance of a robust,
scalable global Internet infrastructure. It is based at the San Diego
Supercomputer Center (SDSC) at the University of California, San Diego
(UCSD) and includes participation by Internet providers and suppliers, as
well as the NSF and the Defense Advanced Research Project Agency (DARPA).
CAIDA focuses on the engineering and traffic analysis requirements of the
commercial Internet community. Current priorities include the development
and deployment of traffic measurement, visualization, and analysis tools
and the analysis of Internet traffic data. For more information, see
http://www.caida.org/, or contact Tracie Monk, CAIDA, 619-822-0943,
<tmonk@caida.org>.
The San Diego Supercomputer Center (SDSC) is a research unit of the
University of California, San Diego, and the leading-edge site of the
National Partnership for Advanced Computational Infrastructure
(http://www.npaci.edu/). SDSC is sponsored by the National Science
Foundation through NPACI and by other federal agencies, the State and
University of California, and private organizations. For additional
information about SDSC, see http://www.sdsc.edu/, or contact Ann Redelfs at
SDSC, 619-534-5032, <redelfs@sdsc.edu>.
###
--
To unsubscribe, send email to majordomo@silug.org with
"unsubscribe silug-discuss" in the body.