[prev in list] [next in list] [prev in thread] [next in thread]
List: best-of-security
Subject: BoS: Firewalls.html
From: Julian Assange <proff () suburbia ! net>
Date: 1995-11-24 4:37:33
[Download RAW message or body]
Data Communications: Lab Tests
_________________________________________________________________
By David Newman, Data Communications, and Brent Melson, National
Software Testing Laboratories Inc.
_________________________________________________________________
CAN FIREWALLS TAKE THE HEAT?
Performance tests show that some Internet firewalls are stronger than others
Top Performers
Top Performer
_________________________________________________________________
Checkpoint Software Technologies' Firewall-1 finished first in most of
our performance tests, suggesting it can withstand even the heaviest
attacks from outsiders. Firewall-1 includes security features not
offered by most other packet-filtering devices, such as the ability to
monitor state information about all open circuits. Another plus:
Firewall-1's graphical user interface is a model of clarity.
_________________________________________________________________
Harris Computer Systems' Cyberguard uses both proxy-service and
packet-filtering mechanisms for a double dose of security. Despite the
performance penalty that proxy services usually extract, Cyberguard
turned in a very strong showing in our performance tests. Cyberguard
also was one of the few products that never lost a session, no matter
how heavy the load.
_________________________________________________________________
Trusted Information Systems' Gauntlet fared well when it came to
handling large numbers of transactions in a hurry. Among the
proxy-service offerings, Gauntlet is a model of transparency: It can
be configured to be completely invisible to end-users, and it allows
new proxies to be defined for applications developed or modified
in-house.
_________________________________________________________________
Lab Test
Global e-mail links. Low-cost WAN bandwidth. Cool Web sites. Internet
services bring a lot to the corporate networking table. Of course,
they also open the door to a host of uninvited guests who can
compromise the confidentiality of corporate data-not to mention the
job security of the poor soul responsible for keeping the network safe
and sound.
No wonder net managers are putting Internet firewalls at the top of
their security shopping lists. Firewalls offer a measure of protection
against invasion via the Internet. By controlling access between
corporate networks and the outside world, firewalls are built to keep
the bad guys out while still giving insiders the ability to tap
Internet resources.
That's a tall order, given the typical mix of Internet and customized
applications running on most corporate nets. The task is getting
tougher by the day, as more users make Internet access part of their
daily work routine. Along with keeping intruders at bay, firewalls
have to be strong enough to handle traffic from hundreds or thousands
of internal users without buckling under the strain.
To find out which products can stand up to the challenge, Data
Communications joined forces with the National Software Testing
Laboratories Inc. (NSTL, Conshohocken, Pa.) to conduct the most
comprehensive evaluation of commercial firewalls to date. Using a
custom application suite developed by NSTL just for this evaluation,
we subjected 15 leading firewalls to rigorous performance tests. We
looked at how firewalls handle a mix of Internet applications under
light, moderate, and heavy loads. We found out which firewalls handle
customized applications transparently-and which are more opaque. And
while we didn't specifically test the security provisions of Internet
firewalls, we did evaluate security in the context of our performance
and transparency tests.
Overall, the tests yielded some encouraging news about Internet
firewalls. Most products proved able to handle moderate traffic loads
sent across T1 (1.544-Mbit/s) links to the Internet. But there's
plenty of room for improvement. Our tests revealed a wide gap between
the fastest and slowest performers, and we found user interfaces on
some products to be cryptic at best.
Application transparency turned out to be a major issue-one that
prevented us from completing testing on two products. Features like
token authentication cards and specially modified client software add
an extra dose of security. But when use of these features is
mandatory, net managers who can't or won't change applications to
accommodate the firewall may be stuck with no solution. In our tests,
we were unable to complete testing on two products-the Borderware
Firewall Server from Border Network Technologies Inc. (Toronto) and
Connect: Firewall from Sterling Software Inc. (Irving, Texas)-because
both required changes to our test application that we couldn't make.
After poring over thousands of data points and hundreds of pages of
configuration manuals (not to mention downing gallons of coffee), we
selected three Internet firewalls as the pick of the pack. Top
Performer awards go to Firewall-1 from Checkpoint Software
Technologies Inc. (Lexington, Mass.), Cyberguard from Harris Computer
Systems Corp. Trusted Systems Division (Fort Lauderdale, Fla.), and
Gauntlet from Trusted Information Systems Inc. (TIS, Glenwood, Md.).
(Note that Firewall-1 also is sold as Solstice Firewall-1 by Sunsoft
Inc. [Mountain View, Calif.].)
We decided to focus our evaluation on performance rather than security
issues for one basic reason. Our take on security is that it's
inherently untestable: With attackers coming up with new tricks all
the time, there's no way to prove that a firewall can withstand all
forms of assault.
That said, however, security is the core function of a firewall, and
it affects all firewall-related issues, including performance and
application transparency. Before buying and installing a firewall,
network managers need to answer two fundamental questions: What needs
to be protected, and from whom must it be protected?
The answers seem self-evident: A firewall should secure an
organization's network resources and data, and those resources and
data should be protected from unauthorized access by anyone outside
the organization. Fair enough. But the Internet is only one of several
potential points of entry into the network. A firewall between the
corporate network and the Internet might keep 'Net surfers away, but a
dial-up communications server that's not connected to a firewall
offers intruders a clear path to the corporate vault. For this reason,
companies need to establish a security policy that covers all access
to and from the internal network. Such a policy should encompass
employee training, logging of access attempts (an important legal
requirement), and security technologies like token authentication
schemes and encryption (see "Token Authentication: The Safety Catch,"
May 1995).
OTHER SIDES
Restricted access is only one aspect of security. Most of the firewall
products we evaluated include "secured" kernels (see Table 1). A
secured kernel is a modified version of an operating system, one that
contains only those services needed to run the firewall. The idea
behind this approach is that attackers can't exploit services that
don't exist.
_________________________________________________________________
TABLE 1: THE FIREWALL FIELD
_________________________________________________________________
Vendor Product Architecture Kernel Other OSs
supported CPU tested/RAM/
disk space required Physical networks
supported/maximum
physical interfaces Console
access Logging Authentication schemes
supported Hardware
included Price (1)
_________________________________________________________________
Advanced Network and Services Inc. (ANS)
Reston, Va.
703-758-8700 Interlock 3.0 Application proxy Secured
Solaris 2.3 AIX Sparc 20/64 Mbytes/
400 Mbytes Ethernet, token ring, FDDI, 100Base-T/2 In-band,
out-of-band, dial-up Service, time, source, destination Kerberos,
SecureID, SSL Yes $21,000 to $28,000 per year for all
scenarios
_________________________________________________________________
Atlantic Systems Group (ASG)
Fredericton, New Brunswick
506-453-3505 Turnstyle Firewall
System 1.1 Packet filter Secured NetBSD SunOS, OSF/1, HP-UX,
BSD 66-MHz Intel 486/
16 Mbytes/250 Mbytes Ethernet, token ring, FDDI/2 Terminal Service,
time, source, destination SHTTP Yes $3,995/$10,000/
not applicable
_________________________________________________________________
Border Network Technologies Inc.
Toronto
416-368-7157 Borderware Firewall Server 3.0.1 Packet filter,
application proxy, network address translator, Internet server Secured
BSD Maintenance-mode kernel
with no networking
functions 90-MHz Intel Pentium/
32 Mbytes/500 Mbytes Ethernet, serial, dial-up/2 In-band, dial-up
Service, time, source, destination, packet trace MD5, SecureID No
$4,000/$7,000/$11,000
_________________________________________________________________
Checkpoint Software Technologies Inc.
Lexington, Mass.
617-859-9501 Firewall-1 1.2(2) Packet filter, application proxy,
network address translator SunOS 4.1.3 Solaris 2.3 Sparcstation 10/
32 Mbytes/10 Mbytes Ethernet, token ring, FDDI, 100Base-T,
100VG-AnyLAN, ATM, serial, dial-up/16 In-band, out-of-band, dial-up
Service, time, source, destination, packet trace MD5, Kerberos,
SecureID, SSL, SHTTP No $10,000 for all scenarios
_________________________________________________________________
Digital Equipment Corp. (DEC)
Maynard, Mass.
508-493-5111 Digital Firewall
for Unix 1.0 Packet filter, application proxy, network address
translator Digital Unix 3.2b (formerly DEC OSF/1) None 233-MHz Alpha/
64 Mbytes/1 Gbyte Ethernet, token ring, FDDI, ATM, serial,
dial-up/1,024 In-band, out-of-band, dial-up Service, time, source,
destination, packet trace Kerberos, SecureID, SSL, SHTTP; Socks
optional No $24,995/$24,995/$31,990
_________________________________________________________________
Harris Computer Systems Corp.
Trusted Systems Division
Fort Lauderdale, Fla.
305-974-1700 Cyberguard 2.0 Packet filter, application proxy, network
address translator Secured Harris Nighthawk None 2 Motorola 88110s/
32 Mbytes/1 Gbyte Ethernet, FDDI, ATM, serial, dial-up/29 Out-of-band,
dial-up Service, source, destination, packet trace Socks optional Yes
$9,999 for all
scenarios
_________________________________________________________________
IBM
Contact local sales office NetSP Secure Network Gateway 2.1 Packet
filter, application proxy Secured AIX None RS/6000 3.2.5/
32 Mbytes/1 Gbyte Ethernet, token ring, FDDI,
serial/hardware-dependent In-band, out-of-band, dial-up Service,
source, destination, packet trace SecureID; Socks optional Yes
$2,900/$7,100/$20,000
_________________________________________________________________
Milkyway Networks Corp.
Ottawa
613-566-4574 Black Hole 2.01 Application proxy, network address
translator Secured SunOS BSD Sparc 20/32 Mbytes/
500 Mbytes Ethernet, token ring/hardware-dependent In-band, dial-up
Service, time, source, destination, packet trace SecureID, SSL No
$2,895 for all
scenarios
_________________________________________________________________
Morning Star Technologies Inc.
Columbus, Ohio
614-451-1883 Secureconnect 1.7 Packet filter Custom AIX, BSD, HP-UX,
Irix,
SCO Unix, Solaris, SunOS, Unixware(3) Intel 486/4 Mbytes/
10 Mbytes Ethernet, serial/4 In-band, out-of-band, dial-up Service,
time, source, destination, packet trace MD5, SecureID Yes $11,900 for
all
scenarios
_________________________________________________________________
Network-1 Software and Technology Inc.
New York
212-293-3068 Network-1 1.0-4 Packet filter Secured DOS None 100-MHz
Intel 486/
8 Mbytes/100 Mbytes Ethernet, token ring/4 In-band Service, time,
source, destination None Yes $8,995/$14,995/$19,995
_________________________________________________________________
Network Translation Inc. (NTI)
Palo Alto, Calif.
415-494-6387 Private Internet
Exchange (PIX) 2.5.17 Network address translator Custom None 75-MHz
Intel Pentium/
8 Mbytes/not applicable (uses RAM) Ethernet/2 In-band, out-of-band,
dial-up Service, time, source, destination, packet trace None Yes
$4,990/$9,990/$18,990
_________________________________________________________________
Raptor Systems Inc.
Waltham, Mass.
617-487-7700 Eagle 3.0 Application proxy Secured
SunOS 4.1.3 AIX, HP-UX Sparc 5/32 Mbytes/
500 Mbytes Ethernet, token ring, FDDI, dial-up/hardware-dependent
In-band Service, time, source, destination MD5, SecureID Yes
$7,000/$15,000/$25,000
_________________________________________________________________
Secure Computing Corp.
Roseville, Minn.
612-628-2700 Sidewinder 2.0 Application proxy Secured BSD
Maintenance-mode kernel
with no networking
functions 90-MHz Intel Pentium/
32 Mbytes/120 Mbytes Ethernet, token ring, FDDI, ATM Out-of-band
Service, time, source, destination, packet trace Kerberos, Digital
Pathways Yes $29,995 for all
scenarios
_________________________________________________________________
Sterling Software Inc.
Irving, Texas
214-868-5000 Connect:Firewall 2.1.00 Application proxy Secured
SunOS 4.1.4 None Sparc 4/32 Mbytes/
2 Gbytes Ethernet, token ring, FDDI, 100Base-T, 100VG-AnyLAN, ATM,
serial, dial-up/hardware-dependent In-band, out-of-band, dial-up
Service, time, source, destination, packet trace MD5, SecureID, SHTTP,
Socks, (4)SSL No $18,900 for all
scenarios
_________________________________________________________________
Trusted Information Systems Inc. (TIS)
Glenwood, Md.
301-854-5550 Gauntlet 3.0 Application proxy Secured BSD HP-UX, SGI/RX,
SunOS 90-MHz Intel Pentium/
32 Mbytes/less than
200 Mbytes Ethernet/2 In-band, out-of-band, dial-up Service, time,
source, destination MD5, SecureID, SHTTP, SSL Yes $15,000 for all
scenarios
_________________________________________________________________
1. Prices are given for 10, 100, and 1,000 nodes handling FTP, HTTP,
mail, telnet, news, and five other services. 3. Requires PPP support.
FTP = File transfer protocol SSL = Secure Sockets Layer nbspThe
10-node configuration assumes one Ethernet and one T1; the 100-node
configuration assumes four Ethernets 4. Socks support is mandatory for
HTTP = Hypertext transfer protocol nbspand one T1; the 1,000-node
configuration assumes one FDDI ring and one T3. nbspWorld Wide Web
browsers. PPP = Point-to-point protocol 2. Also resold as Solstice
Firewall-1 by Sunsoft Inc. (Mountain View, Calif.). SHTTP = Secure
hypertext transfer protocol
_________________________________________________________________
Most of the firewalls in our test come with secured kernels based on
Unix variants. One notable exception is Network-1 from Network-1
Software and Technology Inc. (New York). That product's secured kernel
is based on DOS. Network-1 says DOS's lack of integrated networking
capabilities is an advantage, since it means the firewall can't be
traversed if the firewall application is compromised. One trade-off:
The Network-1 firewall proved to be a slow performer in our tests.
Performance is a security issue for several reasons. A firewall that
slows response time to a crawl may motivate frustrated end-users to
look for ways around the bottleneck. Any path that circumvents the
firewall can present a point of entry to intruders. This problem is
especially acute when firewalls are used between internal networks; a
product built to work with 56-kbit/s or T1 Internet connections will
not be able to keep up with corporate LANs running at 10 Mbit/s or
more. Also, one of the most common forms of assault on the Internet is
the denial-of-service attack, in which invaders try to cut off access
to a given resource. The easiest way to do this is to bombard a
firewall with more requests than it can handle. The weaker a
firewall's performance, the more susceptible it is to a
denial-of-service attack.
Other firewalls require end-users to learn special routines to
navigate the firewall. Again, if firewalls become too inconvenient,
end-users are more likely to look for ways to get around them.
BRICKS IN THE WALL
There are two main building blocks for firewalls: packet filters and
proxy services. Most firewalls include one or both of these elements.
A relatively new type of proxy service, called the network address
translator (NAT), is used in a number of products. Each of these
elements has specific strengths and weaknesses when it comes to
performance, network security, and application transparency. Anyone
choosing a firewall needs to consider the pros and cons of each
approach.
Packet filters do exactly as their name suggests-they decide whether
to forward a packet by filtering IP addresses or TCP port numbers in
that packet's header. Addresses and port numbers are network- and
transport-layer functions, respectively, but packet filters can work
on applications as well. Internet applications typically are
associated with specific port numbers; for example, telnet
applications almost always run on TCP port 23. Thus it's possible to
set a firewall filter to block any attempt to send a telnet packet to
an inside machine.
The main advantage of packet-filtering firewalls is that they can be
added to most corporate networks at little or no cost. That's because
any router can filter traffic, and routers are required for connecting
a network to the Internet. As many as 80 percent of all firewalls now
in use consist of nothing more than a few filters configured on a
router linking an organization to an Internet service provider.
Commercial packet-filter firewalls go far beyond conventional routers
by adding extensive logging capabilities and security features like
detection of IP spoofing (an attack in which an external machine
claims to be a trusted host). The logging feature is vital not only
for analyzing attacks but also for providing legal evidence that an
effort has been made to secure the network. In the U.S., publicly held
companies are legally responsible to shareholders for securing assets,
and detailed evidence is needed for any legal action taken against
attackers.
GO-BETWEENS
Proxy services do not allow traffic to pass directly between internal
and external networks. Instead, a client establishes a circuit with
the firewall, which then sets up a separate circuit to a server. Even
if the firewall fails, a client on the external network cannot gain
access to the internal network. Proxy services typically run on a
firewall equipped with at least two logical circuits, also known as a
dual-homed host.
Specific proxies are required for each application the firewall
handles. The products we tested come with differing levels of
application support, although most include proxies for core TCP/IP
applications like FTP, HTTP (hypertext transfer protocol), and telnet
(see Table 2). Most firewalls also offer a generic proxy for setting
up other applications.
Some proxy services require the installation of special software on
client machines. The most common example is Socks, a software tool kit
for creating proxy services. Socks is an authentication scheme through
which an application negotiates with a firewall to allow connections
through the firewall. Before the normal connection is allowed, the
Socks client connects to the firewall and requests authentication. If
authentication is granted, the requested link to the protected system
is allowed.
Because Socks is generic in nature, support can be built into most
applications. But Socks has a major drawback from an administrative
standpoint: It requires that Socks-capable applications be installed
on all client machines. If an application can't be modified, or if
network managers can't install Socks-capable software on all machines,
Socks can't be used.
We got some firsthand experience with this problem in our test.
Sterling Software's Connect: Firewall includes a Socks proxy service
that requires World Wide Web (WWW) clients to be Socks-compliant.
Because the Web clients in our test application could not be modified
to support Socks, we weren't able to test Sterling's firewall.
Connect:Firewall does work with standard FTP and telnet clients;
Sterling says it is working on adding support for Web clients.
Network address translators represent an innovative twist on proxy
services. NAT boxes were developed specifically to help alleviate the
shortage of IP addresses on the Internet. In IP networks, each device
must have an address that can be recognized by all other IP devices in
all other IP networks. In essence, a NAT box allows private numbering
schemes to be used on internal networks, thus conserving the number of
"public" addresses-those that can be seen on the entire Internet. This
means addresses on the internal network don't have to be unique to the
entire IP universe; instead, an organization can use any IP addressing
scheme for its internal networks.
Because NAT boxes already serve as intermediaries between the internal
and external IP network, they can be outfitted to handle proxy
services as well. Six vendors in our test offer NAT facilities in
their firewalls: Border, Checkpoint, Digital Equipment Corp. (Maynard,
Mass.), Harris, Milkyway Networks Corp. (Ottawa), and Network
Translation Inc. (Palo Alto, Calif.).
TRICKS OF THE TRADE-OFF
Packet filters and proxy services can be used alone or combined in
various ways. Each combination involves trade-offs in security,
performance, and transparency.
Packet filters can be excellent performers if only a few filters are
defined (as was the case in our tests). They also are absolutely
transparent to applications and end-users. But used by themselves,
packet filters offer relatively weak security.
The biggest problem with packet filters is that if a firewall is
compromised, all networks behind it are at risk. For example, if an
attacker manages to log in to the machine doing the packet filtering,
it's a relatively simple matter to gain supervisor rights and define
filters that allow access to any location in the private network. Most
packet filters present cryptic interfaces that make them difficult to
configure; in our test, we found it alarmingly easy to misdefine a
filter to allow access rather than prevent it. (One exception here is
the Checkpoint firewall, which has a graphical interface.)
Another problem with packet filters is that they can't monitor
link-state information, which means they can have trouble with
connectionless datagram-based protocols like NFS (Network File System)
and RPCs (remote procedure calls). Although packet filters generally
offer good performance, throughput does suffer as the number of
defined filters goes up.
Finally, the available rules for filtering don't necessarily give net
managers enough control over their traffic-a limitation we uncovered
when we tested Secureconnect from Morning Star Technologies Inc.
(Columbus, Ohio). Secureconnect comes with some predefined filters,
including an optional filter that blocks all traffic if it receives
probes from Satan (Security Administrator Tool for Analyzing
Networks), a well-known application for finding weaknesses in a TCP/IP
network. One of Satan's invasion techniques is that it sends traffic
using sequential UDP (user datagram protocol) port numbers. If
Secureconnect's anti-Satan filter sees this kind of activity, it
blocks all traffic from the sender. However, this filter also blocks
traffic from legitimate sources that use certain port numbers in
sequence. In our tests, we found that Netbios name broadcasts from
Windows NT set the anti-Satan filter into action. When we enabled
Secureconnect's anti-Satan filter, the firewall was unable to work
with Windows NT servers.
Proxy services are stronger than packet filters for security, but
they're weaker when it comes to performance and transparency. The
major security benefit is in the design of proxy-based firewalls: Even
if a proxy-based firewall is destroyed, there's still no direct route
to the network behind the firewall. With packet-filtering routers, in
contrast, if a filter is modified or destroyed, the firewall could
continue to route packets.
But this security comes at a price: considerable inconvenience for
developers, administrators, and end-users. Each application that runs
through the firewall needs its own proxy. Some proxies also require
every machine that communicates through the firewall to run modified
client and server software. Some firewalls allow unmodified clients to
reach standard services like mail, FTP, and telnet; even here,
however, end-users may need to learn special procedures to negotiate
through the proxy.
The transparency issue really comes to a head with custom protocols or
applications. As we discovered in our tests, even transparent proxies
expect specific applications to use specific TCP or UDP port numbers.
If a site is running a standard application over a nonstandard port (a
fairly common technique for discouraging casual prowlers), the proxy
won't support that application. In many cases, however, the firewall
does allow the administrator to start up two copies of the proxy, one
on the standard port and one on a nonstandard port. The amount of
customization involved depends on the specific product.
Some firewalls also require the use of a token authentication card
before granting access to a service. Border's Borderware Firewall
Server requires the use of a token authentication card for access from
an external network to internal resources-a requirement that precluded
the use of our automated test script. Mandatory use of token
authentication cards proved a problem for us, but some net managers
are likely to view this feature as added security.
Such measures obviously enhance security, but they also add to the
burden on application developers. For our evaluation, NSTL's
developers spent several days modifying the test application to
accommodate these requirements. But some shops may not be willing or
able to change protocols or applications developed painstakingly over
many years. All too often, the "solution" is to allow special traffic
to pass unchallenged through the firewall-leaving a gaping hole for
attackers to exploit.
Vendors of proxy-based firewalls are beginning to deal with this
problem. The proxy-based products we evaluated come with generic
proxies to define custom applications or nonstandard port numbers.
Still, users of proxy-based services may find themselves having to
develop new proxies whenever an application is upgraded; a case in
point is the various security technologies being added to many Web
browsers. Potential buyers should ask vendors to demonstrate all the
applications their firewalls can handle.
THIS AND THAT
Rather than rely on one approach, several firewall vendors offer a
combination of packet filtering and proxy services in their products.
The combinations generally follow one of two designs: screened host or
screened subnet. Under the screened host approach, a packet-filtering
router connects with the Internet, while a dual-homed host resides on
the internal network. Usually, the router's filters are set up so that
the dual-homed host is the only machine directly reachable from the
Internet-a measure meant to secure other internal machines from
unauthorized access.
The screened subnet design reverses the roles: The dual-homed host
resides on its own subnet, separated from the internal network by a
packet-filtering router. In many screened subnets, packet-filtering
routers are placed on both sides of the separate subnet, creating a
"demilitarized zone" (DMZ) on the screened subnet. Internet servers,
such as WWW and FTP servers, typically are situated in these DMZs.
TO THE TEST
For our firewall performance tests, we set up a test bed comprising an
internal network, an external network, and a DMZ subnet, linking all
three to the firewall under test (see "Test Methodology"). Our test
bed included six servers and up to 48 virtual clients across the
various networks.
A virtual client is a single process running on a physical client,
requesting information as quickly as possible. We used virtual clients
to place a far heavier load on the networks; activity from one virtual
client can be equivalent to activity from dozens or even hundreds of
real-world users.
We evaluated performance using three popular Internet applications:
WWW, FTP, and WAIS (Wide Area Information Servers). A customized
version of NSTL's Intermark application suite instructed each virtual
client to request 100 FTP, WWW, and/or WAIS resources from servers
across a firewall. To determine performance, we measured transactions
per minute (tpm) and session loss at various load levels. We measured
the time involved to complete 100 transactions, including data
transfer and associated protocol negotiation. Transaction sizes varied
from less than 1 kbyte to 256 kbytes, with sizes averaging about 85
kbytes.
In addition to tracking tpm data, we also noted the number of failed
sessions during each test. Anytime one virtual client logged two or
more failed requests per 100 in a given test, we discarded the results
for that test; if 5 percent or more of all client requests failed at a
given load level, we considered the firewall under test to be
incapable of handling that many clients at that load level.
We ran three traffic profiles, representing increasing levels of
difficulty for the firewall. In the first scenario, an average of
about 20 percent of all traffic crossed from the internal segment to
the outside network, or from the external to the internal network. The
load increased moderately in the second scenario, with just over 50
percent of all traffic traversing the firewall internally and
externally. The third scenario put the heaviest load on the firewall,
with 90 percent of all internal and external traffic traversing the
firewall. The greatest differences between products showed up in this
heavy-load test, and it's this test for which we report results.
RUNNING THE NUMBERS
Checkpoint's Firewall-1 consistently turned in the highest tpm numbers
in our test (see the figure). That didn't surprise us, since
Checkpoint's firewall performs no proxy services. The big surprise was
the performance of Harris's Cyberguard, a product that uses both
packet filtering and proxy services. Cyberguard outperformed
Firewall-1 when handling requests from 48 virtual clients-the heaviest
load we tested. Products from TIS, Network Translation, and IBM also
turned in strong performance numbers.
THE FIREWALL STRESS TEST
_________________________________________________________________
At the other end of the spectrum, some products were unable to cope
with 24 virtual clients or more. The Eagle firewall from Raptor
Systems Inc. (Waltham, Mass.) wasn't able to service more than 16
virtual clients at a time without at least 5 percent of virtual client
requests timing out. Morning Star's Secureconnect also couldn't handle
large numbers of virtual clients; the vendor notes that it supplied
NSTL with a unit intended for T1 service rather than the Ethernet
segments used in our tests. Network-1's firewall topped out at 24
virtual clients in our heavy-load scenario; however, that product was
able to complete all tests at the lower traffic loads.
Most of the products we tested were able to saturate a T1 pipe.
According to NSTL calculations, a load of about 135 tpm is enough to
fill a T1. Most of the firewalls we evaluated performed well above
that level, which means they won't present a bottleneck for T1
circuits. Of course, the threshold is much higher for firewalls
linking internal networks. Because of packet collisions, the top end
for real-world Ethernet performance is around 5 Mbit/s, or about 440
tpm. Any firewall that operates at this level or better won't be a
bottleneck when used between Ethernets.
Strong tpm results alone don't tell the whole story, since they don't
take into account lost sessions. In our heavy-load scenario, firewalls
had to field more packets in very rapid succession, with each packet
having to be examined and forwarded before subsequent packets arrived.
Firewalls that couldn't respond in time ended up dropping packets. If
enough packets were dropped, retry limits were exceeded, leading to
lost sessions.
For this reason, we also tracked the number of lost sessions for every
1,000 client requests (see Table 3). Two of the firewalls that
finished in the middle of the pack in the tpm tests turned in very
strong session integrity scores: ANS's Interlock dropped only 1
session per 1,000, even at the most stressful levels. Sidewinder from
Secure Computing Corp. (Roseville, Minn.) maintained all sessions, no
matter how heavy the load.
_________________________________________________________________
TABLE 3: LOST SESSIONS
Sessions lost per 1,000 URL requests
_________________________________________________________________
nbsp nbspNumber of virtual circuits
_________________________________________________________________
Vendor 8 16 24 32 40 48
_________________________________________________________________
ANS 0 0 0 0 1 1
_________________________________________________________________
ASG 0 1 7 18 21 22
_________________________________________________________________
Checkpoint 0 0 0 0 0 3
_________________________________________________________________
DEC 1 0 13 42 83 116
_________________________________________________________________
Harris 0 0 0 0 0 0
_________________________________________________________________
IBM 0 0 9 19 36 38
_________________________________________________________________
Milkyway 20 13 13 10 14 16
_________________________________________________________________
Morning Star 20 47 * * * *
_________________________________________________________________
Network-1 3 62 58 * * *
_________________________________________________________________
NTI 0 0 1 1 3 3
_________________________________________________________________
Raptor 1 1 * * * *
_________________________________________________________________
Secure Computing 0 0 0 0 0 0
_________________________________________________________________
TIS 14 7 12 12 16 14
_________________________________________________________________
*Product unable to complete test for this number of virtual clients.
 URL = Uniform resource locator
_________________________________________________________________
Results from our tests suggest that performance doesn't necessarily
have to be sacrificed for session integrity. Two of the top finishers
in the tpm tests also did extremely well at maintaining sessions.
Checkpoint's Firewall-1 dropped only 3 sessions per 1,000 at the most
stressful level. Harris's Cyberguard did even better, maintaining all
sessions regardless of stress level.
_________________________________________________________________
Stephen M. Platt, Intermark development manager at NSTL, served as
project leader, created the test application, and helped interpret
test results for this evaluation. David Newman is testing editor for
Data Communications. His Internet address is dnewman@data.com. Brent
Melson is a senior programming analyst at National Software Testing
Laboratories Inc. (NSTL, Conshohocken, Pa.).
_________________________________________________________________
[ Home ]
[ Registration | Subscriptions ]
[ Contact Us | E-Mail ]
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic