[prev in list] [next in list] [prev in thread] [next in thread] 

List:       firewall-wizards
Subject:    Industry test reports... (was RE: [fw-wiz] Proverbial
From:       "Marcus J. Ranum" <mjr () ranum ! com>
Date:       2002-10-26 18:08:02
[Download RAW message or body]

Ryan, Kennedy wrote:
>Funny thing..  I'm right now looking at a Netscreen commissioned Tolly Group
>test report from March 2002, (no.202121) which amazingly (not) shows
>Netscreen outperforming Checkpoint/Nokia IP740 and Cisco 545 by leaps and
>bounds..    

Tolly group's tests are all bought and paid for. The vendor pays
to have a test of their own devising run, and Tolly reports the
results. Remember the big scandal from last year when Miercomm
published the results of their "test" of Intrusion.com's IDS?
The test was rigged (the protocol was suggested by Intrusion.com)
to yield completely ridiculous (and deliberately deceptive)
results. Typically, in a software company, it's the marketing
organization's budget that pays for the tests. :)  Then the testing
company carefully words the results so that they don't step on
their other customers that have _also_ paid for tests.

The thing to look for is when they run the SAME TEST protocol
every time and make up their own test protocol with input from
outsiders not beholden to the vendors. I know I used to bash
them 8 years ago or so, but ICSA labs is doing some really good
stuff with their firewall tests these days. But they're looking
at functionality, not "marketing numbers" - don't believe _ANY_
of the marketing number tests.

When you buy a car do you believe the manufacturer's claim
that it goes 0-60 in a "blazing" 6 seconds, or Car and Driver's
test results where they used their standard test methods and
got 0-60 in 8 seconds? ;)

mjr.
---
Marcus J. Ranum				http://www.ranum.com
Computer and Communications Security	mjr@ranum.com

_______________________________________________
firewall-wizards mailing list
firewall-wizards@honor.icsalabs.com
http://honor.icsalabs.com/mailman/listinfo/firewall-wizards
[prev in list] [next in list] [prev in thread] [next in thread]