[prev in list] [next in list] [prev in thread] [next in thread] 

List:       best-of-security
Subject:    BoS: Re: Sidewinder Challenge winding up
From:       spaf () cs ! purdue ! edu (Gene Spafford)
Date:       1995-11-27 4:37:53
[Download RAW message or body]

> 
> 
> The first Sidewinder Challenge will be completed on Sunday Dec 3.
> Soon after we will post information on the number of attacks we
> received for the challenge.  After that the system will be down for
> approximately a week while we upgrade the system to our 2.1.2 release.
> At that time we will issue a new challenge complete with a new crypto
> certificate.
> 
> Keep an eye on this the sneakers mailing list or http://www.sidewinder.com 
> for an announcement of the new Challenge.

Dan,

I have heard nothing but good reports about the Sidewinder (but only a
few of those -- few people I know have one).  However, I am very much
*against* the way it is being marketed with this "challenge."  I am
enclosing something derived from a piece I originally wrote for a
Computer Security Institute publication which indicates why I believe
security professionals (such as those on "sneakers") should take their
business elsewhere from companies that use "challenges" as a way of
marketing their products.

--spaf

Vendors who try to establish security by having people make attacks on their 
products (including product challenges like the Sidewinder challenge) should 
be viewed with great skepticism.  The following are known problems
with this overall approach:

  * Few such "challenges" are conducted using established testing
        techniques.  They are ad hoc, random tests.  Thus, there is 
	no way of determining final coverage.  If 90% of all attacks
	are of the same variety, what has the test really shown?
  * That no problems are found does not mean that no problems exist.
        It may mean that the testers didn't expose them.  Doing
	random, black-box testing remotely is not likely to really
	test much of the product.
  * That no problems are reported does not mean that no problems
        exist.  The testers might not have recognized them.  (Look
        at how often software is released with bugs, even after
        careful scrutiny.)
  * That no problems are reported does not mean that no problems
        exist.  How do you know that the testers will report what
        they find?  How do you know the vendor is getting accurate
        data?  If Jane Random Hacker found a way to penetrate a
	Sidewinder that SCTC monitoring didn't expose, you can be
	damn sure she'd find more profitable uses for that
	information than informing Secure Computing about it.
	Further, because of problems with the law or otherwise, people
	might not want to report success and draw attention to themselves.
  * Simply because the vendor does not report a successful penetration
        does not mean that one did not occur -- the vendor may
        choose not to report it because it would reflect poorly
        on their product, or not meet their narrow criteria for a
        "successful" penetration, or they may not be able to detect it 
	happened. (I'm not suggesting it is the case here, but
	how can you *prove* that it is not the case?)
  * It gives potential miscreants some period to practice breaking
        the system without penalty.  Any other time spent hacking at
	one of these might result in legal action or worse.  Isn't it
	nice SCTC is  giving free practice time to the bad guys?  I
	hope all the potential customers are equally pleased at this.
  * It gives miscreants an excuse if they are caught trying to break
        into the system later (e.g., "We thought the contest was
        still on." or "We were just helping out X and Y.")
  * Seldom do the really good experts, on either side of the fence,
        participate in such exercises.  Thus, anything done is usually
        done by amateurs. (The "honor" of having won the challenge is
        not sufficient to lure the good ones into the fray.  Good consultants
        command fees of several thousand $$ per day in some cases -- why
        should they donate their time and name for free advertising?)

Furthermore, the whole process sends the wrong message -- that we
should build things and then try to break them, or that there is
some prestige or glory in breaking systems.  

Security should be carefully designed in and tested using established
methods.  Tiger teams have a role, but using them as a major means
of establishing safety is negligent.  Security "contests" to
demonstrate a system are worse, and should be viewed negatively by
potential customers and professionals.  Such contests should not
establish confidence in a product, are not a good means of testing,
and actually create a climate that may encourage or enable people to
try to break the product after it is in use.

If I was a potential customer of any security product, which of the
following would be more likely to convince me that a company had its
act together?  Which one is the company more likely to be seeking to
sell based on smoke and mirrors?

  Approach A:  We put our product up on the Internet for 6 months, and
offered a nifty backpack and some money to anyone who could break in.
No one claimed the prize.  Obviously, ours is a superior product.

  Approach B: Our company is certfied as an ISO 9000 company.  We used
formal software engineering approaches to design and build our
product, ending in full functional testing, D-U path testing, and
statement coverage to 100%.  We also hired well-known independent
security experts A, B, and C under non-disclosure to examine the code
and identify weaknesses, and then conduct field trials.  Company X and
University Y have also had the opportunity to examine and test our
product, and none of them have found flaws.  


Again, I am not implying that Sidewinder is in any way deficient, or
that SCTC is doing anything underhanded.  But personally, I would take
my business elsewhere from any company using a "challenge" to promote
its products.  I would encourage SCTC to come up with some "real"
proof of robustness, rather than a second "challenge."

[prev in list] [next in list] [prev in thread] [next in thread]