[prev in list] [next in list] [prev in thread] [next in thread] 

List:       wget
Subject:    referer not sent on all recursive requests?
From:       jonah benton <jonah () jonah ! com>
Date:       2006-02-14 22:06:09
Message-ID: 43F25451.8030509 () jonah ! com
[Download RAW message or body]


Hi-

I'm using wget in some qa scripts to recurse through a site I'm 
developing to find 404s and 500s and bad resource references. I'm using 
RHEL4's wget:

GNU Wget 1.10.2 (Red Hat modified)

I'm running it as per below:

   wget \
     -kpSrN \
     -F -i $urls \
     -B $base \
     -D $domain \
     -np \
     -l 2 \
     --user-agent="$ua" \
     --header="$lm" \

$urls is an html file of all relative urls the CMS should be producing.
--user-agent is needed to bypass the paywall.
$base is the absolute base for the relative urls.
$domain is to keep from crawling off-site links.
--header sends a special old last-modified header which the web server 
echoes back in cases when one is not provided by the app server.

In any event, my problem is that I'm finding that wget is sending a 
Referer: along with *most* requests, but not all.

I want Referers sent for everything so I can easily find (at least one 
of) the referring pages for problematic URLs.

Is this a bug?

Thanks,

Jonah



[prev in list] [next in list] [prev in thread] [next in thread]