[prev in list] [next in list] [prev in thread] [next in thread] 

List:       newtontalk
Subject:    Re: [NTLK] Requiring registration to download?
From:       Grant Hutchinson <grant () splorp ! com>
Date:       2017-04-13 20:57:09
Message-ID: 9A6866ED-2550-449D-868E-FAE78B24DB59 () splorp ! com
[Download RAW message or body]


> > If it was way back machine why would they download the same
> > files over and ove again, and ignore robots.txt?
> 
> Can't be the way back machine then.  I was going to ask about your
> robots.txt but that answered it.  Also it wouldn't be every day.
> Changes perhaps, every few days or weeks.  But not every day.

I agree with Dan. It isn't the Internet Archive doing the bulk downloading.

However, you might consider removing the following lines from your robots.txt file, \
because the Wayback Machine isn't currently archiving all of your .css files.

Disallow: /components/
Disallow: /modules/

g.

----------------------------------------------------------------------

Grant Hutchinson
NewtonTalk List Dad & Web Custodian


----------------------------------------------------------------------

http://newtontalk.net/
http://twitter.com/newtontalk


[prev in list] [next in list] [prev in thread] [next in thread]