[prev in list] [next in list] [prev in thread] [next in thread] 

List:       privoxy-developers
Subject:    Re: [privoxy-devel] Patches
From:       Fabian Keil <fk () fabiankeil ! de>
Date:       2012-01-21 18:04:42
Message-ID: 20120121190442.4d16e017.fk () fabiankeil ! de
[Download RAW message or body]

[Attachment #2 (multipart/signed)]


James Vasile <vasile@freedomboxfoundation.org> wrote:

> On Mon, 16 Jan 2012 21:07:12 +0100, Fabian Keil <fk@fabiankeil.de> wrote:
> > James Vasile <vasile@freedomboxfoundation.org> wrote:

> > > I work with the FreedomBox project.  We're integrating privacy, security
> > > and anonymity infrastructure in a plug server for easy end-user use.
> > > Privoxy is one package we are using for improving the web browsing
> > > experience.  Toward that end, we've added all the http-everywhere and
> > > easyprivacy rules to our version of privoxy to upgrade connections to
> > > ssl and to block web tracking.
> > > 
> > > The code is at https://github.com/jvasile/freedombox-privoxy and we're
> > > testing it now for release later this month.
> > 
> > Great. So far I only cloned the repository and skimmed over it,
> > but I'll try to have a closer look in the next days. A converter
> > from Adblock Plus to Privoxy files has been on our TODO list for
> > a while now.
> 
> My converter is somewhat primitive and lossy.  I'm operating on the
> principle that some conversion is better than none.  It still needs work
> and some testing.

I think partial conversion is a good start.

> > > In order to do the https-everywhere ruleset, I had to add a few lines to
> > > privoxy so that redirects could contain multiple regex substitutions.
> > 
> > It's not obvious to me why a https-everywhere rule set would require
> > multiple regex substitutions. Could you give an example?
> 
> Take this https-everywhere rule:
> 
> <!-- blogg.binero.se lacks ssl support. -->
> <ruleset name="Binero.se">
>   <target host="binero.se" />
>   <rule from="^http://binero\.se/" to="https://www.binero.se/"/>
>   <rule from="^http://support\.binero\.se/" to="https://support.binero.se/"/>
>   <rule from="^http://order\.binero\.se/" to="https://order.binero.se/"/>
>   <rule from="^http://www\.binero\.se/" to="https://www.binero.se/"/>
> </ruleset>
> 
> If a request comes in matching binero.se, it needs to be checked against
> all the regexes, but if I do several +redirect lines, privoxy sees each
> one as trumping the earlier ones and only applies the last one.

I'm not familiar with https-everywhere syntax, but I would
have converted the ruleset above to something like:

{+redirect{s@^http://binero\.se/@https://www.binero.se/@i}}
# Redirected URL = http://binero.se/
# Redirect Destination = https://www.binero.se/
binero.se/

{+redirect{s@^http://support\.binero\.se/@https://support.binero.se/@i}}
# Redirected URL = http://support.binero.se/
# Redirect Destination = https://support.binero.se/
support.binero.se/

{+redirect{s@^http://order\.binero\.se/@https://order.binero.se/@i}}
# Redirected URL = http://order.binero.se/
# Redirect Destination = https://order.binero.se/
order.binero.se/

{+redirect{s@^http://www\.binero\.se/@https://www.binero.se/@i}}
# Redirected URL = http://www.binero.se/
# Redirect Destination = https://www.binero.se/
www.binero.se/

Am I missing something?

> > > I'd like to offer changes to Debian and also to the privoxy project, but
> > > I'm not sure if you're interested.  It's a lot of additional rules, but
> > > I think it would be cool if privoxy offered at least some of these
> > > features to all of Debian or to everybody in the world, not just our own
> > > project.
> > 
> > Given that the https-everywhere and "easyprivacy" rules are actively
> > maintained elsewhere and likely frequently updated, while there are
> > usually several months between Privoxy releases, I don't think including
> > translated rules in Privoxy releases would be a good idea.
> 
> From a convenience perspective, my guess is that many users would
> probably like rules included by default, even if they are a couple
> months out of date.  But I take your point.

It's my understanding that https-everywhere rules occasionally
cause site breakage after websites are updated.

A lot of Privoxy users aren't willing or experienced enough
to diagnose and fix such problems, which is why Privoxy's
default configuration is less aggressive than it used to be.

> > Allowing Privoxy users to leverage those rule sources by either
> > downloading already transformed action files or by converting the
> > rules themselves makes a lot of sense, though.
> > 
> > If this really requires modifications to Privoxy itself, I wouldn't
> > mind helping with their implementation or reviewing patches.
> > 
> > > Please let me know the best way to offer changes to the Debian package
> > > or to the privoxy project.  Below is the code for multiple regexes in
> > > redirects.  It is only needed to make https-everywhere work, not
> > > easyprivacy.  I'm open to adjusting my work for better fit with yours.
> > 
> > The patch has a couple of issues:
> > 
> > 1) Privoxy is usually built with threading support so all
> >    the functions it uses need to be either thread-safe or
> >    protected with a mutex lock. Using strtok() the way you
> >    do is likely to cause incorrect redirects under load,
> >    unless you disable threading support.
> 
> Would switching to strtok_r do it?

It should, but it isn't portable as it's not part of any
C standard (I'm aware of). For FreedomBox you can probably
assume that it's available, but upstream we can't take it
for granted and would need a fallback for systems without
it.

> >    You should be able to confirm this with Privoxy-Regression-Test:
> >    https://sourceforge.net/tracker/?func=detail&aid=3429848&group_id=11118&atid=311118
> 
> Here is the result of the test:
> 
> 2012-01-16 21:20:39: Asking Privoxy for the number of action files available ...
> 2012-01-16 21:20:39: Gathering regression tests from 5 action file(s) delivered by Privoxy.
> 2012-01-16 21:20:39: Executing regression tests ...
> 2012-01-16 21:20:44: Executed 298 regression tests. Skipped 14. 298 successes, 0 failures.

By default Privoxy-Regression-Test doesn't execute redirect tests
as test failures could "leak" to the outside. Also most thread-related
issue don't show up unless you use multiple forks, and even then they
may not be obvious from the outside, if the forks don't execute the
tests in different order.

A command like the following should do, assuming there actually
are redirect tests available: 
privoxy-regression-test.pl --forks 100 --min-level 108 --max-level 200 --shuffle-tests

> >    BTW, did you already consider letting your rules converters
> >    generate tests for Privoxy-Regression-Test? If there's
> >    enough information in the source to do this, it should
> >    make testing the translated action files a lot easier.
> 
> No, but I will look into this.

Great.

> > 3) Assuming redirecting a request based on multiple pcrs
> >    commands is really necessary, I think a more elegant
> >    solution would be to allow multiple matching redirect{}
> >    actions, each with a single pcrs command, or to add a
> >    redirect-filter{} action that works similar to the
> >    client-header-filter{} action and only references a
> >    "redirect filter" that is defined in a filter file.
> > 
> >    Doing the latter would have the added benefit that
> >    the "redirect filter" only needs to be compiled once
> >    when loading the filter file and not for every request
> >    the action applies to.
> 
> My approach is definitely a bit of a kludge.  Your makes makes more
> sense.  I will look into your approach.  In the mean time, I've pushed a
> new filters.c to git per your suggestions.  It is below.

It's still not clear to me why multiple pcrs commands are necessary,
but your patch looks like it addresses the issues I mentioned.

Fabian

["signature.asc" (application/pgp-signature)]

------------------------------------------------------------------------------
Try before you buy = See our experts in action!
The most comprehensive online learning library for Microsoft developers
is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3,
Metro Style Apps, more. Free future releases when you subscribe now!
http://p.sf.net/sfu/learndevnow-dev2

_______________________________________________
Ijbswa-developers mailing list
Ijbswa-developers@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ijbswa-developers


[prev in list] [next in list] [prev in thread] [next in thread]