[prev in list] [next in list] [prev in thread] [next in thread]
List: spamassassin-users
Subject: Re: How to create my personal RBL
From: hg user <mercurialuser () gmail ! com>
Date: 2019-07-03 7:22:08
Message-ID: CANRteqCfzkaEdw6xZBsOhxVrUTL3=SJx4-FjYuKLV_r4XGFtFA () mail ! gmail ! com
[Download RAW message or body]
Thank you for this interesting list.
Unfortunately my external MTA is based on exim. I think that a lot of the
checks you list are already performed by exim and by the rules we added to
that MTA. I will check one by one anyway.
Perhaps the most interesting idea, something we were already thinking
about, is to move spamassassin away from zimbra and give it a standalone
server. We will lose something in integration but we can be free from
zimbra release cycles.
On Thu, Jun 27, 2019 at 2:38 PM David Jones <djones@ena.com> wrote:
> On 6/26/19 3:43 AM, hg user wrote:
> > Thank you everybody for your really interesting answers. In this moment
> > I'm just collecting informations.
> >
> > I have one main problem: one of the engines used by our commercial
> > antispam solution returns too many FPs. I'm gradually introducing
> > spamassassin (included in zimbra) and I'd like to mitigate the FPs with
> > some other checks... using a proven, well-known technology like AskDNS
> > seems a quick and viable solution to me.
> >
> > Unfortunately a personal RBL may not cover all the use cases I'm
> > thinking about and looking at the source code of a plugin that queries a
> > sql or redis server can be interesting.
>
> Before you start working on a custom plugin, have you tuned out your MTA
> and SpamAssasin? From my personal experience, I setup an edge MTA as
> the MX and sent filtered mail to Zimbra and smarthosted from Zimbra back
> to the edge MTA. This provides the most flexibility to upgrade perl and
> SpamAssassin to the latest version along with many other benefits.
>
> Tuning out the MTA:
> - Setup Postfix with Postscreen
> - Enable weighted RBLs in Postscreen, lots of them. See the SA mailing
> list archives for "postscreen_dnsbl_sites".
> __This will block 80% or more of spam/junk alone.__
> - Setup postfwd to give extra control to add headers based on SMTP
> conversation time so SA can use those headers later. For example, I set
> headers based on the number of recipients which is very useful when
> email has been BCC'd.
> - Setup sqlgrey and slowly phase it in where users won't even know it.
> - Setup policyd-spf, OpenDMARC, and OpenDKIM
> - Setup fail2ban for repeat spammers/bots
> - Setup Postwhite to whitelist trusted senders by their SPF record.
> This allows for turning up other Postfix config settings
> - Setup TLS with a Letsencrypt certificate
> - Setup rate limiting then put exceptions in
> smtpd_client_event_limit_exceptions.
> - Postfix header_checks, body_checks, smtpd_client_restrictions,
> smtpd_helo_restrictions, smtpd_sender_restrictions,
> smtpd_relay_restrictions, smtpd_recipient_restrictions,
> smtpd_data_restrictions in the main.cf can be tuned over time.
> - Enable reject_unverified_recipient in smtpd_recipient_restrictions so
> Postfix will "look ahead" to Zimbra and not accept invalid recipients.
> -
>
> Tuning out SpamAssassin:
> - Make sure your internal_networks and trusted_networks are correct so
> RBL checks will happen correctly for the last external IP. I have
> extended this out to Google, Office 365, and other major platforms to
> detect the X-Originating-IP of the web/mail client.
> - Install KAM.cf and KAMonly.cf
> - Install DCC, Razor, Pyzor
> - Install ClamAV unofficial (extra) signatures
> - Add local rules to use the headers from OpenDMARC
> - Enable extra RBLs that aren't in the stock SA
> - I use the ShortCircuit plugin heavily, disable the ALL_TRUSTED
> shortcircuit, and enable shortcircuit on a number of the USER_IN_* rules.
> - I have created a massive list of whitelist_auth entries that are
> mostly subdomain senders from trusted senders.
> - Setup a way to train your Bayes easily by dragging email into a Spam
> and Ham folder as things are misclassified to keep the Bayesian DB tuned
> correctly.
> - Get on the latest version of perl even if you have to compile it
> because your OS might be older.
> - Install the latest stable version of SpamAssassin.
> - Many more things covered on this list over the years.
> - I setup local DBLs and DWLs for brand new Office 365 senders and other
> common sources of spam like secureserver.net, unifiedlayer.com,
> websitewelcome.com, myregisteredsite.com, etc to add a couple of points
> for new senders. Then I add good senders on those bad hosting platforms
> to a DWL that subtracts a couple of points and excludes them from other
> meta rules that amplifies certain scores for the spam.
>
> Note that a lot of this can be found by setting up a quick VM and
> installing iRedMail to check out the Postfix configuration for the
> milters mentioned above and the TLS configuration. It uses Amavisnew so
> that might be different from how you want to "glue" SpamAssassin into
> the MTA.
>
> I use MailScanner which has a few extra features of it's own in addition
> to processing emails in batches for high volume mail flow.
>
> After I did all of that work above over many years, my mail filtering
> accuracy is very good for about 80,000 mailboxes. The more mailboxes
> and domains you filter, the more time it takes to tune everything properly.
>
>
> >
> > Thank you
> > Francesco
> >
> > On Tue, Jun 25, 2019 at 10:20 PM Matus UHLAR - fantomas
> > <uhlar@fantomas.sk <mailto:uhlar@fantomas.sk>> wrote:
> >
> > >On Tue, 2019-06-25 at 11:09 -0500, David B Funk wrote:
> > >> that's way overthinking it.
> >
> > On 25.06.19 17:55, Martin Gregorie wrote:
> > >I agree, now that there's a configurable OSS dnsbl server
> available,
> > >that using it is the obvious choice for dealing with a standalone
> > list,
> > >but the OP did ask specifically about using database queries to
> > >implement a blacklist, so I thought it was worthwhile to tell him
> > what's
> > >involved in doing that.
> >
> > No. The OP wanted to store data in DB to avoid restarting SA, not
> > mentioning
> > any other specific reason to use DB.
> >
> > using DNSBL does avoid restarting SA and does not require any
> > plugin, which
> > is a great advantage.
> >
> > we are trying to provide described requirements, while avoiding
> proposed
> > complicated solutions.
> >
> > >For all I know the OP either has a similar archive or is intending
> to
> > >implement one: searching for a specific message with a database
> > tool is
> > >a *lot* faster than ferreting through a set of very large mail
> folders
> > >with your MUA, though of course the effort of creating and
> maintaining
> > >the database, mail loader, query tools and SA plugin is non
> trivial.
> >
> > well, if THIS is the real reason...
> >
> > --
> > Matus UHLAR - fantomas, uhlar@fantomas.sk <mailto:uhlar@fantomas.sk>
> > ; http://www.fantomas.sk/
> > Warning: I wish NOT to receive e-mail advertising to this address.
> > Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
> > Saving Private Ryan...
> > Private Ryan exists. Overwrite? (Y/N)
> >
>
>
> --
> David Jones
>
[Attachment #3 (text/html)]
<div dir="ltr"><div>Thank you for this interesting list.</div><div>Unfortunately my \
external MTA is based on exim. I think that a lot of the checks you list are already \
performed by exim and by the rules we added to that MTA. I will check one by one \
anyway.</div><div><br></div><div>Perhaps the most interesting idea, something we were \
already thinking about, is to move spamassassin away from zimbra and give it a \
standalone server. We will lose something in integration but we can be free from \
zimbra release cycles.<br></div></div><br><div class="gmail_quote"><div \
class="gmail_attr" dir="ltr">On Thu, Jun 27, 2019 at 2:38 PM David Jones <<a \
href="mailto:djones@ena.com">djones@ena.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">On \
6/26/19 3:43 AM, hg user wrote:<br> > Thank you everybody for your really \
interesting answers. In this moment <br> > I'm just collecting \
informations.<br> > <br>
> I have one main problem: one of the engines used by our commercial <br>
> antispam solution returns too many FPs. I'm gradually introducing <br>
> spamassassin (included in zimbra) and I'd like to mitigate the FPs with <br>
> some other checks... using a proven, well-known technology like AskDNS <br>
> seems a quick and viable solution to me.<br>
> <br>
> Unfortunately a personal RBL may not cover all the use cases I'm <br>
> thinking about and looking at the source code of a plugin that queries a <br>
> sql or redis server can be interesting.<br>
<br>
Before you start working on a custom plugin, have you tuned out your MTA <br>
and SpamAssasin? From my personal experience, I setup an edge MTA as <br>
the MX and sent filtered mail to Zimbra and smarthosted from Zimbra back <br>
to the edge MTA. This provides the most flexibility to upgrade perl and <br>
SpamAssassin to the latest version along with many other benefits.<br>
<br>
Tuning out the MTA:<br>
- Setup Postfix with Postscreen<br>
- Enable weighted RBLs in Postscreen, lots of them. See the SA mailing <br>
list archives for "postscreen_dnsbl_sites".<br>
__This will block 80% or more of spam/junk alone.__<br>
- Setup postfwd to give extra control to add headers based on SMTP <br>
conversation time so SA can use those headers later. For example, I set <br>
headers based on the number of recipients which is very useful when <br>
email has been BCC'd.<br>
- Setup sqlgrey and slowly phase it in where users won't even know it.<br>
- Setup policyd-spf, OpenDMARC, and OpenDKIM<br>
- Setup fail2ban for repeat spammers/bots<br>
- Setup Postwhite to whitelist trusted senders by their SPF record. <br>
This allows for turning up other Postfix config settings<br>
- Setup TLS with a Letsencrypt certificate<br>
- Setup rate limiting then put exceptions in <br>
smtpd_client_event_limit_exceptions.<br>
- Postfix header_checks, body_checks, smtpd_client_restrictions, <br>
smtpd_helo_restrictions, smtpd_sender_restrictions, <br>
smtpd_relay_restrictions, smtpd_recipient_restrictions, <br>
smtpd_data_restrictions in the <a href="http://main.cf" target="_blank" \
rel="noreferrer">main.cf</a> can be tuned over time.<br>
- Enable reject_unverified_recipient in smtpd_recipient_restrictions so <br>
Postfix will "look ahead" to Zimbra and not accept invalid recipients.<br>
-<br>
<br>
Tuning out SpamAssassin:<br>
- Make sure your internal_networks and trusted_networks are correct so <br>
RBL checks will happen correctly for the last external IP. I have <br>
extended this out to Google, Office 365, and other major platforms to <br>
detect the X-Originating-IP of the web/mail client.<br>
- Install KAM.cf and KAMonly.cf<br>
- Install DCC, Razor, Pyzor<br>
- Install ClamAV unofficial (extra) signatures<br>
- Add local rules to use the headers from OpenDMARC<br>
- Enable extra RBLs that aren't in the stock SA<br>
- I use the ShortCircuit plugin heavily, disable the ALL_TRUSTED <br>
shortcircuit, and enable shortcircuit on a number of the USER_IN_* rules.<br>
- I have created a massive list of whitelist_auth entries that are <br>
mostly subdomain senders from trusted senders.<br>
- Setup a way to train your Bayes easily by dragging email into a Spam <br>
and Ham folder as things are misclassified to keep the Bayesian DB tuned <br>
correctly.<br>
- Get on the latest version of perl even if you have to compile it <br>
because your OS might be older.<br>
- Install the latest stable version of SpamAssassin.<br>
- Many more things covered on this list over the years.<br>
- I setup local DBLs and DWLs for brand new Office 365 senders and other <br>
common sources of spam like <a href="http://secureserver.net" target="_blank" \
rel="noreferrer">secureserver.net</a>, <a href="http://unifiedlayer.com" \
target="_blank" rel="noreferrer">unifiedlayer.com</a>, <br> <a \
href="http://websitewelcome.com" target="_blank" \
rel="noreferrer">websitewelcome.com</a>, <a href="http://myregisteredsite.com" \
target="_blank" rel="noreferrer">myregisteredsite.com</a>, etc to add a couple of \
points <br> for new senders. Then I add good senders on those bad hosting platforms \
<br> to a DWL that subtracts a couple of points and excludes them from other <br>
meta rules that amplifies certain scores for the spam.<br>
<br>
Note that a lot of this can be found by setting up a quick VM and <br>
installing iRedMail to check out the Postfix configuration for the <br>
milters mentioned above and the TLS configuration. It uses Amavisnew so <br>
that might be different from how you want to "glue" SpamAssassin into <br>
the MTA.<br>
<br>
I use MailScanner which has a few extra features of it's own in addition <br>
to processing emails in batches for high volume mail flow.<br>
<br>
After I did all of that work above over many years, my mail filtering <br>
accuracy is very good for about 80,000 mailboxes. The more mailboxes <br>
and domains you filter, the more time it takes to tune everything properly.<br>
<br>
<br>
> <br>
> Thank you<br>
> Francesco<br>
> <br>
> On Tue, Jun 25, 2019 at 10:20 PM Matus UHLAR - fantomas <br>
> <<a href="mailto:uhlar@fantomas.sk" target="_blank">uhlar@fantomas.sk</a> \
<mailto:<a href="mailto:uhlar@fantomas.sk" \
target="_blank">uhlar@fantomas.sk</a>>> wrote:<br> > <br>
> >On Tue, 2019-06-25 at 11:09 -0500, David B Funk wrote:<br>
> >> that's way overthinking it.<br>
> <br>
> On 25.06.19 17:55, Martin Gregorie wrote:<br>
> >I agree, now that there's a configurable OSS dnsbl server \
available,<br> > >that using it is the obvious choice for dealing with \
a standalone<br> > list,<br>
> >but the OP did ask specifically about using database queries \
to<br> > >implement a blacklist, so I thought it was worthwhile to tell \
him<br> > what's<br>
> >involved in doing that.<br>
> <br>
> No. The OP wanted to store data in DB to avoid restarting SA, not<br>
> mentioning<br>
> any other specific reason to use DB.<br>
> <br>
> using DNSBL does avoid restarting SA and does not require any<br>
> plugin, which<br>
> is a great advantage.<br>
> <br>
> we are trying to provide described requirements, while avoiding \
proposed<br> > complicated solutions.<br>
> <br>
> >For all I know the OP either has a similar archive or is intending \
to<br> > >implement one: searching for a specific message with a \
database<br> > tool is<br>
> >a *lot* faster than ferreting through a set of very large mail \
folders<br> > >with your MUA, though of course the effort of creating \
and maintaining<br> > >the database, mail loader, query tools and SA \
plugin is non trivial.<br> > <br>
> well, if THIS is the real reason...<br>
> <br>
> -- <br>
> Matus UHLAR - fantomas, <a href="mailto:uhlar@fantomas.sk" \
target="_blank">uhlar@fantomas.sk</a> <mailto:<a href="mailto:uhlar@fantomas.sk" \
target="_blank">uhlar@fantomas.sk</a>><br> > ; <a \
href="http://www.fantomas.sk/" target="_blank" \
rel="noreferrer">http://www.fantomas.sk/</a><br> > Warning: I wish NOT to \
receive e-mail advertising to this address.<br> > Varovanie: na tuto adresu \
chcem NEDOSTAVAT akukolvek reklamnu postu.<br> > Saving Private Ryan...<br>
> Private Ryan exists. Overwrite? (Y/N)<br>
> <br>
<br>
<br>
-- <br>
David Jones<br>
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic