[prev in list] [next in list] [prev in thread] [next in thread] 

List:       security-onion
Subject:    Re: [security-onion] A very strange issue with rule-update: "scp: /tmp/snortrules-snapshot-2972.tar.
From:       Doug Burks <doug.burks () gmail ! com>
Date:       2015-08-25 20:32:09
Message-ID: CAK8kjrDL-FS29Fintz7i2z_p0EVk140z1iGmqOsjg9nONobM=w () mail ! gmail ! com
[Download RAW message or body]

Hi Simone,

This is a very strange issue indeed.  I don't believe I've seen any
other reports of it.  But that may be because the code section you're
referring to is only used if LOCAL_NIDS_RULE_TUNING is set to yes in
/etc/nsm/securityonion.conf.  This is not the default and so very few
folks use it.  Do all of your sensors have LOCAL_NIDS_RULE_TUNING=yes?

Chris White originally wrote this section of code and so he may have
further comments.

I haven't had time to look into this in any detail yet, but I went
ahead and created Issue 806 for this:

rule-update: replace for with while when LOCAL_NIDS_RULE_TUNING=yes
https://github.com/Security-Onion-Solutions/security-onion/issues/806

On Fri, Aug 7, 2015 at 9:45 AM, Simone Bonetti
<pascal.c.python@gmail.com> wrote:
> Hi everyone
> 
> Everyday on one of mine snort sensor rule-update runs but doesn't update VRT rules. \
>                 On pulledpork.log I found:
> scp: /tmp/snortrules-snapshot-2972.tar.gz: No such file or directory
> This sensor is updated so there isn't snort 2.9.7.2 on it.
> 
> I tried to debug rule-update. I found the problem before that rule-update invokes \
> pullepork.pl, in particular in this part of code: 
> MASTER_RULES=$(ssh -i "$KEY" $SSH_USERNAME@$SERVERNAME grep '^rule_url' \
> /etc/nsm/pulledpork/pulledpork.conf | awk -F '|' '{print $2}' | sed 's/\./*\./g') \
> LOCAL_RULES=$(grep '^rule_url' /etc/nsm/pulledpork/pulledpork.conf | awk -F '|' \
> '{print $2}' | sed 's/\./*\./g') 
> set -- $MASTER_RULES
> MASTER_LENGTH=${#@}
> set -- $LOCAL_RULES
> LOCAL_LENGTH=${#@}
> 
> # The master must be a super set of all rules.
> # Check to ensure master rule url count is not smaller than local
> # which implies a rule_url was added to the local sensor and not the master.
> # Also warn if local has fewer rule urls than master in case
> # a url was added upstream and not subsequently downstream.
> if [ "$MASTER_LENGTH" -lt "$LOCAL_LENGTH" ]; then
> echo "Error: Master rule_url count is less than Local rule_url count."
> echo "Master must have all rule_urls enabled in its pulledpork.conf."
> elif [ "$MASTER_LENGTH" -gt "$LOCAL_LENGTH" ]; then
> echo "Warning: Local rule_url count is less than Master rule_url count. "
> echo "Ensure this is intended."
> fi
> 
> for COMPRESSED_RULE in $MASTER_RULES; do
> scp -i "$KEY" $SSH_USERNAME@$SERVERNAME:/tmp/$COMPRESSED_RULE /tmp/
> done
> 
> when a sensor copy the rules from server.
> 
> I modified that code to understand the problem so I insert many echoes. I know it's \
> ridiculous but it runs. Now the source code is: MASTER_RULES=$(ssh -i "$KEY" \
> $SSH_USERNAME@$SERVERNAME grep '^rule_url' /etc/nsm/pulledpork/pulledpork.conf | \
> awk -F '|' '{print $2}' | sed 's/\./*\./g') echo
> echo "master rules: $MASTER_RULES"
> LOCAL_RULES=$(grep '^rule_url' /etc/nsm/pulledpork/pulledpork.conf | awk -F '|' \
> '{print $2}' | sed 's/\./*\./g') echo
> echo "local rules: $LOCAL_RULES"
> 
> set -- $MASTER_RULES
> echo
> echo "master rules after set: $MASTER_RULES"
> MASTER_LENGTH=${#@}
> echo
> echo "master rules after master_length: $MASTER_RULES - master_length: \
> $MASTER_LENGTH" set -- $LOCAL_RULES
> echo
> echo "local rules after set: $LOCAL_RULES"
> LOCAL_LENGTH=${#@}
> echo
> echo "local rules after local_length: $LOCAL_RULES - local_length: $LOCAL_LENGTH"
> 
> # The master must be a super set of all rules.
> # Check to ensure master rule url count is not smaller than local
> # which implies a rule_url was added to the local sensor and not the master.
> # Also warn if local has fewer rule urls than master in case
> # a url was added upstream and not subsequently downstream.
> if [ "$MASTER_LENGTH" -lt "$LOCAL_LENGTH" ]; then
> echo "Error: Master rule_url count is less than Local rule_url count."
> echo "Master must have all rule_urls enabled in its pulledpork.conf."
> elif [ "$MASTER_LENGTH" -gt "$LOCAL_LENGTH" ]; then
> echo "Warning: Local rule_url count is less than Master rule_url count. "
> echo "Ensure this is intended."
> fi
> echo
> echo "master rules and local rules after length check: $MASTER_RULES - \
> $LOCAL_RULES" 
> for COMPRESSED_RULE in $MASTER_RULES; do
> echo
> echo "copying: $COMPRESSED_RULE"
> scp -i "$KEY" $SSH_USERNAME@$SERVERNAME:/tmp/$COMPRESSED_RULE /tmp/
> done
> 
> and this is the "funny" output, I preferred use it to explain better where is the \
> problem: master rules: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> local rules: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> master rules after set: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> master rules after master_length: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz - master_length: 2
> 
> local rules after set: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> local rules after local_length: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz - local_length: 2
> 
> master rules and local rules after length check: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz - snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> copying: snortrules-snapshot-2972.tar.gz
> scp: /tmp/snortrules-snapshot-2972.tar.gz: No such file or directory
> 
> copying: emerging.rules.tar.gz
> emerging.rules.tar.gz
> 
> The issue is on the for instruction. I don't know why but bash shell does many \
> jokes when processes strings with special chars. I modified the "for" with a \
> "while" and now all is ok. That's the code without echoes: 
> MASTER_RULES=$(ssh -i "$KEY" $SSH_USERNAME@$SERVERNAME grep '^rule_url' \
> /etc/nsm/pulledpork/pulledpork.conf | awk -F '|' '{print $2}' | sed 's/\./*\./g') \
> LOCAL_RULES=$(grep '^rule_url' /etc/nsm/pulledpork/pulledpork.conf | awk -F '|' \
> '{print $2}' | sed 's/\./*\./g') 
> set -- $MASTER_RULES
> MASTER_LENGTH=${#@}
> set -- $LOCAL_RULES
> LOCAL_LENGTH=${#@}
> 
> # The master must be a super set of all rules.
> # Check to ensure master rule url count is not smaller than local
> # which implies a rule_url was added to the local sensor and not the master.
> # Also warn if local has fewer rule urls than master in case
> # a url was added upstream and not subsequently downstream.
> if [ "$MASTER_LENGTH" -lt "$LOCAL_LENGTH" ]; then
> echo "Error: Master rule_url count is less than Local rule_url count."
> echo "Master must have all rule_urls enabled in its pulledpork.conf."
> elif [ "$MASTER_LENGTH" -gt "$LOCAL_LENGTH" ]; then
> echo "Warning: Local rule_url count is less than Master rule_url count. "
> echo "Ensure this is intended."
> fi
> 
> while read -r COMPRESSED_RULE ; do
> echo
> echo "copying: $COMPRESSED_RULE"
> scp -i "$KEY" $SSH_USERNAME@$SERVERNAME:/tmp/$COMPRESSED_RULE /tmp/
> done <<< "$MASTER_RULES"
> 
> and the funny right output is:
> master rules: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> local rules: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> master rules after set: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> master rules after master_length: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz - master_length: 2
> 
> local rules after set: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> local rules after local_length: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz - local_length: 2
> 
> master rules and local rules after length check: snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz - snortrules-snapshot*.tar*.gz
> emerging*.rules*.tar*.gz
> 
> copying: snortrules-snapshot*.tar*.gz
> snortrules-snapshot-2973.tar.gz                                                     \
> 100%   31MB  31.4MB/s   00:01 
> copying: emerging*.rules*.tar*.gz
> emerging.rules.tar.gz                                                               \
> 100% 1614KB   1.6MB/s   00:00 
> I found the solution on \
> http://superuser.com/questions/284187/bash-iterating-over-lines-in-a-variable. The \
> other snort sensors don't show this problem. That's funny. 
> Bye
> 
> Simone
> 
> --
> You received this message because you are subscribed to the Google Groups \
> "security-onion" group. To unsubscribe from this group and stop receiving emails \
> from it, send an email to security-onion+unsubscribe@googlegroups.com. To post to \
> this group, send email to security-onion@googlegroups.com. Visit this group at \
> http://groups.google.com/group/security-onion. For more options, visit \
> https://groups.google.com/d/optout.



-- 
Doug Burks
Need Security Onion Training or Commercial Support?
http://securityonionsolutions.com

-- 
You received this message because you are subscribed to the Google Groups \
"security-onion" group. To unsubscribe from this group and stop receiving emails from \
it, send an email to security-onion+unsubscribe@googlegroups.com. To post to this \
group, send email to security-onion@googlegroups.com. Visit this group at \
http://groups.google.com/group/security-onion. For more options, visit \
https://groups.google.com/d/optout.


[prev in list] [next in list] [prev in thread] [next in thread]