[prev in list] [next in list] [prev in thread] [next in thread]
List: rpm-devel
Subject: Fwd: Unsnarling an rpmdb from the HR MIB
From: Jeff Johnson <n3npq () mac ! com>
Date: 2008-07-17 16:22:51
Message-ID: 1415DC6D-67C1-4F9B-90C1-79F8428F0F71 () mac ! com
[Download RAW message or body]
Begin forwarded message:
> From: Jeff Johnson <n3npq@mac.com>
> Date: July 17, 2008 12:17:58 PM EDT
> To: Dave Shield <D.T.Shield@liverpool.ac.uk>
> Subject: Re: Unsnarling an rpmdb from the HR MIB
>
>
> On Jul 17, 2008, at 11:51 AM, Dave Shield wrote:
>
>> 2008/7/17 Jeff Johnson <n3npq@mac.com>:
>>> I still suggest eliminating -lrpm entirely.
>>
>> That's clearly a desirable long-term goal.
>> But I suspect it's going to take more than a little while to
>> achieve this. We tend to be reluctant to make drastic
>> changes to the code for a released branch, so this work
>> is probably really looking at 5.5 and above.
>>
>
> Yes. But 0b files in some directory is easily retrofitted
> with either a timer on script execution (the script is trivial),
> or even by a cron script.
>
> The patching in rpm is not hard at all either. Alas, I cannot
> control what @redhat.com and @rpm.org decide to do, but I
> certainly can apply pressure through other means, including
> signing up SElinux types, to club the patch into all versions of
> rpm (with a periodic, say once every hour, cron job script running
> for those who simply cannot upgrade rpm but can upgrade net-snmp).
>
>> One consequence of this is that it's probably worth
>> looking at applying the rpm-5 patched for -lrpm
>> linkage to the current branches.
>> How hairy are these patches likely to be?
>> Or put another way, how long would it take you to
>> construct a suitable AutoFu invocation? :-)
>>
>
> The AutoFu will be nasty, but is doable.
>
> The underlying issue that I waited for a _MAJOR_ release of
> rpm in order to implement, is that the malloc rules have changed.
>
> The rule in rpm-5.0 is
> All data retrieved from a Header MUST be free'd.
> rather than the API Baroqueness of
> Some data must not be free'd.
> that rpm has always had.
>
> See headerFreeData() for where the Baroque'ness is hidden.
>
> Changing malloc rules for retrieved data is best dealt with in
> applications,
> not through legacy retorfits, or additional flags, or wrappers or
> any other
> pretense that the rules have not changed when, in fact, the rules
> have changed.
>
> rpm-4.6 has decided to hedge their bets with -D_COMPAT_4_4 flag,
> but is also
> clearly on a path to
> All data from a Header MUST be free'd (if you access with opt-
> in flag).
> I've chosen a major release path instead, and carry a rpm4compat.h
> set of wrappers
> retrofit in rpm-5.0 instead.
>
> Note that the rpm-5.x "compatibility" through
> #include <rpm4compat.h>
> is not a viable option for net-snmp, the wrappers in rpm4compat.h
> have memory leaks,
> and I strongly believe that applications, not rpm, are the best
> place to fix a change in malloc rules.
>
> But also note that the patching for net-snmp to free all, not just
> some, memory
> when using rpm-5.0 is quite easy to do as well. I just wanted to
> warn you before you
> attempted using the compatibility layer in rpm-5.0.
>
>>
>>
>>
>> One other thing to check - re-reading your earlier message,
>> I get the impression that the /var/cache/hrmib data will
>> be populated automatically by "new" RPM installations.
>> Is this correct?
>>
>
> Yes. Every addition/deletion to an rpmdb is synchronously coupled to
> an addition/deletion in /var/cache/hrmib. You will get exactly the
> same answer
> whether you traverse /var/cache/hrmib or iterate on /var/lib/rpm,
> modulo
> a rather teensy time window that will self correct on next traverse.
>
> There is zero loss of functionality for HR MIB if /var/cache/hrmib
> is used instead
> of /var/lib/rpm/Packages.
>
> You can quibble about time windows, and data consistency, etc etc
> in the HR MIB
> data, but linking -lrpm certainly has far worse DoS flaws because
> of stale
> POSIX mutexes than what I have implemented in /var/cache/hrmib.
>
> If necessary, I can add a fcntl lock mechanism, but I hardly think
> that the HR MIB
> return values have clearly defined semantics when an rpm upgrade is
> concurrently
> running while net-snmp is populating a HR MIB reply.
>
>> (My initial assumption is that this cache would have to
>> be built manually, using some form of cron job. But I
>> now suspect that's not the case. Can you please confirm?)
>>
>
> There's a one-time cost to initially populate /var/cache/hrmib,
> yes. But in
> a "library push" model, that becomes an rpm, not a net-snmp, problem.
>
> Convinced yet? If more is needed, or we have to do the change more
> slowly,
> I can do that too. But I do think -lrpm was a very bad
> architectural mistake on
> my part. Who knew in 1998?
>
> 73 de Jeff
[Attachment #3 (text/html)]
<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
-webkit-line-break: after-white-space; "> <br><div><br><div>Begin forwarded \
message:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
"><font face="Helvetica" size="3" color="#000000" style="font: 12.0px Helvetica; \
color: #000000"><b>From: </b></font><font face="Helvetica" size="3" style="font: \
12.0px Helvetica">Jeff Johnson <<a \
href="mailto:n3npq@mac.com">n3npq@mac.com</a>></font></div><div style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font \
face="Helvetica" size="3" color="#000000" style="font: 12.0px Helvetica; color: \
#000000"><b>Date: </b></font><font face="Helvetica" size="3" style="font: 12.0px \
Helvetica">July 17, 2008 12:17:58 PM EDT</font></div><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" \
size="3" color="#000000" style="font: 12.0px Helvetica; color: #000000"><b>To: \
</b></font><font face="Helvetica" size="3" style="font: 12.0px Helvetica">Dave Shield \
<<a href="mailto:D.T.Shield@liverpool.ac.uk">D.T.Shield@liverpool.ac.uk</a>></font></div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
"><font face="Helvetica" size="3" color="#000000" style="font: 12.0px Helvetica; \
color: #000000"><b>Subject: </b></font><font face="Helvetica" size="3" style="font: \
12.0px Helvetica"><b>Re: Unsnarling an rpmdb from the HR MIB</b></font></div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><br></div> <div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">On \
Jul 17, 2008, at 11:51 AM, Dave Shield wrote:</div><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div> <blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">2008/7/17 Jeff Johnson <<a \
href="mailto:n3npq@mac.com">n3npq@mac.com</a>>:</div> <blockquote type="cite"><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">I \
still suggest eliminating -lrpm entirely.</div> </blockquote><div style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">That's clearly a desirable long-term goal.</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">But I suspect it's going to take more than a little while to</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">achieve this. <span class="Apple-converted-space"> </span>We tend to be reluctant \
to make drastic</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">changes to the code for a released branch, so this \
work</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">is probably really looking at 5.5 and above.</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><br></div> </blockquote><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">Yes. But 0b files in some directory is easily \
retrofitted</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">with either a timer on script execution (the script is \
trivial),</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">or even by a cron script.</div><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">The patching in rpm is not hard at all either. Alas, I \
cannot</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">control what @redhat.com and @rpm.org decide to do, but \
I</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">certainly can apply pressure through other means, \
including</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">signing up SElinux types, to club the patch into all versions \
of</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">rpm (with a periodic, say once every hour, cron job script \
running</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">for those who simply cannot upgrade rpm but can upgrade \
net-snmp).</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><br></div> <blockquote type="cite"><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">One consequence of this is that it's probably worth</div><div style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">looking at applying \
the rpm-5 patched for -lrpm</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">linkage to the current branches.</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
"><span class="Apple-converted-space"> </span>How hairy are these patches likely to \
be?</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">Or put another way, how long would it take you to</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">construct a suitable AutoFu invocation? <span class="Apple-converted-space"> \
</span>:-)</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><br></div> </blockquote><div style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">The AutoFu will be nasty, but is doable.</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">The underlying issue that I waited for a \
_MAJOR_ release of</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">rpm in order to implement, is that the malloc \
rules have changed.</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">The rule in rpm-5.0 is</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; "><span class="Apple-converted-space"> \
</span>All data retrieved from a Header MUST be free'd.</div><div style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">rather than the API \
Baroqueness of</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; "><span class="Apple-converted-space"> </span>Some data \
must not be free'd.</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">that rpm has always had.</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">See headerFreeData() for where the \
Baroque'ness is hidden.</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">Changing malloc rules for retrieved data is best dealt with in \
applications,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">not through legacy retorfits, or additional flags, or \
wrappers or any other</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">pretense that the rules have not changed \
when, in fact, the rules have changed.</div><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">rpm-4.6 has decided to hedge their bets with -D_COMPAT_4_4 flag, \
but is also</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">clearly on a path to</div><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><span \
class="Apple-converted-space"> </span>All data from a Header MUST be free'd (if \
you access with opt-in flag).</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">I've chosen a major release path instead, and \
carry a rpm4compat.h set of wrappers</div><div style="margin-top: 0px; margin-right: \
0px; margin-bottom: 0px; margin-left: 0px; ">retrofit in rpm-5.0 instead.</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">Note that the rpm-5.x "compatibility" \
through</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; "><span class="Apple-converted-space"> </span>#include \
<rpm4compat.h></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">is not a viable option for net-snmp, the wrappers in \
rpm4compat.h have memory leaks,</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">and I strongly believe that applications, not \
rpm, are the best place to fix a change in malloc rules.</div><div style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">But also note that the patching for net-snmp to free all, not \
just some, memory</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">when using rpm-5.0 is quite easy to do as well. I just \
wanted to warn you before you</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">attempted using the compatibility layer in \
rpm-5.0.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><br></div> <blockquote type="cite"><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><br></div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">One other thing to check - re-reading your \
earlier message,</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">I get the impression that the /var/cache/hrmib data \
will</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">be populated automatically by "new" RPM installations.</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
"><span class="Apple-converted-space"> </span>Is this correct?</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><br></div> </blockquote><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">Yes. Every addition/deletion to an rpmdb is synchronously coupled \
to</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">an addition/deletion in /var/cache/hrmib. You will get exactly \
the same answer</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">whether you traverse /var/cache/hrmib or iterate on \
/var/lib/rpm, modulo</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">a rather teensy time window that will self \
correct on next traverse.</div><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; min-height: 14px; "><br></div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">There is zero loss of functionality for HR MIB if /var/cache/hrmib is used \
instead</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">of /var/lib/rpm/Packages.</div><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">You can quibble about time windows, and data consistency, etc etc \
in the HR MIB</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">data, but linking -lrpm certainly has far worse DoS flaws \
because of stale</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">POSIX mutexes than what I have implemented in \
/var/cache/hrmib.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">If necessary, I can add a \
fcntl lock mechanism, but I hardly think that the HR MIB</div><div style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">return values have \
clearly defined semantics when an rpm upgrade is concurrently</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
">running while net-snmp is populating a HR MIB reply.</div><div style="margin-top: \
0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div> <blockquote type="cite"><div style="margin-top: 0px; margin-right: 0px; \
margin-bottom: 0px; margin-left: 0px; ">(My initial assumption is that this cache \
would have to</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: \
0px; margin-left: 0px; ">be built manually, using some form of cron job.<span \
class="Apple-converted-space"> </span>But I</div><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">now suspect that's not the \
case.<span class="Apple-converted-space"> </span>Can you please confirm?)</div><div \
style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; \
min-height: 14px; "><br></div> </blockquote><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; min-height: 14px; \
"><br></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">There's a one-time cost to initially populate /var/cache/hrmib, \
yes. But in</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; ">a "library push" model, that becomes an rpm, not a net-snmp, \
problem.</div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; \
margin-left: 0px; min-height: 14px; "><br></div><div style="margin-top: 0px; \
margin-right: 0px; margin-bottom: 0px; margin-left: 0px; ">Convinced yet? If more is \
______________________________________________________________________
RPM Package Manager http://rpm5.org
Developer Communication List rpm-devel@rpm5.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic