[prev in list] [next in list] [prev in thread] [next in thread]
List: cyrus-info
Subject: Re: Again on squatter and cyrus imap 2.5.11
From: "Gabriele Bulfon via Info" <info () cyrus ! topicbox ! com>
Date: 2021-03-12 7:59:13
Message-ID: 929669976.2391.1615535953535 () www
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi, I turned debug on, but no particular info about any fall back to full indexing.
So I modified squatter.c and rebuilt binary to have some output.
I added some output in squat_single right after the last "goto bail" else section, \
right before starting the actual incremental indexing for cycle. What I discovered is \
that all mailboxes that changed (not skipped) are being indexed incrementally (no \
goto bail for any error), but for some unknown reasons some of them don't get any \
matching UID on the old index file in the for cycle, so they all get indexed. Some of \
them are big archives of years, that receive few new mails almost daily (archived by \
the users), and here I get the whole archive being indexed and take 30-40 mins! Most \
of the other folders are correctly indexed incrementally, e.g. seeing the actual 2 \
new messages of the day indexed. What may be the reason for the following piece of \
code to never skip any matching uid? for (msgno = 1; msgno <= state->exists; \
msgno++) { uint32_t uid = index_getuid(state, msgno);
/* Scan uid_item list for matching UID (ascending order, 0 termination) \
*/ while (uid_item->uid && (uid_item->uid < uid))
uid_item++;
if ((uid_item->uid == uid) && uid_item->flagged)
continue;
/* This UID didn't appear in the old index file */
index_getsearchtext_single(state, msgno, search_text_receiver,
&data);
uid_item->flagged = 1;
}
Gabriele
Sonicle S.r.l. : http://www.sonicle.com
Music: http://www.gabrielebulfon.com
eXoplanets : https://gabrielebulfon.bandcamp.com/album/exoplanets
Da: ellie timoney <ellie@fastmail.com>
A: cyrus info <info@cyrus.topicbox.com>
Data: 4 marzo 2021 23.37.30 CET
Oggetto: Re: Again on squatter and cyrus imap 2.5.11
p.MsoNormal,p.MsoNoSpacing{margin:0}On Wed, 3 Mar 2021, at 3:44 AM, Gabriele Bulfon \
via Info wrote: Problem is that I don't see any debug output from squatter saying \
that something went wrong so it falls back. I have syslog.conf with "local6.debug \
/sonicle/var/log/imap/imapd.log" and I can see all the info logs of squatter there, \
but no debugs.
Is there anything I can do here?
Do you have "debug: yes" in imapd.conf? IIRC Cyrus won't log to LOG_DEBUG at all, \
regardless of your syslog config, unless this setting is also enabled. (I imagine \
this is so that you can leave syslog configured for debug logging all the time, and \
just turn it on and off in Cyrus as you need it/don't need it.) Cyrus / Info / see \
discussions + participants + delivery options Permalink
------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/Te58e1559834056be-M8a5dc80e0771e3e01e5c7bc6
Delivery options: https://cyrus.topicbox.com/groups/info/subscription
[Attachment #5 (text/html)]
<html><html><html><header></header><body><div style="font-family: \
tahoma,arial,helvetica,sans-serif; font-size: 14px;">Hi, I turned debug on, but no \
particular info about any fall back to full indexing.</div><div style="font-family: \
tahoma,arial,helvetica,sans-serif; font-size: 14px;">So I modified squatter.c and \
rebuilt binary to have some output.</div><div style="font-family: \
tahoma,arial,helvetica,sans-serif; font-size: 14px;">I added some output in \
squat_single right after the last "goto bail" else section, right before \
starting the actual incremental indexing for cycle.</div><div style="font-family: \
tahoma,arial,helvetica,sans-serif; font-size: 14px;">What I discovered is that all \
mailboxes that changed (not skipped) are being indexed incrementally (no goto bail \
for any error), but for some unknown reasons some of them don't get any matching \
UID on the old index file in the for cycle, so they all get indexed.</div><div \
style="font-family: tahoma,arial,helvetica,sans-serif; font-size: 14px;">Some of them \
are big archives of years, that receive few new mails almost daily (archived by the \
users), and here I get the whole archive being indexed and take 30-40 mins!<br />Most \
of the other folders are correctly indexed incrementally, e.g. seeing the actual 2 \
new messages of the day indexed.</div><div style="font-family: \
tahoma,arial,helvetica,sans-serif; font-size: 14px;">What may be the reason for the \
following piece of code to never skip any matching uid?<br /><p><span \
style="font-family: 'courier new', courier;"> for (msgno = 1; \
msgno <= state->exists; msgno++) {</span><br /><span style="font-family: \
'courier new', courier;"> uint32_t uid = \
index_getuid(state, msgno);</span><br /><span style="font-family: 'courier \
new', courier;"> /* Scan uid_item list for matching \
UID (ascending order, 0 termination) */</span><br /><span style="font-family: \
'courier new', courier;"> while (uid_item->uid \
&& (uid_item->uid < uid))</span><br /><span style="font-family: \
'courier new', courier;"> \
uid_item++;</span><p><span style="font-family: 'courier new', \
courier;"> if ((uid_item->uid == uid) && \
uid_item->flagged)</span><br /><span style="font-family: 'courier new', \
courier;"> continue;</span><p><span \
style="font-family: 'courier new', courier;"> /* \
This UID didn't appear in the old index file */</span><br /><span \
style="font-family: 'courier new', courier;"> \
index_getsearchtext_single(state, msgno, search_text_receiver,</span><br /><span \
style="font-family: 'courier new', courier;"> \
\
&data);</span><br /><span style="font-family: 'courier new', \
courier;"> uid_item->flagged = 1;</span><br /><span \
style="font-family: 'courier new', courier;"> }</span></div><div \
style="font-family: tahoma,arial,helvetica,sans-serif; font-size: \
14px;">Gabriele</div><div style="font-family: tahoma,arial,helvetica,sans-serif; \
font-size: 14px;"> </div><div id="wt-mailcard"><div> </div><div><span \
style="font-size: 14px; font-family: Helvetica;"><strong>Sonicle \
S.r.l. </strong>: <a href="http://www.sonicle.com/" \
target="_new">http://www.sonicle.com</a></span></div><div><span style="font-size: \
14px; font-family: Helvetica;"><strong>Music: </strong><a \
href="http://www.gabrielebulfon.com/" \
target="_new">http://www.gabrielebulfon.com</a></span></div><div><span \
style="font-size: 14px; font-family: \
Helvetica;"><strong>eXoplanets : </strong><a \
href="https://gabrielebulfon.bandcamp.com/album/exoplanets">https://gabrielebulfon.bandcamp.com/album/exoplanets</a></span></div><div> </div></div><div \
style="font-family: tahoma,arial,helvetica,sans-serif; font-size: \
14px;"> </div><hr /><div style="font-family: tahoma,arial,helvetica,sans-serif; \
font-size: 14px;"><br /><br /><span style="font-family: Arial, Helvetica, sans-serif; \
font-size: small;"><strong>Da:</strong> ellie timoney <ellie@fastmail.com><br \
/><strong>A:</strong> cyrus info <info@cyrus.topicbox.com><br \
/><strong>Data:</strong> 4 marzo 2021 23.37.30 CET<br /><strong>Oggetto:</strong> Re: \
Again on squatter and cyrus imap 2.5.11<br /></span><br /><br /></div><style \
type="text/css">p.MsoNormal,p.MsoNoSpacing{margin:0}</style><blockquote \
style="border-left: #000080 2px solid; margin-left: 5px; padding-left: 5px;"><div>On \
Wed, 3 Mar 2021, at 3:44 AM, Gabriele Bulfon via Info wrote:</div><blockquote \
id="qt"><div style="font-family: tahoma, arial, helvetica, sans-serif; font-size: \
14px;">Problem is that I don't see any debug output from squatter saying that \
something went wrong so it falls back.</div><div style="font-family: tahoma, arial, \
helvetica, sans-serif; font-size: 14px;">I have syslog.conf with "local6.debug \
/sonicle/var/log/imap/imapd.log" and I can see all the info logs of \
squatter there, but no debugs.</div><div style="font-family: tahoma, arial, \
helvetica, sans-serif; font-size: 14px;"> </div><div style="font-family: tahoma, \
arial, helvetica, sans-serif; font-size: 14px;">Is there anything I can do \
here?</div></blockquote><div> </div><div>Do you have "debug: yes" in \
imapd.conf? IIRC Cyrus won't log to LOG_DEBUG at all, regardless of your \
syslog config, unless this setting is also enabled. (I imagine this is so that \
you can leave syslog configured for debug logging all the time, and just turn it on \
and off in Cyrus as you need it/don't need it.)</div></blockquote><div \
id="topicbox-footer" style="margin:10px 0 0;border-top:1px solid \
#ddd;border-color:rgba(0,0,0,.15);padding:7px 0;">
<strong><a href="https://cyrus.topicbox.com/latest" \
style="color:inherit;text-decoration:none">Cyrus</a></strong> / Info / see
<a href="https://cyrus.topicbox.com/groups/info">discussions</a>
+
<a href="https://cyrus.topicbox.com/groups/info/members">participants</a>
+
<a href="https://cyrus.topicbox.com/groups/info/subscription">delivery options</a>
<a href="https://cyrus.topicbox.com/groups/info/Te58e1559834056be-M8a5dc80e0771e3e01e5c7bc6" \
style="float:right">Permalink</a> </div>
</body></html></html></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic