[prev in list] [next in list] [prev in thread] [next in thread]
List: cyrus-info
Subject: Re: Problem with large twoskip mailboxes.db after upgrade
From: Andrzej Kwiatkowski <lists () kwiatek ! eu ! org>
Date: 2021-11-05 12:03:38
Message-ID: 53e3241a-582d-b165-0681-9996c9fe5c28 () kwiatek ! eu ! org
[Download RAW message or body]
Hi,
I've made another test. I was wondering if problem exists in
mailbox/folder name. I generated file for ctl_mboxlist -u with script:
for i in {1..375000} ; do printf
"domainname.pl"'!'"user.imap${i}-test\t0 default
imap${i}-test@domainname.pl\tlrswipkxtecdan\tcyrusadmin\tlrswipxtecda\t\n"
; done
And tested varius scenarios on stopped cyrus
50k rec : import 2min : reconstruct on one mailbox 0,4s
100k rec : import 3,31min : reconstruct on one mailbox 0,66s
250k rec : import 9min : reconstruct on one mailbox 1,63s
375k rec : import 13min : reconstruct on one mailbox 2,55s
550k rec : import 19min : reconstruct on one mailbox 3,4s
It shows that amount of records in mailboxes.db
have coincidence with time needed for operations.
Regards,
AK
W dniu 05.11.2021 o 09:12, Andrzej Kwiatkowski pisze:
> Hi Bron,
>
> Thanks for your attention.
> All is on Centos 8.4. For better tests on the same server we have
> installed 2.5.15 (we are using compiled version), and with 2.5
> reconstruct lasts:
> real 0m0.206s
> with full 550k records in mailboxes.db
>
> With 3.4.2 and with 3.2.8(we also installed this version) about 4s.
>
> I think that problem is with slow searching in twoskip, or some
> misconfiguration. The only difference between 2.5 and 3.x we have found
> is that in 2.5 version inside config/db directory are some DB files :
> __db.001 __db.002 __db.003
>
> and with 3.x this folder is empty.
>
> I'm not sure if this is cause, but we were cleaning this folder , and
> running reconstruct without starting master, and this files appeared.
>
> Our compilation options are:
>
> /configure --prefix=/opt/store21/imapd-2.5.15 --with-cyrus-user=cyrus
> --with-cyrus-group=cyrus --disable-gssapi
> --with-cyrus-prefix=xxxxxxxxxxxx/cyrus
> --with-service-path=xxxxxxxxxxxx/service --with-sasl=xxxxxxxxxxxx
> --sysconfdir=xxxxxxxxxxxx/etc --disable-murder
> --with-pidfile=xxxxxxxxxxxx/var/cyrus-master.pid
> --with-syslogfacility=LOCAL0 --without-perl --without-snmp --enable-idled
>
> (nothing special)
>
> and config file (imapd.conf)
>
> configdirectory: xxxxxxxxxxxxxxxxx/config
> defaultpartition: default
> partition-default: xxxxxxxxxxxxxxx/mailbox
> sievedir: xxxxxxxxxxxxxxxxx/sieve
> annotation_definitions: xxxxxxxxxxxxxxxxxxxxx/etc/imapd.annotations.conf
> servername: xxxxxxxxxxxxxxx
> hashimapspool: true
> fulldirhash: 1
> altnamespace: 1
> allowusermoves: 1
> virtdomains: 1
> unixhierarchysep: 1
> allowplaintext: 1
> sasl_mech_list: plain login
> sasl_pwcheck_method: auxprop saslauthd
> sasl_auxprop_plugin: sasldb
> admins: cyrusadmin
> defaultdomain: xxxxxxxxxxxxxxxxx
> autocreate_quota: 10485760
> autocreate_sieve_folders: Junk
> syslog_prefix: xxxxxxxxxxxxxxxxxxxxx
> lmtp_over_quota_perm_failure: on
> quotawarn: 85
> proxyservers: xxxxxxxxxxxxxxxxxxxxxxx
> proxy_authname: xxxxxxxxxxxxxxxxxxxxxxxxxx
> lmtp_admins: xxxxxxxxxxxxxxxxxxxxxxxxxx
>
> Also nothing special,
> reverseacls: true didn't helped.
>
> This is standalone instalation without murder, replication and other
> features.
>
> Regards
> AK
>
>
> W dniu 05.11.2021 o 01:30, Bron Gondwana pisze:
>> Hi Andrzej,
>>
>> What operating system is this on? I suspect maybe it's that the locks
>> or mmaps are significantly more expensive on this platform than on the
>> places I've used it.
>>
>> Also if you can give specifics of your config that might help a lot in
>> tracking down what's going on.
>>
>> I wonder if setting |reverseacls: true| in your imapd.conf might fix
>> this. If it's locking and unlocking that frequently then it seems like
>> maybe it's doing a mailbox scan that we could be avoiding.
>>
>> Cheers,
>>
>> Bron.
>>
>> On Fri, Nov 5, 2021, at 10:20, Andrzej Kwiatkowski wrote:
>>>
>>> Before Upgrade on old 2.5.x cyrus we have about 250k mailboxes with
>>> about 500k folders.
>>> We made upgrade with imapsync , because of problem with path
>>> compatibility with unixhierarchysep set on yes.
>>>
>>> After upgrade to 3.4.2 we have problem that fcntl locks on mailboxes.db
>>> are very long. For tests we have made copy of one production servers,
>>> and run without load, reconstruct for empty mailbox lasts 4-5 seconds,
>>> and strace shows us:
>>>
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>> stat("/store/usermailboxes//config/mailboxes.db", {st_mode=S_IFREG|0600,
>>> st_size=104625064, ...}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fcntl(3, F_SETLKW, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0,
>>> l_len=0}) = 0
>>> fstat(3, {st_mode=S_IFREG|0600, st_size=104625064, ...}) = 0
>>>
>>> For reconstruct single mailbox there is sometimes 27k operation for RDLCK.
>>>
>>> One of our server have database with only 50k folders, and there is
>>> everythink ok.We also made dump, restore mailboxes.db, conversion to
>>> skipstamp and back for twoskip.
>>> Without progress. With zeroskip is much worse.
>>>
>>> We also test on XFS,EXT4. Mailboxes.db on ramdisk for removing disk
>>> performance issue..
>>>
>>> Any idea ?
>>>
>>> Regards
>>>
>>> AK
>>>
>>> ------------------------------------------
>>> Cyrus: Info
>>> Permalink:
>>> https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f
>>> <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-Mf02e8d0ad71ef6d7ac900a1f>
>>> Delivery options: https://cyrus.topicbox.com/groups/info/subscription
>>> <https://cyrus.topicbox.com/groups/info/subscription>
>>>
>>
>> --
>> Bron Gondwana, CEO, Fastmail Pty Ltd
>> brong@fastmailteam.com
>>
>>
>> *Cyrus <https://cyrus.topicbox.com/latest>* / Info / see discussions
>> <https://cyrus.topicbox.com/groups/info> + participants
>> <https://cyrus.topicbox.com/groups/info/members> + delivery options
>> <https://cyrus.topicbox.com/groups/info/subscription> Permalink
>> <https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M668cd6a988cc43afb98e4755>
>>
------------------------------------------
Cyrus: Info
Permalink: https://cyrus.topicbox.com/groups/info/T61dd54ebdc77fdc5-M099f301bbd68ec13d55942d0
Delivery options: https://cyrus.topicbox.com/groups/info/subscription
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic