[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openldap-technical
Subject:    Re: python-lmdb crashing ( d <= env->me_pglast' failed in mdb_freelist_save()) - unusable 300gb data
From:       Howard Chu <hyc () symas ! com>
Date:       2023-07-18 15:46:31
Message-ID: dd4fbacc-18f5-61fc-c9b2-97b89d6dfdeb () symas ! com
[Download RAW message or body]

mark jayson wrote:
> Hello,
> 
> I've created a program that writes data to an lmdb database using python bindings. \
> I've written about 300gb worth of data but when I was re-reading the database by \
> running my script again, it crashes immediately. Did a backtrace in gdb and this is \
> what I got: 
> Really have no clue what's going on here. I suspect that the server has rebooted \
> (or was rebooted by someone while I was running the script but I thought lmdb is \
> crash-proof.

It is. Extensively tested so, but storage devices can still lie about whether they \
successfully wrote data.

> Do you have any idea how to best troubleshoot this? The database is worth 300gb+ so \
> it's not easy to share it, not to mention it contains sensitive information.

Try using mdb_dump to see if it will backup the contents without crashing.

Also, when asking for help you should specify exactly what version of liblmdb you \
used.

The backtrace is a bit odd because normally just opening an environment shouldn't \
require committing any write txns.
> 
> Starting program: /usr/bin/python3 read_events.py -d jul012022-23/
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> build/lib/mdb.c:3274: Assertion 'len >= 0 && id <= env->me_pglast' failed in \
> mdb_freelist_save() 
> Program received signal SIGABRT, Aborted.
> __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350283264) at \
>                 ./nptl/pthread_kill.c:44
> 44./nptl/pthread_kill.c: No such file or directory.
> (gdb) backtrace
> #0   __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350283264) at \
> ./nptl/pthread_kill.c:44 #1   __pthread_kill_internal (signo=6, \
> threadid=140737350283264) at ./nptl/pthread_kill.c:78 #2   __GI___pthread_kill \
> (threadid=140737350283264, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3   \
> 0x00007ffff7c96476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 \
> #4   0x00007ffff7c7c7f3 in __GI_abort () at ./stdlib/abort.c:79 #5   \
> 0x00007ffff62fb3e2 in mdb_assert_fail (env=0x5555565b49f0, \
> expr_txt=expr_txt@entry=0x7ffff62fe308 "len >= 0 && id <= env->me_pglast",   \
> func=func@entry=0x7ffff62fe930 <__func__.13> "mdb_freelist_save", \
> line=line@entry=3274, file=0x7ffff62fe010 "build/lib/mdb.c") at \
> build/lib/mdb.c:1545 #6   0x00007ffff62f10df in mdb_freelist_save \
> (txn=0x555556752950) at build/lib/mdb.c:3274 #7   mdb_txn_commit \
> (txn=0x555556752950) at build/lib/mdb.c:3646 #8   0x00007ffff62f369b in \
> txn_db_from_name (env=env@entry=0x7ffff61c8090, name=<optimized out>, flags=262144) \
> at lmdb/cpython.c:1017 #9   0x00007ffff62f6e1d in env_open_db (self=0x7ffff61c8090, \
> args=<optimized out>, kwds=<optimized out>) at lmdb/cpython.c:1665 #10 \
> 0x00005555556b23a9 in ?? () #11 0x0000555555699c14 in _PyEval_EvalFrameDefault ()
> #12 0x0000555555696176 in ?? ()
> #13 0x000055555578bc56 in PyEval_EvalCode ()
> #14 0x00005555557b8b18 in ?? ()
> #15 0x00005555557b196b in ?? ()
> #16 0x00005555557b8865 in ?? ()
> #17 0x00005555557b7d48 in _PyRun_SimpleFileObject ()
> #18 0x00005555557b7a43 in _PyRun_AnyFileObject ()
> #19 0x00005555557a8c3e in Py_RunMain ()
> #20 0x000055555577ebcd in Py_BytesMain ()
> #21 0x00007ffff7c7dd90 in __libc_start_call_main (main=main@entry=0x55555577eb90, \
>                 argc=argc@entry=14, argv=argv@entry=0x7fffffffe378) at
> ../sysdeps/nptl/libc_start_call_main.h:58
> #22 0x00007ffff7c7de40 in __libc_start_main_impl (main=0x55555577eb90, argc=14, \
> argv=0x7fffffffe378, init=<optimized out>, fini=<optimized out>, \
> rtld_fini=<optimized out>, stack_end=0x7fffffffe368) at ../csu/libc-start.c:392
> #23 0x000055555577eac5 in _start ()
> (gdb) client_loop: send disconnect: Broken pipe
> 
> 
> For what it's worth, this is how I've opened the database environment:
> 
> dbenv = lmdb.open(db_dir, map_size=1099511627776, max_dbs=11, readahead=False)  
> 
> 


-- 
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/


[prev in list] [next in list] [prev in thread] [next in thread]