'Re: [mico-devel] Crash possibly caused by invalid write'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       mico-announce
Subject:    Re: [mico-devel] Crash possibly caused by invalid write
From:       Karel Gardas <kgardas () objectsecurity ! com>
Date:       2007-12-20 7:53:07
Message-ID: 476A1F63.9070808 () objectsecurity ! com
[Download RAW message or body]

Hello,

first of all, could you be so kind and try to duplicate this issue with
the latest MICO sources in the darcs repository? On linux, just install
darcs and get MICO by this command:

darcs get --set-scripts-executable http://mico.org/mico-darcs-repository
mico

The source tree is saved into the `mico' directory. You'll also probably
need to have autoconf2.13 installed and invoke ./bootstrap.sh script to
generate proper configure script before you try to compile it.

Thanks,
Karel

Gabi Mahu wrote:
> Hello all,
> 
> I am using MICO for a CORBA client application. Everything is ok, except the fact \
> that the application crashes from time to time. Using Valgrind, I managed to get \
> close to a possible cause of the crash, an invalid write in MICO. I obtained a \
> detailed log with the operations around the point of the invalid write, but I can't \
> fully understand what is going on. My guess is that threads are somehow, stepping \
> on each other's foot, at some point. 
> I am listing here some relevant information for my problem and the important parts \
>                 of the Valgrind and MICO logs
> - the MICO version is 2.3.12 with multi thread  turned on (on Linux)
> - I use a single ORB reference
> - the actual CORBA requests are made from different threads, in my client \
>                 application
> - the average number of requests made is around 2-3 per second
> - I simulate some extreme server  failure rate by turning it off and on, about \
>                 every 5 seconds; the time period between off and on state is \
>                 sometimes very small
> - every invalid write is doubled by a CORBA/TRANSIENT exception
> - the server is also MICO-based
> 
> This is the relevant part of the the Valgrind log:
> 
> ==8481== 17 errors in context 1 of 1:
> ==8481== Invalid write of size 4
> ==8481==    at 0x4828CFE: MICO::IIOPProxy::del_invoke(MICO::IIOPProxyInvokeRec*) \
> (orb_mico.h:99) ==8481==    by 0x482934F: \
> MICO::IIOPProxy::abort_invoke(CORBA::ORBInvokeRec*) (iop.cc:3368) ==8481==    by \
> 0x48306BE: MICO::IIOPProxy::kill_conn(MICO::GIOPConn*, unsigned char) (iop.h:661) \
> ==8481==    by 0x48308C3: MICO::IIOPProxy::callback(MICO::GIOPConn*, \
> MICO::GIOPConnCallback::Event) (iop.cc:4459) ==8481==    by 0x498C3AF: \
> MICO::MTDispatcher::process(MICO::msg_type*) (mt_dispatcher.cc:134) ==8481==    by \
> 0x4986C52: MICO::PassiveOperation::_run() (operation.cc:236) ==8481==    by \
> 0x498CD31: MICO::WorkerThread::_run(void*) (mt_manager.h:342) ==8481==    by \
> 0x4985896: MICOMT::Thread::_thr_startup(void*) (pthreads.cc:145) ==8481==    by \
> 0x49859B4: MICOMT::Thread::ThreadWrapper(void*) (pthreads.cc:125) ==8481==    by \
> 0xDE1370: start_thread (in /lib/tls/libpthread-2.3.4.so) ==8481==    by 0xC6B9BD: \
> clone (in /lib/tls/libc-2.3.4.so) ==8481==  Address 0xF01B3F8 is 80 bytes inside a \
> block of size 92 free'd ==8481==    at 0x4005559: operator delete(void*) \
> (vg_replace_malloc.c:244) ==8481==    by 0x47F7795: \
> CORBA::ORBInvokeRec::~ORBInvokeRec() (orb.cc:282) ==8481==    by 0x47F84C1: \
> CORBA::ORB::del_invoke(unsigned long) (stl_tree.h:172) ==8481==    by 0x47F9ED3: \
> CORBA::ORB::get_invoke_reply(CORBA::ORBInvokeRec*, ObjOut<CORBA::Object>, \
> CORBA::ORBRequest*&, short&) (orb_mico.h:109) ==8481==    by 0x485E0CA: \
> CORBA::StaticRequest::invoke() (static.cc:2201) ==8481==    by 0x4B62DB3: \
> CosNaming::NamingContext_stub::resolve(SequenceTmpl<CosNaming::NameComponent, 0> \
> const&) (CosNaming.cc:1548) ==8481==    [CORBA Request Method]
> 
> This is the relevant part of the the MICO log: 
> ...
> ORB::add_invoke (MsgId=51)
> GIOP: sending Request to [server address] msgid is 51
> GIOPConn::deref: 0x42dd500, refcnt: 1, activerefs: 3
> IIOPProxy::add_invoke: rec=0xeffc9b0, id=0xeffb1b8, msgid=51)
> MICO::GIOPConn::output (CORBA::Buffer *b)
> b: 0xeffc728
> Out Data [Actual data - ignored]
> ORB::wait for 0xeffb1b8
> MICO::GIOPConn::input_ready ()
> conn: 0x42dd500
> ev: GIOPConnCallback::InputReady
> t_mod: 0
> pool: 
> conn: 
> req:
> _activerefs: 2
> In Data [Actual data - ignored]
> IIOP: incoming data from [server address]
> GIOP: incoming Reply from [server address] for msgid 51 status is 0
> ORB::get_invoke (MsgId=51)
> IIOPProxy::pull_invoke: id=0xeffb1b8, rec = 0xeffc9b0
> IIOPProxy::handle_invoke_reply: rec=0xeffc9b0)
> IIOPProxy::del_invoke: rec = 0xeffc9b0
> MICO::IIOPProxy::exec_invoke_reply (obj=0, *req=0xa83cf10, *conn=0x42dd500)
> ORB::del_invoke (MsgId=51)
> GIOPConn::deref: 0x42dd500, refcnt: 1, activerefs: 2
> ORB::add_invoke (MsgId=52)
> GIOP: sending Request to [server address] msgid is 52
> IIOPProxy::add_invoke: rec=0xf00ab48, id=0xf008210, msgid=52)
> MICO::GIOPConn::output (CORBA::Buffer *b)
> b: 0xf00a8c0
> Out Data [Actual data - ignored]
> ORB::wait for 0xf008210
> MICO::GIOPConn::input_ready ()
> conn: 0x42dd500
> ev: GIOPConnCallback::InputReady
> t_mod: 0
> pool: 
> conn:
> req:
> _activerefs: 2
> In Data [Actual data - ignored]
> IIOP: incoming data from [server address]
> GIOP: incoming Reply from i[server address] for msgid 52 status is 1
> ORB::get_invoke (MsgId=52)
> IIOPProxy::pull_invoke: id=0xf008210, rec = 0xf00ab48
> IIOPProxy::handle_invoke_reply: rec=0xf00ab48)
> IIOPProxy::del_invoke: rec = 0xf00ab48
> MICO::IIOPProxy::exec_invoke_reply (obj=0, *req=0xb23df10, *conn=0x42dd500)
> GIOPConn::deref: 0x42dd500, refcnt: 1, activerefs: 2
> ORB::del_invoke (MsgId=52)
> MICO::GIOPConn::close_connection()
> conn: 0x42d0258
> > ActiveMsgQueue::put_msg: (0x42b0198) msg: 0xf015de8
> MICO::GIOPConn::close_connection()
> conn: 0x42dd500
> > ActiveMsgQueue::put_msg: (0x42b0198) msg: 0xf015e68
> ORB::add_invoke (MsgId=53)
> GIOP: sending Request to [server address] msgid is 53
> IIOPProxy::add_invoke: rec=0xf01c8f8, id=0xf01b3a8, msgid=53)
> ORB::wait for 0xf01b3a8
> PassiveOperation::put_msg():0xf015e68
> WorkerThread::_run: 
> PassiveOperation::_run():0xf015e68
> MTDispatcher::process
> ORBMsg::CloseConn
> IIOP: connection to [server address] closed or broken
> MICO::GIOPConn::terminate
> GIOPConn::terminated
> GIOPConn::deref: 0x42dd500, refcnt: 0, activerefs: 0
> MICO::GIOPConnCallback::send_orb_msg (GIOPConn *conn)
> conn: 0x42dd500
> ev: 0
> > ActiveMsgQueue::put_msg: (0x42b0198) msg: 0xf0397c0
> PassiveOperation::put_msg():0xf015de8
> WorkerThread::_run: 
> PassiveOperation::_run():0xf015de8
> MTDispatcher::process
> ORBMsg::CloseConn
> IIOP: connection to [server address] closed or broken
> PassiveOperation::put_msg():0xf0397c0
> > ActiveMsgQueue::check_msg: (0x42b0198) msg: 
> void_array::__fast_insert (0xf015ea8):    return 0
> > ActiveMsgQueue::check_msg: (0x42b0198) msg: 
> MICO::GIOPConn::terminate
> GIOPConn::terminated
> GIOP: invocation(0xf01b3a8) aborted
> IIOPProxy::pull_invoke: id=0xf01b3a8, rec = 0xf01c8f8
> WorkerThread::_run: 
> PassiveOperation::_run():0xf0397c0
> MTDispatcher::process
> ORBMsg::KillConn
> GIOPCodec::~GIOPCodec: 0x42dd330
> > ActiveMsgQueue::check_msg: (0x42b0198) msg: 
> void_array::__fast_insert (0xf039848):    return 1
> > ActiveMsgQueue::check_msg: (0x42b0198) msg: 
> ORB::del_invoke (MsgId=53)
> IIOPProxy::del_invoke: rec = 0xf01c8f8
> GIOPConn::deref: 0x42d0258, refcnt: 1, activerefs: 0
> > ActiveMsgQueue::check_msg: (0x42b0198) msg: 
> void_array::__fast_insert (0xf0382d0):    return 2
> > ActiveMsgQueue::check_msg: (0x42b0198) msg: 
> ORB::add_invoke (MsgId=54)
> IIOP: making new GIOP 1.0 connection to [server address]
> IIOP: connect to [server address] failed: Connection refused
> ORB::wait for 0x10e86300
> ORB::del_invoke (MsgId=54)
> ORB::add_invoke (MsgId=55)
> IIOP: making new GIOP 1.0 connection to [server address]
> IIOP: connect to [server address] failed: Connection refused
> ORB::wait for 0x10e917a0
> ORB::del_invoke (MsgId=55)
> ...
> 
> Somewhere in the middle, the CORBA server is stopped. I am not very familiar with \
> MICO internals, so this is a bit confusing for me. Could someone please help me \
> understand what exactly is happening! 
> Thank you for your time,
> Gabriel Mahu
> 
> 
> 
> 
> 
> 
> ____________________________________________________________________________________
>  Be a better friend, newshound, and 
> know-it-all with Yahoo! Mobile.  Try it now.  \
> http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ  
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Mico-devel mailing list
> Mico-devel@mico.org
> http://www.mico.org/mailman/listinfo/mico-devel

-- 
Karel Gardas                  kgardas@objectsecurity.com
ObjectSecurity Ltd.           http://www.objectsecurity.com
_______________________________________________
Mico-devel mailing list
Mico-devel@mico.org
http://www.mico.org/mailman/listinfo/mico-devel

[prev in list] [next in list] [prev in thread] [next in thread] 