[prev in list] [next in list] [prev in thread] [next in thread] 

List:       activemq-users
Subject:    SQL Exception after failover using shared filesystem master/slave
From:       mmp <m.i.p () gmx ! at>
Date:       2008-03-27 10:12:48
Message-ID: 16323606.post () talk ! nabble ! com
[Download RAW message or body]


Hi,

We are having two boxes, with 4 brokers each, which are responsible for one
queue. Failover works most of the time, but sometimes after a broker shuts
down, the one, which is waiting for the lock throws following exception:

java.io.IOException: Failed to start database 'derbydb', see the next
exception for details.
        at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
        at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
  at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.createAdapter(JDBCPersistenceAdapter.java:294)
  at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getAdapter(JDBCPersistenceAdapter.java:249)
  at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.start(JDBCPersistenceAdapter.java:155)
  at
org.apache.activemq.store.journal.JournalPersistenceAdapter.start(JournalPersistenceAdapter.java:226)
  at
org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:1549)
        at
org.apache.activemq.broker.BrokerService.createBroker(BrokerService.java:1502)
        at
org.apache.activemq.broker.BrokerService.getBroker(BrokerService.java:531)
        at
org.apache.activemq.broker.BrokerService.start(BrokerService.java:440)
        at
org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:46)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:128
 )
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1257)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:438)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:383)
  at java.security.AccessController.doPrivileged(Native Method)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:353)
  at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:245)
  at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:169)
  at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:242)
  at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
  at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:400)
  at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:736)
  at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:369)
  at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
  at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
  at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:91)
  at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:51)
        at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
        at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
        at
org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:112)
        at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
        at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51)
        at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:104)
        at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51)
        at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:222)
        at org.apache.activemq.console.Main.main(Main.java:106)
Caused by: SQL Exception: Failed to start database 'derbydb', see the next
exception for details.
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown
Source)
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown
Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
        at
org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown
Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown
Source)
        at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Unknown
Source)
        at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
        at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown
Source)
        at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown
Source)
        at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
                
        ... 40 more
ERROR: java.lang.RuntimeException: Failed to execute start task. Reason:
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class
path resource [activemq.xml]: Invocation of init method failed; nested ex
eption is java.io.IOException: Failed to start database 'derbydb', see the
next exception for details.
java.lang.RuntimeException: Failed to execute start task. Reason:
org.springframework.beans.factory.BeanCreationException: Error creating bean
with n
me 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class path
resource [activemq.xml]: Invocation of init method failed; nested exception
is java.io.IOException: Failed to start database 'derbydb', see the next
exception for details.
        at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:99)
        at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51)
        at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:104)
        at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51)
        at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:222)
        at org.apache.activemq.console.Main.main(Main.java:106)
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' de
ined in class path resource [activemq.xml]: Invocation of init method
failed; nested exception is java.io.IOException: Failed to start database
'derb
db', see the next exception for details.
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1260)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:438)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:383)
  at java.security.AccessController.doPrivileged(Native Method)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:353)
  at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:245)
  at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:169)
  at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:242)
  at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
  at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:400)
  at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:736)
  at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:369)
  at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
  at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
  at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:91)
  at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:51)
        at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
        at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
        at
org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:112)
        at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
        ... 10 more
Caused by: java.io.IOException: Failed to start database 'derbydb', see the
next exception for details.
        at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
        at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
  at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.createAdapter(JDBCPersistenceAdapter.java:294)
  at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getAdapter(JDBCPersistenceAdapter.java:249)
  at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.start(JDBCPersistenceAdapter.java:155)
  at
org.apache.activemq.store.journal.JournalPersistenceAdapter.start(JournalPersistenceAdapter.java:226)
  at
org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:1549)
        at
org.apache.activemq.broker.BrokerService.createBroker(BrokerService.java:1502)
        at
org.apache.activemq.broker.BrokerService.getBroker(BrokerService.java:531)
        at
org.apache.activemq.broker.BrokerService.start(BrokerService.java:440)
        at
org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:46)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:128
 )
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1257)
                
        ... 29 more
Caused by: SQL Exception: Failed to start database 'derbydb', see the next
exception for details.
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown
Source)
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown
Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
        at
org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown
Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown
Source)
        at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Unknown
Source)
        at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
        at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown
Source)
        at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown
Source)
        at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
                
        ... 40 more
ERROR: java.lang.Exception:
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'org.apache.activemq.xbean.XBeanBr
kerService#0' defined in class path resource [activemq.xml]: Invocation of
init method failed; nested exception is java.io.IOException: Failed to sta
t database 'derbydb', see the next exception for details.
java.lang.Exception:
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'org.apache.activemq.xbean.XBeanBrokerSer
ice#0' defined in class path resource [activemq.xml]: Invocation of init
method failed; nested exception is java.io.IOException: Failed to start data
ase 'derbydb', see the next exception for details.
        at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:100)
        at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51)
        at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:104)
        at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:51)
        at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:76)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:222)
        at org.apache.activemq.console.Main.main(Main.java:106)
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0' de
ined in class path resource [activemq.xml]: Invocation of init method
failed; nested exception is java.io.IOException: Failed to start database
'derb
db', see the next exception for details.
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1260)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:438)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory$1.run(AbstractAutowireCapableBeanFactory.java:383)
  at java.security.AccessController.doPrivileged(Native Method)
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:353)
  at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:245)
  at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:169)
  at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:242)
  at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:164)
  at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:400)
  at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:736)
  at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:369)
  at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
  at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
  at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:91)
  at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:51)
        at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
        at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
        at
org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:112)
        at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
        ... 10 more
Caused by: java.io.IOException: Failed to start database 'derbydb', see the
next exception for details.
        at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:45)
        at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:61)
  at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.createAdapter(JDBCPersistenceAdapter.java:294)
  at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.getAdapter(JDBCPersistenceAdapter.java:249)
  at
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.start(JDBCPersistenceAdapter.java:155)
  at
org.apache.activemq.store.journal.JournalPersistenceAdapter.start(JournalPersistenceAdapter.java:226)
  at
org.apache.activemq.broker.BrokerService.createRegionBroker(BrokerService.java:1549)
        at
org.apache.activemq.broker.BrokerService.createBroker(BrokerService.java:1502)
        at
org.apache.activemq.broker.BrokerService.getBroker(BrokerService.java:531)
        at
org.apache.activemq.broker.BrokerService.start(BrokerService.java:440)
        at
org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:46)
  at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:128
 )
        at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1257)
                
        ... 29 more
Caused by: SQL Exception: Failed to start database 'derbydb', see the next
exception for details.
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown
Source)
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown
Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown
Source)
        at
org.apache.derby.impl.jdbc.EmbedConnection.newSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown
Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown
Source)
        at org.apache.derby.jdbc.Driver30.getNewEmbedConnection(Unknown
Source)
        at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
        at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown
Source)
        at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown
Source)
        at
org.apache.activemq.store.jdbc.TransactionContext.getConnection(TransactionContext.java:54)
                
        ... 40 more

After that, clients can not establish a connection and the newly started
broker, which throws the exception is holding the lock. Restarting the
original broker, also doesn't get the lock. It worked fine, after sending
about 34 000 000 messages over the night, having two failovers (which
worked).

This is the config of one the brokers:

  <broker xmlns="http://activemq.org/config/1.0"
brokerName="localhost-My-QUEUE" persistent="false" useJmx="true">
    <!-- The transport connectors ActiveMQ will listen to -->
    <transportConnectors>
       <transportConnector name="tc-localhost-My-QUEUE"
uri="tcp://localhost:61616"/>
    </transportConnectors>

    <persistenceAdapter>
   	<journaledJDBC dataDirectory="\\192.0.0.2\My-QUEUE"/>
    </persistenceAdapter>

    <managementContext>
	<managementContext connectorPort="2011"/>
    </managementContext>
  </broker>


We are using a Windows XP SP2 Share with NTFS.

thx in advance for any help



-- 
View this message in context: \
http://www.nabble.com/SQL-Exception-after-failover-using-shared-filesystem-master-slave-tp16323606s2354p16323606.html
 Sent from the ActiveMQ - User mailing list archive at Nabble.com.


[prev in list] [next in list] [prev in thread] [next in thread] 

Configure | About | News | Add a list | Sponsored by KoreLogic