[prev in list] [next in list] [prev in thread] [next in thread]
List: activemq-users
Subject: =?UTF-8?Q?Re=3A_Artemis_isn=E2=80=99t_starting_up?=
From: Justin Bertram <jbertram () apache ! org>
Date: 2022-05-13 19:46:41
Message-ID: CAF+kE=T8x1=0VOSz_Lt1FK-PSafWQxp9E=+3gVF2GE6LutqLpw () mail ! gmail ! com
[Download RAW message or body]
> So we were thinking d that and append files from the live journal on top ?
I don't understand what you're asking here.
> I keep seeing journal and journal1 directories whereas broker.xml has
journal configured, any idea what is journal1 for ?
I'm not sure. If you're using colocation with replication then that might
be a replica backup caused by failing over and then stopping the backup
before it was able to fail-back. In that situation when the backup restarts
it will move its existing journal to a backup directory and then re-sync
with the live broker.
Out of curiosity, did you rule out patching the broker? If so, why?
Justin
On Fri, May 13, 2022 at 8:59 AM Roy Cohen <roy_cohen@hotmail.com> wrote:
> Thanks again Justin.
>
> Re ARTEMIS-2460 / Upgrade - Sigh and LOL at the same time :@)
>
> So, I was able to find the offending amq file, when I remove it from the
> journal the broker does start up just fine however all queues are showing
> zero count, whereas with a colocated backup restore we at least got a
> "snapshot" of the day of the backup. So we were thinking d that and append
> files from the live journal on top ?
>
> Another question: I keep seeing journal and journal1 directories whereas
> broker.xml has journal configured, any idea what is journal1 for ?
>
>
>
> > On 13 May 2022, at 14:48, Justin Bertram <jbertram@apache.org> wrote:
> >
> > > Would it be possible to just add more files to the journal from the
> > colocated backup to recover missing messages ?
> >
> > If you had a backup broker then it's certainly possible you could replace
> > the offending journal with the journal from the backup. However, I
> wouldn't
> > copy/replace individual journal files. You'd want to do the whole thing.
> >
> > > Alternatively is there a way to know in which journal file that
> offending
> > prepared transaction is, accept the fact that journal file is gone and
> have
> > it removed from journal ?
> >
> > You might be able to comb through the output of the 'data print' command
> to
> > find which file it's in, but I'm not certain even that would be
> effective.
> >
> > In my opinion patching the broker is by far the simplest option (assuming
> > you know Java). It's really just a couple of lines.
> >
> > > While I understand we are far behind sadly that is not something we can
> > do quickly.
> >
> > I've worked in large organizations where upgrading things was difficult
> so
> > I can empathize. That said, it's important to stay up-to-date if you're
> > relying on community (i.e. free) support. If you had commercial support
> > that would, of course, be a different story. You were commenting on
> > ARTEMIS-2460 [1] almost 6 months ago that you were looking at upgrading.
> >
> >
> > Justin
> >
> > [1] https://issues.apache.org/jira/browse/ARTEMIS-2460
> >
> > On Thu, May 12, 2022 at 6:05 PM Roy Cohen <roy_cohen@hotmail.com> wrote:
> >
> > > Alternatively is there a way to know in which journal file that
> offending
> > > prepared transaction is, accept the fact that journal file is gone and
> have
> > > it removed from journal ?
> > >
> > > On 12 May 2022, at 23:08, Roy Cohen <roy_cohen@hotmail.com> wrote:
> > >
> > > Thank you Justin.
> > >
> > > While I understand we are far behind sadly that is not something we can
> do
> > > quickly.
> > >
> > > I did manage to restore the broker using its colocated backup binding
> from
> > > the other broker and it is back up.
> > >
> > > Question: Would it be possible to just add more files to the journal
> from
> > > the colocated backup to recover missing messages ?
> > >
> > >
> > >
> > > > On 12 May 2022, at 21:02, Justin Bertram <jbertram@apache.org> wrote:
> > > >
> > > > As you know, this issue was fixed via ARTEMIS-1764 [1] in 2.6.0. The
> fix
> > > is
> > > > pretty straightforward as it just ignores the prepared transaction
> > > causing
> > > > the problem.
> > > >
> > > > At the moment I see two ways for you to proceed.
> > > >
> > > > 1) Download the latest version of ActiveMQ Artemis [2], create a new
> > > > instance, copy your journal to the instance's, start that instance, and
> > > > then move messages from that instance back to your 1.3.0 broker (e.g.
> via
> > > > Camel, JMS bridge, etc.).
> > > > 2) Download the source-code for 1.3.0 [3], apply the fix [4], build the
> > > > artemis-server module (cd artemis-server; mvn clean install
> -DskipTests),
> > > > deploy the fixed artemis-server-1.3.0.jar to your broker
> > > >
> > > > Aside from that, I strongly recommend you upgrade your broker to the
> > > latest
> > > > version. Version 1.3.0 was released in June 2016, almost 6 years ago
> now.
> > > > There's been 36 releases since then.
> > > >
> > > >
> > > > Justin
> > > >
> > > > [1] https://issues.apache.org/jira/browse/ARTEMIS-1764
> > > > [2] https://activemq.apache.org/components/artemis/download/
> > > > [3]
> > > >
> > >
> https://archive.apache.org/dist/activemq/activemq-artemis/1.3.0/apache-artemis-1.3.0-source-release.tar.gz
>
> > > > [4]
> > > >
> > >
> https://github.com/apache/activemq-artemis/commit/f8547aecf1999c8675f634413ee49f4b0edb64d5
>
> > > >
> > > >
> > > > On Wed, May 11, 2022 at 5:35 PM Roy Cohen <roy_cohen@hotmail.com>
> wrote:
> > > >
> > > > > Hi All
> > > > >
> > > > > We have an urgent situation with our Artemis brokers pair in
> production
> > > > > which somehow has left 01 without any DLQs and 02 in a state where
> it's
> > > not
> > > > > starting up.
> > > > >
> > > > > The one which isn't starting up is throwing this Exception,
> > > unfortunately
> > > > > I haven't been able to find much online. Any help would be greatly
> > > > > appreciated
> > > > >
> > > > > 2022-05-11 20:35:38,673 ERROR
> [org.apache.activemq.artemis.core.server]
> > > > > (main) AMQ224000: Failure in initialisation:
> > > > > java.lang.IllegalStateException: Cannot find queue with id 80
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.core.server.impl.PostOfficeJournalLoader.handlePreparedAcknowledge(PostOfficeJournalLoader.java:281)
>
> > > > > [artemis-server-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadPreparedTransactions(AbstractJournalStorageManager.java:1649)
>
> > > > > [artemis-server-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.loadMessageJournal(AbstractJournalStorageManager.java:1167)
>
> > > > > [artemis-server-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:2054)
>
> > > > > [artemis-server-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1927)
>
> > > > > [artemis-server-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.core.server.impl.SharedNothingLiveActivation.run(SharedNothingLiveActivation.java:100)
>
> > > > > [artemis-server-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.core.server.impl.ColocatedActivation.run(ColocatedActivation.java:108)
>
> > > > > [artemis-server-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:427)
>
> > > > > [artemis-server-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:394)
>
> > > > > [artemis-jms-server-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.integration.FileBroker.start(FileBroker.java:65)
> > > > > [artemis-cli-1.3.0.jar:1.3.0]
> > > > > at
> > > > > org.apache.activemq.artemis.cli.commands.Run.execute(Run.java:83)
> > > > > [artemis-cli-1.3.0.jar:1.3.0]
> > > > > at
> > > > >
> > >
> org.apache.activemq.artemis.cli.Artemis.internalExecute(Artemis.java:110)
> > > > > [artemis-cli-1.3.0.jar:1.3.0]
> > > > > at
> > > > > org.apache.activemq.artemis.cli.Artemis.execute(Artemis.java:72)
> > > > > [artemis-cli-1.3.0.jar:1.3.0]
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > [rt.jar:1.8.0_191]
> > > > > at
> > > > >
> > >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > [rt.jar:1.8.0_191]
> > > > > at
> > > > >
> > >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> > > > > [rt.jar:1.8.0_191]
> > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > [rt.jar:1.8.0_191]
> > > > > at
> > > > > org.apache.activemq.artemis.boot.Artemis.execute(Artemis.java:122)
> > > > > [artemis-boot.jar:1.3.0]
> > > > > at org.apache.activemq.artemis.boot.Artemis.main(Artemis.java:45)
> > > > > [artemis-boot.jar:1.3.0]
> > > > >
> > > > >
> > >
> > >
>
>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic