[prev in list] [next in list] [prev in thread] [next in thread] 

List:       struts-user
Subject:    Re: Issue with ExecuteAndWait when using Spring Session (Redis)
From:       "Burton Rhodes" <burtonrhodes () gmail ! com>
Date:       2023-09-25 15:27:21
Message-ID: em090cf4a7-abb3-4030-9a28-9a257c45b268 () 3f3cef72 ! com
[Download RAW message or body]

Thanks Dale for the detailed responses.  I don't disagree with your 
suggestion as we making plans to migrate our platform to a React 
architecture, but our current application is robust and eliminating all 
ExecuteAndWait actions would be a monumental task - one that doesn't 
make sense in this scenario.  That said, I should have some time to 
investigate this issue further over the next few days and will report 
back.


------ Original Message ------
From "Dale Newfield" <dale@newfield.org>
To "Dale Newfield" <Dale@newfield.org>
Cc "Struts Users Mailing List" <user@struts.apache.org>
Date 9/24/2023 12:32:08 PM
Subject Re: Issue with ExecuteAndWait when using Spring Session (Redis)

> Depending on your deployment architecture, this exec and wait may no longer be the \
> right strategy.  Yes, by managing the executor pool you can bound how many \
> non-http-serving threads (jobs) are executing concurrently on your web server's \
> host, but it is all premised on using your own hardware.  If running "in the \
> cloud", beware that this doesn't lead you to overprovison  (overspend on) always-on \
> infrastructure...and thus limit your ability to scale dynamically for variable load \
> requirements. 
> These days I'd capture the data describing the pending work items somewhere \
> (db/s3/kinesis/lots of other possibilities) and ensure some resilient scheme \
> translates that into a broadcast message for each.  Probably subscribe a queue to \
> that broadcast and use that to trigger the work in parallel.  If queue listeners \
> are executing on ec2/ecs/eks, perhaps scale as that queue size grows.  Or most \
> likely execute the steps with serverless lambdas, potentially with massive \
> concurrency and low cost. Today there are lots of alternatives, depending on the \
> workload: step functions, etc.   Usually you can find a way to ensure at-least-once \
> execution of each task, so this other side of the coin is to make the work \
> idempotent so duplicative execution cannot introduce issues.  Perhaps a DLQ for \
> problematic jobs.  Make sure something monitors that DLQ. 
> When exec-and-wait was originally implemented we lived in an only intermittently \
> connected world and owned all the hardware.  I used it to sync up work done by a \
> disconnected instance with "the real, system" upon reconnection.  (For me this \
> meant photographers at parties, building engagement after the party by virtualizing \
> the real world connections.  It displayed photos live during the event, and allowed \
> assistants trailing behind the photographers to tag event-goers in those photos (by \
> scanning barcodes on their wristbands/entering account ids/collecting email \
> addresses on the spot to create accounts).  This allowed us to email event photos \
> to the subjects in them by the next morning, and gave participants an easy way to \
> connect with each other afterwards just by having a photographer take their picture \
> together.  At the time, facial recognition was just becoming possible and was also \
> leveraged.  But back to the technical side this meant I ran either ran a full \
> instance (db/struts web server/etc) at the front door managing barcode assignments \
> to match prepaid ticket holders, and supporting the whole event via wifi (I'd plug \
> in repeaters as necessary thru the venue and/or stick battery-powered ones in the \
> back pockets of security guards) or that I ran a full instance on the device \
> carried around by the photographer's assistant. 
> This was all before tablets/smartphones/social networks/etc. Today we live in a \
> very different world--always connected, cloud based, etc.  Some tools/techniques \
> still apply.  Many now have much better solutions (more \
> reliable/resilient/scalable/simpler; less expensive to build/launch/operate/etc.).  \
> YMMV. 
> -Dale
> 
> > On Sep 23, 2023, at 9:52 PM, Dale Newfield <Dale@newfield.org> wrote:
> > 
> > If it runs again then look at the logic where it's gotta decide whether this \
> > is the initial call or if the job is already running and this  p my try just a \
> > status check.  Is it looking at the request to figure that out?  If so, and the \
> > expected info isn't found, walk the data path to figure out where it is getting \
> > lost (sounds like the session filter might be a good place to start).  Is it \
> > looking in a database/shared memory/file system?  Perhaps these are not in sync?  \
> > Perhaps the long running job is doing everything in a single transaction, not \
> > committed until the very end, and the "I'm running!" flag set within the \
> > transaction is not visible unless the query is READ_UNCOMMITTED? 
> > This run/not decision is critical, so it should be determined atomically.  \
> > Perhaps one transaction for the decision (which either concludes "report status" \
> > or ensures its record is visible to any other query) and others for each \
> > separable unit of work within the long running job? 
> > Oh, yeah, you said Redis.  I've somehow managed to avoid that, so I don't know \
> > what persistent data check/update can be done atomically, but that's where you're \
> > hanging your hat.  Maybe the response to your "is anyone else already running \
> > this job?" query is being cached, and so the first "no" response is being \
> > returned every time? 
> > So many places this decision could go wrong in a distributed platform.  Every \
> > possible race condition will eventually happen.  Can you confidently say that two \
> > concurrent requests on distinct hosts will always result in a single "winner"? 
> > (Then later on there's a "liveness" detector you'll need to figure out in case \
> > the winner fails.  Perhaps it should periodically update that record to show when \
> > it was last known to be running? (How far into the job it is right then would \
> > also be great to snapshot.)  if that time stamp is older than X, is it safe to \
> > assume it failed and to elect another leader to take its place?  Make sure time \
> > zone differences on different hosts can't impact the timeout checks.  Is it safe, \
> > even in the face of a network disconnection, so eventually the evicted leader can \
> > reappear?  Would one of them detect the issue and kill itself?) 
> > -Dale
> > 
> > > On Sep 23, 2023, at 4:23 PM, Burton Rhodes <burtonrhodes@gmail.com> wrote:
> > > 
> > > I am attempting to implement a centralized session store to an existing \
> > > application using Spring Session (Redis) and Struts. I have everything working \
> > > on a basic level, but I am running into an issue when a Struts Action uses \
> > > ExecuteAndWait.  The refresh attempts don't seem to pick up the running action \
> > > using the provided Struts Token (at least I think this is the case). So on each \
> > > refresh, the action seems to run again as if it was the first time duplicating \
> > > database inserts, etc.  It becomes an infinite process, and Struts never \
> > > returns the final "SUCCESS" result JSP page to the browser.  It's worth noting \
> > > that if I disable the Spring Session filter, everything works fine. Anyone have \
> > > a clue as to what might be going on or how I might begin to troubleshoot? 
> > > I've included a thread dump so you can see the filters involved. The \
> > > "SessionRepositoryFilter" is where the HttpSession is wrapped by Spring (Redis) \
> > > and is before any Struts filters. I've also included a basic action definition \
> > > that is causing the issue (although this is happening on all actions that use \
> > > the ExecuteAndWait logic). Any help is appreciated. 
> > > ** Example Action Definition **
> > > 
> > > <action name="ContactBatchCategories_update" \
> > > class="com.afs.web.struts.action.contact.ContactBatchCategoriesAction" \
> > > method="update"> <interceptor-ref name="myDefaultStack"/>
> > > <interceptor-ref name="openSessionExecuteAndWaitInterceptor" >
> > > <param name="delay">0</param>
> > > </interceptor-ref>
> > > <result name="input">/struts/search/actions/contactBatchCategories_modal.jsp</result>
> > >  <result name="wait">/struts/common/progressMonitorWait_modal.jsp</result>
> > > <result>/struts/common/progressMonitorSuccess_modal.jsp</result>
> > > </action>
> > > 
> > > 
> > > ** Thread Dump **
> > > 
> > > java.lang.Thread.State: RUNNABLE
> > > at com.afs.web.struts.action.contact.ContactBatchCategoriesAction.prepare(ContactBatchCategoriesAction.java:62)
> > >  at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)
> > >  at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
> > >  at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:154)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
> > >  at com.afs.web.config.struts.StrutsAccountInterceptor.intercept(StrutsAccountInterceptor.java:67)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
> > >  at com.afs.web.config.struts.StrutsSessionInterceptor.intercept(StrutsSessionInterceptor.java:44)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
> > >  at com.afs.web.common.struts.interceptor.ExceptionInterceptor.intercept(ExceptionInterceptor.java:33)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.executeConditional(DefaultActionInvocation.java:299)
> > >  at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:253)
> > >  at org.apache.struts2.factory.StrutsActionProxy.execute(StrutsActionProxy.java:48)
> > >  at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:651)
> > > at org.apache.struts2.dispatcher.ExecuteOperations.executeAction(ExecuteOperations.java:79)
> > >  at org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.handleRequest(StrutsPrepareAndExecuteFilter.java:157)
> > >  at org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.tryHandleRequest(StrutsPrepareAndExecuteFilter.java:140)
> > >  at org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:128)
> > >  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
> > > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > >  at com.afs.web.config.filter.MyAppSessionFilter.doFilter(MyAppSessionFilter.java:85)
> > >  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
> > >  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
> > >  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
> > > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:352)
> > >  at org.springframework.security.web.access.intercept.AuthorizationFilter.doFilter(AuthorizationFilter.java:100)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
> > >  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:131)
> > >  at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:85)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
> > >  at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:164)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:151)
> > >  at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:129)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
> > >  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107)
> > >  at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90)
> > >  at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75)
> > >  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
> > >  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62)
> > >  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42)
> > >  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> > >  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:361)
> > >  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:225)
> > >  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:190)
> > >  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
> > >  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
> > >  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
> > > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > >  at org.springframework.orm.jpa.support.OpenEntityManagerInViewFilter.doFilterInternal(OpenEntityManagerInViewFilter.java:186)
> > >  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
> > >  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
> > > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > >  at org.eclipse.jetty.websocket.servlet.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:170)
> > >  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
> > > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > >  at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:142)
> > >  at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82)
> > >  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354)
> > >  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267)
> > >  at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)
> > > at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
> > >  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
> > > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
> > >  at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
> > > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > >  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
> > >  at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
> > >  at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
> > >  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
> > >  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
> > >  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
> > > at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
> > >  at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
> > >  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
> > >  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
> > >  at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
> > >  at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:51)
> > > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
> > >  at org.eclipse.jetty.server.Server.handle(Server.java:563)
> > > at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
> > > at org.eclipse.jetty.server.HttpChannel$$Lambda$1353.1731450897.dispatch(Unknown \
> > > Source:-1) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
> > >  at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
> > > at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
> > > at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
> > >  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
> > > at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
> > >  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
> > >  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
> > >  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
> > >  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
> > >  at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy$$Lambda$1338.510268842.run(Unknown \
> > > Source:-1) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
> > >  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
> > >  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
> > >  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
> > >  at java.lang.Thread.run(Thread.java:834)
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
> For additional commands, e-mail: user-help@struts.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@struts.apache.org
For additional commands, e-mail: user-help@struts.apache.org


[prev in list] [next in list] [prev in thread] [next in thread]