ActiveMQ Artemis 线程转储

4

我有一个与ActiveMQ Artemis 2.10.0版本相关的问题,它正在运行在Docker上。正是一个线程问题(日志如下)。在日志的某个时刻,我看到了这个条目:

Component org.apache.activemq.artemis.core.io.buffer.TimedBuffer is expired on path 0

接下来有一个条目,它宣布经纪人进程将被杀死:

AMQ224079: The process for the virtual machine will be killed, as component org.apache.activemq.artemis.core.io.buffer.TimedBuffer@51b01960 is not responsive

稍后,有一系列带有线程进程 ID 的条目,从 id = 2 开始到线程 id 200(即将变量 thread-pool-max-size=200 scheduled-thread-pool-max-size=20 设置为所述的值,见文档)。

Artemis 网站上我发现,如果 broker 处于不稳定状态,则可以关闭 broker。

为什么会发生这种情况?

日志:

 WARN  [org.apache.activemq.artemis.utils.critical.CriticalMeasure] Component org.apache.activemq.artemis.core.io.buffer.TimedBuffer is expired on path 0
 ERROR [org.apache.activemq.artemis.core.server] AMQ224079: The process for the virtual machine will be killed, as component org.apache.activemq.artemis.core.io.buffer.TimedBuffer@51b01960 is not responsive
 WARN  [org.apache.activemq.artemis.core.server] AMQ222199: Thread dump: *******************************************************************************
Complete Thread dump
"Reference Handler" Id=2 RUNNABLE
        at java.base@11.0.5/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
        at java.base@11.0.5/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
        at java.base@11.0.5/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)


"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@1d3b0b50
        at java.base@11.0.5/java.lang.Object.wait(Native Method)
        -  waiting on java.lang.ref.ReferenceQueue$Lock@1d3b0b50
        at java.base@11.0.5/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
        at java.base@11.0.5/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
        at java.base@11.0.5/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)


"Signal Dispatcher" Id=4 RUNNABLE


"Common-Cleaner" Id=18 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@a9de548
        at java.base@11.0.5/java.lang.Object.wait(Native Method)
        -  waiting on java.lang.ref.ReferenceQueue$Lock@a9de548
        at java.base@11.0.5/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
        at java.base@11.0.5/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
        at java.base@11.0.5/java.lang.Thread.run(Thread.java:834)
        at java.base@11.0.5/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:134)


"oneagentsubpathsender 1.189.238.20200508-120527" Id=20 RUNNABLE (in native)


"oneagentperiodiceventsmanaged" Id=21 RUNNABLE (in native)


"oneagentautosensor" Id=22 RUNNABLE


"oneagentperiodicurls" Id=23 RUNNABLE (in native)


"oneagentperiodicrequests" Id=24 RUNNABLE (in native)


"ActiveMQ Artemis Server Shutdown Timer" Id=27 TIMED_WAITING on java.util.TaskQueue@3cc6ec06
        at java.base@11.0.5/java.lang.Object.wait(Native Method)
        -  waiting on java.util.TaskQueue@3cc6ec06
        at java.base@11.0.5/java.util.TimerThread.mainLoop(Timer.java:553)
        at java.base@11.0.5/java.util.TimerThread.run(Timer.java:506)

"Thread-0 (-scheduled-threads)" Id=29 RUNNABLE
        at java.management@11.0.5/sun.management.ThreadImpl.dumpThreads0(Native Method)
        at java.management@11.0.5/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:502)
        at java.management@11.0.5/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:490)
        at org.apache.activemq.artemis.utils.ThreadDumpUtil.threadDump(ThreadDumpUtil.java:47)
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.threadDump(ActiveMQServerImpl.java:1022)
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.lambda$initializeCriticalAnalyzer$0(ActiveMQServerImpl.java:678)
        at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$$Lambda$127/0x00000001002df840.run(Unknown Source)
        at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.fireAction(CriticalAnalyzerImpl.java:155)
        at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl.check(CriticalAnalyzerImpl.java:140)
        at org.apache.activemq.artemis.utils.critical.CriticalAnalyzerImpl$1.run(CriticalAnalyzerImpl.java:53)
        at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$2.run(ActiveMQScheduledComponent.java:284)
        at org.apache.activemq.artemis.core.server.ActiveMQScheduledComponent$3.run(ActiveMQScheduledComponent.java:294)
        at java.base@11.0.5/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base@11.0.5/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base@11.0.5/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)

        Number of locked synchronizers = 1
        - java.util.concurrent.ThreadPoolExecutor$Worker@1b58ff9e


"Thread-0 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@466cf502)" Id=35 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350d7400
        at java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350d7400
        at java.base@11.0.5/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.5/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)
        at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
        at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)



"Thread-0 (activemq-netty-threads)" Id=112 WAITING on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@50ab59d6
        at java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@50ab59d6
        at java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
        at java.base@11.0.5/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:959)
        at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.page(PagingStoreImpl.java:835)
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.addToPage(AbstractJournalStorageManager.java:2074)
        at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:1308)
        at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:1003)
        at org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:894)
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:2073)
        -  locked org.apache.activemq.artemis.core.server.impl.ServerSessionImpl@7805129
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1712)
        -  locked org.apache.activemq.artemis.core.server.impl.ServerSessionImpl@7805129
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1651)
        -  locked org.apache.activemq.artemis.core.server.impl.ServerSessionImpl@7805129
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1643)
        at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession.lambda$sendShouldBlockProducer$0(AMQSession.java:453)
        at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession$$Lambda$274/0x0000000100597840.run(Unknown Source)
        at org.apache.activemq.artemis.core.paging.impl.PagingStoreImpl.checkMemory(PagingStoreImpl.java:728)
        at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession.sendShouldBlockProducer(AMQSession.java:504)
        at org.apache.activemq.artemis.core.protocol.openwire.amq.AMQSession.send(AMQSession.java:415)
        at org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection$CommandProcessor.processMessage(OpenWireConnection.java:1570)
        at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:768)
        at org.apache.activemq.artemis.core.protocol.openwire.OpenWireConnection.bufferReceived(OpenWireConnection.java:293)
        at org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654)
        at org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Thread-29 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@466cf502)" Id=200 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350d7400
        at java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@350d7400
        at java.base@11.0.5/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.5/java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:458)
        at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:112)
        at org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor$ThreadPoolQueue.poll(ActiveMQThreadPoolExecutor.java:45)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


"Scheduler-143999341" Id=202 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a25f629
        at java.base@11.0.5/jdk.internal.misc.Unsafe.park(Native Method)
        -  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a25f629
        at java.base@11.0.5/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.5/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.5/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)
        at java.base@11.0.5/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.5/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.5/java.lang.Thread.run(Thread.java:834)


"Thread-1 (activemq-netty-threads)" Id=2344 RUNNABLE (in native)
        at io.netty.channel.epoll.Native.epollWait0(Native Method)
        at io.netty.channel.epoll.Native.epollWait(Native.java:114)
        at io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:251)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:276)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
        at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)


为什么会发生这种情况? - avalus
1个回答

6
据我所知,这个问题似乎与线程无关。正如错误消息所指示的那样,“在路径0上,TimedBuffer已过期”,换句话说,“TimedBuffer@51b01960不响应”。 "TimedBuffer"是负责将数据刷新到磁盘的组件,它花费的时间太长,以至于代理的“关键分析器”关闭代理并发出线程转储以进行调试。
“关键分析器”是代理中监视重要任务的服务。如果这些任务花费的时间太长,则关键分析器将采取措施。默认的 broker.xml 包含关键分析器的以下配置:
<critical-analyzer>true</critical-analyzer>
<critical-analyzer-timeout>120000</critical-analyzer-timeout>
<critical-analyzer-check-period>60000</critical-analyzer-check-period>
<critical-analyzer-policy>HALT</critical-analyzer-policy>

因此,每60秒关键分析器将检查各个组件,如果其中任何一个在120秒(即2分钟)内停滞,那么它将停止经纪人。您需要手动重新启动经纪人或者,根据您的平台,如果经纪人作为“服务”运行,则可以自动重启。有关关键分析器的更多信息,请参阅文档

HALT是指代理将停止还是重新启动?此外,TimedBuffer会将数据刷新到磁盘。这是否是使用共享文件存储的HA的一部分?我最近遇到了这个错误,正在尝试理解原因。 - BenW
我更新了我的回答以应对您评论的第一部分。我不太确定您关于HA提出的问题是什么。 TimedBuffer 用于将所有数据刷新到磁盘上,无论是本地还是在远程共享的磁盘上。希望这有所帮助! - Justin Bertram
谢谢。我有这些默认设置,同时在systemctl上启用了Artemis。这个错误昨晚出现了,看起来像是Artemis自己重新启动了。对于TimedBuffer,我正在试图理解是否由于我们用于数据文件夹的NFS挂载文件系统导致了这个错误,以实现HA。 - BenW
1
要明确的是,在这种情况下,平台必须重新启动代理,因为 Artemis 本身没有任何重启功能。我仍然不确定我是否回答了你关于 HA/NFS 的问题。 - Justin Bertram

网页内容由stack overflow 提供, 点击上面的
可以查看英文原文,
原文链接