【springcloud gateway诡异间歇性500异常解决】500 Server Error，ConcurrentModificationException: null

最近在搭建一个新的网关时，发布页面访问或者postman访问，会间歇性,500异常,折腾了很长的时间，搜了国内国外各种网站，都没有具体的解决方案。

    经过2，3天的 关注和分析定位，最终解决，mark一下

具体异常信息

022-02-19 21:50:20.279 ERROR 35366 --- [ctor-http-nio-3] a.w.r.e.AbstractErrorWebExceptionHandler : [84a6cc5f-36]  500 Server Error for HTTP POST "/template/apilog/add"

java.util.ConcurrentModificationException: null
	at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) ~[na:1.8.0_202]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	|_ checkpoint ⇢ org.springframework.web.cors.reactive.CorsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ com.alibaba.csp.sentinel.adapter.spring.webflux.SentinelWebFluxFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
	|_ checkpoint ⇢ HTTP POST "/template/apilog/add" [ExceptionHandlingWebHandler]
Stack trace:
		at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) ~[na:1.8.0_202]
		at java.util.ArrayList$Itr.next(ArrayList.java:859) ~[na:1.8.0_202]
		at reactor.core.publisher.FluxIterable$IterableSubscription.slowPath(FluxIterable.java:254) [reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxIterable$IterableSubscription.request(FluxIterable.java:225) [reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:162) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxConcatMap$ConcatMapImmediate.onSubscribe(FluxConcatMap.java:228) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:90) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:161) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:86) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.Flux.subscribe(Flux.java:8325) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:207) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:80) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4213) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at org.springframework.http.server.reactive.ChannelSendOperator$WriteBarrier.onNext(ChannelSendOperator.java:192) ~[spring-web-5.2.8.RELEASE.jar:5.2.8.RELEASE]
		at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:114) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxBuffer$BufferExactSubscriber.onComplete(FluxBuffer.java:179) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:252) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:136) ~[reactor-core-3.3.9.RELEASE.jar:3.3.9.RELEASE]
		at reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:445) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
		at reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:251) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
		at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:382) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
		at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:373) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
		at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:429) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
		at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:645) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:96) ~[reactor-netty-0.9.11.RELEASE.jar:0.9.11.RELEASE]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) ~[netty-codec-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.51.Final.jar:4.1.51.Final]
		at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_202]

2022-02-19 21:50:20.768  INFO 35366 --- [ctor-http-nio-2] c.y.b.a.filter.WhiteListUrlFilter        : req method:POST,url:/template/apilog/add

  注意异常信息中

com.alibaba.csp.sentinel.adapter.spring.webflux.SentinelWebFluxFilter [DefaultWebFilterChain]
    |_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]

    因为出现异常的情况和概率完全不可控，非常随机。一开始怀疑是网关的的路由配置问题，网关参数配置问题，hystrix 或者 sentinel的问题。各种排查，均没有解决。

    经过国外一个issues 的点拨，扩展开来，怀疑到 spring-boot-starter-actuator 端点监控上来，于是通过 开放端点用于SpringBoot Admin的监控

management.endpoints.web.gateway=false 关闭监控，再次测试仍然会出现。

   为了定位，在网关中把 spring-boot-starter-actuator 全部删除和排除 ，再次重启测试，访问很稳定，没有间歇性出现500的异常。

  分析： 可能是由于 spring-boot-starter-actuator 在获取 网关上的路由配置时，会将路由信息保存在 java.util.ArrayList中，当更新时，需要清空当前路由。然而当页面访问 网关时，actuate的MetricsWebFilter  需要读取路由信息，发现是空的，所以报了 java.util.ConcurrentModificationException: null 异常，当次请求失败。

  解决办法：在网关中把 spring-boot-starter-actuator 全部删除和排除，从根本上避免出现这样的问题。          此为比较粗暴的解决方案，欢迎同行提供更好的解决办法，可与我联系：tangwwk@126.com 

                                                       2022年2月20日  23：25分

有用的链接

https://github.com/spring-cloud/spring-cloud-sleuth/issues/1511

排查SpringCloudGateway的readAddress(..) failed: Connection reset by peer问题（利用arthas）_lsy1162526799的博客-CSDN博客