Java SDK crashes the JVM on Alpine docker image

patlil · January 11, 2022, 7:28am

We are using the Docker image eclipse-temurin:17-alpine in a Spring boot application.

When running Java SDK version 3.0.10 the application works fine. If we upgrade to version 3.1.0 or later the JVM crashes during startup. It seems like the way temp files are created when loading native libs has been changed in this version and I believe that is what’s causing the crash.

# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000003dde, pid=1, tid=8
#
# JRE version: OpenJDK Runtime Environment Temurin-17.0.1+12 (17.0.1+12) (build 17.0.1+12)
# Java VM: OpenJDK 64-Bit Server VM Temurin-17.0.1+12 (17.0.1+12, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# C 0x0000000000003dde
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try “ulimit -c unlimited” before starting Java again
#
# An error report file with more information is saved as:
# //hs_err_pid1.log
#
# If you would like to submit a bug report, please visit:
# Issues · adoptium/adoptium-support · GitHub
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

patlil · January 11, 2022, 4:11pm

I enabled trace logging and these are the last log outputs before the crash happens.

Slightly shortened the Spring stackframes to avoid limit on number of chars on this forum

2022-01-11T07:21:06.533Z [DEBUG] c.c.c.c.d.i.n.u.i.PlatformDependent0 [main] - java.nio.DirectByteBuffer.(long, int): unavailable

2022-01-11T07:21:06.534Z [DEBUG] c.c.c.c.d.i.n.u.i.PlatformDependent [main] - sun.misc.Unsafe: available

2022-01-11T07:21:06.537Z [DEBUG] c.c.c.c.d.i.n.u.i.PlatformDependent [main] - maxDirectMemory: 2415919104 bytes (maybe)

2022-01-11T07:21:06.538Z [DEBUG] c.c.c.c.d.i.n.u.i.PlatformDependent [main] - -Dio.netty.tmpdir: /tmp (java.io.tmpdir)

2022-01-11T07:21:06.538Z [DEBUG] c.c.c.c.d.i.n.u.i.PlatformDependent [main] - -Dio.netty.bitMode: 64 (sun.arch.data.model)

2022-01-11T07:21:06.543Z [DEBUG] c.c.c.c.d.i.n.u.i.PlatformDependent [main] - -Dio.netty.maxDirectMemory: -1 bytes

2022-01-11T07:21:06.543Z [DEBUG] c.c.c.c.d.i.n.u.i.PlatformDependent [main] - -Dio.netty.uninitializedArrayAllocationThreshold: -1

2022-01-11T07:21:06.547Z [DEBUG] c.c.c.c.d.i.n.u.i.CleanerJava9 [main] - java.nio.ByteBuffer.cleaner(): available

2022-01-11T07:21:06.548Z [DEBUG] c.c.c.c.d.i.n.u.i.PlatformDependent [main] - -Dio.netty.noPreferDirect: false

2022-01-11T07:21:06.581Z [DEBUG] c.c.c.c.d.i.n.u.i.PlatformDependent [main] - com.couchbase.client.core.deps.org.jctools-core.MpscChunkedArrayQueue: available

2022-01-11T07:21:06.611Z [DEBUG] c.c.c.c.d.i.n.u.i.NativeLibraryLoader [main] - -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)

2022-01-11T07:21:06.611Z [DEBUG] c.c.c.c.d.i.n.u.i.NativeLibraryLoader [main] - -Dio.netty.native.deleteLibAfterLoading: true

2022-01-11T07:21:06.611Z [DEBUG] c.c.c.c.d.i.n.u.i.NativeLibraryLoader [main] - -Dio.netty.native.tryPatchShadedId: true

2022-01-11T07:21:06.615Z [DEBUG] c.c.c.c.d.i.n.u.i.NativeLibraryLoader [main] - Unable to load the library ‘com_couchbase_client_core_deps_netty_transport_native_epoll_x86_64’, trying other loading mechanism.

java.lang.UnsatisfiedLinkError: no com_couchbase_client_core_deps_netty_transport_native_epoll_x86_64 in java.library.path: /opt/java/openjdk/lib/server:/opt/java/openjdk/lib:/opt/java/openjdk/…/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib

at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2429)

at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)

at java.base/java.lang.System.loadLibrary(System.java:1989)

at com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:385)

at java.base/java.security.AccessController.doPrivileged(AccessController.java:318)

at com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:377)

at com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:341)

at com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)

at com.couchbase.client.core.deps.io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:250)

at com.couchbase.client.core.deps.io.netty.channel.epoll.Native.(Native.java:69)

at com.couchbase.client.core.deps.io.netty.channel.epoll.Epoll.(Epoll.java:39)

at com.couchbase.client.core.env.IoEnvironment.createEventLoopGroup(IoEnvironment.java:430)

at com.couchbase.client.core.env.IoEnvironment.(IoEnvironment.java:264)

at com.couchbase.client.core.env.IoEnvironment.(IoEnvironment.java:67)

at com.couchbase.client.core.env.IoEnvironment$Builder.build(IoEnvironment.java:614)

at com.couchbase.client.core.env.CoreEnvironment.(CoreEnvironment.java:148)

at com.couchbase.client.java.env.ClusterEnvironment.(ClusterEnvironment.java:52)

at com.couchbase.client.java.env.ClusterEnvironment.(ClusterEnvironment.java:45)

at com.couchbase.client.java.env.ClusterEnvironment$Builder.build(ClusterEnvironment.java:212)

at com.unibet.evolution.config.CouchBaseConfig.clusterEnvironment(CouchBaseConfig.java:47)

at com.unibet.evolution.config.CouchBaseConfig$$EnhancerBySpringCGLIB$$4fe55d55.CGLIB$clusterEnvironment$1()

at com.unibet.evolution.config.CouchBaseConfig$$EnhancerBySpringCGLIB$$4fe55d55$$FastClassBySpringCGLIB$$2de3ac82.invoke()

at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)

at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)

at com.unibet.evolution.config.CouchBaseConfig$$EnhancerBySpringCGLIB$$4fe55d55.clusterEnvironment()

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)

at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)

{… shortened spring stack frames …}

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)

at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.resolveBeanReference(ConfigurationClassEnhancer.java:362)

at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:334)

at com.unibet.evolution.config.CouchBaseConfig$$EnhancerBySpringCGLIB$$4fe55d55.clusterEnvironment()

at com.unibet.evolution.config.CouchBaseConfig.cluster(CouchBaseConfig.java:41)

at com.unibet.evolution.config.CouchBaseConfig$$EnhancerBySpringCGLIB$$4fe55d55.CGLIB$cluster$2()

at com.unibet.evolution.config.CouchBaseConfig$$EnhancerBySpringCGLIB$$4fe55d55$$FastClassBySpringCGLIB$$2de3ac82.invoke()

at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)

at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)

at com.unibet.evolution.config.CouchBaseConfig$$EnhancerBySpringCGLIB$$4fe55d55.cluster()

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)

at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)

at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:486)

{… shortened spring stack frames …}

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1431)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:619)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)

at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:671)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeansOfType(DefaultListableBeanFactory.java:659)

at org.springframework.security.config.annotation.web.configuration.AutowiredWebSecurityConfigurersIgnoreParents.getWebSecurityConfigurers(AutowiredWebSecurityConfigurersIgnoreParents.java:51)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.springframework.expression.spel.support.ReflectiveMethodExecutor.execute(ReflectiveMethodExecutor.java:129)

at org.springframework.expression.spel.ast.MethodReference.getValueInternal(MethodReference.java:139)

at org.springframework.expression.spel.ast.MethodReference.access$000(MethodReference.java:55)

at org.springframework.expression.spel.ast.MethodReference$MethodValueRef.getValue(MethodReference.java:387)

at org.springframework.expression.spel.ast.CompoundExpression.getValueInternal(CompoundExpression.java:92)

at org.springframework.expression.spel.ast.SpelNodeImpl.getValue(SpelNodeImpl.java:112)

at org.springframework.expression.spel.standard.SpelExpression.getValue(SpelExpression.java:272)

at org.springframework.context.expression.StandardBeanExpressionResolver.evaluate(StandardBeanExpressionResolver.java:167)

at org.springframework.beans.factory.support.AbstractBeanFactory.evaluateBeanDefinitionString(AbstractBeanFactory.java:1631)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1333)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309)

{… shortened spring stack frames …}

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213)

at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:212)

at org.springframework.boot.web.servlet.ServletContextInitializerBeans.getOrderedBeansOfType(ServletContextInitializerBeans.java:203)

at org.springframework.boot.web.servlet.ServletContextInitializerBeans.addServletContextInitializerBeans(ServletContextInitializerBeans.java:97)

at org.springframework.boot.web.servlet.ServletContextInitializerBeans.(ServletContextInitializerBeans.java:86)

at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.getServletContextInitializerBeans(ServletWebServerApplicationContext.java:260)

at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.selfInitialize(ServletWebServerApplicationContext.java:234)

at org.springframework.boot.web.embedded.jetty.ServletContextInitializerConfiguration.callInitializers(ServletContextInitializerConfiguration.java:65)

at org.springframework.boot.web.embedded.jetty.ServletContextInitializerConfiguration.configure(ServletContextInitializerConfiguration.java:54)

at org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:498)

at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409)

at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:910)

at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)

at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)

at org.eclipse.jetty.server.handler.StatisticsHandler.doStart(StatisticsHandler.java:253)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)

at org.eclipse.jetty.server.Server.start(Server.java:423)

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)

at org.eclipse.jetty.server.Server.doStart(Server.java:387)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)

at org.springframework.boot.web.embedded.jetty.JettyWebServer.initialize(JettyWebServer.java:123)

at org.springframework.boot.web.embedded.jetty.JettyWebServer.(JettyWebServer.java:90)

at org.springframework.boot.web.embedded.jetty.JettyServletWebServerFactory.getJettyWebServer(JettyServletWebServerFactory.java:447)

{… shortened spring stack frames …}

at org.springframework.boot.SpringApplication.run(SpringApplication.java:1301)

at org.springframework.boot.SpringApplication.run(SpringApplication.java:1290)

at com.unibet.evolution.EvolutionApplication.main(EvolutionApplication.java:98)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)

at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)

at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)

at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)

2022-01-11T07:21:06.619Z [DEBUG] c.c.c.c.d.i.n.u.i.NativeLibraryLoader [main] - com_couchbase_client_core_deps_netty_transport_native_epoll_x86_64 cannot be loaded from java.library.path, now trying export to -Dio.netty.native.workdir: /tmp

java.lang.UnsatisfiedLinkError: no com_couchbase_client_core_deps_netty_transport_native_epoll_x86_64 in java.library.path: /opt/java/openjdk/lib/server:/opt/java/openjdk/lib:/opt/java/openjdk/…/lib:/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib

at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2429)

at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)

at java.base/java.lang.System.loadLibrary(System.java:1989)

at com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)

at com.couchbase.client.core.deps.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:351)