word2vecを使うバッチをサーバー上で動かそうとしたらjava.lang.RuntimeException: ND4J is probably missing dependencies.

忘れないうちにメモを残します

word2vecを使って単語間の類似度を測るアプリを作っていて、ローカルPC（Win）では動いていたのにサーバ環境にデプロイしたら動かなくなった
いろいろ調べた結果対処方法がわかったのでそれをメモ

何が起きたのか

ローカルPCでは動いたJavaバッチをサーバ（AmazonLinux2023）にデプロイして実行したら下記の例外が出ました

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'initW2V' defined in class path resource []: Failed to instantiate [void]: Factory method 'initW2V' threw exception with message: null
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:648)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:485)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1335)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1165)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:562)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:975)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:962)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:624)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:334)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1354)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343)
        at com.BatchJavaApplication.java:14)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91)
        at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53)
        at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [void]: Factory method 'initW2V' threw exception with message: null
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:177)
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:644)
        ... 23 common frames omitted
Caused by: java.lang.ExceptionInInitializerError: null
        at org.nd4j.linalg.cpu.nativecpu.ops.NativeOpExecutioner.<init>(NativeOpExecutioner.java:79)
        at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:502)
        at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
        at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:304)
        at java.base/java.lang.Class.newInstance(Class.java:725)
        at org.nd4j.linalg.factory.Nd4j.initWithBackend(Nd4j.java:5152)
        at org.nd4j.linalg.factory.Nd4j.initContext(Nd4j.java:5064)
        at org.nd4j.linalg.factory.Nd4j.<clinit>(Nd4j.java:284)
        at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.readWord2VecModel(WordVectorSerializer.java:2312)
        at org.deeplearning4j.models.embeddings.loader.WordVectorSerializer.readWord2VecModel(WordVectorSerializer.java:2292)
        at com.Word2VecService.initW2V(Word2VecService.java:58)
        at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
        at java.base/java.lang.reflect.Method.invoke(Method.java:580)
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:140)
        ... 24 common frames omitted
Caused by: java.lang.RuntimeException: ND4J is probably missing dependencies. For more information, please refer to: https://deeplearning4j.konduit.ai/nd4j/backend
        at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:107)
        at org.nd4j.nativeblas.NativeOpsHolder.<clinit>(NativeOpsHolder.java:41)
        ... 39 common frames omitted
Caused by: java.lang.UnsatisfiedLinkError: no jnind4jcpu in java.library.path: ./libs
        at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2458)
        at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:916)
        at java.base/java.lang.System.loadLibrary(System.java:2063)
        at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1800)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1402)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1214)
        at org.bytedeco.javacpp.Loader.load(Loader.java:1190)
        at org.nd4j.linalg.cpu.nativecpu.bindings.Nd4jCpu.<clinit>(Nd4jCpu.java:14)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:534)
        at java.base/java.lang.Class.forName(Class.java:513)
        at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:62)
        at org.nd4j.common.config.ND4JClassLoading.loadClassByName(ND4JClassLoading.java:56)
        at org.nd4j.nativeblas.NativeOpsHolder.<init>(NativeOpsHolder.java:97)
        ... 40 common frames omitted
Caused by: java.lang.UnsatisfiedLinkError: Could not find jnind4jcpu in class, module, and library paths.
        at org.bytedeco.javacpp.Loader.loadLibrary(Loader.java:1767)
        ... 50 common frames omitted

ND4Jがdependenciesにない、jnind4jcpuがないと。

どうしたらよいのか？

はて？ローカルでは動いてたのに。と思いつつ、ChatGPTさんに聞いてみたら、

ND4J (N-Dimensional Arrays for Java) のネイティブライブラリが見つからないことが原因で発生しています。jnind4jcpu という名前のネイティブライブラリが java.library.path に存在しないため、UnsatisfiedLinkError が発生しています。

だそうです。

サーバ上で動かすにはdependenciesが足りていないと。

どう対応したか？

build.gradleにはもともと下記を書いていました

build.gradle

    // https://mvnrepository.com/artifact/org.deeplearning4j/deeplearning4j-core
    implementation 'org.deeplearning4j:deeplearning4j-core:1.0.0-M2.1'
    // https://mvnrepository.com/artifact/org.deeplearning4j/deeplearning4j-nlp
    implementation 'org.deeplearning4j:deeplearning4j-nlp:1.0.0-M2.1'
    // https://mvnrepository.com/artifact/org.nd4j/nd4j-native-platform
    implementation 'org.nd4j:nd4j-native-platform:1.0.0-M2.1'

でも他にも org.nd4j:nd4j-native シリーズのjarが存在してました。
コレが必要だったみたい。また、これ自体、アーキテクチャごとにjarが分かれていました

https://repo1.maven.org/maven2/org/nd4j/nd4j-native/1.0.0-M2.1/

nd4j-nativeを導入しました
build.gradleに下記の osdetector を追加し、

build.gradle

plugins {
    id "com.google.osdetector" version "1.6.2"
}

dependenciesはこんな感じで書きました

build.gradle

    def os = project.osdetector.os
    def arch = project.osdetector.arch
    def nd4jVersion = "1.0.0-M2.1"

    // https://mvnrepository.com/artifact/org.nd4j/nd4j-native-platform
    // implementation 'org.nd4j:nd4j-native-platform:1.0.0-M2.1'
    if (os == "linux") {
        if (arch == "arm64") {
            implementation "org.nd4j:nd4j-native:${nd4jVersion}:linux-arm64"
        } else if (arch == "aarch_64") {
            implementation "org.nd4j:nd4j-native:${nd4jVersion}:linux-arm64"
        } else if (arch == "armhf") {
            implementation "org.nd4j:nd4j-native:${nd4jVersion}:linux-armhf"
        } else if (arch == "x86_64") {
            implementation "org.nd4j:nd4j-native:${nd4jVersion}:linux-x86_64-avx2"
        } else {
            throw new GradleException("Unsupported architecture for Linux: $arch")
        }
    } else if (os == "osx") {
        implementation "org.nd4j:nd4j-native:${nd4jVersion}:osx-x86_64"
    } else if (os == "windows") {
        implementation "org.nd4j:nd4j-native:${nd4jVersion}:windows-x86_64"
    } else {
        throw new GradleException("Unsupported OS: $os")
    }

今回は開発者にWindowsもMacもいるのでいろんなパターンを入れましたがもっと減らしてもいいな