Spring-batch(Spring-boot-batch) 入門 - その1 [getting started]を試す -

こんなことがタマにある

ある会社に転職するとする。自分はバリバリのwebのフロントエンジニアだ。
初日の挨拶で意気揚々と
「自分、HTMLとcssとJavascriptを駆使して最高のホームページ🏠が作れます！」

すると、デスクの奥に座っていた部長が書類の束を丸めながら颯爽と近づいてくる。
「おおー新人のドジョウ君だね！きみぃ〜〜、期待してるよ？」

ポンポンと肩に手を回しながら書類の丸まった束で胸をツンツンしてくる。

「きみぃ〜〜、Javaスクリプト、出来るんだって？？」
「は、はい！」
「期待してるよ？、、、まずはじめに、きみにはJavaのバッチを書いてもらう」

「は、はい！！、、、、、、、、、、はぁ？」

「Javaのバッチだよ。Javaスクリプト出来るんだろ？」

「は、はい！！、、、、、、、、、、はぁ？」

「JavaとJavaすくりぷとが同じことぐらいはきみでも知ってるよね？」

「は、はい！！、、、、、、、、、、はぁ？」

以下ループ

この記事は、とあるwebエンジニアがJavaバッチをアサインされた時の記憶の断片である

環境

• mac
• JAVA version "10.0.2" 2018-07-17
• Gradle 4.10.2
• Intellij IDEA 2018.2
• Docker Version 18.06.1-ce-mac73 (26764) ← Docker for Mac

はじめに

Creating a Batch Service
https://spring.io/guides/gs/batch-processing/

こちらを基本にして進めることにした。15分で出来るらしい。

IDEAで初期構築

ドキュメントを読むと
mkdir -p src/main/java/hello
Create a Gradle build file
など書いてあるがIDEA経由で構築することにする。

１. IDEAを起動
２. File　→　New　→　Projectの順で選択
３. Spring Initializarを選択し、[Project SDK]は10(Java)、あとはDefault　→　Next

Project Meta Data
Group:	com.example
Artifact:	demo
Type:	Gradle Project
Language:	Java
Packaging	Jar
Java Version:	8
Version:	0.0.1-SNAPSHOT
Name:	demo
Description:	Demo project for Spring Boot
Package:	com.example.demo

→ Next

４. spring-boot 2.0.5 を確認して他はそのままで → Next
５. 設定は以下の通り。自分のホームディレクトリにworkspaceを作成してその下に配置することにした。

設定
Project name:	demo
Project location:	~/workspace/demo

more settingは触らず　→ Finish!

６. Directory Does Not Exist　→ OK
７. そして立ち上がったwindowでImport Module from Gradleの設定画面で、[Use auto-import]と[Create directories for empty content roots automatically]にチェックを入れておく。

ようやく、初期のプロジェクトができました。（ここまでで15分かかったような、、、。）

ドキュメントを読み進める

とりあえず、今の段階でbuildというものをしてみる。

画面の右上にある三角を押す。

結果

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.0.5.RELEASE)

2018-09-30 10:23:53.417  INFO 12722 --- [           main] com.example.demo.DemoApplication         : Starting DemoApplication on rkataoka-mm.local with PID 12722 (/Users/r_kataoka/workspace/demo/out/production/classes started by r_kataoka in /Users/r_kataoka/workspace/demo)
2018-09-30 10:23:53.420  INFO 12722 --- [           main] com.example.demo.DemoApplication         : No active profile set, falling back to default profiles: default
2018-09-30 10:23:53.463  INFO 12722 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@225129c: startup date [Sun Sep 30 10:23:53 JST 2018]; root of context hierarchy
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/Users/r_kataoka/.gradle/caches/modules-2/files-2.1/org.springframework/spring-core/5.0.9.RELEASE/9f9a828936d81afd49a603bda9cc1aed863a0d85/spring-core-5.0.9.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2018-09-30 10:23:53.931  INFO 12722 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2018-09-30 10:23:53.943  INFO 12722 --- [           main] com.example.demo.DemoApplication         : Started DemoApplication in 0.914 seconds (JVM running for 2.091)
2018-09-30 10:23:53.945  INFO 12722 --- [       Thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@225129c: startup date [Sun Sep 30 10:23:53 JST 2018]; root of context hierarchy
2018-09-30 10:23:53.947  INFO 12722 --- [       Thread-1] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown

Process finished with exit code 0

WARNINGがいつくか出ているがほっておく。
とりあえずbuildは無事に出来た。

1. build.gradle

build.gradleファイルを開いて、
ドキュメントに書いてある以下で上書きする。

buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath("org.springframework.boot:spring-boot-gradle-plugin:2.0.5.RELEASE")
    }
}

apply plugin: 'java'
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'org.springframework.boot'
apply plugin: 'io.spring.dependency-management'

bootJar {
    baseName = 'gs-batch-processing'
    version =  '0.1.0'
}

repositories {
    mavenCentral()
}

sourceCompatibility = 1.8
targetCompatibility = 1.8

dependencies {
    compile("org.springframework.boot:spring-boot-starter-batch")
    compile("org.hsqldb:hsqldb")
    testCompile("junit:junit")
}

編集したら、

右側のタブからGradleを開いて上に出てきたぐるぐる矢印を押すとRefreshされる。

2. Business Data

ドキュメントを見ると

src/main/resources/sample-data.csv

Jill,Doe
Joe,Doe
Justin,Doe
Jane,Doe
John,Doe

ドキュメントと自分のディレクトリ構成は違うので少し読み換える必要がある。
以下の画像のように追加する。

先ほどと同じようにresourcesの中にsqlファイルの作成も行う。

src/main/resources/schema-all.sql

DROP TABLE people IF EXISTS;

CREATE TABLE people  (
    person_id BIGINT IDENTITY NOT NULL PRIMARY KEY,
    first_name VARCHAR(20),
    last_name VARCHAR(20)
);

ここでもう一度buildを行う。すると

2018-09-30 11:42:48.085  INFO 13654 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from URL [file:~/workspace/demo/out/production/resources/schema-all.sql]
2018-09-30 11:42:48.089  INFO 13654 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from URL [file:~/workspace/demo/out/production/resources/schema-all.sql] in 4 ms.

ログの中にsqlファイルが実行された形跡が見つかる。よしよしと。

3. Create a business class

先ほどのbusiness dataに対応するクラスを作る。DBにデータを投入する為に行を定義する。

src/main/java/hello/Person.java

package hello;

public class Person {

    private String lastName;
    private String firstName;

    public Person() {
    }

    public Person(String firstName, String lastName) {
        this.firstName = firstName;
        this.lastName = lastName;
    }

    public void setFirstName(String firstName) {
        this.firstName = firstName;
    }

    public String getFirstName() {
        return firstName;
    }

    public String getLastName() {
        return lastName;
    }

    public void setLastName(String lastName) {
        this.lastName = lastName;
    }

    @Override
    public String toString() {
        return "firstName: " + firstName + ", lastName: " + lastName;
    }

}

先ほどと同じ要領でPersonクラスを作成していく。
ただ、このファイルの内容でそのまま作ってしまうとディレクトリ構成が違うのでpackage部分でエラーが出てしまう。

- package hello;
+ package com.example.demo;

としておこう。

4. Create an intermediate processor

実際に処理するクラス。ここでは文字列を大文字に変換している。

src/main/java/hello/PersonItemProcessor.java

package hello;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.springframework.batch.item.ItemProcessor;

public class PersonItemProcessor implements ItemProcessor<Person, Person> {

    private static final Logger log = LoggerFactory.getLogger(PersonItemProcessor.class);

    @Override
    public Person process(final Person person) throws Exception {
        final String firstName = person.getFirstName().toUpperCase();
        final String lastName = person.getLastName().toUpperCase();

        final Person transformedPerson = new Person(firstName, lastName);

        log.info("Converting (" + person + ") into (" + transformedPerson + ")");

        return transformedPerson;
    }

}

同じ要領で駆け足で進もう！

5. Put together a batch job

バッチの中心的な所。処理の流れなどを記述している。

src/main/java/hello/BatchConfiguration.java

package hello;

import javax.sql.DataSource;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecutionListener;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.support.RunIdIncrementer;
import org.springframework.batch.item.database.BeanPropertyItemSqlParameterSourceProvider;
import org.springframework.batch.item.database.JdbcBatchItemWriter;
import org.springframework.batch.item.database.builder.JdbcBatchItemWriterBuilder;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.builder.FlatFileItemReaderBuilder;
import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.jdbc.core.JdbcTemplate;

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    // tag::readerwriterprocessor[]
    @Bean
    public FlatFileItemReader<Person> reader() {
        return new FlatFileItemReaderBuilder<Person>()
            .name("personItemReader")
            .resource(new ClassPathResource("sample-data.csv"))
            .delimited()
            .names(new String[]{"firstName", "lastName"})
            .fieldSetMapper(new BeanWrapperFieldSetMapper<Person>() {{
                setTargetType(Person.class);
            }})
            .build();
    }

    @Bean
    public PersonItemProcessor processor() {
        return new PersonItemProcessor();
    }

    @Bean
    public JdbcBatchItemWriter<Person> writer(DataSource dataSource) {
        return new JdbcBatchItemWriterBuilder<Person>()
            .itemSqlParameterSourceProvider(new BeanPropertyItemSqlParameterSourceProvider<>())
            .sql("INSERT INTO people (first_name, last_name) VALUES (:firstName, :lastName)")
            .dataSource(dataSource)
            .build();
    }
    // end::readerwriterprocessor[]

    // tag::jobstep[]
    @Bean
    public Job importUserJob(JobCompletionNotificationListener listener, Step step1) {
        return jobBuilderFactory.get("importUserJob")
            .incrementer(new RunIdIncrementer())
            .listener(listener)
            .flow(step1)
            .end()
            .build();
    }

    @Bean
    public Step step1(JdbcBatchItemWriter<Person> writer) {
        return stepBuilderFactory.get("step1")
            .<Person, Person> chunk(10)
            .reader(reader())
            .processor(processor())
            .writer(writer)
            .build();
    }
    // end::jobstep[]
}

処理が完了した時にお知らせしてくれるListenerクラス

src/main/java/hello/JobCompletionNotificationListener.java

package hello;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.batch.core.BatchStatus;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.listener.JobExecutionListenerSupport;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;

@Component
public class JobCompletionNotificationListener extends JobExecutionListenerSupport {

	private static final Logger log = LoggerFactory.getLogger(JobCompletionNotificationListener.class);

	private final JdbcTemplate jdbcTemplate;

	@Autowired
	public JobCompletionNotificationListener(JdbcTemplate jdbcTemplate) {
		this.jdbcTemplate = jdbcTemplate;
	}

	@Override
	public void afterJob(JobExecution jobExecution) {
		if(jobExecution.getStatus() == BatchStatus.COMPLETED) {
			log.info("!!! JOB FINISHED! Time to verify the results");

			jdbcTemplate.query("SELECT first_name, last_name FROM people",
				(rs, row) -> new Person(
					rs.getString(1),
					rs.getString(2))
			).forEach(person -> log.info("Found <" + person + "> in the database."));
		}
	}
}

package部分の修正しながら進めて、

最終的に上記の画像のようになった。
ここでbuildを行う。

結果

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.0.5.RELEASE)

2018-09-30 22:13:45.972  INFO 14694 --- [           main] com.example.demo.DemoApplication         : Starting DemoApplication on rkataoka-mm.local with PID 14694 (/Users/r_kataoka/workspace/demo/out/production/classes started by r_kataoka in /Users/r_kataoka/workspace/demo)
2018-09-30 22:13:45.976  INFO 14694 --- [           main] com.example.demo.DemoApplication         : No active profile set, falling back to default profiles: default
2018-09-30 22:13:46.099  INFO 14694 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@13330ac6: startup date [Sun Sep 30 22:13:46 JST 2018]; root of context hierarchy
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/Users/r_kataoka/.gradle/caches/modules-2/files-2.1/org.springframework/spring-core/5.0.9.RELEASE/9f9a828936d81afd49a603bda9cc1aed863a0d85/spring-core-5.0.9.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2018-09-30 22:13:46.767  INFO 14694 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2018-09-30 22:13:46.769  WARN 14694 --- [           main] com.zaxxer.hikari.util.DriverDataSource  : Registered driver with driverClassName=org.hsqldb.jdbcDriver was not found, trying direct instantiation.
2018-09-30 22:13:46.916  INFO 14694 --- [           main] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Driver does not support get/set network timeout for connections. (feature not supported)
2018-09-30 22:13:46.919  INFO 14694 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2018-09-30 22:13:46.933  INFO 14694 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from URL [file:/Users/r_kataoka/workspace/demo/out/production/resources/schema-all.sql]
2018-09-30 22:13:46.936  INFO 14694 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from URL [file:/Users/r_kataoka/workspace/demo/out/production/resources/schema-all.sql] in 3 ms.
2018-09-30 22:13:47.139  INFO 14694 --- [           main] o.s.b.c.r.s.JobRepositoryFactoryBean     : No database type set, using meta data indicating: HSQL
2018-09-30 22:13:47.225  INFO 14694 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : No TaskExecutor has been set, defaulting to synchronous executor.
2018-09-30 22:13:47.235  INFO 14694 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executing SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql]
2018-09-30 22:13:47.242  INFO 14694 --- [           main] o.s.jdbc.datasource.init.ScriptUtils     : Executed SQL script from class path resource [org/springframework/batch/core/schema-hsqldb.sql] in 7 ms.
2018-09-30 22:13:47.319  INFO 14694 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2018-09-30 22:13:47.319  INFO 14694 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Bean with name 'dataSource' has been autodetected for JMX exposure
2018-09-30 22:13:47.322  INFO 14694 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Located MBean 'dataSource': registering with JMX server as MBean [com.zaxxer.hikari:name=dataSource,type=HikariDataSource]
2018-09-30 22:13:47.335  INFO 14694 --- [           main] com.example.demo.DemoApplication         : Started DemoApplication in 1.766 seconds (JVM running for 2.77)
2018-09-30 22:13:47.336  INFO 14694 --- [           main] o.s.b.a.b.JobLauncherCommandLineRunner   : Running default command line with: []
2018-09-30 22:13:47.414  INFO 14694 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=importUserJob]] launched with the following parameters: [{run.id=1}]
2018-09-30 22:13:47.435  INFO 14694 --- [           main] o.s.batch.core.job.SimpleStepHandler     : Executing step: [step1]
2018-09-30 22:13:47.477  INFO 14694 --- [           main] com.example.demo.PersonItemProcessor     : Converting (firstName: Jill, lastName: Doe) into (firstName: JILL, lastName: DOE)
2018-09-30 22:13:47.477  INFO 14694 --- [           main] com.example.demo.PersonItemProcessor     : Converting (firstName: Joe, lastName: Doe) into (firstName: JOE, lastName: DOE)
2018-09-30 22:13:47.477  INFO 14694 --- [           main] com.example.demo.PersonItemProcessor     : Converting (firstName: Justin, lastName: Doe) into (firstName: JUSTIN, lastName: DOE)
2018-09-30 22:13:47.477  INFO 14694 --- [           main] com.example.demo.PersonItemProcessor     : Converting (firstName: Jane, lastName: Doe) into (firstName: JANE, lastName: DOE)
2018-09-30 22:13:47.477  INFO 14694 --- [           main] com.example.demo.PersonItemProcessor     : Converting (firstName: John, lastName: Doe) into (firstName: JOHN, lastName: DOE)
2018-09-30 22:13:47.485  INFO 14694 --- [           main] c.e.d.JobCompletionNotificationListener  : !!! JOB FINISHED! Time to verify the results
2018-09-30 22:13:47.486  INFO 14694 --- [           main] c.e.d.JobCompletionNotificationListener  : Found <firstName: JILL, lastName: DOE> in the database.
2018-09-30 22:13:47.486  INFO 14694 --- [           main] c.e.d.JobCompletionNotificationListener  : Found <firstName: JOE, lastName: DOE> in the database.
2018-09-30 22:13:47.486  INFO 14694 --- [           main] c.e.d.JobCompletionNotificationListener  : Found <firstName: JUSTIN, lastName: DOE> in the database.
2018-09-30 22:13:47.486  INFO 14694 --- [           main] c.e.d.JobCompletionNotificationListener  : Found <firstName: JANE, lastName: DOE> in the database.
2018-09-30 22:13:47.486  INFO 14694 --- [           main] c.e.d.JobCompletionNotificationListener  : Found <firstName: JOHN, lastName: DOE> in the database.
2018-09-30 22:13:47.487  INFO 14694 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=importUserJob]] completed with the following parameters: [{run.id=1}] and the following status: [COMPLETED]
2018-09-30 22:13:47.489  INFO 14694 --- [       Thread-1] s.c.a.AnnotationConfigApplicationContext : Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@13330ac6: startup date [Sun Sep 30 22:13:46 JST 2018]; root of context hierarchy
2018-09-30 22:13:47.491  INFO 14694 --- [       Thread-1] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
2018-09-30 22:13:47.491  INFO 14694 --- [       Thread-1] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans
2018-09-30 22:13:47.492  INFO 14694 --- [       Thread-1] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
2018-09-30 22:13:47.495  INFO 14694 --- [       Thread-1] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.

Process finished with exit code 0

ログの中にConvertingやFoundが見つかる。
今回のバッチ処理の流れとしては、peopleテーブルを作成して、csvファイルを読み取って小文字を大文字に変換する。そして最後にpeopleに書き込んで終了。確認のため、Listenerの中でselectを行なっている。

とりあえず公式ドキュメントの通りにやって無事に終えたようだ。
15分で出来たであろうか？

本日はとりあえずここまで。部長！お疲れ様でした。