Help us understand the problem. What is going on with this article?

ListObjectsV2Request(AWS SDK for Java)でS3のファイル一覧を取得する

More than 1 year has passed since last update.

概要

  • ファイルの一覧を取得するにはListObjectsV2Requestを利用する
    • ListObjectsV2Requestは一回のリクエストですべての一覧を返さない場合がある
      • ListObjectsV2Result#isTruncatedtrueの場合はすべてのファイル一覧を取得できていない
        • ListObjectsV2Request#setContinuationTokenListObjectsV2Result#getNextContinuationTokenを渡して、再度取得する必要がある。

すべてのファイル一覧を取得するIterator

import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.model.ListObjectsV2Request;
import com.amazonaws.services.s3.model.S3ObjectSummary;
import lombok.val;

import java.util.*;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;

class S3ObjectSummaryIterator implements Iterator<S3ObjectSummary> {

    private final AmazonS3 client;
    private final ListObjectsV2Request request;
    private final Queue<S3ObjectSummary> summaries = new LinkedList<>();
    private boolean isTruncated = false;

    S3ObjectSummaryIterator(AmazonS3 client, ListObjectsV2Request request) {
        this.client = client;
        this.request = request;
        fetch();
    }

    private void fetch() {
        val result = this.client.listObjectsV2(this.request);
        summaries.addAll(result.getObjectSummaries());
        this.isTruncated = result.isTruncated();
        if (result.isTruncated()) {
            request.setContinuationToken(result.getNextContinuationToken());
        }
    }

    @Override
    public boolean hasNext() {
        if (summaries.isEmpty() && isTruncated) {
            fetch();
        }
        return !summaries.isEmpty();
    }

    @Override
    public S3ObjectSummary next() {
        return summaries.poll();
    }

    public static Stream<S3ObjectSummary> stream(AmazonS3 client, ListObjectsV2Request request) {
        val iterator = new S3ObjectSummaryIterator(client, request);
        return StreamSupport.stream(Spliterators.spliteratorUnknownSize(iterator, Spliterator.ORDERED), false);
    }

}

参考

Why not register and get more from Qiita?
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
Comments
No comments
Sign up for free and join this conversation.
If you already have a Qiita account
Why do not you register as a user and use Qiita more conveniently?
You need to log in to use this function. Qiita can be used more conveniently after logging in.
You seem to be reading articles frequently this month. Qiita can be used more conveniently after logging in.
  1. We will deliver articles that match you
    By following users and tags, you can catch up information on technical fields that you are interested in as a whole
  2. you can read useful information later efficiently
    By "stocking" the articles you like, you can search right away
ユーザーは見つかりませんでした