Go to Qiita Advent Calendar 2024 Top
LoginSignup
8
5

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 3 years have passed since last update.

@takanattie(Takashi Natsume)

boto3でのendpoint_url=Noneの指定

Last updated at Posted at 2021-01-01

はじめに

新年明けましておめでとうございます。
本年（2021年）ですが、「技術の目利き」のスキルと技術戦略策定のスキルの向上、エンジニアの採用方法の確立、CTO（Chief Technical/Technology Officer）やTech Leadとの人的ネットワーク構築を目標に頑張っていきたいと考えています。
本年もよろしくお願いいたします。

確認したかったこと

AWS SDKやAWS CLIで使用するAWSのサービスエンドポイントですが、AWSのリファレンスガイドには、以下のような記述があり、特に指定しない場合は自動でエンドポイントが決まる仕様のようです。

AWS SDK と AWS Command Line Interface (AWS CLI) では、AWS リージョンで各サービスのデフォルトのエンドポイントを自動的に使用します。

LocalStackという、ローカル（自分の手元の環境）で動かすAWSを模したテスト用モック環境を利用する場合は、LocalStackのエンドポイントを指定することになります。Boto3だと以下のようにendpoint_urlを指定します。

s3 = boto3.client('s3', endpoint_url='http://localhost:4566')

このときにendpoint_url=Noneを指定した場合の挙動がどうなるのかが確認したかったことです。

私の予想

Boto3のドキュメントによると、boto3.session.Sessionクラスのclientメソッドの引数にendpoint_url=Noneがあり、デフォルト値はNoneになっているようです。ここから考えると、endpoint_url=Noneを指定した場合は、自動的にAWSのエンドポイントが使用されるのではないかと考えましたが、念のため試してみました。

試行のための準備（環境構築）

今回使用したソフトウェアのバージョン一覧

  • Ubuntu 20.04.1 LTS
  • AWS CLI 2.1.15
  • LocalStack 0.12.4
  • Docker 19.03.8
  • python3-pip 20.0.2
  • boto3 1.16.47

AWS CLI

まず、公式のユーザーガイドに従い、以下のコマンドでAWS CLIをインストールします。

wget "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"
unzip awscli-exe-linux-x86_64.zip
sudo ./aws/install
実行結果
stack@stack:~$ wget "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip"
--2021-01-01 01:29:18--  https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip
Resolving awscli.amazonaws.com (awscli.amazonaws.com)... 54.239.169.102, 54.239.169.98, 54.239.169.20, ...
Connecting to awscli.amazonaws.com (awscli.amazonaws.com)|54.239.169.102|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 35197257 (34M) [application/zip]
Saving to: ‘awscli-exe-linux-x86_64.zip’

awscli-exe-linux-x86_64.zip                       100%[=============================================================================================================>]  33.57M  2.57MB/s    in 11s     

2021-01-01 01:29:29 (2.97 MB/s) - ‘awscli-exe-linux-x86_64.zip’ saved [35197257/35197257]

stack@stack:~$ unzip awscli-exe-linux-x86_64.zip
Archive:  awscli-exe-linux-x86_64.zip
   creating: aws/
   creating: aws/dist/
  inflating: aws/install             
  inflating: aws/THIRD_PARTY_LICENSES  
  inflating: aws/README.md           
   creating: aws/dist/_struct/
   creating: aws/dist/awscli/
   creating: aws/dist/botocore/
   creating: aws/dist/cryptography/
   creating: aws/dist/cryptography-2.8-py3.7.egg-info/
   creating: aws/dist/docutils/
   creating: aws/dist/include/
   creating: aws/dist/lib/
   creating: aws/dist/zlib/
  inflating: aws/dist/grp.cpython-37m-x86_64-linux-gnu.so  
（中略）
  inflating: aws/dist/cryptography-2.8-py3.7.egg-info/LICENSE.BSD  
   creating: aws/dist/include/python3.7m/
  inflating: aws/dist/include/python3.7m/pyconfig.h  
stack@stack:~$ sudo ./aws/install
You can now run: /usr/local/bin/aws --version
stack@stack:~$ aws --version
aws-cli/2.1.15 Python/3.7.3 Linux/5.4.0-54-generic exe/x86_64.ubuntu.20 prompt/off

次にAWS ウェブコンソールから生成されたCSV形式の認証情報を「aws configure import」コマンドでインポートします（ユーザーガイドを参照）。今回、私はCSVファイルを編集して、CSVの中のUser name列の値を「default」に変更して、defaultプロファイルになるようにしました。また、デフォルトのリージョン設定を行ないました（aws configure set region コマンド）。

stack@stack:~$ aws configure import --csv file://new_user_credentials.csv
Successfully imported 1 profile(s)
stack@stack:~$ aws configure set region ap-northeast-1
stack@stack:~$ aws configure list-profiles 
default

LocalStack

LocalStackをインストールする前に、以下のコマンドでDockerをインストールして、Dockerのサービスを起動します。

sudo apt-get install docker docker.io
sudo systemctl start docker.service

rootユーザでなくても、Dockerを操作できるように、現在のユーザ（私の場合はstack）をdockerグループに所属させます。（newgrpコマンドは、すぐにdockerグループへの所属を反映させるために行なっています。）

sudo usermod -a -G docker stack
newgrp docker

そして、以下のコマンドでLocalStackをインストールします。pip3をインストールして、その後pip3を使ってLocalStackをインストールします。

sudo apt-get install python3-pip
pip3 install localstack
実行結果
stack@stack:~$ sudo apt-get install python3-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  build-essential dpkg-dev fakeroot g++ g++-9 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libexpat1-dev libfakeroot libpython3-dev libpython3.8-dev libstdc++-9-dev
  python-pip-whl python3-dev python3-wheel python3.8-dev zlib1g zlib1g-dev
Suggested packages:
  debian-keyring g++-multilib g++-9-multilib gcc-9-doc libstdc++-9-doc
The following NEW packages will be installed:
  build-essential dpkg-dev fakeroot g++ g++-9 libalgorithm-diff-perl libalgorithm-diff-xs-perl libalgorithm-merge-perl libexpat1-dev libfakeroot libpython3-dev libpython3.8-dev libstdc++-9-dev
  python-pip-whl python3-dev python3-pip python3-wheel python3.8-dev zlib1g-dev
The following packages will be upgraded:
  zlib1g
1 upgraded, 19 newly installed, 0 to remove and 71 not upgraded.
Need to get 17.8 MB of archives.
After this operation, 75.0 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 zlib1g amd64 1:1.2.11.dfsg-2ubuntu1.2 [53.6 kB]
Get:2 http://jp.archive.ubuntu.com/ubuntu focal-updates/main amd64 libstdc++-9-dev amd64 9.3.0-17ubuntu1~20.04 [1714 kB]
（中略）
Processing triggers for libc-bin (2.31-0ubuntu9) ...
stack@stack:~$ pip3 install localstack
Collecting localstack
  Downloading localstack-0.12.4.tar.gz (279 kB)
     |████████████████████████████████| 279 kB 76 kB/s 
Collecting boto3>=1.14.33
  Downloading boto3-1.16.47.tar.gz (99 kB)
     |████████████████████████████████| 99 kB 131 kB/s 
Collecting dnspython==1.16.0
  Downloading dnspython-1.16.0-py2.py3-none-any.whl (188 kB)
     |████████████████████████████████| 188 kB 132 kB/s 
Collecting docopt>=0.6.2
  Downloading docopt-0.6.2.tar.gz (25 kB)
Collecting localstack-client>=0.14
  Downloading localstack-client-1.10.tar.gz (4.3 kB)
Collecting localstack-ext>=0.11.0
  Downloading localstack-ext-0.12.3.3.tar.gz (716 kB)
     |████████████████████████████████| 716 kB 196 kB/s 
Collecting requests>=2.20.0
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 1.8 MB/s 
Requirement already satisfied: six>=1.12.0 in /usr/lib/python3/dist-packages (from localstack) (1.14.0)
Collecting botocore<1.20.0,>=1.19.47
  Downloading botocore-1.19.47-py2.py3-none-any.whl (7.2 MB)
     |████████████████████████████████| 7.2 MB 2.6 MB/s 
Collecting jmespath<1.0.0,>=0.7.1
  Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
Collecting s3transfer<0.4.0,>=0.3.0
  Downloading s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)
     |████████████████████████████████| 69 kB 1.5 MB/s 
Collecting dnslib>=0.9.10
  Downloading dnslib-0.9.14.tar.gz (72 kB)
     |████████████████████████████████| 72 kB 646 kB/s 
Collecting pyaes>=1.6.0
  Downloading pyaes-1.6.1.tar.gz (28 kB)
Requirement already satisfied: idna<3,>=2.5 in /usr/lib/python3/dist-packages (from requests>=2.20.0->localstack) (2.8)
Collecting certifi>=2017.4.17
  Downloading certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
     |████████████████████████████████| 147 kB 2.1 MB/s 
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests>=2.20.0->localstack) (1.25.8)
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/lib/python3/dist-packages (from requests>=2.20.0->localstack) (3.0.4)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3/dist-packages (from botocore<1.20.0,>=1.19.47->boto3>=1.14.33->localstack) (2.7.3)
Building wheels for collected packages: localstack, boto3, docopt, localstack-client, localstack-ext, dnslib, pyaes
  Building wheel for localstack (setup.py) ... done
  Created wheel for localstack: filename=localstack-0.12.4-py3-none-any.whl size=311544 sha256=79900437f8ab3902817e8a2a65fc9c418cc004a01652d6c77db4b4365a78138a
  Stored in directory: /home/stack/.cache/pip/wheels/6b/96/58/9a299ffed3f2630dddb6ad3cdc17b61f5fb508fe06b71122f3
  Building wheel for boto3 (setup.py) ... done
  Created wheel for boto3: filename=boto3-1.16.47-py2.py3-none-any.whl size=128711 sha256=5f71da0686eed283d5e01894769a08bd0435ceb9583a493bc326697b64084d2b
  Stored in directory: /home/stack/.cache/pip/wheels/6a/90/d3/ebaaf8d74ce53fcdb09354cf37b93aa7e03e43bd8c6558e4cf
  Building wheel for docopt (setup.py) ... done
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13704 sha256=592c6e4c0f8dc7fb2fb27139159ff6365abd4bba7a7cb90f7bfa24ecda0aa9a8
  Stored in directory: /home/stack/.cache/pip/wheels/56/ea/58/ead137b087d9e326852a851351d1debf4ada529b6ac0ec4e8c
  Building wheel for localstack-client (setup.py) ... done
  Created wheel for localstack-client: filename=localstack_client-1.10-py3-none-any.whl size=3810 sha256=4b50d57e15684b2c672a7a6bcf909e098800dbb1d9e03ec55dfe68718ec42a41
  Stored in directory: /home/stack/.cache/pip/wheels/70/fd/cf/27fa4e7b43bf50e49bd08ee4b92356e745059f28fb0b015a62
  Building wheel for localstack-ext (setup.py) ... done
  Created wheel for localstack-ext: filename=localstack_ext-0.12.3.3-py3-none-any.whl size=738468 sha256=6ccfc9c4798f123f03f2e63bc27599a87840eb42d8e2a8bfa886e240dd812553
  Stored in directory: /home/stack/.cache/pip/wheels/e9/cb/3e/efe9714198c2e65df0d1582764b166ebbc3fdf36a65630746c
  Building wheel for dnslib (setup.py) ... done
  Created wheel for dnslib: filename=dnslib-0.9.14-py3-none-any.whl size=54650 sha256=73718ca66fd25b49d34013e5c3b6ef3c4040b826ab365fb370a663d8978b39f1
  Stored in directory: /home/stack/.cache/pip/wheels/8b/3d/dd/faac4d3774f42937334e7b65bfdae225a60204088f04f5b684
  Building wheel for pyaes (setup.py) ... done
  Created wheel for pyaes: filename=pyaes-1.6.1-py3-none-any.whl size=26345 sha256=a92c10cd13668abb873fd630d0996f6de763d08715731af12fcf6b8cc378bad0
  Stored in directory: /home/stack/.cache/pip/wheels/aa/ca/9c/8a3c00512585c703edc457db81c066b9609d76758c74f72ac6
Successfully built localstack boto3 docopt localstack-client localstack-ext dnslib pyaes
Installing collected packages: jmespath, botocore, s3transfer, boto3, dnspython, docopt, localstack-client, dnslib, pyaes, certifi, requests, localstack-ext, localstack
Successfully installed boto3-1.16.47 botocore-1.19.47 certifi-2020.12.5 dnslib-0.9.14 dnspython-1.16.0 docopt-0.6.2 jmespath-0.10.0 localstack-0.12.4 localstack-client-1.10 localstack-ext-0.12.3.3 pyaes-1.6.1 requests-2.25.1 s3transfer-0.3.3

「$HOME/.local/bin」ディレクトリにに「localstack」コマンドがインストールされたので、そのlocalstackコマンドを使用してLocalStackを起動します（localstack start）。

stack@stack:~$ export PATH=$PATH:~/.local/bin
stack@stack:~$ localstack start

「localstack start」を実行して「Ready.」のメッセージが出力されれば準備完了です。ログ出力にもあるとおり、デフォルトでは4566番のポートを使用しています（ポート番号は環境変数EDGE_PORTに設定して変更できるようです）。そのため、「http://localhost:4566」のエンドポイントを使用してアクセスします。

localstack startの実行結果
stack@stack:~$ localstack start
Starting local dev environment. CTRL-C to quit.
docker run -it -e TEST_AWS_ACCOUNT_ID="000000000000" -e LOCALSTACK_HOSTNAME="localhost" -e DEFAULT_REGION="us-east-1" --rm --privileged --name localstack_main -p 4566:4566 -p 4571:4571 -p 8080-8081:8080-8081  -v "/tmp/localstack:/tmp/localstack" -v "/var/run/docker.sock:/var/run/docker.sock" -e DOCKER_HOST="unix:///var/run/docker.sock" -e HOST_TMP_FOLDER="/tmp/localstack" "localstack/localstack" 
Unable to find image 'localstack/localstack:latest' locally
latest: Pulling from localstack/localstack
4880f1046b03: Pull complete 
Digest: sha256:e52abfdd82fbd1a6503a857f1aa6daa81aa46af424a26ad7c98417ea371d9759
Status: Downloaded newer image for localstack/localstack:latest
Waiting for all LocalStack services to be ready
2021-01-01 02:25:30,806 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2021-01-01 02:25:30,811 INFO supervisord started with pid 13
2021-01-01 02:25:31,822 INFO spawned: 'dashboard' with pid 19
2021-01-01 02:25:31,831 INFO spawned: 'infra' with pid 20
2021-01-01 02:25:31,855 INFO success: dashboard entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
(. .venv/bin/activate; exec bin/localstack start --host)
2021-01-01 02:25:32,865 INFO success: infra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2021-01-01 02:25:32,865 INFO exited: dashboard (exit status 0; expected)
Starting local dev environment. CTRL-C to quit.
LocalStack version: 0.12.4
Waiting for all LocalStack services to be ready
Starting edge router (https port 4566)...
Starting mock ACM service on http port 4566 ...
2021-01-01T02:25:39:INFO:localstack.utils.analytics.profiler: Execution of "load_plugin_from_path" took 634.166955947876ms
2021-01-01T02:25:39:INFO:localstack.utils.analytics.profiler: Execution of "load_plugins" took 634.6888542175293ms
2021-01-01T02:25:40:INFO:localstack.multiserver: Starting multi API server process on port 56613
[2021-01-01 02:25:40 +0000] [21] [INFO] Running on https://0.0.0.0:4566 (CTRL + C to quit)
2021-01-01T02:25:40:INFO:hypercorn.error: Running on https://0.0.0.0:4566 (CTRL + C to quit)
[2021-01-01 02:25:40 +0000] [21] [INFO] Running on http://0.0.0.0:56613 (CTRL + C to quit)
2021-01-01T02:25:40:INFO:hypercorn.error: Running on http://0.0.0.0:56613 (CTRL + C to quit)
Starting mock API Gateway service on http port 4566 ...
Starting mock CloudFormation service on http port 4566 ...
Starting mock CloudWatch service on http port 4566 ...
Starting mock DynamoDB service on http port 4566 ...
Starting mock DynamoDB Streams service on http port 4566 ...
Starting mock EC2 service on http port 4566 ...
Starting mock ES service on http port 4566 ...
Starting mock Firehose service on http port 4566 ...
Starting mock IAM service on http port 4566 ...
Starting mock STS service on http port 4566 ...
Starting mock Kinesis service on http port 4566 ...
Starting mock KMS service on http port 4566 ...
Starting mock Lambda service on http port 4566 ...
Starting mock CloudWatch Logs service on http port 4566 ...
Starting mock Redshift service on http port 4566 ...
Starting mock Route53 service on http port 4566 ...
Starting mock S3 service on http port 4566 ...
Starting mock Secrets Manager service on http port 4566 ...
Starting mock SES service on http port 4566 ...
Starting mock SNS service on http port 4566 ...
Starting mock SQS service on http port 4566 ...
Starting mock SSM service on http port 4566 ...
Starting mock Cloudwatch Events service on http port 4566 ...
Starting mock StepFunctions service on http port 4566 ...
Waiting for all LocalStack services to be ready
Waiting for all LocalStack services to be ready
Ready.
2021-01-01T02:25:54:INFO:localstack.utils.analytics.profiler: Execution of "start_api_services" took 14802.880048751831ms

以下はLocalStackにS3のBucketを作成して、一覧取得を行なった結果です。

stack@stack:~$ aws s3 mb s3://test-localstack --endpoint-url http://localhost:4566
make_bucket: test-localstack
stack@stack:~$ aws s3 ls --endpoint-url http://localhost:4566
2021-01-01 02:43:44 test-localstack

試行

以下のプログラムを作成して、エンドポイントが指定された時は、そのエンドポイントを使用してS3のBucketの一覧を取得し、エンドポイントが指定されない場合は、endpoint_url=NoneとしてAWSのエンドポイントを使用してS3のBucketの一覧を取得することにしました。（ソースコードはこちらにもあります。）

listbuckets.py
import sys

import boto3


if len(sys.argv) not in (1, 2):
    print("Usage: python3 listbuckets.py [endpoint_url]")
    sys.exit(1)

endpoint_url = sys.argv[1] if len(sys.argv) == 2 else None

s3 = boto3.client('s3', endpoint_url=endpoint_url)

response = s3.list_buckets()

if 'Buckets' in response:
    for bucket in response['Buckets']:
        print(bucket['Name'])

実行してみると以下のとおりendpoint_url=Noneの場合は、AWSのエンドポイントにアクセスできています。

stack@stack:~$ aws s3 ls --endpoint-url http://localhost:4566
2021-01-01 02:43:44 test-localstack
stack@stack:~$ aws s3 ls
2021-01-01 04:46:14 test-aws-20210101
stack@stack:~$ python3 listbuckets.py http://localhost:4566
test-localstack
stack@stack:~$ python3 listbuckets.py
test-aws-20210101

※ 更新履歴
2021年1月8日：pythonのソースコードについてgithubへのリンクを追加。

8
5
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
8
5

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?