ssm接続失敗する場合に--debugをつけた例で一体何が出力されるのか見てみました。
出力内容が何を指しているのか、基本的に英語で同行に出力があるのでわかりやすかったです。
<実行コマンド>
% AWS_PROFILE=credential-name aws ssm start-session --region=ap-northeast-1 --target i-xxxxxxxx --debug
<cli関連リソースのバージョン1>
2021-11-19 12:05:44,110 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.14 Python/3.8.8 Darwin/20.6.0 exe/x86_64
<引数1>
2021-11-19 12:05:44,110 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ssm', 'start-session', '--region=ap-northeast-1', '--target', 'i-xxxxxxxxxx', '--debug']
<ハンドラー呼び出し>
2021-11-19 12:05:44,131 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7f8af8839a60>
2021-11-19 12:05:44,132 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7f8af7e7a160>
2021-11-19 12:05:44,132 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2021-11-19 12:05:44,132 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f8af7e1e8b0>
2021-11-19 12:05:44,132 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f8af7e288b0>
2021-11-19 12:05:44,132 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7f8af884a4c0>
2021-11-19 12:05:44,132 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7f8af7ec2ee0>
2021-11-19 12:05:44,132 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2021-11-19 12:05:44,132 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7f8af8840700>
<json読み込み>
2021-11-19 12:05:44,133 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/data/cli.json
<ハンドラー呼び出し>
2021-11-19 12:05:44,136 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7f8af7f749d0>
2021-11-19 12:05:44,136 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7f8af7f75550>
2021-11-19 12:05:44,136 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7f8af7f754c0>
2021-11-19 12:05:44,136 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7f8af7f75670>
2021-11-19 12:05:44,136 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7f8af7f755e0>
2021-11-19 12:05:44,136 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7f8af88e3c40>
<regionの変数を設定>
2021-11-19 12:05:44,136 - MainThread - botocore.session - DEBUG - Setting config variable for region to 'ap-northeast-1'
<cli関連リソースのバージョン2 ※1と比べてprompt/offが追加になっている>
2021-11-19 12:05:44,137 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.14 Python/3.8.8 Darwin/20.6.0 exe/x86_64 prompt/off
<引数2>
2021-11-19 12:05:44,137 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ssm', 'start-session', '--region=ap-northeast-1', '--target', 'i-xxxxxxxxxx', '--debug']
<ハンドラー呼び出し>
2021-11-19 12:05:44,137 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7f8af883a0d0>
2021-11-19 12:05:44,137 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7f8af7bd1dc0>
2021-11-19 12:05:44,138 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7f8af88a8b80>
2021-11-19 12:05:44,138 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7f8af7bce280>
2021-11-19 12:05:44,138 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f8af7c31790>
<インスタンスメタデータサービス(IMDS)エンドポイント>
2021-11-19 12:05:44,142 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
<ハンドラー呼び出し>
2021-11-19 12:05:44,149 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f8af7ec2dc0>
2021-11-19 12:05:44,149 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7f8af7e79040>
<json読み込み>
2021-11-19 12:05:44,173 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/botocore/data/ssm/2014-11-06/service-2.json
<ハンドラー呼び出し>
2021-11-19 12:05:44,197 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ssm: calling handler <function add_custom_start_session at 0x7f8af8857430>
2021-11-19 12:05:44,197 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ssm: calling handler <function add_waiters at 0x7f8af8840700>
<json読み込み>
2021-11-19 12:05:44,220 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/botocore/data/ssm/2014-11-06/waiters-2.json
<引数を配列化しているように見える>
2021-11-19 12:05:44,221 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('target', <awscli.arguments.CLIArgument object at 0x7f8af8f93f40>), ('document-name', <awscli.arguments.CLIArgument object at 0x7f8af8f7e280>), ('parameters', <awscli.arguments.CLIArgument object at 0x7f8af8f7e190>)])
<ハンドラー呼び出し>
2021-11-19 12:05:44,221 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ssm.start-session: calling handler <function add_streaming_output_arg at 0x7f8af883a670>
2021-11-19 12:05:44,221 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ssm.start-session: calling handler <function add_cli_input_json at 0x7f8af7c39040>
2021-11-19 12:05:44,222 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ssm.start-session: calling handler <function add_cli_input_yaml at 0x7f8af7c39310>
2021-11-19 12:05:44,222 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ssm.start-session: calling handler <function unify_paging_params at 0x7f8af7e7a790>
<json読み込み>
2021-11-19 12:05:44,244 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/botocore/data/ssm/2014-11-06/paginators-1.json
<ハンドラー呼び出し>
2021-11-19 12:05:44,244 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ssm.start-session: calling handler <function add_generate_skeleton at 0x7f8af7f65f70>
2021-11-19 12:05:44,244 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ssm.start-session: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7f8af8fac0d0>>
2021-11-19 12:05:44,245 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ssm.start-session: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7f8af8fac100>>
2021-11-19 12:05:44,245 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ssm.start-session: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f8af8fac250>>
2021-11-19 12:05:44,245 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ssm.start-session.target: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f8af8a1dd90>
2021-11-19 12:05:44,246 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ssm.start-session: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f8af7bf4f40>
<targetの値を展開?>
2021-11-19 12:05:44,246 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'i-xxxxxxxxxx' for parameter "target": 'i-xxxxxxxxxx'
<ハンドラー呼び出し>
2021-11-19 12:05:44,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ssm.start-session.document-name: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f8af8a1dd90>
2021-11-19 12:05:44,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ssm.start-session.parameters: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f8af8a1dd90>
2021-11-19 12:05:44,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ssm.start-session.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f8af8a1dd90>
2021-11-19 12:05:44,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ssm.start-session.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f8af8a1dd90>
2021-11-19 12:05:44,246 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ssm.start-session.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f8af8a1dd90>
2021-11-19 12:05:44,246 - MainThread - botocore.hooks - DEBUG - Event calling-command.ssm.start-session: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7f8af8fac0d0>>
2021-11-19 12:05:44,246 - MainThread - botocore.hooks - DEBUG - Event calling-command.ssm.start-session: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7f8af8fac100>>
2021-11-19 12:05:44,248 - MainThread - botocore.hooks - DEBUG - Event calling-command.ssm.start-session: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f8af8fac250>>
<credentialsの情報を使って値を探している>
2021-11-19 12:05:44,248 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-11-19 12:05:44,249 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role
2021-11-19 12:05:44,249 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role-with-web-identity
2021-11-19 12:05:44,249 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: sso
2021-11-19 12:05:44,249 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file
2021-11-19 12:05:44,249 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials
<json読み込み>
2021-11-19 12:05:44,251 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/botocore/data/endpoints.json
<ハンドラー呼び出し>
2021-11-19 12:05:44,260 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f8af71f3f70>
2021-11-19 12:05:44,264 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.ssm: calling handler <function add_generate_presigned_url at 0x7f8af71a11f0>
<ssmのタイムアウト値を設定>
2021-11-19 12:05:44,268 - MainThread - botocore.endpoint - DEBUG - Setting ssm timeout as (60, 60)
<ハンドラー呼び出し>
2021-11-19 12:05:44,269 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.ssm.StartSession: calling handler <function base64_decode_input_blobs at 0x7f8af88a9310>
2021-11-19 12:05:44,269 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ssm.StartSession: calling handler <function generate_idempotent_uuid at 0x7f8af7212f70>
2021-11-19 12:05:44,269 - MainThread - botocore.hooks - DEBUG - Event before-call.ssm.StartSession: calling handler <function inject_api_version_header_if_needed at 0x7f8af7219820>
<startsessionのリクエストとパラメータ定義>
2021-11-19 12:05:44,269 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=StartSession) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonSSM.StartSession', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'aws-cli/2.2.14 Python/3.8.8 Darwin/20.6.0 exe/x86_64 prompt/off command/ssm.start-session'}, 'body': b'{"Target": "i-xxxxxxxxxx"}', 'url': 'https://ssm.ap-northeast-1.amazonaws.com/', 'context': {'client_region': 'ap-northeast-1', 'client_config': <botocore.config.Config object at 0x7f8af9006520>, 'has_streaming_input': False, 'auth_type': None}}
<ハンドラー呼び出し>
2021-11-19 12:05:44,270 - MainThread - botocore.hooks - DEBUG - Event request-created.ssm.StartSession: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f8af90064f0>>
2021-11-19 12:05:44,270 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ssm.StartSession: calling handler <function set_operation_specific_signer at 0x7f8af7212e50>
<aws signature認証 ver4で計算>
2021-11-19 12:05:44,270 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
<正規化されたリクエスト?をPOST>
2021-11-19 12:05:44,270 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/
content-type:application/x-amz-json-1.1
host:ssm.ap-northeast-1.amazonaws.com
x-amz-date:20211119T030544Z
x-amz-target:AmazonSSM.StartSession
content-type;host;x-amz-date;x-amz-target
c28b334d3f45995e3de686b8ad27d0xxxxxxxxxx
<暗号化アルゴリズムと暗号化された情報>
2021-11-19 12:05:44,270 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20211119T030544Z
20211119/ap-northeast-1/ssm/aws4_request
fa203ba78860d8f9a6481f9c51fd844aee7d1613ceee66xxxxxxxxxx
<暗号化された署名の情報>
2021-11-19 12:05:44,270 - MainThread - botocore.auth - DEBUG - Signature:
17268c890a3be216ba7fb686bd7f4614760fd5xxxxxxxxxxxxxxx
<httpリクエスト>
2021-11-19 12:05:44,270 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ssm.ap-northeast-1.amazonaws.com/, headers={'X-Amz-Target': b'AmazonSSM.StartSession', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'aws-cli/2.2.14 Python/3.8.8 Darwin/20.6.0 exe/x86_64 prompt/off command/ssm.start-session', 'X-Amz-Date': b'20211119T030544Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=xxxxx/20211119/ap-northeast-1/ssm/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=17268c890a3be216ba7bd7f461476xxxxx', 'Content-Length': '33'}>
<証明書>
2021-11-19 12:05:44,271 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/botocore/cacert.pem
<新規https接続の開始>
2021-11-19 12:05:44,271 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): ssm.ap-northeast-1.amazonaws.com:443
<接続状態:403>
2021-11-19 12:05:44,359 - MainThread - urllib3.connectionpool - DEBUG - https://ssm.ap-northeast-1.amazonaws.com:443 "POST / HTTP/1.1" 403 None
<レスポンスヘッダ>
2021-11-19 12:05:44,360 - MainThread - botocore.parsers - DEBUG - Response headers: {'Server': 'Server', 'Date': 'Fri, 19 Nov 2021 03:05:44 GMT', 'Content-Type': 'application/octet-stream', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive'}
<接続失敗>
2021-11-19 12:05:44,361 - MainThread - botocore.parsers - DEBUG - Response body:
b'Server authentication failed: <UnauthorizedRequest xmlns=""><message>Forbidden.</message></UnauthorizedRequest>\n'
<ハンドラー呼び出し>
2021-11-19 12:05:44,370 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ssm.StartSession: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f8af9295070>>
<リトライはしない>
2021-11-19 12:05:44,370 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
<ハンドラー呼び出し>
2021-11-19 12:05:44,370 - MainThread - botocore.hooks - DEBUG - Event after-call.ssm.StartSession: calling handler <bound method RetryQuotaChecker.release_retry_quota of
<Exceptionの中身>
<botocore.retries.standard.RetryQuotaChecker object at 0x7f8af9006bb0>>
2021-11-19 12:05:44,374 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "awscli/clidriver.py", line 459, in main
File "awscli/clidriver.py", line 594, in __call__
File "awscli/clidriver.py", line 770, in __call__
File "awscli/customizations/sessionmanager.py", line 64, in invoke
File "botocore/client.py", line 278, in _api_call
File "botocore/client.py", line 597, in _make_api_call
botocore.exceptions.ClientError: An error occurred (403) when calling the StartSession operation: Server authentication failed: <UnauthorizedRequest xmlns=""><message>Forbidden.</message></UnauthorizedRequest>
An error occurred (403) when calling the StartSession operation: Server authentication failed: <UnauthorizedRequest xmlns=""><message>Forbidden.</message></UnauthorizedRequest>
ssm接続成功の時のデバッグログは「証明書」の次の行あたりから違ってくる。
<新規https接続の開始>
2021-11-19 12:17:57,889 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): ssm.ap-northeast-1.amazonaws.com:443
<接続状態:200>
2021-11-19 12:17:58,087 - MainThread - urllib3.connectionpool - DEBUG - https://ssm.ap-northeast-1.amazonaws.com:443 "POST / HTTP/1.1" 200 729
<レスポンスヘッダ(失敗の時なかったリクエストIDが確認できる)>
2021-11-19 12:17:58,088 - MainThread - botocore.parsers - DEBUG - Response headers: {'Server': 'Server', 'Date': 'Fri, 19 Nov 2021 03:17:58 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '729', 'Connection': 'keep-alive', 'x-amzn-RequestId': 'd343e616-9d53-4bb2-9474-cdc5fxxxx'}
<レスポンスボディ>
2021-11-19 12:17:58,089 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"SessionId":"user-name-09fdxxxxxxxx","StreamUrl":"wss://ssmmessages.ap-northeast-1.amazonaws.com/v1/data-channel/user-name-0f6fxxxxxxxxx?role=publish_subscribe","TokenValue":"xxxxxxxxxxxxxxxxxxxxx"}'
<ハンドラー呼び出し>
2021-11-19 12:17:58,090 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ssm.StartSession: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f8e7dad5070>>
<リトライはしない>
2021-11-19 12:17:58,090 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
<ハンドラー呼び出し>
2021-11-19 12:17:58,090 - MainThread - botocore.hooks - DEBUG - Event after-call.ssm.StartSession: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7f8e7d845bb0>>
Starting session with SessionId: user-name-0042bxxxxxxxxx
bash
cd ~
sh-4.2$ bash
Last login: Fri Nov 19 11:46:59 JST 2021 on pts/0
[ec2-user@srv ~]$ cd ~
[ec2-user@srv ~]$
調べて為になったけど、ssm接続不可の場合の切り分けのヒントにはあまりならない、、?
ちなみに接続不可の切り分けはcloudtrailにも出力されず(接続成功のログしか出力されず認証で失敗のログは出なかった)この場合は接続先インスタンスの/var/log/amazon/ssm
や/var/lib/amazon/ssm
内のファイルを確認すれば良いそう。(見てない)