目標と手順
目標
- 署名付きURLを作成し、ストレージ上のファイルをダウンロード出来るようにする。
- 署名付きURLを作成し、ストレージ上にファイルをアップロード出来るようにする。
手順
- IBM CloudのObject Storageを作成。
- 左のメニューの「エンドポイント」を選択。自分のサービスまたはアプリケーションと同じ Region に置かれているエンドポイントURLをメモ。
- 左のメニューの「サービス資格情報」を選択。「新規資格情報」を選択して、サービス資格情報を作成し、メモ。
- IBMの公式サイトの「署名付きURLの作成」を参照および編集。(今回はpythonでスクリプト作成します。)
※AWSのCLIもインストールしておいてください。
brew install awscli
そもそも署名付きURLとは?
- アクセス権がなくても、発行し共有することで任意の人がアクセス可能になるリンク。
- Boxの共有リンクのようなもの。
#環境
PC:MacBook Pro (13-inch, 2020, Four Thunderbolt 3 ports) ※Intel版
OS:macOS Monterey 12.1
ブラウザ:Google Chrome
スクリプトの編集
- 公式サイトのPythonのスクリプトはPython2系。3系に対応するためにコードを一部修正。
- 手順の2.と3.で取得したストレージの情報を入力。
【修正前】スクリプト_ダウンロード編
※ 公式サイトの 「オブジェクトをダウンロードするための事前署名URLの作成」より抜粋。
import datetime
import hashlib
import hmac
import requests
from requests.utils import quote
access_key = '{COS_HMAC_ACCESS_KEY_ID}'
secret_key = '{COS_HMAC_SECRET_ACCESS_KEY}'
# request elements
http_method = 'GET'
region = ''
bucket = 'example-bucket'
cos_endpoint = '{endpoint}'
host = cos_endpoint
endpoint = 'https://' + host
object_key = 'example-object'
expiration = 86400 # time in seconds
# hashing methods
def hash(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()
# region is a wildcard value that takes the place of the AWS region value
# as COS doen't use regions like AWS, this parameter can accept any string
def createSignatureKey(key, datestamp, region, service):
keyDate = hash(('AWS4' + key).encode('utf-8'), datestamp)
keyRegion = hash(keyDate, region)
keyService = hash(keyRegion, service)
keySigning = hash(keyService, 'aws4_request')
return keySigning
# assemble the standardized request
time = datetime.datetime.utcnow()
timestamp = time.strftime('%Y%m%dT%H%M%SZ')
datestamp = time.strftime('%Y%m%d')
standardized_querystring = ( 'X-Amz-Algorithm=AWS4-HMAC-SHA256' +
'&X-Amz-Credential=' + access_key + '/' + datestamp + '/' + region + '/s3/aws4_request' +
'&X-Amz-Date=' + timestamp +
'&X-Amz-Expires=' + str(expiration) +
'&X-Amz-SignedHeaders=host' )
standardized_querystring_url_encoded = quote(standardized_querystring, safe='&=')
standardized_resource = '/' + bucket + '/' + object_key
standardized_resource_url_encoded = quote(standardized_resource, safe='&')
payload_hash = 'UNSIGNED-PAYLOAD'
standardized_headers = 'host:' + host
signed_headers = 'host'
standardized_request = (http_method + '\n' +
standardized_resource + '\n' +
standardized_querystring_url_encoded + '\n' +
standardized_headers + '\n' +
'\n' +
signed_headers + '\n' +
payload_hash)
# assemble string-to-sign
hashing_algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + 's3' + '/' + 'aws4_request'
sts = ( hashing_algorithm + '\n' +
timestamp + '\n' +
credential_scope + '\n' +
hashlib.sha256(standardized_request).hexdigest() )
# generate the signature
signature_key = createSignatureKey(secret_key, datestamp, region, 's3')
signature = hmac.new(signature_key,
(sts).encode('utf-8'),
hashlib.sha256).hexdigest()
# create and send the request
# the 'requests' package autmatically adds the required 'host' header
request_url = ( endpoint + '/' +
bucket + '/' +
object_key + '?' +
standardized_querystring_url_encoded +
'&X-Amz-Signature=' +
signature )
print 'request_url: %s' % request_url
print '\nSending `%s` request to IBM COS -----------------------' % http_method
print 'Request URL = ' + request_url
request = requests.get(request_url)
print '\nResponse from IBM COS ---------------------------------'
print 'Response code: %d\n' % request.status_code
print request.text
# this information can be helpful in troubleshooting, or to better
# understand what goes into signature creation
print 'These are the values used to construct this request.'
print 'Request details -----------------------------------------'
print 'http_method: %s' % http_method
print 'host: %s' % host
print 'region: %s' % region
print 'endpoint: %s' % endpoint
print 'bucket: %s' % bucket
print 'object_key: %s' % object_key
print 'timestamp: %s' % timestamp
print 'datestamp: %s' % datestamp
print 'Standardized request details ----------------------------'
print 'standardized_resource: %s' % standardized_resource_url_encoded
print 'standardized_querystring: %s' % standardized_querystring_url_encoded
print 'standardized_headers: %s' % standardized_headers
print 'signed_headers: %s' % signed_headers
print 'payload_hash: %s' % payload_hash
print 'standardized_request: %s' % standardized_request
print 'String-to-sign details ----------------------------------'
print 'credential_scope: %s' % credential_scope
print 'string-to-sign: %s' % sts
print 'signature_key: %s' % signature_key
print 'signature: %s' % signature
print 'Because the signature key has non-ASCII characters, it is'
print 'necessary to create a hexadecimal digest for the purposes'
print 'of checking against this example.'
def hex_hash(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).hexdigest()
def createHexSignatureKey(key, datestamp, region, service):
keyDate = hex_hash(('AWS4' + key).encode('utf-8'), datestamp)
keyRegion = hex_hash(keyDate, region)
keyService = hex_hash(keyRegion, service)
keySigning = hex_hash(keyService, 'aws4_request')
return keySigning
signature_key_hex = createHexSignatureKey(secret_key, datestamp, region, 's3')
print 'signature_key (hexidecimal): %s' % signature_key_hex
print 'Header details ------------------------------------------'
print 'pre-signed url: %s' % request_url
【修正後】スクリプト_ダウンロード編
python2系→3系に対応するための修正箇所
- print文を()で囲む。
- hashlibの部分を修正する。
- 文字列がUnicode型で、これにhashlibが対応していないためエンコードが必要みたいです。
ストレージ情報の入力 (個人情報のため、各自ここを入力してください)
- access_key =<情報を入力> ※手順3.で取得した資格情報を参照
- secret_key =<情報を入力> ※手順3.で取得した資格情報を参照
- bucket = <情報を入力> ※作成したストレージのバケット名を参照
- cos_endpoint = <情報を入力> ※手順2.の情報を参照
- object_key = <情報を入力> ※ストレージ上にあるダウンロードしたいファイル名を指定してください。
- expiration = <情報を入力> ※URLの有効期限。単位は秒。
import datetime
import encodings
import hashlib
import hmac
import requests
from requests.utils import quote
access_key = '<情報を入力>'
secret_key = '<情報を入力>'
# request elements
http_method = 'GET'
region = ''
bucket = '<情報を入力>'
cos_endpoint = '<情報を入力>'
host = cos_endpoint
endpoint = 'https://' + host
object_key = '<情報を入力>'
expiration = 86400 # time in seconds
def hash(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()
# region is a wildcard value that takes the place of the AWS region value
# as COS doen't use regions like AWS, this parameter can accept any string
def createSignatureKey(key, datestamp, region, service):
keyDate = hash(('AWS4' + key).encode('utf-8'), datestamp)
keyRegion = hash(keyDate, region)
keyService = hash(keyRegion, service)
keySigning = hash(keyService, 'aws4_request')
return keySigning
# assemble the standardized request
time = datetime.datetime.utcnow()
timestamp = time.strftime('%Y%m%dT%H%M%SZ')
datestamp = time.strftime('%Y%m%d')
standardized_querystring = ('X-Amz-Algorithm=AWS4-HMAC-SHA256' +
'&X-Amz-Credential=' + access_key + '/' + datestamp + '/' + region + '/s3/aws4_request' +
'&X-Amz-Date=' + timestamp +
'&X-Amz-Expires=' + str(expiration) +
'&X-Amz-SignedHeaders=host')
standardized_querystring_url_encoded = quote(
standardized_querystring, safe='&=')
standardized_resource = '/' + bucket + '/' + object_key
standardized_resource_url_encoded = quote(
standardized_resource, safe='&')
payload_hash = 'UNSIGNED-PAYLOAD'
standardized_headers = 'host:' + host
signed_headers = 'host'
standardized_request = (http_method + '\n' +
standardized_resource + '\n' +
standardized_querystring_url_encoded + '\n' +
standardized_headers + '\n' +
'\n' +
signed_headers + '\n' +
payload_hash)
# assemble string-to-sign
hashing_algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + 's3' + '/' + 'aws4_request'
sts = (hashing_algorithm + '\n' + timestamp + '\n' + credential_scope +
'\n' + hashlib.sha256((standardized_request).encode('utf-8')).hexdigest())
# generate the signature
signature_key = createSignatureKey(secret_key, datestamp, region, 's3')
signature = hmac.new(signature_key,
(sts).encode('utf-8'),
hashlib.sha256).hexdigest()
# create and send the request
# the 'requests' package autmatically adds the required 'host' header
request_url = (endpoint + '/' +
bucket + '/' +
object_key + '?' +
standardized_querystring_url_encoded +
'&X-Amz-Signature=' +
signature)
print('request_url: %s' % request_url)
print('\nSending `%s` request to IBM COS -----------------------' % http_method)
print('Request URL = ' + request_url)
request = requests.get(request_url)
print('\nResponse from IBM COS ---------------------------------')
print('Response code: %d\n' % request.status_code)
print(request.text)
# this information can be helpful in troubleshooting, or to better
# understand what goes into signature creation
print('These are the values used to) construct this request.')
print('Request details -----------------------------------------')
print('http_method: %s' % http_method)
print('host: %s' % host)
print('region: %s' % region)
print('endpoint: %s' % endpoint)
print('bucket: %s' % bucket)
print('object_key: %s' % object_key)
print('timestamp: %s' % timestamp)
print('datestamp: %s' % datestamp)
print('Standardized request details ----------------------------')
print('standardized_resource: %s' % standardized_resource_url_encoded)
print('standardized_querystring: %s' % standardized_querystring_url_encoded)
print('standardized_headers: %s' % standardized_headers)
print('signed_headers: %s' % signed_headers)
print('payload_hash: %s' % payload_hash)
print('standardized_request: %s' % standardized_request)
print('String-to-sign details ----------------------------------')
print('credential_scope: %s' % credential_scope)
print('string-to-sign: %s' % sts)
print('signature_key: %s' % signature_key)
print('signature: %s' % signature)
print('Because the signature key has non-ASCII characters, it is')
print('necessary to create a hexadecimal digest for the purposes')
print('of checking against this example.')
def hex_hash(key, msg):
return hmac.new(key, msg, hashlib.sha256).hexdigest()
def createHexSignatureKey(key, datestamp, region, service):
keyDate = hex_hash(bytes(('AWS4' + key),
encoding='utf-8'), bytes(datestamp, encoding="utf-8"))
keyRegion = hex_hash(bytes(keyDate, encoding="utf-8"),
bytes(region, encoding="utf-8"))
keyService = hex_hash(bytes(keyRegion, encoding="utf-8"),
bytes(service, encoding="utf-8"))
keySigning = hex_hash(bytes(keyService, encoding="utf-8"),
bytes('aws4_request', encoding="utf-8"))
return keySigning
signature_key_hex = createHexSignatureKey(secret_key, datestamp, region, 's3')
print('signature_key (hexidecimal): %s' % signature_key_hex)
print('Header details ------------------------------------------')
print('pre-signed url: %s' % request_url)
上記スクリプトを実行します。うまくいった場合、ログの末尾に以下の内容が出力されます。
pre-signed url: https:<ここにURLの詳細が出ます>
pre-signed urlのリンク先にアクセスすることで、ストレージ上の特定のファイルをダウンロード出来ます。
【修正前】スクリプト_アップロード編
※ 公式サイトの 「オブジェクトをアップロードするための事前署名URLの作成」より抜粋。
import datetime
import hashlib
import hmac
import requests
from requests.utils import quote
access_key = '{COS_HMAC_ACCESS_KEY_ID}'
secret_key = '{COS_HMAC_SECRET_ACCESS_KEY}'
# request elements
http_method = 'PUT'
region = ''
bucket = 'example-bucket'
cos_endpoint = '{endpoint}'
host = cos_endpoint
endpoint = 'https://' + host
object_key = 'example-object'
expiration = 3600 # time in seconds
# hashing methods
def hash(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()
# region is a wildcard value that takes the place of the AWS region value
# as COS doen't use regions like AWS, this parameter can accept any string
def createSignatureKey(key, datestamp, region, service):
keyDate = hash(('AWS4' + key).encode('utf-8'), datestamp)
keyRegion = hash(keyDate, region)
keyService = hash(keyRegion, service)
keySigning = hash(keyService, 'aws4_request')
return keySigning
# assemble the standardized request
time = datetime.datetime.utcnow()
timestamp = time.strftime('%Y%m%dT%H%M%SZ')
datestamp = time.strftime('%Y%m%d')
standardized_querystring = ('X-Amz-Algorithm=AWS4-HMAC-SHA256' +
'&X-Amz-Credential=' + access_key + '/' + datestamp + '/' + region + '/s3/aws4_request' +
'&X-Amz-Date=' + timestamp +
'&X-Amz-Expires=' + str(expiration) +
'&X-Amz-SignedHeaders=host')
standardized_querystring_url_encoded = quote(standardized_querystring, safe='&=')
standardized_resource = '/' + bucket + '/' + object_key
standardized_resource_url_encoded = quote(standardized_resource, safe='&')
payload_hash = 'UNSIGNED-PAYLOAD'
standardized_headers = 'host:' + host + '\n'
signed_headers = 'host'
standardized_request = (http_method + '\n' +
standardized_resource + '\n' +
standardized_querystring_url_encoded + '\n' +
standardized_headers + '\n' +
signed_headers + '\n' +
payload_hash)
# assemble string-to-sign
hashing_algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + 's3' + '/' + 'aws4_request'
sts = (hashing_algorithm + '\n' +
timestamp + '\n' +
credential_scope + '\n' +
hashlib.sha256(standardized_request).hexdigest())
# generate the signature
signature_key = createSignatureKey(secret_key, datestamp, region, 's3')
signature = hmac.new(signature_key,
(sts).encode('utf-8'),
hashlib.sha256).hexdigest()
# create and send the request
# the 'requests' package autmatically adds the required 'host' header
request_url = (endpoint + '/' +
bucket + '/' +
object_key + '?' +
standardized_querystring_url_encoded +
'&X-Amz-Signature=' +
signature)
print 'request_url: %s' % request_url
print '\nSending `%s` request to IBM COS -----------------------' % http_method
print 'Request URL = ' + request_url
request = requests.put(request_url)
print '\nResponse from IBM COS ---------------------------------'
print 'Response code: %d\n' % request.status_code
print request.text
# this information can be helpful in troubleshooting, or to better
# understand what goes into signature creation
print 'These are the values used to construct this request.'
print 'Request details -----------------------------------------'
print 'http_method: %s' % http_method
print 'host: %s' % host
print 'region: %s' % region
print 'endpoint: %s' % endpoint
print 'bucket: %s' % bucket
print 'object_key: %s' % object_key
print 'timestamp: %s' % timestamp
print 'datestamp: %s' % datestamp
print 'Standardized request details ----------------------------'
print 'standardized_resource: %s' % standardized_resource_url_encoded
print 'standardized_querystring: %s' % standardized_querystring_url_encoded
print 'standardized_headers: %s' % standardized_headers
print 'signed_headers: %s' % signed_headers
print 'payload_hash: %s' % payload_hash
print 'standardized_request: %s' % standardized_request
print 'String-to-sign details ----------------------------------'
print 'credential_scope: %s' % credential_scope
print 'string-to-sign: %s' % sts
print 'signature_key: %s' % signature_key
print 'signature: %s' % signature
print 'Because the signature key has non-ASCII characters, it is'
print 'necessary to create a hexadecimal digest for the purposes'
print 'of checking against this example.'
def hex_hash(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).hexdigest()
def createHexSignatureKey(key, datestamp, region, service):
keyDate = hex_hash(('AWS4' + key).encode('utf-8'), datestamp)
keyRegion = hex_hash(keyDate, region)
keyService = hex_hash(keyRegion, service)
keySigning = hex_hash(keyService, 'aws4_request')
return keySigning
signature_key_hex = createHexSignatureKey(secret_key, datestamp, region, 's3')
print 'signature_key (hexidecimal): %s' % signature_key_hex
print 'Header details ------------------------------------------'
print 'pre-signed url: %s' % request_url
【修正後】スクリプト_アップロード編
python2系→3系に対応するための修正箇所
- print文を()で囲む。
- hashlibの部分を修正する。
- 文字列がUnicode型で、これにhashlibが対応していないためエンコードが必要みたいです。
ストレージ情報の入力 (個人情報のため、各自ここを入力してください)
- access_key =<情報を入力> ※手順3.で取得した資格情報を参照
- secret_key =<情報を入力> ※手順3.で取得した資格情報を参照
- bucket = <情報を入力>※作成したストレージのバケット名を参照
- cos_endpoint = <情報を入力> ※手順2.の情報を参照
- object_key = <情報を入力>
※ アップロード後のファイル名を指定できます。この後に記載のcurlを利用したアップロード時はローカルのファイル名を指定します。ここで指定したファイル名と違っていても問題ないです。名前が異なる場合、ストレージ上で保存され確認できる名前は、このobject_keyの名前が反映されます。 - expiration = <情報を入力> ※ URLの有効期限。単位は秒。
import datetime
import hashlib
import hmac
import requests
from requests.utils import quote
access_key = 'b9bc3498718e42c8800d7de740d71031'
secret_key = '01a9ac6ea09ae47bc6bfeed2b1dc34eaf3c32d9356dfb30c'
# request elements
http_method = 'PUT'
region = ''
bucket = '<情報を入力>'
cos_endpoint = '<情報を入力>'
host = cos_endpoint
endpoint = 'https://' + host
object_key = '<情報を入力>'
expiration = 3600 # time in seconds
# hashing methods
def hash(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()
# region is a wildcard value that takes the place of the AWS region value
# as COS doen't use regions like AWS, this parameter can accept any string
def createSignatureKey(key, datestamp, region, service):
keyDate = hash(('AWS4' + key).encode('utf-8'), datestamp)
keyRegion = hash(keyDate, region)
keyService = hash(keyRegion, service)
keySigning = hash(keyService, 'aws4_request')
return keySigning
# assemble the standardized request
time = datetime.datetime.utcnow()
timestamp = time.strftime('%Y%m%dT%H%M%SZ')
datestamp = time.strftime('%Y%m%d')
standardized_querystring = ('X-Amz-Algorithm=AWS4-HMAC-SHA256' +
'&X-Amz-Credential=' + access_key + '/' + datestamp + '/' + region + '/s3/aws4_request' +
'&X-Amz-Date=' + timestamp +
'&X-Amz-Expires=' + str(expiration) +
'&X-Amz-SignedHeaders=host')
standardized_querystring_url_encoded = quote(
standardized_querystring, safe='&=')
standardized_resource = '/' + bucket + '/' + object_key
standardized_resource_url_encoded = quote(standardized_resource, safe='&')
payload_hash = 'UNSIGNED-PAYLOAD'
standardized_headers = 'host:' + host + '\n'
signed_headers = 'host'
standardized_request = (http_method + '\n' +
standardized_resource + '\n' +
standardized_querystring_url_encoded + '\n' +
standardized_headers + '\n' +
signed_headers + '\n' +
payload_hash)
# assemble string-to-sign
hashing_algorithm = 'AWS4-HMAC-SHA256'
credential_scope = datestamp + '/' + region + '/' + 's3' + '/' + 'aws4_request'
sts = (hashing_algorithm + '\n' +
timestamp + '\n' +
credential_scope + '\n' +
hashlib.sha256((standardized_request).encode('utf-8')).hexdigest())
# generate the signature
signature_key = createSignatureKey(secret_key, datestamp, region, 's3')
signature = hmac.new(signature_key,
(sts).encode('utf-8'),
hashlib.sha256).hexdigest()
# create and send the request
# the 'requests' package autmatically adds the required 'host' header
request_url = (endpoint + '/' +
bucket + '/' +
object_key + '?' +
standardized_querystring_url_encoded +
'&X-Amz-Signature=' +
signature)
print('request_url: %s' % request_url)
print('\nSending `%s` request to IBM COS -----------------------' % http_method)
print('Request URL = ' + request_url)
request = requests.put(request_url)
print('\nResponse from IBM COS ---------------------------------')
print('Response code: %d\n' % request.status_code)
print(request.text)
# this information can be helpful in troubleshooting, or to better
# understand what goes into signature creation
print('These are the values used to construct this request.')
print('Request details -----------------------------------------')
print('http_method: %s' % http_method)
print('host: %s' % host)
print('region: %s' % region)
print('endpoint: %s' % endpoint)
print('bucket: %s' % bucket)
print('object_key: %s' % object_key)
print('timestamp: %s' % timestamp)
print('datestamp: %s' % datestamp)
print('Standardized request details ----------------------------')
print('standardized_resource: %s' % standardized_resource_url_encoded)
print('standardized_querystring: %s' % standardized_querystring_url_encoded)
print('standardized_headers: %s' % standardized_headers)
print('signed_headers: %s' % signed_headers)
print('payload_hash: %s' % payload_hash)
print('standardized_request: %s' % standardized_request)
print('String-to-sign details ----------------------------------')
print('credential_scope: %s' % credential_scope)
print('string-to-sign: %s' % sts)
print('signature_key: %s' % signature_key)
print('signature: %s' % signature)
print('Because the signature key has non-ASCII characters, it is')
print('necessary to create a hexadecimal digest for the purposes')
print('of checking against this example.')
def hex_hash(key, msg):
return hmac.new(key, msg, hashlib.sha256).hexdigest()
def createHexSignatureKey(key, datestamp, region, service):
keyDate = hex_hash(bytes(('AWS4' + key),
encoding='utf-8'), bytes(datestamp, encoding="utf-8"))
keyRegion = hex_hash(bytes(keyDate, encoding="utf-8"),
bytes(region, encoding="utf-8"))
keyService = hex_hash(bytes(keyRegion, encoding="utf-8"),
bytes(service, encoding="utf-8"))
keySigning = hex_hash(bytes(keyService, encoding="utf-8"),
bytes('aws4_request', encoding="utf-8"))
return keySigning
signature_key_hex = createHexSignatureKey(secret_key, datestamp, region, 's3')
print('signature_key (hexidecimal): %s' % signature_key_hex)
print('Header details ------------------------------------------')
print('pre-signed url: %s' % request_url)
上記スクリプトを実行します。うまくいった場合、ログの末尾に以下の内容が出力されます。
pre-signed url: https:<ここにURLの詳細が出ます>
pre-signed urlのリンク先をアップロード先として利用します。
リンク先を利用して下記のコマンドを実行します。
curl -T <アップロードしたいファイルのパス> -H 'Content-type: image/jpg' '<pre-signed urlのリンク>'
問題なければ、IBM Cloudのストレージ上にファイルがアップロードされています。
クラウド環境にログインしてご確認ください。
最後に
これにて終了です。
公式サイトはPython3系でスクリプトを書いてほしいなぁと、思いました。