s3 Unexpected error while reading file: expected string or bytes-like object
解決したいこと
- aws Lambdaからs3にアップロードしているファイルの読み取りをしたい
-
Unexpected error while reading file: expected string or bytes-like objectというものの解決方法がわからなくて解決したい
発生している問題・エラー
- 下のエラーが発生している
- Error occurred: Unexpected error while reading file: expected string or bytes-like object
- モジュールのライブラリでやってることをdebugログで取得ししてみた
- ロガーを入れてみると下の箇所で止まっていた
# boto3とbotocoreのロガーを設定
boto3.set_stream_logger('boto3', logging.DEBUG)
boto3.set_stream_logger('botocore', logging.DEBUG)
.
.
.
1730068540959,[DEBUG] 2024-10-27T22:35:40.959Z cee1f473-afd0-4a84-92dc-7c8af65db046 Loading JSON file: /usr/local/lib/python3.8/dist-packages/botocore/data/_retry.json
1730068540959,2024-10-27 22:35:40,959 botocore.client [DEBUG] Registering retry handlers for service: s3
1730068540959,[DEBUG] 2024-10-27T22:35:40.959Z cee1f473-afd0-4a84-92dc-7c8af65db046 Registering retry handlers for service: s3
1730068540959,2024-10-27 22:35:40,959 botocore.utils [DEBUG] Registering S3 region redirector handler
1730068540960,[DEBUG] 2024-10-27T22:35:40.959Z cee1f473-afd0-4a84-92dc-7c8af65db046 Registering S3 region redirector handler
1730068540960,2024-10-27 22:35:40,960 botocore.utils [DEBUG] Registering S3Express Identity Resolver
1730068540960,[DEBUG] 2024-10-27T22:35:40.960Z cee1f473-afd0-4a84-92dc-7c8af65db046 Registering S3Express Identity Resolver
1730068540961,2024-10-27 22:35:40,961 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7fa1f386fc10>
1730068540961,[DEBUG] 2024-10-27T22:35:40.961Z cee1f473-afd0-4a84-92dc-7c8af65db046 Event before-parameter-build.s3.GetObject: calling handler <function sse_md5 at 0x7fa1f386fc10>
1730068540961,2024-10-27 22:35:40,961 botocore.hooks [DEBUG] Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7fa1f386fb80>
1730068540961,[DEBUG] 2024-10-27T22:35:40.961Z cee1f473-afd0-4a84-92dc-7c8af65db046 Event before-parameter-build.s3.GetObject: calling handler <function validate_bucket_name at 0x7fa1f386fb80>
★↑ここで止まっている
ポリシー系
- Lambdaの実行ロールについているポリシー
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "logs:CreateLogGroup",
"Resource": "arn:aws:logs:ap-northeast-1:{user_id}:*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents",
"s3:GetObject",
"s3-object-lambda:GetObject"
],
"Resource": [
"arn:aws:logs:ap-northeast-1:{user_id}:log-group:/aws/lambda/lambda-api:*",
"arn:aws:s3:::{backet_name}/*"
]
}
]
}
- S3バケットポリシー
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowLambdaAccess",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::{user_id}:role/service-role/lambda-api-role-dcozjiys"
},
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::{backet_name}/*"
}
]
}
ソースコード
- ローカル環境で、下のモジュールのテストを行いましたがうまく動いたんですよね...
storage_service.py
import boto3
import json
import logging
from botocore.exceptions import ClientError
# ロギングの設定
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
# boto3とbotocoreのロガーを設定
boto3.set_stream_logger('boto3', logging.DEBUG)
boto3.set_stream_logger('botocore', logging.DEBUG)
class StorageService:
def __init__(self, aws_access_key_id, aws_secret_access_key):
# Amazon S3クライアントの初期化
# 認証情報を明示的に指定
self.s3_client = boto3.client(
's3',
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key)
"""
Amazon S3からJSONファイルを読み取る
:param bucket_name: S3バケット名
:param path: ファイルが格納されているディレクトリのパス
:param file_name: 読み込むファイル名
:return: 読み込んだJSONコンテンツ
"""
def read_json_from_file(self, bucket_name, path, file_name):
try:
full_path = f"{path}/{file_name}"
response = self.s3_client.get_object(Bucket=bucket_name, Key=full_path)
if 'Body' not in response:
raise KeyError("'Body' not found in response")
body = response['Body']
logger.debug(f"Body type: {type(body)}")
if not hasattr(body, 'read'):
raise AttributeError(f"'Body' object has no attribute 'read'. Type: {type(body)}")
contents = body.read()
logger.debug(f"Contents type: {type(contents)}")
if not isinstance(contents, (str, bytes)):
raise TypeError(f"Unexpected content type: {type(contents)}")
if isinstance(contents, bytes):
contents = contents.decode('utf-8')
# JSONとしてパース
return json.loads(contents)
except ClientError as e:
raise RuntimeError(f'Error reading JSON file: {e}')
except json.JSONDecodeError as e:
raise RuntimeError(f'Error decoding JSON file: {e}')
except Exception as e:
raise RuntimeError(f'Unexpected error while reading file: {e}')
0
