概要
Private環境でパスワード認証のTransfer Familyを使ってEFSにファイルをアップロードしたいと思い、色々ぐぐって調べてみました。
その結果、以下のような記事がクラメソさんから上がっていて非常に助かったんですが、残念ながら記事ではPrivate環境用ではなかったので、Private環境用に直した流れを記載します。
参考にしたクラメソさんの記事
構成
構成は以下の通りです。PrivateなEC2からVPCEndpoint経由でEFSにファイルをアップロードします。Transfer Familyはパスワード認証なので、ユーザ名やパスワードをSecrets Managerに持たせて、Lambdaで認証します。
変更した点
・Transfer Family ServerのEndpointを「内部アクセスを持つ Amazon Virtual Private Cloud (Amazon VPC) エンドポイント」に変更
・Transfer FamilyとLambdaの間のAPI Gatewayを削除
・プロトコルをFTP(内部アクセスを持つ Amazon Virtual Private Cloud (Amazon VPC)エンドポイントのみ対応)に変更
・SecretManagerの作成をコードに追加
・EFSの作成をコードに追加
コード
【前提】
Endpoint用のVPC/Subnet/SecurityGroup、EFS用のSubnet/SecurityGroupは作成済とする
---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
TransferSubnetIDs:
Type: String
Default: ''
TransferVPCID:
Type: String
Default: ''
TransferSecurityGroupIDs:
Type: String
Default: ''
EFSMountTargetSubnetID:
Type: String
Default: ''
EFSMountTargetSecurityGroupIDs:
Type: String
Default: ''
SecretManagerUserName:
Type: String
Default: ''
SecretManagerPassword:
Type: String
Default: ''
NoEcho: true
SecretManagerUserID:
Type: String
Default: ''
SecretManagerGroupID:
Type: String
Default: ''
SecretManagerHomeDirectory:
Type: String
Default: ''
Resources:
TransferServer:
Type: AWS::Transfer::Server
Properties:
Domain: EFS
EndpointType: VPC
EndpointDetails:
SubnetIds:
Fn::Split: [',', Ref: TransferSubnetIDs]
VpcId:
Ref: TransferVPCID
SecurityGroupIds:
- Ref: TransferSecurityGroupIDs
IdentityProviderDetails:
Function:
Fn::GetAtt: GetUserConfigLambda.Arn
IdentityProviderType: AWS_LAMBDA
LoggingRole:
Fn::GetAtt: TransferCWLoggingRole.Arn
Protocols:
- FTP
TransferCWLoggingRole:
Description: IAM role used by Transfer to log API requests to CloudWatch
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- transfer.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSTransferLoggingAccess
RoleName: !Sub
- 'iamrole-TransferLog-01'
GetUserConfigLambdaPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:invokeFunction
FunctionName:
Fn::GetAtt: GetUserConfigLambda.Arn
Principal: transfer.amazonaws.com
SourceArn:
Fn::GetAtt: TransferServer.Arn
LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- lambda.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: LambdaSecretsPolicy
PolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Action:
- secretsmanager:GetSecretValue
Resource:
Fn::Sub:
- arn:${AWS::Partition}:secretsmanager:${SecretsRegion}:${AWS::AccountId}:secret:s-*
- SecretsRegion: ap-northeast-1
RoleName: !Sub
- iamrole-FTPLambda-01
GetUserConfigLambda:
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile: |
import os
import json
import boto3
import base64
from ipaddress import ip_network, ip_address
from botocore.exceptions import ClientError
def lambda_handler(event, context):
# Get the required parameters
required_param_list = ["serverId", "username", "protocol", "sourceIp"]
for parameter in required_param_list:
if parameter not in event:
print("Incoming " + parameter + " missing - Unexpected")
return {}
input_serverId = event["serverId"]
input_username = event["username"]
input_protocol = event["protocol"]
input_sourceIp = event["sourceIp"]
input_password = event.get("password", "")
print("ServerId: {}, Username: {}, Protocol: {}, SourceIp: {}"
.format(input_serverId, input_username, input_protocol, input_sourceIp))
# Check for password and set authentication type appropriately.
print("Start User Authentication Flow")
if input_password != "":
print("Using PASSWORD authentication")
authentication_type = "PASSWORD"
else:
if input_protocol == 'FTP' or input_protocol == 'FTPS':
print("Empty password not allowed for FTP/S")
return {}
# Retrieve our user details from the secret. For all key-value pairs stored in SecretManager,
# checking the protocol-specified secret first, then use generic ones.
secret = get_secret(input_serverId + "/" + input_username)
if secret is not None:
secret_dict = json.loads(secret)
# Run our password checks
user_authenticated = authenticate_user(secret_dict, input_password, input_protocol)
# Run sourceIp checks
ip_match = check_ipaddress(secret_dict, input_sourceIp, input_protocol)
if user_authenticated and ip_match:
print("User authenticated, calling build_response with: PASSWORD")
return build_response(secret_dict, input_protocol)
else:
print("User failed authentication return empty response")
return {}
else:
# Otherwise something went wrong. Most likely the object name is not there
print("Secrets Manager exception thrown - Returning empty response")
# Return an empty data response meaning the user was not authenticated
return {}
def lookup(secret_dict, key, input_protocol):
if input_protocol + key in secret_dict:
print("Found protocol-specified {}".format(key))
return secret_dict[input_protocol + key]
else:
return secret_dict.get(key, None)
def check_ipaddress(secret_dict, input_sourceIp, input_protocol):
accepted_ip_network = lookup(secret_dict, "AcceptedIPNetwork", input_protocol)
if not accepted_ip_network:
# No IP provided so skip checks
print("No IP range provided - Skip IP check")
return True
net = ip_network(accepted_ip_network)
if ip_address(input_sourceIp) in net:
print("Source IP address match")
return True
else:
print("Source IP address not in range")
return False
def authenticate_user(secret_dict, input_password, input_protocol):
# Retrieve the password from the secret if exists
password = lookup(secret_dict, "Password", input_protocol)
if not password:
print("Unable to authenticate user - No field match in Secret for password")
return False
if input_password == password:
return True
else:
print("Unable to authenticate user - Incoming password does not match stored")
return False
# Build out our response data for an authenticated response
def build_response(secret_dict, input_protocol):
response_data = {}
# Check for each key value pair. These are required so set to empty string if missing
role = lookup(secret_dict, "Role", input_protocol)
if role:
response_data["Role"] = role
else:
print("No field match for role - Set empty string in response")
response_data["Role"] = ""
# These are optional so ignore if not present
policy = lookup(secret_dict, "Policy", input_protocol)
if policy:
response_data["Policy"] = policy
# External Auth providers support chroot and virtual folder assignments so we'll check for that
home_directory_details = lookup(secret_dict, "HomeDirectoryDetails", input_protocol)
if home_directory_details:
print("HomeDirectoryDetails found - Applying setting for virtual folders - "
"Note: Cannot be used in conjunction with key: HomeDirectory")
response_data["HomeDirectoryDetails"] = home_directory_details
# If we have a virtual folder setup then we also need to set HomeDirectoryType to "Logical"
print("Setting HomeDirectoryType to LOGICAL")
response_data["HomeDirectoryType"] = "LOGICAL"
# Note that HomeDirectory and HomeDirectoryDetails / Logical mode
# can't be used together but we're not checking for this
home_directory = lookup(secret_dict, "HomeDirectory", input_protocol)
if home_directory:
print("HomeDirectory found - Note: Cannot be used in conjunction with key: HomeDirectoryDetails")
response_data["HomeDirectory"] = home_directory
PosixProfile = lookup(secret_dict, "PosixProfile", input_protocol)
if PosixProfile:
print("Uid found")
response_data["PosixProfile"] = PosixProfile
return response_data
def get_secret(id):
region = os.environ["SecretsManagerRegion"]
print("Secrets Manager Region: " + region)
print("Secret Name: " + id)
# Create a Secrets Manager client
client = boto3.session.Session().client(service_name="secretsmanager", region_name=region)
try:
resp = client.get_secret_value(SecretId=id)
# Decrypts secret using the associated KMS CMK.
# Depending on whether the secret is a string or binary, one of these fields will be populated.
if "SecretString" in resp:
print("Found Secret String")
return resp["SecretString"]
else:
print("Found Binary Secret")
return base64.b64decode(resp["SecretBinary"])
except ClientError as err:
print("Error Talking to SecretsManager: " + err.response["Error"]["Code"] + ", Message: " +
err.response["Error"]["Message"])
return None
Description: A function to lookup and return user data from AWS Secrets Manager.
Handler: index.lambda_handler
Role:
Fn::GetAtt: LambdaExecutionRole.Arn
Runtime: python3.7
Environment:
Variables:
SecretsManagerRegion: ap-northeast-1
FunctionName: !Sub
- lambda-ftp
EFS:
Type: AWS::EFS::FileSystem
Properties:
BackupPolicy:
Status: ENABLED
PerformanceMode: generalPurpose
ThroughputMode : bursting
Encrypted: true
LifecyclePolicies:
- TransitionToIA: AFTER_30_DAYS
- TransitionToPrimaryStorageClass: AFTER_1_ACCESS
FileSystemTags:
- Key: Name
Value: !Sub
- efs-01
MountTarget:
Type: AWS::EFS::MountTarget
Properties:
FileSystemId: !Ref EFS
SubnetId: !Ref EFSMountTargetSubnetID
SecurityGroups:
- Ref: EFSMountTargetSecurityGroupIDs
TransferRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: '2012-10-17'
Statement:
- Effect: Allow
Principal:
Service:
- transfer.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: !Sub
- iampolicy-TransferFamily-01
PolicyDocument:
Version: "2012-10-17"
Statement:
Effect: Allow
Action:
- 'elasticfilesystem:ClientMount'
- 'elasticfilesystem:ClientWrite'
- 'elasticfilesystem:DescribeMountTargets'
Resource: !Sub
- 'arn:aws:elasticfilesystem:ap-northeast-1:${AWS::AccountId}:file-system/${EFSID}'
- EFSID: !Ref EFS
RoleName: !Sub
- 'iamrole-Transfer-01'
DependsOn: EFS
SecretManager:
Type: AWS::SecretsManager::Secret
Properties:
Name: !Sub
- '${TFServerID}/${UserName}'
- TFServerID:
Fn::GetAtt: TransferServer.ServerId
UserName: !Ref SecretManagerUserName
Description: !Sub
- "This secret has a ${UserName}'s Information."
- UserName: !Ref SecretManagerUserName
SecretString: !Sub
- '{ "Password":"${Password}", "Role":"${IAMARN}", "HomeDirectoryDetails":"[{\"Entry\": \"/\", \"Target\": \"/${EFSID}${HomeDirectory}\"}]", "PosixProfile": { "Uid": "${UserID}", "Gid": "${GroupID}" } }'
- Password: !Ref SecretManagerPassword
IAMARN:
Fn::GetAtt: TransferRole.Arn
EFSID: !Ref EFS
HomeDirectory: !Ref SecretManagerHomeDirectory
UserID: !Ref SecretManagerUserID
GroupID: !Ref SecretManagerGroupID
Transfer FamilyのVPCエンドポイントについて
VPCエンドポイントのリソースページからTransfer FamilyのVPCエンドポイントを作成しようとすると以下の3種類、もしくは1と2の2種類のサービスが存在します。
- com.amazonaws.ap-northeast-1.transfer
- com.amazonaws.ap-northeast-1.transfer.server
- com.amazonaws.ap-northeast-1.transfer.server.x-xxxx
それぞれの用途は以下の通りです。上記コードを利用/流用される場合は意識する必要はありませんが、個別に設定を行う場合はご注意下さい。
- com.amazonaws.ap-northeast-1.transfer
Transfer Family Serverにアクセスするものではなく、Transfer FamilyのAPIを実行するためのエンドポイントです - com.amazonaws.ap-northeast-1.transfer.server
Transfer Familyのサービス開始当初に3のVPC エンドポイントが自動生成されなかったため、このエンドポイントを手動作成し、サーバに紐づけて利用していたエンドポイントです - com.amazonaws.ap-northeast-1.transfer.server.x-xxxx
Transfer Family Server作成時のエンドポイントを VPC に設定すると自動で作成されるエンドポイントで、Transfer Family Serverに対してアクセスするためのエンドポイントです
We Are Hiring!