Private環境でパスワード認証(SecretManager + Lambda)のTransfer Familyを使ってEFSにファイルをアップロードしてみた(FTP)

概要

Private環境でパスワード認証のTransfer Familyを使ってEFSにファイルをアップロードしたいと思い、色々ぐぐって調べてみました。

その結果、以下のような記事がクラメソさんから上がっていて非常に助かったんですが、残念ながら記事ではPrivate環境用ではなかったので、Private環境用に直した流れを記載します。
参考にしたクラメソさんの記事

構成

構成は以下の通りです。PrivateなEC2からVPCEndpoint経由でEFSにファイルをアップロードします。Transfer Familyはパスワード認証なので、ユーザ名やパスワードをSecrets Managerに持たせて、Lambdaで認証します。

変更した点

・Transfer Family ServerのEndpointを「内部アクセスを持つ Amazon Virtual Private Cloud (Amazon VPC) エンドポイント」に変更
・Transfer FamilyとLambdaの間のAPI Gatewayを削除
・プロトコルをFTP(内部アクセスを持つ Amazon Virtual Private Cloud (Amazon VPC)エンドポイントのみ対応)に変更
・SecretManagerの作成をコードに追加
・EFSの作成をコードに追加

コード

【前提】
Endpoint用のVPC/Subnet/SecurityGroup、EFS用のSubnet/SecurityGroupは作成済とする

---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Parameters:
  TransferSubnetIDs:
    Type: String
    Default: ''
  TransferVPCID:
    Type: String
    Default: ''
  TransferSecurityGroupIDs:
    Type: String
    Default: ''
  EFSMountTargetSubnetID:
    Type: String
    Default: ''
  EFSMountTargetSecurityGroupIDs:
    Type: String
    Default: ''
  SecretManagerUserName:
    Type: String
    Default: ''
  SecretManagerPassword:
    Type: String
    Default: ''
    NoEcho: true
  SecretManagerUserID:
    Type: String
    Default: ''
  SecretManagerGroupID:
    Type: String
    Default: ''
  SecretManagerHomeDirectory:
    Type: String
    Default: ''
Resources:
  TransferServer:
    Type: AWS::Transfer::Server
    Properties:
      Domain: EFS
      EndpointType: VPC
      EndpointDetails:
        SubnetIds:
          Fn::Split: [',', Ref: TransferSubnetIDs]
        VpcId:
          Ref: TransferVPCID
        SecurityGroupIds:
          - Ref: TransferSecurityGroupIDs
      IdentityProviderDetails:
        Function:
          Fn::GetAtt: GetUserConfigLambda.Arn
      IdentityProviderType: AWS_LAMBDA
      LoggingRole:
        Fn::GetAtt: TransferCWLoggingRole.Arn
      Protocols:
        - FTP
  TransferCWLoggingRole:
    Description: IAM role used by Transfer to log API requests to CloudWatch
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - transfer.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSTransferLoggingAccess
      RoleName: !Sub
          - 'iamrole-TransferLog-01'
  GetUserConfigLambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:invokeFunction
      FunctionName:
        Fn::GetAtt: GetUserConfigLambda.Arn
      Principal: transfer.amazonaws.com
      SourceArn:
        Fn::GetAtt: TransferServer.Arn
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      ManagedPolicyArns:
      - Fn::Sub: arn:${AWS::Partition}:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
      - PolicyName: LambdaSecretsPolicy
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - secretsmanager:GetSecretValue
            Resource:
              Fn::Sub:
                - arn:${AWS::Partition}:secretsmanager:${SecretsRegion}:${AWS::AccountId}:secret:s-*
                - SecretsRegion: ap-northeast-1
      RoleName: !Sub
        - iamrole-FTPLambda-01
  GetUserConfigLambda:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          import os
          import json
          import boto3
          import base64
          from ipaddress import ip_network, ip_address
          from botocore.exceptions import ClientError


          def lambda_handler(event, context):
              # Get the required parameters
              required_param_list = ["serverId", "username", "protocol", "sourceIp"]
              for parameter in required_param_list:
                  if parameter not in event:
                      print("Incoming " + parameter + " missing - Unexpected")
                      return {}

              input_serverId = event["serverId"]
              input_username = event["username"]
              input_protocol = event["protocol"]
              input_sourceIp = event["sourceIp"]
              input_password = event.get("password", "")

              print("ServerId: {}, Username: {}, Protocol: {}, SourceIp: {}"
                    .format(input_serverId, input_username, input_protocol, input_sourceIp))

              # Check for password and set authentication type appropriately.
              print("Start User Authentication Flow")
              if input_password != "":
                  print("Using PASSWORD authentication")
                  authentication_type = "PASSWORD"
              else:
                  if input_protocol == 'FTP' or input_protocol == 'FTPS':
                      print("Empty password not allowed for FTP/S")
                      return {}

              # Retrieve our user details from the secret. For all key-value pairs stored in SecretManager,
              # checking the protocol-specified secret first, then use generic ones.
              secret = get_secret(input_serverId + "/" + input_username)

              if secret is not None:
                  secret_dict = json.loads(secret)
                  # Run our password checks
                  user_authenticated = authenticate_user(secret_dict, input_password, input_protocol)
                  # Run sourceIp checks
                  ip_match = check_ipaddress(secret_dict, input_sourceIp, input_protocol)

                  if user_authenticated and ip_match:
                      print("User authenticated, calling build_response with: PASSWORD")
                      return build_response(secret_dict, input_protocol)
                  else:
                      print("User failed authentication return empty response")
                      return {}
              else:
                  # Otherwise something went wrong. Most likely the object name is not there
                  print("Secrets Manager exception thrown - Returning empty response")
                  # Return an empty data response meaning the user was not authenticated
                  return {}


          def lookup(secret_dict, key, input_protocol):
              if input_protocol + key in secret_dict:
                  print("Found protocol-specified {}".format(key))
                  return secret_dict[input_protocol + key]
              else:
                  return secret_dict.get(key, None)


          def check_ipaddress(secret_dict, input_sourceIp, input_protocol):
              accepted_ip_network = lookup(secret_dict, "AcceptedIPNetwork", input_protocol)
              if not accepted_ip_network:
                  # No IP provided so skip checks
                  print("No IP range provided - Skip IP check")
                  return True

              net = ip_network(accepted_ip_network)
              if ip_address(input_sourceIp) in net:
                  print("Source IP address match")
                  return True
              else:
                  print("Source IP address not in range")
                  return False


          def authenticate_user(secret_dict, input_password, input_protocol):
              # Retrieve the password from the secret if exists
              password = lookup(secret_dict, "Password", input_protocol)
              if not password:
                  print("Unable to authenticate user - No field match in Secret for password")
                  return False

              if input_password == password:
                  return True
              else:
                  print("Unable to authenticate user - Incoming password does not match stored")
                  return False


          # Build out our response data for an authenticated response
          def build_response(secret_dict, input_protocol):
              response_data = {}
              # Check for each key value pair. These are required so set to empty string if missing
              role = lookup(secret_dict, "Role", input_protocol)
              if role:
                  response_data["Role"] = role
              else:
                  print("No field match for role - Set empty string in response")
                  response_data["Role"] = ""

              # These are optional so ignore if not present
              policy = lookup(secret_dict, "Policy", input_protocol)
              if policy:
                  response_data["Policy"] = policy

              # External Auth providers support chroot and virtual folder assignments so we'll check for that
              home_directory_details = lookup(secret_dict, "HomeDirectoryDetails", input_protocol)
              if home_directory_details:
                  print("HomeDirectoryDetails found - Applying setting for virtual folders - "
                        "Note: Cannot be used in conjunction with key: HomeDirectory")
                  response_data["HomeDirectoryDetails"] = home_directory_details
                  # If we have a virtual folder setup then we also need to set HomeDirectoryType to "Logical"
                  print("Setting HomeDirectoryType to LOGICAL")
                  response_data["HomeDirectoryType"] = "LOGICAL"

              # Note that HomeDirectory and HomeDirectoryDetails / Logical mode
              # can't be used together but we're not checking for this
              home_directory = lookup(secret_dict, "HomeDirectory", input_protocol)
              if home_directory:
                  print("HomeDirectory found - Note: Cannot be used in conjunction with key: HomeDirectoryDetails")
                  response_data["HomeDirectory"] = home_directory

              PosixProfile = lookup(secret_dict, "PosixProfile", input_protocol)
              if PosixProfile:
                  print("Uid found")
                  response_data["PosixProfile"] = PosixProfile

              return response_data

          def get_secret(id):
              region = os.environ["SecretsManagerRegion"]
              print("Secrets Manager Region: " + region)
              print("Secret Name: " + id)

              # Create a Secrets Manager client
              client = boto3.session.Session().client(service_name="secretsmanager", region_name=region)

              try:
                  resp = client.get_secret_value(SecretId=id)
                  # Decrypts secret using the associated KMS CMK.
                  # Depending on whether the secret is a string or binary, one of these fields will be populated.
                  if "SecretString" in resp:
                      print("Found Secret String")
                      return resp["SecretString"]
                  else:
                      print("Found Binary Secret")
                      return base64.b64decode(resp["SecretBinary"])
              except ClientError as err:
                  print("Error Talking to SecretsManager: " + err.response["Error"]["Code"] + ", Message: " +
                        err.response["Error"]["Message"])
                  return None
      Description: A function to lookup and return user data from AWS Secrets Manager.
      Handler: index.lambda_handler
      Role:
        Fn::GetAtt: LambdaExecutionRole.Arn
      Runtime: python3.7
      Environment:
        Variables:
          SecretsManagerRegion: ap-northeast-1
      FunctionName: !Sub
        - lambda-ftp
  EFS:
    Type: AWS::EFS::FileSystem
    Properties:
      BackupPolicy:
        Status: ENABLED
      PerformanceMode: generalPurpose
      ThroughputMode : bursting
      Encrypted: true
      LifecyclePolicies:
        - TransitionToIA: AFTER_30_DAYS
        - TransitionToPrimaryStorageClass: AFTER_1_ACCESS
      FileSystemTags:
        - Key: Name
          Value: !Sub
            - efs-01
  MountTarget:
    Type: AWS::EFS::MountTarget
    Properties:
      FileSystemId: !Ref EFS
      SubnetId: !Ref EFSMountTargetSubnetID
      SecurityGroups:
        - Ref: EFSMountTargetSecurityGroupIDs
  TransferRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - transfer.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: !Sub
          - iampolicy-TransferFamily-01
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              Effect: Allow
              Action:
                - 'elasticfilesystem:ClientMount'
                - 'elasticfilesystem:ClientWrite'
                - 'elasticfilesystem:DescribeMountTargets'
              Resource: !Sub
                - 'arn:aws:elasticfilesystem:ap-northeast-1:${AWS::AccountId}:file-system/${EFSID}'
                - EFSID: !Ref EFS
      RoleName: !Sub
          - 'iamrole-Transfer-01'
    DependsOn: EFS
  SecretManager:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: !Sub
        - '${TFServerID}/${UserName}'
        - TFServerID:
            Fn::GetAtt: TransferServer.ServerId
          UserName: !Ref SecretManagerUserName
      Description: !Sub
        - "This secret has a ${UserName}'s Information."
        - UserName: !Ref SecretManagerUserName
      SecretString: !Sub
        - '{ "Password":"${Password}", "Role":"${IAMARN}", "HomeDirectoryDetails":"[{\"Entry\": \"/\", \"Target\": \"/${EFSID}${HomeDirectory}\"}]", "PosixProfile": { "Uid": "${UserID}", "Gid": "${GroupID}" } }'
        - Password: !Ref SecretManagerPassword
          IAMARN:
            Fn::GetAtt: TransferRole.Arn
          EFSID: !Ref EFS
          HomeDirectory: !Ref SecretManagerHomeDirectory
          UserID: !Ref SecretManagerUserID
          GroupID: !Ref SecretManagerGroupID

Transfer FamilyのVPCエンドポイントについて

VPCエンドポイントのリソースページからTransfer FamilyのVPCエンドポイントを作成しようとすると以下の3種類、もしくは1と2の2種類のサービスが存在します。

1. com.amazonaws.ap-northeast-1.transfer
2. com.amazonaws.ap-northeast-1.transfer.server
3. com.amazonaws.ap-northeast-1.transfer.server.x-xxxx

それぞれの用途は以下の通りです。上記コードを利用/流用される場合は意識する必要はありませんが、個別に設定を行う場合はご注意下さい。

1. com.amazonaws.ap-northeast-1.transfer
   Transfer Family Serverにアクセスするものではなく、Transfer FamilyのAPIを実行するためのエンドポイントです
2. com.amazonaws.ap-northeast-1.transfer.server
   Transfer Familyのサービス開始当初に3のVPC エンドポイントが自動生成されなかったため、このエンドポイントを手動作成し、サーバに紐づけて利用していたエンドポイントです
3. com.amazonaws.ap-northeast-1.transfer.server.x-xxxx
   Transfer Family Server作成時のエンドポイントを VPC に設定すると自動で作成されるエンドポイントで、Transfer Family Serverに対してアクセスするためのエンドポイントです

We Are Hiring!