3
3

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

More than 5 years have passed since last update.

AWS CodeCommit への Git Push 時の変更内容を、Lambda (Python スクリプト) で作成し、SNS 経由でメール通知してみる

Posted at

0.はじめに

AWS CodeCommit を利用していたんですが、

Git Push した時の変更情報の通知が欲しかったので、試してみました。

大枠の流れは、以下。

Qiita_画像_02.001.jpeg

1.SNS のトピックを作成し、所定のメールアドレスを登録する。

基本的に、以下のページの手順に従って設定します。

  1. SNS のトピックを作成します。設定は、以下。

    • トピック名 : ※任意
    • アクセスポリシー : ※以下参考。

{
"Version": "2008-10-17",
"Id": "__default_policy_ID",
"Statement": [
{
"Sid": "__default_statement_ID",
"Effect": "Allow",
"Principal": {
"AWS": "*"
},
"Action": [
"SNS:GetTopicAttributes",
"SNS:SetTopicAttributes",
"SNS:AddPermission",
"SNS:RemovePermission",
"SNS:DeleteTopic",
"SNS:Subscribe",
"SNS:ListSubscriptionsByTopic",
"SNS:Publish",
"SNS:Receive"
],
"Resource": "arn:aws:sns:ap-northeast-1:XXXXXXXXXXXX:SysOps-Lambda-Mnt",
"Condition": {
"StringEquals": {
"AWS:SourceOwner": "XXXXXXXXXXXX"
}
}
}
]
}
```
* FireShot Capture 110 - Simple Notification Service - ap-northeast-1.console.aws.amazon.com.png


  1. 作成した SNS のトピックへ、所定のメールアドレスをサブスクリプションとして登録する。
    • FireShot Capture 111 - Simple Notification Service - ap-northeast-1.console.aws.amazon.com.png
       
  2. 登録後、作成した SNS のトピックへ、パブリッシュして SNS トピックが正常に設定されているか、確認する。

2.Lambda ファンクションを作成する。

  1. IAM ロールを作成します。設定は、以下。

    • ロール名 : ※任意
    • ポリシー名 : ※任意
    • ポリシー : ※以下参考。

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "sns:Publish",
"Resource": [
"arn:aws:sns::XXXXXXXXXXXX:SysOps-Lambda-DLQ",
"arn:aws:sns:ap-northeast-1:XXXXXXXXXXXX:DevOps-Team-XXXX",
"arn:aws:sns:ap-northeast-1:XXXXXXXXXXXX:SysOps-Lambda-Mnt"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"codecommit:GetCommit",
"logs:CreateLogStream",
"codecommit:GetDifferences",
"codecommit:GetBlob",
"logs:CreateLogGroup",
"logs:PutLogEvents"
],
"Resource": "
"
}
]
}
```
* FireShot Capture 107 - IAM Management Console - console.aws.amazon.com.png
* _Key.png


  1. Lambda ファンクションを作成します。設定は、以下。
    • ファンクション名 : ※任意
    • 実行ロール : ※作成した IAM ロール
    • コード : ※以下参考。
#!/usr/bin/env python
# -*- coding: utf-8-unix; -*-
"""AWS Lambda Function - Maintenance DevOps CodeCommit Push to SNS
"""
from __future__ import print_function

import logging
import boto3
import os
import difflib

# 「python - aws lambda Unable to import module 'lambda_function': No module named 'requests' - Stack Overflow」
# https://stackoverflow.com/questions/48912253/aws-lambda-unable-to-import-module-lambda-function-no-module-named-requests
from botocore.vendored import requests

# 「Python 3 で少しだけ便利になった datetime の新機能 - Qiita」
# <https://qiita.com/methane/items/d7ac3c9af5a2c659bc51>
from datetime import datetime, timezone, timedelta
TimeZone = timezone(timedelta(hours=+9), 'JST')

# ISO 8601の日付フォーマットをPythonでparseするには
# http://blog.kzfmix.com/entry/1311336119
import dateutil.parser

# 「Lambdaの本番業務利用を考える① - ログ出力とエラーハンドリング | ナレコムAWSレシピ」
# <https://recipe.kc-cloud.jp/archives/9968>
logger = logging.getLogger()
logLevelTable={'DEBUG':logging.DEBUG,'INFO':logging.INFO,'WARNING':logging.WARNING,'ERROR':logging.ERROR,'CRITICAL':logging.CRITICAL}
if os.environ.get('logging_level') in logLevelTable :
    logger.setLevel(logLevelTable[os.environ['logging_level']])
else:
    logger.setLevel(logging.WARNING)

#
StartDateTime = datetime.now(TimeZone)

# Aws

# Mail
MailSubjectTemplate = "AWS CodeCommit Push ({0}/{1}) By {2} At {3:%Y/%m/%dT%H:%M:%S.%f%z}"

MailMessageTemplate00 = "\
\n\
{0}\n\
\n\
 ■処理時間 : {1:%Y/%m/%d %H:%M:%S} ~ {2:%Y/%m/%d %H:%M:%S}\n\
\n\
 ■コミット\n\
  ・ID     : {3}\n\
  ・Last ID   : {4}\n\
  ・Comment   :\n\
----\n\
{5}\
----\n\
  ・Author   : {6}\n\
  ・Committer  : {7}\n\
  ・AddData   :\n\
----\n\
{8}\
----\n\
\n\
 ■差分\n\
"

MailMessageTemplate01 = "{0}\n\
[{1}] {2}\n\
{3}\
"

MailMessageTemplate02 = "{0}\n\
\n\
----\n\
"

# ------------------------------------------------------------------------------
# CodeCommit Info Get
# ------------------------------------------------------------------------------
def CodeCommitInfoGet(prmRepositoryName, prmBranchName, prmCommitID, prmInfo):
    logging.info("prmRepositoryName:[%s]", prmRepositoryName)
    logging.info("prmBranchName:[%s]", prmBranchName)
    logging.info("prmCommitID:[%s]", prmCommitID)
    result = 0
    try:
        # CodeCommit — Boto 3 Docs 1.9.95 documentation
        # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codecommit.html
        # Amazon CodeCommit にPushしたファイルを S3 に自動保存してみた - 挫折から何かしら学んでいきたい
        # https://rioner2525.hatenablog.com/entry/2018/11/10/161030
        # Automatically publish to S3 using AWS CodeCommit and Lambda – nlytn
        # https://nlytn.me/archives/2018/09/06/204
        codecommit = boto3.client('codecommit')
        commit = codecommit.get_commit(repositoryName=prmRepositoryName,commitId=prmCommitID)
        logging.info("commit:[%s]", commit)
        prmInfo['commit'] = commit['commit']
        prmInfo['differences'] = {}
        if 'commit' in commit:
            if 'parents' in commit['commit'] and len(commit['commit']['parents']) > 0:
                differences = []
                response = codecommit.get_differences(
                    repositoryName=prmRepositoryName,
                    beforeCommitSpecifier=commit['commit']['parents'][0],
                    afterCommitSpecifier=prmCommitID)
                while 'nextToken' in response:
                    differences += response['differences']
                    response = codecommit.get_differences(
                        repositoryName=prmRepositoryName,
                        beforeCommitSpecifier=commit['commit']['parents'][0],
                        afterCommitSpecifier=prmCommitID,
                        nextToken=response['nextToken'])
                else:
                    differences += response['differences']
                logging.info("differences:[%s]", differences)
                for difference in differences:
                    logging.info("difference:[%s]", difference)
                    changeType = difference.get('changeType', '')
                    beforeBlobPath = ''
                    beforeBlobContent = ''
                    if 'beforeBlob' in difference:
                        beforeBlobPath = difference['beforeBlob'].get('path', '')
                        blobID = difference['beforeBlob'].get('blobId', '')
                        if blobID:
                            try:
                                beforeBlobContent = codecommit.get_blob(repositoryName=prmRepositoryName, blobId=blobID)['content'].decode('utf-8')
                            except UnicodeDecodeError:
                                beforeBlobContent = "??? beforeBlobContent UnicodeDecodeError ???\n"
                    logging.info("beforeBlobPath:[%s]", beforeBlobPath)
                    logging.info("beforeBlobContent:[%s]", beforeBlobContent)
                    afterBlobPath = ''
                    afterBlobContent = ''
                    if 'afterBlob' in difference:
                        afterBlobPath = difference['afterBlob'].get('path', '')
                        blobID = difference['afterBlob'].get('blobId', '')
                        if blobID:
                            try:
                                afterBlobContent = codecommit.get_blob(repositoryName=prmRepositoryName, blobId=blobID)['content'].decode('utf-8')
                            except UnicodeDecodeError:
                                afterBlobContent = "??? afterBlobContent UnicodeDecodeError ???\n"
                    logging.info("afterBlobPath:[%s]", afterBlobPath)
                    logging.info("afterBlobContent:[%s]", afterBlobContent)
                    path = ''
                    blobid = ''
                    if changeType:
                        if beforeBlobPath:
                            path = beforeBlobPath
                        elif afterBlobPath:
                            path = afterBlobPath
                    if path:
                        key = changeType + '_' + path
                        prmInfo['differences'][key] = {
                            'changeType': changeType,
                            'path': path,
                            'difference': difference,
                            'beforeBlobContent': beforeBlobContent,
                            'afterBlobContent': afterBlobContent,
                        }
    except Exception as e:
        logger.exception("Error dosomething: %s", e)
        result = 1
        raise
    else:
        pass
    finally:
        pass
    return result
    
# 「【Python】Lambdaからメールを送信 | ハックノート」
# https://hacknote.jp/archives/35679/
def MailSend(prmTopic, prmSubject, prmMessage):
    result = -1
    try:
        sns = boto3.client('sns')
        response = sns.publish(
            TopicArn = 'arn:aws:sns:ap-northeast-1:XXXXXXXXXXXX:' + prmTopic,
            Message = prmMessage,
            Subject = prmSubject
        )
    except Exception as e:
        logger.exception("Error dosomething: %s", e)
        result = -1
        raise
    else:
        pass
    finally:
        pass
    return result

def lambda_handler(event, context):
    try:
        global StartDateTime
        StartDateTime = datetime.now(TimeZone)
        logger.info("StartDateTime:[%s]", StartDateTime)
        #
        logger.info("event:[%s]", event)
        commitID = ""
        commitRef = ""
        eventTime = ""
        eventSourceARN = ""
        userIdentityARN = ""
        customData = ""
        if 'Records' in event and len(event['Records']) > 0:
            eventTime = event['Records'][0].get('eventTime', '')
            eventTime = dateutil.parser.parse(eventTime).astimezone(TimeZone)
            eventSourceARN = event['Records'][0].get('eventSourceARN', '')
            userIdentityARN = event['Records'][0].get('userIdentityARN', '')
            customData = event['Records'][0].get('customData', '')
            if 'codecommit' in event['Records'][0] and 'references' in event['Records'][0]['codecommit'] and len(event['Records'][0]['codecommit']['references']) > 0:
                commitID = event['Records'][0]['codecommit']['references'][0]['commit']
                commitRef = event['Records'][0]['codecommit']['references'][0]['ref']
        logger.info("commitID:[%s]", commitID)
        logger.info("commitRef:[%s]", commitRef)
        logger.info("eventTime:[%s]", eventTime.isoformat())
        logger.info("eventSourceARN:[%s]", eventSourceARN)
        logger.info("userIdentityARN:[%s]", userIdentityARN)
        logger.info("customData:[%s]", customData)
        logger.info("context:[%s]", context)
        #
        branchName = os.path.basename(commitRef)
        userName = os.path.basename(userIdentityARN)
        repositoryName = eventSourceARN.split(":")[-1]
        logger.info("branchName:[%s]", branchName)
        logger.info("userName:[%s]", userName)
        logger.info("repositoryName:[%s]", repositoryName)
        # ----------------------------------
        # CodeCommit Info Get
        # ----------------------------------
        dicInfo = {}
        CodeCommitInfoGet(repositoryName, branchName, commitID, dicInfo)
        logging.info("dicInfo:[%s]", dicInfo)
        # ----------------------------------
        # Mail Create Message
        # ----------------------------------
        # Subject
        tmpSubject = MailSubjectTemplate.format(repositoryName, branchName, userName, eventTime)
        # Message
        tmpMessage = MailMessageTemplate00.format(
            tmpSubject,
            StartDateTime,
            datetime.now(TimeZone),
            dicInfo['commit']['commitId'],
            dicInfo['commit']['parents'],
            dicInfo['commit']['message'],
            dicInfo['commit']['author'],
            dicInfo['commit']['committer'],
            dicInfo['commit']['additionalData'])
        #
        for key in sorted(dicInfo['differences']):
            logging.info("key:[%s]", key)
            logging.info("difference:[%s]", dicInfo['differences'][key])
            # 6.3. difflib — 差分の計算を助ける — Python 3.6.5 ドキュメント
            # https://docs.python.jp/3/library/difflib.html
            # text 文字列 差分 - ファイルを比較するpython difflib - CODE Q&A 問題解決
            # https://code.i-harness.com/ja-jp/q/f21341
            # Pythonのdifflibモジュールを用いて複数行テキストどうしの差分を取得する - 試験運用中なLinux備忘録
            # http://d.hatena.ne.jp/kakurasan/20100308/p1
            diff = difflib.unified_diff(
                dicInfo['differences'][key].get('beforeBlobContent', '').splitlines (True),
                dicInfo['differences'][key].get('afterBlobContent', '').splitlines (True),
                fromfile=dicInfo['differences'][key].get('path', ''),
                tofile=dicInfo['differences'][key].get('path', ''),
                lineterm='\n')
            changes = ""
            for l in diff:
                changes += l
            #changes = [l for l in diff if l.startswith('+ ') or l.startswith('- ')]
            logging.info("changes:[%s]", changes)
            tmpMessage = MailMessageTemplate01.format(
                tmpMessage,
                dicInfo['differences'][key].get('changeType'),
                dicInfo['differences'][key].get('path'),
                changes)
        #
        tmpMessage = MailMessageTemplate02.format(tmpMessage)
        
        # Mail Send
        tmpTopic = 'SysOps-Lambda-Mnt'
        if customData:
            tmpTopic = customData
        logging.info("tmpTopic:[%s]", tmpTopic)
        logging.info("tmpSubject:[%s]", tmpSubject)
        logging.info("tmpMessage:[%s]", tmpMessage)
        MailSend(tmpTopic, tmpSubject, tmpMessage)
        
    except Exception as e:
        logger.exception("Error dosomething: %s", e)
    else:
        pass
    finally:
        pass
    #
    return "normal end"
  • DevOps_CodeCommit_Push_to_SNS.png
  • _AWSCodeCommit.png
  • _AmazonCloudWatchLogs.png
  • _AmazonSNS.png

3.AWS CodeCommit のトリガーを追加する。

  1. Lambda のコンソールから、「トリガーを追加」ボタンを押下し、「CodeCommit」を選択する。


  2. Lambda のコンソールから、「トリガーを追加」ボタンを押下し、「CodeCommit」を選択する。
    • リポジトリ名 : ※所定のリポジトリ
    • トリガー名 : ※任意
    • イベント : ※任意
    • ブランチ名 : ※任意


    • _CodeCommit.png

99.ハマりポイント

  • Python スクリプトで、Diff 情報を作成するのが結構面倒でしたね…。

XX.まとめ

ご参考になれば♪

3
3
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
3
3

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?