LoginSignup
3
2

More than 1 year has passed since last update.

@ryotayaguchi(Ryota Yaguchi)

雲の上でクラウドサービスを展開する

Posted at

クラウドと言うからには雲の上で展開できなきゃ嘘だよなぁ。
というわけで本日のアーキテクチャは以下のような形となっております。
image.png

必要なもの

  • ノートパソコン1台

飛行機に搭乗する前の準備

1. そこはかとなくK8sを用意する

たぶんK8sなら何でも良いと思うのですが、以下の例ではVirtualBox上のCentOSにminikubeをインストールします。アレルギーや宗教上の理由でCentOSを利用できない場合はご自身に合わせた環境を用意して下さい。

https://docs.docker.com/engine/install/centos/ 

yum install -y update
sudo reboot
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce
sudo systemctl enable docker; sudo systemctl start docker; systemctl status docker
sudo usermod -aG docker `whoami`
exit
docker run hello-world

https://minikube.sigs.k8s.io/docs/start/ 

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube
minikube config set memory 4096
minikube config set cpus 4
minikube start

2. なんとなくゲストブックを展開してみる

公式チュートリアルのやつだから特に面白みは無いです。

https://kubernetes.io/docs/tutorials/stateless-application/guestbook/ 

minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/redis-leader-deployment.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/redis-leader-service.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/redis-follower-deployment.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/redis-follower-service.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/frontend-deployment.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/frontend-service.yaml
minikube kubectl -- rollout status deployment/frontend
minikube kubectl -- rollout status deployment/redis-follower
minikube kubectl -- rollout status deployment/redis-leader
minikube kubectl -- get svc | grep -E "frontend|redis"

3. さすがに殺風景なのでイタズラを仕込む

URLはお好みの画像に差し替えて下さい。

for P in $(minikube kubectl -- get pod -o name | grep "frontend"); do minikube kubectl -- exec -it ${P} -- sed -i -e 's/<form>/<img width=200 height=200 src="https:\/\/raw.githubusercontent.com\/ryotayaguchi\/kubecat\/main\/img\/wheel_is_yummy.png"><form>/g' /var/www/html/index.html; done

4. それとなくプロキシを1台追加しておく

このままだと svc が ClusterIP なので kubectl port-forward とか kubectl proxy とかを使う羽目になります。それでは誰もインターネット経由で接続してきてくれません。小細工としてプロキシ用のPODを追加で1台デプロイします。

たぶん誰も気にしないと思いますが補足しておくと、ボリュームのマウントが変なのはK8sでsystemd使う為です。あんまり詳しくないのですが、これで動いたので大体合ってるんだと思うことにします。 ~どうせ仕事じゃ使わんし

cat <<EOF > nginx.repo
[nginx-stable]
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF

cat <<EOF > nginx.conf
upstream guestbook{
  server        frontend:80;
}

server {
  listen        80;
  server_name   _;

  location / {
    proxy_pass  http://guestbook;
  }
}
EOF

cat <<EOF > Dockerfile.rmt3
FROM centos/systemd
ADD nginx.repo /etc/yum.repos.d/nginx.repo
RUN yum -y install epel-release; yum -y install sudo iproute jq nginx
RUN yum -y install https://downloads.remote.it/remoteit/v4.13.6/remoteit-4.13.6-1.x86_64.rpm
ADD nginx.conf /etc/nginx/conf.d/default.conf
RUN yum clean all; systemctl enable nginx; rm /etc/remoteit/config.json
STOPSIGNAL SIGRTMIN+3
CMD ["/usr/sbin/init"]
EOF

cat <<EOF > rmt3.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rmt3
  labels:
    app: rmt3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: rmt3
  template:
    metadata:
      labels:
        app: rmt3
    spec:
      containers:
      - name: rmt3
        image: rmt3
        imagePullPolicy: Never
        volumeMounts:
        - mountPath: /tmp
          name: tmp
        - mountPath: /run
          name: run
        - mountPath: /sys/fs/cgroup
          name: cgroup
          readOnly: true
      volumes:
      - name: tmp
        emptyDir: {}
      - name: run
        emptyDir: {}
      - name: cgroup
        hostPath:
          path: /sys/fs/cgroup
          type: Directory
EOF

eval $(minikube docker-env)
docker build --rm --no-cache -t rmt3 . -f Dockerfile.rmt3
minikube kubectl -- apply -f rmt3.yml
minikube kubectl -- rollout status deployment/rmt3

5. インターネット公開用URLを生成して接続テストする

URL作成に必要なAPIを叩くスクリプトを書きます。kubectl の出力を Python Requests に入れてAPIを叩くだけです。
普通はWebブラウザから管理コンソールにアクセスして設定するんだと思いますがGUIは甘え。

sudo yum -y install python3 jq
sudo pip3 install --upgrade pip
pip3 install requests requests_http_signature
rmt3.py
import sys
import os
import json
import requests
from requests_http_signature import HTTPSignatureAuth
from base64 import b64decode

def invoke_rest_api(url_path, http_method, body):
    key_id = os.environ.get('R3_ACCESS_KEY_ID')
    key_secret_id = os.environ.get('R3_SECRET_ACCESS_KEY')
    api_key = os.environ.get('R3_DEVELOPER_API_KEY')

    host = 'api.remote.it'
    content_type_header = 'application/json'
    content_length_header = str(len(body))
    headers = {
        'host': host,
        'content-type': content_type_header,
        'content-length': content_length_header,
        'DeveloperKey': api_key
    }

    auth=HTTPSignatureAuth(algorithm="hmac-sha256",
                           key=b64decode(key_secret_id),
                           key_id=key_id,
                           headers=[
                               '(request-target)', 'host',
                               'date', 'content-type',
                               'content-length'
                           ])
    if http_method == "get":
        response = requests.get('https://' + host + url_path, auth=auth, headers=headers)
    if http_method == "post":
        response = requests.post('https://' + host + url_path, auth=auth, headers=headers, data=body.encode('utf-8'))
    if "response" not in locals():
        print("No response found")
        os.exit()

    if response.status_code == 200:
        print(response.text)
    else:
        print(response.status_code)
        print(response.text)

def get_payload_list():
    query="""query getDevices($size: Int, $from: Int, $sort: String, $state: String, $name: String) {
    login {
        devices(size: $size, from: $from, sort: $sort, state: $state, name: $name) {
            total
            hasMore
            items {
                id
                name
                hardwareId
                state
                services {
                    id
                    name
                    enabled
                    state
                    protocol
                    port
                    title
                }
            }
        }
    }
}"""
    payload={
        "query": query,
        "variables": {
            "size": 1000,
            "from": 0,
            "sort": 'name',
        }
    }
    return json.dumps(payload)


def get_payload_register(code):
    query="""mutation query($code: String!){
    claimDevice(code: $code){
        id
        name
    }
}"""
    payload={
        "query": query,
        "variables": {
            "code": code
        }
    }
    return json.dumps(payload)

def get_payload_addsvc(hwid):
    query="""mutation query($deviceId: String!, $name: String, $application: Int, $host: String, $port: Int, $enabled: Boolean) {
    addService(
        deviceId: $deviceId,
        name: $name,
        application: $application,
        host: $host,
        port: $port,
        enabled: $enabled,
    ) {
      id
    }
}"""
    payload={
        "query": query,
        "variables": {
            "deviceId": hwid,
            "name": "Web protocol",
            "application": 7,
            "host": "127.0.0.1",
            "port": 80,
            "enabled": True
        }
    }
    return json.dumps(payload)

def main():
    f=sys.argv[1]
    if f == "list":
        invoke_rest_api('/graphql/v1', 'post', get_payload_list())
        sys.exit()
    if f == "register":
        invoke_rest_api('/graphql/v1', 'post', get_payload_register(sys.argv[2]))
        sys.exit()
    if f == "rename":
        invoke_rest_api('/apv/v27/device/name', 'post', sys.argv[2])
        sys.exit()
    if f == "addsvc":
        invoke_rest_api('/graphql/v1', 'post', get_payload_addsvc(sys.argv[2]))
        sys.exit()
    if f == "geturl":
        invoke_rest_api('/apv/v27/device/connect', 'post', sys.argv[2])
        sys.exit()
    if f == "delete":
        invoke_rest_api('/apv/v27/developer/device/delete/registered/' + sys.argv[2], 'post', '{}')
        sys.exit()
    print("What's wrong?")

if __name__ == "__main__":
    main()
rmt3.sh
#!/bin/sh
function get_pod_name(){
    PODS=$(minikube kubectl -- get pod --selector app=rmt3 -o jsonpath={.items..metadata.name})
}

if [ "${1}" == "list" ]; then
    echo "--- running pods ---"
    get_pod_name
    for P in ${PODS}; do
        D=$(minikube kubectl -- exec -it ${P} -- cat /etc/remoteit/config.json | jq -er ".device | {\"claim\":.claim,\"id\":.id} | tostring")
        echo "{\"pod\":\"${P}\",\"rmt3\":${D}}" | jq .
    done
    echo "--- registered devices ---"
    python3 rmt3.py list | jq .
    exit
fi

if [ "${1}" == "register" ]; then
    get_pod_name
    for P in ${PODS}; do
        echo "For a pod: ${P}"
        D=$(minikube kubectl -- exec -it ${P} -- cat /etc/remoteit/config.json | jq -er ".device | {\"claim\":.claim,\"id\":.id} | tostring")
        CODE=$(echo ${D} | jq -r ".claim")
        HWID=$(echo ${D} | jq -r ".id")
        if [ "${CODE}" != "null" ]; then
            echo "Registering ${HWID} with ${CODE} ... "
            python3 rmt3.py register ${CODE}
            echo "done"
        else
            echo "No claim code found. skipping ..."
        fi
        echo "Renaming the device with ${P} ... "
        python3 rmt3.py rename "{\"deviceaddress\":\"${HWID}\",\"devicealias\":\"${P}\"}"
        echo "Finding HTTP service ..."
        SVCS=$(python3 rmt3.py list | jq -er ".data.login.devices.items[] | select(.hardwareId==\"${HWID}\") | .services[] | select(.title==\"HTTP\")")
        if [ -z "${SVCS}" ]; then
            echo "No HTTP service found"
            echo "Adding HTTP service ..."
            python3 rmt3.py addsvc "${HWID}"
        else
            echo "Found HTTP service"
            echo "${SVCS}"
        fi
    done
    exit
fi

if [ "${1}" == "geturl" ]; then
    get_pod_name
    for P in ${PODS}; do
        echo "For a pod: ${P}"
        HWID=$(minikube kubectl -- exec -it ${P} -- cat /etc/remoteit/config.json | jq -er ".device.id | tostring")
        echo "Hardware ID: ${HWID}"
        echo "Finding HTTP service ..."
        SVC=$(python3 rmt3.py list | jq -er ".data.login.devices.items[] | select(.hardwareId==\"${HWID}\") | .services | select(.[0].title==\"HTTP\")[0] | {\"id\":.id,\"state\":.state} | tostring")
        SVCID=$(echo ${SVC} | jq -er ".id")
        SVCST=$(echo ${SVC} | jq -er ".state")
        if [ -z "${SVCID}" ]; then
            echo "No HTTP service found"
            break
        fi
        if [ "${SVCST}" != "active" ]; then
            echo "HTTP service is not active"
            break
        fi
        echo "Found device address: ${SVCID}"
        echo "Creating URL ... "
        BODY="{\"deviceaddress\":\"${SVCID}\",\"hostip\":\"255.255.255.255\",\"devicetype\":7,\"isolate\":\"domain=app.remote.it\",\"wait\":\"true\"}"
        python3 rmt3.py geturl "${BODY}" | jq -er ".connection.proxy"
    done
    exit
fi

if [ "${1}" == "delete" ]; then
    HWIDS=$(python3 rmt3.py list | jq -er ".data.login.devices.items[] | select(.state==\"inactive\") | .hardwareId")
    for HWID in ${HWIDS}; do
        echo -n "Deleting a device ${HWID} ..."
        python3 rmt3.py delete "${HWID}"
    done
    exit
fi

スクリプト実行する前にAPIキーを環境変数に放り込みます。

export R3_ACCESS_KEY_ID='**************************'
export R3_SECRET_ACCESS_KEY='**************************'
export R3_DEVELOPER_API_KEY='**************************'

API叩くのに必要な3個のキーは https://remote.it/ でアカウント作成してログインしたらアカウント設定画面から以下をコピペすれば手に入ります。
- Developer API Key
- ACCESS KEY ID
- SECRET ACCESS KEY

誰ですかね。GUIは甘えとか言った人は。

image.png

とりあえずスクリプトの動作確認も兼ねてPODと登録済みデバイスの一覧を表示します。

chmod +x rmt3.sh
./rmt3.sh list

--- running pods ---
{
  "pod": "rmt3-89f7dc959-kf7v5",
  "rmt3": {
    "claim": "AQP37JNM",
    "id": "80:00:01:7F:7E:00:69:E6"
  }
}
--- registered devices ---
{
  "data": {
    "login": {
      "devices": {
        "total": 0,
        "hasMore": false,
        "items": []
      }
    }
  }
}

まだ最初なんで当然ながら登録済みのデバイスなんぞ出てきません。
次に起動中のPODをデバイスとして登録します。

./rmt3.sh register

For a pod: rmt3-89f7dc959-kf7v5
Registering 80:00:01:7F:7E:00:69:F0 with 24KYI3TK ...
{"data":{"claimDevice":{"id":"80:00:01:7F:7E:00:69:F0","name":"My Device"}}}
done
Renaming the device with rmt3-89f7dc959-kf7v5 ...
{"status":"true"}
Finding HTTP service ...
No HTTP service found
Adding HTTP service ...
{"data":{"addService":{"id":"80:00:01:7F:7E:00:69:F1"}}}

最後にデバイスとして登録されたPODのTCP:80をインターネット経由でアクセスできるようにします。

./rmt3.sh geturl

For a pod: rmt3-89f7dc959-kf7v5
Hardware ID: 80:00:01:7F:7E:00:69:F0
Finding HTTP service ...
Found device address: 80:00:01:7F:7E:00:69:F1
Creating URL ...
https://z8dra3r2e5l.p72.rt3.io

出てきたURLにアクセスしてゲストブックが使えたらテスト完了。

image.png

6. 粛々と片づける

コンテナイメージは消さないようにしましょう。機内でダウンロードなんかしたら単なる営業妨害です。

minikube kubectl -- delete deployment rmt3
minikube kubectl -- delete deployment -l app=redis
minikube kubectl -- delete service -l app=redis
minikube kubectl -- delete deployment frontend
minikube kubectl -- delete service frontend
minikube kubectl -- get pod
minikube stop

離陸後の作業

  1. シートベルト着用サインが消えるのを待つ
  2. ノートパソコンを広げてWiFiに接続する

しめやかにクラウドサービスを展開する

すでに仕込み済みなのでチョロいですね。

minikube start
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/redis-leader-deployment.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/redis-leader-service.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/redis-follower-deployment.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/redis-follower-service.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/frontend-deployment.yaml
minikube kubectl -- apply -f https://k8s.io/examples/application/guestbook/frontend-service.yaml
eval $(minikube docker-env)
minikube kubectl -- apply -f rmt3.yml

minikube kubectl -- rollout status deployment/frontend
minikube kubectl -- rollout status deployment/redis-follower
minikube kubectl -- rollout status deployment/redis-leader
minikube kubectl -- get svc | grep -E "frontend|redis"
minikube kubectl -- rollout status deployment/rmt3

for P in $(minikube kubectl -- get pod -o name | grep "frontend"); do minikube kubectl -- exec -it ${P} -- sed -i -e 's/<form>/<img width=200 height=200 src="https:\/\/raw.githubusercontent.com\/ryotayaguchi\/kubecat\/main\/img\/wheel_is_yummy.png"><form>/g' /var/www/html/index.html; done

./rmt3.sh list
./rmt3.sh register
./rmt3.sh geturl

補足

図にするとこんな感じ。
と思ったけど図にするほどの内容でも無かったですね。。。

image.png

3
2
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
Sign upLogin
3
2