自動化：CLIを使用してCDF-PCでデータフローを作成／インポートし、それを環境にデプロイする

目的：
CLIを使用してCDF-PCでデータフローを作成／インポートし、それを環境にデプロイする

1) CDP Cli インストール

MacOSを使用しているので、この方法でcdpcliをインストールします。
https://docs.cloudera.com/cdp-public-cloud/cloud/cli/topics/mc-install-cdp-client-on-macos.html#mc-install-cdp-client-on-macos

pip3 install cdpcli

インストール結果を確認：

[zzeng@zeng-mbp ~]$ ll ~/Library/Python/3.9/bin/ | grep cdp
-rwxr-xr-x@ 1 zzeng  staff   250 Mar  4 12:22 cdp
-rwxr-xr-x@ 1 zzeng  staff   250 Mar  4 12:22 cdp_completer
[zzeng@zeng-mbp ~]$ export PATH="$HOME/Library/Python/3.9/bin:$PATH"
[zzeng@zeng-mbp ~]$ cdp --version
0.9.107
[zzeng@zeng-mbp ~]$

2) CDP cli設定

公式ドキュメント参照URL:
https://docs.cloudera.com/cdp-public-cloud/cloud/cli/topics/mc-configuring-cdp-client-with-the-api-access-key.html

[Management Console] ->[User] -> [Profile]

上記メニューでAPI keyを確認（もしくは作成。）

Cloudera Data PlatformのAPI Keyをメモった後、CLI側で設定する。
ドキュメント：
https://docs.cloudera.com/cdp-public-cloud/cloud/cli/topics/mc-configuring-cdp-client-with-the-api-access-key.html

cdp configure

設定後、下記コマンドで状態確認。

[zzeng@zeng-mbp ~]$ cdp iam get-user
{
    "user": {
        "userId": *****
        "status": "ACTIVE",
        "workloadPasswordDetails": {
            "isPasswordSet": true
        }
    }
}

3) CDP CLIでDataFlowを作成（Import）

コマンドフォーマット：

cdp df import-flow-definition \
  --name "zzeng2-fetch_from_S3_folder" \
  --file "/<<PATH_TO_UPDATE>>/fetch_from_S3_folder.json" \
  --comments "Initial Version"

例：

$ cdp df import-flow-definition   --name "zzeng-fetch_from_S3_folder"   --description "Description for this flow"   --file "/Users/zzeng/Library/CloudStorage/OneDrive-Personal/38_CLDR_Docs/50_demo/FetchFromS3Folder/fetch_from_S3_folder.json"   --comments "Initial Version"
{
    "crn": "crn:cdp:df:us-west-1:******:flow:zzeng-fetch_from_S3_folder",
    "name": "zzeng-fetch_from_S3_folder",
    "versionCount": 1,
    "createdTimestamp": 1709632435790,
    "description": "Description for this flow",
    "modifiedTimestamp": 1709632435790,
    "versions": [
        {
            "crn": "crn:cdp:df:us-west-1:******:flow:zzeng-fetch_from_S3_folder/v.1",
            "bucketIdentifier": "https://s3.us-west-2.amazonaws.com/*****.cloudera.com/******",
            "author": "Zhen Zeng",
            "version": 1,
            "timestamp": 1709632435792,
            "deploymentCount": 0,
            "comments": "Initial Version",
            "draftCount": 0,
            "tags": []
        }
    ]
}

4) 該当DataFlowをDeployする

CDF-PCの画面でWizard式入力して作るか、CLIで作るか、どちらでもできます。
CLIを作る一つ楽な方法は、CDF-PCの画面でWizard式入力して、最後のDeploy画面まで進んだら、「CLIを表示メニュー」があります。
それをクリックしたら、DataFlowをDeployするCLIが自動的に生成されている。

cdp df create-deployment \
  --service-crn crn:cdp:df:us-west-1:558bc1d2-8867-4357-8524-311d51259233:service:08280633-a615-41b4-953d-87b74d5f7fd9 \
  --flow-version-crn "crn:cdp:df:us-west-1:558bc1d2-8867-4357-8524-311d51259233:flow:zzeng-fetch_from_S3_folder/v.1" \
  --deployment-name "zzeng-deploy-01" \
  --project-crn "crn:cdp:df:us-west-1:558bc1d2-8867-4357-8524-311d51259233:project:fd081b85-736a-4648-956e-e80b8848c8db" \
  --cfm-nifi-version 1.24.0.2.3.13.0-9 \
  --auto-start-flow \
  --cluster-size-name EXTRA_SMALL \
  --static-node-count 1 \
  --no-auto-scaling-enabled

例：

$ cdp df create-deployment \
>   --service-crn crn:cdp:df:us-west-1:******:service:***** \
>   --flow-version-crn "crn:cdp:df:us-west-1:*****:flow:zzeng-fetch_from_S3_folder/v.1" \
>   --deployment-name "zzeng-deploy-01" \
>   --project-crn "crn:cdp:df:us-west-1:****:project:*****" \
>   --cfm-nifi-version 1.24.0.2.3.13.0-9 \
>   --auto-start-flow \
>   --cluster-size-name EXTRA_SMALL \
>   --static-node-count 1 \
>   --no-auto-scaling-enabled
{
    "deploymentCrn": "crn:cdp:df:us-west-1:******:deployment:*****/*****"
}

5) cdp-cliのバージョンアップ

pip3 install cdpcli --upgrade --user

例：

[zzeng@zeng-mbp ~/.cdp]$ cdp --version
0.9.107

[zzeng@zeng-mbp ~]$ pip3 install cdpcli --upgrade --user
Requirement already satisfied: cdpcli in ./Library/Python/3.9/lib/python/site-packages (0.9.107)
Collecting cdpcli
  Downloading cdpcli-0.9.125.tar.gz (343 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 343.3/343.3 kB 486.1 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./Library/Python/3.9/lib/python/site-packages (from cdpcli) (2.9.0.post0)
Requirement already satisfied: docutils==0.14 in ./Library/Python/3.9/lib/python/site-packages (from cdpcli) (0.14)
Requirement already satisfied: pyyaml>=3.11 in ./Library/Python/3.9/lib/python/site-packages (from cdpcli) (6.0.1)
Requirement already satisfied: colorama!=0.3.8,!=0.3.9,!=0.4.2,>=0.2.5 in ./Library/Python/3.9/lib/python/site-packages (from cdpcli) (0.4.6)
Requirement already satisfied: asn1crypto>=0.21.1 in ./Library/Python/3.9/lib/python/site-packages (from cdpcli) (1.5.1)
Requirement already satisfied: rsa>=3.4.2 in ./Library/Python/3.9/lib/python/site-packages (from cdpcli) (4.9)
Requirement already satisfied: pure25519>=0.0.1 in ./Library/Python/3.9/lib/python/site-packages (from cdpcli) (0.0.1)
Requirement already satisfied: ecdsa>=0.17.0 in ./Library/Python/3.9/lib/python/site-packages (from cdpcli) (0.18.0)
Requirement already satisfied: requests>=2.21.0 in ./Library/Python/3.9/lib/python/site-packages (from cdpcli) (2.31.0)
Requirement already satisfied: urllib3>=1.21.1 in /Library/Python/3.9/site-packages (from cdpcli) (1.26.18)
Requirement already satisfied: six>=1.9.0 in /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages (from ecdsa>=0.17.0->cdpcli) (1.15.0)
Requirement already satisfied: charset-normalizer<4,>=2 in ./Library/Python/3.9/lib/python/site-packages (from requests>=2.21.0->cdpcli) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in ./Library/Python/3.9/lib/python/site-packages (from requests>=2.21.0->cdpcli) (3.6)
Requirement already satisfied: certifi>=2017.4.17 in ./Library/Python/3.9/lib/python/site-packages (from requests>=2.21.0->cdpcli) (2024.2.2)
Requirement already satisfied: pyasn1>=0.1.3 in ./Library/Python/3.9/lib/python/site-packages (from rsa>=3.4.2->cdpcli) (0.5.1)
Building wheels for collected packages: cdpcli
  Building wheel for cdpcli (setup.py) ... done
  Created wheel for cdpcli: filename=cdpcli-0.9.125-py3-none-any.whl size=379568 sha256=43be2cd77a29a0678a92bb87338ae378d42eb8aaae748adc57f16c9e02a5cbbe
  Stored in directory: /Users/zzeng/Library/Caches/pip/wheels/1d/94/e2/4a74ace6a271b5cd1192486585a26969bb9cc7bf4bfb5f4adb
Successfully built cdpcli
Installing collected packages: cdpcli
  Attempting uninstall: cdpcli
    Found existing installation: cdpcli 0.9.107
    Uninstalling cdpcli-0.9.107:
      Successfully uninstalled cdpcli-0.9.107
Successfully installed cdpcli-0.9.125

[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: /Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip
[zzeng@zeng-mbp ~]$ cdp --version
0.9.125
[zzeng@zeng-mbp ~]$

参考記事：

• https://cloudera.github.io/cdp-dev-docs/cli-docs/dfworkload/update-deployment.html?highlight=update%20deployment
• https://docs.cloudera.com/cdp-public-cloud/cloud/cli/topics/mc-cli-client-setup.html
• https://docs.cloudera.com/dataflow/cloud/deploy-flows/topics/cdf-deploy-flow-cli.html