目的

SageMakerのEstimator APIの検証のために WindowsのWSL環境で「CIFAR-10分類のための機械学習トレーニングスクリプト (PyTorch Lightning使用)」を試しに実行してみた。検証メモです。

環境

WSL(Ubuntu)
Linux DESKTOP-C0A2DE7 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Jupyter notebook(WSLのUbuntuに入れて実行)

主な機能:

SageMaker Estimatorのセットアップ。
PyTorch Lightningスクリプトを指定してトレーニングを実行。
ログとモデルアーティファクトをS3に保存。

トレーニングジョブのコンテナ環境

コンテナイメージに使うDockerfileは下記になります。

FROM nvidia/cuda:11.6.1-cudnn8-devel-ubuntu20.04

ENV TZ=Asia/Tokyo
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone

# 必要なパッケージをインストール
RUN apt-get update && \
    apt-get install -yq --no-install-recommends \
        software-properties-common && \
    add-apt-repository ppa:deadsnakes/ppa && \
    apt-get update && \
    apt-get install -yq --no-install-recommends python3.9 python3.9-dev python3.9-distutils \
        build-essential \
        wget \
        git \
        unzip \
        curl \
        tzdata && \
    apt-get clean

# シンボリックリンクを作成
RUN ln -sf /usr/bin/python3.9 /usr/bin/python

# pipのインストール
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python

# 必要なPythonパッケージをインストール
RUN pip install --no-cache-dir torch torchvision --extra-index-url https://download.pytorch.org/whl/cu116
RUN pip install --no-cache-dir pytorch-lightning \
        matplotlib \
        scikit-learn \
        timm \
        sagemaker-training \
        tensorboard

Dockerビルドしてイメージを作成します。sagemaker.gpuが作られましたね。

@DESKTOP-C0A2DE7:~/09_local_mode/docker$ docker build -t sagemaker.gpu -f Dockerfile_local_sagemaker_gpu .

@DESKTOP-C0A2DE7:~/09_local_mode/docker$ docker images
REPOSITORY                                                         TAG                       IMAGE ID       CREATED          SIZE
sagemaker.gpu                                                      latest                    cce745b2c2e9   37 minutes ago   14.6GB
<none>                                                             <none>                    c881fdb09124   55 minutes ago   14.6GB
ubuntu22.04.python                                                 latest                    a8d5dd4a7c63   2 weeks ago      272MB
ubuntu22.04.vim                                                    latest                    001a4fe504ab   2 weeks ago      196MB
public.ecr.aws/aws-cli/aws-cli                                     latest                    cab764e2e45d   2 weeks ago      418MB
nginx                                                              latest                    f876bfc1cc63   7 weeks ago      192MB
255456539449.dkr.ecr.ap-northeast-1.amazonaws.com/test_container   ubuntu.20.04              6013ae1a63c2   3 months ago     72.8MB
ubuntu                                                             20.04                     6013ae1a63c2   3 months ago     72.8MB
ubuntu                                                             22.04                     97271d29cb79   4 months ago     77.9MB
nvidia/cuda                                                        11.0.3-base-ubuntu20.04   97dfa1ef5ee7   14 months ago    122MB

ジョブスクリプト

トレーニングジョブを呼びだすpythonスクリプトは下記の通り。
上記で作成したDockerイメージも
　image_uri="sagemaker.gpu:latest"
で指定しています。

run_local_job.py

import boto3
import sagemaker
from sagemaker.estimator import Estimator

def main():
    # Boto3セッションを初期化 (AWSのプロファイルを指定)
    boto_session = boto3.Session(profile_name="develop")
     
    # SageMakerローカルセッションを作成
    sagemaker_session = sagemaker.LocalSession(boto_session=boto_session)
    # 実行に必要なロールARNを取得
    role = sagemaker_session.get_caller_identity_arn()
  
    # Estimatorの設定
    estimator = Estimator(
        image_uri="sagemaker.gpu:latest",  # 使用するDockerイメージ
        entry_point="09_cifar.py",  # 実行するスクリプト
        role=role,  # 実行権限
        instance_count=1,  # 使用するインスタンス数
        instance_type="local",  # ローカル実行 (CPUの場合)
        output_path="s3://takuma-s3-sagemaker-kensho/sagemaker/logs",  # 出力先S3バケット
        code_location="s3://takuma-s3-sagemaker-kensho/sagemaker/logs",  # スクリプト保存先
        sagemaker_session=sagemaker_session,  # SageMakerセッション
        base_job_name="local-cifar",  # ベースジョブ名
        hyperparameters={  # ハイパーパラメータ
            "ckpt_dir": "/opt/ml/model"  # モデル保存先 (コンテナ内)
        }
    )
    # トレーニングジョブの実行
    estimator.fit()
 
if __name__ == "__main__":
    # メイン関数の実行
    main()

CIFAR-10分類のための機械学習トレーニングスクリプト (PyTorch Lightning使用)

python.py


'''
CIFAR-10分類のための機械学習トレーニングスクリプト (PyTorch Lightning使用)

このスクリプトは、CIFAR-10データセットを分類するための畳み込みニューラルネットワーク (ConvNet) モデルを実装しています。
PyTorch Lightningを使用して、トレーニングおよびバリデーションループの管理、ログ記録、チェックポイントの保存を行います。

主な機能:
- LightningDataModule (`CIFARModule`) を使用したデータ準備。
- PyTorch Lightningモジュールとして定義されたConvNetモデル (`ConvNet`)。
- TensorBoardLoggerを使用したトレーニングとバリデーションのメトリクス記録。
- バリデーション精度に基づくチェックポイント保存。

使用方法:
1. スクリプト内のハイパーパラメータやパスを調整するか、コマンドライン引数として指定してください。
2. 必要な依存関係がインストールされたPython環境でスクリプトを実行してください。
'''

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader
import torchvision
import torchmetrics
import pytorch_lightning as pl
import argparse
 
class ConvNet(pl.LightningModule):
    def __init__(self):
        super().__init__()
        # モデルの各層を定義するリスト
        self.layers = nn.ModuleList()
        for i in range(3):  # 畳み込みブロックを3つ作成
            for j in range(3):  # 各ブロックに3層ずつ追加
                if i == 0 and j == 0:
                    in_chs = 3  # CIFAR-10はRGB画像なので入力チャネル数は3
                elif j == 0:
                    in_chs = 64 * (2**(i-1))  # ブロック間でチャネル数を増加
                else:
                    in_chs = 64 * (2**i)  # ブロック内では一定のチャネル数
                out_chs = 64 * (2**i)
                # 畳み込み層を追加
                self.layers.append(nn.Conv2d(in_chs, out_chs, 3, padding=1))
                # バッチ正規化層を追加
                self.layers.append(nn.BatchNorm2d(out_chs))
                
                # 活性化関数を追加（最終ブロックの最後には追加しない）
                if i != 2 and j != 2:
                    self.layers.append(nn.ReLU(inplace=True))
                # プーリング層を追加（最終ブロック以外の最後に追加）
                if i != 2 and j == 2:
                    self.layers.append(nn.AvgPool2d(2))

        # グローバルプーリング層
        self.global_pool = nn.AdaptiveAvgPool2d((1, 1))
        # 全結合層（10クラス分類用）
        self.fc = nn.Linear(256, 10)

        # トレーニングとバリデーション用の精度メトリクス
        self.train_acc = torchmetrics.Accuracy(task="multiclass", num_classes=10)
        self.val_acc = torchmetrics.Accuracy(task="multiclass", num_classes=10)
    
    def forward(self, inputs):
        # 順伝播の定義
        x = inputs
        for l in self.layers:
            x = l(x)
        # グローバルプーリング後に平坦化
        x = self.global_pool(x).view(x.shape[0], 256)
        # 全結合層を通過
        x = self.fc(x)
        return x
 
    def configure_optimizers(self):
        # オプティマイザ（SGD）と学習率スケジューラの設定
        optimizer = torch.optim.SGD(self.parameters(), lr=1e-2, momentum=0.9)
        scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, [60, 80], gamma=0.2)
        return [optimizer], [scheduler]
 
    def training_step(self, train_batch, batch_idx):
        # トレーニングデータの1バッチを処理
        x, y_true = train_batch  # 入力データとラベル
        y_pred = self.forward(x)  # モデルの順伝播
        loss = F.cross_entropy(y_pred, y_true)  # クロスエントロピー損失
        y_pred_label = torch.argmax(y_pred, dim=-1)  # 予測ラベル
        acc = self.train_acc(y_pred_label, y_true)  # 精度を計算
        # 損失と精度をログに記録
        self.log("train_loss", loss, prog_bar=False, logger=True)
        self.log("train_acc", acc, prog_bar=True, logger=True)
        return loss
 
    def validation_step(self, val_batch, batch_idx):
        # バリデーションデータの1バッチを処理
        x, y_true = val_batch  # 入力データとラベル
        y_pred = self.forward(x)  # モデルの順伝播
        loss = F.cross_entropy(y_pred, y_true)  # クロスエントロピー損失
        y_pred_label = torch.argmax(y_pred, dim=-1)  # 予測ラベル
        acc = self.val_acc(y_pred_label, y_true)  # 精度を計算
        # 損失と精度をログに記録
        self.log("val_loss", loss, prog_bar=False, logger=True)
        self.log("val_acc", acc, prog_bar=True, logger=True)
 
    def on_validation_epoch_end(self):
        # エポック終了時にバリデーション精度を表示し、メトリクスをリセット
        print(f"\nEpoch {self.current_epoch:03} | ValidationAcc={self.val_acc.compute():.2%}")
        self.val_acc.reset()
        
class CIFARModule(pl.LightningDataModule):
    def __init__(self, opt):
        super().__init__()
        self.opt = opt
 
    def prepare_data(self):
        # CIFAR-10データセットの準備
        self.train_dataset = torchvision.datasets.CIFAR10(
            self.opt.data_dir, train=True, download=True,
            transform=torchvision.transforms.Compose([
                torchvision.transforms.RandomCrop((32, 32), padding=2),  # ランダムクロップ
                torchvision.transforms.RandomHorizontalFlip(),  # ランダム水平反転
                torchvision.transforms.ToTensor()  # テンソルに変換
            ]))
        self.val_dataset = torchvision.datasets.CIFAR10(
            self.opt.data_dir, train=False, download=True,
            transform=torchvision.transforms.ToTensor())
 
    def train_dataloader(self):
        # トレーニングデータのデータローダー
        return DataLoader(self.train_dataset, batch_size=128, num_workers=4, shuffle=True)
 
    def val_dataloader(self):
        # バリデーションデータのデータローダー
        return DataLoader(self.val_dataset, batch_size=128, num_workers=4, shuffle=False)
 
def main(opt):
    # ConvNetモデルを初期化
    model = ConvNet()
    # CIFAR-10データモジュールを初期化
    data = CIFARModule(opt)
    # TensorBoardLoggerを設定
    logger = pl.loggers.TensorBoardLogger(f"{opt.ckpt_dir}/logs")
 
    # モデルのチェックポイント設定
    ckpt = pl.callbacks.ModelCheckpoint(
        monitor="val_acc",  # 監視するメトリクス
        dirpath=opt.ckpt_dir,  # 保存先ディレクトリ
        filename="cifar-10-{epoch:03d}-{val_acc:.4f}",  # 保存ファイル名
        save_top_k=3,  # 上位3つのモデルを保存
        mode="max"  # 最大値を基準にする
    )
 
    # トレーナーの設定
    trainer = pl.Trainer(
        max_epochs=100,  # 最大エポック数
        callbacks=[ckpt],  # コールバックの設定
        accelerator="auto",  # 使用可能なデバイスを自動検出
        devices="auto",  # 利用可能なGPUまたはCPUを使用
        logger=logger  # ロガーを指定
    )
    # トレーニングの実行
    trainer.fit(model, data)
 
if __name__ == "__main__":
    # コマンドライン引数の設定
    parser = argparse.ArgumentParser(description="MNIST Training")
    parser.add_argument("--data_dir", type=str, default="./data")  # データセットの保存先
    parser.add_argument("--ckpt_dir", type=str, default="./ckpt")  # チェックポイントの保存先
 
    opt = parser.parse_args()
 
    # メイン関数の実行
    main(opt)

実行結果

time="2025-01-19T22:43:48+09:00" level=warning msg="/tmp/tmpm1ue0tm7/docker-compose.yaml: the attribute version is obsolete, it will be ignored, please remove it to avoid potential confusion"
time="2025-01-19T22:43:48+09:00" level=warning msg="a network with name sagemaker-local exists but was not created for project "tmpm1ue0tm7".\nSet external: true to use an existing network"
Container d7n2uflacp-algo-1-wmrbd Creating
Container d7n2uflacp-algo-1-wmrbd Created
Attaching to d7n2uflacp-algo-1-wmrbd
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | ==========
d7n2uflacp-algo-1-wmrbd | == CUDA ==
d7n2uflacp-algo-1-wmrbd | ==========
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | CUDA Version 11.6.1
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | This container image and its contents are governed by the NVIDIA Deep Learning Container License.
d7n2uflacp-algo-1-wmrbd | By pulling and using the container, you accept the terms and conditions of this license:
d7n2uflacp-algo-1-wmrbd | https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | WARNING: The NVIDIA Driver was not detected. GPU functionality will not be available.
d7n2uflacp-algo-1-wmrbd | Use the NVIDIA Container Toolkit to start this container with GPU support; see
d7n2uflacp-algo-1-wmrbd | https://docs.nvidia.com/datacenter/cloud-native/ .
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | *************************
d7n2uflacp-algo-1-wmrbd | ** DEPRECATION NOTICE! **
d7n2uflacp-algo-1-wmrbd | *************************
d7n2uflacp-algo-1-wmrbd | THIS IMAGE IS DEPRECATED and is scheduled for DELETION.
d7n2uflacp-algo-1-wmrbd | https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/support-policy.md
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,443 botocore.credentials INFO Found credentials in environment variables.
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,652 botocore.httpchecksum INFO Skipping checksum validation. Response did not contain one of the following algorithms: ['crc32', 'sha1', 'sha256'].
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,656 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,656 sagemaker-training-toolkit INFO Failed to parse hyperparameter ckpt_dir value /opt/ml/model to Json.
d7n2uflacp-algo-1-wmrbd | Returning the value itself
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,658 sagemaker-training-toolkit INFO No Neurons detected (normal if no neurons installed)
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,663 sagemaker-training-toolkit INFO instance_groups entry not present in resource_config
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,669 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,669 sagemaker-training-toolkit INFO Failed to parse hyperparameter ckpt_dir value /opt/ml/model to Json.
d7n2uflacp-algo-1-wmrbd | Returning the value itself
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,670 sagemaker-training-toolkit INFO No Neurons detected (normal if no neurons installed)
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,677 sagemaker-training-toolkit INFO instance_groups entry not present in resource_config
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,682 sagemaker-training-toolkit INFO No GPUs detected (normal if no gpus installed)
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,682 sagemaker-training-toolkit INFO Failed to parse hyperparameter ckpt_dir value /opt/ml/model to Json.
d7n2uflacp-algo-1-wmrbd | Returning the value itself
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,683 sagemaker-training-toolkit INFO No Neurons detected (normal if no neurons installed)
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,688 sagemaker-training-toolkit INFO instance_groups entry not present in resource_config
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,692 sagemaker-training-toolkit INFO Invoking user script
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | Training Env:
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | {
d7n2uflacp-algo-1-wmrbd | "additional_framework_parameters": {},
d7n2uflacp-algo-1-wmrbd | "channel_input_dirs": {},
d7n2uflacp-algo-1-wmrbd | "current_host": "algo-1-wmrbd",
d7n2uflacp-algo-1-wmrbd | "current_instance_group": "homogeneousCluster",
d7n2uflacp-algo-1-wmrbd | "current_instance_group_hosts": [],
d7n2uflacp-algo-1-wmrbd | "current_instance_type": "local",
d7n2uflacp-algo-1-wmrbd | "distribution_hosts": [
d7n2uflacp-algo-1-wmrbd | "algo-1-wmrbd"
d7n2uflacp-algo-1-wmrbd | ],
d7n2uflacp-algo-1-wmrbd | "distribution_instance_groups": [],
d7n2uflacp-algo-1-wmrbd | "framework_module": null,
d7n2uflacp-algo-1-wmrbd | "hosts": [
d7n2uflacp-algo-1-wmrbd | "algo-1-wmrbd"
d7n2uflacp-algo-1-wmrbd | ],
d7n2uflacp-algo-1-wmrbd | "hyperparameters": {
d7n2uflacp-algo-1-wmrbd | "ckpt_dir": "/opt/ml/model"
d7n2uflacp-algo-1-wmrbd | },
d7n2uflacp-algo-1-wmrbd | "input_config_dir": "/opt/ml/input/config",
d7n2uflacp-algo-1-wmrbd | "input_data_config": {},
d7n2uflacp-algo-1-wmrbd | "input_dir": "/opt/ml/input",
d7n2uflacp-algo-1-wmrbd | "instance_groups": [],
d7n2uflacp-algo-1-wmrbd | "instance_groups_dict": {},
d7n2uflacp-algo-1-wmrbd | "is_hetero": false,
d7n2uflacp-algo-1-wmrbd | "is_master": true,
d7n2uflacp-algo-1-wmrbd | "is_modelparallel_enabled": null,
d7n2uflacp-algo-1-wmrbd | "is_smddpmprun_installed": false,
d7n2uflacp-algo-1-wmrbd | "is_smddprun_installed": false,
d7n2uflacp-algo-1-wmrbd | "job_name": "local-cifar-2025-01-19-13-43-47-954",
d7n2uflacp-algo-1-wmrbd | "log_level": 20,
d7n2uflacp-algo-1-wmrbd | "master_hostname": "algo-1-wmrbd",
d7n2uflacp-algo-1-wmrbd | "model_dir": "/opt/ml/model",
d7n2uflacp-algo-1-wmrbd | "module_dir": "s3://takuma-s3-sagemaker-kensho/sagemaker/logs/local-cifar-2025-01-19-13-43-47-954/source/sourcedir.tar.gz",
d7n2uflacp-algo-1-wmrbd | "module_name": "09_cifar",
d7n2uflacp-algo-1-wmrbd | "network_interface_name": "eth0",
d7n2uflacp-algo-1-wmrbd | "num_cpus": 16,
d7n2uflacp-algo-1-wmrbd | "num_gpus": 0,
d7n2uflacp-algo-1-wmrbd | "num_neurons": 0,
d7n2uflacp-algo-1-wmrbd | "output_data_dir": "/opt/ml/output/data",
d7n2uflacp-algo-1-wmrbd | "output_dir": "/opt/ml/output",
d7n2uflacp-algo-1-wmrbd | "output_intermediate_dir": "/opt/ml/output/intermediate",
d7n2uflacp-algo-1-wmrbd | "resource_config": {
d7n2uflacp-algo-1-wmrbd | "current_host": "algo-1-wmrbd",
d7n2uflacp-algo-1-wmrbd | "hosts": [
d7n2uflacp-algo-1-wmrbd | "algo-1-wmrbd"
d7n2uflacp-algo-1-wmrbd | ]
d7n2uflacp-algo-1-wmrbd | },
d7n2uflacp-algo-1-wmrbd | "user_entry_point": "09_cifar.py"
d7n2uflacp-algo-1-wmrbd | }
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | Environment variables:
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | SM_HOSTS=["algo-1-wmrbd"]
d7n2uflacp-algo-1-wmrbd | SM_NETWORK_INTERFACE_NAME=eth0
d7n2uflacp-algo-1-wmrbd | SM_HPS={"ckpt_dir":"/opt/ml/model"}
d7n2uflacp-algo-1-wmrbd | SM_USER_ENTRY_POINT=09_cifar.py
d7n2uflacp-algo-1-wmrbd | SM_FRAMEWORK_PARAMS={}
d7n2uflacp-algo-1-wmrbd | SM_RESOURCE_CONFIG={"current_host":"algo-1-wmrbd","hosts":["algo-1-wmrbd"]}
d7n2uflacp-algo-1-wmrbd | SM_INPUT_DATA_CONFIG={}
d7n2uflacp-algo-1-wmrbd | SM_OUTPUT_DATA_DIR=/opt/ml/output/data
d7n2uflacp-algo-1-wmrbd | SM_CHANNELS=[]
d7n2uflacp-algo-1-wmrbd | SM_CURRENT_HOST=algo-1-wmrbd
d7n2uflacp-algo-1-wmrbd | SM_CURRENT_INSTANCE_TYPE=local
d7n2uflacp-algo-1-wmrbd | SM_CURRENT_INSTANCE_GROUP=homogeneousCluster
d7n2uflacp-algo-1-wmrbd | SM_CURRENT_INSTANCE_GROUP_HOSTS=[]
d7n2uflacp-algo-1-wmrbd | SM_INSTANCE_GROUPS=[]
d7n2uflacp-algo-1-wmrbd | SM_INSTANCE_GROUPS_DICT={}
d7n2uflacp-algo-1-wmrbd | SM_DISTRIBUTION_INSTANCE_GROUPS=[]
d7n2uflacp-algo-1-wmrbd | SM_IS_HETERO=false
d7n2uflacp-algo-1-wmrbd | SM_MODULE_NAME=09_cifar
d7n2uflacp-algo-1-wmrbd | SM_LOG_LEVEL=20
d7n2uflacp-algo-1-wmrbd | SM_FRAMEWORK_MODULE=
d7n2uflacp-algo-1-wmrbd | SM_INPUT_DIR=/opt/ml/input
d7n2uflacp-algo-1-wmrbd | SM_INPUT_CONFIG_DIR=/opt/ml/input/config
d7n2uflacp-algo-1-wmrbd | SM_OUTPUT_DIR=/opt/ml/output
d7n2uflacp-algo-1-wmrbd | SM_NUM_CPUS=16
d7n2uflacp-algo-1-wmrbd | SM_NUM_GPUS=0
d7n2uflacp-algo-1-wmrbd | SM_NUM_NEURONS=0
d7n2uflacp-algo-1-wmrbd | SM_MODEL_DIR=/opt/ml/model
d7n2uflacp-algo-1-wmrbd | SM_MODULE_DIR=s3://takuma-s3-sagemaker-kensho/sagemaker/logs/local-cifar-2025-01-19-13-43-47-954/source/sourcedir.tar.gz
d7n2uflacp-algo-1-wmrbd | SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{},"current_host":"algo-1-wmrbd","current_instance_group":"homogeneousCluster","current_instance_group_hosts":[],"current_instance_type":"local","distribution_hosts":["algo-1-wmrbd"],"distribution_instance_groups":[],"framework_module":null,"hosts":["algo-1-wmrbd"],"hyperparameters":{"ckpt_dir":"/opt/ml/model"},"input_config_dir":"/opt/ml/input/config","input_data_config":{},"input_dir":"/opt/ml/input","instance_groups":[],"instance_groups_dict":{},"is_hetero":false,"is_master":true,"is_modelparallel_enabled":null,"is_smddpmprun_installed":false,"is_smddprun_installed":false,"job_name":"local-cifar-2025-01-19-13-43-47-954","log_level":20,"master_hostname":"algo-1-wmrbd","model_dir":"/opt/ml/model","module_dir":"s3://takuma-s3-sagemaker-kensho/sagemaker/logs/local-cifar-2025-01-19-13-43-47-954/source/sourcedir.tar.gz","module_name":"09_cifar","network_interface_name":"eth0","num_cpus":16,"num_gpus":0,"num_neurons":0,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1-wmrbd","hosts":["algo-1-wmrbd"]},"user_entry_point":"09_cifar.py"}
d7n2uflacp-algo-1-wmrbd | SM_USER_ARGS=["--ckpt_dir","/opt/ml/model"]
d7n2uflacp-algo-1-wmrbd | SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate
d7n2uflacp-algo-1-wmrbd | SM_HP_CKPT_DIR=/opt/ml/model
d7n2uflacp-algo-1-wmrbd | PYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/lib/python39.zip:/usr/lib/python3.9:/usr/lib/python3.9/lib-dynload:/usr/local/lib/python3.9/dist-packages:/usr/lib/python3/dist-packages
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | Invoking script with the following command:
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | /usr/bin/python 09_cifar.py --ckpt_dir /opt/ml/model
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,692 sagemaker-training-toolkit INFO Exceptions not imported for SageMaker Debugger as it is not installed.
d7n2uflacp-algo-1-wmrbd | 2025-01-19 22:43:49,692 sagemaker-training-toolkit INFO Exceptions not imported for SageMaker TF as Tensorflow is not installed.
d7n2uflacp-algo-1-wmrbd | GPU available: False, used: False
d7n2uflacp-algo-1-wmrbd | TPU available: False, using: 0 TPU cores
d7n2uflacp-algo-1-wmrbd | HPU available: False, using: 0 HPUs
d7n2uflacp-algo-1-wmrbd | Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data/cifar-10-python.tar.gz
100% 170M/170M [00:42<00:00, 4.03MB/s]
d7n2uflacp-algo-1-wmrbd | Extracting ./data/cifar-10-python.tar.gz to ./data
d7n2uflacp-algo-1-wmrbd | Files already downloaded and verified
d7n2uflacp-algo-1-wmrbd | /usr/local/lib/python3.9/dist-packages/pytorch_lightning/callbacks/model_checkpoint.py:654: Checkpoint directory /opt/ml/model exists and is not empty.
d7n2uflacp-algo-1-wmrbd |
d7n2uflacp-algo-1-wmrbd | | Name | Type | Params | Mode
d7n2uflacp-algo-1-wmrbd | -----------------------------------------------------------
d7n2uflacp-algo-1-wmrbd | 0 | layers | ModuleList | 1.9 M | train
d7n2uflacp-algo-1-wmrbd | 1 | global_pool | AdaptiveAvgPool2d | 0 | train
d7n2uflacp-algo-1-wmrbd | 2 | fc | Linear | 2.6 K | train
d7n2uflacp-algo-1-wmrbd | 3 | train_acc | MulticlassAccuracy | 0 | train
d7n2uflacp-algo-1-wmrbd | 4 | val_acc | MulticlassAccuracy | 0 | train
d7n2uflacp-algo-1-wmrbd | -----------------------------------------------------------
d7n2uflacp-algo-1-wmrbd | 1.9 M Trainable params
d7n2uflacp-algo-1-wmrbd | 0 Non-trainable params
d7n2uflacp-algo-1-wmrbd | 1.9 M Total params
d7n2uflacp-algo-1-wmrbd | 7.701 Total estimated model params size (MB)
d7n2uflacp-algo-1-wmrbd | 29 Modules in train mode
d7n2uflacp-algo-1-wmrbd | 0 Modules in eval mode
Sanity Checking DataLoader 0: 100% 2/2 [00:00<00:00, 3.98it/s]
d7n2uflacp-algo-1-wmrbd | Epoch 000 | ValidationAcc=7.81%
Epoch 0: 100% 391/391 [04:25<00:00, 1.48it/s, v_num=0, train_acc=0.525]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: 0% 0/79 [00:00<?, ?it/s]
Validation DataLoader 0: 0% 0/79 [00:00<?, ?it/s]
Validation DataLoader 0: 1% 1/79 [00:00<00:19, 3.90it/s]
Validation DataLoader 0: 3% 2/79 [00:00<00:17, 4.39it/s]
Validation DataLoader 0: 4% 3/79 [00:00<00:16, 4.57it/s]
Validation DataLoader 0: 5% 4/79 [00:00<00:15, 4.71it/s]
Validation DataLoader 0: 6% 5/79 [00:01<00:15, 4.74it/s]
Validation DataLoader 0: 8% 6/79 [00:01<00:15, 4.76it/s]
Validation DataLoader 0: 9% 7/79 [00:01<00:14, 4.80it/s]
Validation DataLoader 0: 10% 8/79 [00:01<00:14, 4.82it/s]
Validation DataLoader 0: 11% 9/79 [00:01<00:14, 4.84it/s]
Validation DataLoader 0: 13% 10/79 [00:02<00:14, 4.85it/s]
Validation DataLoader 0: 14% 11/79 [00:02<00:14, 4.83it/s]
Validation DataLoader 0: 15% 12/79 [00:02<00:13, 4.86it/s]
Validation DataLoader 0: 16% 13/79 [00:02<00:13, 4.87it/s]
Validation DataLoader 0: 18% 14/79 [00:02<00:13, 4.87it/s]
Validation DataLoader 0: 19% 15/79 [00:03<00:13, 4.86it/s]
Validation DataLoader 0: 20% 16/79 [00:03<00:12, 4.85it/s]
Validation DataLoader 0: 22% 17/79 [00:03<00:12, 4.85it/s]
Validation DataLoader 0: 23% 18/79 [00:03<00:12, 4.87it/s]
Validation DataLoader 0: 24% 19/79 [00:03<00:12, 4.88it/s]
Validation DataLoader 0: 25% 20/79 [00:04<00:12, 4.89it/s]
Validation DataLoader 0: 27% 21/79 [00:04<00:11, 4.88it/s]
Validation DataLoader 0: 28% 22/79 [00:04<00:11, 4.89it/s]
Validation DataLoader 0: 29% 23/79 [00:04<00:11, 4.89it/s]
Validation DataLoader 0: 30% 24/79 [00:04<00:11, 4.90it/s]
Validation DataLoader 0: 32% 25/79 [00:05<00:10, 4.92it/s]
Validation DataLoader 0: 33% 26/79 [00:05<00:10, 4.91it/s]
Validation DataLoader 0: 34% 27/79 [00:05<00:10, 4.92it/s]
Validation DataLoader 0: 35% 28/79 [00:05<00:10, 4.92it/s]
Validation DataLoader 0: 37% 29/79 [00:05<00:10, 4.92it/s]
Validation DataLoader 0: 38% 30/79 [00:06<00:09, 4.93it/s]
Validation DataLoader 0: 39% 31/79 [00:06<00:09, 4.93it/s]
Validation DataLoader 0: 41% 32/79 [00:06<00:09, 4.93it/s]
Validation DataLoader 0: 42% 33/79 [00:06<00:09, 4.93it/s]
Validation DataLoader 0: 43% 34/79 [00:06<00:09, 4.93it/s]
Validation DataLoader 0: 44% 35/79 [00:07<00:08, 4.93it/s]
Validation DataLoader 0: 46% 36/79 [00:07<00:08, 4.93it/s]
Validation DataLoader 0: 47% 37/79 [00:07<00:08, 4.93it/s]
Validation DataLoader 0: 48% 38/79 [00:07<00:08, 4.94it/s]
Validation DataLoader 0: 49% 39/79 [00:07<00:08, 4.94it/s]
Validation DataLoader 0: 51% 40/79 [00:08<00:07, 4.95it/s]
Validation DataLoader 0: 52% 41/79 [00:08<00:07, 4.94it/s]
Validation DataLoader 0: 53% 42/79 [00:08<00:07, 4.94it/s]
Validation DataLoader 0: 54% 43/79 [00:08<00:07, 4.93it/s]
Validation DataLoader 0: 56% 44/79 [00:08<00:07, 4.93it/s]
Validation DataLoader 0: 57% 45/79 [00:09<00:06, 4.92it/s]
Validation DataLoader 0: 58% 46/79 [00:09<00:06, 4.92it/s]
Validation DataLoader 0: 59% 47/79 [00:09<00:06, 4.93it/s]
Validation DataLoader 0: 61% 48/79 [00:09<00:06, 4.93it/s]
Validation DataLoader 0: 62% 49/79 [00:09<00:06, 4.93it/s]
Validation DataLoader 0: 63% 50/79 [00:10<00:05, 4.93it/s]
Validation DataLoader 0: 65% 51/79 [00:10<00:05, 4.93it/s]
Validation DataLoader 0: 66% 52/79 [00:10<00:05, 4.93it/s]
Validation DataLoader 0: 67% 53/79 [00:10<00:05, 4.93it/s]
Validation DataLoader 0: 68% 54/79 [00:10<00:05, 4.93it/s]
Validation DataLoader 0: 70% 55/79 [00:11<00:04, 4.92it/s]
Validation DataLoader 0: 71% 56/79 [00:11<00:04, 4.92it/s]
Validation DataLoader 0: 72% 57/79 [00:11<00:04, 4.92it/s]
Validation DataLoader 0: 73% 58/79 [00:11<00:04, 4.94it/s]
Validation DataLoader 0: 75% 59/79 [00:11<00:04, 4.94it/s]
Validation DataLoader 0: 76% 60/79 [00:12<00:03, 4.95it/s]
Validation DataLoader 0: 77% 61/79 [00:12<00:03, 4.95it/s]
Validation DataLoader 0: 78% 62/79 [00:12<00:03, 4.95it/s]
Validation DataLoader 0: 80% 63/79 [00:12<00:03, 4.95it/s]
Validation DataLoader 0: 81% 64/79 [00:12<00:03, 4.95it/s]
Validation DataLoader 0: 82% 65/79 [00:13<00:02, 4.95it/s]
Validation DataLoader 0: 84% 66/79 [00:13<00:02, 4.95it/s]
Validation DataLoader 0: 85% 67/79 [00:13<00:02, 4.95it/s]
Validation DataLoader 0: 86% 68/79 [00:13<00:02, 4.95it/s]
Validation DataLoader 0: 87% 69/79 [00:13<00:02, 4.95it/s]
Validation DataLoader 0: 89% 70/79 [00:14<00:01, 4.95it/s]
Validation DataLoader 0: 90% 71/79 [00:14<00:01, 4.95it/s]
Validation DataLoader 0: 91% 72/79 [00:14<00:01, 4.94it/s]
Validation DataLoader 0: 92% 73/79 [00:14<00:01, 4.95it/s]
Validation DataLoader 0: 94% 74/79 [00:14<00:01, 4.95it/s]
Validation DataLoader 0: 95% 75/79 [00:15<00:00, 4.94it/s]
Validation DataLoader 0: 96% 76/79 [00:15<00:00, 4.94it/s]
Validation DataLoader 0: 97% 77/79 [00:15<00:00, 4.94it/s]
Validation DataLoader 0: 99% 78/79 [00:15<00:00, 4.95it/s]
Validation DataLoader 0: 100% 79/79 [00:15<00:00, 5.00it/s]
d7n2uflacp-algo-1-wmrbd | Epoch 000 | ValidationAcc=51.52%
d7n2uflacp-algo-1-wmrbd |
Epoch 1: 100% 391/391 [04:23<00:00, 1.48it/s, v_num=0, train_acc=0.663, val_acc=0.515]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: 0% 0/79 [00:00<?, ?it/s]
Validation DataLoader 0: 0% 0/79 [00:00<?, ?it/s]
Validation DataLoader 0: 1% 1/79 [00:00<00:20, 3.89it/s]
Validation DataLoader 0: 3% 2/79 [00:00<00:18, 4.16it/s]
Validation DataLoader 0: 4% 3/79 [00:00<00:17, 4.34it/s]
Validation DataLoader 0: 5% 4/79 [00:00<00:16, 4.46it/s]
Validation DataLoader 0: 6% 5/79 [00:01<00:16, 4.51it/s]
Validation DataLoader 0: 8% 6/79 [00:01<00:15, 4.60it/s]
Validation DataLoader 0: 9% 7/79 [00:01<00:15, 4.62it/s]
Validation DataLoader 0: 10% 8/79 [00:01<00:15, 4.64it/s]
Validation DataLoader 0: 11% 9/79 [00:01<00:15, 4.65it/s]
Validation DataLoader 0: 13% 10/79 [00:02<00:14, 4.66it/s]
Validation DataLoader 0: 14% 11/79 [00:02<00:14, 4.67it/s]
Validation DataLoader 0: 15% 12/79 [00:02<00:14, 4.70it/s]
Validation DataLoader 0: 16% 13/79 [00:02<00:14, 4.71it/s]
Validation DataLoader 0: 18% 14/79 [00:02<00:13, 4.72it/s]
Validation DataLoader 0: 19% 15/79 [00:03<00:13, 4.72it/s]
Validation DataLoader 0: 20% 16/79 [00:03<00:13, 4.73it/s]
Validation DataLoader 0: 22% 17/79 [00:03<00:13, 4.75it/s]
Validation DataLoader 0: 23% 18/79 [00:03<00:12, 4.76it/s]
Validation DataLoader 0: 24% 19/79 [00:03<00:12, 4.77it/s]
Validation DataLoader 0: 25% 20/79 [00:04<00:12, 4.77it/s]
Validation DataLoader 0: 27% 21/79 [00:04<00:12, 4.79it/s]
Validation DataLoader 0: 28% 22/79 [00:04<00:11, 4.79it/s]
Validation DataLoader 0: 29% 23/79 [00:04<00:11, 4.79it/s]
Validation DataLoader 0: 30% 24/79 [00:05<00:11, 4.79it/s]
Validation DataLoader 0: 32% 25/79 [00:05<00:11, 4.81it/s]
Validation DataLoader 0: 33% 26/79 [00:05<00:11, 4.81it/s]
Validation DataLoader 0: 34% 27/79 [00:05<00:10, 4.81it/s]
Validation DataLoader 0: 35% 28/79 [00:05<00:10, 4.82it/s]
Validation DataLoader 0: 37% 29/79 [00:06<00:10, 4.82it/s]
Validation DataLoader 0: 38% 30/79 [00:06<00:10, 4.82it/s]
Validation DataLoader 0: 39% 31/79 [00:06<00:09, 4.83it/s]
Validation DataLoader 0: 41% 32/79 [00:06<00:09, 4.83it/s]
Validation DataLoader 0: 42% 33/79 [00:06<00:09, 4.84it/s]
Validation DataLoader 0: 43% 34/79 [00:07<00:09, 4.84it/s]
Validation DataLoader 0: 44% 35/79 [00:07<00:09, 4.84it/s]
Validation DataLoader 0: 46% 36/79 [00:07<00:08, 4.85it/s]
Validation DataLoader 0: 47% 37/79 [00:07<00:08, 4.85it/s]
Validation DataLoader 0: 48% 38/79 [00:07<00:08, 4.85it/s]
Validation DataLoader 0: 49% 39/79 [00:08<00:08, 4.85it/s]
Validation DataLoader 0: 51% 40/79 [00:08<00:08, 4.85it/s]
Validation DataLoader 0: 52% 41/79 [00:08<00:07, 4.85it/s]
Validation DataLoader 0: 53% 42/79 [00:08<00:07, 4.86it/s]
Validation DataLoader 0: 54% 43/79 [00:08<00:07, 4.86it/s]
Validation DataLoader 0: 56% 44/79 [00:09<00:07, 4.86it/s]
Validation DataLoader 0: 57% 45/79 [00:09<00:06, 4.87it/s]
Validation DataLoader 0: 58% 46/79 [00:09<00:06, 4.87it/s]
Validation DataLoader 0: 59% 47/79 [00:09<00:06, 4.87it/s]
Validation DataLoader 0: 61% 48/79 [00:09<00:06, 4.87it/s]
Validation DataLoader 0: 62% 49/79 [00:10<00:06, 4.88it/s]
Validation DataLoader 0: 63% 50/79 [00:10<00:05, 4.87it/s]
Validation DataLoader 0: 65% 51/79 [00:10<00:05, 4.87it/s]
Validation DataLoader 0: 66% 52/79 [00:10<00:05, 4.88it/s]
Validation DataLoader 0: 67% 53/79 [00:10<00:05, 4.88it/s]
Validation DataLoader 0: 68% 54/79 [00:11<00:05, 4.88it/s]
Validation DataLoader 0: 70% 55/79 [00:11<00:04, 4.88it/s]
Validation DataLoader 0: 71% 56/79 [00:11<00:04, 4.88it/s]
Validation DataLoader 0: 72% 57/79 [00:11<00:04, 4.88it/s]
Validation DataLoader 0: 73% 58/79 [00:11<00:04, 4.88it/s]
Validation DataLoader 0: 75% 59/79 [00:12<00:04, 4.87it/s]
Validation DataLoader 0: 76% 60/79 [00:12<00:03, 4.88it/s]
Validation DataLoader 0: 77% 61/79 [00:12<00:03, 4.88it/s]
Validation DataLoader 0: 78% 62/79 [00:12<00:03, 4.88it/s]
Validation DataLoader 0: 80% 63/79 [00:12<00:03, 4.88it/s]
Validation DataLoader 0: 81% 64/79 [00:13<00:03, 4.88it/s]
Validation DataLoader 0: 82% 65/79 [00:13<00:02, 4.88it/s]
Validation DataLoader 0: 84% 66/79 [00:13<00:02, 4.88it/s]
Validation DataLoader 0: 85% 67/79 [00:13<00:02, 4.89it/s]
Validation DataLoader 0: 86% 68/79 [00:13<00:02, 4.89it/s]
Validation DataLoader 0: 87% 69/79 [00:14<00:02, 4.89it/s]
Validation DataLoader 0: 89% 70/79 [00:14<00:01, 4.88it/s]
Validation DataLoader 0: 90% 71/79 [00:14<00:01, 4.86it/s]
Validation DataLoader 0: 91% 72/79 [00:14<00:01, 4.86it/s]
Validation DataLoader 0: 92% 73/79 [00:15<00:01, 4.85it/s]
Validation DataLoader 0: 94% 74/79 [00:15<00:01, 4.85it/s]
Validation DataLoader 0: 95% 75/79 [00:15<00:00, 4.85it/s]
Validation DataLoader 0: 96% 76/79 [00:15<00:00, 4.84it/s]
Validation DataLoader 0: 97% 77/79 [00:15<00:00, 4.84it/s]
Validation DataLoader 0: 99% 78/79 [00:16<00:00, 4.84it/s]
Validation DataLoader 0: 100% 79/79 [00:16<00:00, 4.89it/s]
d7n2uflacp-algo-1-wmrbd | Epoch 001 | ValidationAcc=53.24%
d7n2uflacp-algo-1-wmrbd |
Epoch 2: 100% 391/391 [04:24<00:00, 1.48it/s, v_num=0, train_acc=0.663, val_acc=0.532]
Validation: | | 0/? [00:00<?, ?it/s]
Validation: 0% 0/79 [00:00<?, ?it/s]
Validation DataLoader 0: 0% 0/79 [00:00<?, ?it/s]
Validation DataLoader 0: 1% 1/79 [00:00<00:20, 3.88it/s]
Validation DataLoader 0: 3% 2/79 [00:00<00:18, 4.22it/s]
Validation DataLoader 0: 4% 3/79 [00:00<00:17, 4.36it/s]
Validation DataLoader 0: 5% 4/79 [00:00<00:17, 4.41it/s]
Validation DataLoader 0: 6% 5/79 [00:01<00:16, 4.43it/s]
以下省略

まとめ

ローカル環境で、トレーニングジョブを実行してみました。
ローカルのGPUではない環境では、トレーニングジョブの一部修正が必要でした。