動かしたスクリプト
Neural Style TransferのLua実装をMacbook（macOS Catalina)で動かしてみました。
MacbookでLuaを動かすための環境構築については、この記事の後半に記載しています。
なお、本手法の背景にある技術については、次の2つがわかりやすいです。
実行コード
- __thは、Lua言語__の実行コマンドです。
- （コマンドライン引数） -style_image : 指定した画像ファイルの「画風」を模倣します。
- （コマンドライン引数） -content_image : 指定した画像ファイルを、上記の「画風」の筆致に書き換えます。
Terminal
% th neural_style.lua -style_image examples/inputs/ghibli_portrait.jpg -content_image examples/inputs/nozomi2.jpg -gpu -1
出力結果
__th neural_style.lua__の実行場所（カレント・ディレクトリ）に、以下のファイルが出力されます。
末尾の数字が大きくなるほど、処理が進んだあとに出力されたファイルです。
__out_900.png__が__最終的に出力された納品ファイル__です。
- out.png
- out_100.png
- out_200.png
- out_300.png
- out_400.png
- out_500.png
- out_600.png
- out_700.png
- out_800.png
- out_900.png
（ 生成された画像 ）
（ その他の画像も試してみました ）
環境構築
Terminal
electron@diynoMacBook-Pro ~ % cd Desktop
electron@diynoMacBook-Pro Desktop % git clone https://github.com/jcjohnson/neural-style.git
Cloning into 'neural-style'...
remote: Enumerating objects: 579, done.
remote: Total 579 (delta 0), reused 0 (delta 0), pack-reused 579
Receiving objects: 100% (579/579), 47.06 MiB | 12.79 MiB/s, done.
Resolving deltas: 100% (269/269), done.
electron@diynoMacBook-Pro Desktop %
中身を確認
Terminal
electron@diynoMacBook-Pro Desktop % cd neural-style
electron@diynoMacBook-Pro neural-style % ls
INSTALL.md LICENSE README.md examples models neural_style.lua
必要なライブラリを追加で入れる
Terminal
electron@diynoMacBook-Pro Desktop % brew install protobuf
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 2 taps (homebrew/core and homebrew/cask).
==> New Formulae
bupstash
==> Updated Formulae
Updated 11 formulae.
==> Updated Casks
Updated 2 casks.
Warning: protobuf 3.17.3 is already installed and up-to-date.
To reinstall 3.17.3, run:
brew reinstall protobuf
electron@diynoMacBook-Pro Desktop %
Terminal
electron@diynoMacBook-Pro Desktop % luarocks install loadcaffe
学習済みのモデルを取得
公式GitHubの説明に従い、__download_models.sh__を実行
Terminal
electron@diynoMacBook-Pro Desktop % cd neural-style
electron@diynoMacBook-Pro neural-style % ls models/download_models.sh
models/download_models.sh
electron@diynoMacBook-Pro neural-style % sh models/download_models.sh
（ 省略 ）
2021-08-09 02:15:52 (6.50 MB/s) - `VGG_ILSVRC_19_layers.caffemodel' へ保存完了 [574671192/574671192]
electron@diynoMacBook-Pro neural-style %
出来上がったディレクトリ構成
Terminal
electron@diynoMacBook-Pro neural-style % tree
.
├── INSTALL.md
├── LICENSE
├── README.md
├── examples
│ ├── inputs
│ │ ├── brad_pitt.jpg
│ │ ├── escher_sphere.jpg
│ │ ├── frida_kahlo.jpg
│ │ ├── golden_gate.jpg
│ │ ├── hoovertowernight.jpg
│ │ ├── picasso_selfport1907.jpg
│ │ ├── seated-nude.jpg
│ │ ├── shipwreck.jpg
│ │ ├── starry_night.jpg
│ │ ├── starry_night_crop.png
│ │ ├── starry_night_google.jpg
│ │ ├── the_scream.jpg
│ │ ├── tubingen.jpg
│ │ └── woman-with-hat-matisse.jpg
│ ├── multigpu_scripts
│ │ └── starry_stanford.sh
│ └── outputs
│ ├── golden_gate_escher.png
│ ├── golden_gate_kahlo.png
│ ├── golden_gate_matisse.png
│ ├── golden_gate_scream.png
│ ├── golden_gate_seated.png
│ ├── golden_gate_starry.png
│ ├── golden_gate_starry_scale05.png
│ ├── golden_gate_starry_scale1.png
│ ├── golden_gate_starry_scale2.png
│ ├── golden_gate_starry_scream_1_9.png
│ ├── golden_gate_starry_scream_3_7.png
│ ├── golden_gate_starry_scream_5_5.png
│ ├── golden_gate_starry_scream_7_3.png
│ ├── golden_gate_starry_scream_9_1.png
│ ├── original_color
│ │ ├── tubingen_composition_vii.png
│ │ ├── tubingen_scream.png
│ │ ├── tubingen_seated_nude.png
│ │ ├── tubingen_shipwreck.png
│ │ └── tubingen_starry.png
│ ├── pitt_picasso_content_0025_style_10.png
│ ├── pitt_picasso_content_01_style_10.png
│ ├── pitt_picasso_content_1_style_10.png
│ ├── pitt_picasso_content_5_style_10.png
│ ├── pitt_picasso_nin_opencl.png
│ ├── starry_stanford_big.png
│ ├── starry_stanford_big_2.png
│ ├── starry_stanford_bigger.png
│ ├── tubingen_composition_vii.png
│ ├── tubingen_scream.png
│ ├── tubingen_scream_composition_vii.png
│ ├── tubingen_seated_nude.png
│ ├── tubingen_seated_nude_composition_vii.png
│ ├── tubingen_shipwreck.png
│ ├── tubingen_starry.png
│ ├── tubingen_starry_scream.png
│ └── tubingen_starry_seated.png
├── models
│ ├── VGG_ILSVRC_19_layers.caffemodel
│ ├── VGG_ILSVRC_19_layers_deploy.prototxt
│ ├── download_models.sh
│ └── vgg_normalised.caffemodel
└── neural_style.lua
6 directories, 59 files
electron@diynoMacBook-Pro neural-style %
取得した学習済みモデルを確認
Terminal
electron@diynoMacBook-Pro neural-style % ls models
VGG_ILSVRC_19_layers.caffemodel VGG_ILSVRC_19_layers_deploy.prototxt download_models.sh vgg_normalised.caffemodel
electron@diynoMacBook-Pro neural-style %
画風変換させたい画像ファイルを置く
（ 格納場所 ）
examples/input直下
Terminal
electron@diynoMacBook-Pro neural-style % ls examples/inputs
brad_pitt.jpg golden_gate.jpg nozomi2.jpg shipwreck.jpg the_scream.jpg
escher_sphere.jpg hoovertowernight.jpg picasso_selfport1907.jpg starry_night.jpg tubingen.jpg
frida_kahlo.jpg nanao.webp renoir_portrait.jpg starry_night_crop.png woman-with-hat-matisse.jpg
ghibli_portrait.jpg nozomi.jpg seated-nude.jpg starry_night_google.jpg
electron@diynoMacBook-Pro neural-style %
参照先の画風（が映った）画像ファイルを置く。
（ 格納場所 ）
examples/input直下
今回は、Google画像検索でwebから取得してきたルノワール作の肖像画の画像ファイルをおきます。
- renoir_portrait.jpg
画風変換処理の実行
Terminal
electron@diynoMacBook-Pro neural-style % th neural_style.lua -style_image examples/inputs/ghibli_portrait.jpg -content_image examples/inputs/nozomi2.jpg -gpu -1
Successfully loaded models/VGG_ILSVRC_19_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv3_4: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv4_4: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
conv5_4: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
Setting up style layer 2 : relu1_1
Setting up style layer 7 : relu2_1
Setting up style layer 12 : relu3_1
Setting up style layer 21 : relu4_1
Setting up content layer 23 : relu4_2
Setting up style layer 30 : relu5_1
Capturing content targets
nn.Sequential {
[input -> (1) -> (2) -> (3) -> (4) -> (5) -> (6) -> (7) -> (8) -> (9) -> (10) -> (11) -> (12) -> (13) -> (14) -> (15) -> (16) -> (17) -> (18) -> (19) -> (20) -> (21) -> (22) -> (23) -> (24) -> (25) -> (26) -> (27) -> (28) -> (29) -> (30) -> (31) -> (32) -> (33) -> (34) -> (35) -> (36) -> (37) -> output]
(1): nn.TVLoss
(2): nn.SpatialConvolution(3 -> 64, 3x3, 1,1, 1,1)
(3): nn.ReLU
(4): nn.StyleLoss
(5): nn.SpatialConvolution(64 -> 64, 3x3, 1,1, 1,1)
(6): nn.ReLU
(7): nn.SpatialMaxPooling(2x2, 2,2)
(8): nn.SpatialConvolution(64 -> 128, 3x3, 1,1, 1,1)
(9): nn.ReLU
(10): nn.StyleLoss
(11): nn.SpatialConvolution(128 -> 128, 3x3, 1,1, 1,1)
(12): nn.ReLU
(13): nn.SpatialMaxPooling(2x2, 2,2)
(14): nn.SpatialConvolution(128 -> 256, 3x3, 1,1, 1,1)
(15): nn.ReLU
(16): nn.StyleLoss
(17): nn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
(18): nn.ReLU
(19): nn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
(20): nn.ReLU
(21): nn.SpatialConvolution(256 -> 256, 3x3, 1,1, 1,1)
(22): nn.ReLU
(23): nn.SpatialMaxPooling(2x2, 2,2)
(24): nn.SpatialConvolution(256 -> 512, 3x3, 1,1, 1,1)
(25): nn.ReLU
(26): nn.StyleLoss
(27): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
(28): nn.ReLU
(29): nn.ContentLoss
(30): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
(31): nn.ReLU
(32): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
(33): nn.ReLU
(34): nn.SpatialMaxPooling(2x2, 2,2)
(35): nn.SpatialConvolution(512 -> 512, 3x3, 1,1, 1,1)
(36): nn.ReLU
(37): nn.StyleLoss
}
Capturing style target 1
Running optimization with L-BFGS
<optim.lbfgs> creating recyclable direction/step/history buffers
Iteration 50 / 1000
Content 1 loss: 2231081.250000
Style 1 loss: 26797.116089
Style 2 loss: 429528.759766
Style 3 loss: 237259.130859
Style 4 loss: 935653.027344
Style 5 loss: 1072.086716
Total loss: 3861391.370773
Iteration 100 / 1000
Content 1 loss: 1698894.843750
Style 1 loss: 20820.108032
Style 2 loss: 217066.943359
Style 3 loss: 105750.048828
Style 4 loss: 440574.316406
Style 5 loss: 1030.835247
Total loss: 2484137.095623
Iteration 150 / 1000
Content 1 loss: 1498420.468750
Style 1 loss: 16410.388184
Style 2 loss: 134153.186035
Style 3 loss: 68532.604980
Style 4 loss: 308941.406250
Style 5 loss: 1057.839489
Total loss: 2027515.893688
Iteration 200 / 1000
Content 1 loss: 1372491.875000
Style 1 loss: 12948.892212
Style 2 loss: 89896.453857
Style 3 loss: 48842.700195
Style 4 loss: 267218.017578
Style 5 loss: 1086.896801
Total loss: 1792484.835644
Iteration 250 / 1000
Content 1 loss: 1301307.109375
Style 1 loss: 10226.798248
Style 2 loss: 63556.909180
Style 3 loss: 38473.391724
Style 4 loss: 244541.113281
Style 5 loss: 1106.688213
Total loss: 1659212.010021
Iteration 300 / 1000
Content 1 loss: 1246721.328125
Style 1 loss: 7762.709045
Style 2 loss: 44931.973267
Style 3 loss: 31255.328369
Style 4 loss: 234602.148438
Style 5 loss: 1122.317696
Total loss: 1566395.804939
Iteration 350 / 1000
Content 1 loss: 1212946.796875
Style 1 loss: 5538.634109
Style 2 loss: 31816.052246
Style 3 loss: 26545.123291
Style 4 loss: 224237.402344
Style 5 loss: 1124.403763
Total loss: 1502208.412628
Iteration 400 / 1000
Content 1 loss: 1184374.921875
Style 1 loss: 3667.434692
Style 2 loss: 22393.768311
Style 3 loss: 22884.857178
Style 4 loss: 217541.186523
Style 5 loss: 1136.209774
Total loss: 1451998.378353
Iteration 450 / 1000
Content 1 loss: 1162057.890625
Style 1 loss: 2371.082497
Style 2 loss: 16727.882385
Style 3 loss: 20576.927185
Style 4 loss: 213357.006836
Style 5 loss: 1143.882465
Total loss: 1416234.671993
Iteration 500 / 1000
Content 1 loss: 1142415.234375
Style 1 loss: 1541.375446
Style 2 loss: 13419.105530
Style 3 loss: 18727.278137
Style 4 loss: 211519.213867
Style 5 loss: 1154.151821
Total loss: 1388776.359177
Iteration 550 / 1000
Content 1 loss: 1127063.281250
Style 1 loss: 1043.627071
Style 2 loss: 11696.353149
Style 3 loss: 17628.172302
Style 4 loss: 210361.962891
Style 5 loss: 1168.090439
Total loss: 1368961.487103
Iteration 600 / 1000
Content 1 loss: 1114870.156250
Style 1 loss: 739.654446
Style 2 loss: 10621.449280
Style 3 loss: 16743.675232
Style 4 loss: 209458.129883
Style 5 loss: 1180.265617
Total loss: 1353613.330708
Iteration 650 / 1000
Content 1 loss: 1105306.796875
Style 1 loss: 577.065754
Style 2 loss: 9999.113464
Style 3 loss: 16162.327576
Style 4 loss: 209075.170898
Style 5 loss: 1185.475159
Total loss: 1342305.949726
Iteration 700 / 1000
Content 1 loss: 1097775.546875
Style 1 loss: 509.323359
Style 2 loss: 9730.662537
Style 3 loss: 15708.894348
Style 4 loss: 208598.071289
Style 5 loss: 1191.439152
Total loss: 1333513.937559
Iteration 750 / 1000
Content 1 loss: 1091324.140625
Style 1 loss: 497.316933
Style 2 loss: 9601.439667
Style 3 loss: 15372.019958
Style 4 loss: 208578.417969
Style 5 loss: 1194.777966
Total loss: 1326568.113117
Iteration 800 / 1000
Content 1 loss: 1086343.281250
Style 1 loss: 509.478331
Style 2 loss: 9581.963348
Style 3 loss: 15085.220337
Style 4 loss: 208392.163086
Style 5 loss: 1198.876667
Total loss: 1321110.983019
Iteration 850 / 1000
Content 1 loss: 1082268.046875
Style 1 loss: 525.061035
Style 2 loss: 9557.526398
Style 3 loss: 14868.060303
Style 4 loss: 208396.752930
Style 5 loss: 1200.657749
Total loss: 1316816.105289
Iteration 900 / 1000
Content 1 loss: 1079115.000000
Style 1 loss: 538.488197
Style 2 loss: 9574.259186
Style 3 loss: 14729.005432
Style 4 loss: 208432.568359
Style 5 loss: 1201.611710
Total loss: 1313590.932884
Iteration 950 / 1000
Content 1 loss: 1076711.718750
Style 1 loss: 550.451899
Style 2 loss: 9597.094727
Style 3 loss: 14648.091125
Style 4 loss: 208304.809570
Style 5 loss: 1201.183987
Total loss: 1311013.350058
Iteration 1000 / 1000
Content 1 loss: 1074437.421875
Style 1 loss: 560.825729
Style 2 loss: 9608.381653
Style 3 loss: 14544.635010
Style 4 loss: 208378.100586
Style 5 loss: 1202.579880
Total loss: 1308731.944733
<optim.lbfgs> reached max number of iterations
electron@diynoMacBook-Pro neural-style %
出力ファイルを確認
カレントディレクトリに、以下のファイルが出力される。
- out.png
- out_100.png
- out_200.png
- out_300.png
- out_400.png
- out_500.png
- out_600.png
- out_700.png
- out_800.png
- out_900.png
Terminal
electron@diynoMacBook-Pro neural-style % open out_900.png