Fortran 多次元配列のアクセス順序による計算時間の違い
https://qiita.com/YPonz1/items/66494bb93a5c89437db2
をdockerで実行し、hubに登録しようと作業中。
Fortranをインストールしよう(Ubuntu)
https://www.linuxmania.jp/aptget-site.html
<この項は書きかけです。順次追記します。>
実行環境
macOS 10.12.6, 16GBメモリ, 3GH Intel Corei7, Mac mini(Late 2014)
Docker Desktop version 2.0.0.2(30215) stable: Engine 18.09.1,Compose: 1.23.2, Machine 0.16.1
作業記録
macOS
$ docker run -it ubuntu /bin/bash
docker/ubuntuが起動したら。
docker/ubuntu
# apt update; apt -y upgrade
# apt install -y vim sudo apt-utils wget gfortran
# gfortran cal_array.f90
cal_array.f90:14:8:
do i = 1,n
1
Error: Symbol 'i' at (1) has no IMPLICIT type
cal_array.f90:15:11:
do j = 1,n
1
Error: Symbol 'j' at (1) has no IMPLICIT type
cal_array.f90:16:14:
do k = 1,n
1
Error: Symbol 'k' at (1) has no IMPLICIT type
ソースを見ると
cal_array.f90
program main
implicit none
integer :: n,m,l
...
オリジナルの場所とinteger i,j,kの追加。
cal_array-a.f90
! https://qiita.com/YPonz1/items/66494bb93a5c89437db2
program main
implicit none
integer :: n,m,l,i,j,k
...
実行
# gfortran cal_array-a.f90
# ./a.out
50 0.0000000000000000 0.0000000000000000
75 1.0000000000000000E-002 0.0000000000000000
100 1.9999999999999997E-002 9.9999999999999950E-003
125 5.0000000000000003E-002 9.9999999999999811E-003
150 9.9999999999999978E-002 2.0000000000000018E-002
175 0.22999999999999998 3.0000000000000027E-002
200 0.38000000000000012 4.0000000000000036E-002
225 0.60000000000000009 5.9999999999999609E-002
250 0.79000000000000004 8.0000000000000071E-002
275 1.3199999999999994 0.11000000000000032
300 1.9899999999999993 0.14000000000000057
325 2.4299999999999997 0.16999999999999993
350 3.1799999999999997 0.21000000000000085
375 4.0399999999999991 0.25000000000000000
400 4.8399999999999999 0.30999999999999872
425 6.8299999999999983 0.36999999999999744
450 8.5899999999999963 0.45000000000000284
475 10.210000000000001 0.53000000000000114
10分経っても、ここからびくともしない。
30分経過し、断念。leftだけ計算するように修正して実行。
cal_array-al.f90
! https://qiita.com/YPonz1/items/66494bb93a5c89437db2
! add integer :: i,j,k by @kaizen_nagoya
! left only by @kaizen_nagoya
program main
implicit none
integer :: n,m,l, i, j, k
real(kind=8),allocatable :: a(:,:,:) , b(:,:,:)
real(kind=8) :: t1,t2,s1,s2
do n = 50,1000,25
allocate(a(n,n,n))
allocate(b(n,n,n))
call random_number(a)
call random_number(b)
!----------- left-----------!
call cpu_time(s1)
do i = 1,n
do j = 1,n
do k = 1,n
a(k,j,i) = a(k,j,i) * b(k,j,i)
end do
end do
end do
call cpu_time(s2)
!----------- left -----------!
print*, n, s2-s1
deallocate(a)
deallocate(b)
end do
end program main
実行
docker/ubuntu
# gfortran cal_array-al.f90
# ./a.out
50 0.0000000000000000
75 0.0000000000000000
100 1.0000000000000002E-002
125 9.9999999999999950E-003
150 1.9999999999999990E-002
175 3.0000000000000027E-002
200 4.0000000000000036E-002
225 5.9999999999999942E-002
250 8.0000000000000071E-002
275 0.11000000000000010
300 0.13999999999999968
325 0.17999999999999972
350 0.20999999999999996
375 0.26000000000000068
400 0.31999999999999940
425 0.37999999999999901
450 0.46000000000000085
475 0.53999999999999915
500 2.3099999999999987
525 2.6700000000000017
550 3.1999999999999993
killed
ほっておいたら、いつのまにかkilled。再実行。
# ./a.out
50 0.0000000000000000
75 0.0000000000000000
100 1.0000000000000002E-002
125 2.0000000000000004E-002
150 1.9999999999999990E-002
175 3.0000000000000027E-002
200 4.0000000000000036E-002
225 6.0000000000000053E-002
250 7.9999999999999849E-002
275 0.10999999999999988
300 0.12999999999999989
325 0.16999999999999993
350 0.21999999999999975
375 0.26999999999999957
400 0.35999999999999943
425 0.46999999999999886
450 0.46999999999999886
475 0.62999999999999901
500 2.4800000000000004
525 3.1000000000000014
550 3.6199999999999974
Killed
やっぱり、誰かに殺された。
コンパイラのオプションなどを捜索。
docker hub登録
捜索中につき、現状をdocker hubに保存。
macOS
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d557aa91ea17 ubuntu "/bin/bash" About an hour ago Up About an hour crazy_tesla
$ docker commit d557aa91ea17 kaizenjapan/fortran
sha256:2a8bd111810c2d7d70b68a160f2f3e7a50d2d2cadeec3d3effce04297d8826a8
$ docker push kaizenjapan/fortran
The push refers to repository [docker.io/kaizenjapan/fortran]
545ccf59ca4d: Pushed
b079b3fa8d1b: Mounted from library/ubuntu
a31dbd3063d7: Mounted from library/ubuntu
c56e09e1bd18: Mounted from library/ubuntu
543791078bdb: Mounted from library/ubuntu
latest: digest: sha256:79feba280e0082593b67df42fdd5f17c914e0587ba56d98fabdcccba8acf286e size: 1364
docker hubからfortranを実行するやつを起動する場合は、
$ docker run -it kaizenjapan/fortran /bin/bash
dockerが起動したら
docker/ubuntu
# cd fortran
でいけるはず。
# ループを短く
1000を475に変更。cal_array-b.f90
# gfortran cal_array-b.f90
# ./a.out
50 0.0000000000000000 0.0000000000000000
75 1.0000000000000000E-002 0.0000000000000000
100 1.9999999999999997E-002 9.9999999999999950E-003
125 5.0000000000000003E-002 1.0000000000000009E-002
150 9.0000000000000024E-002 1.9999999999999962E-002
175 0.21999999999999992 3.0000000000000027E-002
200 0.38000000000000000 4.0000000000000036E-002
225 0.57000000000000006 4.9999999999999822E-002
250 0.81000000000000005 7.9999999999999627E-002
275 1.2500000000000000 9.9999999999999645E-002
300 1.6100000000000003 0.12999999999999989
325 2.3300000000000001 0.16999999999999993
350 3.2699999999999996 0.20999999999999908
375 4.1900000000000013 0.25000000000000000
400 4.7900000000000027 0.30999999999999872
425 6.8300000000000018 0.36999999999999744
450 8.3700000000000045 0.44999999999999574
475 9.8999999999999986 0.52000000000000313
# ./a.out
50 0.0000000000000000 0.0000000000000000
75 1.0000000000000000E-002 0.0000000000000000
100 1.9999999999999997E-002 9.9999999999999950E-003
125 5.0000000000000003E-002 9.9999999999999811E-003
150 0.10999999999999999 2.0000000000000018E-002
175 0.23999999999999999 2.0000000000000018E-002
200 0.42000000000000004 4.0000000000000036E-002
225 0.58000000000000007 5.0000000000000266E-002
250 0.83000000000000007 8.0000000000000071E-002
275 1.2500000000000000 0.10000000000000053
300 1.6200000000000001 0.12999999999999989
325 2.3300000000000001 0.16999999999999993
350 3.3000000000000007 0.21000000000000085
375 4.0999999999999996 0.25999999999999801
400 4.8299999999999983 0.30999999999999872
425 6.9000000000000021 0.37999999999999545
450 8.3999999999999986 0.45000000000000284
475 9.9699999999999989 0.52000000000000313
help
docker/ubuntu
# gfortran --help
Usage: gfortran [options] file...
Options:
-pass-exit-codes Exit with highest error code from a phase.
--help Display this information.
--target-help Display target specific command line options.
--help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
Display specific types of command line options.
(Use '-v --help' to display command line options of sub-processes).
--version Display compiler version information.
-dumpspecs Display all of the built in spec strings.
-dumpversion Display the version of the compiler.
-dumpmachine Display the compiler's target processor.
-print-search-dirs Display the directories in the compiler's search path.
-print-libgcc-file-name Display the name of the compiler's companion library.
-print-file-name=<lib> Display the full path to library <lib>.
-print-prog-name=<prog> Display the full path to compiler component <prog>.
-print-multiarch Display the target's normalized GNU triplet, used as
a component in the library path.
-print-multi-directory Display the root directory for versions of libgcc.
-print-multi-lib Display the mapping between command line options and
multiple library search directories.
-print-multi-os-directory Display the relative path to OS libraries.
-print-sysroot Display the target libraries directory.
-print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
-Wa,<options> Pass comma-separated <options> on to the assembler.
-Wp,<options> Pass comma-separated <options> on to the preprocessor.
-Wl,<options> Pass comma-separated <options> on to the linker.
-Xassembler <arg> Pass <arg> on to the assembler.
-Xpreprocessor <arg> Pass <arg> on to the preprocessor.
-Xlinker <arg> Pass <arg> on to the linker.
-save-temps Do not delete intermediate files.
-save-temps=<arg> Do not delete intermediate files.
-no-canonical-prefixes Do not canonicalize paths when building relative
prefixes to other gcc components.
-pipe Use pipes rather than intermediate files.
-time Time the execution of each subprocess.
-specs=<file> Override built-in specs with the contents of <file>.
-std=<standard> Assume that the input sources are for <standard>.
--sysroot=<directory> Use <directory> as the root directory for headers
and libraries.
-B <directory> Add <directory> to the compiler's search paths.
-v Display the programs invoked by the compiler.
-### Like -v but options quoted and commands not executed.
-E Preprocess only; do not compile, assemble or link.
-S Compile only; do not assemble or link.
-c Compile and assemble, but do not link.
-o <file> Place the output into <file>.
-pie Create a position independent executable.
-shared Create a shared library.
-x <language> Specify the language of the following input files.
Permissible languages include: c c++ assembler none
'none' means revert to the default behavior of
guessing the language based on the file's extension.
Options starting with -g, -f, -m, -O, -W, or --param are automatically
passed on to the various sub-processes invoked by gfortran. In order to pass
other options on to these processes the -W<letter> options must be used.
For bug reporting instructions, please see:
<file:///usr/share/doc/gcc-7/README.Bugs>.
最適化
ファイルの大きさを比較するため -o で出力ファイル指定。
docker/ubuntu
# gfortran cal_array-b.f90 -o calb
# ./calb
50 0.0000000000000000 0.0000000000000000
75 1.0000000000000000E-002 0.0000000000000000
100 1.9999999999999997E-002 9.9999999999999950E-003
125 5.0000000000000003E-002 9.9999999999999811E-003
150 0.10999999999999999 2.0000000000000018E-002
175 0.23999999999999999 2.0000000000000018E-002
200 0.38000000000000000 4.0000000000000036E-002
225 0.56999999999999984 4.9999999999999822E-002
250 0.81000000000000005 8.0000000000000071E-002
275 1.1900000000000004 9.9999999999999645E-002
300 1.6200000000000001 0.14000000000000057
325 2.3300000000000001 0.16999999999999993
350 3.1899999999999995 0.20999999999999908
375 4.0800000000000001 0.25000000000000000
400 4.8500000000000014 0.32000000000000028
425 6.7600000000000016 0.36999999999999744
450 8.4900000000000020 0.44999999999999574
475 10.130000000000003 0.53999999999999915
最適化 -O3
docker/ubuntu
# gfortran cal_array-b.f90 -o calbO3 -O3
# ./calbO3
50 0.0000000000000000 0.0000000000000000
75 0.0000000000000000 0.0000000000000000
100 1.9999999999999997E-002 0.0000000000000000
125 2.0000000000000004E-002 9.9999999999999950E-003
150 5.9999999999999998E-002 0.0000000000000000
175 0.12000000000000000 1.0000000000000009E-002
200 0.22999999999999998 1.0000000000000009E-002
225 0.32000000000000006 2.0000000000000018E-002
250 0.48999999999999999 2.0000000000000018E-002
275 0.83999999999999986 3.0000000000000249E-002
300 1.2199999999999998 3.0000000000000249E-002
325 1.9600000000000000 4.9999999999999822E-002
350 2.5999999999999996 7.0000000000000284E-002
375 3.5600000000000005 8.0000000000000071E-002
400 4.1299999999999990 8.9999999999999858E-002
425 6.5000000000000000 0.10999999999999943
450 8.0800000000000018 0.12999999999999545
475 9.2100000000000009 0.14999999999999858
# ls -al calb*
-rwxr-xr-x 1 root root 13176 Aug 1 02:11 calb
-rwxr-xr-x 1 root root 13112 Aug 1 02:14 calbO3
大きさはあまり違いがない。rightの実行時間は少し速く、leftの実行時間はちょっとは速くかも。
ここまでの記録をdocker hubに再度上げ直しました。
dockerでも1000まで事項させる方法など、なにかありましたら、続きをよろしくお願いします。
docker/ubuntu
# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 61255492 2420008 55694160 5% /
tmpfs 65536 0 65536 0% /dev
tmpfs 1023564 0 1023564 0% /sys/fs/cgroup
shm 65536 0 65536 0% /dev/shm
/dev/sda1 61255492 2420008 55694160 5% /etc/hosts
tmpfs 1023564 0 1023564 0% /proc/acpi
tmpfs 1023564 0 1023564 0% /sys/firmware
# free -m
total used free shared buff/cache available
Mem: 1999 106 1671 0 220 1749
Swap: 1023 198 825
docker メモリ設定
docker containerにもっと多くのメモリを割り当てる方法
https://codeday.me/jp/qa/20190301/341141.html
macOSのdockerアイコンで、Preferenceを選び、Advancedタグを選択すると
メモリ2Gb, Swap1Gbになっていた。
メモリ6Gb, Swap2Gbに変更
1000まで最適化してから実行。
docker/ubuntu
# gfortran cal_array-a.f90 -o cala
# gfortran cal_array-a.f90 -o calaO3 -O3
# la -al cala*
-rwxr-xr-x 1 root root 13176 Aug 1 03:53 cala
-rwxr-xr-x 1 root root 13112 Aug 1 03:53 calaO3
# ./calaO3
50 0.0000000000000000 0.0000000000000000
75 0.0000000000000000 0.0000000000000000
100 1.0000000000000002E-002 0.0000000000000000
125 1.9999999999999990E-002 1.0000000000000009E-002
150 5.0000000000000017E-002 0.0000000000000000
175 0.10999999999999999 1.0000000000000009E-002
200 0.19999999999999996 1.0000000000000009E-002
225 0.30000000000000004 2.0000000000000018E-002
250 0.45999999999999996 1.9999999999999574E-002
275 0.80999999999999961 3.0000000000000249E-002
300 1.0800000000000001 4.0000000000000036E-002
325 1.7999999999999998 4.9999999999999822E-002
350 2.5999999999999996 7.0000000000000284E-002
375 3.2799999999999994 8.0000000000000071E-002
400 3.9100000000000001 8.9999999999999858E-002
425 6.1600000000000001 0.10999999999999943
450 7.6300000000000026 0.13000000000000256
475 8.8299999999999983 0.16000000000000369
500 14.489999999999995 0.18000000000000682
525 14.009999999999991 0.21000000000000796
550 16.019999999999996 0.25000000000000000
575 21.170000000000002 0.34000000000000341
600 22.659999999999997 0.40999999999999659
625 29.090000000000003 0.38999999999998636
650 31.830000000000013 0.40000000000000568
675 34.540000000000020 0.44999999999998863
700 39.010000000000048 0.48999999999995225
725 70.210000000000036 1.3100000000000023
ERRO[1445] error waiting for container: EOF
プログラムの大きさは、475まででも1000まででも、大きさは同じ。
最適化の有無の差も同じ。
実行させたが、10分待ちきれない。
メモリ12Gb, SWAP4Gbに変更。
docker/ubuntu
# ./calaO3
50 0.0000000000000000 0.0000000000000000
75 0.0000000000000000 0.0000000000000000
100 1.9999999999999997E-002 0.0000000000000000
125 2.9999999999999999E-002 0.0000000000000000
150 4.0000000000000008E-002 9.9999999999999811E-003
175 0.11000000000000004 9.9999999999999534E-003
200 0.20999999999999996 1.0000000000000009E-002
225 0.31000000000000005 2.0000000000000018E-002
250 0.46999999999999975 3.0000000000000249E-002
275 0.82000000000000028 2.9999999999999805E-002
300 1.0300000000000002 4.0000000000000036E-002
325 1.6299999999999999 4.9999999999999822E-002
350 2.3900000000000006 6.0000000000000497E-002
375 3.5100000000000016 6.9999999999998508E-002
400 4.5299999999999976 0.10999999999999943
425 8.2399999999999984 0.10999999999999943
450 8.3500000000000014 0.14000000000000057
475 10.590000000000003 0.15999999999999659
500 11.549999999999997 0.18000000000000682
525 14.200000000000003 0.21000000000000796
550 18.790000000000006 0.23999999999999488
575 20.820000000000007 0.28000000000000114
600 23.989999999999981 0.33000000000001251
625 26.549999999999983 0.36000000000001364
650 32.520000000000010 0.39999999999997726
675 46.919999999999987 0.43999999999999773
700 48.669999999999959 0.58000000000004093
725 67.829999999999984 0.56000000000000227
750 71.959999999999980 0.68000000000006366
775 83.670000000000073 0.76999999999998181
800 83.899999999999977 1.1600000000000819
825 99.600000000000023 0.96000000000003638
850 108.47000000000003 0.97000000000002728
875 133.09000000000003 0.97000000000002728
900 151.77999999999997 1.0599999999999454
ERRO[1842] error waiting for container: EOF
ここまで25分くらい。待ちきれない。断念。
まとめ
| メモリ | 実行範囲 |
|---|---|
| 2Gb | 475 |
| 6Gb | 725 |
| 12Gb | 900 |
16Gbくらいないと1000まで実行できなさそう。
参考資料
Dockerのメモリ制限に関するメモ
https://qiita.com/irotoris/items/944aba5e448a8e723ff6
今日のdocker error
https://qiita.com/kaizen_nagoya/items/63ece3ddab07a4340dfd
docker rm, docker rmiして、macOSを再起動してから、fortran作業始めています。
＜この記事は個人の過去の経験に基づく個人の感想です。現在所属する組織、業務とは関係がありません。＞
Reference
Ethernet 記事一覧 Ethernet(0)
https://qiita.com/kaizen_nagoya/items/88d35e99f74aefc98794
Wireshark 一覧 wireshark(0)、Ethernet(48)
https://qiita.com/kaizen_nagoya/items/fbed841f61875c4731d0
線網（Wi-Fi）空中線(antenna)(0) 記事一覧(118/300目標)
https://qiita.com/kaizen_nagoya/items/5e5464ac2b24bd4cd001
C++ Support(0)
https://qiita.com/kaizen_nagoya/items/8720d26f762369a80514
Coding Rules(0) C Secure , MISRA and so on
https://qiita.com/kaizen_nagoya/items/400725644a8a0e90fbb0
Autosar Guidelines C++14 example code compile list(1-169)
https://qiita.com/kaizen_nagoya/items/8ccbf6675c3494d57a76
Error一覧(C/C++, python, bash...) Error(0)
https://qiita.com/kaizen_nagoya/items/48b6cbc8d68eae2c42b8
なぜdockerで機械学習するか 書籍・ソース一覧作成中 (目標100)
https://qiita.com/kaizen_nagoya/items/ddd12477544bf5ba85e2
言語処理100本ノックをdockerで。python覚えるのに最適。:10+12
https://qiita.com/kaizen_nagoya/items/7e7eb7c543e0c18438c4
プログラムちょい替え（0）一覧:4件
https://qiita.com/kaizen_nagoya/items/296d87ef4bfd516bc394
一覧の一覧( The directory of directories of mine.) Qiita(100)
https://qiita.com/kaizen_nagoya/items/7eb0e006543886138f39
官公庁・学校・公的団体（NPOを含む）システムの課題、官（０）
https://qiita.com/kaizen_nagoya/items/04ee6eaf7ec13d3af4c3
プログラマが知っていると良い「公序良俗」
https://qiita.com/kaizen_nagoya/items/9fe7c0dfac2fbd77a945
自動制御、制御工学一覧（０）
https://qiita.com/kaizen_nagoya/items/7767a4e19a6ae1479e6b
Rust(0) 一覧
https://qiita.com/kaizen_nagoya/items/5e8bb080ba6ca0281927
小川清最終講義、最終講義（再）計画, Ethernet(100) 英語(100) 安全(100)
https://qiita.com/kaizen_nagoya/items/e2df642e3951e35e6a53
データサイエンティストの気づき『勉強だけして仕事に役立てない人。大嫌い』それ自分かもってなった。
https://qiita.com/kaizen_nagoya/items/d85830d58d8dd7f71d07
文書履歴(document history)
ver. 0.01 初稿 20190801 午前9時
ver. 0.02 半分にしてみた 20190801 午前10時
ver. 0.03 docker hubにひとまず登録 午前11時
ver. 0.04 dockerへのメモリ割り当てを変更 正午
ver. 0.05 標題追記 20191231
ver. 0.06 表題追記 20210327
ver. 0.07 データサイエンティストの気づき『勉強だけして仕事に役立てない人。大嫌い』それ自分かもってなった。追記 20210919
最後までおよみいただきありがとうございました。
いいね 💚、フォローをお願いします。
Thank you very much for reading to the last sentence.
Please press the like icon 💚 and follow me for your happy life.
