Fortran 多次元配列のアクセス順序による計算時間の違い。docker(96)プログラムちょい替え(11)

Fortran 多次元配列のアクセス順序による計算時間の違い

をdockerで実行し、hubに登録しようと作業中。

Fortranをインストールしよう(Ubuntu)
https://www.linuxmania.jp/aptget-site.html

<この項は書きかけです。順次追記します。>

実行環境

macOS 10.12.6, 16GBメモリ, 3GH Intel Corei7, Mac mini(Late 2014)
Docker Desktop version 2.0.0.2(30215) stable: Engine 18.09.1,Compose: 1.23.2, Machine 0.16.1

作業記録

macOS

$ docker run -it ubuntu /bin/bash

docker/ubuntuが起動したら。

docker/ubuntu

# apt update; apt -y upgrade
# apt install -y vim sudo apt-utils wget gfortran
# gfortran cal_array.f90 
cal_array.f90:14:8:

     do i = 1,n
        1
Error: Symbol 'i' at (1) has no IMPLICIT type
cal_array.f90:15:11:

        do j = 1,n
           1
Error: Symbol 'j' at (1) has no IMPLICIT type
cal_array.f90:16:14:

           do k = 1,n
              1
Error: Symbol 'k' at (1) has no IMPLICIT type

ソースを見ると

cal_array.f90

program main
  implicit none
  integer :: n,m,l
...

オリジナルの場所とinteger i,j,kの追加。

cal_array-a.f90

! https://qiita.com/YPonz1/items/66494bb93a5c89437db2
program main
  implicit none
  integer :: n,m,l,i,j,k
...

実行

# gfortran cal_array-a.f90 
# ./a.out
          50   0.0000000000000000        0.0000000000000000     
          75   1.0000000000000000E-002   0.0000000000000000     
         100   1.9999999999999997E-002   9.9999999999999950E-003
         125   5.0000000000000003E-002   9.9999999999999811E-003
         150   9.9999999999999978E-002   2.0000000000000018E-002
         175  0.22999999999999998        3.0000000000000027E-002
         200  0.38000000000000012        4.0000000000000036E-002
         225  0.60000000000000009        5.9999999999999609E-002
         250  0.79000000000000004        8.0000000000000071E-002
         275   1.3199999999999994       0.11000000000000032     
         300   1.9899999999999993       0.14000000000000057     
         325   2.4299999999999997       0.16999999999999993     
         350   3.1799999999999997       0.21000000000000085     
         375   4.0399999999999991       0.25000000000000000     
         400   4.8399999999999999       0.30999999999999872     
         425   6.8299999999999983       0.36999999999999744     
         450   8.5899999999999963       0.45000000000000284     
         475   10.210000000000001       0.53000000000000114     

10分経っても、ここからびくともしない。

30分経過し、断念。leftだけ計算するように修正して実行。

cal_array-al.f90

! https://qiita.com/YPonz1/items/66494bb93a5c89437db2
! add integer :: i,j,k by @kaizen_nagoya
! left only by @kaizen_nagoya
program main
  implicit none
  integer :: n,m,l, i, j, k
  real(kind=8),allocatable ::  a(:,:,:) , b(:,:,:)
  real(kind=8) :: t1,t2,s1,s2

  do  n = 50,1000,25
    allocate(a(n,n,n))
    allocate(b(n,n,n))
    call random_number(a)
    call random_number(b)
 !----------- left-----------!
    call cpu_time(s1)
    do i = 1,n
       do j = 1,n
          do k = 1,n
             a(k,j,i) = a(k,j,i) * b(k,j,i)
          end do
       end do
    end do
    call cpu_time(s2)
 !----------- left -----------!
  print*, n, s2-s1
  deallocate(a)
  deallocate(b)
  end do
end     program main

実行

docker/ubuntu

# gfortran cal_array-al.f90 
# ./a.out
          50   0.0000000000000000     
          75   0.0000000000000000     
         100   1.0000000000000002E-002
         125   9.9999999999999950E-003
         150   1.9999999999999990E-002
         175   3.0000000000000027E-002
         200   4.0000000000000036E-002
         225   5.9999999999999942E-002
         250   8.0000000000000071E-002
         275  0.11000000000000010     
         300  0.13999999999999968     
         325  0.17999999999999972     
         350  0.20999999999999996     
         375  0.26000000000000068     
         400  0.31999999999999940     
         425  0.37999999999999901     
         450  0.46000000000000085     
         475  0.53999999999999915     
         500   2.3099999999999987     
         525   2.6700000000000017     
         550   3.1999999999999993  

killed   

ほっておいたら、いつのまにかkilled。再実行。

# ./a.out
          50   0.0000000000000000     
          75   0.0000000000000000     
         100   1.0000000000000002E-002
         125   2.0000000000000004E-002
         150   1.9999999999999990E-002
         175   3.0000000000000027E-002
         200   4.0000000000000036E-002
         225   6.0000000000000053E-002
         250   7.9999999999999849E-002
         275  0.10999999999999988     
         300  0.12999999999999989     
         325  0.16999999999999993     
         350  0.21999999999999975     
         375  0.26999999999999957     
         400  0.35999999999999943     
         425  0.46999999999999886     
         450  0.46999999999999886     
         475  0.62999999999999901     
         500   2.4800000000000004     
         525   3.1000000000000014     
         550   3.6199999999999974     
Killed

やっぱり、誰かに殺された。
コンパイラのオプションなどを捜索。

docker hub登録

捜索中につき、現状をdocker hubに保存。

macOS

$ docker ps 
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
d557aa91ea17        ubuntu              "/bin/bash"         About an hour ago   Up About an hour                        crazy_tesla

$ docker commit d557aa91ea17 kaizenjapan/fortran
sha256:2a8bd111810c2d7d70b68a160f2f3e7a50d2d2cadeec3d3effce04297d8826a8

$ docker push kaizenjapan/fortran
The push refers to repository [docker.io/kaizenjapan/fortran]
545ccf59ca4d: Pushed 
b079b3fa8d1b: Mounted from library/ubuntu 
a31dbd3063d7: Mounted from library/ubuntu 
c56e09e1bd18: Mounted from library/ubuntu 
543791078bdb: Mounted from library/ubuntu 
latest: digest: sha256:79feba280e0082593b67df42fdd5f17c914e0587ba56d98fabdcccba8acf286e size: 1364

docker hubからfortranを実行するやつを起動する場合は、

$ docker run -it kaizenjapan/fortran /bin/bash

dockerが起動したら

docker/ubuntu

# cd fortran

でいけるはず。

#　ループを短く
1000を475に変更。cal_array-b.f90

# gfortran cal_array-b.f90  
# ./a.out
          50   0.0000000000000000        0.0000000000000000     
          75   1.0000000000000000E-002   0.0000000000000000     
         100   1.9999999999999997E-002   9.9999999999999950E-003
         125   5.0000000000000003E-002   1.0000000000000009E-002
         150   9.0000000000000024E-002   1.9999999999999962E-002
         175  0.21999999999999992        3.0000000000000027E-002
         200  0.38000000000000000        4.0000000000000036E-002
         225  0.57000000000000006        4.9999999999999822E-002
         250  0.81000000000000005        7.9999999999999627E-002
         275   1.2500000000000000        9.9999999999999645E-002
         300   1.6100000000000003       0.12999999999999989     

         325   2.3300000000000001       0.16999999999999993     
         350   3.2699999999999996       0.20999999999999908     
         375   4.1900000000000013       0.25000000000000000     
         400   4.7900000000000027       0.30999999999999872     
         425   6.8300000000000018       0.36999999999999744     
         450   8.3700000000000045       0.44999999999999574     
         475   9.8999999999999986       0.52000000000000313  
# ./a.out
          50   0.0000000000000000        0.0000000000000000     
          75   1.0000000000000000E-002   0.0000000000000000     
         100   1.9999999999999997E-002   9.9999999999999950E-003
         125   5.0000000000000003E-002   9.9999999999999811E-003
         150  0.10999999999999999        2.0000000000000018E-002
         175  0.23999999999999999        2.0000000000000018E-002
         200  0.42000000000000004        4.0000000000000036E-002
         225  0.58000000000000007        5.0000000000000266E-002
         250  0.83000000000000007        8.0000000000000071E-002
         275   1.2500000000000000       0.10000000000000053     
         300   1.6200000000000001       0.12999999999999989     
         325   2.3300000000000001       0.16999999999999993     
         350   3.3000000000000007       0.21000000000000085     
         375   4.0999999999999996       0.25999999999999801     
         400   4.8299999999999983       0.30999999999999872     
         425   6.9000000000000021       0.37999999999999545     
         450   8.3999999999999986       0.45000000000000284     
         475   9.9699999999999989       0.52000000000000313       

help

docker/ubuntu

# gfortran --help
Usage: gfortran [options] file...
Options:
  -pass-exit-codes         Exit with highest error code from a phase.
  --help                   Display this information.
  --target-help            Display target specific command line options.
  --help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
                           Display specific types of command line options.
  (Use '-v --help' to display command line options of sub-processes).
  --version                Display compiler version information.
  -dumpspecs               Display all of the built in spec strings.
  -dumpversion             Display the version of the compiler.
  -dumpmachine             Display the compiler's target processor.
  -print-search-dirs       Display the directories in the compiler's search path.
  -print-libgcc-file-name  Display the name of the compiler's companion library.
  -print-file-name=<lib>   Display the full path to library <lib>.
  -print-prog-name=<prog>  Display the full path to compiler component <prog>.
  -print-multiarch         Display the target's normalized GNU triplet, used as
                           a component in the library path.
  -print-multi-directory   Display the root directory for versions of libgcc.
  -print-multi-lib         Display the mapping between command line options and
                           multiple library search directories.
  -print-multi-os-directory Display the relative path to OS libraries.
  -print-sysroot           Display the target libraries directory.
  -print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
  -Wa,<options>            Pass comma-separated <options> on to the assembler.
  -Wp,<options>            Pass comma-separated <options> on to the preprocessor.
  -Wl,<options>            Pass comma-separated <options> on to the linker.
  -Xassembler <arg>        Pass <arg> on to the assembler.
  -Xpreprocessor <arg>     Pass <arg> on to the preprocessor.
  -Xlinker <arg>           Pass <arg> on to the linker.
  -save-temps              Do not delete intermediate files.
  -save-temps=<arg>        Do not delete intermediate files.
  -no-canonical-prefixes   Do not canonicalize paths when building relative
                           prefixes to other gcc components.
  -pipe                    Use pipes rather than intermediate files.
  -time                    Time the execution of each subprocess.
  -specs=<file>            Override built-in specs with the contents of <file>.
  -std=<standard>          Assume that the input sources are for <standard>.
  --sysroot=<directory>    Use <directory> as the root directory for headers
                           and libraries.
  -B <directory>           Add <directory> to the compiler's search paths.
  -v                       Display the programs invoked by the compiler.
  -###                     Like -v but options quoted and commands not executed.
  -E                       Preprocess only; do not compile, assemble or link.
  -S                       Compile only; do not assemble or link.
  -c                       Compile and assemble, but do not link.
  -o <file>                Place the output into <file>.
  -pie                     Create a position independent executable.
  -shared                  Create a shared library.
  -x <language>            Specify the language of the following input files.
                           Permissible languages include: c c++ assembler none
                           'none' means revert to the default behavior of
                           guessing the language based on the file's extension.

Options starting with -g, -f, -m, -O, -W, or --param are automatically
 passed on to the various sub-processes invoked by gfortran.  In order to pass
 other options on to these processes the -W<letter> options must be used.

For bug reporting instructions, please see:
<file:///usr/share/doc/gcc-7/README.Bugs>.

最適化

ファイルの大きさを比較するため -o で出力ファイル指定。

docker/ubuntu

# gfortran cal_array-b.f90 -o calb
# ./calb
          50   0.0000000000000000        0.0000000000000000     
          75   1.0000000000000000E-002   0.0000000000000000     
         100   1.9999999999999997E-002   9.9999999999999950E-003
         125   5.0000000000000003E-002   9.9999999999999811E-003
         150  0.10999999999999999        2.0000000000000018E-002
         175  0.23999999999999999        2.0000000000000018E-002
         200  0.38000000000000000        4.0000000000000036E-002
         225  0.56999999999999984        4.9999999999999822E-002
         250  0.81000000000000005        8.0000000000000071E-002
         275   1.1900000000000004        9.9999999999999645E-002
         300   1.6200000000000001       0.14000000000000057     
         325   2.3300000000000001       0.16999999999999993     
         350   3.1899999999999995       0.20999999999999908     
         375   4.0800000000000001       0.25000000000000000     
         400   4.8500000000000014       0.32000000000000028     
         425   6.7600000000000016       0.36999999999999744     
         450   8.4900000000000020       0.44999999999999574     
         475   10.130000000000003       0.53999999999999915     

最適化 -O3

docker/ubuntu

# gfortran cal_array-b.f90 -o calbO3 -O3
# ./calbO3
          50   0.0000000000000000        0.0000000000000000     
          75   0.0000000000000000        0.0000000000000000     
         100   1.9999999999999997E-002   0.0000000000000000     
         125   2.0000000000000004E-002   9.9999999999999950E-003
         150   5.9999999999999998E-002   0.0000000000000000     
         175  0.12000000000000000        1.0000000000000009E-002
         200  0.22999999999999998        1.0000000000000009E-002
         225  0.32000000000000006        2.0000000000000018E-002
         250  0.48999999999999999        2.0000000000000018E-002
         275  0.83999999999999986        3.0000000000000249E-002
         300   1.2199999999999998        3.0000000000000249E-002
         325   1.9600000000000000        4.9999999999999822E-002
         350   2.5999999999999996        7.0000000000000284E-002
         375   3.5600000000000005        8.0000000000000071E-002
         400   4.1299999999999990        8.9999999999999858E-002
         425   6.5000000000000000       0.10999999999999943     
         450   8.0800000000000018       0.12999999999999545     
         475   9.2100000000000009       0.14999999999999858   
# ls -al calb*
-rwxr-xr-x 1 root root 13176 Aug  1 02:11 calb
-rwxr-xr-x 1 root root 13112 Aug  1 02:14 calbO3

大きさはあまり違いがない。rightの実行時間は少し速く、leftの実行時間はちょっとは速くかも。

ここまでの記録をdocker hubに再度上げ直しました。

dockerでも1000まで事項させる方法など、なにかありましたら、続きをよろしくお願いします。

docker/ubuntu

# df -k
Filesystem     1K-blocks    Used Available Use% Mounted on
overlay         61255492 2420008  55694160   5% /
tmpfs              65536       0     65536   0% /dev
tmpfs            1023564       0   1023564   0% /sys/fs/cgroup
shm                65536       0     65536   0% /dev/shm
/dev/sda1       61255492 2420008  55694160   5% /etc/hosts
tmpfs            1023564       0   1023564   0% /proc/acpi
tmpfs            1023564       0   1023564   0% /sys/firmware

# free -m
              total        used        free      shared  buff/cache   available
Mem:           1999         106        1671           0         220        1749
Swap:          1023         198         825

docker　メモリ設定

docker containerにもっと多くのメモリを割り当てる方法
https://codeday.me/jp/qa/20190301/341141.html

macOSのdockerアイコンで、Preferenceを選び、Advancedタグを選択すると
メモリ2Gb, Swap1Gbになっていた。

メモリ6Gb, Swap2Gbに変更

1000まで最適化してから実行。

docker/ubuntu

# gfortran cal_array-a.f90 -o cala
# gfortran cal_array-a.f90 -o calaO3 -O3
# la -al cala*
-rwxr-xr-x 1 root root 13176 Aug  1 03:53 cala
-rwxr-xr-x 1 root root 13112 Aug  1 03:53 calaO3
# ./calaO3
          50   0.0000000000000000        0.0000000000000000     
          75   0.0000000000000000        0.0000000000000000     
         100   1.0000000000000002E-002   0.0000000000000000     
         125   1.9999999999999990E-002   1.0000000000000009E-002
         150   5.0000000000000017E-002   0.0000000000000000     
         175  0.10999999999999999        1.0000000000000009E-002
         200  0.19999999999999996        1.0000000000000009E-002
         225  0.30000000000000004        2.0000000000000018E-002
         250  0.45999999999999996        1.9999999999999574E-002
         275  0.80999999999999961        3.0000000000000249E-002
         300   1.0800000000000001        4.0000000000000036E-002
         325   1.7999999999999998        4.9999999999999822E-002
         350   2.5999999999999996        7.0000000000000284E-002
         375   3.2799999999999994        8.0000000000000071E-002
         400   3.9100000000000001        8.9999999999999858E-002
         425   6.1600000000000001       0.10999999999999943     
         450   7.6300000000000026       0.13000000000000256     
         475   8.8299999999999983       0.16000000000000369     
         500   14.489999999999995       0.18000000000000682     
         525   14.009999999999991       0.21000000000000796     
         550   16.019999999999996       0.25000000000000000     
         575   21.170000000000002       0.34000000000000341     
         600   22.659999999999997       0.40999999999999659     
         625   29.090000000000003       0.38999999999998636     
         650   31.830000000000013       0.40000000000000568     
         675   34.540000000000020       0.44999999999998863     
         700   39.010000000000048       0.48999999999995225  
         725   70.210000000000036        1.3100000000000023     
ERRO[1445] error waiting for container: EOF     

プログラムの大きさは、475まででも1000まででも、大きさは同じ。
最適化の有無の差も同じ。

実行させたが、10分待ちきれない。

メモリ12Gb, SWAP4Gbに変更。

docker/ubuntu

# ./calaO3
          50   0.0000000000000000        0.0000000000000000     
          75   0.0000000000000000        0.0000000000000000     
         100   1.9999999999999997E-002   0.0000000000000000     
         125   2.9999999999999999E-002   0.0000000000000000     
         150   4.0000000000000008E-002   9.9999999999999811E-003
         175  0.11000000000000004        9.9999999999999534E-003
         200  0.20999999999999996        1.0000000000000009E-002
         225  0.31000000000000005        2.0000000000000018E-002
         250  0.46999999999999975        3.0000000000000249E-002
         275  0.82000000000000028        2.9999999999999805E-002
         300   1.0300000000000002        4.0000000000000036E-002
         325   1.6299999999999999        4.9999999999999822E-002
         350   2.3900000000000006        6.0000000000000497E-002
         375   3.5100000000000016        6.9999999999998508E-002
         400   4.5299999999999976       0.10999999999999943     
         425   8.2399999999999984       0.10999999999999943     
         450   8.3500000000000014       0.14000000000000057     
         475   10.590000000000003       0.15999999999999659     
         500   11.549999999999997       0.18000000000000682     
         525   14.200000000000003       0.21000000000000796     
         550   18.790000000000006       0.23999999999999488     
         575   20.820000000000007       0.28000000000000114     
         600   23.989999999999981       0.33000000000001251     
         625   26.549999999999983       0.36000000000001364     
         650   32.520000000000010       0.39999999999997726     
         675   46.919999999999987       0.43999999999999773     
         700   48.669999999999959       0.58000000000004093     
         725   67.829999999999984       0.56000000000000227     
         750   71.959999999999980       0.68000000000006366     
         775   83.670000000000073       0.76999999999998181     
         800   83.899999999999977        1.1600000000000819     
         825   99.600000000000023       0.96000000000003638     
         850   108.47000000000003       0.97000000000002728     
         875   133.09000000000003       0.97000000000002728     
         900   151.77999999999997        1.0599999999999454 
ERRO[1842] error waiting for container: EOF   

ここまで25分くらい。待ちきれない。断念。

まとめ

メモリ	実行範囲
2Gb	475
6Gb	725
12Gb	900

16Gbくらいないと1000まで実行できなさそう。

参考資料

Dockerのメモリ制限に関するメモ
https://qiita.com/irotoris/items/944aba5e448a8e723ff6

今日のdocker error
https://qiita.com/kaizen_nagoya/items/63ece3ddab07a4340dfd

docker rm, docker rmiして、macOSを再起動してから、fortran作業始めています。

＜この記事は個人の過去の経験に基づく個人の感想です。現在所属する組織、業務とは関係がありません。＞

データサイエンティストの気づき『勉強だけして仕事に役立てない人。大嫌い』それ自分かもってなった。

文書履歴(document history)

ver. 0.01 初稿 20190801 午前9時
ver. 0.02 半分にしてみた 20190801 午前10時
ver. 0.03 docker hubにひとまず登録 午前11時
ver. 0.04 dockerへのメモリ割り当てを変更 正午
ver. 0.05 標題追記 20191231
ver. 0.06 表題追記 20210327
ver. 0.07 データサイエンティストの気づき『勉強だけして仕事に役立てない人。大嫌い』それ自分かもってなった。追記 20210919

最後までおよみいただきありがとうございました。

いいね　💚、フォローをお願いします。

Thank you very much for reading to the last sentence.

Please press the like icon 💚　and follow me for your happy life.

https://b.hatena.ne.jp/guide/bbutton