前回

前回の下調べ結果：
ByteTrackのtools/track.pyにimport motmetricsの記述有．
やはり公式っぽいライブラリpy-motmetricsを使っていた．
MOT評価指標については，以下にも使い方が書いてある．
マイフレームの結果をMOTAccumulatorに蓄積して評価するようだ．
真値GTと推定値の1-IoUをスコアとして，bounding boxがかぶらない物体は0点のようだ．

今回の概要

tools/track.pyを動かしてMOT評価する
デバッガを使ってデータ構造を把握する

`tools/track.py`を動かしてMOT評価する

README.mdのTracking>Evaluation on MOT17 half valを実行してみると，jsonが無いと言われる．
README.mdを上から辿ると，Data preparationのconvert_mot17_to_coco.pyで作られるようだ．
COCOデータフォーマットというのがAI分野のアルゴリズム評価検証用のアノテーションデータフォーマットで有名らしい．(理解には負荷かかりそう...)

以下のようにmot配下にMOT17データセットをunzip・配置する．

$ python3 tools/convert_mot17_to_coco.py
MOT17-02-FRCNN: 600 images
600 ann images
42 73
:
MOT17-13-FRCNN: 750 images
750 ann images
359 170
loaded train_half for 2664 images and 58407 samples
MOT17-02-FRCNN: 600 images
600 ann images
53 83
:
MOT17-13-FRCNN: 750 images
750 ann images
339 165
loaded val_half for 2652 images and 53890 samples
MOT17-02-FRCNN: 600 images
600 ann images
62 83
:
MOT17-13-FRCNN: 750 images
750 ann images
546 170
loaded train for 5316 images and 112297 samples
MOT17-01-DPM: 450 images
0 -1
:
MOT17-14-SDP: 750 images
0 -1
loaded test for 17757 images and 0 samples

mot/annotation配下にjsonファイルが作成される．
(testの数値が0 -1となるのが気になるが問題なさそう)

FileNotFoundError: [Errno 2] No such file or directory: '/workspace/ByteTrack/datasets/mot/annotations/val_half.json'

workspacesや開いているディレクトリの位置が違ってパスが合わない問題のようだ
↓
VSCodeでDevContainer: Open Folders in Containerをしたときに，
なぜか/workspace/ByteTrackではなく，/workspaces/ByteTrackに変わっていて，
これまでの変更履歴の一切ない同じファイルが詰まっていた...
しかもlocateコマンドでファイル探しても見つからない. カオス．
実行時に動くのは前者なので，変更履歴を合わせてもう一度動かす．

➡動作した！

user@a9c39c9e5784:/workspace/ByteTrack$  cd /workspace/ByteTrack ; /usr/bin/env /bin/python3.8 /home/user/.vscode-server/extensions/ms-python.debugpy-2024.4.0/bundled/libs/debugpy/adapter/../../debugpy/launcher 52907 -- /workspace/ByteTrack/tools/track.py -f exps/example/mot/yolox_x_ablation.py -c pretrained/bytetrack_ablation.pth.tar -b 1 -d 1 --fp16 --fuse 
2024-05-02 06:22:00 | INFO     | __main__:154 - Args: Namespace(batch_size=1, ckpt='pretrained/bytetrack_ablation.pth.tar', conf=0.01, devices=1, dist_backend='nccl', dist_url=None, exp_file='exps/example/mot/yolox_x_ablation.py', experiment_name='yolox_x_ablation', fp16=True, fuse=True, local_rank=0, machine_rank=0, match_thresh=0.9, min_box_area=100, mot20=False, name=None, nms=0.7, num_machines=1, opts=[], seed=None, speed=False, test=False, track_buffer=30, track_thresh=0.6, trt=False, tsize=None)
2024-05-02 06:22:01 | INFO     | __main__:164 - Model Summary: Params: 99.00M, Gflops: 793.21
2024-05-02 06:22:01 | INFO     | yolox.data.datasets.mot:39 - loading annotations into memory...
2024-05-02 06:22:02 | INFO     | yolox.data.datasets.mot:39 - Done (t=0.22s)
2024-05-02 06:22:02 | INFO     | pycocotools.coco:88 - creating index...
2024-05-02 06:22:02 | INFO     | pycocotools.coco:88 - index created!
2024-05-02 06:22:22 | INFO     | __main__:186 - loading checkpoint
2024-05-02 06:22:24 | INFO     | __main__:191 - loaded checkpoint done.
2024-05-02 06:22:24 | INFO     | __main__:197 -         Fusing model...
/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:561: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more information.
  if param.grad is not None:
 11%|#1        | 299/2652 [00:30<03:24, 11.53it/s]2024-05-02 06:22:55 | INFO     | yolox.evaluators.mot_evaluator:39 - save results to ./YOLOX_outputs/yolox_x_ablation/track_results/MOT17-02-FRCNN.txt
 31%|###       | 822/2652 [01:18<02:56, 10.39it/s]2024-05-02 06:23:44 | INFO     | yolox.evaluators.mot_evaluator:39 - save results to ./YOLOX_outputs/yolox_x_ablation/track_results/MOT17-04-FRCNN.txt
 47%|####6     | 1240/2652 [01:56<02:03, 11.41it/s]2024-05-02 06:24:21 | INFO     | yolox.evaluators.mot_evaluator:39 - save results to ./YOLOX_outputs/yolox_x_ablation/track_results/MOT17-05-FRCNN.txt
 57%|#####6    | 1502/2652 [02:19<01:39, 11.60it/s]2024-05-02 06:24:44 | INFO     | yolox.evaluators.mot_evaluator:39 - save results to ./YOLOX_outputs/yolox_x_ablation/track_results/MOT17-09-FRCNN.txt
 69%|######8   | 1828/2652 [02:48<01:11, 11.52it/s]2024-05-02 06:25:13 | INFO     | yolox.evaluators.mot_evaluator:39 - save results to ./YOLOX_outputs/yolox_x_ablation/track_results/MOT17-10-FRCNN.txt
 86%|########5 | 2278/2652 [03:27<00:32, 11.48it/s]2024-05-02 06:25:52 | INFO     | yolox.evaluators.mot_evaluator:39 - save results to ./YOLOX_outputs/yolox_x_ablation/track_results/MOT17-11-FRCNN.txt
100%|##########| 2652/2652 [03:59<00:00, 10.94it/s]2024-05-02 06:26:25 | INFO     | yolox.evaluators.mot_evaluator:39 - save results to ./YOLOX_outputs/yolox_x_ablation/track_results/MOT17-13-FRCNN.txt
100%|##########| 2652/2652 [04:00<00:00, 11.02it/s]
2024-05-02 06:26:25 | INFO     | yolox.evaluators.mot_evaluator:628 - Evaluate in main process...
2024-05-02 06:26:34 | INFO     | yolox.evaluators.mot_evaluator:657 - Loading and preparing results...
2024-05-02 06:26:34 | INFO     | yolox.evaluators.mot_evaluator:657 - DONE (t=0.39s)
2024-05-02 06:26:34 | INFO     | pycocotools.coco:363 - creating index...
2024-05-02 06:26:34 | INFO     | pycocotools.coco:363 - index created!
Running per image evaluation...
Evaluate annotation type *bbox*
COCOeval_opt.evaluate() finished in 1.10 seconds.
Accumulating evaluation results...
COCOeval_opt.accumulate() finished in 0.13 seconds.
gt_type _val_half
2024-05-02 06:26:35 | INFO     | __main__:218 - 
Average forward time: 81.14 ms, Average track time: 4.83 ms, Average inference time: 85.97 ms
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.624
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.893
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.707
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.219
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.542
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.729
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.043
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.331
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.671
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.303
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.610
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.767

gt_files ['datasets/mot/train/MOT17-11-FRCNN/gt/gt_val_half.txt', 'datasets/mot/train/MOT17-09-FRCNN/gt/gt_val_half.txt', 'datasets/mot/train/MOT17-04-FRCNN/gt/gt_val_half.txt', 'datasets/mot/train/MOT17-02-FRCNN/gt/gt_val_half.txt', 'datasets/mot/train/MOT17-13-FRCNN/gt/gt_val_half.txt', 'datasets/mot/train/MOT17-05-FRCNN/gt/gt_val_half.txt', 'datasets/mot/train/MOT17-10-FRCNN/gt/gt_val_half.txt']
2024-05-02 06:26:35 | INFO     | __main__:235 - Found 7 groundtruths and 7 test files.
2024-05-02 06:26:35 | INFO     | __main__:236 - Available LAP solvers ['lap', 'scipy']
2024-05-02 06:26:35 | INFO     | __main__:237 - Default LAP solver 'lap'
2024-05-02 06:26:35 | INFO     | __main__:238 - Loading files.
2024-05-02 06:26:37 | INFO     | __main__:118 - Comparing MOT17-05-FRCNN...
2024-05-02 06:26:37 | INFO     | __main__:118 - Comparing MOT17-04-FRCNN...
2024-05-02 06:26:38 | INFO     | __main__:118 - Comparing MOT17-02-FRCNN...
2024-05-02 06:26:39 | INFO     | __main__:118 - Comparing MOT17-10-FRCNN...
2024-05-02 06:26:39 | INFO     | __main__:118 - Comparing MOT17-13-FRCNN...
2024-05-02 06:26:39 | INFO     | __main__:118 - Comparing MOT17-11-FRCNN...
2024-05-02 06:26:40 | INFO     | __main__:118 - Comparing MOT17-09-FRCNN...
2024-05-02 06:26:40 | INFO     | __main__:246 - Running metrics
                Rcll  Prcn  GT    MT    PT    ML    FP    FN  IDs   FM  MOTA  MOTP num_objects
MOT17-05-FRCNN 81.8% 95.0%  71 52.1% 36.6% 11.3%  4.3% 18.2% 0.8% 1.5% 76.6% 0.180        3357
MOT17-04-FRCNN 93.3% 94.4%  69 87.0% 10.1%  2.9%  5.5%  6.7% 0.1% 0.4% 87.7% 0.128       24178
MOT17-02-FRCNN 64.4% 86.2%  53 34.0% 49.1% 17.0% 10.4% 35.6% 0.5% 1.8% 53.5% 0.195        9880
MOT17-10-FRCNN 75.6% 93.7%  36 44.4% 50.0%  5.6%  5.1% 24.4% 0.4% 1.5% 70.1% 0.219        5923
MOT17-13-FRCNN 80.2% 97.3%  44 65.9% 18.2% 15.9%  2.2% 19.8% 0.4% 0.5% 77.6% 0.197        3156
MOT17-11-FRCNN 81.4% 87.8%  44 50.0% 25.0% 25.0% 11.3% 18.6% 0.2% 0.6% 69.8% 0.128        4517
MOT17-09-FRCNN 83.9% 99.6%  22 72.7% 22.7%  4.5%  0.3% 16.1% 0.4% 0.9% 83.2% 0.152        2879
OVERALL        83.1% 93.0% 339 58.4% 29.8% 11.8%  6.3% 16.9% 0.3% 0.9% 76.5% 0.155       53890
                IDF1   IDP   IDR  Rcll  Prcn  GT  MT  PT ML   FP   FN IDs   FM  MOTA  MOTP IDt IDa IDm num_objects
MOT17-05-FRCNN 76.2% 82.3% 70.9% 81.8% 95.0%  71  37  26  8  145  612  28   49 76.6% 0.180  26  11  13        3357
MOT17-04-FRCNN 90.8% 91.4% 90.3% 93.3% 94.4%  69  60   7  2 1332 1615  22   92 87.7% 0.128  11  11   2       24178
MOT17-02-FRCNN 58.4% 68.3% 51.0% 64.4% 86.2%  53  18  26  9 1023 3516  54  174 53.5% 0.195  37  17   5        9880
MOT17-10-FRCNN 68.6% 76.8% 62.0% 75.6% 93.7%  36  16  18  2  303 1445  25   87 70.1% 0.219  17  14   6        5923
MOT17-13-FRCNN 85.5% 94.7% 77.9% 80.2% 97.3%  44  29   8  7   69  626  12   16 77.6% 0.197  14   1   4        3156
MOT17-11-FRCNN 70.8% 73.6% 68.2% 81.4% 87.8%  44  22  11 11  512  842  11   28 69.8% 0.128   4   9   2        4517
MOT17-09-FRCNN 77.9% 85.2% 71.7% 83.9% 99.6%  22  16   5  1   10  464  11   25 83.2% 0.152  10   4   4        2879
OVERALL        79.4% 84.2% 75.2% 83.1% 93.0% 339 198 101 40 3394 9120 163  471 76.5% 0.155 119  67  36       53890
2024-05-02 06:26:42 | INFO     | __main__:271 - Completed
user@a9c39c9e5784:/workspace/ByteTrack$

デバッガを使ってデータ構造を把握する

デバッガを実行して挙動を確認することができた.

以下，雑だが概観．

トラッキングはevaluator.evaluateの中で実行．BYTETrackerが全てのSTrackクラスの追跡オブジェクトをリスト管理している
評価は真値にmot/train/*/gt/gt.txtを使用しており，motmetricsのパッケージを活用しているが理解に時間がかかりそう

個人の所感

個人でシステム開発経験が無かったため，昨年度の開発ではシステム設計・実装の気持ちが未だわかっていなくて，関係者に酷く迷惑をかけてしまっていた．
もう遅いけれど，基本的な設計理念を把握できて良かった．
ByteTrackは評価に組み込み内ポストプロセスのmotmetrics-pyを使用しており，導入負荷が高そうなことと，実装時のメモリ蓄積に課題を感じた．
一方で，DeepSORTでは評価に別コードのTrackEvalを使用しており，楽そうな上に，公式っぽい(?)．
また，ByteTrackよりもDeepSORTの方が構造が全体的に簡素であることがわかったので，今後はDeepSORTに切り替える．