目的

深層学習による画像認識の結果をみたときに、**凄すぎる!と思う場合と何故（これが検出できない）?!**と思う場合がある。
前者は、結構、部分勝負で検出しているだけの場合があるが。。。

以下のモデルで、現場力を比較してみる。

Vision Transformer (ViT)
ResNet50

ともに、Imagenet学習済モデル。

	比較の観点
第１ラウンド	重なった？映り込み（想定）画像
第２ラウンド	寄りの画像

第１ラウンド:重なった？映り込み（想定）画像

対象画像

以下の記事で、2つのわかりやすい画像を重ねた画像をつくった。

Attention Is All You Need の『Positional Encoding』の作用の仕組みを探る（まだ、始め）

これを検出させてみる。
これは、現実世界では、ガラスへの映り込みとか、いろいろなケースであり得る画像だと思う。

結果

両方ともダメ。
一応100位とか、200位とかでは検出しているが。。。
ViTでは、bananaが1番にはなっているが、pandaが全くだめ。

予想以上にダメ、がっかり。 (この絵文字じゃないな。。。）

Vision Transformer (ViT)

使ったソース：
https://github.com/lukemelas/PyTorch-Pretrained-ViT

モデル：B_16_imagenet1k
　　※大きいほうでないもの。大きいほうのモデルだと頑張るんだろうか？

[954] banana ★★                                                                     (62.86%)
[955] jackfruit, jak, jack                                                        (24.47%)
[956] custard apple                                                               (1.86%)
[64] green mamba                                                                 (1.15%)
[953] pineapple, ananas                                                           (0.64%)
[951] lemon                                                                       (0.43%)
[55] green snake, grass snake                                                    (0.27%)
[950] orange                                                                      (0.22%)
[114] slug                                                                        (0.20%)
[59] vine snake                                                                  (0.19%)
[63] Indian cobra, Naja naja                                                     (0.18%)
[986] yellow lady's slipper, yellow lady-slipper, Cypripedium calceolus, Cypripedium parviflorum (0.15%)
[952] fig                                                                         (0.15%)
[47] African chameleon, Chamaeleo chamaeleon                                     (0.14%)
[113] snail                                                                       (0.14%)
[957] pomegranate                                                                 (0.14%)
[66] horned viper, cerastes, sand viper, horned asp, Cerastes cornutus           (0.11%)
[31] tree frog, tree-frog                                                        (0.10%)
[599] honeycomb                                                                   (0.10%)
[96] toucan                                                                      (0.09%)
[93] hornbill                                                                    (0.08%)
[944] artichoke, globe artichoke                                                  (0.07%)
[990] buckeye, horse chestnut, conker                                             (0.07%)
[738] pot, flowerpot                                                              (0.07%)
[940] spaghetti squash                                                            (0.07%)
[38] banded gecko                                                                (0.06%)
[948] Granny Smith                                                                (0.06%)
[997] bolete                                                                      (0.06%)
[995] earthstar                                                                   (0.06%)
[61] boa constrictor, Constrictor constrictor                                    (0.06%)
[949] strawberry                                                                  (0.06%)
[364] three-toed sloth, ai, Bradypus tridactylus                                  (0.06%)
[380] titi, titi monkey                                                           (0.05%)
[68] sidewinder, horned rattlesnake, Crotalus cerastes                           (0.05%)
[701] parachute, chute                                                            (0.05%)
[382] squirrel monkey, Saimiri sciureus                                           (0.05%)
[60] night snake, Hypsiglena torquata                                            (0.04%)
[994] stinkhorn, carrion fungus                                                   (0.04%)
[62] rock python, rock snake, Python sebae                                       (0.04%)
[989] hip, rose hip, rosehip                                                      (0.04%)
[384] indri, indris, Indri indri, Indri brevicaudatus                             (0.04%)
[947] mushroom                                                                    (0.04%)
[988] acorn                                                                       (0.04%)
[133] bittern                                                                     (0.03%)
[126] isopod                                                                      (0.03%)
[998] ear, spike, capitulum                                                       (0.03%)
[46] green lizard, Lacerta viridis                                               (0.03%)
[54] hognose snake, puff adder, sand viper                                       (0.03%)
[110] flatworm, platyhelminth                                                     (0.03%)
[82] ruffed grouse, partridge, Bonasa umbellus                                   (0.03%)
[410] apiary, bee house                                                           (0.03%)
[112] conch                                                                       (0.03%)
[356] weasel                                                                      (0.03%)
[387] lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens         (0.03%)
[95] jacamar                                                                     (0.03%)
[378] capuchin, ringtail, Cebus capucinus                                         (0.02%)
[993] gyromitra                                                                   (0.02%)
[117] chambered nautilus, pearly nautilus, nautilus                               (0.02%)
[306] rhinoceros beetle                                                           (0.02%)
[404] airliner                                                                    (0.02%)
[116] chiton, coat-of-mail shell, sea cradle, polyplacophore                      (0.02%)
[924] guacamole                                                                   (0.02%)
[43] frilled lizard, Chlamydosaurus kingi                                        (0.02%)
[5] electric ray, crampfish, numbfish, torpedo                                  (0.02%)
[105] koala, koala bear, kangaroo bear, native bear, Phascolarctos cinereus       (0.02%)
[48] Komodo dragon, Komodo lizard, dragon lizard, giant lizard, Varanus komodoensis (0.02%)
[299] meerkat, mierkat                                                            (0.02%)
[52] thunder snake, worm snake, Carphophis amoenus                               (0.02%)
[83] prairie chicken, prairie grouse, prairie fowl                               (0.02%)
[669] mosquito net                                                                (0.02%)
[32] tailed frog, bell toad, ribbed toad, tailed toad, Ascaphus trui             (0.02%)
[16] bulbul                                                                      (0.02%)
[319] dragonfly, darning needle, devil's darning needle, sewing needle, snake feeder, snake doctor, mosquito hawk, skeeter hawk (0.02%)
[417] balloon                                                                     (0.02%)
[506] coil, spiral, volute, whorl, helix                                          (0.02%)
[325] sulphur butterfly, sulfur butterfly                                         (0.02%)
[86] partridge                                                                   (0.02%)
[936] head cabbage                                                                (0.02%)
[941] acorn squash                                                                (0.02%)
[938] cauliflower                                                                 (0.02%)
[45] Gila monster, Heloderma suspectum                                           (0.02%)
[987] corn                                                                        (0.02%)
[996] hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa   (0.02%)
[390] eel                                                                         (0.02%)
[81] ptarmigan                                                                   (0.02%)
[40] American chameleon, anole, Anolis carolinensis                              (0.02%)
[326] lycaenid, lycaenid butterfly                                                (0.02%)
[331] hare                                                                        (0.02%)
[88] macaw                                                                       (0.02%)
[377] marmoset                                                                    (0.02%)
[843] swing                                                                       (0.02%)
[73] barn spider, Araneus cavaticus                                              (0.02%)
[130] flamingo                                                                    (0.02%)
[363] armadillo                                                                   (0.02%)
[70] harvestman, daddy longlegs, Phalangium opilio                               (0.02%)
[53] ringneck snake, ring-necked snake, ring snake                               (0.02%)
[295] American black bear, black bear, Ursus americanus, Euarctos americanus      (0.02%)
[972] cliff, drop, drop-off                                                       (0.02%)
[21] kite                                                                        (0.02%)
[392] rock beauty, Holocanthus tricolor                                           (0.02%)
[26] common newt, Triturus vulgaris                                              (0.02%)
[335] fox squirrel, eastern fox squirrel, Sciurus niger                           (0.02%)
[67] diamondback, diamondback rattlesnake, Crotalus adamanteus                   (0.02%)
[588] hamper                                                                      (0.02%)
[908] wing                                                                        (0.02%)
[367] chimpanzee, chimp, Pan troglodytes                                          (0.02%)
[405] airship, dirigible                                                          (0.01%)
[298] mongoose                                                                    (0.01%)
[728] plastic bag                                                                 (0.01%)
[65] sea snake                                                                   (0.01%)
[51] triceratops                                                                 (0.01%)
[101] tusker                                                                      (0.01%)
[374] langur                                                                      (0.01%)
[76] tarantula                                                                   (0.01%)
[991] coral fungus                                                                (0.01%)
[365] orangutan, orang, orangutang, Pongo pygmaeus                                (0.01%)
[388] giant panda ★★, panda, panda bear, coon bear, Ailuropoda melanoleuca           (0.01%)
[943] cucumber, cuke                                                              (0.01%)
[30] bullfrog, Rana catesbeiana                                                  (0.01%)
[109] brain coral                                                                 (0.01%)
[80] black grouse                                                                (0.01%)
[946] cardoon                                                                     (0.01%)
[322] ringlet, ringlet butterfly                                                  (0.01%)
[315] mantis, mantid                                                              (0.01%)
[316] cicada, cicala                                                              (0.01%)
[376] proboscis monkey, Nasalis larvatus                                          (0.01%)
[385] Indian elephant, Elephas maximus                                            (0.01%)
[168] redbone                                                                     (0.01%)
[92] bee eater                                                                   (0.01%)
[379] howler monkey, howler                                                       (0.01%)
[90] lorikeet                                                                    (0.01%)
[383] Madagascar cat, ring-tailed lemur, Lemur catta                              (0.01%)
[607] jack-o'-lantern

ResNet50

使ったソース：https://keras.io/ja/applications/
モデル:ResNet50(weights='imagenet')

[('n03188531', 'diaper', 0.06685701),
 ('n04209133', 'shower_cap', 0.06336665),
 ('n01748264', 'Indian_cobra', 0.053833533),
 ('n03958227', 'plastic_bag', 0.0477978),
 ('n07880968', 'burrito', 0.0431565),
 ('n04033995', 'quilt', 0.03513578),
 ('n07718747', 'artichoke', 0.02602887),
 ('n04235860', 'sleeping_bag', 0.025143128),
 ('n07860988', 'dough', 0.024247095),
 ('n01667778', 'terrapin', 0.022566179),
 ('n01664065', 'loggerhead', 0.021968575),
 ('n07714571', 'head_cabbage', 0.021425359),
 ('n02074367', 'dugong', 0.020652529),
 ('n01440764', 'tench', 0.019023761),
 ('n01698640', 'American_alligator', 0.016179482),
 ('n03888257', 'parachute', 0.013034657),
 ('n01677366', 'common_iguana', 0.012845791),
 ('n01749939', 'green_mamba', 0.012838124),
 ('n01944390', 'snail', 0.010219791),
 ('n01669191', 'box_turtle', 0.009841825),
 ('n01695060', 'Komodo_dragon', 0.00868533),
 ('n01667114', 'mud_turtle', 0.008071985),
 ('n01737021', 'water_snake', 0.0073151086),
 ('n03627232', 'knot', 0.006370477),
 ('n07697313', 'cheeseburger', 0.0062961983),
 ('n03871628', 'packet', 0.0060379985),
 ('n03485794', 'handkerchief', 0.006006749),
 ('n04367480', 'swab', 0.00592959),
 ('n01697457', 'African_crocodile', 0.005014863),
 ('n03868863', 'oxygen_mask', 0.004961956),
 ('n02110958', 'pug', 0.0048995228),
 ('n04399382', 'teddy', 0.004425834),
 ('n04133789', 'sandal', 0.004231881),
 ('n03938244', 'pillow', 0.0041891537),
 ('n01768244', 'trilobite', 0.004171828),
 ('n01986214', 'hermit_crab', 0.0040465007),
 ('n02869837', 'bonnet', 0.003970811),
 ('n02909870', 'bucket', 0.003894355),
 ('n01883070', 'wombat', 0.0038887928),
 ('n07684084', 'French_loaf', 0.003779268),
 ('n07695742', 'pretzel', 0.0037406099),
 ('n07579787', 'plate', 0.00372636),
 ('n07753592', 'banana'　★★, 0.0037171696),
 ('n01753488', 'horned_viper', 0.0037020848),
 ('n07717410', 'acorn_squash', 0.0036752466),
 ('n01755581', 'diamondback', 0.0035176787),
 ('n01704323', 'triceratops', 0.0033780576),
 ('n01955084', 'chiton', 0.0032795744),
 ('n03786901', 'mortar', 0.0032332388),
 ('n07697537', 'hotdog', 0.003205056),
 ('n02134084', 'ice_bear', 0.0031007996),
 ('n02398521', 'hippopotamus', 0.0030010697),
 ('n09468604', 'valley', 0.0028716042),
 ('n01744401', 'rock_python', 0.0028035499),
 ('n02526121', 'eel', 0.0027678132),
 ('n07760859', 'custard_apple', 0.002687465),
 ('n01692333', 'Gila_monster', 0.0026803724),
 ('n04458633', 'totem_pole', 0.0025927152),
 ('n02129604', 'tiger', 0.0025757342),
 ('n02093428', 'American_Staffordshire_terrier', 0.0024840261),
 ('n01943899', 'conch', 0.0024245891),
 ('n02795169', 'barrel', 0.0023302436),
 ('n02480495', 'orangutan', 0.0022904908),
 ('n02093256', 'Staffordshire_bullterrier', 0.0022238945),
 ('n03788365', 'mosquito_net', 0.0020587973),
 ('n03887697', 'paper_towel', 0.0019693126),
 ('n02108089', 'boxer', 0.001964624),
 ('n04493381', 'tub', 0.0018422267),
 ('n02640242', 'sturgeon', 0.0018405094),
 ('n02808440', 'bathtub', 0.0018323634),
 ('n01729977', 'green_snake', 0.0017895664),
 ('n09332890', 'lakeside', 0.0017504566),
 ('n01675722', 'banded_gecko', 0.0017276495),
 ('n03929855', 'pickelhaube', 0.0017105255),
 ('n02443114', 'polecat', 0.0016949566),
 ('n01751748', 'sea_snake', 0.0016778206),
 ('n02085936', 'Maltese_dog', 0.0016648371),
 ('n02099601', 'golden_retriever', 0.0016648251),
 ('n12144580', 'corn', 0.0016592229),
 ('n01629819', 'European_fire_salamander', 0.0016194135),
 ('n03935335', 'piggy_bank', 0.00161333),
 ('n04259630', 'sombrero', 0.0016044596),
 ('n03014705', 'chest', 0.0016021407),
 ('n13133613', 'ear', 0.001600591),
 ('n02837789', 'bikini', 0.001596331),
 ('n09256479', 'coral_reef', 0.0015835938),
 ('n07584110', 'consomme', 0.0015825942),
 ('n02504458', 'African_elephant', 0.0015425562),
 ('n01491361', 'tiger_shark', 0.0015347875),
 ('n03792972', 'mountain_tent', 0.0015005224),
 ('n03991062', 'pot', 0.0014792152),
 ('n01740131', 'night_snake', 0.0014612696),
 ('n01819313', 'sulphur-crested_cockatoo', 0.0014487966),
 ('n02095570', 'Lakeland_terrier', 0.0014417097),
 ('n03065424', 'coil', 0.0014214374),
 ('n02090379', 'redbone', 0.001331581),
 ('n04251144', 'snorkel', 0.0013039334),
 ('n02107312', 'miniature_pinscher', 0.0012931975),
 ('n04522168', 'vase', 0.001289689),
 ('n01990800', 'isopod', 0.0012486882),
 ('n02098413', 'Lhasa', 0.0012427429),
 ('n01983481', 'American_lobster', 0.0012367677),
 ('n01665541', 'leatherback_turtle', 0.0012171009),
 ('n01978287', 'Dungeness_crab', 0.0012116744),
 ('n02088466', 'bloodhound', 0.0012091378),
 ('n02444819', 'otter', 0.0012075476),
 ('n02786058', 'Band_Aid', 0.0011942227),
 ('n01945685', 'slug', 0.0011502523),
 ('n04162706', 'seat_belt', 0.001143836),
 ('n07715103', 'cauliflower', 0.0011376633),
 ('n04336792', 'stretcher', 0.0011328654),
 ('n01689811', 'alligator_lizard', 0.0010975883),
 ('n04398044', 'teapot', 0.0010951292),
 ('n01917289', 'brain_coral', 0.0010893045),
 ('n03724870', 'mask', 0.0010842477),
 ('n02363005', 'beaver', 0.0010828231),
 ('n04371774', 'swing', 0.0010819857),
 ('n02493509', 'titi', 0.0010802051),
 ('n02106550', 'Rottweiler', 0.0010796139),
 ('n03710637', 'maillot', 0.0010739941),
 ('n07930864', 'cup', 0.0010585586),
 ('n01641577', 'bullfrog', 0.0010573524),
 ('n03623198', 'knee_pad', 0.0010541431),
 ('n01644900', 'tailed_frog', 0.001051543),
 ('n04560804', 'water_jug', 0.0010370798),
 ('n07754684', 'jackfruit', 0.00102939),
 ('n07614500', 'ice_cream', 0.0009983778),
 ('n02319095', 'sea_urchin', 0.000994882),
 ('n02106382', 'Bouvier_des_Flandres', 0.0009813908),
 ('n03496892', 'harvester', 0.0009761827),
 ('n02747177', 'ashcan', 0.00097544893),
 ('n03388043', 'fountain', 0.00097304664),
 ('n02097047', 'miniature_schnauzer', 0.0009551877),
 ('n04229816', 'ski_mask', 0.00093830726),
 ('n02102480', 'Sussex_spaniel', 0.00093589386),
 ('n02514041', 'barracouta', 0.00093452935),
 ('n02808304', 'bath_towel', 0.0009257423),
 ('n02807133', 'bathing_cap', 0.00092556357),
 ('n01685808', 'whiptail', 0.0009166785),
 ('n02097658', 'silky_terrier', 0.0009142907),
 ('n04613696', 'yurt', 0.0009025597),
 ('n02951358', 'canoe', 0.0009017497),
 ('n01914609', 'sea_anemone', 0.0008968752),
 ('n04201297', 'shoji', 0.00088736176),
 ('n04090263', 'rifle', 0.00088089885),
 ('n01756291', 'sidewinder', 0.0008525167),
 ('n03992509', "potter's_wheel", 0.00085035834),
 ('n01688243', 'frilled_lizard', 0.0008264672),
 ('n02096585', 'Boston_bull', 0.0008254309),
 ('n02098286', 'West_Highland_white_terrier', 0.0008137851),
 ('n02782093', 'balloon', 0.0008128203),
 ('n07753275', 'pineapple', 0.000812721),
 ('n03590841', "jack-o'-lantern", 0.00081251684),
 ('n01984695', 'spiny_lobster', 0.00080373336),
 ('n03355925', 'flagpole', 0.0007974661),
 ('n02481823', 'chimpanzee', 0.0007972197),
 ('n01818515', 'macaw', 0.0007904845),
 ('n01980166', 'fiddler_crab', 0.00077911804),
 ('n07693725', 'bagel', 0.00077835674),
 ('n07749582', 'lemon', 0.000773825),
 ('n03127747', 'crash_helmet', 0.0007644384),
 ('n07742313', 'Granny_Smith', 0.00075247313),
 ('n04525038', 'velvet', 0.00074524723),
 ('n03873416', 'paddle', 0.0007378166),
 ('n02494079', 'squirrel_monkey', 0.0007339725),
 ('n02107574', 'Greater_Swiss_Mountain_dog', 0.00073071686),
 ('n01873310', 'platypus', 0.0007291971),
 ('n09421951', 'sandbar', 0.00072215486),
 ('n02087394', 'Rhodesian_ridgeback', 0.00072002097),
 ('n02112137', 'chow', 0.0007164359),
 ('n04540053', 'volleyball', 0.00069893955),
 ('n02804414', 'bassinet', 0.000693986),
 ('n03637318', 'lampshade', 0.0006926213),
 ('n02504013', 'Indian_elephant', 0.0006885995),
 ('n01742172', 'boa_constrictor', 0.0006875997),
 ('n02108915', 'French_bulldog', 0.00068659685),
 ('n04355933', 'sunglass', 0.0006827656),
 ('n02088238', 'basset', 0.00068103475),
 ('n15075141', 'toilet_tissue', 0.00068031746),
 ('n09246464', 'cliff', 0.00067776826),
 ('n02606052', 'rock_beauty', 0.0006772798),
 ('n02281406', 'sulphur_butterfly', 0.00067677),
 ('n13054560', 'bolete', 0.0006723335),
 ('n04326547', 'stone_wall', 0.0006707245),
 ('n01735189', 'garter_snake', 0.00066899415),
 ('n02105162', 'malinois', 0.0006637893),
 ('n03998194', 'prayer_rug', 0.00066320336),
 ('n01496331', 'electric_ray', 0.00066319166),
 ('n07248320', 'book_jacket', 0.0006618011),
 ('n01981276', 'king_crab', 0.00066063984),
 ('n01494475', 'hammerhead', 0.0006595979),
 ('n02085620', 'Chihuahua', 0.0006546478),
 ('n03916031', 'perfume', 0.0006438952),
 ('n02097474', 'Tibetan_terrier', 0.0006358973),
 ('n02099849', 'Chesapeake_Bay_retriever', 0.00062768563),
 ('n04462240', 'toyshop', 0.00062643486),
 ('n02454379', 'armadillo', 0.00062574615),
 ('n03825788', 'nipple', 0.00062463415),
 ('n02096177', 'cairn', 0.0006217132),
 ('n04557648', 'water_bottle', 0.0006131043),
 ('n01847000', 'drake', 0.00060888304),
 ('n04532106', 'vestment', 0.00060648005),
 ('n02492660', 'howler_monkey', 0.00059765484),
 ('n02361337', 'marmot', 0.0005971751),
 ('n02510455', 'giant_panda'　★★, 0.0005930799),
 ('n02104365', 'schipperke', 0.00059168396),
 ('n02132136', 'brown_bear', 0.00058264274),
 ('n02088364', 'beagle', 0.00057981495),
 ('n03131574', 'crib', 0.00056277745),
 ('n02493793', 'spider_monkey', 0.000553812),
 ('n01739381', 'vine_snake', 0.00055204815),
 ('n02910353', 'buckle', 0.00055191625)]

第２ラウンド:寄りの画像

対象画像

以下の記事で、使った「寄り」の画像。

全22行のコードでGradCAM。tf_explainは、使い易いかも、お薦め！

結果

両方とも、まあまあ。
ViTでは、tabby, tabby catが1番にはなっているので、いい？

Vision Transformer (ViT)

使ったソース：
https://github.com/lukemelas/PyTorch-Pretrained-ViT

モデル：B_16_imagenet1k

[281] tabby, tabby cat(31.92%)
[285] Egyptian cat(29.05%)
[282] tiger cat(28.46%)
[287] lynx, catamount (2.49%)
[283] Persian cat (1.06%)
[284] Siamese cat, Siamese  (0.47%)
[478] carton  (0.13%)
[728] plastic bag  (0.12%)
[678] neck brace  (0.09%)
[903] wig (0.08%)
[292] tiger, Panthera tigris  (0.08%)
[722] ping-pong ball   (0.07%)
[332] Angora, Angora rabbit (0.06%)
[620] laptop, laptop computer(0.06%)
[700] paper towel    (0.06%)

ResNet50

[('n02124075', 'Egyptian_cat', 0.5787574),
 ('n02123159', 'tiger_cat', 0.32389614),
 ('n02123045', 'tabby', 0.06504008),
 ('n02127052', 'lynx', 0.020585762),
 ('n02123394', 'Persian_cat', 0.0019484512),
 ('n04589890', 'window_screen', 0.001253177),
 ('n03958227', 'plastic_bag', 0.0010556503),
 ('n02123597', 'Siamese_cat', 0.0009966693),
 ('n02129604', 'tiger', 0.00061496574),
 ('n04235860', 'sleeping_bag', 0.00025958236),
 ('n07714990', 'broccoli', 0.00025066984),
 ('n04023962', 'punching_bag', 0.00023452094),
 ('n02971356', 'carton', 0.00023311013),
 ('n03724870', 'mask', 0.00018257847),
 ('n02808304', 'bath_towel', 0.00015606672),

まとめ

特にありません。
このレベルだと、これらの検出器は、部品という感じですね。前後の処理がだいぶないと、現場力がない。
（augmentationとかも、なんだかな。。。という気持ちになる。）
こういうのに強いネットワークとかモデルも、既に、どこかにあるんでしょうね。