「スケーリング法則が何故スケーラブルなのか?」川西発之 LLM memo 2024100801 AI(27)

Last updated at 2024-11-23Posted at 2024-10-08

松尾研 LLM コミュニティ "Paper & Hacks Vol.20"
https://matsuolab-community.connpass.com/event/332695/

川西発之

Reference

[1] 岡崎直観 (2023), 大規模言語モデルの驚異と脅威 - Speaker Deck
https://speakerdeck.com/chokkan/20230327_riken_llm

[2] Ilya Sutskever et al. (2014), “sequence to sequence learning with neural networks”, NeurIPS2014
https://papers.nips.cc/paper_files/paper/2014/hash/a14ac55a4f27472c5d894ec1c3c743d2-Abstract.html

[3] Tomáš Mikolov, et al. (2010), “recurrent neural network based language model”, Proc. Interspeech 2010, 1045-1048
https://www.isca-speech.org/archive/interspeech_2010/mikolov10_interspeech.html

[5] Dzmitry Bahdanau et al. (2014), “Neural machine translation by jointly learning to align and translate”, arXiv:1409.0473
https://arxiv.org/abs/1409.0473

[6] Masaki Hayashi (2022), Transformer と seq2seq with attention の違いは？系列変換モデル【Q and A 記事】 | CVMLエキスパートガイド
https://cvml-expertguide.net/column/qa/difference-between-transformer-and-seq2seq/

[7] Ashish Vaswani et al. (2017), “Attention Is All You Need”, NeurIPS2017
https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

[8] OpenAI (2023) “GPT-4 Technical Report”, arXiv:2303.08774
https://arxiv.org/abs/2303.18027

[9] Jungo Kasai et al. (2023), “Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinations”, arXiv:2303.18027
https://arxiv.org/abs/2303.08774

[10] Harsha Nori et al. (2023), “Capabilities of GPT-4 on Medical Challenge Problems”, Microsoft
https://www.microsoft.com/en-us/research/publication/capabilities-of-gpt-4-on-medical-challenge-problems/

[11] 日本経済新聞 (2023), 生成AIが司法試験「合格水準」東大発スタートアップ、一部科目で - 日本経済新聞
https://www.nikkei.com/article/DGXZQOUC317WP0R30C23A5000000/

[12] 映画『トランスフォーマー/ビースト覚醒』公式サイト, https://tf-movie.jp/

[13] BRIDGE (2023), ジェネレーティブAIの基礎を築いた論文「Attention Is All You Need」著者たちの今——期待される〝OpenAIマフィア〟の出現 - BRIDGE（ブリッジ）テクノロジー＆スタートアップ情報, https://thebridge.jp/2023/05/expected-emergence-of-openai-mafia

[14] ASCII.jp (2023), 元グーグルのトップAI研究者、東京にAI企業「Sakana.ai」立ち上げ, https://ascii.jp/elem/000/004/150/4150456/,

[15] Shraddha Anala (2020), A Guide to Word Embedding. What are they? How are they more useful... | by Shraddha Anala | Towards Data Science, https://towardsdatascience.com/a-guide-to-word-embeddings-8a23817ab60f

[16] John Hewitt, Natural Language Processing with Deep Learning CS224N/Ling284,
https://web.stanford.edu/class/cs224n/slides/cs224n-2023-lecture08-transformers.pdf,

[17] Raimi Karim (2019) Illustrated: Self-Attention. A step-by-step guide to self-attention... | by Raimi Karim | Towards Data Science, https://towardsdatascience.com/illustrated-self-attention-2d627e33b20a

[18] Jay Alammar (2018) The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time. https://jalammar.github.io/illustrated-transformer/

[19] Mor Geva et al. (2021), “Transformer Feed-Forward Layers Are Key-Value Memories”, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484–5495
https://aclanthology.org/2021.emnlp-main.446/

[20] Kaiming He et al. (2016), “Deep Residual Learning for Image Recognition”,2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) pages 770-778
https://www.computer.org/csdl/proceedings-article/cvpr/2016/8851a770/12OmNxvwoXv

[21] Jimmy Lei Ba et al. (2016), "Layer Normalization”, arXiv:1607.06450
https://arxiv.org/abs/1607.06450

[22] Rishi Bommasani et al. (2021), “On the Opportunities and Risks of Foundation Models”, arXiv:2108.07258
https://arxiv.org/abs/2108.07258

[23] Hugo Touvron et al. (2023), “LLaMA: Open and Efficient Foundation Language Models”, arXiv:2302.13971
https://arxiv.org/abs/2302.13971

[24] Tom Brown et al. (2020), “Language Models are Few-Shot Learners”, NeurIPS2020
https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html

[25] Luca Soldaini (2023), AI2 Dolma: 3 Trillion Token Open Corpus for LLMs | AI2 Blog, https://blog.allenai.org/dolma-3-trillion-tokens-open-llm-corpus-9a0ff4b8da64

[26] Wayne Xin Zhao et al. (2023), “A Survey of Large Language Models”, arXiv:2303.18223
https://arxiv.org/abs/2303.18223

[27] Guilherme Penedo et al.(2023), “The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only” arXiv: 2306.01116
[28] Hugo Touvron et al. (2023), "Llama 2: Open Foundation and Fine-Tuned Chat Models” arXiv:2307.09288
https://arxiv.org/abs/2307.09288

[29] Fuzhao Xue et al. (2023), “To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis”, arXiv:2305.13230

[30] Niklas Muennighoff et al. (2023), “Scaling Data-Constrained Language Models”, arXiv:2305.16264

[31] Stas Bekman (2022) The Technology Behind BLOOM Training https://huggingface.co/blog/bloom-megatron-deepspeed

[32] suchenxang (2023), metaseq/projects/OPT/chronicles/OPT175B_Logbook.pdf at main · facebookresearch/metaseq · GitHub,
https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf

[33] Diederik P. Kingma & Jimmy Ba, (2014), “Adam: A Method for Stochastic Optimization”, arXiv:1412.6980

[34] Ilya Loshchilov & Frank Hutter, (2017), “Decoupled Weight Decay Regularization”, arXiv:1711.05101

[35] Shikoan’s ML Blog (2021), Cosine DecayとWarmupを同時にこなすスケジューラー（timm）| Shikoan's ML Blog, https://blog.shikoan.com/?s=Cosine

[36] Chip Huyen (2019) Evaluation Metrics for Language Modeling, https://thegradient.pub/understanding-evaluation-metrics-for-language-models/

[37] Kaito Sugimoto (2021) テキスト生成における decoding テクニック: Greedy search, Beam search, Top-K, Top-p https://zenn.dev/hellorusk/articles/1c0bef15057b1d

[38] mm_0824 (2020) ビームサーチ(Beam Search)を理解する| 楽しみながら理解するAI・機械学習入門 https://data-analytics.fun/2020/12/16/understanding-beamsearch/

[39] cohere, Temperature, https://docs.cohere.com/docs/temperature,

[40] Ari Holtzman et al. (2019), “The Curious Case of Neural Text Degeneration”, arXiv:1904.09751

[41] Colin Raffel et al. (2020), “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, The Journal of Machine Learning Research, Volume 21, Issue 1,Article No.: 140, pp 5485–5551

[42] Alexis Conneau et al. (2020), “Unsupervised Cross-lingual Representation Learning at Scale”, Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, pages 8440–8451

[43] Linting Xue et al. (2021), “mT5: A massively multilingual pre-trained text-to-text transformer”, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498

[44] Leo Gao et al. (2020), “The Pile: An 800GB Dataset of Diverse Text for Language Modeling”, arXiv:2101.00027

[45] Luca Soldaini et al. (2023) “Dolma: An Open Corpus of 3 Trillion Tokens for Language Model Pretraining Research”

[46] masao-classcat (2021), HuggingFace Transformers 4.5 : 利用方法 : トークナイザーの要点 – Transformers, Diffusers | ClassCat® Chatbot,
https://torch.classcat.com/2021/05/14/huggingface-transformers-4-5-tokenizer-summary/

[47] Linting Xue et al. (2022), “ByT5: Towards a token-free future with pre-trained byte-to-byte models” Transactions of the Association for Computational Linguistics, vol. 10, pp.291–306

[48] Jacob Devlin et al. (2019), “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”, Proceedings of NAACL-HLT 2019, pages 4171–4186

[49] Yinhan Liu et al. (2019), “RoBERTa: A Robustly Optimized BERT Pretraining Approach”, arXiv:1907.11692

[50] Zhenzhong Lan et al. (2020), “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations” ICLR2020

[51] Mike Lewis et al. (2020), “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension” Proceedings of the 58th
Annual Meeting of the Association for Computational Linguistics, pages 7871–7880

[52] 清野舜 (2022), より良いTransformerをつくる - Speaker Deck https://speakerdeck.com/butsugiri/yoriliang-itransformerwotukuru

[53] Ofir Press et al. (2021), “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation” arXiv:2108.12409

[54] Manzil Zaheer et al. (2020), “Big Bird: Transformers for Longer Sequences” NeurIPS2020

[55] Iz Beltagy et al. (2020), “Longformer: The Long-Document Transformer”, arXiv:2004.05150

[56] Joshua Ainslie et al. (2023), “GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints” arXiv:2305.13245

[57] Li Dong et al. (2019), “Unified Language Model Pre-training for Natural Language Understanding and Generation” NeurIPS2019

[58] Yi Tay et al. (2022), “UL2: Unifying Language Learning Paradigms” arXiv:2205.05131

[59] Yupeng Chang et al. (2023), “A Survey on Evaluation of Large Language Models” arXiv:2307.03109

[60] Seonghyeon Ye et al. (2023), “FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets” arXiv:2307.10928

[61] Andrea Galassi et al. (2019), “Attention in Natural Language Processing” arXiv:1902.02181

[62] Samira Abnar & Willem Zuidema (2020), “Quantifying Attention Flow in Transformers” Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, pages 4190–4197

[63] Yi Tay et al. (2021), “Are Pre-trained Convolutions Better than Pre-trained Transformers?” Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4349–4359

[64] Jean-Baptiste et al. (2020), “On the Relationship between Self-Attention and Convolutional Layers” ICLR2020

[65] Jack Morris (2020), “What are adversarial examples in NLP?”, https://towardsdatascience.com/what-are-adversarial-examples-in-nlp-f928c574478e

[66] Yonatan Belinkov (2022), “Probing Classifiers: Promises, Shortcomings, and Advances” Computational Linguistics, Volume 48, Issue 1 - March 2022 pages 207-119

<この項は書きかけです。順次追記します。>
This article is not completed. I will add some words and/or centences in order.

Qiita Calendar 2024

2024 参加・主催Calendarと投稿記事一覧 Qiita(248)
https://qiita.com/kaizen_nagoya/items/d80b8fbac2496df7827f

主催Calendar2024分析 Qiita(254)
https://qiita.com/kaizen_nagoya/items/15807336d583076f70bc

博士論文 Calendar 2024 を開催します。
https://qiita.com/kaizen_nagoya/items/51601357efbcaf1057d0

博士論文(0)関連記事一覧
https://qiita.com/kaizen_nagoya/items/8f223a760e607b705e78

自己記事一覧

Qiitaで逆リンクを表示しなくなったような気がする。時々、スマフォで表示するとあらわっることがあり、完全に削除したのではなさそう。

４月以降、せっせとリンクリストを作り、統計を取って確率を説明しようとしている。
2025年２月末を目標にしている。

一覧の一覧( The directory of directories of mine.) Qiita(100)
https://qiita.com/kaizen_nagoya/items/7eb0e006543886138f39

仮説（0）一覧（目標100現在40）
https://qiita.com/kaizen_nagoya/items/f000506fe1837b3590df

Qiita(0)Qiita関連記事一覧（自分）
https://qiita.com/kaizen_nagoya/items/58db5fbf036b28e9dfa6

Error一覧 error(0)
https://qiita.com/kaizen_nagoya/items/48b6cbc8d68eae2c42b8

C++ Support(0)　
https://qiita.com/kaizen_nagoya/items/8720d26f762369a80514

Coding(0) Rules, C, Secure, MISRA and so on
https://qiita.com/kaizen_nagoya/items/400725644a8a0e90fbb0

Ethernet 記事一覧　Ethernet(0)
https://qiita.com/kaizen_nagoya/items/88d35e99f74aefc98794

Wireshark 一覧 wireshark(0)、Ethernet(48)
https://qiita.com/kaizen_nagoya/items/fbed841f61875c4731d0

線網（Wi-Fi）空中線(antenna)(0) 記事一覧(118/300目標)
https://qiita.com/kaizen_nagoya/items/5e5464ac2b24bd4cd001

なぜdockerで機械学習するか書籍・ソース一覧作成中 (目標100)
https://qiita.com/kaizen_nagoya/items/ddd12477544bf5ba85e2

プログラムちょい替え（0）一覧:4件
https://qiita.com/kaizen_nagoya/items/296d87ef4bfd516bc394

言語処理100本ノックをdockerで。python覚えるのに最適。:10+12
https://qiita.com/kaizen_nagoya/items/7e7eb7c543e0c18438c4

Python(0)記事をまとめたい。
https://qiita.com/kaizen_nagoya/items/088c57d70ab6904ebb53

安全（0）安全工学シンポジウムに向けて: 21
https://qiita.com/kaizen_nagoya/items/c5d78f3def8195cb2409

プログラマによる、プログラマのための、統計(0)と確率のプログラミングとその後
https://qiita.com/kaizen_nagoya/items/6e9897eb641268766909

転職(0)一覧
https://qiita.com/kaizen_nagoya/items/f77520d378d33451d6fe

技術士(0)一覧
https://qiita.com/kaizen_nagoya/items/ce4ccf4eb9c5600b89ea

Reserchmap(0) 一覧
https://qiita.com/kaizen_nagoya/items/506c79e562f406c4257e

物理記事　上位100
https://qiita.com/kaizen_nagoya/items/66e90fe31fbe3facc6ff

量子(0) 計算機, 量子力学
https://qiita.com/kaizen_nagoya/items/1cd954cb0eed92879fd4

数学関連記事１００
https://qiita.com/kaizen_nagoya/items/d8dadb49a6397e854c6d

coq(0) 一覧
https://qiita.com/kaizen_nagoya/items/d22f9995cf2173bc3b13

統計(0)一覧
https://qiita.com/kaizen_nagoya/items/80d3b221807e53e88aba

図(0) state, sequence and timing. UML and お絵描き
https://qiita.com/kaizen_nagoya/items/60440a882146aeee9e8f

色(0) 記事100書く切り口
https://qiita.com/kaizen_nagoya/items/22331c0335ed34326b9b

品質一覧
https://qiita.com/kaizen_nagoya/items/2b99b8e9db6d94b2e971

言語・文学記事　１００
https://qiita.com/kaizen_nagoya/items/42d58d5ef7fb53c407d6

医工連携関連記事一覧
https://qiita.com/kaizen_nagoya/items/6ab51c12ba51bc260a82

水の資料集(0)　方針と成果
https://qiita.com/kaizen_nagoya/items/f5dbb30087ea732b52aa

自動車　記事　１００
https://qiita.com/kaizen_nagoya/items/f7f0b9ab36569ad409c5

通信記事１００
https://qiita.com/kaizen_nagoya/items/1d67de5e1cd207b05ef7

日本語（０）一欄
https://qiita.com/kaizen_nagoya/items/7498dcfa3a9ba7fd1e68

英語(0) 一覧
https://qiita.com/kaizen_nagoya/items/680e3f5cbf9430486c7d

音楽　一覧(0)
https://qiita.com/kaizen_nagoya/items/b6e5f42bbfe3bbe40f5d

「@kazuo_reve 新人の方によく展開している有益な情報」確認一覧
https://qiita.com/kaizen_nagoya/items/b9380888d1e5a042646b

鉄道（０）鉄道のシステム考察はてっちゃんがてつだってくれる
https://qiita.com/kaizen_nagoya/items/faa4ea03d91d901a618a

OSEK OS設計の基礎　OSEK(100)
https://qiita.com/kaizen_nagoya/items/7528a22a14242d2d58a3

coding (101) 一覧を作成し始めた。omake:最近のQiitaで表示しない5つの事象
https://qiita.com/kaizen_nagoya/items/20667f09f19598aedb68

官公庁・学校・公的団体（NPOを含む）システムの課題、官（０）
https://qiita.com/kaizen_nagoya/items/04ee6eaf7ec13d3af4c3

「はじめての」シリーズ　ベクタージャパン　
https://qiita.com/kaizen_nagoya/items/2e41634f6e21a3cf74eb

AUTOSAR(0)Qiita記事一覧, OSEK(75)
https://qiita.com/kaizen_nagoya/items/89c07961b59a8754c869

プログラマが知っていると良い「公序良俗」
https://qiita.com/kaizen_nagoya/items/9fe7c0dfac2fbd77a945

LaTeX(0) 一覧　
https://qiita.com/kaizen_nagoya/items/e3f7dafacab58c499792

自動制御、制御工学一覧（０）
https://qiita.com/kaizen_nagoya/items/7767a4e19a6ae1479e6b

Rust(0) 一覧　
https://qiita.com/kaizen_nagoya/items/5e8bb080ba6ca0281927

「スケーリング法則が何故スケーラブルなのか?」川西発之 LLM memo 2024100801 AI(27)

川西発之

Reference

Qiita Calendar 2024

自己記事一覧

関連資料

Engineering Festa 2024前に必読記事一覧

文書履歴(document history)

最後までおよみいただきありがとうございました。

Thank you very much for reading to the last sentence.

「スケーリング法則が何故スケーラブルなのか?」 川西 発之 LLM memo 2024100801 AI(27)

川西発之

Reference

Qiita Calendar 2024

自己記事一覧

関連資料

Engineering Festa 2024前に必読記事一覧

文書履歴(document history)

最後までおよみいただきありがとうございました。

Thank you very much for reading to the last sentence.

「スケーリング法則が何故スケーラブルなのか?」川西発之 LLM memo 2024100801 AI(27)