LLMはハルシネーションを自覚しているか 松尾研 LLM コミュニティ "Paper & Hacks Vol.24" AI(6)

生成AI Calendar 2024 @k-keita　
https://qiita.com/advent-calendar/2024/generateai
17日投稿予定の記事です。

松尾研 LLM コミュニティ "Paper & Hacks Vol.24"
https://matsuolab-community.connpass.com/event/335384/

発表者: 松尾研LLMコミュニティメンバー 下村 晃生（九州工業大学工学部宇宙システム工学科/松尾研大規模言語モデル講座第９回担当）
テーマ: LLMはハルシネーションを自覚しているか
論文リンク：Physics of Language Models: Part2.2 How to Learn From Mistakes on Grade-School Math Problems
https://arxiv.org/abs/2408.16293

自然言語処理をdocker(186)で
https://qiita.com/kaizen_nagoya/items/e29cbaed8370e7913487

参考文献の参考文献を一覧とするとともに単語帳を作る。（途中）
単語帳は、２回出現する単語までをここに記載している。１度だけの単語はDocker上にある。

<この項は書きかけです。順次追記します。>
This article is not completed. I will add some words and/or centences in order.

Reference

[1] Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36, 2024.
https://arxiv.org/abs/2306.00297
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 1, Learning Hierarchical Language Structures. ArXiv e-prints, abs/2305.13673, May 2023. Full version available at http://arxiv.org/abs/2305.13673.
[3] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.2, Knowledge Manipulation. ArXiv e-prints, abs/2309.14402, September 2023. Full version available at http://arxiv.org/abs/2309.14402.
[4] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. In ICML, 2024. Full version available at http://arxiv.org/abs/2309.14316.
[5] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. ArXiv e-prints, abs/2404.05405, April 2024. Full version available at http://arxiv.org/abs/ 2404.05405.
[6] Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. Advances in Neural Information Processing Systems, 35:38546–38556, 2022.
[7] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022. URL https://arxiv.org/abs/2204.06745.
[8] S ́ebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
[10] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In ICLR, 2021.
[11] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023.
[12] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5315–5333, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.291. URL https://aclanthology.org/ 2023.acl-long.291.
[13] Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. Making language models better reasoners with step-aware verifier. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5315–5333, 2023.
[14] Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. TinyGSM: achieving > 80% on GSM8k with small language models. arXiv preprint arXiv:2312.09241, 2023.
[15] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
[16] Marah Abdin et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024.
[17] John Miller, Karl Krauth, Benjamin Recht, and Ludwig Schmidt. The effect of natural distribution shift on question answering models. In International conference on machine learning, pages 6905–6916. PMLR, 2020.
[18] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.
[19] Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. arXiv preprint arXiv:2308.03188, 2023.
[20] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
[21] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116–12128, 2021.
[22] Yiheng Shu and Zhiwei Yu. Distribution shifts are bottlenecks: Extensive evaluation for grounding language models to knowledge bases. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 71–88, 2024.
[23] Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Al ́an Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv preprint arXiv:2303.14100, 2023.
[24] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
[25] Llama Team. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[26] Fei Wang, Chao Shang, Sarthak Jain, Shuai Wang, Qiang Ning, Bonan Min, Vittorio Castelli, Yassine Benajiba, and Dan Roth. From instructions to constraints: Language model alignment with automatic constraint verification. arXiv preprint arXiv:2403.06326, 2024.
[27] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09561, 2022.
[28] Kaiyu Yang, Jia Deng, and Danqi Chen. Generating natural language proofs with verifier-guided search. arXiv preprint arXiv:2205.12443, 2022.
[29] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade- School Math and the Hidden Reasoning Process. arXiv e-prints, abs/2407.20311, 2024. Full version available at http://arxiv.org/abs/2407.20311.
[30] Yunxiang Zhang, Muhammad Khalifa, Lajanugen Logeswaran, Jaekyeom Kim, Moontae Lee, Honglak Lee, and Lu Wang. Small language models need strong verifiers to self-correct reasoning. arXiv preprint arXiv:2404.17140, 2024.
[31] Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. What algorithms can transformers learn? a study in length generalization. arXiv preprint arXiv:2310.16028, 2023.

Reference on the Reference

1 Kwangjun Ahn

Kwangjun Ahn, Xiang Cheng, Minhak Song, Chulhee Yun, Ali Jadbabaie, and Suvrit Sra. Lin- ear attention is (maybe) all you need (to understand transformer optimization). arXiv preprint arXiv:2310.01082, 2023.
Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. International Conference on Learning Representations, 2022.
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 1, context-free grammar. arXiv preprint arXiv:2305.13673, 2023.
Noga Alon and Joel H Spencer. The probabilistic method. John Wiley & Sons, 2016.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. Proceedings of BigScience – Workshop on Challenges & Perspectives in Creating Large Language Models, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Neural Information Processing Systems, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, 2019.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Benjamin L Edelman, Surbhi Goel, Sham Kakade, and Cyril Zhang. Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning (ICML), 2022.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https://transformer-circuits.pub/2021/framework/index.html.
Murat A Erdogdu, Lee H Dicker, and Mohsen Bayati. Scaled least squares estimator for glms in large-scale problems. Advances in Neural Information Processing Systems, 29, 2016.
Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598, 2022.
Angeliki Giannou, Shashank Rajput, Jy-yong Sohn, Kangwook Lee, Jason D Lee, and Dimitris Pa- pailiopoulos. Looped transformers as programmable computers. arXiv preprint arXiv:2301.13196, 2023.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997.
Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=SJa9iHgAZ.
Ke Li and Jitendra Malik. Learning to optimize. In International Conference on Learning Represen- tations, 2017.
11
Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evaluation. White Paper. AI21 Labs, 2021.
Arvind Mahankali, Tatsunori B Hashimoto, and Tengyu Ma. One step of gradient descent is provably the optimal in-context learner with one layer of linear self-attention. arXiv preprint arXiv:2307.03576, 2023.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022.
Jorge Pérez, Pablo Barceló, and Javier Marinkovic. Attention is turing complete. The Journal of Machine Learning Research, 2021.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.
Imanol Schlag, Kazuki Irie, and Jürgen Schmidhuber. Linear transformers are secretly fast weight programmers. In International Conference on Machine Learning, pages 9355–9366. PMLR, 2021.
Hava T Siegelmann and Eduardo D Sontag. On the computational power of neural nets. In Proceedings of Workshop on Computational learning theory, 1992.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 2017.
Johannes von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151–35174. PMLR, 2023.
Colin Wei, Yining Chen, and Tengyu Ma. Statistically meaningful approximation: a case study on approximating turing machines with transformers. Advances in Neural Information Processing Systems, 35:12071–12083, 2022.
Mitchell Wortsman, Jaehoon Lee, Justin Gilmer, and Simon Kornblith. Replacing softmax with relu in vision transformers. arXiv preprint arXiv:2309.08586, 2023.
Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. International Conference on Learning Representations, 2021.
Ruiqi Zhang, Spencer Frei, and Peter L Bartlett. Trained transformers learn linear models in-context. arXiv preprint arXiv:2306.09927, 2023.
Haoyu Zhao, Abhishek Panigrahi, Rong Ge, and Sanjeev Arora. Do transformers parse while predicting the masked word? arXiv preprint arXiv:2303.08117, 2023.

2

[1] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. ArXiv e-prints, abs/2309.14316, September 2023. Full version available at http://arxiv. org/abs/2309.14316.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.2, Knowledge Manipulation. ArXiv e-prints, abs/2309.14402, September 2023. Full version available at http://arxiv.org/abs/ 2309.14402.
[3] Zeyuan Allen-Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep learning. In COLT, 2023. Full version available at http://arxiv.org/abs/2001.04413.
[4] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. ArXiv e-prints, abs/2404.05405, April 2024. Full version available at http://arxiv.org/abs/ 2404.05405.
[5] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In ICML, 2019. Full version available at http://arxiv.org/abs/1811.03962.
[6] Sanjeev Arora and Yi Zhang. Do gans actually learn the distribution? an empirical study. arXiv preprint arXiv:1706.08224, 2017.
[7] David Arps, Younes Samih, Laura Kallmeyer, and Hassan Sajjad. Probing for constituency structure in neural language models. arXiv preprint arXiv:2204.06201, 2022.
[8] James K Baker. Trainable grammars for speech recognition. The Journal of the Acoustical Society of America, 65(S1):S132–S132, 1979.
[9] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022. URL https://arxiv.org/abs/2204.06745.
[10] Gregoire Deletang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, et al. Neural networks and the chomsky hierar- chy. In ICLR, 2023.
[11] Brian DuSell and David Chiang. Learning hierarchical structures with differentiable nondeterministic stacks. In ICLR, 2022.
[12] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1, 2021.
[13] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020.
[14] John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word represen- tations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1419. URL https://aclanthology.org/N19-1419.
[15] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186, 2019.
[16] Christopher D Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48):30046–30054, 2020.
[17] Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330, 1993. URL https: //aclanthology.org/J93-2004.
[18] Rowan Hall Maudslay and Ryan Cotterell. Do syntactic probes probe syntax? experiments with jabberwocky probing. arXiv preprint arXiv:2106.02559, 2021.
[19] Milad Moradi and Matthias Samwald. Evaluating the robustness of neural language models to input perturbations. arXiv preprint arXiv:2108.12237, 2021.
[20] Shikhar Murty, Pratyusha Sharma, Jacob Andreas, and Christopher D Manning. Characterizing intrin- sic compositionality in transformers with tree projections. In ICLR, 2023.
[21] Neel Nanda, Lawrence Chan, Tom Liberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217, 2023.
[22] Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022.
[23] OpenAI. Gpt-4 technical report, 2023.
[24] Matt Post and Shane Bergsma. Explicit and implicit syntactic features for text classification. In
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 866–872, 2013.
[25] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[26] Itiroo Sakai. Syntax in universal translation. In Proceedings of the International Conference on Machine Translation and Applied Language Analysis, 1961.
[27] Hui Shi, Sicun Gao, Yuandong Tian, Xinyun Chen, and Jishen Zhao. Learning bounded context-free- grammar via lstm and the transformer: Difference and the explanations. In Proceedings of the AAAI
[28] Michael Sipser. Introduction to the Theory of Computation. Cengage Learning, 2012.
[29] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021.
[30] Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621–633, 2020.
[31] David Vilares, Michalina Strzyz, Anders Søgaard, and Carlos G ́omez-Rodr ́ıguez. Parsing as pretraining. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9114–9121, 2020.
[32] KevinWang, AlexandreVariengien, ArthurConmy, BuckShlegeris,and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. arXiv preprint arXiv:2211.00593, 2022.
[33] Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. Perturbed masking: Parameter-free probing for analyzing and interpreting bert. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166–4176, 2020.
[34] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. arXiv preprint, 2024. to appear.
[35] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems. arXiv preprint, 2024. to appear.
[36] Shizhuo Dylan Zhang, Curt Tigges, Stella Biderman, Maxim Raginsky, and Talia Ringer. Can trans- formers learn to solve problems recursively? arXiv preprint arXiv:2305.14699, 2023.
[37] Haoyu Zhao, Abhishek Panigrahi, Rong Ge, and Sanjeev Arora. Do transformers parse while predicting the masked word? arXiv preprint arXiv:2303.08117, 2023.

3

[1] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 1, Learning Hierarchical Language Structures. ArXiv e-prints, abs/2305.13673, May 2023. Full version available at http://arxiv.org/ abs/2305.13673.
[2] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.1, Knowledge Storage and Extraction. In ICML, 2024. Full version available at http://arxiv.org/abs/2309.14316.
[3] Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws. ArXiv e-prints, abs/2404.05405, April 2024. Full version available at http://arxiv.org/abs/ 2404.05405.
[4] Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. The Reversal Curse: LLMs trained on ”A is B” fail to learn ”B is A”. arXiv preprint arXiv:2309.12288, September 2023.
[5] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT-NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models, 2022. URL https://arxiv.org/abs/2204.06745.
[6] Deng Cai, Yan Wang, Lemao Liu, and Shuming Shi. Recent advances in retrieval-augmented text gen- eration. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3417–3419, 2022.
[7] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361, 2021.
[8] Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. arXiv preprint arXiv:2404.19737, 2024. [9] Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, and Sainbayar Sukhbaatar. Reverse training to nurse the reversal curse. arXiv preprint arXiv:2403.13799, 2024. [10] Qingyan Guo, Rui Wang, Junliang Guo, Xu Tan, Jiang Bian, and Yujiu Yang. Mitigating reversal curse via semantic-aware permutation training. arXiv preprint arXiv:2403.00758, 2024. [11] Evan Hernandez, Belinda Z Li, and Jacob Andreas. Measuring and manipulating knowledge represen- tations in language models. arXiv preprint arXiv:2304.00740, 2023. [12] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. LoRA: Low-Rank Adaptation of Large Language Models. In ICLR, 2021. [13] AlbertQJiang,AlexandreSablayrolles, ArthurMensch, ChrisBamford, DevendraSinghChaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [14] Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983, 2023. [15] Mojtaba Komeili, Kurt Shuster, and Jason Weston. Internet-augmented dialogue generation. arXiv preprint arXiv:2107.07566, 2021. [16] R ́emi Lebret, David Grangier, and Michael Auli. Generating text from structured data with application to the biography domain. CoRR, abs/1603.07771, 2016. URL http://arxiv.org/abs/1603.07771. [17] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Ku ̈ttler, Mike Lewis, Wen-tau Yih, Tim Rockt ̈aschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/ paper_files/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf. [18] Shangqing Liu, Yu Chen, Xiaofei Xie, Jingkai Siow, and Yang Liu. Retrieval-augmented generation for code summarization via hybrid gnn. arXiv preprint arXiv:2006.05405, 2020. [19] Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. Generation-augmented retrieval for open-domain question answering. arXiv preprint arXiv:2009.08553, 2020. [20] Tahira Naseem, Srinivas Ravishankar, Nandana Mihindukulasooriya, Ibrahim Abdelaziz, Young-Suk Lee, Pavan Kapanipathi, Salim Roukos, Alfio Gliozzo, and Alexander Gray. A semantics-aware trans- former model of relation linking for knowledge base question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 256–262, Online, August 2021. Association for Computational Linguistics. [21] Anh Nguyen, Nikos Karampatziakis, and Weizhu Chen. Meet in the middle: A new pre-training paradigm. Advances in Neural Information Processing Systems, 36, 2024. [22] Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. Chatgpt versus traditional ques- tion answering for knowledge graphs: Current status and future directions towards knowledge graph chatbots. arXiv preprint arXiv:2302.06466, 2023. [23] OpenAI. Gpt-4 technical report, 2023. [24] Md Rizwan Parvez, Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. Retrieval augmented code generation and summarization. arXiv preprint arXiv:2108.11601, 2021. [25] Hao Peng, Xiaozhi Wang, Shengding Hu, Hailong Jin, Lei Hou, Juanzi Li, Zhiyuan Liu, and Qun Liu. Copen: Probing conceptual knowledge in pre-trained language models. arXiv preprint arXiv:2211.04079, 2022. [26] Fabio Petroni, Tim Rockt ̈aschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066, 2019. [27] Jacob Pfau, William Merrill, and Samuel R Bowman. Let’s think dot by dot: Hidden computation in transformer language models. arXiv preprint arXiv:2404.15758, 2024. [28] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [29] OriRam,YoavLevine,ItayDalmedigos,DorMuhlgay,AmnonShashua,KevinLeyton-Brown,andYoav Shoham. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083, 2023. [30] Kyle Richardson and Ashish Sabharwal. What does my QA model know? devising controlled probes using expert knowledge. Transactions of the Association for Computational Linguistics, 8:572–588, 2020. doi: 10.1162/tacl a 00331. URL https://aclanthology.org/2020.tacl-1.37. [31] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022. [32] Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. Transactions of the Association for Computational Linguistics, 11:1–17, 2023. [33] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2021. [34] Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. Head-to-tail: How knowl- edgeable are large language models (llm)? aka will llms replace knowledge graphs? arXiv preprint arXiv:2308.10168, 2023. [35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ́ee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[36] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
[37] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. arXiv preprint arXiv:xxxx.xxxxx, 2024. to appear.
[38] Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems. arXiv preprint arXiv:xxxx.xxxxx, 2024. to appear.
[39] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023.

単語帳

単語帳の作成目的は、英語論文を読む際に、頻出単語上位で知らない単語がないことを確認するためでした。
国際規格の翻訳業務をしていたときには、予定訳語を想定し、同じ文脈では同じ訳語を使うための基準として作成していました。
機械翻訳ソフトの開発に参加してからは、機械翻訳ソフトの精度確認のため、予定訳語を想定し、文脈上どのように訳しているかを確認するために利用していました。

LLM論文の翻訳はあまりしておらず、今回は、まだ訳語はつけていません。何本か単語帳を作ったら、１００後から１０００語くらいまでは訳語をつけようかなって思っています。

PDFファイルで保存してそのままTXTファイルにしています。論文内部ではない単語も混ざっています。ごめんなさい。経験上1%を超えないことと、定型的なものがあり論文共通の文字列だったりすることがあります。誤差ということでご勘弁。

自然言語処理をdocker(186)で
https://qiita.com/kaizen_nagoya/items/e29cbaed8370e7913487

次回からは

bash

docker run -v /tmp/llm:/rmp/llm -it kaizenjapan/llm /bin/bash

	term	count
1	the	540
2	retry	329
3	to	252
4	of	249
5	a	237
6	s	221
7	with	210
8	op	202
9	and	199
10	in	182
11	we	165
12	data	160
13	retryrate	160
14	is	141
15	for	140
16	model	131
17	each	123
18	as	119
19	on	117
20	this	116
21	can	103
22	it	99
23	error	90
24	e	88
25	lora	82
26	that	81
27	finetune	80
28	figure	76
29	box	70
30	define	70
31	unnecessary	69
32	models	67
33	language	63
34	pretrain	60
35	so	60
36	number	59
37	qv	57
38	igsm	52
39	mask	52
40	not	52
41	be	49
42	correct	49
43	from	49
44	pretrained	49
45	rate	48
46	accuracy	47
47	b	47
48	mistakes	47
49	such	47
50	using	45
51	cheese	44
52	market	44
53	or	42
54	param	41
55	solution	41
56	beam	40
57	math	40
58	studio	39
59	arxiv	38
60	maskretryrate	38
61	training	38
62	its	37
63	are	36
64	free	35
65	pretraining	35
66	parameters	34
67	reasoning	34
68	equals	33
69	has	33
70	if	33
71	steps	33
72	also	32
73	errors	32
74	problems	32
75	retry_weak	32
76	solutions	32
77	even	31
78	an	30
79	film	30
80	high	30
81	upon	29
82	use	29
83	regret	28
84	school	28
85	g	27
86	gpt	27
87	have	27
88	x	27
89	r	26
90	tokens	26
91	when	26
92	by	25
93	jim	25
94	probing	25
95	section	25
96	w	25
97	finetuning	24
98	next	24
99	only	24
100	process	24
101	see	24
102	correction	23
103	generation	23
104	learning	23
105	original	23
106	result	23
107	same	23
108	version	23
109	after	22
110	c	22
111	dist	22
112	i	22
113	international	22
114	parmesan	22
115	than	22
116	weight	22
117	at	21
118	but	21
119	cream	21
120	full	21
121	jungle	21
122	mistake	21
123	our	21
124	reask	21
125	backpack	20
126	district	20
127	ice	20
128	inside	20
129	ood	20
130	p	20
131	parameter	19
132	problem	19
133	rank	19
134	results	19
135	sentence	19
136	step	19
137	their	19
138	time	19
139	used	19
140	back	18
141	distout	18
142	many	18
143	more	18
144	one	18
145	own	18
146	central	17
147	example	17
148	paper	17
149	still	17
150	v	17
151	which	17
152	all	16
153	during	16
154	grape	16
155	make	16
156	no	16
157	pounds	16
158	u	16
159	al	15
160	d	15
161	detection	15
162	how	15
163	improve	15
164	k	15
165	masking	15
166	retry_miss	15
167	they	15
168	context	14
169	continued	14
170	datasets	14
171	decay	14
172	directly	14
173	does	14
174	hard	14
175	l	14
176	length	14
177	part	14
178	preprint	14
179	q	14
180	small	14
181	weak	14
182	while	14
183	appendix	13
184	corrections	13
185	fine	13
186	fresh	13
187	generate	13
188	learn	13
189	li	13
190	search	13
191	size	13
192	synthetic	13
193	trained	13
194	wrong	13
195	y	13
196	best	12
197	daypack	12
198	double	12
199	label	12
200	made	12
201	med	12
202	pineapple	12
203	self	12
204	there	12
205	very	12
206	accuracies	11
207	batch	11
208	better	11
209	cannot	11
210	details	11
211	et	11
212	experiments	11
213	higher	11
214	however	11
215	o	11
216	over	11
217	perfect	11
218	test	11
219	FALSE	11
220	abs	10
221	chen	10
222	college	10
223	compared	10
224	different	10
225	distribution	10
226	embedding	10
227	experiment	10
228	goat	10
229	inference	10
230	joe	10
231	may	10
232	miss	10
233	need	10
234	residential	10
235	retries	10
236	skill	10
237	t	10
238	uses	10
239	vs	10
240	weights	10
241	where	10
242	without	10
243	yuanzhi	10
244	among	9
245	approach	9
246	call	9
247	dance	9
248	do	9
249	generated	9
250	j	9
251	large	9
252	layer	9
253	llms	9
254	med_pqigsm	9
255	stage	9
256	sum	9
257	versionp	9
258	what	9
259	would	9
260	zhu	9
261	TRUE	9
262	additional	8
263	allen	8
264	average	8
265	because	8
266	comparing	8
267	comparison	8
268	dataset	8
269	erroneous	8
270	evaluation	8
271	f	8
272	fake	8
273	followed	8
274	following	8
275	future	8
276	goal	8
277	including	8
278	instance	8
279	internal	8
280	knowledge	8
281	line	8
282	med_qpigsm	8
283	messenger	8
284	much	8
285	product	8
286	prompting	8
287	simply	8
288	times	8
289	total	8
290	tuning	8
291	two	8
292	verify	8
293	wang	8
294	already	7
295	answer	7
296	arithmetic	7
297	compare	7
298	detect	7
299	detector	7
300	easily	7
301	first	7
302	generating	7
303	level	7
304	like	7
305	meta	7
306	needed	7
307	new	7
308	org	7
309	other	7
310	output	7
311	perform	7
312	physics	7
313	question	7
314	similar	7
315	thus	7
316	trader	7
317	trainable	7
318	via	7
319	zeyuan	7
320	adding	6
321	additionally	6
322	available	6
323	before	6
324	compute	6
325	dosample	6
326	especially	6
327	except	6
328	few	6
329	harder	6
330	here	6
331	immediately	6
332	important	6
333	knows	6
334	linear	6
335	low	6
336	making	6
337	might	6
338	most	6
339	n	6
340	note	6
341	operations	6
342	out	6
343	pear	6
344	perfectly	6
345	presented	6
346	qp	6
347	random	6
348	re	6
349	riverview	6
350	samples	6
351	seeds	6
352	significantly	6
353	states	6
354	sufficiently	6
355	tasks	6
356	teach	6
357	those	6
358	twice	6
359	update	6
360	value	6
361	why	6
362	work	6
363	ye	6
364	accurate	5
365	achieve	5
366	achieved	5
367	amount	5
368	arts	5
369	autoregressive	5
370	based	5
371	been	5
372	both	5
373	campus	5
374	cell	5
375	ch	5
376	choices	5
377	complexity	5
378	computations	5
379	computed	5
380	contains	5
381	controlled	5
382	correctness	5
383	counts	5
384	difficulties	5
385	discover	5
386	end	5
387	explore	5
388	fair	5
389	find	5
390	focus	5
391	grade	5
392	h	5
393	http	5
394	immediate	5
395	include	5
396	information	5
397	ingredient	5
398	instead	5
399	into	5
400	long	5
401	matrices	5
402	multi	5
403	necessary	5
404	needs	5
405	neural	5
406	observation	5
407	per	5
408	please	5
409	prepare	5
410	present	5
411	probability	5
412	provide	5
413	query	5
414	randomly	5
415	regretful	5
416	seasons	5
417	setting	5
418	shortest	5
419	should	5
420	show	5
421	shows	5
422	solve	5
423	some	5
424	statistics	5
425	these	5
426	through	5
427	token	5
428	type	5
429	up	5
430	us	5
431	verifier	5
432	were	5
433	works	5
434	yet	5
435	zhang	5
436	actually	4
437	advances	4
438	almost	4
439	another	4
440	any	4
441	association	4
442	attention	4
443	carefully	4
444	cases	4
445	changes	4
446	check	4
447	common	4
448	computation	4
449	computational	4
450	conclusion	4
451	consider	4
452	contrast	4
453	controllable	4
454	could	4
455	difference	4
456	due	4
457	easy	4
458	effective	4
459	exhibit	4
460	fewer	4
461	final	4
462	found	4
463	fully	4
464	given	4
465	gpu	4
466	graph	4
467	hidden	4
468	improvement	4
469	included	4
470	introduce	4
471	just	4
472	larger	4
473	lee	4
474	linguistics	4
475	liu	4
476	llama	4
477	longer	4
478	makes	4
479	matrix	4
480	medop	4
481	method	4
482	multiple	4
483	natural	4
484	observations	4
485	obtain	4
486	oh	4
487	originalunnecessary	4
488	pages	4
489	possible	4
490	pq	4
491	prints	4
492	proceedings	4
493	processing	4
494	range	4
495	ready	4
496	real	4
497	regenerate	4
498	requires	4
499	round	4
500	select	4
501	sense	4
502	shown	4
503	since	4
504	skip	4
505	statement	4
506	strong	4
507	systems	4
508	tells	4
509	them	4
510	then	4
511	though	4
512	towards	4
513	transformers	4
514	trivial	4
515	understanding	4
516	verification	4
517	vocational	4
518	weighs	4
519	whether	4
520	within	4
521	your	4
522	z	4
523	about	3
524	accurately	3
525	acl	3
526	adamw	3
527	add	3
528	again	3
529	alignment	3
530	allow	3
531	although	3
532	auto	3
533	automatically	3
534	banana	3
535	below	3
536	between	3
537	case	3
538	chance	3
539	change	3
540	choose	3
541	com	3
542	commercial	3
543	complements	3
544	correcting	3
545	cosine	3
546	count	3
547	creating	3
548	determine	3
549	did	3
550	differs	3
551	discard	3
552	down	3
553	eese	3
554	effectively	3
555	efficient	3
556	encourages	3
557	entire	3
558	evaluated	3
559	extreme	3
560	failure	3
561	finetuned	3
562	format	3
563	gain	3
564	general	3
565	go	3
566	half	3
567	hardest	3
568	hardpq	3
569	head	3
570	highly	3
571	https	3
572	ideally	3
573	identical	3
574	illustration	3
575	implies	3
576	improves	3
577	improving	3
578	indicating	3
579	intelligence	3
580	interestingly	3
581	intuitively	3
582	keep	3
583	least	3
584	let	3
585	life	3
586	lin	3
587	lu	3
588	maximum	3
589	methods	3
590	mlp	3
591	nearly	3
592	non	3
593	notably	3
594	observe	3
595	once	3
596	overall	3
597	pattern	3
598	peft	3
599	performance	3
600	performs	3
601	position	3
602	practice	3
603	prediction	3
604	previous	3
605	produce	3
606	prompts	3
607	qiang	3
608	randomness	3
609	rather	3
610	realize	3
611	reasoners	3
612	right	3
613	rows	3
614	sample	3
615	sampling	3
616	save	3
617	scheduling	3
618	sentences	3
619	set	3
620	shift	3
621	significant	3
622	simplify	3
623	sinternation	3
624	solving	3
625	specifically	3
626	st	3
627	stating	3
628	strikethrough	3
629	success	3
630	sufficient	3
631	supermarket	3
632	table	3
633	task	3
634	temperature	3
635	too	3
636	top	3
637	tried	3
638	truly	3
639	try	3
640	tunes	3
641	understand	3
642	unless	3
643	updates	3
644	versions	3
645	want	3
646	was	3
647	weizhu	3
648	will	3
649	writing	3
650	xu	3
651	yang	3
652	yu	3
653	zhou	3
654	above	2
655	abstract	2
656	achieving	2
657	acquire	2
658	acquired	2
659	actual	2
660	adaptation	2
661	adapted	2
662	against	2
663	algorithm	2
664	allowing	2
665	alters	2
666	anna	2
667	annual	2
668	appeared	2
669	appears	2
670	applied	2
671	apply	2
672	approaches	2
673	architecture	2
674	article	2
675	aspect	2
676	aspects	2
677	augment	2
678	auxiliary	2
679	avoid	2
680	aware	2
681	backpacks	2
682	becomes	2
683	begin	2
684	bei	2
685	ben	2
686	benefit	2
687	betas	2
688	bin	2
689	broken	2
690	bubeck	2
691	calculate	2
692	capability	2
693	capacity	2
694	checking	2
695	cmu	2
696	comes	2
697	comparable	2
698	complex	2
699	conclusions	2
700	conference	2
701	configurations	2
702	conjecture	2
703	consists	2
704	constructed	2
705	contained	2
706	corpus	2
707	correctly	2
708	corrects	2
709	correspond	2
710	cot	2
711	crucial	2
712	current	2
713	daypacks	2
714	decoder	2
715	decoding	2
716	decreased	2
717	deferred	2
718	definethefresh	2
719	demonstrate	2
720	denote	2
721	dependencyunused	2
722	depends	2
723	description	2
724	desirable	2
725	detected	2
726	detecting	2
727	difficulty	2
728	doesn	2
729	easier	2
730	efforts	2
731	either	2
732	eldan	2
733	eliminating	2
734	encourage	2
735	english	2
736	ensuring	2
737	eric	2
738	essentially	2
739	evidence	2
740	exact	2
741	examples	2
742	exploring	2
743	extremely	2
744	fei	2
745	finally	2
746	focuses	2
747	follow	2
748	framework	2
749	fu	2
750	gao	2
751	generalization	2
752	generates	2
753	generative	2
754	give	2
755	grammar	2
756	ground	2
757	gsm	2
758	guang	2
759	guide	2
760	guided	2
761	he	2
762	help	2
763	hierarchical	2
764	hig	2
765	hu	2
766	icecreamasp	2
767	idea	2
768	implement	2
769	implemented	2
770	incentivized	2
771	increase	2
772	increases	2
773	indicates	2
774	initial	2
775	insert	2
776	integers	2
777	interested	2
778	introduced	2
779	introduction	2
780	issue	2
781	iterative	2
782	jian	2
783	job	2
784	jun	2
785	karl	2
786	last	2
787	later	2
788	layers	2
789	leading	2
790	leads	2
791	learned	2
792	learns	2
793	less	2
794	lets	2
795	limit	2
796	llm	2
797	location	2
798	logic	2
799	lou	2
800	lower	2
801	lr	2
802	mann	2
803	manner	2
804	maskigsm	2
805	masks	2
806	mbzuai	2
807	meaning	2
808	means	2
809	medpq	2
810	meeting	2
811	michael	2
812	miller	2
813	multinomial	2
814	name	2
815	now	2
816	occurs	2
817	often	2
818	open	2
819	otherwise	2
820	outperform	2
821	pan	2
822	papers	2
823	paramterabstract	2
824	particularly	2
825	performed	2
826	perhaps	2
827	positional	2
828	potentially	2
829	practical	2
830	predict	2
831	predicting	2
832	prepared	2
833	promising	2
834	purposes	2
835	ramp	2
836	rarely	2
837	rates	2
838	realistic	2
839	reason	2
840	reasonable	2
841	recall	2
842	recently	2
843	regressive	2
844	reliably	2
845	remains	2
846	remove	2
847	require	2
848	research	2
849	resp	2
850	rewrite	2
851	rict	2
852	ronen	2
853	rotary	2
854	rounds	2
855	rumored	2
856	safe	2
857	say	2
858	sbananaask	2
859	sees	2
860	selecting	2
861	shifts	2
862	shizhuo	2
863	short	2
864	shot	2
865	showcase	2
866	simple	2
867	simulate	2
868	single	2
869	skills	2
870	slightly	2
871	smaller	2
872	solely	2
873	sometimes	2
874	sop	2
875	source	2
876	soy	2
877	special	2
878	stands	2
879	starting	2
880	strongly	2
881	structures	2
882	studies	2
883	study	2
884	summarize	2
885	support	2
886	sure	2
887	teaches	2
888	team	2
889	technique	2
890	tempting	2
891	tend	2
892	testing	2
893	text	2
894	therefore	2
895	three	2
896	throughout	2
897	tian	2
898	train	2
899	transformer	2
900	truth	2
901	tuned	2
902	typically	2
903	underlined	2
904	unlike	2
905	unlikely	2
906	until	2
907	url	2
908	usefulness	2
909	user	2
910	usually	2
911	various	2
912	verifiers	2
913	volume	2
914	well	2
915	wide	2
916	widely	2
917	window	2
918	words	2
919	workshop	2
920	world	2
921	wu	2
922	yifei	2
923	you	2
924	zeqi	2
925	zicheng	2

ひさびさにdocker(186)使う
https://qiita.com/kaizen_nagoya/items/e29cbaed8370e7913487
のawkで処理。

PDFからTXTにする際に、複合語が分離できていない課題がある。理由は不明。