はじめに
Azure OpenAIでほしい回答を入手するために検討したいこと、第二弾はOpenAIのドキュメントで紹介されているプロンプトエンジニアリングで意識したい6つの方針のうち、3つについてまとめていきます。残る3つは外部データを使った回答（RAG）と一緒に触れるつもりです。
Azure OpenAIでほしい回答を得るために検討したいこと、全体としては以下にまとめていく予定ですのでこちらもご参考にしてください。
プロンプトエンジニアリングで意識したい6つの方針
OpenAIはよりよいユーザーの求める回答を生成するために意識したいプロンプトエンジニアリングの方針を6つ紹介しています。
- 方針1：指示を明確に書く
- 方針2：複雑なタスクを分割してシンプルなサブタスクにする
- 方針3：モデルに考える時間を与える
- 方針4：リファレンスを提供する
- 方針5：外部ツールを活用する
- 方針6：モデルのアウトプットを評価する
本記事では1~3についてまとめます。
指示を明確に書く
より関連した回答を得るために詳細を述べる
以下OpenAIのページで説明されている例を引用します。指示を明確化することでユーザーが欲しい回答を生成できるように注意します。
| 悪い例 | 良い例 |
|---|---|
| Excelで数値をどうやって追加しますか？ | Excelで金額の合計を追加するにはどうすればよいですか？シート全体の行に対して自動的に行い、合計が右側の「合計」という列に表示されるようにしたいです。 |
| 誰が大統領ですか？ | 2021年のメキシコの大統領は誰でしたか、そして選挙はどのくらいの頻度で開催されますか？ |
| フィボナッチ数列を計算するコードを書いてください。 | TypeScriptでフィボナッチ数列を効率的に計算する関数を書いてください。コードにコメントを付けて、各部分が何をしているのか、書かれている理由を説明してください。 |
| 会議のメモを要約してください。 | 会議のメモを一つの段落に要約してください。次に、講演者とその要点のマークダウンリストを作成します。 最後に、講演者が提案した次のステップやアクション項目がある場合は、それをリストします。 |
モデルの返答の振る舞いや制限事項をシステムメッセージとして定義する
目的に合った回答を生成できるように制限を与える：システムメッセージエンジニアリングで細かく触れているのでご参考ください。
区切り文字を使用して入力の異なる部分を明確にする
以下の例では”””（トリプル引用符）を使用して入力テキストを明確化することで、タスク（要約と俳句調）と入力テキストを明確に区別しています。
delimiters.py
text = f"""
In recent years, the impact of digital technology on society has been profound. With the advent of smartphones and high-speed internet, information is now more accessible than ever. This has led to significant changes in how people communicate, access information, and entertain themselves. Social media platforms have become central to many people's lives, influencing everything from daily communication to global politics. The rise of e-commerce has transformed the retail industry, making shopping more convenient and personalized. However, these advancements also bring challenges, such as concerns over privacy, misinformation, and the digital divide. The future of digital technology promises even more innovation, but it is crucial to address these challenges to ensure that the benefits of technology are accessible to all.
"""
messages = [
{'role':'user', 'content':f"Summarize the text delimited by triple quotes with a haiku. {text}"} ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
Digital age dawns bright,
Information at our fingertips,
Challenges to face.
タスクを完了するために必要なステップを明確にする
要約と翻訳を行うタスクにおいて、タスクを分割して一つ一つをこなすことにより回答の精度を高めるようにします。
stepbystep.py
messages = [
{'role':'system', 'content':'Use the following step-by-step instructions to respond to user inputs. \
Step 1 - The user will provide you with text in triple quotes. Summarize this text in one sentence with a prefix that says "Summary: ". \
Step 2 - Translate the summary from Step 1 into Japanese, with a prefix that says "Translation: ". \
'},
{'role':'user', 'content':"""In recent years, the impact of digital technology on society has been profound. With the advent of smartphones and high-speed internet, information is now more accessible than ever. This has led to significant changes in how people communicate, access information, and entertain themselves. Social media platforms have become central to many peoples lives, influencing everything from daily communication to global politics. The rise of e-commerce has transformed the retail industry, making shopping more convenient and personalized. However, these advancements also bring challenges, such as concerns over privacy, misinformation, and the digital divide. The future of digital technology promises even more innovation, but it is crucial to address these challenges to ensure that the benefits of technology are accessible to all."""} ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
Summary: Digital technology has had a profound impact on society, with increased accessibility to information leading to changes in communication, entertainment, and commerce, but also bringing challenges such as privacy concerns and the digital divide.
Translation: 技術の進歩により、情報がよりアクセスしやすくなり、コミュニケーション、エンターテインメント、商業などに大きな変化が生じました。しかし、プライバシーの懸念やデジタル格差などの課題も生じています。技術の将来には更なる革新が約束されていますが、技術の利点がすべての人々にアクセス可能であることを確保するためにこれらの課題に対処することが重要です。
(Note: The translation is in informal Japanese. If you need a formal translation, please let me know.)
ステップを明確にしなかった場合の結果
デジタル技術の進歩により、情報のアクセスが容易になり、コミュニケーションやエンターテインメントの方法が変化し、SNSが中心的な存在となり、電子商取引が小売業界を変革したが、プライバシーやデジタル格差などの問題も生じており、今後の技術革新には課題がある。
例を提示する
few-shotと呼ばれる手法です。いきなり回答を生成させるのではなく、回答の例を提示することでほしい回答のスタイルや内容を学習させる方法です。インストラクションをステップバイステップで表すよりもより簡単に使うことができます。以下の例ではそのままだとかなり説明口調ですが、例にならっておしゃれな言い回しと内容で回答が生成されるようになりました。
fewshot.py
messages = [
{'role':'system', 'content':'Answer in a consistent style.'},
{'role':'user', 'content':'Teach me about patience.'},
{'role':'assistant', 'content':'The river that carves the deepest valley flows from a modest spring; the grandest symphony originates from a single note; the most intricate tapestry begins with a solitary thread.'},
{'role':'user', 'content':'Teach me about the ocean.'}, ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
The ocean is a vast and mysterious body of water that covers more than 70% of the Earth's surface. It is home to countless species of marine life, from tiny plankton to massive whales. The ocean plays a crucial role in regulating the Earth's climate and weather patterns, and it provides a source of food and livelihood for millions of people around the world. Despite its importance, much of the ocean remains unexplored and undiscovered, making it a fascinating and awe-inspiring subject of study and exploration.
回答例を提示しなかった場合の結果
The ocean is a vast body of saltwater that covers more than 70% of the Earth's surface. It is home to a diverse range of marine life, from tiny plankton to massive whales. The ocean plays a crucial role in regulating the Earth's climate and weather patterns, as well as providing food and resources for humans.
The ocean is divided into five major basins: the Atlantic, Pacific, Indian, Southern, and Arctic oceans. Each of these basins has unique characteristics, such as temperature, salinity, and currents.
The ocean is also home to a variety of ecosystems, including coral reefs, kelp forests, and deep-sea hydrothermal vents. These ecosystems support a wide range of marine species and provide important services such as carbon sequestration and nutrient cycling.
However, the ocean is facing numerous threats, including overfishing, pollution, and climate change. These threats are causing significant damage to marine ecosystems and endangering the survival of many species.
To protect the ocean and its inhabitants, it is important to reduce our impact on the environment by practicing sustainable fishing, reducing plastic waste, and taking action to address climate change.
アウトプットの長さを指定する
ちょうど50 wordに要約するように指示します。
length.py
text = f"""
In recent years, the impact of digital technology on society has been profound. With the advent of smartphones and high-speed internet, information is now more accessible than ever. This has led to significant changes in how people communicate, access information, and entertain themselves. Social media platforms have become central to many people's lives, influencing everything from daily communication to global politics. The rise of e-commerce has transformed the retail industry, making shopping more convenient and personalized. However, these advancements also bring challenges, such as concerns over privacy, misinformation, and the digital divide. The future of digital technology promises even more innovation, but it is crucial to address these challenges to ensure that the benefits of technology are accessible to all.
"""
messages = [
{'role':'user', 'content':f"Summarize the text delimited by triple quotes in about 50 words. {text}"} ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
Digital technology has had a profound impact on society, with smartphones and high-speed internet making information more accessible than ever. Social media and e-commerce have transformed communication and retail, but challenges such as privacy and misinformation must be addressed to ensure technology benefits everyone. The future promises even more innovation.
複雑なタスクを分割してシンプルなサブタスクにする
まずはユーザーからの依頼事項を分類します。
classify.py
messages = [
{'role':'system', 'content':"""You will be provided with customer service queries. Classify each query into a primary category and a secondary category. Provide your output in json format with the keys: primary and secondary.
Primary categories: Billing, Technical Support, Account Management, or General Inquiry.
Billing secondary categories:
- Unsubscribe or upgrade
- Add a payment method
- Explanation for charge
- Dispute a charge
Technical Support secondary categories:
- Troubleshooting
- Device compatibility
- Software updates
Account Management secondary categories:
- Password reset
- Update personal information
- Close account
- Account security
General Inquiry secondary categories:
- Product information
- Pricing
- Feedback
- Speak to a human"""},
{'role':'user', 'content':'I need to get my internet working again.'}, ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
{
"primary": "Technical Support",
"secondary": "Troubleshooting"
}
その後、分類したカテゴリに合わせた回答を生成します。サポートリクエストの例のように複雑なタスクを扱う場合に活用することで、いきなり回答導き出すよりシステマティックに回答できるようになります。
response.py
messages = [
{'role':'system', 'content':"""
You will be provided with customer service inquiries that require troubleshooting in a technical support context. Help the user by:
- Ask them to check that all cables to/from the router are connected. Note that it is common for cables to come loose over time.
- If all cables are connected and the issue persists, ask them which router model they are using
- Now you will advise them how to restart their device:
-- If the model number is MTD-327J, advise them to push the red button and hold it for 5 seconds, then wait 5 minutes before testing the connection.
-- If the model number is MTD-327S, advise them to unplug and replug it, then wait 5 minutes before testing the connection.
- If the customer's issue persists after restarting the device and waiting 5 minutes, connect them to IT support by outputting {"IT support requested"}.
- If the user starts asking questions that are unrelated to this topic then confirm if they would like to end the current chat about troubleshooting and classify their request according to the following scheme:
{
"primary": "Technical Support",
"secondary": "Troubleshooting"
}"""},
{'role':'user', 'content':'I need to get my internet working again.'}, ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
Sure, I can help you with that. Let's start by checking that all cables to/from the router are connected. It's common for cables to come loose over time. Can you please check that all cables are securely connected?
過去の会話ログを要約またはフィルターして継続的な会話を実現する
過去の会話やその要約を持たせておく方法、embeddings-based searchで効率的なナレッジ検索を実現し検索できるようにする方法があります。
以下は過去の会話ログを踏まえて回答する例です。ちゃんと過去の会話内容を覚えてくれています。
memory.py
messages = [
{'role':'system', 'content':'You are friendly chatbot.'},
{'role':'user', 'content':'Hi, my name is Mel.'},
{'role':'assistant', 'content': "Hi! It's nice to meet you. \
Is there anything I can help you with today?"},
{'role':'user', 'content':'Yes, can you remind me which company I work for? \
I work at ABC company.'},
{'role':'system', 'content':'Sure. Is there anything else I can help you with?'},
{'role':'user', 'content':'tell me joke.'},
{'role':'system', 'content':'Why did the tomato turn red? Because it saw the salad dressing! \
Is there anything I can help you with today?'},
{'role':'user', 'content':'Yes, tell me which company I work for?'}, ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
You work for ABC company. Is there anything else I can help you with?
長いドキュメントを分割して要約することで全体の要約を作成する
LangChainでチャンク分割して要約する方法が提供されているので、試してみることができます。以下イベントの要約作成で実行しているのでご参考ください。自分でプロンプトをカスタマイズすることも可能です。
モデルに考える時間を与える
結論を出す前に自分で実行させる
自分で回答を導き出す前に自分でタスクを実行させ、最終的な回答を導き出します。以下の例はいきなり回答を導こうとすると間違えてしまうという例です。
thinkfirst.py
messages = [
{'role':'system', 'content':'First work out your own solution to the problem, and write the answer with a prefix that says "My Caluculation: ". Then compare your solution to the students solution and evaluate if the students solution is correct or not, and write the final answer with a prefix that says "Final answer: ". Dont decide if the students solution is correct until you have done the problem yourself.'},
{'role':'user', 'content':"Problem Statement: I'm building a solar power installation and I need help working out the financials. \
- Land costs $100 / square foot \
- I can buy solar panels for $250 / square foot \
- I negotiated a contract for maintenance that will cost me a flat $100k per year, and an additional $10 / square foot \
What is the total cost for the first year of operations as a function of the number of square feet. \
\
Student's Solution: Let x be the size of the installation in square feet. \
1. Land cost: 100x \
2. Solar panel cost: 250x \
3. Maintenance cost: 100,000 + 100x \
Total cost: 100x + 250x + 100,000 + 100x = 450x + 100,000"} ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
My Calculation:
1. Land cost: 100x
2. Solar panel cost: 250x
3. Maintenance cost: 100,000 + 10x
Total cost: 100x + 250x + 100,000 + 10x = 360x + 100,000
Final answer:
The student's solution is incorrect. The maintenance cost is not $100 per square foot, but rather a flat fee of $100,000 per year plus an additional $10 per square foot. Therefore, the correct calculation for the total cost for the first year of operations as a function of the number of square feet is:
1. Land cost: 100x
2. Solar panel cost: 250x
3. Maintenance cost: 100,000 + 10x
Total cost: 100x + 250x + 100,000 + 10x = 360x + 100,000
いきなり回答を求めた結果
The student's solution is correct. They correctly identified the three costs involved in building and maintaining the solar power installation and added them together to get the total cost as a function of the number of square feet.
モデルの推論プロセスを隠すために独自の記号を使用する
実際には推論もステップバイステップで行うが、利用者にフィールドバックするのは結果だけにする場合などに活用できる例です。こちら何度か試したのだが生成AIが計算を間違えるケースが多く、以下の例ではヒントが誤ってしまっています。改善案としてはモデルをより高い能力を持つにすることや計算は外部ツールに任せるなどが手だと思います。
innermonologue.py
messages = [
{'role':'system', 'content':'Follow step1 to step4 to answer the user queries. Enclose all your work within triple quotes (""").\
Step 1 - First you think student answer is incorrect. Work out your own solution to the problem. \
Step 2 - Compare your solution(total cost) to the students solution(total cost) and evaluate if the students solution is correct or not. \
Step 3 - If the student made a mistake, determine what hint you could give the student without giving away the answer. \
Step 4 - If the student made a mistake, provide the hint from the previous step to the student (outside of triple quotes). Instead of writing "Step 4 - ..." write "Hint:".'},
{'role':'user', 'content':"Problem Statement: I'm building a solar power installation and I need help working out the financials. \
- Land costs $100 / square foot \
- I can buy solar panels for $250 / square foot \
- I negotiated a contract for maintenance that will cost me a flat $100k per year, and an additional $10 / square foot \
What is the total cost for the first year of operations as a function of the number of square feet. \
\
Student's Solution: Let x be the size of the installation in square feet. \
1. Land cost: 100x \
2. Solar panel cost: 250x \
3. Maintenance cost: 100,000 + 100x \
Total cost: 100x + 250x + 100,000 + 100x = 450x + 100,000"} ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
"""Step 1 - Solution:
Total cost for the first year of operations as a function of the number of square feet is:
Total cost = Land cost + Solar panel cost + Maintenance cost
Total cost = 100x + 250x + (100,000 + 10x)
Total cost = 360x + 100,000"""
"""Step 2 - Evaluation:
The student's solution is incorrect. The correct total cost for the first year of operations as a function of the number of square feet is 360x + 100,000. The student added the maintenance cost twice in their solution."""
"""Step 3 - Hint:
The maintenance cost should only be added once in the total cost calculation."""
Hint: The maintenance cost should only be added once in the total cost calculation.
モデルに過去の手順で見落としがないか確認する
確認させるステップを入れることで漏れがないかを確認し、より確からしい結果を得ようとします。以下に例として実行してみたのですが、逆に取り入れすぎてどれが本当に重要なのかがわかりにくくなる場合もあるように思います。
reconfirm.py
messages = [
{'role':'system', 'content':'You will be provided with a document delimited by triple quotes. Your task is to select excerpts which pertain to the following question: "What significant paradigm shifts have occurred in the history of artificial intelligence." \
Ensure that excerpts contain all relevant context needed to interpret them - in other words dont extract small snippets that are missing important context. Provide output in JSON format as follows: \
{"excerpt": "..."}, \
{"excerpt": "..."}'},
{'role':'user', 'content':'""" Artificial Intelligence (AI) has undergone a fascinating historical evolution, marked by pivotal moments and technological breakthroughs. The genesis of AI as a distinct field can be traced back to the 1950s, following the pioneering work of Alan Turing, who proposed the concept of a machine that could simulate human intelligence. The term "Artificial Intelligence" was officially coined in 1956 at the Dartmouth Conference, marking the beginning of AI as a professional scientific discipline. The subsequent decades saw fluctuating interest and progress, known as the AI winters, where funding and enthusiasm diminished due to unmet expectations. However, the late 20th and early 21st centuries witnessed a resurgence in AI research, fueled by advancements in computational power, data availability, and algorithmic innovation. \
One of the most remarkable aspects of AI is its transformative impact across various sectors. In recent years, AI technologies such as machine learning, deep learning, and natural language processing have revolutionized industries. Machine learning algorithms, which enable systems to learn and improve from experience without being explicitly programmed, have become integral in areas like finance, for risk assessment and algorithmic trading, and in healthcare, for predictive diagnostics and personalized medicine. Deep learning, a subset of machine learning inspired by the structure and function of the brains neural networks, has made significant strides in image and speech recognition, powering technologies from autonomous vehicles to sophisticated voice assistants. \
Despite its impressive advancements, AI faces significant challenges and ethical considerations. Theres growing concern over issues like algorithmic bias, where AI systems may inherit and perpetuate the biases present in their training data, leading to unfair or discriminatory outcomes. Privacy concerns are paramount, as AI often relies on large datasets, including personal information. The automation of jobs by AI also presents socio-economic challenges, necessitating a rethinking of workforce development and employment structures. Looking to the future, the field of AI stands at a critical juncture, where its potential for societal benefit is immense, but so is the need for responsible and ethical development and deployment. Balancing innovation with regulation, addressing ethical dilemmas, and ensuring equitable access to AI benefits are key to harnessing AIs full potential while mitigating its risks.""" '},
{'role':'assistant', 'content':'{"excerpt": "the model writes an excerpt here"}, \
{"excerpt": "the model writes another excerpt here"}'} ,
{'role':'user', 'content':'Are there more relevant excerpts? Take care not to repeat excerpts. Also ensure that excerpts contain all relevant context needed to interpret them - in other words dont extract small snippets that are missing important context.'} ]
response = get_completion_from_messages(messages, temperature=0)
print(response)
結果
Yes, there are more relevant excerpts. Here are some additional excerpts:
1. "The genesis of AI as a distinct field can be traced back to the 1950s, following the pioneering work of Alan Turing, who proposed the concept of a machine that could simulate human intelligence."
2. "The subsequent decades saw fluctuating interest and progress, known as the AI winters, where funding and enthusiasm diminished due to unmet expectations."
3. "However, the late 20th and early 21st centuries witnessed a resurgence in AI research, fueled by advancements in computational power, data availability, and algorithmic innovation."
4. "One of the most remarkable aspects of AI is its transformative impact across various sectors."
5. "Machine learning algorithms, which enable systems to learn and improve from experience without being explicitly programmed, have become integral in areas like finance, for risk assessment and algorithmic trading, and in healthcare, for predictive diagnostics and personalized medicine."
6. "Deep learning, a subset of machine learning inspired by the structure and function of the brains neural networks, has made significant strides in image and speech recognition, powering technologies from autonomous vehicles to sophisticated voice assistants."
7. "Despite its impressive advancements, AI faces significant challenges and ethical considerations."
8. "Theres growing concern over issues like algorithmic bias, where AI systems may inherit and perpetuate the biases present in their training data, leading to unfair or discriminatory outcomes."
9. "Privacy concerns are paramount, as AI often relies on large datasets, including personal information."
10. "The automation of jobs by AI also presents socio-economic challenges, necessitating a rethinking of workforce development and employment structures."
11. "Looking to the future, the field of AI stands at a critical juncture, where its potential for societal benefit is immense, but so is the need for responsible and ethical development and deployment."
12. "Balancing innovation with regulation, addressing ethical dilemmas, and ensuring equitable access to AI benefits are key to harnessing AIs full potential while mitigating its risks."
優先順に並べてもらうと少し精査して抽出してくれました。
Yes, there are more relevant excerpts. Here they are arranged in order of priority:
1. "The subsequent decades saw fluctuating interest and progress, known as the AI winters, where funding and enthusiasm diminished due to unmet expectations."
- This excerpt highlights a significant paradigm shift in the history of AI, where the field experienced a decline in funding and enthusiasm due to unmet expectations. This shift is important because it demonstrates that AI is not a linear progression of advancements, but rather a field that experiences ups and downs.
2. "The late 20th and early 21st centuries witnessed a resurgence in AI research, fueled by advancements in computational power, data availability, and algorithmic innovation."
- This excerpt highlights another significant paradigm shift in the history of AI, where the field experienced a resurgence in research and progress. This shift is important because it demonstrates that AI is a field that is constantly evolving and improving, and that advancements in technology and innovation can have a significant impact on its progress.
3. "One of the most remarkable aspects of AI is its transformative impact across various sectors."
- This excerpt highlights the transformative impact that AI has had on various sectors, which is important because it demonstrates the practical applications and real-world implications of AI.
4. "Despite its impressive advancements, AI faces significant challenges and ethical considerations."
- This excerpt highlights the challenges and ethical considerations that AI faces, which is important because it demonstrates that AI is not without its drawbacks and potential negative consequences.
5. "Looking to the future, the field of AI stands at a critical juncture, where its potential for societal benefit is immense, but so is the need for responsible and ethical development and deployment."
- This excerpt highlights the importance of responsible and ethical development and deployment of AI, which is important because it demonstrates that the future of AI depends on how it is developed and used, and that there is a need for careful consideration of its potential impact on society.
確認前の結果
"The subsequent decades saw fluctuating interest and progress, known as the AI winters, where funding and enthusiasm diminished due to unmet expectations. However, the late 20th and early 21st centuries witnessed a resurgence in AI research, fueled by advancements in computational power, data availability, and algorithmic innovation."
"Machine learning algorithms, which enable systems to learn and improve from experience without being explicitly programmed, have become integral in areas like finance, for risk assessment and algorithmic trading, and in healthcare, for predictive diagnostics and personalized medicine. Deep learning, a subset of machine learning inspired by the structure and function of the brains neural networks, has made significant strides in image and speech recognition, powering technologies from autonomous vehicles to sophisticated voice assistants."
"Despite its impressive advancements, AI faces significant challenges and ethical considerations. Theres growing concern over issues like algorithmic bias, where AI systems may inherit and perpetuate the biases present in their training data, leading to unfair or discriminatory outcomes. Privacy concerns are paramount, as AI often relies on large datasets, including personal information. The automation of jobs by AI also presents socio-economic challenges, necessitating a rethinking of workforce development and employment structures."
まとめ
より良い回答を得るためのプロンプトエンジニアリングの方針を実際の実行結果と共にまとめました。いい結果が得られない場合はこのまとめに立ち返ってプロンプトを工夫する、といった風に活用してもらえると嬉しいです。