こちらの記事を翻訳しました。
次にこちらのサンプルをウォークスルーします。
サンプルのウォークスルー
この記事では、DSPyライブラリを活用する最小のサンプルをウォークスルーします。
DSPyにおけるプロンプティングタスクをシミュレーションするために、GSM8KデータセットとOpenAI GPT-3.5-turboモデルを活用します。
Minimal Working Example | DSPy
%pip install dspy-ai
dbutils.library.restartPython()
セットアップ
サンプルに踏み込む前に、環境が適切に設定されていることを確認しましょう。必要なモジュールをインポートし、言語モデルを設定することからスタートしましょう。
import os
os.environ["OPENAI_API_KEY"] = dbutils.secrets.get("demo-token-takaaki.yayoi", "openai_api_key")
import dspy
from dspy.datasets.gsm8k import GSM8K, gsm8k_metric
# LMのセットアップ
turbo = dspy.OpenAI(model='gpt-3.5-turbo-instruct', max_tokens=250)
dspy.settings.configure(lm=turbo)
# GSM8Kデータセットから数学の質問をロード
gsm8k = GSM8K()
gsm8k_trainset, gsm8k_devset = gsm8k.train[:10], gsm8k.dev[:10]
モジュールの定義
環境をセットアップしたら、回答を生成するためにステップバイステップの理由づけを行うために、ChainOfThoughtモジュールを活用するカスタムプログラムを定義しましょう。
class CoT(dspy.Module):
def __init__(self):
super().__init__()
self.prog = dspy.ChainOfThought("question -> answer")
def forward(self, question):
return self.prog(question=question)
モデルのコンパイルと評価
シンプルなプログラムを準備したら、BootstrapFewShotWithRandomSearchテレプロンプターを用いた最適化に進みましょう。
from dspy.teleprompt import BootstrapFewShot
# オプティマイザのセットアップ: 我々のCoTプログラムに4-shotの例を"bootstrap"(自己生成)させたいと考えています
config = dict(max_bootstrapped_demos=4, max_labeled_demos=4)
# 最適化！ここでは、`gsm8k_metric`を用います。通常、このメトリックはオプティマイザにどれだけうまく行っているのかを伝えることになります
teleprompter = BootstrapFewShot(metric=gsm8k_metric, **config)
optimized_cot = teleprompter.compile(CoT(), trainset=gsm8k_trainset, valset=gsm8k_devset)
50%|█████ | 5/10 [00:06<00:06, 1.32s/it]
Bootstrapped 4 full traces after 6 examples in round 0.
評価
コンパイルした(最適化された)DSPyプログラムを入手したので、devデータセットに対するパフォーマンスの評価に進みましょう。
from dspy.evaluate import Evaluate
# 複数回使用できるエバリュエーターをセットアップします
evaluate = Evaluate(devset=gsm8k_devset, metric=gsm8k_metric, num_threads=4, display_progress=True, display_table=0)
# 我々の`optimized_cot`プログラムを評価します
evaluate(optimized_cot)
Average Metric: 6 / 10 (60.0): 100%|██████████| 10/10 [00:04<00:00, 2.24it/s]
Average Metric: 6 / 10 (60.0%)
60.0
モデルの履歴の調査
モデルとのインタラクションをより理解するために、モデルの履歴の調査を通じて最新の生成結果を確認することができます。
turbo.inspect_history(n=1)
Given the fields `question`, produce the fields `answer`.
---
Follow the following format.
Question: ${question}
Reasoning: Let's think step by step in order to ${produce the answer}. We ...
Answer: ${answer}
---
Question: The result from the 40-item Statistics exam Marion and Ella took already came out. Ella got 4 incorrect answers while Marion got 6 more than half the score of Ella. What is Marion's score?
Reasoning: Let's think step by step in order to find Marion's score. We know that Ella got 4 incorrect answers, which means she got 36 correct answers out of 40. We also know that Marion got 6 more than half of Ella's score, which is 6 more than 36/2 = 18. Therefore, Marion's score is 18 + 6 = 24.
Answer: 24
---
Question: Bridget counted 14 shooting stars in the night sky. Reginald counted two fewer shooting stars than did Bridget, but Sam counted four more shooting stars than did Reginald. How many more shooting stars did Sam count in the night sky than was the average number of shooting stars observed for the three of them?
Reasoning: Let's think step by step in order to find the answer. First, we know that Bridget counted 14 shooting stars. Then, we know that Reginald counted two fewer than Bridget, so he counted 12 shooting stars. Finally, we know that Sam counted four more than Reginald, so he counted 16 shooting stars. To find the average number of shooting stars, we add up the total number of shooting stars (14 + 12 + 16 = 42) and divide by the number of people (3). This gives us an average of 14 shooting stars per person. Since Sam counted 16 shooting stars, he counted 2 more than the average.
Answer: 2
---
Question: Sarah buys 20 pencils on Monday. Then she buys 18 more pencils on Tuesday. On Wednesday she buys triple the number of pencils she did on Tuesday. How many pencils does she have?
Reasoning: Let's think step by step in order to find the total number of pencils Sarah has. First, we know that she bought 20 pencils on Monday and 18 more on Tuesday, so she has 20 + 18 = 38 pencils. Then, on Wednesday she bought triple the number of pencils she did on Tuesday, which is 3 * 18 = 54 pencils. Therefore, the total number of pencils Sarah has is 38 + 54 = 92.
Answer: 92
---
Question: Rookie police officers have to buy duty shoes at the full price of $85, but officers who have served at least a year get a 20% discount. Officers who have served at least three years get an additional 25% off the discounted price. How much does an officer who has served at least three years have to pay for shoes?
Reasoning: Let's think step by step in order to calculate the final price. First, we calculate the discounted price for an officer who has served at least a year. This is 85 * 0.8 = 68. Then, we calculate the additional discount for an officer who has served at least three years. This is 68 * 0.75 = 51. Finally, the officer has to pay this final price of 51.
Answer: $51
---
Question: Trey is raising money for a new bike that costs $112. He plans to spend the next two weeks selling bracelets for $1 each. On average, how many bracelets does he need to sell each day?
Reasoning: Let's think step by step in order to find the average number of bracelets Trey needs to sell each day. First, we need to find the total amount of money Trey needs to raise, which is $112. Then, we divide this amount by the price of each bracelet, which is $1. This gives us 112 bracelets. Since Trey has two weeks to sell these bracelets, he needs to sell 112 / 14 = 8 bracelets per day on average.
Answer: 8