LLMを動かしながらプロンプトエンジニアリングを試してみました。超軽量のflan-t5-baseを使って文章要約タスクを試しています。パラメータ数が248Mと少なく、手軽に動かすには都合が良かったです。その反面日本語ではプロンプトできないですし、精度も良くないです。Google ColabのCPUで動かしています。
実行環境
2024/3/26時点のGoogle Colab で、Python3.10.12で、以下のパッケージを使っています。
| Package | Version | 備考 |
|---|---|---|
| torch | 2.1.0+cu121 | Google Colabにプリインストールされていたバージョン |
| transformers | 4.38.2 | Google Colabにプリインストールされていたバージョン |
| datasets | 2.18.0 | 追加でpip install |
サマリ
以下のダイアログ要約をパターン変えてさせています。
knkarthick/dialogsum test-11
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
正解ラベルと各パターン別結果
正解ラベルと、各試行パターンごとの生成値です(tempertureはdefault1)。
モデルがあんまり良くないので0 shotと 1 shotで違いが微妙だし、2shotでも大して良くない。
tempertureを2にすると滅茶苦茶になりますね。
| 種別 | 結果 |
|---|---|
| 正解ラベル | #Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming. |
| 0 shot without instruction | Brian, thank you for coming to our party. |
| 0 shot | Brian's birthday is coming up. |
| 1 shot | Brian's birthday is coming up. |
| 2 shot(temperture=1) | Brian's birthday is coming up. Brian wants to have a dance with Brian. |
| 2 shot(temperture=0.1) | Brian's birthday is coming up. Brian will have a dance with Brian. |
| 2 shot(temperture=0.5) | Brian's birthday is coming up. Brian invites everyone to come over and have a dance. |
| 2 shot(temperture=2) | #p and Others enjoy and love a private dance, with dancing of celebrities' ward. |
実行プログラム
1. pip intall
Google Colab にプレインストールされていないdatasetsをインストール。
!pip install datasets
2. Impot packages
PythonパッケージをImport。AutoModelForSeq2SeqLM(Encoder-Decoder)は珍しく、GPTなどDecoder-onlyが主流(その場合はAutoModelForCausalLM)。
rom datasets import load_dataset
# AutoModelForSeq2SeqLM は Encoder-decoder用。 AutoModelForCausalLM だと auto-regressive(Decoder-only)
from transformers import AutoModelForSeq2SeqLM
from transformers import AutoTokenizer
from transformers import GenerationConfig
3. Load Data
以下のdialogsumのデータを使っています。ダイアログと要約のデータがあります。
データをロードします。
dataset = load_dataset('knkarthick/dialogsum')
print(dataset)
出力結果
DatasetDict({
train: Dataset({
features: ['id', 'dialogue', 'summary', 'topic'],
num_rows: 12460
})
validation: Dataset({
features: ['id', 'dialogue', 'summary', 'topic'],
num_rows: 500
})
test: Dataset({
features: ['id', 'dialogue', 'summary', 'topic'],
num_rows: 1500
})
})
データの中身を見てみます。
# index 1, 2は同じダイアログだったので離れたIndexにした
examples = [1, 10]
dash_line = '-'*100
def output_prompt(dataset, i, prompt):
print(dash_line)
print('例: ', i + 1)
print(dash_line)
print('プロンプト:')
print(prompt)
print(dash_line)
def output_result(y_true, output=None):
print('正解ラベル:')
print(y_true)
print(dash_line)
if output != None:
print(f'モデル生成結果:\n{output}\n')
print()
for i in examples:
output_prompt(dataset, i, dataset['test'][i]['dialogue'])
output_result(dataset['test'][i]['summary'])
出力結果
----------------------------------------------------------------------------------------------------
例: 2
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.
----------------------------------------------------------------------------------------------------
正解ラベル:
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
----------------------------------------------------------------------------------------------------
----------------------------------------------------------------------------------------------------
例: 11
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
4. Load model
Flan-T5 releaseのbaseサイズを使いました。パラメータ数は248Mで昨今のLLMに比べると非常に少ないです。
# https://huggingface.co/google/flan-t5-small
# use Encoder-decoder small model
#model_name='google/flan-t5-small'
model_name='google/flan-t5-base'
#model_name='google/flan-t5-large'
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
Tokenizerを少し試してみます。
# 日本語で試したが当然、Encodeできない
sentence = "What time is it, Tom?"
# PyTorchのテンソル型で返すためにreturn_tensors='pt'
sentence_encoded = tokenizer(sentence, return_tensors='pt')
sentence_decoded = tokenizer.decode(
sentence_encoded["input_ids"][0],
skip_special_tokens=True
)
print(f'{tokenizer.max_model_input_sizes=}')
print('ENCODED SENTENCE:')
print(sentence_encoded["input_ids"][0])
print('\nENCODED Tokens:')
print(tokenizer.convert_ids_to_tokens(sentence_encoded["input_ids"][0]))
print('\nDECODED SENTENCE:')
print(sentence_decoded)
出力結果
tokenizer.max_model_input_sizes={'google-t5/t5-small': 512, 'google-t5/t5-base': 512, 'google-t5/t5-large': 512, 'google-t5/t5-3b': 512, 'google-t5/t5-11b': 512}
ENCODED SENTENCE:
tensor([ 363, 97, 19, 34, 6, 3059, 58, 1])
ENCODED Tokens:
['▁What', '▁time', '▁is', '▁it', ',', '▁Tom', '?', '</s>']
DECODED SENTENCE:
What time is it, Tom?
5. Prompt Engineering
ここから本題です。
事前にサマリを生成する関数を定義しておきます。
def generate_summary(tokenizer, prompt, config=GenerationConfig(max_new_tokens=50)):
inputs = tokenizer(prompt, return_tensors='pt')
output = tokenizer.decode(
model.generate(
inputs["input_ids"],
max_new_tokens=50,
generation_config=config
)[0],
skip_special_tokens=True
)
return output
5.1. Zero-Shot without instruction
Zero-ShotでインストラクションなしでLLMに投げます。
for i in examples:
dialogue = dataset['test'][i]['dialogue']
output = generate_summary(tokenizer, dialogue)
output_prompt(dataset, i, dialogue)
output_result(dataset['test'][i]['summary'], output)
出力結果
----------------------------------------------------------------------------------------------------
例: 2
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.
----------------------------------------------------------------------------------------------------
正解ラベル:
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
----------------------------------------------------------------------------------------------------
モデル生成結果:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
----------------------------------------------------------------------------------------------------
例: 11
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian, thank you for coming to our party.
5.2. Zero-Shot with instruction
Zero-Shotでインストラクションとして「What was going on?」を付加してみます。もう少しいいインストラクションあるかもしれませんが、試していません。
for i in examples:
dialogue = dataset['test'][i]['dialogue']
prompt = f"""
Dialogue:
{dialogue}
What was going on?
"""
output = generate_summary(tokenizer, prompt)
output_prompt(dataset, i, prompt)
output_result(dataset['test'][i]['summary'], output)
出力結果
----------------------------------------------------------------------------------------------------
例: 2
----------------------------------------------------------------------------------------------------
Dialogue:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.
What was going on?
----------------------------------------------------------------------------------------------------
正解ラベル:
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
----------------------------------------------------------------------------------------------------
モデル生成結果:
The memo is to be distributed to all employees by this afternoon.
----------------------------------------------------------------------------------------------------
例: 11
----------------------------------------------------------------------------------------------------
プロンプト:
Dialogue:
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
What was going on?
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up.
長いので、Instruction有無でどう変わったか改めて表にします。少し良くなっていますね。
| 例 | Without Instruction | With Instruction |
|---|---|---|
| 2 | #Person1#: Ms. Dawson, I need you to take a dictation for me. | The memo is to be distributed to all employees by this afternoon. |
| 11 | Brian, thank you for coming to our party. | Brian's birthday is coming up. |
5.3. One-Shot
次はOne-Shotで試してみます。
def make_prompt(example_indices_full, example_index_to_summarize):
prompt = ''
for index in example_indices_full:
# '{summary}の後の\n\n\n'がFLAN-T5にとっては重要らしい
prompt += f"""
Dialogue:
{dataset['test'][index]['dialogue']}
What was going on?
{dataset['test'][index]['summary']}
"""
prompt += f"""
Dialogue:
{dataset['test'][example_index_to_summarize]['dialogue']}
What was going on?
"""
return prompt
output = generate_summary(tokenizer, one_shot_prompt)
output_result(dataset['test'][examples[1]]['summary'], output)
プロンプトの中身です。
Prompt(One-Shot)
Dialogue:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.
What was going on?
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
Dialogue:
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
What was going on?
ZeroーShotと変わらなかったです。
出力結果
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up.
5.４. Few-Shot
Few-Shot(2 Shot)で試してみます。
few_shot_prompt = make_prompt([examples[0], 80], examples[1])
print(few_shot_prompt)
Promptの内容です。
Prompt(Few-Shot)
Dialogue:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.
What was going on?
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
Dialogue:
#Person1#: May, do you mind helping me prepare for the picnic?
#Person2#: Sure. Have you checked the weather report?
#Person1#: Yes. It says it will be sunny all day. No sign of rain at all. This is your father's favorite sausage. Sandwiches for you and Daniel.
#Person2#: No, thanks Mom. I'd like some toast and chicken wings.
#Person1#: Okay. Please take some fruit salad and crackers for me.
#Person2#: Done. Oh, don't forget to take napkins disposable plates, cups and picnic blanket.
#Person1#: All set. May, can you help me take all these things to the living room?
#Person2#: Yes, madam.
#Person1#: Ask Daniel to give you a hand?
#Person2#: No, mom, I can manage it by myself. His help just causes more trouble.
What was going on?
Mom asks May to help to prepare for the picnic and May agrees.
Dialogue:
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
What was going on?
生成結果です。BrianはBrianと踊りたい、というのは意味がわからないですね。
出力結果
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. Brian wants to have a dance with Brian.
5.5. Configuration
Configurationでtemperatureを変えてFew-Shotでやってみます。temperatureが高いほどランダム性が高くなります。
# temperature が高いとランダム性が高い(Softmaxレイヤの計算制御)
for temperature in [0.1, 0.5, 1.0, 2.0]:
generation_config = GenerationConfig(max_new_tokens=50,
do_sample=True, temperature=temperature)
print(dash_line)
print(f'{temperature=}')
print(dash_line)
output = generate_summary(tokenizer, few_shot_prompt, generation_config)
output_result(dataset['test'][examples[1]]['summary'], output)
出力結果
----------------------------------------------------------------------------------------------------
temperature=0.1
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. Brian will have a dance with Brian.
----------------------------------------------------------------------------------------------------
temperature=0.5
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. Brian invites everyone to come over and have a dance.
----------------------------------------------------------------------------------------------------
temperature=1.0
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. People congratulate him on his appearance.
----------------------------------------------------------------------------------------------------
temperature=2.0
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
#p and Others enjoy and love a private dance, with dancing of celebrities' ward.