LLMを動かしながらプロンプトエンジニアリングを試してみました。超軽量のflan-t5-baseを使って文章要約タスクを試しています。パラメータ数が248Mと少なく、手軽に動かすには都合が良かったです。その反面日本語ではプロンプトできないですし、精度も良くないです。Google ColabのCPUで動かしています。

実行環境

2024/3/26時点のGoogle Colab で、Python3.10.12で、以下のパッケージを使っています。

Package	Version	備考
torch	2.1.0+cu121	Google Colabにプリインストールされていたバージョン
transformers	4.38.2	Google Colabにプリインストールされていたバージョン
datasets	2.18.0	追加でpip install

サマリ

以下のダイアログ要約をパターン変えてさせています。

knkarthick/dialogsum test-11

#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday

正解ラベルと各パターン別結果

正解ラベルと、各試行パターンごとの生成値です(tempertureはdefault1)。
モデルがあんまり良くないので0 shotと 1 shotで違いが微妙だし、2shotでも大して良くない。
tempertureを2にすると滅茶苦茶になりますね。

種別	結果
正解ラベル	#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
0 shot without instruction	Brian, thank you for coming to our party.
0 shot	Brian's birthday is coming up.
1 shot	Brian's birthday is coming up.
2 shot(temperture=1)	Brian's birthday is coming up. Brian wants to have a dance with Brian.
2 shot(temperture=0.1)	Brian's birthday is coming up. Brian will have a dance with Brian.
2 shot(temperture=0.5)	Brian's birthday is coming up. Brian invites everyone to come over and have a dance.
2 shot(temperture=2)	#p and Others enjoy and love a private dance, with dancing of celebrities' ward.

実行プログラム

1. pip intall

Google Colab にプレインストールされていないdatasetsをインストール。

!pip install datasets

2. Impot packages

PythonパッケージをImport。AutoModelForSeq2SeqLM(Encoder-Decoder)は珍しく、GPTなどDecoder-onlyが主流(その場合はAutoModelForCausalLM)。

rom datasets import load_dataset

# AutoModelForSeq2SeqLM は Encoder-decoder用。 AutoModelForCausalLM だと auto-regressive(Decoder-only)
from transformers import AutoModelForSeq2SeqLM
from transformers import AutoTokenizer
from transformers import GenerationConfig

3. Load Data

以下のdialogsumのデータを使っています。ダイアログと要約のデータがあります。

データをロードします。

dataset = load_dataset('knkarthick/dialogsum')
print(dataset)

出力結果

DatasetDict({
    train: Dataset({
        features: ['id', 'dialogue', 'summary', 'topic'],
        num_rows: 12460
    })
    validation: Dataset({
        features: ['id', 'dialogue', 'summary', 'topic'],
        num_rows: 500
    })
    test: Dataset({
        features: ['id', 'dialogue', 'summary', 'topic'],
        num_rows: 1500
    })
})

データの中身を見てみます。

# index 1, 2は同じダイアログだったので離れたIndexにした
examples = [1, 10]
dash_line = '-'*100

def output_prompt(dataset, i, prompt):
    print(dash_line)
    print('例: ', i + 1)
    print(dash_line)
    print('プロンプト:')
    print(prompt)
    print(dash_line)

def output_result(y_true, output=None):
    print('正解ラベル:')
    print(y_true)
    print(dash_line)
    if output != None:
        print(f'モデル生成結果:\n{output}\n')
    print()

for i in examples:
    output_prompt(dataset, i, dataset['test'][i]['dialogue'])
    output_result(dataset['test'][i]['summary'])

出力結果

----------------------------------------------------------------------------------------------------
例:  2
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.
----------------------------------------------------------------------------------------------------
正解ラベル:
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
----------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------
例:  11
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------

4. Load model

Flan-T5 releaseのbaseサイズを使いました。パラメータ数は248Mで昨今のLLMに比べると非常に少ないです。

# https://huggingface.co/google/flan-t5-small
# use Encoder-decoder small model
#model_name='google/flan-t5-small'
model_name='google/flan-t5-base'
#model_name='google/flan-t5-large'
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)

Tokenizerを少し試してみます。

# 日本語で試したが当然、Encodeできない
sentence = "What time is it, Tom?"

# PyTorchのテンソル型で返すためにreturn_tensors='pt'
sentence_encoded = tokenizer(sentence, return_tensors='pt')

sentence_decoded = tokenizer.decode(
        sentence_encoded["input_ids"][0],
        skip_special_tokens=True
    )

print(f'{tokenizer.max_model_input_sizes=}')

print('ENCODED SENTENCE:')
print(sentence_encoded["input_ids"][0])

print('\nENCODED Tokens:')
print(tokenizer.convert_ids_to_tokens(sentence_encoded["input_ids"][0]))

print('\nDECODED SENTENCE:')
print(sentence_decoded)

出力結果

tokenizer.max_model_input_sizes={'google-t5/t5-small': 512, 'google-t5/t5-base': 512, 'google-t5/t5-large': 512, 'google-t5/t5-3b': 512, 'google-t5/t5-11b': 512}
ENCODED SENTENCE:
tensor([ 363,   97,   19,   34,    6, 3059,   58,    1])

ENCODED Tokens:
['▁What', '▁time', '▁is', '▁it', ',', '▁Tom', '?', '</s>']

DECODED SENTENCE:
What time is it, Tom?

5. Prompt Engineering

ここから本題です。
事前にサマリを生成する関数を定義しておきます。

def generate_summary(tokenizer, prompt, config=GenerationConfig(max_new_tokens=50)):
    inputs = tokenizer(prompt, return_tensors='pt')
    output = tokenizer.decode(
        model.generate(
            inputs["input_ids"],
            max_new_tokens=50,
            generation_config=config
        )[0],
        skip_special_tokens=True
    )
    return output

5.1. Zero-Shot without instruction

Zero-ShotでインストラクションなしでLLMに投げます。

for i in examples:
    dialogue = dataset['test'][i]['dialogue']
    output = generate_summary(tokenizer, dialogue)
    output_prompt(dataset, i, dialogue)
    output_result(dataset['test'][i]['summary'], output)

出力結果

----------------------------------------------------------------------------------------------------
例:  2
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.
----------------------------------------------------------------------------------------------------
正解ラベル:
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
----------------------------------------------------------------------------------------------------
モデル生成結果:
#Person1#: Ms. Dawson, I need you to take a dictation for me.


----------------------------------------------------------------------------------------------------
例:  11
----------------------------------------------------------------------------------------------------
プロンプト:
#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian, thank you for coming to our party.

5.2. Zero-Shot with instruction

Zero-Shotでインストラクションとして「What was going on?」を付加してみます。もう少しいいインストラクションあるかもしれませんが、試していません。

for i in examples:
    dialogue = dataset['test'][i]['dialogue']

    prompt = f"""
Dialogue:

{dialogue}

What was going on?
"""

    output = generate_summary(tokenizer, prompt)
    output_prompt(dataset, i, prompt)
    output_result(dataset['test'][i]['summary'], output)

出力結果

----------------------------------------------------------------------------------------------------
例:  2
----------------------------------------------------------------------------------------------------
Dialogue:

#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.

What was going on?

----------------------------------------------------------------------------------------------------
正解ラベル:
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.
----------------------------------------------------------------------------------------------------
モデル生成結果:
The memo is to be distributed to all employees by this afternoon.


----------------------------------------------------------------------------------------------------
例:  11
----------------------------------------------------------------------------------------------------
プロンプト:

Dialogue:

#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday

What was going on?

----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up.

長いので、Instruction有無でどう変わったか改めて表にします。少し良くなっていますね。

例	Without Instruction	With Instruction
2	#Person1#: Ms. Dawson, I need you to take a dictation for me.	The memo is to be distributed to all employees by this afternoon.
11	Brian, thank you for coming to our party.	Brian's birthday is coming up.

5.3. One-Shot

次はOne-Shotで試してみます。

def make_prompt(example_indices_full, example_index_to_summarize):
    prompt = ''
    for index in example_indices_full:

        # '{summary}の後の\n\n\n'がFLAN-T5にとっては重要らしい
        prompt += f"""
Dialogue:

{dataset['test'][index]['dialogue']}

What was going on?
{dataset['test'][index]['summary']}


"""

    prompt += f"""
Dialogue:

{dataset['test'][example_index_to_summarize]['dialogue']}

What was going on?
"""

    return prompt

output = generate_summary(tokenizer, one_shot_prompt)
output_result(dataset['test'][examples[1]]['summary'], output)

プロンプトの中身です。

Prompt(One-Shot)

Dialogue:

#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.

What was going on?
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.



Dialogue:

#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday

What was going on?

ZeroーShotと変わらなかったです。

出力結果

正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up.

5.４. Few-Shot

Few-Shot(2 Shot)で試してみます。

few_shot_prompt = make_prompt([examples[0], 80], examples[1])
print(few_shot_prompt)

Promptの内容です。

Prompt(Few-Shot)

Dialogue:

#Person1#: Ms. Dawson, I need you to take a dictation for me.
#Person2#: Yes, sir...
#Person1#: This should go out as an intra-office memorandum to all employees by this afternoon. Are you ready?
#Person2#: Yes, sir. Go ahead.
#Person1#: Attention all staff... Effective immediately, all office communications are restricted to email correspondence and official memos. The use of Instant Message programs by employees during working hours is strictly prohibited.
#Person2#: Sir, does this apply to intra-office communications only? Or will it also restrict external communications?
#Person1#: It should apply to all communications, not only in this office between employees, but also any outside communications.
#Person2#: But sir, many employees use Instant Messaging to communicate with their clients.
#Person1#: They will just have to change their communication methods. I don't want any - one using Instant Messaging in this office. It wastes too much time! Now, please continue with the memo. Where were we?
#Person2#: This applies to internal and external communications.
#Person1#: Yes. Any employee who persists in using Instant Messaging will first receive a warning and be placed on probation. At second offense, the employee will face termination. Any questions regarding this new policy may be directed to department heads.
#Person2#: Is that all?
#Person1#: Yes. Please get this memo typed up and distributed to all employees before 4 pm.

What was going on?
In order to prevent employees from wasting time on Instant Message programs, #Person1# decides to terminate the use of those programs and asks Ms. Dawson to send out a memo to all employees by the afternoon.



Dialogue:

#Person1#: May, do you mind helping me prepare for the picnic?
#Person2#: Sure. Have you checked the weather report?
#Person1#: Yes. It says it will be sunny all day. No sign of rain at all. This is your father's favorite sausage. Sandwiches for you and Daniel.
#Person2#: No, thanks Mom. I'd like some toast and chicken wings.
#Person1#: Okay. Please take some fruit salad and crackers for me.
#Person2#: Done. Oh, don't forget to take napkins disposable plates, cups and picnic blanket.
#Person1#: All set. May, can you help me take all these things to the living room?
#Person2#: Yes, madam.
#Person1#: Ask Daniel to give you a hand?
#Person2#: No, mom, I can manage it by myself. His help just causes more trouble.

What was going on?
Mom asks May to help to prepare for the picnic and May agrees.



Dialogue:

#Person1#: Happy Birthday, this is for you, Brian.
#Person2#: I'm so happy you remember, please come in and enjoy the party. Everyone's here, I'm sure you have a good time.
#Person1#: Brian, may I have a pleasure to have a dance with you?
#Person2#: Ok.
#Person1#: This is really wonderful party.
#Person2#: Yes, you are always popular with everyone. and you look very pretty today.
#Person1#: Thanks, that's very kind of you to say. I hope my necklace goes with my dress, and they both make me look good I feel.
#Person2#: You look great, you are absolutely glowing.
#Person1#: Thanks, this is a fine party. We should have a drink together to celebrate your birthday

What was going on?

生成結果です。BrianはBrianと踊りたい、というのは意味がわからないですね。

出力結果

正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. Brian wants to have a dance with Brian.

5.5. Configuration

Configurationでtemperatureを変えてFew-Shotでやってみます。temperatureが高いほどランダム性が高くなります。

# temperature が高いとランダム性が高い(Softmaxレイヤの計算制御)
for temperature in [0.1, 0.5, 1.0, 2.0]:
    generation_config = GenerationConfig(max_new_tokens=50,
                                         do_sample=True, temperature=temperature)
    print(dash_line)
    print(f'{temperature=}')
    print(dash_line)
    output = generate_summary(tokenizer, few_shot_prompt, generation_config)
    output_result(dataset['test'][examples[1]]['summary'], output)

出力結果

----------------------------------------------------------------------------------------------------
temperature=0.1
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. Brian will have a dance with Brian.


----------------------------------------------------------------------------------------------------
temperature=0.5
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. Brian invites everyone to come over and have a dance.


----------------------------------------------------------------------------------------------------
temperature=1.0
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
Brian's birthday is coming up. People congratulate him on his appearance.


----------------------------------------------------------------------------------------------------
temperature=2.0
----------------------------------------------------------------------------------------------------
正解ラベル:
#Person1# attends Brian's birthday party. Brian thinks #Person1# looks great and charming.
----------------------------------------------------------------------------------------------------
モデル生成結果:
#p and Others enjoy and love a private dance, with dancing of celebrities' ward.