BedrockのLlama 3.1を呼ぶ（InvokeModelとConverse、ToolやCode Interpreterも）

Posted at 2024-07-24

新モデルが来たら、やるでしょ

BedrockのドキュメントにはまだLlama 3.1のプロンプト例がなかったので、Metaのドキュメントから拝借しました。

https://llama.meta.com/docs/model-cards-and-prompt-formats/llama3_1/

InvokeModel API

プロンプトが独特。

import json
import boto3

client = boto3.Session(region_name="us-west-2").client("bedrock-runtime")

model_id = "meta.llama3-1-70b-instruct-v1:0"

prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Cutting Knowledge Date: December 2023
Today Date: 23 Jul 2024

You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

What is the capital for France?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

response = client.invoke_model(
    body=json.dumps(
        {
            "prompt": prompt,
        }
    ),
    modelId=model_id,
)
response_json = json.loads(response["body"].read().decode("utf-8"))

print(json.dumps(response_json, indent=2, ensure_ascii=False))

Response

{
  "generation": "The capital of France is Paris.",
  "prompt_token_count": 47,
  "generation_token_count": 8,
  "stop_reason": "stop"
}

Converse API

Converse APIもいけます。プロンプトのお決まりのところを書かなくていいので、見やすい！

import json
import boto3

client = boto3.Session(region_name="us-west-2").client("bedrock-runtime")

model_id = "meta.llama3-1-70b-instruct-v1:0"

messages = {}

system_prompt = """Cutting Knowledge Date: December 2023
Today Date: 23 Jul 2024

You are a helpful assistant
"""

user_prompt = """What is the capital for France?
"""

response = client.converse(
    system=[{"text": system_prompt}],
    messages=[{"role": "user", "content": [{"text": user_prompt}]}],
    modelId=model_id,
)

print(json.dumps(response, indent=2, ensure_ascii=False))

Response

{
  "ResponseMetadata": {
    "RequestId": "944c59d7-600b-4aa0-b2b0-c79fbc259031",
    "HTTPStatusCode": 200,
    "HTTPHeaders": {
      "date": "Wed, 24 Jul 2024 13:11:56 GMT",
      "content-type": "application/json",
      "content-length": "215",
      "connection": "keep-alive",
      "x-amzn-requestid": "944c59d7-600b-4aa0-b2b0-c79fbc259031"
    },
    "RetryAttempts": 2
  },
  "output": {
    "message": {
      "role": "assistant",
      "content": [
        {
          "text": "\n\nThe capital of France is Paris."
        }
      ]
    }
  },
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 47,
    "outputTokens": 9,
    "totalTokens": 56
  },
  "metrics": {
    "latencyMs": 813
  }
}

InvokeModel API + Tool

Llama 3.1はなんと、 ビルドインツール があります！

Brave Search: Web 検索を実行するためのツール呼び出し
Wolfram Alpha: 複雑な数学計算を実行するためのツール呼び出し
Code Interpreter: モデルが Python コードを実行

Environment: ipython をシステムプロンプトに書くだけで、コードインタープリターが有効になって、他の２つのツールはToolsで指定するとのことです。

import json
import boto3

client = boto3.Session(region_name="us-west-2").client("bedrock-runtime")

model_id = "meta.llama3-1-70b-instruct-v1:0"

prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Environment: ipython
Tools: brave_search, wolfram_alpha

Cutting Knowledge Date: December 2023
Today Date: 23 Jul 2024

You are a helpful assistant<|eot_id|>
<|start_header_id|>user<|end_header_id|>

What is the current weather in Menlo Park, California?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

response = client.invoke_model(
    body=json.dumps(
        {
            "prompt": prompt,
        }
    ),
    modelId=model_id,
)
response_json = json.loads(response["body"].read().decode("utf-8"))

print(json.dumps(response_json, indent=2, ensure_ascii=False))

Response

{
  "generation": "<|python_tag|>brave_search.call(query=\"current weather in Menlo Park, California\")",
  "prompt_token_count": 67,
  "generation_token_count": 17,
  "stop_reason": "stop"
}

出力はちょっと加工が必要そうですね。

Converse API + Tool

Converse APIではこうなります

import json
import boto3

client = boto3.Session(region_name="us-west-2").client("bedrock-runtime")

model_id = "meta.llama3-1-70b-instruct-v1:0"

messages = {}

system_prompt = """Environment: ipython
Tools: brave_search, wolfram_alpha

Cutting Knowledge Date: December 2023
Today Date: 23 Jul 2024

You are a helpful assistant
"""

user_prompt = """What is the current weather in Menlo Park, California?
"""

response = client.converse(
    system=[{"text": system_prompt}],
    messages=[{"role": "user", "content": [{"text": user_prompt}]}],
    modelId=model_id,
)

print(json.dumps(response, indent=2, ensure_ascii=False))

Response

{
  "ResponseMetadata": {
    "RequestId": "87d57430-45ce-4ecb-90ad-6e11d76b1f65",
    "HTTPStatusCode": 200,
    "HTTPHeaders": {
      "date": "Wed, 24 Jul 2024 13:19:49 GMT",
      "content-type": "application/json",
      "content-length": "270",
      "connection": "keep-alive",
      "x-amzn-requestid": "87d57430-45ce-4ecb-90ad-6e11d76b1f65"
    },
    "RetryAttempts": 0
  },
  "output": {
    "message": {
      "role": "assistant",
      "content": [
        {
          "text": "\n\n<|python_tag|>brave_search.call(query=\"current weather in Menlo Park, California\")"
        }
      ]
    }
  },
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 66,
    "outputTokens": 18,
    "totalTokens": 84
  },
  "metrics": {
    "latencyMs": 1108
  }
}

出力に加工が必要そうなのは変わらずですが、システムプロンプトはわかりやすいです。

InvokeModel APIでCode Interpreterのみ

import json
import boto3

client = boto3.Session(region_name="us-west-2").client("bedrock-runtime")

model_id = "meta.llama3-1-70b-instruct-v1:0"

prompt = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Environment: ipython

Cutting Knowledge Date: December 2023
Today Date: 23 Jul 2024

You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>

以下のCSVデータをグラフにしてください。

1, 10
2, 11
3, 12
4, 13
<|eot_id|><|start_header_id|>assistant<|end_header_id|>
"""

response = client.invoke_model(
    body=json.dumps(
        {
            "prompt": prompt,
        }
    ),
    modelId=model_id,
)
response_json = json.loads(response["body"].read().decode("utf-8"))

print(json.dumps(response_json, indent=2, ensure_ascii=False))

Response

{
  "generation": "import matplotlib.pyplot as plt\nimport csv\n\n# CSVデータを読み込む\nwith open('data.csv', 'r') as f:\n    reader = csv.reader(f)\n    data = list(reader)\n\n# データを整理する\nx = [int(row[0]) for row in data]\ny = [int(row[1]) for row in data]\n\n# グラフを作成する\nplt.plot(x, y)\n\n# グラフを表示する\nplt.show()",
  "prompt_token_count": 75,
  "generation_token_count": 103,
  "stop_reason": "stop"
}

generationの部分のみをPythonコードとして出力するとこの通り

import matplotlib.pyplot as plt
import csv

# CSVデータを読み込む
with open('data.csv', 'r') as f:
    reader = csv.reader(f)
    data = list(reader)

# データを整理する
x = [int(row[0]) for row in data]
y = [int(row[1]) for row in data]

# グラフを作成する
plt.plot(x, y)

# グラフを表示する
plt.show()

いいですね

Converse APIでCode Interpreterのみ

理由は不明ですが、これがうまくいきませんでした。
（「Sure! Here is a simple line graph based」に続けてコードが出力される感じ）
コード間違ってる？？

import json
import boto3

client = boto3.Session(region_name="us-west-2").client("bedrock-runtime")

model_id = "meta.llama3-1-70b-instruct-v1:0"

messages = {}

system_prompt = """Environment: ipython

Cutting Knowledge Date: December 2023
Today Date: 23 Jul 2024

You are a helpful assistant
"""

user_prompt = """以下のCSVデータをグラフにしてください。

1, 10
2, 11
3, 12
4, 13

"""

response = client.converse(
    system=[{"text": system_prompt}],
    messages=[{"role": "user", "content": [{"text": user_prompt}]}],
    modelId=model_id,
)

print(json.dumps(response, indent=2, ensure_ascii=False))

Response

{
  "ResponseMetadata": {
    "RequestId": "2c77f9f0-f821-4465-879d-c94ba2302712",
    "HTTPStatusCode": 200,
    "HTTPHeaders": {
      "date": "Wed, 24 Jul 2024 13:33:16 GMT",
      "content-type": "application/json",
      "content-length": "1055",
      "connection": "keep-alive",
      "x-amzn-requestid": "2c77f9f0-f821-4465-879d-c94ba2302712"
    },
    "RetryAttempts": 0
  },
  "output": {
    "message": {
      "role": "assistant",
      "content": [
        {
          "text": "\n\nSure! Here is a simple line graph based on the provided CSV data:\n\n```\nimport matplotlib.pyplot as plt\n\n# Define the data\nx = [1, 2, 3, 4]\ny = [10, 11, 12, 13]\n\n# Create the figure and axis\nfig, ax = plt.subplots()\n\n# Plot the data\nax.plot(x, y)\n\n# Set the title and labels\nax.set_title('Line Graph')\nax.set_xlabel('X')\nax.set_ylabel('Y')\n\n# Show the plot\nplt.show()\n```\n\nThis code will generate a simple line graph with the x-values on the horizontal axis and the y-values on the vertical axis. The title of the graph is \"Line Graph\", and the labels for the x and y axes are \"X\" and \"Y\", respectively.\n\nPlease note that you need to have the `matplotlib` library installed in your Python environment to run this code. If you don't have it installed, you can do so by running `pip install matplotlib` in your terminal or command prompt."
        }
      ]
    }
  },
  "stopReason": "end_turn",
  "usage": {
    "inputTokens": 75,
    "outputTokens": 221,
    "totalTokens": 296
  },
  "metrics": {
    "latencyMs": 7641
  }
}

You get articles that match your needs
You can efficiently read back useful information
You can use dark theme

What you can do with signing up