jev は TypeSafe AI のモデル。テキストと型付きの質問を送ると、選択・スコア・yes/no の確率を返す。

公式ドキュメントを読みながら Python SDK で一通り動かし、公式に載っていない使い方も試した。コード例と実行結果は、すべて実際に jev で実行したもの。

動作確認した環境

モデル: jev-1.13.0
SDK: typesafe-sdk 0.7.1
確認日: 2026-09-22

数値は同じ入力でも実行ごとに少し変わる。

セットアップと最初の呼び出し

インストール

uv add typesafe-sdk

pip の場合

pip install typesafe-sdk

API キー

API キーを環境変数 TYPESAFE_API_KEY に入れる。クライアントはこの変数を自動で読む。

export TYPESAFE_API_KEY=<キー>

コードで渡すときは TypeSafeClient(api_key=...) を使う。

最初の呼び出し

state に評価させる内容、questions に質問を入れて system_one を呼ぶ。

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="I was charged twice. Please fix this ASAP.",
        questions={
            "department": Choice(
                instructions="Which team should handle this?",
                criteria={
                    "billing": "Payment or subscription issues",
                    "technical": "Bugs or integration problems",
                    "sales": "Pricing or account questions",
                },
            ),
            "frustration": Score(
                instructions="How frustrated does the customer appear?",
                criteria=[
                    "Calm, just stating facts",
                    "Frustrated but civil",
                    "Very angry, strong language",
                ],
            ),
            "is_urgent": Noul(instructions="Does the message convey urgency?"),
        },
    )

print(response.answers["department"].choice)
print(response.answers["frustration"].score)
print(response.answers["is_urgent"].noul)

実行結果

billing
1.0
0.97

レスポンスの中身

属性	内容
`answers`	質問 ID → 答え
`choices` / `scores` / `nouls`	型ごとに分けた答え
`model`	実際に答えたモデルの版
`usage`	入力と出力のトークン数
`request_id`	リクエストの ID

from typesafe_sdk import Noul, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="I was charged twice.",
        questions={"billing": Noul(instructions="Is this about billing?")},
    )

print(response.model)
print(response.usage)
print(response.nouls["billing"].noul)

実行結果

jev-1.13.0
input_tokens=276 output_tokens=20
0.96

model を指定しないと jev-latest が使われ、レスポンスの model には答えた版の ID が入る。

state

state は jev に評価させる内容。1 回のリクエストで送る state は 1 つで、すべての質問が同じ state を見る。

形式

文字列・dict・list を渡せる。

from typesafe_sdk import Noul, TypeSafeClient

states = [
    "My card was charged twice.",
    {"message": "My card was charged twice.", "order_id": "A-104"},
    ["Hi", "My customer number is TS1337.", "My card was charged twice."],
]

with TypeSafeClient() as client:
    for state in states:
        response = client.system_one(
            state=state,
            questions={"billing": Noul(instructions="Is this about a billing problem?")},
        )
        print(type(state).__name__, response.nouls["billing"].noul)

実行結果

str 0.98
dict 0.98
list 0.98

state の一部を質問で指す

質問の instructions に、state 内のパスをバッククォートで囲んで書く。

from typesafe_sdk import Noul, TypeSafeClient

state = {
    "ticket": {
        "messages": [
            {"from": "customer", "text": "I was charged twice for order A-104. Please refund the duplicate."},
        ],
    },
    "order": {
        "id": "A-104",
        "charges": [
            {"amount_usd": 49, "status": "captured"},
            {"amount_usd": 49, "status": "captured"},
        ],
    },
    "refund_policy": "Duplicate charges are eligible for a refund.",
}

with TypeSafeClient() as client:
    response = client.system_one(
        state=state,
        questions={
            "refund_requested": Noul(
                instructions="Does `ticket.messages[0].text` request a refund?",
            ),
            "policy_supports_refund": Noul(
                instructions="Does `refund_policy` support the refund requested in `ticket.messages[0].text`, given `order.charges`?",
            ),
        },
    )

for question_id, answer in response.nouls.items():
    print(question_id, answer.noul)

実行結果

refund_requested 0.99
policy_supports_refund 0.98

Choice

選択肢から 1 つを選ばせる質問。

書き方

from typesafe_sdk import Choice, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="My running shoes arrived in the wrong size. Can I swap them for a size 10?",
        questions={
            "department": Choice(
                instructions="Which team should handle this?",
                criteria={
                    "returns": "Exchanges, wrong or damaged items",
                    "shipping": "Delivery status, delays, lost packages",
                    "billing": "Charges, invoices, payment problems",
                },
            ),
        },
    )

answer = response.choices["department"]
print(answer.choice)
print(answer.confidence)
print(answer.probabilities)

実行結果

returns
1.0
{'billing': 0.0, 'returns': 1.0, 'shipping': 0.0}

criteria は選択肢名 → 説明の dict
選択肢は最大 255 個。256 個以上は TypeSafeBadRequestError (400) になる

答え

属性	内容
`choice`	確率が最も高い選択肢
`probabilities`	選択肢ごとの確率。合計は 1
`confidence`	0〜1。確率が 1 つに集中すると 1 になる

選択肢名と説明

選択肢名と説明のどちらも判断に使われる。名前だけで分かる選択肢は、説明を None にできる。

from typesafe_sdk import Choice, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="My package never arrived.",
        questions={
            "description_only": Choice(
                instructions="Which team should handle this?",
                criteria={"a": "Delivery problems and lost packages", "b": "Charges and refunds"},
            ),
            "name_only": Choice(
                instructions="Which team should handle this?",
                criteria={"shipping": None, "billing": None},
            ),
        },
    )

for question_id, answer in response.choices.items():
    print(question_id, answer.choice, answer.probabilities)

実行結果

description_only a {'b': 0.0, 'a': 1.0}
name_only shipping {'billing': 0.0, 'shipping': 1.0}

other を足す

どの選択肢にも当てはまらない入力で、other の有無を比べる。

from typesafe_sdk import Choice, TypeSafeClient

TEAMS = {
    "returns": "Exchanges, wrong or damaged items",
    "shipping": "Delivery status, delays, lost packages",
    "billing": "Charges, invoices, payment problems",
}
message = "Do you have any job openings in your marketing team?"

with TypeSafeClient() as client:
    response = client.system_one(
        state=message,
        questions={
            "without_other": Choice(instructions="Which team should handle this?", criteria=TEAMS),
            "with_other": Choice(
                instructions="Which team should handle this?",
                criteria={**TEAMS, "other": "None of the teams above"},
            ),
        },
    )

for question_id, answer in response.choices.items():
    print(question_id, answer.choice, answer.confidence)

実行結果

without_other billing 0.42
with_other other 1.0

other が無いと、当てはまらない入力でも既存の選択肢のどれかが選ばれる。

Score

段階のある尺度で、state の位置を測る質問。

書き方

from typesafe_sdk import Score, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="The export button crashes the settings page in Safari. It works in Chrome, but a few of our customers only use Safari.",
        questions={
            "bug_severity": Score(
                instructions="How severe is the reported issue?",
                criteria=[
                    "Cosmetic; no impact to functionality",
                    "Broken or degraded feature, but workaround exists",
                    "Blocking issue; no workaround exists",
                ],
            ),
        },
    )

answer = response.scores["bug_severity"]
print(answer.score)
print(answer.confidence)
print(answer.probabilities)
print(answer.legend)

実行結果

1.42
0.36
{0: 0.0, 1: 0.58, 2: 0.42}
{0: 'Cosmetic; no impact to functionality', 1: 'Broken or degraded feature, but workaround exists', 2: 'Blocking issue; no workaround exists'}

criteria は段階の説明のリスト。段階の番号はリストの位置で、0 から始まる
段階は最大 10。11 以上は TypeSafeBadRequestError (400) になる

答え

属性	内容
`score`	段階番号 × 確率の合計。段階の間の値にもなる
`probabilities`	段階番号 → 確率
`legend`	段階番号 → 段階の説明
`confidence`	0〜1。確率が 1 つの段階に集中すると 1 になる

SDK では probabilities と legend のキーは int。

段階は状況で書く

段階を数字だけで書いた場合と、状況で書いた場合を比べる。

from typesafe_sdk import Score, TypeSafeClient

report = "The export button is misaligned by a few pixels on the settings page."

with TypeSafeClient() as client:
    response = client.system_one(
        state=report,
        questions={
            "numbers_only": Score(
                instructions="Rate severity from 0 to 2, where 2 is worst",
                criteria=["0", "1", "2"],
            ),
            "descriptive": Score(
                instructions="How severe is the reported issue?",
                criteria=[
                    "Cosmetic; no impact to functionality",
                    "Broken or degraded feature, but workaround exists",
                    "Blocking issue; no workaround exists",
                ],
            ),
        },
    )

for question_id, answer in response.scores.items():
    print(question_id, answer.score, answer.confidence)

実行結果

numbers_only 0.59 0.38
descriptive 0.0 1.0

この例では、数字だけの段階は判断が割れ (confidence 0.38)、状況で書いた段階は 1 つに決まった (confidence 1.0)。

Noul

yes/no の質問。答えの noul は yes の確率 (0〜1)。

書き方

from typesafe_sdk import Noul, NoulCriteria, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="I have asked three times now. Can I please just talk to a real person?",
        questions={
            "is_human_escalation": Noul(
                instructions="Is the customer asking for a human agent?",
            ),
            "is_repeat_contact": Noul(
                instructions="Has the customer contacted support about this before?",
                criteria=NoulCriteria(
                    true="Mentions a prior attempt, ticket, or that they have asked before",
                    false="No sign of any previous contact",
                ),
            ),
        },
    )

for question_id, answer in response.nouls.items():
    print(question_id, answer.noul)

実行結果

is_human_escalation 0.99
is_repeat_contact 0.95

criteria は省略できる。付けるときは NoulCriteria(true=..., false=...) で yes と no の意味を書く
Noul の答えに confidence は無い

1 問に条件は 1 つ

2 つの条件を 1 問で聞いた場合と、分けて聞いた場合を比べる。

from typesafe_sdk import Noul, TypeSafeClient

with TypeSafeClient() as client:
    response = client.system_one(
        state="This is the worst service ever. I am furious that my order is late.",
        questions={
            "angry_and_refund": Noul(instructions="Is the customer angry and asking for a refund?"),
            "angry": Noul(instructions="Is the customer angry?"),
            "refund": Noul(instructions="Is the customer asking for a refund?"),
        },
    )

for question_id, answer in response.nouls.items():
    print(question_id, answer.noul)

実行結果

angry_and_refund 0.21
angry 0.99
refund 0.12

1 問で聞くと、どちらの条件が満たされていないのかが分からない。

しきい値で分岐する

from typesafe_sdk import Noul, TypeSafeClient

YES = 0.8
NO = 0.2

messages = [
    "Thanks, that fixed it!",
    "Are you a bot?",
    "Is there any way to speak to someone about my invoice?",
]

with TypeSafeClient() as client:
    for message in messages:
        response = client.system_one(
            state=message,
            questions={"wants_human": Noul(instructions="Is the customer asking for a human agent?")},
        )
        value = response.nouls["wants_human"].noul
        route = "agent" if value >= YES else "bot" if value <= NO else "review"
        print(f"{value:.2f} {route} {message}")

実行結果

0.02 bot Thanks, that fixed it!
0.39 review Are you a bot?
0.84 agent Is there any way to speak to someone about my invoice?

人を求めているかはっきりしない「Are you a bot?」は 0.39 になり、review に振り分けられた。

複数の質問をまとめる

同じ state に対する質問は、1 回のリクエストにまとめて送れる。Choice・Score・Noul は混ぜられる。

同じ state で 1 問と 20 問を 5 回ずつ送った実測

質問数	応答時間 (中央値)	入力トークン	共通の 1 問の答え
1	289ms	292	0.99
20	284ms	605	0.99

質問を増やしても応答時間はほとんど変わらない
増えるのは入力トークン
同じ質問の答えは、他の質問を足しても変わらない

使わないかもしれない質問も送る

分類の結果によって要る質問も、最初から一緒に送る。使わない答えはコードで無視する。

from typesafe_sdk import Choice, Noul, Score, TypeSafeClient

QUESTIONS = {
    "category": Choice(
        instructions="Determine the broad category of this support ticket",
        criteria={
            "bug_report": "The user is reporting something that is broken or producing errors",
            "billing": "Charges, invoices, refunds, subscriptions",
            "feature_request": "The user is requesting new functionality",
        },
    ),
    "bug_severity": Score(
        instructions="How severe is the reported issue?",
        criteria=[
            "Cosmetic; no impact to functionality",
            "Broken or degraded feature; workaround exists",
            "Blocking issue; no workaround exists",
        ],
    ),
    "refund_requested": Noul(instructions="The user is explicitly asking for a refund or credit"),
}

tickets = [
    "Nobody on our team can log in since this morning. We get a 500 error on every attempt.",
    "I was charged twice this month. Please refund the extra charge.",
]

with TypeSafeClient() as client:
    for ticket in tickets:
        answers = client.system_one(state=ticket, questions=QUESTIONS).answers
        category = answers["category"].choice
        if category == "bug_report":
            print(category, "severity", answers["bug_severity"].score)
        elif category == "billing":
            print(category, "refund", answers["refund_requested"].noul)
        else:
            print(category)

実行結果

bug_report severity 2.0
billing refund 0.99

判断を分けてコードで組み合わせる

複数の要素に依る判断は、要素ごとに Score で聞き、コードで重みを付けて合算する。尺度の長さが違うので、最上位の段階番号で割って 0〜1 にそろえてから合算する。

from typesafe_sdk import Score, TypeSafeClient

QUESTIONS = {
    "severity": Score(
        instructions="How severe is the reported issue?",
        criteria=[
            "Cosmetic; no impact to functionality",
            "Broken or degraded feature, but workaround exists",
            "Blocking issue; no workaround exists",
        ],
    ),
    "frustration": Score(
        instructions="How frustrated is the customer?",
        criteria=[
            "Calm, just stating facts",
            "Frustrated but civil",
            "Very angry, strong language or threatening to leave",
        ],
    ),
    "report_quality": Score(
        instructions="How much does the report give an engineer to work with?",
        criteria=[
            "No detail; just says something is broken",
            "Names the feature but no steps or environment",
            "Steps to reproduce or environment, but not both",
            "Steps to reproduce and environment",
        ],
    ),
}
WEIGHTS = {"severity": 0.6, "frustration": 0.3, "report_quality": 0.1}

ticket = (
    "Export to PDF fails with a spinner that never finishes. "
    "This is the third time I'm writing in and honestly I'm done. "
    "Steps: open any report, click Export, choose PDF. Chrome 128 on macOS."
)

with TypeSafeClient() as client:
    answers = client.system_one(state=ticket, questions=QUESTIONS).answers

priority = sum(
    weight * answers[question_id].score / (len(QUESTIONS[question_id].criteria) - 1)
    for question_id, weight in WEIGHTS.items()
)
print(round(priority, 2))

実行結果

0.87

confidence で分岐する

Choice と Score の答えには confidence (0〜1) が付く。確率が 1 つに集中すると 1 になり、分散すると下がる。

操作ごとにしきい値を変える

confidence がしきい値より低い答えは人に回し、送金の承認だけしきい値を高くする。

from typesafe_sdk import Choice, TypeSafeClient

INTENT = Choice(
    instructions="What action is the user requesting?",
    criteria={
        "check_balance": "Check the balance of an account",
        "approve_transfer": "Approve the pending transfer request",
        "other": "Something else",
    },
)

commands = [
    "What's my balance?",
    "Yes, go ahead and approve that transfer.",
    "Hmm, the transfer... I guess, maybe, not sure.",
]

with TypeSafeClient() as client:
    for command in commands:
        intent = client.system_one(state=command, questions={"intent": INTENT}).choices["intent"]
        if intent.confidence < 0.6:
            action = "support agent"
        elif intent.choice == "check_balance":
            action = "show balance"
        elif intent.choice == "approve_transfer":
            action = "approve" if intent.confidence > 0.85 else "ask to confirm"
        else:
            action = "support agent"
        print(f"{intent.choice} {intent.confidence:.2f} {action}")

実行結果

check_balance 1.00 show balance
approve_transfer 1.00 approve
other 0.61 support agent

あいまいな 3 つ目の指示は confidence が 0.61 に下がり、other として人に回った。

SDK の設定

モデルを指定する

使えるモデル名は client.models.list() で取れる。model で指定する。

from typesafe_sdk import Noul, TypeSafeClient

with TypeSafeClient() as client:
    for model in client.models.list().models:
        print(model.name, "-", model.description)

    response = client.system_one(
        model="jev-1.13.0",
        state="I was charged twice.",
        questions={"billing": Noul(instructions="Is this about billing?")},
    )
    print(response.model)

実行結果

jev-latest - The latest iteration of TypeSafe's System One Model: Jev
jev-preview - A preview version of `jev-latest`: should be better in most ways
jev-1.13.0

一覧に無い jev-1.13.0 のような版の ID も指定できる
存在しないモデル名は TypeSafeBadRequestError (400) になる

型付きのレスポンス

response_model にクラスを渡すと、答えを属性で読める。

from typesafe_sdk import Noul, NoulAnswer, SystemOneResponse, TypeSafeClient


class BillingResponse(SystemOneResponse):
    billing: NoulAnswer


with TypeSafeClient() as client:
    response = client.system_one(
        state="I was charged twice.",
        questions={"billing": Noul(instructions="Is this about billing?")},
        response_model=BillingResponse,
    )

print(response.billing.noul)

実行結果

0.96

非同期

import asyncio

from typesafe_sdk import AsyncTypeSafeClient, Noul


async def main() -> None:
    async with AsyncTypeSafeClient() as client:
        response = await client.system_one(
            state="I was charged twice.",
            questions={"billing": Noul(instructions="Is this about billing?")},
        )
    print(response.nouls["billing"].noul)


asyncio.run(main())

実行結果

0.97

エラー

API のエラーは TypeSafeAPIError で受け、status でステータスを読む。

from typesafe_sdk import Choice, TypeSafeAPIError, TypeSafeClient

with TypeSafeClient() as client:
    try:
        client.system_one(
            state="I was charged twice.",
            questions={"team": Choice(instructions="Which team?", criteria={})},
        )
    except TypeSafeAPIError as error:
        print(type(error).__name__, error.status)

実行結果

TypeSafeBadRequestError 400

実測した例

状況	例外
空の `criteria`	`TypeSafeBadRequestError` (400)
存在しないモデル名	`TypeSafeBadRequestError` (400)
選択肢 256 個以上・段階 11 以上	`TypeSafeBadRequestError` (400)
誤った API キー	`TypeSafeAuthenticationError` (401)
空白を含むなど形式が不正な API キー	クライアント作成時に `TypeSafeError`
サーバーが 503 を返す	`TypeSafeInternalServerError`
接続できない URL	`TypeSafeAPIConnectionError`

リトライ

SDK は失敗したリクエストを自動でリトライする。503 を返すサーバーに送ると、既定では 3 回 (初回 + リトライ 2 回) リクエストした。回数は RetryPolicy で変えられる。

from typesafe_sdk import RetryPolicy, TypeSafeClient

client = TypeSafeClient(retry=RetryPolicy(max_retries=3, timeout=10.0))

max_retries=3 では 4 回リクエストした。

ログ

ログは typesafe_sdk ロガーに出る。表示するには logging.basicConfig() などでハンドラを設定する。

import logging

from typesafe_sdk import Noul, TypeSafeClient

logging.basicConfig()
logging.getLogger("typesafe_sdk").setLevel(logging.INFO)

TypeSafeClient().system_one(state="x", questions={"q": Noul(instructions="Is this empty?")})

実行結果

INFO:typesafe_sdk:POST https://api.typesafe.ai/v1/systemone <- 200 in 563ms (request req_01a0c92c53c8724fbab737670d5fd85b)

レベルは環境変数 TYPESAFE_LOG_LEVEL でも設定できる。成功したリクエスト 1 回で出た行数

値	行数
`debug`	3
`info`	1
`warning` / `error` / `off`	0

環境変数

変数	内容	既定値
`TYPESAFE_API_KEY`	API キー	なし
`TYPESAFE_BASE_URL`	API の URL	`https://api.typesafe.ai`
`TYPESAFE_DEFAULT_MODEL`	`model` を省略したときのモデル	`jev-latest`
`TYPESAFE_LOG_LEVEL`	ログレベル	未設定

試してみた使い方

公式ドキュメントに載っていない使い方を、実際に試した結果。

選択肢の順番を入れ替える

同じ選択肢を順番だけ変えて送る。

from typesafe_sdk import Choice, TypeSafeClient

TEAMS = {
    "billing": "Payments, invoices, refund requests",
    "technical": "Bugs, outages, login problems",
    "sales": "Pricing plans, contract changes",
}
message = "The app has not started since last month, but I am still being charged."

with TypeSafeClient() as client:
    response = client.system_one(
        state=message,
        questions={
            "original": Choice(instructions="Which team should handle this?", criteria=TEAMS),
            "reversed": Choice(
                instructions="Which team should handle this?",
                criteria=dict(reversed(TEAMS.items())),
            ),
        },
    )

for question_id, answer in response.choices.items():
    print(question_id, answer.choice, dict(sorted(answer.probabilities.items())))

実行結果

original billing {'billing': 0.99, 'sales': 0.0, 'technical': 0.01}
reversed billing {'billing': 0.98, 'sales': 0.0, 'technical': 0.02}

順番を入れ替えても選ばれた選択肢は同じで、確率の差は 0.01 だった。

Noul を否定形で聞く

同じことを肯定形と否定形で聞き、値の和を見る。

from typesafe_sdk import Noul, TypeSafeClient

messages = [
    "Call me back at 090-1234-5678 after 6 pm.",
    "Please call me back after 6 pm.",
    "My old number was 090-1234-5678 but it no longer works.",
]

with TypeSafeClient() as client:
    for message in messages:
        nouls = client.system_one(
            state=message,
            questions={
                "has_phone": Noul(instructions="Does the text include a phone number?"),
                "no_phone": Noul(instructions="Is the text without any phone number?"),
            },
        ).nouls
        total = nouls["has_phone"].noul + nouls["no_phone"].noul
        print(nouls["has_phone"].noul, nouls["no_phone"].noul, round(total, 2), message)

実行結果

0.99 0.01 1.0 Call me back at 090-1234-5678 after 6 pm.
0.01 0.97 0.98 Please call me back after 6 pm.
0.98 0.02 1.0 My old number was 090-1234-5678 but it no longer works.

肯定形と否定形の値の和は、3 文とも 0.98〜1.0 だった。「もう使っていない番号」も電話番号ありと判定された。

日本語で使う

日本語の問い合わせを、英語の質問と日本語の質問で分類する。

from typesafe_sdk import Choice, TypeSafeClient

QUESTIONS = {
    "english": Choice(
        instructions="Which department should handle this inquiry?",
        criteria={
            "billing": "Charges, payments, refunds",
            "technical": "Bugs, outages, login",
            "sales": "Pricing plans, contract changes",
        },
    ),
    "japanese": Choice(
        instructions="この問い合わせはどの部署が担当すべきですか?",
        criteria={
            "経理": "請求・支払い・返金",
            "技術": "不具合・障害・ログイン",
            "営業": "料金プラン・契約変更",
        },
    ),
}
messages = [
    "パスワードを忘れてログインできません。",
    "来月から社員が10人増えるので、上のプランに変えたいです。",
    "請求書の宛名を会社名に変えてください。",
    "先月からアプリが起動しないのに、料金だけは引き落とされています。",
]

with TypeSafeClient() as client:
    for message in messages:
        choices = client.system_one(state=message, questions=QUESTIONS).choices
        print(
            f"{choices['english'].choice} {choices['english'].confidence:.2f}",
            f"{choices['japanese'].choice} {choices['japanese'].confidence:.2f}",
            message,
        )

実行結果

technical 1.00 技術 1.00 パスワードを忘れてログインできません。
sales 1.00 営業 1.00 来月から社員が10人増えるので、上のプランに変えたいです。
billing 0.98 経理 0.99 請求書の宛名を会社名に変えてください。
billing 0.39 技術 0.55 先月からアプリが起動しないのに、料金だけは引き落とされています。

はっきりした 3 件は、英語の質問でも日本語の質問でも同じ答えになった。経理と技術の両方に当てはまる 4 件目は、英語の質問では billing (confidence 0.39)、日本語の質問では技術 (confidence 0.55) と答えが分かれた。

複数の文書を 1 回で分類する

state に文書のリストを入れ、パスで 1 件ずつ指す質問を作る。1 件ずつ送った場合と答えとトークン数を比べる。

from typesafe_sdk import Choice, TypeSafeClient

CRITERIA = {
    "billing": "Payments, invoices, refund requests",
    "technical": "Bugs, outages, login problems",
    "sales": "Pricing plans, contract changes",
}
tickets = [
    "I forgot my password and cannot log in.",
    "We are adding 10 people next month and want a bigger plan.",
    "Please change the name on my invoice to our company name.",
    "The dashboard shows a 500 error every time I open it.",
    "I was charged twice this month.",
]

with TypeSafeClient() as client:
    batch = client.system_one(
        state={"tickets": tickets},
        questions={
            f"t{i}": Choice(instructions=f"Which team should handle `tickets[{i}]`?", criteria=CRITERIA)
            for i in range(len(tickets))
        },
    )
    single_tokens = 0
    for i, ticket in enumerate(tickets):
        single = client.system_one(
            state=ticket,
            questions={"team": Choice(instructions="Which team should handle this?", criteria=CRITERIA)},
        )
        single_tokens += single.usage.input_tokens
        print(batch.choices[f"t{i}"].choice, single.choices["team"].choice, ticket)

print("input tokens", batch.usage.input_tokens, single_tokens)

実行結果

technical technical I forgot my password and cannot log in.
sales sales We are adding 10 people next month and want a bigger plan.
billing billing Please change the name on my invoice to our company name.
technical technical The dashboard shows a 500 error every time I open it.
billing billing I was charged twice this month.
input tokens 742 1733

5 件とも 1 件ずつ送った場合と同じ答えになり、入力トークンは 1733 から 742 に減った。

質問を大量に送る

1 回のリクエストに入れる質問数を増やし、応答時間と入力トークンを見る。

import time

from typesafe_sdk import Noul, TypeSafeClient

state = "I was charged twice this month and nobody answers my emails."

with TypeSafeClient() as client:
    client.system_one(state=state, questions={"q": Noul(instructions="Is this about billing?")})
    for count in [1, 10, 50, 100, 200]:
        questions = {f"q{i}": Noul(instructions=f"Does the message mention the number {i}?") for i in range(count)}
        start = time.perf_counter()
        response = client.system_one(state=state, questions=questions)
        elapsed = (time.perf_counter() - start) * 1000
        print(count, f"{elapsed:.0f}ms", response.usage.input_tokens)

実行結果

1 310ms 287
10 363ms 431
50 264ms 1111
100 258ms 1961
200 281ms 3761

質問を 200 問にしても応答時間は 250〜370ms の範囲のままで、増えたのは入力トークンだった。

少しずつ強くなる文を Score で測る

同じ出来事を、少しずつ強い言い方にした 6 文で測る。Score が文の順に上がるかを見る。

from typesafe_sdk import Score, TypeSafeClient

ANGER = Score(
    instructions="How angry is the customer?",
    criteria=[
        "Not upset at all",
        "Mildly disappointed",
        "Clearly annoyed",
        "Angry and demanding action",
        "Furious, insulting or threatening",
    ],
)
messages = [
    "My order arrived a day late. No problem at all.",
    "My order arrived a day late. A bit disappointing.",
    "My order arrived a day late again. This is getting annoying.",
    "My order is late for the third time. Fix this now.",
    "Third late order. Your company is useless and I want a refund today.",
    "Third late order. You are useless idiots and I will make sure everyone knows.",
]

with TypeSafeClient() as client:
    for message in messages:
        answer = client.system_one(state=message, questions={"anger": ANGER}).scores["anger"]
        print(f"{answer.score:.2f} {answer.confidence:.2f} {message}")

実行結果

0.79 0.34 My order arrived a day late. No problem at all.
1.00 1.00 My order arrived a day late. A bit disappointing.
2.00 1.00 My order arrived a day late again. This is getting annoying.
3.00 1.00 My order is late for the third time. Fix this now.
3.66 0.72 Third late order. Your company is useless and I want a refund today.
4.00 1.00 Third late order. You are useless idiots and I will make sure everyone knows.

Score は文の順に 0.79 → 1 → 2 → 3 → 3.66 → 4 と上がった。1 文目は「No problem at all」と書いてあっても 0 にならず、confidence も 0.34 と低かった。

皮肉と絵文字を読ませる

字面は丁寧でも中身は怒っている文や、絵文字だけの文を Score で測る。

from typesafe_sdk import Score, TypeSafeClient

FRUSTRATION = Score(
    instructions="How frustrated is the customer?",
    criteria=[
        "Calm or satisfied",
        "Somewhat frustrated",
        "Very frustrated",
    ],
)
messages = [
    "Thanks for fixing it so quickly!",
    "Wow, a third outage this week. Truly amazing work, thanks so much.",
    "Oh great, the login page is down again. Love it.",
    "😡😡😡",
    "🙏✨",
]

with TypeSafeClient() as client:
    for message in messages:
        answer = client.system_one(state=message, questions={"frustration": FRUSTRATION}).scores["frustration"]
        print(f"{answer.score:.2f} {answer.confidence:.2f} {message}")

実行結果

0.00 1.00 Thanks for fixing it so quickly!
1.99 0.99 Wow, a third outage this week. Truly amazing work, thanks so much.
1.54 0.31 Oh great, the login page is down again. Love it.
2.00 1.00 😡😡😡
0.01 0.99 🙏✨

字面が褒め言葉の皮肉 (2 文目) は、最も高い段階 (1.99) と判定された。「Love it」で終わる 3 文目は 1.54 で、confidence は 0.31 と迷っていた。絵文字だけの文も、😡 は最も高い段階、🙏✨ は最も低い段階になった。

Choice と選択肢ごとの Noul を比べる

同じ判断を、1 つの Choice と、選択肢ごとの Noul で聞く。Choice は 1 つを選ぶが、Noul は選択肢ごとに独立して答える。

from typesafe_sdk import Choice, Noul, TypeSafeClient

TEAMS = {
    "billing": "Payments, invoices, refund requests",
    "technical": "Bugs, outages, login problems",
    "sales": "Pricing plans, contract changes",
}
messages = [
    "I cannot log in.",
    "The app has not started since last month, but I am still being charged.",
    "What is the weather like in Tokyo?",
]

with TypeSafeClient() as client:
    for message in messages:
        questions = {"choice": Choice(instructions="Which team should handle this?", criteria=TEAMS)}
        for team, description in TEAMS.items():
            questions[team] = Noul(instructions=f"Should the team for '{description}' handle this?")
        response = client.system_one(state=message, questions=questions)
        choice = response.choices["choice"].probabilities
        print(message)
        for team in TEAMS:
            print(f"  {team:9} choice {choice[team]:.2f} noul {response.nouls[team].noul:.2f}")

実行結果

I cannot log in.
  billing   choice 0.00 noul 0.09
  technical choice 1.00 noul 0.94
  sales     choice 0.00 noul 0.06
The app has not started since last month, but I am still being charged.
  billing   choice 0.99 noul 0.93
  technical choice 0.01 noul 0.34
  sales     choice 0.00 noul 0.64
What is the weather like in Tokyo?
  billing   choice 0.00 noul 0.02
  technical choice 0.99 noul 0.04
  sales     choice 0.01 noul 0.03

当てはまる選択肢が無い天気の質問でも、Choice は technical を confidence 0.99 で選んだ。Noul では 3 つとも 0.04 以下になり、どれにも当てはまらないことが分かった。両方に当てはまる 2 文目は、Choice が billing に 0.99 を集めた一方、Noul は billing 0.93・technical 0.34 と独立に答えた。

言い換えた質問をまとめて聞く

同じ判断を 3 通りの言い方で 1 回のリクエストに入れ、答えの揺れと平均を見る。

from typesafe_sdk import Noul, TypeSafeClient

PHRASINGS = [
    "Is the customer asking for their money back?",
    "Does the customer want a refund?",
    "Is the customer requesting that a payment be returned to them?",
]
messages = [
    "Please refund my last payment.",
    "I was charged twice. What are you going to do about it?",
    "Can I get a discount on next month instead?",
]

with TypeSafeClient() as client:
    for message in messages:
        nouls = client.system_one(
            state=message,
            questions={f"p{i}": Noul(instructions=text) for i, text in enumerate(PHRASINGS)},
        ).nouls
        values = [nouls[f"p{i}"].noul for i in range(len(PHRASINGS))]
        print(values, f"mean {sum(values) / len(values):.2f}", message)

実行結果

[0.98, 0.98, 0.98] mean 0.98 Please refund my last payment.
[0.83, 0.88, 0.84] mean 0.85 I was charged twice. What are you going to do about it?
[0.05, 0.09, 0.07] mean 0.07 Can I get a discount on next month instead?

言い換えによる値の差は最大 0.05 だった。はっきりしない 2 文目 (0.83〜0.88) で差が最も大きかった。

無関係な文を state に混ぜる

判断に関係のない文を state に足していき、答えがどう変わるかを見る。

from typesafe_sdk import Choice, TypeSafeClient

TEAMS = {
    "billing": "Payments, invoices, refund requests",
    "technical": "Bugs, outages, login problems",
    "sales": "Pricing plans, contract changes",
}
message = "I was charged twice this month."
filler = [f"Note {i}: the office plant on floor {i % 7} was watered today." for i in range(300)]

with TypeSafeClient() as client:
    for count in [0, 10, 50, 150, 300]:
        state = {"notes": filler[:count], "message": message}
        response = client.system_one(
            state=state,
            questions={"team": Choice(instructions="Which team should handle `message`?", criteria=TEAMS)},
        )
        answer = response.choices["team"]
        print(count, answer.choice, f"{answer.confidence:.2f}", response.usage.input_tokens)

実行結果

0 billing 1.00 357
10 billing 1.00 556
50 billing 1.00 1396
150 billing 1.00 3546
300 billing 1.00 6846

無関係なメモを 300 件足しても、答えも confidence も変わらなかった。増えたのは入力トークンで、357 から 6846 になった。

出典

公式ドキュメント: https://docs.typesafe.ai/