Kong AI GatewayとDocker Model Runnerをつないでみる

はじめに

今回は Kong の機能にある AI Gatewayを利用するために
Docker Model runner と接続するところまで目標に実行してみます

Kong AI Gateway

Kong AI Gatewayについてはすでにまとめてくださっている記事があるのでいただいてきます( @m5takeda さんありがとうございます)

Docker Model Runner

Docker Model Runner の概要は過去記事転載します
今回は、Modelとして ai/smollm3:Q4_K_M を利用しますので、事前に取得をしてある前提としています

実装

サンプル用意しました

Kong Gateway 利用方式

Kong 設定値は以下です

_format_version: "3.0"
services:
- name: dmr
  url: http://model-runner.docker.internal/engines/v1/chat/completions
  routes:
  - name: dmr-route
    paths:
    - /

Kongにアクセスしたリクエストはすべて Model Runnerに送付設定としています

以下、 リクエストを投げてみました
(回答はめちゃくちゃですが、、、)

$ curl http://localhost/ -H "Content-Type: application/json"   -d '{
    "model": "ai/smollm3:Q4_K_M",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Please write 100 words about abi inc."}
    ]
  }'
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"Abi Inc. is a technology company that focuses on developing innovative artificial intelligence solutions. Founded in 2001, the company has grown significantly since its inception, leveraging its expertise in AI to create cutting-edge products and services. Abi Inc. is known for its work in natural language processing, computer vision, and robotics, with clients ranging from Fortune 500 companies to startups in various industries. The company's commitment to research and development has allowed it to stay ahead of the curve in the rapidly evolving AI landscape. Abi Inc. is headquartered in San Francisco, California, and is dedicated to making AI accessible to a wider audience."}}],"created":1763095768,"model":"ai/smollm3:Q4_K_M","system_fingerprint":"b1-97d5117","object":"chat.completion","usage":{"completion_tokens":132,"prompt_tokens":60,"total_tokens":192},"id":"chatcmpl-zsFtppbXbKMZkVU6PovqRF1rYEwNaVns","timings":{"cache_n":59,"prompt_n":1,"prompt_ms":37.417,"prompt_per_token_ms":37.417,"prompt_per_second":26.725819814522808,"predicted_n":132,"predicted_ms":4827.782,"predicted_per_token_ms":36.57410606060606,"predicted_per_second":27.34174823966782}}

AI Proxy化

AI ProxyというPluginを用いて再現してみます
ここからがAI Gatewayの本領発揮です
なお、この場合ServiceのURLは使用しなくなります

_format_version: "3.0"
services:
- name: dmr
  url: http://localhost
  routes:
  - name: dmr-route
    paths:
    - /
  plugins:
  - name: ai-proxy
    config:
      route_type: llm/v1/chat
      model:
        provider: llama2
        name: ai/smollm3:Q4_K_M
        options:
          llama2_format: openai
          upstream_url: http://model-runner.docker.internal:80/engines/v1/chat/completions

upstream_urlに port:80 を明示する必要があったので注意

$ curl http://localhost/ -H "Content-Type: application/json"   -d '{
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Please write 100 words about ABI inc."}
    ]
  }'
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","content":"ABI International, established in 1965, is a leading provider of software solutions and services for the enterprise market. They specialize in enterprise resource planning (ERP) and supply chain management (SCM) software, offering platforms like ABMs and APICS. ABI's cloud-based solutions help businesses streamline operations, improve efficiency, and enhance decision-making processes. With a focus on innovation and customer satisfaction, ABI aims to deliver scalable and cost-effective solutions tailored to meet the evolving needs of businesses worldwide."}}],"created":1763095970,"model":"ai/smollm3:Q4_K_M","system_fingerprint":"b1-97d5117","object":"chat.completion","usage":{"completion_tokens":103,"prompt_tokens":60,"total_tokens":163},"id":"chatcmpl-dz2zAWoC9tzqELr8wX4jzrXpFjx23hgT","timings":{"cache_n":59,"prompt_n":1,"prompt_ms":37.0,"prompt_per_token_ms":37.0,"prompt_per_second":27.027027027027028,"predicted_n":103,"predicted_ms":3763.646,"predicted_per_token_ms":36.54025242718447,"predicted_per_second":27.367079688153453}}

Modelの制約をPlugin側でつけたため、リクエストでの指定が不要になりました

AI Prompt Decorator 追加

PromptをGatewayで付与することができます
こいつも追加してみます

_format_version: "3.0"
services:
- name: dmr
  url: http://localhost
  routes:
  - name: dmr-route
    paths:
    - /
  plugins:
  - name: ai-proxy
    config:
      route_type: llm/v1/chat
      model:
        provider: llama2
        name: ai/smollm3:Q4_K_M
        options:
          llama2_format: openai
          upstream_url: http://model-runner.docker.internal:80/engines/v1/chat/completions
  - name: ai-prompt-decorator
    config:
      prompts: 
        prepend:
        - role: system
          content: You are a helpful assistant. Always respond in Japanese.

リクエストを投げてみます

$ curl http://localhost/ -H "Content-Type: application/json"   -d '{
    "messages": [
      {"role": "user", "content": "Please write 100 words about ABI inc."}
    ]
  }'
{"choices":[{"finish_reason":"stop","index":0,"message":{"role":"assistant","reasoning_content":"Okay, the user asked for a 100-word summary about ABI Inc. Let me start by recalling what I know about ABI. ABI is a company known for its semiconductor division, particularly in the field of microprocessors. They have been around for a while, maybe since the 1980s or 90s. I think they were involved in developing the PowerPC architecture, which was a significant contribution to the computer industry.\n\nI should mention their current business areas, like semiconductor design and manufacturing. They might also have a focus on energy efficiency and sustainability, which is a big trend in the tech world. It's important to highlight their market position, maybe their ranking in the semiconductor industry. They're based in the US, so that's another point to include.\n\nI need to make sure the summary is concise and covers the key aspects: history, products, market focus, and maybe their impact on the industry. Let me check if there are any recent developments, like mergers or new product lines. If I don't have the latest info, it's better to stick with well-established facts. Also, I should avoid any jargon that might not be familiar to everyone. Let me structure the summary in a logical flow, starting with their founding and key products, then their current focus areas, and their market significance. That should fit within 100 words.","content":"ABI, Inc. は、1980年代から半導体設計と製造に特化した企業として知られ、特にPowerPCプロセッサの開発に重要な役割を果たしてきた。米国・カリフォルニア州に本拠を置き、半導体設計から製造、そしてソフトウェアまで幅広いサービスを提供する企業として、業界のパートナーと位置づけられている。PowerPCプロセッサは、IBMとAppleの協業プロジェクトによって生まれたもので、当時のPCやラップトップ業界に大きな影響を与えた。近年ではエネルギー効率化やエコフレンドリーな製造プロセスにも注力し、環境問題への取り組みを強化している。ABIの製品は、モバイルデバイスから産業用機器まで、幅広い用途に活用されているが、業界のトップレベルに位置することは少ない。2023年現在は、IntelとARMに匹敵する市場シェアを保ちながら、業界リーダーとしての地位を維持している。"}}],"created":1763096986,"model":"ai/smollm3:Q4_K_M","system_fingerprint":"b1-97d5117","object":"chat.completion","usage":{"completion_tokens":539,"prompt_tokens":65,"total_tokens":604},"id":"chatcmpl-6uFonzTJdTgumrpKXXcz6wHXhFSdgUrU","timings":{"cache_n":47,"prompt_n":18,"prompt_ms":547.931,"prompt_per_token_ms":30.440611111111114,"prompt_per_second":32.85085165832924,"predicted_n":539,"predicted_ms":23585.893,"predicted_per_token_ms":43.758614100185525,"predicted_per_second":22.852643315222366}}

日本語で帰ってくるようになりました
(例によって我々と違うABI社の内容ですが、、、)