0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?

導入メモ(gemma4 12B) - vLLM

0
Posted at

環境メモ:Ubuntu / RTX 5080(VRAM 16GB)/ RAM 96GB。16GBでロードできるか未確認。Gemma 4 12BのQAT W4A16を使う。

環境を作る

nvidia-smi
curl -LsSf https://astral.sh/uv/install.sh | sh
# uvが見つからなければ新しい端末で続きから
uv python install 3.12
mkdir -p ~/venvs
uv venv --python 3.12 ~/venvs/gemma4-vllm
source ~/venvs/gemma4-vllm/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
python -c 'import torch, vllm; print(torch.__version__, vllm.__version__, torch.cuda.is_available())'

起動

vllm serve google/gemma-4-12B-it-qat-w4a16-ct \
  --host 127.0.0.1 --port 8000 \
  --max-model-len 4096 --max-num-seqs 1 \
  --gpu-memory-utilization 0.85
curl -sS http://127.0.0.1:8000/v1/models
curl -sS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"google/gemma-4-12B-it-qat-w4a16-ct","messages":[{"role":"user","content":"Pythonの例外処理を日本語で説明して"}],"max_tokens":128}'
  • OOMなら --max-model-len 2048。それでも厳しければE4BのQATで起動だけ先に確認。
  • CUDAが見えないときは、ドライバとPyTorchのCUDAランタイムを確認。

あとで記録:vLLM・torch・CUDAの版、ロード時間、VRAM、tokens/s、エラー。

Gemma公式の手順 / vLLMのGPUインストール

0
0
0

Register as a new user and use Qiita more conveniently

  1. You get articles that match your needs
  2. You can efficiently read back useful information
  3. You can use dark theme
What you can do with signing up
0
0

Delete article

Deleted articles cannot be recovered.

Draft of this article would be also deleted.

Are you sure you want to delete this article?