LM/vLLM
< LM
compose
services:
vllm-xpu:
image: vllm/vllm-openai-xpu:v0.30.0
container_name: vllm-xpu-qwen38-exp
devices:
- /dev/dri:/dev/dri
volumes:
- ${HOME}/.cache/huggingface:/root/.cache/huggingface
environment:
- HF_HOME=/root/.cache/huggingface
- HF_TOKEN=${HF_TOKEN:-}
- no_proxy=localhost,127.0.0.1
- ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
- ZE_AFFINITY_MASK=0
- VLLM_WORKER_MULTIPROC_METHOD=spawn
- PYTORCH_ALLOC_CONF=expandable_segments:True
- VLLM_XPU_ENABLE_XPU_GRAPH=1
ports:
- "9000:9000"
entrypoint: ["vllm", "serve"]
command:
- "SergiioB/Qwen3.8-27B-GPTQ-Int4-sym-G128-MTP-BF16"
- "--served-model-name=qwen38"
- "--port=9000"
- "--host=0.0.0.0"
- "--quantization=gptq"
- "--dtype=float16"
- "--max-model-len=131072"
- "--gpu-memory-utilization=0.88"
- "--kv-cache-dtype=fp8"
- "--max-num-seqs=1"
- "--max-num-batched-tokens=8192"
- "--trust-remote-code"
- "--enable-auto-tool-choice"
- "--tool-call-parser=qwen3_xml"
- "--reasoning-parser=qwen3"
- "--limit-mm-per-prompt={\"image\":10}"
- "--speculative-config={\"method\":\"mtp\",\"num_speculative_tokens\":4}"
restart: no
Verify environments in container
| Command | Results in vllm v0.30.0 |
|---|---|
docker exec vllm-xpu-qwen38-prod bash -c \
"pip list | awk 'NR<3 || tolower(\$0) ~ /torch|intel|triton|vllm|oneccl|mkl|dpc|level/'"
|
Package Version Editable project location ---------------------------------------- --------------- ------------------------------------- dpcpp-cpp-rt 2026.0.0 intel-cmplr-lib-rt 2026.0.0 intel-cmplr-lib-ur 2026.0.0 intel-cmplr-lic-rt 2026.0.0 intel-opencl-rt 2026.0.0 intel-openmp 2026.0.0 intel-pti 0.17.0 intel-sycl-rt 2026.0.0 mkl 2026.0.0 oneccl 2022.0.0 oneccl-devel 2022.0.0 onemkl-license 2026.0.0 onemkl-sycl-blas 2026.0.0 onemkl-sycl-dft 2026.0.0 onemkl-sycl-lapack 2026.0.0 onemkl-sycl-rng 2026.0.0 onemkl-sycl-sparse 2026.0.0 open_clip_torch 2.32.0 torch 2.13.0+xpu torchaudio 2.11.0+xpu torchcodec 0.16.0+cpu torchvision 0.28.0+xpu triton 3.7.2+xpu triton-xpu 3.7.2 vllm 0.30.0+xpu vllm_test_utils 0.1 /workspace/vllm/tests/vllm_test_utils vllm-xpu-kernels 0.1.14.1 |
docker exec vllm-xpu-qwen38-prod bash -c 'pip show vllm-xpu-kernels'
|
Name: vllm-xpu-kernels Version: 0.1.14.1 Summary: A high-throughput and memory-efficient inference and serving engine for LLMs Home-page: https://github.com/vllm-project/vllm-xpu-kernels Author: vLLM Team Author-email: License-Expression: Apache-2.0 Location: /opt/venv/lib/python3.12/site-packages Requires: Required-by: vllm |
docker exec vllm-xpu-qwen38-prod bash -c 'lsb_release -a'
|
Distributor ID: Ubuntu Description: Ubuntu 24.04.5 LTS Release: 24.04 Codename: noble |
docker exec -it vllm-xpu-qwen38-prod bash
|
TODO
- https://docs.vllm.ai/en/stable/cli/serve/
- https://lcz.me/topic/1293/5
- https://jonathanmann.tech/blog/qwen38-intel-arc-b70/
- https://hub.docker.com/r/intel/vllm
- https://hub.docker.com/u/intel/?page=1&search=vllm
- https://docs.vllm.ai/en/stable/getting_started/installation/gpu/#requirements
- https://www.pugetsystems.com/labs/articles/intel-arc-pro-b70-multi-gpu-ai-inference-performance/
- https://github.com/Hal9000AIML/arc-pro-b70-inference-setup-ubuntu-server
- https://sergiiob.dev/posts/intel-arc-b70-vllm-initial-mxfp4-test/
- https://forum.level1techs.com/t/intel-b70-launch-unboxed-and-tested/247873
- https://huggingface.co/blog/MatrixYao/intel-gpu