模型部署

Florin Wang

2026-07-15

TECH

玩了玩vllm,然后炸了.

vllm

vllm是一个推理加速库.原始的模型已经包含了如何推理的完整方法,但是时间复杂度爆炸.如何做优化成了重点,所以有了很多推理引擎,比如vllm,SGLang以及ollama.cpp.

部署

Fluu使用WSL+vllm作为技术支撑.

环境

卡:rtx4060
KMD Version: 610.62
CUDA UMD Version: 13.3
vllm=0.9.2
transformer=4.51.3

先装vllm,然后降级transformers库,环境就配好了.
Fluu主机上的py环境相当恐怖,有一个 3.11.4 的主环境,什么东西都往里装,还有一个 3.8.? 的版本是给ciphey特供的,当时不知道有个东西叫conda.
而到了接触wsl的时候了解了更多的技术,也就有了更多的选择…

特别说明:vllm对wsl支持的不好,直接装最新版wsl最后运行的时候会报错,详见下面,有懂的可以交流一下,反正GPT说就是uva cache在wsl里面没有或者找不到,最后炸了,也就是兼容问题.

(FluGPU1) fluorine@NaraFluorine:/mnt/d/SucksFluorine/cpp/Contests$ VLLM_USE_UVA=0 vllm serve "Qwen/Qwen2.5-0.5B" --gpu-memory-utilization 0.7 --max-model-len 4096(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339](APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339]        █     █     █▄   ▄█(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339]  ▄▄ ▄█ █     █     █ ▀▄▀ █  version 0.25.0(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339]   █▄█▀ █     █     █     █  model   Qwen/Qwen2.5-0.5B(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339]    ▀▀  ▀▀▀▀▀ ▀▀▀▀▀ ▀     ▀(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339](APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:273] non-default args: {'model_tag': 'Qwen/Qwen2.5-0.5B', 'model': 'Qwen/Qwen2.5-0.5B', 'max_model_len': 4096, 'gpu_memory_utilization': 0.7}(APIServer pid=502) WARNING 07-13 13:30:09 [envs.py:2041] Unknown vLLM environment variable detected: VLLM_USE_UVA(APIServer pid=502) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.(APIServer pid=502) INFO 07-13 13:30:15 [model.py:619] Resolved architecture: Qwen2ForCausalLM(APIServer pid=502) INFO 07-13 13:30:15 [model.py:1776] Using max model len 4096(APIServer pid=502) INFO 07-13 13:30:15 [vllm.py:1042] Asynchronous scheduling is enabled.(APIServer pid=502) INFO 07-13 13:30:15 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])(EngineCore pid=559) INFO 07-13 13:32:07 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='Qwen/Qwen2.5-0.5B', speculative_config=None, tokenizer='Qwen/Qwen2.5-0.5B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Qwen/Qwen2.5-0.5B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto')(EngineCore pid=559) INFO 07-13 13:32:19 [parallel_state.py:1607] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.25.103.83:59337 backend=nccl(EngineCore pid=559) INFO 07-13 13:32:19 [parallel_state.py:1942] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A(EngineCore pid=559) INFO 07-13 13:32:19 [gpu_worker.py:360] Using V2 Model Runner(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] EngineCore failed to start.(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] Traceback (most recent call last):(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     return func(*args, **kwargs)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 966, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     super().init((EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 123, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     self.model_executor = executor_class(vllm_config)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     return func(*args, **kwargs)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/executor/abstract.py", line 109, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     self._init_executor()(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/executor/uniproc_executor.py", line 63, in _init_executor(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     self.driver_worker.init_device()(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/worker_base.py", line 331, in init_device(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     self.worker.init_device()  # type: ignore(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     ^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     return func(*args, **kwargs)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]            ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 390, in init_device(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     self.model_runner: GPUModelRunner = GPUModelRunnerV2(  # type: ignore(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/model_runner.py", line 213, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     self.req_states = RequestState((EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]                       ^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/states.py", line 33, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     self.all_token_ids = StagedWriteTensor((EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]                          ^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/buffer_utils.py", line 141, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     self._uva_buf = UvaBuffer(size, dtype)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]                     ^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/buffer_utils.py", line 47, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231]     raise RuntimeError("UVA is not available")(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] RuntimeError: UVA is not available(EngineCore pid=559) Process EngineCore:(EngineCore pid=559) Traceback (most recent call last):(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap(EngineCore pid=559)     self.run()(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/multiprocessing/process.py", line 108, in run(EngineCore pid=559)     self._target(*self._args, **self._kwargs)(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 1235, in run_engine_core(EngineCore pid=559)     raise e(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core(EngineCore pid=559)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)(EngineCore pid=559)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559)     return func(*args, **kwargs)(EngineCore pid=559)            ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 966, in init(EngineCore pid=559)     super().init((EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 123, in init(EngineCore pid=559)     self.model_executor = executor_class(vllm_config)(EngineCore pid=559)                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559)     return func(*args, **kwargs)(EngineCore pid=559)            ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/executor/abstract.py", line 109, in init(EngineCore pid=559)     self._init_executor()(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/executor/uniproc_executor.py", line 63, in _init_executor(EngineCore pid=559)     self.driver_worker.init_device()(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/worker_base.py", line 331, in init_device(EngineCore pid=559)     self.worker.init_device()  # type: ignore(EngineCore pid=559)     ^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559)     return func(*args, **kwargs)(EngineCore pid=559)            ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 390, in init_device(EngineCore pid=559)     self.model_runner: GPUModelRunner = GPUModelRunnerV2(  # type: ignore(EngineCore pid=559)                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/model_runner.py", line 213, in init(EngineCore pid=559)     self.req_states = RequestState((EngineCore pid=559)                       ^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/states.py", line 33, in init(EngineCore pid=559)     self.all_token_ids = StagedWriteTensor((EngineCore pid=559)                          ^^^^^^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/buffer_utils.py", line 141, in init(EngineCore pid=559)     self._uva_buf = UvaBuffer(size, dtype)(EngineCore pid=559)                     ^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/buffer_utils.py", line 47, in init(EngineCore pid=559)     raise RuntimeError("UVA is not available")(EngineCore pid=559) RuntimeError: UVA is not available[rank0]:[W713 13:32:20.457347550 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())(APIServer pid=502) Traceback (most recent call last):(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/bin/vllm", line 6, in <module>(APIServer pid=502)     sys.exit(main())(APIServer pid=502)              ^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/cli/main.py", line 95, in main(APIServer pid=502)     args.dispatch_function(args)(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/cli/serve.py", line 148, in cmd(APIServer pid=502)     uvloop.run(run_server(args))(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/uvloop/init.py", line 92, in run(APIServer pid=502)     return runner.run(wrapper())(APIServer pid=502)            ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/asyncio/runners.py", line 118, in run(APIServer pid=502)     return self._loop.run_until_complete(task)(APIServer pid=502)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/uvloop/init.py", line 48, in wrapper(APIServer pid=502)     return await main(APIServer pid=502)            ^^^^^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 698, in run_server(APIServer pid=502)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 712, in run_server_worker(APIServer pid=502)     async with build_async_engine_client((APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/contextlib.py", line 210, in aenter(APIServer pid=502)     return await anext(self.gen)(APIServer pid=502)            ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client(APIServer pid=502)     async with build_async_engine_client_from_engine_args((APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/contextlib.py", line 210, in aenter(APIServer pid=502)     return await anext(self.gen)(APIServer pid=502)            ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args(APIServer pid=502)     async_llm = AsyncLLM.from_vllm_config((APIServer pid=502)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config(APIServer pid=502)     return cls((APIServer pid=502)            ^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 146, in init(APIServer pid=502)     self.engine_core = EngineCoreClient.make_async_mp_client((APIServer pid=502)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(APIServer pid=502)     return func(*args, **kwargs)(APIServer pid=502)            ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 132, in make_async_mp_client(APIServer pid=502)     return AsyncMPClient(*client_args)(APIServer pid=502)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(APIServer pid=502)     return func(*args, **kwargs)(APIServer pid=502)            ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 963, in init(APIServer pid=502)     super().init((APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 573, in init(APIServer pid=502)     with launch_core_engines((APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/contextlib.py", line 144, in exit(APIServer pid=502)     next(self.gen)(APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/utils.py", line 1213, in launch_core_engines(APIServer pid=502)     wait_for_engine_startup((APIServer pid=502)   File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/utils.py", line 1272, in wait_for_engine_startup(APIServer pid=502)     raise RuntimeError((APIServer pid=502) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}

装一下老版的vllm就没事了,gpt说新版本的vllm大量引入uva所以才有问题.

体验

装好vllm之后直接上手,也算是用到自己部署的模型了.

下面的模型直接接cherry shudio选本地模型提供商就行,api空着.

Qwen2.5-0.5B

1 2	vllm serve Qwen/Qwen2.5-0.5B \ --gpu-memory-utilization 0.7

vllm默认好像显卡会占0.9,然而我的电脑还得跑桌面所以会崩,此时要调一下占比,0.7就够了.

qwen模型很蠢,写一个最简单的编程都写不对.
不过也不怪qwen,模型参数量确实太小了,当成玩具还差不多,本地部署的推理速度还挺快,可能是模型真的太小了.

做了一个简单的编程测试,qwen连括号匹配都做不好.

DeepSeek-R1-Distill-Qwen-1.5B

1
2
3

vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.7

deepseek默认的上下文非常长,默认占用的空间会很大占爆内存,需要限制一下.

Qwen/Qwen2.5-Coder-1.5B-Instruct

做了做abc的AB题还行,C就不行了.1.5B还是有点吃力.

Qwen2.5-Coder-3B-Instruct

部署失败,因为显存不够,开的最大0.85占比的GPU和2048的上下文.
问gpt说vllm追求高吞吐量就会疯狂占显存,所以就不够用,不要折腾vllm了.

nano-vllm

这个一千二百行的代码能实现一个简易的vllm…
因为新版本的vllm大量引入uva,而这个特性是wsl不支持的,所以只能装旧版本的vllm,而旧版本的不支持qwen3…

而且vllm为了高并发会占很高很高的显存,很容易oom(在服务正式启动之前分显存的时候就oom了)

Qwen—Qwen2.5-Coder-1.5B-Instruct

部署失败(基本成功,部署成功但是推理失败),原因是新版本的Attention和旧版本不一致,所以就炸了.
来看看吃错药的Qwen2.5:(下面的是最逆天的一段)

{'text': '18岁，已经过了成年期，所以你不能再生育了，所以你不 能结婚。 对吗？ 不，你不能结婚，但可以生育。 成年期并不是不能结婚的年龄，而是要达到一定的生育能力和生育意愿。如果你已经过了成年期，那么你就可以结婚，', 'token_ids': [16, 23, 92015, 3837, 99461, 102346, 12857, 7948, 22704, 3837, 99999, 56568, 112268, 103042, 34187, 3837, 99999, 56568, 53153, 104388, 1773, 69162, 101037, 11319, 86009, 3837, 56568, 53153, 104388, 3837, 77288, 73670, 103042, 1773, 86546, 7948, 22704, 102095, 53153, 104388, 9370, 102185, 3837, 103955, 30534, 100366, 102495, 103042, 106712, 103042, 107869, 1773, 102056, 99461, 102346, 12857, 7948, 22704, 3837, 100624, 56568, 102003, 104388, 3837]}

因为模型在扯淡所以推理速度非常快:153/153 00:02<00:00, 55.94it/s, Prefill=801tok/s, Decode=3730tok/s.

Qwen3-1.7B

部署成功. Prefill=10tok/s, Decode=38tok/s 属于勉强能用的水平,4060卡还是不要奢求太多…

经过测试,这个1.7B的模型不能过cf的A,主要原因是思维链太低效了,10000的max token都没拿下A,废物,给段样例:

Prompt: '<|im_start|>user\nhow many "r" characters are in the word "strawberry"?<|im_end|>\n<|im_start|>assistant\n'
Completion: '<think>\nOkay, let me try to figure out how many "r" characters are in the word "strawberry". 

First, I need to write down the word and check each letter one by one.\n\nThe word is "strawberry". Let me spell it out: S-T-R-A-W-B-E-R-R-Y. Wait, is that right? Let me confirm. S-T-R-A-W-B-E-R-R-Y. Yeah, that\'s correct. Let me break it down letter by letter.\n\nStarting with the first letter: S. Then T. Next is R. So that\'s the first R. Then A, W, B, E, then R again. So that\'s the second R. Then another R? Wait, after E, there\'s R, then another R? Let me check again. The word is S-T-R-A-W-B-E-R-R-Y. So after the E, there\'s R, then another R. So that\'s two R\'s in the middle. Wait, let me count again.\n\nBreaking it down:\n\n1. S\n2. T\n3. R (first R)\n4. A\n5. W\n6. B\n7. E\n8. R (second R)\n9. R (third R)\n10. Y\n\nSo that\'s three R\'s? 

Wait, no. Wait, the word is S-T-R-A-W-B-E-R-R-Y. So positions 3, 8, and 9. So that\'s three R\'s? Wait, but when I first thought, I thought there were two. Wait, maybe I miscounted. Let me write it out again:\n\nS T R A W B E R R Y\n\nSo after the E, there\'s R, then another R. So that\'s two R\'s. Wait, but the third R is the one after the second R? Wait, no. Let me count the letters:\n\nLetters in "strawberry" are:\n\n1. S\n2. T\n3. R\n4. A\n5. W\n6. B\n7. E\n8. R\n9. R\n10. Y\n\nSo positions 3, 8, and 9. That\'s three R\'s. Wait, but when I first thought, I thought there were two. Maybe I made a mistake earlier. 

Let me check again. The word is strawberry. Let me spell it out again: S-T-R-A-W-B-E-R-R-Y. Yes, so after the E, there are two R\'s. So the third R is the second R in the middle. Wait, no, the letters are S-T-R-A-W-B-E-R-R-Y. So the third letter is R, then after E, it\'s R, then another R. So that\'s three R\'s. Wait, but maybe the user is asking for the number of R\'s, so the answer is three?\n\n

Wait, but I need to make sure. Let me check the spelling of "strawberry". The correct spelling is S-T-R-A-W-B-E-R-R-Y. So yes, that\'s three R\'s. The first R is at position 3, then the next R is at 8, and the third R is at 9. So three R\'s. Therefore, the answer should be 3.\n\n

But wait, maybe I\'m missing something. Let me check again. Sometimes people might miscount. Let me write the letters with numbers:\n\n1: S\n2: T\n3: R\n4: A\n5: W\n6: B\n7: E\n8: R\n9: R\n10: Y\n\nYes, that\'s three R\'s. So the answer is 3. Therefore, the number of "r" characters in "strawberry" is 3.\n</think>\n\n

The word "strawberry" is spelled as **S-T-R-A-W-B-E-R-R-Y**. By examining each letter:\n\n1. **S**  \n2. **T**  \n3. **R** (first "r")  \n4. **A**  \n5. **W**  \n6. **B**  \n7. **E**  \n8. **R** (second "r")  \n9. **R** (third "r")  \n10. **Y**\n\nThere are **three "r" characters** in the word.\n\n**Answer:** 3<|im_end|>'

数一个字符个数,Qwen足足检查了4遍,而且不算一遍计算中间的重新检查.

Qwen3.5

ASR

语音识别模型