玩了玩vllm,然后炸了.
vllm
vllm是一个推理加速库.原始的模型已经包含了如何推理的完整方法,但是时间复杂度爆炸.如何做优化成了重点,所以有了很多推理引擎,比如vllm,SGLang以及ollama.cpp.
部署
Fluu使用WSL+vllm作为技术支撑.
环境
卡:rtx4060
KMD Version: 610.62
CUDA UMD Version: 13.3
vllm=0.9.2
transformer=4.51.3
先装vllm,然后降级transformers库,环境就配好了.
Fluu主机上的py环境相当恐怖,有一个 3.11.4 的主环境,什么东西都往里装,还有一个 3.8.? 的版本是给ciphey特供的,当时不知道有个东西叫conda.
而到了接触wsl的时候了解了更多的技术,也就有了更多的选择…
特别说明:vllm对wsl支持的不好,直接装最新版wsl最后运行的时候会报错,详见下面,有懂的可以交流一下,反正GPT说就是uva cache在wsl里面没有或者找不到,最后炸了,也就是兼容问题.
1 | (FluGPU1) fluorine@NaraFluorine:/mnt/d/SucksFluorine/cpp/Contests$ VLLM_USE_UVA=0 vllm serve "Qwen/Qwen2.5-0.5B" --gpu-memory-utilization 0.7 --max-model-len 4096(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339](APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339] █ █ █▄ ▄█(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.25.0(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339] █▄█▀ █ █ █ █ model Qwen/Qwen2.5-0.5B(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀(APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:339](APIServer pid=502) INFO 07-13 13:30:09 [api_utils.py:273] non-default args: {'model_tag': 'Qwen/Qwen2.5-0.5B', 'model': 'Qwen/Qwen2.5-0.5B', 'max_model_len': 4096, 'gpu_memory_utilization': 0.7}(APIServer pid=502) WARNING 07-13 13:30:09 [envs.py:2041] Unknown vLLM environment variable detected: VLLM_USE_UVA(APIServer pid=502) Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.(APIServer pid=502) INFO 07-13 13:30:15 [model.py:619] Resolved architecture: Qwen2ForCausalLM(APIServer pid=502) INFO 07-13 13:30:15 [model.py:1776] Using max model len 4096(APIServer pid=502) INFO 07-13 13:30:15 [vllm.py:1042] Asynchronous scheduling is enabled.(APIServer pid=502) INFO 07-13 13:30:15 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])(EngineCore pid=559) INFO 07-13 13:32:07 [core.py:114] Initializing a V1 LLM engine (v0.25.0) with config: model='Qwen/Qwen2.5-0.5B', speculative_config=None, tokenizer='Qwen/Qwen2.5-0.5B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=Qwen/Qwen2.5-0.5B, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto')(EngineCore pid=559) INFO 07-13 13:32:19 [parallel_state.py:1607] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.25.103.83:59337 backend=nccl(EngineCore pid=559) INFO 07-13 13:32:19 [parallel_state.py:1942] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A(EngineCore pid=559) INFO 07-13 13:32:19 [gpu_worker.py:360] Using V2 Model Runner(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] EngineCore failed to start.(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] Traceback (most recent call last):(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] return func(*args, **kwargs)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 966, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] super().init((EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 123, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] self.model_executor = executor_class(vllm_config)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] return func(*args, **kwargs)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/executor/abstract.py", line 109, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] self._init_executor()(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/executor/uniproc_executor.py", line 63, in _init_executor(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] self.driver_worker.init_device()(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/worker_base.py", line 331, in init_device(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] self.worker.init_device() # type: ignore(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] return func(*args, **kwargs)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 390, in init_device(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] self.model_runner: GPUModelRunner = GPUModelRunnerV2( # type: ignore(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/model_runner.py", line 213, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] self.req_states = RequestState((EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/states.py", line 33, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] self.all_token_ids = StagedWriteTensor((EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/buffer_utils.py", line 141, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] self._uva_buf = UvaBuffer(size, dtype)(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] ^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/buffer_utils.py", line 47, in init(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] raise RuntimeError("UVA is not available")(EngineCore pid=559) ERROR 07-13 13:32:19 [core.py:1231] RuntimeError: UVA is not available(EngineCore pid=559) Process EngineCore:(EngineCore pid=559) Traceback (most recent call last):(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap(EngineCore pid=559) self.run()(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/multiprocessing/process.py", line 108, in run(EngineCore pid=559) self._target(*self._args, **self._kwargs)(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 1235, in run_engine_core(EngineCore pid=559) raise e(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 1200, in run_engine_core(EngineCore pid=559) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)(EngineCore pid=559) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559) return func(*args, **kwargs)(EngineCore pid=559) ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 966, in init(EngineCore pid=559) super().init((EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 123, in init(EngineCore pid=559) self.model_executor = executor_class(vllm_config)(EngineCore pid=559) ^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559) return func(*args, **kwargs)(EngineCore pid=559) ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/executor/abstract.py", line 109, in init(EngineCore pid=559) self._init_executor()(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/executor/uniproc_executor.py", line 63, in _init_executor(EngineCore pid=559) self.driver_worker.init_device()(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/worker_base.py", line 331, in init_device(EngineCore pid=559) self.worker.init_device() # type: ignore(EngineCore pid=559) ^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(EngineCore pid=559) return func(*args, **kwargs)(EngineCore pid=559) ^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 390, in init_device(EngineCore pid=559) self.model_runner: GPUModelRunner = GPUModelRunnerV2( # type: ignore(EngineCore pid=559) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/model_runner.py", line 213, in init(EngineCore pid=559) self.req_states = RequestState((EngineCore pid=559) ^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/states.py", line 33, in init(EngineCore pid=559) self.all_token_ids = StagedWriteTensor((EngineCore pid=559) ^^^^^^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/buffer_utils.py", line 141, in init(EngineCore pid=559) self._uva_buf = UvaBuffer(size, dtype)(EngineCore pid=559) ^^^^^^^^^^^^^^^^^^^^^^(EngineCore pid=559) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/worker/gpu/buffer_utils.py", line 47, in init(EngineCore pid=559) raise RuntimeError("UVA is not available")(EngineCore pid=559) RuntimeError: UVA is not available[rank0]:[W713 13:32:20.457347550 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())(APIServer pid=502) Traceback (most recent call last):(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/bin/vllm", line 6, in <module>(APIServer pid=502) sys.exit(main())(APIServer pid=502) ^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/cli/main.py", line 95, in main(APIServer pid=502) args.dispatch_function(args)(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/cli/serve.py", line 148, in cmd(APIServer pid=502) uvloop.run(run_server(args))(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/uvloop/init.py", line 92, in run(APIServer pid=502) return runner.run(wrapper())(APIServer pid=502) ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/asyncio/runners.py", line 118, in run(APIServer pid=502) return self._loop.run_until_complete(task)(APIServer pid=502) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/uvloop/init.py", line 48, in wrapper(APIServer pid=502) return await main(APIServer pid=502) ^^^^^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 698, in run_server(APIServer pid=502) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 712, in run_server_worker(APIServer pid=502) async with build_async_engine_client((APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/contextlib.py", line 210, in aenter(APIServer pid=502) return await anext(self.gen)(APIServer pid=502) ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client(APIServer pid=502) async with build_async_engine_client_from_engine_args((APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/contextlib.py", line 210, in aenter(APIServer pid=502) return await anext(self.gen)(APIServer pid=502) ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args(APIServer pid=502) async_llm = AsyncLLM.from_vllm_config((APIServer pid=502) ^^^^^^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config(APIServer pid=502) return cls((APIServer pid=502) ^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/async_llm.py", line 146, in init(APIServer pid=502) self.engine_core = EngineCoreClient.make_async_mp_client((APIServer pid=502) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(APIServer pid=502) return func(*args, **kwargs)(APIServer pid=502) ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 132, in make_async_mp_client(APIServer pid=502) return AsyncMPClient(*client_args)(APIServer pid=502) ^^^^^^^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper(APIServer pid=502) return func(*args, **kwargs)(APIServer pid=502) ^^^^^^^^^^^^^^^^^^^^^(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 963, in init(APIServer pid=502) super().init((APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/core_client.py", line 573, in init(APIServer pid=502) with launch_core_engines((APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/contextlib.py", line 144, in exit(APIServer pid=502) next(self.gen)(APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/utils.py", line 1213, in launch_core_engines(APIServer pid=502) wait_for_engine_startup((APIServer pid=502) File "/mnt/d/ZRepo/conda/miniforge3/envs/FluGPU1/lib/python3.11/site-packages/vllm/v1/engine/utils.py", line 1272, in wait_for_engine_startup(APIServer pid=502) raise RuntimeError((APIServer pid=502) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {} |
装一下老版的vllm就没事了,gpt说新版本的vllm大量引入uva所以才有问题.
体验
装好vllm之后直接上手,也算是用到自己部署的模型了.
下面的模型直接接cherry shudio选本地模型提供商就行,api空着.
Qwen2.5-0.5B
1 | vllm serve Qwen/Qwen2.5-0.5B \ |
vllm默认好像显卡会占0.9,然而我的电脑还得跑桌面所以会崩,此时要调一下占比,0.7就够了.
qwen模型很蠢,写一个最简单的编程都写不对.
不过也不怪qwen,模型参数量确实太小了,当成玩具还差不多,本地部署的推理速度还挺快,可能是模型真的太小了.
做了一个简单的编程测试,qwen连括号匹配都做不好.
DeepSeek-R1-Distill-Qwen-1.5B
1 | vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B \ |
deepseek默认的上下文非常长,默认占用的空间会很大占爆内存,需要限制一下.
abc463测试
Qwen/Qwen2.5-Coder-1.5B-Instruct
做了做abc的AB题还行,C就不行了.1.5B还是有点吃力.
Qwen3
部署失败,因为Fluu的vllm版本太低,而版本高一些用uva之后就不能跑在wsl了.
Qwen2.5-Coder-3B-Instruct
部署失败,因为显存不够,开的最大0.85占比的GPU和2048的上下文.
问gpt说vllm追求高吞吐量就会疯狂占显存,所以就不够用,不要折腾vllm了.
ASR
语音识别模型