0.6.1.post1

Highlights

This release features important bug fixes and enhancements for

• Pixtral models. (#8415, #8425, #8399, #8431)
  • Chunked scheduling has been turned off for vision models. Please replace --max_num_batched_tokens 16384 with --max-model-len 16384
• Multistep scheduling. (#8417, #7928, #8427)
• Tool use. (#8423, #8366)

Also

• support multiple images for qwen-vl (#8247)
• removes engine_use_ray (#8126)
• add engine option to return only deltas or final output (#7381)
• add bitsandbytes support for Gemma2 (#8338)

What's Changed

• [MISC] Dump model runner inputs when crashing by @comaniac in https://github.com/vllm-project/vllm/pull/8305
• [misc] remove engine_use_ray by @youkaichao in https://github.com/vllm-project/vllm/pull/8126
• [TPU] Use Ray for default distributed backend by @WoosukKwon in https://github.com/vllm-project/vllm/pull/8389
• Fix the AMD weight loading tests by @mgoin in https://github.com/vllm-project/vllm/pull/8390
• [Bugfix]: Fix the logic for deciding if tool parsing is used by @tomeras91 in https://github.com/vllm-project/vllm/pull/8366
• [Gemma2] add bitsandbytes support for Gemma2 by @blueyo0 in https://github.com/vllm-project/vllm/pull/8338
• [Misc] Raise error when using encoder/decoder model with cpu backend by @kevin314 in https://github.com/vllm-project/vllm/pull/8355
• [Misc] Use RoPE cache for MRoPE by @WoosukKwon in https://github.com/vllm-project/vllm/pull/8396
• [torch.compile] hide slicing under custom op for inductor by @youkaichao in https://github.com/vllm-project/vllm/pull/8384
• [Hotfix][VLM] Fixing max position embeddings for Pixtral by @ywang96 in https://github.com/vllm-project/vllm/pull/8399
• [Bugfix] Fix InternVL2 inference with various num_patches by @Isotr0py in https://github.com/vllm-project/vllm/pull/8375
• [Model] Support multiple images for qwen-vl by @alex-jw-brooks in https://github.com/vllm-project/vllm/pull/8247
• [BugFix] lazy init _copy_stream to avoid torch init wrong gpu instance by @lnykww in https://github.com/vllm-project/vllm/pull/8403
• [BugFix] Fix Duplicate Assignment of Class Variable in Hermes2ProToolParser by @vegaluisjose in https://github.com/vllm-project/vllm/pull/8423
• [Bugfix] Offline mode fix by @joerunde in https://github.com/vllm-project/vllm/pull/8376
• [multi-step] add flashinfer backend by @SolitaryThinker in https://github.com/vllm-project/vllm/pull/7928
• [Core] Add engine option to return only deltas or final output by @njhill in https://github.com/vllm-project/vllm/pull/7381
• [Bugfix] multi-step + flashinfer: ensure cuda graph compatible by @alexm-neuralmagic in https://github.com/vllm-project/vllm/pull/8427
• [Hotfix][Core][VLM] Disable chunked prefill by default and prefix caching for multimodal models by @ywang96 in https://github.com/vllm-project/vllm/pull/8425
• [CI/Build] Disable multi-node test for InternVL2 by @ywang96 in https://github.com/vllm-project/vllm/pull/8428
• [Hotfix][Pixtral] Fix multiple images bugs by @patrickvonplaten in https://github.com/vllm-project/vllm/pull/8415
• [Bugfix] Fix weight loading issue by rename variable. by @wenxcs in https://github.com/vllm-project/vllm/pull/8293
• [Misc] Update Pixtral example by @ywang96 in https://github.com/vllm-project/vllm/pull/8431
• [BugFix] fix group_topk by @dsikka in https://github.com/vllm-project/vllm/pull/8430
• [Core] Factor out input preprocessing to a separate class by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/7329
• [Bugfix] Mapping physical device indices for e2e test utils by @ShangmingCai in https://github.com/vllm-project/vllm/pull/8290
• [Bugfix] Bump fastapi and pydantic version by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/8435
• [CI/Build] Update pixtral tests to use JSON by @DarkLight1337 in https://github.com/vllm-project/vllm/pull/8436
• [Bugfix] Fix async log stats by @alexm-neuralmagic in https://github.com/vllm-project/vllm/pull/8417
• [bugfix] torch profiler bug for single gpu with GPUExecutor by @SolitaryThinker in https://github.com/vllm-project/vllm/pull/8354
• bump version to v0.6.1.post1 by @simon-mo in https://github.com/vllm-project/vllm/pull/8440

Full Changelog: https://github.com/vllm-project/vllm/compare/v0.6.1...v0.6.1.post1