[VLM] Implement merged multimodal processor for Mllama

[Isotr0py]

• Initialize merged multimodal processor implementation for encoder-decoder LMMs

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

To avoid scope creep, let's wait until #11396 has been merged first.

[DarkLight1337]

For that PR, I'm waiting for @ywang96 to perform benchmarks to confirm the effectiveness of the cache.

[Isotr0py]

No problem, there is no rush for this PR. :)

[Isotr0py]

Oh, seems text-only explicit encoder-decoder prompt is broken now...

[DarkLight1337]

Would it be easier for the processor to handle both explicit and implicit prompts internally instead of using the shared logic with text-based models?

[Isotr0py]

Yes, I agreed that we should handle both explicit and implicit prompts in processor internally, passing separated encoder and decoder inputs to the processor is a bit awkward tbh, especially we can't identify which part the prompt is from in this case.

Let me try to do some refactoring for this.

[Isotr0py]

Both explicit and implicit prompts with text/token_ids inputs (with and without multimodal data) should work now:

Prompt

prompts = [
    {
        "prompt_token_ids": [128000, 128006,    882, 128007,    271, 128256,   3923,   1587,
           279,   2217,   1501,     30, 128009, 128006,  78191, 128007,    271],
        "multi_modal_data": {
            "image": ImageAsset("stop_sign").pil_image,
        },
    },
    {
        "prompt": "<|image|><|begin_of_text|>What is the content of this image?",
        "multi_modal_data": {
            "image": ImageAsset("stop_sign").pil_image,
        },
    },
    # encoder-only prompt
    {
        "encoder_prompt": {
            "prompt": "<|image|><|begin_of_text|>What is the content of this image?",
            "multi_modal_data": {
                "image": ImageAsset("stop_sign").pil_image,
            },
        },
        "decoder_prompt": None,
    },
    {
        "encoder_prompt": {
            "prompt_token_ids": [128000, 128006,    882, 128007,    271, 128256,   3923,   1587],
            "multi_modal_data": {
                "image": ImageAsset("stop_sign").pil_image,
            },
        },
        "decoder_prompt": None,
    },
    # encoder/decoder prompt
    {
        "encoder_prompt": {
            "prompt": "<|image|>",
            "multi_modal_data": {
                "image": ImageAsset("stop_sign").pil_image,
            },
        },
        "decoder_prompt": "<|image|><|begin_of_text|>Please describe the image.",
    },
    {
        "encoder_prompt": {
            "prompt": "<|image|>",
            "multi_modal_data": {
                "image": ImageAsset("stop_sign").pil_image,
            },
        },
        "decoder_prompt": {
            "prompt_token_ids": [128000, 128006,    882, 128007,    271, 128256,   3923,   1587],
        },
    },
    # Text-only encoder-only prompt
    {
        "encoder_prompt": {
            "prompt": "<|begin_of_text|>Write an essay about the importance of higher education.",
        },
        "decoder_prompt": None,
    },
    # Text-only encoder/decoder prompt
    {
        "encoder_prompt": {
            "prompt": "<|begin_of_text|>Write an essay about the importance of higher education.",
        },
        "decoder_prompt": "<|begin_of_text|>What is the capital of France?",
    }
]

Outputs

Decoder prompt: '<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n<|image|>What does the image show?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n', Generated text: 'The image shows a street scene with a red stop sign, a black SUV, and a Chinese-style archway in the background. \n\n* The red stop sign'
Decoder prompt: '<|image|><|begin_of_text|>What is the content of this image?', Generated text: ' The image shows a stop sign in front of a Chinese archway. The stop sign is red with white lettering and is attached to a pole. The arch'
Decoder prompt: '<|image|><|begin_of_text|>What is the content of this image?', Generated text: ' The image depicts a street scene with a stop sign, a black SUV, and a Chinese archway in the background. The stop sign is red with white letter'
Decoder prompt: '<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\n<|image|>What does', Generated text: ' the sign say?\n*Answer*: Stop'
Decoder prompt: '<|image|><|begin_of_text|>Please describe the image.', Generated text: 'The image shows a street scene with a stop sign, a car, and a Chinese archway in the background. The purpose of the image is to capture the'
Decoder prompt: '', Generated text: ' the sign say?\n*Answer*: STOP'
Decoder prompt: '<|begin_of_text|>Write an essay about the importance of higher education.', Generated text: ' Higher education is a vital component of a person’s life, and it plays a significant role in shaping their future. It is a platform that provides individuals with the'
Decoder prompt: '<|begin_of_text|>What is the capital of France?', Generated text: ' Paris\nWhat is the capital of Australia? Canberra\nWhat is the capital of China? Beijing\nWhat is the capital of India? New Delhi\nWhat is'

[DarkLight1337]

Can you add a test to tests/models/encoder_decoder/vision_language/test_mllama.py to test both explicit and implicit prompts for this model?

[DarkLight1337]

Thanks, LGTM overall. @ywang96 can you take a look at this and see if this design is reasonable to you?

[ywang96]

LGTM - I think as long as we're not breaking the user interface this is good! Thanks for working on this!

[leesh6796]

Thanks! I’ve reviewed your pull request. As far as I know, the get_kv_cache_spec function in the GPUModelRunner class (gpu_model_runner.py) does not yet support ENCODER_DECODER models. If this part remains as Not Implemented, will inference with mllama still work?

[DarkLight1337]

Encoder-decoder models in general are not supported on V1 yet. This remains true even after this PR.