[Model] Ultravox Model: Support v0.5 Release

[farzadab]

Hi there!

We (cc @petersalas) are preparing for the v0.5 release of Ultravox audio/text to text model. This release does some minor adjustments to the architecture (moves a layer norm before the last layer of the projector).
The previous versions are still supported through the config parameter projector_ln_mid.

I'm gonna mark this as draft for now until I can verify everything. I'm also gonna add a test for v0.5 going forward once it becomes public.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[ywang96]

@farzadab @petersalas Thanks for the contribution! Feel freel to ping me here or on slack whenever this is ready for review!

[petersalas]

I'm also gonna add a test for v0.5 going forward once it becomes public.

I suspect we should probably just update the test to use the newest model (after smoke testing that the new code still works on 0.3 and 0.4.1)? And maybe switch to using the 1B model as long as it can demonstrate some understanding of the sample audios.