Errors in kubernetes yaml after upgrading to spark 3.5.0

[paulpaul1076]

After we updated spark operator to 3.5.0, we now get this error:

Error from server (BadRequest): error when creating "C:\Users\wnwnn\IdeaProjects\firestarter\firestarter\src\main\resources\kubernetes\app-action-event-batch.yml": SparkApplication in version "v1alpha1" cannot be handled as a SparkApplication: strict decoding error: unknown field "spec.driver.resources", unknown field "spec.driver.volumeMounts", unknown field "spec.executor.instances", unknown field "spec.executor.resources", unknown field "spec.executor.volumeMounts", unknown field "spec.sparkImagePullPolicy"

Did something change? How do you specify these things after the upgrade?

[adwk67]

Thanks for the feedback @paulpaul1076! There were a number of breaking changes in the 23.11 release and we have endeavoured to flag these up and to document how to redefine the resource yamls in the release notes. For the spark-k8s-operator you can find this here. In a nutshell you will need to move the configuration elements (volumes, logging, resources etc.) down one level to be underneath "config", using the new product image structure, and renaming the executor instances to replicas.

[adwk67]

And....apologies for not consistently carrying this through to the operator docs: I see that here the config level has not yet been included. I'll open an issue for that.

[adwk67]

Issue opened here.