Truncated errors logs after upgrading from v1beta2-1.6.1 to v2.1.0

[NicoMovh]

What happened?

I've encountered problems when upgrading from v1beta2-1.6.1 to v2.1.0. I manage to make the controller works but the spark-submit fails everytime without proper error message.
The error message is truncated and make debugging almost impossible.
SparkApplication fails with truncated error message after NativeCodeLoader warning
Error only shows:
Error Message: failed to run spark-submit: failed to run spark-submit:

My guess is an issues with permissions or resources are missing that was added in the v2.1.0 and is not allowed on my Kubernetes Cluster for securities reasons. But I can't be sure of that because there's no proper error message. But it was working flawlessly on v1beta2-1.6.1 and there has been a lot of changes.

Reproduction Code

1. Deploy spark-operator v2.1.0 with default configurations
2. Deploy basic SparkPi application
3. Observe truncated error message in events and logs

Expected behavior

Clear error message
Full stack trace or detailed error information about the spark-submit failure

Actual behavior

When I start my SparkPi exemple application. I get have this status:
Status: Application State: Error Message: failed to run spark-submit: failed to run spark-submit:
On the Controller:
INFO sparkapplication/controller.go:663 Failed to submit SparkApplication {"name": "spark-pi3", "namespace": "namespace-workers", "state": "SUBMISSION_FAILED", "error": "failed to run spark-submit: failed to run spark-submit:
DEBUG events recorder/recorder.go:104 failed to submit SparkApplication spark-pi3: failed to run spark-submit: failed to run spark-submit:

Environment & Versions

• Kubernetes Version: v1.29.9
• Spark Operator Version: v2.1.0
• Apache Spark Version: 3.5.3

Additional context

The operator should provide more verbose error reporting
Consider adding a validation mode to check permissions before attempting spark-submit

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍