-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[QUESTION] My SparkApplication won't start after migrating to spark:3.5.0 #2088
Comments
Is there any specific reason for setting the working directory to |
@networkingana Have you tried running the app with user 185 and group 185, as the spark user's uid and gid are 185. $ kubectl run --rm --restart=Never -i -t spark --image=spark:3.5.0 -- bash
If you don't see a command prompt, try pressing enter.
spark@spark:/opt/spark/work-dir$ id
uid=185(spark) gid=185(spark) groups=185(spark) |
I resolved this issue with changing my Dockerfile, correcting the permissions and ownership of my files. One strange thing is that my application doesn't print its logs
i'm using the same log4j2.xml file as with older spark versions |
@networkingana |
Same issue here, do you have any updates on it ? |
I ended up writing my dockerfile like this
I think the key thing was going back to workdir /opt/spark/work-dir and as well owning my jar files by the spark user which is in the spark:3.5.0 base image |
I changed the base image from gcr.io spark:3.1.1 to spark:3.5.0 and built my dockerfile from there where it has my DataAnalyticsReporting.jar
This is the error when the spark-operator tries to start my application
`
This is my SparkApplication YAML
and this is my Dockerfile
`
I tried to play and change different users, securityContext and podSecurityContext, but with no success, please let me know how should i configure this
The text was updated successfully, but these errors were encountered: