"Command failed with exit code X" in shell script mode is not too informative.

[ThePlenkov]

Context

I'm building some Glue shell python Job in Glue. I want to introduce a required parameter. My code looks like this:

import sys
from awsglue.utils import getResolvedOptions

def validate_arguments():
    # This will raise a RuntimeError if 'required_arg' is not provided
    return getResolvedOptions(sys.argv, ['required_arg'])

try:
    validate_arguments()
except Exception as e:  # Catch any exception
    print(f"Caught an exception of type {type(e).__name__}: {e}")

Problem

Of course, when I run this code without a given parameter - it should fail. But currently I only have this: Command failed with exit code 2 as the error message in UI.

Here is how it looks:

If I go to error log I can see there the actual error:

So it seems like my fallback logic is not even picked up and the process stops immeadiately.

I also noticed that in the job run attributes Glue version is 1. Why like that? I don't even have the option to choose it.

Happy to hear any suggestions.

Thanks

Petr

[ThePlenkov]

I think I can narrow down the scope now:

If I run the script as simple as that:

import sys
sys.exit("My own reason to exit this script which I want to see in UI")

I have following error in Shell Python:

Command failed with exit code 1

but same script running in Spark script ( Glue 5 ) returns at least this:

Error Category: SYSTEM_EXIT_ERROR; Failed Line Number: 3; SystemExit: My own reason to exit this script which I want to see in UI

So as you can see at least we can say what went' wrong on high level and react on message without going to AWS logs which takes extra time.

How difficult is to adjust Python Shell to support this?

Why Python shell is preferable - I noticed that it takes around 45 seconds to start the script for Spark mode but it takes only up to 15 seconds in a Python Shell mode.

[ThePlenkov]

at same time, if we don't want to exit with a custom code, but just running a code like this:

import sys
from awsglue.utils import getResolvedOptions
arg = getResolvedOptions(sys.argv, ['required_arg'])
print(arg)

In Spark we have

Error Category: INVALID_ARGUMENT_ERROR; Failed Line Number: 3; GlueArgumentError: the following arguments are required: --required_arg. 

and as it was initially mentioned in shell python:

Command failed with exit code 2

So same here - it doesn't matter we handle exception or not - the only thing what changes is the exit code in shell script mode