travis.sh and run_tests.py reported success when it a was failure

[evandrocoan]

Job: https://travis-ci.org/ehuss/Sublime-Wrap-Plus/jobs/523336576

AssertionError: 106 not greater than 106
----------------------------------------------------------------------
Ran 53 tests in 4.851s
FAILED (failures=1)
Name             Stmts   Miss  Cover
------------------------------------
py_textwrap.py     161    127    21%
wrap_plus.py       828    363    56%
------------------------------------
TOTAL              989    490    50%
UnitTesting: Done.
The command "sh travis.sh run_tests --coverage" exited with 0.

travis.sh

RunTests() {
    if [ -z "$1" ]; then
        python "$STP/UnitTesting/sbin/run_tests.py" "$PACKAGE"
    else
        python "$STP/UnitTesting/sbin/run_tests.py" "$@" "$PACKAGE"
    fi

    pkill "[Ss]ubl" || true
    pkill 'plugin_host' || true
    sleep 2
}

run_tests.py

def read_output(path):
    # todo: use notification instead of polling
    success = None
    start_time = time.time()

    def check_is_success(result):
        try:
            return RX_RESULT.search(result).group('result') == 'OK'
        except AttributeError:
            return success

    def check_is_done(result):
        return RX_DONE.search(result) is not None

    with open(path, 'r') as f:
        while True:
            offset = f.tell()
            result = f.read()

            print(result, end="")

            # Keep checking while we don't have a definite result.
            success = check_is_success(result)

            if check_is_done(result):
                assert success is not None, 'final test result must not be None'
                break
            elif not result:
                f.seek(offset)

            time.sleep(0.2)

            if time.time() - start_time > 1000:
                break

    return success

def main(default_schedule_info):
    package_under_test = default_schedule_info['package']
    output_dir = os.path.join(UT_OUTPUT_DIR_PATH, package_under_test)
    output_file = os.path.join(output_dir, "result")
    coverage_file = os.path.join(output_dir, "coverage")

    default_schedule_info['output'] = output_file

    create_dir_if_not_exists(output_dir)
    delete_file_if_exists(output_file)
    delete_file_if_exists(coverage_file)

    create_schedule(package_under_test, output_file, default_schedule_info)
    delete_file_if_exists(SCHEDULE_RUNNER_TARGET)
    copy_file_if_not_exists(SCHEDULE_RUNNER_SOURCE, SCHEDULE_RUNNER_TARGET)

    start_sublime_text()

    try:
        print("Wait for tests output...", end="")
        wait_for_output(output_file, SCHEDULE_RUNNER_TARGET)

        print("Start to read output...")
        if not read_output(output_file):
            sys.exit(1)
    except ValueError:
        print("Timeout: Could not obtain tests output.")
        print("Maybe Sublime Text is not responding or the tests output"
              "is being written to the wrong file.")
        sys.exit(1)
    finally:
        restore_coverage_file(coverage_file, package_under_test)
        delete_file_if_exists(SCHEDULE_RUNNER_TARGET)

Related issues:

1. improve script structure for run_tests.py #70 improve script structure for run_tests.py
2. generalizing travis.sh for linux/macos-based ci hosts #71 generalizing travis.sh for linux/macos-based ci hosts

[randy3k]

To be honest, I can't immediately spot the cause of the error.

[randy3k]

I tested the exit code on my repo AlignTab and it seems working.

https://travis-ci.org/randy3k/AlignTab/jobs/523692004

[evandrocoan]

I compared both outputs, and they are almost equal:

It is strange that it could correctly catch the UnitTesting: Done, but missed the FAILED string.

I will try to debug it.

[evandrocoan]

After 5 ours working on it, I got this:

read_output, success False # run_tests.py
Exiting with error! False END! # on run_tests.py
travis.sh SYSTEM EXIT CODE: 4005! # on travis.sh

1. run_tests.py is working 100% OK.
2. set -e is not working.

Python is exiting with code 4005, but the script is not failing. After searching, I see set -e does not work on nested shells: https://unix.stackexchange.com/questions/65532/why-does-set-e-not-work-inside-subshells-with-parenthesis-followed-by-an-or

I am not sure how I am inside a nested shell when running the function RunTests().

But just adding || exit $? to run_tests.py fixed the issue.
Example: https://travis-ci.org/evandrocoan/WrapPlus/jobs/524306626

RunTests() {
    if [ -z "$1" ]; then
        python "$STP/UnitTesting/sbin/run_tests.py" "$PACKAGE" || exit $?
    else
        python "$STP/UnitTesting/sbin/run_tests.py" "$@" "$PACKAGE" || exit $?
    fi
    printf 'travis.sh SYSTEM EXIT CODE: %s!' "${$?}"

    pkill "[Ss]ubl" || true
    pkill 'plugin_host' || true
    sleep 2
}

Before I was doing this:

    "run_tests")
        RunTests "$@" || ShowFullSublimeTextConsole

The function ShowFullSublimeTextConsole would print all Sublime Text console saved to a file by some other script I added on bootstrap.

But, after adding || exit $? it did not worked too. Even after removing || ShowFullSublimeTextConsole did not fixed the initial problem.

My final solution was:

1. To remove the || ShowFullSublimeTextConsole because it was not working anyways.
2. To add || exit $? after run_tests.py

[randy3k]

Ok. I believe that the bug was introduced by your fork, to be specific

https://github.com/evandrocoan/UnitTesting/blob/1421b21730798c9834846bdeddc24689db061b3d/sbin/travis.sh#L261

If RunTest fails, it runs ShowFullSublimeTextConsole instead of raising an exiting code.

[evandrocoan]

If RunTest fails, it runs ShowFullSublimeTextConsole instead of raising an exiting code.

Yes, the function should run, but it did not ran because none of its prints are showed. Also, notice the function ends with exit 1, so it will just forward the error.

But as I say earlier, even after I removed the || ShowFullSublimeTextConsole, it still not working.

[randy3k]

I still fail to see why it doesn't forward the return code. At the very least, we don't see any issues using our master https://travis-ci.org/randy3k/AlignTab/jobs/523692004

[randy3k]

To demonstrate that it is the issue of || ShowFullSublimeTextConsole, see the following.

(randyimac)-~$ cat foo
#!/bin/bash

set -e

Foo() {
    echo "foo"
    python -c 'import sys; sys.exit(1)'
    echo "bar"
}

Foo || true  # bar is printed
(randyimac)-~$ sh foo
foo
bar
(randyimac)-~$ cat foo
#!/bin/bash

set -e

Foo() {
    echo "foo"
    python -c 'import sys; sys.exit(1)'
    echo "bar"
}

Foo  # bar is not printed
(randyimac)-~$ sh foo
foo

You can see that if || is used, the function will not be terminated earlier.

[randy3k]

Actually, it is documented that

Exit immediately if a pipeline (see Pipelines), which may consist of a single simple command (see Simple Commands), a list (see Lists), or a compound command (see Compound Commands) returns a non-zero status. The shell does not exit if the command that fails is part of the command list immediately following a while or until keyword, part of the test in an if statement, part of any command executed in a && or || list except the command following the final && or ||, any command in a pipeline but the last, or if the command’s return status is being inverted with !. If a compound command other than a subshell returns a non-zero status because a command failed while -e was being ignored, the shell does not exit. A trap on ERR, if set, is executed before the shell exits.

[evandrocoan]

What!!! Shell is insane! Thanks for the full example.

1. Either I must be doing some pipe || somewhere else on the script, which is breaking everything.
2. Or when I removed the pipe on || ShowFullSublimeTextConsole "$?", the Travis CI downloaded some cached version of the travis.sh.