[PATCH STABLE] tests: run test-check-code* last

[Mads Kiilerich]

On 09/10/2012 05:14 AM, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison <matt_harbison at yahoo.com>
> # Date 1347244676 14400
> # Branch stable
> # Node ID 299d1005b72ce3f803f6b1dff0a7788fc73710f0
> # Parent  3ee5d3c372fabcf57c305835dac98da78bdc1837
> tests: run test-check-code* last
>
> Part of check-code.py scans the output of the tests and helpfully indicates that
> (glob) needs to be used on paths for Windows compatibility.  But since the tests
> are run in sorted order, the majority of the tests get run after check-code and
> will need a second run of the tests to pick up the problem.

It is not clear to me what problem you are solving and how you are 
solving it by this change. Is the hidden use case that you are running 
the test suite with -i and only want to run it once?

I would argue that it is a good idea to run the test suite again after 
making any changes anyway.

It could also be argued that check-code by checking everything and 
"failing" quite often would be a good candidate for running very early.

> I don't see a mechanism in place to ensure these are run only after all others
> complete in parallel mode, but this is still an improvement in that the vast
> majority of tests will be covered.

It could be nice to have a mechanism for smart ordering of the tests.
* tests that have failed recently should be run first.
* "important" tests should be run first. I guess that some 5% of the 
tests covers 95% of the code and would catch 95% of the failures.
* some dependencies. If the basic tests for some area fails then it 
probably would make less sense to test more complex scenarios in the 
same area.
* tests should be scheduled evenly across multiple runners to ensure 
that all runners are kept busy until they all finish, for example by 
using execution times from last run and run slow tests first and fill up 
with faster tests.

That mechanism could also be used to run check-code early or late if we 
could agree on which is best. I don't think it is a good idea to 
hardcode a policy now.

> diff --git a/tests/run-tests.py b/tests/run-tests.py
> --- a/tests/run-tests.py
> +++ b/tests/run-tests.py
> @@ -1210,7 +1210,14 @@
>           args = os.listdir(".")
>       args.sort()
>   
> -    tests = args
> +    notcheck, check = [], []
> +    for f in args:
> +        if f.startswith('test-check-code'):
> +            check.append(f)
> +        else:
> +            notcheck.append(f)
> +
> +    tests = notcheck + check

It is not obvious that 'notcheck' is a good name for what it does.

/Mads