Please read: only you can prevent test suite bloat

[Matt Mackall]

On Wed, 2012-10-31 at 10:21 -0700, Bryan O'Sullivan wrote:
> On Tue, Oct 30, 2012 at 4:18 PM, Matt Mackall <mpm at selenic.com> wrote:
>         And yet I'm seeing new patches submitted on a nearly daily
>         basis from subscribers to this list
>         that ignore this, some of them being accepted by crew members
>         who
>         haven't read it either.
>         
> 
> Since this is clearly aimed at me:

Actually, it's intentional not aimed at you. But I guess the subject
line isn't universally recognizable:

https://en.wikipedia.org/wiki/Smokey_Bear

It's aimed for maximal distribution, precisely because responding
individually 100x was not working. There are obviously LOTS of regular
contributors on the list who haven't figured out what the issue here is
yet.

The policy has existed since around the 1.9 sprint, where Nic cut out a
big chunk of test redundancy. Here's what's happened to the test suite
from 2.0 to today:

2.0:
# Ran 426 tests, 6 skipped, 5 failed.

real	4m50.134s
user	13m22.381s
sys	3m32.613s

tip:
# Ran 448 tests, 6 skipped, 1 failed.

real	6m46.688s
user	19m0.012s
sys	5m6.593s

That's 40% bloat in just one year AFTER we declared war on test suite
bloat. And we obviously don't have 40% new features/code/bugs to test.
Nor is this sustainable: our laptops are not getting 40% faster per year
anymore. And if you're running something other than a Core i7 Linux box
that can do -j8 or higher, you're probably seeing more pain here.
Asking OS X and Windows devs to run the full test suite today is asking
a lot.

Another way to quantify it: I've personally probably run the test suite
>10 times per day in the past year. Assuming a linear increase, that
change amounts to about 1.5 weeks of lost productivity for me this year
just in me waiting for tests to finish. This doesn't count lots of extra
round trips on patches because people didn't run the whole test suite
because it was too slow.

> I read that wiki page, and I don't push changes that add new tests.

Great. That was indeed the most common manifestation of the underlying
problem. But as the section explains, the problem is the total amount of
time spent running redundant commands, not the number of *.t files. So
simply taking the contents of what would have been a wholly new .t file
and appending it to an existing test doesn't help. (That entire page
could stand to be rewritten to emphasize this more forcefully, but I've
avoided editing it while quoting it for people... which means editing it
has been starved.)

For reviewers, top signs that a test contains new bloat:

- new *.t file
- appending a large, independent block to a test
- calling hg init
- multiple new calls to commit and merge
- could have been a doctest

In short, if it's a nice pretty test case like we used to make, it's
probably contributing to bloat.

-- 
Mathematics is the supreme nostalgia of our time.