[PATCH 2 of 3 V3] color: enable ANSI support on Windows 10

[Matt Harbison]

# HG changeset patch
# User Matt Harbison <matt_harbison at yahoo.com>
# Date 1495506038 14400
#      Mon May 22 22:20:38 2017 -0400
# Node ID d0dcc7b57bab42b1e79931ef5cab4ba66a0cc58e
# Parent  79425b258088af20ae520e24bd60ab471dd62ec4
color: enable ANSI support on Windows 10

This will display color if "color.mode=ansi", and default to 'ansi' if the mode
is set to 'auto'.  The 'debugcolor' command also reflects this policy.
Previously, "color.mode=ansi" on Windows printed jibberish around the normal
text.  Using ANSI color is better, as it avoids the normal loss of color when
the default pager is enabled on Windows.  See also issue5570.

When the underlying function fails (e.g. when run on older Windows), 'auto'
still falls back to 'win32'.  Apparently, Microsoft originally had this feature
turned on by default, and then made it opt-in[1].  Therefore, not enabling it
unconditionally seems safer.  Instead, only do it after processing the existing
check for support in a Unix-like environment.

[1] https://github.com/symfony/symfony/issues/17499#issuecomment-243481052

diff --git a/mercurial/color.py b/mercurial/color.py
--- a/mercurial/color.py
+++ b/mercurial/color.py
@@ -215,24 +215,34 @@
         mode = ui.config('color', 'pagermode', mode)
 
     realmode = mode
-    if mode == 'auto':
-        if pycompat.osname == 'nt':
-            term = encoding.environ.get('TERM')
-            # TERM won't be defined in a vanilla cmd.exe environment.
+    if pycompat.osname == 'nt':
+        from . import win32
+
+        term = encoding.environ.get('TERM')
+        # TERM won't be defined in a vanilla cmd.exe environment.
 
-            # UNIX-like environments on Windows such as Cygwin and MSYS will
-            # set TERM. They appear to make a best effort attempt at setting it
-            # to something appropriate. However, not all environments with TERM
-            # defined support ANSI. Since "ansi" could result in terminal
-            # gibberish, we error on the side of selecting "win32". However, if
-            # w32effects is not defined, we almost certainly don't support
-            # "win32", so don't even try.
-            if (term and 'xterm' in term) or not w32effects:
+        # UNIX-like environments on Windows such as Cygwin and MSYS will
+        # set TERM. They appear to make a best effort attempt at setting it
+        # to something appropriate. However, not all environments with TERM
+        # defined support ANSI.
+        ansienviron = term and 'xterm' in term
+
+        if mode == 'auto':
+            # Since "ansi" could result in terminal gibberish, we error on the
+            # side of selecting "win32". However, if w32effects is not defined,
+            # we almost certainly don't support "win32", so don't even try.
+            # w32ffects is not populated when stdout is redirected, so checking
+            # it first avoids win32 calls in a state known to error out.
+            if ansienviron or not w32effects or win32.enablevtmode():
                 realmode = 'ansi'
             else:
                 realmode = 'win32'
-        else:
-            realmode = 'ansi'
+        # An empty w32effects is a clue that stdout is redirected, and thus
+        # cannot enable VT mode.
+        elif mode == 'ansi' and w32effects and not ansienviron:
+            win32.enablevtmode()
+    elif mode =='auto':
+        realmode = 'ansi'
 
     def modewarn():
         # only warn if color.mode was explicitly set and we're in