[PATCH 1 of 5 RFC] util: add platform-agnostic absjoin function and some long-paths win utils

[Kostia Balytskyi]

> -----Original Message-----
> From: Gregory Szorc [mailto:gregory.szorc at gmail.com]
> Sent: Monday, 14 August, 2017 01:35
> To: Kostia Balytskyi <ikostia at fb.com>
> Cc: mercurial-devel at mercurial-scm.org
> Subject: Re: [PATCH 1 of 5 RFC] util: add platform-agnostic absjoin function
> and some long-paths win utils
> 
> 
> 
> > On Aug 12, 2017, at 01:22, Kostia Balytskyi <ikostia at fb.com> wrote:
> >
> > # HG changeset patch
> > # User Kostia Balytskyi <ikostia at fb.com> # Date 1502481241 25200
> > #      Fri Aug 11 12:54:01 2017 -0700
> > # Node ID 586870aa06799e2ef49a2be4c8feab1464ad54b1
> > # Parent  609606d217659e0a6c1cf6f907b6512be5340e57
> > util: add platform-agnostic absjoin function and some long-paths win
> > utils
> >
> > The goal of this seies is to add support for long paths on Windows.
> > Context here is that Win32 API CreateFileA (which opens a file handle
> > and which is what CreateFile macro is resolved to by default) does not
> > recognize paths longer than MAX_PATH constant (260 chars), unless the
> > path starts with \\?\, which causes API to not do any checks or
> > preprocessing of the file path. Therefore, it is posible to use long
> > paths with this prefix, but it comes at a cost: forward slashes are
> > not automatically replaced by baclslashes, . and .. are not automatically
> resolved to appropriate dirs.
> >
> > Also, it is hard to change paths to use //?/ everywhere, so I propose
> > that the approach is to handle both types of paths simultaneously.
> >
> > Also, even if this series is not accepted as it is, I would like to
> > start a discussion about solving this problem in general.
> >
> > This particular patch adds some first steps in reaching this goal:
> > - it adds helper functions hasntprefix and converttontpath
> > - it adds absjoin function which is supposed to replace _join method
> > of dirstate
> 
> Cool series! This feature is long overdue.
> 
> At the point proper path support on Windows is fully implemented in
> Mercurial, how much of Python 3's pathlib have we reinvented? In other
> words, should we start coding to the pathlib API now so all kinds of path
> operations "just work?"
I am not familiar with python3's pathlib, let me have a read about it. But I think I had a somewhat similar idea: move everything form os.path to util.py (or have a similar wrapper) for path utils and never use os.path in our codebase directly. That way we can avoid errors going forward. Things are a tiny bit complicated by the fact that Mercurial uses CreateFile Win32 API directly in some c flies.

> 
> >
> > diff --git a/mercurial/posix.py b/mercurial/posix.py
> > --- a/mercurial/posix.py
> > +++ b/mercurial/posix.py
> > @@ -666,3 +666,9 @@ def bindunixsocket(sock, path):
> >     if bakwdfd:
> >         os.fchdir(bakwdfd)
> >         os.close(bakwdfd)
> > +
> > +def absjoin(absprefix, relpath):
> > +    """Join a relative path to an absolute path. This function assumes that
> > +    the passed absolute path already contains a terminating directory
> > +    separator."""
> > +    return absprefix + relpath
> > diff --git a/mercurial/util.py b/mercurial/util.py
> > --- a/mercurial/util.py
> > +++ b/mercurial/util.py
> > @@ -149,6 +149,7 @@ testpid = platform.testpid umask = platform.umask
> > unlink = platform.unlink username = platform.username
> > +absjoin = platform.absjoin
> >
> > try:
> >     recvfds = osutil.recvfds
> > diff --git a/mercurial/windows.py b/mercurial/windows.py
> > --- a/mercurial/windows.py
> > +++ b/mercurial/windows.py
> > @@ -236,6 +236,28 @@ def normpath(path):
> > def normcase(path):
> >     return encoding.upper(path) # NTFS compares via upper()
> >
> > +def converttontpath(path):
> > +    """Prepend \\?\ prefix to a path and perform slash replacements
> > +    The '\\?\' prefix tells Win32 API to not perform userland path checks,
> > +    therefore allowing long paths. This also means that / are not replaced
> > +    with \ by Win32, so we need to do it manually. See MSDN entry on
> > +    CreateFile function for more details"""
> > +    return '\\\\?\\' + os.path.normpath(path)
> > +
> > +def hasntprefix(path):
> > +    """Check if path starts with a \\?\ prefix"""
> > +    return path.startswith('\\\\?\\')
> > +
> > +def absjoin(absprefix, relpath):
> > +    """See docblock for posix.absjoin for explanation of what this does."""
> > +    if hasntprefix(absprefix):
> > +        # we assume that if absprefix starts with \\?\, this means that it
> > +        # was already preprocessed by converttontpath and therefore does
> > +        # not need to passed to `localpath`
> > +        return absprefix + localpath(relpath)
> > +    else:
> > +        return converttontpath(absprefix + relpath)
> > +
> > # see posix.py for definitions
> > normcasespec = encoding.normcasespecs.upper
> > normcasefallback = encoding.upperfallback
> > _______________________________________________
> > Mercurial-devel mailing list
> > Mercurial-devel at mercurial-scm.org
> > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-
> 2Dscm.org_mailman_listinfo_mercurial-
> 2Ddevel&d=DwIFAg&c=5VD0RTtNlTh3ycd41b3MUw&r=Pp-
> gQYFgs4tKlSFPF5kfCw&m=cxy7gI2F_Xb52Km13M6_ZcLKLEEXWKPuemgMhXL
> zb-A&s=04-3IPa7GiyozU1lv7iydrhxQAuUVOGWJnwkeoli18Q&e=