Long pathnames

[Adrian Buehlmann]

On 21.07.2008 04:19, Arne Babenhauserheide wrote:
> Am Sonntag 20 Juli 2008 19:50:36 schrieben Sie:
>>> 2007-11-14/www.daskochrezept.de/kuechengeraete/index.html?param_search_st
>>> ring=bob+home+staubsauger+\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"crazy+vac\\\\\\\
>>> \\\\\\\\\\\\\\\\\\\\\\\\"!
>> Is this a valid filename on Linux?
> 
> wget created it, so: yes. 
> 
>> What version of Mercurial?
> 
> 1.0.1
> 
>> What platform exactly?
> 
> Gentoo GNU/Linux, CPU: amd64
> 
>> What hg command exactly?
> 
> hg add ... ; hg commit -m "..."
> 
>> What's the output of that command when run with --traceback?
> 
> The following: 
> 
> hg --traceback commit -m "2007-11-14/www.tortenformel.de"
> trouble committing 
> 2007-11-14/www.daskochrezept.de/kuechengeraete/index.html?param_search_string=bob+home+staubsauger+\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"crazy+vac\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"!
> transaction abort!
> rollback completed
> Traceback (most recent call last):
>   File "/usr/lib64/python2.5/site-packages/mercurial/dispatch.py", line 45, in 
> _runcatch
>     return _dispatch(ui, args)
>   File "/usr/lib64/python2.5/site-packages/mercurial/dispatch.py", line 364, 
> in _dispatch
>     ret = _runcommand(ui, options, cmd, d)
>   File "/usr/lib64/python2.5/site-packages/mercurial/dispatch.py", line 417, 
> in _runcommand
>     return checkargs()
>   File "/usr/lib64/python2.5/site-packages/mercurial/dispatch.py", line 373, 
> in checkargs
>     return cmdfunc()
>   File "/usr/lib64/python2.5/site-packages/mercurial/dispatch.py", line 356, 
> in <lambda>
>     d = lambda: func(ui, repo, *args, **cmdoptions)
>   File "/usr/lib64/python2.5/site-packages/mercurial/commands.py", line 557, 
> in commit
>     node = cmdutil.commit(ui, repo, commitfunc, pats, opts)
>   File "/usr/lib64/python2.5/site-packages/mercurial/cmdutil.py", line 1179, 
> in commit
>     return commitfunc(ui, repo, files, message, match, opts)
>   File "/usr/lib64/python2.5/site-packages/mercurial/commands.py", line 555, 
> in commitfunc
>     force_editor=opts.get('force_editor'))
>   File "/usr/lib64/python2.5/site-packages/mercurial/localrepo.py", line 832, 
> in commit
>     new[f] = self.filecommit(f, m1, m2, linkrev, trp, changed)
>   File "/usr/lib64/python2.5/site-packages/mercurial/localrepo.py", line 685, 
> in filecommit
>     fl = self.file(fn)
>   File "/usr/lib64/python2.5/site-packages/mercurial/localrepo.py", line 482, 
> in file
>     return filelog.filelog(self.sopener, f)
>   File "/usr/lib64/python2.5/site-packages/mercurial/filelog.py", line 14, in 
> __init__
>     "/".join(("data", self.encodedir(path + ".i"))))
>   File "/usr/lib64/python2.5/site-packages/mercurial/revlog.py", line 462, in 
> __init__
>     f = self.opener(self.indexfile)
>   File "/usr/lib64/python2.5/site-packages/mercurial/util.py", line 1305, in o
>     return openerfn(fn(path), *args, **kw)
>   File "/usr/lib64/python2.5/site-packages/mercurial/util.py", line 1442, in 
> __call__
>     fp = posixfile(f, mode)
> IOError: [Errno 36] File name too 
> long: '/home/arne/Dokumente/Kochrezepte/.hg/store/data/2007-11-14/www.daskochrezept.de/kuechengeraete/index.html~3fparam__search__string=bob+home+staubsauger+~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~22crazy+vac~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~22.i'
> abort: File name too 
> long: /home/arne/Dokumente/Kochrezepte/.hg/store/data/2007-11-14/www.daskochrezept.de/kuechengeraete/index.html~3fparam__search__string=bob+home+staubsauger+~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~22crazy+vac~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~5c~22.i
> 

Thanks. Now I know that it is an IOError.

>>> Would it be possible to shorten these long paths to
>>> store/data/path/beginning_of_the_filename-HASH_OVER_THE_END.i
>> Did you read what I wrote?
> 
> Does "No Hash" mean "doesn't work in my case"? 
> Then I'd oppose it (naturally :) )

No. No reason to oppose here ;-) (unless you have other issues...)

> Is hasing a part among the fixes you include? 

Yes. It uses hashing.

My patch ensures that the length of all path parts below .hg/store
doesn't get longer than 120 chars.

In your pathological case, Mercurial encodes path characters as ~xx
because they would be illegal on Windows, where xx is the two digit hex
code of that char ('~5c' in your case). In your case, that apparently
made the encoded path hit one of the Kernel length limits (as posted by Matt).

Mercurial applies said encoding by default, to ensure that Mercurial
repositories can be copied verbatim on the file level between Windows (or Mac)
and Linux. For example, said encoding also encodes all uppercase characters
X to _x (underbar plus lowercase), which also increases path length.
This is done because Windows -- and Mac, apparently -- are case folding, which
means you can't have a file 'Foo' and a file 'foo' at the same time in the same
dir.

As a workaround, you could use the other (non-default) non-cross-platform-compatible
repo format of Mercurial, which doesn't encode the paths in the store. It can
be enabled by setting

  [format]
  usestore = False

in your hgrc (read http://www.selenic.com/mercurial/hgrc.5.html#format before
using it), or with

  hg init --config format.usestore=0

or

  hg clone --pull --config format.usestore=0 ...

for individual repos (the "--pull" is needed if the source repo is a local
one and you want to have a different repo layout than what the source repo
uses).

As a side note, I think your example would work with my filenamelog patch
applied to Mercurial.

Since I don't have your platform (only Windows XP and FreeBSD 6.2 + 6.3 here),
you could actually try my patch series on your platform and see
yourself if your example works.

The currently published version of my patch series is:
http://selenic.com/pipermail/mercurial-devel/2008-July/007156.html
http://selenic.com/pipermail/mercurial-devel/2008-July/007157.html
http://selenic.com/pipermail/mercurial-devel/2008-July/007158.html

I currently have a newer version in my local (private) repo here, which
tries to address Matt's latest comments. I hope to be able to publish
that update soon on the Mercuial-devel list (in case you are interested).