Created attachment 1997 [details]
A sample repo

I can see how this is a problem.  The short term work around would be to add the 'lfs' line to the requires file, so it doesn't run that hook.

How do I use the sample repo?  Everything shows as clean, and when I tried to edit and commit either of the two files, it worked.  (I've never used narrow before.)

It's meant to be cloned via http or ssh, otherwise it doesn't do anything.
In the attached repo the ACLs are set so that it will only clone the public folder.

Oops, I didn't scroll up high enough- this hook was originally only for commits.  I forgot about transactions.

So I guess I need to know from a narrow expert how not to trip over this.  It looks like maybe the more efficient way that I was looking for is to call repo.changelog.readfiles(), but I have no idea if that will make narrow happy.

Perhaps it's easiest to move the narrowmatch() method (and its deps) from narrowrepo into localrepo. Then you should be able to filter by that in the lfs extension.

Fixed by https://mercurial-scm.org/repo/hg/rev/4d63f3bc1e1a
Matt Harbison <matt_harbison@yahoo.com>
lfs: respect narrowmatcher when testing to add 'lfs' requirement (issue5794)

There's a similar test in lfs.wrapper.convertsink(), but I didn't update that
because I don't think that the sink repo in a convert can be narrow.

It seems reasonable that a narrow clone of an LFS repo may not necessarily be an
LFS repo.  The only potential issue is that LFS has a hard requirement for
changegroup v3, which that extension enables.  The use of treemanifest will
enable changegroup v3 in narrow clones, because allsupportedversions() in
changegroup.py preserves it when it sees a 'treemanifest' requirement.  But I
don't see where changegroup v3 is enabled for a flat manifest.

(please test the fix)

Bug was set to TESTING for 7 days, resolving

I'm actually still having an issue.
Using hg 4.6, cloning via ssh.

This is .hg/hgrc in the source repo:
---------------------
[extensions]
narrow=
lfs=

[hooks]
pretxncommit.acl =

[lfs]
threshold=1K

[experimental]
changegroup3=true

[narrowhgacl]
default.includes=p1
---------------------

there are two large files named update_build.sh, one under p1, and another under p2.
Getting this:

requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 1 changes to 1 files
error: pretxnchangegroup.lfs hook raised an exception: data/update_build.sh.i@af6024a705c8: no match found
transaction abort!
rollback completed
abort: data/update_build.sh.i@af6024a705c8: no match found!

Bug was set to TESTING for 7 days, resolving

Bug was set to UNCONFIRMED for 30 days, bumping

Bug was set to UNCONFIRMED for 31 days, bumping

Bug was set to UNCONFIRMED for 30 days, bumping

Is this still broken for you? I know a lot has been going on here.

(Please update the bug to "CONFIRMED" if you're still seeing this.)

Once it's not broken, someone should add test cases for it. I may very well break it more (or less, I really don't know) with some of unsent patches I have for narrow.

Yes, it's still happening with 4.7.2

I need to see if I can reproduce it with the sample repo, but I can definitely cause problems in non-clone operations by enabling lfs in some of test-narrow* on default, so marking this confirmed again.

I fixed a crash when trying to extract pointers from a ctx (e.g. to upload blobs, update, etc).  I'm puzzled by the remaining one in clone.

Given this test change:

diff --git a/tests/test-narrow-acl.t b/tests/test-narrow-acl.t
--- a/tests/test-narrow-acl.t
+++ b/tests/test-narrow-acl.t
@@ -1,3 +1,12 @@
+#testcases lfs-on lfs-off
+
+#if lfs-on
+  $ cat >> $HGRCPATH <<EOF
+  > [extensions]
+  > lfs =
+  > EOF
+#endif
+
 Make a narrow clone then archive it
   $ . "$TESTDIR/narrow-library.sh"

@@ -17,7 +26,7 @@ Make a narrow clone then archive it
   $ cat hg.pid >> "$DAEMON_PIDS"

   $ cd ..
-  $ hg clone http://localhost:$HGPORT1 narrowclone1
+  $ hg clone http://localhost:$HGPORT1 narrowclone1 --traceback
   requesting all changes
   adding changesets
   adding manifests


... this stacktrace happens:

--- c:/Users/Matt/projects/hg/tests/test-narrow-acl.t
+++ c:/Users/Matt/projects/hg/tests/test-narrow-acl.t#lfs-on.err
@@ -32,20 +32,92 @@
   adding manifests
   adding file changes
   added 3 changesets with 2 changes to 2 files
-  new changesets * (glob)
-  updating to branch default
-  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  error: pretxnchangegroup.lfs hook raised an exception: data/f3.i@2661d26c6496: no match found
+  transaction abort!
+  rollback completed
+  Traceback (most recent call last):
+    File "c:\Users\Matt\projects\hg\mercurial\scmutil.py", line 166, in callcatch
+      return func()
+    File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 350, in _runcatchfunc
+      return _dispatch(req)
+    File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 987, in _dispatch
+      cmdpats, cmdoptions)
+    File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 733, in runcommand
+      ret = _runcommand(ui, options, cmd, d)
+    File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 996, in _runcommand
+      return cmdfunc()
+    File "c:\Users\Matt\projects\hg\mercurial\dispatch.py", line 984, in <lambda>
+      d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
+    File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1644, in check
+      return func(*args, **kwargs)
+    File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1644, in check
+      return func(*args, **kwargs)
+    File "c:\Users\Matt\projects\hg\hgext\narrow\narrowcommands.py", line 101, in clonenarrowcmd
+      return orig(ui, repo, *args, **pycompat.strkwargs(opts))
+    File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1644, in check
+      return func(*args, **kwargs)
+    File "c:\Users\Matt\projects\hg\mercurial\commands.py", line 1556, in clone
+      storeexcludepats=excludepats)
+    File "c:\Users\Matt\projects\hg\mercurial\hg.py", line 753, in clone
+      depth=depth)
+    File "c:\Users\Matt\projects\hg\mercurial\exchange.py", line 1528, in pull
+      _fullpullbundle2(repo, pullop)
+    File "c:\Users\Matt\projects\hg\mercurial\exchange.py", line 1443, in _fullpullbundle2
+      _pullbundle2(pullop)
+    File "c:\Users\Matt\projects\hg\mercurial\exchange.py", line 1711, in _pullbundle2
+      bundle2.processbundle(pullop.repo, bundle, op=op)
+    File "c:\Users\Matt\projects\hg\mercurial\bundle2.py", line 460, in processbundle
+      processparts(repo, op, unbundler)
+    File "c:\Users\Matt\projects\hg\mercurial\bundle2.py", line 467, in processparts
+      _processpart(op, part)
+    File "c:\Users\Matt\projects\hg\mercurial\bundle2.py", line 534, in _processpart
+      handler(op, part)
+    File "c:\Users\Matt\projects\hg\hgext\narrow\narrowbundle2.py", line 284, in wrappedcghandler
+      origcghandler(op, inpart)
+    File "c:\Users\Matt\projects\hg\mercurial\bundle2.py", line 1819, in handlechangegroup
+      expectedtotal=nbchangesets, **extrakwargs)
+    File "c:\Users\Matt\projects\hg\mercurial\bundle2.py", line 470, in _processchangegroup
+      ret = cg.apply(op.repo, tr, source, url, **kwargs)
+    File "c:\Users\Matt\projects\hg\mercurial\changegroup.py", line 366, in apply
+      throw=True, **pycompat.strkwargs(hookargs))
+    File "c:\Users\Matt\projects\hg\mercurial\localrepo.py", line 1370, in hook
+      return hook.hook(self.ui, self, name, throw, **args)
+    File "c:\Users\Matt\projects\hg\mercurial\hook.py", line 218, in hook
+      res = runhooks(ui, repo, htype, hooks, throw=throw, **args)
+    File "c:\Users\Matt\projects\hg\mercurial\hook.py", line 254, in runhooks
+      throw)
+    File "c:\Users\Matt\projects\hg\mercurial\hook.py", line 98, in pythonhook
+      r = obj(ui=ui, repo=repo, hooktype=htype, **pycompat.strkwargs(args))
+    File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 252, in checkrequireslfs
+      if any(match(f) and ctx[f].islfs() for f in ctx.files()
+    File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in <genexpr>
+      if f in ctx):
+    File "c:\Users\Matt\projects\hg\hgext\lfs\wrapper.py", line 192, in filectxislfs
+      return _islfs(self.filelog(), self.filenode())
+    File "c:\Users\Matt\projects\hg\mercurial\context.py", line 631, in filenode
+      return self._filenode
+    File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1528, in __get__
+      result = self.func(obj)
+    File "c:\Users\Matt\projects\hg\mercurial\context.py", line 579, in _filenode
+      return self._filelog.lookup(self._fileid)
+    File "c:\Users\Matt\projects\hg\mercurial\filelog.py", line 53, in lookup
+      self._revlog.indexfile)
+    File "c:\Users\Matt\projects\hg\mercurial\utils\storageutil.py", line 217, in fileidlookup
+      raise error.LookupError(fileid, identifier, _('no match found'))
+  LookupError: data/f3.i@2661d26c6496: no match found
+  abort: data/f3.i@2661d26c6496: no match found!
+  [255]

Any thoughts on why the narrowmatcher appears to be wider than it should be Martin?

The ACL stuff was written by someone else and we don't use it, so I don't know very well how it works. The same is true for lfs. Perhaps the narrowing happens after that pretxnchangegroup.lfs?

(In reply to Martin von Zweigbergk from comment #18)

That seems like a reasonable explanation.  But I don’t know anything about narrow to know when it should be narrowing.  (It seems like filelogs are already missing, so it should be at least that narrow by that point?)  I submitted a patch, and it mentions that test-narrow-exchange.t also breaks if lfs is enabled.  Maybe you’re more familiar with those tests?

Fixed by https://mercurial-scm.org/repo/hg/rev/4a81d82474e9
Matt Harbison <matt_harbison@yahoo.com>
lfs: consult the narrow matcher when extracting pointers from ctx (issue5794)

I added a testcase for lfs to all narrow tests, and the following failed:

    test-narrow-acl.t
    test-narrow-exchange.t
    test-narrow-patterns.t
    test-narrow-strip.t
    test-narrow-trackedcmd.t
    test-narrow-widen.t
    test-narrow.t

The first two still have errors in the pretxnchangegroup on clone and (receiving
a) push, which I'm still looking into (4d63f3bc1e1a fixed something in this area
already).  These two modified tests seem to cover the things that failed in the
remaining narrow tests, i.e. `hg tracked` and `hg strip`, so I didn't bother
enabling the testcases elsewhere.  Maybe we should, but it's 68 tests total.

(please test the fix)

Still need to handle some clone/push issues referenced, so back to IN_PROGRESS.  To hack around the problem, add ‘lfs’ to the requires file. The problem code only runs when the extension is loaded, but the requirement isn’t present.

Bug was set to IN_PROGRESS for 14 days, moving back to confirmed

Fixed by https://mercurial-scm.org/repo/hg/rev/7c5a922be068
Matt Harbison <matt_harbison@yahoo.com>
tests: document a known failing interaction between narrow and lfs

This is one of the two remaining aborts I found looking into issue5794.  I've
got no idea what's wrong with the hook, since the changes there fixed the other
two problems noted in that bug report.  It seems like it might go away when the
narrow issue is fixed, but let's make sure this doesn't get lost.

The stacktrace for the hook seems to indicate that the missing file *is* in ctx:

  remote: Traceback (most recent call last):
  remote:   File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in checkrequireslfs
  remote:     if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()):
  remote:   File "c:\Users\Matt\projects\hg\hgext\lfs\__init__.py", line 253, in <genexpr>
  remote:     if any(f in ctx and match(f) and ctx[f].islfs() for f in ctx.files()):
  remote:   File "c:\Users\Matt\projects\hg\hgext\lfs\wrapper.py", line 191, in filectxislfs
  remote:     return _islfs(self.filelog(), self.filenode())
  remote:   File "c:\Users\Matt\projects\hg\mercurial\context.py", line 631, in filenode
  remote:     return self._filenode
  remote:   File "c:\Users\Matt\projects\hg\mercurial\util.py", line 1528, in __get__
  remote:     result = self.func(obj)
  remote:   File "c:\Users\Matt\projects\hg\mercurial\context.py", line 579, in _filenode
  remote:     return self._filelog.lookup(self._fileid)
  remote:   File "c:\Users\Matt\projects\hg\mercurial\filelog.py", line 68, in lookup
  remote:     self._revlog.indexfile)
  remote:   File "c:\Users\Matt\projects\hg\mercurial\utils\storageutil.py", line 218, in fileidlookup
  remote:     raise error.LookupError(fileid, identifier, _('no match found'))
  remote: LookupError: data/inside2/f.i@f59b4e021835: no match found

(please test the fix)

Bug was set to TESTING for 7 days, resolving