/!\ This page is primarily intended for Mercurial's developers.

Better Matcher API and File Patterns Plan

Status: Project

Main proponents: KatsunoriFujiwara, RodrigoDamazio

/!\ This is a speculative project and does not represent any firm decisions on future behavior.

{X} Add a short summary of the idea here.

1. Goal

2. Detailed description

2.1. Sprint Notes

Non-recursive globs (Rodrigo, spectral, Durham, :
    Issue is that * is sometimes recursive
    matcher API is a mess
    Should we re-write match.py or just add fileglob?
    Suggestion: add fileglob via a new, cleaner API, then migrate others over time
    Possible FB use case: pick parts of a tree to include and exclude (would add ordering dependency instead of excludes always trumping includes?)
    matcher API should be extensible
    matcher composition: anyof, allof, negate, per-file-type, etc.
    Inconsistencies in pattern behavior between hgignore, --include/--exclude, etc.
    FB: conversion between matchers and watchman expressions
    Proposal: wiki page, first group to have a use case proposes the initial API

2.2. Current Status

2.2.1. Summary of mode, relative-to, and recursion of each types

mode

root-ed

cwd-ed

any-of-path

control recursion by pattern

context depend recursion

wildcard

-

glob:

relglob:

by **

o

regexp

re:

-

relre:

by $

x (*A)

raw string

path:

relpath:

-

(always)

x

2.2.2. The list of contexts, in which pattern is specified

pattern for

default type

recursion of wildcard

related API

fileset

glob:

x

ctx.match()

files() template function

glob:

x

ctx.match()

diff() template function

glob:

o (*1)

ctx.match()

file() revset predicate

glob:

x

match.match()

follow() revset predicate

path: (*3)

x

match.match()

--include/--exclude

glob:

o (*1)

match.match()

hgignore

relre:

o (*1)

match.match()

archive web command

path:

- (*2)

scmutil.match()

hg locate

relglob:

x

scmutil.match()

hg log

relpath:

x

scmutil.matchandpats()

others (e.g. hg files)

relpath:

x

scmutil.match()

For "recursion of wildcard":

For example, file foo/bar/baz is:

The last case seems to cause the issue mentioned by Rodrigo in "match: adding non-recursive directory matching". And the second case can be used as instant work around for that issue.

Table below re-summarizes about recursion (= matching against intermediate directory) of each modes.

mode

-I/-X

in "set:"

-I/-X with "set:"

wildcard

always

endswith("**")

endswith("**")

regexp

not endswith("$")

not endswith("$")

not endswith("$")

raw string

always

always

always

"Recursion of wildcard" of the pattern from a file follows one of what tries to read that file in. For example:

2.2.3. Reading patterns from file

read in by

type substitution

default type for hgignore

default type for otherwise

include:FILE

o

relre:

relre:

listfile:FILE

x

(*X)

(*Y)

If "type substitution", substitutions below occur always at reading patterns from file. This is mentioned in "hg help patterns" and "hg help hgignore", but type relglob: and relre: themselves aren't explained.

Reading from .hgignore and "[ui] ignore" is treated as a variant of include: internally (e.g. include:$REPOROOT/.hgignore)

2.2.4. Recursion of ignore patterns

As a ignore pattern, "wildcard" and "raw string" modes are obviously recursive, because:

On the other hand, "regexp" mode itself is non-recursive. For example, with "re:^foo$" in .hgignore, "hg debugignore" shows the regexp, which doesn't match against file foo/bar.

But actually, "re:^foo$" in .hgignore ignores file foo/bar, because dirstate (and "hg debugignore") examines whether specified file does:

and that file is ignored, if one of conditions above is true.

Therefore, "regexp" ignore pattern is recursive, even if it uses "$".

In conclusion, all ignore patterns are treated as recursive, regardless of pattern types.

This special recursion of "regexp" mode is specific for ignore patterns. In other cases, "regexp" mode pattern isn't recursive, if it uses "$".

2.3. Proposal by foozy

By introducing systematic new pattern types, both "start point" and "recursion" of matching can be fully controlled arbitrarily in any contexts (as PATTERN, -I/-X, and so on).

2.3.1. Control start point of matching arbitrarily

New types and their start point.

mode

root-ed

cwd-ed

any-of-path

wildcard

rootglob:

cwdglob:

anyglob:

regexp

rootre:

cwdre:

anyre:

raw string

rootpath:

cwdpath:

anypath:

New "wildcard" and "regexp" types other than anyglob: match recursively, fully according to specified pattern, as below. anyglob: pattern should be always recursive, because this type is "any-of-path" matching.

type

recursive

rootglob:

endswith("**")

cwdglob:

endswith("**")

anyglob:

always

rootre:

not endswith("$")

cwdre:

not endswith("$")

anyre:

not endswith("$")

rootpath:

always

cwdpath:

always

anypath:

always

2.3.2. Emulate legacy types as an alias of new types

Current match.py implementation adds prefix/suffix regexp below to the specified pattern internally, according to what it is used for. See implementation of _regex() and match._normalize(), and _buildmatch() invocations in match.__init__() in match.py, for detail.

type

used for

prefix

suffix

recursive

glob:

pattern

"$CWD/"

"$"

endswith("**")

include/exclude

"$CWD/"

"(?:/|$)"

always

relglob:

pattern

"(?:|.*/)"

"$"

endswith("**")

include/exclude

"(?:|.*/)"

"(?:/|$)"

always

re:

(always)

(none)

(none)

not endswith("$")

relre:

(always)

".*" (*1)

(none)

not endswith("$")

path:

(always)

"^" (*2)

"(?:/|$)"

always

relpath:

(always)

"$CWD/"

"(?:/|$)"

always

So, at first, let assume that newly introduced types use additional prefix/suffix regexp below BY DEFAULT (now, controlling recursion in "wildcard" and "regexp" mode is user responsibility).

type

prefix

suffix

recursive

rootglob:

(none)

"$"

endswith("**")

cwdglob:

"$CWD/"

"$"

endswith("**")

anyglob:

"(?:|.*/)"

"(?:/|$)"

always

rootre:

(none)

(none)

not endswith("$")

cwdre:

"$CWD/"

(none)

not endswith("$")

anyre:

".*"

(none)

not endswith("$")

rootpath:

(none)

"(?:/|$)"

always

cwdpath:

"$CWD/"

"(?:/|$)"

always

anypath:

"(?:|.*/)"

"(?:/|$)"

always

Then, legacy types can be emulated as an alias of newly introduced type as below:

type

used as

alias of

needed suffix

glob:

pattern

cwdglob:

"$" (= default of cwdglob:)

include/exclude

cwdglob:

"(?:/|$)"

relglob:

pattern

anyglob:

"$"

include/exclude

anyglob:

"(?:/|$)" (= default of anyglob:)

re:

(always)

rootre:

(none) (= default of rootre:)

relre:

(always)

anyre:

(none) (= default of anyre:)

path:

(always)

rootpath:

"(?:/|$)" (= default of rootpath:)

relpath:

(always)

cwdpath:

"(?:/|$)" (= default of cwdpath:)

At this point, using suffix below forcibly for legacy glob: and relglob: is as same as current match.py implementation.

Therefore,aliasing should be emulated easily.

2.3.3. Control recursion of matching arbitrarily

With current Mercurial (at least, 4.0 or earlier), recursion of each pattern types can be controlled by:

mode

recursive

wildcard

endswith("**")

regexp

not endswith("$")

raw string

always

User can't control recursion of matching with "raw string" pattern arbitrarily (it matches against both directory and file).

Therefore, how about introducing two more additional modes "raw file" and "raw dir" ? Additional suffix regexp can control these matching.

mode

recursive

suffix

raw file name

never

"$"

raw dir name

always, but matches against only directory

"/"

After adding these modes, there are 5 (modes) x 3 (start points) = 15 types

mode

root-ed

cwd-ed

any-of-path

wildcard

rootglob:

cwdglob:

anyglob:

regexp

rootre:

cwdre:

anyre:

raw string

rootpath:

cwdpath:

anypath:

raw file name

rootfile:

cwdfile:

anyfile:

raw dir name

rootdir:

cwddir:

anydir:

2.4. Proposal by Rodrigo

Add rootglob: to get over the issue of -I/-X patterns.

https://patchwork.mercurial-scm.org/patch/17311/

3. Roadmap

{X}

4. See Also


CategoryDeveloper CategoryNewFeatures

FileNamePatternsPlan (last edited 2016-12-05 13:08:40 by YuyaNishihara)