[PATCH 01 of 13 sparse] sparse: vendor Facebook-developed extension

[Martin von Zweigbergk]

On Wed, Jul 5, 2017 at 4:47 PM, Durham Goode <durham at fb.com> wrote:
>
>
> On 7/5/17 3:37 PM, Augie Fackler wrote:
>>
>>
>>> On Jul 5, 2017, at 18:34, Gregory Szorc <gregory.szorc at gmail.com> wrote:
>>>
>>>> I'm happy to see sparse in tree and in core. There are some things we
>>>> need to discuss, but I think we can discuss those as things are being
>>>> moved into core. For example, should "hg clone --include X" be a
>>>> narrow clone of files (and directories, if using treemanifests)
>>>> matching X or a full clone with a sparse checkout of files matching X?
>>>>
>>> I'm going to defer integrating sparse with commands as long as possible.
>>> Before we even talk about UX, I'd like to get the "profiles" formalized so
>>> they are compatible with narrow. Right now, the profiles are strictly files
>>> in the checkout. But presumably we'll also want a way to specify the depth
>>> and "width" of files and changelog history to facilitate narrow and shallow
>>> clone. I can make a compelling argument that all these properties should be
>>> defined in the same "profile." That means designing the profiles so they can
>>> be extended to narrow and shallow later. I don't think this should be too
>>> much work, especially if we're willing to break BC in certain ways when we
>>> introduce narrow/shallow (like having a clone using a sparse profile imply a
>>> narrow clone).
>>
>>
>> Yeah, if we can figure out how to define profiles cleanly so that they're
>> narrow compatible, that'll be awesome.
>>
>> Things narrow wants to set, for consideration:
>>
>>  * should narrow be active, or the store be full?
>>  * depth of clone (if the first is yes)
>>  * should changes outside the tracked spec be elided? (if the first is
>> yes)
>>
>> That's all I can think of offhand.
>
>
> (This is probably more design than we should do in this initial thread, but
> I didn't think about that until after I wrote it)
>
> To be clear on terminology (for anyone reading along), a sparse config
> refers to a set of includes/excludes (the user's sparse config is stored in
> .hg/sparse).  A sparse profile is just a sparse config that is checked into
> the repository and tracked over time.
>
> Sparse
> ===
> For sparse, profiles have two purposes:
>
> - Deploying sparse configs for others to use
> - Tracking changes to a particular config over time
>
>
> Narrow (without shallow)
> ===
> Seems like it could have a few variants:

You missed the obvious one:

0. Profiles contain no info about narrow. Narrow is controlled
independently. Sparse wider than narrow will have no effect (other
than perhaps print a warning).

Oh, I guess the user should not be using a profile but instead a
custom config (i.e. one not under version control) if they don't want
"hg checkout" to risk talking to the server?

>
> 1. Profiles contain no info about narrow. Narrow is widened whenever the
> sparse profile brings in a new file, and narrow is narrowed only by explicit
> user commands.
> 2. Profiles do contain information about narrow. So on clone, the profile is
> used to determine what files to checkout *and* what revlogs to download
> (which must be a superset of what files to checkout). As the narrow portion
> of the profile changes over time, revlogs are widened automatically.
>
> Shallow
> ===
> Shallow doesn't seem to have any tracking-changes-over-time requirement, but
> could benefit from the deploying-configs part at clone time (so at initial
> clone it could read what the intended initial depth is). But that could be
> done by normal configs at clone time, and so maybe shouldn't be checked in
> as part of a profile anywhere.
>
> Proposal
> ===
> We could initially define a sparse profile as a series of
> [include]/[exclude]/%include patterns that affect the working copy if sparse
> is enabled. If narrow is also enabled, it will be automatically widened to
> accommodate the sparse spec. If narrow is initially disabled, and later
> enabled, existing repos are unaffected since narrow only widens the revlogs,
> and they are already completely widened. This seems sufficient for the 99%
> case.  The decision to use narrow or not is not encoded in the profile, and
> is instead a user or system level config decision or clone time decision.
>
> We can later support [store.include]/[store.exclude] as variants that
> additionally widen the store (in addition to anything in the normal working
> copy includes that were already in the store).
>
> And we don't include any info about depth or eliding in the profiles,
> leaving that to user or system configs or clone time decision.