User metadata support

[Guenther Brunthaler]

Hi all,

I'm rather new to Mercurial, but I have used a lot of different SCMs before.

Mercurial looks in many aspects like the "right thing" to me.

I used to check out Monotone some time ago, but it had just too many
shortcomings to be useful - most of which Mercurial managed to avoid.

Actually, there is only one single big thing left (except for symlink
support) which would make everyone happy: User metadata support.

With user metadata, I mean something like the "properties" of Subversion.

In essence, it's just a version-controlled key/value list associated
with each file.

You might think nobody actually needs such a thing?

Let's illustrate a few cases where such metadata would be highly useful:

* Additional permission bits. Currently, Mercurial supports the
executable bit right out of the box. Fine. But what if more permission
bits should be associated with a file, such as the sticky bit. Or
creating a file as read-only. Or a special POSIX ACL. If hooks for
checkin/checkout had access to file metadata, the hooks could set the 
appropriate bits on checkout as required, and without a need to 
integrate such features into the Mercurial core.

* Actually, even the executable bit needed not to be supported directly
my the core Mercurial any more: Hooks could take over that job too, 
provided they have access to metadata items such as to a property named
"hg:executable".

* Line-ending conversion. While I agree that line-ending conversions
should normally be performed based on heuristics because users tend to 
forget about setting special properties, there are exceptions. What if a 
file with extension .txt is a texture in some project subdirectory 
rather than a text file like in the rest of the project? If the 
autodetection heuristics for binary files fails, we'll be screwed as 
soon as line-ending conversion will be attempted on that file. Using a 
property such as "hg:eol-style" set to "binary" would let a hook script 
override autodetection in such cases.

* Character set conversion. What if a single directory contains text
files in different character set encodings? Just think of text files on
a Windows machine which shall also be edited on a UTF-8 Linux 
workstation: On the Windows side, most files will be using the "ANSI" 
character set (in fact WINDOWS-1252 because of that EURO-Symbol), but 
some files intended to be used by the Console are instead represented 
using the "OEM" character set (IBM CP 437 or CP 850). Using the same 
conversion for all text files cannot work in this case. And they all 
share the same filename extension. It is necessary to override the 
conversions on a per-file basis. Metadata properties would allow the 
hook to also take care of this.

* Stream metadata. Machines like the Apple Macintosh can use different
streams in a file, the so-called "data fork" and "resource fork". Think
of it as a kind of sub-file. Same for NTFS which also supports streams.
Each stream of a file has the potential to contain data a user would 
like to be subject to version control. The "normal file contents" are 
just the contents of the default stream of each file. But the best is 
yet to come:

* Symlinks could be implemented using hooks having access to stream
metadata! In this case, a symlink would be treated as a stream with a
reserved name of a file which has no default stream (and thus no normal 
file data) at all. That means, when checking out, there will no file be 
created. But the checkout hooks (which will be run for all streams, not 
just for file data contents) can check the stream type and create a symlink.

* Any kind of additional information to be attached to files/streams,
such as MIME types etc. The hooks can make use of this information if
required.

* Directory attributes! Directory can also also have metadata streams,
storing version-controlled metadata about the directories, such as
"hg:ignore".

* Tracking even empty directories and renaming or moving of directories. 
If we assign an (empty) dummy stream such as "hg:dir" to each directory, 
we can deduce the existence of a directory from the mere existence of 
that stream. Which means we won't need things like dummy-".keep" any more.

You see, metadata support would in fact be exceptionally useful for
everyone, and would even make implementation of some things easier.

For instance, you could forget about the executable bit or symlinks in
the core Mercurial project, and delegate such problems to the hook scripts.

Of course, metadata support should be implemented in a way that requires
the least changes to the existing implementation.

First, what is actually needed:

* Not files are versioned, but streams are. It's pretty much the same
from the perspective of a revlog, but we have to add an additional
level below the leaf level (as it is now).

For instance, instead of having a

./hg/data/somedir/somefile.d

we then could have a

./hg/data/somedir/somefile/hg_data.d

which means: This represents a stream with name "hg:data" of the version
controlled "object somedir/somefile". We say "object" here rather than
"file", because that object could be a symlink as well - depending on 
which stream properties it has.

In this case, it is a file, because it has a "hg:data" property (which 
contains the actual file contents).

But it could have additional properties as well:

./hg/data/somedir/somefile/hg_executable.d

could indicate the fact that file "somedir/somefile" has an additional 
property "hg:executable", which means its executable bit should be set 
on checkout.

"hg:executable" is also the example for a "switch"-style property: It's 
mere existence indicates something; the actual revlog contents will 
typically an empty file because all that matters is whether this 
property is there or not.

You can also see: Streams and Properties are pretty much the same in 
this model - from the viewpoint of the revlog they are just more files 
to be version-controlled.

It's the only *interpretation* as streams/properties which makes them 
special.

Another example: In order to save a symlink instead of a file using the
same name as above, we could use a property-revlog like this:

./hg/data/somedir/somefile2/hg_symlink.d

which represents a stream with name "hg:symlink", and the contents of
the current revision of that revlog contain the symlink target.

And now how to store a "hg:ignore" property for directory
"somedir/somesubdir":

./hg/data/somedir/somedir/somesubdir/hg_ignore.d

So, the most important thing to be changed for implementing that feature 
is to add an additional subdirectory level at the leaves of the 
version-controlled directory tree that is omitted when checking out a 
revision, but available to the hooks.

In the output of "hg manifest" the streams could be displayed using the 
-v option, while the "hg:data" stream should be suppressed from output
in the normal case (because it is the default).

For instance,

$ hg manifest

hexstuff... somedir/somefile
hexstuff... somedir/somefile2 [hg:symlink]
hexstuff... somedir/somesubdir [hg:ignore]

$ hg manifest -v

hexstuff... somedir/somefile [hg:data]
hexstuff... somedir/somefile2 [hg:symlink]
hexstuff... somedir/somesubdir [hg:ignore]

Of course, streams could also be displayed in any different way as well;
it's just an example.

The manifest internal format needed to be updated as well:

$ hg debugdata .hg/00manifest.d 0

somedir/somefile/hg_data<hexstuff>
somedir/somefile2/hg_symlink<hexstuff>
somedir/somesubdir/hg_ignore<hexstuff>

So, actually it does NOT need to be changed, but rather includes the
*uninterpreted* contents of the .gh/data directory, including the leaf 
revlogs which always represent streams.

To be more precise: *All* the revlogs now represent streams! Because the 
actual files or directories or symlinks will be represented by 
subdirectories now which contain the stream revlogs. And whether such a 
directory will be interpreted as the name of a version-controlled file, 
directory, symlink, fifo, device file or something different depends 
solely on which stream revlogs exist in that directory.

And to make the best of properties, there should be a means of
*inheriting* them from the parent directory, possibly overriding them in
nested subdirectories. But that's worth its own thread I think.

Regarding stream names, it might be wise to enforce a naming policy to 
avoid name clashes with user-defined properties.

A suggestion of such a policy:

* All property/stream names optionally start with a namespace prefix, 
followed by a colon, and then an identifier. For instance, in "hg:data", 
"hg" is the name of the namespace, and "data" is the namespace-relative 
name of the stream.

* Namespace "hg" is reserved for Mercurial's "well-known" or specially 
interpreted streams. For instance, while "hg:executable" could be a 
user-defined property as well which is only of interest for user-defined 
hooks, "hg:data" is clearly of essential interest for the internal 
checkout and checkin functions of Mercurial.

* Namespace "urn" is reserved for property names which conform to the 
URN syntax, e. g. globally unique and *persistent* identifiers. 
(Persistency is also the big difference between an URL and an URN. URLs 
cannot truly be considered to be persistent: Domains come into existence 
and go away all the time.) For instance, there is a "urn:uuid" scheme 
which allows to create URNs based on UUIDs for those who like this. But 
numerous other schemes exist as well.

* Namespace "rdn" is reserved for the "reversed domain name" identifiers 
which are so popular in JAVA (or Monotone). This specifies properties 
such as "rdn:com.sun.java/bigproject/specialstream". However, as stated 
in the previous paragraph, DNS names might be not be the best choice to 
guarantee uniqueness of a name - at least not over time. URNs, in 
contrast, will do (if an appropriate URN scheme is chosen, such as 
"urn:uuid").

* All other names with or without a namespace prefix are free to be used 
by users in any way they like.

However, there is a problem here: The "urn" and "rdn" namespaces allow 
to include slashes, colons and other characters better to be avoded in 
filenames. Especially under Windows.

So I suggest a simple name mapping strategy:

* We add a pseudo-namespace "b32" which encodes whatever follows it in 
BASE-32 encoding.

* That pseudo-namespace will only be recognized at the beginning of a 
stream name and will encode whichever follows it in BASE-32.

* colon characters are mapped into underscores.

Here are some examples of stream names and the mapped revlog file names 
which will represent them:

"hg:data" -> "hg_data2.d"
"hg:symlink" -> "hg_symlink.d"
"plain" -> "plain.d"
"usernamespace:whatever" -> "usernamespace_whatever.d"
"funny_name" -> "b32_<base32stuff>.d"
"urn:uuid:11223344-5566-3353-aabbccddeeff" -> "b32_<base32stuff>.d"

In those examples, <base32stuff> is a placeholder for the BASE-32 
encoding of the string on the left side.

Why BASE-32 instead of BASE-64 one might ask?

Because BASE-32 does not use both upper- and lower case characters in 
the encoding it generates, which eliminates problems which filesystems 
that do not preserve letter case in file or directory names. (See the 
RFC about BASE-32 encoding for more details.)

Anyway, all the above is a mere suggestion to show how streams could be 
implemented; I'll be happy to keep it open to discussion.

But I would really be happy to see properties supported by Mercurial 
some day, which will also be the day I convert my SVK repositories into 
Mercurial!

Currently I cannot use Mercurial because I have lots of symlinks under 
version control in SVK.

I am using SVK because it is still the best distributed SCM I have 
encountered so far: It has (nearly) all the features of Subversion, but 
adds fully disconnected (off-line) operation.

SVK has also some disadvantages. The biggest disadvantage of SVK is its 
lack of concise documentation and it's largely intransparent operation. 
It's very obscure. *And* written in Perl. ;-)

Mercurial clearly excels here: All basic data structures (i. e. the 
revlog) are well defined and the interconnection between the components 
of the data structures (revlog, nodeid, manifest, etc) are nicely 
explained. This is how it should be.

In SVK I do not even fully understand the options and operation modes of 
its 3 merge commands... especially in disconnected or mirrored operation.

However, it works.

Somehow.

But I would really prefer Mercurial - if it only could support support 
properties like symlinks and character conversion attributes.


Greetings from Vienna,
Guenther