/!\ This page is primarily intended for Mercurial's developers.

Tracking File Encoding in Mercurial

Clients displaying text files do not currently have enough information available to always reliably decode file contents to unicode, leading to incorrectly decoded and rendered text under some circumstances.

This proposal describes a mechanism for users to explicitly declare file encoding on an individual file level as an aid to the client. Encoding meta data is versioned and stored alongside the repository's contents.

Problem Statement

For files that are not using a self-identifying character encoding like a UTF BOM and not plain ASCII/latin-1, the client is left guessing how to interpret the contents. Different clients apply different approaches with varying levels of success.

As an example, Bitbucket assumes all files in all repositories to use UTF-8. If decoding fails, it then runs the first kilobyte through an encoding detection library that applies heuristics to guess the encoding. If decoding fails again, it switched back to UTF-8, this time forcing decoding, replacing invalid codepoints, visibly mutilating the contents.

In cases where the detection succeeds, there is still no guarantee that the detected encoding was the one used by the user, as encoding schemes overlap and a particular byte sequence can be valid for multiple schemes, mapping to distinct unicode code points.

Current Situation

Mercurial currently offers a global way to provide an encoding hint through $HGENCODING. This has limitations however:

  1. It is not versioned or stored alongside the repo's contents, meaning that forks/upstreams do not inherit this meta data, so that each service must be configured individually and knowledge must be exchanged through separate means.
  2. A single encoding is inappropriate for repositories that use multiple file encodings. An example could be a project that switched encodings at some point in its history, or large "mono-repos" containing diverse, independent sub projects.

Suggested Solution

Following .hgignore and .hgeol's lead, a new file .hgencoding is created containing rules that map file patterns to character encodings. This file is checked into the repository and lives at the root level.

A file's character encoding is then determined at runtime by matching its path against the patterns. If multiple patterns match the same path, the first rule that matches will take precedence.

It is possible to have subdirectories contain .hgencoding files whose patterns only apply to their sub-tree of the repository. Each nested .hgencoding file must be explicitly included, either directly or indirectly through a chain of intermediary files, from the one in the project's root.

File Format and Syntax

Similar to .hgignore, .hgencoding contains one file matching pattern per line, but unlike .hgignore, each pattern is followed by an encoding, e.g.:

    syntax: glob
    **.cs = windows-1252

Empty lines are skipped. The # character is treated as a comment character, and the \ character is treated as an escape character.

Mercurial supports several pattern syntaxes. The default syntax used is Python/Perl-style regular expressions.

To change the syntax used, use a line of the following form:

    syntax: NAME

where NAME is one of the following:

The chosen syntax stays in effect when parsing all patterns that follow, until another syntax is selected.

Neither glob nor regexp patterns are rooted. A glob-syntax pattern of the form *.c will match a file ending in .c in any directory, and a regexp pattern of the form \.c$ will do the same. To root a regexp pattern, start it with ^.

Subdirectories can have their own .hgencoding settings by adding subinclude:path/to/subdir/.hgencoding to the root .hgencoding. See hg help patterns for details on subinclude: and include:.

Contract

The rules established in .hgencoding are meant to provide decoding hints to clients opting in to this system. However, there is no requirement for Mercurial, or any part of its ecosystem to strictly enforce them.

It is the expectation that graphical display clients that currently rely on decoding file contents to unicode, like web apps and desktop clients, will take advantage of the additional meta data, while core Mercurial utilities including hg merge and hg diff will not.

In the event that a rule's encoding fails to decode a file's contents, clients are free to fall back to current methods of interpretation, including heuristic-based guessing, forced decoding, or treating the file as binary. No logging or prompting is required.

The same is true for values that are not valid encoding names, or encodings whose definition is unknown to the client.

File Name Encoding

Decoding ambiguities apply to file contents, as well as file names in the bytes-based manifest. This spec applies to the former only and does not address manifest parsing.

Precedents From Other SCMs

While the problem of tracking encoding meta data for repository contents is not unique to Mercurial, the proposal set forth herein is. The situation in Git in particular is very similar, yet we choose to not adopt its approach verbatim.

Git offers a somewhat generic system for tracking file meta data through its .gitattributes files that are also checked into the repository, apply glob-based path matching, allow hierarchical file overrides and are meant to be hints, rather than strict decoding requirements, with many tools, including all of its own non-graphical tools ignoring it.

Yet our approach varies subtly.

These deviations are deliberate, as following Git directly would cause inconsistencies with existing mechanisms introduced by like .hgignore and .hgeol. It could also implicitly set unrealistic expectations with the user, with regards to other things managed by .gitattributes that Mercurial already does differently (e.g. line-endings).


CategoryDeveloper CategoryNewFeatures

Tracking File Encoding (last edited 2015-09-18 22:06:48 by Erik van Zijst)