[RFC] Designing a store format that uses fewer files

[Gregory Szorc]

Issue 4889 (creating/appending thousands of files is absurdly slow on NTFS)
is kind of a big deal at Mozilla since we have a number of developers and
automated processes running Windows. As I've documented in commit messages
like
https://selenic.com/pipermail/mercurial-devel/2015-September/073788.html,
this NTFS behavior adds up to some operations taking several minutes longer
on Windows than other platforms. Moving file closing to a background thread
pool shows promise and I intend to get those patches landed for 3.7.
However, the underlying performance issue stems from Mercurial's store
model of 1 file per store path.

While I feel like 1 file per store path is good enough for most people,
cracks do form at scale *on all platforms*. So, I've been thinking of new
store formats that rely on fewer files so filesystems won't interfere as
much.

I wrote up https://www.mercurial-scm.org/wiki/PackedRepoPlan with the
beginnings of a proposal.

I was hoping to keep it simple. But somewhere along the way I realized it
overlaps significantly with reader locks, better transactions, ending data
duplication for copies and moves, and keeping the performance impact of
obsolete/hidden changesets in check. I may have accidentally scope bloated
to a grand unified store format. Oops.

I'm really curious what others think about the proposal. If I break 1 file
per store path to make Windows scale, I introduce a new store requirement.
And new store requirements mean you can break the world because there are
no backwards compatibility concerns. So I think designing the new format to
be compatible with all the other crazy stuff people have talked about is
justified.

I'm sure there are many flaws in my proposal. Perhaps some fatal ones. I'd
love to hear what they are.

Gregory
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20151126/87d0fa4c/attachment.html>