[PATCH 1 of 3] sparse-read: move from a recursive-based approach to a heap-based one

[Paul Morelle]

# HG changeset patch
# User Boris Feld <boris.feld at octobus.net>
# Date 1508323980 -7200
#      Wed Oct 18 12:53:00 2017 +0200
# Node ID 495cbf44112b9872f2803ccd836de3ffae30b28c
# Parent  537de0b14030868e3e850ae388b08f88cabc88e8
# EXP-Topic optimized-read
# Available At https://bitbucket.org/octobus/mercurial-devel/
#              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 495cbf44112b
sparse-read: move from a recursive-based approach to a heap-based one

The previous recursive approach was trying to optimise each read slice to have
a good density. It had the tendency to over-optimize smaller slices while
leaving larger hole in others.

The new approach focuses on improving the combined density of all the reads,
instead of the individual slices. It slices at the largest gaps first, as they
reduce the total amount of read data the most efficiently.

Another benefit of this approach is that we iterate over the delta chain only
once, reducing the overhead of slicing long delta chains.

On the repository we use for tests, the new approach shows similar or faster
performance than the current default linear full read.

The repository contains about 450,000 revisions with many concurrent
topological branches. Tests have been run on two versions of the repository:
one built with the current delta constraint, and the other with an unlimited
delta span (using 'experimental.maxdeltachainspan=0')

Below are timings for building 1% of all the revision in the manifest log using
'hg perfrevlogrevisions -m'. Times are given in seconds. They include the new
couple of follow-up changeset in this series.

delta-span  standard    unlimited
linear-read     922s         632s
sparse-read     814s         566s

diff -r 537de0b14030 -r 495cbf44112b mercurial/revlog.py
--- a/mercurial/revlog.py	Wed Sep 20 19:38:06 2017 +0200
+++ b/mercurial/revlog.py	Wed Oct 18 12:53:00 2017 +0200
@@ -17,6 +17,7 @@
 import collections
 import errno
 import hashlib
+import heapq
 import os
 import struct
 import zlib
@@ -170,49 +171,59 @@
     start = revlog.start
     length = revlog.length
 
-    chunkqueue = collections.deque()
-    chunkqueue.append((revs, 0))
+    if len(revs) <= 1:
+        yield revs
+        return
 
-    while chunkqueue:
-        revs, depth = chunkqueue.popleft()
+    startbyte = start(revs[0])
+    endbyte = start(revs[-1]) + length(revs[-1])
+    readdata = deltachainspan = endbyte - startbyte
+
+    chainpayload = sum(length(r) for r in revs)
 
-        startbyte = start(revs[0])
-        endbyte = start(revs[-1]) + length(revs[-1])
-        deltachainspan = endbyte - startbyte
+    if deltachainspan:
+        density = chainpayload / float(deltachainspan)
+    else:
+        density = 1.0
 
-        if deltachainspan <= revlog._srminblocksize or len(revs) <= 1:
-            yield revs
-            continue
+    # Store the gaps in a heap to have them sorted by decreasing size
+    gapsheap = []
+    heapq.heapify(gapsheap)
+    prevend = None
+    for i, rev in enumerate(revs):
+        revstart = start(rev)
+        revlen = length(rev)
 
-        # Find where is the largest hole (this is where we would split) and
-        # sum up the lengths of useful data to compute the density of the span
-        textlen = 0
-        prevend = None
-        largesthole = 0
-        idxlargesthole = -1
-        for i, rev in enumerate(revs):
-            revstart = start(rev)
-            revlen = length(rev)
+        if prevend is not None:
+            gapsize = revstart - prevend
+            if gapsize:
+                heapq.heappush(gapsheap, (-gapsize, i))
+
+        prevend = revstart + revlen
+
+    # Collect the indices of the largest holes until the density is acceptable
+    indicesheap = []
+    heapq.heapify(indicesheap)
+    while gapsheap and density < revlog._srdensitythreshold:
+        oppgapsize, gapidx = heapq.heappop(gapsheap)
+
+        heapq.heappush(indicesheap, gapidx)
 
-            if prevend is not None:
-                hole = revstart - prevend
-                if hole > largesthole:
-                    largesthole = hole
-                    idxlargesthole = i
-
-            textlen += revlen
-            prevend = revstart + revlen
+        # the gap sizes are stored as negatives to be sorted decreasingly
+        # by the heap
+        readdata -= (-oppgapsize)
+        if readdata > 0:
+            density = chainpayload / float(readdata)
+        else:
+            density = 1.0
 
-        density = textlen / float(deltachainspan) if deltachainspan > 0 else 1.0
-
-        if density > revlog._srdensitythreshold:
-            yield revs
-            continue
-
-        # Add the left and right parts so that they will be sliced
-        # recursively too
-        chunkqueue.append((revs[:idxlargesthole], depth + 1))
-        chunkqueue.append((revs[idxlargesthole:], depth + 1))
+    # Cut the revs at collected indices
+    previdx = 0
+    while indicesheap:
+        idx = heapq.heappop(indicesheap)
+        yield revs[previdx:idx]
+        previdx = idx
+    yield revs[previdx:]
 
 # index v0:
 #  4 bytes: offset