[PATCH 3 of 3] Fetch revision log on demand (in batches of 50 revisions)

Daniel Holth dholth at fastmail.fm
Sun Jul 1 20:47:30 CDT 2007


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Edouard Gomez wrote:
> Merged the convert_svn.rev method bit into ./svn/enh-add-svn-
> converter.patch.
>
> I advise putting the 50 revision fetching constant somewhere near the top
> so we can try a few values easily for big repos (mplayer > 23000 commits)
I have fine results with a svnsync of The Subversion Repository, with
upwards of 7500 revs at a time.

As for fetching the log in pieces, it should be quite simple to
measure the point where the overhead of asking for a log is more than
getting the log messages themselves. If 1.4.2 bindings run out of ram
with 100 revs??? then we could play with better memory pool
management, explicitly closing memory pools and so forth. The memory
pool management is one of the things that has been made more automatic
in the bindings over time.

We also really need to invent a way for the main converter to provide
scratch space for convert_svn (a shelve file in dest/.hg/ would be
fine). If the converter_source implementation had a way to even look
at the shamap file, which happens to encode the last branch/revision
we fetched, it would also be a great clue about which revisions we
need to log for.

And if it is really a problem to merely fetch the log, it's not any
slower to parse the XML output of "svn log". The big wins from using
the API come from not reconnecting to the source repository so many
times, and from not actually typing "svn update" on some working copy
for each fetched revision.

I should mention that there's a lot to be said for using the *working
bindings*...  they aren't that hard to get...

>
> And the bad part:
>
> The rest of the patch doesn't work for me on a very simple repo.
>
> I uploaded the problematic svn repo at:
> http://ed.gomez.free.fr/mercurial/rawstudio.repo.svn.tar.bz2
>
> It's a svnsync of https://rawstudio.org/svn/rawstudio/
>
> It breaks at fetching rev 1172.
I was able to fix a problem with "copy a directory and delete a file
in the same commit" with a 3-revision repository. If we can figure out
which cases are causing problems, we should make some tiny test
repositories to share and include in unit tests.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGiFkxVh4W2pVfoMsRAshBAJ9DDD3cXDo3HxYNtC+5ZIACBbGQ0QCgsXL+
i8GgWn5Bfexg0l1lFbgm41U=
=IsSj
-----END PGP SIGNATURE-----



More information about the Mercurial-devel mailing list