Allow largefiles to be at a different location - Implementation details

[Piotr Listkiewicz]

I have done further research and played with current solution and i think
the best will be to reuse proto.statlfile to return location.
proto.statfile is currently used only for checking for file availability,
checking content was disabled by design.

proto.statlfile can return:
0 - file available in the store
2 - file missing

Third possible return value of proto.statlfile will be a string containing
location of largefile if the file is not currently in the store and we cant
say that this file is not available at all - nor in server nor in different
location.

In new solution

1) verifying files in remote store will send stat to server, for files the
stat will be 0 or 2 it will print the result, for location string it will
download result from this location and check file locally.

2) getting files from remote store will send stat to server first, for
files the stat will return 0 it will get files as usual by getlfiles, for
files that stat will returns 2 it will abort, for files the stat will
return location string it will download them from those locations.

Version compability

Old Mercurial on Client, New Mercurial on Server using this feature
- Old mercurial Client will fail with unexpected response from stat for
location string

New Mercurial on Client, Old Mercurial on Server without this feature
Works as usual, because stat will return 0 and client will download files
with getlfile

What do you think about it?





2016-05-24 13:03 GMT+02:00 Piotr Listkiewicz <piotr.listkiewicz at gmail.com>:

> Unfortunately, I'm not familiar at all with the wire protocol.  It will
>> take me a few days to catch up.
>
>
> It would be awesome if you made an effort. BTW whats the best hours to
> meet You in irc, what is Your nick?
>
> That would rely on the other side closing the socket to signal EOF.  I'm
>> guessing there's some SSH or HTTP server or proxy that doesn't do that
>> reliably?
>
>
> I found the answer in proto.getlfile function:
>
> # Since we can't set an HTTP content-length header here, and
> # Mercurial core provides no way to give the length of a streamres
> # (and reading the entire file into RAM would be ill-advised), we
> # just send the length on the first line of the response, like the
> # ssh proto does for string responses.
>
> So i guess length must stay. So probably we must ditinguish between
> getfile from server where it gets length at the beggining of file stream
> and getting file from different location where it reads it to the end of
> stream.
>
>
> 2016-05-24 4:29 GMT+02:00 Matt Harbison <mharbison72 at gmail.com>:
>
>> On Mon, 23 May 2016 16:52:43 -0400, Piotr Listkiewicz <
>> piotr.listkiewicz at gmail.com> wrote:
>>
>> There are several ways to implement getting large files from different
>>> location and no matter which solution is chosen there is a problem when
>>> client uses hg that doesn't support new feature to connect to repository
>>> with this feature enabled. I would like to know which of the following
>>> solutions you find best or do you have any comments etc:
>>>
>>
>> Unfortunately, I'm not familiar at all with the wire protocol.  It will
>> take me a few days to catch up.
>>
>> [snip]
>>
>>
>>> Any of those ideas have problems with knowing size of largefile, are
>>> there
>>> any reason that proto.getlfile must include size at the beggining of the
>>> stream - wouldn't it be easier if client just download all data from
>>> stream?
>>>
>>
>> That would rely on the other side closing the socket to signal EOF.  I'm
>> guessing there's some SSH or HTTP server or proxy that doesn't do that
>> reliably?
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.mercurial-scm.org/pipermail/mercurial-devel/attachments/20160524/320413f6/attachment.html>