[PATCH 0 of 2 bfiles] support HTTP stores (as client and server)

[alexandru]

On 2010-06-04, at 1:52 PM, Greg Ward wrote:

> On Thu, Jun 3, 2010 at 6:47 PM, alexandru <alex at hackd.net> wrote:
>> This series implements HTTP stores, so you can push/pull/verify big files over HTTP.
> 
> Thank you!  This looks very good.  Some random comments:
> 
>  * Would you mind using your full name in the commit?  Pretty soon
> now I have to write down the list of contributors, and just plain
> "alexandru" is a bit vague.

I've changed the patches to now include my full name, if you prefer to use that.

> 
>  * I think your patches are in the wrong order: after applying the
> client-side changes, bfput over HTTP cannot possibly work.  Apply the
> server-side change first, and we avoid that problem.

While it's possible I messed up while playing with mq (I've re-done the patch queue on BitBucket) bfput *should* work only with the httpput patch, without requiring the server-side changes. The way I approached it, bfput doesn't care (or know) what server is one the other end — as long as it understands GET and PUT (and HEAD for bfverify, though some answers from that command are ignored).

The biggest shortcoming is that it doesn't currently do basic or digest auth if the server wants/asks for it, but I'm working to add that.


Is there something in the design of bfiles that I missed, that would tie bfput into bfserve? Or did I just swap the patches by accident previously? (as I said, first timer with mq)

> 
>  * I'm unclear how this would be deployed.  If I understand, it looks
> like the admin of the machine where the central store lives will have
> to arrange for "hg bfserve --http" to be called in some circumstances.
> When?  On every request? At system startup?  What if I want to use
> (say) Apache as the front-end (for SSL, authentication, etc.) -- would
> I have to setup mod_proxy?

bfserve listens for files until it gets killed, so having it start with the system should be enough. It does use Python's BaseHTTPServer which isn't as grown up a webserver as, say, Apache. I don't really see it as a permanent option, more for developers to send bfiles to one another (i.e. over a LAN link)

> 
>  * tests?

Oops, I ran the test suite that came with bfiles as I saw some http-related tests there. I'll look into adding my own as soon as possible.

> 
>  * terminology: I deliberately use "put" rather than "push".  You
> push changesets, but you put big files.  I saw one or two comments
> that said "push" but should say "put".  Likewise you pull changesets
> but get big files.  (Early on, bfupdate was going to be called bfget,
> but then I realized it has to delete files too.)

Fixed.

> 
> Another way to provide the server-side component of this would be as a
> CGI or WSGI script -- e.g. "hg bfserve --cgi /path/to/store".  Then
> the admin has to arrange for Apache to run that for certain URLs.
> That's more consistent with how hgweb works, which I think is nice.
> 
> OTOH, a standalone HTTP server is much easier to write tests for.  Hmmmm.

I can look at what hgweb does and what would be needed to make bfserve run as CGI/WSGI, so it can be plugged into Apache. But I tried to make bfput agnostic in terms of the server it's talking to, so as long as the store implements HTTP PUT, all should be fine.

I did write a lightweight, CherryPy-based server when I was implementing the client support, so if you want to go that route I can finish it up and write some tests. From what I gather, CherryPy apps can be run as WSGI, so it would work in Apache/mod_wsgi. OTOH, it's an extra dependency which isn't as nice as having everything built-in.


Cheers.

--
@

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 314 bytes
Desc: not available
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20100605/245896ea/attachment.pgp>