Mercurial does not support TLS server name indication (SNI), as reported in Issue2811. However, Python 2.x itself does not support SNI and will never do (http://bugs.python.org/issue5639#msg141913). Thus, in order to archive support for SNI, Mercurial would have to support Python 3.2, which does support SNI (http://bugs.python.org/issue5639#msg141950).
That's the best rationale for Python 3 support so far ;-) Optional use of some external crypto library is however more likely to happen, IMHO.
Yeah, this is the first carrot I've seen WRT Python3.
Maybe python-gnutls? GNU TLS supports SNI: http://www.gnu.org/software/gnutls/manual/html_node/TLS-Extensions.html M2Crypto might support SNI, I could not figure out whether it supports SNI or not (there is an open question at Stack Overflow: http://stackoverflow.com/questions/7961547/does-m2crypto-have-client-side- server-name-indication-sni-support)
Adjusting title.
--- Bug imported by bugzilla@serpentine.com 2012-05-12 09:25 EDT --- This bug was previously known as _bug_ 3090 at http://mercurial.selenic.com/bts/issue3090
*** Bug 3505 has been marked as a duplicate of this bug. ***
I have a patch for this pretty much done. I just need to write a test for this. I need some help with this. There are two ways of doing this. First, somehow get hg serve to serve up a different cert based on hostname and then tell the test to use both 127.0.0.1 and localhost and make sure it accepts both certs. The other way is to get someone to setup a reliable host somewhere that we can then use for testing. Way number one sounds better but it also seems like it would take a lot of useless code to change hg serve to serve up different certificates. Useless because no one really has a use other than testing for hg serving multiple certs.
(In reply to comment #7) How did you get an SSL library that supports SNI on Python 2.x?
Using pyOpenSSL to communicate and a combination of pyasn1 and code ripped and slightly modified from ndg.httpsclient to decode the subjectAltName portion of the cert. Should I just submit the patch, let people look at it and then get it revised once we have a test?
I think at this point seeing the patch might be the most helpful thing, yes.
That said, please wait until after burning man, as I want mpm to see the patch and I believe he's going to declare bankruptcy on patches when he returns.
Well, until then here's the temporary solution: http://pastebin.com/cWFRYPuL .
One high-level comment: httpclient is actually a disjoint project, https://code.google.com/p/httpplus - any necessary patches to that directory should go there. It's a Google-owned project, so there's a contributor license agreement (not copyright assignment) required for any patches there. Does this patch work without pyopenssl? I wonder if it'd be easier to just do pyopenssl stuff in httpplus, and not even worry about supporting SNI outside http2?
Perhaps you could give me an overview of how httpplus fits in. I couldn't exactly work out where it gets used to makes connections vs where sslutils gets used. Also, the code in sslutils seems to duplicate a lot of what's done in socketutil.
(In reply to comment #13) Sorry, I forgot to respond to your question about working without pyopenssl. It works in the sense that pages that don't require SNI will still work, but the SNI aspect requires pyopenssl and pyasn1.
Sure. httpplus is a complete rewrite of httplib, with an eye towards fixing major problems for Mercurial. I'm hoping (some day) to be able to show up at the stdlib with a battle-tested version of it that'll drop into some version of Python 3, but that's still a ways off. Anything we do to support SNI needs to work with httpplus, but taht probably means at least some small patches to httpplus. If you'd rather, I can do the patches to httpplus, but you should leave httpclient unmodified in hg - we can just document there's work to do for SNI and httpplus, worst case. My gut says we should have some sort of ssl utility module that handles all of the wrap_socket behavior, and works consistently across machines with and without pyopenssl. I'd be fine putting such a function in httpplus and not even doing SNI for the rest of hg, but I'm not sure if that's a good plan. We're hoping to enable the http2 flag by default as soon as in the next couple of weeks.
I'm still confused as to what http2 is. Also, I'm still lost as to where sslutils is used vs where httpplus/httpclient is used. I just did a test now to try to help me get a handle on it, and it appears that when I do: 'hg clone https://someplace/repo/' that it uses the ssl_wrap_socket function in sslutil. As for putting the code in httpplus, according to the google license agreement I can't because it's not all my code. Some of it is just code that I grabbed that has a BSD license on it.
If you can stop by the irc channel (#mercurial on freenode) I can try and answer questions interactively. At this point, I think you should patch mainline Mercurial, and explicitly not touch anything in mercurial/httpclient - I'll fix httpclient to do the right thing "somehow" once we have an interface defined. Does that sound at least somewhat workable?
Yep, I'm on the irc. I tried to get some response a few times and nobody said anything, and you responded in here, so I figured this was the more reliable way to communicate. I'm CrazyCasta in #mercurial and if you pm me I should respond if I'm there.
Per some IRC discussion, it sounds like Alex will take a run at packaging what he's got on PyPI (it sounds wonderfully generally useful), and then we'll use the library in hg.
As Augie has said I've put all my SSL SNI code into a PyPI package called ssl_sni. The new patch is at http://pastebin.com/MvYgzjKv. I'm testing with two virtualenvs. One includes only pyOpenSSL (required to run test-https.t), the other includes ssl_sni which pulls in pyOpenSSL and pyasn1. The test-https.t has passed on both and I'm running the full suite of tests now. Assuming they pass I'll renumber ssl_sni to 0.1 to represent that it is should be ready for use with mercurial (and hopefully httpplus).
I have now run the full test suite. The hash for only pyOpenSSL is 2762200200, with ssl_sni it is 2320926546. The following is the output of run-tests.py after the first (hash seed) line. I have included it to show which tests were skipped due to my missing components in case anyone notices something that should get tested. .s.....s.s........ss.s.......s......s.......................................s...s........s......................................s..........s............s..s...............s.........s.s..ss..........................s.s...........................s...................................................s.........................................................s.........s................ss...............s........................................................... Skipped test-convert-svn-move.t: missing feature: subversion python bindings Skipped test-convert-svn-source.t: missing feature: subversion python bindings Skipped test-convert-svn-encoding.t: missing feature: subversion python bindings Skipped test-convert-svn-branches.t: missing feature: subversion python bindings Skipped test-convert-hg-svn.t: missing feature: subversion python bindings Skipped test-highlight.t: missing feature: Pygments source highlighting library Skipped test-convert-p4-filetypes.t: missing feature: Perforce server and client Skipped test-convert-svn-startrev.t: missing feature: subversion python bindings Skipped test-convert-svn-tags.t: missing feature: subversion python bindings Skipped test-convert-mtn.t: missing feature: monotone client (>= 1.0) Skipped test-convert-cvs.t: missing feature: cvs client/server Skipped test-convert-bzr.t: missing feature: Canonical's Bazaar client Skipped test-casecollision-merge.t: missing feature: case insensitive file system Skipped test-convert-cvs-detectmerge.t: missing feature: cvs client/server Skipped test-convert-baz.t: missing feature: GNU Arch baz client Skipped test-convert-cvs-synthetic.t: missing feature: cvs client/server Skipped test-convert-bzr-directories.t: missing feature: Canonical's Bazaar client Skipped test-convert-cvsnt-mergepoints.t: missing feature: cvs client/server Skipped test-convert-tla.t: missing feature: GNU Arch tla client Skipped test-convert-p4.t: missing feature: Perforce server and client Skipped test-casefolding.t: missing feature: case insensitive file system Skipped test-convert-cvs-branch.t: missing feature: cvs client/server Skipped test-convert-darcs.t: missing feature: darcs client Skipped test-convert-bzr-merges.t: missing feature: Canonical's Bazaar client Skipped test-no-symlinks.t: system supports symbolic links Skipped test-check-pyflakes.t: missing feature: Pyflakes python linter Skipped test-convert-bzr-ghosts.t: missing feature: Canonical's Bazaar client Skipped test-convert-bzr-114.t: missing feature: Canonical's Bazaar client >= 1.14 Skipped test-convert-bzr-treeroot.t: missing feature: Canonical's Bazaar client # Ran 429 tests, 29 skipped, 0 failed.
Please note that Python 2.7.7 will be adding SNI support in the next few weeks: http://legacy.python.org/dev/peps/pep-0466/
Python 2.7.9 was released on 12/10/14 with support for SNI. See: https://www.python.org/downloads/release/python-279/ I can confirm that when running on 2.7.9 Mercurial works with https on servers that require SNI.
Note that there still appear to be some issues, namely: Mercurial still reports certificate fingerprint warnings despite certificate validation being done by default in the Python https client (see https://www.python.org/dev/peps/pep-0476/ ). Also the certificate fingerprint that Mercurial reports appears to be the certificate obtained before SNI (ie: the default certificate on the server, not the proper certificate for the requested host).
I'm curious what version of mercurial you're using. I've tried 3.2.3 (the version installed on my computer) as well as the latest from the repo. Neither one works for me, both say: abort: error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581) I'm going to submit my new patch (to use SSLContext from 2.7.9) to the list, because as far as I can tell, it's necessary.
Fixed by http://selenic.com/repo/hg/rev/bf07c19b4c82 Alex Orange <crazycasta@gmail.com> https: support tls sni (server name indication) for https urls (issue3090) SNI is a common way of sharing servers across multiple domains using separate SSL certificates. As of Python 2.7.9 SSLContext has been backported from Python 3. This patch changes sslutil's ssl_wrap_socket to use SSLContext and take a server hostname as and argument. It also changes the url module to make use of this argument. The new code for 2.7.9 achieves it's task by attempting to get the SSLContext object from the ssl module. If this fails the try/except goes back to what was there before with the exception that the ssl_wrap_socket functions take a server_hostname argument that doesn't get used. Assuming the SSLContext exists, the arguments to wrap_socket at the module level are emulated on the SSLContext. The SSLContext is initialized with the specified ssl_version. If certfile is not None load_cert_chain is called with certfile and keyfile. keyfile being None is not a problem, load_cert_chain will simply expect the private key to be in the certificate file. verify_mode is set to cert_reqs. If ca_certs is not None load_verify_locations is called with ca_certs as the cafile. Finally the wrap_socket method of the SSLContext is called with the socket and server hostname. Finally, this fails test-check-commit-hg.t because the "new" function ssl_wrap_socket has underscores in its names and underscores in its arguments. All the underscore identifiers are taken from the other functions and as such can't be changed to match naming conventions. (please test the fix)
Bulk testing -> fixed