Mercurial tries to pick which program to call to edit a commit message by trying the following (in order):
HGEDITOR environment variable
editor configuration option in [ui] section (in hgrc or passed with --config ui.editor command-line option).
VISUAL environment variable
EDITOR environment variable
vi, if none of the above is set
On Windows, the installer places a configuration file inside %programfiles%\Mercurial\hgrc.d\ that contains the editor settings (notepad by default). Thus, if you want to override it, you either have to redefine the configuration option or use an HGEDITOR environment variable (VISUAL and EDITOR will be ignored).