hg 4.8 sprint notes
Taken from: https://public.etherpad-mozilla.org/p/sprint-hg4.8-NOSPAMREMOVETHATLASTPART (drop the anti spam part)
Mercurial 4.8 Sprint
https://www.mercurial-scm.org/wiki/4.8sprint
Possible Topics
KitKat distribution (please tell me your expression about each flavor > foozy) (probably the hardest discussion)
[SAT] [indygreg] Wire protocol futures
[indygreg] Repository / storage interfaces / alternate storage backends
https://docs.google.com/presentation/d/1DltJOrFndBvVuFDvMaqe9LHfJqErbynEcXs3si3-72k/edit?usp=sharing
[augie] git solution
[pulkit,smf] do something to prevent breaking extensions
[SAT afternoon] Performance
[boris] performance benchmark and regression
[spectral/rdamazio] reference cycles and garbage collection
[the31k, pulkit] hg cat, stat, ls: (fs-like iface, cat speed) (discussed briefly)
[SAT] Oxidation
[SAT bikeshed] [hooper] has anyone tried "hg fix"?
[indygreg] bonsai changesets / changeset namespaces / sha-1 transition
[SAT bikeshed] unification of annotate and fastannotate
[SAT] improving UI around partial clones narrow + shallow (remotefilelog)
[DISCUSSED] [av6] IRC, opers for #mercurial, hgbot and related infra
[SAT] log aliases /templates (in last sprint discussion didn't happen because smf could not make it)
[SAT afternoon] [boris] stable-range (and other obsolescence related caches)
[SUN] [boris/spectral] next sprint location (asia please, japan maybe +3)
[smf] diff annotation https://www.youtube.com/watch?v=eUQ6KXvItK8
[SAT] [pulkit] merge diffs
improving our website (and wiki) look and content (pulkit is ready to mentor for this)
[SAT bikeshed] [augie] tweakdefaults - what should we be shipping by default, or almost-default, to minimize setup overhead?
[DISCUSSED] hg undo/rewind (everyone wants it, but what do they want exactly? undo a rebase, amend, commit etc?)
Big Discussions
[PSA - augie] Don't expect tests to be fast on macOS with APFS
[smf,pulkit,augie] code of conduct
[rdamazio] Future strategy / competitive landscape / bla bla
Need to make obsmarker exchange faster
Improve website
Improve communication around new features - especially to non-Mercurial users (blog, wiki, reddit, twitter, ...)
Improve default install
Git core features nos shipped in Mercurial (like mercurial-keyring)
Upstreaminig some of Google's trainingwheels
Upstreaming Google's improved help
Modern book / tutorials / long form documentation
https://www.mercurial-scm.org/wiki/MercurialBookPlan
http://book.mercurial-scm.org
Playground with repository visualization?
https://bitbucket.org/Unity-Technologies/satoru/src/default/ might be a good start
How to foster innovation on the collaborative platforms solutions for Mercurial?
Have Mercurial core using one of the solution
Have mercurial features integrated with the collaboration platform (auto-formatting, LFS, narrow...)
[augie] oxidization
Demos
[boris] experimentation around the test runner
[hooper] multi-commit build/test (done)
[augie] catapult tracing in hg and the test runner (done)
[boris] heptapod (gitlab + hg) (done)
[sangeet] a fixed hg grep
[rodrigo] a better hg help (done)
Possible Hacking topics
Narrow / shallow / partial clone
Python 3 (pulkit)
Oxidation
Auto-formatting state of extensions
memory merge
[augie] absorb tweaks
[SAT afternoon] State of Mercurial at [company]
[marcink] state of Mercurial at RhodeCode
[spectral] state of Mercurial at Google
[evzijst] state of Mercurial at Bitbucket
[sheehan?] state of Mercurial at Mozilla
[the31k,pulkit] state of Mercurial at Yandex
[sintzoff] Mercurial usage at unnamed company
[mbthomas] state of Mercurial at Facebook
[boris] state of Mercurial at Octobus
Code of Conduct:
- Exemples: PSF Coc and enforcements, Rust Coc
Todo:
- Add git log -S to githelp extension
Rodrigo's help demo
Help is divided into various sections and looks nice. Has ability to hide some commands. It's long but organized better.
# hg undo:
hg rewind creates new obsmarkers
time machine? would you be able to undo a rebase you did some commands ago? how often do people undo a not-most-recent command?
probably need obsmarker cycles support
Scenarios:
- Undoing an amend
- Undoing a rebase
- Undoing an absorb
- Undoing an histedit
- Undoing a commit
- Undoing a fix (auto-formatting)
- Undoing a evolve
- Undoing split
- Undoing fold
- Redo support?
Open questions:
- Should we able to undo other people operations?
- Partial undo (undo operation on a single changeset)?
- Absorb and fix should be treated an atomic operation [kyle]
-
-> Do git-less has undo support?
Other scenarios (do we want to support it):
- Undoing an update
- Undoing an update with merge? - (update --abort might be sufficient for that)?
- Undoing a bookmark move
- Undoing a pull
- Undoing a push
- Undoing an unbundle
Kyle: we should think about undoing some normal operations like bookmark creation, update etc. because nobody kind of requires them. The person who need most probably already know how to do that. (+1 from pulkit)
Reflexions:
- Local only log journal to undo commands + per-command support for undo (un-absorb, un-rebase, ...)
We can have read-only time machine quite easily but indygreg is worried about mutating after going back time
How ZFS, butterfs design their API for snapshot?
- (jeffpc answers via etherpad without attending) ZFS uses a ioctl to manage snapshots, by default it is a privileged op, but it can be delegated to any user (at least on illumos, not sure about FreeBSD/Linux/etc.)
Notes:
- No strip support
- Based on obs-markers except for commit
- We can store command id in obs-markers
- We have a local log journal of command + the commands to undo it
- Support for obsmarkers cycle
- Always be explicit about which operation to undo
We need a new abstraction which is not obsmarker, visibility abstraction
Steps:
- Add command id in UI
- Store it in obs-marker
- Solve the obsmarker cycle support
- Ensure that for each history rewriting command and commit, we have the command to reverse it
- Start experimenting with UI
# Demos
- Heptapod: Mercurial support for Gitlab
Video: https://peertube.mastodon.host/videos/watch/eadd6b5e-2339-4193-b591-ca8aa55619eb?start=
Oxidation:
Decisions about rust for next 6 months and will revisit them after 6 months:
Making hg a rust binary is explict non-goal but depnding on what goal are we can see patches
rust support will be treated on case by case basis
Quick Bikeshedding session (5 mean each topic) (kinda like old batch bikeshedding with mpm)
* hg fix (hooper): has anyone outside google tried it? No! (expected)
* let's get auto-formatting for the hg repo itself
* ui.tweakdefaults:
* should we encourage packagers (debian,etc) to enable this for their users by default
* or not.. let's just turn it on by default. (should we rename it ui.untweakdefaults? (no))
* things currently slow/broken in large repos:
* status --copies
* terse (was removed?)
* needs better documentation (including how to turn off individual tweaks)
* ACTION ITEM: everyone turn it on. Right now. Stop reading this and turn it on
* (10:56) absorb improvements: Mark thomas made some improvements and changes default to -pn and show summary and then show you a prompt asking whether you want to do that
* there can be improvements to -i and -e flags
* we should not take the experimental flag yet, getting shipped in 4.8
* (10:58) fastannotate and annotate should be merged. Discuss:
* why they are different: fa requires you to define some revset, linelogs can only store a long(?) linear history
* fa has one feature which is interesting, --deleted output mode which shows all the lines which ever existed in the file
* as of now we don't want it
* (11:06) log templates/show
* smf has log templates, there is JordiGH wip, facebook smartlog, google xl/ll
* google one is a complicated descendant of mpm templates
* there is inbuilt show extension which is niceeeeee!!!!!! (pulkit: I have aliased hg show work to hg work and I don't use log anymore)
* (similarly, almost no googlers use `hg log`)
* smf: there are two things here: revsets and color/templates
* contrib/examples directory for hgrcs?
* add a `[ui]color-scheme = ` or something config knob that could give you themes to adjust color defaults (smf will like to take a stab)
* show should investigate using the log template
* (11:15) holy hand grenade
* mass rewrite source code to have b'' strings
* we now have skipblame, so this is much less painful than previously
* just do it in first month of next cycle
* smf: send an email to mercurial-dev explaining why we are doing it right now and why we didn't do it before
* durin42: send an email with solutions to problems which can be faced during rebasing
* the https://bitbucket.org/octobus/format-source/src/default/ extension can apply code transforming (auto-formatting for example) tools before doing a merge
* can be upstreamed, ideally should use the same config file as fix
* (11:21) PEP 8 rewrite transformation
* should we make our code PEP 8 compatible to prevent editors yelling at people
* nobody objects to two blank lines in top level identifiers
* nobody objects to use underscores in variable names but people don't look happy, previous attempt: https://pha
* (11:15) holy hand grenade
* mass rewrite source code to have b'' strings
* we now have skipblame, so this is much less painful than previously
* just do it in first month of next cycle
* smf: send an email to mercurial-dev explaining why we are doing it right now and why we didn't do it before
* durin42: send an email with solutions to problems which can be faced during rebasing
* the https://bitbucket.org/octobus/format-source/src/default/ extension can apply code transforming (auto-formatting for example) tools before doing a merge
* can be upstreamed, ideally should use the same config file as fix
* (11:21) PEP 8 rewrite transformation
* should we make our code PEP 8 compatible to prevent editors yelling at people
* nobody objects to two blank lines in top level identifiers
* nobody objects to use underscores in variable names but people don't look happy, previous attempt: https://pha
* (11:15) holy hand grenade
* mass rewrite source code to have b'' strings
* we now have skipblame, so this is much less painful than previously
* just do it in first month of next cycle
* smf: send an email to mercurial-dev explaining why we are doing it right now and why we didn't do it before
* durin42: send an email with solutions to problems which can be faced during rebasing
* the https://bitbucket.org/octobus/format-source/src/default/ extension can apply code transforming (auto-formatting for example) tools before doing a merge
* can be upstreamed, ideally should use the same config file as fix
* (11:21) PEP 8 rewrite transformation
* should we make our code PEP 8 compatible to prevent editors yelling at people
* nobody objects to two blank lines in top level identifiers
* nobody objects to use underscores in variable names but people don't look happy, previous attempt: https://phab.mercurial-scm.org/D2010
* alternate take: everyone is *super* happy with _s and unhappy with current situation ;)
* lets send a summary to list to give people last chance to share their opinions
* we don't want to rewrite the old ones
* new options name should use '-' which is already accepted
* funny reminder that tweakdefaults is two words
* some black and style discussions
* BLACK IS BLACK
* Everyone should use it
* Except Anish Kapoor(I know 3 of them, which one? (the artist who can't use BLACK 2.0))
b.mercurial-scm.org/D2010
* alternate take: everyone is *super* happy with _s and unhappy with current situation ;)
* lets send a summary to list to give people last chance to share their opinions
* we don't want to rewrite the old ones
* new options name should use '-' which is already accepted
* funny reminder that tweakdefaults is two words
* some black and style discussions
* BLACK IS BLACK
* Everyone should use it
* Except Anish Kapoor(I know 3 of them, which one? (the artist who can't use BLACK 2.0))
b.mercurial-scm.org/D2010
* alternate take: everyone is *super* happy with _s and unhappy with current situation ;)
* lets send a summary to list to give people last chance to share their opinions
* we don't want to rewrite the old ones
* new options name should use '-' which is already accepted
* funny reminder that tweakdefaults is two words
* some black and style discussions
* BLACK IS BLACK
* Everyone should use it
* Except Anish Kapoor(I know 3 of them, which one? (the artist who can't use BLACK 2.0))
* (11:32) histedit-like domain specific language
* possibly useful to have a "commit rewriting plan" that mercurial can commit as an atomic transaction
* (possibly related to `hg undo`?)
* PlanPlan: https://www.mercurial-scm.org/wiki/PlanHistoryRewritePlan
* Octobus is planning to pay Sangeet to move rewrite related API from evolve to core
Python 3 test coordination: https://public.etherpad-mozilla.org/p/hg-py3-test-coordination
obsmarker exchange:
* pretty useful
* "overdue" that we figure this out
* stablerange cache is there in evolve, Octobus feels the algorithm is good, implementation is not yet convincing and can be made much faster
* stablerange algo can be used in pullbundles and Octobus has seen promising results using that B
* Augie adds that the cache file should not be very large
* Boris agrees and says that's one of the important thing which they are trying to solve and current problem was more of because of using sqlite
* Boris: we should do discovery of obsmarkers and changesets should be done together, right now we first discover changesets and then discover obsmarker
* (some demo about obsmarkers discovery) (I am not sure I understand this part, anyone else fill this)
* demo of pullbundles using stablerange cache, some working prototype
* Boris: stablerange is about how to cut csets in ranges, how to store them in caches and how to plug that into discovery parts etc.
* Augie: at what rate does the cache grows, linearly, sub-linearly etc
* Boris: pullbundle cache is log based 2, obsmarker cache is linearly in number of obsmarkers ([boris] by reviewing my notes, both will actually be NLog(N))
* indygreg: stablerange might be useful for wireprotocol v2 work and can help upstreaming with reviewing
* Boris wanted to get this in 4.8 but Augie has set expectations set high and says it's too ambitious
* indygreg: (fill me, some basic concept) why it's not useful and enough?
* augie: exchange any markers that are can be reached by walking backwards along the marker graph from the changesets which are exchanged
* augie: this is a marketing pitch as it's a feature and prunes are the edge case here
* boris: we want to improve on cset ownership and obsmarker exchange
* augie: i have 6 year evolve experience and I have a lot unrequired markers and prune markers
* We will use internal phase to hide rewritten changesets
* hg pull / hg unbundle receiving an internal phase changeset will bump the phase and unhide the changeset
* We will continue to write backup bundles. So restoring hidden changesets will mean applying backup bundle - same as today
* There are some complications related to interaction of internal phase and obsmarkers. To be figured out later.
* We will enable internal phase by default in 4.9 cycle hopefully
* Need to teach `hg debugupgraderepo` to enable internal phases on existing repos
* Steps to upstream and activate obs-markers exchange by default:
* Upstream stablerange algortihm
* Upstream the obshashrange cache without using Sqlite
* Upstream the obs-marker discovery and merge it with changeset discovery
* Merge diffs
* more than just merge-commits
* when we have a pull request/whatever, useful to determine conflicts that will arise when landing before actually attempting to land
* right now most diff views are between (common ancestor of branch) and (pull request)
* if a change is made since the common ancestor that conflicts with pull request, it's not detected until very late
* Freezes
* Currently: Merge default to stable, run tests, tag/push/accept, nothing lands on 'default' for two weeks, only bugfixes (on 'stable')
* Benefits of Freeze:
* extension maintainers able to catch up
* performance tests, etc. are able to be run
* mini vacation (for both authors *and reviewers*)
* Downsides of Freeze:
* causes large drops after freeze
* Experiment with continued development on 'default'?
* Let's do it for the next release: able to mail stuff for 'default' during the two weeks before actual release
* https://www.mercurial-scm.org/wiki/TimeBasedReleasePlan
* Benefits of current release process:
* packagers are able to handle this better
* help to avoid bad releases (head has passing tests but broken in some way)
* easier / able to "backport" bug fix releases (especially for packagers)
* easier to make "LTS" releases? (18-24 months?)
* Wireproto
* Look at GraphQL for inspiration?