D6288: automation: shore up rebooting behavior

indygreg (Gregory Szorc) phabricator at mercurial-scm.org
Fri Apr 19 10:41:40 EDT 2019


indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  There was a race condition in the old code. Use
  instance.stop()/instance.start() to eliminate it.
  
  As part of debugging this, I also found another race condition
  related to PowerShell permissions after the reboot. Unfortunately,
  I'm not sure the best way to work around it. I've added a comment
  for now.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6288

AFFECTED FILES
  contrib/automation/hgautomation/aws.py

CHANGE DETAILS

diff --git a/contrib/automation/hgautomation/aws.py b/contrib/automation/hgautomation/aws.py
--- a/contrib/automation/hgautomation/aws.py
+++ b/contrib/automation/hgautomation/aws.py
@@ -808,10 +808,26 @@
         )
 
         # Reboot so all updates are fully applied.
+        #
+        # We don't use instance.reboot() here because it is asynchronous and
+        # we don't know when exactly the instance has rebooted. It could take
+        # a while to stop and we may start trying to interact with the instance
+        # before it has rebooted.
         print('rebooting instance %s' % instance.id)
-        ec2client.reboot_instances(InstanceIds=[instance.id])
+        instance.stop()
+        ec2client.get_waiter('instance_stopped').wait(
+            InstanceIds=[instance.id],
+            WaiterConfig={
+                'Delay': 5,
+            })
 
-        time.sleep(15)
+        instance.start()
+        wait_for_ip_addresses([instance])
+
+        # There is a race condition here between the User Data PS script running
+        # and us connecting to WinRM. This can manifest as
+        # "AuthorizationManager check failed" failures during run_powershell().
+        # TODO figure out a workaround.
 
         print('waiting for Windows Remote Management to come back...')
         client = wait_for_winrm(instance.public_ip_address, 'Administrator',



To: indygreg, #hg-reviewers
Cc: mercurial-devel


More information about the Mercurial-devel mailing list