It is shaping up to be a Black Wednesday for providers of online services after Microsoft Azure Virtual Machines users suffered lingering near-global glitches that prevented them from spinning up new Windows-based systems.
According to Microsoft’s status page, the downtime happened “as early as” 05:12 UTC today and has yet to be fixed, with the company saying it is “Applying Mitigation”, which we are sure is of some comfort to irritated users.
The message states that a “subset of customers using Windows Virtual Machines may experience failure notifications when performing service management operations – such as start, create, update, delete.”
As such, “deployments of new VMs and any updates to extensions may fail. Non-Windows Virtual Machines, and existing running Windows Virtual Machines should not be impacted by this issue. Services with dependencies on Windows VMs may also experience failures when creating resources.”
Services are impacted everywhere from the Americas, to Europe, the Middle East, Africa, and Asia Pacific. Azure Government and Azure China were not listed as running borked VM services.
The Azure Support page on Twitter made no mention of the incident but confirmed to a customer it was “aware of this issue” and that “engineering teams are actively collaborating to resolve this.”
As of 11:03 UTC, Microsoft said it had “identified that calls made during service management operations are failing due to a required artifact version not returning as expected during query.
“The failure shows that that a required extension cannot be located. We are currently implementing a mitigation option to force a refresh of the extension and are seeing signs of recovery where mitigation has been deployed. We expect recovery to be observed as the mitigation progresses across regions. The next update will be provided within 60 minutes, or as events warrant.”
So sysadmins, it might be best to explain the situation to your users and then pop off for some lunch. It’ll all be fixed by the time you return, right? At least all of Microsoft’s other services appear to be working… for now.
As revealed earlier today, European great cloud hope OVH suffered some downtime after a planned network reconfiguration turned into something less routine, with human error blamed. ®