Microsoft can’t tell time.
On April 3, an odd glitch caused computers all over the world to show the wrong time. Three of my clients called me because all the computers in their offices were six minutes off. I researched it and fixed it and moved on. It was another example of Microsoft carelessness but appeared to have been fixed.
And then three more clients called yesterday, April 12, because all the computers in the office were ten minutes off. A second outage in a service that ought to be rock solid 100% of the time – that’s just weird.
Loyal readers: you don’t have to do anything. This is a story about a glitch that has been resolved – I hope. Don’t panic. It’s just an anecdote.
Our world is full of time servers – highly accurate time sources used by computers, servers, apps, security features, and many other things in our technology world. There are public time servers run by government organizations, and there are time servers run as a public service by private companies.
Several years ago Microsoft set up time.windows.com as the time server that is consulted by default by Windows servers and non-domain individual Windows computers. In a company with a Windows domain controller, the domain server consults Microsoft’s time server for the correct time, then passes the time out to all the domain workstations. It’s crucial for computers to share the correct time. A number of things start to fall apart when the time isn’t set correctly.
Presumably there are complicated bits behind the scenes about running a time server, but the basics seem pretty obvious. Its job is to know the time, and to provide the time to devices that ask for it. That’s the job. That’s the only job.
On April 3, people began to report that their computers were showing the wrong time. Reports came in on Reddit and Twitter from all over the world. Clocks were off by a few seconds, or by an hour. Minutes had 61 seconds. There were reports of the time jumping around crazily. “It’s 10:53… then 10:20… then 10:35…” one affected user wrote on Reddit.
The reports continued for at least ten hours, although Microsoft’s official statement said that “we investigated and quickly resolved the issue our time service experienced.”
It’s not just a harmless mistake. Events on computers get the wrong timestamp. Programs and apps that depend on the time can crash or go off at the wrong time. Many security mechanisms rely on time.
So a ten-hour worldwide time outage on April 3. We all survived. Microsoft learned its lesson and that will never happen again, right?
Except it did. I have no details about what happened on April 12, more than a week later. All I know is that there was another outage that lasted at least long enough for more of my clients to notice and call me.
Sigh. What in the world is going on at Microsoft?
If there are any domain admins who don’t have this info in their notes, these are the steps on domain servers if you want to switch to a different NTP provider. There are rich debates about which time service to use. Feel free to research and form your own strong opinion. I’ve been using time.nist.gov and it’s been working fine.
From an elevated command prompt, issue these commands:
w32tm /config /manualpeerlist:time.nist.gov,0x8 /syncfromflags:MANUAL
net stop w32time
net start w32time
w32tm /resync
I’ve taken all my clients’ domain controllers off time.windows.com. Knowing the time was its only job!
