At VaiSulWeb virtualization is so pervasive that basically there aren’t physical machines other than virtualization hosts or storage since about 2008 or 2009. That means about 10 years now. And counting. Microsoft Hyper-V served us very well and over time our reliance on that technology increased with happy results. It has been solid and consistent and allowed us to scale up more and more, adding new technologies and solutions and enabling us to virtualize roles and workloads that seemed a bit difficult to virtualize.

As many other providers, we recently started integrating AMD CPUs into our infrastructure, given the terrific advantages that those could bring to the datacenter especially for generic or mixed workloads. Only a subset of our infrastructure has been migrated to the new AMD servers but things got easier because of the many advantages that the Windows Server (+ Hyper-V) platform could provide and we carefully started to fill those hosts by migrating workloads. Results have been very pleasing.

Performance improvements and security

As we had planned, we immediately started benefiting of performance improvements. Not only CPU-bound tasks were faster but also I/O performance were terrific.

One of our goals was also to improve security by using AMD technologies that showed better resiliency when dealing with security issues when compared to Intel chipsets and CPUs and we were also prepared to face a few limitations. For example, Hyper-V isolated containers are not supported on non-Intel CPUs and thus cannot be used.

We started migrating our workloads to the new servers and Hyper-V functionalities like shared-nothing live migration and live migrations made the process quite easy and straightforward. In a few days about 80% of the target workloads have been migrated with minimal or no disruption at all. So far so good.

Unexpected crashes

Then something odd happened: one of the hosts started to crash. That was surely something we rarely faced in our 10+-years-long experience with Hyper-V but that specific host was crashing up to 2-3 times per day and while it usually was back online in about 2 minutes with all of its VMs restarted, the virtual machines that it was hosting were obviously also crashing causing downtimes. That was very surprising since similar machines (basically identical since they were using the same components) were not exhibiting any issue even after running for weeks in production and even more in our labs.

That machine had been running for days without issues then started crashing 2-3 times per day with no traceable pattern. Sometimes it could run for hours (10 or more) without issues, sometimes it was crashing two times in 15 minutes. Weird. And scary.

We decided to halt our migration to ensure that we didn’t miss any incompatibility between Windows Server 2019 and those AMD servers yet other servers were not having any issue and tracing back the issues we had, that specific machine had not been exhibiting issues for days before it started crashing so often.

Diagnosing the issues

The first thing that you might want to do in such cases is to ensure [...]