At VaiSulWeb virtualization is so pervasive that basically there aren’t physical machines other than virtualization hosts or storage since about 2008 or 2009. That means about 10 years now. And counting. Microsoft Hyper-V served us very well and over time our reliance on that technology increased with happy results. It has been solid and consistent and allowed us to scale up more and more, adding new technologies and solutions and enabling us to virtualize roles and workloads that seemed a bit difficult to virtualize.

As many other providers, we recently started integrating AMD CPUs into our infrastructure, given the terrific advantages that those could bring to the datacenter especially for generic or mixed workloads. Only a subset of our infrastructure has been migrated to the new AMD servers but things got easier because of the many advantages that the Windows Server (+ Hyper-V) platform could provide and we carefully started to fill those hosts by migrating workloads. Results have been very pleasing.

Performance improvements and security

As we had planned, we immediately started benefiting of performance improvements. Not only CPU-bound tasks were faster but also I/O performance were terrific.

One of our goals was also to improve security by using AMD technologies that showed better resiliency when dealing with security issues when compared to Intel chipsets and CPUs and we were also prepared to face a few limitations. For example, Hyper-V isolated containers are not supported on non-Intel CPUs and thus cannot be used.

We started migrating our workloads to the new servers and Hyper-V functionalities like shared-nothing live migration and live migrations made the process quite easy and straightforward. In a few days about 80% of the target workloads have been migrated with minimal or no disruption at all. So far so good.

Unexpected crashes

Then something odd happened: one of the hosts started to crash. That was surely something we rarely faced in our 10+-years-long experience with Hyper-V but that specific host was crashing up to 2-3 times per day and while it usually was back online in about 2 minutes with all of its VMs restarted, the virtual machines that it was hosting were obviously also crashing causing downtimes. That was very surprising since similar machines (basically identical since they were using the same components) were not exhibiting any issue even after running for weeks in production and even more in our labs.

That machine had been running for days without issues then started crashing 2-3 times per day with no traceable pattern. Sometimes it could run for hours (10 or more) without issues, sometimes it was crashing two times in 15 minutes. Weird. And scary.

We decided to halt our migration to ensure that we didn’t miss any incompatibility between Windows Server 2019 and those AMD servers yet other servers were not having any issue and tracing back the issues we had, that specific machine had not been exhibiting issues for days before it started crashing so often.

Diagnosing the issues

The first thing that you might want to do in such cases is to ensure [...]

If you’re anything like me, virtualization has a great and important role in your everyday work. I use to virtualize most of my development machines, with Visual Studio and the whole stack. That allows me to avoid reconfiguring the environments whenever I need to switch my main machine, plus you get all the benefits of virtualization like for example the ability to backup the whole machine and move it elsewhere.

Up until now, such deployments had just one limitation: you could not use them to run mobile apps since nested virtualization was not supported on Windows 8 or Windows Server 2012 R2 so you could not run emulators that needed Hyper-V. Things changed with Windows 10 and Windows Server 2016 since nested virtualization is now allowed so I thought that was perfect chance to redeploy machines I used for mobile development into VMs.

So Visual Studio 2015 is running and Xamarin ready, you create a new Xamarin Forms project and you want to test that your environment is working fine. You start debug for the Android application but the Visual Studio Android emulator is not working. A “Hyper-V component is not running” :

Something happened while starting a virtual machine: ‘VS Emulator 5-inch KitKat (4.4) XXHDPI Phone.macoswin’ failed to start. (Virtual machine ID 77D02994-C4A1-4A43-9AE7-7E55C03521D4) The Virtual Machine Management Service failed to start the virtual machine ‘VS Emulator 5-inch KitKat (4.4) XXHDPI Phone.macoswin’ because one of the Hyper-V components is not running (Virtual machine ID 77D02994-C4A1-4A43-9AE7-7E55C03521D4). _

Hyper-V seems to be working fine plus it has been installed, something that never succeeded in past Windows (Server) versions but emulators don’t work. The reason for this is your host is not enabling all Hyper-V features to the guest machine. That’s probably related to the fact that Hyper-V is also needed for Containers but I’m not sure about that.

Turns out that you need to tell host that you want to expose ALL Hyper-V functionalities if you want to enable full nested virtualization so you have to log into the host (which might simply be your own notebook or desktop PC or even a Surface Pro, like in my case) and run this command using an elevated shell:

Set-VMProcessor -VMName <VMName> -ExposeVirtualizationExtensions $true

<VMName> obviously is the name of the virtual machine that you want to enable nested virtualization for. Also ensure that such VM has been shut down (no saving, no suspension). Once you reboot, emulators will start working. You might need to re-deploy the emulators VMs just to ensure that everything will be redeployed in a smooth way. You simply need to delete those emulators inside Hyper-V Manager.

The problem is: when you will deploy your test app – even the simplest one, the one generated by the template – it will not run inside the emulator. It just starts and shuts down almost instantly, probably crashing.

To fix this problem, you need to change a setting in the virtual machine that is hosting Visual Studio and the emulators as well:

Again, you need to shut the VM down before changing [...]

Nel suo processo di trasformazione verso architetture completamente virtualizzate (SDDC, software defined datacenter), Bank of America ha attivato due infrastrutture parallele, una basata su OpenStack e l’altra sul software di un vendor che ha deciso di non rendere pubblico. In questo momento, l’infrastruttura fornita dal vendor sta prevalendo mentre BofA ritiene OpenStack “non ancora pronto”. Pochi dei 1500 servizi che la banca sta trasferendo ogni mese nella nuova struttura vengono infatti spostati sulla piattaforma OpenStack a causa di non meglio precisati “problemi” che sono stati aggirati trasferendo circa 6500 servizi sulla seconda infrastruttura, quella proprietaria.

“All’inizio di quest’anno abbiamo scoperto che OpenStack non è pronto e non lo è neanche per eseguire i servizi non mission-critical che attualmente abbiamo attivi” è stato il commento di David Reilly, il responsabile generale delle infrastrutture della banca, la seconda più grande degli Stati Uniti con circa 2,100 miliardi di asset, una dimensione che le consente di sperimentare le tecnologie più innovative senza grandi problemi di costi. L’obiettivo di Bank of America è quello di virtualizzare tutti gli elementi della propria infrastruttura – il networking, lo storage e la sicurezza – in modo da poter modificare velocemente le proprie applicazioni o attivarne di nuove quando richiesto dai clienti.

Mentre oggi la banca usa la nuova infrastruttura principalmente per attività non critiche come lo sviluppo ed il testing, a partire dal prossimo anno prevede di trasferirvi i circa 70mila servizi attualmente attivi. Non è però chiaro quanti di questi opereranno su OpenStack. Reilly ha confermato che gli ingegneri di BofA hanno risolto i problemi principali e che il giudizio non è legato alla tecnologia di base che “funziona molto, molto bene”. Il problema principale è che la piattaforma non è pronta per gestire migliaia di servizi e la banca ha dovuto anche creare tool di gestione che sostituissero quelli di OpenStack, ritenuti inadatti a gestire una soluzione così complessa ed automatizzare le operazioni. “Se dobbiamo migrare circa 1500 servizi al mese dobbiamo farlo in modo automatico, non possiamo dipendere dall’intervento umano” ha concluso Reilly.

Nonostante queste difficoltà, BofA non ha intenzione di cancellare il proprio investimento nella piattaforma OpenStack anche se ritiene che l’infrastruttura creata sulla piattaforma del vendor ed alternativa a quella open-source sia più indicata per l’esecuzione e gestione dei servizi più critici e che facciano un uso intensivo delle risorse, soprattutto lo storage.

Man mano che la piattaforma di OpenStack maturerà, Reilly pensa che sarà piano piano trasferire i servizi della banca su quest’ultima. “E’ difficile competere con qualcosa che è gratis. La differenza di prezzo comincerà a giocare un ruolo importante quando le funzionalità di OpenStack miglioreranno.”.