Acronis Storage Gateway is the HCI solution from Acronis. The solution, which is provided to partners, allows to build a cluster of storage and computing nodes that can become both the target for the Acronis CyberProtect service, that is an on-premise storage solution for backups, but also an hyper-converged solution that allows to deploy a wide range of storage and computing services, including virtual machines.

In this post we will describe how to perform a typical maintenance operation for the Backup Storage service, that is the removal of a node from the cluster. It happens, as in our case, when you want to decommission a node that you’re going to replace either because it is not working anymore or because you want to change the topology of the cluster.

In our case we needed to replace an old node that had a specific amount of storage space with a bigger node on a faster server and with more storage space.

Step 1: release node from the Backup Storage

The first step you need to perform is releasing the node from the Backup Storage service, going to Storage Services and the choosing Backup Storage.

The menu items you need to use

Once there you will see the current topology of the Backup storage, that is all the nodes that you have allocated for that specific service. In our case, you can see only two nodes because we have already removed the third one, the one we’re going to replace.

List of the nodes currently used by the backup storage service

Should you have any other node added to your cluster but not yet used, you would see another list of available nodes here, ready to included into available backup storage space.

Now this is the part where Acronis documentation is not clear enough. They explain how to remove a node from the infrastructure, i.e. out of the whole cluster, but they’re not clear how to do that if the node is allocated. That wouldn’t be difficult per se but the real question we faced was how to ensure that no data loss will happen.

It is relatively easy to understand that you can go to the Infrastructure menu, enter Nodes and attempt to remove the node. You would easily know that once you attempt to release a node

you will be presented a choice to release the node, optionally deciding not to migrate data to other nodes thereby incurring in a data loss. However, that won’t work if you don’t deallocate the node out of existing services.

So the first step is going to Storage services > Backup storage, selecting the node that you would like to start releasing and then click on Release

Step 2: releasing the node out of the cluster

Once you click on “Release” as explained above, nothing much seems to happen. Acronis Gateway will migrate active services out of the node (for example, Primary metadata or metadata and the Storage service) and it will exclude the node out of rotations so no new data will be stored on [...]

We have been fighting for months with a weird behaviour where Windows Server 2022 all of sudden was becoming no more accessible via RDP after a while. That was happening only on some VMs, not all of them.

The symptom was simple: you were trying to connect to a Windows Server 2022 server hosted in a VM, the RDP connection screen was showing to you the classic certificate warning and when you decide to go on server was logging you in for a very short time then the connection crashed. We were able to login via Hyper-V console but RDP was no-go. Nothing was working while the fact you were correctly logging in was proved by the fact that another session started from Hyper-V console was being forced out when you were attempting to login by the same username. However, logging in never completed.

We were quite upset since that seemed to affect and increasingly number of VMs until we discovered a shocking thing: the issue was caused by the removal of Users group access to C:\ folder. 😟😟

That’s a change we usually perform on Web Servers and actually Web Servers were the most affected kind of servers. Not sure which is the connection between Users group access to C:\ folder but if we re-enable the access for that group, RDP starts working again.

Just crazy! One the strangest thing that happened to us. We used to remove Users access to C:\ for ages and that showed no issues on any Windows Server version since 2003 or so. Hope this might help anyone else!

(credit where is due)

A few months ago we decided to change the configuration of some of our virtualization servers that were using Hyper-V replica to create replicas that could be used in case of a disaster. We used to replicate such machines so we could restore them quickly in case of issues or even start them on the destination host, if needed.

I will deal with the reasons why we decided to switch technology, from Hyper-V replica to Windows Storage Replica, in a different, bigger post but one thing we noticed quite soon during our first migrations was that sometimes our replicated volumes were apparently hanging in a WaitingForDestination status. As the name implies, it seemed that such status should mean that the source volume cannot contact the destination one so the replica is suspended.

We tried to diagnose the issue for a few time. We were sure that it wasn’t an issue with communications between the two machines because the connection from source to destination was clearly available plus some machines also had volumes replicated onto each other, that is Server 1 was replicating volume X to Server 2 and Server 2 was replicating volume Z to Server 1; and in such cases the replication was suspended one way, say from Server 1 to Server 2, but not suspended and working fine from Server 2 to Server 1. Clearly that was not a connection issue.

At first we thought that server would eventually re-establish a working connection to the destination, after trying to connect from time to time but we noticed that in one case the source suspended replication for 3 days without (self-)healing. Which was somewhat odd since during the same period of time the inverse operation had been working fine. No issues during the same period of time when the other replica was waiting for the destination while that other replica task was also displaying a Failed status on the destination instead of the usual ContinuouslyReplicating.

Clearly the replica was stuck and I noticed that there are a very few pages on the Internet dealing with this status and the same situation. In a couple of cases someone suggested to restart the replica service which is an option but not a very handy one since to restart the service we needed to shutdown ALL of the virtual machines hosted on that server.

Let me be clear here because I see this one rarely mentioned in blog posts and even in the documentation: restarting replica service will crash your services because the source volume, not the destination, will be briefly dismounted. So before you restart the service on source server, stop any service or application that is using your volume.

When we tried to shutdown the VMs that were running on that host and actually restarting the storage replica service, replication started right away without issues so it was clear that something was wrong with the storage replica service. Some users reported that they solved a WaitingForDestination status by issuing a

Get-SRGroup | Sync-SRGroup

command, which is supposed to [...]

You might find youserlf in a situation where you need to clear Powershell command history. For example, for us it was needed because we needed to prepare templates for virtual machines deployment and we needed to activate a specific software before finalizing the template.

Unfortunately you need to type a command to enter the code and activate the software and those templates could be used by our customers that might be able to retrieve the correct code by simply striking arrow-up. If you look for a solution you will find that Powershell has a specific command to do that:

Clear-History

However, you might find that this won’t solve your problem. Some history gets cancelled and some don’t or sometimes nothing at all gets removed and all of your history is still available to anyone logging using that account.

Turned out that there’s a way to check actually where Powershell stores your history. Just issue this command:

PS C:\Users\Administrator> (Get-PSReadlineOption).HistorySavePath C:\Users\Administrator\AppData\Roaming\Microsoft\Windows\PowerShell\PSReadLine\ConsoleHost_history.txt

The result is the location of the history file. Just edit this file and remove those lines you don’t want other users to access.

Neat trick but actually we’re not sure why the dedicated command is not working as expected.

Microsoft released Windows Server 2022 a few weeks ago. It did so earlier than expected, which is pretty odd, but it seems that the development of new features was already complete and mature so there was no point to wait a few months more. The last sentence pretty much explains the current status of Windows Server: so few changes, albeit interesting ones, that the product can be released months earlier than planned and without much fanfare.

While generally speaking the development of new features and functionalities for server applications mostly moved out of the base operating system and while we can also concede that Windows Server has many features at the OS level so it is difficult to design new ones, the fact each new release of Windows Server keeps bringing few new features is definitely because of Microsoft errors. Server applications today are either simple, think about a website or a Web application, or complicated. In the latter case, you generally think about microservices, Kubernetes and so on.

So we might say that since a few years Windows Server became more a tool to build base infrastructures than the base for new applications. That means that you usually install something on top of Windows Server to manage and deploy applications and you don’t usually rely on OS services only to develop your application. It is safe to say that it is a common scenario because Linux too is experiencing the same transformation.

However, Microsoft could have responded to Linux popularity by upping the ante and push more innovation into Windows Server and they didn’t.

Windows Server 2022 – continued fragmentation

The new Windows Server version has some nice and neat features. Besides a better Kubernetes compatibility, which is basically mandatory today, it features and big work to improve security. Credit where is due. That improves a lot both standalone servers security but it is also a fundamental improvement when deploying clusters, the famous infrastructures.

Other improvements include TLS 1.3 enabled by default and a few improvements around SMB, all of them improve security.

There would be other two important features to mention, one of them would be very important, but surprise: they won’t be available to all customers but only to those using Azure. At least for now.

Microsoft introduced Windows Server Datacenter: Azure Edition, another attempt to fragment Windows Server but the first one to draw a line between “standard” Windows Server and Azure versions. And we cannot welcome this kind of behavior because it basically starts to leave non-Azure Windows Server installations behind in an attempt to force Windows customers to move to Azure. Quite frankly, this is indecent especially when you factor how much Windows Server costs when compared to its competitor.

Features that Microsoft is withholding from its non-Azure customers are very important.

First, there is the much awaited hot-patching, that is a way to patch Windows without rebooting. Microsoft aims to use hot-patching especially for security patches and it would be a very ground-breaking innovation when considering how many emergency updates are needed [...]