Acronis Storage Gateway is the HCI solution from Acronis. The solution, which is provided to partners, allows to build a cluster of storage and computing nodes that can become both the target for the Acronis CyberProtect service, that is an on-premise storage solution for backups, but also an hyper-converged solution that allows to deploy a wide range of storage and computing services, including virtual machines.

In this post we will describe how to perform a typical maintenance operation for the Backup Storage service, that is the removal of a node from the cluster. It happens, as in our case, when you want to decommission a node that you’re going to replace either because it is not working anymore or because you want to change the topology of the cluster.

In our case we needed to replace an old node that had a specific amount of storage space with a bigger node on a faster server and with more storage space.

Step 1: release node from the Backup Storage

The first step you need to perform is releasing the node from the Backup Storage service, going to Storage Services and the choosing Backup Storage.

The menu items you need to use

Once there you will see the current topology of the Backup storage, that is all the nodes that you have allocated for that specific service. In our case, you can see only two nodes because we have already removed the third one, the one we’re going to replace.

List of the nodes currently used by the backup storage service

Should you have any other node added to your cluster but not yet used, you would see another list of available nodes here, ready to included into available backup storage space.

Now this is the part where Acronis documentation is not clear enough. They explain how to remove a node from the infrastructure, i.e. out of the whole cluster, but they’re not clear how to do that if the node is allocated. That wouldn’t be difficult per se but the real question we faced was how to ensure that no data loss will happen.

It is relatively easy to understand that you can go to the Infrastructure menu, enter Nodes and attempt to remove the node. You would easily know that once you attempt to release a node

you will be presented a choice to release the node, optionally deciding not to migrate data to other nodes thereby incurring in a data loss. However, that won’t work if you don’t deallocate the node out of existing services.

So the first step is going to Storage services > Backup storage, selecting the node that you would like to start releasing and then click on Release

Step 2: releasing the node out of the cluster

Once you click on “Release” as explained above, nothing much seems to happen. Acronis Gateway will migrate active services out of the node (for example, Primary metadata or metadata and the Storage service) and it will exclude the node out of rotations so no new data will be stored on [...]

We have been fighting for months with a weird behaviour where Windows Server 2022 all of sudden was becoming no more accessible via RDP after a while. That was happening only on some VMs, not all of them.

The symptom was simple: you were trying to connect to a Windows Server 2022 server hosted in a VM, the RDP connection screen was showing to you the classic certificate warning and when you decide to go on server was logging you in for a very short time then the connection crashed. We were able to login via Hyper-V console but RDP was no-go. Nothing was working while the fact you were correctly logging in was proved by the fact that another session started from Hyper-V console was being forced out when you were attempting to login by the same username. However, logging in never completed.

We were quite upset since that seemed to affect and increasingly number of VMs until we discovered a shocking thing: the issue was caused by the removal of Users group access to C:\ folder. 😟😟

That’s a change we usually perform on Web Servers and actually Web Servers were the most affected kind of servers. Not sure which is the connection between Users group access to C:\ folder but if we re-enable the access for that group, RDP starts working again.

Just crazy! One the strangest thing that happened to us. We used to remove Users access to C:\ for ages and that showed no issues on any Windows Server version since 2003 or so. Hope this might help anyone else!

(credit where is due)

You might find youserlf in a situation where you need to clear Powershell command history. For example, for us it was needed because we needed to prepare templates for virtual machines deployment and we needed to activate a specific software before finalizing the template.

Unfortunately you need to type a command to enter the code and activate the software and those templates could be used by our customers that might be able to retrieve the correct code by simply striking arrow-up. If you look for a solution you will find that Powershell has a specific command to do that:

Clear-History

However, you might find that this won’t solve your problem. Some history gets cancelled and some don’t or sometimes nothing at all gets removed and all of your history is still available to anyone logging using that account.

Turned out that there’s a way to check actually where Powershell stores your history. Just issue this command:

PS C:\Users\Administrator> (Get-PSReadlineOption).HistorySavePath C:\Users\Administrator\AppData\Roaming\Microsoft\Windows\PowerShell\PSReadLine\ConsoleHost_history.txt

The result is the location of the history file. Just edit this file and remove those lines you don’t want other users to access.

Neat trick but actually we’re not sure why the dedicated command is not working as expected.

Microsoft released Windows Server 2022 a few weeks ago. It did so earlier than expected, which is pretty odd, but it seems that the development of new features was already complete and mature so there was no point to wait a few months more. The last sentence pretty much explains the current status of Windows Server: so few changes, albeit interesting ones, that the product can be released months earlier than planned and without much fanfare.

While generally speaking the development of new features and functionalities for server applications mostly moved out of the base operating system and while we can also concede that Windows Server has many features at the OS level so it is difficult to design new ones, the fact each new release of Windows Server keeps bringing few new features is definitely because of Microsoft errors. Server applications today are either simple, think about a website or a Web application, or complicated. In the latter case, you generally think about microservices, Kubernetes and so on.

So we might say that since a few years Windows Server became more a tool to build base infrastructures than the base for new applications. That means that you usually install something on top of Windows Server to manage and deploy applications and you don’t usually rely on OS services only to develop your application. It is safe to say that it is a common scenario because Linux too is experiencing the same transformation.

However, Microsoft could have responded to Linux popularity by upping the ante and push more innovation into Windows Server and they didn’t.

Windows Server 2022 – continued fragmentation

The new Windows Server version has some nice and neat features. Besides a better Kubernetes compatibility, which is basically mandatory today, it features and big work to improve security. Credit where is due. That improves a lot both standalone servers security but it is also a fundamental improvement when deploying clusters, the famous infrastructures.

Other improvements include TLS 1.3 enabled by default and a few improvements around SMB, all of them improve security.

There would be other two important features to mention, one of them would be very important, but surprise: they won’t be available to all customers but only to those using Azure. At least for now.

Microsoft introduced Windows Server Datacenter: Azure Edition, another attempt to fragment Windows Server but the first one to draw a line between “standard” Windows Server and Azure versions. And we cannot welcome this kind of behavior because it basically starts to leave non-Azure Windows Server installations behind in an attempt to force Windows customers to move to Azure. Quite frankly, this is indecent especially when you factor how much Windows Server costs when compared to its competitor.

Features that Microsoft is withholding from its non-Azure customers are very important.

First, there is the much awaited hot-patching, that is a way to patch Windows without rebooting. Microsoft aims to use hot-patching especially for security patches and it would be a very ground-breaking innovation when considering how many emergency updates are needed [...]

We had previously identified what happened to those two boxes but that explanation is out of scope. This post is about how you can quickly recover form that situation.

A few days ago we had a weird issue on a couple of servers after upgrading Acronis agent. Console was unable to start any backup plan, either automatically or manually, complaining that the agent was not registered anymore. That looked weird since we had updated our agents many times in the past but this time, somehow, agents unregistered or were unable to re-register.

Actually, this round of updates had caused issues on a couple of Linux servers too but those had been quickly fixed by rebooting those machines. We quickly noticed that servers that were failing both had Windows Server Core as their base OS and SQL Server was installed on them too and in the end we decided to access those machines and manually re-register those agents as detailed inside Acronis documentation. It was surprising to find that agent was unable to re-register, complaining that the endpoint certificate was invalid or, more specifically, the Certification Authority that signed the certificate was unknown.

Thing is that both those server were working fine before the update plus we seemed to be unable to re-register those agents even when using Acronis own endpoint. The same error was reported, which was so weird seems it could well be (though unlikely) that our own customized certificate had something wrong – even if it was working fine when accessed through browsers or on other machines – but Acronis own certificate could not have had any issue.

Updating root certificates

The very single fact that agent was reporting an unknown certificate authority was pretty indicative but we also had troubles connecting to the endpoint on our own, for example when using Powershell

Invoke-WebRequest -UseBasicParsing -Uri https://cloud.acronis.com

Definitely there was a problem connecting to SSL endpoints. While PowerShell was only complaining that the underlying connection was closed, Acronis agent reported a much more specific issue so we decided to investigate and check if we had any problems with our root certificates. We decided to update them altogether, just in case:

md C:\Temp cd C:\Temp # Download ALL root certificates from Windows Update certutil.exe -generateSSTFromWU roots.sst # Import ALL root certificates back inside root authorities $sstContainer = (Get-ChildItem -Path C:\temp\roots.sst) $sstContainer | Import-Certificate -CertStoreLocation Cert:\LocalMachine\Root

Right after you do this, you need to reboot the affected machine to let it reload all root certificates.

Encryption issue

That didn’t fix the problem per se. Once the first machine was back, any attempt to connect to our endpoint or Acronis own one using Powershell was still failing. So we decided to investigate to check if machine was not using the proper cypher and we decided to force TLS1.2 in Powershell by using:

# Force PowerShell to use TLS1.2 [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Then you have to remember not to close that session or your will need to issue that command again. So now we attempted to connect to our endpoint or Acronis endpoint again and… bingo! It was working fine!

PS [...]