How the Restart-Netadapter command fixed an inaccessible Windows Cluster

Introduction.

Recently a Windows Server OS upgrade of a two node cluster caused some head scratching issues.  This cluster consisted of two physical servers, each with ILO connectivity.  The cluster had on it several Hyper-V virtual servers - thus this upgrade was indeed a high impact change.  Working on physical servers remotely does require a steady hand.  Obviously with a VM server we can revert back to a snapshot quickly if something goes wrong - provided you have remembered to take a snapshot beforehand.

I did upgrade one node to Server 2022 and it appeared to go well.  But things started falling apart when upgrading the second node.

Why did the Cluster Collapse?

I had moved all the Hyper-V machines onto the upgraded server, in readiness for upgrading the now Hyper-V VM empty Windows Server.  The upgrade was from 2016 to 2022.  The upgrade of the second Windows 2016 server to Server 2022 went well and it was believed all stable and OK.  But all was not well.  The problem was that the first server to be upgraded was now no longer accessible.  It could not be pinged nor could you RDP onto it.  

In addition the cluster was unmanageable using the Failover Cluster Manager tool.  An error message indicated one of the nodes was not contactable.

So what had happened?  

Thankfully I was able to access the offending server using the ILO portal and was able to sign into Windows using a local administrator account.  I performed an iponfig command and realized that it now had an APIPA address.

An APIPA address is a self-assigned IPv4 address in the range 169.254.0.1 to 169.254.255.254 with a subnet mask of 255.255.0.0, used when a device cannot obtain an address from a DHCP server.

This was most unusual because the NIC was configured with a static address and not a DHCP address.  

I cannot be sure how this did occur.  Nevertheless I was able to duplicate the issue in my lab by shutting the cluster node down.  I then assigned its IP address to another network device.  I then powered on the node - this replicated this issue using DAD (Duplicate Address Detection).

The Repair

I could not just restart the server node because it had live Hyper-V machines on it.  And I could not risk right clicking on the server's production nick and disabling it - as I was not sure I would not lose connectivity altogether, and not be able to enable it again.  So I tried the Restart-NetAdapter PowerShell command.  In my lab the command was:

restart-netadapter -name "Ethernet"

Initially this command failed because I had not started the cmd console with admin permissions.  When I did start with admin rights, the command executed successfully and my node now had its proper IP Address.

This then allowed me to connect to the cluster using the Failover Cluster Manager Tool.

Conclusion

We know how procedures should work - but oftentimes they do not work as expected.  In live environments and under strict change control timeframes it is important to keep cool.  Examine the logs, protect whatever connectivity you do have and resolve the issue - try not to panic.  I hope you have enjoyed reading about my cluster upgrade issues - and I wish your own troubleshooting endeavors as much success.