Resolving client to MP communication issue - using IIS Crypto 3.2

Introduction

Recently I performed an upgrade of an SCCM site from 2103 to 2107 and the organization also upgraded the site systems' operating systems from Windows Server 2012 to Windows Server 2019.  And this went well but with one issue.  A number of servers did not upgrade the client agent - approximately 25% of all servers were in this state.

I advised the operations engineer to manually uninstall the old client and reinstall the new client.  This failed using the client push installation method.  It did work by copying the client source files locally, running ccmsetup.exe /uninstall and then running ccmsetup.exe smssitecode=xxx from the source files directory.

But although the new client installed it would not communicate with either of the two management points.

The client CcmMessaging.log file showed a number of errors such as

Post to https://xxxx.xxx.xxx/ccm_system/request failed with 0x87d00231. CcmMessaging 28/10/2021 10:01:51 17280 (0x4380)

and

Failed in WinHttpSendRequest API, ErrorCode = 0x2efe

and

Post to https://xxx.xxx.xxx/ccm_system_windowsauth/request failed with 0x87d00231.

In the SCCM console these devices had a strange ? icon next to their entry in the Devices node.

The Things I Tried

I won't turn this post into a final year dissertation by detailing everything I tried to get the SCCM client communicating with the management points.  But in summary:

1) Attempted every suggestion in every related google article returned from a search of any of these error return codes.

2) Requested the networks team to investigate the possibility of any network related issues that might be blocking the flow of traffic.

3) Re-installed the Management Points.

4) Re-installed IIS and the Management Points.

5) Ensured .net 4.8 was installed on the Management Points.

6) Reinstalled the certificates - client and Management Points.

7) Repaired the WMI repository on a failing client.

8) Checked the DCOM configuration on a failing client.

9) Tried a TCP/IP reset on a failing client.

10) Disabled Microsoft Endpoint Protection anti-virus scanning.

11) Analysed group policy settings for differences between a communicating and non communicating client.

12) Ensured BITS service was running.

13) Checked permissions on the IIS virtual directories

14) Ensured webdav was enabled on the Management Point IIS configuration.

15) Scanned the IIS logs for clues - the failing client's IP addresses did not appear in these logs.

16) And many many more things.

Nothing would make these server SCCM clients register or communicate with the Management Points and therefore they were, for all real purposes, orphaned and unmanaged.

It was a very difficult problem and I did get some inspiration from Churchill's famous quote - Never give in. Never give in. Never, never, never

The IIS Crypto 3.2 Tool Breakthrough

One change that had taken place in this organisation, previously and before I began providing  this organisation with my services, was the disabling of the TLS 1.1 and 1.0 protocols - and enabling the TLS 1.2 protocol.

I did check the SCHANNEL key under: HKLM\System\CurrentControlSet\Control\SecurityProviders\SCHANNEL

I found no differences between a working SCCM client and a failing SCCM client.

Finally I downloaded the IIS Crypto 3.2 tool from Nartac.

https://www.nartac.com/Products/IISCrypto/Download

I ran this tool on a working server and a failing server and it showed no differences on the Schannel blade.

I then clicked on the Cipher Suites blade and I did notice that a failing server had a number of Cipher Suites not selected - Cipher Suites that were selected on the working servers.

Namely, the following Cipher Suites were not enabled on the failing servers.

And so I did enable these cipher suites, clicked on apply and rebooted the server.  And after the reboot everything started working, and the client registered itself correctly in the SCCM database and was now fully manageable from within the SCCM console.

The Road to Microsoft Endpoint Manager with co-management

The road to MEMCM co-management 

In the world of systems management, Microsoft’s Endpoint Manager (Configuration Manager) solution, is and has been one of the most powerful and flexible solutions for many years.  However, in the modern day where IT administrators need to manage devices that may only be internet connected, without a VPN connection; relying on MEMCM alone will not necessarily be a viable solution.  MEMCM can be configured to manage internet only connected devices, but to do so does rely on a complex set of requirements, design considerations and corresponding configuration decisions.  MEMCM internet-based management of devices and users is referred to as IBCM.  In the past it has been called SCCM IBCM.  I will refer to this solution as MEMCM IBCM.

Alternatively, MEMCM can be configured to use a Cloud Management Gateway (CMG) approach which is much simpler to setup but more expensive to maintain.  This approach will be referred to as MEMCM CMG.  Previously it may have been referred to as SCCM CMG

 

MEMCM IBCM 

In summary MEMCM IBCM implementations follow one of three designs. 

1. A MEMCM internet offering server is placed in the perimeter network. That network also has a read-only domain controller to authenticate users. A firewall between the perimeter and internal networks allows Active Directory packets.
2. A MEMCM internet offering server is placed in a perimeter-based forest. The perimeter forest trusts the internal forest. A firewall between the perimeter and internal networks allows the trafficking of authentication packets. 
3. A MEMCM internet offering server is in the intranet-based forest (Data Centre). The MEMCM internet offering system is published to the internet with a web proxy server. 

• 
  

A MEMCM IBCM solution to internet connected devices without a VPN can be complex but is possible.  Due to this complexity however, along with associated security concerns of exposing internal systems publicly, MEMCM introduced the CMG (Cloud Management Gateway) method of managing internet facing MEMCM devices. 

MEMCM CMG 

The cloud management gateway (CMG) provides a simple way to manage Configuration Manager client agents over the internet. The CMG is a cloud service in Microsoft Azure that does not require additional on-premise infrastructure.  Devices that are internet connected and without a VPN connection can retrieve MEMCM policies and because of this can install deployed applications or have their devices configured according to deployed policies. 

This solution relies on an on-premise MEMCM site system being configured with a CMG connection point.  The CMG connection point, as the name suggests, connects the on-premise MEMCM site to the CMG in the cloud.  The Service connection point, responsible for MEMCM release updates and fixes, also communicates with the CMG in the cloud. 

 

 

If the MEMCM CMG approach is used then, at the time of writing, the best pathway is to ensure that the on-premise MEMCM site is upgraded to 2107. In version 2107 you deploy the CMG with a virtual machine scale set. 

An Azure virtual machine scale set manages the load for a group of VMs. VM instances can automatically increase or decrease in number in response to demand. Thus, with scale sets there is high availability and simplicity in managing a multitude of VMs. 

When you create the CMG in the MEMCM console, the default option is to allow the CMG to function as a cloud distribution point.  And so, we have here a very simple way of providing content and policy to MEMCM internet only roaming clients.  There are of course, additional Azure costs incurred when this solution is implemented, even if it does not incur the costs of setting up additional on-premise infrastructure. 

MEMCM – Tenant Attach 

Many organisations do not want to allow their devices to be without a connection to the corporate network.  These machines, they would argue, are in a rogue environment and MEMCM has no mandate to provide managing capability to such machines.  But as for machines that are connected to the corporate network, the IT administrator should be able to manage these machines from a web portal from anywhere.  

Even from their mobile phone if necessary.  Thus, here arrives the concept of Tenant Attach.  The MEMCM devices are uploaded to the cloud and typical MEMCM console actions such as forcing a machine policy refresh, or viewing the devices details using resource explorer, can be done from the Microsoft Endpoint Manager Admin Center portal.  The clients themselves however, must be connected to the corporate network, either by being located at a corporate office, or connected via a VPN. 

MEMCM – Co-Management 

The underlying cloud management solution that makes Tenant Attach possible is Intune. 

Intune is a complete Modern Device Management (MDM) solution in itself, and can be used even to manage devices that are not part of a corporate Active Directory. 

There is no reason therefore, why Tenant Attach cannot be extended to provide management of clients that belong to a corporate Active Directory and are part of a MEMCM infrastructure - but are not connected to a VPN or directly to Active Directory. 

 

Further, Intune is quite good at managing Windows 10 devices whether connected to the corporate network, or not connected.  So why not provide the ability to switch the responsibility of a particular MEMCM workload, such as software updates, from MEMCM to Intune? 

When MEMCM is configured in this way, so that the responsibly for certain management capabilities is handed over to Intune – then we have a MEMCM site in a co-managed configuration.