4: Troubleshooting

This lesson explains | introduces | provides a detailed …

What You Will Learn

After completing this lesson, you will be able to:

·         Describe the basic WinRM dependencies required for operation

·         Troubleshoot communication problems

·         Gather required information for in depth analysis

When WinRM is not functioning correctly the problem often breaks down into one of two areas: Communications and Permissions. Communications refers to any point between two systems including the systems themselves. Firewalls are a common example. Permission issues are identified as performing actions under credentials that do not have adequate rights, or the systems are not in an environment that matches the authentication method being used. Kerberos authentication, for example, will not succeed if there is no domain trust as communication mechanisms are not in place to verify permissions.

WinRM operations usually involve two systems. If either of these systems block or interrogate communications WinRM could be affected. Local system refers to a single system, but for the purpose of WinRM communications, it instead implies both local systems as both must pass the following tests.

DNS Resolution

Very commonly a customer's environment will have incorrect or missing DNS entries. Mulitple DNS suffixes can cause issues as well. Verify DNS resolution from System A to System B, and vice versa. Ping the NETBIOS name of the remote system and note both the IP address and FQDN returned. Ping the FQDN exactly as returned by the NETBIOS ping. This should resolve and should match the IP address of the NETBIOS ping. In most cases this is sufficient testing.

Listeners

WinRM communicates through http.sys, so anything using the same ports will cause a problem. This was more of an issue prior to WinRM 2.0 as ports 80 and 443 are commonly used. Check that no other processes are configured to listen on the same port as the System process, PID 4.

Netstat identifies processes using a port specified. In this example we filter on port 80. Other recommended ports to check are 443, 5985 and 5986. The first column is the local system, the second the remote system connecting, the connection status followed by the process ID (PID). The first column is the only one of interest as it represents what ports our local system is listening on. All zero's (0.0.0.0) means 'all interfaces.'

C:\>netstat -ano | find "80"

  TCP    0.0.0.0:80             0.0.0.0:0              LISTENING       4

  TCP    0.0.0.0:49152          0.0.0.0:0              LISTENING       680

  TCP    0.0.0.0:49155          0.0.0.0:0              LISTENING       180

  TCP    65.53.5.178:56456      65.53.63.31:80         ESTABLISHED     3580

  TCP    65.53.5.178:56457      65.53.63.31:80         ESTABLISHED     3580

Certificates

WinRM communicates through http.sys, so anything that is bound to http.sys using the same port as WinRM will cause a problem.

Antivirus and Antimalware

Until the last few years, most protection software did not monitor traffic protocols but instead the files arriving and leaving the system. Most vendors now include a form of protocol scanning that can disrupt WinRM sessions. When protection software is found to be running on a system and is suspected:

·         Open 'Programs and Features' in the Control Panel to identify all components installed. Perform an internet search of the exact product and release. Key words such as 'endpoint protection' and 'internet security' are clear indications of protocol scanning.

·         Temporarily disable all protection services and processes on both systems during testing

·         Disabling protection software, all services and startup items as listed in 'MSConfig', does not prevent filter drivers from loading and possibly continuing to impact communications.

·         Ask that software be removed if no other communication problems seem probable

·         Update all protection software (not just definitions) as vendors do identify problems and release updates to address them

Disconnected Networks

Systems that have multiple network cards (NICs) can contribute to communications issues, even if intermittently.

·         Control Panel> Network and Sharing Center> Change Adapter Settings

·         Disable any devices that show status 'Disconnected'

Firewalls

Enabling WinRM through 'winrm qc','Set-WSManQuickConfig' or 'Enable-PSRemoting' should open all necessary ports in the Windows firewall. If in doubt, run the commands again at any time.

If the Windows Firewall is disabled preventing the WinRM enablement process from completing, perform these manual commands.

1.       sc config "WinRM" start= auto

2.       net start WinRM

3.       winrm create winrm/config/listener?Address=*+Transport=HTTP

4.       netsh firewall add portopening TCP 5985 "Windows Remote Management"

Third party firewalls should be entirely disabled until testing is performed that rules them out.

Routers and Switches

Routers, especially Point to Point devices, generally connect networks and have sophisticated management options. It is very common for a customer to believe all ports are open for all protocols when in fact they are not. Switches can similarly be problematic at times as layer three switches learn MAC addresses. Consult a network engineer for assistance if in doubt.

Generally, the user or computer account (Local System) performing a WinRM operation locally will be the one used for authentication and authorization on the remote system, unless credentials are explicitly provided.

Group Membership

Membership in the Local Administrators group on the remote system is usually sufficient to perform the actions requested by WinRM.

Group Policy

If permissions appear sufficient yet WinRM is still failing due to rights, run a resultant set of policy to determine if any specific policies are in place. A troubleshooting step is to move the systems into a new OU and block inheritance. Force group policy update or restart the systems following this move.

WinRM Group Policy Settings

·         Computer Configuration\Policies\Administrative Templates\Windows Components\Windows Remote Management (WinRM)

·         Computer Configuration\Policies\Administrative Templates\Windows Components\Windows Remote Shell

Despite the seemingly endless combination of WinRM settings and permissions that can be tested and verified, the first steps should be the simplest.

Without basic network communication and resolution, everything else is a waste of time. Over and again, SCVMM cases are resolved by testing what is overlooked and assumed healthy. When your car stalls on the highway you first check the fuel gauge, not call a mechanic. Make the same logical choice here.

Correct IP and DNS Resolution

Follow this test in the order presented. Once this has been performed from system A to B, do the same from system B to A. This verifies IP and DNS correctly.

1.       Ping the remote system NETBIOS name

2.       Ping the FQDN that is returned by the NETBIOS name ping

3.       Verify three items:

a.       NETBIOS ping returns an FQDN? If not, resolve DNS issues first.

Highlighted in Yellow.

b.       NETBIOS IP and FQDN IP addresses match.

Highlighted in Green.

c.       FQDN returned by NETBIOS ping matches 'ipconfig /all' output on remote system. Concatenate the 'Host Name' and 'Primary DNS Suffix' and compare.

Highlighted in Blue below.

4.       A reverse lookup of the IP address to verify that it resolves the correct system name is optional, but a good check.

Highlighted in Pink below.

C:\>ping cssvmmr2

Pinging cssvmmr2.northamerica.corp.microsoft.com [2001:4898:0:fff:200:5efe:65.53.5.62] with 32 bytes

 of data:

Reply from 2001:4898:0:fff:200:5efe:65.53.5.62: time=1ms

Reply from 2001:4898:0:fff:200:5efe:65.53.5.62: time<1ms

Reply from 2001:4898:0:fff:200:5efe:65.53.5.62: time<1ms

Reply from 2001:4898:0:fff:200:5efe:65.53.5.62: time<1ms

Ping statistics for 2001:4898:0:fff:200:5efe:65.53.5.62:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 1ms, Average = 0ms

C:\>ping cssvmmr2.northamerica.corp.microsoft.com

Pinging cssvmmr2.northamerica.corp.microsoft.com [2001:4898:0:fff:200:5efe:65.53.5.62] with 32 bytes

 of data:

Reply from 2001:4898:0:fff:200:5efe:65.53.5.62: time=2ms

Reply from 2001:4898:0:fff:200:5efe:65.53.5.62: time<1ms

Reply from 2001:4898:0:fff:200:5efe:65.53.5.62: time<1ms

Reply from 2001:4898:0:fff:200:5efe:65.53.5.62: time<1ms

Ping statistics for 2001:4898:0:fff:200:5efe:65.53.5.62:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 2ms, Average = 0ms

C:\>ping -a 65.53.5.62

Pinging CSSVMMR2 [65.53.5.62] with 32 bytes of data:

Reply from 65.53.5.62: bytes=32 time=2ms TTL=128

Reply from 65.53.5.62: bytes=32 time<1ms TTL=128

Reply from 65.53.5.62: bytes=32 time<1ms TTL=128

Reply from 65.53.5.62: bytes=32 time<1ms TTL=128

Ping statistics for 65.53.5.62:

    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

    Minimum = 0ms, Maximum = 2ms, Average = 0ms

Remote system 'ipconfig /all'

C:\>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : CSSVMMR2

   Primary Dns Suffix  . . . . . . . : northamerica.corp.microsoft.com

   Node Type . . . . . . . . . . . . : Hybrid

   IP Routing Enabled. . . . . . . . : No

   WINS Proxy Enabled. . . . . . . . : No

   DNS Suffix Search List. . . . . . : northamerica.corp.microsoft.com

                                       corp.microsoft.com

With the items above verified as correct, it's time to move on to the WinRM Ping Test.

Using the same idea for testing IP communications, we verify WinRM functionality. The most important point to keep in mind during this test is that it is two-way. If communications are not tested in both directions, the test is not conclusive. WinRM follows a client-server model and so each 'ping' direction tests a different component.

Ping Test

This simple test confirms WinRM communications in the same way an IP 'ping' command does.

1.       From System A, ping System B. If this succeeds as below, one direction of communication has been verified.

C:\>winrm id -r:SystemB

IdentifyResponse

    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd

    ProductVendor = Microsoft Corporation

    ProductVersion = OS: 6.1.7600 SP: 0.0 Stack: 2.0

2.       From System B, ping System A. If this succeeds as below, both directions of communication have been verified.

Resolving a Failed Ping Test - Local Validation

The purpose of this exercise is to pinpoint the location of the problem. Possibilities are either of the communicating systems, or the communication path between them. This section verifies local settings.

Is the Local WinRM Service Responding?

Perform this test locally on both systems. This verifies local functionality.

1.       Verify basic loopback connectivity to WinRM service

2.       Output below indicates all is well. If there were a problem, perform 'winrm qc' then try again.

C:\>winrm id

IdentifyResponse

    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd

    ProductVendor = Microsoft Corporation

    ProductVersion = OS: 6.1.7600 SP: 0.0 Stack: 2.0

Are Local Listeners, Service, Client, and WinRS configured correctly?

Enumerate listeners and compare to default settings in the 'Working with WinRM' module.

winrm e winrm/config/listener

1.       If there are no listeners, restore settings then perform 'winrm qc' again. Listeners should then be listed.

C:\>winrm invoke Restore winrm/Config @{}

Restore_OUTPUT

C:\>winrm qc

2.       To enable the "compatibility" HTTP listener on port 80 for communicating with legacy WinRM 1.1 clients, perform:

winrm set winrm/config/service @{EnableCompatibilityHttpListener="true"}

3.       To enable the "compatibility" HTTPS listener on port 443: (this step is not required in most cases)

winrm set winrm/config/service @{EnableCompatibilityHttpsListener="true"}

Get configuration for Service, Client and WinRS. Compare to default settings in the 'Working with WinRM' module.

winrm g winrm/config

Resolving a Failed Ping Test - Remote Validation

The purpose of this exercise is to pinpoint the location of the problem. Possibilities are either of the communicating systems, or the communication path between them. This section verifies communication in two directions. This build on the tests already performed in the previous step, 'Resolving a Failed Ping Test - Local Validation.'

On System A, can WinRM information be pulled from System B?

C:\>winrm id -r:SystemB

IdentifyResponse

    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd

    ProductVendor = Microsoft Corporation

    ProductVersion = OS: 6.1.7600 SP: 0.0 Stack: 2.0

Output shows it can. If this failed, there will be an error along with a hex return code. Read the error, keeping it in mind as these possibilities are reviewed:

·         NETBIOS or DNS resolution failure.

·         Routing or network problems between systems.

·         Credentials used on local system are not authorized on the remote system.

·         Firewall settings on the remote system, B.

·         A Firewall on the network blocking communication by port or protocol (WinRM).

·         Listeners on remote system not configured (this should have been resolved in the previous section)

On System B, can WinRM information be pulled from System A?

C:\>winrm id -r:SystemA

IdentifyResponse

    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd

    ProductVendor = Microsoft Corporation

    ProductVersion = OS: 6.1.7600 SP: 0.0 Stack: 2.0

There are two possibilities. This succeeds or fails. This tells us a great bit.

Success - If this succeeded…

·         It is highly likely there is not a problem on the network affecting communications between systems.

·         It is true that the credentials used on System B to communicate with System A are valid and are authorized to connect.

o   Use these same credentials now when pulling data from System B on System A.

o   Also, make sure these credentials are in the local Administrators group on System B.

Failure - If this failed…

·         There may be problems on the network affecting communication.

·         Credentials used may not have permissions needed. Make sure the domain account logged in with is in the Administrators group on both systems

·         Verify that there are no GPOs enabled that configure WinRM

WinRM communication has been established at this point, but as most WinRM work is WMI related it a good idea to perform one further test.

System A: Can WinRM communicate with WMI locally?

C:\>winrm enum wmi/root/cimv2/Win32_ComputerSystem

Win32_ComputerSystem

    AdminPasswordStatus = 3

    AutomaticManagedPagefile = false

    AutomaticResetBootOption = true

    AutomaticResetCapability = true

    ...

Yes. If not, first step would be to click Start and begin typing 'wbemtest.' When this shows up in the Start Menu right click it and run as Administrator.

·         Click 'Connect…'

·         Click 'Open Class…'

·         Type 'win32_computersystem'

·         If class opens, success.

System A: Can WinRM communicate with WMI on System B?

C:\>winrm enum wmi/root/cimv2/Win32_ComputerSystem -r:SystemB

Win32_ComputerSystem

    AdminPasswordStatus = 3

    AutomaticManagedPagefile = false

    AutomaticResetBootOption = true

    AutomaticResetCapability = true

    ...

Yes. Now perform the same tests from System B to System A.

If communication failed:

·         Perform the same local winrm enumeration on System B

·         If this fails test WMI directly on System B with 'wbemtest'

Finally, the SCVMM Agent

Using the 'Ping Test' remote WinRM and WMI communication was verified. As WinRM is used to communicate with the VMMAgent running on Hosts it is necessary to verify that this agent responds remotely.

Verify remote VMMAgent response.

winrm invoke GetVersion wmi/root/scvmm/AgentManagement -r:SystemB @{}

GetVersion_OUTPUT

    Version = 2.0.4275.0

    IsAmd64 = true

    ErrorInfo = null

    ReturnValue = 0

Success!

Using the same methods as previously mentioned, the following namespaces are valuable tests to determine what WinRM can and cannot connect to remotely. If none can be connected to, there is likely a general WMI problem. If only a few fails, target those namespaces for WMI troubleshooting.

Perform all from an elevated command prompt. The commands below pull information from a remote system. To perform the same tests locally simply remove:

-r:<remoteserver>

Windows System

winrm wmi/root/cimv2/Win32_ComputerSystem -r:<remoteserver>

winrm wmi/root/cimv2/win32_OperatingSystem -r:<remoteserver>

winrm wmi/root/cimv2/Win32_PerfRawData_PerfOS_Memory -r:<remoteserver>

Failover Cluster

winrm wmi/root/mscluster/MSCluster_ClusterToNode -r:<remoteserver>

winrm wmi/root/mscluster/MSCluster_Cluster -r:<remoteserver>

winrm wmi/root/mscluster/MSCluster_ResourceGroup -r:<remoteserver>

winrm wmi/root/mscluster/MSCluster_ResourceGroupToResource -r:<remoteserver>

winrm wmi/root/mscluster/MSCluster_Service -r:<remoteserver>

winrm wmi/root/mscluster/MSCluster_ResourceToDisk -r:<remoteserver>

SCVMM Agent

winrm invoke GetVersion wmi/root/scvmm/AgentManagement -r:<remoteserver> @{}

Test Script

The commands above can be performed together using the script below. Execute from an elevated command prompt. If no system name is added following the name of the batch file you create, the local system will be tested. If a system name is added following the batch file name it will be treated as a remote system.

Example

Save the commands below to WinRM_WMI.cmd

At a command prompt type:

·         ' WinRM_WMI.cmd' and the local system will be tested.

·         'WinRM_WMI.cmd SystemA' and SystemA will be tested.

setlocal

If .%1 EQU . (set hostname=%computername%)&(set name=)&(Goto :Start)

Set hostname=%1

set name=-r:%1

:Start

echo . >%hostname%.txt

echo . >%hostname%.Error.txt

:Ping

echo ping %hostname% >>%hostname%.txt

echo ping %hostname% >>%hostname%.Error.txt

echo +++++++++++++++++++++++++++++++++++++++++++++  >>%hostname%.txt

echo +++++++++++++++++++++++++++++++++++++++++++++  >>%hostname%.Error.txt

ping %hostname% >>%hostname%.txt 2>>%hostname%.Error.txt

echo ================================================================= >>%hostname%.txt

echo ================================================================= >>%hostname%.Error.txt

:VMMAgent

echo winrm invoke GetVersion wmi/root/scvmm/AgentManagement %name% @{}  >>%hostname%.txt

echo winrm invoke GetVersion wmi/root/scvmm/AgentManagement %name% @{}  >>%hostname%.Error.txt

echo +++++++++++++++++++++++++++++++++++++++++++++  >>%hostname%.txt

echo +++++++++++++++++++++++++++++++++++++++++++++  >>%hostname%.Error.txt

call winrm invoke GetVersion wmi/root/scvmm/AgentManagement %name% @{} >>%hostname%.txt 2>>%hostname%.Error.txt

echo ================================================================= >>%hostname%.txt

echo ================================================================= >>%hostname%.Error.txt

:Cluster

:: SCVMM cluster information pulled

Call :WinRM wmi/root/mscluster/MSCluster_ClusterToNode

Call :WinRM wmi/root/mscluster/MSCluster_Cluster

Call :WinRM wmi/root/mscluster/MSCluster_ResourceGroup

Call :WinRM wmi/root/mscluster/MSCluster_ResourceGroupToResource

Call :WinRM wmi/root/mscluster/MSCluster_Service

:: During host refresh, for example

Call :WinRM wmi/root/mscluster/MSCluster_ResourceToDisk

:System

:: Initial system scan during P2V, for example

Call :WinRM wmi/root/cimv2/Win32_ComputerSystem

Call :WinRM wmi/root/cimv2/win32_OperatingSystem

:: Collects avail memory during host refresh, for example

Call :WinRM wmi/root/cimv2/Win32_PerfRawData_PerfOS_Memory

:End

exit /b

:: ----------

:WinRM

echo winrm enum %1 %name% >>%hostname%.txt

echo winrm enum %1 %name%>>%hostname%.Error.txt

echo +++++++++++++++++++++++++++++++++++++++++++++  >>%hostname%.txt

echo +++++++++++++++++++++++++++++++++++++++++++++  >>%hostname%.Error.txt

Call winrm enum %1 %name% >>%hostname%.txt 2>>%hostname%.Error.txt

echo ================================================================= >>%hostname%.txt

echo ================================================================= >>%hostname%.Error.txt

echo.  >>%hostname%.txt

echo.  >>%hostname%.Error.txt

Goto :eof

Depending on how WinRM is being used, there should be either output errors or event log entries to review. WinRM errors are typically specific and should point in the direction of the problem.

Console

If using a command prompt or PowerShell the error will output to the console.

C:\>winrm id -r:cssvmm-n1

WSManFault

    Message = The WinRM client cannot complete the operation within the time specified. Check if the

 machine name is valid and is reachable over the network and firewall exception for Windows Remote M

anagement service is enabled.

Error number:  -2144108250 0x80338126

The WinRM client cannot complete the operation within the time specified. Check if the machine name

is valid and is reachable over the network and firewall exception for Windows Remote Management serv

ice is enabled.

In the event that the remote system is using Compatibility Listeners only as it is running WinRM 1.1, specify port 80 with the 'http' designation.

C:\>winrm id -r:http://cssvmm-n1

This returned the same error, but could be relevant if the remote system is a Windows Vista or prior release.

The error above provides a great deal of information. Of most importance is the hex code returned. Although in the example above this return code is translated, it is not always. To figure out what a hex error means use the WinRM help function.

C:\>winrm helpmsg 0x80338126

The WinRM client cannot complete the operation within the time specified. Check if the machine name is valid and is reachable over the network and firewall exception for Windows Remote Management service is enabled.

In the above example we cannot communicate with 'CSSVMM-N1'. Log into the remote system and starting with the basics determine what the problem is.

C:\>winrm id

IdentifyResponse

    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd

    ProductVendor = Microsoft Corporation

    ProductVersion = OS: 6.1.7600 SP: 0.0 Stack: 2.0

The WinRM service appears to be running. Check the rest of the settings now.

C:\>winrm get winrm/config

Config

    MaxEnvelopeSizekb = 150

    MaxTimeoutms = 60000

    MaxBatchItems = 32000

    MaxProviderRequests = 4294967295

    Client

        NetworkDelayms = 5000

        URLPrefix = wsman

        AllowUnencrypted = false

        Auth

            Basic = true

            Digest = true

            Kerberos = true

            Negotiate = true

            Certificate = true

            CredSSP = false

        DefaultPorts

            HTTP = 5985

            HTTPS = 5986

        TrustedHosts

    Service

        RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)S:P(AU;FA;GA;;;WD)(AU;SA;GWGX;;;WD)

        MaxConcurrentOperations = 4294967295

        MaxConcurrentOperationsPerUser = 15

        EnumerationTimeoutms = 60000

        MaxConnections = 25

        MaxPacketRetrievalTimeSeconds = 120

        AllowUnencrypted = false

        Auth

            Basic = false

            Kerberos = true

            Negotiate = true

            Certificate = false

            CredSSP = false

            CbtHardeningLevel = Relaxed

        DefaultPorts

            HTTP = 5985

            HTTPS = 5986

        IPv4Filter = *

        IPv6Filter = *

        EnableCompatibilityHttpListener = false

        EnableCompatibilityHttpsListener = false

        CertificateThumbprint

    Winrs

        AllowRemoteShellAccess = true

        IdleTimeout = 180000

        MaxConcurrentUsers = 5

        MaxShellRunTime = 2147483647

        MaxProcessesPerShell = 15

        MaxMemoryPerShellMB = 150

        MaxShellsPerUser = 5

C:\>winrm e winrm/config/listener

C:\>

Two items are relevant. First, if this system did require Compatibility Listeners, it no longer has them. The second command shows that there are in fact no listeners at all. WinRM cannot communicate this way. We easily correct this.

C:\>winrm set winrm/config/service @{EnableCompatibilityHttpListener="true"}

Service

    RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)S:P(AU;FA;GA;;;WD)(AU;SA;GWGX;;;WD)

    MaxConcurrentOperations = 4294967295

    MaxConcurrentOperationsPerUser = 15

    EnumerationTimeoutms = 60000

    MaxConnections = 25

    MaxPacketRetrievalTimeSeconds = 120

    AllowUnencrypted = false

    Auth

        Basic = false

        Kerberos = true

        Negotiate = true

        Certificate = false

        CredSSP = false

        CbtHardeningLevel = Relaxed

    DefaultPorts

        HTTP = 5985

        HTTPS = 5986

    IPv4Filter = *

    IPv6Filter = *

    EnableCompatibilityHttpListener = true

    EnableCompatibilityHttpsListener = false

    CertificateThumbprint

C:\>winrm qc

WinRM already is set up to receive requests on this machine.

WinRM is not set up to allow remote access to this machine for management.

The following changes must be made:

Create a WinRM listener on HTTP://* to accept WS-Man requests to any IP on this machine.

Make these changes [y/n]? y

WinRM has been updated for remote management.

Created a WinRM listener on HTTP://* to accept WS-Man requests to any IP on this machine.

Both remote connections now work again.

C:\>winrm id -r:cssvmm-n1

IdentifyResponse

    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd

    ProductVendor = Microsoft Corporation

    ProductVersion = OS: 6.1.7600 SP: 0.0 Stack: 2.0

C:\>winrm id -r:http://cssvmm-n1

IdentifyResponse

    ProtocolVersion = http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd

    ProductVendor = Microsoft Corporation

    ProductVersion = OS: 6.1.7600 SP: 0.0 Stack: 2.0

Event Logs

Most WinRM related events appear in the System or Application event logs. 'Service Control Manager' often contains the error as the WinRM service has terminated or restarted for some reason. These are often well known and have either been corrected by an update or hotfix, or documented in the KB. A few examples below.

Event Type: Error

Event Source: Service Control Manager

Event Category: None

Event ID: 7023

Date: 10/17/2010

Time: 6:45:56 PM

User: N/A

Computer: SystemB

Description:

The Windows Remote Management (WS-Management) service terminated with the following error:

Not all privileges referenced are assigned to the caller.

WinRM runs under the NETWORK SERVICE account, so in this case verify that all privileges the account should have are in place.

WinRM Event Log

WinRM has an event log under 'Event Viewer', though it is a bit hidden.

·         Open Event Viewer

·         Expand Windows Logs

·         Expand Applications and Services Logs

·         Expand Microsoft

·         Expand Windows

·         Expand Windows Remote Management

·         View the 'Operational' log

This event was thrown on the local system attempting to contact the remote system in the previous section.

Log Name:      Microsoft-Windows-WinRM/Operational

Source:        Microsoft-Windows-WinRM

Date:          1/1/2011 7:31:43 PM

Event ID:      168

Task Category: User authentication

Level:         Error

Keywords:      Security,Server

User:          NETWORK SERVICE

Computer:      CSSVMM-N1.Contoso.com

Description:

Sending HTTP 401 response to the client and disconnect the connection after sending the response

WinRM traces provide great detail of process activity. There many methods of collecting information. Common methods are examined here. Three types of WSMan provider logging are available in event logs: Operational, Analytic and Debug.

Operational

These are diagnostic messages, high level, and simple. The admin can see them using the EventViewer. To locate this event log:

·         Open Event Viewer in Server Manager

·         Open Applications and Services Logs

·         Open Microsoft

·         Scroll down to Windows Remote Management

Analytic and Debug

These are administrative trace messages. The ETW analytic channel is designed for events published in high volumes. These are used for logging administrative trace messages which will have to be later viewed using tracing tools/Event Viewer.

These event logs are located in the same location as 'Operational' events, but 'Show Analytic and Debug Logs' must be enabled from the 'View' menu.

Enable Analytic or Debug logging

Right click on the event log and select 'Enable Log.' Reproduce your issue, then right click again and select 'Disable Log.' Save all events to an .EVTX file then clear the log.

Analytic Logging Using Wevtutil

Use the following to show and enable the Analytic log by script. This can be opened in Event Viewer.

Begin logging.

wevtutil.exe sl Microsoft-Windows-Winrm/Analytic /e:true /q

Reproduce the issue. Then…

Disable logging.

wevtutil.exe sl Microsoft-Windows-Winrm/Analytic /e:false /q

Export the log.

wevtutil epl Microsoft-Windows-Winrm/Analytic C:\temp\winrm-analytic.evtx

Finally, clear the log.

wevtutil cl Microsoft-Windows-Winrm/Analytic

A file named 'winrm-analytic.evtx' will exist in the specified directory.

Traces generally gather data in a verbose manner and are viewable in text editors. Some traces must be converted from their native binary format to text prior to viewing.

WSManTrace

This is an extraordinarily simple method. This uses PowerShell and can be initiated on both machines to be traced prior to reproducing the issue. The resulting file will be unreadable, and must be converted.

Open an elevated PowerShell console and begin tracing.

Import-Module psdiagnostics

Enable-WSManTrace

Reproduce the issue. Then…

Disable tracing.

Disable-wsmantrace

Output will be available in %windir%\system32\wsmtraces.log.

Converting a Trace

In order to read the output collected in 'wsmtraces.log' above, perform these steps.

·         Go to http://logeater

·         Click 'Choose File' to upload the 'wsmtraces.log'

·         Click 'Submit'

·         Download the resulting 'wsmtraces_formatted.txt' or 'wsmtraces_formatted.csv' file depending on what you selected

This site is available internal to Microsoft only. There is no need to track down the correct TMF files to convert the trace as the entire process is automated for you.

Logman

Logman can also be used to collect an ETW trace. This must be converted prior to viewing.

Begin logging.

logman.exe start winrmtrace -p Microsoft-Windows-Winrm -o winrmtrace.etl -ets

Reproduce the issue. Then…

Disable logging.

logman.exe stop winrmtrace -ets

A file named 'winrmtrace.etl' will exist in the current console directory.

WinRM Traces with Performance Monitor

Perform step ‘A. Configuration’ steps on all systems where WinRM trace information will be recorded. For SCVMM, this normally means the SCVMM Server and the Host(s) involved. Do not proceed to step ‘B. Reproduce Issue’ until this section is complete on all systems. This trace should be viewable using 'tracerpt.exe' as covered later.

A: Configuration

1.       Open Perfmon.msc and navigate to Event Trace Sessions. In your case you will run this from a non-core box and connect to the core box first.

2.       Right click Event Trace Sessions and choose New Data Collector Set.

3.       Name it WinRM, select “Create manually” and click Next

4.       Click “Add” when prompted “Which event trace providers would you like to enable?”

5.       Select “Windows Remote Management Trace”.

6.       Select the ‘Keywords(Any)’ Property and choose Edit.

7.       Select all Values and click OK.

8.       Select the ‘Level’ Property and choose Edit.

9.       Select the Manual option, change the value to 0x05, and click OK.

10.   Click Next.

11.   Change the ‘Root Directory’ from ‘%LOCALAPPDATA%\’ to the location you would like to save the log. Then click Next.

12.   Leave ‘Save and close’ selected and click Finish

 B: Reproduce Issue

Perform the following steps on all systems you wish to trace WinRM. For SCVMM, that normally means the SCVMM Server and Host(s).

1.       Start the trace by right clicking on the new Data Collector, ‘WinRM’ and select Start (on all systems you configured). A ‘Running’ icon will appear on the WinRM data collector.

2.       Reproduce the issue now. Once the issue has been reproduced, continue to the next step.

3.       Stop the trace by right clicking the WinRM data collector and selecting Stop on both systems.

4.       You will now have a file named ‘winrm.etl’ saved to the location you specified in the configuration section. 

ETW File Conversion

To convert the ETL logs to various formats perform any of the following.

·         XML

tracerpt.exe winrmtrace.etl -of XML -o winrmtrace.xml

·         CSV

tracerpt.exe winrmtrace.etl -of CSV -o winrmtrace.csv

·         TXT

netsh trace convert winrmtrace.etl dump=TXT

Scripts to Convert WinRM .ETL Files

If you prefer to convert traces without LogEater, the scripts below can be used. They do not require access to TMFs beyond referencing them. Scripts provided courtesy of Tony Gaston.

Decode Windows 2008 R2 and Windows 7:

@IF "%_ECHO%" == "" ECHO OFF

REM

REM Batch file to decode ETL trace logs

REM execute with file name to decode, can be ran remotely from my sharepoint

REM Written by TonyGa

REM

IF {%1} == {} GOTO USAGE

IF {%BRIEF%} == {} GOTO VERBOSE

REM User requested brief output,

\\cpr-web\cprnt\tools\Tracing\Tools\%processor_architecture%\tracefmt.exe %1 -p \\cpr-web\cprnt\tools\Tracing\TMF\Windows2008R2-Win7 -o %1-formatted.log

GOTO END

:VERBOSE

@setlocal DISABLEDELAYEDEXPANSION

REM

REM Format string below translates to

REM [CPUNumber]ProcessID.ThreadID::SystemTime [MessageGUIDFriendlyName](ComponentName:FunctionName:Filename_LineNumber)

REM

SET TRACE_FORMAT_PREFIX= [%%9!d!] %%8!04X!.%%3!04X!::%%4!s! [%%1!s!](%%!COMPNAME!:%%!FUNC!:%%2!s!)

@setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION

\\cpr-web\cprnt\tools\Tracing\Tools\%processor_architecture%\tracefmt.exe %1 -p \\cpr-web\cprnt\tools\Tracing\TMF\Windows2008R2-Win7 -o %1-formatted.log

GOTO END

:USAGE

ECHO.

ECHO. Missing file name

ECHO.

ECHO A script to decode ETL trace logs

ECHO.

ECHO Usage: %~nx0 FileName_To_Decode

ECHO.

echo To switch to default output sent the environment variable below

ECHO SET BRIEF=1

GOTO END

:END

Decode Windows 2008 and Windows Vista:

@IF "%_ECHO%" == "" ECHO OFF

REM

REM Batch file to decode ETL trace logs

REM execute with file name to decode, can be ran remotely from my sharepoint

REM Written by TonyGa

REM

IF {%1} == {} GOTO USAGE

IF {%BRIEF%} == {} GOTO VERBOSE

REM User requested brief output,

\\cpr-web\cprnt\tools\Tracing\Tools\%processor_architecture%\tracefmt.exe %1 -p \\cpr-web\cprnt\tools\Tracing\TMF\Windows2008-Vista -o %1-formatted.log

GOTO END

:VERBOSE

@setlocal DISABLEDELAYEDEXPANSION

REM

REM Format string below translates to

REM [CPUNumber]ProcessID.ThreadID::SystemTime [MessageGUIDFriendlyName](ComponentName:FunctionName:Filename_LineNumber)

REM

SET TRACE_FORMAT_PREFIX= [%%9!d!] %%8!04X!.%%3!04X!::%%4!s! [%%1!s!](%%!COMPNAME!:%%!FUNC!:%%2!s!)

@setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION

\\cpr-web\cprnt\tools\Tracing\Tools\%processor_architecture%\tracefmt.exe %1 -p \\cpr-web\cprnt\tools\Tracing\TMF\Windows2008-Vista -o %1-formatted.log

GOTO END

:USAGE

ECHO.

ECHO. Missing file name

ECHO.

ECHO A script to decode ETL trace logs

ECHO.

ECHO Usage: %~nx0 FileName_To_Decode

ECHO.

echo To switch to default output sent the environment variable below

ECHO SET BRIEF=1

GOTO END

:END

Decode Windows 2008 and Windows Vista:

@IF "%_ECHO%" == "" ECHO OFF

REM

REM Batch file to decode ETL trace logs

REM execute with file name to decode, can be ran remotely from my sharepoint

REM Written by TonyGa

REM

IF {%1} == {} GOTO USAGE

IF {%BRIEF%} == {} GOTO VERBOSE

REM User requested brief output,

\\cpr-web\cprnt\tools\Tracing\Tools\%processor_architecture%\tracefmt.exe  %1 -p \\cpr-web\cprnt\tools\Tracing\TMF\Windows2003 -o %1-formatted.log

GOTO END

:VERBOSE

@setlocal DISABLEDELAYEDEXPANSION

REM

REM Format string below translates to

REM [CPUNumber]ProcessID.ThreadID::SystemTime [MessageGUIDFriendlyName](ComponentName:FunctionName:Filename_LineNumber)

REM

SET TRACE_FORMAT_PREFIX= [%%9!d!] %%8!04X!.%%3!04X!::%%4!s! [%%1!s!](%%!COMPNAME!:%%!FUNC!:%%2!s!)

@setlocal ENABLEEXTENSIONS ENABLEDELAYEDEXPANSION

\\cpr-web\cprnt\tools\Tracing\Tools\%processor_architecture%\tracefmt.exe  %1 -p \\cpr-web\cprnt\tools\Tracing\TMF\Windows2003 -o %1-formatted.log

GOTO END

:USAGE

ECHO.

ECHO. Missing file name

ECHO.

ECHO A script to decode ETL trace logs

ECHO.

ECHO Usage: %~nx0 FileName_To_Decode

ECHO.

echo To switch to default output sent the environment variable below

ECHO SET BRIEF=1

GOTO END

:END

ETL traces are collected to examine exactly what transpires in SCVMM and related technologies. As WinRM is the technology used to communicate control data between SCVMM Server and Hosts, WSMAN entries make up a good bit of the trace commands.

Understanding the WinRM calls made, and being able to reproduce these calls from an elevated command prompt is an important troubleshooting skill. This allows immediate testing of commands run just prior to, and on the same thread, as exceptions thrown.

Below is an example trace that is followed by the WinRM commands that would be performed to emulate the same calls.

GREEN entries are the initiating PowerShell commands from VMMService

CRIMSON entries contain WinRM [ENUMERATE] calls

BLUE entries contain Exceptions

GOLD entries are messages

5086 00005084    132.19062805    [6096] 17D0.0744::01/28-18:29:22.378#18:MonadUtils.cs(293): Cmdlet Refresh-VM:

5164 00005162    132.24555969    [4608] 1200.0A10::01/28-18:29:22.425#04:WsmanAPIWrapper.cs(1163): WSMAN: URL: [http://server.contoso.com:80] Verb: [ENUMERATE],  resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/virtualization/Msvm_ComputerSystem], wqlQuery: []

5178 00005176    132.30389404    [4608] 1200.0A10::01/28-18:29:22.488#04:WsmanAPIWrapper.cs(1163): WSMAN: URL: [http://server.contoso.com:80] Verb: [ENUMERATE],  resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/virtualization/*], wqlQuery: [select * from Msvm_MemorySettingData where InstanceID like "Microsoft:7E2BD20C-581F-4C6D-AFAA-A4D64B1221C0%" and ResourceType=4]

5180 00005178    132.31896973    [4608] 1200.0A10::01/28-18:29:22.503#04:WsmanAPIWrapper.cs(1163): WSMAN: URL: [http://server.contoso.com:80] Verb: [ENUMERATE],  resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/virtualization/Msvm_PreviousSettingData], wqlQuery: []

5182 00005180    132.32810974    [4608] 1200.0A10::01/28-18:29:22.503#04:WsmanAPIWrapper.cs(1163): WSMAN: URL: [http://server.contoso.com:80] Verb: [ENUMERATE],  resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/virtualization/Msvm_ResourceAllocationSettingData], wqlQuery: []

5190 00005188    132.49446106    [4608] 1200.0A10::01/28-18:29:22.675#04:WsmanAPIWrapper.cs(1163): WSMAN: URL: [http://server.contoso.com:80] Verb: [ENUMERATE],  resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_Volume], wqlQuery: []

…

5262 00005260    132.66345215    [4608] 1200.0A10::01/28-18:29:22.831#26:VmRefresher.cs(4876): VM V-MBX1 has an invalid bus specification or bad path or path that we cannot access and will be marked unsupported

5263 00005261    132.66349792    [4608] 1200.0A10::01/28-18:29:22.831#26:VmRefresher.cs(4876): Microsoft.VirtualManager.Engine.WorkloadCreation.DriveSlotIsBusy: Another device is already at the location (SCSI, 0, 0).

5264 00005262    132.66349792    [4608] Select another destination for the disk, and then try the operation again.

5265 00005263    132.66349792    [4608]    at Microsoft.VirtualManager.Engine.ImageLibrary.VM.CheckForBindingViolations(IList`1 vddList, IList`1 dvdList, IList`1 scsiList)

5266 00005264    132.66349792    [4608]    at Microsoft.VirtualManager.Engine.WorkloadCreation.VirtualDiskDrive.SetAdapterBinding(VHDBusType busType, Byte bus, Byte lun, Boolean checkForBindingViolations)

5267 00005265    132.66349792    [4608]    at Microsoft.VirtualManager.Engine.WorkloadCreation.VirtualDiskDrive.SetAdapterBinding(VHDBusType busType, Byte bus, Byte lun)

5268 00005266    132.66349792    [4608]    at Microsoft.VirtualManager.Engine.Backup.DifferencingDisksRefresher.RemoveOrUpdatePassThruDisksInDB(ImgLibCatalog imgLibCatalog, CommonVMBase vmBase)

5269 00005267    132.66349792    [4608]    at Microsoft.VirtualManager.Engine.Backup.DifferencingDisksRefresher.UpdateObjectsAndRelationsInDB(Guid taskID, CommonVMBase vmBase, List`1 vhdList, ImgLibCatalog imgLibCatalog)

5270 00005268    132.66349792    [4608]    at Microsoft.VirtualManager.Engine.Backup.DifferencingDisksRefresher.UpdateDB(Guid taskID, CommonVMBase vmBase)

5271 00005269    132.66349792    [4608]    at Microsoft.VirtualManager.Engine.BitBos.VMRefresherBase.UpdateDisks(IVMComputerSystem vmComputer)

5272 00005270    132.66349792    [4608] *** Carmine error was: BusSlotIsBusy (659); 0

5273 00005271    132.66349792    [4608] *** V-MBX1 ** SCSI ** 0 ** 0 **

WSMAN command translation

Line 5262 contains the error. Moving up in the trace along the same Process ID (PID) and Thread ID (TID), the most recent WinRM command called is on line 5190. This entry, being the last command executed prior to the exception, is likely related to the error. We will turn this into a command that can be called from an elevated command prompt on the SCVMM Server.

5190 00005188    132.49446106    [4608] 1200.0A10::01/28-18:29:22.675#04:WsmanAPIWrapper.cs(1163): WSMAN: URL: [http://server.contoso.com:80] Verb: [ENUMERATE],  resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_Volume], wqlQuery: []

This entry breaks down into a few components.

WSMAN: URL: [http://server.contoso.com:80] Verb: [ENUMERATE],  resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_Volume], wqlQuery: []

Target system:

http://server.contoso.com:80

WMI Namespace:

http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_Volume

Verb action:

Verb: [ENUMERATE]

Create URI from this information. Start at the wmi/.

winrm enum wmi/root/cimv2/Win32_Volume

To initiate this command on a remote system simply add -r:<servername>

winrm enum wmi/root/cimv2/Win32_Volume -r:server

The output from this will fill a few pages. If this data can be pulled, examine it. If the command pulls nothing, or produces an error, examine it more thoroughly.

This section addresses specific problems that are encountered on a regular basis in relation to SCVMM. System Center Virtual Machine Manager heavily utilizes WinRM for Control Data (communications) between the SCVMM server and Hosts it manages. This is not true of performing Physical to Virtual (P2V) conversions or interaction with VMware.

Errors in SCVMM appear in the Admin Console as a general error number, followed by a problem description, a possible workaround, and a resulting hex code. The general error number is specific to the product and is useful for reference. More important is the hex code. A sample return code might be '0x803381A6.' Simply deciphering this code often points to proper corrective actions.

Return Code 0x803381A6

An HResult of 0x803381A6 is very specific and directly applies here. Excerpts of SCVMM data are provided to illustrate the type of information available and steps taken to remedy the issue.

First, what does 0x803381A6 mean?

C:\>winrm helpmsg 0x803381A6

The WS-Management service cannot process the request. The maximum number of concurrent operations for this user has been exceeded. Close existing operations for this user, or raise the quota for this user.

Clearly, the maximum number of concurrent connections has been exceeded. Simple enough. Let's look at the trace as well.

SCVMM Trace Example

47233,14:26:14.297 07-01-2010,0x0890,0x0484,26,VmRefresher.cs,901,0x00000000,Failure in getting the vm service and the vms on that host vmserver.contoso.com,{00000000-0000-0000-0000-000000000000},1,

…

47234,14:26:14.298 07-01-2010,0x0890,0x0484,26,VmRefresher.cs,901,0x00000000,"Microsoft.Carmine.WSManWrappers.WSManProviderException: A Hardware Management error has occurred trying to contact server vmserver.contoso.com.

Check that WinRM is installed and running on server vmserver.contoso.com. For more information use the command ""winrm helpmsg hresult"". ---> System.Runtime.InteropServices.COMException (0x803381A6): The WS-Management service cannot process the request. The maximum number of concurrent operations for this user has been exceeded. Close existing operations for this user; or raise the quota for this user.

…

47984,14:26:25.356 07-01-2010,0x0890,0x10E4,19,Task.cs,246,0x00000000,"Task failed error HostAgentWSManError (2927); HR: 0x803381A6 with exception Microsoft.Carmine.WSManWrappers.WSManProviderException: A Hardware Management error has occurred trying to contact server vmserver.contoso.com.

Troubleshooting 0x803381A6

Step 1

On each server open 'services.msc' and stop, not restart, WinRM (Windows Remote Management). This should take a few seconds. If it takes 30 seconds or more follow the steps below. Otherwise, start WinRM again and move to ‘Step 2’.

From an elevated command prompt type the command below. Note the space after the "=".

sc config winrm type= own

Start the WinRM service after running this command.

This change will take effect after a Host server restart. Perform a restart when possible.

Step 2

On each server open an elevated command prompt to perform the following steps.

Windows 2008R2 AND Windows 2008 SP2 machines. Use the following command to increase the default value for WinRM timeout.

winrm set winrm/config @{MaxTimeoutms = "1800000"}

Windows 2008R2 AND Windows 2008 SP2 machines. (Windows 2008 R2 servers may report not needing this step)

winrm set winrm/config/Service @{MaxConcurrentOperations="200"}

Windows 2008R2 AND Windows 2008 SP2 machines. (Windows 2008 SP2 servers may report not needing this step)

winrm set winrm/config/Service @{MaxConcurrentOperationsPerUser="400"}

Windows 2008R2 AND Windows 2008 SP2 machines:

net stop winrm

net start winrm

net start vmmagent

Appendix A: Glossary

wsman:Items

A WS-Management protocol element returned in an enumeration that obtains both the instances and the instance EPRs. wsman:Items is a container that holds an instance and its EPR. This type of enumeration is initiated when the WSManFlagReturnObjectAndEPR flag is set in the request.

B

baseboard management controller (BMC)

A microcontroller attached locally to a server. BMCs have sensors that monitor the physical state of the server and a separate network connection that can communicate over the network, even if the server is offline. You have access to BMC data through the Intelligent Platform Management Interface (IPMI) WMI provider.

basic authentication

The user name and password sent in the authentication exchange. Basic authentication can be configured to use either HTTP or HTTPS transport in a domain or workgroup. This method is the least secure method of authentication.

BMC

See baseboard management controller (BMC).

C

CIM

See Common Information Model (CIM).

client

The client application using the WS-Management protocol to access the management service on either the local or a remote computer.

Common Information Model (CIM)

The Distributed Management Task Force (DMTF) model that describes how to represent real-world computer and network objects. CIM uses an object-oriented paradigm, where managed objects are modeled using the concepts of classes and instances.

D

digest authentication

An exchange wherein the server receives a request from a client and sends data about the client to an authenticating server, typically a domain controller. If the client is authenticated, then the server receives a Digest session key used to authenticate subsequent requests from the client.

Distributed Management Task Force (DMTF)

The industry organization developing management standards and integration technology for enterprise and Internet environments.

DMTF

See Distributed Management Task Force (DMTF).

E

endpoint

A resource that can be addressed by an endpoint reference (EPR).

endpoint reference (EPR)

A combination of WS-Addressing and WS-Management addressing elements that together describe an address for a resource in the message SOAP header. This is a Web service term.

enumeration

A set, or collection, of resource instances or the action of requesting such a set. In WS-Management protocol, WS-Enumeration is used to obtain the collection. In the WinRM service scripting implementation of enumeration, Session.Enumerate and the Enumerator object are used. The corresponding C++ method and interface are IWSManSession::Enumerate and IWSManEnumerator.

EPR

See endpoint reference (EPR).

Event Collection Service

The operating system component that receives events from the BMC and other operating system components or applications.

event forwarding

A notification of events that occur on remote computers can be sent to subscribing applications. Event forwarding is not a feature of WinRM, but of the Windows Event Log. Event forwarding becomes available for the first time in Windows Vista. The Management applications, such as Microsoft Operations Manager (MOM) use event forwarding.

F

filter

A query mechanism for specifying a limited set of instances in the request for a resource. You can specify a filter parameter on calls to Session.Enumerate or IWSManSession::Enumerate.

filter dialect

An XML string that identifies the XML dialect used to specify a filter in a call to Session.Enumerate or IWSManSession::Enumerate. The WinRM service supports WQL as a filter dialect when receiving requests.

fragment

An XML string that identifies part of an instance of a resource rather than the entire resource. Fragment support is found in the ResourceLocator object.

fragment dialect

An XML string that identifies the XML dialect used to specify a fragment. Fragment support is found in the ResourceLocator object. The WinRM service supports XPath for fragment dialect when receiving requests.

I

in-band

The client application obtains BMC data through the WinRM listener in the operating system.

Intelligent Platform Management Interface (IPMI)

An IT industry standard for the architecture of baseboard management controller (BMC). The Windows hardware management features supply an IPMI driver and a WMI IPMI provider that allow management scripts, command-line tools, and applications to obtain BMC data. The IPMI provider has WMI classes.

IPMI

See Intelligent Platform Management Interface (IMPI).

IPMI driver

The kernel driver that enables access to baseboard management controller (BMC) devices from the operating system components.

IPMI provider

A standard WMI provider that defines classes modeling the IPMI and its data. The IPMI provider is a COM DLL that obtains data from the BMC.

K

KCS

See Keyboard Controller Style (KCS).

Kerberos authentication

A method of mutual authentication between the client and server that uses encrypted keys. For computers running on a Windows-based operating system, the client account must be a domain account in the same domain as the server. When a client uses default credentials, Kerberos is the authentication method if the connection string is not one of the following: localhost, 127.0.0.1, or [::1].

Keyboard Controller Style (KCS)

The protocol used by the IPMI driver to communicate with the baseboard management controller (BMC).

L

listener

A management service that implements WS-Management protocol to send and receive messages. WinRM is a listener service. A listener is defined by a transport (HTTP or HTTPS) and an IPv4 or IPv6 address. You can create more than one WinRM listener instance on a single computer by giving them a different TCP/IP address or port number.

M

message

A package of information transmitted between computers or separate networks constructed with the SOAP protocol. The package has a header, that describes the message target and transport, and a body that contains the content to be used when the message arrives. A message is a combination of elements from specifications such as WS-Addressing, WS-Transfer, and WS-Management.

N

namespace

A WMI namespace, which is a logical grouping of WMI classes and instances to control scope and access. The most frequent source of management data for systems running Windows is the root\cimv2 namespace, which contains classes such as Win32_Process. Namespaces appear in the resource URI for WMI classes, for example http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_Service.

negotiate authentication

A negotiated, single sign on type of authentication that is the Windows implementation of Simple and Protected GSSAPI Negotiation Mechanism (SPNEGO). SPNEGO negotiation determines whether authentication is handled by Kerberos or NTLM. Kerberos is the preferred mechanism. Negotiate authentication on Windows-based systems is also called Windows Integrated Authentication.

numeric sensor

A numeric type of sensor in a baseboard management controller (BMC), for example temperature or voltage.

O

option

The additional data required by the resource provider to process a request. For example, some WMI providers require additional data supplied as IWbemContext or SWbemNamedValueSet objects. Option support is found in the ResourceLocator object.

out-of-band

The client application obtains data directly from the baseboard management controller (BMC) of another computer, rather than through the WinRM listener in the operating system.

P

pull

A WS-Enumeration pull message is sent to continue an enumeration started by an initial call to WS-Enumeration:Enumerate. The pull operation in the WinRM service is performed by Enumerator.ReadItem or IWSManEnumerator::ReadItem.

R

resource

An endpoint that represents a distinct type of management operation or value. A service exposes one or more resources and some resources can have more than one instance. A management resource is similar to a WMI class or a database table, and an instance is similar to an instance of the class or a row in the table. For example, the Win32_LogicalDisk class represents a resource. Win32_LogicalDisk="C:\" is a specific instance of the resource.

resource URI

The uniform resource identifier (URI) used to identify a specific type of resource, such as disks or processes, on a system.

S

SEL

See System Event Log (SEL).

SEL adapter

The adapter that sends baseboard management controller (BMC) data to the Event Collector.

selector

A name and value pair that represents a particular instance of a resource. This is essentially a filter or "key" that identifies the desired instance of the resource. Selector support is found in the ResourceLocator object.

sensor

A measurement device in a baseboard management controller (BMC).

service

An application that provides management services to clients through the WS-Management protocol and other Web services. A service is usually the same as the listener on a network. The service has a physical transport address.

session

A connection between a Windows Remote Management client and the local or remote WinRM listener, or service. This connection is similar to the connection between a WMI client script and WMI on a remote server. The session operations, such as enumerating a resource (Enumerate), getting an instance of a resource (Get), or running a resource method (Invoke) are methods of the Session object. A Session object is created by WSMan.CreateSession.

Simple and Protected GSS-API Negotiation Mechanism (SPNEGO)

An authentication mechanism used by the client or server receiving requests for data through the WinRM in an Active Directory context. SPNEGO is based on an Request For Comments (RFC) protocol produced by the Internet Engineering Task Force (IETF). SPNEGO is also known as Windows Integrated Authentication, the term used in the Windows Remote Management help topics.

Simple Object Access Protocol (SOAP)

An XML-based protocol used by Web services.

SOAP

See Simple Object Access Protocol (SOAP).

SPNEGO

See Simple and Protected GSS-API Negotiation Mechanism (SPNEGO).

System Event Log (SEL)

The database of events in the baseboard management controller (BMC) hardware. The SEL adapter conveys these events to the operating system.

U

uniform resource identifier (URI)

A string that identifies a resource in the enterprise, such as a computer, a process, a disk drive, or a temperature sensor in a baseboard management controller (BMC). The URI is the Web service addressing mechanism defined in Internet Engineering Task Force (IETF) Uniform Resource Identifier (URI): Generic Syntax [RFC3986].

URI

See uniform resource identifier (URI).

W

Web service

A software application used for interaction between computers across the Internet or a network. Web services are described by the Web Service Description Language (WSDL). The specific description of the Web service tells other services how to interact with the Web service using SOAP messages. The messages are conveyed between computers by a transport, typically HTTP or HTTPS. WS-Addressing, WS-Eventing, and WS-Management are examples of protocols used by Web service applications to communicate with each other.

Web Service Description Language (WSDL)

An XML-based language used to define how to interact with a Web service. Typically, the WSDL describes what SOAP messages the Web service requires to return data or carry out operations. The WSDL allows an implementation from one operating system to communicate with the Web service implemented on another operating system, as long as the requirements of the WSDL are met.

Windows Integrated Authentication

See negotiate.

Windows Management Instrumentation (WMI)

The Microsoft implementation of the Web-Based Enterprise Management (WBEM) standard published by the Distributed Management Task Force (DMTF). WMI allows you to manage local and remote computers and models computer and network objects using an extension of the Common Information Model (CIM) standard.

Windows Remote Management (WinRM)

The Microsoft implementation of a management Web service based on the public standard WS-Management protocol.

Windows Remote Shell (WinRS)

A shell tool that relies on WinRM to execute remote commands, especially for headless servers. The command-line tool is Winrs.

WMI

See Windows Management Instrumentation (WMI).

WMI plug-in

The WinRM plug-in that makes WMI data available to WinRM clients.

WS-Addressing (wsa)

A public standard protocol, which is SOAP-based, that creates an addressing system used in the headers of messages sent across the Internet. The standard defines how resources can be located across networks and firewalls. WS-Addressing is one of the Web service protocols which compose the WS-Management protocol.

WS-Enumeration (wsen)

A public standard protocol, which is SOAP-based, for enumerating a sequence of XML elements that may represent data collections, logs, or other linear information structures. WS-Enumeration is one of the Web service protocols which compose the WS-Management protocol.

WS-Eventing (wse)

A public standard protocol, which is SOAP-based, that allows one Web service (the subscriber) to subscribe to and accept event notification messages from another Web service (the event source). WS-Eventing is one of the Web service protocols which compose the WS-Management protocol.

WS-Management

A public standard protocol, which is SOAP-based, for sharing management data among all operating systems, computers, and devices. All messages sent by the WinRM client or server components use this protocol.

WS-Transfer (wxf)

A public standard protocol, which is SOAP-based, for accessing XML representations of Web service-based resources through a simple set of verbs such as Get, Put, Invoke, or Delete. WS-Transfer defines operations for sending and receiving the representation of a particular resource and operations for creating or deleting a resource and its corresponding representation.

X

XPath

A path notation for addressing parts of an XML document, similar to a URL. An XPath expression is a sequence of phrases to get from the current location in the XML document to another node or set of nodes. The phrases are separated by forward-slash ("/") characters. The WinRM service supports XPath for fragment dialect.

Source: http://msdn.microsoft.com/en-us/library/aa384465(v=VS.85).aspx

Appendix B: Details on secure port configuration for WS-Management in Windows 7 and Windows Server 2008 R2

Details on secure port configuration for WS-Management in Windows 7 and Windows Server 2008 R2

http://technet.microsoft.com/en-us/library/ee922665(WS.10).aspx

Updated: December 30, 2009

Applies To: Windows Server 2008 R2

This article provides some important details on secure port configuration for WS-Management in WinRM 2.0, which is included in Windows 7 and Windows Server 2008 R2. WinRM can be downloaded separately as part of the Windows Management Framework for Windows XP, Windows Server 2003, Windows Vista, and Windows Server 2008.

Secure port configuration details

1.       Summary

2.       Functional behavior

a.       New default values

b.       Compatibility listeners

c.       Listener migration

d.       PowerShell-specific changes

e.       WinRM-specific changes

f.        Working in mixed environments

g.       Firewall rules

h.       Event forwarding

Summary

Servers connected to the internet are typically configured to allow traffic from many different users who warrant various levels of trust. Exposing a management interface over traditional ports potentially exposes important system-altering capabilities to malicious users. For this reason, it is more secure to expose a management interface through a port that does not handle web traffic.

In order to provide more secure default configuration, and in order to lessen the likelihood that a user will unknowingly create a WS-Management listener on these common ports, the default configuration was changed in WinRM 2.0 to a different set of ports. When upgrading to WinRM 2.0, any listener that exists on port 80 or 443 will automatically be migrated to the new default ports of 5985 for HTTP and 5986 for HTTPS.

Functional behavior

New default values

The default ports used when creating new WS-Management listeners will be changed from 80 for HTTP and 443 for HTTPS to 5985 for HTTP and 5986 for HTTPS.  Any listeners created manually without specifying a port, or listeners created using quick config, will listen on these new ports. Similarly, the default ports used when issuing requests from the client will also be changed to the new ports.  Any client requests sent out without specifying a port will be sent to these new ports.

Compatibility listeners

In order to aid interoperability with computers that have not been upgraded, new configuration settings have been introduced to create compatibility listeners on ports 80 and 443 for HTTP and HTTPS, respectively. These settings can be enabled directly through the WinRM configuration console command line call as follows.

·         winrm set winrm/config/service @{EnableCompatibilityHttpListener="true"}

·         winrm set winrm/config/service @{EnableCompatibilityHttpsListener="true"}

The compatibility listeners can also be enabled via Group Policy using the new settings under Computer Configuration/Administrative Templates/Windows Components/Windows Remote Management (WinRM)/WinRM Service.

These compatibility listeners are not directly addressable, meaning they cannot be retrieved via winrm get and their settings cannot be modified. When enumerated, these listeners will be tagged with the string Source=”Compatibility” in order to identify them, as in the following example.

C:\Windows\system32>winrm e winrm/config/listener

Listener

    Address = *

    Transport = HTTP

    Port = 5985

    Hostname

    Enabled = true

    URLPrefix = wsman

    CertificateThumbprint

    ListeningOn = 127.0.0.1, 157.59.85.27, ::1, 2001:4898:0:fff:200:5efe:157.59.85.27, 2001:4898:2b:3:594:e48f:e99a:de17, 2001:4898:2b:3:4c54:7d1e:a5f6:6ea, fe80::100:7f:fffe%11, fe80::200:5efe:157.59.85.27%13, fe80::4c54:7d1e:a5f6:6ea%12

Listener [Source="Compatibility"]

    Address = *

    Transport = HTTP

    Port = 80

    Hostname

    Enabled = true

    URLPrefix = wsman

    CertificateThumbprint

    ListeningOn = 127.0.0.1, 157.59.85.27, ::1, 2001:4898:0:fff:200:5efe:157.59.85.27, 2001:4898:2b:3:594:e48f:e99a:de17, 2001:4898:2b:3:4c54:7d1e:a5f6:6ea, fe80::100:7f:fffe%11, fe80::200:5efe:157.59.85.27%13, fe80::4c54:7d1e:a5f6:6ea%12

By default these settings are both set to Enabled = False for new installations of WinRM 2.0. For upgrades, the setting values will depend on the existing configuration. See the following section for more details.

Listener migration

The following table defines the listener settings that exist after upgrading to WinRM 2.0.

PowerShell-specific changes

The New-PSSession PowerShell cmdlet has been modified to take these changes into account.  New-PsSession now takes on two different parameter sets:  ComputerName and ConnectionURI.

·         When the ComputerName parameter set is specified, then the default client port setting is used in the WinRM 2.0 configuration unless a port is explicitly specified.

·         When the ConnectionUri parameter is passed, then the string will be interpreted using ports 80 for HTTP and 443 for HTTPS by default. 

The following WSMan cmdlets have also undergone the same change as described above for New-PSSession.

·         Connect-WSMan

·         Get-WSManInstance

·         Invoke-WSManAction

·         New-WSManInstance

·         Remove-WSManInstance

·         Set-WSManInstance

WinRM-specific changes

The WinRM cmd line tool will share the behavior of the New-PSSession cmdlet in the following scenarios.

·         If a user passes a computer name to the winrm cmd line using the –r:<Computername> switch, then WinRM 2.0 will use the default client HTTP port, port 5985.

·         If a user passes a computer name to the winrm cmd line using the –r:<Computername> switch, and the –UseSSL flag is passed in, then WinRM 2.0 will use the default client HTTPS port, port 5986.

·         If a user passes a computer name to the winrm cmd line using the –r:<Computername> switch, and if a port is specified, then WinRM 2.0 will use the specified port.

·         If a user passes the connection URI to the winrm cmd line, as in the example –r:http://Mycomputer/wsman, then WinRM 2.0 will interpret the URI using ports 80 for HTTP and 443 for HTTPS by default.

Working in mixed environments

If you have a mixed environment of computers, where a number of them have WinRM 2.0 installed and a number do not, then there are a few scenarios that will require administrative action in order to maintain proper functionality.

·         If the client is updated so the default ports are 5985/5986 but the server is unchanged and it is still listening on ports 80/443, the client must either explicitly specify ports 80/443 when connecting or specify the remote machine using a ConnectionUri.

·         If the client operating system has a new installation of Windows 7, and the default ports are 5985 for HTTP and 5986 for HTTPS, but the server is unchanged so that it is still listening on 80/443, then the situation is the same. The client must either explicitly specify ports 80/443 when connecting or specify the remote server using a ConnectionUri.

·         If the server operating system has a new installation of Windows 7, and it is listening on ports 5985 for HTTP and 5986 for HTTPS, but the client is unchanged so that its default ports are still 80/443, the administrator must explicitly create listeners on ports 80/443.

Firewall rules

There are two firewall rules within the same group: one for port 80 and one for port 5985. The following table defines the status of these two rules in various upgrade scenarios.

Event forwarding

The tables below lay out the various situations that may occur with event forwarding, and what steps need to be taken to correct potential issues.

Collector-initiated PUSH/PULL subscriptions

Source-initiated PUSH subscriptions with existing subscriptions specifying collector as Server=<FQDN>

Source-initiated PUSH subscriptions with existing subscriptions specifying collector as Server=http://<FQDN>

Appendix B: WinRM Event IDs

The WinRM service processes WSMan requests received over the network. It uses HTTP.sys to listen on the network. The following is a list of all aspects that are part of this managed entity as identified in WinRM Event logs. Solutions to many errors are provided.

Applies To: Windows Server 2008 R2

You can configure various aspects of the WinRM service. The configuration information is stored in the registry.

Applies To: Windows Server 2008 R2

The Listener is a component of the WinRM service that indicates to the service which IP addresses and ports to listen on for WS-Management related requests. Users can create one or more listeners.

Applies To: Windows Server 2008 R2

Plug-ins are loaded and used by the WinRM service to provide various functions including: configuration, WMI access, and event forwarding.

Appendix C: WMI Code Reference

As WinRM is deeply intertwined with WMI there are often return codes, error constants and so on that, while reported by WinRM, originated in and relate to WMI. Knowing the meaning of these codes and what aspect of Windows they relate to can improve troubleshooting knowledge.

This section proides a list of the WMI return codes that may be returned by scripts, C++ applications, or Wmic commands.

If an error occurs, WMI returns one of the following error codes as a 32-bit value where the two high-order bits indicate the severity code of the message.

To obtain more information about the source of the problem, you can download and run the WMI Diagnosis Utility diagnostic command line tool. This tool produces a report that can usually isolate the source of the problem and provide instructions on how to fix it. The report also aids Microsoft support services in assisting you.

If an error occurs, WMI returns an error code as an HRESULT value. These codes may be returned by scripts, C++ applications, or Wmic.

Note  If WMI returns error messages, be aware that they may not indicate problems in the WMI service or in WMI providers. Failures can originate in other parts of the operating system and emerge as errors through WMI. Under any circumstances, do not delete the WMI repository as a first action because deleting the repository can cause damage to the system or to installed applications.

Some methods in WMI classes can return system and network error codes (64 for example). You can check the definition of these types of error codes by using the net helpmsg command in the command prompt window. For example, the command net helpmsg 64 returns the message: The specified network name is no longer available.

Figure  SEQ Figure \* ARABIC 8: Common ranges of WMI errors

Some methods in WMI classes can return system and network error codes (64 for example). You can check the definition of these types of error codes by using the net helpmsg command in the command prompt window. For example, the command net helpmsg 64 returns the message: The specified network name is no longer available. In C++, you can call FormatMessage and specify C:\Windows\System32\wbem\wmiutils.dll as the message module.

Windows 2000, Windows NT 4.0, and Windows Me/98/95:  Use C:\Winnt\System32\wbem\wbemcomn.dll as the message module.

Figure  SEQ Figure \* ARABIC 9: Net Helpmsg return codes

If an operation does not result in an error, WMI returns one of the following codes as an HRESULT that indicates the status of the operation.

Note  Some methods in WMI classes can return system and network error codes (64 for example). You can check the definition of these types of error codes by using the net helpmsg command in the command prompt window. For example, the command net helpmsg 64 returns the message: The specified network name is no longer available.

In C++, you can call FormatMessage and specify C:\Windows\System32\wbem\wmiutils.dll as the message module.

Windows 2000 and Windows NT 4.0:  Use C:\Winnt\System32\wbem\wbemcomn.dll as the message module.

Figure  SEQ Figure \* ARABIC 10: If an operation does not result in an error, WMI returns one of the following codes as an HRESULT that indicates the status of the operation