Server locator service call had a communication error

In our previous blog post we covered an overview of what migration endpoints are, how to find them and what makes them tick. In this post, we will cover related troubleshooting. Note that this post has some in-depth troubleshooting steps, so it is not necessarily something that you’ll read for fun, but we wanted to make it available for those times when you run into trouble!

Now let’s cover some troubleshooting!

Things that are commonly mis-configured

Before getting to the part where we troubleshoot specific migration endpoint issues, be aware of the following top reasons why a migration endpoint cannot be created:

• TLS1.2 should be enabled in the on-premises infrastructure.
• MRSproxy needs to be enabled and running in Exchange on-premises.
• Exchange Online requires Negotiate (NTLM) authentication for MRSProxy.
• On-premises migration Admin needs to have the minimum required permissions and valid credentials.
• Load-balancing Exchange 2010 MRSproxy servers requires IP persistence (affinity).
• SSL offloading is not supported for MRSProxy.
• For Classic hybrid – where we require inbound connectivity from Exchange Online to on-premises Exchange, allow all Exchange Online IP addresses to connect to on-premises EWS / Autodiscover.
• For Classic hybrid, pre-authentication for EWS / Autodiscover virtual directories is not supported.
• For Classic hybrid, a valid 3^rd party certificate is required for EWS / IIS. Also see this.

Troubleshooting

Whether you are having trouble creating new migration endpoints in Office 365 Exchange Online or are not able to migrate anymore to or from Exchange Online using an existing migration endpoint, the cmdlet Test-MigrationServerAvailability is your dear friend. Always run this cmdlet in Exchange Online PowerShell, not from on-premises Exchange Management Shell.

We will focus on hybrid migration endpoint issues and the cmdlet syntax for this endpoint type. Below we will show you 3 commands that can help you check for underlying issues or error messages:

1. Hybrid remote move endpoint with Autodiscover

Test-MigrationServerAvailability -ExchangeRemoteMove -Autodiscover -EmailAddress user@contoso.com -Credentials (get-credential contosoadministrator)

Note: The option with Autodiscover is not used in Modern hybrid as we go directly to EWS server(s) for both Migration Endpoints and Free/Busy configuration (Cloud Intra-Organization Connectors and Organization Relationships have TargetSharingEpr set to the EWS namespace.)

2. Hybrid remote move endpoint without Autodiscover (testing EWS directly)

Test-MigrationServerAvailability -ExchangeRemoteMove –RemoteServer mail.contoso.com -Credentials(get-credential contosoadministrator)

Note: In Modern hybrid, the RemoteServer is in the format of ‘<GUID>.resource.mailboxmigration.his.msappproxy.net’, where <GUID> is unique for each organization. This is randomly generated and stored encoded base 64 in the OnPremisesOrganization object’s Comment in Exchange Online when you will run Modern HCW and it gets to that configuration part.  This GUID will be stamped on the Migration Endpoint in the RemoteServer value for both Full and Minimal Modern Hybrid Topologies and also on the TargetSharingEpr values for Cloud Intra-Organization Connector / Organization Relationship . You can check the GUID in the HCW log and on Get-MigrationEndpoint / Get-IntraOrganizationConnector / Get-OrganizationRelationship EXO cmdlets.

Cmdlets ran above:

Get-IntraOrganizationConnector | fl targetsharingepr
Get-OrganizationRelationship | fl targetsharingepr
Get-MigrationEndpoint
$strdata = (Get-OnPremisesOrganization).comment
$bytes = [Convert]::FromBase64String($strdata)
$ms = New-Object System.IO.MemoryStream(@(,$bytes))
$deflate = New-Object System.IO.Compression.DeflateStream($ms, [System.IO.Compression.CompressionMode]::Decompress)
$reader = New-Object System.IO.StreamReader($deflate)
$text = $reader.ReadToEnd()
$text

3. Testing an existing hybrid remote move endpoint

Test-MigrationServerAvailability -Endpoint <Identity of the Endpoint>

OK, I ran these and found errors; now what?

Let’s walk through some common Test-MigrationServerAvailability errors and how to troubleshoot these in Classic / Modern hybrid.
Tip: Whenever a command returns an error in PowerShell, you should run the command $Error[0].Exception |fl -f to get more details on the exception thrown.

However, test-migrationserveravailability failures are not actually those red errors that you get when a command does not work in PowerShell. So, in this situation, we can use New-MoveRequest command to throw the same error and get the serialized exception from this one.
Test-MigrationServerAvailability which returns the result:

Running the New-MoveRequest command to test migration of a synced user will give me same error message but in “red” and we can then get the serialized exception from it.

Based on these error messages, we gathered most common scenarios.

Scenario 1: Test-MigrationServerAvailability fails because of connectivity / timeout errors

Examples of some of those errors:

The call to 'https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' timed out. Error details: The request channel timed out attempting to send after 00:00:00. The time allotted to this operation may have been a portion of a longer timeout.

The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' timed out. Error details: The operation did not complete within the allotted timeout of 00:00:50. The time allotted to this operation may have been a portion of a longer timeout. 

The request channel timed out while waiting for a reply after 00:00:09.9996191. The time allotted to this operation may have been a portion of a longer timeout. The remote server returned an error: (504) Gateway Timeout.

Troubleshooting these timeout errors in Modern hybrid:

During the Modern hybrid configuration, you will be asked to input the credentials for the on-premises migration admin – these can be the same credentials inserted in the beginning of HCW or new ones. The on-premises Migration admin credentials are needed by HCW in order to run the Test-MigrationServerAvailability cmdlet and this account can have lesser privileges than the admin account running HCW (which is Organization Management role). Once Modern HCW has tested the on-premises migration server availability, we will know if we are able to create the migration endpoint in Exchange Online or not and also in this step we would see the dynamically generated GUID for your on-premises MRSProxy server(s): https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc.
Supposing that you get a timeout error in HCW when testing migration server availability, you would first search in the HCW log for the Test-MigrationServerAvailability cmdlet executed by HCW, copy-paste the exact command to a Notepad file (or at least make a note of the RemoteServer value) and then connect to Exchange Online PowerShell and run the same command to confirm the error message thrown by HCW.

Connect to Exchange Online PowerShell using an O365 Global Admin account, you can use for example this one line command:

Import-PSSession $(New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Authentication Basic –AllowRedirection -Credential $(Get-Credential))

Once you are successfully connected to EXO PS, run the same command that HCW ran in the log file to see if you get same error or different one, providing the credentials for the on-premises migration admin and your RemoteServer value:

Test-MigrationServerAvailability -ExchangeRemoteMove: $true -RemoteServer '<GUID>.resource.mailboxmigration.his.msappproxy.net' -Credentials (Get-Credential domainadmin)

Most likely you will get same result as HCW did when you will run the same command in PowerShell. Although this step may seem redundant, it is always a good idea to analyze the command ran by HCW (if you selected Classic Hybrid option during HCW, then HCW should not try to create the migration endpoint to <GUID>.resource.mailboxmigration.his.msappproxy.net which is for Modern Hybrid topology) and confirm at the same time that the error was not for example a transient one.

Once you confirm that you still get the error that HCW was complaining about and you are running Modern Hybrid Mode, you would need to investigate it.

On the server where you ran the Modern HCW, you need to check the Hybrid Agent Status (ACTIVE or INACTIVE). There are two ways to do this:

1. Re-run Modern HCW and check it in the GUI, reference here.
2. Through PowerShell as described below, reference here.

NOTE: Make sure you have installed Azure PowerShell Module before proceeding further.
Import the Hybrid Management Module in Windows PowerShell (ran as Administrator) from Program FilesMicrosoft Hybrid Service directory:

Import-Module .HybridManagement.psm1

Run the Get-HybridAgent command:

Get-HybridAgent -credential (get-credential)    ## cloud admin credentials

Check that the Hybrid Agent(s) is ACTIVE. 

• If you switched from Modern hybrid to Classic hybrid (as this would uninstall the agent). If that is the case – you proceeded with Classic hybrid topology and this successfully uninstalled the Hybrid Agent, then you can ignore this warning thrown by HCW related to the migration endpoint for <GUID>.resource.mailboxmigration.his.msappproxy.net and you should create the migration endpoint in EAC using Autodiscover or your published EWS URL. At the moment of the writing of this blog post, if you switch from Modern to Classic successfully – HCW still tries to create the migration endpoint going to the Hybrid Agent proxy instead of using your published EWS URL.
• If Hybrid Service is installed on the machine and is up and running and the Hybrid Agent machine itself is running
• Check all things from Additional Information here to see if the Agent is installed properly

Most important, if Hybrid Agent is ACTIVE but you get the error ‘unable to connect to the server’ in Test-MigrationServerAvailability, check and confirm with Performance Monitor that you see the requests.
If the request counters (for #of requests) go up on the Agent machine when you do Test-MigrationServerAvailability to the Hybrid Agent, the problem is likely on the on-premises server; if they don’t, the problem is probably with either the connector or the cloud configuration. If we suspect on-premises (most likely the case), you need to check again the on-premises infrastructure, especially proxy and firewall settings. Several things that might be helpful: install requirements, system requirements, port and protocol requirements.

Troubleshooting these errors in Classic hybrid:

Location of IIS and HTTPProxy logs:

• IIS logs Default Web Site (DWS): %SystemDrive%inetpublogsLogFilesW3SVC1

Example: C:inetpublogsLogFilesW3SVC1

The name of the IIS logs contains the date of the log, for example u_ex190930.log is from Sept 30, 2019.

• HTTPProxy logs for EWS:  %ExchangeInstallPath%LoggingHttpProxyEws

Example: C:Program FilesMicrosoftExchange ServerV15LoggingHttpProxyEws

The name of the HTTPProxy logs contains the date and hour starting to log, for example HttpProxy_2019093014-10.LOG (10^th log from Sept 30, 2019, starting hour 14:00 UTC)

Below we have some examples of entries from IIS logs when we get a successful request (200 OK) and a failed request (500) that could correspond with a timeout error in Test-MigrationServerAvailability if the request reached IIS / Exchange Server.

IIS logs - 200 OK
2019-08-28 06:57:38 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 0
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 15
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contosoadministrator 4.4.0.1 - 200 0 0 125
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contosoadministrator 4.4.0.1 - 200 0 0 296

IIS logs - 500 error
2019-08-28 07:15:48 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 2 5 4890
2019-08-28 07:15:52 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 0
2019-08-28 07:15:55 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contosoadministrator 4.4.0.1 - 500 0 0 2562
2019-08-28 07:15:55 192.168.2.50 POST /EWS/mrsproxy.svc - 443 contosoadministrator 4.4.0.1 - 500 0 0 93

If you don’t see the failed requests in IIS logs, make sure you allow all Exchange Online IP addresses to connect to your on-premises and check firewall logs to see if connections were blocked.

This is an extract from HTTP proxy logs with a 500 error code and a timeout when proxying to another Exchange server:

2019-09-30T12:02:55.930Z,a0ff365c-741b-4e59-b2e3-760991d3a27a,15,1,1713,5,,Ews,exch01.contoso.com,/EWS/mrsproxy.svc,,Negotiate,true,contosoadmin,,OrganizationId~OrganizationAnchor@,,40.100.175.55,exch01,500,,ServerLocatorError,POST,,,,,ForestWideOrganization,,,2807,664,,,,1,2819,0;,0,,0,8;2;,10,10,,0,2821,0,,,,,,,,,0,2819,0,,2819,,2820,2820,,,,BeginRequest=2019-09-30T12:02:53.109Z;CorrelationID=<empty>;ProxyState-Run=None;ServerLocatorCall=DM:a10ad628-e020-409e-9f1e-22a700182ac1~~contoso.structure;ProxyState-Complete=CalculateBackEnd;SharedCacheGuard=0;EndRequest=2019-09-30T12:02:55.930Z;S:ServiceCommonMetadata.Cookie=ee4af79a0a144bcaa9a5bc0af6eec215;I32:ADR.C[DC04]=1;F:ADR.AL[DC04]=1.554922;I32:ATE.C[DC04.contoso.local]=2;F:ATE.AL[DC04.contoso.local]=1;I32:ADS.C[DC04]=1;F:ADS.AL[DC04]=2.897117,HttpProxyException=Microsoft.Exchange.HttpProxy.HttpProxyException: Server Locator Service call had a communication error. ---> Microsoft.Exchange.Data.ServerLocator.ServerLocatorClientTransientException: Server Locator Service call had a communication error. ---> System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://exch02.contoso.local:64337/Exchange.HighAvailability/ServerLocator. The connection attempt lasted for a time span of 00:00:02.8010812. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time  or established connection failed because connected host has failed to respond 10.2.2.1::64337.  ---> System.Net.Sockets.SocketException: A connection attempt failed because the connected party did not properly respond after a period of time  or established connection failed because connected host has failed to respond 10.2.2.1::64337    at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)    at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)    at System.ServiceModel.Channels.SocketConnectionInitiator.ConnectAsyncResult.OnConnect(IAsyncResult result)    --- End of inner exception stack trace ---

Scenario 2: Test-MigrationServerAvailability fails with 403 Forbidden

The connection to the server '<GUID>.resource.mailboxmigration.his.msappproxy.net' could not be completed., The call to 'https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP request was forbidden with client authentication scheme 'Negotiate'. --> The remote server returned an error: (403) Forbidden.., The HTTP request was forbidden with client authentication scheme 'Negotiate'., The remote server returned an error: (403) Forbidden.

Follow this article, as it is applicable for both Modern and Classic hybrid topologies.

Scenario 3: Test-MigrationServerAvailability fails with 401 Unauthorized, Access denied or Invalid credentials

Check this support article.

Check authentication methods on all EWS virtual directories in IIS and make sure Negotiate provider under Windows Authentication is enabled for all EWS.

Make sure on-premises migration Admin has at least Exchange Recipient Admin permissions (or Recipient Management, depending on the Exchange version). If you are running Modern HCW, usually this is the same on-premises account that has Organization Management rights, but if you change the account, you would see here if the permissions are right or the credentials are correct when we would input account credentials. We recommend that you create (do not copy account) another on-premises account that purely has Exchange Recipient Admin permissions, then test with that account (Test-MigrationServerAvailability):

Do you have devices that pre-authenticate the requests coming from Exchange Online to Exchange on-premises servers (EWS and Autodiscover paths)? If yes, this is not supported. If you are not sure of it, we recommend you temporarily bypass network devices in front of Exchange Servers, allow direct access to Exchange Servers and see if same error when doing Test-MigrationServerAvailability

When you run Test-MigrationServerAvailability, make a note of the timestamp when you get the error, then check IIS logs on each Exchange Client Access Server (logs are UTC timezone) around the exact time when Test-MigrationServerAvailability has been ran (HH:MM:SS) and check entries for mrsproxy.svc and see the statuses mentioned. Normally, there following are the first 3 IIS entries for 1 successful request (Test-MigrationServerAvailability):

Successful
2019-08-28 06:57:38 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 0
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 15
2019-08-28 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc - 443 miryadministrator 4.4.0.1 - 200 0 0 125

Some other error examples:

Issue: wrong credentials of migration admin

Message            : The connection to the server 'mail.contoso.com' could not be completed.
ErrorDetail        : Microsoft.Exchange.MailboxReplicationService.RemotePermanentException: The Mailbox Replication Service was unable to connect to the remote server using the credentials provided. Please check the credentials and try again. The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' failed. Error details: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Negotiate,NTLM'. --> The remote server returned an error: (401) Unauthorized.

IIS Logs
---------
2019-08-28 07:19:05 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 0 0 2390
2019-08-28 07:19:10 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074254 0
2019-08-28 07:19:10 192.168.2.50 POST /EWS/mrsproxy.svc - 443 - 4.4.0.1 - 401 1 2148074252 46

Issue: authentication scheme mismatch (EXO requires Negotiate / NTLM, on-premises gives us Basic only)

Message         : The connection to the server 'mail.contoso.com' could not be completed
ErrorDetail      : Microsoft.Exchange.MailboxReplicationService.RemotePermanentException
The Mailbox Replication Service was unable to connect to the remote server using the credentials provided. Please check the credentials and try again. The call to 'https://mail.contoso.com/EWS/mrsproxy.svc failed. Error details: The HTTP request is unauthorized with client authentication scheme 'Negotiate'. The authentication header received from the server was 'Basic Realm="mail.contoso.com"'. --> The remote server returned an error: (401) Unauthorized.

First, make sure that NTLM is enabled on the EWS virtual directory. You would check with Get-WebServicesVirtualDirectory |FL cmdlet if NTLM is present in the Authentication Methods. You should also double check in IIS Manager, to make sure that the Negotiate provider is present as well under Windows Authentication. Negotiate provider is falling back to NTLM if Kerberos is not possible and with external clients that is always the case. So, Exchange Online MRS requires NTLM inside the Negotiate Provider on the EWS on-premises virtual directory.

If you are using Azure AD App Proxy as Reverse Proxy for your MRSProxy servers, then you should be aware that there is a limitation of AADAP that cannot present both Negotiate and NTLM providers in the WWW-Authenticate header, no matter the order of the providers in IIS. If you have this setup, you can remove NTLM provider from Windows Authentication on EWS in IIS Manager (leave only Negotiate provider that does also NTLM) or bypass AADAP.
If that is fine (configuration on Exchange and IIS side), then you would check the publishing rules for EWS on your firewall/reverse proxy.

You can also use this mini PowerShell script to check the Authentication Methods advertised by your on-premises (replace the URL with your on-premises MRSProxy /EWS namespace).  It is also a good practice to check the WWW-Authenticate headers both from external and internal PC and notice if there are differences in the output (for example you see Basic, Negotiate, NTLM from internal – all ok but you see only Basic from external – not ok).

$req = [System.Net.HttpWebRequest]::Create("https://mail.contoso.com/ews/MRSProxy.svc")
$req.UseDefaultCredentials = $false
$req.GetResponse()
# Expected error: Exception calling "GetResponse" with "0" argument(s):
# "The remote server returned an error: (401) Unauthorized."
$ex = $error[0].Exception
$resp = $ex.InnerException.Response
$resp.Headers["WWW-Authenticate"]

Scenario 4: Test-MigrationServerAvailability fails with SSL / TLS error

Microsoft.Exchange.Migration.MigrationServerConnectionFailedException: The connection to the server'hybrid.contoso.com' could not be completed. --->
Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The Mailbox Replication Service could not connect to the remote server because the certificate is invalid. The call to 'https://hybrid.contoso.com/EWS/mrsproxy.svc' failed. Error details: Could not establish trust relationship for the SSL/TLS secure channel with authority 'hybrid.contoso.com'. -->The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. --> The remote certificate is invalid according to the validation procedure.

Whenever you see SSL/TLS errors, you would check the following:

For Classic hybrid, you can use this PowerShell command while logged to your Office 365 Exchange Online tenant to test the network request on TLS1.2 protocol and see if you also get a SSL/TLS error here:
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12;Invoke-WebRequest https://<endpoint FQDN>/ews/mrsproxy.svc -Verbose

For Modern Hybrid, you can check Test-HybridConnectivity which also checks for TLS1.2 and this is mandatory step when configuring / installing the Hybrid Agent. See Verifying Connectivity section from here.

You can use the following command in Exchange Management Shell to quickly check the Exchange certificates and some properties on them from all your Exchange servers in the organization:

Foreach ($i in (Get-ExchangeServer)) {Write-Host $i.FQDN; Get-ExchangeCertificate -Server $i.Identity | FT Thumbprint, Status, RootCAtype, Services, Subject}

Also, you should be able to access the CRLs for the certificate.

Scenario 5: Test-MigrationServerAvailability fails with 503 Service Unavailable

The call to 'https://<GUID> resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc' failed. Error details: The HTTP service located at https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc is unavailable.  This could be because the service is too busy or because no endpoint was found listening at the specified address. Please ensure that the address is correct and try accessing the service again later. --> The remote server returned an error: (503) Server Unavailable

For this error, best is to check HTTPProxy logs for EWS and see if it tried to proxy to a server that is unavailable, for example Server Wide Offline in Get-ServerComponentState or maybe EWS application pool is not started or MRS service is stopped. One thing to add here is that, if you have Exchange 2013/2016 in coexistence with Exchange 2010, the Exchange 2013/ 2016 servers will proxy to the same version server (2013/2016) and will not proxy down to Exchange 2010.

Scenario 6: Test-MigrationServerAvailability fails with 400 Bad Request

There was no endpoint listening at https://<GUID>.resource.mailboxmigration.his.msappproxy.net/EWS/mrsproxy.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. —> The remote server returned an error: (400) Bad Request.

For both Modern and Classic Hybrid Configurations, check if you see the corresponding 400 entries both in IIS logs and HTTPProxy logs (for ews/mrsproxy.svc) when running Test-MigrationServerAvailability command for ExchangeRemoteMove.

Example of a 400 entry in IIS logs:

2020-02-20 06:57:42 192.168.2.50 POST /EWS/mrsproxy.svc — 443 miryadministrator 4.4.0.1 — 400 0 0 125

If you see it in IIS logs, check HTTPProxy logs (if Exchange 2013 or higher) and Failed Request Traces in IIS to get more info on the error. See here how to enable the former ones. You can narrow down the statuses between 400-499 as an example for this situation.

If you don’t see the 400 entries in IIS logs, this means the request from Exchange Online didn’t reach the IIS Server / Exchange CAS. You would then check HTTPerr logs (HTTP.sys would be before IIS from a routing point of view), these are found in %SystemRoot%System32LogFilesHTTPERR.

Example of a 400 entry in HTTPerr log:

2020-02-20 12:06:34 192.168.2.50 41743 192.168.2.50 443 HTTP/1.1 POST /EWS/mrsproxy.svc 400 — Hostname —

For this particular error, where we have the keyword “hostname”, you would check the IIS bindings, if there are hostnames populated for HTTPS and HTTP (screenshot below) , if we have HTTP redirection or URL Rewrite in place that would alter the host headers.

If you don’t see the 400 entries in the HTTPerr logs either, then likely the request didn’t reach the Exchange Server at all. You would check the network devices that are in front of the Exchange Servers (firewall, reverse proxy, load balancer).

In case of Modern Hybrid Configuration, you would check if the Hybrid Connector is up and running. You can check its status in HCW GUI or using the PowerShell method.

If the Hybrid Agent is inactive, then check if the Hybrid Service is installed and running, reference here. You would install the agent automatically via HCW when choosing Modern Hybrid Topology or manually.

The Agent would be uninstalled if for example you switched from Modern to Classic Hybrid Topology or manually uninstall of the Microsoft Hybrid Service in Programs and Feature.

If you didn’t uninstall the Microsoft Hybrid Service and the service is started and running, then you would need to check the Hybrid Service logs. Describing below on how to enable this type of logging and where to get the logs from.

To enable the logging, you would have to navigate to Hybrid Service installation path, for example C:Program FilesMicrosoft Hybrid Service. In this folder, there is a config file of the Hybrid Service called Microsoft.Online.EME.Hybrid.Agent.Service.exe.config. You would run the Notepad.exe program as Administrator and then open this config file to edit it. Remove the <!—and —> characters (uncomment the XML comments) from the config file and save it. The file should look like this:

Restart the Microsoft Hybrid Services in services.msc. Follow the procedure from here to attempt to reproduce the problem. Navigate to these 2 folders and check the HybridService logs:

• C:programdataMicrosoft Hybrid ServiceLogging will have logs related to connector registration and startup.
• C:programdataMicrosoftMicrosoft Hybrid ServiceTrace will have logs regarding application requests.

If the Hybrid Agent is active, then you would check with performance monitor that the #requests go up, as mentioned here. If they go up, this means the Hybrid Agent is fine and that the Exchange Online Request reached the Hybrid Connector. You would further need to check the connector route to make sure it is able to reach Exchange /IIS server.  The Hybrid Connector would be pointing to your Exchange CAS or your load balancer. You can see this in Get-HybridApplication cmdlet where appId is your Remote Server in Test-MigrationServerAvailability cmdlet and the credentials are the Office 365 Global Admin. See below example of the cmdlet:

From the Hybrid Connector machine, you would check if you are able to resolve the internalURL FQDN to the internal IP of your Exchange Server. For testing purposes, try running Update-HybridApplication and point it to a single Exchange Server instead of a load balancer and check if it still fails. If, after pointing to the Exchange Server, you still get the 400 requests, go check HTTPerr, IIS, HTTPProxy and Event viewer logs on that Exchange server and see if you can track the request.

This brings us to the end of this post! Hope you find this useful if you want to learn a bit more about how migration endpoints work, or you need to troubleshoot this area.

I wanted to thank Brad Hughes, Jason Nelson, Nino Bilic and Greg Taylor for their review of this post.

Exchange server 2013 OWA problem

We have Windows 2008 DC’s on HO and DR site. Both sites are connected with 2MBPS WAN using MPLS. The HO site has Exchange 2013 on Windows 2012 and in DR site also Exchange 2013 on Windows 2012 server. DAG is setup and primary has active and DR exchange has passive database copy.

When i check the ECP DR db properties it shows following error attached.
A server-side administrative operation has failed. The Microsoft Exchange Replication service may not be running on server DREXCH01.domain.local. Specific RPC error message: Error 0x71a (The remote procedure call was cancelled) from RpccGetCopyStatusEx4

When i ran the Get-MailboxDatabase copy on DR site i get following output :
[PS] C:>Get-MailboxDatabaseCopystatus *

Name                                                                   Status          CopyQueue ReplayQueue LastInspectedLogTime   Conte
                                                                                                                                    Length    Length                             State
—-                                                                             ——          ——— ———— ———————   ——
Mailbox Database 1361361745EX2013           ServiceDown            0         0                                  Unkno
Mailbox Database PFEX2013                             ServiceDown            0         0                                  Unkno
Mailbox Database 1458444762DREXCH01     Mounting                  0         0                                  Faile
Mailbox Database 1361361745DREXCH01     Disconnected…   219       104         1/19/2016 5:49:32 PM   Faile
DRDBDREXCH01                                                  Mounted                   0         0                                  Healt

When i do the same in Production LAN :
[PS] C:>Get-MailboxDatabaseCopyStatus *

Name                                          Status          CopyQueue ReplayQueue LastInspectedLogTime   Content
                                                              Length    Length                             State
—-                                          ——          ——— ———— ———————   ——-
Mailbox Database 1361361745EX2013            Mounted         0         0                                  Healthy
Mailbox Database PFEX2013                               Mounted         0         0                                  Healthy
Mailbox Database 1458444762DREXCH01          ServiceDown     0         0                                  Unknown
Mailbox Database 1361361745DREXCH01          ServiceDown     0         0                                  Unknown
DRDBDREXCH01                                                      ServiceDown     0         0                                  Unknown

Both end there is no port been blocked. DC replication are running fine and servers are reachable. I am unsure where to start, i am assuming it is network issue but i need solid proof to throw the issue to network team. Any suggestion and ideas are appreciated.

These are my troubleshooting notes. Thought it could benefit the community.

Error when installing Exchange Server 2013 CU 7:

Issue:

When installing Exchange 2013CU7, it fails to update with the following error: We will not be able to uninstall or continue further with the installation:

Exception : Microsoft.Exchange.Management.Deployment.ScriptExecutionException: The following error was generated when “$error.Clear();
          Enable-RpcOverTcpPortScaling; 
          Enable-ExtendedTcpPortRange
        ” was run: “System.Management.Automation.ActionPreferenceStopException: The running command stopped because the preference variable “ErrorActionPreference” or common parameter is set to Stop: The registry key at the specified path does not exist.
at System.Management.Automation.PowerShell.EndInvoke(IAsyncResult asyncResult)
at Microsoft.Exchange.Configuration.MonadDataProvider.MonadPipelineProxy.ClosePipeline(MonadAsyncResult asyncResult)”. —> System.Management.Automation.ActionPreferenceStopException: The running command stopped because the preference variable “ErrorActionPreference” or common parameter is set to Stop: The registry key at the specified path does not exist.
at System.Management.Automation.PowerShell.EndInvoke(IAsyncResult asyncResult)
at Microsoft.Exchange.Configuration.MonadDataProvider.MonadPipelineProxy.ClosePipeline(MonadAsyncResult asyncResult)
— End of inner exception stack trace —
at Microsoft.Exchange.Management.Deployment.ComponentInfoBasedTask.InternalExecuteScript(String script, Boolean handleError, Int32 subSteps, LocalizedString statusDescription)
at Microsoft.Exchange.Management.Deployment.ComponentInfoBasedTask.ExecuteScript(String script, Boolean handleError, Int32 subSteps, LocalizedString statusDescription)
at Microsoft.Exchange.Management.Deployment.ComponentInfoBasedTask.GenerateAndExecuteTaskScript(InstallationCircumstances installationCircumstance)
[01/30/2015 19:47:58.0635] [1] [ERROR] The following error was generated when “$error.Clear();
. “$RoleInstallPathScriptsConfigureNetworkProtocolParameters.ps1″;
Enable-RpcOverTcpPortScaling;
Enable-ExtendedTcpPortRange
” was run: “System.Management.Automation.ActionPreferenceStopException: The running command stopped because the preference variable “ErrorActionPreference” or common parameter is set to Stop: The registry key at the specified path does not exist.
at System.Management.Automation.PowerShell.EndInvoke(IAsyncResult asyncResult)
at Microsoft.Exchange.Configuration.MonadDataProvider.MonadPipelineProxy.ClosePipeline(MonadAsyncResult asyncResult)”.
[01/30/2015 19:47:58.0635] [1] [ERROR] The running command stopped because the preference variable “ErrorActionPreference” or common parameter is set to Stop: The registry key at the specified path does not exist.
[01/30/2015 19:47:58.0635] [1] [WARNING] An unexpected error has occurred and a Watson dump is being generated: The following error was generated when “$error.Clear();
. “$RoleInstallPathScriptsConfigureNetworkProtocolParameters.ps1″;
Enable-RpcOverTcpPortScaling;
Enable-ExtendedTcpPortRange
” was run: “System.Management.Automation.ActionPreferenceStopException: The running command stopped because the preference variable “ErrorActionPreference” or common parameter is set to Stop: The registry key at the specified path does not exist.
at System.Management.Automation.PowerShell.EndInvoke(IAsyncResult asyncResult)
at Microsoft.Exchange.Configuration.MonadDataProvider.MonadPipelineProxy.ClosePipeline(MonadAsyncResult asyncResult)”.

Resolution:

• Check if the registry key is present “HKLMSOFTWAREPoliciesMicrosoftWindows NTRpc” and a value EnableTcpPortScaling. If this registry key is missing, please try to create it. Go to HKLMSOFTWAREPoliciesMicrosoftWindows NTRpc  -> Click Edit -> New -> DWORD -> enter EnableTcpPortScaling -> double click it and change the value data to 1.
  
• Disable any third party security software ( Symantec Endpoint, Sopohs etc.) & try installing it again
  

Unable to initialize the Microsoft Exchange Information Store service. – Error 0x96f. Exchange fails to start with the following error:”Unable to initialize the Microsoft Exchange Information Store service.   – Error 0x96f. “    

Resolution:

• Re-enabling IPv6.
  

https://social.technet.microsoft.com/Forums/office/en-US/b89d34a7-c8c8-4433-89ab-90114d1b7012/unable-to-initialize-the-microsoft-exchange-information-store-service-error-0x96f?forum=exchangesvrdevelopment

Getting error “Multiple potential Migration mailboxes were found for this organization. Please specify a particular Partition mailbox to use” when trying to set up mailbox migrating through EAC.

When trying to migrate mailbox from Exchange 2010 to Exchange 2013 using Exchange EAC we get the following error in the migration wizard:

“Multiple potential Migration mailboxes were found for this organization. Please specify a particular Partition mailbox to use.”

Resolution:

• It is recommended to remove all migrated data first.
  

• To remove run the following command:

  Remove-MoveRequest 
  

  Identity: Mailbox@ domain.com
  

  Select [A]

• Then using the following command try to migrate the mailbox:
  

  ‘mail@domain.com’ | New-MoveRequest -TargetDatabase “Database_Name” -BadItemLimit 200 (if you have corrupted mails)

Exchange management shell on Exchange 2013  server locally it would fail to connect locally with error:

Failed to connect to an Exchange server in the current site.x.x.x. New-PSSession : [mail.abc.com] Connecting to remote server server.domain.com failed with the following error message :WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer server.domain.com

Issue:

When opening Exchange Management Shell in Exchange 2013 using the Administrator account it fails with the following error. However we can create a new account in the Organization Management Permission it will connect successfully.

Failed to connect to an Exchange server in the current site.

Enter the server FQDN where you want to connect.: SERVER.DOMAIN.COM

VERBOSE: Connecting to mail.*.*.

New-PSSession : [server.domain.com] Connecting to remote server SERVER.DOMAIN.COM failed with the following error message :

WinRM cannot process the request. The following error occurred while using Kerberos authentication: Cannot find the computer Server.domain.com. Verify that the computer exists on the network and that the name provided is spelled correctly. For more information, see the about_Remote_Troubleshooting Help topic.

At line:1 char:1

+ New-PSSession -ConnectionURI “$connectionUri” -ConfigurationName Microsoft.Excha …

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : OpenError: (System.Manageme….RemoteRunspace:RemoteRunspace) [New-PSSession], PSRemotingTransportException

    + FullyQualifiedErrorId : NetworkPathNotFound,PSSessionOpenFailed

[PS] C:Windowssystem32>

Resolution:

• Check if there is any permission missing for the Administrator account.
  
• Try deleteting the Administrator profile using the following steps:
  

1.     Launch=> Control Panel => System and Security => System
   
2.        In the dialog click => Advanced system settings
   
3.        System Properties dialog will be displayed.
   
4.        Make sure you are in the “Advanced” register
   
5.        In User Profiles click =>Settings
   
6.        The “User Profiles” dialog is displayed
   
7.        Select administrator account and Delete it.
   

Exchange Replication service crashes in Exchange Server 2013 with even ID: 4999 & 4401

Issue:

The Microsoft Exchange Replication service crashes frequently after installing Cumulative Update 6 for Exchange Server 2013. We get the following event ID:

Log Name: Application
Source: MSExchange Common
Event ID: 4999
Task Category: General
Level: Error
Description:
Watson report about to be sent for process id: 14020, with parameters: E12IIS, c-RTL-AMD64, 15.00.0995.029, msexchangerepl, M.Exchange.Common, M.E.C.H.DatabaseFailureItem.Parse, System.ArgumentOutOfRangeException,version.
ErrorReportingEnabled: True

Log Name: Application
Source: MSExchangeRepl
Event ID: 4401
Task Category: Service
Level: Error
Description:
Microsoft Exchange Server Locator Service failed to find active server for database. Error: An Active Manager operation failed. Error: Invalid Active Manager Configuration. Error: Active Manager hasn’t completed configuration initialization.

Resolution:

• Installing Cumulative Update 7 for Exchange Server 2013
  

• Or run the following command in the Exchange server using command prompt:
  

  Wevtutil.exe cl “Microsoft-Exchange-MailboxDatabaseFailureItems/Operational”

Getting errors accessing Exchange Management Shell, Exchange Admin Console, Exchange Control Panel, or OWA in Exchange Server 2013

Issue:

• When users try to access EAC it fails with the following error:

  403 Sorry! Access denied L
  

  You don’t have permission to open this page. If you’re a new user or were recently assigned credentials, please wait 15 minutes and try again.

  We might get the following event ID:

  Log Name:      Application
  Source:        MSExchange Control Panel
  Event ID:      4
  Task Category: General
  Level:         Error
  Description:
  Request for URL ‘https://server.domain.com:444/ecp/default.aspx(https:/CASserver.domain.com/ecp/)’ failed with the following error:
  Microsoft.Exchange.Configuration.Authorization.CmdletAccessDeniedException: The user “domain.com/Computers/CAS1” isn’t assigned to any management roles.

• When users tries to access OWA it fails with the following error:

  L Something went wrong
  

  A problem occurred while you were trying to use your mailbox.

• Also when trying to access the Exchange management Shell it fails with the following error:

  VERBOSE: Connecting to CasServer.domain.com.
  New-PSSession : [CASserver.domain.com] Connecting to remote server CASserver.domain.com failed with the following error
  message : [Server= CASserver ,RequestId=xxxxxx,TimeStamp=xxxxxx] Access is
  denied.

  Resolution:

  • Remove the computer object from the restricted group.
    

  • You can run the following command to view all the groups denied permission
    

    Get-ADPermission -Identity <ExchangeServer> | where {($_.ExtendedRights -like “ms-Exch-EPI-Token-Serialization”) -and ($_.Deny -like “True”)} | ft -autosize User,ExtendedRights

Messages are stuck in a remote delivery queue in a Microsoft Exchange Server with event ID:1035 & queue viewer shows: 454 4.7.0 Temporary authentication failure.

Issue:

When few mails are stuck in the remote delivery queue with the following error: In Addition to it , event id: 1035 keeps logging.

451 4.4.0 Primary target IP address responded with: "454 4.7.0 Temporary authentication failure." Attempted failover to alternate host, but that did not succeed. Either there are no alternate hosts, or delivery failed to all alternate hosts.

Event Type: Error

Event Source: MSExchangeTransport

Event Category: SmtpReceive 

Event ID: 1035

Description:

Inbound authentication failed with error IllegalMessage for Receive connector Default <Server>. The authentication mechanism is ExchangeAuth. The source IP address of the client who tried to authenticate to Microsoft Exchange is xxx.

Resolution:

• Make sure that the clock on both servers and domain controllers display the same time.
  
• Check the Service Principal Name (SPN) for SMTPSVC is registered correctly on the target server. You can use SetSPN –L <Exchange server name> . Also its advisible to verify if there are any duplicate SPNs using the command SetSPN –x
  

• Check if ports required for Kerberos are enabled. We can add the following registry key:
  

  Open Regedit , navigate to HKEY_LOCAL_MACHINESYSTEMCurrentControlSetControlLsaKerberosParameters . Click New => DWORD Value=> enter the value as LogLevel => now right click & modify. => Edit DWORD value => Decimal. Add the Value data => 1. & click ok.
  

The Exchange Server 2013 Client RPC Service keeps on crashing.

Issue:

The Exchange server Client RPC service keeps crashing after a server reboot with Event ID 1006, Event ID 1025, Event ID 3010, Event ID: 1028

Resolution:

• Run the following command in Exchange Management Shell :
  

Set-RpcClientAccess -Server Bucksrv70 -EncryptionRequired $false

Set-RpcClientAccess -Server Bucksrv70 -EncryptionRequired $true

(These commands rebuilt the configuration file for the RPC Client Access Connector)

• Now Start the Service.
  

Unable to Start Exchange server Mailbox service using NT AUTHORITYNetwork Service.

Issue:

When installing Exchange 2013, Mailbox service fails to start using NT AUTHORITYNetwork Service. When checking Event viewer we will see a message exchange server could not access AD using account NT AUTHORITYNetwork Service.

Resolution:

• Check the permission in the ADSI EDIT & give read permission to this security groups : Authenticated users and Network Service account .
  

Updating Receive Connector in Exchange server gets overwritten by TMG

Issue:

If Exchange 2010 Server Edge role is installed with TMG 2010 & Forefront Protection for Exchange in the same machine, the Edge servers Default receive connector gets disabled.

Resolution:

• Enable TMG’s Integration mode to resolve the issue. This issue is caused by the Forefront TMG. During the TMG set up property IntegrationEnabled of the SmtpProtectionConfiguration COM object is wrongly set to TRUE by default.
  
• If you don’t prefer to use the intergartion mode, you can workaround by setting the Email Policy Integration mode to Disabled. ( in the TMG Console go to Troubleshooting => on the right pane click Control Email policy Configure Integration => Set the Email policy Integration mode to “Disabled“, save settings.

After running Windows Update in Exchange Server 2013 Event ID: 23 & Event ID: 258 appear.

Issue:

After running Windows Update in Exchange Server 2013, we keep getting Event ID 23 & Event ID 258. In addition to it we will not be able to open the ECP & will not be able to open Exchange Management Shell:

Resolution:

• Open Regedit & Navigate to HKEY_LOCAL_MACHINESOFTWAREMicrosoftPowerShell1.
  
• Create a Key named “PowerShellEngine” .
  
• Create all the following 6 Strings under “PowerShellEngine” and copy the values from Windows Server 2008: ApplicationBase, ConsoleHostAssemblyName, ConsoleHostModuleName, PowerShellVersion, PSCompatibleVersion and RuntimeVersion.
  
• Restart the server after creating the Key and Strings.

Unable to access Exchange Management Shell after migrating CAS role to another Server.

Issue:

After moving the CAS server role to HUB/MAILBOX server to have all roles in a single server, getting the following error when trying to access the Exchange Management Shell:

IIS 7.5 Detailed Error – 500.19 – Internal Server Error
The requested page cannot be accessed because the related configuration data for the page is invalid.
C:Program FilesMicrosoftExchange ServerV15ClientAccessPowerShell-Proxyweb.config

Resolution:

• Install WinRM-IIS-Ext feature .

Remove- InboxRule not working

Issue:

When trying to remove some Inbox rules, Exchange won’t allow & says there are duplicate inbox rules. When using the following command:

Remove-InboxRule -Mailbox Joe@Contoso.com -Identity “Project-MoveToFolderB”

Resolution:

Running the following command fixes the issue:

$mailboxes = Get-Mailbox -Organization myorganization

$mailboxes | % { Get-InboxRule -Mailbox $_.alias | ?{ $_.Forwardto -match “email@domain.com” } | Remove-inboxRule }

Ratish Nair

Microsoft MVP | Exchange Server

Team @MSExchangeGuru

Posted March 16th, 2015 under Exchange 2013.
RSS 2.0 feed.

Leave a response, or trackback.