Sacl watcher servicelet encountered an error while monitoring sacl change

old DC failed:-
removed from sites and services
removed from AD
check any remnants in ADSI Edit
should there be anything else?

heres the event (ARTY is the failed DC)

Log Name:      Application
Source:        MSExchange SACL Watcher
Date:          07/08/2012 07:32:26
Event ID:      6003
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EXCHANGE.domain.local
Description:
SACL Watcher servicelet encountered an error while monitoring SACL change.
Got error 1722 opening group policy on system ARTY.domain.local in domain works.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchange SACL Watcher" />
    <EventID Qualifiers="49152">6003</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-08-07T06:32:26.000000000Z" />
    <EventRecordID>270494</EventRecordID>
    <Channel>Application</Channel>
    <Computer>EXCHANGE.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>ARTY.domain.local</Data>
    <Data>works</Data>
    <Data>1722</Data>
  </EventData>
</Event>

Open in new window

im also getting this quite often too —  ive checked and the exchange server is in that group in AD

Log Name:      Application
Source:        MSExchangeSA
Date:          07/08/2012 07:18:35
Event ID:      9385
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      EXCHANGE.domain.local
Description:
Microsoft Exchange System Attendant failed to read the membership of the universal security group '/dc=local/dc=domain/ou=Microsoft Exchange Security Groups/cn=Exchange Servers'; the error code was '8007203a'. The problem might be that the Microsoft Exchange System Attendant does not have permission to read the membership of the group. 

If this computer is not a member of the group '/dc=local/dc=domain/ou=Microsoft Exchange Security Groups/cn=Exchange Servers', you should manually stop all Microsoft Exchange services, run the task 'add-ExchangeServerGroupMember,' and then restart all Microsoft Exchange services. 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="MSExchangeSA" />
    <EventID Qualifiers="49152">9385</EventID>
    <Level>2</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2012-08-07T06:18:35.000000000Z" />
    <EventRecordID>270487</EventRecordID>
    <Channel>Application</Channel>
    <Computer>EXCHANGE.domain.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>/dc=local/dc=domain/ou=Microsoft Exchange Security Groups/cn=Exchange Servers</Data>
    <Data>8007203a</Data>
  </EventData>
</Event>

Open in new window

Thanks Guys

Approximately every 3 days, Exchange loses contact with all the domain controllers and fails. The only way to resolve the issue is to restart the server. Once restarted, it functions perfectly with clean event logs right up until the next failure. I’ve been working on this for around 4 weeks now, since the problem began, but I am unable to find the root cause. I am also unable to tie the start of these problems with any particular change to the configuration of the network.

We have 2 x 2008R2 domain controllers and a domain running at 2008R2 functional level. Exchange 2010 is also installed on Server 2008R2, and all three of these servers are virtualised on VMWare ESXi 4.1.

——

The last good application event log entry looks like this:

Event 2080, MSExchange ADAccess

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1992). Exchange Active Directory Provider has discovered the following servers with the following characteristics:
(Server name | Roles | Enabled | Reachability | Synchronized | GC capable | PDC | SACL right | Critical Data | Netlogon | OS Version)
In-site:
DC01.domain.com CDG 1 7 7 1 0 1 1 7 1
DC02.domain.com CDG 1 7 7 1 0 1 1 7 1

Then an error:

Event 1009, MSExchangeMailSubmission

The Microsoft Exchange Mail Submission service is currently unable to contact any Hub Transport servers in the local Active Directory site. The servers may be too busy to accept new connections at this time.

Another error:

Event 6003, MSExchange SACL Watcher

SACL Watcher servicelet encountered an error while monitoring SACL change.
Got error 1722 opening group policy on system DC01.domain.com in domain domain.com

A warning:

Event 1007, MSExchange Mailbox Replication

The Mailbox Replication service was unable to determine the set of active mailbox databases on a mailbox server.
Mailbox server: EXCHANGE.domain.com
Error: MapiExceptionNetworkError: Unable to make admin interface connection to server. (hr=0x80040115, ec=-2147221227)

An informational event:

Event 2070, MSExchange ADAccess

Process STORE.EXE (PID=5012). Exchange Active Directory Provider lost contact with domain controller . Error was 0x80040951 (LDAP_SERVER_DOWN (Cannot contact the LDAP server)) (). Exchange Active Directory Provider will attempt to reconnect with this domain controller when it is reachable.

An error:

Event 2104, MSExchange ADAccess

Process STORE.EXE (PID=5012). Topology discovery failed due to LDAP_SERVER_DOWN error. This event can occur if one or more domain controllers in local or all domains become unreachable because of network problems. Use the Ping or PathPing command line tools to test network connectivity to local domain controllers. Run the Dcdiag command line tool to test domain controller health.

A warning:

Event 2121, MSExchange ADAccess

Process STORE.EXE (PID=5012). Exchange Active Directory Provider is unable to connect to any domain controller in domain domain.com although DNS was successfully queried for the service location (SRV) resource record used to locate a domain controller for that domain.
The query was for the SRV record for _ldap._tcp.dc._msdcs.domain.com
The following domain controllers were identified by the query:
dc02.domain.com
dc01.domain.com

Meanwhile, the system log shows:

Event 5719, NETLOGON

This computer was not able to set up a secure session with a domain controller in domain DOMAIN due to the following:
The RPC server is unavailable.
This may lead to authentication problems. Make sure that this computer is connected to the network. If the problem persists, please contact your domain administrator.

——

From this point onwards the event log just fills up with errors and warnings and all the Outlook clients get kicked off. From the Exchange server, via RDP, I can ping both domain controllers no problem.

One other error I am seeing is an SMB error when I try to browse the network from the Exchange server (while it has failed). I get a message: «The name limit for the local computer network adapter card was exceeded«. The documentation I have found for this is quite old and the suggested registry key changes for TcpTimedWaitDelay and MaxUserPort are already set as recommended.

Once the server has been restarted, I have little avenue for further investigation as the event logs run clean and everything seems fine. Even when the server fails, everything else on the network functions perfectly and there are no errors in the domain controllers’ event logs.

I’ve been down numerous avenues here, but I’m running out of ideas and I would really really appreciate some help with this problem.

Many thanks in advance,
Steve…

I spent a few hours today trying to figure out why the AD tools (AD Users & Computer, AD Sites & Services, etc) were not working on an Exchange 2010 server, running on Windows 2008 R2, today. The odd thing was that Exchange was working fine but the following errors were being generated in the Application log about every five minutes.

Event ID: 6003

Source: MSExchange SACL Watcher

SACL Watcher servicelet encountered an error while monitoring SACL change.

Got error 1722 opening group policy on system wfsad02.company.local in domain company.

DCDIAG was also failing on multiple tests, see “DCDIAG results” at the end of this post. The odd thing was that DCDIAG test were working fine against DC outside of the AD Site the Exchange server was in.

After running multiple tests from different servers this server was the only one having these issues. So I then decided to check some NIC settings and discovered the issue:

The above settings would be OK if this NIC was being used for iSCSI communications. But for client traffic ALL of the above should be checkedenabled. For DAG replication traffic TCP/IPv4, TCP/IPv6, and the two Link-Layer options should also be checked.

So to break the AD tools and cause DCDIAG errors just uncheck these options. After doing this you won’t find much help searching for the errors as I found out. So I wrote this post to hopefully help others who have a misconfigured NIC on a Windows server.

DCDIAG results

Testing server: WFSWFSAD01

Starting test: Advertising

Fatal Error:DsGetDcName (WFSAD01) call failed, error 1722

The Locator could not find the server.

……………………. WFSAD01 failed test Advertising

Starting test: SysVolCheck

[WFSAD01] An net use or LsaPolicy operation failed with error 1231,

The network location cannot be reached. For information about network troubleshooting, see Windows Help..

……………………. WFSAD01 failed test SysVolCheck

Starting test: MachineAccount

Could not open pipe with [WFSAD01]:failed with 1231:

The network location cannot be reached. For information about network troubleshooting, see Windows Help.

Could not get NetBIOSDomainName

Failed can not test for HOST SPN

Failed can not test for HOST SPN

Starting test: NetLogons

[WFSAD01] An net use or LsaPolicy operation failed with error 1231,

The network location cannot be reached. For information about network troubleshooting, see Windows Help..

……………………. WFSAD01 failed test NetLogons

Starting test: Services

Could not open Remote ipc to [WFSAD01.company.local]: error 0x4cf

“The network location cannot be reached. For information about network troubleshooting, see Windows Help.”

……………………. WFSAD01 failed test Services

Running enterprise tests on : company.local

Starting test: LocatorCheck

Warning: DcGetDcName(GC_SERVER_REQUIRED) call failed, error 1722

A Global Catalog Server could not be located – All GC’s are down.

Warning: DcGetDcName(PDC_REQUIRED) call failed, error 1722

A Primary Domain Controller could not be located.

The server holding the PDC role is down.

Warning: DcGetDcName(TIME_SERVER) call failed, error 1722

A Time Server could not be located.

The server holding the PDC role is down.

Warning: DcGetDcName(GOOD_TIME_SERVER_PREFERRED) call failed, error 1722

A Good Time Server could not be located.

Warning: DcGetDcName(KDC_REQUIRED) call failed, error 1722

A KDC could not be located – All the KDCs are down.

……………………. company.local failed test LocatorCheck

• Note: Per KB2512643 the DFSREvent, FrsEvent, KccEvent, & SystemLog “The RPC server in unavailable” expected when Windows Firewall is enabled on DCs
• I skipped these test using the following command, so their results were not included in the above DCDIAG output:
  dcdiag /s:wfsad02 /e /skip:kccevent /skip:systemlog /skip:DFSREvent /skip:FrsEvent