BESx 5 & The RefreshGALUsingMAPI Error of Doom!

So my boss messages me to say his Blackberry is playing up and randomly sending some emails, receiving some emails, not syncing the calendar, etc.  The usual fix is to sign in to Bes Manager, find the user, find the device and then push a full sync.  After having done this three times and the same issues occurring I noticed something very odd.

The Users account had marked itself for Blackberry Services removal.  No one else has access to this service so that intrigued me but also worried me.  At the same time, my blackberry also started to play up.  I deleted the users account, waited 5 minutes for it to clear and tried to re-add it but to no avail, it claimed the user still existed.  At this point I decided it would be best to restart all the Blackberry services.  In to Admin/Services.  I stopped all the Blackberry ones, then tried to restart them.  This is where all BESx failed monumentally.  All of a sudden in my eventlog I started to see this repeatedly:

MailStoreExchange::RefreshGALUsingMAPI – Error in HrFindExchangeGlobalAddressList, result=0x80004005.
[ODBCRecord:DeleteRecord] No rows found to delete.  SQL = “DELETE FROM MsAddresses WHERE MailStoreDomainId = ? AND SequenceNumber > ?”.
[ODBCRecord:DeleteRecord] DATA = “MailStoreDomainId[1] SequenceNumber[0] “.

Well that was not a good sign, and then to add to fold this one started appearing too from MS Exchange 2007:

* Source: MSExchangeSA
* Event Log: Application
* Type: Error Event
* Event ID: 9385
* Microsoft Exchange System Attendant failed to read the membership of the
universal security group ‘/dc=com/dc=Domain/ou=Microsoft Exchange Security
Groups/cn=Exchange Servers’; the error code was ‘8007203a’.
The problem might be that the Microsoft Exchange System Attendant does not
have permission to read the membership of the group.

If this computer is not a member of the group
‘/dc=com/dc=domainname/ou=Microsoft Exchange Security Groups/cn=Exchange
Servers’, you should manually stop all Microsoft Exchange services, run the
task ‘add-ExchangeServerGroupMember,’ and then restart all Microsoft Exchange
services.

Now between Microsoft and Research In Motion’s technical support, I got underway trying to resolve the issues.  First thing I did was log in as the BesAdmin account locally and try and sort out the MAPI config in the way RIM suggest.  Firstly stopping all the Blackberry Services, I fired up regedit, drilled down to HKEY_CURRENT_USER\Software\MicrosoftWindows NT\CurrentVersion\Windows Messaging Subsystem\Profiles, and deleted it!  Cue the absolute horror as the server hung for over 20 mins and regedit just sat there saying NOT RESPONDING.  I eventually bit the bullet and killed the regedit process, dove back in to the registry, expanded out the profiles subkeys and time consumingly deleted each subkey (over 100 for some reason) which took ages.

Next I fired up the Blackberry Server Configuration Tool, Blackberry Server Tab, and hit Edit MAPI.  I whacked in the server and the mailbox name and hit Check Name…. To be confronted with an error stating that the mailbox can not be found.  Highly frustrated I came to the conclusion that it wasn’t BESx all at fault here so started down the “let’s fix MS Exchange 2007 first” route.

Fixing error 9385 isn’t actually all that difficult it turned out.  After checking all the permissions and discovering that Microsoft tell you to move the server back to the USERS OU on the A/D, it turns out you need to move the server to the Computers OU and then restart the Microsoft Exchange System Attendant service.  Once this is done, Error 9385 will disappear.

By resolving this, it also resolved my issue with MAPI not being able to look up the BES ADMIN mailbox.  I got that sorted, and restarted the Blackberry services.  Let the EventLog Viewer become awash in:

agent x: will not restart – reached the maximum of 10 restarts per 24 hours.

&

MailStoreExchange::RefreshGALUsingMAPI – Error in HrFindExchangeGlobalAddressList, result=0x80004005.

I obviously hadn’t quite cleared the RefreshGALUsingMAPI issue.  A lot more research led me to the fact that the Global Catalog (GC) was probably not defined in the registry under the MAPI profile area.  Regedit up, back down to Profiles and then in to:

HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows Messaging Subsystem\Profiles\BlackBerryServer\dca740c8c04210 1ab4b908002b2fe182

Add a string value called 001e6602 and set the value to a FQDN of a Global Catalog server.  Go back and stop all your Blackberry Services.  Now start them all back up, making sure the Blackberry Controller and Blackberry Dispatcher services come up first.  Fingers crossed it should all be resolved!