Seeing a lot of duplication in journaling mailboxes while using standard MS Exchange Journaling?

Are you seeing duplicates in MS Exchange Journal mailboxes ?
Are you using standard journaling on Exchange 2007 or Exchange 2010 ?
Mostly seeing duplicate when emails are sent to large distribution lists ?

If answers to above mentioned questions is 'YES' then there are chances it is working as designed even though when reading some MS Articles about Journaling and transport journaling might confuse you a little. I am not trying to argue with MS articles (http://technet.microsoft.com/en-us/library/bb738122(v=EXCHG.80).aspx) , I am just trying to explain how that can be broken down for understanding:

There are two scenarios where you see duplication:

1. Sites with 1 Journal Mailbox: In this case you will see that when an email is sent to distribution list especially if that distribution list contained thousands of users then you will see approximately journaling mailbox having bifurcated copies that will be using formula of

Number of recipients/1000

This is because by default in Exchange 2010 and Exchange 2007 within EdgeTransport.exe.config file variable "ExpansionSizeLimit" is set to 1000. There is no garuntee through each bifurcated copy will have exact 1000 recipients information in its envalope. Sometimes you will see nine hundred something and sometimes it would few more than 1000.

Now when an email is sent to a distribution group and only one journal mailbox is configured in your environment you will see multiple copies of the message. What happens if that due to above setting in place, to make routing/delivery quicker message is broken into multiple copies by explanding the distribution list. The tests have I have apprears to suggest that breakdown is sorted alphabatically. MessageID for every single copy in journaling mailbox will be same, just the recipient information on the envalope will be different.

Above mentioned setting can be changed BUT with caution as it can impact your system if it is in production. Reffer to above mentioned article for more information.

2. Sites with more than 1 journal mailbox: This scenario gets a little more tricky. In this scenario you might even see that not only journaling mailbox that is responsible for journaling from DB where sender is location had journaled multiple copies but other journaling mailboxes also has multiple copies. If you have distribution groups with many thousand users and many journal mailboxes to spead the load of journaling you might see a lot more duplicates. Here is how I have put it in my tick skull:

(a) When a sender sends the email to the distibution list (DL) with large number of users. The journal mailbox responsible for journaling DB where sender's mailbox is located will bifurcate message to many messages in the journaling mailbox where it is journaling. Now to this point we are all good as mentioned above.

 

(b) Now if we go and check other journal mailboxes we may find similar if not exact number of messages sitting in it. The reason is that there are other mailbox DBs that are journaling to other journaling mailboxes. Usually they are only meant to have bifurcated copies of the messages that had users belonging to mailbox DBs in question within the envelop of the message. What happens is that many a times we do not divide users based on their names while assigning them mailbox DBs. Especially for a company with global presence exchange servers could be country specific or region specific. Now when first transport server would have created the bifurcated messages all other servers would have got all the copies due to one or more user being present in the envaople of the each bifurcated message. For testing in house I did the following which might explain what happened:

- CREATED USERS ON MULIPLE DBs

TEST_User1 - EXCHANGE\First Storage Group\Mailbox Database
TEST_User2 - EXCHANGE\First Storage Group\Mailbox Database
TEST_User3 - EXCHANGE\First Storage Group\Mailbox Database
TEST_User4 - EXCHANGE\Third Storage Group\Mailbox Database
TEST_User5 - EXCHANGE\Third Storage Group\Mailbox Database
TEST_User6 - EXCHANGE\Third Storage Group\Mailbox Database
TEST_User7 - EXCHANGE2\First Storage Group\Mailbox Database
TEST_User8 - EXCHANGE2\First Storage Group\Mailbox Database
TEST_User9 - EXCHANGE2\First Storage Group\Mailbox Database
TEST_User10 - EXCHANGE2\First Storage Group\Mailbox DB2
TEST_User11 - EXCHANGE2\First Storage Group\Mailbox DB2

EXCHANGE\First Storage Group\Mailbox Database   - Journal_01_2007
EXCHANGE\Third Storage Group\Mailbox Database   - Journal_02_2007
EXCHANGE2\First Storage Group\Mailbox Database  - Journal_03_2007
EXCHANGE2\First Storage Group\Mailbox DB2       - Journal_04_2007

TEST ALL - Distribution Group with all 11 users created above.

Set value of ExpansionSizeLimit to 2 and restarted Transport Service on both Exchange servers

Test CASE 1 - Sent an email from KATWAL_Service account to itself in To and "TEST ALL" in BCC

Following are the number of messages found in journal mailboxes
Journal_01_2007 = 7 messages
Journal_02_2007 = 2 messages
Journal_03_2007 = 2 messages
Journal_04_2007 = 2 messages

Breakdown of recipients on the envalipes per journal mailbox

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Journal_01_2007 = 7 messages

MSG1
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_User11@katwal.local, Expanded: TESTALL@katwal.local

MSG2
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_user10@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User9@katwal.local, Expanded: TESTALL@katwal.local

MSG3
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_User8@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User7@katwal.local, Expanded: TESTALL@katwal.local

MSG4
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_User6@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_user5@katwal.local, Expanded: TESTALL@katwal.local

MSG5
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_User4@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User3@katwal.local, Expanded: TESTALL@katwal.local

MSG6
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_user2@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User1@katwal.local, Expanded: TESTALL@katwal.local

MSG7
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
To: Katwal_service@katwal.local
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Journal_02_2007 = 2 messages

MSG1
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_User6@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_user5@katwal.local, Expanded: TESTALL@katwal.local

MSG2
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_User4@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User3@katwal.local, Expanded: TESTALL@katwal.local
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Journal_03_2007 = 2 messages

MSG1
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_user10@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User9@katwal.local, Expanded: TESTALL@katwal.local

MSG2
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_User8@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User7@katwal.local, Expanded: TESTALL@katwal.local
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Journal_04_2007 = 2 messages

MSG1
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_User11@katwal.local, Expanded: TESTALL@katwal.local

MSG2
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434D@EXCHANGE.katwal.local>
Bcc: TEST_user10@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User9@katwal.local, Expanded: TESTALL@katwal.local
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


=====================================================================================
TEST CASE 2 - We configure all DBs on each exchange server to journal to only one journal mailbox. That way we have One journal mailbox per server.

Configured Journaling in the following manner

EXCHANGE\First Storage Group\Mailbox Database   - Journal_02_2007
EXCHANGE\Third Storage Group\Mailbox Database   - Journal_02_2007
EXCHANGE2\First Storage Group\Mailbox Database  - Journal_04_2007
EXCHANGE2\First Storage Group\Mailbox DB2       - Journal_04_2007

- Tested that changes made took place in exchange
- ExpansionSizeLimit is still set to 2
- Sent email to TEST ALL in BCC and Katwal_Service in TO list

Journal_02_2007 = 7 messages
Journal_04_2007 = 3 messages

- 7 messages breakdown is similar to what we found earlier.
- Breakdown of 3 messages in Journal_04_2007 are as follows.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Journal_04_2007 = 3 messages (from bottom to top)
MSG1
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434F@EXCHANGE.katwal.local>
Bcc: TEST_User8@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User7@katwal.local, Expanded: TESTALL@katwal.local

MSG2
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434F@EXCHANGE.katwal.local>
Bcc: TEST_user10@katwal.local, Expanded: TESTALL@katwal.local
Bcc: TEST_User9@katwal.local, Expanded: TESTALL@katwal.local

MSG3
Message-Id: <D91E43950FB2744E94C5A4E6C78E815606857157434F@EXCHANGE.katwal.local>
Bcc: TEST_User11@katwal.local, Expanded: TESTALL@katwal.local
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

=====================================================================================

TEST CASE 3 - To confirm what we found in the testing above is true; we configured all DBs to journal into Journal_04_2007

EXCHANGE\First Storage Group\Mailbox Database   - Journal_04_2007
EXCHANGE\Third Storage Group\Mailbox Database   - Journal_04_2007
EXCHANGE2\First Storage Group\Mailbox Database  - Journal_04_2007
EXCHANGE2\First Storage Group\Mailbox DB2       - Journal_04_2007

- There were only 7 messages in the Journal_04_2007 which were created the same way in above test cases.


In my testing number of mailboxes were very small leading to small number of bifurcated messages. Number of bifurcated messages will depend on configuration usesed at your site.

(NOTE: I hope others will find it useful. All above mentioned content is based on my understaning and testing performed. If you see any issues in your environment you may want to consult software vendor of yours.)