Exchange 2003 transaction log files filling up very quickly


----------------

I haven’t blogged about any IT-related issues in a while, and I thought that the solution to this little-known exchange 2003 problem would probably interest the SysAdmin crowd among you guys.

If one day, you log into your inbox and find an email waiting for you announcing that the disk containing your exchange 2003 transaction log files has almost no more disk space, you may be experiencing the following issue.

As I’m sure you’re aware, the E00*.log files located in your \exchsrvr\MDBDATA are transaction logs containing everything that happened to your Information Store since it was last backed up. As soon as you perform a backup of the store using an exchange-aware agent, the log files will be cleared. However, in this particular situation, the transaction files are filling up the drive too quickly and will consume all of the remaining disk space before you have the time to back up your store.

In this case, you may want to check out your Outlook 2007 clients that have the “exchange cache mode” option enabled. The station that causes the problem will probably have a large number of “sync issues” in the Outlook folder of the same name. I’m not sure of this though. The client who was causing the issue happened to have this problem. Can anyone confirm this? After locating the rogue workstation, the only thing you have to do to fix the problem is to disable the “exchange cache mode”. If you want to re-enable it after, be sure to delete the old OST file and have Outlook create a new one.

But after fixing the issue, your disk will likely still be out of space, so you’ll need to clear those log files before mounting the exchange store again.

WARNING: NEVER, EVER delete the E00 log files manually. The proper way to remove them in this situation is:

1- Make a full backup of your exchange store using an exchange-aware agent…. OR
2- Enable Circular logging. (Don’t forget to disable the option and restart the information store service after you get your disk space back)

Voilà! Now if your Exchange transaction log files folder start filling very quickly, this little tutorial will give you an extra thing to check for to solve your problem!







36 Responses to Exchange 2003 transaction log files filling up very quickly

  1. Have you ever had the logs fill up due to different issues? Also did you experience the logs filling up and eating the Hard Drive space on just one day out of the week? Please let me know. Thank you.

  2. Have you ever had the logs fill up due to different issues? Also did you experience the logs filling up and eating the Hard Drive space on just one day out of the week? Please let me know. Thank you.

  3. Oh Yeah, there’s A LOT of different possibilities. One of the possibilities is that a message is caught in a never ending loop, and keeps bouncing around your infrastructure…

    One of hte thing you can do to see if a client is the problem is to open your exchange system manager and go to the mailbox section (where you see all your users mailbox)… now on the RIGHT section of the manager, add the “total ops” or “ops” column… if you see a client generating A LOT more operations than the others, this will probably be your culprit..

    Let me know if it works!

  4. Oh Yeah, there's A LOT of different possibilities. One of the possibilities is that a message is caught in a never ending loop, and keeps bouncing around your infrastructure…

    One of hte thing you can do to see if a client is the problem is to open your exchange system manager and go to the mailbox section (where you see all your users mailbox)… now on the RIGHT section of the manager, add the "total ops" or "ops" column… if you see a client generating A LOT more operations than the others, this will probably be your culprit..

    Let me know if it works!

  5. I’m looking in the System Management and do not see or can’t add a column called OPS or total ops. Is this something that has to be added to Exchange?

  6. I'm looking in the System Management and do not see or can't add a column called OPS or total ops. Is this something that has to be added to Exchange?

  7. Ok got it, sorry, it’s under the Logon section, –

    administrative groups – first administrative groups – server – yourservername – first storage group – mailbox store – logons

    From there, go on the “View” menu on top, and select “add /remove column”.. then add the column named “Total Ops

    After, sort your “ops” column, and if one user is making *WAY* more operations than the others, you’ll probably have your culprit.

  8. Ok got it, sorry, it's under the Logon section, –

    administrative groups – first administrative groups – server – yourservername – first storage group – mailbox store – logons

    From there, go on the "View" menu on top, and select "add /remove column".. then add the column named "Total Ops"

    After, sort your "ops" column, and if one user is making *WAY* more operations than the others, you'll probably have your culprit.

  9. Do you remember which client version your issue system had? I was wondering if this had something to do with it.

  10. Do you remember which client version your issue system had? I was wondering if this had something to do with it.

  11. No. The client version under logons. (System Manager.)
    Examples include 12.0.6300.5000, 11.0.8200.0, ect.)

    • Hmmm, no idea.. I’m in vacation right now and don’t have access to the console.. but this should be available under all versions of the EX2k3 manager.. I run the latest SP with all the most recent patches..

  12. No. The client version under logons. (System Manager.)

    Examples include 12.0.6300.5000, 11.0.8200.0, ect.)

    • Hmmm, no idea.. I'm in vacation right now and don't have access to the console.. but this should be available under all versions of the EX2k3 manager.. I run the latest SP with all the most recent patches..

  13. I have a few good questions to ask you. Did you issue happen a particular day of the week, or every day. The second question is did the rouge computer you found, was the Outlook client up and running when it was filling up the logs or was the user logged off? Please let me know. Thank you.

    • The size of the logs were only growing when the client was up… if the computer was powered down, everything was going fine.. The only thing I had to do was to delete and reconfigure the outlook 2007 profile on the client that was causing the problem..

      After, I had to do an offline defrag to shrink down the size of the store.

  14. I have a few good questions to ask you. Did you issue happen a particular day of the week, or every day. The second question is did the rouge computer you found, was the Outlook client up and running when it was filling up the logs or was the user logged off? Please let me know. Thank you.

    • The size of the logs were only growing when the client was up… if the computer was powered down, everything was going fine.. The only thing I had to do was to delete and reconfigure the outlook 2007 profile on the client that was causing the problem..

      After, I had to do an offline defrag to shrink down the size of the store.

  15. Say the user is not logged into their machine but the machine is still on. Would this still cause the logs to fill up? And our issue is that this, the issue, for us only happens on Sundays. Any reason why it would do that? Thank you.

  16. Say the user is not logged into their machine but the machine is still on. Would this still cause the logs to fill up? And our issue is that this, the issue, for us only happens on Sundays. Any reason why it would do that? Thank you.

  17. Hmmm, in my situation, as soon as outlook was close, the problem stopped occuring.

    Is it possible that someone has an event that only runs on sunday? Something like a recurring task… I heard that rules can sometimes cause this kind of problem too..

  18. Hmmm, in my situation, as soon as outlook was close, the problem stopped occuring.

    Is it possible that someone has an event that only runs on sunday? Something like a recurring task… I heard that rules can sometimes cause this kind of problem too..

  19. We’ve had this problem several times in the last couple of months. It seems to be caused by a corrupted OST and the quickest way to fix is to find the offending user (ExMon is good for this), shut down Outlook, delete their ost file and open Outlook again. This will rebuild it and should sort the problem. All the users were using Outlook 2007, and at least half of them reported a mail being stuck in their outbox when the problem occurred.

    It doesn’t seem to fall into any kind of pattern either, we’ve had this happen on different days of the week and at different times.

    More worryingly than just transaction logs filling up, we’ve also found that it can quickly fill up a store with blank space. We’re running Exchange 2003 Enterprise edition so fortunately we’re not being hit with store size limits, but disk space is a problem when this kicks off and only recreating stores and juggling mailboxes can reclaim that space.

    I’m hoping MS manage to fix this one and get a patch for Outlook 2007 soon, else this could cause major problems if it keeps on happening.

    • Have you gat a response from Microsoft.
      It looks we are have the same problem, suddenly Exchange is filling the log disk, until is dismounts the stores.

  20. We've had this problem several times in the last couple of months. It seems to be caused by a corrupted OST and the quickest way to fix is to find the offending user (ExMon is good for this), shut down Outlook, delete their ost file and open Outlook again. This will rebuild it and should sort the problem. All the users were using Outlook 2007, and at least half of them reported a mail being stuck in their outbox when the problem occurred.

    It doesn't seem to fall into any kind of pattern either, we've had this happen on different days of the week and at different times.

    More worryingly than just transaction logs filling up, we've also found that it can quickly fill up a store with blank space. We're running Exchange 2003 Enterprise edition so fortunately we're not being hit with store size limits, but disk space is a problem when this kicks off and only recreating stores and juggling mailboxes can reclaim that space.

    I'm hoping MS manage to fix this one and get a patch for Outlook 2007 soon, else this could cause major problems if it keeps on happening.

    • Have you gat a response from Microsoft.

      It looks we are have the same problem, suddenly Exchange is filling the log disk, until is dismounts the stores.

  21. I have just had the same problem my system has started generating 40gig of log files a day for 100 users we only have a 30gig edb file . i’m running ex server 2003 all clients are outlook 2007. I’m having trouble trying to identify the offending client.

    Any suggestions would be helpfull

  22. I have just had the same problem my system has started generating 40gig of log files a day for 100 users we only have a 30gig edb file . i'm running ex server 2003 all clients are outlook 2007. I'm having trouble trying to identify the offending client.

    Any suggestions would be helpfull

  23. For this business i usually use next utility-Recovery Toolbox for Outlook,also as how as i remember tool has free status,it restore all your contacts, emails, meetings and tasks are typically stored on server,permit to open *.ost files and convert them into *.pst files, that can be opened by any program, compatible with Microsoft Outlook email client,allows opening ost files and to store your personal data independently from Microsoft Exchange Server, it is very important, if you’d like to access your contacts and email archive from outside of your company,can extract the files in *.vcf, *.eml and *.txt formats.

  24. For this business i usually use next utility-Recovery Toolbox for Outlook,also as how as i remember tool has free status,it restore all your contacts, emails, meetings and tasks are typically stored on server,permit to open *.ost files and convert them into *.pst files, that can be opened by any program, compatible with Microsoft Outlook email client,allows opening ost files and to store your personal data independently from Microsoft Exchange Server, it is very important, if you’d like to access your contacts and email archive from outside of your company,can extract the files in *.vcf, *.eml and *.txt formats.

  25. A couple things.

    1. You may want to add a note that the process will not start deleting the log files until the current log file finishes. To expedite the process, you can send yourself an email with a 5MB attachment (assuming you’re using the default log size).

    2. You said to backup the info store BEFORE enabling circular logging. According to Microsoft…

    “You cannot replay log files unless all uncommitted log files from the time the database was last running are available. You must have all log files starting from the checkpoint at the time the database was backed up. You can then replay log files from this point as long as they follow an unbroken sequence. If there is a single log file missing in the middle or from the beginning of the sequence, replay stops there.”

    It would seem to me, that based on this, it would be proper practice to run a backup of the info store immediately AFTER disabling circular logging.

    Am I missing something? This is not meant to be a flame by any means. I am no expert in this regard.

  26. A couple things.

    1. You may want to add a note that the process will not start deleting the log files until the current log file finishes. To expedite the process, you can send yourself an email with a 5MB attachment (assuming you're using the default log size).

    2. You said to backup the info store BEFORE enabling circular logging. According to Microsoft…

    "You cannot replay log files unless all uncommitted log files from the time the database was last running are available. You must have all log files starting from the checkpoint at the time the database was backed up. You can then replay log files from this point as long as they follow an unbroken sequence. If there is a single log file missing in the middle or from the beginning of the sequence, replay stops there."

    It would seem to me, that based on this, it would be proper practice to run a backup of the info store immediately AFTER disabling circular logging.

    Am I missing something? This is not meant to be a flame by any means. I am no expert in this regard.