Today's Issues

This past weekend, we applied an update to the mail environments that has successfully addressed the original issue. Thank you for your patience as we worked to strategically repair functionality and prevent further user impact.

While mail delivery has not been impacted, users may continue to experience the symptoms listed below. We have included remediation steps as well.

1. Unable to login to Webmail, POP, IMAP: These issues are auto-detected and fixed daily.

2. Additional folder named “lost-�: Users can move any email in these folders back to the original folder. If empty, extraneous “lost� folders may be safely deleted.

3. Missing folders or messages: Administrators should contact support with affected user’s email address and folder names, subject lines or dates of any messages that appear to be missing.

We have made progress in repairing the impacted mailboxes and will continue to run the processes listed below:

We auto detect the following issue by scanning system logs for errors that are generated when a user logs in to access their email:

- for those users who have not logged in since the system load issue, there is a small chance that an error in the Inbox will cause them to receive an error upon login. This issue may also present itself with intermittent errors when attempting to connect to a mail client using POP or IMAP protocols. We are continuously scanning for these errors and automatically fixing them several times each day.

Users may clean-up affected folders for the following issue:

- if a user has extraneous folders beginning with the word "lost-", any email in these folders can be moved back to the original folder and the extraneous folder may be deleted.

Administrators should contact the support team for the following issue:

- if a user believes that there may be missing messages or folders, the administrator should contact support with that user's email address & information regarding what appears to be missing. Because there are cases in which we cannot automatically detect this issue, contacting support is necessary.

We do not anticipate issuing further updates for this incident.

The primary focus early this week was to establish stable mail delivery and login access. While we have made significant progress, some work remains. Our current focus is on individual mailbox repairs due to issues indirectly caused when the system was under high load. Now that the system is stabilized, these mailbox repairs are processing more quickly. While we can automatically detect and remediate some issues, others require you to contact the support team.

There are certain issues which we are unable to detect If a user has not logged in. Because some users may go days or even weeks without logging in, we anticipate the symptoms described below will persist over a period of time. Administrators should contact the support team for the following issue:

- if a user believes that there may be missing messages or folders, the administrator should contact support with that user's email address & information regarding what appears to be missing. Because there are cases in which we cannot automatically detect this issue, contacting support is necessary.

We auto detect the following issues by scanning system logs for errors that are generated when a user logs in to access their email:

- for those users who have not logged in since the system load issue last week, there is a small chance that an error in the Inbox will cause them to receive an error upon login. This issue may also present itself with intermittent errors when attempting to connect to a mail client using POP or IMAP protocols. We are continuously scanning for these errors and automatically fixing them several times each day.

- if a user has extraneous folders beginning with the word "lost-" these errors are automatically detected by our log scan. While this requires no action on your part to be cleaned, the user may also safely delete them if they do not contain any email. If a folder containing email is deleted, users may check the Trash or Recover Deleted Items folders to retrieve the email.

These processes will continue to run through the weekend. We expect to provide the next update on Monday, July 24, 2017.

We continue to perform mailbox level cleanups to address the following issues, which we have identified proactively from system logs:

- mailboxes with login/access issues: this issue may cause intermittent errors for users that connect via POP or IMAP.

- extraneous folders prefixed with "lost-"

- missing messages or folders: the information is present, but not visible to email programs, including Webmail.

We recommend that those users who experience intermittent errors connecting via POP or IMAP utilize Webmail instead.

While we are working to resolve all symptoms, our highest priority is ensuring all affected users have access to their mailbox.

We will post the next update related to the mailbox cleanup efforts by 17:30 CDT/ 22:30 GMT on Thursday, July 20, 2017.

The initial passes to fix affected mailboxes were successful and the hosted email environment has stabilized.

We will continue to actively run the bulk mailbox level cleanup to address the following issues we have identified proactively from system logs:

- missing messages or folders; the information is present but not visible to email programs including Webmail
- extraneous folders starting with "lost-"
- mailboxes that we detect as having login/access issues

Some mailboxes may require additional attention to fully resolve, and our team is actively working through those.

We will provide another update to the mailbox cleanup efforts tomorrow afternoon.

Our initial pass of fixes for mailboxes that we detected as having login/access issues was successful. We will run a pass for newly discovered mailboxes that our logs have indicated are having issues.

Once complete, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:

- missing messages or folders; the information is present but not visible to email programs including Webmail
- extraneous folders starting with "lost-"

As we continue to progress through our mailbox cleanup efforts, some users may see slowness within Webmail.

Some mailboxes may require additional attention to resolve some of the issues experienced, and our team is actively working through those.

We will provide updates as we progress through these cleanup efforts.

We are continuing to focus on maintaining access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load.

We have begun running a fix to address the issues for users that have not yet been able to successfully access their mailbox. During this time, some users may see some slowness within Webmail.

Once completed, we will begin the bulk mailbox level cleanup to address the issues we have identified proactively from system logs. These issues include:

- missing messages or folders; the information is present but not visible to email programs including Webmail
- extraneous folders starting with "lost-"

Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.

We will continue to work on these mailboxes through the evening. We will provide our next update during the morning of Tuesday, July 18th.

Over the weekend we have worked through a system-wide cleanup to address the after-effects of the high system load experienced last week. The load was the result of an unknown bug that impacted an effort to proactively double system capacity. Please see the system status update posted 06:49 PM EDT on 07/15/17 for a brief, preliminary review of the causes of the system disruption.

Our priority today is to maintain good access and performance while assisting individual users who may have specific issues arising as a result of the previous high system load. In an abundance of caution, we are limiting the number of simultaneous connections to a single mailbox to 2 POP connections and 4 IMAP connections. Mail programs that exceed the limit will usually retry and users are unlikely to see errors due to connection limits.

We are also rate limiting email delivery for mailboxes that receive hundreds or thousands of email messages per minute; that email will safely queue for delivery over a longer period of time. If a user normally receives less email than this, they should see normal delivery speeds.

After peak load hours today, usually finished before 2pm Eastern, we will consider a bulk mailbox level cleanup to address several issues we have identified proactively from system logs. These issues include:

- some users have not yet been able to successfully access their mailbox
- missing messages or folders; the information is present but not visible to email programs including Webmail
- extraneous folders starting with "lost-"

Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.

We will continue to provide updates as they become available.

Our efforts this weekend have been focused on system wide cleanup to address the consequences of high system load after expanding capacity. These efforts are nearing completion. We will soon begin a more detailed mailbox-level cleanup effort to address some issues that we have seen among some users. While those updates happen there may be very brief periods where accessing email is slow or gives an error; trying again a few minutes later should be successful.

The mailbox level cleanup will address, in an automated fashion, the following issues for users we have been able to identify proactively:

- some users have not yet been able to successfully access their mailbox
- missing messages or folders; the information is present but not visible to email programs including Webmail
- extraneous folders starting with "lost-"

Some mailboxes may require additional attention to resolve some of the issues experienced and our team is actively working through those.

We will continue to provide updates as they become available.

Previous updates have been focused on resolution status as we worked through issues from the evening of Wednesday, July 12, 2017 through today, Saturday, July 15. We have compiled a preliminary view into the causes behind the system issues this week, and will have a final report available in the coming weeks. We recognize that these issues were significant for many of our customers, and we apologize for the difficulties you have experienced.

Here is a summary of what happened:

- We identified an issue with system performance during the week of July 3. While minor, we were concerned that it had the potential to grow worse over time. After discussing the issue with our vendors and consultants, we elected to perform proactive maintenance designed to double our total capacity. The additional capacity would provide plenty of headroom and prevent the issue.

- On the evening of Wednesday, July 12, the capacity was ready, and engineers initiated the process to rebalance users across the new hardware.

- An unknown bug associated with moving users caused high system load across affected systems, and impacted only those users who had been moved.

- During the time of impact, when an affected user tried to access their mailbox, they would have seen access errors, incomplete message listings, or other errors. Mail delivery was also slowed for a time, and this would have affected a broader set of users.

- Email messages were not lost or destroyed, but would have been inaccessible by some users during the time of impact.

- Our efforts through the end of the week of July 10 and into this weekend have been focused on reducing and shifting load, and then preparing to automatically clean up side effects where possible.

The high system load directly caused by the bug is now resolved, and performance and access have improved. We are in the process of preparing our systems to do a set of automated fixes for issues caused as side effects - the most serious of these are messages or folders that appear to be missing, but are actually present.

Please contact us if you have specific questions or concerns about your users.

We know that the email system is important to you and will continue to provide updates when we have more to share.

We have completed another phase of repairs and have initiated the next phase. During this time, some users may experience sporadic delays along with Webmail, POP & IMAP connectivity issues.

We will continue to provide updates as they become available.

We have made progress in repairing the impacted mailboxes. Some users may experience sporadic delays today while the repair continues.

We will continue to provide updates as they become available.

System load is progressing towards normal levels and most users should find the email system operational. Until the system is fully restored, users who experience occasional timeouts and connection errors should refresh their connection.

- Folder or Message Visibility Issue: Affected users have partial or no access to their email and/or folders are able to log in, but do not see all expected folders or messages. Engineers are scanning system logs to detect these issues and compiling a list of affected users. We have prepared an automated fix to be applied following restoration of systems and a complete database cleanup.

- Folder Deletion Issue: Affected users are unable to delete folders in Webmail, receiving the error Message: "The selected folder does not exist...." This issue is related to the Visibility Issue noted above and will be remediated in parallel.

- "Lost" Folders Issue: Affected users note a group of folders that have the word "lost" in the folder name, followed by a string of characters. Example: "lost-asfkjl3q5kjafls". This issue will be addressed for many users during the weekend. If the issue persists following the weekend, please contact a member of your support team.

- POP/IMAP Setup & Webmail Last Page Issues: Affected users receive an error when trying to configure a mail client or navigate to last page in Inbox in Webmail (error code undefined). These issues are related to the timeouts and connection errors noted above. Currently, most issues are resolved. Any remaining issues should be resolved once load returns to normal levels.

- Message duplication issue: Affected users may experience duplication of recently sent messages. This issue is a side effect of the system issue. Users can safely delete the duplicates. If manual deletion of a large number of duplicate messages is impractical, please contact a member of your support team, as we can assist with de-duplication once the system is fully restored and our maintenance is complete.

- Unread Items Issue: For affected users, a number of messages or entire folder changed from "Read" to "Unread". Affected users may safely mark the messages as Read. We urge those users who find it impractical to mark a large number of messages as Read to contact a member of your support team, as we can assist with this task once the system is fully restored and our maintenance is complete.

We will continue to provide updates as they become available.

Impacted users no longer need contact their support team to provide mailbox details.

We continue to monitor the environment and will apply fixes to resolve the aforementioned symptoms for affected users. We will continue to provide updates as they become available.

Progress is being made in making adjustments to provide relief to impacted users. While we make these changes, users may continue to experience the aforementioned symptoms. We will continue to provide updates as they become available.

We are continuing to monitor the environment and are making adjustments to provide relief to impacted users. While we make these changes, users may continue to experience email delivery delays.

We encourage those users who are experiencing any of the following specific persistent issues to contact their support team for resolution:

- Unread Items Issue: A large number (or entire folder) of messages changed from "Read" to "Unread"

- Missing Subfolders Issue: Missing Subfolders removed without intervention. (These folders are hidden and not actually missing)

- "Lost" Folders Issue: a group of folders that have the word "lost" in the folder name, followed by a string of characters. Example: "lost-asfkjl3q5kjafls"

- POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to last page in Inbox in Webmail (error code undefined).

- Folder Deletion Issue: Users unable to delete folders in Webmail. Error Message: "The selected folder does not exist...."

- Webmail Access Issue: Users unable to access their email through Webmail or Outlook, could receive error message web hard hat or "an error occurred ...", depending on the operation.

- Message duplication issue: Users may experience duplication of recently sent messages.

We will continue to provide updates as they become available.

We continue to monitor the environment as more users access the system throughout the day and are doing everything we can to drive system load down and improve responsiveness.

Users may experience:
- Latency in receiving and accessing mail or mail folders, - POP/IMAP access issues, and/or

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

In addition, there are persistent issues with a small portion of mailboxes due to an emergency maintenance performed on 12 July 2017 at 21:00 EDT. Symptoms include, but are not limited to, the following:

- Unread Items Issue: A large number (or entire folder) of messages changed from "Read" to "Unread" - Missing Subfolders Issue: Missing Subfolders removed without intervention. (These folders are hidden and not actually missing)

- "Lost" Folders Issue: a group of folders that have the word "lost" in the folder name, followed by a string of characters. Example: "lost-asfkjl3q5kjafls"

- POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to last page in Inbox in webmail (error code undefined).

- Folder Deletion Issue: Users unable to delete folders in Webmail. Error Message: "The selected folder does not exist...."

- Webmail Access Issue: Users unable to access their email through webmail or Outlook, could receive error message web hard hat or "an error occurred ...", depending on the operation.

- Message duplication issue: Users may experience duplication of recently sent messages.

The next update will be provided by 13:30 EDT / 17:30 PM GMT.

As system load increases throughout the day users may experience:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

- Latency in accessing mail or mail folders.- POP/IMAP access issues.- Delayed email delivery.

There are persisting issues with a small portion of mailboxes due to the emergency maintenance that was performed at 9:00 PM EDT 7/12/2017. Symptoms include but are not limited to:

- Unread Items Issue: A large number (or entire folder) of messages has changed from "Read" to "Unread"

- Missing Subfolders Issue: Missing Subfolders that seemingly disappeared without intervention. (These folders are hidden and not actually missing)

- "Lost" Folders Issue: User will report a group of folders that have "lost" in the folder name followed by a string of characters "Ex: lost-asfkjl3q5kjafls"

- POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to last page in Inbox in webmail (error code undefined).

- Folder Deletion Issue: Users unable to delete folders in Webmail. Error received will be "The selected folder does not exist...."

- Webmail Access Issue: Users unable to access their email through webmail or Outlook and receive a hard hat or "an error occurred ..." message.

- Message duplication issue: Users may experience duplication of recently sent messages.

The next update will be provided by 12:00 PM EDT / 4:00 PM GMT.

At this time, message delivery times and webmail responsiveness have remained stable but we will continue to monitor the environment as system load increases throughout the day. As system load increases throughout the day users may experience:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."
- Latency in accessing mail or mail folders.
- POP/IMAP access issues.

There are persisting issues with a small portion of mailboxes due to the emergency maintenance that was performed at 9:00 PM EDT 7/12/2017. Symptoms include but are not limited to:

- Unread Items Issue: A large number (or entire folder) of messages has changed from "Read" to "Unread"

- Missing Subfolders Issue: Missing Subfolders that seemingly disappeared without intervention. (These folders are hidden and not actually missing)

- "Lost" Folders Issue: User will report a group of folders that have "lost" in the folder name followed by a string of characters "Ex: lost-asfkjl3q5kjafls"

- POP / Folder Issue: Error when trying to configure POP in a mail client. Unable to navigate to last page in Inbox in webmail (error code undefined).

- Folder Deletion Issue: Users unable to delete folders in Webmail. Error received will be "The selected folder does not exist...."

- Webmail Access Issue Users unable to access their email through webmail or Outlook and receive a hard hat or "an error occurred ..." message.

The next update will be provided by 10:00 AM EDT / 2:00 PM GMT.

Message delivery times remain at normal levels.

System performance has stabilized to normal levels.

We will continue to monitor the environment as system load increases throughout the day.
As system load increases throughout the day users may experience:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

- Latency in accessing mail or mail folders.

- POP/IMAP access issues.The next update will be provided by 8:00 AM EDT / 12:00 PM GMT.

Message delivery times remain at normal levels.

We continue to improve system performance across our Hosted Email environment.

During this increased system load, users may continue to see the following:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

- Latency in accessing mail or mail folders.

- POP/IMAP access issues.

The next update will be provided by 5:00 AM EDT / 9:00 AM GMT.

Message delivery times have been restored to normal levels.

We continue to monitor system load and actively improve system performance across our Hosted Email environment.

During this increased system load, users may continue to see the following:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

- Latency in accessing mail or mail folders.

- POP/IMAP access issues.

The next update will be provided by 3:00 AM EDT / 7.00 AM GMT.

We continue to monitor system load, and are actively improving system performance across our Hosted Email environment.

During this increased system load, users may continue to see the following:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

- Latency in accessing mail or mail folders.

- Delivery delays continue to fall and are nearing resolution.

- POP/IMAP access issues.These symptoms will resolve themselves once the issue has been resolved.

The Next update will be provided by 1:00 AM EDT / 05.00 AM GMT.

We continue to monitor system load, and are actively improving performance across our Hosted Email Platform.

During this increased system load, users may continue to see the following:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

- Latency in accessing mail or mail folders.

- Delivery delays (currently 10 minutes).

- POP/IMAP access issues.These symptoms will resolve themselves once the issue has been resolved.

The next update will be provided by 10:00 PM EDT.

No additional updates or ETA available at this time.

Our teams continue to work on resolving the issue.

Users may continue to experience the symptoms below.

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

- Latency in accessing mail or mail folders- Delivery delays (approximately 1.3 hours)

- Intermittent access to POP/IMAP

These symptoms will resolve themselves once the issue has been resolved.

The next update will be provided by 7:00 PM EDT.

Our teams continue to actively work on reducing load and returning the system to normal operation.

In the meantime, users affected by this may notice the following:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

- Latency in accessing mail or mail folders
- Delivery delays (approximately 37 minutes)
- Intermittent access to POP/IMAP

These specific symptoms will resolve themselves once the issue has been resolved.

Another update will be posted by 4:00 PM EDT.

Our teams are actively working to reduce load and return to normal operation.

In the meantime, users affected by this may also experience the following:

- Error Message: "Webmail: "[SERVERBUG] Internal error occurred. Refer to server log for more information."

- Latency in accessing mail or mail folders

- Delivery delays (approximately 22 minutes)

These specific symptoms will resolve themselves once the issue has been resolved.

Another update will be posted at 1:00 PM CDT.

Due to an emergency maintenance that was performed at 9:00 PM EDT 7/12/2017, a portion of users may notice a disruption of service with their Hosted Email mailbox.

Some of the symptoms that are being reported would include:

- Latency when navigating the Webmail interface- Slowness upon logging in to the Webmail interface

- Delayed email delivery

- POP Error 'Your incoming (POP3) email server has reported an internal error. If you continue to receive this message, contact your server administrator or Internet service provider (ISP). The server responded: - ERR [SYS/TEMP] Internal error occurred. Refer to server log for more information.

- Users unable to delete folders in Webmail "The selected folder does not exist...."

- Messages previously marked as read are now unread.

- "Lost Folders" appear in webmail

- Delayed email delivery

We are aware of an issue that is affecting our Hosted Email environment. More details will be posted as they become available.