XPLIVE

Additional Monitoring

2009-01-22T17:00:00.000-08:00

We've added Chicago and New York to our list of active monitors. After the long lasting DNS outage that has occured on the West Coast of the USA today, more monitoring was necessary to get a better picture of the wider internet availability of the service.

View the full report links to see how the service has performed since April of 2008.

Unplanned DNS Outage at 12:10 Pacific

2009-01-22T12:44:00.001-08:00

XPLive and its related web sites experienced an unplanned DNS outage that apparently started at 12:10 PM Pacific. Our San Francisco monitor was triggered at that time and our support staff started to diagnose the problem. It appeared to take about 20 minutes for the error to unroll and the planner was again available by 12:30 PM Pacific.

This did not trigger our London monitor, so it appeared to be a US West Coast anomaly related to DNS. We were able to remotely log into the literal IP addresses of the domains without any problems during the outage.

You may experience some slow load times on the login page. This is the result of the SSL Site Seal from Network Solutions loading slowly. We are working to remove that site seal from the login page.

[Update] 1:30 PM Pacific: It appears as though the AT&T network is experiencing some problems with routing packets through their US West Coast presence. Users outside of the USA appear to be unaffected, while users on the West Coast of the USA are experiencing intermittent timeouts caused by packet delay.

UTF8 Encoding Fix

2008-07-02T22:13:00.000-07:00

We have updated the planner to support UTF-8 encoding properly. This appears to have been broken since we added a display tag filter to our filter chain a few months ago. Our customers using Arabic characters reported that their display characters were not being preserved correctly, and thus the need for this update.

With the release from today, this appears to be fixed. Yet, if you have any display issues with your localized language, please let us know by sending email to support@extremeplannerlive.com.

Updated SSL Certificates

2008-06-26T21:42:00.001-07:00

We updated our SSL certificates as part of our yearly renewal. The new certificates should continue to work with all browsers. If there are any problems with SSL or identity verification, please contact support@extremeplannerlive.com. The SSL certificate update affected the planner site as well as the commerce site.

60 Minute Outage

2008-06-06T09:41:00.000-07:00

Today we updated the kernel on our load balancer and attempted a warm reset of the system. That, unfortunately, did not work well, and the load balancer did not recover nicely. What should have been a five minute reset turned into 60 minutes because we had to physically reset the load balancer (cold reset) to make it recover.

Lesson learned and we'll be doing kernel updates with warm resets on-site rather than remotely.

Maintenance Window June 7th

2008-06-02T19:43:00.000-07:00

On June 7th between 00:00 and 02:00 Pacific time (GMT-8), we will be making changes to the primary routing infrastructure of our network. During this time there may be periods of higher latency and possible service loss. We recommend that all users make sure that this time period is not critical for their workflow. Please accept our apologies in advance of any inconveniences caused by this planned maintenance. Know that these changes are designed to improve the upstream capacity and speed of the network.

Uptime Report

2008-05-31T21:31:00.000-07:00

As of May 31st, 2008, we've been actively monitoring the XPLive system from two continental locations (USA and Great Britain). This constitutes about 61 days of active monitoring of the service since the last major outage.

The uptime report shows that it is running at:

99.969%, or 27.23 minutes of down time total (commerce site)
99.988%, or 10.54 minutes of down time total (planner)
99.981%, or 16.69 minutes of down time total (planner)

Our average amount of down time is 18.15 minutes, which is about 18 seconds per day, relative to the source of the monitoring.

This monitoring doesn't take into account network issues that occur between the monitoring system and our servers. We have tried to black-out planned maintenance windows so that those outages do not skew the uptime reports.

VisualStudio 2008 Support

2008-05-31T21:25:00.000-07:00

We are pleased to announce support for Microsoft Visual Studio 2008 in our plugin for XPLive. The installer version 1.3.7 installs for both VS2005 and VS2008. You can get the updated installer at: XPLive.

Please note that if you get an error when VisualStudio loads after installing the plugin, then likely you are in a locale that we do not support. Send an email to support@extremeplannerlive.com and let us know your locale so that we can distribute an update to the installer.

Network Maintenance Scheduled

2008-05-16T10:23:00.000-07:00

One of our hosting facilities has scheduled two maintenance windows where additional bandwidth capacity will be added and some routine upgrades of routers will occur. These windows will occur on May 22nd between 9AM and 1PM GMT, and then on May 23rd between 8AM and 10AM GMT.

Each of these windows should not produce any loss of connectivity, and should take less than 20 minutes each to complete. During the windows, please plan for the unexpected and try to schedule your usage accordingly.

Quota Whine Emails

2008-05-13T12:28:00.001-07:00

Starting May 13th, 2008, accounts that are over-quota in their storage will get whine emails on a nightly basis. We are working on more stringent controls over the storage quota system to prevent rampant abuse of our lenient and generous enforcement policies. While we had hoped that self-policing of the storage quota was enough enforcement, the reality is that many accounts did not police themselves. Therefore, we are going to enforce the quota strictly and send you email notifications when you exceed the quota.

Storage Price Reduction

2008-05-13T12:23:00.001-07:00

Many customers have asked why we charged so much for our storage. At $1.38 per MB, we were far above the industry average. In response, we've lowered our price for storage to $0.25 per MB. This new price point reflects a growing trend in the marketplace where storage costs are low. Yet, we continue to face increased bandwidth and power costs associated with storing large volumes of data. We hope that this new price point for storage will be much more useful for our valuable customers.

Web Usage Statistics

2008-05-13T12:21:00.000-07:00

Starting in May of 2008 you will be able to see your web utilization for your account. We are analyzing our logs on a per-customer basis so that you can see the amount of usage that your account is getting. This is another free value-added service that we offer to you that makes the management process easier!

Version 3.2 Upgrade

2008-04-17T08:08:00.001-07:00

We updated the service to ExtremePlanner 3.2 over the weekend of April 13th, 2008. This update went smooth and with little interruption to service. The monitoring service reported about 3 minutes of downtime during the update.

You can see the list of changes in version 3.2 by visiting the ExtremePlanner Web Site.

During the upgrade we made configuration changes to our application server. These changes inadvertently broke compatibility with the Eclipse plugin. Compatibility has been restored as of April 16th, 2008.

We hope that you enjoy the updated version of the planner. If you find any bugs or usability problems with this release, please send us email.

Cisco IOS Failure

2008-04-07T07:07:00.000-07:00

Many of you already know that we had two outages in March, one that was about an hour long, and another that was 4 hours long. These outages, at first, appeared to be the result of administrative incompetence. When the second attack occured, though, we soon realized that this was the result of a sophisticated exploit on a bug in the Cisco IOS used to manage our BGP routers.

CERT issued a vulnerability notice on March 27, 2008. You can read it here (VU936177).

Since this IOS update release, the attacks have been successfully managed and we are regaining our confidence in our colocation partners.

This type of attack can affect any ISP in any part of the world. For that reason, we are carefully screening our backup ISP providers to ensure that they have the proper updates applied to their infrastructure. We can't afford to subject our customers to another one of these attacks when it can be prevented with a bug fix to software.

We are working to expand our reliability infrastructure and provide outstanding service to all of our customers. Our screening process does take time, though, and we need to make sure that our decision does not adversely affect the distributed information sharing that all of our customers require.

24/7 Monitoring

2008-04-07T07:00:00.000-07:00

We have implemented 24/7 monitoring and early outage reporting through SiteUptime (www.siteuptime.com). The connectivity issues we had in March made us realize that our current local monitoring startegy did not allow us to react to the failures adequately.

The SiteUptime monitoring service will be monitoring from San Francisco and London. If necessary, we'll expand that monitoring to include other key Internet relay hubs that represent more of our global customer base.

XPLive is dedicated to keeping your service operational 24/7. We can not guarantee 100% availability because we are at the mercy of our vendors, such as Cisco. More changes to our reliability infrastructure are coming.

The Beginning

2008-03-19T19:58:00.000-07:00

This is the official blog of XPLive, the hosted service for ExtremePlanner. If you are wondering what these are, please visit XPLive or ExtremePlanner.

On this blog you will find information about the network status, business operations, and helpful hints for the XPLive service.