status.ReadyWire.com

Status Notification System

16. Jul, 2010

In an effort to better serve you, ReadyWire will be performing maintenance on Turbine, our uptime notification on Monday, July 19th.

During the maintenance window, our automated systems may post, tweet, and email out notifications that servers/services are unavailable. Note: Should a real outage or issue occur, we will reschedule this maintenance window for another day.

WINDOW: 7/19 — 12:00pm CDT thru 2:00pm CDT

CUSTOMER IMPACT: No downtime will occur as a result of this work.

Please do not hesitate to open a ticket should you have questions regarding this post.

[RESOLVED] Incident- ReadyWire Website/Help Desk

07. May, 2010

ReadyWire datacenter technicians are investigating an issue with our corporate web server.  As soon as additional info is available, we’ll post it here.  (Update:  9:07AM CDT – 5/7/10).

This issue has been resolved.  We’re sorry for any inconvenience!  (UPDATE:  9:30AM CDT – 5/7/10).

[COMPLETED] Corporate Servers- Scheduled Maintenance

30. Apr, 2010

ReadyWire datacenter technicians will be performing routine maintenance on our corporate servers during the following window of time:

DATE:  Saturday, May 8th 2010
TIME:12:00AM thru 1:00 AM CDT
CUSTOMER IMPACT:  No clients will be impacted by this maintenance window.  However, because a reboot of a number of servers will be required, customers may have a difficult time reaching the the customer portal, our website, and our VOIP phone system.  Total downtime is expected to be no more than 15 minutes, so we’ll be back online as quickly as possible.  Because this blog is hosted in a remote location, you should still be able to reach http://status.readywire.com/, where we will post additional information once we start our work.

If you have any questions before, during, and/or after this maintenance window, please do not hesitate to open a support ticket by visiting https://secure.readywire.com/.

UPDATE: Due to this morning’s events, this maintenance alert has been canceled.

[COMPLETED] Adams- Scheduled Maintenance

30. Apr, 2010

ReadyWire datacenter technicians will be performing routine maintenance on Adams during the following window of time:

DATE: Saturday, May 8th 2010
TIME: 12:00AM thru 1:00 AM CDT
CUSTOMER IMPACT: Although we do not expect clients to be greatly impacted by this maintenance window, ReadyWire technicians will need to reboot the server.  As a result, up to fifteen minutes of downtime should be expected.

If you have any questions before, during, and/or after this maintenance window, please do not hesitate to open a support ticket by visiting https://secure.readywire.com/.

UPDATE: We have started our work on Adams.  As soon as work has been completed, we’ll update this thread.  (12:27AM CDT 5-8-10)

UPDATE: Work on this VPS node has been completed.  We’re rebooting all of the containers now.  Thank you so much for your patience and understanding!  (12:40AM CDT on 5-8-10).

[COMPLETED] Jefferson- Scheduled Maintenance

30. Apr, 2010

ReadyWire datacenter technicians will be performing routine maintenance on Jefferson during the following window of time:

DATE: Saturday, May 8th 2010
TIME:12:00AM thru 1:00 AM CDT
CUSTOMER IMPACT: Although we do not expect clients to be greatly impacted by this maintenance window, ReadyWire technicians will need to reboot the server.  As a result, up to fifteen minutes of downtime should be expected.

If you have any questions before, during, and/or after this maintenance window, please do not hesitate to open a support ticket by visiting https://secure.readywire.com/.

UPDATE: We have started our work on Jefferson.  As soon as work has been completed, we’ll update this ticket.  (12:18AM CDT 5-8-10)

UPDATE: Work on Jefferson has been completed.  Thanks for your patience and understanding!    (12:25AM on 5-8-10)

[COMPLETED] Washington- Scheduled Maintenance

30. Apr, 2010

ReadyWire datacenter technicians will be performing routine maintenance on Washington during the following window of time:

DATE: Saturday, May 8th 2010
TIME:12:00AM thru 1:00 AM CDT
CUSTOMER IMPACT: Although we do not expect clients to be greatly impacted by this maintenance window, ReadyWire technicians will need to reboot the server.  As a result, up to fifteen minutes of downtime should be expected.

If you have any questions before, during, and/or after this maintenance window, please do not hesitate to open a support ticket by visiting https://secure.readywire.com/.

UPDATE: Work on Washington has been completed.  Thanks for your patience and understanding!    (12:16AM on 5-8-10)

[COMPLETED] Network Maintenance- April 28th 10-11PM

27. Apr, 2010

Our datacenter will be performing emergency maintenance work on the network.

Maintenance Window: 04/28/10 2200 CDT – 04/28/10 2300 CDT

Expected Impact: 15 minutes of downtime while cabinet switch is replaced

Maintenance Description: Network staff will be replacing a switch that has been experiencing hardware issues in order to avoid further unscheduled downtime.  Some customers may experience an interruption in service as the switch is replaced.

Please note: Every effort is made to accurately represent the maintenance window, time frame, and level of customer impact. However, due to factors outside our control we may affect customer connectivity and, need to either start maintenance late or complete the maintenance window late. Thus, the time frames, and customer effects specified above, should be considered approximate and subject to change.

If after the maintenance you have any concerns regarding your service, please do not hesitate to contact us by submitting a ticket.

UPDATE:  We have received word from the datacenter that this went off without a hitch!  Thanks for your patience and understanding.

[RESOLVED] Phone Difficulties

18. Apr, 2010

ReadyWire is currently encountering issues with our VOIP server.  As a result, we are unable to make or receive phone calls.  If you are in need of technical assistance, please open a support ticket by visiting https://secure.readywire.com/.

(Last Updated: 4/18/10 — 1:11PM CDT)

UPDATE: Our phone system is back online and we’re now able to accept calls. (4/19/10 – 6:27PM CDT)

[RESOLVED] Power Outage at NOC

15. Apr, 2010

Our network operations center has been without power now for several hours and ComEd has yet to provide us with any real indication as to when we will have service restored.  As a result, we are unable to offer telephone support at this time.  As soon as power has been restored, we’ll update this ticket.  In the mean time, our technicians are working remotely this evening in an effort to provide the usual level of technical support.  If you are in need of assistance, please do not hesitate to open a support ticket by visiting https://secure.readywire.com/.

Please note, our NOC is NOT located within the same location as our datacenter.  This power outage is NOT impacting any customer servers.

Last Updated:  7:30PM CDT

UPDATE: After hours and hours of being without power, ComEd has finally restored power to our offices.  If you have been trying to reach us, our call center is now open and ready to speak with you.  (2:30PM CDT 4/16/10)

[RESOLVED] Washington- Apache inaccessable

11. Apr, 2010

We have received reports that Apache on Washington is not responding.  We have esclated this issue to our datacenter team and hope to have an update ASAP!  Thank you for your patience.  (4/10/10 – 11:47PM CDT)

UPDATE: Our datacenter team has informed us that the server has been rebooted in an effort to get service back online as quickly as possible.  We’ll bring another update as soon as possible.  (4/11/10 — 12:02AM CDT)

UPDATE: Washington is back online, however the server load is still a little high as things are quickly returning online:

root@washington [/scripts]# uptime
00:13:13 up 11 min,  1 user,  load average: 10.70, 8.39, 4.02

This should return to normal within the next 10 to 15 minutes.  Our datacenter team will continue to investigate the cause of this incident and will post an update here shortly.  We sincerely apologize for this disruption to your ReadyWire account.  (4/11/10 — 12:14AM CDT)

UPDATE: Upon further investigation, it appears as if this server encountered a rather large amount of traffic between the hours of 9PM CDT and the time the server went down for a reboot at 12:02AM CDT.  We have identified the associated accounts that were the target of such traffic and they have been suspended as a result.  Please note that ReadyWire takes such service disruptions seriously and any accounts that contribute to such a disruption will be dealt with accordingly.

Running a shared hosting server is a lot like managing an apartment rental community.  In order to provide for a quality living situation, order must exist within the community.  Loud music late at night and glass bottles by the pool are not allowed at most rental communities and hosting accounts that cause issues with other accounts are not allowed at ReadyWire.

All in all, please know that our Abuse/Security department is in the process of notifying customers involved and we’ll continue to monitor the situation throughout the night to ensure that this doesn’t occur again in the future.  We sincerely appreciate your understanding.  If you would like one-on-one assistance regarding this situation, please do not hesitate to open a support ticket by emailing support@readywire.com or by visiting http://secure.readywire.com.  (4/12/10 — 12:45AM CDT)