WP.com Downtime Summary
Today WordPress.com was down for approximately 110 minutes, our worst downtime in four years. The outage affected 10.2 million blogs, including our VIPs, and appears to have deprived those blogs of about 5.5 million pageviews.
What Happened: We are still gathering details, but it appears an unscheduled change to a core router by one of our datacenter providers messed up our network in a way we haven’t experienced before, and broke the site. It also broke all the mechanisms for failover between our locations in San Antonio and Chicago. All of your data was safe and secure, we just couldn’t serve it.
What we’re doing: We need to dig deeper and find out exactly what happened, why, and how to recover more gracefully next time and isolate problems like this so they don’t affect our other locations.
I will update this post as we find out more, and have a more concrete plan for the future.
I know this sucked for you guys as much as it did for us — the entire team was on pins and needles trying to get your blogs back as soon as possible. I hope it will be much longer than four years before we face a problem like this again.
Update 1: We’ve gathered more details about what happened. There was a latent misconfiguration, specifically a cable plugged someplace it shouldn’t have been, from a few months ago. Something called the spanning tree protocol kicked in and started trying to route all of our private network traffic to a public network over a link that was much too small and slow to handle even 10% of our traffic which caused high packet loss. This “sort of working” state was much worse than if it had just gone down and confused our systems team and our failsafe systems. It is not clear yet why the misconfiguration bit us yesterday and not earlier. Even though the network issue was unfortunate, we responded too slowly in pinpointing the issue and taking steps to resolve it using alternate routes, extending the downtime 3-4x longer than it should have been.
- February 19, 2010
- WordPress.com
WordPress.com News 
thanks for being honest about a free service, what an usual thing in today’s world. Taking responsibility and truthful are traits I cherish the most and you made the cut.
LikeLike
Among the best things about wordpress.com during these last 4 years has been the up time. Thanks for communicating with us swiftly and bringing the site back up as quickly as was humanly possible. In investigating and creating a partial shutdown strategy fro the first areas affected is a good idea, and I’m looking forward to hearing the investigation results.
LikeLike
This was definitely the longest downtime I’ve experienced since I’ve been with WordPress, but outages are few and far between. The advantages of being hosted here still greatly outweigh any negatives. Thanks for getting things back up and going, and thank you for the real time updates via Twitter!
LikeLike
Thanks for fix and for letting us know…
I feel very happy that my blog is in the hands of folks who care so much.
LikeLike
Well, I was one of the users who didn’t even notice, which surprises me. Your transparency is impressive. That’s what I’ll remember. Furthermore, it’s high time I thanked you for nearly three years of free hosting and a fine blogging platform that keeps getting better.
LikeLike
thanks for the prompt explanation guys! 🙂
LikeLike
I’m just logging in now, [20:48, EST] so, Gosh, I missed all the Excitement! Thanks for the announcement in the sidebar that it has happened. I’m confident that you will find the bug soon. meanwhile, won’t it be interesting if all 10 million of us post these ‘get well cards!’
🙂
LikeLike
Thanks Matt and the rest of the team.
For a while I was horrified and I thought it may be the same kind of attack by Iranian Cyber Army which hacked twitter.com before. Iranian state hates so much blogosphere (in fact my own blog has just been filtered & blocked inside iran) that I am afraid they try to attack wordpress.com as well.
Please be vigilant which I am sure you are.
Good luck
Bamdad
LikeLike
110 minutes including WordPress VIP! Certainly the longest downtime. I missed it! I keep researching on hosting and enjoy monitoring uptime! Bad for Techcrunch who recently shifted to VIP!
I have about 12 active blogs on wp.com and I am still happy! 🙂
LikeLike
My first down time! Thanks for your hard work getting things back online. WordPress is the best!
LikeLike
I can live with downtime every 4 years.
If you look at it like… 110 minutes, out of the 2,102,400 minutes across 4 years.
That does not suck.
LikeLike
Very impressed with how you guys are keeping people informed of the outage. Good work getting things back up and running!
LikeLike
Guys, I don’t know if I am the only one, but I posted something just after the system came back. I added a picture with a link to its source and it seems impossible to make this right. I will explain myself: I want to link to http://www.lidialuz.blogspot.com, but no matter what I do, it links to http://deepbrazil.com/www.lidialuz.blogspot.com
It never happened to me before. Is there any chance this was related to the episode?
Any tips?
Regina
LikeLike
Deep Brazil: Please contact support, they’ll be able to help: https://wordpress.com/support/contact/
LikeLike
Love the WP staff for being this transparent, you just earned a little more loyalty from all your users through this!
keep up the good work!
LikeLike
I’ll admit I freaked out briefly (15 minutes) as I’m addicted to the attention I get from my site. But I do regular back ups so I wasn’t worried about losing content.
Great job on getting things back up and for being so up front about the problem.
I appreciate your dedication and professionalism.
LikeLike
Great job WordPress team, we never doubted your abilities to bring it all back to life. We at SimplyCast.com thank you for providing us with a great service.
LikeLike
Miracles Happen too you know! (Or some Angel causes them) keep up the hard work
LikeLike
Matt: don’t stress. We can all handle being down for two hours. Thanks for keeping us informed, and don’t lose any sleep over this.
LikeLike
I am fairly new to word press and have been very impressed with the ease of starting a blog and the quality that comes with it. Thanks for getting back online so quick. Cheers!
LikeLike
आपली कामगिरी कौतुकास्पद आहे.थोड्याच वेळात सगळ ठाक ठिक केलेत आपण ! जे झालं ते आपण मनमोकळेपणाने सांगितलं हा आपला मोठेपणा आहे. धन्यवाद वर्ड्प्रेस डाट काम !!!
LikeLike
If only everyone fixed things fast and just said what happened! Thanks!
LikeLike
I’m new to this and didn’t know what was happening – so thanks for the update
LikeLike
Thank you . . .
It’s all good !
LikeLike
You were down? Thank you for the free blog that I wouldn’t have without you.
Sharky
LikeLike
Gold stars, high fives and COLD bevies all around!
Well done fixit-folks. You are loved!
LikeLike
Many Thanks to keep us update, may God bless you for your efforts
LikeLike
As soon as I figured out it was your problem, not my computer, I relaxed and began oozing empathy for you guys. It must have been a terrible, horrible, no good very bad time for you. Welcome back!
LikeLike
I am a new user and I was very concerned but feel much better knowing today’s event was the worst event in 4 years. Thanks for keeping us informed and responding with a sense of urgency. it’s a good example.
LikeLike
Is there affect to myblog?
I hope, everything will be better!
LikeLike
Akhmad: There was no data loss, we just had a network problem that prevented us from serving data yesterday. Your blog should be fine.
LikeLike
Thanks for keeping us informed. I much appreciate the high performance of WordPress.
LikeLike
You are the best, things happen, thanks for the info!
LikeLike
No sweat, you guys rock.
LikeLike
Glad I caught this; I was on LI at the time…and thought THEY had dropped the ball, with all of their goofy updates of late [as my Blog was all of a sudden not showing up there.] Glad you, i.e. ‘we’, are “live” again…
LikeLike
Glad to know it wasn’t an attack.Give us an update when you find out what happened.
Your AutoSave feature saved me some minor aggravation, too. Rock on.
LikeLike
great crisis communication… congrats! and thanks for informing us of the reasons and what you’re trying to fix it!
LikeLike
It is sure nice of you to let us know what happened. Stuff happens. We’re just glad you guys are great at figuring things out and getting it straightened out. Thank you and keep up the good work.
LikeLike
Thanks for the update. Great recovery.
LikeLike
Great work. I feel that when such downtime occurs, it is equally important that WordPress keep its user updated. So kudos to you people.
LikeLike
110 minutes in four years… that’s an AH-mazing track record. I’m new to WP and have loved every minute of it, and really appreciate everything the WP team does. You folks ROCK!
LikeLike
Transparency makes WordPress a better place… Thanks
LikeLike
You guys are great! Keep up the good work.
LikeLike
Went to the gym in the meantime, maybe a blessing in disguise! 🙂
LikeLike
wow, thanks. we were a little freaked out at the time, then realized that probably the right people were on the job. yay you! thanks for the valiant effort.
LikeLike
“Stuff happens” Thanks for paying attention to the details and keeping us informed. That is what really matters.
LikeLike
no wonder i found it hard to open my blog.,.,so is it the page views stats we received lately is fake?
thanks for the information
LikeLike
I didn’t noticed it. Probably it happened when I was asleep. Lucky me 🙂
Tnx for the message.
Brabosh,
Antwerpen / Belgium 5:17 am
LikeLike
All is well. We will survive. PRZ
LikeLike
Thanks for the update and getting it back up and running. Seeing that WP is awesome and I have not had any complaints since I have been blogging I still cant seem to find a reason to complain. Things happen and as inconvienent as they are at times the track record for WP is outstanding. Thanks again for giving us a place share our thougths and engage with others WP rocks.
LikeLike
Thanks for such constant hardwork.well done.
LikeLike
Not to worry. Worse things have happened over time. I shall stay loyal. Good luck! Richard
LikeLike
Thanks for the heads up. Currently trying to add a new post creates a never ending loop that starts and ends at the dashboard. Maybe tomorrow.
LikeLike
I didn’t notice it but thank you very much for the info! I appreciate your honesty and likewise hope it doesn’t happen again soon 🙂
LikeLike
Thanks for the update. . . it was weird alright, and I hated it. Thought my computer was going over the hill until I proved to myself, that I could reach other contacts online, so it became apparent the problem was over there. I wish I could say now that I prayed for you, but I didn’t – – I just sat here and growled. I’m sorry – I know better.
No harm done. All is well and you guys remain great in my book.
Life being what it is – there is bound to be a next time. But I know you will handle it and come out on top.
Thanks for all you do. Jan
LikeLike
Dear Matt,
I thank you for the briefing. I am a new user,but I am impressed by your frankness. It is all right.Triubles will crop up forever in some form or the other.They actually reinforce our resolve to do better.Tushar Bhatt,
Ahmedabad,India
LikeLike
Thanks for the heads up. Good luck on solving the problem!
LikeLike
Hey i think i missed the chance to experience this, just knew it after i read this post.
LikeLike
LikeLike
Any explanations forthcoming as to why the stats graph has disappeared? I’ve only had a blank space (day, week or month) now for at least FOUR DAYS! I can’t understand how that was allowed to happen, and no comment on it.
LikeLike
Please contact support, they’ll help you troubleshoot your stats: https://wordpress.com/support/contact/
LikeLike
Hey, they’re just blogs, get a life I say! Anyway, the transparency is greatly appreciated, this is what all SaaS providers should be doing, not just hiding what really happened (you know who you are).
LikeLike
Longest 110 minutes EVAR in the nearly 3 years I’ve been here. Thanks for letting us know what broke and for being transparent. Now, don’t do that again! 🙂
LikeLike
Man, I shocked when I heard you guys were down. At least you guys came up in the same day. It could always be worse. I’d have 110 minutes of downtime of a day any day.
LikeLike
Never have I seen such fawning and lock-tugging as in these comments. What, is NOBODY even slightly angry about this? Is “oh thank you Mr. WordPress sir” the ONLY thing that you people can find to say? Oh come on now.
Well, for the record – it really sucked that the site was down for two hours. If you have 10 million blogs you might also have a back-up plan to cover most problems and more than one “core router”. 2 hours is far FAR too long, especially as it happened during the day. So, for the record, sloppy work WordPress.
LikeLike
It’s Thursday in a long work week – I think I was down for at least three hours myself!
Seriously, thanks for the communication; always up-front and honest. (Why can’t more companies figure out how refreshing honesty is to their customers?
LikeLike
Guess I wasn’t online at the time. Lucky me 🙂 . But thanks for being honest with us.
LikeLike
Thanks for the quick info update and restoration!
LikeLike
Thank you!!!
You guys are awesome! 110 minutes is so little, and yet, so long too. And to log on and see you guys taking this so seriously (even though we don’t pay) is absolutely aweomse. Love you guys. 🙂
LikeLike
Thank you for the professional transparency and for the amazing way you take care of all of us WordPress bloggers. I have learned so much from WordPress since that faithful day in November 2008 when I wrote my first blog post. Thank you Matt and crew – you totally get the power of superlative customer service! Rock on…
😉
LikeLike
happens. 🙂
LikeLike
Don’t crucify yourselves, guys (and gals), you’re doing a helluvva good job and the blogosphere is proud of you all.Moreover – and I hate to use a dirty word – all of you are HUMAN, a species know for making errors. Keep up the great work!!!
LikeLike
I’m glad I have my site backed up on my mac just in case
LikeLike
Everything will be fine, it’s great that you tell us about that.
I just start to learn how to use my blog, but you earn my confidence.
Thank you!
LikeLike
I know now what happened this morning. Thanks a lot for you all, for really great work. Best regards.
LikeLike
Thanks ….:*:*:*:*
LikeLike
thanks for that.
LikeLike
Thanks for keeping us in the loop – we were worried enough last night (Irish time), but relieved now that we know all.
LikeLike
hmm. link to my blog http:\\hkokko.wordpress.com comes out as “this is not a weblog” and the page shows only the last post written there. Clicking on the post brings 404…
–h
LikeLike
Your blog is showing up normally for me, and I see three posts there. Please contact support if you’re still having problems, and we’ll get you help: https://wordpress.com/support/contact/
LikeLike