Some Turbulence
You’ve probably noticed that the ride has been a bit bumpy on WP.com the past week with a widely covered DDoS attack last week and then another interruption a few hours ago from an unrelated network maintenance issue. Our core mission is enabling the world to publish, and anything that affects the availability of your site or dashboard impedes that.
Anything less than 100% availability drives us to work harder. Rest assured that behind-the-scenes here at Automattic we’re learning as much as we can from these incidents so we’re even better prepared and more able to adapt next time something unexpected comes our way. (We weather DDoS attacks every day on WP.com and 99.9% of them have no user impact.)
In a previous post I wrote about downtime I ended with the sentence, “For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”
That last part about “adding new servers” was especially true, for WP.com we’re now at 8,921 CPU cores across 2,475 physical processors, 8,200 gigabytes of memory (RAM). We’ve changed how we’ve done storage, but now that layer includes on its own 1.3 terabytes of RAM, 1.3 petabytes of storage, and 8.9 terabytes of solid state disks. (Plus Amazon S3, which we use as backup to our internal systems.) Also, we’re adding new servers constantly. 🙂
Keep blogging, and we’ll keep doing our best to make sure you never have to think or worry about this stuff.
- March 8, 2011
WordPress.com News 
Good job keeping up WordPress! 😉
LikeLike
Understanding! WordPress is the BEST!
LikeLike
We love WordPress.com 😀
LikeLike
Thanks for sharing that. I was afraid that everybody just thought yesterday’s blog sucked.
LikeLike
I could pretend I have the slightest clue about what all that means… or I could just say I’m really glad you guys are there to make this whole thing run smoothly and I appreciate your efforts!! Thanks!
LikeLike
No one is perfect but we appreciate that you try to be!
LikeLike
Thank You!
LikeLike
wow, that is a lot of storage and processors and RAM and so much more WordPress goodness 😀
LikeLike
Thanks, guys! It gives a good feel to know there is a hand-in-hand type of work understanding. We keep on blogging, you keep on docking! Pretty much the best kind of motivator for us to be or to become good bloggers too. With best regards, Walter.
LikeLike
Iran’s government has filtered all of wordpress.com.
LikeLike
Wow. That’s some horsepower.
I’m curious about the timing – WP shuts off spam referrers using short URLs, WP gets hit by DDoS attack. Hmmmm….
Especially since the trend was heading toward 30% of all page requests being spam referrers.
LikeLike
8,921 cores and 8,200 gigabytes of RAM. What a render farm that would make.
Keep up the great work.
LikeLike
Thanks for all your hard work, guys!
LikeLike
I never worry about WordPress going down, or stopping. It plugs along and I love to write! The support and encouragement is wonderful. I love WordPress.com and I am thankful for those who work behind the scene to keep it worry free for all of us.
LikeLike
Thanks for the update Matt! I had noticed a degradation in response times and availability this past week, it’s good to know it’s not just due to the popularity of WordPress! 🙂 Best of luck supporting such a gargantuan network of servers and storage! I myself have a lot of experience supporting huge databases in an enterprise environment, with the goal of keeping everything transparent to the end users. You guys do a great job at WP.com. My husband, son and I all maintain WP blogs and love your software and the overall development environment. Keep up the good work!
LikeLike
You guys at WP are the best there is! Given that we have a large number of bloggers in our wordpress community who fight the good fight against repressive regimes. These kind of attacks are to be expected. Nobody keeps the presses rolling like wordpress! Thanks for the fine work you all do making it possible for us to speak our speak and be heard!
It’s high time the winter of our discontent
become the not so silent spring of ACTION!
LikeLike
Keep up the good work. We believe in you guys 🙂
LikeLike
Thank you Matt! WordPress is amazing! 🙂
LikeLike
You guys are GREAT! Thanks for working so hard to fix everything.
LikeLike
Glad everything is not only okay but getting better with new servers and all. Good luck!!!
LikeLike
“You’ve probably noticed that the ride has been a bit bumpy on WP.com the past week with a widely covered DDoS attack last week …” – yes I noticed that and was irritated, didn’t know the reason. Thanks for information!
LikeLike
Thank You for the info and please keep up the good work You are doing.
LikeLike
Thank you!! It’s always a bit disturbing to hit turbulence but it’s great to trust my pilot WordPress!
LikeLike
Great job, WordPress team! You guys do really great work and I hope WordPress just grows and grows. Maybe if WordPress has downtime, you can create an image like the Twitter Fail Whale?
LikeLike
I’m not sure if we could have an illustration awesome enough.
LikeLike
May you grow from strength to strength. 🙂
LikeLike
As we discuss in forums (https://wordpress.com/forums/), we all are wondering about what happened. But as you describe, it could always be resolved.
Thanks to wordpress’s staff who are always ready to overcome all existing problems.
We are sure, you do the best thing that should be done.
GBU…
LikeLike
New servers, yay! We’ll keep bloggin’. 🙂
LikeLike
You are all working very hard. I have no complaints. Keep up the good work.
LikeLike
We appreciate your hard work guys! I Love WordPress!
LikeLike
Thanks! 🙂
LikeLike
Thank you for the hard work! 🙂
LikeLike
I’m glad to know everything is in good hands. 🙂
LikeLike
I love those stats!
LikeLike
Thanks for the update! Sorry to hear about all the disruptances and such and hope things are coming together again. You guys do great work for us and should know that we’re thankful for it! (Or at least I am.) It’s great to know of the hard work behind the scenes. Makes me appreciate it even more. Cheers!
LikeLike
Thank you! Whew! That is some hardware and software! Best to you and all you do! Thank you again!
LikeLike
Good! I’d hate to go back to blogspot 😦
LikeLike
Thank you for all you do for us bloggers.
LikeLike
Thank you very much. I am from Morocco and I am so happy that I can publish freely through WordPress.com. Your help is very important for our people. Our society is changing toward a better social and political progress. In some weeks, I will send the URL of my blog to hundreds, then thousands of young, activists. This helps us share and perform our cultural, social and political ideas. So thank you very much, and to all your collaborators.
LikeLike
Wow — best of luck with getting your message out there, and be careful.
LikeLike
I’m new at WP, but I’ve been impressed with your organization so far. Keep up the good work.
LikeLike
Thank you for all of your hard work and for keeping us posted on what’s going on.
LikeLike
I didn’t even notice! Keep up the good work guys!!!
LikeLike
Thank you!
LikeLike
Thank you so much for your hard work!!!
LikeLike
Thank you for all your efforts!
LikeLike
Your loyal users will hang in there as you sort out all those tricky bits! Thanks for all the behind the scenes work on our behalf. I heart wordpress!
LikeLike
Thanks for all the work y’all do to make blogging a pleasant experience! I’ve looked hard at several other blogging “options” (Blogger, etc.) and none of them have been so easy to get around or as inviting as WP! Keep up the good work!
LikeLike
Thanks for your dedication
LikeLike
Thanks guys, you are doing a great job!
LikeLike
Capable “pilots” always manage to eliminate turbulence..
That’s why we are always flying with you….
Very informative, indeed.The fact that you keep us posted about every detail,shows how much you respect your bloggers.We do appreciate it immensely.
Thank you
LikeLike
But capable pilots also aren’t allowed to while flying… I think we found our problem. 😉
LikeLike
No problem Matt- keep up the good work 🙂
LikeLike
Keep adding servers 😉
LikeLike
Thanks for keeping things and us updated.
LikeLike
Keep up the good work!
LikeLike
Just to say, thanks guys. Phil
LikeLike
I’m sure glad you are staying on top of interruptions. It’s sad that anyone would launch a DDoS against WordPress, but that’s he kind of CyberWorld we’re living in. Someone’s got a beef, they don’t take it up with the source, they just strike out because they can.
Keep up the good work. I am not often long enough on WordPress itself to feel the impact of any interruptions, and that’s good, but I appreciate that you guys are continuing to learn from these experiences. That’s how I moved ahead with my computing knowledge and ability.
LikeLike
Great stuff guys. Your hard work and dedication is appreciated very, very much.
JZ
LikeLike
Thanks much for keeping us all informed.
LikeLike
I have to say I am impressed with the speed in which you got WP up and running again. I had logged in and done some changes to my draft post. When I went to save them, I got the message of a problem and that, for the time being, read-only was active. I’ve never been on a site that doesn’t have some downtime for one reason or another, so I shrugged my shoulders and went to do something else for a while. About 10-15 minutes later, I thought it was a waste of time to try again so early, but I was eager to finish my draft. Lo and behold, the service was back up. Bravo!!
LikeLike
Thanks for working so hard for us!
I bought the adfree extra so I could show some appreciation which puts something in your pocketbook.
All the best
LikeLike
Thank you so much. I appreciate your diligence and commitment…all of the folks at WordPress. I am new to this world. And am just grateful that one of my friends told me about your site.
LikeLike
Thanks Matt for the explaining thingy. I understand now what’s up with that. WP rocks.
LikeLike
thank you for the information. 🙂
LikeLike
I did notice some slow response to my typing, but in spanish we have a saying that goes like this – when someone gives you a horse as a gift, you don’t go around checking its teeth. I am grateful I am able to share my thoughts with my friends, and I can do that through you for a song. Thank you again for allowing me to do just that.
LikeLike
Thank you WordPress for doing such a superb job, I am glad to be a part of WordPress.
LikeLike
Just a simple and well deserved, Thank you.
LikeLike
Yours is an unappreciated field of endeavor. Thanks.
LikeLike
Keep up the good work. I for one appreciate WordPress. I now have my main blog where I post new stuff on WordPress. After I got the hang of it I found that it is a great blog program for me.
LikeLike
While the hiccups were a bit disappointing, I’ve never been happier with a blogging solution. Especially when I consider how affordable WP is. Thanks for all you do! Keep up the great work!
LikeLike
WordPress does a great job! I rarely have trouble with it even though I’m practically computer illiterate. Thanks for the update.
LikeLike
Thank you for the wonderful outlet bumps come and bumps go. Your attitude and provision of the marvelous tools provided by you are sincerely appreciated.
LikeLike
Thank you Matt, subsequently WordPress.com!
LikeLike
Am so glad you are aware and working on it. It has been frustrating not to receive notifications or have my new post notifications sent. I just starting blogging July 31 and my life has become so enriched by the people with whom I send and receive. I don’t think I could live without WordPress. Thank you so much.
LikeLike
Thank you for working so hard to make sure that we don’t have to worry.
Your service is awesome and much appreciated.
LikeLike
The end of the world is nigh, Matt, even for Automatic 😉
LikeLike
It’s okay I took a survival course.
LikeLike
Now that is some serious hardware. Keep up the good work. I read a little bit about the attack today coming from China. This too shall pass.
LikeLike
Thanks for all your hard work. WordPress is the best.
LikeLike
Well, I have to be honest and tell you that most of the technical terminology in this report is “all Greek” to me, but that doesn’t matter. It’s like a little child knowing that Mom and Dad know what they’re doing when it comes to providing a home, food, clothing, and safety. That little child doesn’t worry it’s head about all those details as long as he trusts Mom and Dad. Just want you guys to know that I started with WordPress because everything I heard about you and everything I read that you said about yourself gave me a safe feeling about blogging with you. (Since I was really new at blogging, that safe feeling was important.) So far, I have two blogs, and all of my experience with you assures me that I made the right decision. So I’m not going to worry about all those terms or what they mean. I’m just going sit back and relax, knowing you guys know what they all mean and how to keep us safe from the problems they cause.
Thanks a whole lot!!!
LikeLike
Hey Matt,
Your post reminds me of a byegone era, when I was an IT manager. It was so long ago we had a precious 2Mb of internal memory to play with for an installation supporting 3,500 users.
In those days, I had to write this sort of apologetic stuff too. Like when some numbskull operator deleted all our monthly back-up files inadvertently. Or when the system date was set by manual input at start-of-day, and (another) numbskull input it incorrectly, so we had to fix thousands of financial transactions with the wrong date.You probably don’t have a numbskull working for you, but you get the picture.
Anyhow, the point is 100% availability wasn’t feasible then, and it probably isn’t now, not unless you spend gazillions of dollars in mirrored environments. Users – me included – should be prepared to accept some interruption to service. Unless we want you to charge the kind of fees that would enable you to provide world class availability.
WP.com offers a great service. Like you said, “Keep blogging, and we’ll keep doing our best to make sure you never have to think or worry about this stuff.”
Sure we will – but only because you guys are worrying about it 24/7. It’s appreciated.
LikeLike
We do mirror our environments, and it’s not a gazillion anymore, just a few million, but still they can all get hit sometimes, as this shows. You’re correct that the biggest errors are often human and not hardware.
LikeLike
Bless you!
LikeLike
Thank you for all your updates and hard work. I am new to this, but have found the service and accessability to be wonderful. Good job and thank you.
Trish
LikeLike
You work well, I know.
LikeLike
Wow! I have to confess, I never do think about this stuff, just take it all for granted that it will work. A huge thank you to the team that makes it all happen is long overdue!
LikeLike
I am so glad, wordpress.com, is my avenue to blog. You folks, have done an outstanding job, in my estimation. There are things, here, that are offered, that maybe someone would not even know about. As you mention, these maintenance issues. Or all of the technical enhancements, that to most people, cannot, even comprehend, or understand. But you folks do, and your considerations, even here, to communicate, for our best interests, touches my heart, to know I blog with the best, of the best. WordPress.com. God bless you. Thank you for taking care of us, so we really do not have to worry….I appreciate you folks more than I can say.
LikeLike
I’ve read about attacks to your company; however I never experienced any problems using WP in my blog. I know how much you care for your users, and understand how you must feel about the interruptions of your service. It really doesn’t bother me to wait a few hours if your site is being under siege, because I know how hard you work for us. I’m impressed with the resources you have to keep WP operating. After reading this post, I’m more assured, WP is the best blogging tools around. Won’t change to another site even if they promise me the world. Thank you for your sacrifice and support for all of us.
LikeLike
Keep up the great work. I have received many tributes to the way my blog is presented. I always answer it’s easy using wordpress.
LikeLike
BRAVO!!!
LikeLike
Excellent work. Haven’t noticed a single hitch.
LikeLike
I think you guys do an amazing job. Thank you. Keep up the good work. God bless you all.
LikeLike
Thanks for letting us know!
LikeLike
If this affected the speed with which WP responds, then my suspicions that accessing my blog was a bit slow are now confirmed. I have premium cable service and really thought the issue was from my provider and not you.. so, I promptly called them and gave them a piece of my mind.. 🙂 Maybe I should call back and apologize?
LikeLike
I should add.. YOU PEOPLE ROCK! Keep up the SUPERIOR work!
LikeLike
Thank you. I do appreciate the information and your efforts to keep things ticking over correctly.
Best, Robert Linn.
LikeLike
Amen and thank you!!
LikeLike
You guys should run the world. In all my years in business (50) and dealing with people I have never seen better communications or customer/client service from any organization.
LikeLike
Honestly….most of that is like a foreign language to me but I’m sure it impresses someone out there! If there are blurps and bleeps occasionally….I’m sure we’re all accepting of that. Everyone has had the lights go out on them or dropped a call….you get the idea.
Thanks for the update!
LikeLike
Impressive! Like all the things you guys do!
LikeLike
Thank you guys for the sterling work you are doing. We really do appreciate not having “to think or worry about this stuff.”.
LikeLike
Wow. That’s a LOT of robo-power keeping our little blogs running happily. Thanks for all that you do!
LikeLike
Really appreciate your efforts and the open communication about what goes on behind-the-scenes. Your work enables us to do some of our best work and gratitude is thick. A sense of community comes from understanding the aggregate of the technology that enables us all, which is an interesting side effect when considering the magnitude of the blogging world. In short, thanks!
LikeLike
Thank you for all the work you do keep the servers afloat:) I (and I’m sure others agree with me) sincerely appreciate your effort.
LikeLike