Some Turbulence

March 8, 2011

Matt

You’ve probably noticed that the ride has been a bit bumpy on WP.com the past week with a widely covered DDoS attack last week and then another interruption a few hours ago from an unrelated network maintenance issue. Our core mission is enabling the world to publish, and anything that affects the availability of your site or dashboard impedes that.

Anything less than 100% availability drives us to work harder. Rest assured that behind-the-scenes here at Automattic we’re learning as much as we can from these incidents so we’re even better prepared and more able to adapt next time something unexpected comes our way. (We weather DDoS attacks every day on WP.com and 99.9% of them have no user impact.)

In a previous post I wrote about downtime I ended with the sentence, “For those following along at home, we’re now powered by 152 physical processors, 511 gigabytes of memory (RAM), 174 hard disks with several terabytes of storage, and we’re adding new servers constantly.”

That last part about “adding new servers” was especially true, for WP.com we’re now at 8,921 CPU cores across 2,475 physical processors, 8,200 gigabytes of memory (RAM). We’ve changed how we’ve done storage, but now that layer includes on its own 1.3 terabytes of RAM, 1.3 petabytes of storage, and 8.9 terabytes of solid state disks. (Plus Amazon S3, which we use as backup to our internal systems.) Also, we’re adding new servers constantly. 🙂

Keep blogging, and we’ll keep doing our best to make sure you never have to think or worry about this stuff.

Missing out on the latest WordPress.com developments? Enter your email below to receive future announcements direct to your inbox. An email confirmation will be sent before you will start receiving notifications—please check your spam folder if you don't receive this.

Join 110.6M other subscribers

364 Comments

Comments are closed.

« Older Comments
Newer Comments »
  1. Treasureyourhealth Mar 8th at 9:45 pm

    Thanks for that heads up! I love that I have finally turned to blogging. Still learning how to maneuver WP, but am thrilled with all
    the possibilities. And, remember, keep taking those deep breaths! (*_~)

    Like

  2. diogeneseassociates Mar 8th at 9:53 pm

    Y’know what? Considering the cost and quality of the WordPress service, a little downtime due to the odd hiccup is a small price to pay.

    Thanks for keeping us informed and thanks for providing such a great service.

    Like

  3. dreamsburnred Mar 8th at 10:06 pm

    A little downtime for a great service isnt a issue. WordPress.com has and always will be my blog host.

    Like

  4. cookingoodies Mar 8th at 10:17 pm

    I hadnt even noticed ! WordPress is wonderful!

    Like

  5. mindblowingfood Mar 8th at 10:39 pm

    Thanks for making such a brilliant tool to connect people and let us share our passions!

    Like

  6. dontpokeme Mar 8th at 10:43 pm

    Please show the negative..You are a wonderful service-host but…we know there are glitches that we all feel…

    Like

  7. Kerrina Mar 8th at 10:48 pm

    Thank you for all your hard work!

    Like

  8. Craig Mar 8th at 10:49 pm

    Kudos to Automattic … because server up-time and fending off various DDoS attacks is not automatic. This blogger thanks you.

    Like

  9. terrydarc Mar 8th at 10:51 pm

    Thanks for the great job, WP! We all appreciate your taking the time/effort to keep us going and for informing us what was going on. I hope that information continues in the future.

    On the attack itself: sounds like a concerted effort by a large group of computers (this is not the sort of thing a disgruntled techie launches from his/her PC) but a concerted effort to control the msg. As shown in the Mideast, the 21st century lives free (and dies) by the message getting out. My sincere thanks.
    -Terry

    Like

  10. alligramz Mar 8th at 10:54 pm

    Ya Done Good, Guys! Thank you for watching out for us!

    Like

  11. hljourney Mar 8th at 10:57 pm

    Thank you so very much for all that you do behind the scenes to make everything work correctly!! I greatly appreciate it!

    Like

  12. gatherer4 Mar 8th at 10:57 pm

    Hey, I just started with WordPress and I have no complaints. It’s nice to know there are people out there working on your behalf. Thank you, Matt.

    Like

  13. Harold Mar 8th at 11:01 pm

    Thanks for the work behind the curtains! And a peek every once in a while!

    Like

  14. Taufik Mar 8th at 11:02 pm

    several times down here in Indonesia . . . but its OK
    You’ve done a great work guys 🙂

    Like

  15. HannibalTC Mar 8th at 11:04 pm

    Keep up the good work!

    Like

  16. markgorman Mar 8th at 11:08 pm

    Cool. You are the guys. In all my spheres of commercial interest I demand WordPress as the blogging platform and it’s ability to integrate into websites is legendary. Much love.

    Like

  17. italianhandful Mar 8th at 11:15 pm

    So appreciate the update and info. Thanks. Adagio

    Like

  18. Pingback: That’s A Lot Of Hardware
  19. silvercannon Mar 8th at 11:21 pm

    Thanks! You all do great work and I appreciate having this available.

    Like

  20. Steve Mar 8th at 11:33 pm

    I most certainly appreciate all that WordPress does to allow worldwide publishing to be within reach of the average person. I run several WP websites, and while an occasional hiccup can be taxing during a time of heavy uploading or site modification, you folks always get it handled quickly and efficiently, so I cannot complain (especially since the price is most assuredly right)! You have earned my strong loyalty over the course of the past two-plus years, which has led to several other author/publishers signing on at my highest recommendation. Thank you!

    Like

  21. bccmee Mar 8th at 11:37 pm

    This is my favorite blog provider and even if there are glitches, you *communicate* and you *care*!

    Like

  22. thf2 Mar 8th at 11:37 pm

    I appreciate you guys!! Keep pressin’!!

    Like

  23. debbyfriselladesigns Mar 8th at 11:53 pm

    Thank you for keeping us posted on the details…I love WordPress just for that reason!!!

    Like

  24. What's for tea tonight dear? Mar 8th at 11:55 pm

    We do appreciate your efforts to keep the show on the road, all the more so when you’re up front if there are problems. Keep up the good work!

    Like

  25. My Camera, My Friend Mar 9th at 12:01 am

    Thank you for letting us know what’s going on.

    Like

  26. chandrapzm Mar 9th at 12:04 am

    Thank you for all the WP teams have done. I really appreciate it. 🙂

    Like

  27. Ћэ §åŀэςmãη Mar 9th at 12:06 am

    Wow, Keep up with keeping WP safe!

    Like

  28. Success Coaching Group Mar 9th at 12:24 am

    If all businesses were so transparent and forthcoming…ahhhh, I can dream 🙂
    Thanks for the updates!

    Like

  29. Martin Scherer Mar 9th at 12:36 am

    Bastards

    Anything we can do to help?

    Like

  30. undeniablymaybe Mar 9th at 1:12 am

    You guys are the best, which is why, when I recently made the decision to start a real blog again after so many years away, I chose WP. Thanks, keep up the good work.

    Like

  31. andro51 Mar 9th at 1:19 am

    Thank you for keeping us all up to speed on things, you’re doing a fantastic job by the way…

    Like

  32. tknologlvr Mar 9th at 1:46 am

    Thanks for all you guys and gals do and for keeping us informed.

    Like

  33. Trang Đài Mar 9th at 1:48 am

    Love you WordPress! ❤

    Like

  34. fullphaser Mar 9th at 1:48 am

    There’s really no need to apologize for being knocked out by a DDoS attack, in those situations the best you can do is try desperately to retune network switches and packet filters and ride the wave.

    Like

  35. kbelle1 Mar 9th at 2:03 am

    Thanks for your dedication. Looking forward to smoother sailing!

    Like

  36. Mira Faraday Mar 9th at 2:18 am

    Thank you guys and gals, for what you do for all of us. We greatly appreciate your hard work and dedication that all of us benefit from each day.
    Yes, when things go down we all may panic. We as well know ourselves when our own work or home gadgets go down frustration occurs too.
    My hat is off to each of you for all that you do to keep us happy! Have a wonderful day when everything is fixed, the you can go home. ( teasing)
    Many smiles across the miles to let you know your appreciated.

    Like

  37. Achmad Fauzi Mar 9th at 2:22 am

    Thanks for Your best job … 😀

    Like

  38. John Ryan Recabar Mar 9th at 2:24 am

    the mark of an excellent service is that when you do not even noticed you are being served because it is seamless as seamless can go. that’s why i chose wordpress.

    thanks, wp.

    Like

  39. metalodyssey Mar 9th at 2:25 am

    You dudes ROCK! This stuff happens… and we carry on bloggin’! 🙂

    Like

  40. eClinik Mar 9th at 3:17 am

    Wish I could have the same firepower in my basement. 😉

    Like

  41. LDMartin1959 Mar 9th at 3:30 am

    I guess I don’t get it. It doesn’t make sense to me that anyone would DDoS a blog site. Unless ya’ll really, really, really pissed someone off.

    Like

  42. technogran1 Mar 9th at 3:34 am

    Well done all at WP keeping everything running smoothly. I also have to confess I never noticed any problem.

    Like

  43. Margaret Mar 9th at 3:45 am

    Just want to thank you all for your dedicated hard work to make WordPress the very best place to post all of our words, words, and more words. … Your expertise shows and puts everyone at ease… Thanks again.

    Like

  44. elcampeador Mar 9th at 4:13 am

    Hat is off to you folks. I’d assist if I could, unfortunately some wise guy just encircled my abode with kryptonite. I can’t get to a phone booth..

    Like

  45. horse11 Mar 9th at 4:24 am

    I Love wp as you folks are humble and willing to communicate and share!!

    Like

  46. elteee Mar 9th at 4:30 am

    Thank you for taking the time to explain what the turbulence was for those who felt it. Even though I don’t really understand what you were saying (it’s like you’re speaking another language!) it’s so comforting to know that you and your team are there to help make our ride as smooth as you possibly can. It’s a GREAT ride! Thanks for all you do. It’s very much appreciated!

    Like

  47. excogitator Mar 9th at 4:33 am

    Never mind buddy, WordPress still rocks!!!

    Appreciate your efforts and honesty.

    Like

  48. mymysticmuse Mar 9th at 4:42 am

    WP is awesome. I’m grateful for the space you’ve created for us here Many blessings to you all.

    Like

  49. Lucky Teacher Mar 9th at 4:48 am

    I started my blog on another site but grew too frustrated within two weeks. So glad I then found WordPress. The features WordPress offers make that other service seem even lamer. Even more so, I appreciate the ease with which I can find blogs on similar subjects. THANK YOU WORDPRESS.

    Like

  50. Rahul Bakale Mar 9th at 5:05 am

    Thanks for this update and for your services; we appreciate this openness in communication. It is good to know what WordPress offers us in terms of the hardware.

    Like

  51. Abigail Mar 9th at 5:07 am

    Didn’t have a single problem, so obviously your work paid off. Not to mention I’m getting more visits than ever. 🙂

    Like

  52. Walter Allievi Mar 9th at 6:17 am

    keep up the good work!!!

    Like

  53. Alexander Kormilitsyn Mar 9th at 6:19 am

    hi! can’t imagine those figures you are writing about. petabytes… must be something exorbital! thanks for sharing, as it’s always interesting to know the details of what been doing behind. wishing you all the best in your efforts!

    Like

  54. Ian Gardner Mar 9th at 6:23 am

    GOOD SHOW! Keep it up.
    🙂

    Like

  55. Lady Marilyn Kay Dennis Mar 9th at 6:33 am

    I only understood a very small part of all that, but the little that I did understand provokes the comment “Wow!” Thank you for keeping things running smoothly.

    Like

  56. hanin Mar 9th at 6:37 am

    thanks, you guys let me sit back and relax while you banging your head over those turbulence. way more than free service I expect, then…

    Like

  57. Matthew Lowes Mar 9th at 6:41 am

    Dittos. Really appreciate WordPress!!!

    Like

  58. apollodorosh Mar 9th at 6:44 am

    Keep it up, so we can keep going 😀

    Like

  59. Admin Mar 9th at 6:55 am

    Türkiye’den Teşekkürler. [Thanks from Turkey.]

    Like

  60. prank1989 Mar 9th at 6:56 am

    Matt,

    Thanks alot Matt for your this blog post. Great work your doing. I wish wordpress will be at the top of the edge. Keep it up dude. 😉

    Like

  61. bagnidilucca Mar 9th at 6:56 am

    Thank you for the explanation. I thought I must have been doing something wrong.

    Like

  62. mels63galaxie Mar 9th at 7:14 am

    it all looks great to me, and I am just a budding blogger. keep up the good work and thanks for keeping us informed in language that I can understand!

    Like

  63. Joey Chong Mar 9th at 7:29 am

    Thank you for solving all problems and technical issues WordPress!!
    Can’t tell you how reckless I felt without my blog..
    But I know it will be up and running in no time =)

    Like

  64. Admin Mar 9th at 7:41 am

    Hey Matt,
    thanks for giving wordpress users such a huge space!

    Like

  65. r3dg3 Mar 9th at 8:00 am

    Thanks, wordpress! You rock!

    Like

  66. kweide Mar 9th at 8:13 am

    Well done

    Like

  67. Shailesh Mar 9th at 9:42 am

    Thanks for updating us. I am using WP since long time. Presently I am working on a blog -Environment, Energy, Sustainability and Technology. WP helped me a lot in this initiative.

    Thanks!

    Like

  68. gonecycling Mar 9th at 9:42 am

    It’s all greatly appreciated – thanks for keeping us up-and-writing.

    Like

  69. nita36 Mar 9th at 10:09 am

    Thank you for your efforts to keep everything going well. I appreciate your effort as well.
    from Nita

    Like

  70. jeauxmoer Mar 9th at 10:13 am

    This is exceptional! Those new phrases and hardware specs leave an old techie like me speechless.

    Like

  71. cinnamon_dragon Mar 9th at 10:30 am

    WoredPress is the best. Keep on going! I would be soooooo sad, if there would be no wordpress 😦

    Like

  72. menchu4901 Mar 9th at 10:42 am

    I now understand what those odd behaviours were a few times I was doing site admin, but nothin’ to worry me as I know WordPress handles things well. Thanks guys! Keep up the good work.
    PS: With all that RAM, core processors, servers, I think it must all make up a small nation in size. :< )

    Like

  73. Follygirl Mar 9th at 10:45 am

    Thank you!!!
    Greetings from Germany

    Like

  74. Debbie Bates Mar 9th at 11:46 am

    Thanks WP. Our bloggy world reflects the Big Blue Marble in general- World keeps getting bigger; world has some bad days; tomorrow is another day; WP fixes big problems faster. Thank you.

    Like

  75. R. Mar 9th at 12:05 pm

    I LOVE WordPress. And I always will!!!

    Like

  76. BornToInspire Barbara Sherry Rose, PhD Mar 9th at 12:06 pm

    Matt, You and all of your team are doing a superior, outstanding job! Your dedication and commitment to excellence is so evident, and the care you have for all who blog on WordPress shines right through. Just keep going – the world needs you!

    Like

  77. Cicci Mar 9th at 12:13 pm

    No one can be perfect all the time.. 😉

    Like

  78. ahmad Mar 9th at 12:46 pm

    thanks matt…

    Like

  79. Nyoman Wiastra Mar 9th at 12:47 pm

    Yeah I do believe on what you are doing on WP!

    Like

  80. illage3 Mar 9th at 1:26 pm

    Oh well, these things happen. I am glad that you are doing something about the issues, and hope to see more themes being implemented. It must take a lot of time and money to have all those servers up and running. Keep up the good work.

    Like

  81. nobusysignal Mar 9th at 1:49 pm

    Kudos WordPress.

    Like

  82. Ethan Banks Mar 9th at 1:58 pm

    We love you, WordPress & Automattic.

    Like

  83. Olga SE Mar 9th at 2:02 pm

    Yes, I did notice that and frankly speaking I started to worry. But after reading this post I feel reassured. Thank you for sharing your difficulties with us. It’s so nice to realize that my small blog is being taken care of.

    Like

  84. kyriako Mar 9th at 2:04 pm

    Very good job… thanks for the excellent quality hosting..

    Like

  85. Maxi Mar 9th at 3:01 pm

    There is no substitute for hard work … and no one works harder than WordPress.

    Thank you.

    Like

  86. ScientologyParent Mar 9th at 3:07 pm

    Love the insight into your infrastructure! And I can appreciate what it takes to keep online through DDoS attacks – so annoying.

    But thanks for keeping it all together. We appreciate it!

    Like

  87. Hosk Mar 9th at 3:23 pm

    I noticed it a couple of times but you guys fixed it so quickly I wasn’t sure if I imagined it.

    Like

  88. Marissa Jean Mar 9th at 4:19 pm

    I’m so happy I understood most of that! Thank goodness wordpress knows what it’s doing, tumblr is constantly going down…

    Like

  89. fridal12 Mar 9th at 4:23 pm

    You are fantastic!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

    Like

  90. arzki Mar 9th at 4:24 pm

    That’s lot of hardware. Good thing you have some redundant systems 🙂 From a guy coming from technical background, I know what I am talking about. Cheers!

    Like

  91. Chen Li Yong Mar 9th at 4:33 pm

    It’s great to know that our blog are in such a very good hand. And for worrying our blog 24/7, we will thinking to expand our blog’s content 24/7 too. Thanks for the great work. I really appreciate it.

    Like

  92. metaglossia Mar 9th at 4:44 pm

    I just wish to join my voice to the others in encouraging WP to keep up the good work and remain in the lead in the the blogging world. You have shown beyond all doubts that you have just what it takes to face all sorts of difficulties. Keep on and thanks.

    Like

  93. Charlousie Mar 9th at 4:48 pm

    Sounds good! 😉

    Like

  94. Hans Mar 9th at 5:00 pm

    Keep the good work up. I even didnt notice it that you were suffering from a DDoS

    Like

  95. জামাল উদ্দিন Mar 9th at 5:17 pm

    Thanks for keeping us up-to-date .
    WP Rocks ! Keep it up .

    Like

  96. JG Mar 9th at 5:51 pm

    Keeping those numbers in mind, you have some serious horsepower supporting WP and I don’t want sound ungrateful for the service we receive but I could imagine that in order to support your power house you need some huge generators to keep all those processors. I do hope that some of the energy consumed is coming from renewable energy sources … does it ?

    Like

  97. Carlos Acosta Mar 9th at 6:49 pm

    Thanks for the info, impressive numbers…

    Like

  98. horsewhispertje Mar 9th at 7:19 pm

    Thank You!

    Like

  99. sylviahubbard1 Mar 9th at 8:35 pm

    man that sounds like a lot of computers. i knew you guys were on the job. I wasn’t worried… much!

    Like

  100. neelyssimpson Mar 9th at 9:13 pm

    Thanks for all the hard work you do!

    Like

« Older Comments
Newer Comments »

Create your new blog or website for free

Get Started