What is the biggest disaster to ever hit your supply chain? What caused this dilemma and how did your company recover from the situation?

Images of devastation are appearing on television screens around the world out of Brownsville, Texas and surrounding areas. Please keep in mind that no one will ever see the images of those whose lives will be wiped out from this storm. That is because those who will be most harshly dealt with are not living in Texas, but across the Rio Grande in Mexico.

There are, no doubt, pockets of extreme poverty all along the Rio Grande valley in Texas. That is not the issue here. The real issue deals with the lack of understanding regarding just how bad conditions are on the Mexican side of the valley. One must keep in mind that few Americans ever venture into the horrible neighborhoods of most Mexican cities where people live in cardboard houses (at best) and survive on nearly nothing. Welcome to the “Third World”.

One does not have to go to countries in Africa or Asia to find living conditions so brutal that no one in this country could imagine. Poverty so deep that there are no solutions and political systems so rife with corruption that there is no hope are the norm in most third world countries. Too many times Americans picture Mexico as a resort country due to visits to Cancun or Acapulco. Also, too many times Americans associate Mexico only with the huge and sprawling capital of Mexico City.

Outside of these areas, there is a huge country locked in poverty with few governmental services available and areas run more by drug lords than any official government. The people live in utter squalor and have for many generations. From these dire straits comes the yearning to come to America where there are jobs (which most Americans refuse to do) and money to be legitimately made. Most of what is made by Mexicans in America goes back to Mexico to support their families.

As fewer and fewer Americans are willing to make a living doing physical work, somebody must to it or it won’t get done. If it were not for the Mexicans who went to New Orleans and the Mississippi coast after Katrina, hardly any roofs would have been repaired or replaced. Americans are simply not willing to climb up on scalding hot roofs and do the hard work needed to make the repairs.

We all know that we would starve to death if not for the migrant workers who make their living picking the fruits and vegetables we enjoy so much. These people spend all their lives doing a job very few Americans would do even if paid $30 an hour. Americans are simply not willing to get down on their hands and knees to do what is needed no matter how difficult or physically taxing.

Yet, in spite of all the good things most of our neighbors from the south have done for us, they are hated and despised and huge numbers of Americans want to deport anyone who snuck into this country. Giant fences are being put up to keep people who are trying to survive out of our pristine land. Since when is this country so damn righteous that we shut our doors to those who are only trying to literally live? Since when are we so high and mighty that we say NO to those who are gladly willing to do the jobs we refuse to do.

Nothing has changed since the days of slavery. Just because official slavery was banned by the Constitution doesn’t mean it does not still exist. What else do you call the importation of legal and illegal aliens to do jobs no one else will do? Slaves were imported to work in the cotton fields and to take care of children and make meals. Slaves were imported to do all the dirty work so that the wealthy land owners could busy themselves with making more money, socializing and traveling.

Those wealthy enough to hire “household servants” usually don’t get them from the local labor pool. No, they scarf them up from Guatemala or El Salvador or Mexico. Why? So they don’t have to pay them what an American would demand in wages. This exact same thing goes on with migrant workers and other menial jobs Americans will not lower themselves to do unless they receive a king’s ransom for doing them.

As Dolly plows up the Rio Grande River valley, it will literally wipe out hundreds of thousands of cardboard homes and probably end of killing thousands of poor people the world will never hear about. Every major hurricane has seen far more casualties than are officially recorded. Eyewitnesses swore under oath that they saw trucks loaded with bodies in body bags leaving the devastation after Hurricane Andrew. The same can be said after Katrina.

Officials refuse to accurately give statistics for fatalities due to fears that they will be held responsible financially or in the court of public opinion. Instead, they downplay the numbers of people killed, especially when they are peasants, homeless, illegal aliens or ultra poor. These tactics are employed in the United States, Mexico Miramar, China, Russia and just about every other country in the world. Ruling governments do not want the world to know just how badly they screwed up by not preparing for a major storm or helping them quickly afterwards.

Please keep all this in mind as you see the handpicked images of particular destruction that will grace our televisions over the next few days. Rest assured that a week or so from now, Dolly will be forgotten except for those directly impacted. The rest of the country will be far more concerned about politics and sports than how “forgotten” people will figure out how to move on when they have lost literally everything they had in this life.

In any major disaster there are those who we see on the news who somehow manage to get all the help they need both economically as well as volunteers. The people we never see are the ones who either fall through the cracks and somehow miss out on all the help provided by social safety nets or worse yet; as far as the government was concerned—they never were there to begin with. God help the invisible forgotten masses whose only means for survival comes from our prayers and the willingness on the part of a few true heroes who look for them and help them exclusively.

I thank God for the small independent relief agencies whose mission is to find those people neglected by everyone else. I thank God for those who are willing to avoid the headlines and the photo ops to search out the one lost sheep no one else cares about. In my books, those who are worthy of being called “relief workers” are those who go where no one else is willing to go to help those no one else will help.

Most small businesses back up nightly at best, which means you could easily lose a full day's data if something serious goes wrong. That can be a lot of work to recreate. Top that off with the hassle of swapping tapes, lugging hard drives, or dealing with slow offsite backups that aren't done by morning. No wonder backups are so hard to keep running, and such a source of ongoing frustration for business owners.

Now, you can now use a combination solution that keeps risk of data loss to an absolute minimum, with fast recovery in the event of disaster. And while it's more expensive than basic backup options, it's affordable. A bargain compared to hours, days, or weeks of downtime.

First, there's a hardware device onsite at your office, a network attached storage device (NAS). The software takes snapshots as frequently as every 15 minutes. The way the software is written, it only copies what's changed at a bits and bytes level rather than document by document. It also sends the data changes offsite. That's usually a little less frequent, to keep from using up your Internet bandwidth. And then all this is monitored round the clock to make sure it's running smoothly.

If something happens, the recovery options are really impressive, fast and comprehensive. Let's say your server has a hardware failure, and it's going to take a couple of days to get replacement parts. Using virtual server technology, we recreate your server on the NAS, on a temporary basis, to keep your business up and running. It can take as little as an hour to rebuild. It might be a little slow compared to your normal server, but it keeps your staff working instead of waiting.

If it's a worse-case scenario, like fire or flood, have a NAS overnighted and get that up and running the very next day. You'll have plenty of other problems to deal with, but at least your computer systems won't be one of them.

To begin, what exactly is CDP?

In short, it is continuous backup. The ability to split every write ingress to a protected volume (LUN),  instantaneously. This process is performed  out-of-band or out of the data path, to an appliance (in this case a RecoverPoint Appliance). One item to note here, the concept of out-of-band is so important as introducing any devices in the data path has the tendency to impede I/O. Impeding I/O affects application performance, poor application performance leads to loss of productivity, etc,etc.. see where I am going with this....

Let's continue...this "write splitter" can reside on the host (Windows Only) , on the array (CX and CX3 series arrays only (iSCSI and FC)-Support for AIX, HP-UX, Linux, Solaris, VMware and Windows) or within the Fabric (ie. Cisco's SANTAP, Brocade's Fabric Application Platform, RP version only). So a couple of things here, there is actually two versions of RP..as noted below..both offer local and remote replication but RP seems to provide the added benefit of fabric based write splits and multi-array functionality as well as the following: BW reduction, Heterogeneous Multi-pathing support, and up to 8 appliances per site, to name a few.

RecoveryPoint- Geared toward multi-storage vendor environments. Comes with the CDP module for local synchronous replication intra-array or another same site array and the CRR module or Continuous  Remote Replication for remote asynchronous replication between multiple arrays. Both allow for data rewind (think TiVo) for point in time recovery.

RecoveryPoint/SE (Single Edition? Array to Array implication)- Geared specifically toward CX and CX3 series arrays only, noted above. CDP and CRR modules exist as well  but purely for Clariion to Clariion replication with the added limitation of an 8TB cap. In addition, there is support for block based storage on the NS20, 40 and 80 FC models, but no support for the fabric write splitter among other things.

We all know a picture is worth a thousand words, so lets look at a typical RecoverPoint (forget RP/SE for now) environment and an explanation of each component (images by way of EMC)...

  The RP Splitter Drivers

• Out of Band write mirroring to the RP appliance
• Function can exist on the Host, on the Array, or within the Fabric

The RP Appliance-Thanks Kashya

• Runs intelligent RP software
• Handles Bi-directional replication
• Adherence to write order, Consistency Groups
• Maintains complete management and monitoring capabilities

The Importance of the Journal

• Delta tracker for all protected volumes-Stored in a compressed format for Point-In-Time rollback or rewind (did I mention TiVo)
• Bookmark Tracker for application aware recovery
• Maintains reserved pool to track changes to PIT (Point in Time) copies that have been recovered (target side processing space)
• SAN based LUN, easily expandable via concatenation or striped LUN expansions

Advance Networking Capabilities

• Pre-Flight data compression
• Eliminates the need for pricey FC/IP converters

The "Spread the Love" Support Banner

• Heterogeneous third party storage support, but please check the EMC Support Matrix
• Storage agnostic, any to any replication

Here is another great image depicting RP/SE, Clariion to Clariion Remote Replication...

What about licensing you ask?  Important question, so here is the answer...be sure to talk to you local EMC rep about specifics

• RecoverPoint is licensed by a per replicated capacity
• RecoverPoint/SE is licensed per array and ultimately that depends on what model of Clariion

Beyond this basic run through, which I hope was helpful, there are numerous integration points with this product to talk about. But there is one that I personally am very excited about, VMware Site Recovery Manager. So hold on to you hats as the next post should be a dooozy...;)

De "Protection Suite for XenServer" repliceert virtuele machines primair vanuit hypervisor-niveau en dus niet vanuit de guest operating systemen die binnen de virtuele machines zelf worden geïnstalleerd.

De volgende key features worden ondersteund:

• WAN replicatie voor disaster recovery over meerdere datacenters
• LAN replicatie voor snelle lokale herstel
• Selectieve replicatie waardoor alleen bedrijfskritische virtuele machines worden gerepliceerd
• Virtual-to-virtual, virtual-to-physical en physical-to-virtual replicatie en failover
• Agents voor Linux en Windows guests voor verfijnde monitoring en recovery
• Configureerbaar via een gebruikersinterface en via een API
• Ondersteunt SCSI, FC, NAS en data replicatie
• Ondersteunt zowel XenServer Standard Edition als Enterprise Edition

De oplossing lijkt op datgene wat VMware's Site Recovery Manager biedt voor de VMware VI 3.5 omgeving. Prijzen beginnen bij $2495 per fysieke server voor een onbeperkt aantal virtuele machines.

Zowel Virtualization.info als Daniel Kusnetzky geven meer achtergrondinformatie. Informatie over de Protection Suite zelf is uiteraard bij SteelEye te vinden.

Well, after seeing System Error 25 VolSnap too many times and realizing each time it appears all  Volume Shadow Copy snapshots were being dumped, I decided to dig into this.  Why?  Because those VSS snapshots are very valuable safety nets that users rely on.  Things like redirected My Documents ...etc are highly valuable and worth having an extra version to fall back to.

What I discovered is that whenever a defragmenter runs against the default 4 KB (SBS/Win2k3) cluster size these events occur.  I experienced this testing out Diskeeper and then PerfectDisk (Raxco) but also see this when running the native Microsoft defragmenter.  Further research shows that the cluster size needs to be 16 KB for VSS.

So next time you create a data partition my best practice advise to you is to raise the cluster size up from the default to 16 KB (or a greater size).

Although this blog was written a couple of years ago for Windows Server 2003 this still applies in the Server 2008 product.  How do I know?  I asked the Server 2008 VSS team in their blog at the time of launch earlier this year.  What was amazing is that they gave me the impression by the tone of their response that didn't seem to understand that this issue was relevant or significant.

Here's the 3/24/2008 response I got from Greg Jacklin via Jim Benton:

"Although volsnap and defrag/ntfs do their best to get along, it is still
best to format with a 16K cluster size for performance.
-Greg"

BTW if you think your heard about this before but you can't remember where perhaps it was via Wayne Small or Microsoft.  Below are these historical references:

• Hack#95 - Work with Shadow Copies
• KB833167 - A Volume Shadow Copy Service (VSS) update package is available for Windows Server 2003

There are some other planning & "hot fix" articles from MS that address some aspects of snapshot dumping; nevertheless, remember that the root cause is that your cluster size isn't large enough.

Overview Info:

• Default cluster size for FAT and NTFS
• How Shadow Copies for Shared Folders Work
• Volume Shadow Copy Service Technical Reference (Oct'03)
• Best Practices for Using Volume Shadow Copy Service with Exchange Server 2003 (for larger than SBS environs)

Hotfixes:

• A Volume Shadow Copy Service (VSS) update package is available for Windows Server 2003 (Oct '05)
• Time-out errors occur in Volume Shadow Copy service writers, and shadow copies are lost during backup and during times when there are high levels of input/output

The big question for me as I look to the private sector is, if the real professionals have challenges during an emergency, what would it look like from an individual's company's perspective following a major business disruption? Are you prepared? Would communications flow freely? Could you respond to a business disruption? Could you resume your business? Recover your business over time? Unless you test your plan (assuming that you have one), you will never know! My theory is, better to find out during an exercise vs. during an emergency. If you are not going to make the effort to exercise your plan - good luck following the disruption we all hope never occurs.

The legal industry adheres to some pretty rigorous document retention policies where a certain number of files need to remain online for 90 days before they can be archived to a near line solution, which may or may not be at a remote data center. I have also seen other legal firms implement a 2 month - 2 year policy where documents remain online to the local facility for 2 months then can be online at a remote facility for 2 years before they can be archived offsite for sometimes periods upwards of 7 years. Now you can imagine the amount of storage that would be required for any sized law firm to retain even just two months of documents or image tape and storage cost to archive for 7 years. I have seen these storage requirements upwards of several terabytes which really require a way to optimize storage for only the documents needed based on defined retention periods. Retention periods should be able to be defined at least the following parameters:

• Last Access Date
• Size of file
• Storage threshold of available storage

These three primary settings for storage optimization are critical to keep the storage requirements to only what is needed and easily manageable for the IT manager. Once these archive parameters are set then it is nice to have a scheduled process or even better automatic process to help manage the storage requirements by identifying the defined rules and then executing that process so you don’t have to manually.

A good resource for technical information for the legal industry is The International Legal Technology Association (ILTA) is a collegial group of members who all communicate.

This year is the 31^st annual Educational Conference of the International Legal Technology Association (ILTA) and is being held in Grapevine, Texas August 25-28^th. Due to its strong relationship with ILTA members, Double-Take Software immediately tops the list for data protection solutions. Double-Take Software will be attending the show again this year in booth #915 and demonstrating the latest in high availability products for mission critical legal applications.

All of our videos can be found on the Double-Take website here or on the DBTK group on YouTube here: http://www.youtube.com/group/dbtk

What I have seen in several blogs is that Business Continuity will look at how to recover the business and Disaster Recovery is about recovering IT Infrastructure. This is mentioned not only in some of the blogs but also some of the institutes preach the same. This is not to comment that they are wrong, but to spread the word and understand what others think of the same.

Looking at the statement that says Business Continuity (BC) is looking at how to recover business, does this mean that those organizations that use IT Services for carrying out their business would be recovered to a certain extent without recovering any IT infrastructure…??? I wouldn’t agree with that…. Also it states that those organizations which do not use any IT services will not have any Disaster Recovery Plans (DRP).

Let’s say an incident had occurred and the organization is unable to deliver its services. The incident could have been caused due to a natural disaster such as fire. If the fire has occurred at a location where it has no impact on business, this can be controllable and immediate actions can be taken. If you look at a scenario where the entire building is under fire, then your Emergency Response plan would kick-in and evacuations, contacts with authorities etc, public addressing will be exercised.

The next action is to continue providing the services of the organization to its clients. So the organization might have to start operating from a different location with minimal resources (which include IT and People). This is triggered from the business continuity plan and is looking at continuing the business which will also require the support of IT Infrastructure.

Now going back to the original site and recovering back that site to its normal working condition is triggered by the Disaster Recovery Plan, which is also a part of the Business Continuity Plan (BCP). When I say normal working conditions, it means to recover the entire site to how it was before and the organization start to function as before from that location (or a new location). Now the organization will move from its minimal service delivery to full service delivery with all its force.

So according to me

Business Continuity (BC) = Business Continuity Plan (BCP) + Emergency Recovery Plan (ERP) + Disaster Recovery Plan (DRP)

Irrespective or whether the organization uses IT Infrastructure for its operations or not.

Now a lot of things we think of as disasters have warning systems in place if we would only pay attention to them. Weather is the first one that comes to mind and has several methods ranging from almost free to ones with substantial cost.  On the almost free end is NOAA's All Hazards Radio: http://www.weather.gov/nwr/.

"NOAA Weather Radio All Hazards (NWR) is a nationwide network of radio stations broadcasting continuous weather information directly from the nearest National Weather Service office. NWR broadcasts official Weather Service warnings, watches, forecasts and other hazard information 24 hours a day, 7 days a week."

With the use of this system monitoring of severe weather is pretty straight forward.  However businesses that are effected more severely by weather may want to subscribe to more sophisticated systems such as those provided by Accuweather.com: (http://www.accuweather.com/adc2004/pub/products_services/construction.asp?partner=accuweather&traveler=0&header=b)

The most comprehensive and expensive service is the one called Skyguard (form the web site): "Our comprehensive weather risk management and monitoring system combines a dedicated team of experienced meteorologists and business analysts with robust software to help you manage the impact of weather on your people, your property and, ultimately, your bottom-line profitability. SkyGuard®'s weather risk management solutions provide:

	Site-specific risk assessment
	Weather Impact analysis & system design
	SmartRAD Weather Information & Monitoring System
	Redundant weather risk warning system
	24x7 access to meteorologists
	High availability with minimal impact on your infrastructure
	Metrics to measure success
	SkyGuard Extensions

Another system that has been around in some form is the Emergency Alert System (http://www.fcc.gov/pshs/services/eas/).  This system is that one were the local broadcasters are diverted to provide emergency news and information.  Of course this is only effective if you are monitoring local radio or TV during the work day or after some emergency has already happened and you are already having to respond.

There are also numerous options at various news websites for email, or text message to your cell phone the local news headlines or breaking news.  If you are the person tasked with the emergency response there is not cheaper way to be tapped into the local and national news.

These options I have been discussing should help you be more prepared to meet an emergency situation that you could possibly see coming like those related to weather and civil unrest.  Items such as earthquakes (I live in Southern California), terror attacks and catastrophic infrastructure failures aren't predictable, at least yet.

Next time I will discuss communication in the middle and after an event.  Have a great week.
As always I thank you for your time and interest. Please take the time to Digg, Stumble Upon or add to the other social network of your choice to help me spread the word about these issues. Please forward any questions or suggestions to: askthefm@gmail.com

Social Bookmarks:

[gallery]

Perhaps many of you have read another of my blog posts about the successful effort by Tri Coastal Community Outreach in Mobile county Alabama.  Since writing that blog many months ago, the TCCO has continued to grow and expand its efforts into long term programs to enable those it serves to become self-sufficient and self-reliant. 

It has a very successful program called “It’s All About You” (IAAY) which has captured the attention of many people and groups nationwide because it teaches people basic construction skills among other things so they can fix and repair their own house instead of waiting for someone else to do it.  IAAY also allows local community service agencies to come to TCCO to present material on health and other vital topics. 

Two incidents took place on Tuesday, July 8 which are both miraculous as well as quite challenging to their operation.  I will attempt to share this with a minimum of personal comments so as to not take up too much of your time.

The oral surgery had been scheduled for Tuesday morning since early June.  Everything had been covered at TCCO to allow its director this week to recover from surgery to cut out a wisdom tooth.  All was going smoothly until 1:30 Monday night. 

The director has grown children, many of whom have children of their own.  The youngest child did not have any children and is on the heavy side.  She had been feeling quite ill for weeks and finally her older sister made her go to the hospital Monday night at 1:30 a.m.  She was in horrible pain and her blood pressure was sky high.  The sister who was with her called her oldest sister who had been trying to get the one in pain to go to the hospital for almost a month and told her the hospital had given her sister morphine.

The oldest sister freaked out and said to run back and tell the doctors that only the night before her sister had tested positive on a pregnancy test.  She did and the hospital did their own test and discovered she was not only pregnant, she was 39 weeks pregnant.  Somehow they deduced her water had broken 10 days earlier and still, somehow the baby survived.  They called their mom and awoke her to tell her to get to the hospital as soon as possible.

When she did, the daughter’s blood pressure was 220/117 and she was about to go into kidney failure and/or have a stroke.  The doctors said she had to have a C Section immediately, which she did.  They miraculously delivered a very healthy 6 pound 9 ounce baby girl to a woman who did not even know she was pregnant and still does not believe the baby is hers.  It took some work, but they got her blood pressure back to a safe level and the new grandmother went home to catch a couple of hours sleep before her own oral surgery early that morning.

She had her surgery as planned, and thankfully there were no complications.  While she was coming out of anesthesia, she received an urgent call from her assistant at the center.  She simply told her; “at least she’s still living”. 

Her oldest daughter had left the hospital earlier that morning driving TCCO’s van back to the center.  She was never supposed to be allowed to drive alone, but she was.  About half way there, she had a Grand Mall epileptic seizure (which is why she was never to drive alone).  She slumped down in the driver’s seat and leaned to the left against the door.  The van careened across the four lane highway just missing one oncoming vehicle after another.

The van started hitting telephone poles which pierced the windshield of the van like giant spears.  The van slowed but still barreled through a fence which had numerous large wooden posts which snapped and also went through the windshield of the van.  Finally the van came to a stop in the yard of a PREACHER.  The preacher came running out and immediately laid hands on the driver and prayed.

Before the state police or other emergency vehicles could get there, somehow the oldest daughter came out of her seizure.   They got a door open and she got out with only a gash on her forehead.  The preacher and his wife held her, prayed for her and kept her warm till the emergency personnel got there a few minutes later.

In the words of the State Police officer; “there is no way on earth anyone should have survived that accident”.  Evidently everyone who saw the van was totally amazed that the driver was not killed by the various huge spears poking through the windshield.  In fact, if there had been a passenger, he would have been killed instantly by one such projectile.  The stunned girl was taken to the same hospital her younger sister is at.  What a reunion. 

Please keep this whole situation in your prayers for there are, needless to say, many issues which must be sorted out and dealt with in due time.  Thank God both daughters and the new granddaughter are doing well.  But, among the many problems which must be dealt with is the need for a replacement vehicle to haul supplies up and down the coast of Mississippi and Alabama.  If anyone reading this has access to any kind of cargo van/truck that could be donated to TCCO; please contact Tri Coastal or Carolyn Thompson.  Their number is 251-865-9731 or www.tricoastalcommunity.org on the web. 

But for now; I pray hearing about this incredible day will help strengthen your faith and give you the extra bit of spiritual energy to get you over whatever hump stands in your path.  We truly have an awesome God who knows what is happening and will do His best to watch over and take care of situations as He is allowed to.   He most certainly did at TCCO.

Thu, June 26, 2008 — CIO — Designing a disaster recovery plan has traditionally forced companies to strike a delicate balance. To create a plan that restores operations quickly, an enterprise needs to invest significant capital. On the other hand, costs can be cut dramatically if an enterprise is willing to withstand longer periods of operations downtime. During the planning stages and while the computer network runs properly, the forces to reduce costs are felt the strongest and often prevail. But when disaster strikes and the network goes down, everyone starts screaming to get the network up and running again, as fast as possible. Finding a way to walk this tightrope is a major challenge, but with the advent of virtualization, deploying disaster recovery plans that restore operations quickly—and at a reasonable cost—is quite possible.

At Transplace, we developed a new disaster recovery plan based on virtualization when we moved our infrastructure to a new production data center in 2007. We also took that time to refresh our hardware and review our overall architecture. Previously, we ran daily backups and physically moved the data to an off-site location. With this process, we risked being down for a half day if we experienced a problem in the middle of the day. This type of plan also limited us in that we only backed-up once a day, which meant we risked losing a day's worth of work. This plan also required us to have dedicated servers that sat idle except when we executed a recovery.

After we moved into our new data center in Dallas at the end of 2007, we began to plan our new disaster recovery data center in Arkansas, into which we moved in February 2008. At the storage level, we deployed network-attached storage and SnapMirror software from Network Appliance to create virtual storage for our database and application servers. SnapMirror allows us to send copies of all changes to our backup facility on a near real-time basis without impacting the performance of the applications. Anytime a record changes in production, it sends a copy to our disaster recovery facility. This shared-storage approach also allows us to manage storage centrally. We buy storage only when we need it.

At the database level, we deployed IBM P570s with AIX as the operating system, leveraging its logical partitioning technology. This combination allows us to partition each server to look like multiple servers, and we can run multiple database servers by sharing the capacity of the individual servers. In the disaster recovery facility, the database server runs four to six copies of Oracle that we use for testing and development most of the time, but if the need arises, we can shut down the virtual servers and run the disaster recovery instance of Oracle on that same server. This also allows us to make the most efficient use of our Oracle licensing costs, which are charged by each physical CPU core.

the application server level, where we run VMware and Windows on Dell servers, the content of each virtual machine is also replicated to the disaster recovery site anytime an update occurs. With VMware and IBM database servers, we use a set of servers for testing and development. When we need to run a disaster recovery restore, we turn off the virtual servers for test and development, bring-up the ones for disaster recovery, and we're good to go. All the data and content of the servers is quickly copied over.

Four-Step Disaster Recovery Process

For enterprises ready to develop a disaster recovery plan, we recommend a four-step process that helps frame the project and ensures a reliable disaster recovery process:

Step 1: Enablement
Make sure all the data is properly transferring to the disaster recovery data center. Ensure that all the proper hardware in the disaster recovery data center is in place, will remain stable and is running on up-to-date operating systems. Also, review all applications and decide how long you can you go without each one. This helps prioritize the most crucial applications. Some applications might need to be restored in less than an hour while you might be able to do without others for up to 12 hours. This part of the plan becomes an internal SLA.

Step 2: Testing
Develop detailed procedures and processes on how and how often to test the disaster recovery plan. We recommend at least once per quarter. You also need to determine how to measure success so that you can evaluate the testing and document the findings to compare one test to another with a high level of validity.

Step 3: Cutover Documentation
You need to document exactly how you will cut over if and when a disaster strikes. There will be some elements similar to the test process, but there will also be differences for how you execute procedures while under a live disaster recovery. With all the pressure your IT staff will be under, it's critical that this step be clearly and thoroughly documented.

Step 4: Returning to Normal Production Infrastructure
Just as important as how to cut over to your disaster recovery infrastructure is knowing how to return to your normal production infrastructure. It's not always a case of doing things in reverse, and it's a process you should also test.

Lessons Learned

It's important to bring all of the key vendors and your internal IT team into the same room at the same time. This gives everyone a chance to voice concerns, explain how their piece of the puzzle contributes to the overall project, and to understand the functions of the other parts of the project. If you get yourself into a position where you act as the go-between among your vendors, important information will undoubtedly be lost in translation.

Enterprises should take a good look at compression technologies. With all of the data that needs to be copied to the disaster recovery site all day long, it's important to reduce the amount of bandwidth you require so that your network runs efficiently.

Looking back on the disaster recovery plan we started in 2007, we feel we have achieved the ultimate balance: a simple way to recover operations fast—but at a relatively lower cost than traditional disaster recovery systems. Without a doubt, virtualization played a vital role in helping us achieve this mission.

Vincent Biddlecombe is the CTO of Transplace. He has more than 15 years of experience in IT consulting with an emphasis on transportation management systems.

Mungkin banyak diantara anda yang sudah mengenal SLA. Well, dalam dunia Disaster Recovery, anda akan banyak bertemu dengan istilah ini. Mumpung masih belum terjun ke dunia tersebut, anda bisa belajar dulu disini :) .

Apa itu SLA ? Saya akan berikan gambaran SLA di dunia perbankan. Setiap bank memiliki beberapa jenis aplikasi yang berbeda2 untuk mendukung bisnisnya. Sebagai contoh: ATM, Core Banking, e-Banking, mulai dari proses external sampai ke proses internal diatur oleh aplikasi2 yang berbeda2. Nah, aplikasi2 ini memiliki SLA yang berbeda2 pula. Sebagai contoh: ATM dan website harus dapat diakses 24 jam/hari, dan bank tersebut mungkin akan memberikan garansi ke customernya. Garansi inilah yang disebut SLA Misalnya, SLA ATM dan website untuk uptime adalah 99% dalam 1 tahun. Artinya 1% boleh downtime. 1% ini adalah waktu dalam 1 tahun. Kalau kita buat perhitungan satu tahun jam adalah :

Satu tahun jam : 24 jam x 365 hari = 8760 jam.

Nah, dalam klausul SLA tersebut, 1% downtime adalah 1% dari 8760 jam = 87.6 jam/tahun atau 1.68 Jam/minggu. Artinya sistem tersebut boleh mati selama 1.68 jam/minggu. Well, terkejut ? Dalam dunia banking, layanan pelanggan adalah nomor satu. Tak heran jika investasi bank untuk mendapatkan SLA seperti itu sangatlah besar. Menarik ?

Jadi secara garis besar, SLA merupakan persetujuan formal antara dua belah pihak, yaitu pihak customer dan service provider mengenai hal performance dan service delivery. Perjanjian ini memuat pengertian terhadap services, prioritas, tanggung jawab dan jaminan yang disebut "Level of Service". Dalam IT Service Management di dunia Call Center atau Service Desk, SLA ini ada dalam beberapa matrik:

• ABA (Abandon Rate) : Persentase call yang terputus saat menunggu jawaban
• ASA (Average Speed to Answer) : Waktu rata2 call yang masuk dijawab oleh Service Desk
• TSF (Time Service Factor) : Persentasi call yang dijawab dalam timeframe tertentu. Misalnya 80% dalam 20 detik
• FCR (First Call Resolution) : Persentasi call yang bisa diselesaikan dalam sekali call.

In summary, SLA memiliki point2 sebagai berikut:

• Komitmen dari penyedia jasa (service provider) ke customer. Misalnya Aplikasi X available dalam 99% 24x7
• Blueprint dari available solutions
• Membutuhkan keterlibatan business unit dan IT
• Prioritas ke sistem yang paling kritikal atau perlu perhatian lebih
• Fokus ke business availability daripada technology availability
• Membuat solusi menjadi dapat diukur

Mengerti ? Syukurlah kalo sudah mengerti :) . Kalau belum mengerti bisa sekali lagi membaca artikel di atas atau mencari di google.

Baik, di atas saya sering menyebut kata2 Downtime dan Uptime. Dari bahasa inggrisnya sepertinya anda sudah mengerti artinya. Yup, downtime adalah masa dimana sistem dari suatu organisasi itu tidak dapat diakses. Sedangkan uptime adalah masa dimana sistem dari suatu organisasi itu running dan dapat digunakan.

Nah, apa aja sih penyebab sistem itu downtime ? Bisa dari unplanned atau planned downtime. Unplanned downtime adalah downtime karena sesuatu yang tidak terduga, seperti: kebakaran, gempa bumi, banjir, sehingga kita tidak siap untuk melakukan persiapan terhadap bencana tersebut. Biasanya terjadi data loss disini, sehingga proses recovery lebih lama.

Sedangkan Planned downtime adalah downtime yang sudah kita persiapkan atau rencanakan. Misalnya ada upgrade mesin, atau mencoba sistem DRC (Disaster Recovery Center). Biasanya tidak ada data loss karena semua downtime direncanakan, sehingga kita memiliki langkah2 yang benar dalam melakukannya.

Jika terjadi unplanned downtime, maka ada kerugian yang harus ditanggung oleh suatu organisasi. Berikut ini adalah Cost of Downtime:

Wow...ternyata downtime bisa menyebabkan kerugian yang sangat besar !! Oleh karena itu, di jaman globalisasi ini, suatu organisasi sudah seharusnya memiliki suatu mekanisme untuk mengatasi atau meminimalisasi resiko kerugian akibat downtime dengan mengimplementasikan solusi High Availability and Disaster Recovery agar SLA yang diinginkan dapat dicapai.

Solusi untuk mencapai level SLA yang diinginkan, tentunya ada cost. Semakin tinggi SLA, misalnya 99.999%, maka semakin besar cost yang dibutuhkan untuk solusinya. Biasanya solusi untuk SLA yang sangat tinggi seperti itu digunakan software Replikasi Data atau Storage. Jika SLA yang dibutuhkan tidak terlalu tinggi, solusi lain bisa digunakan, seperti backup restore ke tape.

Pusing ? Tenang saja, mungkin di awal2 anda pusing karena mengenal istilah2 yang benar2 baru. Bisa juga anda pusing karena mungkin pekerjaan anda sangat technical seperti Programming atau Administrator, sehingga belum menyentuh hal2 berbau IT Management seperti ini. Tapi ingat, bidang IT itu luas sekali :) . Seorang IT Professional yang baik tidak boleh hanya menguasai technical saja, namun perlu juga kemampuan IT management. Saya akan kembali dengan menulis artikel lain seputar IT Management, seperti Disaster Recovery dan High Availability. Maju terus IT Indonesia !!

Regards,

Aris Kumara Prabhawa
ABCP®, SCJP, ITIL®, MIMIX® Certified, EchoSuite® Certified

Segunda lição: Conceitos, alguns meses atrás escutava um amigo: "existem muitas pessoas no mercado c/ nomes "fortes" que não conhecem conceito, isso o mercado esta cheio", ou seja, fala de algo com toda a segurança mas não conhece na essência o assunto e acaba ensinando conceitos errados a quem não conhece; pensei bom quem sou eu para dizer algo. Não estou aqui para ser advogado do diabo, mas pude recentemente comprovar isso pessoalmente, falávamos de RTO, a questão foi sobre o entendimento do que era, quando era iniciado o RTO e hoje o grande motivador da minha escrita foi lendo um post que me agradou pelo título cujo tratava sobre Continuidade de Negócios; rapidamente já fui abrindo o post para ler, até que me deparo com: "Este tempo é chamado de RTO (Return to Operation)", li novamente, e comecei a rir e a lembrar do meu amigo Wagner Elias falando sobre os conceitos.

O Significado de RTO é (Recovery Time Objective), não vou falar o mesmo, se alguém tiver dúvidas recomendo o excelente post do WElias.

Volto a dizer, também cometo enganos e não estou aqui para arbitrar, eu particularmente já aprendi as minhas lições esta semana: Não postar nada sem certeza e entender muito bem do assunto antes de falar de "sopa de letrinhas". Aguardem, estou preparando um pequeno artigo sobre PRD e estarei indicando onde o mesmo será publicado aqui no blog.

Até a próxima.

We enjoyed beignets and café au lait at the mandatory 24-hour tourist magnet called Café du Monde. A guitar player on the sidewalk plucked the theme from “Fiddler on the Roof,” followed by “Raindrops Keep Fallin’ on my Head.”

New Orleans’ French Quarter is slate sidewalks and jazz, art and antique shops and carriages pulled by donkeys instead of horses because, I overheard a driver tell her riders, “They deal with heat better than horses do.”

At night, Bourbon Street is a raucous footpath through loud music and varying degrees of naughtiness.

We had Cheeseburgers in Paradise and Land Shark beer, and stood on a street corner watching one of several active crews filming “Mardi Gras,” the working name of a movie someone said would star Carmen Electra.

It was a fun town, people were genuinely friendly, and we want to go back.

But something was missing. The young Margaritaville waiter said the reason the streets and clubs were not stuffed with people is summer is off season for New Orleans. New Orleans doesn’t have a shore, he said, and summer visitors want the ocean.

Then we met 60-something Joe Favalora.

He lived most of his life in or near the French Quarter, where Bienville settled in 1718 because King Louis IV wanted to move his New World capital and that was the highest spot of land in the area, 15 feet above sea level.

About 25 years ago Favalora moved to Slidell, a town on the northeast side of Lake Pontchartrain, where wind blew trees into his house and forced him to seek shelter elsewhere. National Guard soldiers of Bravo Battery, 1st-108 Field Artillery, a Gettysburg and Waynesboro Pa. unit, spent a month near there in September 2005, helping distribute food and other supplies to residents forced from their homes by Hurricane Katrina, and by the waters of Lake Pontchartrain when its levee collapsed.

It may have been them of whom Favalora said: Them Pennsylvania boys, that's where I got my water.

Volunteers had come from Pennsylvania — “church groups, the people with the long dresses … they worked hard, those good people, believe me.”

They cut wind-felled trees from his house and sprayed it with bleach to kill the mold, and gave him back his home eight months after Katrina.

In the weeks following the hurricane, and the collapse of the levee holding Lake Pontchartrain back from the neighborhoods on the north side of New Orleans, we saw plenty on television news of the 9th Ward. We saw little or none on national television of local news people who went out to document the carnage in its early stages, and realized their homes also were gone.

Allow me also to introduce St. Bernard Parish, population in 2000, according to the U.S. Census Bureau, 67,229 souls; 88.3 percent White, 7.6 percent African-American; nearly one-third in the family family-raising 25-44 group; median household income $35,939.

They thought they’d dodged the bullet — and then the water came.

“When I say the water came,” Chalmette High School Principal Wayne Warner told us Saturday afternoon, “we went from no water to eight feet in 15 minutes.”

Chalmette High School is the highest point in St. Bernard Parish, eight feet above sea level, and still it was under eight feet of water. For a week, survivors holed up on the school’s second floor, before finally finding ways to leave their destroyed homes.

Gone were the electricians and plumbers and carpenters — the very people needed to rebuild the erased communities and destroyed economy.

To drive through the community of Chalmette is to drive through the set of a disaster movie, without Bruce Willis, Tommy Lee Jones, or Dennis Quaid to save the citizens. Concrete slabs lay where they once had supported homes. Tiles cling to the otherwise barren floors where kitchens and bathrooms had served.

It was like a movie, as long as you could forget real people had really lived there, before Aug. 29, 2005. Three years later, families still are scattered across the nation from homes passed through generations.

Warner said about 20,000 former residents have come back to homes volunteers from around the nation have helped them salvage.

They still need money, workers and supervisors to help rebuild the residential areas of city.

And mental health workers.

“They’re alive,” Warner said of the returning and never-left, “but they’re not living.”

Normally, he said, mental health begins to improve about two years after a disaster. In New Orleans, it is getting worse.

“Everything is not all right,” he said. “My grandmother is not living around the corner.”

(Pictures from our trip are here. Check back — I'm still culling and adding.)

©2008 Readers may contact John Messeder at jmesseder@comcast.net

Nightly backups are the norm, usually to tape, an external hard drive or network-attached storage device (NAS), or online backups to a remote server.

You need your backup easily accessible for data recovery when there's a problem; at the same time you want a copy of your data offsite - somewhere other than your office - in case of fire, theft, or other serious disaster.

Comparing backup options...a NAS is fast but cannot be easily taken offsite. External hard drives can be removed from the office but must be handled with care to avoid damage. Tapes are less reliable but easy to transport - just avoid leaving them in the car on a hot summer day. Online backups require a speedy internet connection and ongoing subscription fees but are offsite by definition.

Combining options gets you the best of all worlds. Use a NAS to backup everything nightly. Then protect your most critical data, like company financials, with an online backup service to ensure you've got a copy offsite.

Don't forget to create copies of all your original software CDs and DVDs along with all the license keys. Keep these in a safe location (offsite) so that you can fully restore all your computers even if you have to reinstall every single program. This will eliminate the need to buy replacement software in the event of a serious catastrophe.

If loss of a full day's work makes you queasy, nightly backups may not be enough for your business. Affordable backup options are now available to small businesses that provide a higher level of protection. These copy your data throughout the day, providing fast recovery in the event of disaster. They combine a NAS, offsite backup, remote monitoring and service, plus new technologies that quickly "virtualize" your server if there's a problem. You can be back up and running in as little as an hour.

Just now I sat down to work on a critically important document that I spent hours on last week and need to finish today. On opening it, I discovered that the version that had been retained in the cloud and then pushed to my notebook when the dust settled this weekend was on from early next week, not Friday. After a momentary panic, I looked on my network share and found a copy, right where I'd stashed it Friday at the end of the day. I was concerned about the flakiness of my iDisk and so took the time to save a copy in a known safe location.

Moral of the story: always be sure to have two or more up-to-date copies of critical work in known good locations. Often a Mac Time Machine backup will deal with this, but not always. Check.

Yesterday morning we had a major server outage affecting our 1.0 customers. We completely lost one of our database servers. The day was spent rebuilding and restoring everything we possibly could.

There were a handful of accounts that the restore completely failed on. Yours was one of those accounts. We have exhausted all available avenues for restoring the account data with no positive results.

Quite simply, I find this astounding! I would have thought that should have had a mirrored/clustered database server, and at the very least held an offline backup of my information. Obviously not.

In my previous life prior to YouDo, I spent countless hours with auditors going over BCP and DR plans, and countless hours making sure we'd avoid this kind of screw up ever happening to us. I used to hate it. Really hate it.

But now I've been on the receiving end of what seems like an inadequate disaster recovery plan, and can relate to the pain that our customers could feel - it's going back near the top of my agenda.

SRM is namelijk bedoeld voor disaster recovery situaties en niet per se voor het verbeteren van de ICT-security. Bij uitval van een volledig datacenter zorgt SRM ervoor dat de functies op een uitwijklocatie worden herstart. Daarvoor moet je op die uitwijk natuurlijk wel de data en de virtuele machines tot je beschikking hebben. Deze dienen dan ook door een Storage Array gerepliceerd te worden van de primaire locatie naar de backup locatie.

Een voorwaarde voor het gebruik van SRM is daarom dat de storage leverancier zich door VMware laat certificeren. Want, alhoewel de replicatie van data op storage-niveau plaatsvindt, wordt de coördinatie van die replicatie door SRM geregeld.

SRM maakt duidelijk dat het verhogen van de beschikbaarheid eenvoudiger wordt als je de servers hebt gevirtualiseerd. En dat geldt zowel binnen het datacenter (met VMware's High Availability oplossing) als tussen twee datacenters met SRM.

Since then, I've been back to Windsor on numerous occasions, working with The Salvation Army Emergency Disaster Services in its efforts to provide three hot meals a day plus 24-hour snacks and drinks to all comers. I have found the experience to be profoundly satisfying, in fact, opening a doorway into a new world of service that continues to beckon me with greater insistency.

Herewith, some more notes I have compiled...

The Link Between Disasters and Astrology

Curious to know more about the etymology of the word "disaster," I was intrigued to learn that the word has its roots in astrology. From a website called takeourword.com (which seems to have "gone dark," yet still has a very interesting archives for word enthusiasts to poke through), comes this:

"This word was first used in astrological terms to refer to an unfavorable aspect of a star. English took the word from French d'sastre, which referred to 'a disaster, misfortune, calamatie, misadventure, hard chance,' as it was defined in the mid-16th century. It was taken by the French from Italian disastra, which was formed from the earlier disastrata "ill-starred." Its components are dis- "bad" and astra "star" (ultimately from the Greek astran, which also gave us such words as astronomy and aster, a type of flower which is "star-shaped." ) At one time, all events in human existence were attributed to the motions of the planets. Thus, in 1684, a diarist recorded, "I am very sick with a disaster upon my stomach." This usage parallels that of 'influenza,' which derives from the Italian phrase 'influenza degli steiil' or "influence of the stars."

Regardless of Popular Opinion, Tis Still Easier to Give than Receive

Whatever their reasons, motivations, desires or callings, people are coming forth in droves around the globe to offer their hearts, hands, and heads in service. Alas, the capacity to receive is still snarled by cultural conditioning, pride, and the ego mind. There remains so much discomfort around receiving that which is freely offered. What a joy it is to gently remind people that we are steeped in abundance and that they may have as much as they like, as often as they like, without hesitation, guilt, or embarrassment. Even better, I am able to do this not by telling, but through grounded action. A hot meal, a bag of snacks, a jug of water, a big old bear hug, carry a much deeper message than words ever could.

Finding The New Paradigm in the Least Likely Places

In my home county, there seems to be undue focus on the trappings of consciousness. One must have the politically (and spiritually?) correct home furnishings, vehicles, clothing, activities, workshops, classes, adventure trips. In the parking lot where hundreds of people are fed daily, each day is marked by fully-embodied demonstrations of prosperity, unconditional love, non-judgment, community, gratitude, and connection. No one is talking about finding their purpose in life, discussing the need to create greater satisfaction through service, frantically looking for proof of the presence of Creation in its angelic and divine forms: we're all simply living it. Not so much talk, way more walk...

Seen Handwritten on a T-Shirt: Mother Nature's A Bitch

Well, not exactly. Yet She does seem to be frustrated and more than a bit unhappy. Like many of us, Mother Nature/Mother Earth is purging and releasing, fiercely declaring "Enough!," and going through massive upheavals of transformation. Not always a pretty sight. My spiritual worldview includes a knowing that incarnate souls (that would be you and me, by and large) do make free-will choices and, at some infinitely deep level, have agreed to the experiences wrought by natural disasters. This neither diminishes nor denies the material and emotional losses many sustain, though, for me at least, it provides a greater context in which to view things.

The Time To Create Community is Now, Before Disaster Strikes

If it takes large-scale events to bring Good Samaritans out of the closet and into the thick of it, that's a good thing. If we connect more with the other folks in our communities before we are all thrust into chaotic worlds gone upside-down, that's a great thing. I'm planning to go through some training so that I can be of greater service in a variety of situations. How about you? Besides the American Red Cross, there are a variety of large, well-organized, faith-based groups that effectively support people (and their animals) in emergency disaster situations -- including The Salvation Army, The Adventists, The Southern Baptists, and many local/regional churches.

No matter what your own spiritual/religious leanings, these are powerful vehicles in which you can join up with others and walk your talk in service. The qualifications are simple: Open hearts, open hands, open minds, and that still, small voice whispering to you, "Get out there now and share your light with the world."