HOW A WELL PLANNED IT STRATEGY SAVED A WELDING COMPANY




This tale of a welding company is full of lessons. It's an actual company and our client.

Companies, like products, have a life cycle.

A start-up would have a different set of priorities from a mature company. A growing company taking giant steps to reach the next level would also differ from a market leader trying to maintain the course.

Business objectives can be categorized by time period. There are near-term, intermediate, and long-term objectives.

Objectives will also differ based on the company’s place in its life cycle and on the ambitiousness of its owners. Does it seek to simply maintain the status quo? To advance ahead of the pack? Or to innovate, jump to a higher level, and even create a new industry?



Whether they realize it or not, all organizations operate within this framework. Leaders who recognize this can tailor their strategies to optimize the investments they make to achieve their objectives.

Optimize, as used here, means reaping the highest possible return for the dollars and the effort.

Objectives dictate strategy. Strengths, weaknesses, opportunities, and threats shape it further.

Objectives must also be appropriate. How likely will a community bank’s plan to become a downtown commercial bank succeed? Within three years, probably not. Within 10 years, maybe.

Smart and purposeful strategies place the company’s objectives within this matrix.

Enter technology. We use it to achieve business objectives. That said, how do we connect IT strategy with this matrix of objectives?

ALIGNING YOUR IT STRATEGY WITH YOUR BUSINESS OBJECTIVES

Let’s take an actual company, trace its path in its life cycle, and show how it made smart investments in IT in order to meet the objectives in its matrix.

The subject is a growing manufacturer of welding supplies. Its premium products are sold to other manufacturers who share one common attribute. They’re all known for producing high quality products. Its welding supplies are used wherever superior welds are necessary. Its customers are in Japan, Germany, the U.S., and other industrialized countries. This manufacturer is competing globally in a niche with few players. It has decided that it wants to increase its market share and accomplish that by buying a larger but floundering competitor. Its matrix looks like this:



The numbers show how its IT investments are allocated. Notice how the allocation changes according to the nature of the business situation which, itself, depends on the company’s place in its life cycle as well as its place in its long-term strategy.

Let’s understand the nature of the objectives first.

THREE DIFFERENT KINDS OF BUSINESS OBJECTIVES

“Status quo” refers to simply maintaining the necessary infrastructure to operate the business. These investments cover such functions as e-mail, website & web-based services, and company records. These are core operations but also included are some not-so-obvious necessities like regulatory compliance.

The corporate scandals of 2001 spawned a host of legal and regulatory requirements. All publicly-traded companies, for instance, must comply with the Sarbanes-Oxley Act of 2002 (otherwise known as SOX).

SOX is a complex piece of legislation implemented by the Securities & Exchange Commission (SEC). SOX was written with broad brush strokes and this has allowed the SEC wide latitude in interpreting its provisions. It matters for many reasons and only one has to be stated here.

It lifts the mantle of accountability all the way up to the Board of Directors for essentially confirming the truth and accuracy of the company’s financial statements. This sharpens the line between the legal persona of a corporation and the actual persons who run it. Before, the corporation was its own legal entity and its officers stood legally distinct from it. SOX, arguably, still recognizes that but it also recognizes that real people run the corporation. SOX detractors make good points against it but, personally, I believe that SOX, overall, is justified because it brings the legal persona of a corporation closer and more down to earth. Corporate officers are real people. Corporate officers are stewards of money raised from the public. They use other people’s money and if some of that was your money, wouldn’t you demand some accountability for the way they used it?

Thanks to SOX, the Board and every corporate officer down the chain of command are now liable for violations of this law. This naturally compelled companies to establish oversight mechanisms for financial reporting. This means all publicly traded firms must spend IT dollars complying with SOX simply as a part of doing business.

“Advance” refers to advancing ahead of the pack. How do you advance? By creating competitive advantages. That management-speak refers to either of two things. Your costs of doing business are lower or your business is more productive than your competitors.

LOWER COSTS

Assume a competitor and your company spend $5 to create and sell one welding rod. You invested IT dollars to create a web-based service that allows your marketing, sales, manufacturing, and purchasing people to coordinate your company’s purchases of raw materials. This investment pays for itself by minimizing errors in ordering quantities and optimizing the timing of those orders so that you receive them only when you need them.

In management-speak, you created an intranet to improve your supply chain. If you opened up the intranet (that’s a company-wide internet meaning it’s only accessible within the organization) to your suppliers, your intranet has expanded to become an extranet. How do these “nets” help? They should lower your costs by extracting savings. The savings come from committing fewer errors, buying only when you need to, and, of course, from automating the entire process.

If I went grocery shopping and bought two extra milk jugs for a party this weekend and learned that the party had been postponed but I wasn’t informed, then I wasted money by buying two extra jugs, ahead of time, requiring me to choose between spending more time and gas to return the jugs or to keep them and have them spoil. All of these could have been avoided if I had just been informed.

GREATER PRODUCTIVITY

Your company is now using your intranet. Now purchasing is buying your raw materials in the right quantities at the right times and from the most competitive suppliers. Given all that, wouldn’t it be reasonable to expect a more efficient production floor? For example, I would expect fewer incidents of production shutdown because of raw material shortages.

That translates into increased productivity and you can measure it by dividing your overhead into your output of finished goods.

If your competitor and you had similar plants that produced 100,000 welding tips at a cost of $50,000 a month and your output increased to 125,000 simply because you never had shortages, your productivity quotient improved from 2.0 to 2.5

“Innovate” refers to revolutionary, as opposed to evolutionary, changes in the way you do business. The two terms, “revolutionary” and “evolutionary,” allude to the magnitude of change. The former means a radical change and the latter means a gradual change.

Apple’s iPods and personal GPS systems (Global Positioning Satellite) are two familiar consumer examples of product innovations. They were so revolutionary that:

• they didn’t fit in any existing product category; and
• they created new markets and will, in most likelihood, spawn new industries.

Apple knew it had a hit when iPod became a noun (just like “Google” became a verb) and companies in other industries created products to complement the iPod.

Bose®, for example, created its SoundDock® for the iPod.

The Bose® SoundDock® digital music system was specifically designed to expand and enhance your enjoyment of the music stored on your iPod. Just slip it into the iPod docking station for the Bose sound your favorite songs deserve. The iPod charges as it plays, so you enjoy music without interruption.

Innovative IT investments are undertaken on the premise and promise of revolutionary outcomes. Bear in mind that, as with most things, the potential for a greater reward is counterbalanced by the investment’s higher risk. What specifically makes innovative IT investments riskier?

1. The technology might be nascent, i.e., it recently came into existence.
2. The technology might have never been used for the purpose your company is planning to use it for.
3. There are typically fewer (or even just one) companies supporting the technology. What happens if they go out of business?
4. There are fewer qualified personnel to operate the technology.
5. The technology will be expensive. Early adopters of new technology pay a premium for the privilege of being the first consumers. On the other hand, their Return On Investment (ROI) might also be substantially higher than average.

Some risks can be mitigated while others may not.

Let me share these details about the high-risk, high-reward program of our subject welding company. The company is currently in the first half of a 24- to 30-month long $10-million dollar program consisting of three projects.

A program consists of two or more projects. A program has a large scope. Sending man to the moon is an example of a program.

The project managers are introducing a piece of the new technology as each project finishes. So far, so good. If it succeeds this company will be poised to offer new products and services faster than ever before. It will also help the company forge closer ties to its customers -- more so than its competitors presently can.

A PRIMER ON THE WELDING INDUSTRY

Talk about unglamorous but vital. Until I began working with this company, I didn’t give welding a second thought. I found welding to be as interesting as buttons. They hold things together and their absence would change our lives drastically. Unfortunately, they’re so mundane that they don’t even register on my radar screen.

Click here to read about the Welding Industry.

BEFORE WE ANALYZE THOSE NUMBERS

Here, again, is the IT Investment & Strategy Matrix.



At this point, we know the row headings classify business objectives by their nature. Is the objective to simply maintain the status quo? Is it to advance ahead of the pack? Or is it to innovate and aim to leapfrog the industry?

We also know the long-term plan of the company. Let’s refresh ourselves:

The subject is a growing manufacturer of welding supplies. Its premium products are sold to other manufacturers who share one common attribute. They’re all known for producing high quality products. Its welding supplies are used wherever superior welds are necessary. Its customers are in Japan, Germany, the U.S., and other industrialized countries. This manufacturer is competing globally in a niche with few players. It has decided that it wants to increase its market share and accomplish that by buying a larger but floundering competitor.

To that, I’ll add this much more.

THE CURRENT STATE OF THE WELDING INDUSTRY

The metal welding equipment and supplies industry primarily serves mature markets with fully developed technology. Major portions of the welding market are flat or growing proportionately with the Gross Domestic Product.

The U.S. welding industry faces three major challenges:

1. a chronic shortage of trained welders;
2. an increasing number of low-priced foreign competitors; and
3. the continued growth of alternative metal-joining technologies and alternative fabrication technologies.

You can substitute practically any industry in that paragraph and the list will not materially change. In other words, these are not new challenges.

My client realized this and concluded, correctly in my opinion, that the battle has to be taken overseas. Welding is one industry that clearly feels the impact of globalization. The battleground is not in North America; it is everywhere.

Take China’s emergence as an economic power. The average American doesn’t realize it but China has turned into a giant; it’s the biggest news outside the U.S. Its transformation into an industrialized country, has fueled an enormous demand for infrastructure. Infrastructure means roads, buildings, bridges, et al. And infrastructure requires a lot of welding.

It’s impossible to compete against inexpensive Chinese manpower but China requires high-quality welding equipment and supplies. Chinese imitations don’t make the grade. Yet.

Figures are hard to come by, even by the industry’s trade associations, because the industry lagged behind in uniting and creating common standards. In addition, most welding end-users perceive welding as a necessary production input for which costs must be controlled. Consequently, the overwhelming majority evaluate welding with the objective of reducing costs. Very few study the economics associated with it with the objective of increasing welding’s value-added contribution.

The size of the U.S. market was estimated at about $6 to $8 billion dollars in 2005. We’re talking about the market size for welding equipment and supplies. It’s so vague that my client could only make a rough guess about the size of the global market. His low and high estimates were $12 to $15 billion dollars. The $3 billion dollar variance doesn’t inspire confidence (it’s 25% of his low estimate). Independent reports are available but for this article, I wasn't going to spend $4,200 to buy the report to submit a more accurate figure.

We started working with him in 2004. Teamed up with one of my consultants, the owner and his staff, we created the matrix. His company is not publicly traded so SOX doesn’t concern him. On the other hand, in early-2004, his primary factory nearly burned down and took with it almost all his records. Like too many end-users, he ignored the peril of data loss. He paid lip service to a systematic approach to data backups.

That lackadaisical approach nearly killed his business. Studies show that as much as 50% of ongoing businesses that lose most of their data go out of business within 12 months. Another 25% will close its doors by the end of the 24th month. (This subject will be covered in a forthcoming blog post.)

He became our client after that near-disaster.

Our background information is complete.

ANALYZING THOSE NUMBERS

Here, again, is the IT Investment & Strategy Matrix.



In the first two years, my client devoted 80% of his IT budget to rebuilding and creating a robust IT infrastructure.

The phrase, “robust IT infrastructure,” sounds impressive and it is impressive. The IT infrastructure refers to the roads and parking lots that data is channeled through. These are the servers, networking gear, and security-ware.

Robust (as opposed to frail) refers to creating an IT department and a secure physical location staffed with the right people implementing the right processes.

Give data its due respect. My client had a small business mentality before his accident. IT was an afterthought (just like welding is for most end-users!). One observation will suffice. His son and his son’s friend were his part-time IT staff! This, for a business with 150 employees in the $40 million dollar range.

In '04 and ’05, he invested 20% of his IT budget in people and processes to advance his business ahead of the pack.

IT veterans may disagree (as I originally did) but, in this particular case, those investments consisted of creating the disaster recovery and business continuity processes. This consisted of storage and backup equipment and the services of an outside data center. And, of course, people. Always, people.

We agreed to classify this investment as one whose objective was to advance ahead of the pack because his competitors were still using their children to run their “data centers.”

Financial institutions, which are bound by more stringent requirements to protect their records, would classify this investment under the objective of simply maintaining the status quo.

I think you’ll agree that this investment gave him a competitive advantage; a significant one at that.

THE INTERMEDIATE TIME PERIOD: YEARS 3 AND 4

The past two years, 2006 and 2007, revealed the conviction of his aspirations. His company strengthened its market position and become financially stable. Forty percent of his IT dollars went to maintaining the status quo. Half of his budget went to beefing up his infrastructure and developing web-based services (his intranet). This made sense since his operations started leaning more and more on IT. He had to upgrade not only his existing capability but also his IT redundancy.

“Redundancy” refers to the fallback system that he could fall back on whenever his primary system went down.

This is where we patted ourselves on the back. His total operating budget in 2007 was about 30% larger than it was in 2004. This roughly mirrored his company’s growth.

Now for his IT budget. In 2004, his entire IT budget was a third of his total operating budget. Three years later, in 2007, IT consumed only 15% of his total.

IT Budget as a portion of the Total Budget

He was able to devote less of his total dollars to IT since he (with our guidance) had the foresight to build a strong foundation.

A strong foundation only requires regular preventive maintenance. Like a new car, it only requires fuel, oil (regular fluid changes), and lubrication. More significant expenses are incurred only when major components like tires and belts need to be replaced.



When the matrix is charted (below), you clearly see how his IT strategy directed the allocation of his IT dollars to accomplish specific objectives.



With time, he spends less and less of his IT dollars on maintaining his core operations. At the same time, he increases his investments in the Intermediate period to accommodate the needs of his organization after he makes the acquisition. It is also during this time period that he starts investing in cutting-edge concepts. He’s not actually using unproven technology. Rather he’s configuring them in ways intended to make his ideas a reality.

When the third phase begins in 2011, he plans to ramp up his investment in those cutting-edge concepts. He hopes to create a new kind of supply chain for the upstream and downstream. If he succeeds, he will change the rules not only of the welding industry and make an impact on the nature of supply chains in general.

He also thinks that within the next five years, his company could grow to become the third largest in the world. (The current number three grossed $153 million dollars in 2001. I know that was seven years ago but that’s the most current reliable figure I got for this article.)

WHAT DID WE LEARN?

I hope you enjoyed this story as much as you learned from it.

I’d like to emphasize these points.

1. Your data is valuable. Streamline the process of capturing the data. Massage it and make it available for use.
2. Protect it. Treat data for what it is—invaluable information. It’s intangible but in many ways, it’s more valuable than property, plant, and equipment.
3. Use technology to help you accomplish your business objectives. In management-speak, this is about aligning your IT strategy to drive your goals.
4. Understand and differentiate between the nature of your objectives. Align your IT investments accordingly. Budget your IT dollars purposefully.

Your comments are welcome.

BUSINESS CONTINUITY

One step beyond Disaster Recovery

I recently advised a medium-sized commercial bank in the Philippines about a stalled project to create a business continuity solution.

Financial institutions in the Philippines do not face equivalent data integrity and safety requirements as they do here in the U.S. Still, management knew that they had to improve their IT capabilities. Their primary data center is located in their head office and it’s vulnerability surfaced at every coup attempt.

They learned about me from another client. Click here for that story.

The bank was trying to install an EMC Asynchronous SRDF solution.

I briefly worked for EMC U.S.A. as a systems engineer. I’m familiar with the product line and the subject of disaster recovery & business continuity in general.

Disaster Recovery (DR) aptly describes the process of recovering from a disaster.

DR can be illustrated with the knowledge that all hard drives crash. It’s not a question of “if,” but a question of “when.” When the drives of a “production box” crash, business grinds to a halt unless and until the data can be restored and the server restarted. The process of restoring the data and restarting the server is disaster recovery.

A “production box” is tech-speak for a computer server that’s serving a live network.

Operations can grind to a halt for any number of reasons. Fire, a software crash, human error, network failure, and a power blackout are common culprits.

DR planning begins by defining the acceptable minimum values of two factors. The first is called the Recovery Time Objective (RTO) and the second is the Recovery Point Objective (RPO).

RTO is the amount of time you require to recover your lost or damaged data in order to become operational again. Can your business tolerate being down for several days or several hours? Whether it’s days or hours, this figure is your RTO.

RPO, on the other hand, is the amount of data accumulated over time that you can tolerate losing. Can your business afford to lose a day’s worth of data? If so, then your data must be backed up on a daily basis. A retail operation, like a supermarket, that logs hundreds or thousands of transactions a day may require several backups made during the course of the day.

“Business Continuity” (BC) extends the scope of preparation, plans, and resources past DR. Those two factors, RTO and RPO, figure into this as well.

BC’s goal is to ensure the business will be able to continue operating through crises and disasters. Accomplishing that requires going beyond the processes and equipment for restoring data and replacing equipment. Indeed, BC refers to making plans and preparing resources that, among other things, will prevent the loss of data. It refers to advance preparation in order to cope with the unexpected.

A good BC plan has:

1. identified the most likely disaster scenarios and their impact on the business;
2. determined the “mission-critical,” important, and less-important processes, systems, and services of the company;
3. established its priorities for supporting the mission-critical components;
4. developed and implemented the most redundant and fault-tolerant system possible within its budget;
5. several alternate strategies
6. taught and regularly practice the plan with its people; and
7. the continuing support of senior management.

“Mission-critical” is tech-speak for the most important processes, systems, and services that a business must have in order to fulfill its mission. What is a mission? For a hospital, it could be the 24/7 availability of patient information.

“Redundant” is tech-speak for a backup that can temporarily take the place of a failed primary system.

“Fault-tolerant” is tech-speak for the characteristic of being able to withstand glitches.

Certain industries and companies require uninterrupted IT services. For them, BC is mandatory. The airline industry and financial institutions are examples. The financial sector, in fact, has to follow stringent guidelines for protecting and maintaining the security of its data. These companies must have minimal downtime. How minimal?

A calendar year has 8,760 hours. To give you an idea of the pressure to perform, consider that a 99.9% uptime is “only” equivalent to 8,751 hours.

Imagine the trouble a bank would face if it's nine non-operational hours occurred on the 15th. Employees would not receive their pay.

It turns out that a 99.99% uptime is required to stay operational 8,759 hours of the year! That’s still one hour short of the goal!

When the availability or integrity of data is compromised for any reason, businesses risk losing revenue and market share, experiencing decreased productivity, damaging their reputation, eroding their customers’ loyalty, and, in certain industries, being penalized for failing to comply with mandated regulations.

I enjoy BC planning because it's an activity that can incorporate numerous improvements for a little or no additional cost. It's a rare opportunity to deliver a lot of added value beyond the client's initial expectations.

There are several ways to go with DR and BC. You can create it in-house or outsource some or all of its aspects.

I'll cover both but the next entry will focus on the offerings of two established players in the field of storage, DR, and BC. These are the two I’m familiar with, EMC and NetApp.

DISASTER RECOVERY FROM A COUP D'ETAT

Business continuity in action!

Four years ago this month, a disaster recovery solution we created proved its worth. It saved our client, an international property consulting firm, from tanking after an attempted coup d’etat in 2003.

The Philippines has been wracked by four or five coup attempts in the last ten years. That averages to one every 24 months!

The renegades, 300 heavily armed soldiers and their leaders, barricaded themselves in several buildings in Makati’s business district. The standoff lasted for 19 hours before they surrendered. During the crisis, authorities turned off the power grid that served the contested area. My client’s office was within that grid. After it was over, her staff returned to a ransacked office.

Other companies in her building were not so fortunate. In fact, my client was the only one who was able to restore her data and resume operations as if nothing had happened.

About half of her neighbors—branch offices of large companies as well as individual businesses—did not back up at all. As for the other half, the IT manager kept their backup data offsite by bringing the media home. I learned that many of the ones who backed up discovered that their backups were too old or could not be restored.

The latter didn’t surprise me. In smaller shops, many IT administrators diligently back up their data but neglect to regularly test the media’s integrity by doing test restores. Back in the days when I was a network engineer, I was installing Citrix in a 25-desktop network. They had two Windows NT servers and had always been using Windows NT’s built-in backup utility. I knew about that utility’s notorious reputation so I challenged them to restore the data (prior to my continuing my work). Their office manager, who doubled as the IT administrator, pulled out seven tape cartridges. One by one, she tried to restore the contents of each tape. And one by one, she discovered they were empty. In fact, if I recall correctly, the MS-DOS directory listing revealed one empty folder in each tape. That’s how we sold a lot of ARCserve software back then!

As for my client, months earlier, we added additional storage. I persuaded them to configure it to do double-duty as a disaster recovery system. The hardware was kept in a cabinet closet (literally) down the hall. The system consisted of a NetApp NAS (Network Attached Storage) appliance and Symantec’s Backup Exec (System Recovery version).

I was back in the U.S. when this happened and I was able to talk them through the procedure. They were up and running by the end of the day!

BUSINESS CONTINUITY

How to turn it into a competitive advantage.

Business continuity. Disaster recovery. Colocation. Backup service providers. All of these share at least one thing in common and that’s storage.

SAAS. SOA. High availability. Regulatory compliance. Risk management. Virtualization. These topics were steps in a thread that followed this sequence:

We plan to deploy our proprietary applications over the web to serve our user
communities (Software As A Service, or SAAS).

In fact, we should start re-orienting our entire information system to become more agile and flexible (Service-Oriented Architecture or SOA).

If we do that, we’ve got to ensure our user communities have 24x7x365 accessibility (high availability).

At the same time, the law requires us to safeguard our clients’ information from unauthorized access (regulatory compliance).

Now, what are the risks that stand in our way? How can we mitigate most or all of those (risk management)?

A primary way to do that is to create shared pools of computing assets (server and storage virtualization).

I illustrated the actual sequence of thought of a client's CIO. It eventually became the company’s blueprint for the nature and sequence of IT investments. Senior management gave him the mission of ensuring that IT would support the company’s business goals. At that time, the company was a four-year old start-up that specialized in providing backroom services to hospitals.

I became their consultant based on my experience when I worked for the second largest U.S. provider of the same service. We provided the service through WANs. That was in the late-90s. The CIO above planned to provide it through web-based applications. It’s really remarkable how fast things change in eight years.

WANs are Wide-Area Networks. These are networks that connect far-flung branches to the mother ship and, occasionally, directly to each other. WANs are ubiquitous. Walgreens, Wal-mart, and Home Depot are examples of companies that use WANs extensively. You’ll know you’re dealing with a WAN when you can buy from one store and return it at another. Their WANs might not necessarily be working in real-time but here’s one example that does: bank ATMs (or Automatic Teller Machines). “Real-time” means occurring as it happens. Monday night football, for instance, is aired in real-time.

Here’s an overview of these IT objectives. I’ve been on both sides of the fence—as a CIO and as a vendor—so I can present both perspectives.

The IT goal exists to accomplish the Business Objective.

The IT Deliverable will indicate the attainment of the IT Goal.

The IT Goal can be classified according to its nature. It may be an “O,” which stands for Offensive, or a “D,” which stands for Defensive. An O goal indicates the objective confers the organization with a competitive advantage. A D is meant to protect the organization’s assets. It turns out that these are all Ds. Later on, I'll explain how these Ds can become Os.

I only presented the first four issues. They all revolve around storage.

---

If the IT GOAL is Business Continuity, then the BUSINESS OBJECTIVE must have been:

to keep the business operating through crises and disasters. While it may not be possible to operate every aspect of the business, all mission-critical aspects must continue to operate. Interruptions should be kept to a minimum. The severity of the impact should be limited. Minimize financial losses!

The IT DELIVERABLE, in that case, would have been:

an infrastructure equipped with the appropriate technology and trained people that are always prepared to provide IT services in the event of crises and disasters. Components of this IT deliverable include the IT goals described below, namely, disaster recovery, colocation, and backup service providers. The organization should be trained and periodically rehearsed.

---

If the IT GOAL is Disaster Recovery, then the BUSINESS OBJECTIVE must have been:

to get back up and running as quickly as possible in the event of a major disruption.

The IT DELIVERABLE, in that case, would have been:

a combination of elements that will allow operations to resume in the event of a disaster. Some of these elements are the colocation services and backup service providers discussed below.

---

If the IT GOAL is Colocated Services, then the BUSINESS OBJECTIVE must have been:

to provide stand-by redundancy and data protection from a geographically distant location.

The IT DELIVERABLE, in that case, would have been:

a fully-functional IT configuration housed in a facility located some distance from the primary data center. "Colocate" was a term coined to describe the service of renting space in a "hardened" facility for the purpose of housing backup equipment.

If live production data is continuously sent to the backup, it's referred to as a "hot" configuration. If not, it's a "cold" one. Regardless, the purpose of collocated service is to take over in the event of an emergency. A hot site should be able to transition within 24 hours of the start of the incident. A cold one will take a lot longer. The transition time will be measured in days.

---

If the IT GOAL is a Remote Backup Service, then the BUSINESS OBJECTIVE must have been:

to safely store data that stays accessible (through the Internet) in the event of a major disruption.

The IT DELIVERABLE, in that case, would have been:

an ongoing subscription to the services of a third-party company. Your data is stored on their equipment and administered by their people. You should pay for the highest level of support with this option. It will take days to restore your data and resume operations.

Think about it. First, you might need to replace your equipment. Second, you would need to prepare them (format the drives, install the software, etc.). Third, you might need to restore the previous IT environment (user accounts, etc.). Fourth, you need to retrieve your data (on CDs probably). Finally, you can resume operations. This description is overly simplified. I've gone through this for different clients and it's always caused them a huge headache and given us a small windfall.

In one incident involving a metro trucking company, a lightning storm fried most of their desktops and everything in their server room. It took us five calendar days (we worked through a weekend) to receive the correct hardware and software. We bought replacement servers and desktops and laptops. We had to re-order again because of incorrect or incomplete items. We also had to buy an upgraded version of their proprietary software. We dealt with five different vendors, three of which required immediate payment in full. This, in turn, put additional pressure on the owners. They were already scrambling. They were dealing with insurance, their bank, the landlord, and their staff. They already had to hire extra help since they were using paperwork to run their business.

It took a day and a half to prepare everything. Both steps, after nearly seven days, required two of us. It took another two days to configure their IT environment. We learned that their proprietary software required agents installed in all of its clients. In other words, their trucking industry software required each and every desktop that used it to be specifically prepared. We were on the phone constantly with them. Our hassles didn't stop there. They had been using an older version of this software so when we installed the current version, our client's analyst had to adapt the stored data to the newer version. So finally, nine days after the incident occurred, we began restoring the data.

I skimmed over many details. For instance, I didn't mention the hotel room we rented. Or the Internet connection that had to be set up (this happened in the late-90s). We moved back into the office on the 14th calendar day and they felt comfortable enough to let us go after the 16th day.

This experience taught us many lessons. One of them is the importance of selecting the appropriate contingency solution.

---

Everything begins with a formal process of contingency planning. Senior management should support this. It’s highly advisable for the Chief Operating Officer (COO) and the Chief Information Officer (CIO) to become active members of the planning team. In broad strokes, the team should do a comprehensive assessment of the risks the company faces, their likely impact, and the company’s various plans of action.

Contingency planning is important for another reason. The discussion can take a proactive turn if the technical side is able to enlighten the business side of the potential of carefully configuring the business continuity solution.

I’ve seen it happen several times. The light bulb goes on and suddenly the business side gets “it.”

IT being the proposition that a carefully configured solution will enable new processes. These new processes, in turn, can become competitive advantages. In so doing, the Ds suddenly turn into Os that deliver competitive advantages.

I'll explain that in a forthcoming article.

EFFECTIVE DISASTER RECOVERY & BUSINESS CONTINUITY

Best Practices


If I had to distill the best practices of an effective DR and BC program, it would boil down to a short list of four concepts. Each concept directs you to pursue a course of action that contributes to a successful Disaster Recovery (DR) & Business Continuity (BC) program.

These four concepts are:

• Work out a realistic vision of your organization’s survival objectives and develop your plan based on it. The key word is “realistic.” You need to know your organization's objectives are realistic because your plans need management's support.

• Review and regularly refine your plans. Be proactive. Your DR and BC plans are meant to be living documents. They’re useless if they sit on the shelf. Review the plan internally as well as through the eyes of experts. Unearth shortcomings. Cover the gaps. America’s leaders don’t think it’s a question of “if” but rather a question of “when” before terrorism strikes at our heartland again.

• Anticipate and adjust to your environment. Continuously. There are “big” changes like new regulations and new technologies and “small” changes like employee turnover and new phone numbers. Big or small, these changes must find their way into your plans.

• Practice for the real thing. There’s no substitute for going through the exercise. It’s probably not possible to exercise all or even most of the plan but that shouldn’t stop you from doing parts at a time. Seek senior-level sponsorship especially for this next piece. Exercise your plan with as little advance notice as possible. Disasters don’t usually announce themselves—they just happen, don’t they? Practice benefits you an important way. It exposes your plan’s flaws. Most of the time, you’ll find it’s the people that “betrays” you. Their apparent apathy is behind their lack of preparation. This reminds me of a fire drill years ago at our office on the umpteenth floor of a high-rise. Nobody took the drills seriously—even the “old-timers”—until building management hired a retired fire chief to conduct the drill. In gruff tones and with piercing eyes, he told us how quickly the flames would spread and why we would probably not burn to death. The smoke, he growled, would kill us first.

There you have it. Four common sense concepts:

1. Identify and fill in a realistic vision of your organization’s survival objectives. What are the goals and what will it take to achieve them?
2. Review and regularly refine your plans. Don’t wait for a disruptive event to update your plan. Be proactive.
3. Anticipate and adjust to your environment. Ensure senior management is involved. BC and DR are not IT concerns. They’re business concerns. IT just happens to be the one tasked with the program.
4. Practice for the real thing. Flush out your deficiencies. You can bet there’ll be many. Hire a fire chief and then begin a regular disaster awareness and training program.

Good luck!

BUSINESS CONTINUITY PLANNING

Planning takes four steps

It took a while but business continuity planning (BCP) has finally become visible on the radar screen of managers and owners of smaller businesses (< $100 million sales). It’s about time too. The state of the world today is far more volatile than it was a mere eight years ago. Nine 11 did change everything.

Every organization should plan for its continued existence in the event of a major disruption. How will it continue to operate if its operation—and existence—is disrupted by any number of natural or man-made disasters?

The practice of Business Continuity Planning (BCP) has evolved into a recognized field. Job titles that carry or imply this area now exist. Practitioners can join any number of reputable associations that promote this field. Several recognized certifications can now be earned as well.

I had the good fortune of working as a Sales Systems Engineer for the world’s largest enterprise storage vendor just before the dot com crash. I’m referring to EMC, the 800-pound gorilla of the enterprise storage space. At that time, the basic rationale behind EMC’s fabulously expensive SRDF (Symmetrix Remote Data Facility) was real-time replication for disaster recovery (DR). Under the proper guidance, it can be a short leap from DR to BCP. And that is where SRDF is now positioned—as the lynchpin of the data side of business continuity planning.

The mission of a Systems Engineer who works in Sales is to support his sales reps by designing the storage and DR solutions for customers and prospects alike. To him fell the task of dealing with the technical aspect of any proposal or project. This frequently involved making technical presentations for prospects and serving as the single point-of-contact for existing customers that were contemplating system upgrades.

Disaster recovery (DR) is a subset of the BC solution. Many fine definitions of the term abound so rather than reinvent the wheel, I will quote some of the better ones. Disaster recovery is:

• the process, policies and procedures of restoring operations that are critical to the resumption of business [Wikipedia].
• the ability of an organization to respond to a disaster or an interruption in services by implementing a disaster recovery plan to stabilize and restore the organization’s critical functions. [Disaster Recovery Journal].

Wikipedia goes on to say that…

• a disaster recovery plan (DRP) should include plans for coping with the unexpected or sudden loss of communications and/or key personnel, although these are not covered in this article, the focus of which is data protection. Disaster recovery planning is part of a larger process known as business continuity planning (BCP).

Disaster Recovery Journal continues as well…

• The management approved document that defines the resources, actions, tasks and data required to manage the technology recovery effort. Usually refers to the technology recovery effort. This is a component of the Business Continuity Management Program.

The two share the common thread in their reference to business continuity planning and its inclusion of disaster recovery within its larger scope.

I will continue this in a subsequent post. For now, let me break down the steps that BCP entails. The process follows these four steps in a logical sequence.

Identification

Identify risks and hazards that confront your business. These can be natural hazards, e.g., flooding and earthquake, or man-made risks, e.g., power outage, theft, fire, attack against your computer network. Obviously you have to draw the line at some point since it is impractical to anticipate some risks regardless of their severity. For example, two key project members in an SAP implementation project I participated in literally met an unfortunate and fatal accident. That incident delayed a major portion of the entire project until replacement personnel were hired.

Assessment

It is possible to quantitatively and qualitatively determine the likelihood, magnitude, and duration of the identified risks. Assessing risks this way allows you to prioritize them. When risks are categorized this way, you can budget your resources more rationally.

Plan Development

You now have the information to create the plans and procedures for preparing your organization to respond to and recover from interruptions. This is a high-level step and as the saying goes, the devil is in the details. This is where senior management, which should have initiated this project to begin with, should return and visibly support the BCP team. The team will need the time to extensively discuss the risks and possible solutions with functional heads. Without that support, the team will find it difficult to get the attention of the functional heads, much less their full-hearted cooperation.

Exercise

In this final step you must exercise the plan. This is the only way to learn what works and what does not. Needless to say, this is another step that senior management must support. Exercising the plan is a continuing activity. In fact, this entire process is performed iteratively. Exercising the BC plans will refine those plans and, more importantly, teach the employees how to respond if and when the real event happens.