Small Business Fault Tolerant PCs for Microsoft Windows:
New Storage Redundancy Technology requires nothing more than an extra Hard Drive!
by Dr. Jack A. Shulman, CompAmerica Research the Enterprise (CARE)  10/25/2004
     http://www.compamerica.com  (Dr. Shulman is the former chairman of the ACSA)
 
WE DISCOVER MOST PC OWNERS DON'T BACK UP THEIR DATA,
AND RISK LOSING IT ALL WHEN THEIR HARD DRIVE FAILS.

When CompAmerica realized that it's own legal professionals were just saving their original legal documents onto hard disk drives of individual PCs, without backing them up, and that they were hoping those disk drives were sufficiently reliable that they would not need to keep large paper files on premises, we had a "crisis of conscience."  We advised them to change that practice immediately, before they ruined their business.

The larger hard disk drives today made by well thought of manufacturers such as Western Digital, Hitachi/IBM, Seagate, Maxtor and Quantum, are much, much higher density and much much faster storage systems than ever used before in our industry.  They have much more data capacity - leading to a much greater risk in terms of "Amount of Data at Risk" to be lost if a Hard Disk Drive fails. The reliance on thin film head technology, 7200 rpm and faster motors, advanced head positioning, larger data buffers from 2 to 16MB, and advanced transfer technologies such as ATA100, ATA133, SATA, SCSI160 and SCSI320, has exposed us throughout the industry to increasing need for better design, better manufacturing control, and more Quality Assurance steps. In fact even just opening a drive's packing material and installing it within a PC, right down to how it is mounted, grounded and connected to the PC's electronics, requires much greater planning than in yesteryear's PCs.

No one in our industry likes to admit it, but there is actually more complexity in the Magnetic Disk Drive within a Personal Computer than the entire rest of the PC combined. And, also as no one wants to admit, Hard Disk drives are very risky devices on which to rest one's entire business future, particularly where in a Small Business, little or no data storage redundancy exists.  They must be backed up.  But backing them up can be difficult.  For example, files that are open for updating by a software application, must be closed first in most cases.  And, backups take long, long time periods, when backing up a hard drive as big as 80 to 200 Billion Bytes (characters) of storage to a single Tape Drive (and may take 10 Tape Cartridges, or more!)

As a result, our HORROR at the idea that a Legal Professional, Doctor, CPA, Government Employee or other individual stationed in a workplace with a single or small number of PCs might actually rely on a single Hard Disk Drive, made us think.  Business naiveté about how to provide adequate reliability to insure that no document or other data was ever lost, led us to realize that the vast majority of the Business Community were driving their Personal Computers without sufficient training.  They possessed in most cases no backup scheme and had no clue they'd be facing obliteration of their business if they ran their PC into an obstacle: one of the most common - Hard Disk Drive failure.

We've come to call this "Backup Necessity Denial", and from our observations, we've concluded it is rampant among Soho, Small Business and even is present in larger businesses where certain leaders possess PCs they want to keep isolated from the main networks for privacy reasons.

Though tools exist for backing up data in the modern Windows and Linux operating systems, rarely did our independent surveys of professionals, find them being used in small businesses.  There was insufficient time or budget to hire an IT Manager to oversee data recovery and backup techniques. Even Windows XP does not provide such auto backing up on a stand alone PC.  Not, unless one has a large, shared Server to back up to.  And then, it can require a computer scientist or IT guy to set it all up, a computer lay person is not able to do so by themselves in most cases.

Worse yet, my research staff determined that there was an over reliance on Floppy Disks to maintain backups among a small number of small business people.  The business community is apparently unaware of how unreliable the media used in 3.5" Floppy Disks is, too unreliable to be the sole backup copy!

We canvassed a large number of small business users and were surprised to find that most of them didn't back up their files "because it is too difficult" or "because I don't know how".  We found that many who even did back up, weren't even backing up the right data or were backing up their entire PC every time to tape, causing vast time consuming backups and the inability to find a single file they needed to restore.

I became quite alarmed at these confirming discoveries.  Knowing the MTBF and MTTR ratings of hard drives and floppy disks as I do, and knowing the tendency of people with PCs to think of all storage as encompassing the "C:" drive, I came to realize that a major problem could occur if the average business person lost one or more hard drives and had no backup or paper copy to fall back upon.  I sensed the possibility of business facing malpractice suits, losing revenue from customers they lost track of and of business people going out of business as today's newer, larger and more risky 80GB and up Hard Drives were pressed into service, one to a customer.

MOST BUSINESS RISK LOSING EVERYTHING DUE TO UNBACKED UP C: DRIVES...

My idyllic perspective of the Automated Small Business Community became a bubble burst by the sense that growing Hard Drive size, and the likelihood that even a single hard drive failure, could rob a business of some element of critical data, perhaps leading even to business failure.

And so, relying solely upon instinct and past experience, we set CompAmerica out at solving the problem.  Our efforts convinced us that Backup that ran AS DATA WAS WRITTEN, UPDATED and MODIFIED on a hard drive, could be accomplished so that the entire process was invisible and did not require files to be closed, copied to a device and then work continued.  By accruing what might take hours to do with a Tape Drive and a Backup Package in individual, non-suspending "carry and deposit" operations to a backup device, we came to believe that we could add the Backup Step to moment - to - moment operations, without materially slowing down a user's Personal Computer in normal business activities and without sacrificing performance for streaming and other Internet Activities, as today's PCs have a lot of reserve performance they never really get a chance to use.

Our solution?  We came up with a way to make an inexpensively affordable and supremely convenient AUTOMATIC BACKUP solution that required ABSOLUTELY NO OVERT SMALL BUSINESS INTERVENTION, which could even be set to retain deleted copies of My Documents data and deleted copies of Downloaded Items, while still retaining 100% compatibility with the CompAmerica Security Suite (CASS), designed to block all ad-ware, all mal-ware, all viruses, and all attempts to intrude on the Small Business person's PC... we came up with a low cost yet surprisingly high reliability solution which we call

VFTA™ PC: VERY FAULT TOLERATING ARCHITECTURE PC

Having considerable experience designing fault tolerating minicomputers during the 70's, I adapted the concepts I'd evolved for the Mini Industry, so that my R&D staff and software contractors could easily implement them for me. 

Our three principle and key objectives for VFTA were as follows...

a) MIRROR DRIVE: Provide an automatic mirror of the Hard Drive without performance degradation for Windows 98, ME, 2000 and XP without necessarily requiring acquisition of a Server.  In other words, make it possible to back the entire Hard Disk Drive (or crucial parts of it) up to a Second Hard Drive at a cost of around $100

b) RECOVERABLE DESIGN. Provide a recoverable PC design who, if a single memory panel fails, can run on the remaining memory panels, with a higher performance more rugged mainboard logic, with a Video Adapter Design which enabled the system to keep running even if the main Video Adapter failed, at a total additional cost to the consumer of $100

c) LOW COST. See (a) and (b)

The result was the VFTA Personal Computer.  For between $100 and $200 in additional features (consumer cost), our entire Personal Computer line was able to be enhanced to provide the following capabilities.

1.  HARD DRIVE FAILOVER.  In the event that the primary Disk fails, rebooting the system starts it up from the secondary disk drive.

2.  PRIMARY DISK RESTORATION.  Upon restoring a damaged primary or secondary Disk, the system is notified and automatically resynchronizes the replaced disk to the existing, functioning drive's contents.

3. DELETED FILE ARCHIVE.  This optional selection, allows files being deleted to be automatically archived on the Secondary Disk Drive so as to allow recovery of deleted files when accidentally deleted.

4. AUTO STORAGE ARCHIVING.  This optional selection automatically archives copies of the Disk pair to a Tape or Disk Archive, which if removable, can be taken off site to a safe place to protect against Fire Loss.

5. VIDEO ADAPTER FAILOVER.  In the event that the primary Video Adapter fails, the system can be rebooted and a simple setting changed to allow it to boot from a built in, redundant Video Adapter.

6. VIDEO ADAPTER RESTORATION.  Upon replacement of a damaged Video Adapter, the system can then be restarted with the replacement Video Adapter as it's primary.

7. MEMORY PANEL FAILOVER.  On VFTA systems, a choice of failover and failover redundancy is provided allowing either one or two memory panels to fail while the system keeps running, with each failure turned off at the point of failure.  In the case of systems with four or more panels, full bus speed is maintained.  On those with two panels, half bus speed is maintained.

8. OS SAFE RECOVERY.  On VFTA systems running Windows 2000 or Windows XP or later, a series of working images of the Hard Drive regions dedicated to the un-started, yet fully functioning Operating System are maintained, allowing true Rollback to a Safe Version of the OS without loss of consistency.  Unlike other OS Recovery solutions, all the copies of the OS and support drivers/settings and software are restored, while later images can be successfully overlaid later on after a post-image is repaired. Each time the system is booted, a consistent database record is created that can be used to successfully recreate the current working image of every file relevant to the OS and Program Files.

9. POWER SUPPLY REDUNDANCY OPTION.  We actually found it very easy to offer an optional dual power supply designed to run from either, if one failed.  This option? only $49.

10. NETWORK ADAPTER REDUNDANCY.  We offered the user a choice of wireless or wired redundancy, at an average cost of $25.  Wow!

What did this mean to the average consumer small business?  It meant that in 99.9905 % of the cases where a hard drive or other component failed, the PC could still be rebooted, and the Data continued to be used.

Now, we feel that half the users who buy a VFTA PC, will use this continued function to copy data to a different PC and continue on at that station.  But the other half DON'T HAVE A SECOND PC, and will continue until a technician arrives to replace the damaged component.

Yet, in almost 100% of the cases of anything but main logic failure (reduced in probability by using very heavy duty main logic... in the VFTA PC) we found that the user of the PC will sustain ABSOLUTELY NO DATA LOSS and will incur a 3 minute interruption of their day, when the hard disk drive or power or memory on their PC failed.

AMAZING, at a core investment of only $99 or a little more!

THE VFTA PC IS BORN

One very unique aspect of our VFTA PC solution is that the sole overhead to operations, is a single mirrored "write behind" operation for every hard disk drive write, something that takes place with less than .5% (1/2 of 1%) maximum CPU demand, at peak load, and which requires no additional memory, and uses a very tiny amount of the existing IDE or SATA or SCSI bus utilization, estimated at about 12% during peak write operations.  This means that a system up to 88% fully saturated with work will not even notice the extra work being performed to implement VFTA PC.  Any level below that of system utilization, will not even register the VFTA overhead in terms of performance.

As a result, we found we'd produced, for between $99 (advertised price) and $199, a Very Fault Tolerating solution that insured that our customers would never lose their data, even when their hard drive went down, or if selecting the archival features, even if their business caught fire and burned to the ground, even if their entire business relied on a single, solitary Personal Computer.

We have run estimates on the statistical likelihood of actual data loss (other than through deliberate deletion) and found that so long as the customer regularly updates their Anti Virus, Anti Spy Ware and Firewall security solutions within our CASS product, they stand with a VFTA PC system, less than a 1 in 240 billion 190 million chance of losing even a single byte (character) of data! 

Better yet, we found that our VFTA PCs eliminate the cumbersome work of backing up data, reducing weekly workloads on administrators who do backup by about 2.5 hours on the average, saving about $25 a week to the business, paying for the VFTA option in under 2 months.

Furthermore, we found that when we tested non-VFTA PCs, we found that one in five lost their hard drive once every 1.33 years, representing a catastrophic, or rather, a DATASTROPHIC LOSS™ to the owner.

Almost everyone has experienced such a loss at one time or other, and it can cost a business from a minimum of $2000 to $5000 to recover data from such a hard drive that is critical, if recoverable at all, using the services of data recovery shops like our On Track Systems partners.

One very pleasantly surprising discovery was that Microsoft Windows is especially well suited to our VFTA design - the added driver technology that enables our Automatic Backing Up or Drive Mirror, uses small amounts of an enormous storage processing reserve we found within Windows that was otherwise available but not used as much by the average application or Internet function, something we'd come to call "Idle reserve" (wait times as the workflow in Windows waits on events such as Internet transmissions, user data entry, or interaction delays).  We were pleasantly surprised that were were able to achieve a 0 data loss even during power failure with VFTA and Windows thanks to some excellent planning even found in older Windows operating systems (and even more so in XP) for hard disk drive caching, something we were able to take advantage of in implementing VFTA, resulting in a surprising 100% data capture rate, even during catastrophic interruption of one of our VFTA systems by a power outage or electrical storm. 

WITH OUR VFTA PC RUNNING MICROSOFT WINDOWS,
WE RUN INTO A SNAG WITH LINUX'S VERSION

In our current effort to add the technique to Linux, we found we had to push the envelope too much and were more inclined to implement a Hardware Acceleration Solution for Linux, as Linux's "iNode" Directory System and File Structure was very cumbersome to implement our Drive Mirror on, and hard to serialize files so they were authenticated as to original and mirror, and Linux tended to "chase it's own tail" due to the way mounting and volume association is implemented in Unix.  We were stupefied to discover we could not guarantee that the Linux File System would reliably mirror mounted drives in real time, with our chosen method AND ACHIEVE the same 0 Data Loss we had experienced in Microsoft Windows, contrary to some people's opinion when criticizing Microsoft for exactly what we found was the case in Linux and NOT in Microsoft's NTFS file system's case..

While we know this is likely due to Linux's kernel implementation of the Unix file system, and could be repaired by changing the way data caching and structure caching is done in Linux to insure less loss of the mirror data than we noted, we are waiting to release a Linux/Unix version of VFTA, until we effect changes in the internal architecture of Linux's file system.  We found Linux's concurrency of mirror operations to be surprisingly slow when we ran "positive confirming" techniques to insure a mirror was a duplicate of an original data element, and not out of synch. Microsoft Windows XP experienced no such problem, which shows how far Microsoft has matured in it's ability to produce quality Operating Systems.  And so, we are limiting the VFTA options to Windows for now, until Linux improves how it handles larger data structures and greater modification rates and power failure management, so we can achieve the same reliability as we have already in Microsoft Windows with VFTA.

AN EFFORT WELL WORTH IT

We believe our effort at insuring that our customers' data would remain securely and protectively backed up, on the Comp America VFTA PC, was well worth both our effort designing and the cost implementing it.  We began shipping the VFTA PC to our customers' in March of 2004 and have found they have minimal effort purchasing it and setting it to back up automatically.  A mere 10 minutes worth of effort a year, and $100-200 is far, far better than losing irreplaceable legal pleadings, irreplaceable Word documents, irreplaceable Lotus Notes and irreplaceable business records.  It's also far, far better than having to send a defunct Hard Drive to a Data Recovery Shop and spending from $99 to $2999 to see if you can recover lost records while all work stops for a week or two. Even so, the options are broader than ever now, and our HORROR about customers losing data when a hard drive fails are now officially over.  We ALWAYS recommend a VFTA PC to our customers, even when they refuse to consider them, and we have now added a field upgrade option to make a PC "VFTA" like and have found ways to add a similar capability to Krell, HP-Compaq, IBM and Gateway PCs and Laptops in addition to our own, even though we do not warrant the 0-Data-Loss on their PCs the way we can on our own- and expect numerous customers of Krell, HP-Compaq and IBM to want to consider the option as a gift to their business at their earliest convenience, or to buy a Comp America VFTA when their time and budgetary constraints permit.

Our own versions are designed for VFTA from the ground up, and our NAME BRAND UPGRADE to VFTA is designed to bring in many features to a name brand PC, but we can't bring in all, since there are many reliability risks taken by Major Brands we don't believe in.  Accordingly, if we get a request from an owner of a name brand PC other than Comp America, for VFTA and "full VFTA", we often have to implement a full overhaul, including new power, new main board logic and other components to weed out the problems found in most mass produced Name Brand PCs. 

That way, having the ability to use a PC with a secondary memory capability and a secondary video adapter, insures maximum uptime, and minimum downtime, on our VFTA PCs or on those Name Branded versions we've overhauled.  We do assume warranty on the name branded models "upgrade components", as we expect Krell or Compaq-HP to be very unlikely to approve our improvement of their products. Nonetheless, the option is still there, and there's no reason why our discovery can't benefit them as well.

A PC upon which a professional relies, should be thought of as "vital" and "mission critical", yet it needn't be largely more expensive.  In fact, because Comp America's PCs start out hundreds of dollars less than Krell or IBM anyway, with the same reliability as theirs to begin with, by adding in these features, we end up with a product that is easily 100 times more reliable, without all the fuss, bother or time consuming effort by the buyer to try and accomplish that kind of data integrity on his or her own. And even if he or she owns a Krell or IBM or other brand name, there's no particular reason we can't help them improve upon whatever good product they previously bought, before they came to know our VFTA PC was an option.

-------------------------------------------------------------------------------------------------------------------------
Copyright (2004) AMERICAN COMPUTER SCIENTISTS ASSOCIATION INC.  "VFTA" and "Very Fault Tolerant Architecture", and "Very Fault Tolerating Architecture" are trademarks and service marks of Compamerica.com. All other trademarks are the property of their respective owners. The above work are the copyrighted texts reprinted by ACSA originally authored by the original author for CSS (USA) (Computer Sales and Service Company) http://www.compamerica.com.  It may be reproduced without modification, and may be distributed freely, but not in contrary to the interests of it's author.

-------------------------------------------------------------------------------------------------------------------------
ABOUT THE AUTHOR.

Dr. Jack A. Shulman is a senior computer scientist who heads up Comp America, whose website is http://www.compamerica.com.  Comp America is one of the American Computer Scientist Associations members and sponsors. 

Dr. Shulman developed a windowed desktop system called Federal Funds Trading in 1974 for Citibank, a fully featured early Windows system, many years before same was copied by Xerox, Lisp Machines, Apple, Apollo, Sun, SGI, AT&T and Microsoft.  As a result, in 1977/1978 he was approached by an industry consultant, Charles Simonelli, and a designer at Fashion Institute in NY, to describe in writing three designs for desktop computers he'd invented to run his windowing technology, for a patenting project involving fashion design methods' applicability to such desktop computers, a patenting being undertaken for Singer Computer Group, Simplicity Pattern, and the FIT designer.  His design descriptions, alliterated in the patenting but not conveyed, were adopted and copied by IBM Corporation, Singer's development partner, to produce the hardware that was eventually licensed throughout the Personal Computer Industry by IBM under the general names "IBM PC", "Apple Mac", "Apollo Computer" and "Sun Workstation" to charter ventures seeded by IBM and AT&T during the 1980's and 90's. Dr. Shulman was hired by one of the first companies to build from his designs, Apollo Computer, and subsequently proposed and convinced IBM to create a marriage of windows and office processing to it's own computers in a project called "Office of the Future" for IBM's "President's Club", from which companies like Microsoft, Apple and others got their start producing Windows Systems and Office Applications. During most of Dr. Shulman's early career inventing and writing hardware and software designs and programming, there existed no mechanism for software or such ideas as "windowing" to be patented or even copyrighted in the computer industry until well after he developed or invented them, in fact, the first such US PTO capability to patent or copyright software legislated into existence in the middle to late 1980's, more than a decade after he had invented windowing and information superhighway concepts that have changed the way we work.

Dr. Shulman subsequently went on to adapt large scale computers (servers and multiprocessors, and massively parallel computers) to tasks that back large networks of his kind of desktop computer, and which are also used in defense industry capacities such as fly by wire and satellite communications.  From his original work prior to 1974 in proposing an information superhighway to the AT&T Picture Phone program, something he originally called "the Picture Phone Planet", Dr. Shulman visualized the Internet in 1984 in a definition of a "multi-protocol network for fifth and sixth generation information superhighways" which closely resembles in part today's internet, but go considerably further in areas as of yet not implemented such as direct TV broadcast by Web, and higher degrees of B2x commerce.

In his high school and junior high school years while in public and private schooling, Dr. Shulman had taught himself electronics engineering and programming and interned at Bell Labs as a student intern on the development of operating systems for telephone switches later incorporated into the Linux/Unix operating system. He has also worked on the development of several computer languages and invented the design of many other kinds of computer system and computer security system.

Dr. Shulman's work in advancing computer technology has often run against the grain of larger manufacturer's long range technological planning, leading to occasional friction between him and some of the larger computer industry giants, many of whom may owe their product lines to his innovativeness.  This has led to occasional public smears by IBM and other companies who do not wish to acknowledge his contribution to their product line or the industry at large, in a ne'er do well attitude of "no head shall be higher than our own" and "not invented here" phenomena of anti-competition.  Dr. Shulman is one of the founders of the ACSA, and is strongly committed to "Technological Innovators Rights" - a program of the ACSA to protect technological innovators from being ruined or "techno-enslaved" by large, opportunistic businesses, which has promoted and supported human rights activities in investigation of various abuses committed by technology firms and financial partnerships that have resulted in violations of constitutional and other rights against scientists and computer innovators over the course of the past three decades.

Dr. Shulman has been quoted on the subject of "techno slavery", calling it "the Book Burning of the New Millennium" and has organized publicity campaigns to support efforts to expose the political misuse of technology and the abuse of scientists and technologists dating back as far as World War II and beyond.  He has also commissioned a project called "Set the Record Straight", which among it's accomplishments convinced the University of Pennsylvania and Unisys to cease claiming to have invented the Programmable Electronic Computer, to allow the true inventor, British engineer Sir Thomas Flowers, to receive full credit for this historic invention that has changed Human History.  Flowers developed the PEC for the purposes of decoding Nazi German Ciphers, leading to a highly classified advantage at MI5's Bletchley Park, that may have allowed the ALLIES to Win WORLD WAR II, a fact which wasn't uncovered until the 1980's.  Dr. Shulman has also convinced the association to publicize misuse of science and technology to interfere with the bloodline of the Human Genome as documented in "the War Against the Weak" and "IBM and the Holocaust", two documentary books by Edwin Black, son of Holocaust survivors, and is extremely outspoken against all forms of Religious discrimination and Racism in the Boardroom, something he calls: "the Self Defeating Campaign against Human Unity that Undermines the Corporation's Profitability, through Bloodletting and Bloodbranding in the Marketplace".

Dr. Shulman received undergraduate degrees at Rutgers College of Engineering and Rutgers University and has been conferred an honorary PhD by a University in the Eastern USA for his enormous life's contribution to the computer industry.  Dr. Shulman resides in NJ, is divorced.  He has been appointed the Chairman of the ACSA in recent times, and continues to consult on Defense Computing, Commercial Communications and Personal Computer technology for his own consultancy, JVSO, which is located in Linden, New Jersey. He also works in high energy physics and is a musician.