The improvements in magnetic storage technology are almost unbelievable. I recall the days, only 20 years ago when data processing systems had to manipulate data between a fixed 5MB and a removable 5MB cartridge.
The multi-platter CDC drives with 50MB capacity were a wonder to behold; like tape drives, they were essential for movie makers because you could see something happening through the perspex covers. Nowadays I sometimes wonder if I have enough capacity on my PC drive, is 6GB enough? And not only has the capacity increased out of all proportion, but so has the speed and, even more, the size. These 2.5 cm drives they put in cameras hold a gigabyte!
These phenomenal advances have had significant impact on the whole IT industry. It is not just a matter of the same as before but cheaper, it is a whole new range of applications that have been enabled by availability of storage. The obvious advance is in the ability to process graphics. Even word processors now use graphical representation of text and my PC disc is full of photographs and sound tracks. The next trend is to include more moving imagery, whether we want it or not. The important thing to note about all this progress is that it introduces new systems into the IT world, but doesn’t replace the earlier requirements. We still have to run the payroll and invoicing systems! But these too have been influenced by technical advances. Payroll systems now need to handle communication with banks and they have evolved into human resource systems, which now include employee photographs, yet another concept enabled by storage advances. Historic data is now kept rather than updated, for CRM applications.
The outcome of all this is that storage technology advances have affected all sectors of IT. But the development has not been planned, rather it has "happened". Thus since all these developments took place under different managers (and from different budgets) the physical storage is all over the place. This would not be such a problem if all these developments stayed independent of each other, but they haven’t. Text is stored on individual PCs in many cases, most of which is valueless, but manuals, documents, groupware, etc., all need communal storage. Data processing systems should have been easier to manage, but in practice this proved difficult. Today there has been a lot of talk about Enterprise Resource Planning (ERP), a key feature of which is to base various application sub-systems on a common database, but historically new application requirements were introduced as new systems and data was transferred between the applications as needed. Separate applications have created serious storage management problems, but in fact it is worse because many applications were introduced on new database management software and on new and different hardware and operating systems.
In an ideal world all the existing systems could be replaced by one integrated system, with only one RDBMS and one physical storage pool, but it will never happen. Already there are a lot of complaints that after all the effort of introducing an ERP system, there have been few gains, it is just too complex. Thus we will have to live with the problems of running and managing disparate systems, probably spending more on automating workflow with management tools.
It would be very nice to have a common database, which could then be run as a separate entity to which a variety of applications could be interfaced from a wide range of clients, both local and remote. But even this is being too ambitious. And so attention has turned to the lowest common denominator, the physical storage. The idea of a common file system has been extensively used in PC local area networks, Novell Netware being the most common, but it was also common with NFS on Unix servers. NFS and extensions has been used for large scale systems, but the shear size of the storage requirements and the need to continually expand it has lead to a compromise in which various computers are connected by a very high speed network to a storage sub-system. Multiple sub-systems can be added as each one reaches its capacity and there lies one of the key problems. If users are not to be tied into one supplier, then they need to be able to add storage sub-systems from mixed suppliers, and that means standards, both physical, networking protocols and software. It is not likely to happen in the IT industry and we will still be tied to specific suppliers.
