This is not a new requirement, it has been going on for years. In a perfect world the interfaces to system services and probably to other applications would be consistent and well documented but they aren't. The interfaces would also be consistent from one type of system to another, and again they aren't. Thus it was inevitable that some of the load of developing software to help interface to the system was taken on by "third party" software houses specialising in such products, rather than in applications. In effect these third party products provided a layer of software to improve and simplify the interfaces provided by operating systems such as MVS, Windows or Unix.
The other requirement for third party software was created by the need for much better management tools than provided by IBM, et al. This requirement grew as major products were added to the basic operating system, in particular database management systems, job (workload/batch) schedulers and transaction processing managers. And then came the emphasis on communications and a further glut of actual communication software products and the inevitable management add-ons.
All this software, which is neither the core operating system, nor an application is loosely referred to as "middleware". In the early days third party TP monitors could be considered as middleware, but such products have been eliminated by core products such as CICS or Tuxedo. The same is happening with Internet related software. In the beginning of the e-commerce revolution there were no standard products and so numerous products appeared to couple the Web to applications and databases. Now Microsoft's ASP, BEA WebLogic and IBM's WebSphere have taken this middleware market over, pushing the emphasis on other middleware towards application packages and development tools.
The pattern is becoming clear now. The deficiencies in the core software systems creates a market for a variety of third party products, which are generally successfully sold because the application developers want to concentrate on the application itself and not on management and interfacing software. Gradually, with each release, IBM, Oracle, et al improve their own software, both as a natural progression and because it is an opportunity to get some of the revenue being paid to third parties. Over time the third party software house must continue to develop new products to meet the demands of any new developments or go out of business. The latter is all too often the case, but that brings with it another problem. Most users of a specific middleware product will progress to the new core software, but they will not upgrade everything overnight just to suit their primary supplier. Thus the existing third party software products must be maintained for some years after they have ceased to generate enough revenue to keep the original vendor in business. And this is where Computer Associates stepped in. CA were one of the early third party software developers, but they grew to be such a huge company by taking over other third party software as it was in its death throes. Development ceased and CA made a killing from the maintenance revenues. Eventually the customers replaced the old software with upgrades to the core systems, but by that time there were plenty of other fading software houses to feed CA.
But this situation is changing again. The PC growth upset CA, although they were able to progress by taking over their own competitors, primarily Legent, and by continuing genuine development of their Unicenter systems management tool. But once a PC software product is no longer supported, it is dead; there is no product loyalty and the old products are easier replaced than equivalent products in mainframe systems. As a result there was nothing in the PC sector for CA to feed off!
There is no doubt that things will get a bit better as the open systems movement at last gains in strength through the Internet, simplifying the interfaces between different systems and subsystems. As a result there will be less need for middleware software of all sorts. It is a sad conclusion that there are going to be a large number of failed IT companies to add to the dot.com pile.