Enmotus Blog

A.I. For Storage

Posted by Jim O'Reilly on Dec 18, 2017 2:12:46 PM
Find me on:

As we saw in the previous part of this two-part series, “Storage for A.I.”, the performance demands of A.I. will combine with technical advances in non-volatile memory to dramatically increase performance and scale within the storage pool and also move addressing of data to a much finer granularity, the byte level rather than 4KB block. This all creates a manageability challenge that must be resolved if we are to attain the potential of A.I. systems (and next-gen computing in general).

Simply put, storage is getting complex and will become ever more so as we expand the size and use of Big Data. Rapid and agile monetization of data will be the mantra of the next decade. Consequentially, the IT industry is starting to look for ways to migrate from today’s essentially manual storage management paradigms to emulate and exceed the automation of control demonstrated in 

public clouds.C3PO head.jpg

The first step in this process is software-defined storage (SDS). Just beginning to move mainstream, this is a concept whereby storage software is divorced from the underlying hardware to allow more scalable and agile services, while reducing vendor lock-in and associated high costs. The hope is that the SDS model will engender innovation and competition, while leading to new ways to build an IT cluster.

All the evidence today, as we move to the early adopter phase of SDS, is that these goals are being met and the result is that pundits like Gartner see a strong growth in adoption over the next few years. Even so, there is a recognition that some fundamental thought processes in storage need revisiting. Computing is moving away from an IT-centric model to become business-centric. Bluntly, IT will cease to be a cost of business and become a monetizable asset.

All of this leaves manual administration in the dust. SDS, with its current benchmark standard being “policy-driven”, is an interim fix to the need for self-managing storage, but, realistically, we are not yet advanced enough for mass deployment of “self-aware” systems based on A.I. approaches. So, the near future is a migration to semi-manual methods that leverage valuable and scarce admin expertise, while allowing delegation of control to an extent to departmental staff.

A.I. storage management is the ultimate step to meet needs of an A.I. computing world. The complexities of monitoring tuning and repairing a system beginning in the early 2020’s will demand that IT eats its own dog-food. Several major storage appliance vendors have announced projects aimed at bringing A.I. to bear on specific management tasks, so we can be confident that activity is beginning, but we still need an infrastructure around storage that allows A.I. the level of storage system knowledge that it takes to make smart, timely decisions.

Storage monitoring – devices, links, software and such – will need a major upgrade. We are looking at real-time detection and a much finer granularity of information here. But that’s not enough! Remember, we are heading towards “business-centricity” and that implies that any instrumentation needs to cover applications’ use of storage and understand actual content and how it is being used.

As in many A.I. environments, this is a completely open-ended search for knowledge. A.I. works better the more it knows! One can reasonably expect that storage A.I. will be a long-term business segment with an ongoing need for innovation.

What sorts of metrics are involved? New storage paradigms point to extended metadata as a key element as a key control mechanism. Content awareness, such as whole object indexing and data typing, are facilitated by the performance boosts coming from solid-state storage. The speed of networks and the ability to add service functions into a virtual server pool mean that adding detection of network and drive bottlenecks and failures becomes easier, while that same SSD performance caters well to background data tiering.

Metrication is well under way and already leading A.I. to a great extent. The problem is the reducing of those metrics to information in near-real-time, which is where we are seeing projects to create (interim) point solutions today and evolve these to A.I. as that technology and the underlying hardware accelerators mature.

I’ve seen comments that manual storage administration was the “Stone Age” of computing and SDS is the “Neolithic”. A.I. will move the pace of IT and of the businesses IT serves to a newer level of achievement. This is a game-changer!


Ready for a Free Trial?    Free Trial

Topics: NVMe, Data Center, NVMe over Fibre, enmotus, data analytics, NVDIMM, artificial intelligence