For over three decades, we’ve lived with a boring truth. Disk drive performance was stuck in a rut, only doubling over all that time. One consequence was that storage architecture became frozen, with little real innovation. RAID added a boost, but at a high price. In fact, we didn’t get a break until SSDs arrived on the scene.
SSDs really upset the applecart. Per drive performance increased 1000X in just a few years and all bets were off at that point. Little did we realize that the potential of SSDs reached into stratospheric levels of millions of IOPS per drive.
All of this performance broke the standard SCSI model of the storage stack in the operating system. An interrupt-driven, verbose stack with up to seven levels of address translation just doesn’t cut the I/O rate needed. The answer is the NVMe stack, which consolidates I/O’s and interrupts efficiently and uses the power of RDMA to reduce round-trip counts and overhead dramatically. IOPS rates in excess of 20M IOPS have been demonstrated and there is still room to speed up the protocol.