Samsung Announces Standards-Compliant Key-Value SSD Prototype
by Billy Tallis on September 5, 2019 11:00 AM EST- Posted in
- SSDs
- Storage
- Samsung
- Enterprise SSDs
Samsung has announced a new prototype key-value SSD that is compatible with the first industry standard API for key-value storage devices. Earlier this year, the Object Drives working group of Storage Networking Industry Association (SNIA) published version 1.0 of the Key Value Storage API Specification. Samsung has added support for this new API to their ongoing key-value SSD project.
Most hard drives and SSDs expose their storage capacity through a block storage interface, where the drive stores blocks of a fixed size (typically 512 bytes or 4kB) and they are identified by Logical Block Addresses that are usually 48 or 64 bits. Key-value drives extend that model so that a drive can support variable-sized keys instead of fixed-sized LBAs, and variable-sized values instead of fixed 512B or 4kB blocks. This allows a key-value drive to be used more or less as a drop-in replacement for software key-value databases like RocksDB, and as a backend for applications built atop key-value databases.
Key-value SSDs have the potential to offload significant work from a server's CPUs when used to replace a software-based key-value database. More importantly, moving the key-value interface into the SSD itself means it can be tightly integrated with the SSD's flash translation layer, cutting out the overhead of emulating a block storage device and layering a variable-sized storage system on top of that. This means key-value SSDs can operate with much lower write amplification and higher performance than software key-value databases, with only one layer of garbage collection in the stack instead of one in the SSD and one in the database.
Samsung has been working on key-value SSDs for quite a while, and they have been publicly developing open-source software to support KV SSDs for over a year, including the basic libraries and drivers needed to access KV SSDs as well as a sample benchmarking tool and a Ceph backend. The prototype drives they have previously discussed have been based on their PM983 datacenter NVMe drives with TLC NAND, using custom firmware to enable the key-value interface. Those drives support key lengths from 4 to 255 bytes and value lengths up to 2MB, and it is likely that Samsung's new prototype is based on the same hardware platform and retains similar size limits.
Samsung's Platform Development Kit software for key-value SSDs originally supported their own software API, but now additionally supports the vendor-neutral SNIA standard API. The prototype drives are currently available for companies that are interested in developing software to use KV SSDs. Samsung's KV SSDs probably will not move from prototype status to being mass production products until after the corresponding key-value command set extension to NVMe is finalized, so that KV SSDs can be supported without needing a custom NVMe driver. The SNIA standard API for key-value drives is a high-level transport-agnostic API that can support drives using NVMe, SAS or SATA interfaces, but each of those protocols needs to be extended with key-value support.
Facebook
Twitter
RSS
48 Comments
View All Comments
Billy Tallis - Thursday, September 5, 2019 - link
Page sizes for current flash memory tend to be on the order of eg. 16kB. Even block-oriented SSDs have to do partial page programming a lot, and benefit when they can cache writes in order to program full pages at a time.I don't know what kind of data structures Samsung is using under the hood, but it has to be more complex than the flat address indirection tables used by typical SSDs, and probably looks more like a filesystem. So keys and length info are probably packed into some kind of index, while values (at least larger ones) probably get allocated whole NAND pages.
lkcl - Thursday, September 5, 2019 - link
my guess is that they found a random piece of free software off the internet, ported it to run on an embedded linux distribution such as openwrt on what is almost certainly an ARM core on the PCB of the SSD, and slapped a "server" version of the exact same API code they just released on github.that "server" version will receive the API calls over the NVMe bus, unpack them, and call whatever free software key-value library is being run on the SSD's processor.
what would actually be far more sensible - and this is something that the RISE Group at IIT Madras is giving actual serious consideration to - would be to allow the *USERS* to run the Embedded OS *of their own choice* on the actual SSD's embedded processor.
this would allow developers to put the best key-value store *OF THEIR CHOICE* on the actual bare metal (ok NVMe) SSD, whilst also allowing the manufacturers to actually still sell useful product and make a lot of money.
iaw4 - Friday, September 6, 2019 - link
I would love to experiment with my own SSD firmware. I have been looking for such a device for a long time. does anyone know where I could find a reasonably-priced SSD that allows this??lkcl - Friday, September 6, 2019 - link
some quick google searches:https://news.ycombinator.com/item?id=8791274
which in turn links hee:
http://www.openssd-project.org/
that would be a good place to start, and it turns out that "The Indilinx's BarefootTM controller is an ARM-based SATA controller used in numerous high-performance SSDs such as Corsair Memory's Extreme/Nova, Crucial Technology's M225, G.Skill's Falcon, A-RAM's Pro series, OCZ's "
iaw4 - Friday, September 6, 2019 - link
sadly, the openssd project is a $3,500 card to experiment with software. it's great, but there seems to be no path to developing one's own firmware and load it onto a commercially reasonably priced ssd. even for the indilinx chip, one would presumably still have to be able to sign code to upload it. oh, and indilinx is dead.lkcl - Sunday, September 8, 2019 - link
argh sigh, that's a pity. it means starting again from scratch, with a more up-to-date chip. well, sigh, this is precisely why IIT Madras want to actually develop their own SSD ASIC, with *actual* open standards, so that people can put the processing right where the data is.in the meantime, there is... sort-of... a way that you could produce your own sort-of SSD, and it's by using something like the Allwinner A20 (or the R40) and attach multiple ONFI NAND ICs to it. i believe that the SATA interface on the A20 and R40 can be "turned around" so that you can use that processor *as* a SATA client. it's a full ARM Cortex A7 Dual-Core (Quad Core for the R40) so is not messing about.
once you have full control over the NAND ICs like that (running your own File system, or treating the entire SSD as a block device), the sky's the limit.
this really isn't technically hard (except for dealing with wear-levelling, and there's NAND filesystem drivers to take care of that), and the sole major difference between the above and "an SSD" is... well... um... one's proprietary and mass-produced, and the other isn't.
FunBunny2 - Thursday, September 5, 2019 - link
"I am curious how the move from fixed block-oriented to dynamic-key-oriented storage was implemented in firmware."There's nothing special about fixed block hard drive. This reads more like a clone of IBM 360 C-K-D structure. Which IBM has been emulating in PC harddrives for a couple of decades.
mode_13h - Sunday, September 8, 2019 - link
I'm concerned about how well these will handle larger data. 2 MB sounds like a lot, but a lot of audio, video, and even image data is much larger than that. Any chance these drives will support partitions of both types - block and KV?There's also the issue of scalability beyond a single drive, but allegedly the standard allows for some kind of "linking" between drives?
Finally, what's the solution for replication/fault-tolerance? Does that fall entirely on the client side of the API?