S3 is the New Narrow Waist

Over the past 15 years we’ve experienced a true revolution in data, the ramifications of which are still playing out. No, I’m not talking about the cloud, although it plays a key role. I’m talking about something more fundamental: the fact that most of the data we’re generating today is just blobs of bytes — unstructured, as they say.

It’s the incredible masses of video, audio, photos, logs, backups and text we’re collecting from every corner of the network. And it’s dwarfing everything else. In fact, by 2025, 80% of the data we generate will be unstructured, according to IDC.

It’s a story you’ve undoubtedly heard. And as you might already know, most of these bytes are stored as objects, often in absolutely massive object storage systems. How massive? The granddaddy, Amazon Simple Storage Service (S3), now holds well over 100 trillion objects — that’s 12,500 for every human on the planet.

But why is object storage so successful?

Object storage, as opposed to the file systems, block devices and databases we know and love, provides a super-simple, scalable and easily connected solution for storing blobs and their metadata. It was born cloud native to operate in a networked world of REST-based protocols. Blobs themselves can be nearly any size, from literally zero to 50 billion bytes. They can be easily referenced, accessed, replicated, cached, you name it. And the hierarchy is almost completely flat. As the name implies, it’s a beautifully simple storage service.

Databases, of course, are still an essential part of the picture. It’s just that we use them to store the addresses to these blobs — addresses that are universally unique and fetchable over the network via simple REST calls. 

And what are those REST calls? There is only one dominant answer anymore, and it’s the S3 API. There have been many contenders over the years, from open source ones such as Swift from OpenStack to proprietary ones provided by the familiar storage vendors to other hyperscale APIs. But today nearly all systems that want to interface with object storage speak the S3 API.  The market set the standard, and it’s long been game over for how we GET and PUT our data.

In short, S3 is the new narrow waist for storage. 

Wait, what? 

Let’s go back to the early days of the internet and this guy, Vint Cerf, who, along with Bob Kahn, gave us the internet protocol.

What you can’t quite read on his shirt is the pithy phrase, “IP on everything,” which was shorthand for, “Hey, let’s standardize on IP as the protocol for the network layer.” At the time there were many possibilities: DECNet, IPX, Vines, etc., all of which, in their heterogeneity, made interconnectivity, and hence the singular internet, impossible. What was needed was a common lingua franca.

It’s obvious once you picture it:

At the waist, everything speaks internet protocol, while above and below, any number of different protocols can flourish. This facilitates both the freedom and rapid evolution we’re familiar with on the internet, while also providing the intercommunication that allows us all to connect.

We’ve seen the identical effect in databases with SQL, as the brilliant Ajay Kulkarni and Mike Freedman from Timescale point out. This pattern seems to repeat itself both in technology and even biology, where DNA acts as the common language for animals and the viruses that prey on them. 

OK, cool. But why do I care? 

Well, this standardization on object storage with S3 as the API creates a powerful connection point that empowers us to manage multiple disruptive and inescapable trends in our industry, namely:

1. We’re creating unstructured data at an absolutely explosive rate.

2. We’re creating that data at all points in the network — edge, data center and cloud — and consequently we have to move it around to wherever we need it, whenever we need it.

3. We’re using that data in unprecedented volumes for traditional analytics, machine learning training and real-time responsiveness, without which our businesses can no longer compete.

And simultaneously we still need to achieve all the traditional must-haves for data management around retention, protection and availability, with ever fewer humans to manage it and at ever lower costs per byte. No wonder we’re all stressed out! 

The good news is that object storage with the S3 API as the narrow waist dramatically simplifies all these challenges by providing interoperability. And it’s interoperability that creates the essential advantages we need to manage these massive disruptions.  Critically, this combo circumvents platform lock-in and fosters competition, which provides us options in the marketplace. This consequently drives down costs and encourages innovation while simultaneously separating concerns and allowing specialization at different layers of the stack.

There simply is no magic bullet or one-size-fits-all solution that can provide for our unstructured data use cases. Definitely not AWS by itself. But object storage and the S3 API as a common language enables us to compose a constellation of interoperable tools into an end-to-end solution for each of our data challenges. 

This is why, for example, when Cloudflare launched its new object storage service, R2, it also provided a slurper to pull objects straight out of AWS. It’s why many providers formed the Bandwidth Alliance to drive down egress fees and accelerate object portability. It’s why tools such as Rook serve to make Ceph and other object storage systems seamlessly available to every container in every pod on every platform. And it’s why literally all the higher-level services in data retention, protection, visibility and portability speak the S3 API. 

Given this is the world we already live in, you might picture the new S3 narrow waist hourglass like this: 

With this in mind, one can see how we can create a true hybrid cloud for data management, with S3 as the lingua franca and placement decisions made based on cost, performance, compliance and ease of management. 

Where Platina Comes In

This is the vision that motivates us here at Platina: a world where you have the ability to put your unstructured object data where you need it, when you need it, at the lowest possible cost, in on-premises data centers and edge locations that you control.

It’s a vision that makes the provisioning and management of on-prem object storage as easy and fluid as in the public cloud, and it opens the door to truly controlling where your data lives. You don’t have to be held hostage by punitive egress charges or exorbitantly expensive legacy storage appliances.

We’re making it easy to run a modern, cloud-like infrastructure on-premises. By automating the day-to-day operations of the systems you need to run cloud native applications at scale, you can allocate IT resources more efficiently and unlock the full value of your data.

Want to learn more? Let’s talk!