Posted by
Raj Bala on Wed, Jan 04, 2012 @ 12:05 PM
I read a post recently about Instagram, the mobile photo sharing phenomenon, written by their engineering team that describes how the service is built and how it scales. Instagram allows users to take photos on their mobile phones, add a filter to the image to change the appearance, upload it to the Instagram service and then share it on various social platforms such as Facebook and Twitter.
Instagram encountered many technical scaling issues early in the life of the application because of the incredible user and data growth. They're nearing 20 million users and 50 objects created a second. That sort of growth exposes a thorny set of issues that most platforms won't ever encounter. For instance simply creating a unique ID to track each object in a distributed system with that sort of volume becomes a challenge.
One of the most interesting parts of that post to me, unsurprisingly, was about how the actual photographs are stored.
Here's what they had to say:
"The photos themselves go straight to Amazon S3, which currently stores several terabytes of photo data for us."
In other words, an application server running in a compute instance isn't a gateway (or a bottleneck) to the storage layer. The Instagram app (on the iPhone perhaps) communicates directly with the storage layer using REST over the Internet.
It wouldn't work as well any other way.
RESTful webservices are fundamental to the mobile Internet because users themselves are more mobile now than ever. Traditional ways of thinking about storage are almost irrelevant in the mobile Internet particularly where unstructured data is concerned.
The design pattern of mobile apps that access RESTful webservices will not just continue, but it will increase.
HTTP is the language of the Internet, but particularly mobile apps. It's the protocol that apps want to speak. And it's why we built Atmos.
(Photograph of the MIT Stata Center in Cambridge, MA taken with Instagram)
Regardless of what holiday you observe this time of year, I’m sure there’s some form of gift giving involved.
Whether you’re a parent of small children or, like me, find yourself providing the “mission critical” IT services to your parents like programming whatever new gadgets they get, these 3 little words “Some Assembly Required” strike fear into the heart of anyone responsible for providing the gift or service.
Do you really want to stay up all night assembling a bike? or a doll house? or pinning the desktop resolution to 800x600 for your folks? Probably not.
Life is short. Holiday time is precious. God Particle or not, some laws of Physics still apply and we can't bend the time / space continuum.
Liberation from tedious assembly has value.
Your time is probably better spent with family, friends, or significant others. If you’re passionate about your work, maybe you’d rather spend your time building a new app.
Having things thoughtfully pre-assembled for you can eliminate the effort of assembly and can give you the time to add a creative personal touch on a gift, have a 2nd cup of coffee or adult beverage with a friend, or just get more sleep and be less stressed out.
We feel the same way about storage.
When we came to market a decade ago with EMC Centera we quickly started seeing it solve difficult problems of storing unstructured content. “Assembling” your own capabilities like compliance, content authenticity and low-touch operation at cloud scale, across multiple sites, on a file system, took a lot of effort and a lot of money.
In the elapsed time, we've continued to innovate and bake-in the components you'll need for storage as a service.
So just in time for the final holiday rush, here's your check-list of capabilities we've built in that you’d otherwise have to assemble yourself:
With Atmos you can:
Or better yet?
So,
- If you’re trying to manage mobile apps, get control over an explosion of unstructured content, or
- Quickly stand up all the software needed for delivering software into a Service , or build the next killer app yourself and skip the tedious steps and click Watch Video here
- See how easy it is to build the next cool cloud app (that I may one day have to then configure for my folks) take a workshop
- See for yourself how easily storage as a service can be for your customers, developers or end-users
Freedom from tedious assembly has value. Since the first Centera, we’ve exhaustively worked to drive the complexity and tedium out of storing, managing and protecting content - regardless of where it comes from, where it needs to be stored, and whatever its business or clinical value.
In return, you can free yourself from the ‘some assembly required’ complexity at work.
Now, with the time you've saved, and tedium you’ve avoided, please invest that time by spending it with those you care about.
Happy Holidays
Posted by
Raj Bala on Tue, Dec 06, 2011 @ 08:09 AM
"Developers, developers, developers!" That refrain from Steve Ballmer from a few years ago accurately describes our focus with Atmos to provide application developers with a scalable storage back-end using modern architectures and protocols.
We're constantly thinking about how developers can utilize Atmos with their applications in an easy and efficient manner. Part of that thinking involves pattern recognition and being able to peer around corners to see how developers will build and deliver their applications to users and how we can help.
Clearly application development in the mobile market using specific platform tools (iOS, Android, etc) has a great deal of developer mind share. We already have tools that make it easy to build on those platforms, but developers are also building similarly rich experiences using technologies such as HTML5 and JavaScript without being limited to or dependent on a single client-side platform. By using HTML5 and various mobile toolkits such as Sencha and jQuery developers can build thin client applications in the browser that have similar characteristics to their thicker client counterparts.
Developers are also starting to build server-side web applications using JavaScript with Node.js. Node's primary value is that requests are all non-blocking in nature. Requests are processed asynchronously so that resources are not consumed simply waiting for other requests to complete. The net result is that more requests can be served because resources are being used more efficiently.
In all of these scenarios applications still need access to a storage infrastructure designed with mobility and scale in mind. We recently released Atmos support for JavaScript by developing a wrapper that performs the heavy lifting required to communicate with our REST API. The wrapper signs HTTP requests, sends them to Atmos, and parses the responses. The wrapper supports asynchronous I/O and event-driven programming, so it's written using the same paradigms that developers would expect when being used with AJAX, Node, etc.
The wrapper is being released under the open source BSD license; use it, modify it, and perhaps build on it too. One can get access to it by checking out a copy from the Google Code SVN.
So, yes: developers, developers, developers. That's our focus. And we're continuing to innovate to make it easier for developers to implement cloud storage.
Posted by
Raj Bala on Mon, Nov 21, 2011 @ 01:36 PM
We announced an update to our Atmos Cloud Delivery Platform (ACDP) today. ACDP is a software layer deployed on Atmos that allows service providers to deliver metered cloud storage resources in a self-service manner.
Who's a service provider these days? Everyone is a service provider. Historically, we only thought of service providers as being organizations that operated as public utilities whose mission it was to serve otherwise unrelated 3rd parties.
But that's changing; nearly every category of customer wants to deliver metered, monitored, and scalable resources to users in a self-service manner. This includes "internal customers" of an IT organization and "external customers" of a public utility.
Why would an enterprise IT organization need the same infrastructure tools as a public utility? Because enterprises want to bill internal departments only for the resources that they use --- just like a public utility does with external customers.
It's more efficient for departments to only pay for the resources they consume rather than over provisioning resources up front. IT gets a better understanding of which applications and departments consume resources and can appropriately charge back.
Enterprise IT gets many of the benefits of operating a public utility while still maintaining control. This is IT-as-a-service. Because today everyone is a service provider.
Posted by
Raj Bala on Wed, Nov 09, 2011 @ 08:05 AM
I've been thinking a lot lately about the genesis of technology products, the teams that build them, and the pace at which they're delivered to end-users. Part of what's prompting my focus on product evolution stems from an observation of an industry-wide attempt by technology companies to re-architect legacy products for this new era of cloud that we're experiencing. It's not going to work well. Let me explain.
Products need to be purposefully designed from the beginning to deliver predictable results in the cloud. Legacy products can have fundamental architecture issues that results in a fundamental incompatibility with the cloud. Certain foundational aspects of products designed specifically for cloud can't be added as an afterthought without significant effort and yet still an unnatural result is possible. Multi-tenancy is a great example. Multi-tenant architectures allow users and applications to be logically segregated at the infrastructure layer for purposes of security, isolation, consumption based billing, management, etc.
Adding multi-tenancy to an existing product is akin to adding a foundation to a house. It just can't be done seamlessly without re-designing (and rebuilding) significant parts of the entire structure. One can add piers to a insufficient foundation, but it's merely a temporary fix: the house will still have fundamental issues at the core.
Atmos is an organic product developed within EMC specifically for the Cloud. I like to think of Atmos as being 'net native because of the individual foundational elements that make up the product.
When you sequence the Atmos DNA it looks something like this:
- Speaks HTTP natively
- Multi-tenant foundation
- Scale-out architecture
- Distributed metadata
- Pragmatic design
Forging prosthetic extremities onto technology products can have odd ways of materializing. This is evident because technology products have personality traits and lineage. One could even say that products have DNA. And our DNA was clearly designed for the Cloud.
One of the most compelling, yet challenging, aspects of cloud storage is the new economic paradigm it presents. This paradigm promises tremendous cost savings and efficiencies, but in order to deliver on these promises, they demand a keen understanding of the underlying tenets which make them possible and the critical technological and process drivers necessary to exploit them. In the world of Atmos, we have synthesized these tenets into three key principles: Scale, Utilization and Variable Cost, and we’ve internalized them as the foundation on which we build our technology.
Scale
The concept of scale focuses on the notion that every environment has an inevitable amount of fixed costs, be it facilities, labor, core infrastructure, etc. By scaling resources around them (in this case storage) customers are able to spread these fixed costs over an enormous amount of capacity, thereby achieving “economies of scale” as the cost per incremental resource approaches zero. While such scale may appear easy enough to achieve (to start, one simply needs a blank check) in reality, scale is often limited by the technology one chooses to deploy.
In Atmos, scale is primarily attacked from two dimensions. To begin, Atmos focuses on automation, system healing and self-service functionality to reduce the number one cost in any environment, human capital, by dramatically increasing the amount of storage a single admin can manage. Second, Atmos’ support of a vast array of access methods, all through a single, flexible API, means customers can build a ubiquitous storage layer, across a variety of use cases, that allows them to achieve a scale unmatched by otherwise siloed, purpose-built solutions.
Utilization
While a large, well-scaled environment is a critical first step, ultimately its value is lost if not fully utilized. The concept of utilization focuses on minimizing unused, or overhead, capacity with the knowledge that every dollar spent on such resources is a dollar lost to the bottom line. While this may seem rather straight-forward and measurable, in truth an environment’s utilization is quietly under attack at every turn; from the amount of overhead required to protect one’s data, to wasted capacity provisioned to users or held back as system, or site, reserve.
In the case of Atmos, utilization is first addressed through a true multi-tenant and multi-site architecture, which allows the pooling of storage resources across countless users and numerous data centers. Second, by offering highly efficient, object-level protection schemas such as Atmos GeoProtect®, customers have powerful tools to optimize the protection they place on their data. Finally, by designing Atmos as a scale out, node-based architecture, customers can expand their environment quickly and efficiently, thereby allowing them to realize a just-in-time operational model.
Variable Cost
Once an environment is both scaled and fully utilized, a customer has reached a point where fixed costs are immaterial and the marginal cost of adding the next unit of capacity is all that remains. This final cost element is truly variable in that it grows as capacity grows and thus it defines a customer’s long term potential for economic success. While it may be tempting to avoid grappling with such long term issues up front, once a large deployment exists, customers are all but stuck with the economics they have wrought. Thus understanding the inherent variable cost implications of a given technology is a critical day one requirement.
Atmos is designed to reward customers for achieving scale, by both directly and indirectly lowering their variable costs. By tiering its pricing as capacity grows, Atmos enables customers to see direct evidence of variable cost declines with each new investment. At the same time, by developing the intelligence necessary to automate the entire storage user lifecycle, customers indirectly gain variable cost efficiencies by avoiding incremental investments in human capital and management overhead as their environments grow.
While these three principles were not born in the cloud, leveraging technologies that were provide unique opportunities to exploit and reshape them. Ultimately, when customers, partners and vendors embrace these concepts, put them into practice and measure their impact, the true power and economic benefits of the cloud are unleashed.
Scott is a Principal Marketing Business Analyst with EMC’s Cloud Infrastructure Group. He’s worked with Atmos for the past 4 years developing both business and operational models for cloud storage offerings.
Posted by
Raj Bala on Mon, Oct 17, 2011 @ 08:54 AM
Apple released Siri on the iPhone 4S this past Friday. Siri is an automated, voice directed assistant that can schedule reminders, find restaurants, and generally answer questions in natural language format. I spent a few minutes on Friday evening asking Siri questions that I would normally have asked Google on my mobile.
Siri is largely disintermediating the user from Google search to have a direct relationship with the user. Apple is removing the layers in between the user's questions and related answers with a neat interface that does natural language processing. But this post isn't necessarily meant to be about Apple --- it's about resource disintermediation.
Cloud is also about the direct to consumer model. Cloud removes the barriers to efficiently allocating resources so developers can build and deploy applications quickly. No longer are developers constrained by the time it takes to setup and tear down compute and storage resources. They can self-provision resources using point-and-click portals rather than submit requests for resources that could take weeks and perhaps months to build and deploy.
In the case of cloud, the consumers are mostly developers who are not building applications based on IT practices of the past. Developers are not writing boiler-plate code when building web and mobile applications -- they're using frameworks that provide a massive amount of functionality at the offset. Signing up to use an API is the new partnering model --- they're not waiting around for business development people to make decisions in order to begin hacking together apps to prove out concepts.
Inefficient layers are being removed to provide quicker access to resources in intuitive ways. We're living in the direct to consumer era. Self-service is the new normal.
Posted by
Raj Bala on Wed, Oct 05, 2011 @ 12:03 PM
All software, even infrastructure-based software components, should increase user experience. And it's not just limited to the interaction focused on user interface elements. As software developers we're failing if we don't keep both the user's implicit and explicit experience in mind.
The explicit experience is concerned with the more traditional user interface questions. Is the user interface intuitive? Does the UI make once complicated tasks trivial? Does the experience make the product a pleasure to use? Apple is a great example of a company with an incredible focus on user interaction in terms of UI. They obsess over it and are arguably the best in the business at explicit user experience.
We're concerned with user experience with Atmos too. We don't describe Atmos as being focused on user experience in the marketing data sheets. And that's not our leading pitch when we describe it to customers, but we certainly lead up to the conclusion that users are better served with Atmos because of our architecture and implementation.
But this is largely a transparent experience; it's the implicit experience of using a scalable cloud storage backend to applications. We're thinking about the user at every turn. Does the user get faster access to data? Can users securely access data anywhere and on any device? Is the data protected? Can developers build and deploy applications faster? Is it easier to manage?
These are just some of the questions that we ask ourselves at every step when considering new features and updates. EMC is a great example of a company with an incredible focus on the implicit user experience. We obsess over it. And we're arguably the best in the business at it.
Posted by
Raj Bala on Wed, Sep 28, 2011 @ 10:33 AM
We've been building Atmos, our cloud storage product for nearly 4 years now. We have learned a lot about how both enterprises and service providers want to sell, deploy, meter, and manage their infrastructure. In the process of enabling customers to deliver elastic storage services we've added to and adapted the product to meet market requirements.
One of the key components that we added to the Atmos family of products is the Atmos Cloud Delivery Platform (ACDP). ACDP is software designed for end-users to self-provision storage resources and for administrators to manage it better. Gone are the days when the provisioning process was the biggest barrier to getting access to storage resources quickly and efficiently. We're in the phase of IT evolution where services are delivered to end-users in metered, managed methods in both the public and private cloud.
We're holding a webcast on October 6, 2011 to share how service provider partners have built storage-as-a-service on EMC Atmos and the ACDP. We'll also be sharing updates to the ACDP 1.1 release.
If you're building a storage-as-a-service platform as a service provider or an enterprise I would highly recommend that you attend this event.
Posted by
Raj Bala on Wed, Sep 14, 2011 @ 12:56 PM
I read an interesting post by Scott Weiss titled "The Big Data Conundrum." Scott opens the post by saying:
"As data storage costs plummet, the world is storing data that would be impractical to keep just a few years ago. Think video camera feeds, web logs and GPS tracking information. We used to throw away or sample this data, but now we can store and explore it."
This echoes a sentiment that I made in a post just two days before where I said:
"The fundamental bet that businesses should be making today should be loosely based upon the ability to store extensive amounts of data efficiently, manage it with less resources, and extract value from the data that was otherwise not possible. That also happens to be the definition of Big Data."
We're entering the store everything era. The trend of storing data that we previously threw away will not just continue --- it will grow. The data exhaust created by applications such as web servers, routers, and other devices contain valuable information, but it's not necessarily the sort that needs to be stored and managed in uncompressed format and kept on tier-1 storage.
That's a great use case for Atmos, our scale-out cloud storage product. Atmos is ideal as an archive platform for data such as web server logs that need to be retained and analyzed, but not necessarily stored on the fastest disk possible. (We'll sell you that too if you really want it though.)
Developers can read and write log files (or perhaps other unstructured, file-based data) to Atmos using our REST API. One of the big differences between storing the data on Atmos and just storing the data on ordinary NAS is that developers can control various actions that can applied to an object once it's stored simply by tagging the object with descriptive key/value pairs.
Web server log files should, in most cases, be stored compressed. Compression is one of the actions that Atmos can automatically apply server-side to objects based on a configured policy. If the compression policy is configured to be triggered when metadata "key = compression" and the metadata "value = true" then any object written using this key/value pair will be stored compressed and then decompressed on a read request. Automatically. Neat.
We even optimize the way the object is compressed so that the entire file need not be decompressed just to access a portion for read or update. Balanced storage efficiency and performance? Very neat.
Here's an example of a policy created server-side that identifies which key/value pairs will trigger compression:
In this particular policy as each object is created and stored with "compression equals true" key/value pairs it will be compressed and stored by Atmos appropriately.
Business logic drives storage policy. It's the way it should work.