Dispersed Storage Blog

Self-organizing systems (Part 2)
Last week, we kicked-off a blog series titled Trends in the advancement of storage virtualization whose impetus stemmed from the recent SNW in Phoenix, and the conclusion that the storage virtualization products on the market today are a good first step, but are going to have to evolve. Have tackled the trend of the advanced data virtualization in our first post, this blog will focus on the trend of self-organizing systems.

Current products focus on virtualizing storage pools and access from storage hardware. This too is a good step towards avoiding siloed storage systems, however, it appears there is still too much management burden placed on the storage administrator. The management systems do have self-discovery, but that simply means listing the hardware nodes that have been added to the system. The burden is still on the storage administrator to determine where and how to deploy those nodes.

To simplify the management burden, another step must occur – the system must evolve to self organize. Self-organizing systems are made up of small units that can determine an inherent order collectively. Instead of a storage administrator having to determine which pools and tiers to add storage nodes to, the nodes themselves will evolve to contain metadata and rules, and inherently place themselves within the storage tiers as the system requires.

An example of metadata and rules could be related to disk characteristics –SSD could know it is suited for tier 1 performance scenarios. Another example would be related to GPS location – storage nodes could know which data center they have been installed within, and determine which storage pools they need to join.

Beyond the storage system self-organizing in terms of provisioning hardware, systems will also self organize the tiers. Storage administrators will define the requirements for tiers (QoS, data reliability, performance), and the storage nodes will self organize underneath them. That means that when capacity and performance nodes are added to the system, the system will also determine which tiers need those resources.

Emergent patterns will surface once the storage nodes have metadata and rules, and the toolset for managing the system will change. Rather than managing the physical hardware, and individual storage pools and access, management will occur at a system level – what storage is required in which locations based on how information is dynamically moving across storage nodes and tiers.

We’ll discuss location fault tolerance in more detail in the next post.