What M&E Can Teach the AV Industry About Data

June 15, 2019

Jason Coari and Mark Pastor

At first glance, autonomous vehicles would seem to have little in common with the Media and Entertainment (M&E) industry, beyond action-movie car chases and in-vehicle entertainment screens.

But AVs and M&E do have a lot in common when it comes to their data. Both generate hundreds of petabytes of data per year. Both need to manage that data for retrieval and value, and both need to store that data with security and data protection.

Despite these differences, these two industries share two major challenges around massive data retention and accessibility: storage scalability and data lifecycle management. Storage scalability is about how to afford data storage and storage management when the data is growing so fast. Economies of scale are critical to affording massive capacity storage systems. Data lifecycle management is how to keep massive amounts of data searchable, accessible, and protected. Both industries store large amounts of highly valuable content that they need to protect and keep accessible for authorized users.

Breaking out the challenges

M&E is a mature industry that makes heavy use of disk/flash and tape systems, and integration with sophisticated editing software and workflow tools. Yet even they are under pressure to do more and do it faster. For example, higher-resolution cameras are producing larger files for ingest. This is happening in real-time: 4K resolution is common, and native 8K is here.

Multiple editing teams work on film, sound, special effects, animation and color; they output the final work by collaborating with each other and with directors and producers. This level of collaboration and intensive processing requires storage with high performance and massive capacity.

Once the works are broadcast, M&E continues to monetize them. The rerun is the granddaddy of monetization, but M&E also incorporates sections of existing works to create new ones.

Cyberthreats to M&E are also a major concern: witness Sony losing millions of dollars to a multi-pronged attack in 2014. M&E risks losing monetary and reputational value when something happens to its data. Advanced storage systems must support equally advanced security and data protection.

Automated vehicles are a much newer business than M&E, but the industry also depends on generating and working with massive amounts of data. The automotive industry already generates significant data from its AV test vehicles, typically producing 5 terabytes (TB) to 30 TB per vehicle, per testing day. Researchers build and improve the data intake devices, analyze massive incoming data, and adjust their algorithms for better analysis and testing.

Long-term data retention and accessibility are crucial to the industry. AV retains data for new research projects, and to validate current ones. As a highly regulated industry, most companies plan to keep 20 years of test data for liability protection.

The industry is not yet a popular hacker target like M&E. But data is not only at risk for malware; IT errors, lost mobile devices, failed hardware and software data and even the old-fashioned natural disaster can lose or corrupt data. An industry that depends on original data for its success cannot afford that level of loss.

Best Practices for data storage

Plan your infrastructure for both today and tomorrow’s projected data amounts. Buying for current data is a cheap option in the short-term, but data growth is accelerating and will likely outpace the system’s capacity and scalability in 1-2 years. Replacing the system will cost even more money.

Plan for a highly scalable system that will grow along with big data and user needs:

• Don’t depend on disk/flash storage alone. If you depend on storing data on your production systems, affordability will go out the window. Combine high performance disk/SSD production systems with tape libraries for dynamic scalability and highly accessible archives.
• Invest in a storage solution with integrated data protection. AV and M&E need to protect their valuable data against intrusion and data loss. Use updated data protection and remember that tape’s “air gap defense” is impassible for would-be hackers (malware cannot traverse the separation between a system and an off-line cartridge). For multi-petabyte environments that include active archive data, traditional backup apps are not very efficient or effective. A self-protecting active archive infrastructure provides the data durability and protection needed while also offering active access to all user data.
• Don’t only depend on cloud-based cold storage. Even if you can achieve data transport speeds back and forth from the cloud, large-scale storage and retrieval from cold tiers is slow and expensive. Use post-production cloud applications or cloud-bursting to support intensive processing but keep massive data storage and active archives on-site.

Build the foundation

Data-intensive industries like AV and M&E need dynamic scalability without sacrificing availability or performance. These industries also need storage with user-friendly, sophisticated management tools, and smooth integration with workflow and analysis software.

Build your infrastructure from the start to accommodate all the scalability, flexibility, protection, and accessibility you are going to need for the foreseeable future. Do it smart from the beginning and let your system grow with you.

About the authors:

Jason Coari is Director, Scale-out Storage Solutions at Quantum. He leads the company’s product and commercial strategy for scale-out storage across all industries.

Mark Pastor is director for archive solutions at Quantum. He represents Quantum within the Active Archive Alliance, the LTO Consortium and the Object Storage Alliance. He regularly blogs on topics relating to data protection and archival storage.

---