The AI Data Cycle: Understanding the optimal storage mix for AI workloads at scale

In his recent Thought Leadership piece, “The AI Data Cycle: Understanding the Optimal Storage Mix for AI Workloads at Scale,” Ghassan Azzi, the Sales Director, Africa at Western Digital, provides a comprehensive analysis of the critical relationship between AI development and data storage. As AI continues to transform industries and inspire new applications, it fundamentally relies on the utilization and generation of data.

Azzi discusses how the AI industry is building a massive infrastructure to train AI models and offer AI services (inference). This infrastructure has significant implications for data storage, emphasizing that storage technology plays a crucial role in the cost and power-efficiency of various stages. As AI systems process and analyze data, they create new data, which is often stored due to its usefulness or entertainment value. This leads to a virtuous cycle where increased data generation fuels expanded data storage, further fueling data generation—what Azzi terms the “AI Data Cycle.”

Enterprise data center planners need to understand the dynamic interplay between AI and data storage. Azzi outlines the storage priorities for AI workloads at scale across six stages, highlighting how storage component manufacturers, like Western Digital, are adjusting their product roadmaps to maximize performance and minimize total cost of ownership (TCO).

In the initial stage, raw data is collected and stored securely and efficiently from various sources. The quality and diversity of collected data are critical, setting the foundation for everything that follows. Capacity enterprise hard disk drives (eHDDs) are preferred for bulk data storage, offering the highest capacity per drive and lowest cost per bit.

During data preparation and ingestion, data is processed, cleaned, and transformed for model training. To support this, data center owners are implementing upgraded storage infrastructure such as fast data lakes. All-flash storage systems incorporating high-capacity enterprise solid state drives (eSSDs) are being deployed to augment HDD-based repositories or within new all-flash storage tiers.
AI model training involves iterative training of AI models to make accurate predictions based on the training data. This stage relies heavily on high-performance supercomputers, with training efficiency depending on maximizing GPU utilization. High-bandwidth flash storage near training servers, such as high-performance (PCIe® Gen. 5) and low-latency compute-optimized eSSDs, is crucial for this process.

Inference and prompting entail creating user-friendly interfaces for AI models, including APIs, dashboards, and tools that combine context-specific data with end-user prompts. AI models are integrated into existing internet and client applications, enhancing them without replacing current systems. This integration drives the need for upgraded storage systems with additional eHDD and eSSD capacity. Larger, higher performance client SSDs (cSSDs) for PCs and laptops, and higher capacity embedded flash devices for mobile phones, IoT systems, and automotive applications, are also necessary.

The AI inference engine stage is where real-time deployment of trained models occurs, allowing them to analyze new data and provide predictions or generate content. Efficiency here is crucial for timely and accurate AI responses, requiring high-capacity eSSDs for streaming context or model data to inference servers. High-performance compute eSSDs may be deployed for caching, and high-capacity cSSDs and embedded flash modules are needed in AI-enabled edge devices.

In the final stage, new content is created from AI insights, generating new data that is stored for future use. This perpetuates the AI Data Cycle, driving continuous improvement and innovation. Generated content typically lands back in capacity enterprise eHDDs for archival storage and in high-capacity cSSDs and embedded flash devices in AI-enabled edge devices.
Azzi emphasizes that this continuous loop of data generation and consumption is accelerating the demand for performance-driven and scalable storage technologies. Ed Burns, research director at IDC, notes that the implications for storage are significant as access to data influences the speed, efficiency, and accuracy of AI models, particularly as larger and higher-quality data sets become more prevalent.

As AI technologies become embedded across various industries, storage component providers are increasingly tailoring their products to meet the needs of each stage in the AI Data Cycle. The ongoing innovation in storage technology is crucial for managing large AI data sets and ensuring the efficient re-factoring of complex data, driving further advancements in AI.

This comprehensive analysis by Ghassan Azzi provides valuable insights for understanding the optimal storage mix for AI workloads at scale, highlighting the importance of storage technology in the AI revolution.