Design of ai-based viewport-adaptive immersive video streaming systems

Immersive Virtual Reality (VR) applications have been widely developed in recent years. However, streaming of 360◦ video content is a key issue in multimedia networking. Compared to traditional videos, 360◦ video content captures the whole spherical view, while the user only watches a small portion of the video called Field of View (FoV) at any time. Conventional systems stream the entire projected panoramic video frame at the same quality, which results in bandwidth loss as a user’s FoV contains only a limited part of the full video. This thesis presents the experimental design and analysis of AI-based viewport-adaptive streaming systems, which aim at overcoming this inefficient bandwidth utilization by integrating a Machine Learning (ML)-based viewport prediction model with multi-bitrate tile-based video delivery using the Quick UDP Internet Connections (QUIC) transport protocol. This work is classified into two parts where the first part focuses on comparing viewport prediction results obtained by various ML models. Furthermore, a comparison between Long Short-Term Memory (LSTM) model results and state-of-the-art approaches is also carried out. Systematic comparison includes six ML models–LSTM, Recurrent Neural Network (RNN), Gated Recurrent Unit (GRU), Convolutional Neural Network (CNN), Linear Regression (LR) and Multi-layer Perceptron (MLP)–to predict user FoV based on Head-Mounted Display (HMD) motion traces with horizons starting from next-frame up to five seconds ahead considering a frame rate of 30fps. Results indicate that the CNN model provides the best performance in terms of short-term horizon (next-frame viewport prediction), whereas GRU and LSTM models are more performative in the long-term horizon (5-seconds) as these models have the ability to capture temporal dynamics in the head motion. Comparing viewport prediction results of LSTM model with a state-of-the-art approach called Multi-modal Fusion Transformer (MFTR) shows that the LSTM model outperforms the other in terms of the Average Prediction (AP) accuracy across all considered horizons. This shows that recurrent models with appropriate parameter configuration perform better with less computational effort compared to much more complex architectures. Second part of thesis presents an end-to-end streaming architecture that combines viewport prediction using an LSTM model with adaptive tile-based bitrate allocation over QUIC. The corresponding 2D projection of the 360° video frame is divided into a 3 × 3 tile grid, where the predicted FoV tiles are assigned a higher bitrate of 1.0 Mbps, with neighboring and non-visible tiles are allocated lower bitrates. Multiplexed streams in QUIC perform parallel transmission of tiles with two priorities (high and low) without Head-of-Line (HoL) blocking, which reduces the latency of FoV content delivery. Experimental setup contains real user HMD traces which are used to predict the users’ FoVs. The prediction results are integrated into an adaptive bitrate decision mechanism that accounts for network bandwidth constraints. The system maintains a stable playback buffer by ensuring that the total bitrate assigned to all tiles remains within the available bandwidth. Moreover, the FoV missing ratio is compared with overall missing-tile ratio across the streaming session, showing that the average FoV missing ratio remains lower than the missing ratio computed over all tiles. Although the results prove the effectiveness of streaming system, there are still some limitations that should be considered. Viewport prediction accuracy is high for next-frame prediction; however, it gradually decreases as prediction horizon increases. Future work will consider the following steps: integration of attention-based and hybrid neural network structures, joint bitrate and tile-prioritization decisions based on Deep Reinforcement Learning (DRL), evaluation using different network traces such as mobile 5G and edge computing platform, and real-world deployment.