How to reduce iOS AVPlayer start delay

Note, for the below question: All assets are local on the device -- no network streaming is taking place. The videos contain audio tracks.

I'm working on an iOS application that requires playing video files with minimum delay to start the video clip in question. Unfortunately we do not know what specific video clip is next until we actually need to start it up. Specifically: When one video clip is playing, we will know what the next set of (roughly) 10 video clips are, but we don't know which one exactly, until it comes time to 'immediately' play the next clip.

What I've done to look at actual start delays is to call addBoundaryTimeObserverForTimes on the video player, with a time period of one millisecond to see when the video actually started to play, and I take the difference of that time stamp with the first place in the code that indicates which asset to start playing.

From what I've seen thus-far, I have found that using the combination of AVAsset loading, and then creating an AVPlayerItem from that once it's ready, and then waiting for AVPlayerStatusReadyToPlay before I call play, tends to take between 1 and 3 seconds to start the clip.

I've since switched to what I think is roughly equivalent: calling [AVPlayerItem playerItemWithURL:] and waiting for AVPlayerItemStatusReadyToPlay to play. Roughly same performance.

One thing I'm observing is that the first AVPlayer item load is slower than the rest. Seems one idea is to pre-flight the AVPlayer with a short / empty asset before trying to play the first video might be of good general practice. [Slow start for AVAudioPlayer the first time a sound is played

I'd love to get the video start times down as much as possible, and have some ideas of things to experiment with, but would like some guidance from anyone that might be able to help.

Update: idea 7, below, as-implemented yields switching times of around 500 ms. This is an improvement, but it it'd be nice to get this even faster.

Idea 1: Use N AVPlayers (won't work)

Using ~ 10 AVPPlayer objects and start-and-pause all ~ 10 clips, and once we know which one we really need, switch to, and un-pause the correct AVPlayer, and start all over again for the next cycle.

I don't think this works, because I've read there is roughly a limit of 4 active AVPlayer's in iOS. There was someone asking about this on StackOverflow here, and found out about the 4 AVPlayer limit: fast-switching-between-videos-using-avfoundation

Idea 2: Use AVQueuePlayer (won't work)

I don't believe that shoving 10 AVPlayerItems into an AVQueuePlayer would pre-load them all for seamless start. AVQueuePlayer is a queue, and I think it really only makes the next video in the queue ready for immediate playback. I don't know which one out of ~10 videos we do want to play back, until it's time to start that one. ios-avplayer-video-preloading

Idea 3: Load, Play, and retain AVPlayerItems in background (not 100% sure yet -- but not looking good)

I'm looking at if there is any benefit to load and play the first second of each video clip in the background (suppress video and audio output), and keep a reference to each AVPlayerItem, and when we know which item needs to be played for real, swap that one in, and swap the background AVPlayer with the active one. Rinse and Repeat.

The theory would be that recently played AVPlayer/AVPlayerItem's may still hold some prepared resources which would make subsequent playback faster. So far, I have not seen benefits from this, but I might not have the AVPlayerLayer setup correctly for the background. I doubt this will really improve things from what I've seen.

Idea 4: Use a different file format -- maybe one that is faster to load?

I'm currently using .m4v's (video-MPEG4) H.264 format. H.264 has a lot of different codec options, so it's possible that some options are faster to seek than others. I have found that using more advanced settings that make the file size smaller increase the seek time, but have not found any options that go the other way.

Idea 5: Combination of lossless video format + AVQueuePlayer

If there is a video format that is fast to load, but maybe where the file size is insane, one idea might be to pre-prepare the first 10 seconds of each video clip with a version that is bloated but faster to load, but back that up with an asset that is encoded in H.264. Use an AVQueuePlayer, and add the first 10 seconds in the uncompressed file format, and follow that up with one that is in H.264 which gets up to 10 seconds of prepare/preload time. So I'd get 'the best' of both worlds: fast start times, but also benefits from a more compact format.

Idea 6: Use a non-standard AVPlayer / write my own / use someone else's

Given my needs, maybe I can't use AVPlayer, but have to resort to AVAssetReader, and decode the first few seconds (possibly write raw file to disk), and when it comes to playback, make use of the raw format to play it back fast. Seems like a huge project to me, and if I go about it in a naive way, it's unclear / unlikely to even work better. Each decoded and uncompressed video frame is 2.25 MB. Naively speaking -- if we go with ~ 30 fps for the video, I'd end up with ~60 MB/s read-from-disk requirement, which is probably impossible / pushing it. Obviously we'd have to do some level of image compression (perhaps native openGL/es compression formats via PVRTC)... but that's kind crazy. Maybe there is a library out there that I can use?

Idea 7: Combine everything into a single movie asset, and seekToTime

One idea that might be easier than some of the above, is to combine everything into a single movie, and use seekToTime. The thing is that we'd be jumping all around the place. Essentially random access into the movie. I think this may actually work out okay: avplayer-movie-playing-lag-in-ios5

Which approach do you think would be best? So far, I've not made that much progress in terms of reducing the lag.

28754

小开

Without having done anything like this in the past, based on your thoughts and experiences I would try a combination of 7 and 1: Preload one AVPlayer with the first couple of seconds of the 10 follow up videos. Then skipping will very likely be faster and more reliable due to less data. While you are playing the selected piece, you have enough time to prepare the AVPlayer for the rest of the selected follow up video in the background. When the beginning is finished, you switch to the prepared AVPlayer. So in total, you at any given time have a maximum of 2 AVPlayers loaded.

Of course I don't know whether the switching can be done so smoothly that it does not disturb playback.

(Would have added this as a comment if I could.)

Best, Peter

小开

The asset may not ready once you create it, it may does calculations like duration of movie, be sure to contain all metadata of the movie in the file.

小开

You should try option #7 first, just to see if you can get that working. I suspect that it will not actually work for your needs since the seek time will likely not be fast enough to give you seamless switching between clips. If you try that and it fails, then I would advise that you do option 4/6 and take a look at my iOS library designed specifically for this purpose, just do a quick google search on AVAnimator to find out more. My library makes it possible to implement seamless loops and switching from one clip to another, it is very fast because video has to be decoded into a file before hand. In your case, all 10 video clips would get decoded into files before you begin, but then switching between them would be fast.

小开

Here are several properties and methods provided by the AVAsset class that may help:

- (void)_pu_setCachedDuration:(id)arg1;
- (id)pu_cachedDuration;
- (struct
{
long long x1;
int x2;
unsigned int x3;
long long x4;
})pu_duration;
- (void)pu_loadDurationWithCompletionHandler:(id /* block */)arg1;

小开

If I understood your issue correctly, it seems that you have one continuous video to which you need to load the audio track for on a moment's notice.

If that is the case I suggest looking into BASS. BASS is an audio library much like AVPlayer that gives you (relatively) easy access to the low-level API's of the AudioUnits framework in iOS. What does the mean for you? It means that with a tiny bit of buffer manipulation (you may not even need it, depends on how tiny you want the delay) you can start playing music instantly.

The limitations however extend to video, as I said, it is an audio library so any video manipulation will still have to be done with AVPlayer. However using-seekToTime:toleranfeBefore:toleranceAfter: you should be able to achieve fast seeking within the video as long as you preroll with all the necessary options.

If you're syncing across multiple devices (which your application might suggest) just leave a comment and I'd be happy to edit my answer.

PS: BASS may look daunting at first because of it's C-like format, but it's really really easy to use for what it is.

小开

For iOS 10.x and greater to reduce AVPlayer start delay I set: avplayer.automaticallyWaitsToMinimizeStalling = false; and that seemed to fix it for me. This could have other consequences, but I haven't hit those yet.

I got the idea for it from: https://stackoverflow.com/a/50598525/9620547