[question] how to configure PlayerView's surface type in such a way that it can play both av1 and non-av1 videos

[warren-bank]

versions used in tests:

• ExoPlayer r2.14.2
• av1 extension compiled on 2021-05-22

observed behavior:

• using SimpleExoPlayer

  • with:
    • <com.google.android.exoplayer2.ui.PlayerView app:surface_type="surface_view" />
    • ConcatenatingMediaSource
    • av1 extension, using: DefaultRenderersFactory.EXTENSION_RENDERER_MODE_ON
  • when:
    • ConcatenatingMediaSource playlist contains:
      1. any video format that doesn't require the av1 extension
      2. any video format that does require the av1 extension
      3. any video format that doesn't require the av1 extension
  • behavior:
    • 1st video plays in SurfaceView
    • 2nd video plays in ANativeWindow
    • 3rd video does not play
      • timeline shows length of video
      • playback cannot begin

• using SimpleExoPlayer

  • with:
    • <com.google.android.exoplayer2.ui.PlayerView app:surface_type="video_decoder_gl_surface_view" />
    • ConcatenatingMediaSource
    • av1 extension, using: DefaultRenderersFactory.EXTENSION_RENDERER_MODE_ON
  • when:
    • ConcatenatingMediaSource playlist contains:
      1. any video format that doesn't require the av1 extension
      2. any video format that does require the av1 extension
      3. any video format that doesn't require the av1 extension
  • behavior:
    • 1st video does not render any visible video in VideoDecoderGLSurfaceView
      • playback occurs with audio and subtitles
      • screen is black
    • 2nd video plays in VideoDecoderGLSurfaceView
    • 3rd video does not render any visible video in VideoDecoderGLSurfaceView
      • playback occurs with audio and subtitles
      • screen is black

• VideoDecoderGLSurfaceView seems to:

  • work better than ANativeWindow with the av1 extension
  • not render any video when no extension is required

question:

• what is the suggested approach for being able to successfully play the aforementioned ConcatenatingMediaSource playlist?
  • preferably, using:
    • SurfaceView for all video formats that don't require the av1 extension
    • VideoDecoderGLSurfaceView for video formats that do require the av1 extension

[andrewlewis]

This is tricky because I think VideoDecoderGLSurfaceView will only receive data from extension video renderers (the vp9 and av1 extensions) as they take a VideoDecoderOutputBufferRenderer, unlike MediaCodecVideoRenderer which just wants a Surface. This is probably why in the second scenario under 'observed behavior' you don't get any output for the first and third videos in the playlist.

For the first scenario, I expect this is running into the limitation quoted on #9459 -- after the extension renderer has rendered via ANativeWindow_lock, MediaCodec can no longer use the output surface.

We heard that the VideoDecoderGLSurfaceView approach gives better performance for the av1 extension so it seems best to try to keep using that for the av1 extension, and find a way to make this work in conjunction with normal SurfaceView rendering with the other playlist items. As an experiment I'd try creating two separate output views, a VideoDecoderGLSurfaceView and a normal SurfaceView then passing the relevant one to the MediaCodecVideoRenderer and the Libgav1VideoRenderer separately (so they each have a different output), then updating the view hierarchy z-order when transitioning between videos. SimpleExoPlayer has code to listen to the surface lifecycle (for surface views) and pass the output Surface to the video renderers, and that needs not to run because it will "overwrite" any surface you've set yourself.

I'll mark this as an enhancement (though I guess it could be considered a bug), to find a way to handle this in the PlayerView but as it's quite a niche use case so I doubt we will get round to it soon.

[warren-bank]

versions used in tests:

• ExoPlayer r2.15.1
• av1 extension compiled on 2021-10-03
• ffmpeg extension compiled on 2021-08-12

another interesting observation:

• using SimpleExoPlayer
  • with:
    • <com.google.android.exoplayer2.ui.PlayerView app:surface_type="surface_view" />
    • ConcatenatingMediaSource
    • av1 extension, using: DefaultRenderersFactory.EXTENSION_RENDERER_MODE_ON
  • when:
    • ConcatenatingMediaSource playlist contains:
      1. any video format that doesn't require the av1 extension
      2. any video format that does require the av1 extension
      3. flac audio file that does require the ffmpeg extension
      4. any video format that doesn't require the av1 extension
  • behavior:
    • 1st video plays in SurfaceView
    • 2nd video plays in ANativeWindow
    • 3rd flac audio displays a still frame containing cover art
    • 4th video plays in SurfaceView
  • notes:
    • the ffmpeg extension changes the Surface so the next non-av1 video will play