custom implemented FfmpegVideoRenderer crashes with YUV12 on some devices

[sam-m3]

The Issue:

I have an HLS stream which is played by android tv devices, stb devices and also mobile devices;
One of the formats of the video streams is video/mpeg2 which is not supported by new android phones.
As a workaround I implemented an FfmpegVideoRenderer and FfmpegVideoDecoder based on #7132 pull request.

I have two issues with it;

First:

The software decoder worked just fine on my phone which is Galaxy A31 / SM-A31F/DS, but tried it out on anther phone, the app crashed immediately or after a second of playing, Other device is Galaxy A71 / SM-A7115F/DS.
Down below I will provide the log I got for this error and further more explanation.
note: It cannot be a hardware issue, because I got the same error on 2 or 3 other devices including an Android TV.

Second

On my device which I said the video was working fine, The FfmpegVideoRenderer was doing a fantastic job rendering my video; But I only want video/mpe2 streams to be decoded by software decoder, For Other streams I want it to be rendered by the regular MediaCodedVideoRenderer; I did that by only supporting video/mpeg type for my software decoder, It works fine when switching from a stream rendered by MediaCodecVideoRenderer to FfmpedVideoRenderer, but the other way around, I get error saying that Decoder init failed OMX........
I think it has something to do with the FfmpegVideoRenderer not being released or what so ever, but couldn't find anything to help me with the issue.

Issue logs and files:

From the file below it contains:

• ffmpeg_jni.cc
• both FfmpegVideoDecoder and FfmpegVideoRenderer
• build_ffmpeg.sh
• error logs for both issues in separate text files

ffmpegSofwareDecoder.zip

Extra info:

In the title I said it has issue with YUV, I found that out by playing around in the ffmpeg_jni.cc file; I commented out the U plane from YUV line 727 of the ffmpeg_jni.cc file, and the video started playing without any crashes, but as I did that, the video has lost it's original color and it has a yellowish color.
I don't know that much about c++, that's why it took me a day or two just to figure out where the issue was.

I'll be thankful if you could help me with it and make it easier on me.
Regards.

[krocard]

@ojw28 what is the status of the FfmpegVideoRenderer ? Are there known issues or progress compare to the pull request?

[sam-m3]

@ojw28 @krocard
I managed to solve the first issue.
changing the while(height--) from the ffmpeg-jni.cc, line 448 to while(--height) solved the issue, even though the last pixel line from the bottom is distorted a little bit, but supporting video/mpeg2 format for mobile devices makes up for it xD.

The second issue still remains, If you could help me with that one, I'll be really thankful.

[ojw28]

I did that by only supporting video/mpeg type for my software decoder, It works fine when switching from a stream rendered by MediaCodecVideoRenderer to FfmpedVideoRenderer, but the other way around, I get error saying that Decoder init failed OMX........

Unfortunately, I think this is an Android platform limitation that's described on this page:

Caution: You can't draw on a surface with GLES or send it frames from a video decoder if you've ever called lockCanvas(). lockCanvas() connects the CPU renderer to the producer side of the BufferQueue and doesn't disconnect until the surface is destroyed. The canvas-based CPU renderer can't be disconnected and reconnected to a surface, unlike most producers (like GLES or Vulkan).

FfmpegVideoDecoder (and similar video decoder extensions) connect the CPU renderer, and once that's happened it's not then possible to connect a platform video decoder to the Surface. You'd need to somehow destroy the Surface and create a new instance. We previously filed [Internal ref: b/157473162] to request removing this limitation, but the issue hasn't received attention as of now.

That aside, there are known issues with FfmpegVideoRenderer, and we haven't managed to dedicate time toward fixing them. Doing so is tracked by #2159. I think we can just mark this issue as a duplicate of that one.