HLS playback issues using integer precision segment durations

[laurafuente]

Description:
For some HLS streams the player goes to STATE_BUFFERING forever when seeking to a different position than 0. If we seek again to 0 position the stream starts it reproduction
This is happening in streams that needs the ID3 header to be synchronized.

Steps to reproduce at demo app:

1. Add a new entry above HLS to the stream passed to the e-mail [email protected]
2. Start the reproduction of the stream
3. seek to a different position than 0

This issue is taking place at exoplayer 2.1.0, I can't reproduce it at 2.0.4.
The device I'm using is a Samsung S6 with 6.0.1 as android version

[laurafuente]

I've attached a video showing the problem using the stream passed at the e-mail [email protected]
HLS_seek_problem.mp4.zip

[ojw28]

Note: Interestingly, seeking does appear to work up to about 20:15.
Correction: It looks like playback becomes more prone to getting stuck the closer to 20:15 you get (and is completely broken from that point onward).

[ojw28]

The stream in question does not work correctly on 2.0.4. On 2.0.4 seeking does not cause the player to get stuck, but audio/video sync is completely broken if you seek to the middle of the content. It's a couple of minutes out at least. On 2.1.0 audio/video sync is corrected because we started looking at the ID3 header, as a player is supposed to do, but the knock-on effect is that the player can get stuck buffering.

I think the root cause is that the segment durations listed in the audio media playlists are rounded to integer precision. The sum of the durations then drifts over time. Since it's a long piece of media, by the middle the drift very significant. This may not be a technical violation of the HLS spec, but does mean it's impossible for any player to efficiently seek in the media. The best guess for which segment to seek to for this media will result in a player requesting a segment that's multiple minutes off from the requested seek position. A player can then "guess" again based on this information, to eventually find the correct segment, however this is really quite inefficient. We do not do this in ExoPlayer currently, which I think is why the playback then gets stuck.

The best solution here would be to correct the media playlists to use floating-point precision segment durations. This will resolve the issue when using ExoPlayer, and also make seeking more efficient when using any other player. Recent versions of the HLS spec say this about segment durations:

Generally, durations SHOULD be decimal-floating-point, with enough accuracy to avoid perceptible error when segment durations are accumulated.

We should also think about handling this kind of media better in ExoPlayer, but it's likely to be treated as a low priority issue because the media is quite sub-optimal to start with, and we tend to focus more energy on efficiently supporting well prepared content, rather than on handling all legacy cases where the experience will be sub-optimal no matter how well we handle it.

[rchauderlot]

According to the latest definition of the HLS standard ( https://tools.ietf.org/html/draft-pantos-http-live-streaming-20#section-4.3.2.1 ):

The EXTINF tag specifies the duration of a Media Segment. It applies
only to the next Media Segment. This tag is REQUIRED for each Media
Segment. Its format is:

#EXTINF:,[<title>]

where duration is a decimal-floating-point or decimal-integer number
(as described in Section 4.2) that specifies the duration of the
Media Segment in seconds. Generally, durations SHOULD be decimal-
floating-point, with enough accuracy to avoid perceptible error when
segment durations are accumulated. If the compatibility version
number is less than 3, durations MUST be integers. Durations that
are reported as integers SHOULD be rounded to the nearest integer.
The remainder of the line following the comma is an optional human-
readable informative title of the Media Segment expressed as raw
UTF-8 text.

In the version 5 of the HLS standard (https://tools.ietf.org/html/draft-pantos-http-live-streaming-11#section-3.3.2) the segment duration could use integer precision:

The EXTINF tag specifies the duration of a media segment. It applies
only to the media segment that follows it. Each media segment MUST
be preceded by an EXTINF tag. Its format is:

#EXTINF:,<title>

"duration" is an integer or floating-point number in decimal
positional notation that specifies the duration of the media segment
in seconds. Durations that are reported as integers SHOULD be
rounded to the nearest integer. Durations MUST be integers if the
protocol version of the Playlist file is less than 3. The remainder
of the line following the comma is an optional human-readable
informative title of the media segment.

It seems that the integer precision segment durations are mandatory in versions lower than 3, but, valid in versions v3, v4, and v5, the more common ones.

[ojw28]

I'm unsure what your point is, since I didn't say they were invalid. I explicitly stated they weren't a violation of the spec. But the fact they're valid doesn't make them in any way a sensible idea, for the reasons specified above (i.e. it's impossible for any player to perform an efficient seek if the accumulated error grows significant).

Note also that versions prior to V4 are not relevant to this issue, since EXT-X-MEDIA wasn't added prior to V4.

[rchauderlot]

Im sorry, my intention was not to annoy you.

My point was that this problem is a very common problem.

Well formed assets using the HLS 3, 4 and 5 versions with integer precision EXTINF tags will accumulate errors if they meet this specification requirement:

Durations that are reported as integers SHOULD be rounded to the nearest integer.

And, I fully agree with you. This HLS definition will lead in a suboptimal solution in every possible player implementation :(

Inmho, the pity is that those standard versions was very popular some time ago, and users not consuming their own generated content, which could not control the hls versions, will find those problems easily.

[ojw28]

Whilst it might be common for older streams to use integer precision EXTINF durations, I doubt it's common at all for streams to use integer precision EXTINF durations and EXT-X-MEDIA tags, which is when this becomes a bigger issue for ExoPlayer.

[ojw28]

As an additional piece of information, it seems like streams affected by this issue violate Apple's HLS authoring specifications for Apple devices:

8.1. You MUST use sufficiently accurate segment durations to ensure that the sum of the EXTINF durations of any contiguous group of segments is within one video frame duration of the actual duration of the content.

[christosts]

This came up in #9411. As of 2.13.1, the rounded audio segment durations manifested with audio being out of sync with video. I updated the title of the issue since as of 2.13.1, the issue may manifest in different ways.

[gaberogan]

Possible fix/workaround that worked for me:

I created a CustomLoadControl with the same code as DefaultLoadControl but updated shouldContinueLoading to keep loading content when the buffer size drops below half full. (see BUGFIX lines) We had ID3 metadata in our AAC files overestimating their duration which we think caused the player to overestimate the amount of content it had buffered, causing shouldContinueLoading to return false while the buffer was empty and produce an infinite STATE_BUFFERING state.

  @Override
  public boolean shouldContinueLoading(
      long playbackPositionUs, long bufferedDurationUs, float playbackSpeed) {
    boolean targetBufferSizeReached = allocator.getTotalBytesAllocated() >= targetBufferBytes;
    long minBufferUs = this.minBufferUs;
    if (playbackSpeed > 1) {
      // The playback speed is faster than real time, so scale up the minimum required media
      // duration to keep enough media buffered for a playout duration of minBufferUs.
      long mediaDurationMinBufferUs =
          Util.getMediaDurationForPlayoutDuration(minBufferUs, playbackSpeed);
      minBufferUs = min(mediaDurationMinBufferUs, maxBufferUs);
    }
    // Prevent playback from getting stuck if minBufferUs is too small.
    minBufferUs = max(minBufferUs, 500_000);

    // BUGFIX: If half of the target buffer is not filled, we need to start loading
    if (allocator.getTotalBytesAllocated() <= targetBufferBytes / 2) {
      isLoading = true;
    }

    if (bufferedDurationUs < minBufferUs) {
      isLoading = prioritizeTimeOverSizeThresholds || !targetBufferSizeReached;
      if (!isLoading && bufferedDurationUs < 500_000) {
        Log.w(
            "CustomLoadControl",
            "Target buffer size reached with less than 500ms of buffered media data.");
      }
    }

    // BUGFIX: Ensure both time buffer and size buffer are reached before we stop loading
    if (bufferedDurationUs >= maxBufferUs && targetBufferSizeReached) {
      isLoading = false;
    }

    return isLoading;
  }