Exoplayer: How to extract timed metadata from HLS stream?

Created on 6 Oct 2019  路  7Comments  路  Source: google/ExoPlayer

[REQUIRED] Searched documentation and issues
I've searched through the ExoPlayer issues , and I've done some R&D & experimentation using different streams including Apple's test stream (http://devimages.apple.com/samplecode/adDemo/ad.m3u8).

[END RESULT OF ALL EXPERIMENT]:

  • Apple's test stream (http://devimages.apple.com/samplecode/adDemo/ad.m3u8):- This stream not even playing in Exoplayer,

  • https://vcloud.blueframetech.com/file/hls/13836.m3u8 :- This stream playing & @Override public void onMetadata(Metadata metadata) { Log.d("METADATA", metadata.toString()); } call back event triggered perfectly but other two call back not triggering.
    But, this stream is not as per the specification, data_alignment_indicator

[REQUIRED] Question
I have an HLS video (sometimes live, sometimes not) and I wish to extract timed ID3 metadata frames from it.

The content can be viewed here:
https://adinsertplayer.s3.ap-south-1.amazonaws.com/sampleoutput0302_modified_1_Live.m3u8

Alternatively, just the ID3 metadata packets can be seen in HLS file in following format:

#EXTM3U
#EXT-X-VERSION:1
#EXT-X-TARGETDURATION:17
#EXTINF:16.729778,
pre_process_output1.ts
#EXT-X-CUE-OUT:16.729778
#EXTINF:16.729778,
pre_process_output2.ts
#EXTINF:16.729778,
pre_process_output3.ts
#EXT-X-CUE-IN
#EXTINF:16.683333,
pre_process_output4.ts
#EXTINF:16.683333,
pre_process_output5.ts
#EXT-X-CUE-OUT:16.729778
#EXTINF:16.729778,
pre_process_output6.ts
#EXT-X-CUE-IN
#EXTINF:16.683333,
pre_process_output7.ts
#EXTINF:16.683333,
pre_process_output8.ts
#EXT-X-CUE-OUT:16.729778
#EXTINF:16.729778,
pre_process_output9.ts

Requirement is to trigger an ad event & play an in stream ima adwhen we hit this metadata(
#EXT-X-CUE-IN & #EXT-X-CUE-OUT). Here is what I've tried using demo-ima :

/*
 * Copyright (C) 2017 The Android Open Source Project
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package com.google.android.exoplayer2.imademo;

import android.content.Context;
import android.net.Uri;
import android.util.Log;
import com.google.android.exoplayer2.C;
import com.google.android.exoplayer2.C.ContentType;
import com.google.android.exoplayer2.ExoPlayer;
import com.google.android.exoplayer2.ExoPlayerFactory;
import com.google.android.exoplayer2.Player;
import com.google.android.exoplayer2.SimpleExoPlayer;
import com.google.android.exoplayer2.ext.ima.ImaAdsLoader;
import com.google.android.exoplayer2.metadata.Metadata;
import com.google.android.exoplayer2.metadata.MetadataOutput;
import com.google.android.exoplayer2.source.MediaSource;
import com.google.android.exoplayer2.source.MediaSourceFactory;
import com.google.android.exoplayer2.source.ProgressiveMediaSource;
import com.google.android.exoplayer2.source.TrackGroup;
import com.google.android.exoplayer2.source.TrackGroupArray;
import com.google.android.exoplayer2.source.ads.AdsMediaSource;
import com.google.android.exoplayer2.source.dash.DashMediaSource;
import com.google.android.exoplayer2.source.hls.HlsMediaSource;
import com.google.android.exoplayer2.source.smoothstreaming.SsMediaSource;
import com.google.android.exoplayer2.text.Cue;
import com.google.android.exoplayer2.text.TextOutput;
import com.google.android.exoplayer2.trackselection.TrackSelectionArray;
import com.google.android.exoplayer2.ui.PlayerView;
import com.google.android.exoplayer2.upstream.DataSource;
import com.google.android.exoplayer2.upstream.DefaultDataSourceFactory;
import com.google.android.exoplayer2.upstream.DefaultHttpDataSourceFactory;
import com.google.android.exoplayer2.util.Util;
import java.util.List;

/**
 * Manages the {@link ExoPlayer}, the IMA plugin and all video playback.
 */
/* package */ final class PlayerManager implements MediaSourceFactory, MetadataOutput, TextOutput,
    Player.EventListener {

  private final ImaAdsLoader adsLoader;
  private final DataSource.Factory dataSourceFactory;

  private SimpleExoPlayer player;
  private long contentPosition;

  public PlayerManager(Context context) {
    String adTag = context.getString(R.string.[ad_tag_url](https://pubads.g.doubleclick.net/gampad/ads?sz=640x480&iu=/124319096/external/single_ad_samples&ciu_szs=300x250&impl=s&gdfp_req=1&env=vp&output=vast&unviewed_position_start=1&cust_params=deployment%3Ddevsite%26sample_ct%3Dlinear&correlator=));
    adsLoader = new ImaAdsLoader(context, Uri.parse(adTag));
    dataSourceFactory =
        new DefaultDataSourceFactory(
            context, Util.getUserAgent(context, context.getString(R.string.application_name)));
  }

  public void init(Context context, PlayerView playerView) {
    // Create a player instance.
    player = new SimpleExoPlayer.Builder(context).build();
    adsLoader.setPlayer(player);
    playerView.setPlayer(player);

    // This is the MediaSource representing the content media (i.e. not the ad).
    String contentUrl = context.getString(R.string.content_url);
    MediaSource contentMediaSource = buildMediaSource(Uri.parse([contentUrl](https://adinsertplayer.s3.ap-south-1.amazonaws.com/sampleoutput0302_modified_1_Live.m3u8)));

    // Compose the content media source into a new AdsMediaSource with both ads and content.
    MediaSource mediaSourceWithAds =
        new AdsMediaSource(
            contentMediaSource,  /*adMediaSourceFactory= */ this, adsLoader, playerView);

    player.addMetadataOutput(this);
    player.addTextOutput(this);
    player.addListener(this);

    player.seekTo(contentPosition);
    player.prepare(mediaSourceWithAds);

    player.setPlayWhenReady(true);
  }


  @Override
  public void onTracksChanged(TrackGroupArray trackGroups, TrackSelectionArray trackSelections) {
    for (int i = 0; i < trackGroups.length; i++) {
      TrackGroup trackGroup = trackGroups.get(i);
      for (int j = 0; j < trackGroup.length; j++) {
        Metadata trackMetadata = trackGroup.getFormat(j).metadata;
        if (trackMetadata != null) {
          Log.d("METADATA TRACK", trackMetadata.toString());
        }
      }
    }
  }

  @Override
  public void onMetadata(Metadata metadata) {
    Log.d("METADATA", metadata.toString());
  }

  @Override
  public void onCues(List<Cue> cues) {
    Log.d("CUES", cues.toString());
  }

  public void reset() {
    if (player != null) {
      contentPosition = player.getContentPosition();
      player.release();
      player = null;
      adsLoader.setPlayer(null);
    }
  }

  public void release() {
    if (player != null) {
      player.release();
      player = null;
    }
    adsLoader.release();
  }

  // MediaSourceFactory implementation.

  @Override
  public MediaSource createMediaSource(Uri uri) {
    return buildMediaSource(uri);
  }

  @Override
  public int[] getSupportedTypes() {
    // IMA does not support Smooth Streaming ads.
    return new int[]{C.TYPE_DASH, C.TYPE_HLS, C.TYPE_OTHER};
  }

  // Internal methods.

  private MediaSource buildMediaSource(Uri uri) {
    @ContentType int type = Util.inferContentType(uri);
    switch (type) {
      case C.TYPE_DASH:
        return new DashMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
      case C.TYPE_SS:
        return new SsMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
      case C.TYPE_HLS:
        return new HlsMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
      case C.TYPE_OTHER:
        return new ProgressiveMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
      default:
        throw new IllegalStateException("Unsupported type: " + type);
    }
  }

}

The METADATA TRACK METADATA and CUES logs never happen.

Any clue or expedite help on this would highly appreciated 馃憤
Thanks in advance :)

question
Source
picture sunilkumarjena21

All 7 comments

Apple's test stream (http://devimages.apple.com/samplecode/adDemo/ad.m3u8) - This stream not event playing in Exoplayer

This stream works fine for me (and outputs metadata as expected). Note that you'll need to add android:usesCleartextTraffic="true" to the application element of your AndroidManifest.xml, since it's using http rather than https.

The content can be viewed here: https://adinsertplayer.s3.ap-south-1.amazonaws.com/sampleoutput0302_modified_1_Live.m3u8

Does this stream contain ID3 metadata, which is what we support (and is what the Apple test stream contains)? Note that EXT-X-CUE-IN and EXT-X-CUE-OUT are not part of the HLS specification.

ojw28 picture ojw28 on 6 Oct 2019
👍1

Hello @ojw28 ,
Thanks for your quick response :)

Apple's test stream (http://devimages.apple.com/samplecode/adDemo/ad.m3u8) - Now this stream is playing perfectly fine in Exoplayer after adding the http convention

Only the below call back method is triggering when used the above Apple's test stream
But other two callback onCues & 'onTracksChanged' not triggering.

@Override public void onMetadata(Metadata metadata) { Log.d("METADATA", metadata.toString()); }

[REQUIRED]Question:-
Would be great if you could suggest the process to implement client side in-stream ad insertion, when we hit this metadata ID3 event.
Curently we want to have a demo using the above Apple's test stream.

Note:The existing demo-ima only having the pre-roll & post-roll client side ad integration guide.

Any demo or example link will do the job for us.

Thanks in advance 馃憤

sunilkumarjena21 picture sunilkumarjena21 on 6 Oct 2019

But other two callback onCues & 'onTracksChanged' not triggering.

Why are you expecting these to be triggered? The former is related to subtitles and is documented as such. The latter is related to tracks changing, which isn't something that happens for the test stream. Metadata is output via onMetadata, which you're seeing is being fired correctly.

ojw28 picture ojw28 on 6 Oct 2019
👍1

Hi @ojw28 ,
Thanks for your expedite support.

Can you please answer the later part of our query.
Actually we need your guidance onclient side in-stream ad insertion.

[REQUIRED]Question:-
Would be great if you could suggest the process to implement client side in-stream ad insertion, when we hit this metadata ID3 event.
Curently we want to have a demo using the above Apple's test stream.

Note:The existing demo-ima only having the pre-roll & post-roll client side ad integration guide.

Any demo or example link will do the job for us.

Thanks in advance 馃憤

sunilkumarjena21 picture sunilkumarjena21 on 6 Oct 2019

First, it should be noted that Apple's test stream is an example of server-side ad insertion. The purpose of the ID3 metadata is simply to indicate to the client when the ad starts and ends (and progress within it). It's not intended that the metadata be used for actually triggering a client-side ad to be inserted.

Note:The existing demo-ima only having the pre-roll & post-roll client side ad integration guide.

ExoPlayer's main demo app has additional examples that demonstrate mid-roll ad insertion as well. Note that you need to build the withExtensions build variant for them to work, as documented here.

Would be great if you could suggest the process to implement client side in-stream ad insertion, when we hit this metadata ID3 event.

This seems like custom behavior that you want for your application, and so is outside of scope for this issue tracker. If you really want to do this, then roughly speaking, I guess you should listen to the ID3 event via onMetadata, store the current playback position, re-prepare the player to play the ad, and then when the ad is completed re-prepare the player with the content MediaSource, restoring the stored playback position. This is something you'll have to figure out for yourself, however.

ojw28 picture ojw28 on 7 Oct 2019
👍1

Hi @ojw28 ,
Thanks for your valuable suggestion on the custom requirement part :)
As per your suggestion,I have successfully implemented the mid-roll ad insertion using the ima-demo as documented here.

then when the ad is completed re-prepare the player with the content MediaSource, restoring the stored playback position

  • Requirement is to play multiple mid-roll ads till the duration or the end time received from the Metadata .So that we can discardAdbreak().

SAMPLE DURATION:-
`#EXT-X-CUE-OUT:16.729778

EXTINF:16.729778,

pre_process_output6.ts

EXT-X-CUE-IN`

[QUESTION]
Apple's test stream (http://devimages.apple.com/samplecode/adDemo/ad.m3u8) :- The stream containing metadata don't have any such timeStamp (Either duration or end Time till the ad should play).So that we will play the mid-roll ad till that time.

Thanks in advance for your input for this. 馃憤

[Suggested Implementation]

`
/**

  • Manages the {@link ExoPlayer}, the IMA plugin and all video playback.
    /
    /     package */ final class PlayerManager implements MediaSourceFactory, AdEvent.AdEventListener,
    AdErrorEvent.AdErrorListener, MetadataOutput,
    Player.EventListener {

//private final ImaAdsLoader adsLoader;
private final DataSource.Factory dataSourceFactory;

// private SimpleExoPlayer player;
private long contentPosition;

// The video player.
private SimpleExoPlayer mVideoPlayer;

// Factory class for creating SDK objects.
private ImaSdkFactory mSdkFactory;

// The AdsLoader instance exposes the requestAds method.
private AdsLoader mAdsLoader;

// AdsManager exposes methods to control ad playback and listen to ad events.
private AdsManager mAdsManager;

// Whether an ad is displayed.
private boolean mIsAdDisplayed;

MediaSource contentMediaSource;

public PlayerManager(Context context) {
dataSourceFactory =
new DefaultDataSourceFactory(
context, Util.getUserAgent(context, context.getString(R.string.application_name)));
}

public void init(Context context, PlayerView playerView, ViewGroup mAdUiContainer,
View mPlayButton) {

// Create an AdsLoader.
mSdkFactory = ImaSdkFactory.getInstance();
AdDisplayContainer adDisplayContainer = mSdkFactory.createAdDisplayContainer();
adDisplayContainer.setAdContainer(mAdUiContainer);
ImaSdkSettings settings = mSdkFactory.createImaSdkSettings();
settings.setAutoPlayAdBreaks(false);
mAdsLoader = mSdkFactory.createAdsLoader(
    context, settings, adDisplayContainer);

String adTag = context.getString(R.string.ad_tag_url);
requestAds(adTag);

// Add listeners for when ads are loaded and for errors.
mAdsLoader.addAdErrorListener(this);
mAdsLoader.addAdsLoadedListener(new AdsLoader.AdsLoadedListener() {
  @Override
  public void onAdsManagerLoaded(AdsManagerLoadedEvent adsManagerLoadedEvent) {
    // Ads were successfully loaded, so get the AdsManager instance. AdsManager has
    // events for ad playback and errors.
    mAdsManager = adsManagerLoadedEvent.getAdsManager();

    // Attach event and error event listeners.
    mAdsManager.addAdErrorListener(PlayerManager.this);
    mAdsManager.addAdEventListener(PlayerManager.this);
    mAdsManager.init();
  }
});

// Create a player instance.
mVideoPlayer = new SimpleExoPlayer.Builder(context).build();
//mAdsLoader.setPlayer(mVideoPlayer);
playerView.setPlayer(mVideoPlayer);

// This is the MediaSource representing the content media (i.e. not the ad).
String contentUrl = context.getString(R.string.content_url);
contentMediaSource = buildMediaSource(Uri.parse(contentUrl));

/* // Compose the content media source into a new AdsMediaSource with both ads and content.
MediaSource mediaSourceWithAds =
new AdsMediaSource(
contentMediaSource, // adMediaSourceFactory= // this, adsLoader, playerView);*/
mVideoPlayer.addMetadataOutput(this);
mVideoPlayer.addListener(this);
// Prepare the player with the source.
mVideoPlayer.seekTo(contentPosition);
mVideoPlayer.prepare(contentMediaSource);
mVideoPlayer.setPlayWhenReady(true);

}

/**

  • Request video ads from the given VAST ad tag.
    *
  • @param adTagUrl URL of the ad's VAST XML
    */
    private void requestAds(String adTagUrl) {
    // Create the ads request.
    AdsRequest request = mSdkFactory.createAdsRequest();
    request.setAdTagUrl(adTagUrl);
    request.setContentProgressProvider(new ContentProgressProvider() {
    @Override
    public VideoProgressUpdate getContentProgress() {
    if (mIsAdDisplayed || mVideoPlayer == null || mVideoPlayer.getDuration() <= 0) {
    return VideoProgressUpdate.VIDEO_TIME_NOT_READY;
    }
    return new VideoProgressUpdate(mVideoPlayer.getCurrentPosition(),
    mVideoPlayer.getDuration());
    }
    });
// Request the ad. After the ad is loaded, onAdsManagerLoaded() will be called.
mAdsLoader.requestAds(request);

}

public void reset() {
if (mVideoPlayer != null) {
contentPosition = mVideoPlayer.getContentPosition();
mVideoPlayer.release();
mVideoPlayer = null;
//mAdsLoader.setPlayer(null);
}
}

public void release() {
if (mVideoPlayer != null) {
mVideoPlayer.release();
mVideoPlayer = null;
}
//mAdsLoader.release();
}

// MediaSourceFactory implementation.

@Override
public MediaSource createMediaSource(Uri uri) {
return buildMediaSource(uri);
}

@Override
public int[] getSupportedTypes() {
// IMA does not support Smooth Streaming ads.
return new int[]{C.TYPE_DASH, C.TYPE_HLS, C.TYPE_OTHER};
}

// Internal methods.

private MediaSource buildMediaSource(Uri uri) {
@ContentType int type = Util.inferContentType(uri);
switch (type) {
case C.TYPE_DASH:
return new DashMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
case C.TYPE_SS:
return new SsMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
case C.TYPE_HLS:
return new HlsMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
case C.TYPE_OTHER:
return new ProgressiveMediaSource.Factory(dataSourceFactory).createMediaSource(uri);
default:
throw new IllegalStateException("Unsupported type: " + type);
}
}

@Override
public void onAdEvent(AdEvent adEvent) {
Log.i(TAG, "Event: " + adEvent.getType());

// These are the suggested event types to handle. For full list of all ad event
// types, see the documentation for AdEvent.AdEventType.
switch (adEvent.getType()) {
  case LOADED:
    // AdEventType.LOADED will be fired when ads are ready to be played.
    // AdsManager.start() begins ad playback. This method is ignored for VMAP or
    // ad rules playlists, as the SDK will automatically start executing the
    // playlist.
    mAdsManager.start();
    break;
  case CONTENT_PAUSE_REQUESTED:
    // AdEventType.CONTENT_PAUSE_REQUESTED is fired immediately before a video
    // ad is played.
    mIsAdDisplayed = true;
    mVideoPlayer.stop();
    break;
  case CONTENT_RESUME_REQUESTED:
    // AdEventType.CONTENT_RESUME_REQUESTED is fired when the ad is completed
    // and you should start playing your content.
    mIsAdDisplayed = false;
    mVideoPlayer.setPlayWhenReady(true);
    break;
  case ALL_ADS_COMPLETED:
    if (mAdsManager != null) {
      mAdsManager.destroy();
      mAdsManager = null;
    }
    break;
  default:
    break;
}

}

@Override
public void onAdError(AdErrorEvent adErrorEvent) {
Log.e(TAG, "Ad Error: " + adErrorEvent.getError().getMessage());
mVideoPlayer.setPlayWhenReady(true);
}

@Override
public void onMetadata(Metadata metadata) {
//Log.d("Sunil","Inside advert!!");
for (int i = 0; i < metadata.length(); i++) {
Metadata.Entry entry = metadata.get(i);
if (entry instanceof GeobFrame) {
GeobFrame geobFrame = (GeobFrame) entry;
Log.d(TAG, String.format("%s: mimeType=%s, filename=%s, description=%s",
geobFrame.id, geobFrame.mimeType, geobFrame.filename, geobFrame.description));
if (geobFrame.filename.contains("ad") && android.text.TextUtils
.isDigitsOnly(String.valueOf(geobFrame.filename.charAt(2)))) {
Log.d("Sunil","Inside advert!!");
mAdsManager.start();
}
}
}
}

}

sunilkumarjena21 picture sunilkumarjena21 on 8 Oct 2019

Apple's test stream (http://devimages.apple.com/samplecode/adDemo/ad.m3u8) :- The stream containing metadata don't have any such timeStamp (Either duration or end Time till the ad should play).So that we will play the mid-roll ad till that time.

As above Apple's test stream is an example of server-side ad insertion. The ad is already inserted directly into the content (it might not look like an actual ad, because it's a test stream).

Closing, since I don't think these questions are directly related to ExoPlayer.

ojw28 picture ojw28 on 8 Oct 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

stari4ek picture stari4ek  路  3Comments
eneim picture eneim  路  3Comments
qeadsw picture qeadsw  路  3Comments
arjun-redspace picture arjun-redspace  路  3Comments
AndrewJack picture AndrewJack  路  3Comments