-
Notifications
You must be signed in to change notification settings - Fork 15
/
Copy pathindex.html
462 lines (462 loc) · 21.2 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
<!DOCTYPE html>
<html lang="en-us">
<head>
<link href="fromelement.css" rel="stylesheet" type="text/css">
<title>
Media Capture from DOM Elements
</title>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<script class="remove" src="https://www.w3.org/Tools/respec/respec-w3c">
//<![CDATA[
<!-- keep this comment: what a frightening kludge -->
//]]>
</script>
<script class="remove" src="fromelement.js">
//<![CDATA[
<!-- keep this comment -->
//]]>
</script>
</head>
<body>
<section id="abstract">
<p>
This document defines how a stream of media can be captured from a DOM element, such as a
[^video^], [^audio^], or [^canvas^]
element, in the form of a {{MediaStream}} [[GETUSERMEDIA]].
</p>
</section>
<section id="sotd">
<p>
This document is not complete. It is subject to major changes and, while early
experimentations are encouraged, it is therefore not intended for implementation.
</p>
</section>
<section class="informative" id="intro">
<h2>
Introduction
</h2>
<p>
This document describes an extension to both HTML media elements and the HTML canvas
element that enables the capture of the output of the element in the form of streaming
media.
</p>
<p>
The captured media is formed into a {{MediaStream}} [[GETUSERMEDIA]], which can
then be consumed by the various APIs that process streams of media, such as WebRTC
[[WEBRTC]], or Web Audio [[WEBAUDIO]].
</p>
</section>
<section id="conformance">
<p>
This specification defines conformance criteria that apply to a single product: the
<dfn>user agent</dfn> that implements the interfaces that it contains.
</p>
<p>
Implementations that use ECMAScript to implement the APIs defined in this specification
must implement them in a manner consistent with the ECMAScript Bindings defined in the Web
IDL specification [[!WEBIDL]], as this specification uses that specification and
terminology.
</p>
</section>
<section>
<h2>
HTML Media Element Media Capture Extensions
</h2>
<p>
The method {{HTMLMediaElement/captureStream}} is added on HTML [[!HTML5]] media elements.
Methods for capture are added to both {{HTMLMediaElement}}
and {{HTMLCanvasElement}}.
</p>
<p>
Both {{MediaStream}} and {{HTMLMediaElement}} expose the concept
of a <q>track</q>. Since there is no common type used for
{{HTMLMediaElement}}, this document uses the term <dfn>track</dfn> to refer
to either {{VideoTrack}}
or {{AudioTrack}}.
{{MediaStreamTrack}}
is used to identify the media in a {{MediaStream}}.
</p>
<div>
<pre class="idl">partial interface HTMLMediaElement {
MediaStream captureStream ();
};</pre>
<section>
<h2>
Methods
</h2>
<dl class="methods">
<dt>
{{HTMLMediaElement/captureStream}}
</dt>
<dd>
<p>
The <dfn data-dfn-for="HTMLMediaElement">captureStream()</dfn> method produces a real-time capture of
the media that is rendered to the media element.
</p>
<p>
The captured {{MediaStream}} comprises of {{MediaStreamTrack}}s
that render the content from the set of <a href=
"https://html.spec.whatwg.org/multipage/media.html#dom-videotracklist-selectedindex">
selected</a> (for {{VideoTrack}}s, or other exclusively selected
<a>track</a> types) or <a href=
"https://html.spec.whatwg.org/multipage/media.html#dom-audiotrack-enabled">enabled</a>
(for {{AudioTrack}}s, or other <a>track</a> types that support
multiple selections) <a>track</a>s from the media element. If the media element
does not have a selected or enabled <a>track</a>s of a given type, then no
{{MediaStreamTrack}} of that type is present in the captured stream.
</p>
<p>
A [^video^] element can therefore capture a video
{{MediaStreamTrack}} and any number of audio
{{MediaStreamTrack}}s. An [^audio^] element can capture
any number of audio {{MediaStreamTrack}}s. In both cases, the set of
captured {{MediaStreamTrack}}s could be empty.
</p>
<p>
Unless and until there is a <a>track</a> of given type that is selected or enabled,
no {{MediaStreamTrack}} of that type is present in the captured stream. In
particular, if the media element does not have a source assigned, then the captured
{{MediaStream}} has no tracks. Consequently, a media element with a ready
state of <a href=
"https://html.spec.whatwg.org/multipage/media.html#dom-media-have_nothing">HAVE_NOTHING</a>
produces no captured {{MediaStreamTrack}} instances. Once metadata is
available and the selected or enabled <a>track</a>s are determined, new captured
{{MediaStreamTrack}} instances are created and added to the
{{MediaStream}}.
</p>
<p>
A captured {{MediaStreamTrack}} <a href=
"https://w3c.github.io/mediacapture-main/#ends-nostop">ends</a> when <a href=
"https://html.spec.whatwg.org/multipage/media.html#ended-playback">playback
ends</a> (and the <code>ended</code> event fires) or when the track that it
captures is no longer selected or enabled for playback. A track is no longer
selected or enabled if the source is changed by setting the {{HTMLMediaElement/src}} or
{{HTMLMediaElement/srcObject}} attributes of the media element.
</p>
<p>
The set of captured {{MediaStreamTrack}}s change if the source of the
media element changes. If the source for the media element ends, a different source
is selected.
</p>
<p>
If the selected {{VideoTrack}} or enabled {{AudioTrack}}s for the media
element change, a <code><a href=
"https://w3c.github.io/mediacapture-main/#event-mediastream-addtrack">addtrack</a></code>
event with a new {{MediaStreamTrack}} is generated for each <a>track</a>
that was not previously selected or enabled; and a <code><a href=
"https://w3c.github.io/mediacapture-main/#event-mediastream-removetrack">removetrack</a></code>
events is generated for each <a>track</a> that ceases to be selected or enabled. A
{{MediaStreamTrack}} MUST end prior to being removed from the
{{MediaStream}}.
</p>
<p>
Since a {{MediaStreamTrack}} can only end once, a track that is enabled,
disabled and re-enabled will be captured as two separate tracks. Similarly,
restarting playback after playback ends causes a new set of captured
{{MediaStreamTrack}} instances to be created. Seeking during playback
without changing track selection does not generate events or cause a captured
{{MediaStreamTrack}} to end.
</p>
<p>
The {{MediaStreamTrack}}s that comprise the captured
{{MediaStream}} become muted or unmuted as the tracks they capture change
state. At any time, a media element might not have active content available for
capture on a given track for a variety of reasons:
</p>
<ul>
<li>Media playback could be paused.
</li>
<li>A <a>track</a> might not have content for the current playback time if that
time is either before the content of that track starts or after the content ends.
</li>
<li>A {{MediaStreamTrack}} that is acting as a source could be
<dfn><a href="https://w3c.github.io/mediacapture-main/#track-muted">muted</a></dfn>
or <dfn><a href=
"https://w3c.github.io/mediacapture-main/#track-enabled">disabled</a></dfn>.
</li>
<li>The contents of the <a>track</a> might become inaccessible to the current
origin due to cross-origin protections. For instance, content that is rendered from
an HTTP URL can be subject to a redirect on a request for partial content, or the
enabled or selected tracks can change to include cross-origin content.
</li>
</ul>
<p>
Absence of content is reflected in captured tracks through the <code><a href=
"https://w3c.github.io/mediacapture-main/#dom-mediastreamtrack-muted">muted</a></code>
attribute. A captured {{MediaStreamTrack}} MUST have a
<code><a>muted</a></code> attribute set to <code>true</code> if its corresponding
source <a>track</a> does not have available and accessible content. A
<code><a href="https://w3c.github.io/mediacapture-main/#event-mediastreamtrack-mute">
mute</a></code> event is raised on the {{MediaStreamTrack}} when content
availability changes.
</p>
<p>
What output a muted capture produces as a result will vary based on the type of
media: a {{VideoTrack}} ceases to capture new frames when muted,
causing the captured stream to show the last captured frame; a muted
{{AudioTrack}} produces silence.
</p>
<p>
Whether a media element is actively rendering content (e.g., to a screen or audio
device) has no effect on the content of captured streams. Muting the audio on a
media element does not cause the capture to produce silence, nor does hiding a
media element cause captured video to stop. Similarly, the audio level or volume of
the media element does not affect the volume of captured audio.
</p>
<p>
Captured audio from an element with an <a href=
"https://html.spec.whatwg.org/multipage/media.html#dom-media-playbackrate">effective
playback rate</a> other than 1.0 MUST be time-stretched. An unplayable playback
rate causes the captured audio track to become muted.
</p>
</dd>
</dl>
</section>
</div>
</section>
<section>
<h2>
HTML Canvas Element Media Capture Extensions
</h2>
<p>
The {{HTMLCanvasElement/captureStream}} method is added to the HTML [[!HTML5]] canvas
element. The resulting {{CanvasCaptureMediaStreamTrack}} provides methods
that allow for controlling when frames are sampled from the canvas.
</p>
<div>
<pre class="idl">partial interface HTMLCanvasElement {
MediaStream captureStream (optional double frameRequestRate);
};</pre>
<section>
<h2>
Methods
</h2>
<dl class="methods">
<dt>
{{HTMLCanvasElement/captureStream}}
</dt>
<dd>
<p>
The <dfn data-dfn-for="HTMLCanvasElement">captureStream()</dfn> method produces a real-time video
capture of the surface of the canvas. The resulting media stream has a single video
{{CanvasCaptureMediaStreamTrack}} that matches the dimensions of
the canvas element.
</p>
<p>
Content from a canvas that is not <dfn><a href=
"https://html.spec.whatwg.org/multipage/canvas.html#concept-canvas-origin-clean">
origin-clean</a></dfn> MUST NOT be captured. This method throws a {{SecurityError}}
exception if the canvas is not <a>origin-clean</a>.
</p>
<p>
A captured stream MUST immediately cease to capture content if the
[= origin-clean =] flag of the source canvas becomes false after the stream is
created by {{HTMLCanvasElement/captureStream()}}. The captured {{MediaStreamTrack}}
MUST become <a>muted</a>, producing no new content while the canvas remains in this
state.
</p>
<p>
Each track that captures a canvas has a
<dfn data-dfn-for="track">{{track/[[frameCaptureRequested]]}}</dfn> internal slot that is set to <code>true</code> when a
new frame is requested from the canvas.
</p>
<p>
The value of {{track/[[frameCaptureRequested]]}} on all new
tracks is set to <code>true</code> when the track is created. On creation of the
captured track with a specific, non-zero <var>frameRequestRate</var>, the <a>user
agent</a> starts a periodic timer at an interval of <code>1/<var>frameRequestRate</var></code>
seconds. At each activation of the timer,
{{track/[[frameCaptureRequested]]}} is set to <code>true</code>.
</p>
<p>
In order to support manual control of frame capture with the
{{CanvasCaptureMediaStreamTrack/requestFrame}}() method, browsers MUST support a value of 0 for
<var>frameRequestRate</var>. However, a captured stream MUST request capture of a
frame when created, even if <var>frameRequestRate</var> is zero.
</p>
<p>
This method throws a {{NotSupportedError}} if <var>frameRequestRate</var> is negative.
</p>
<p>
A new frame is requested from the canvas when
{{track/[[frameCaptureRequested]]}} is true and the canvas is painted. Each
time that the captured canvas is painted, the following steps are executed:
</p>
<ol>
<li>For each <var>track</var> capturing from the canvas execute the following
steps:
<ol>
<li>If new content has been drawn to the canvas since it was last painted, and
if the {{track/[[frameCaptureRequested]]}} internal slot of
<var>track</var> is set, add a new frame to <var>track</var> containing what
was painted to the canvas.
</li>
<li>If a <var>frameRequestRate</var> value was specified, set the
{{track/[[frameCaptureRequested]]}} internal slot of <var>track</var>
to <code>false</code>.
</li>
</ol>
</li>
</ol>
<p>
When adding new frames to <var>track</var> containing what was painted to the
canvas, the alpha channel content of the canvas must be captured and preserved if
the canvas is not fully opaque. The consumers of this <var>track</var> might not
preserve the alpha channel.
</p>
<p class="note">
This algorithm results in a captured track not starting until something changes in
the canvas.
</p>
<table class="parameters">
<tbody>
<tr>
<th>
Parameter
</th>
<th>
Type
</th>
<th>
Nullable
</th>
<th>
Optional
</th>
<th>
Description
</th>
</tr>
<tr>
<td class="prmName">
frameRequestRate
</td>
<td class="prmType">
<code>double</code>
</td>
<td class="prmNullFalse">
<span role="img" aria-label="False">✘</span>
</td>
<td class="prmOptTrue">
<span role="img" aria-label="True">✔</span>
</td>
<td class="prmDesc"></td>
</tr>
</tbody>
</table>
<div>
<em>Return type:</em> {{MediaStream}}
</div>
</dd>
</dl>
</section>
</div>
<section>
<h2>
The {{CanvasCaptureMediaStreamTrack}}
</h2>
<p>
The <dfn>CanvasCaptureMediaStreamTrack</dfn> is an extension of
{{MediaStreamTrack}} that provide a single {{CanvasCaptureMediaStreamTrack/requestFrame()}} method.
Applications that depend on tight control over the rendering of content to the media
stream can use this method to control when frames from the canvas are captured.
</p>
<div>
<pre class="idl">[Exposed=Window] interface CanvasCaptureMediaStreamTrack : MediaStreamTrack {
readonly attribute HTMLCanvasElement canvas;
undefined requestFrame ();
};</pre>
<section>
<h2>
Attributes
</h2>
<dl data-link-for="CanvasCaptureMediaStreamTrack" data-dfn-for=
"CanvasCaptureMediaStreamTrack" class="attributes">
<dt>
<dfn><code>canvas</code></dfn> of type {{HTMLCanvasElement}}, readonly
</dt>
<dd>
The canvas element that this media stream captures.
</dd>
</dl>
</section>
<section>
<h2>
Methods
</h2>
<dl data-link-for="CanvasCaptureMediaStreamTrack" data-dfn-for=
"CanvasCaptureMediaStreamTrack" class="methods">
<dt>
{{requestFrame}}
</dt>
<dd>
<p>
The <dfn>requestFrame</dfn>() method allows applications to manually
request that a frame from the canvas be captured and rendered into the track. In
cases where applications progressively render to a canvas, this allows
applications to avoid capturing a partially rendered frame.
</p>
<p class="note">
As currently specified, this results in no {{SecurityError}} or other
error feedback if the canvas is not origin-clean. In part, this is because we
don't track where requests for frames come from. Do we want to highlight that?
</p>
<div>
<em>No parameters.</em>
</div>
<div>
<em>Return type:</em> <code>undefined</code>
</div>
</dd>
</dl>
</section>
</div>
</section>
</section>
<section>
<h2>
Security Considerations
</h2>
<p>
Media elements can render media resources from origins that differ from the origin of the
media element. In those cases, the contents of the resulting {{MediaStreamTrack}}
MUST be protected from access by the document origin.
</p>
<p>
How this protection manifests will differ, depending on how the content is accessed. For
instance, rendering inaccessible video to a [^canvas^] element [[HTML]]
causes the [= origin-clean =]
flag of the canvas to become <code>false</code>; attempting to create a Web Audio
{{MediaStreamAudioSourceNode}} [[WEBAUDIO]] succeeds, but produces no information
to the document origin (that is, only silence is transmitted into the audio context);
attempting to transfer the media using WebRTC [[WEBRTC]] results in no information being
transmitted.
</p>
<p>
The origin of the media that is rendered by a media element can change at any time. This is
even the case for a single media resource. User agents MUST ensure that a change in the
origin of media doesn't result in exposure of cross origin content.
</p>
</section>
<section>
<h2>
Change Log
</h2>
<p>
This section will be removed before publication.
</p>
<h2>
Changes since 2015-tbd-tbd
</h2>
</section>
<section class="appendix">
<h2>
Acknowledgements
</h2>
<p>
This document is based on the stream processing specification [[streamproc]] originally
developed by Robert O'Callahan.
</p>
</section>
</body>
</html>