-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathDocumentation000_audio_streaming.txt
192 lines (124 loc) · 13.2 KB
/
Documentation000_audio_streaming.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
::v3.0::1o.2o22::2019.4.0::
================================================================================
AudioStream Usage instructions:
================================================================================
(Note: for better iOS integration - with just playback and limited formats - see also AudioStreamIce [https://assetstore.unity.com/packages/slug/223601?aid=1100l7sC8])
There are two main components which stream and play audio from generic location (network/filesystem):
- the first completely bypasses Unity audio (playing at stream specified sample rate), but is capable of playing just the audio signal from the stream 1:1, while the second one behaves like a standard AudioSource enabling all usual functionality such as Unity 3D spatialisation, effecting and mixing with other Unity audio sources.
First component is
AudioStreamMinimal,
component for 'normal' Unity AudioSource is called AudioStream.
Both consume (compressed) audio data retrieved either from network via UnityWebRequest, or directly from file system
- UnityWebRequest is used for all communication, except when attempting to resolve an Url redirect for initial connection -
for that platform System.Net.HttpWebRequest is used - this is the deafult behaviour which can be turned off via 'attemptToResolveUrlRedirection'
The 'infinite' radio streams are limited to 4 GB download ( which amounts to about 20 hours of playback of 2 channel, 44100 Hz audio )
Both can optionally save their streamed content into local cache for offline playback later - see 'Downloading the original stream', and 'AudioStreamRuntimeImport' below.
Each component further provides selective Console logging and UnityEvent messages.
As a source a http/s link to a PLS or M3U/8 playlist distributed commonly by web/net radio stations, a direct link to a remote file such as podcast which is then streamed in full,
or a local filesystem path to a file can be specified.
::: by default FMOD autodetects format of the stream automatically. This works reliably on desktops and Android, but iOS requires selecting correct audio format for each respective stream manually.
It is important to select correct audio type, though - if wrong format is selected ( for example mpeg for Ogg Vorbis radio/stream ), FMOD will probably not play it and might run unrecoverable problems when starting/Playing and Stopping/releasing the stream.
:::
Filesystem relative paths: you can reference file in e.g. StreamingAssets using relative paths such as ./Assets/StreamingAssets/FILE.mp3 as url
::: the first entry from playlist is played. This is usually the correct stream, but if there are more than one streams specified
you'd have to extract desired stream/link from the downloaded playlist manually first.
Note that M3U/8 support is limited - only direct, non chunked and non recursive playlist (in other words non HLS streams) can be played with first detected source as mentioned above.
:::
UnityWebRequest by default accepts all certificates and their public keys. In 2018.1 and up this is handled by custom certificate handler so it can be overriden if needed.
::: Any necessary resampling is done by Unity automatically as needed.
::: AudioStreamMinimal uses less resources since it doesn't go through Unity AudioSourcem, so if all Unity audio features are not needed, I generally recommend using it for just for 1:1 playback.
::: The amount of independently running AudioStreamMinimal/AudioStream components created at the same time is limited only by available system resources
For networking enabled component see notes below.
Streaming was tested on iOS, Android, Windows and OS X, and user project on HoloLens.
:::::: Be sure to have have 'Run in Background' turned on in Player settings and you need to switch to another application while still needing audio being processed by the Editor/application.
====
NOTE: All components - except AudioSourceOutputDevice and FMOD direct (Minimal/Media) components - can be used as input for Unity Mixer since they use usual Unity AudioClip
AudioSourceOutputDevice bypasses it completely (since it redirects signal to a device Unity has no knowledge of currently) - but can be attached to main listener in which case the whole application audio (and final mix) ends up on chosen device.
====
! Please be aware that each component which manipulates and manages its own AudioSource makes the Unity AudioSource component on GameObject unusable by user/Unity/other components at runtime
So in general it's advisable to place each AudioStream* component which uses AudioSource on its own exclusive GameObject
====
The whole system is designed to be modular with each module being independent - you should be able to e.g. delete any folder in AudioStream/Scripts (except AudioStreamSupport) - with associated demo scene/s -
if not needed, without impacting the rest.
====
! iOS NOTE: If you want to use Bluetooth connected devices for playback you have to enable iOS player for recording and set project #define symbol
Please see iOS recording specific notes in 'Documentation060_mobiles.txt' next to this file
================================================================================
Network/web proxy:
================================================================================
All UnityWebRequests and HttpWebRequest used by components should follow operating system setting for proxy.
(see Unity manual e.g. https://docs.unity3d.com/2019.4/Documentation/ScriptReference/Networking.UnityWebRequest.html)
================================================================================
Supported formats:
================================================================================
AudioStream can stream all formats FMOD recognises with the exceptions below for audio streamed over the network. For complete list see https://en.wikipedia.org/wiki/FMOD#File_formats
RAW format is supported also by exposing format values in the AudioStream/AudioStreamMinimal Inspector since these must be set explicitly by user.
For Resonance specific formats see below.
'Blockalign' and 'Blockalign download multiplier':
- 'Blockalign' defines chunk size for the decoder to read (smaller values help initial latency), ['Blockalign' times 'Blockalign download multiplier'] defines how much data has to be downloaded prior to playback attempt -
make this as small as possible until the stream still starts, e.g. for common net radios 2-4k times 1 - 2 should work
Some (mp3) files with embedded artwork need this to be read completely so for them 64k times 10 or higher might be needed.
!! Currently ( 2.0 < ) there is a format limitation though:
Ogg/Vorbis format can't be played until the whole file/media is accessible. This means that the whole file has to be downloaded first in order to successfully play it.
Please set 'blockalign' to physical (compressed) file size and 'blockalignDownloadMultiplier' to 1 for Ogg/Vorbis files.
I tried to resolve this with FMOD, but the situation is unresolved for the time being.
Note that the above applies only for audio streamed from network - local files are not affected.
*AAC format*:
there's no special setup or any special considerations regarding this format. The *ONLY* actual difference compared to other formats is that the media *must* be played on an actual physical device (e.g. iOS/Andoid phone) which has hardware decoding capability for this format. That means that you will not be able to test it in e.g. Editor or standalone builds.
One thing might be necessary (although sometimes FMOD can cope without it) - you might want to set 'Stream Type' to 'AUTODETECT'.
It's not possible to netstream AAC [though .caf or .m4a were not tested on iOS]
*MIDI format*:
- audio MIDI files playback is supported
================================================================================
Downloading the original stream:
================================================================================
If AudioStream/AudioStreamMinimal url points to network (i.e. the url begins with 'http'),
'Play From Cache' and 'Download To Cache' options become available:
- 'Download To Cache' will be simultaneously saving incoming audio into local cache in its original format
Note: currently any previous save for given Url will be automatically overwritten
- when started with 'Play From Cache' On, the component will try to locate existing cached media for given Url and will play that instead of live stream if it exists
The identifier/filename in the cache is derived from entered Url; (the original Url will be stripped of non filesystem valid characters, and will be truncated to 1oo characters)
Cache is located at Application.persistentDataPath by default and can be customized - please see 'Local disk cache' in Documentation.txt
When the playabck is stopped/finished and 'Download To Cache' is On, the full resulting path of saved file is passed via 'OnPlaybackStopped' event - see AudioStreamDemo.cs
Note: when using 'Download To Cache' for playlists Url, the cache/filename is derived from original playlist Url (although its content is played)
- this is because to determine its content it has to be downloaded first - and thus to connect to the network - the first item in it is not known yet when started
So e.g. a cache item might be saved with extension '.pls' although it's mpeg audio - you might want to change the extension afterwards for usage outside the app if you know the format;
[the component itself ignores the extension if the stream type is set expliciately]
================================================================================
AudioStreamRuntimeImport:
================================================================================
This allows to stream a file/audio faster than realtime and use its data to create Unity AudioClip which can then be used/played as normally while still going via FMOD decoder first at runtime.
This way it's also possible to create an AudioClip from audio files which formats Unity normally doesn't but FMOD does support.
'Why is import of audio files in the Editor fast and why this can't be that fast' ? was often a question -
the answer is Unity uses specialized tool for importing audio assets in the Editor which is not available at runtime.
AudioStreamRuntimeImport is almost identical to AudioStream component, except that it doesn't run at realtime speed but faster and caches all decoded audio as raw samples.
- running this with normal web audio streams / radios is not recommended (the bandwidth is not enough and decoding won't work)
- it's suitable for higher bandwidth LANs, and for local filesystem media
- Tags are supported the same way as in normal streaming components
- Download progress is being reported via decoded_bytes and file_size public members, note that:
1] file_size is the size of the encoded file (either on the filesystem or as reported from the network), streamed content reports -1
2] decoded_bytes are actual bytes decoded by FMOD
That means that decoded_bytes will be usually much higher than the size of the file (so you get no proper upper bound for the download progress currently)
Downloaded data is progressively cached in application's set cache directory
- it's filename is derived from hash of AudioStreamRuntimeImport's url, with appended '.raw' extension
- original samplerate and channels are saved at the beginning of the file, so raw format can be played back later
- (samplerate and channels occupy 8 bytes together so this has no impact on playback)
- these are followed by raw interleaved stream of PCMFLOAT serialized as bytes (sizeof(float) bytes per sample)
- sudio data is cached as decoded RAW PCM bytes - these can get quite large.
- please also see 'Local disk cache' in Documentation.txt
If 'overwriteCachedDownload' is not set, network is not even started if a file associated with AudioStreamRuntimeImport's url is found in the cache, instead AudioClip is constructed and played immediately.
When it's finished a new AudioClip containing the decoded audio is returned via 'OnAudioClipCreated' Unity event.
Actual samples used for the AudioClip can be also retrieved in 'OnSamplesCreated' event which is invoked immediately before 'OnAudioClipCreated'
- this event is also useful for projects which run with Unity audio disabled - final samples are returned and AudioClip creation will be ignored
- samples are in PCM FLOAT format, with samplerate and channels passed as parameters in the event
See AudioStreamRuntimeImportDemo scene for example usage.
================================================================================
AudioStreamMemory:
================================================================================
Works similarly as 'AudioStreamRuntimeImport' above, but decodes audio from in-memory location
- set buffer IntPtr and its length via memoryLocation and memoryLength fields, and call Play() on the component.
All features of AudioStream such as choosing audio type, tags, logging and Unity events are supported.
Cache is not reused - each Play() call overwrites any existing/previous run.
- please also see AudioStreamRuntimeImport above
If you're using this to decode/stream in-memory raw PCM data, be sure to set correct sound type on the component (i.e. RAW format, and correct samplerate + channels) - otherwise it won't work properly.
See AudioStreamMemoryDemo scene for example usage.