Data
Augmenta OSC Protocol v2.0
Note : All coordinates are (otherwise specified) normalized [0-1] with a top left origin.
When an object enters, is updated or leaves the scene
/object/enter arg0 arg1 ... argN
/object/update arg0 arg1 ... argN
/object/leave arg0 arg1 ... argN
where args are :
0: frame (int) // Scene frame number
1: id (int) // id ex : 42th object to enter stage has id=42
2: oid (int) // Ordered id ex : if 3 objects on stage, 43th object can have oid=2
3: age (float) // Alive time (in s)
4: centroid.x (float 0:1) // Position projected to the ground (normalized)
5: centroid.y (float 0:1)
6: velocity.x (float) // Speed and direction vector (in unit.s-1) (normalized)
7: velocity.y (float)
8: orientation (float 0:360) // With respect to horizontal axis right (0° = (1,0)), rotate counterclockwise
Estimation of the object orientation from its rotation and velocity
9: boundingRect.x (float 0:1) // Bounding box center coord (normalized)
10: boundingRect.y (float 0:1)
11: boundingRect.width (float 0:1) // Bounding box width (normalized)
12: boundingRect.height (float 0:1)
13: boundingRect.rotation (float 0:360)// With respect to horizontal axis right counterclockwise
14: height (float) // Height of the object (in m) (absolute)
/object/enter/extra arg0 arg1 ... argN
/object/update/extra arg0 arg1 ... argN
/object/leave/extra arg0 arg1 ... argN
0: frame (int) // Scene frame number
1: id (int) // id ex : 42th object to enter scene has pid=42
2: oid (int) // Ordered id ex : if 3 object on stage, 43th object might have oid=2
3: highest.x (float 0:1) // Highest point placement (normalized)
4: highest.y (float 0:1)
5: distance (float) // Sensor distance (in m)
6: reflectivity (float) // Reflection value from Lidar sensor
Scene information
/scene arg0 arg1 ... argN
0: frame (int) // Scene frame number
1: objectCount (int) // Number of objects
2: scene.width (float) // Size (in m)
3: scene.height (float)
Fusion software specific information
/fusion arg0 arg1 ... argN
0: videoOut.offset.x (float) // Offset from scene top left (in m)
1: videoOut.offset.y (float)
2: videoOut.width (float) // Size (in m)
3: videoOut.height (float)
4: videoOut.widthInPixels (int) // Resolution (in pixels)
5: videoOut.heightInPixels (int)
Legacy OSC protocols can be found here
Standard TUIO 1.1 protocol plus /scene and /fusion messages from the OSC protocol.
Note : TUIO Protocol does not give access to the complete data collection from the Augmenta protocol. As the TUIO protocol doesn't support sending both the bounding box and the centroid position and orientation, only the bounding box information are sent with the blob descriptors mode. In the case of cursor and object descriptors mode, the centroid information is sent.
Here is the correspondence table between Augmenta data and TUIO data:
| TUIO | Augmenta (cursor and object TUIO mode) | Augmenta (blob TUIO mode) |
|---|---|---|
| s | id | id |
| i | oid | oid |
| x, y, z | centroid.x, centroid.y, height/20 | boundingRect.x, boundingRect.y, height/20 |
| a, b, c | orientation, 0, 0 | boundingRect.rotation, 0, 0 |
| w, h, d | - | boundingRect.width, boundingRect.height, height/10 |
| f, v | - | w x h, w x h x d |
| X, Y, Z | velocity.x, velocity.y, 0 | velocity.x, velocity.y, 0 |
| A, B, C | 0, 0, 0 | 0, 0, 0 |
| m | 0 | 0 |
| r | 0 | 0 |
Reminder of TUIO 1.1 semantic types:
| TUIO Data | Description | Type |
|---|---|---|
| s | Session ID (temporary objcet ID) | int32 |
| i | Class ID (e.g. marker ID) | int32 |
| x, y, z | Position |
float32 [0,1] |
| a, b, c | Angle |
float32 [0,2PI] |
| w, h, d | Dimension |
float32 [0,1] |
| f, v | Area, Volume |
float32 [0,1] |
| X, Y, Y | Velocity vector (motion speed and direction) | float32 |
| A, B, C | Rotation velocity vector (rotation speed and direction) | float32 |
| m | Motion acceleration | float32 |
| r | Rotation acceleration | float32 |
Presets are implemented in fusion to help you choose the best TUIO format:
| Presets | Supported descriptors | Supported dimensions | Default | Description |
|---|---|---|---|---|
| None | Cursor, Blob, Object | 2D, 2.5D, 3D | - | - |
| Minimal | Cursor | 2D | 2D Cursor | - |
| Best | Blob | 2.5D, 3D | 2.5D Blob | Best fit for Augmenta's data |
| Notch | Cursor, Object, Blob | 2D | 2D Blob | Supported formats in Notch |
| Touch Designer | Cursor, Object, Blob | 2D, 2.5D, 3D | 3D Blob | Supported formats in Touch Designer |
Note : By default, OSC and TUIO are sent on the same port for Touch Designer's preset as you can receive both OSC messages and TUIO messages on the same connection. For Notch's preset they are sent on separate ports.
Augmenta JSON Protocol v2.0
Note : All coordinates are (otherwise specified) normalized [0-1] with a top left origin.
When an object enters the scene
{"object":
{ "enter": {
"frame": frame,
"id": id,
"oid": oid,
"age": age,
"centroid": {
"x": centroid.x,
"y": centroid.y
},
"velocity": {
"x": velocity.x,
"y": velocity.y
},
"orientation": orientation,
"boundingRect": {
"x": boundingRect.x,
"y": boundinRect.y,
"width": boundingRect.width,
"height": boundingRect.height,
"rotation": boundingRect.rotation
},
"height": height,
"extra": {
"frame": extra.frame,
"id": extra.id,
"oid": extra.oid,
"highest": {
"x": extra.highest.x,
"y": extra.highest.y
},
"distance": extra.distance,
"reflectivity": extra.reflectivity
}
}
}
When an object is updated (same format than enters)
{"object":
{ "update": {
...
}
}
}
When an object leaves the scene
{"object":
{ "leave": {
...
}
}
}
Scene information
{"scene": {
"frame": frame,
"objectCount": objectCount,
"scene": {
"width": scene.width,
"height": scene.height
}
}
Fusion software specific information
{"fusion":
"textureOffset": {
"x": textureOffset.x,
"y": textureOffset.y
},
"textureBounds": {
"x": textureBounds.x,
"y": textureBounds.y
},
"targetOutSize": {
"x": targetOutSize.x,
"y": targetOutSize.y
}
}
Augmenta OSC Protocol v3.0
Each frame is sent as an udp osc bundle
- i : int
- f : float (if not specified, type is float)
- t : osc timestamp format
Note : This is the current internal coordinate system of Augmenta, the origin and axis are freely tunable when sending data.
- Origin : Top-left of detected plane
- Axis : left-handed, Z up
- System : metric
- Better adapted to OSC support (simplicity of parsing in nodal software)
- Robust point deletion mechanism (no need to have a timeout to clear objects if no update is received)
This protocol is designed for 2 types of use:
- protocol easy to exploit for nodal creation software
- protocol easy to understand and to support in software that allows scripting and that allows to be robust
We have two types of bundles. The first one is sent to each frame, it contains the information of all the objects in the scene and the frame number. The second one contains the information of the scene (its size, its position etc), it is sent at a frequency of 10 Hz.
You can choose between absolute or relative arguments. You can also can check a case to have split coordinates, if it is unchecked you received one line information for x, y and z.
// Object updates must be packed into bundles. Each bundle must include a frame id as well as the list of the active objects for this frame.
# Bundle header
/frame/id i // frame id (0 is reserved, resets at MAX_INT32)
# Objects updates (in same bundle)
## Mandatory args
### When relative and split are unchecked
/N/uid i // unique ID ex : 42th object to enter stage has uid=41 (resets at MAX_INT32)
/N/pos/abs x y z // centroid position 3d coordinates
### if relative and split are both checked for example
/N/uid i // unique ID ex : 42th object to enter stage has uid=41 (resets at MAX_INT32)
/N/pos/rel/x x // centroid X coordinate
/N/pos/rel/y y // centroid Y coordinate
/N/pos/rel/z z // centroid Z coordinate
## Optional args
/N/height f // height, altitude of highest point (alongside normal axis from detected plane)
/N/velocity x y z // velocity vector coordinates
/N/speed f // velocity norm in m/s
/N/weight w // confidence value
/N/state // state of the object (enter, leave, update)
/N/box/angle r // WARNING CoordSys ! rotation angle in deg (counterclockwise around normal axis from plane, 0° at top view horizontal axis)
/N/direction x y z // vector coordinates of current direction (equivalent to velocity while moving)
### If relative is unchecked
/N/box/abs x y z w h l // x y z : bounding box center coordinates, w h l : length alongside x y z axis
### ... and split is checked
/N/box/abs/x x
/N/box/abs/y y
/N/box/abs/z z
### If relative is unchecked
/N/box/rel x y z w h l // bounding box in relative coordinates
### ... and split is checked
/N/box/rel/x x
/N/box/rel/y y
/N/box/rel/z z
## Bundle end
/frame/numobj i // number of objects
/frame/objlist i... // list of objects in the scene
where N is the object id (also called slot id), starting at index 1. // Insert page : More about the slot ID algorithm
where ... is
# At least 10Hz
/scene/pos x y z // scene center 3d coordinate
/scene/size w h d // scene size
# Optional scene infos also at least 10Hz
/scene/video/pos x y z // video center 3d coordinate
/scene/video/size w h d // video surface size (in m)
/scene/video/res i i // resolution in pixels (square)
/scene/zoneN/abs x y z
/scene/zoneN/rel x y z
/scene/zoneN/size w l h
/scene/zoneN/relsize w l h
# sent when installing the server and activating the Clean Scene trigger.
/scene/clear // reset signal -> clear all points
// clear signal sent on (re)connection
scene/clear
// scene infos bundle @10Hz and when (re)connecting
{
scene/pos 0 0 0 //scene info
scene/size 10 15 3 //scene info
}
... // a lot of frames are received
# another method to clear the scene
{
scene/frm 1
scene/numobj 0
scn/objlist
}
# objects update bundle
{
/frame/id 21375 // update frame nbr
/1/uid 13
/1/pos/rel 0.21 0.08 0.4
/1/height 1.78
/1/velocity 0.8 0.72 0.0
/1/speed 1.114
/1/box 0.7 0.45 1.78
/1/box/angle 50.3
/1/weight 0.8
/1/state update
/2/uid 34
/2/pos/rel 0.95 0.2 0.35
/2/height 1.6
/2/velocity 0.52 0.4 0.0
/2/speed 0.656
/2/box 0.7 0.45 1.78
/2/box/angle 110
/2/wgt 0.0
/2/stt leave
/4/uid 512
/4/pos/rel 0.53 0.29 0.4
... // other infos
/4/weight 0.1
/4/state enter
frame/numobj 4
frame/objlist 13 34 512 147 // objects list, so we can clear whatever is not in this list
}
# second bundle for this update
{
/frame/id 21375
/5/uid 147
/5/pos/rel 0.5 0.5 0.5
...
/5/weight 1.0
/5/state update
frame/numobj 4
frame/objlist 13 34 512 147
}
When we can't script, we can use the weight value as an alpha value of an asset that would represent the user (1.0 when an object is correctly detected, 0.0 when it disappears, alpha value can vary [0,1] depending on the detection quality).
When we can script/program, we can use the /frame/objlist message to remove all objects that are not part of this list. This is a global state of all currently active objects. This information is transmitted to each frame, so if we lose one of them we can have the information in the next frame.
It is possible to change the sending addresses. For example, you can change /N/pos/abs to /N/position/abs.
The osc protocol is designed for quick and easy use. This comes at the expense of precision and performance. Here are some limitations.
- Only one angle of rotation for the bounding box is supported
- 0° rotation (bounding box) is defined around the horizontal axis, which has some limitations regarding to the potential position with an angle of the scene
- Scene does not have any rotation angle in space
- Height data is not defined mathematicaly, it's up to the user to use the correct side of the scene