Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BEP for audio/video capture of behaving subjects #1771

Open
bendichter opened this issue Apr 11, 2024 · 50 comments
Open

BEP for audio/video capture of behaving subjects #1771

bendichter opened this issue Apr 11, 2024 · 50 comments
Labels
BEP opinions wanted Please read and offer your opinion on this matter raw

Comments

@bendichter
Copy link
Contributor

I would like to create a BEP to store the audio and/or video recordings of behaving subjects.

While this would obviously be problematic for sharing human data, it would be useful to internal human data and for internal and shared data of non-human subjects.

Following the structure of the Task Events we will define types of files that can be placed in various data_type directories.

sub-<label>/[ses-<label>]
    <data_type>/
        <matches>_behcapture.mp3|.wav|.mp4|.mkv|.avi
        <matches>_behcapture.json

This schema will follow the standard principles of BIDS, listed here for clarity:

  • If no relevant <data_type> exists, use beh/.
  • Video or audio files that are continuous recordings split into files will use the _split- entity.
  • Video or audio files that are recorded simultaneously but from different angles or at different locations would use the _recording- entity to differentiate. We will need to modify the definition of this term to generalize it a bit to accommodate this usage. This entity would also be used to differentiate if a video and audio were recorded simultaneously but from different devices. Not that simply using the file extension to differentiate would not work because it would not be clear which file the .json maps to.
  • The start time of each audio or video recording should be noted in the scans.tsv file.

The JSON would define "streams" which would define each stream in the file.

The *_beh.json would looks like this:

{
  "device": "Field Recorder X200",
  "streams": [
    {
      "type": "audio",
      "sampling_rate": 44100.0,
      "description": "High-quality stereo audio stream."
    },
    {
      "type": "video",
      "sampling_rate": 30.0,
      "description": "Standard 1080p video stream."
    }
  ]
}

To be specific, it would follow this JSON Schema structure:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "properties": {
    "device": {
      "type": "string"
    },
    "streams": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "type": {
            "type": "string",
            "enum": ["audio", "video"]
          },
          "sampling_rate": {
            "type": "number",
            "format": "float"
          },
          "description": {
            "type": "string"
          }
        },
        "required": ["type", "sampling_rate"],
        "additionalProperties": false
      }
    }
  },
  "required": ["device", "streams"],
  "additionalProperties": false
}

This BEP would be specifically for audio and/or video, and would not include related data like eye tracking, point tracking, pose estimation, or behavioral segmentation. All of these would be considered derived and are reserved for another BEP.

@bendichter
Copy link
Contributor Author

cc @yarikoptic who is providing guidance on this concept.

@bendichter
Copy link
Contributor Author

An alternative idea is to name the files "_video.mp4|avi|mkv|..." and "_audio.mp3|wav|...". The advantage of this is it may be more clear what these files are. The disadvantages are that this does not make it clear that it's a recording of the subject as opposed to a stimulus, and that it's not clear what you should do if you have an audio/video recording.

@bendichter
Copy link
Contributor Author

bendichter commented Apr 11, 2024

Another alternative idea is to have the files called "_beh.mp3|.wav|.mp4|.mkv|.avi|...", though this conflicts with the current beh modality. If there is a beh.tsv file in the beh/ directory, then it will have an accompanying beh.json file, which would conflict with the json file that corresponds to the data (e.g. beh.mp3) file

@Remi-Gau
Copy link
Collaborator

@Remi-Gau
Copy link
Collaborator

This BEP would be specifically for audio and/or video, and would not include related data like eye tracking, point tracking, pose estimation, or behavioral segmentation. All of these would be considered derived and are reserved for another BEP.

Some of this may already be covered by the BIDS support for motion data and look at the the eyetracking BEP (PR and HTML)

@Remi-Gau
Copy link
Collaborator

tagging @gdevenyi who I think mentioned wanting to work on something like this last time I saw him.

@VisLab
Copy link
Member

VisLab commented Apr 12, 2024

The ideas for allowing annotations of movies and audios as expressed in issue #153 could be expanded to allow annotation of participant video/audio but in the imaging directories themselves with appropriate file structure to distinguish.
@neuromechanist @Remi-Gau @yarikoptic @adelavega @dungscout96 @dorahermes @arnodelorme
.

@Remi-Gau
Copy link
Collaborator

I like how those different initiatives are synching up.

Wouldn't those annotations of videos using HED when experimenters "code" their video be more appropriate as a derivative though.

@VisLab
Copy link
Member

VisLab commented Apr 12, 2024

Wouldn't those annotations of videos using HED when experimenters "code" their video be more appropriate as a derivative though.

Not necessarily.... in one group I worked with on experiments on stuttering -- the speech pathologist's annotations were definitely considered part of the original data. Most markers that you see in typical event files didn't come from the imaging equipment but are extracted from the control software or external devices. The eye trackers have algorithms to mark saccades and blinks and these are written as original data.

In my mind, if the annotations pertain to data that has been "calculated" from the original experimental data it should go into the derivatives folder. Annotations pertaining to data acquired during the experiment itself should probably go in the main folder.

@Remi-Gau
Copy link
Collaborator

I see I was more thinking of cases where videos of an animal behavior have to be annotated to code when certain behavior happened. Given this is not automated and can happen long time after data acquisition I would have seen this as more derivatives. But your examples show that the answer like in many cases will be "it depends".

@gdevenyi
Copy link

We have potential animal applications in both domains:

  1. Video with annotation timestreams coming from automated touchscreen-based animal behaviour systems.
  2. Videos of animals in classic "open field test" and similar setups where poostprocessing analysis collects a variety of annotations of the video determined by behaviour.

also I guess a:
3. Manual human annotation of videos of animals in naturalistic environments, like maternal care events

@Remi-Gau Remi-Gau added the opinions wanted Please read and offer your opinion on this matter label Apr 16, 2024
@Remi-Gau Remi-Gau changed the title RFC: BEP for audio/video capture of behaving subjects BEP for audio/video capture of behaving subjects Apr 16, 2024
@Remi-Gau Remi-Gau added the BEP label Apr 16, 2024
@DimitriPapadopoulos
Copy link
Collaborator

Would non-contiguous recordings (using the same setup) end up in the same or distinct files?

As an example, there could be cases where video recording has been stopped while taking care of a crying baby and resumed later on. Should BIDS try to enforce anything here, or leave it to end users (and data providers)?

What about other types of "time-series" data? Not sure about MEG, for EEG I know the EDF+ format allows discontinuous recordings:

EDF+ allows storage of several NON-CONTIGUOUS recordings into one file. This is the only incompatibility with EDF. All other features are EDF compatible. In fact, old EDF viewers still work and display EDF+ recordings as if they were continuous. Therefore, we recommend EDF+ files of EEG or PSG studies to be continuous if there are no good reasons for the opposite.

@bendichter
Copy link
Contributor Author

@DimitriPapadopoulos I believe this would be different runs. You would specify the start time of each run in the scans file

@yarikoptic
Copy link
Collaborator

I think there might be multiple scenarios (entities) how it could be handled:

  • runs - if e.g. this corresponds also to separate runs of neural data if any acquired along, so primarily as "this is how we intended this all to be".
  • But I wonder if we should look into adopting/extending (currently they are too narrowly focused) any of other entities meaning of which relate somehow to have "pieces of" (using term which is not yet an entity): split, part, chunk.

@neuromechanist
Copy link
Member

We have potential animal applications in both domains

From the annotation perspective in #153, an annot- entity enables multiple annotations per _media file. It might be useful here as well.

But I wonder if we should look into adopting/extending (currently they are too narrowly focused) any of the other entities meaning of which relate somehow to have "pieces of" (using term which is not yet an entity): split, part, chunk.

Any of them seems great, I currently suggested part- as an entity to use. But I can see any of the three work.

Video or audio files recorded simultaneously but from different angles or at different locations would use the _recording- entity to differentiate.

This is similar to having a stimulus with multiple tracks (left or right video streams, multiple audio channels, or separate video and audio), but they are not recording- per se. So, we might be able to look for a common entity that covers both potentially. We have two suggestions in #153 for now, (1) stream- and (2) track-. Would be happy to have any additional suggestions.

@neuromechanist
Copy link
Member

Also, as @bendichter mentioned, this proposal will very soon find its audience in human neuroscience, especially with DeepLabCut adding subject masking capabilities and newer modalities such as LiDAR and wifi motion capture comes into play.

It might be useful to have Motion-BIDS maintainers' (@sjeung and @JuliusWelzel) opinions as well.

@bendichter
Copy link
Contributor Author

How do we feel about this naming convention?

sub-<label>/[ses-<label>]
    <data_type>/
        <matches>_behcapture.mp3|.wav|.mp4|.mkv|.avi
        <matches>_behcapture.json

I'm not 100% on it myself but I can't think of anything better. Other options:

  • "_video.mp4|avi|mkv|..." and "_audio.mp3|wav|...".
  • "_behvideo.mp4|avi|mkv|..." and "_behaudio.mp3|wav|...".
  • "_behmedia.mp4|avi|mkv|mp3|wav|..."

Is there any precedence from other standards we could use here?

@gdevenyi
Copy link

Is there any precedence from other standards we could use here?

Technically mkv is container format, it could have different kinds of video/audio streams.

Should we specify non-patent-encumbered video compression formats?

@dorahermes
Copy link
Member

dorahermes commented Aug 6, 2024

@bendichter would be good to have your input on the proposed entities here.

A specific point of discussion is how open the description of the proposed -annot entity would be: for stimuli only or also for other types of annotations as discussed above?

@bendichter
Copy link
Contributor Author

@dorahermes I like the idea of a general text annotations file that annotates a media file, and I think that could certainly be relevant downstream of these behavioral capture files.

I think the needs of stimuli storage and behavioral capture storage are different. With stimuli, you often have a single file that you play many times across different subjects, sessions, and trials, so it makes sense to have a root folder for these where they can be referenced repeatedly. For behavioral captures, every capture is unique, so it would make more sense to store these alongside other types of recordings. So I like what is going on with stimuli, but I don't want that that to engulf these ideas about how to represent behavioral capture.

I also am trying to keep this to an MVP, so I'd like to push off discussion of annotations, though I will say I think the general approach you link to will probably work for behcapture as well with minimal adjustments.

@bendichter
Copy link
Contributor Author

Should we specify non-patent-encumbered video compression formats?

The most likely culprit here would be H264, which is used in mpeg files, however it seems that would be a non-issue since this would be covered under the "Free Internet Broadcasting" consideration (source)

@yarikoptic
Copy link
Collaborator

  • "_behvideo.mp4|avi|mkv|..." and "_behaudio.mp3|wav|...".

FWIW, I also think that we should have "audio", "video" in suffix (ref elsewhere) but do not think we should want to collapse an "intent" (beh) into it, moreover since we do have datatype beh and even modality so in principle could be depicted as _mod-beh but AFAIK so far we never did such way to associate (e.g. for _events.tsv).

@yarikoptic
Copy link
Collaborator

yarikoptic commented Oct 2, 2024

I think there is a good amount of overlap (datatypes, extensions) with "stimuli" BEP044. @bendichter when you get a chance, have a look at that BEP google doc.

@yarikoptic
Copy link
Collaborator

@satra pointed to https://docs.b2ai-voice.org/ where BIDS-inspired organization was also used and looked like

└── sub-<participant_id>
    ├── ses-<another_session_id>
    │   ├── beh
    │   │   ├── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_metadata.json
    │   │   └── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_response.json
    │   └── voice
    │       ├── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_audio.wav
    │       ├── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_features.pt
    │       ├── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_metadata.json
    │       └── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_transcript.txt

@niksirbi
Copy link

niksirbi commented Jan 17, 2025

Very cool initiative, I think we really need support for such modalities in BIDS, to keep up with recent trends in neuroscience.
I'm interested in following up and getting involved.

Regarding @bendichter's question on naming:

How do we feel about this naming convention?

sub-<label>/[ses-<label>]
    <data_type>/
        <matches>_behcapture.mp3|.wav|.mp4|.mkv|.avi
        <matches>_behcapture.json

I'm not 100% on it myself but I can't think of anything better. Other options:

  • "_video.mp4|avi|mkv|..." and "_audio.mp3|wav|...".
  • "_behvideo.mp4|avi|mkv|..." and "_behaudio.mp3|wav|...".
  • "_behmedia.mp4|avi|mkv|mp3|wav|..."

Is there any precedence from other standards we could use here?

How about using _videocapture / audiocapture as suffixes. I like having "video"/"audio" in the filename instead of the generic "beh", and the term "capture" disambiguates it from video/audio files used as stimuli.

@bendichter
Copy link
Contributor Author

bendichter commented Jan 17, 2025

Welcome to the thread, @niksirbi! This is a good time to start pushing on this again.

I like _videocapture / audiocapture! I like "capture" over "beh" as it is a bit more broad- not necessarily capturing "behavior." Maybe there would be situations where we want to capture video or audio that isn't a result of behavior but also isn't a stimulus? Then maybe we could have some kind of metadata categories that can give a rough idea of what we are capturing. Maybe HED could help with this?

But is it important for us to differentiate between videos with and without audio streams in the filenames? How would we do that?

@bendichter
Copy link
Contributor Author

@satra pointed to https://docs.b2ai-voice.org/ where BIDS-inspired organization was also used and looked like

└── sub-<participant_id>
    ├── ses-<another_session_id>
    │   ├── beh
    │   │   ├── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_metadata.json
    │   │   └── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_response.json
    │   └── voice
    │       ├── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_audio.wav
    │       ├── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_features.pt
    │       ├── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_metadata.json
    │       └── sub-<participant_id>_ses-<session_id>_task-<task_name>_run-<index>_transcript.txt

Thanks for the pointer, @yarikoptic! Maybe we could take some inspiration from this.

I think "voice" would not work for us. I definitely want to make this more broad than speech studies.

I also would consider features and transcript to be derived data, so out of scope for this initial draft, but good to think about roughly how that would look.

@VisLab
Copy link
Member

VisLab commented Jan 17, 2025

The naming of audio and video streams is also being discussed in PR#2022 with respect to BEP044 on stimuli. It might be useful to have some consistency in naming, if possible, as these initiatives move forward.

@bendichter
Copy link
Contributor Author

bendichter commented Jan 17, 2025

thanks for the pointer to BEP044, @VisLab and @yarikoptic

Things we can carry over from that:

  • filetype support looks good. I'd be happy to simply adopt that without any changes.
  • Filenames: They have: stim-[_{audio, image, video, audiovideo}].ext. We could do something analogous. Maybe capture-[_{audio, image, video, audiovideo}].ext?

Things we should not carry over:

  • BEP044 uses events for the timing of stimuli. That is appropriate for their use-case but I think for us using the scans.tsv file to indicate start times would be more consistent with BIDS conventions.

@neuromechanist
Copy link
Member

neuromechanist commented Jan 17, 2025

Using audio, audiovideo, and video here would NOT cause any confusion with stimuli. Stimulus files start with a stim_id, behavior capture files should be tied to a participant_id(?), so they will be clearly distinguishable from the first entity (also, stimulus files will not have other entities that the data files have, such as task, run, session, and desc).

IMHO,these suffixes indicate what the file is, such as being a video or audiovideo, irrespective of the purpose of the file (being a stimulus or behavior recording).

@bendichter bendichter reopened this Jan 17, 2025
@bendichter
Copy link
Contributor Author

bendichter commented Jan 17, 2025

Sorry I did not mean to close. It appears this new GitHub web UI is a bit buggy.

@niksirbi
Copy link

niksirbi commented Jan 17, 2025

But is it important for us to differentiate between videos with and without audio streams in the filenames? How would we do that?

For me that's not so important, but I'm definitely biased as I primarily work with the image content of videos.
In my mind "video" is a series of image frames with or without a corresponding audio stream. I definitely would want to know if an audio stream is present, but I don't necessarily expect that to be indicated in the file name (having that specified in the .json file would suffice for me).

In theory, we could offer a third option, _avcapture (meaning audio-video capture), but I wouldn't advocate for that.
That said, happy to hear from more audio experts on this issue.

EDIT

Just saw that in #2022 they propose audio, video, audiovideo as suffixes for the corresponding stimulus files, thanks @VisLab.
We could follow that schema for consistency and add capture at the end? The drawback being that audiovideocapture is a mouthful.

@bendichter
Copy link
Contributor Author

Are stimuli always in a directory named "stimuli"? If so, we could just use the same names: audio, video, audiovideo without "capture" and there would technically not be a collision with stimuli. Even if this is technically true, we should consider that this might be confusing to users who do not know about the stimuli standard. If they just see "...-audio.mp3" in a session dir of a BIDS dataset, it would not be immediately clear that this is a captured audio. "...-audiocapture.mp3" would be much more clear IMHO.

@neuromechanist
Copy link
Member

Are stimuli always in a directory named "stimuli"

Yes, according to the proposed specifications.

@bendichter
Copy link
Contributor Author

OK, in that case I would be happy to consider using the audio, video, audiovideo naming convention that is already being established there, though I have a slight preference for adding capture just simply for clarity.

@niksirbi
Copy link

Are stimuli always in a directory named "stimuli"

Yes, according to the proposed specifications.

Given this piece of information, I'm also fine with dropping capture.
In reality, there could be no difference between a video file presented as a stimulus vs one captured to track behaviour. In fact, one could film a mouse and show that video to another mouse as a "stimulus". It's a bit of a contrived example but it's to illustrate that nothing differentiates the two files other than intent, and having the stimulus files under a "stimuli" folder captures that intent sufficiently imo.

@bendichter
Copy link
Contributor Author

@niksirbi I'm not bothered by the length of audiovideocapture. Brevity of filenames has clearly not been a high priority for BIDS 🤣

While I agree the "stimuli" folder captures intent, the lack of stimulus folder does not capture the intent of it not being a stimulus unless the user is already aware of the stimulus rule. In that way, I think it falls just a bit short of the BIDS goal of being self-describing.

@niksirbi
Copy link

I kind of see your point, but we also have an additional differentiator. The BEP044 filenames start with stim- and ours will presumably start with something else, indicative of our intent.

@bendichter
Copy link
Contributor Author

I'm not so sure that it would. I know I said that earlier in the thread but I think I was confused. The filename would be something like: sub-001/ses-002/sub-001_ses-002_recording-right_audiocapture.wav

@niksirbi
Copy link

I'm not so sure that it would. I know I said that earlier in the thread but I think I was confused. The filename would be something like: sub-001/ses-002/sub-001_ses-002_recording-right_audiocapture.wav

That raises a different question (apologies in advance for derailing this conversation on suffixes, which is closing in on some consensus):

How would we store video/audio files capturing multiple subjects at once? It's quite common to acquire videos with multiple subjects. It will be tricky to do given BIDS' subject-first approach. But perhaps this is out-of-scope, in which case ignore my question.

@satra
Copy link
Collaborator

satra commented Jan 17, 2025

some of those derivatives files are an artifact of the past. those are no longer really there.

in terms of the broader space for audio-video capture, in the BBQS consortium where some of this is highly relevant, there are projects on dyadic interaction, navigation in the wild, in specific settings (e.g. hospital rooms and houses), multiple cameras/devices, etc.,. so capturing context will be as important as storing the streams.

@bendichter
Copy link
Contributor Author

bendichter commented Jan 17, 2025

@niksirbi, this is tricky. BIDS' file hierarchy assumes a subject -> session structure where a session has a single subject. My first thought would be to create two different session folders, one for each subject, and use a softlink to link them together. Fortunately, in DANDI (and maybe OpenNeuro?), files that are exactly identical are de-duplicated, so it would be not problem to have multiple sessions with the same video capture file, even if that file takes different names.

@niksirbi
Copy link

My first thought would be to create two different session folders, one for each subject, and use a softlink to link them together.

Something like that seems like a good compromise.

@yarikoptic
Copy link
Collaborator

first thought would be to create two different session folders, one for each subject, and use a softlink to link them together.

softlink/symlink is filesystem specific solution, thus not encouraged/used anywhere in BIDS.

BIDS' file hierarchy assumes a subject -> session structure where a session has a single subject.

  • with a solution for Make it possible to specify folders layout to be other than sub-{label}/[ses-{label}/] bids-2-devel#54 (BIDS 2̶.̶0̶1.0: flex BIDS layout (bids-2-devel/issues/54) #1809) we should be able to get away from such "hardcoded" assumption
  • BEP044 (stimuli) referenced above already somewhat provides an example where "BIDS naming" is used under stimuli/ without sub- prefix; For multi-subject recordings, it might have made sense, if we had that extra entity recording or alike, to have recordings/ folder to collect them all and then similarly to stimuli somehow point to those recordings from within "per subject" folders. (just trying to come up with some generic principle)
  • workaround already could be in coming up with "metasubjects" like sub-s1s2/ which would have behavioral recordings of multiple subjects (s1 and s2) and providing further details on which particular subjects in e.g. participants.tsv or alike. could even have ses- subfolders if multiple sessions are expected/collected per each groupping of subjects. It would not be 100% clearly formalized, hence "workaround".

In reality, there could be no difference between a video file presented as a stimulus vs one captured to track behaviour.

That is exactly why I also would prefer to stay away from using suffix for depicting intention/purpose for the file. So far we mostly avoided that in BIDS, as suffixes describe content not intention per se. E.g. someone could potentially use _T1w anatomicals to research "behavior" of participants during anatomical sequences.

Also I dislike "capture" since too generic -- all data is "captured". Even movie videos from Hollywood are "captured" by video cameras.

But I do confirm that we do have potential for a conflict ambiguity ATM! E.g.,how do we organize and name files for a session where subject was presented with a particular to that subject/session movie stimuli [*], and subject's behavior was captured on camera (audio video as well)

Hence -- we have two sub-X/ses-Y/beh/sub-X_ses-Y_..._audiovideo.mp4 files (let's assume it is purely behavioral experiment). I think that is where we are getting into one of the principles of BIDS on filename construction: come up with a minimal (usually a single entity) addition to filename which would tell two files apart. @bendichter suggests _recording- but IMHO similarly to "capture" it is too generic and does not "capture" clearly difference between such files (also we do not get into the domain of "inheritance principle" if we add an entity to only one of them). Given that we do have stimuli/ folder, and with BEP044 we would get stimulus entity (plural stimuli), which is pretty much aligns with "modality" folder which at least in case of original MRI domain describes "intent" (anatomy vs functional, behavioral, ...) we could "(ab)use" _mod- entity here [**]? (related: bids-standard/bids-2-devel#55). Then they would gain _mod-stim and _mod-beh (beh takes as the name of directory we have and stim as the short version of stimulus entity from BEP044; but we might want to think this better through if decide to go this route). WDYT?

[*] hence does not make sense for placing into top level stimuli/ folder
[**] original and the only use of mod- ATM is to contain original suffix in case of disambiguation of different _defacemasks...

@yarikoptic
Copy link
Collaborator

softlink/symlink is filesystem specific solution, thus not encouraged/used anywhere in BIDS.

I have said that but forgot about BIDS URIs, https://bids-specification.readthedocs.io/en/stable/common-principles.html#bids-uri , so those could potentially be used I guess (ATM we have those + ad-hoc pointers like stimuli_relative)

@neuromechanist
Copy link
Member

[*] hence does not make sense for placing into top-level stimuli/ folder

I believe neither this BEP nor BEP044 proposes a solution for this case (stimulus files per subject in the subject/session directories).

I agree that with some use cases that this BEP will accommodate, it is only natural to have an entity to determine the scope of the recording.

An example that comes to my mind is the STRUM task/dataset, in which two subjects collaborate in a first-person game environment with recordings from (among many datastreams) EEG, eye-tracking, and videos of the participants' faces (behavior), screens (stimulus), and eye-gaze (both behavior and stimulus). Sample videos are here (I removed the face camera video because they are not anonymized).

@bendichter
Copy link
Contributor Author

I think I may be a bit confused. My reading of BEP044 is that all stimuli go in a stimuli directory at the root of the dataset, even if they are subject- or session-specific stimuli. If we wanted to modify it, we could allow for a stimuli directory at the subject or session level. Then we wouldn't ever have a naming collision with these captured videos.

@bendichter
Copy link
Contributor Author

@neuromechanist

If I understand correctly, you are talking about differentiating between multiple simultaneous video recordings, right? I proposed this in the initial comment

Video or audio files that are recorded simultaneously but from different angles or at different locations would use the _recording- entity to differentiate

would that handle this or am I missing something about your example?

@neuromechanist
Copy link
Member

AFAIK, the stimulus directory is intended to store stimuli used across the dataset (that is presented to multiple subjects), hence the directory is at the root of the dataset. However, the current spec does not restrict it. For now at least, subject-specific stimuli have not been addressed (or discussed) in BEP044.

Video or audio files that are recorded simultaneously but from different angles or at different locations would use the _recording- entity to differentiate

Yes, but as in the example, and the case @yarikoptic raised, some videos may not have any behavior in them (being stimulus presentation). It might be beneficial to have a way to differentiate them, either with the _recording entity, or _mod. Probably having both also work, as _recording would indicate multi-camera or multi-angle support, while _mod or some other entity would indicate intent.

@yarikoptic
Copy link
Collaborator

FTR, some of the openneuro datasets with videos under per-subj folders, likely with beh recordings
*$> for ds in ds*; do find $ds/sub-* -iname *.avi -o -iname *.mp4 -o -iname *.mkv | head; done
find: ‘ds001107/sub-*’: No such file or directory
find: ‘ds003676/sub-*’: No such file or directory
ds004505/sub-06/video/sub-06_trial-01.mp4
ds004505/sub-06/video/sub-06_trial-02.mp4
ds004505/sub-06/video/sub-06_trial-03.mp4
ds004505/sub-06/video/sub-06_trial-04.mp4
ds004505/sub-06/video/sub-06_trial-05.mp4
ds004505/sub-06/video/sub-06_trial-06.mp4
ds004505/sub-06/video/sub-06_trial-07.mp4
ds004505/sub-06/video/sub-06_trial-08.mp4
ds004505/sub-06/video/sub-06_trial-09.mp4
ds004505/sub-06/video/sub-06_trial-10.mp4
ds004598/sub-01/ses-1/eeg/sub-01_ses-1_task-LinearTrack_video.avi
ds004598/sub-02/ses-1/eeg/sub-02_ses-1_task-LinearTrack_video.avi
ds004598/sub-02/ses-2/eeg/sub-02_ses-2_task-LinearTrack_video.avi
ds004598/sub-02/ses-3/eeg/sub-02_ses-3_task-LinearTrack_video.avi
ds004598/sub-03/ses-1/eeg/sub-03_ses-1_task-LinearTrack_video.avi
ds004598/sub-03/ses-2/eeg/sub-03_ses-2_task-LinearTrack_video.avi
ds004598/sub-04/ses-1/eeg/sub-04_ses-1_task-LinearTrack_video.avi
ds004598/sub-04/ses-2/eeg/sub-04_ses-2_task-LinearTrack_video.avi
ds004598/sub-05/ses-1/eeg/sub-05_ses-1_task-LinearTrack_video.avi
ds004598/sub-05/ses-2/eeg/sub-05_ses-2_task-LinearTrack_video.avi
find: ‘ds004643/sub-*’: No such file or directory
ds005127/sub-00002/ses-1/video/sub-00002_ses-1_task-sleep_run-20160531_2257.avi
ds005127/sub-00002/ses-1/video/sub-00002_ses-1_task-sleep_run-20160531_2330.avi
ds005127/sub-00003/ses-1/video/sub-00003_ses-1_task-sleep_run-20160712_2255.avi
ds005127/sub-00003/ses-1/video/sub-00003_ses-1_task-sleep_run-20160712_2350.avi
ds005127/sub-00003/ses-1/video/sub-00003_ses-1_task-sleep_run-20160713_0013.avi
ds005127/sub-00003/ses-1/video/sub-00003_ses-1_task-sleep_run-20160713_0142.avi
ds005127/sub-00003/ses-1/video/sub-00003_ses-1_task-sleep_run-20160713_0222.avi
ds005127/sub-00003/ses-1/video/sub-00003_ses-1_task-sleep_run-20160713_0310.avi
ds005127/sub-00003/ses-1/video/sub-00003_ses-1_task-sleep_run-20160713_0338.avi
ds005127/sub-00003/ses-1/video/sub-00003_ses-1_task-sleep_run-20160713_0459.avi
find: ‘ds005443/sub-*’: No such file or directory
find: ‘ds005590/sub-*’: No such file or directory

Example(s) in dandi (not yet BIDS):

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BEP opinions wanted Please read and offer your opinion on this matter raw
Projects
None yet
Development

No branches or pull requests

10 participants