bnd : BeNeuroLab Data Organization

A lightweight collection of functions for managing the experimental data recorded in the BeNeuro Lab, and a CLI tool called bnd for easy access to this functionality.

Play around with it and raise Github issues if anything fails

Setting up

1. Install conda

   • You can use either Miniconda or Miniforge

2. Clone repo

   git clone [email protected]:BeNeuroLab/bnd.git
   cd ./beneuro_experimental_data_organization

3. Open either Miniconda prompt or Miniforge promt and run the following command. This may take some time:

   conda env create --file=env.yml

   or if you want the processing depedencies:

   conda env create --file=processing_env.yml

   If you installed the base environment and want to update later on:

   conda env --file=processing_env.yml

4. Create your configuration file:

   bnd init  # Provide the path to local and remote data storage
   bnd --help # Start reading about the functions!
   

Example usage

Complete your experimental session on animal M099. Then:

bnd up M099

Now, you want to process your data into a pyaldata format. Its a good idea to do this on one of the lab workstations:

bnd dl M099_2025_01_01_10_00 -v  # Downloads everything
bnd to-pyal M099_2025_01_01_10_00  # Run kilosort, nwb conversion, and pyaldata conversion
bnd up M099_2025_01_01_10_00  # Uploads new files to server

If you want specific things during your pipeline (e.g., dont run kilosort, use a custom channel map) read the API below.

API

Config

bnd init

Create a .env file (if there isnt one) to store the paths to the local and remote data storage.

bnd show-config

Show the contents of the config file.

Updating

bnd check-updates

Check if there are any new commits on the repo's main branch.

bnd self-update

Update the bnd tool by pulling the latest commits from the repo's main branch.

Data Transfer

bnd up <session_or_animal_name>

Upload data from session or animal name to the server. If the file exists on the server, it won't be replaced. Every file in the session folder will get uploaded.

Example usage to upload everything of a given session:

bnd up M017_2024_03_12_18_45
bnd up M017

bnd dl <session>

Download experimental data from a given session from the remote server.

Example usage to download everything:

bnd dl M017_2024_03_12_18_45 -v  # will download everything, including videos
bnd dl M017_2024_03_12_18_45  # will download everything, except videos
bnd dl M017_2024_03_12_18_45 --max-size=50  # will download files smaller than 50MB

Pipeline

bnd to-pyal <session>

Convert session data into a pyaldata dataframe and saves it as a .mat

If no .nwb file is present it will automatically generate one and if a nwb file is present it will skip it. If you want to generate a new one run bnd to-nwb

If no kilosorted data is available it will not kilosort by default. If you want to kilosort add the flag -k

Example usage:

bnd to-pyal M037_2024_01_01_10_00  # Kilosorts data, runs nwb and converts to pyaldata
bnd to-pyal M037_2024_01_01_10_00 -K  # converts to pyaldata without kilosorting (if no .nwb file is present)
bnd to-pyal M037_2024_01_01_10_00 -c  # Use custom mapping during nwb conversion if custom_map.json is available (see template in repo). -C uses available default mapping

bnd to-nwb <session>

Convert session data into a nwb file and saves it as a .nwb

If no kilosorted data is available it will not kilosort by default. If you want to kilosort add the flag -k

Example usage:

bnd to-nwb M037_2024_01_01_10_00  # Kilosorts data and run nwb
bnd to-nwb M037_2024_01_01_10_00 -K  # converts to nwb without kilosorting (if no .nwb file is present)
bnd to-nwb M037_2024_01_01_10_00 -c  # Use custom mapping during conversion if custom_map.json is available (see template in repo). Option `-C` uses available default mapping

bnd ksort <session>

Kilosorts data from a single session on all available probes and recordings

Example usage:

bnd ksort M037_2024_01_01_10_00

TODOs:

• Add AniposeInterface in nwb conversion
• Implement Npx2.0 functionality