Skip to content

Commit

Permalink
update conver_notebooks to produce a json with html outputs organized…
Browse files Browse the repository at this point in the history
… by headers in hierarchical structure
  • Loading branch information
dylansdaniels committed Feb 3, 2025
1 parent d77160b commit 1e70e12
Show file tree
Hide file tree
Showing 9 changed files with 198 additions and 40 deletions.
2 changes: 1 addition & 1 deletion content/05_erps/02_hnn_core.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@
# Christopher Bailey <[email protected]>
-->

[[plot_simulate_evoked.ipynb]]
[[plot_simulate_evoked.ipynb]]
Binary file modified content/05_erps/output_nb_plot_simulate_evoked/fig_03.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified content/05_erps/output_nb_plot_simulate_evoked/fig_04.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
21 changes: 10 additions & 11 deletions content/05_erps/plot_simulate_evoked.html
Original file line number Diff line number Diff line change
Expand Up @@ -152,26 +152,25 @@ <h1>01. Simulate Event Related Potentials (ERPs)</h1>

[Done]
Trial 1: 0.03 ms...
Trial 1: 10.0 ms...

Trial 1: 10.0 ms...
Trial 1: 20.0 ms...
Trial 1: 20.0 ms...
Trial 1: 30.0 ms...
Trial 1: 40.0 ms...
Trial 1: 50.0 ms...
Trial 1: 60.0 ms...
Trial 1: 70.0 ms...
Trial 1: 80.0 ms...
Trial 1: 90.0 ms...

Trial 1: 90.0 ms...
Trial 1: 100.0 ms...
Trial 1: 100.0 ms...
Trial 1: 110.0 ms...
Trial 1: 120.0 ms...
Trial 1: 130.0 ms...
Trial 1: 140.0 ms...
Trial 1: 150.0 ms...
Trial 1: 160.0 ms...

Building the NEURON model
Building the NEURON model

[Done]
Trial 2: 0.03 ms...
Expand All @@ -184,9 +183,9 @@ <h1>01. Simulate Event Related Potentials (ERPs)</h1>
Trial 2: 60.0 ms...
Trial 2: 70.0 ms...
Trial 2: 80.0 ms...
Trial 2: 90.0 ms...

Trial 2: 100.0 ms...
Trial 2: 90.0 ms...
Trial 2: 100.0 ms...
Trial 2: 110.0 ms...
Trial 2: 120.0 ms...
Trial 2: 130.0 ms...
Expand Down Expand Up @@ -328,10 +327,10 @@ <h1>01. Simulate Event Related Potentials (ERPs)</h1>
Trial 1: 60.0 ms...
Trial 1: 70.0 ms...
Trial 1: 80.0 ms...

Trial 1: 90.0 ms...
Trial 1: 90.0 ms...
Trial 1: 100.0 ms...
Trial 1: 110.0 ms...

Trial 1: 110.0 ms...
Trial 1: 120.0 ms...
Trial 1: 130.0 ms...
Trial 1: 140.0 ms...
Expand Down
17 changes: 16 additions & 1 deletion content/05_erps/plot_simulate_evoked.json
Original file line number Diff line number Diff line change
@@ -1 +1,16 @@
{"plot_simulate_evoked.ipynb": {"01. Simulate Event Related Potentials (ERPs)": {"contents": "<h1>01. Simulate Event Related Potentials (ERPs)</h1>", "sections": {"References": {"contents": "<h2>References</h2><p>.. [1] Jones, Stephanie R., et al. \"Neural correlates of tactile detection:\n a combined magnetoencephalography and biophysically based computational\n modeling study.\" Journal of Neuroscience 27.40 (2007): 10751-10764.</p>"}}}}}
{
"plot_simulate_evoked.ipynb": {
"01. Simulate Event Related Potentials (ERPs)": {
"title": "01. Simulate Event Related Potentials (ERPs)",
"level": 1,
"html": "<div class='markdown-cell'>\n<h1>01. Simulate Event Related Potentials (ERPs)</h1>\n</div>\n<div class='markdown-cell'>\n<p>This example demonstrates how to simulate a threshold level tactile\nevoked response, as detailed in the <a href=\"https://jonescompneurolab.github.io/hnn-tutorials/erp/erp\">HNN GUI ERP tutorial</a>,\nusing HNN-core. We recommend you first review the GUI tutorial.</p>\n<p>The workflow below recreates an example of the threshold level tactile\nevoked response, as observed in Jones et al. J. Neuroscience 2007 [1]_\n(e.g. Figure 7 in the GUI tutorial), albeit without a direct comparison\nto the recorded data.</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\n# Authors: Mainak Jas <[email protected]>\n# Sam Neymotin <[email protected]>\n# Blake Caldwell <[email protected]>\n# Christopher Bailey <[email protected]>\n# sphinx_gallery_thumbnail_number = 3\nimport os.path as op\nimport tempfile\nimport matplotlib.pyplot as plt\n</code>\n</div>\n<div class='markdown-cell'>\n<p>Let us import hnn_core</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nimport hnn_core\nfrom hnn_core import simulate_dipole, jones_2009_model\nfrom hnn_core.viz import plot_dipole\n</code>\n</div>\n<div class='markdown-cell'>\n<p>Let us first create our default network and visualize the cells\ninside it.</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nnet = jones_2009_model()\nnet.plot_cells()\nnet.cell_types['L5_pyramidal'].plot_morphology()\n</code>\n</div>\n<div class='output-cell'><div class='output-label'>\nOut:\n</div>\n<div class='output-code'>\n&lt;Figure size 640x480 with 1 Axes&gt;\n</div>\n</div>\n<div class='output-cell'>\n<img src='output_nb_plot_simulate_evoked/fig_01.png'/>\n</div>\n<div class='output-cell'><div class='output-label'>\nOut:\n</div>\n<div class='output-code'>\n&lt;Figure size 640x480 with 1 Axes&gt;\n</div>\n</div>\n<div class='output-cell'>\n<img src='output_nb_plot_simulate_evoked/fig_02.png'/>\n</div>\n<div class='output-cell'><div class='output-label'>\nOut:\n</div>\n<div class='output-code'>\n&lt;Axes3D: &gt;\n</div>\n</div>\n<div class='markdown-cell'>\n<p>The network of cells is now defined, to which we add external drives as\nrequired. Weights are prescribed separately for AMPA and NMDA receptors\n(receptors that are not used can be omitted or set to zero). The possible\ndrive types include the following (click on the links for documentation):</p>\n<ul>\n<li>:meth:<code>hnn_core.Network.add_evoked_drive</code></li>\n<li>:meth:<code>hnn_core.Network.add_poisson_drive</code></li>\n<li>:meth:<code>hnn_core.Network.add_bursty_drive</code></li>\n</ul>\n</div>\n<div class='markdown-cell'>\n<p>First, we add a distal evoked drive</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nweights_ampa_d1 = {'L2_basket': 0.006562, 'L2_pyramidal': .000007,\n'L5_pyramidal': 0.142300}\nweights_nmda_d1 = {'L2_basket': 0.019482, 'L2_pyramidal': 0.004317,\n'L5_pyramidal': 0.080074}\nsynaptic_delays_d1 = {'L2_basket': 0.1, 'L2_pyramidal': 0.1,\n'L5_pyramidal': 0.1}\nnet.add_evoked_drive(\n'evdist1', mu=63.53, sigma=3.85, numspikes=1, weights_ampa=weights_ampa_d1,\nweights_nmda=weights_nmda_d1, location='distal',\nsynaptic_delays=synaptic_delays_d1, event_seed=274)\n</code>\n</div>\n<div class='markdown-cell'>\n<p>Then, we add two proximal drives</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nweights_ampa_p1 = {'L2_basket': 0.08831, 'L2_pyramidal': 0.01525,\n'L5_basket': 0.19934, 'L5_pyramidal': 0.00865}\nsynaptic_delays_prox = {'L2_basket': 0.1, 'L2_pyramidal': 0.1,\n'L5_basket': 1., 'L5_pyramidal': 1.}\n# all NMDA weights are zero; pass None explicitly\nnet.add_evoked_drive(\n'evprox1', mu=26.61, sigma=2.47, numspikes=1, weights_ampa=weights_ampa_p1,\nweights_nmda=None, location='proximal',\nsynaptic_delays=synaptic_delays_prox, event_seed=544)\n# Second proximal evoked drive. NB: only AMPA weights differ from first\nweights_ampa_p2 = {'L2_basket': 0.000003, 'L2_pyramidal': 1.438840,\n'L5_basket': 0.008958, 'L5_pyramidal': 0.684013}\n# all NMDA weights are zero; omit weights_nmda (defaults to None)\nnet.add_evoked_drive(\n'evprox2', mu=137.12, sigma=8.33, numspikes=1,\nweights_ampa=weights_ampa_p2, location='proximal',\nsynaptic_delays=synaptic_delays_prox, event_seed=814)\n</code>\n</div>\n<div class='markdown-cell'>\n<p>Now let&#x27;s simulate the dipole, running 2 trials with the\n<code>hnn_core.parallel_backends.Joblib</code> backend.\nTo run them in parallel we could set <code>n_jobs</code> to equal the number of\ntrials. The <code>Joblib</code> backend allows running the simulations in parallel\nacross trials.</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nfrom hnn_core import JoblibBackend\nwith JoblibBackend(n_jobs=2):\ndpls = simulate_dipole(net, tstop=170., n_trials=2)\n</code>\n</div>\n<div class='output-cell'><div class='output-label'>\nOut:\n</div>\n<div class='output-code'>\nJoblib will run 2 trial(s) in parallel by distributing trials over 2 jobs.\nLoading custom mechanism files from /opt/homebrew/Caskroom/miniconda/base/envs/py311/lib/python3.11/site-packages/hnn_core/mod/arm64/.libs/libnrnmech.so\nBuilding the NEURON model\n[Done]\nTrial 1: 0.03 ms...\nTrial 1: 10.0 ms...\nTrial 1: 20.0 ms...\nTrial 1: 30.0 ms...\nTrial 1: 40.0 ms...\nTrial 1: 50.0 ms...\nTrial 1: 60.0 ms...\nTrial 1: 70.0 ms...\nTrial 1: 80.0 ms...\nTrial 1: 90.0 ms...\nTrial 1: 100.0 ms...\nTrial 1: 110.0 ms...\nTrial 1: 120.0 ms...\nTrial 1: 130.0 ms...\nTrial 1: 140.0 ms...\nTrial 1: 150.0 ms...\nTrial 1: 160.0 ms...\nBuilding the NEURON model\n[Done]\nTrial 2: 0.03 ms...\nTrial 2: 10.0 ms...\nTrial 2: 20.0 ms...\nTrial 2: 30.0 ms...\nTrial 2: 40.0 ms...\nTrial 2: 50.0 ms...\nTrial 2: 60.0 ms...\nTrial 2: 70.0 ms...\nTrial 2: 80.0 ms...\nTrial 2: 90.0 ms...\nTrial 2: 100.0 ms...\nTrial 2: 110.0 ms...\nTrial 2: 120.0 ms...\nTrial 2: 130.0 ms...\nTrial 2: 140.0 ms...\nTrial 2: 150.0 ms...\nTrial 2: 160.0 ms...\n</div>\n</div>\n<div class='markdown-cell'>\n<p>Rather than reading smoothing and scaling parameters from file, we recommend\nexplicit use of the <code>~hnn_core.dipole.Dipole.smooth</code> and\n<code>~hnn_core.dipole.Dipole.scale</code> methods instead. Note that both methods\noperate in-place, i.e., the objects are modified.</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nwindow_len, scaling_factor = 30, 3000\nfor dpl in dpls:\ndpl.smooth(window_len).scale(scaling_factor)\n</code>\n</div>\n<div class='markdown-cell'>\n<p>Plot the amplitudes of the simulated aggregate dipole moments over time</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nimport matplotlib.pyplot as plt\nfig, axes = plt.subplots(2, 1, sharex=True, figsize=(6, 6),\nconstrained_layout=True)\nplot_dipole(dpls, ax=axes[0], layer='agg', show=False)\nnet.cell_response.plot_spikes_hist(ax=axes[1],\nspike_types=['evprox', 'evdist']);\n</code>\n</div>\n<div class='output-cell'><div class='output-label'>\nOut:\n</div>\n<div class='output-code'>\n&lt;Figure size 600x600 with 2 Axes&gt;\n</div>\n</div>\n<div class='output-cell'>\n<img src='output_nb_plot_simulate_evoked/fig_03.png'/>\n</div>\n<div class='markdown-cell'>\n<p>If you want to analyze how the different cortical layers contribute to\ndifferent net waveform features, then instead of passing <code>&amp;#x27;agg&amp;#x27;</code> to\n<code>layer</code>, you can provide a list of layers to be visualized and optionally\na list of axes to <code>ax</code> to visualize the dipole moments separately.</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nplot_dipole(dpls, average=False, layer=['L2', 'L5', 'agg'], show=False);\n</code>\n</div>\n<div class='output-cell'><div class='output-label'>\nOut:\n</div>\n<div class='output-code'>\n&lt;Figure size 640x480 with 3 Axes&gt;\n</div>\n</div>\n<div class='output-cell'>\n<img src='output_nb_plot_simulate_evoked/fig_04.png'/>\n</div>\n<div class='markdown-cell'>\n<p>Now, let us try to make the exogenous driving inputs to the cells\nsynchronous and see what happens. This is achieved by setting\n<code>n_drive_cells=1</code> and <code>cell_specific=False</code> when adding each drive.</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nnet_sync = jones_2009_model()\nn_drive_cells=1\ncell_specific=False\nnet_sync.add_evoked_drive(\n'evdist1', mu=63.53, sigma=3.85, numspikes=1, weights_ampa=weights_ampa_d1,\nweights_nmda=weights_nmda_d1, location='distal', n_drive_cells=n_drive_cells,\ncell_specific=cell_specific, synaptic_delays=synaptic_delays_d1, event_seed=274)\nnet_sync.add_evoked_drive(\n'evprox1', mu=26.61, sigma=2.47, numspikes=1, weights_ampa=weights_ampa_p1,\nweights_nmda=None, location='proximal', n_drive_cells=n_drive_cells,\ncell_specific=cell_specific, synaptic_delays=synaptic_delays_prox, event_seed=544)\nnet_sync.add_evoked_drive(\n'evprox2', mu=137.12, sigma=8.33, numspikes=1,\nweights_ampa=weights_ampa_p2, location='proximal', n_drive_cells=n_drive_cells,\ncell_specific=cell_specific, synaptic_delays=synaptic_delays_prox, event_seed=814)\n</code>\n</div>\n<div class='markdown-cell'>\n<p>You may interrogate current values defining the spike event time dynamics by</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\nprint(net_sync.external_drives['evdist1']['dynamics'])\n</code>\n</div>\n<div class='output-cell'><div class='output-label'>\nOut:\n</div>\n<div class='output-code'>\n{&#x27;mu&#x27;: 63.53, &#x27;sigma&#x27;: 3.85, &#x27;numspikes&#x27;: 1}\n</div>\n</div>\n<div class='markdown-cell'>\n<p>Finally, let&#x27;s simulate this network. Rather than modifying the dipole\nobject, this time we make a copy of it before smoothing and scaling.</p>\n</div>\n<div class='code-cell'>\n<code class='language-python'>\ndpls_sync = simulate_dipole(net_sync, tstop=170., n_trials=1)\ntrial_idx = 0\ndpls_sync[trial_idx].copy().smooth(window_len).scale(scaling_factor).plot()\nnet_sync.cell_response.plot_spikes_hist();\n</code>\n</div>\n<div class='output-cell'><div class='output-label'>\nOut:\n</div>\n<div class='output-code'>\nJoblib will run 1 trial(s) in parallel by distributing trials over 1 jobs.\nBuilding the NEURON model\n[Done]\nTrial 1: 0.03 ms...\nTrial 1: 10.0 ms...\nTrial 1: 20.0 ms...\nTrial 1: 30.0 ms...\nTrial 1: 40.0 ms...\nTrial 1: 50.0 ms...\nTrial 1: 60.0 ms...\nTrial 1: 70.0 ms...\nTrial 1: 80.0 ms...\nTrial 1: 90.0 ms...\nTrial 1: 100.0 ms...\nTrial 1: 110.0 ms...\nTrial 1: 120.0 ms...\nTrial 1: 130.0 ms...\nTrial 1: 140.0 ms...\nTrial 1: 150.0 ms...\nTrial 1: 160.0 ms...\n&lt;Figure size 640x480 with 1 Axes&gt;\n</div>\n</div>\n<div class='output-cell'>\n<img src='output_nb_plot_simulate_evoked/fig_05.png'/>\n</div>\n<div class='output-cell'><div class='output-label'>\nOut:\n</div>\n<div class='output-code'>\n&lt;Figure size 640x480 with 1 Axes&gt;\n</div>\n</div>\n<div class='output-cell'>\n<img src='output_nb_plot_simulate_evoked/fig_06.png'/>\n</div>\n<div class='markdown-cell'>\n<p>&lt;h4&gt;Warning&lt;/h4&gt;</p>\n<ul>\n<li>\n<p>Always look at dipoles in conjunction with raster plots and spike histogram to avoid misinterpretation.</p>\n</li>\n<li>\n<p>Run multiple trials of your simulation to get an average of different drives seeds before drawing conclusions.</p>\n</li>\n</ul>\n</div>",
"sub-sections": [
{
"title": "References",
"level": 2,
"html": "<div class='markdown-cell'>\n<h2>References</h2>\n<p>.. [1] Jones, Stephanie R., et al. &quot;Neural correlates of tactile detection:\na combined magnetoencephalography and biophysically based computational\nmodeling study.&quot; Journal of Neuroscience 27.40 (2007): 10751-10764.</p>\n</div>"
}
]
}
}
}
100 changes: 95 additions & 5 deletions scripts/convert_notebooks.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ def save_plot_as_image(img_data, img_filename, output_dir):
return


def html_to_hierarchical_json(html: str, filename: str):
def html_to_json(html: str, filename: str):
"""
Convert html into hierarchical json
"""
Expand Down Expand Up @@ -59,6 +59,7 @@ def html_to_hierarchical_json(html: str, filename: str):

# get the title, level of the new section
current_level = line_match.group(1).strip()
current_level = int(current_level.lstrip('<h').rstrip('>'))
current_title = line_match.group(2).strip()

# start a new section with the previous line
Expand All @@ -82,6 +83,90 @@ def html_to_hierarchical_json(html: str, filename: str):
return contents


def structure_json(contents):
"""
Determine the hierarchy of sections based on levels without adding content.
Returns a list of sections in order of their hierarchy.
"""
hierarchy = {}

for filename, sections in contents.items():
hierarchy[filename] = {}

# list to track parent sections for potential nesting
parent_stack = []

for section_title, section_data in sections.items():
level = section_data['level']
html_contents = section_data['html']

# Create a section dict with 'title', 'level', and 'sub-sections'
section_info = {
'title': section_title,
'level': level,
'html': html_contents,
'sub-sections': []
}

# Ensure only sections with a level greater than the current
# section remain in the stack as potential parents
while parent_stack and parent_stack[-1]['level'] >= level:
parent_stack.pop()

if parent_stack:
# Add the section as a child of the last parent
parent_stack[-1]['sub-sections'].append(section_info)
else:
# Add the section as a top-level section
hierarchy[filename][section_title] = section_info

# Add the current section to the parent stack for future nesting
parent_stack.append(section_info)

def remove_blank_subsections(sections):
seek = 'sub-sections'

for k, v in list(sections.items()):
print('key:', k)
print('value:', v)
print('-' * 30)

if isinstance(v, dict):
# check for 'sub-sections' key in dict
if seek in v:
print('Subsections present:', (seek in v))
print('Subsection value:', v[seek])

# delete empty sub-sections
if v[seek] == []:
print('Blank section found')
del v[seek]

# Recursively check all sub-sections
else:
print('Checking sub-sections recursively...')
for sub_section in v[seek]:
remove_blank_subsections(sub_section)

print('#' * 30)

elif isinstance(v, list):
# if v is an empty list, delete it
if v == []:
print('Blank section found in list')
del sections[k]
# if v is a list of dicts, iterate through the dicts
else:
for dictionary in v:
remove_blank_subsections(dictionary)

return sections

hierarchy[filename] = remove_blank_subsections(hierarchy[filename])

return hierarchy


def extract_html_from_notebook(
notebook,
input_dir, # changed
Expand Down Expand Up @@ -258,17 +343,23 @@ def convert_notebooks_to_html(
f.write(html_content)
f.write("\n</body></html>")

nb_html_json = html_to_hierarchical_json(
# flat json
nb_html_json = html_to_json(
html_content,
filename,
)

# nested json
nb_html_json = structure_json(
nb_html_json
)

output_json = os.path.join(
input_folder, f"{os.path.splitext(filename)[0]}.json"
)

with open(output_json, "w") as f:
json.dump(nb_html_json, f)
json.dump(nb_html_json, f, indent=4)

print(f"Successfully converted '{filename}'")

Expand All @@ -277,6 +368,7 @@ def convert_notebooks_to_html(
def test_nb_conversion():

input_folder = "../tests"
input_folder = "../content/05_erps"

convert_notebooks_to_html(
input_folder,
Expand All @@ -286,5 +378,3 @@ def test_nb_conversion():


test_nb_conversion()

# %%
Loading

0 comments on commit 1e70e12

Please sign in to comment.