intracellular introduced, and inputs.toml

- create developer's guide
  - start using `#!` for comments to stay
  - this is only info in guide for now
- broader use of `haskey`
- `InputFolder` class now includes: `location`, `basename`, `required`, `varied`, and `path_from_inputs`
  - Inner constructors help initialize from just `location` + (`id` OR `folder` OR both)
- `InputFolders` now a wrapper for a `NamedTuple` where the keys are locations (stored in `project_locations.all`) and the values are of type `InputFolder`
  - users are expected to use either the keyword arg constructor (`InputFolders(; kwargs...)`) or the parse-time defined `InputFolders(<required_locations...>; <optional_locations...>)` where the `inputs.toml` determines the parameter/keyword arg names
  - users could create a vector of pairs, but this is expected to be used internally
  - `getindex` is implemented to allow for `input_folders[:config]` to access the values of the `NamedTuple`
- `VariationID` replaces `VariationIDs` and is now just a wrapper for a `NamedTuple`
  - handles construction of default variation ids based on the folder ID being -1 or not
  - `getindex` implemented to allow for `variation_id[:config]` to access the id for the config variation
- better checks on instantiating `Simulation`s
- make use of `project_locations` in just so many helpful ways
- functions to simplify adding output types to vct.db
- one function each for setting table names (`"$(loc)s"`), loc ids (`"$(loc)_id"`) and loc var ids (`"$(loc)_variation_id"`)
- support for roadrunner sims!
  - will download roadrunner if needed and then go ahead and run the sims
  - checks for environment variables needed, sets in the julia runtime if need be, and helps the user to fix this
  - printls helpful message if not present
- `locationPath` to standardize how to access `inputs`
- starting to support components that containt, well, components of simulation inputs
  - these will live in `data/components` and can be used outside of `run` to set up different input/location folders
  - added `VCTComponents.jl` (though it could be renamed to `VCTIntracellularComponents.jl`) to manage combining of components together
- standardize loading of input folders using `createXMLFile`
- `inputs.toml` now allows users to decide on the structure of their project at runtime
- rewrite of VCTDatabase using inputs.toml
- `initializeVCT` replaced by `initializeModelManager` and can accept no arguments to initialize from the current directory
- `make clean` only in the `temp_physicell_dir`
- compilation `cmd` gets the `ENV` passed in (so the librr var can be included) and also the `temp_physicell_dir` to make sure everything is in that location
- print entire compile `cmd` rather than just the `cflags` (though first get rid of the env and dir info because those make it hard to read)
- on compilation failure, print the log and err to console
- `prepareInputFolder` function now gives a generic interface for initializing each input
- `cmd` for running the sim now has `env` and `dir` set to ensure necessary flags are passed in and that it is being run from teh PHysiCell directory
- `--chdir=$(physicell_dir)` now set for `sbatch` command to make sure it is being run from that directory. still need to see if the env is being properly set there or if that requires extra attention
- bundle results of `addVariations` into an `AddVariationsResult` type so those only output one thing
- `sqliteDataType(ev::ElementaryVariation)` to standardize how all database types are determined
- VCTComponents doc

.github/workflows/CI.yml

@@ -46,13 +46,13 @@ jobs:
             arch: "arm64"
           # - os: "windows-latest"
           #   julia-version: '1'
-          #   shell: "pwsh"
+          #   shell: "msys2"
           #   compiler: "g++"
           #   arch: "x64"
 
     defaults:
       run:
-        shell: ${{ matrix.shell }}
+        shell: ${{ matrix.shell }} {0}
 
     steps:
     - name: Checkout repository
@@ -65,28 +65,53 @@ jobs:
         brew install ffmpeg
       
     - name: Install MSYS2 on Windows
-      if: matrix.os.name == 'windows-latest'
+      if: matrix.os == 'windows-latest'
       uses: msys2/setup-msys2@v2
       with:
         update: true
         install: mingw-w64-x86_64-binutils mingw-w64-x86_64-gcc mingw-w64-x86_64-headers-git mingw-w64-x86_64-gcc-libs mingw-w64-x86_64-libwinpthread-git mingw-w64-x86_64-lapack mingw-w64-x86_64-openblas mingw-w64-x86_64-libxml2 mingw-w64-x86_64-bzip2 mingw-w64-x86_64-python mingw-w64-x86_64-python-zstandard mingw-w64-x86_64-python-cffi make bison flex mingw-w64-x86_64-ca-certificates mingw-w64-x86_64-diffutils
+
     - uses: julia-actions/setup-julia@v2
       with:
         version: ${{ matrix.julia-version }}
         arch: ${{ matrix.arch }}
+
+    - name: Add Julia to PATH on Windows
+      if: matrix.os == 'windows-latest'
+      run: echo "C:\\hostedtoolcache\\windows\\julia\\${{ matrix.julia-version }}\\${{ matrix.arch }}\\bin" >> $GITHUB_PATH
+
     - name: Set environment variable PHYSICELL_CPP
       run: echo "PHYSICELL_CPP=${{ matrix.compiler }}" >> $GITHUB_ENV
+
     - uses: julia-actions/cache@v2
-    - name: Add PCVCTRegistry
+
+    - name: Add PCVCTRegistry (Windows)
+      if: matrix.os == 'windows-latest'
+      shell: pwsh
       run: julia -e 'import Pkg; Pkg.Registry.add("General"); Pkg.Registry.add(Pkg.RegistrySpec(url="https://github.com/drbergman/PCVCTRegistry.git"))'
+
+    - name: Add PCVCTRegistry (non-Windows)
+      if: matrix.os != 'windows-latest'
+      run: julia -e 'import Pkg; Pkg.Registry.add("General"); Pkg.Registry.add(Pkg.RegistrySpec(url="https://github.com/drbergman/PCVCTRegistry.git"))'
+
     - uses: julia-actions/julia-buildpkg@v1
+
+    - name: Install libRoadRunner dependencies on Ubuntu
+      if: matrix.os == 'ubuntu-latest'
+      run: |
+        sudo apt-get update
+        wget http://security.ubuntu.com/ubuntu/pool/universe/n/ncurses/libtinfo5_6.3-2ubuntu0.1_amd64.deb
+        sudo apt-get install -y ./libtinfo5_6.3-2ubuntu0.1_amd64.deb
+
     - name: Run all tests
       uses: julia-actions/julia-runtest@v1
       env:
         PCVCT_NUM_PARALLEL_SIMS: 8
         PHYSICELL_CPP: ${{ matrix.compiler }} # maybe necessary for windows??
         PCVCT_PUBLIC_REPO_AUTH: ${{ secrets.PUBLIC_REPO_AUTH }}
+
     - uses: julia-actions/julia-processcoverage@v1
+
     - uses: codecov/codecov-action@v5
       with:
         files: lcov.info
@@ -102,23 +127,30 @@ jobs:
       statuses: write
     steps:
       - uses: actions/checkout@v4
+
       - uses: julia-actions/setup-julia@v2
         with:
           version: '1'
+
       - uses: julia-actions/cache@v2
+
       - name: Add PCVCTRegistry
         run: julia -e 'import Pkg; Pkg.Registry.add("General"); Pkg.Registry.add(Pkg.RegistrySpec(url="https://github.com/drbergman/PCVCTRegistry.git"))'
+
       - name: Configure doc environment
         shell: julia --project=docs --color=yes {0}
         run: |
           using Pkg
           Pkg.develop(PackageSpec(path=pwd()))
           Pkg.instantiate()
+          
       - uses: julia-actions/julia-buildpkg@v1
+
       - uses: julia-actions/julia-docdeploy@v1
         env:
           GITHUB_TOKEN: ${{ secrets.GH_TOKEN }}
           DOCUMENTER_KEY: ${{ secrets.DOCUMENTER_KEY }}
+
       - name: Run doctests
         shell: julia --project=docs --color=yes {0}
         run: |

Project.toml

@@ -1,9 +1,10 @@
 name = "pcvct"
 uuid = "3c374bc7-7384-4f83-8ca0-87b8c727e6ff"
 authors = ["Daniel Bergman <[email protected]> and contributors"]
-version = "0.0.15"
+version = "0.0.16"
 
 [deps]
+AutoHashEquals = "15f4f7f2-30c1-5605-9d31-71845cf9641f"
 CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
 DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
 Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
@@ -28,9 +29,11 @@ RecipesBase = "3cdcf5f2-1ef4-517c-9805-6587b60abb01"
 SQLite = "0aa819cd-b072-5ff4-a722-6bc24af294d9"
 Sobol = "ed01d8cd-4d21-5b2a-85b4-cc3bdc58bad4"
 Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"
+TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
 Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"
 
 [compat]
+AutoHashEquals = "2.2.0"
 CSV = "0.10"
 DataFrames = "1"
 Distributions = "0.25"
@@ -53,6 +56,7 @@ RecipesBase = "1.3.4"
 SQLite = "1"
 Sobol = "1"
 Statistics = "1"
+TOML = "1.0.3"
 Tables = "1"
 julia = "1.6.7"
 

docs/make.jl

@@ -18,9 +18,11 @@ makedocs(;
             "Getting started" => "man/getting_started.md",
             "CoVariations" => "man/covariations.md",
             "Data directory" => "man/data_directory.md",
+            "Intracellular inputs" => "man/intracellular_inputs.md",
             "Known limitations" => "man/known_limitations.md",
             "PhysiCell Studio" => "man/physicell_studio.md",
             "Sensitivity analysis" => "man/sensitivity_analysis.md",
+            "Developer guide" => "man/developer_guide.md",
         ],
         "Documentation" => map(
             s -> "lib/$(s)",

docs/src/lib/VCTComponents.md

@@ -0,0 +1,14 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTComponents
+
+Allows for combining PhysiCell input components into whole inputs.
+
+Currently, only supports this for intracellular ODE (libRoadRunner) models.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTComponents.jl"]
+```

docs/src/lib/VCTConfiguration.md

@@ -6,7 +6,7 @@ CollapsedDocStrings = true
 
 Interface with the configuration file necessary for PhysiCell simulations.
 
-Provide functionality for accessing and modifying elements in any XML, including the PhysiCell configuration file, XML rules file, and XML IC cell file.
+Provide functionality for accessing and modifying elements in any XML, including the PhysiCell configuration file, XML rules file, combined intracellular XML file, XML IC cell file, and XML IC ECM file.
 
 ```@autodocs
 Modules = [pcvct]

docs/src/man/data_directory.md

@@ -13,6 +13,7 @@ project-dir/
 │       │   ├── dcs/
 │       │   ├── ecms/
 │       │   └── substrates/
+│       ├── intracellulars/
 │       ├── rulesets_collections/
 ...
 ```
@@ -31,6 +32,23 @@ Add within `data/inputs/custom_codes/default/` the following, each exactly as is
 - `Makefile`
 - `custom_modules/`
 
+## Rulesets collections
+
+Add a single file within `data/inputs/rulesets_collections/default/` called `base_rulesets.csv` with the base ruleset collection for your PhysiCell project.
+If your project does not use rules, you can skip this step.
+
+You may also place an XML file here. Use [PhysiCellXMLRules.jl](https://github.com/drbergman/PhysiCellXMLRules.jl) to create one from a standard CSV version of the rules.
+
+**Important**: In either case, the variations you define *must* be on the XML version.
+After calling `initializeModelManager()`, any folder with `base_rulesets.csv` will now be populated with a `base_rulesets.xml` file that can be reference to set the XML paths.
+
+## Intracellulars
+
+Add a single XML file within `data/inputs/intracellulars/default/` called `intracellular.xml` in which the root has two child elements: `cell_definitions` and `intracellulars`.
+This currently only supports libRoadRunner, i.e., ODEs.
+See the `sample_projects_intracellular/combined/template-combined` for an example.
+See [Intracellular inputs](@ref) for much more information.
+
 ## ICs
 
 These folders are optional as not every model includes initial conditions as separate files.
@@ -47,7 +65,14 @@ cells/
 
 Proceed similarly for `dcs/`, `ecms/`, and `substrates/`, renaming those files to `dcs.csv`, `ecm.csv`, and `substrates.csv`, respectively.
 
-## Rulesets collections
+### IC cells
 
-Add a single file within `data/inputs/rulesets_collections/default/` called `base_rulesets.csv` with the base ruleset collection for your PhysiCell project.
-If your project does not use rules, you can skip this step.
+pcvct uses [PhysiCellCellCreator.jl](https://github.com/drbergman/PhysiCellCellCreator.jl) to allow for creation of `cells.csv` files based on geometries defined in a `cells.xml` file.
+To use this, first create such an XML document (see [PhysiCellCellCreator.jl](https://github.com/drbergman/PhysiCellCellCreator.jl) for details) and place this in place of the `cells.csv` file.
+You may make variations on this in the same was as for `config` and `rulesets_collection`.
+
+### IC ecm
+
+pcvct uses [PhysiCellECMCreator.jl](https://github.com/drbergman/PhysiCellECMCreator.jl) to allow for creation of `ecm.csv` files based on the structure defined in a `ecm.xml` file.
+To use this, first create such an XML document (see [PhysiCellECMCreator.jl](https://github.com/drbergman/PhysiCellECMCreator.jl) for details) and place this in place of the `ecm.csv` file.
+You may make variations on this in the same was as for `config` and `rulesets_collection`.

docs/src/man/developer_guide.md

@@ -0,0 +1,6 @@
+# Developer guide
+
+## Style guide
+- Use `#!` for comments that are informative
+  - This helps find code lines commented out in development.
+  - Using the regexp `^(\s+)?# .+\n` seems to work well for finding commented out code lines.

docs/src/man/intracellular_inputs.md

@@ -0,0 +1,31 @@
+# Intracellular inputs
+
+pcvct currently only supports ODE intracellular models using libRoadRunner.
+It uses a specialized format to achieve this, creating the SBML files needed by libRoadRunner at PhysiCell runtime.
+Briefly, the `intracellular.xml` file defines a mapping between cell definitions and intracellular models.
+See the template provided [here](https://github.com/drbergman/PhysiCell/blob/my-physicell/sample_projects_intracellular/combined/template-combined/config/sample_combined_sbmls.xml).
+
+To facilitate creation of such files, and to make it easy to mix-and-match intracellular models, users can place the SBML files that define the ODEs into `data/components/roadrunner` and then simply reference those to construct the specialized XMLs needed.
+For example, place the `Toy_Metabolic_Model.xml` from [sample_projects_intracellular/ode/ode_energy/config/](https://github.com/drbergman/PhysiCell/blob/my-physicell/sample_projects_intracellular/ode/ode_energy/config) into `data/components/roadrunner` and assemble the XML as follows
+
+```julia
+cell_type = "default" # name of the cell type using this intracellular model
+component = PhysiCellComponent("roadrunner", "Toy_Metabolic_Model.xml") # pass in the type of the component and the name of the file to use
+cell_type_to_component = Dict{String, PhysiCellComponent}(cell_type => component) # add other entries to this Dict for other cell types using an intracellular model
+intracellular_folder = assembleIntracellular!(cell_type_to_component; name="toy_metabolic") # will return "toy_metabolic" or "toy_metabolic_n"
+```
+
+This creates a folder at `data/inputs/intracellulars/` with the name stored in `intracellular_folder`.
+Also, the `!` in `assembleIntracellular!` references how the components in the `cell_type_to_component` `Dict` are updated to match those in `data/inputs/intracellulars/$(intracellular_folder)/intracellular.xml`.
+Use these IDs to make variations on the components by using
+
+```julia
+xml_path = ["intracellulars", "intracellular:ID:$(component.id)", ...]
+```
+
+where the `...` is the path starting with the root of the XML file (`sbml` for SBML files).
+
+Finally, pass this folder into `InputFolders` to use this input in simulation runs:
+```julia
+inputs = InputFolders(...; ..., intracellular=intracellular_folder, ...)
+```

docs/src/man/known_limitations.md

@@ -7,4 +7,8 @@ If you do need an upper bound on the number of simulations in such a grouping, s
 It is assumed that most, if not all use cases, will benefit from more simulations.
 
 ## Initial conditions not loaded when launching PhysiCell Studio for a simulation.
-When launching PhysiCell Studio from pcvct, the initial conditions (cells and substrates) are not loaded.
+When launching PhysiCell Studio from pcvct, the initial conditions (cells and substrates) are not loaded.
+
+## Limited intracellular models
+Currently only supports ODE intracellular models (using libRoadRunner).
+Does not support MaBoSS or dFBA.

src/VCTAnalysis/motility.jl

@@ -34,14 +34,14 @@ function meanSpeed(p; direction=:any)::NTuple{3,Dict{String,Float64}}
     distance_dict = Dict{String, Float64}(zip(cell_type_names, zeros(Float64, length(cell_type_names))))
     time_dict = Dict{String, Float64}(zip(cell_type_names, zeros(Float64, length(cell_type_names))))
     while start_ind <= length(type_change) 
-        I = findfirst(type_change[start_ind:end]) # from s to I, cell_type_name is constant. at I+1 it changes
-        I = isnothing(I) ? length(type_change)+2-start_ind : I # if the cell_type_name is constant till the end, set I to be at the end
-        # If start_ind = 1 (at start of sim) and I = 2 (so cell_type_name[3] != cell_type_name[2], meaning that for steps [1,2] cell_type_name is constnat), only use dx in stepping from 1->2 since somewhere in 2->3 the type changes. That is, use dx[1]
-        distance_dict[cell_type_name[start_ind]] += sum(dist_fn.(dx[start_ind:I-1], dy[start_ind:I-1], dz[start_ind:I-1]))  # only count distance travelled while remaining in the initial cell_type_name
-        time_dict[cell_type_name[start_ind]] += p.time[start_ind+I-1] - p.time[start_ind] # record time spent in this cell_type_name (note p.time is not diffs like dx and dy are, hence the difference in indices)
-        start_ind += I # advance the start to the first instance of a new cell_type_name
+        I = findfirst(type_change[start_ind:end]) #! from s to I, cell_type_name is constant. at I+1 it changes
+        I = isnothing(I) ? length(type_change)+2-start_ind : I #! if the cell_type_name is constant till the end, set I to be at the end
+        #! If start_ind = 1 (at start of sim) and I = 2 (so cell_type_name[3] != cell_type_name[2], meaning that for steps [1,2] cell_type_name is constnat), only use dx in stepping from 1->2 since somewhere in 2->3 the type changes. That is, use dx[1]
+        distance_dict[cell_type_name[start_ind]] += sum(dist_fn.(dx[start_ind:I-1], dy[start_ind:I-1], dz[start_ind:I-1]))  #! only count distance travelled while remaining in the initial cell_type_name
+        time_dict[cell_type_name[start_ind]] += p.time[start_ind+I-1] - p.time[start_ind] #! record time spent in this cell_type_name (note p.time is not diffs like dx and dy are, hence the difference in indices)
+        start_ind += I #! advance the start to the first instance of a new cell_type_name
     end
-    speed_dict = [k => distance_dict[k] / time_dict[k] for k in cell_type_names] |> Dict{String,Float64} # convert to speed
+    speed_dict = [k => distance_dict[k] / time_dict[k] for k in cell_type_names] |> Dict{String,Float64} #! convert to speed
     return speed_dict, distance_dict, time_dict
 end
 

src/VCTAnalysis/population.jl

@@ -163,7 +163,7 @@ function MonadPopulationTimeSeries(monad::Monad; include_dead::Bool=false)
             @assert time == spts.time "Simulations $(simulation_ids[1]) and $(simulation_id) in monad $(monad.id) have different times in their time series."
         end
         for (name, cell_count) in pairs(spts.cell_count)
-            if !(name in keys(cell_count_arrays))
+            if !haskey(cell_count_arrays, name)
                 cell_count_arrays[name] = zeros(Int, length(time), monad_length)
             end
             cell_count_arrays[name][:,i] = cell_count
@@ -195,18 +195,18 @@ function populationTimeSeries(M::AbstractMonad; include_dead::Bool=false)
     end
 end
 
-# plot recipes
+#! plot recipes
 getMeanCounts(s::SimulationPopulationTimeSeries) = s.cell_count
 getMeanCounts(m::MonadPopulationTimeSeries) = m.cell_count_means
 
 @recipe function f(M::AbstractMonad; include_dead=false, include_cell_types=:all, exclude_cell_types=String[])
     pts = populationTimeSeries(M; include_dead=include_dead)
-    # allow for single string input for either of these
+    #! allow for single string input for either of these
     include_cell_types = include_cell_types == :all ? :all : (include_cell_types isa String ? [include_cell_types] : include_cell_types)
     exclude_cell_types = exclude_cell_types isa String ? [exclude_cell_types] : exclude_cell_types
     for (name, counts) in pairs(getMeanCounts(pts))
-        skip = include_cell_types != :all && !(name in include_cell_types) # skip this cell type as only a subset was requested and this was not in it
-        skip = skip || name in exclude_cell_types # skip this cell type as it was requested to be excluded
+        skip = include_cell_types != :all && !(name in include_cell_types) #! skip this cell type as only a subset was requested and this was not in it
+        skip = skip || name in exclude_cell_types #! skip this cell type as it was requested to be excluded
         if skip
             continue 
         end
@@ -232,15 +232,15 @@ end
         sim_id = monad.simulation_ids[1]
         row_ind = findfirst(df[!, :SimID] .== sim_id)
         row = df[row_ind, :]
-        title_tuple = [row[name] for name in names(row) if !(name in ["SimID", "ConfigVarID", "RulesVarID"])]
+        title_tuple = [row[name] for name in names(row) if !(name in ["SimID"; shortLocationVariationID.(String, project_locations.varied)])]
         push!(title_tuples, title_tuple)
     end
-    
+
     order = sortperm(title_tuples)
     title_tuples = title_tuples[order]
     monads = monads[order]
 
-    layout  --> (length(monads), 1) # easy room for improvement here
+    layout  --> (length(monads), 1) #! easy room for improvement here
 
     for (i, (monad, title_tuple)) in enumerate(zip(monads, title_tuples))
         @series begin
@@ -306,7 +306,7 @@ end
             end
             if cell_types == :all
                 for (name, cell_count) in pairs(spts.cell_count)
-                    if !(name in keys(cell_count_arrays))
+                    if !haskey(cell_count_arrays, name)
                         cell_count_arrays[name] = zeros(Int, length(time), monad_length)
                     end
                     cell_count_arrays[name][:,i] = cell_count
@@ -319,7 +319,7 @@ end
                     if cell_type isa String
                         cell_type = [cell_type]
                     end
-                    if !(cell_type in keys(cell_count_arrays))
+                    if !haskey(cell_count_arrays, cell_type)
                         cell_count_arrays[cell_type] = zeros(Int, length(time), monad_length)
                     end
                     cell_count_arrays[cell_type][:,i] = sum([spts.cell_count[ct] for ct in cell_type])
@@ -336,7 +336,7 @@ end
         monad_summary[monad.id] = (time=time, cell_count_means=cell_count_means, cell_count_stds=cell_count_stds)
     end
 
-    layout  --> (length(all_cell_types), 1) # easy room for improvement here
+    layout  --> (length(all_cell_types), 1) #! easy room for improvement here
 
     for (i, cell_type) in enumerate(all_cell_types)
         @series begin