Merge pull request #89 from drbergman/development

0.0.15

.gitignore

@@ -3,6 +3,7 @@ test.jl
 /data
 *.db
 .vscode
+studio_debug.log
 
 # from PkgTemplate.jl
 *.jl.*.cov

Project.toml

@@ -1,7 +1,7 @@
 name = "pcvct"
 uuid = "3c374bc7-7384-4f83-8ca0-87b8c727e6ff"
 authors = ["Daniel Bergman <[email protected]> and contributors"]
-version = "0.0.14"
+version = "0.0.15"
 
 [deps]
 CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
@@ -18,6 +18,8 @@ LightXML = "9c8b4983-aa76-5018-a973-4c85ecc9e179"
 LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
 MAT = "23992714-dd62-5051-b70f-ba57cb901cac"
 Parameters = "d96e819e-fc66-5662-9728-84c9c7592b0a"
+PhysiCellCellCreator = "eac91067-0c6e-4a29-b848-3b4e7cf67017"
+PhysiCellECMCreator = "515ed340-d2f8-4e13-b70e-c9ebab3e992f"
 PhysiCellXMLRules = "bad69253-e2d7-49f9-a0cf-2fd158c7c32b"
 Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
 QuasiMonteCarlo = "8a4e6c94-4038-4cdc-81c3-7e6ffdb2a71b"
@@ -42,7 +44,9 @@ LightXML = "0.9"
 LinearAlgebra = "1.10"
 MAT = "0.10"
 Parameters = "0.12"
-PhysiCellXMLRules = "0.0.3"
+PhysiCellCellCreator = "0.0.3"
+PhysiCellECMCreator = "0.0.3"
+PhysiCellXMLRules = "0.0.4"
 Pkg = "1.10.0"
 QuasiMonteCarlo = "0.3"
 RecipesBase = "1.3.4"

README.md

@@ -5,153 +5,5 @@
 [![Build Status](https://github.com/drbergman/pcvct/actions/workflows/CI.yml/badge.svg?branch=main)](https://github.com/drbergman/pcvct/actions/workflows/CI.yml?query=branch%3Amain)
 [![Coverage](https://codecov.io/gh/drbergman/pcvct/branch/main/graph/badge.svg)](https://codecov.io/gh/drbergman/pcvct)
 
-Check out [Getting started](#getting-started) for a quick guide to using pcvct.
-Make sure you are familiar with the [Best practices](#best-practices) section before using pcvct.
-
-# Getting started
-1. Download julia (see [here](https://julialang.org/downloads/) for more options):
-```
-curl -fsSL https://install.julialang.org | sh
-```
-Note: this command also installs the [JuliaUp](https://github.com/JuliaLang/juliaup) installation manager, which will automatically install julia and help keep it up to date.
-
-2. Add the PCVCTRegistry:
-Launch julia by running `julia` in a shell.
-Then, enter the Pkg REPL by pressing `]`.
-Finally, add the PCVCTRegistry by running:
-```
-registry add https://github.com/drbergman/PCVCTRegistry
-```
-
-3. Install pcvct:
-Still in the Pkg REPL, run:
-```
-add pcvct
-```
-
-4. Create a pcvct-enabled project:
-Leave the Pkg REPL by pressing `backspace`.
-Load the pcvct module by running:
-```
-using pcvct
-```
-Then, create a new project by running:
-```
-createProject(path_to_project_folder) # createProject() will use the current directory as the project folder
-```
-This creates three folders inside the `path_to_project_folder` folder: `data/`, `PhysiCell/`, and `VCT/`.
-<!-- The `data/` folder will currently contain the PhysiCell template project split across multiple folders within `data/inputs/`.
-After initializing the project, a `data/vct.db` database will be created to store metadata about the project and its simulations. -->
-
-## Importing from `user_projects`
-If you have a project in the `PhysiCell/user_projects/` folder that you would like to import, you can do so by running:
-```
-importProject(path_to_project_folder)
-```
-Note: This function assumes your project files are in the standard `PhysiCell/user_projects/` format. In particular, it assumes that...
-- The config file is `PhysiCell/user_projects/[project_name]/config/PhysiCell_settings.xml`
-- The `main.cpp` file is `PhysiCell/user_projects/[project_name]/main.cpp`
-- The `Makefile` is `PhysiCell/user_projects/[project_name]/Makefile`
-- The custom modules are in `PhysiCell/user_projects/[project_name]/custom_modules/`
-- (if used) The rules file is `PhysiCell/user_projects/[project_name]/config/cell_rules.csv`
-- (if used) The cell initial conditions are in `PhysiCell/user_projects/[project_name]/config/cells.csv` 
-- (if used) the substrate initial conditions are in `PhysiCell/user_projects/[project_name]/config/substrates.csv`
-If any of these files are not located in the standard location, you can define a dictionary with keys from {`config`, ` main`, `makefile`, `custom_modules`, `rules`, `ic_cell`, `ic_substrate`} to specify the location of each file.
-Put the path relative to the project folder. For example, if the config file is instead located at `PhysiCell/user_projects/[project_name]/config/config.xml`, you would run:
-```
-src = Dict("config" => "config/config.xml")
-```
-Additional entries can be added in a comma-separated list into `Dict` or added later with `src[key] = rel_path`.
-
-## Running first trial
-The `createProject()` command creates three folder, including a `VCT` folder with a single file: `VCT/GenerateData.jl`.
-The name of this folder and this file are purely convention, change them as you like.
-To run your first pcvct trial, you can run the GenerateData.jl script from the shell:
-```
-julia VCT/GenerateData.jl
-```
-Note: if you want to parallelize these 9 runs, you can set the shell environment variable `PCVCT_NUM_PARALLEL_SIMS` to the number of parallel simulations you want to run. For example, to run 9 parallel simulations, you would run:
-```
-export PCVCT_NUM_PARALLEL_SIMS=9
-julia VCT/GenerateData.jl
-```
-Or for a one-off solution:
-```
-PCVCT_NUM_PARALLEL_SIMS=9 julia VCT/GenerateData.jl
-```
-Alternatively, you can run the script via the REPL.
-
-Run the script a second time and observe that no new simulations are run.
-This is because pcvct looks for matching simulations first before running new ones.
-The `use_previous` optional keyword argument can control this behavior if new simulations are desired.
-
-## Using PhysiCell Studio
-If you want to use PhysiCell Studio to visualize the output of your simulations, first launch julia.
-Then, make sure the project is initialized by running:
-```
-# if you used createProject(), these are the first two lines of GenerateData.jl
-using pcvct
-initializeVCT(path_to_physicell, path_to_data)
-```
-Finally, run the following command to launch PhysiCell Studio for a simulation with id `sim_id`:
-```
-runStudio(sim_id; python_path=path_to_python, studio_path=path_to_studio)
-```
-where `path_to_python` is the path to the python executable and `path_to_studio` is the path to the PhysiCell Studio __folder__.
-For example,
-```
-runStudio(1; python_path="/usr/bin/python3", studio_path="/home/user/PhysiCell-Studio")
-```
-Note: if using a python executable that is on your PATH, you can supply just the name of it, e.g. `path_to_python="python3"`.
-
-The shell environment variables `PCVCT_PYTHON_PATH` and `PCVCT_STUDIO_PATH` can be set to avoid needing to pass these arguments each time.
-
-Running the studio this way generates temporary config and rules files.
-Any edits to the parameters in studio will be lost when the studio is closed.
-
-# Best practices
-
-1. Do NOT manually edit files inside `inputs`.
-If parameter values need to be changed, use variations as shown in `VCT/GenerateData.jl`.
-Let pcvct manage the databases that track simulation parameters.
-
-2. Anytime a group of simulation replicates (a `Monad` in pcvct internals) is requested, all simulations in that group are used, regardless of the value of `n_replicates`.
-If the number of simulations in the group is less than `n_replicates`, then additional simulations are run to reach `n_replicates`.
-Note: if `use_previous=false`, then `n_replicates` will be run regardless.
-If you do need an upper bound on the number of simulations in such a grouping, submit an issue.
-It is assumed that most, if not all use cases, will benefit from more simulations.
-
-# Database upgrade notes:
-
-## to v0.0.3
-Introduce XML-based cell initial conditions. This introduces `ic_cell_variations`. Also, standardized the use of `config_variation` in place of `variation`. Key changes include:
-- Renaming the `variation_id` column in the `simulations` and `monads` tables to `config_variation_id`.
-- Adding the `ic_cell_variation_id` column to the `simulations` and `monads` tables.
-- In `data/inputs/configs`, renaming all instances of "variation" to "config_variation" in filenames and databases.
-
-## to v0.0.10
-Start tracking the PhysiCell version used in the simulation.
-This introduces the `physicell_versions` table which tracks the PhysiCell versions used in simulations.
-Currently, only supports reading the PhysiCell version, not setting it (e.g., through git commands).
-Key changes include:
-- Adding the `physicell_version_id` column to the `simulations`, `monads`, and `samplings` tables.
-- Adding the `physicell_versions` table.
-  - If `PhysiCell` is a git-tracked repo, this will store the commit hash as well as any tag and repo owner it can find based on the remotes. It will also store the date of the commit.
-  - If `PhysiCell` is not a git-tracked repo, it will read the `VERSION.txt` file and store that as the `commit_hash` with `-download` appended to the version.
-
-# Notes
-When an object `T <: AbstractTrial` is instantiated, immediately add it to the database AND to the CSV.
-If a simulation fails, remove it from the CSV without removing it from the database/output.
-
-# To dos
-- Rename for Julia registry. It will be so nice to have user Pkg.add("pcvct") and have it work.
-  - PhysiCellVT.jl
-  - PhysiVT.jl (possible confusion with the OpenVT project where VT = virtual tissue)
-  - PhysiCellCohorts.jl
-  - PhysiCellTrials.jl
-  - PhysiVirtualTrials.jl
-  - PhysiCellBatch.jl
-  - PhysiBatch.jl
-  - PhysiCellDB.jl
-  - PhysiDB.jl (the clear name for make the database portion a separate package)
-  - PhysiCell.jl (kinda self-important to assume this will be all the PhysiCell stuff in Julia)
+Check out [Getting started](https://drbergman.github.io/pcvct/stable/man/getting_started/) for a quick guide to using pcvct.
+Make sure you are familiar with the [Best practices](https://drbergman.github.io/pcvct/stable/man/best_practices/) section before using pcvct.

codecov.yml

@@ -1,4 +1,3 @@
 ignore:
   - "src/public.jl"
-  - "src/VCTPhysiCellStudio.jl"
   - "src/VCTUp.jl"

docs/Project.toml

@@ -1,3 +1,5 @@
 [deps]
 Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
 pcvct = "3c374bc7-7384-4f83-8ca0-87b8c727e6ff"
+Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
+Random = "9a3f8284-a2c9-5f02-9a11-845980a1fd5c"

docs/make.jl

@@ -1,5 +1,4 @@
-using pcvct
-using Documenter
+using Documenter, pcvct
 
 DocMeta.setdocmeta!(pcvct, :DocTestSetup, :(using pcvct); recursive=true)
 
@@ -15,16 +14,27 @@ makedocs(;
     pages=[
         "Home" => "index.md",
         "Manual" => Any[
-            "Guide" => "man/guide.md",
+            "Best practices" => "man/best_practices.md",
+            "Getting started" => "man/getting_started.md",
+            "CoVariations" => "man/covariations.md",
+            "Data directory" => "man/data_directory.md",
+            "Known limitations" => "man/known_limitations.md",
+            "PhysiCell Studio" => "man/physicell_studio.md",
+            "Sensitivity analysis" => "man/sensitivity_analysis.md",
         ],
         "Documentation" => map(
             s -> "lib/$(s)",
             sort(readdir(joinpath(@__DIR__, "src/lib")))
         ),
+        "Miscellaneous" => Any[
+            "Database upgrades" => "misc/database_upgrades.md",
+            "Renaming" => "misc/renaming.md",
+        ],
     ],
 )
 
 deploydocs(;
     repo="github.com/drbergman/pcvct",
-    devbranch="main",
+    devbranch="development",
+    push_preview=true,
 )

docs/src/index.md

@@ -3,8 +3,12 @@ CurrentModule = pcvct
 ```
 
 # pcvct
+The [pcvct](https://github.com/drbergman/pcvct) (PhysiCell Virtual Clinical Trials) package is a Julia package that provides a framework for running virtual clinical trials using [PhysiCell](https://github.com/MathCancer/PhysiCell) simulations. See [Getting started](@ref) for getting pcvct set up and running.
 
-Documentation for [pcvct](https://github.com/drbergman/pcvct).
+## Issues
+Have an issue? First check the [Known limitations](@ref) and [Best practices](@ref) sections. If you still have an issue, please submit it [here](https://github.com/drbergman/pcvct/issues).
 
+
+## Index
 ```@index
 ```

docs/src/lib/VCTAnalysis.md

@@ -8,5 +8,5 @@ Analyze output from a pcvct project.
 
 ```@autodocs
 Modules = [pcvct]
-Pages = ["population.jl", "substrate.jl"]
+Pages = ["population.jl", "substrate.jl", "motility.jl"]
 ```

docs/src/lib/VCTConfiguration.md

@@ -0,0 +1,14 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTConfiguration
+
+Interface with the configuration file necessary for PhysiCell simulations.
+
+Provide functionality for accessing and modifying elements in any XML, including the PhysiCell configuration file, XML rules file, and XML IC cell file.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTConfiguration.jl"]
+```

docs/src/lib/VCTICCell.md

@@ -0,0 +1,12 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTICCell
+
+Functionality for the using [PhysiCellCellCreator.jl](https://github.com/drbergman/PhysiCellECMCreator.jl) to create and use PhysiCell IC cell XML files.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTICCell.jl"]
+```

docs/src/lib/VCTICECM.md

@@ -0,0 +1,12 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTICECM
+
+Functionality for the using [PhysiCellECMCreator.jl](https://github.com/drbergman/PhysiCellECMCreator.jl) to create and use PhysiCell ECM XML files.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTICECM.jl"]
+```

docs/src/lib/VCTLoader.md

@@ -0,0 +1,12 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTLoader
+
+Load PhysiCell data into useful forms for downstream analysis.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTLoader.jl"]
+```

docs/src/lib/VCTMovie.md

@@ -0,0 +1,12 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTMovie
+
+Make movies for simulations in the database.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTMovie.jl"]
+```

docs/src/lib/VCTPhysiCellStudio.md

@@ -0,0 +1,12 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTPhysiCellStudio
+
+Launch PhysiCell Studio for a simulation run in pcvct.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTPhysiCellStudio.jl"]
+```

docs/src/lib/VCTPruner.md

@@ -0,0 +1,14 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTPruner
+
+Prune files from a simulation immediately after finishing the simulation.
+
+To motivate this functionality, consider the following scenario. A user has been testing their model, including making movies, and is ready to do a large virtual clinical trial with thousands of simulations. Saving all the SVGs will require gigabytes of storage, which is not ideal for the user. The user could choose to create a new variation on the SVG parameters (e.g., increase the SVG save interval), but then pcvct will not be able to reuse previous simulations as they have different variation IDs. Alternatively, the user can use the `PruneOptions` to delete the SVGs after each simulation is finished. This way, there are fewer variations in the database and more capability to reuse simulations.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTPruner.jl"]
+```

docs/src/lib/VCTSensitivity.md

@@ -0,0 +1,12 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTSensitivity
+
+Run sensitivity analyses on a model.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTSensitivity.jl"]
+```

docs/src/lib/VCTUserAPI.md

@@ -0,0 +1,12 @@
+```@meta
+CollapsedDocStrings = true
+```
+
+# VCTUserAPI
+
+Main functions users will use to create and run simulations, monads, samplings, and trials.
+
+```@autodocs
+Modules = [pcvct]
+Pages = ["VCTUserAPI.jl"]
+```