Skip to content

Commit

Permalink
Merge pull request #21 from sybila/validation
Browse files Browse the repository at this point in the history
Validation of SBML rules for MathML
  • Loading branch information
SolunarNexus authored Feb 25, 2024
2 parents 7096d92 + dd8d655 commit 8d65f65
Show file tree
Hide file tree
Showing 32 changed files with 2,644 additions and 517 deletions.
41 changes: 35 additions & 6 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
env:
RUSTFLAGS: "-D warnings"
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
toolchain: ${{ env.RUST_VERSION }}
Expand All @@ -38,7 +38,7 @@ jobs:
env:
RUSTFLAGS: "-D warnings"
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
toolchain: ${{ env.RUST_VERSION }}
Expand All @@ -52,7 +52,9 @@ jobs:
env:
RUSTFLAGS: "-D warnings"
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- run: wget https://github.com/sbmlteam/sbml-test-suite/releases/download/3.4.0/syntactic_tests.v3.4.0.zip
- run: unzip syntactic_tests.v3.4.0.zip
- uses: dtolnay/rust-toolchain@stable
with:
toolchain: ${{ env.RUST_VERSION }}
Expand All @@ -67,20 +69,47 @@ jobs:
env:
RUSTFLAGS: "-D warnings"
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
with:
toolchain: ${{ env.RUST_VERSION }}
components: clippy
- run: cargo clippy --all-features

# Run SBML test suite
sbml-test-suite:
needs: check
name: SBML Test Suite
runs-on: ubuntu-latest
env:
RUSTFLAGS: "-D warnings"
steps:
- uses: actions/checkout@v4
- run: wget https://github.com/sbmlteam/sbml-test-suite/releases/download/3.4.0/syntactic_tests.v3.4.0.zip
- run: unzip syntactic_tests.v3.4.0.zip
- uses: dtolnay/rust-toolchain@stable
with:
toolchain: ${{ env.RUST_VERSION }}
components: rustfmt
- run: cargo run --release --example test-suite-syntactic -- `cat validated-rules.txt`
- run: zip test-results.zip ./test_suite_error.txt ./test_suite_info.txt ./test_suite_warning.txt
if: always()
- uses: actions/upload-artifact@v4
if: always()
with:
name: results
path: ./test-results.zip

# Compute code coverage
codecov:
needs: test
name: Code coverage
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
# Should allow tarpaulin to also measure coverage for the SBML test suite.
- run: wget https://github.com/sbmlteam/sbml-test-suite/releases/download/3.4.0/syntactic_tests.v3.4.0.zip
- run: unzip syntactic_tests.v3.4.0.zip
- uses: dtolnay/rust-toolchain@stable
with:
toolchain: ${{ env.RUST_VERSION }}
Expand All @@ -90,7 +119,7 @@ jobs:
tool: cargo-tarpaulin
- run: cargo tarpaulin --verbose --lib --examples --all-features --out xml
- name: Upload to codecov.io
uses: codecov/codecov-action@v3
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
- name: Archive code coverage results
Expand Down
8 changes: 7 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,4 +20,10 @@ Cargo.lock
.idea

# VScode project folder
.vscode
.vscode

# The SBML test suite files.
syntactic
test_suite_error.txt
test_suite_warning.txt
test_suite_info.txt
5 changes: 5 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@ edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[features]
# When enabled, rust the SBML syntactic test suite as part of unit tests.
# This is mainly used for the purpose of code coverage computation.
sbml_test_suite = []

[dependencies]
const_format = "0.2.31"
macros = { path = "macros" }
Expand Down
47 changes: 47 additions & 0 deletions VALIDATION.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Biodivine SBML (validation)

Currently, the correctness of the implementation is ensured using two
mechanisms:

- Unit tests.
- SBML syntactic test suite.

### Unit tests

These are "straightforward" in the sense that they use standard `cargo`
functionality, and should be thus familiar to anyone testing in the
[Rust ecosystem](https://doc.rust-lang.org/book/ch11-01-writing-tests.html).

### SBML test suite

For further validation, we compare our results to the
[SBML test suite](https://github.com/sbmlteam/sbml-test-suite/releases).
Specifically, we only use the `syntactic` subset of the test cases,
because the rest is concerned with simulation, which we do not support.

This test suite is not part of the standard `cargo test` process. In fact,
it is present twice in the codebase.

> Both cases expect that the latest version of the syntactic test suite
> is downloaded and extracted in the `./syntactic` folder (you can get
> the zip file in the release section of the repository linked above).
First, it is present as a "silent" test in `core::validation::test_suite`
module that is enabled by the `sbml_test_suite` feature. To run it, execute
`cargo test --all-features`. Here, we execute all the test cases in the suite,
but we don't check if the result is correct. This is mainly to (a) measure code
coverage, and (b) detect if the library outright fails on some of the examples.

Second, a dedicated "example" binary (executed as
`cargo run --example test-suite-syntactic`) provides additional options to
tune the testing process. First, it prints additional info about the results
and actually checks that the results conform to the expected outputs. However,
you can request to only test a specific subset of rules by specifying their IDs
as the command line arguments (e.g. `cargo run --example test-suite-syntactic -- 10201 10202`).
This still runs all tests cases, but only shows
an error for the cases where an inconsistency is detected for one of the
requested rules.

For the purposes of automated testing, we apply the list of rules in the
`validated-rules.txt` file. As such, if you extend the list of actually validated
rules, you have to also extend this file.
14 changes: 9 additions & 5 deletions examples/basic_example.rs
Original file line number Diff line number Diff line change
@@ -1,19 +1,23 @@
use biodivine_lib_sbml::{Sbml, SbmlIssue};
use biodivine_lib_sbml::Sbml;

// To run this example, execute `cargo run --example basic_example`.
// If you want to add command line arguments, you can use
// `cargo run --example basic_example -- ARG_1 ARG_2`.
// Note the use of `--` to indicate that ARG_x values are meant as arguments
// for the example binary, not for `cargo` itself.
fn main() {
let doc = Sbml::read_path("test-inputs/COVID19_immunotherapy_Mathematical_Model.xml").unwrap();
// let doc = Sbml::read_path("test-inputs/COVID19_immunotherapy_Mathematical_Model.xml").unwrap();
// let doc = Sbml::read_path("test-inputs/cholesterol_metabolism_and_atherosclerosis.xml").unwrap();
let doc = Sbml::read_path("test-inputs/Mukandavire2020.xml").unwrap();

// let model = doc.model().get().unwrap();
// Print the whole document:
// println!("{}", model.read_doc().write_str().unwrap());
let mut issues: Vec<SbmlIssue> = Vec::new();
doc.validate(&mut issues);
let issues = doc.validate();

println!("No. of issues: {}", issues.len());
println!("{:?}", issues);
for issue in issues {
println!("{:?}", issue);
}
// assert_eq!(issues.len(), 0);
}
Loading

0 comments on commit 8d65f65

Please sign in to comment.