Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TASK-7134 - Re-implement Aggregations Stats for all Catalog Browsers #2529

Open
wants to merge 32 commits into
base: develop
Choose a base branch
from
Open

TASK-7134 - Re-implement Aggregations Stats for all Catalog Browsers #2529

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
666fb1d
catalog: add facet interfaces, #TASK-7134
pfurio Oct 23, 2024
017857d
catalog: add facet dbAdaptor layer, #TASK-7134
pfurio Oct 23, 2024
eb2cf3b
catalog: add facets for samples, and generate clients, #TASK-7152, #T…
jtarraga Oct 24, 2024
fb1b8cb
catalog: remove duplicated parameter in SampleWSServer aggregation st…
jtarraga Oct 25, 2024
ea623ef
catalog: use the MongoDB to FacetField converter in aggregation stats…
jtarraga Oct 25, 2024
cad2d3a
catalog: add facet to FileManager, #TASK-7152, TASK-7134
jtarraga Oct 25, 2024
d43005e
storage: update according to java-common-libs changes, #TASK-7152, #T…
jtarraga Oct 28, 2024
7bbfea6
catalog: add aggregation stats (facets) for cohorts, clinical analyse…
jtarraga Oct 28, 2024
68c8290
catalog: add log message, #TASK-7152, #TASK-7134
jtarraga Oct 28, 2024
1982e4c
server: add facets (aggregation stats) for clinical analyses and inte…
jtarraga Oct 28, 2024
9285d0a
catalog: update manager facet functions by removing protected functio…
jtarraga Oct 28, 2024
7a836bf
test: add facet JUnit test for Files, #TASK-7152, #TASK-7134
jtarraga Oct 29, 2024
f99dbd4
test: add facet JUnit tests for managers, #TASK-7152, #TASK-7134
jtarraga Oct 29, 2024
0902b25
catalog: update according to java-common-libs, #TASK-7152, #TASK-7134
jtarraga Nov 11, 2024
12a6e49
catalog: restore dot notation after applying faceting, #TASK-7152, #T…
jtarraga Nov 12, 2024
5f1dcdb
server: add parameters to endpoing aggregation stats, #TASK-7152, #TA…
jtarraga Nov 13, 2024
cae0d34
Merge branch 'develop' into TASK-7134
imedina Dec 8, 2024
d1989f9
Merge branch 'develop' into TASK-7134
jtarraga Dec 9, 2024
1ad680a
Merge branch 'TASK-7134' of https://github.com/opencb/opencga into TA…
jtarraga Dec 9, 2024
630cacc
catalog: minor improvement in 'facet' logger
imedina Dec 12, 2024
4f25cc8
server: improve parameter description, #TASK-7152, #TASK-7134
jtarraga Dec 13, 2024
9761058
client: restore the previous name for the facet parameter, #TASK-7152…
jtarraga Jan 16, 2025
6dbb931
client: improve description of the parameter 'field' for aggregations…
jtarraga Jan 17, 2025
c64b82b
catalog: implement facet in JobMongoDBAdaptor, #TASK-7152, #TASK-7134
jtarraga Jan 20, 2025
5c2d858
test: add JUnit test for MongoDB facets using the new aggregations: y…
jtarraga Jan 30, 2025
f716812
test: fix JUnit tests according to biodata changes, #TASK-7152, #TASK…
jtarraga Jan 31, 2025
97094e0
test: update according to biodata changes, #TASKL-7152, #TASK-7134
jtarraga Jan 31, 2025
50fa1b2
test: update JUnit test according to java-commons-lib changes, #TASk-…
jtarraga Jan 31, 2025
4b7c38b
test: add more JUnit tests for aggregations, #TASK-7152, #TASK-7134
jtarraga Jan 31, 2025
c03e278
Merge branch 'develop' into TASK-7134
jtarraga Feb 3, 2025
191de31
Merge branch 'develop' into TASK-7134
jtarraga Feb 5, 2025
6481ba1
Merge branch 'develop' into TASK-7134
jtarraga Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions opencga-app/src/main/java/org/opencb/opencga/app/cli/main/OpenCgaCompleter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,17 +39,17 @@ public abstract class OpenCgaCompleter implements Completer {
.map(Candidate::new)
.collect(toList());

private List<Candidate> panelsList = asList( "acl-update","create","distinct","import","search","acl","delete","info","update")
private List<Candidate> panelsList = asList( "acl-update","aggregationstats","create","distinct","import","search","acl","delete","info","update")
.stream()
.map(Candidate::new)
.collect(toList());

private List<Candidate> clinicalList = asList( "acl-update","annotation-sets-load","clinical-configuration-update","create","distinct","interpretation-distinct","interpretation-search","interpretation-info","interpreter-cancer-tiering-run","interpreter-exomiser-run","interpreter-team-run","interpreter-tiering-run","interpreter-zetta-run","load","rga-aggregation-stats","rga-gene-query","rga-gene-summary","rga-index-run","rga-individual-query","rga-individual-summary","rga-variant-query","rga-variant-summary","search","variant-query","acl","delete","update","annotation-sets-annotations-update","info","interpretation-create","interpretation-clear","interpretation-delete","interpretation-revert","interpretation-update","report-update")
private List<Candidate> clinicalList = asList( "acl-update","aggregationstats","annotation-sets-load","clinical-configuration-update","create","distinct","interpretation-aggregation-stats","interpretation-distinct","interpretation-search","interpretation-info","interpreter-cancer-tiering-run","interpreter-exomiser-run","interpreter-team-run","interpreter-tiering-run","interpreter-zetta-run","load","rga-aggregation-stats","rga-gene-query","rga-gene-summary","rga-index-run","rga-individual-query","rga-individual-summary","rga-variant-query","rga-variant-summary","search","variant-query","acl","delete","update","annotation-sets-annotations-update","info","interpretation-create","interpretation-clear","interpretation-delete","interpretation-revert","interpretation-update","report-update")
.stream()
.map(Candidate::new)
.collect(toList());

private List<Candidate> jobsList = asList( "acl-update","create","distinct","retry","search","top","acl","delete","info","update","kill","log-head","log-tail")
private List<Candidate> jobsList = asList( "acl-update","aggregationstats","create","distinct","retry","search","top","acl","delete","info","update","kill","log-head","log-tail")
.stream()
.map(Candidate::new)
.collect(toList());
Expand All @@ -59,12 +59,12 @@ public abstract class OpenCgaCompleter implements Completer {
.map(Candidate::new)
.collect(toList());

private List<Candidate> individualsList = asList( "acl-update","annotation-sets-load","create","distinct","search","acl","delete","info","update","annotation-sets-annotations-update","relatives")
private List<Candidate> individualsList = asList( "acl-update","aggregationstats","annotation-sets-load","create","distinct","search","acl","delete","info","update","annotation-sets-annotations-update","relatives")
.stream()
.map(Candidate::new)
.collect(toList());

private List<Candidate> familiesList = asList( "acl-update","annotation-sets-load","create","distinct","search","acl","delete","info","update","annotation-sets-annotations-update")
private List<Candidate> familiesList = asList( "acl-update","aggregationstats","annotation-sets-load","create","distinct","search","acl","delete","info","update","annotation-sets-annotations-update")
.stream()
.map(Candidate::new)
.collect(toList());
Expand All @@ -74,7 +74,7 @@ public abstract class OpenCgaCompleter implements Completer {
.map(Candidate::new)
.collect(toList());

private List<Candidate> samplesList = asList( "acl-update","annotation-sets-load","create","distinct","load","search","acl","delete","info","update","annotation-sets-annotations-update")
private List<Candidate> samplesList = asList( "acl-update","aggregationstats","annotation-sets-load","create","distinct","load","search","acl","delete","info","update","annotation-sets-annotations-update")
.stream()
.map(Candidate::new)
.collect(toList());
Expand All @@ -99,7 +99,7 @@ public abstract class OpenCgaCompleter implements Completer {
.map(Candidate::new)
.collect(toList());

private List<Candidate> filesList = asList( "acl-update","annotation-sets-load","bioformats","create","distinct","fetch","formats","link","link-run","postlink-run","search","upload","acl","delete","info","unlink","update","annotation-sets-annotations-update","download","grep","head","image","move","refresh","tail","list","tree")
private List<Candidate> filesList = asList( "acl-update","aggregationstats","annotation-sets-load","bioformats","create","distinct","fetch","formats","link","link-run","postlink-run","search","upload","acl","delete","info","unlink","update","annotation-sets-annotations-update","download","grep","head","image","move","refresh","tail","list","tree")
.stream()
.map(Candidate::new)
.collect(toList());
Expand All @@ -109,7 +109,7 @@ public abstract class OpenCgaCompleter implements Completer {
.map(Candidate::new)
.collect(toList());

private List<Candidate> cohortsList = asList( "acl-update","annotation-sets-load","create","distinct","generate","search","acl","delete","info","update","annotation-sets-annotations-update")
private List<Candidate> cohortsList = asList( "acl-update","aggregationstats","annotation-sets-load","create","distinct","generate","search","acl","delete","info","update","annotation-sets-annotations-update")
.stream()
.map(Candidate::new)
.collect(toList());
Expand Down
9 changes: 9 additions & 0 deletions opencga-app/src/main/java/org/opencb/opencga/app/cli/main/OpencgaCliOptionsParser.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ public OpencgaCliOptionsParser() {
jCommander.addCommand("panels", diseasePanelsCommandOptions);
JCommander diseasePanelsSubCommands = jCommander.getCommands().get("panels");
diseasePanelsSubCommands.addCommand("acl-update", diseasePanelsCommandOptions.updateAclCommandOptions);
diseasePanelsSubCommands.addCommand("aggregationstats", diseasePanelsCommandOptions.aggregationStatsCommandOptions);
diseasePanelsSubCommands.addCommand("create", diseasePanelsCommandOptions.createCommandOptions);
diseasePanelsSubCommands.addCommand("distinct", diseasePanelsCommandOptions.distinctCommandOptions);
diseasePanelsSubCommands.addCommand("import", diseasePanelsCommandOptions.importCommandOptions);
Expand All @@ -112,10 +113,12 @@ public OpencgaCliOptionsParser() {
jCommander.addCommand("clinical", analysisClinicalCommandOptions);
JCommander analysisClinicalSubCommands = jCommander.getCommands().get("clinical");
analysisClinicalSubCommands.addCommand("acl-update", analysisClinicalCommandOptions.updateAclCommandOptions);
analysisClinicalSubCommands.addCommand("aggregationstats", analysisClinicalCommandOptions.aggregationStatsCommandOptions);
analysisClinicalSubCommands.addCommand("annotation-sets-load", analysisClinicalCommandOptions.loadAnnotationSetsCommandOptions);
analysisClinicalSubCommands.addCommand("clinical-configuration-update", analysisClinicalCommandOptions.updateClinicalConfigurationCommandOptions);
analysisClinicalSubCommands.addCommand("create", analysisClinicalCommandOptions.createCommandOptions);
analysisClinicalSubCommands.addCommand("distinct", analysisClinicalCommandOptions.distinctCommandOptions);
analysisClinicalSubCommands.addCommand("interpretation-aggregation-stats", analysisClinicalCommandOptions.aggregationStatsInterpretationCommandOptions);
analysisClinicalSubCommands.addCommand("interpretation-distinct", analysisClinicalCommandOptions.distinctInterpretationCommandOptions);
analysisClinicalSubCommands.addCommand("interpretation-search", analysisClinicalCommandOptions.searchInterpretationCommandOptions);
analysisClinicalSubCommands.addCommand("interpretation-info", analysisClinicalCommandOptions.infoInterpretationCommandOptions);
Expand Down Expand Up @@ -151,6 +154,7 @@ public OpencgaCliOptionsParser() {
jCommander.addCommand("jobs", jobsCommandOptions);
JCommander jobsSubCommands = jCommander.getCommands().get("jobs");
jobsSubCommands.addCommand("acl-update", jobsCommandOptions.updateAclCommandOptions);
jobsSubCommands.addCommand("aggregationstats", jobsCommandOptions.aggregationStatsCommandOptions);
jobsSubCommands.addCommand("create", jobsCommandOptions.createCommandOptions);
jobsSubCommands.addCommand("distinct", jobsCommandOptions.distinctCommandOptions);
jobsSubCommands.addCommand("retry", jobsCommandOptions.retryCommandOptions);
Expand Down Expand Up @@ -182,6 +186,7 @@ public OpencgaCliOptionsParser() {
jCommander.addCommand("individuals", individualsCommandOptions);
JCommander individualsSubCommands = jCommander.getCommands().get("individuals");
individualsSubCommands.addCommand("acl-update", individualsCommandOptions.updateAclCommandOptions);
individualsSubCommands.addCommand("aggregationstats", individualsCommandOptions.aggregationStatsCommandOptions);
individualsSubCommands.addCommand("annotation-sets-load", individualsCommandOptions.loadAnnotationSetsCommandOptions);
individualsSubCommands.addCommand("create", individualsCommandOptions.createCommandOptions);
individualsSubCommands.addCommand("distinct", individualsCommandOptions.distinctCommandOptions);
Expand All @@ -197,6 +202,7 @@ public OpencgaCliOptionsParser() {
jCommander.addCommand("families", familiesCommandOptions);
JCommander familiesSubCommands = jCommander.getCommands().get("families");
familiesSubCommands.addCommand("acl-update", familiesCommandOptions.updateAclCommandOptions);
familiesSubCommands.addCommand("aggregationstats", familiesCommandOptions.aggregationStatsCommandOptions);
familiesSubCommands.addCommand("annotation-sets-load", familiesCommandOptions.loadAnnotationSetsCommandOptions);
familiesSubCommands.addCommand("create", familiesCommandOptions.createCommandOptions);
familiesSubCommands.addCommand("distinct", familiesCommandOptions.distinctCommandOptions);
Expand Down Expand Up @@ -227,6 +233,7 @@ public OpencgaCliOptionsParser() {
jCommander.addCommand("samples", samplesCommandOptions);
JCommander samplesSubCommands = jCommander.getCommands().get("samples");
samplesSubCommands.addCommand("acl-update", samplesCommandOptions.updateAclCommandOptions);
samplesSubCommands.addCommand("aggregationstats", samplesCommandOptions.aggregationStatsCommandOptions);
samplesSubCommands.addCommand("annotation-sets-load", samplesCommandOptions.loadAnnotationSetsCommandOptions);
samplesSubCommands.addCommand("create", samplesCommandOptions.createCommandOptions);
samplesSubCommands.addCommand("distinct", samplesCommandOptions.distinctCommandOptions);
Expand Down Expand Up @@ -309,6 +316,7 @@ public OpencgaCliOptionsParser() {
jCommander.addCommand("files", filesCommandOptions);
JCommander filesSubCommands = jCommander.getCommands().get("files");
filesSubCommands.addCommand("acl-update", filesCommandOptions.updateAclCommandOptions);
filesSubCommands.addCommand("aggregationstats", filesCommandOptions.aggregationStatsCommandOptions);
filesSubCommands.addCommand("annotation-sets-load", filesCommandOptions.loadAnnotationSetsCommandOptions);
filesSubCommands.addCommand("bioformats", filesCommandOptions.bioformatsCommandOptions);
filesSubCommands.addCommand("create", filesCommandOptions.createCommandOptions);
Expand Down Expand Up @@ -373,6 +381,7 @@ public OpencgaCliOptionsParser() {
jCommander.addCommand("cohorts", cohortsCommandOptions);
JCommander cohortsSubCommands = jCommander.getCommands().get("cohorts");
cohortsSubCommands.addCommand("acl-update", cohortsCommandOptions.updateAclCommandOptions);
cohortsSubCommands.addCommand("aggregationstats", cohortsCommandOptions.aggregationStatsCommandOptions);
cohortsSubCommands.addCommand("annotation-sets-load", cohortsCommandOptions.loadAnnotationSetsCommandOptions);
cohortsSubCommands.addCommand("create", cohortsCommandOptions.createCommandOptions);
cohortsSubCommands.addCommand("distinct", cohortsCommandOptions.distinctCommandOptions);
Expand Down
77 changes: 77 additions & 0 deletions .../main/java/org/opencb/opencga/app/cli/main/executors/AnalysisClinicalCommandExecutor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,9 @@ public void execute() throws Exception {
case "acl-update":
queryResponse = updateAcl();
break;
case "aggregationstats":
queryResponse = aggregationStats();
break;
case "annotation-sets-load":
queryResponse = loadAnnotationSets();
break;
Expand All @@ -115,6 +118,9 @@ public void execute() throws Exception {
case "distinct":
queryResponse = distinct();
break;
case "interpretation-aggregation-stats":
queryResponse = aggregationStatsInterpretation();
break;
case "interpretation-distinct":
queryResponse = distinctInterpretation();
break;
Expand Down Expand Up @@ -248,6 +254,48 @@ private RestResponse<ClinicalAnalysisAclEntryList> updateAcl() throws Exception
return openCGAClient.getClinicalAnalysisClient().updateAcl(commandOptions.members, commandOptions.action, clinicalAnalysisAclUpdateParams, queryParams);
}

private RestResponse<FacetField> aggregationStats() throws Exception {
logger.debug("Executing aggregationStats in Analysis - Clinical command line");

AnalysisClinicalCommandOptions.AggregationStatsCommandOptions commandOptions = analysisClinicalCommandOptions.aggregationStatsCommandOptions;

ObjectMap queryParams = new ObjectMap();
queryParams.putIfNotEmpty("study", commandOptions.study);
queryParams.putIfNotEmpty("id", commandOptions.id);
queryParams.putIfNotEmpty("uuid", commandOptions.uuid);
queryParams.putIfNotEmpty("type", commandOptions.type);
queryParams.putIfNotEmpty("disorder", commandOptions.disorder);
queryParams.putIfNotEmpty("files", commandOptions.files);
queryParams.putIfNotEmpty("sample", commandOptions.sample);
queryParams.putIfNotEmpty("individual", commandOptions.individual);
queryParams.putIfNotEmpty("proband", commandOptions.proband);
queryParams.putIfNotEmpty("probandSamples", commandOptions.probandSamples);
queryParams.putIfNotEmpty("family", commandOptions.family);
queryParams.putIfNotEmpty("familyMembers", commandOptions.familyMembers);
queryParams.putIfNotEmpty("familyMemberSamples", commandOptions.familyMemberSamples);
queryParams.putIfNotEmpty("panels", commandOptions.panels);
queryParams.putIfNotNull("locked", commandOptions.locked);
queryParams.putIfNotEmpty("analystId", commandOptions.analystId);
queryParams.putIfNotEmpty("priority", commandOptions.priority);
queryParams.putIfNotEmpty("flags", commandOptions.flags);
queryParams.putIfNotEmpty("creationDate", commandOptions.creationDate);
queryParams.putIfNotEmpty("modificationDate", commandOptions.modificationDate);
queryParams.putIfNotEmpty("dueDate", commandOptions.dueDate);
queryParams.putIfNotEmpty("qualityControlSummary", commandOptions.qualityControlSummary);
queryParams.putIfNotEmpty("release", commandOptions.release);
queryParams.putIfNotNull("snapshot", commandOptions.snapshot);
queryParams.putIfNotEmpty("status", commandOptions.status);
queryParams.putIfNotEmpty("internalStatus", commandOptions.internalStatus);
queryParams.putIfNotEmpty("annotation", commandOptions.annotation);
queryParams.putIfNotNull("deleted", commandOptions.deleted);
queryParams.putIfNotEmpty("field", commandOptions.field);
if (queryParams.get("study") == null && OpencgaMain.isShellMode()) {
queryParams.putIfNotEmpty("study", sessionManager.getSession().getCurrentStudy());
}

return openCGAClient.getClinicalAnalysisClient().aggregationStats(queryParams);
}

private RestResponse<Job> loadAnnotationSets() throws Exception {
logger.debug("Executing loadAnnotationSets in Analysis - Clinical command line");

Expand Down Expand Up @@ -432,6 +480,35 @@ private RestResponse<ObjectMap> distinct() throws Exception {
return openCGAClient.getClinicalAnalysisClient().distinct(commandOptions.field, queryParams);
}

private RestResponse<FacetField> aggregationStatsInterpretation() throws Exception {
logger.debug("Executing aggregationStatsInterpretation in Analysis - Clinical command line");

AnalysisClinicalCommandOptions.AggregationStatsInterpretationCommandOptions commandOptions = analysisClinicalCommandOptions.aggregationStatsInterpretationCommandOptions;

ObjectMap queryParams = new ObjectMap();
queryParams.putIfNotEmpty("study", commandOptions.study);
queryParams.putIfNotEmpty("id", commandOptions.id);
queryParams.putIfNotEmpty("uuid", commandOptions.uuid);
queryParams.putIfNotEmpty("name", commandOptions.name);
queryParams.putIfNotEmpty("clinicalAnalysisId", commandOptions.clinicalAnalysisId);
queryParams.putIfNotEmpty("analystId", commandOptions.analystId);
queryParams.putIfNotEmpty("methodName", commandOptions.methodName);
queryParams.putIfNotEmpty("panels", commandOptions.panels);
queryParams.putIfNotEmpty("primaryFindings", commandOptions.primaryFindings);
queryParams.putIfNotEmpty("secondaryFindings", commandOptions.secondaryFindings);
queryParams.putIfNotEmpty("creationDate", commandOptions.creationDate);
queryParams.putIfNotEmpty("modificationDate", commandOptions.modificationDate);
queryParams.putIfNotEmpty("status", commandOptions.status);
queryParams.putIfNotEmpty("internalStatus", commandOptions.internalStatus);
queryParams.putIfNotEmpty("release", commandOptions.release);
queryParams.putIfNotEmpty("field", commandOptions.field);
if (queryParams.get("study") == null && OpencgaMain.isShellMode()) {
queryParams.putIfNotEmpty("study", sessionManager.getSession().getCurrentStudy());
}

return openCGAClient.getClinicalAnalysisClient().aggregationStatsInterpretation(queryParams);
}

private RestResponse<ObjectMap> distinctInterpretation() throws Exception {
logger.debug("Executing distinctInterpretation in Analysis - Clinical command line");

Expand Down
Loading