Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TASK-6442 - Add Resources Management functionality #2535

Open
wants to merge 71 commits into
base: develop
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
0ee51eb
core: add ResourceManager and JUnit tests, #TASK-7047, #TASK-6442
jtarraga Oct 3, 2024
c6f84ef
catalog: create default RESOURCE folder for studies, #TASK-6442
pfurio Oct 7, 2024
77cccad
core: remove old FileCreateParams, #TASK-6442
pfurio Oct 7, 2024
b754a47
core: add an MD5 check when the resource manager downloads a file, #T…
jtarraga Oct 10, 2024
7923785
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 10, 2024
6fb08b9
analysis: implement the tool DownloadResourcesTool to download all an…
jtarraga Oct 11, 2024
2fd7678
core: minor improvements
imedina Oct 11, 2024
0e80c4d
core: first download all resources to a temp folder, then copy them t…
jtarraga Oct 14, 2024
8673ebd
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 14, 2024
3baf7c9
core: rename DownloadResourceTool to ResourceFetcherTool, #TASK-7088,…
jtarraga Oct 14, 2024
8885f30
master: call the ResourceFetcherTool by the monitor service on initia…
jtarraga Oct 14, 2024
774c286
client: generate clients after renaming some fields, #TASK-7088, #TAS…
jtarraga Oct 14, 2024
2b6bfa0
core: use .md5 file to check if the resource file must be downloaded,…
jtarraga Oct 14, 2024
d6007c9
app: add migration to create missing RESOURCES folder, #TASK-6442
pfurio Oct 16, 2024
c4b872c
analysis: fetch all resources only is exececuted by admins, #TASK-708…
jtarraga Oct 17, 2024
6629f3f
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 17, 2024
90e7280
catalog: add resource checks, #TASK-6442
pfurio Oct 17, 2024
701a076
catalog: fix several tests, #TASK-6442
pfurio Oct 18, 2024
297009c
analysis: improve resource suppport and add resource exception, #TASK…
jtarraga Oct 18, 2024
fa2cba7
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 18, 2024
4fad723
analysis: simplify ResourceManager code, #TASK-7047, #TASK-6442
jtarraga Oct 22, 2024
0791fe0
catalog: fix some tests, #TASK-6442
pfurio Oct 22, 2024
e1b5b70
Merge branch 'TASK-6442' of github.com:opencb/opencga into TASK-6442
pfurio Oct 22, 2024
973d72d
analysis: update Exomiser analysis to use ResourceManager, #TASK-7125…
jtarraga Oct 23, 2024
379a0bf
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Oct 23, 2024
9807f37
analysis: add the action UNZIP to unzip Exomiser files after download…
jtarraga Oct 23, 2024
97ce73f
analysis: update mutational signature analysis according to the new r…
jtarraga Oct 23, 2024
ff513c2
core: change resource metadata to be independent/standalone, and rena…
jtarraga Oct 24, 2024
990b757
analysis: remove ResourceUtils and update relatedness analysis by usi…
jtarraga Oct 24, 2024
159e76a
analysis: fix exomiser resource path, #TASK-7125, #TASK-6442
jtarraga Oct 24, 2024
481f2e8
analysis: fix mutational signature resource path, #TASK-7127, #TASK-6442
jtarraga Oct 24, 2024
4a76f4b
analysis: mount exomiser data resources as readonly, #TASK-7125, #TAS…
jtarraga Oct 24, 2024
ba9d0bc
core: add new RESOURCES constant, #TASK-6442
pfurio Oct 28, 2024
eb37a22
catalog: allow filtering by new resource field, #TASK-6442
pfurio Oct 30, 2024
9aafcb3
Merge branch 'develop' into TASK-6442
pfurio Oct 30, 2024
e1ed083
app: fix uri generation in migration code, #TASK-6442
pfurio Oct 31, 2024
f7e0b53
catalog: improve resource manager, #TASK-7047, #TASK-6442
jtarraga Nov 6, 2024
1184713
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Nov 6, 2024
86b0194
catalog: avoid moving non-resource file to Resource folder, #TASK-6442
pfurio Nov 28, 2024
62db7e8
Merge branch 'develop' into TASK-6442
pfurio Nov 28, 2024
9434fe1
test: update JUnit tests, #TASK-7047, #TASK-6442
jtarraga Dec 3, 2024
65e51c7
catalog: improve resource manager, #TASK-7088, #TASK-6442
jtarraga Dec 4, 2024
023f04d
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Dec 4, 2024
3d66280
core: implement ResourceConfiguration and update according to this, #…
jtarraga Dec 9, 2024
70ee362
catalog: use constants where possible, #TASK-6442
pfurio Dec 10, 2024
0eb2ad3
Merge branch 'develop' into TASK-6442
pfurio Dec 10, 2024
9d8ddb2
catalog: improve ResourceManager, #TASK-7047, #TASK-6442
jtarraga Dec 11, 2024
e0d0f9f
catalog: remove unused import, #TASK-7047, #TASK-6442
jtarraga Dec 11, 2024
eed1973
analysis: update relatedness/family-qc according to resource manager …
jtarraga Dec 11, 2024
dc0de9c
analysis: update exomiser according to resource manager changes, #TAS…
jtarraga Dec 11, 2024
6e6141c
analysis: update mutational signature according to resource manager c…
jtarraga Dec 11, 2024
44d3c3f
analysis: update fetch resource tool according to resource manager ch…
jtarraga Dec 11, 2024
8aed078
test: fix JUnit test and update configuration files, #TASK-7047, #TAS…
jtarraga Dec 11, 2024
943e4de
Merge branch 'TASK-6442' of https://github.com/opencb/opencga into TA…
jtarraga Dec 11, 2024
a5cee2a
client: generate clients, #TASK-6442
jtarraga Dec 11, 2024
218510b
core: add relatedness resources in the configuration file, #TASK-6442
jtarraga Dec 12, 2024
0956608
catalog: update JUnit tests for resource manager, #TASK-7047, #TASK-6442
jtarraga Dec 12, 2024
ad213ec
master: use the admin study for the resource fetcher job, #TASK-7088,…
jtarraga Dec 12, 2024
f9347a8
core: complete the section resource in the configuration file, #TASK-…
jtarraga Dec 12, 2024
a05d8dc
server: rename endpoint resource/downloadAll to resource/fetch, #TASK…
jtarraga Dec 12, 2024
e162072
catalog: try workaround for test env. failure, #TASK-6442
jtarraga Dec 12, 2024
e157cd0
core: set resources in fetchOnInit in the configuration file, #TASK-6442
jtarraga Dec 13, 2024
152ec14
catalog: fix Exomiser resources in the configuration file, #TASK-7047…
jtarraga Dec 15, 2024
0c50eab
core: fix configuration files for Exomiser resources, #TASK-7047, #TA…
jtarraga Dec 16, 2024
134c540
analysis: update exomiser according to changes, #TASK-7125, #TASK-6442
jtarraga Dec 16, 2024
c181ab4
analysis: fix Exomiser JUnit tests, #TASK-7125, #TASK-6442
jtarraga Dec 16, 2024
6c2e33c
catalog: remove Exomiser tmp files, #TASK-7125, #TASK-6442
jtarraga Dec 16, 2024
c833066
catalog: check if files exit before deleting them, #TASK-7047, #TASK-…
jtarraga Dec 16, 2024
310f6d6
analysis: delay GRCh37 for Exomiser analysis and update the configura…
jtarraga Dec 16, 2024
86109f3
Merge branch 'develop' into TASK-6442
jtarraga Dec 19, 2024
02e3ae4
Merge branch 'develop' into TASK-6442
jtarraga Jan 7, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,15 +1,21 @@
package org.opencb.opencga.analysis;

import org.apache.commons.collections4.CollectionUtils;
import org.opencb.commons.datastore.core.Query;
import org.opencb.commons.datastore.core.QueryOptions;
import org.opencb.opencga.catalog.db.api.FileDBAdaptor;
import org.opencb.opencga.catalog.db.api.ProjectDBAdaptor;
import org.opencb.opencga.catalog.exceptions.CatalogException;
import org.opencb.opencga.catalog.managers.CatalogManager;
import org.opencb.opencga.catalog.managers.FileManager;
import org.opencb.opencga.catalog.managers.JobManager;
import org.opencb.opencga.catalog.utils.CatalogFqn;
import org.opencb.opencga.core.exceptions.ToolException;
import org.opencb.opencga.core.models.JwtPayload;
import org.opencb.opencga.core.models.common.Enums;
import org.opencb.opencga.core.models.file.File;
import org.opencb.opencga.core.models.job.Job;
import org.opencb.opencga.core.models.project.Project;
import org.opencb.opencga.core.response.OpenCGAResult;

import java.io.BufferedReader;
Expand All @@ -20,6 +26,13 @@

public class AnalysisUtils {

public static String REFERENCE_GENOME_GRCH38_FA = "REFERENCE_GENOME_GRCH38_FA";
public static String REFERENCE_GENOME_GRCH38_FAI= "REFERENCE_GENOME_GRCH38_FAI";
public static String REFERENCE_GENOME_GRCH38_GZI = "REFERENCE_GENOME_GRCH38_GZI";
public static String REFERENCE_GENOME_GRCH37_FA= "REFERENCE_GENOME_GRCH37_FA";
public static String REFERENCE_GENOME_GRCH37_FAI = "REFERENCE_GENOME_GRCH37_FAI";
public static String REFERENCE_GENOME_GRCH37_GZI= "REFERENCE_GENOME_GRCH37_GZI";

public static boolean isSupportedCommand(String commands) {
Set<String> commandSet = new HashSet<>(Arrays.asList(commands.replace(" ", "").split(",")));
if (!commandSet.contains(commands)) {
Expand Down Expand Up @@ -193,4 +206,22 @@ public static String getJobFileRelativePath(String path) {
}
return path.substring(index + 5);
}

public static String getAssembly(CatalogManager catalogManager, String studyId, String sessionId) throws CatalogException {
String assembly = "";
OpenCGAResult<Project> projectQueryResult;

JwtPayload jwtPayload = catalogManager.getUserManager().validateToken(sessionId);
CatalogFqn studyFqn = CatalogFqn.extractFqnFromStudy(studyId, jwtPayload);
String organizationId = studyFqn.getOrganizationId();

projectQueryResult = catalogManager.getProjectManager().search(organizationId, new Query(ProjectDBAdaptor.QueryParams.STUDY.key(), studyId),
new QueryOptions(QueryOptions.INCLUDE, ProjectDBAdaptor.QueryParams.ORGANISM.key()), sessionId);
if (CollectionUtils.isNotEmpty(projectQueryResult.getResults())
&& projectQueryResult.first().getOrganism() != null
&& projectQueryResult.first().getOrganism().getAssembly() != null) {
assembly = projectQueryResult.first().getOrganism().getAssembly();
}
return assembly;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import org.opencb.commons.utils.FileUtils;
import org.opencb.opencga.core.config.AnalysisTool;
import org.opencb.opencga.core.config.Configuration;
import org.opencb.opencga.core.config.ResourceFile;
import org.opencb.opencga.core.config.storage.StorageConfiguration;
import org.opencb.opencga.core.exceptions.ToolException;
import org.slf4j.Logger;
Expand Down Expand Up @@ -121,4 +122,39 @@ public static String getToolDefaultVersion(String toolId, Configuration configur
}
return defaultVersion;
}


public static AnalysisTool getAnalysisTool(String toolId, String version, Configuration configuration) throws ToolException {
for (AnalysisTool tool : configuration.getAnalysis().getTools()) {
if (toolId.equals(tool.getId())) {
if (StringUtils.isEmpty(version) || version.equals(tool.getVersion())) {
return tool;
}
}
}
throw new ToolException("Missing analysis tool (ID = " + toolId + ", version = " + version + ") in configuration file");
}

public static Path getToolResourcePath(String resourceId, Configuration configuration) throws ToolException {
// Get resources from the configuration file
for (ResourceFile resourceFile : configuration.getAnalysis().getResource().getFiles()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

configuration analysis.resource is a new field in the configuration file. It might be null when upgrading from old versions.
Either add a non-null check, or a migration that ensures that the field is correctly defined (where it would only check for the value, and fail until manually fixed)

if (resourceFile.getId().equalsIgnoreCase(resourceId)) {
Path resourcePath = configuration.getAnalysis().getResource().getBasePath().resolve(resourceFile.getPath());
if (!Files.exists(resourcePath)) {
throw new ToolException("Error getting resource '" + resourceId + "': it is not installed (i.e., missing file '"
+ resourcePath + "')");
}
return resourcePath;
}
}
throw new ToolException("Error getting resource '" + resourceId + "': it does not exist in the configuration file");
}

public static void checkAnalysisResources(String toolId, String version, Configuration configuration) throws ToolException {
AnalysisTool analysisTool = getAnalysisTool(toolId, version, configuration);
for (String resourceId : analysisTool.getResources()) {
getToolResourcePath(resourceId, configuration);
}

}
}

This file was deleted.

Loading
Loading