Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SB-24793 - Assessment data archival data product job #425

Open
wants to merge 34 commits into
base: release-4.2.0
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
a022ddb
Issue SB-24793 feat: Assessment data archival data product job imple…
manjudr Jul 14, 2021
fd2ac9c
Issue SB-24793 feat: Assessment data archival data product job imple…
manjudr Jul 14, 2021
b42df8f
Issue SB-24793 feat: removed the hardcoded column names in groupBy
manjudr Jul 14, 2021
4ca0b4c
Issue SB-24793 feat: Assessment data archival data product job imple…
manjudr Jul 14, 2021
b6d3733
Revert "Issue SB-24793 feat: Assessment data archival data product j…
manjudr Jul 14, 2021
f4ca1cb
Issue SB-24793 feat: Assessment data archival data product job imple…
manjudr Jul 14, 2021
acd9a11
Issue SB-24793 feat: Assessment data archival cassandra connection …
manjudr Jul 14, 2021
9d75256
Issue SB-24793 feat: Assessment data archival question data seriali…
manjudr Jul 16, 2021
3bb90fb
Issue SB-24793 feat: Assessment data archival question data seriali…
manjudr Jul 16, 2021
358b96a
Issue SB-24793 feat: Assessment data archival question data seriali…
manjudr Jul 16, 2021
e46e0a1
Issue SB-24793 feat: Assessment data archival question data seriali…
manjudr Jul 16, 2021
423f977
Issue SB-24793 feat: Assessment data archival question data seriali…
manjudr Jul 16, 2021
9cea5eb
Issue SB-24793 feat: removing the unwanted csv files
manjudr Jul 16, 2021
ec710cd
Issue SB-24793 feat: removing the unwanted imports
manjudr Jul 16, 2021
e29b23e
Issue SB-25481 feat: Assessment archival data product archive the da…
manjudr Jul 19, 2021
7e4ea76
Issue SB-25481 feat: Assessment archival data product archive the da…
manjudr Jul 19, 2021
b32d1e6
Issue SB-25481 feat: Assessment archival data product archive the da…
manjudr Jul 19, 2021
22f76e9
Issue SB-25481 feat: Assessment archival data product archive the da…
manjudr Jul 19, 2021
ffa1c5f
Issue SB-25481 feat: Assessment archival data product archive the da…
manjudr Jul 19, 2021
5db16a1
Issue SB-25481 feat: Assessment archival data product cache issue fix
manjudr Jul 19, 2021
df8cf7e
Issue SB-25481 feat: Assessment archival data product cache issue fix
manjudr Jul 20, 2021
1f19060
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
a3245f8
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
4380e66
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
acff429
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
e96faaf
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
94b6269
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
10c2069
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
0d0ae86
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
753d763
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
2b467d4
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
ce5b0c8
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
00fdc7c
Issue SB-25481 feat: Assessment archival data product persist issue
manjudr Jul 20, 2021
9d0e8b1
Issue SB-24793 feat: Assessment data archival configuration update
manjudr Aug 9, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Revert "Issue SB-24793 feat: Assessment data archival data product jo…
…b implementation and testcase"

This reverts commit 4ca0b4c.
  • Loading branch information
manjudr committed Jul 14, 2021
commit b6d373378e5fa53082132925f0abce371d73c750
Original file line number Diff line number Diff line change
@@ -34,14 +34,14 @@ object AssessmentArchivalJob extends optional.Application with IJob with BaseRep
val truncateData: Boolean = modelParams.getOrElse("truncateData", "false").asInstanceOf[Boolean]
try {
val res = CommonUtil.time(archiveData(spark, fetchData, jobConfig))
val total_archived_files = res._2.length
if (truncateData) deleteRecords(spark, assessmentAggDBSettings.getOrElse("keyspace", "sunbird_courses"), assessmentAggDBSettings.getOrElse("table", "assessment_aggregator")) else JobLogger.log(s"Skipping the ${assessmentAggDBSettings.getOrElse("table", "assessment_aggregator")} truncate process", None, INFO)
JobLogger.end(s"$jobName completed execution", "SUCCESS", Option(Map("timeTaken" -> res._1, "total_archived_files" -> total_archived_files)))
JobLogger.end(s"$jobName completed execution", "SUCCESS", Option(Map("timeTaken" -> res._1, "total_archived_files" -> res._2.length)))
} finally {
frameworkContext.closeContext()
spark.close()
}


}

// $COVERAGE-ON$
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

This file was deleted.

This file was deleted.

This file was deleted.

This file was deleted.

Binary file not shown.