Skip to content
This repository was archived by the owner on Aug 16, 2019. It is now read-only.

Commit

Permalink
Update migration.md
Browse files Browse the repository at this point in the history
  • Loading branch information
@Sunitashelar
Sunitashelar authored Jul 25, 2018
1 parent fbf2ced commit 01b06d6
Showing 1 changed file with 53 additions and 53 deletions.
106 changes: 53 additions & 53 deletions pages/developer-docs/installation/migration.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,57 +10,52 @@ allowSearch: true

## Overview

Sunbird provides support for multiple external IDs for a user. In release 1.8, the table name and schema used to store user external IDs has changed.
* external id is the identity of user out-side of sunbird system.
Example: John belongs to organisation **X** and in **X** his identity is **jo_123** and idType is **employeeId**,
now he is trying to register with sunbird, then in sunbird he can use externalId as **jo_123** and provider as **X** and idType as **employeeId**.
* Similarly John can be part of multiple organisations and each organisations can have there own identity to recognise John, in that case during sunbird registration (create/update user api), he can pass all the externalIds and provider and idType.
Sunbird supports multiple external IDs for a user. An external ID is the user's identity outside the Sunbird system. For example; John is an employee of **XYZ Corporation**. In **XYZ Corporation**, his identity(ID) is **jo_123** and ID Type is **employeeId**.
When he registers with Sunbird, he can specify his external ID as **jo_123**, provider as **XYZ Corporation** and ID Type as **employeeId**. Similarly, John, being part of multiple organizations, (like also being an account holder in **ABC bank**), has a separate identity in each organization. The ID provided by that organization identifies and recognises John as being part of that particular organization. When John is registered in Sunbird, using the create or update user API, he can specify all the external IDs, providers and ID types as values in the associated parameters.

The purpose of this document is to describe the steps for migration of user external IDs from old "user_external_identity" table to new "usr_external_identity" table format.
As part of Sunbird release version 1.8, the table holding user external ID data is modified as follows:

Following are the changes:
* The table name changed from "user_external_identity" to "usr_external_identity".
* A new column "idtype" is introduced in "usr_external_identity" table for storing external ID type. e.g. UIDAI is provider of Aadhaar (ID type).
* In release 1.7, the external ID was unencrypted. In release 1.8, encryption of external ID is also performed to ensure privacy of user external ID information.
* The table name is modified from **user_external_identity** to **usr_external_identity**.
* A new column **idtype** is introduced in the **usr_external_identity** table to store external ID type. For example, UIDAI is the provider for ID type Aadhaar.
* In Sunbird release version 1.7, the external ID was unencrypted. In release 1.8, the external ID is encrypted to ensure privacy of user external ID information.

**Note:** This migration of data needs to be performed after deploying Sunbird learner service release version 1.8.
**Note:** This data migration should be done after deploying Sunbird learner service release version 1.8.

## Prerequisites

Following are the pre-requisites before running the migration script:
To run the migration script:

1. Define a mapping file which specifies ID type for each unique external ID provider available in "user_external_identity" table.
A. Get list of all providers using CQL query: select provider from user_external_identity;

B. Create a CSV file (e.g. mapping.csv) which maps each **unique** external ID to desired ID type.
e.g.
1.Define a mapping file that specifies the ID type for each unique external ID provider available in the **user_external_identity** table.
a.Get a list of all providers using CQL query: `select provider from user_external_identity`
b.Create a CSV file (mapping.csv) that maps each unique external ID to its associated ID type
For example;
```
provider,idType
ka,kaIdType
ap,apIdType
```

2. Before performing any migration take a backup of data in sunbird keyspace within Cassandra DB.
2. Before migrating any data, take a backup in Sunbird keyspace within the Cassandra DB.

## Configuration Parameters
The following are the configuration parameters:
The following parameters need to be configured for migrating user data:

| S.No. | Configuration parameter | Description |
| 1 | cassandra_server | Cassandra DB IP Address|
| 2 | cassandra_port | Cassandra DB Port Number |
| 3 | keyspace_name | Cassandra DB Keyspace Name|
| 4 | encryption_key | Key used in Sunbird for encryption of private user information |
| 5 | provider_idtype_mapping_file_path | Path to CSV file which defines mapping between external ID provider and type |
| S.No. | Parameter | Description | Example |
| 1 | cassandra_server | Cassandra DB IP Address| 50.34.16.33|
| 2 | cassandra_port | Cassandra DB Port Number | 9145 |
| 3 | keyspace_name | Cassandra DB Keyspace Name| demodb |
| 4 | encryption_key | Key used in Sunbird for encryption of private user information | 60B568970ASDFer321 |
| 5 | provider_idtype_mapping_file_path | Path to CSV file which defines mapping between external ID provider and type | \john\mapfile.csv |

## Running Migration Script
## Run the Migration Script

In order to migrate the user external ID data, follow below mentioned steps:
To migrate user external ID data:

1. Extract archive file (sunbird-utils/cassandra-migration-etl/r1.8/UserExternalIdentityMigrationBin.zip) which contains script for user external identity table migration.
1.Extract the archive file (sunbird-utils/cassandra-migration-etl/r1.8/UserExternalIdentityMigrationBin.zip) that contains the script for user external identity table migration

2. Based on OS, run the corresponding script for performing the migration.
2.Based on the OS, run the corresponding script to migrate the data

- For Linux, run the following command:
- For Linux, run the following command:
```
UserExternalIdentityMigration_run.sh --context_param cassandra_server="{cassandra_server}" --context_param cassandra_port="{cassandra_port}" --context_param cassandra_keyspace="{keyspace_name}" --context_param sunbird_encryption_key="{encryption_key}" --context_param provider_idtype_mapping_file_path="{provider_idtype_mapping_file_path}"
```
Expand All @@ -71,34 +66,23 @@ In order to migrate the user external ID data, follow below mentioned steps:
UserExternalIdentityMigration_run.bat --context_param cassandra_server="{cassandra_server}" --context_param cassandra_port="{cassandra_port}" --context_param cassandra_keyspace="{keyspace_name}" --context_param sunbird_encryption_key="{encryption_key}" --context_param provider_idtype_mapping_file_path="{provider_idtype_mapping_file_path}"
```
3. Verify that external IDs are updated in usr_external_identity table with desired ID type and provider using CQL query: select * from usr_external_identity;
3. Verify that the external IDs are updated in the **usr_external_identity** table with the desired ID type and provider using CQL query: `select * from usr_external_identity;`
## Table Details
### Details description of old table user_external_identity
| S.No. | Field | Description |
| 1 | id | identity of the recoreds.Primary key|
| 2 | createdby | User id , By whom recoreds is created |
| 3 | createdon | On which date and time records inserted into database|
| 4 | externalid | Identity of user apart from sunbird system, so that when other system fetch records from sunbird, they can co-relate user within there system using externalid and provider combination |
| 5 | provider | provider can be any organisation|
| 6 | lastupdatedon | When recoreds was last updated|
| 7 | userid | User identity within sunbird|
## user_external_identity Table Details
**Note** The **user_external_identity** table is used prior to Sunbird release version 1.8
### Details description of new table usr_external_identity
| S.No. | Field | Description |
| 1 | provider | provider can be any organisation|
| 2 | idtype | idtype can be employeeId, studentid etc |
| 3 | externalid | Identity of user apart from sunbird system, so that when other system fetch records from sunbird, they can co-relate user within there system using externalid and provider combination|
| 4 | createdby | User id , By whom recoreds is created |
| 5 | createdon | On which date and time records inserted into database|
| 6 | lastupdatedon | When recoreds was last updated|
| 7 | userid | User identity within sunbird|
##Note : Combination of provider,idtype,externalid must be unique. System is using these combination as primaryKey.
| 1 | id | Identity of the record. This is the primary key|
| 2 | createdby | The User ID of the person who created the record |
| 3 | createdon | The date and time when the record was inserted into the database|
| 4 | externalid | The user's identity apart from the Sunbird system. This field allows other systems fetch records from Sunbird and co-relate the user within their system using a combination of the externalid and provider |
| 5 | provider | The provider of the external ID. This can be any organisation|
| 6 | lastupdatedon | The date on which the record was last updated|
| 7 | userid | The User's identity within Sunbird|
The schema of the existing table is as follows:
### user_external_identity Table Schema
```
user_external_identity (
Expand All @@ -113,7 +97,23 @@ user_external_identity (
)
```
The schema of the new table is as follows:
### usr_external_identity Table Details
**Note** The **usr_external_identity** table is used from Sunbird release version 1.8
| S.No. | Field | Description |
| 1 | provider | The provider of the external ID. This can be any organisation|
| 2 | idtype | The type of ID for example; employee ID, student ID, etc. |
| 3 | externalid | The user's identity apart from the Sunbird system. This field allows other systems fetch records from Sunbird and co-relate the user within their system using a combination of the externalid and provider |
| 4 | createdby | The User ID of the person who created the record |
| 5 | createdon | The date and time when the record was inserted into the database |
| 6 | lastupdatedon | The date on which the record was last updated |
| 7 | userid | The User's identity within Sunbird|
**Note:** The combination of provider, idtype and externalid must be unique. The system uses this combination as a primary Key.
### usr_external_identity Table Schema
```
usr_external_identity (
Expand Down

0 comments on commit 01b06d6

Please sign in to comment.