Skip to content
This repository has been archived by the owner on Oct 5, 2021. It is now read-only.

Add copy graph script, update README.md #165

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,10 @@ comic books in which they appeared.

```bash
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo \
-O /etc/yum.repos.d/epel-apache-maven.repo
-O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum update -y && sudo yum upgrade -y
sudo yum install -y apache-maven sqlite-devel git java-1.8.0-openjdk-devel
sudo yum update -y && sudo yum upgrade -y && sudo yum install -y \
apache-maven sqlite-devel git java-1.8.0-openjdk-devel
sudo alternatives --set java /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java
sudo alternatives --set javac /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/javac
git clone https://github.com/awslabs/dynamodb-titan-storage-backend.git
Expand Down Expand Up @@ -389,10 +389,10 @@ credential configuration.

```bash
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo \
-O /etc/yum.repos.d/epel-apache-maven.repo
-O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum update -y && sudo yum upgrade -y
sudo yum install -y apache-maven sqlite-devel git java-1.8.0-openjdk-devel
sudo yum update -y && sudo yum upgrade -y && sudo yum install -y \
apache-maven sqlite-devel git java-1.8.0-openjdk-devel
sudo alternatives --set java /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java
sudo alternatives --set javac /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/javac
git clone https://github.com/awslabs/dynamodb-titan-storage-backend.git
Expand Down
49 changes: 49 additions & 0 deletions src/test/resources/copyGraph.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
#!/bin/bash
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add license header

wget https://github.com/amcp/dynamodb-import-export-tool/archive/createTable.zip
unzip -q createTable.zip
rm createTable.zip
pushd dynamodb-import-export-tool-createTable
mvn install
pushd target

export ORIGINAL_TABLE_ENDPOINT=$1
export ORIGINAL_TABLE_REGION=$2
export ORIGINAL_TABLE_PREFIX=$3
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

take in old table name and new table name as parameters and do not iterate over all graph stores in this file. Need flexibility to support the upgrade path from Titan 1.0.0 to JanusGraph 0.1.0 given the titan_ids/janusgraph_ids table name difference.

export NEW_TABLE_ENDPOINT=$4
export NEW_TABLE_REGION=$5
export NEW_TABLE_PREFIX=$6

declare -a tableSuffixes=("edgestore" "graphindex" "system_properties" "systemlog" "titan_ids" "txlog")

for tableSuffix in "${tableSuffixes[@]}"; do
ORIGINAL_TABLE_NAME="${ORIGINAL_TABLE_PREFIX}_${tableSuffix}"
NEW_TABLE_NAME="${NEW_TABLE_PREFIX}_${tableSuffix}"
# prepare graph for cross-region replication
# https://github.com/awslabs/dynamodb-cross-region-library#step-1-optional-table-copy-bootstrapping-existing-data

# enable stream in source tables
aws dynamodb update-table \
--endpoint ${ORIGINAL_TABLE_ENDPOINT} \
--table-name ${ORIGINAL_TABLE_NAME} \
--stream-specification "StreamEnabled=true,StreamViewType=NEW_AND_OLD_IMAGES"

# bootstrap destination tables
java -jar dynamodb-import-export-tool-1.1.0.jar \
--sourceSigningRegion ${ORIGINAL_TABLE_REGION} \
--sourceEndpoint ${ORIGINAL_TABLE_ENDPOINT} \
--sourceTable ${ORIGINAL_TABLE_NAME} \
--destinationSigningRegion ${NEW_TABLE_REGION} \
--destinationEndpoint ${NEW_TABLE_ENDPOINT} \
--destinationTable ${NEW_TABLE_NAME} \
--readThroughputRatio 0.5 \
--writeThroughputRatio 1 \
--createDestination

echo "copied table ${ORIGINAL_TABLE_NAME} at ${ORIGINAL_TABLE_ENDPOINT} to ${NEW_TABLE_NAME} at ${NEW_TABLE_ENDPOINT}"
done

popd
popd

# clean up
rm -rf dynamodb-import-export-tool-createTable