Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for new wema and platform config formats #57

Draft
wants to merge 72 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
72 commits
Select commit Hold shift + click to select a range
d19b7af
Update dependencies
darrenhunt2 Jul 4, 2022
a4e185d
Update Github Actions, Python version
darrenhunt2 Jul 4, 2022
c80f481
Improve formatting
darrenhunt2 Jul 4, 2022
f82c1ec
Finish docstrings in handler; update readme draft
darrenhunt2 Jul 4, 2022
b5f7664
Delete unzip_requirements.py
darrenhunt2 Jul 4, 2022
b128817
Delete serverless-deploy-helpers directory
darrenhunt2 Jul 4, 2022
6071af6
Delete de421.bsp
darrenhunt2 Jul 4, 2022
35993f9
Delete examples.py
darrenhunt2 Jul 4, 2022
44728ab
Add back deleted file
darrenhunt2 Jul 4, 2022
57a716f
Delete de421.bsp
darrenhunt2 Jul 4, 2022
6cfffab
Add docstrings to several files
darrenhunt2 Jul 5, 2022
2b544ff
Add docstrings and cleanup to db.py and remotehq_helpers.py
darrenhunt2 Jul 6, 2022
a8aeef6
Remove unused weather.py
darrenhunt2 Jul 8, 2022
f46e330
Combine Github Actions into one file
darrenhunt2 Jul 8, 2022
060ce15
Syntax
darrenhunt2 Jul 8, 2022
a4f8dfb
Readme updates; minor docstring formatting
darrenhunt2 Jul 12, 2022
c8dd96b
Missing word
darrenhunt2 Jul 12, 2022
cd3f653
Fix typo in readme
darrenhunt2 Jul 12, 2022
13120f5
Minor changes according to review comments
darrenhunt2 Jul 12, 2022
996882a
Fix wrong version for a Python requirement
darrenhunt2 Jul 12, 2022
c926260
Small changes to the Actions file
darrenhunt2 Jul 12, 2022
57835e8
Missing dollar sign in front of variable
darrenhunt2 Jul 12, 2022
1f8a4ae
Add back & update Dockerfile to reduce deployment size
darrenhunt2 Jul 13, 2022
3d767a8
Attempt 1 at de-fzing a fits file to download
mfitzasp Sep 29, 2022
a5dd35c
Revert "Bz2 tofz"
mfitzasp Sep 29, 2022
3b72207
adjusted file pointers from bz2 to fz
mfitzasp Sep 29, 2022
d076ff0
added in helper functions
mfitzasp Sep 29, 2022
9f2c75e
removed stretch from save_fz_to_fits function
mfitzasp Sep 30, 2022
147836b
fixed up the fits file naming strings in save_fz_to_fits
mfitzasp Sep 30, 2022
45e3ea2
strangely had the one important function commented out
mfitzasp Sep 30, 2022
29072cb
hunting formatting of filename in save_fz_to_fits
mfitzasp Sep 30, 2022
18805aa
updating tif function to use fz file
mfitzasp Sep 30, 2022
31ad9d4
updating tif function to use fz file #1
mfitzasp Sep 30, 2022
79686be
removed a bz2 reference for tif making
mfitzasp Sep 30, 2022
692cc67
a + rather than a '. So embarassing.
mfitzasp Sep 30, 2022
d3e1864
Now figuring out what is occuring with the small fits file
mfitzasp Sep 30, 2022
58d9259
Removed logging code and added in some function comments
mfitzasp Sep 30, 2022
d778177
Add simple change to test deploy
timbeccue Feb 15, 2023
24a4840
Update output syntax
timbeccue Feb 15, 2023
b8b764f
Change broken job env
timbeccue Feb 15, 2023
7817390
Try differnt env access
timbeccue Feb 15, 2023
4b0b623
Change env routine
timbeccue Feb 15, 2023
edfac03
Combine the determine-stage and deploy jobs
timbeccue Feb 15, 2023
841fc72
Clean up deploy script
timbeccue Feb 15, 2023
7686086
Update aws credential job to use node 16
timbeccue Feb 15, 2023
93fca25
Remove Dockerfile previously used for deploys
timbeccue Feb 16, 2023
5742e30
Remove unused serverless plugins
timbeccue Feb 16, 2023
86f2283
Add delete endpoint for night log
timbeccue Feb 22, 2023
cbdf5a8
Add custom ttl to night log notes
timbeccue Feb 22, 2023
e0026c8
Fix bug with nightlog note ttl type
timbeccue Feb 22, 2023
dc16347
Fix bug with response when getting night log
timbeccue Feb 22, 2023
769ec4d
Change http response for empty night log queries
timbeccue Feb 22, 2023
6632f9f
Return no_content http status if no info image exists
timbeccue Feb 22, 2023
26afbdd
Update test to reflect change in http response
timbeccue Feb 22, 2023
8ebc6a2
Update github actions checkout to v3
timbeccue Feb 22, 2023
ec35671
Update sqlalchemy lambda layer
timbeccue Mar 2, 2023
b103005
Merge branch 'main' into dev
timbeccue Apr 11, 2023
8b824e0
Add endpoints for wema/platform configs
timbeccue Apr 1, 2023
57afdac
Add pytest_mock dependency
timbeccue Apr 1, 2023
f610cfa
Update requirements
timbeccue Apr 1, 2023
0de0d66
Validate wema/platform configs
timbeccue Apr 1, 2023
d48e409
Update config validation and clean serverless.yml
timbeccue Apr 11, 2023
09bfb93
Fix dynamodb resource loading
timbeccue Apr 11, 2023
94faae1
Merge pull request #56 from LCOGT/feature/new-configs
timbeccue Apr 11, 2023
d2d803f
Fix package-lock out of sync
timbeccue Apr 11, 2023
929eebb
Update python requirements
timbeccue Apr 11, 2023
8b4f1e7
Convert floats to decimals for dynamodb
timbeccue Apr 11, 2023
9d1d22f
Add missing json dumps before writing to ddb
timbeccue Apr 11, 2023
c9265f5
Update tests to account for serialization
timbeccue Apr 11, 2023
49af747
Update platform config validation
timbeccue Apr 11, 2023
c08e872
Add another platform config to tests
timbeccue Apr 11, 2023
5252359
Change response format for associated wema+platforms
timbeccue Apr 11, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ var/

# Serverless directories
.serverless
.dynamodb

# pycache
__pycache__
Expand Down
248 changes: 248 additions & 0 deletions api/configs/configs.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,248 @@
import json
import jsonschema
from http import HTTPStatus
import boto3

from api.helpers import dynamodb_r
from api.helpers import DecimalEncoder
from api.helpers import json_dumps_ddb
from api.configs.validation_schemas import wema_config_schema
from api.configs.validation_schemas import platform_config_schema

table = dynamodb_r.Table("WemaPlatformConfigs")

######################################
###### DynamoDB Table Design ########
######################################

"""
TABLE STRUCTURE

Store the data in the table:

To store a WEMA config:
ConfigID: [wema_id]
ConfigType: WEMA
Config: {WEMA JSON}
To store a platform:

To store a platform config:
ConfigID: [platform_id]
ConfigType: PLATFORM
Config: {Platform JSON}
WemaID: [Associated wema_id] (Store the associated wema_id as an attribute in the platform item)
"""

"""
QUERY AND WRITE PATTERNS

Get a WEMA:
Query by the partition key (ConfigID = [wema_id]) and sort key (ConfigType = WEMA).

Get a WEMA and all its platforms:
Scan the table for items with the sort key ConfigType = PLATFORM and filter the results by the wema_id attribute.

Get a platform and its associated WEMA:
Query the platform by the partition key (ConfigID = [platform_id]) and sort key (ConfigType = PLATFORM).
Get the associated WEMA by querying the partition key (ConfigID = [Associated wema_id]) and
sort key (ConfigType = WEMA).

Write a WEMA:
Insert a new item with the partition key (ConfigID = [wema_id]), sort key (ConfigType = WEMA), and the
WEMA JSON as the Config attribute.

Write a platform:
Insert a new item with the partition key (ConfigID = [platform_id]), sort key (ConfigType = PLATFORM),
the platform JSON as the Config attribute, and the associated wema_id.
"""


def filter_json(data, schema):
if 'properties' in schema:
filtered_data = {}
for key, subschema in schema['properties'].items():
if key in data:
filtered_data[key] = filter_json(data[key], subschema)
return filtered_data
elif 'type' in schema and schema['type'] == 'array' and 'items' in schema:
return [filter_json(item, schema['items']) for item in data]
else:
return data

#####################################
##### Query and Write Methods #####
#####################################

def get_wema(wema_id):
response = table.get_item(
Key={
'ConfigID': wema_id,
'ConfigType': 'WEMA'
}
)
return response.get('Item')

def get_wema_and_all_platforms(wema_id):
response = table.scan(
FilterExpression="ConfigType = :platform AND WemaID = :wema_id",
ExpressionAttributeValues={
":platform": "PLATFORM",
":wema_id": wema_id
}
)
wema = get_wema(wema_id)
platforms_array = response.get('Items')
platforms_dict = {p["ConfigID"]: p["Config"] for p in platforms_array}
return wema, platforms_dict

def get_platform_and_associated_wema(platform_id):
response = table.get_item(
Key={
'ConfigID': platform_id,
'ConfigType': 'PLATFORM'
}
)
platform = response.get('Item')
if platform:
wema_id = platform.get('WemaID')
wema = get_wema(wema_id)
return platform, wema
return None, None

def write_wema(wema_id, wema_config):
table.put_item(
Item={
'ConfigID': wema_id,
'ConfigType': 'WEMA',
'Config': wema_config
}
)

def write_platform(platform_id, wema_id, platform_config):
table.put_item(
Item={
'ConfigID': platform_id,
'ConfigType': 'PLATFORM',
'WemaID': wema_id,
'Config': platform_config
}
)

def get_all_wemas():
response = table.scan(
FilterExpression="ConfigType = :wema",
ExpressionAttributeValues={
":wema": "WEMA"
}
)
wemas = response.get('Items')
wemas_dict = {wema['ConfigID']: wema['Config'] for wema in wemas}
return wemas_dict


############################
###### Handlers ###########
############################

def get_wema_handler(event, context):
wema_id = event['pathParameters']['wema_id']
wema = get_wema(wema_id)

if wema:
return {
'statusCode': HTTPStatus.OK,
'body': json.dumps(wema)
}
else:
return {
'statusCode': HTTPStatus.NOT_FOUND, # 404
'body': 'WEMA not found'
}

def get_wema_and_all_platforms_handler(event, context):
wema_id = event['pathParameters']['wema_id']
wema, platforms = get_wema_and_all_platforms(wema_id)

if wema:
return {
'statusCode': HTTPStatus.OK, # 200
'body': json.dumps({
'wema': wema,
'platforms': platforms
})
}
else:
return {
'statusCode': HTTPStatus.NOT_FOUND, # 404
'body': 'WEMA not found'
}

def get_platform_and_associated_wema_handler(event, context):
platform_id = event['pathParameters']['platform_id']
platform, wema = get_platform_and_associated_wema(platform_id)

if platform:
return {
'statusCode': HTTPStatus.OK, # 200
'body': json.dumps({
'platform': platform,
'wema': wema
})
}
else:
return {
'statusCode': HTTPStatus.NOT_FOUND, # 404
'body': 'Platform not found'
}

def write_wema_handler(event, context):
request_body = json.loads(event['body'])
wema_config = request_body['config']
wema_id = request_body['wema_id']

# Validate the config against the JSON schema
try:
jsonschema.validate(wema_config, wema_config_schema)
except jsonschema.ValidationError as e:
return {
'statusCode': HTTPStatus.BAD_REQUEST, # 400
'body': str(e)
}

simple_wema_config = filter_json(wema_config, wema_config_schema)
write_wema(wema_id, json_dumps_ddb(wema_config))

return {
'statusCode': HTTPStatus.CREATED, # 201
'body': 'WEMA created'
}

def write_platform_handler(event, context):
request_body = json.loads(event['body'])
platform_id = request_body['platform_id']
wema_id = request_body['wema_id']
platform_config = request_body['config']

# Validate the config against the JSON schema
try:
jsonschema.validate(platform_config, platform_config_schema)
except jsonschema.ValidationError as e:
return {
'statusCode': HTTPStatus.BAD_REQUEST, # 400
'body': str(e)
}

write_platform(platform_id, wema_id, json_dumps_ddb(platform_config))

return {
'statusCode': HTTPStatus.CREATED, # 201
'body': 'Platform created'
}

def get_all_wemas_handler(event, context):
wemas = get_all_wemas()

return {
'statusCode': HTTPStatus.OK, # 200
'body': json.dumps(wemas)
}
Loading