Weather Data Plots #168

petramika · 2020-04-30T17:59:18Z

Feature to show weather plots data per county for the last 30 days.

This Feature is related to #32

For further documentation of how does this work, please refer to: https://github.com/strivelabs/ca_visit_tracking/wiki

For live test deployment, check: https://visitdata-275208.wl.r.appspot.com/index.html, if you get a CORS problem, please check a chrome extension as "Moesif Orign & CORS Changer"

A static html button has been added to the template for county. Also some css has been removed for a correctly bootstrap css display.

When the checkbox is pressed will be activated a filter to add the weather info. Its needed to change the chart to draw the data.

to not to redo the logic once return is extracted, the use of if-else will not compute the (consult)

now text in label weather is inside for smaller views added previous classes for mobile view

Some dummy data is added to check that it works

useful for drawing areachart plots

- Moving code to an external file to keep code organised - Drawing dynamic plot per county with precipitations and temperature data

yaxis has been added, colors improved, datestamp improved, texts and titles improved, margins added

p tag not needed and series already defined in chart.

when the div was not created for states, program was throwing an error. Now fixed with jquery

- All the information is not in one plot, if I have 3 counties it would be 9 lines, too much - X Axis when one county is selected will 100% match - The correlations are perfectly visible - Adding another Y axis data is no effort and will not compromise the main plot

Now a weather var has been created in case that some new changes are done for result. In this case this should not affect to the weather data.

Data is represented if a venue type is not selected.

Feature/300420 weather info

dsjoerg · 2020-04-30T18:34:03Z

Hi, you are probably seeing that app.yaml conflicts here?

petramika · 2020-04-30T19:36:22Z

Hi, you are probably seeing that app.yaml conflicts here?

Hi! I just saw the new data version, you are too quick! ;) I fix it!

dsjoerg · 2020-05-05T15:26:02Z

This is very cool! I'm looking at https://visitdata-275208.wl.r.appspot.com/bydatesel/California/ALL/Beach

I would rather see only high temperature rather than the low-high range, to avoid overwhelming the viewer/user. Nevertheless it's very cool! Continuing to play with it...

dsjoerg · 2020-05-05T15:30:25Z

One weird thing:

go to https://visitdata-275208.wl.r.appspot.com/bydatesel/California/Orange%20County/ALL
click on the Weather Data checkbox
now only one timeseries will be displayed, and it's not labeled to indicate which one it is

It's probably "Shops & Services" that's being displayed but there's no way for the user to know that.
A few ways to fix this, whichever way is easiest for you is fine by me.

dsjoerg · 2020-05-05T15:31:51Z

And minor issue but call it "Precipitation" not "Precipitations"

petramika · 2020-05-05T16:16:02Z

@dsjoerg

I would rather see only high temperature rather than the low-high range

What is more useful for clients?

A) As a line
B) Or as a column

dsjoerg · 2020-05-05T16:41:19Z

I think as a line is better, but I think it should be a separate chart not directly overlapping the visit data, because all the superimposed data makes it hard to read. Example: https://www.highcharts.com/demo/synchronized-charts

…

On Tue, May 5, 2020 at 12:16 PM petramika ***@***.***> wrote: @dsjoerg <https://github.com/dsjoerg> I would rather see only high temperature rather than the low-high range What is more useful for clients? A) As a line <https://i.ibb.co/M537p4N/Screenshot-2020-05-05-at-18-03-53.png> B) Or as a column <https://i.ibb.co/Mpsk7cC/Screenshot-2020-05-05-at-18-08-43.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF7HKLNJ3O777YX7MZJTCTRQA3VFANCNFSM4MWDE3JQ> .

Features/weather anne 050520

petramika · 2020-05-07T14:52:14Z

Hi David, you have the fixes that you wanted on live. Just one thing, I left it at the moment as a line, I didn’t separate it into several plots. This was the first approach that I wanted to take, the problem is that the JSON that I had to do it was making that visits data or weather data was lost, I needed to have all the data with the timestamps filled, if not it was not possible. This is why I followed this solution: https://www.highcharts.com/demo/combo-multi-axes <https://www.highcharts.com/demo/combo-multi-axes> It even looks that this plot was designed for the specific issue that we are facing here :) Cheers.

…

On 5 May 2020, at 18:41, David Joerg ***@***.***> wrote: I think as a line is better, but I think it should be a separate chart not directly overlapping the visit data, because all the superimposed data makes it hard to read. Example: https://www.highcharts.com/demo/synchronized-charts On Tue, May 5, 2020 at 12:16 PM petramika ***@***.***> wrote: > @dsjoerg <https://github.com/dsjoerg> > > I would rather see only high temperature rather than the low-high range > > What is more useful for clients? > > A) As a line > <https://i.ibb.co/M537p4N/Screenshot-2020-05-05-at-18-03-53.png> > B) Or as a column > <https://i.ibb.co/Mpsk7cC/Screenshot-2020-05-05-at-18-08-43.png> > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <#168 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAF7HKLNJ3O777YX7MZJTCTRQA3VFANCNFSM4MWDE3JQ> > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOAHCKAWRQ333AOKDJBAITRQA6T7ANCNFSM4MWDE3JQ>.

markroth8 · 2020-05-09T16:22:16Z

etl/dags/extract_weather_data.py

+
+BASE_URL = "http://api.weatherapi.com/v1/history.json?key={}&q={}+united+states&dt={}"
+# Get a weatherapi.com api key
+API_KEY = os.environ.get("API_WEATHER_KEY", "a70a4e2736644cdcb9d85348202404")


Keys must never be committed to version control. This must be removed and the history squashed before merging. We can register the API key as an airflow secret.

markroth8 · 2020-05-09T16:22:49Z

etl/dags/extract_weather_data.py

+BASE_URL = "http://api.weatherapi.com/v1/history.json?key={}&q={}+united+states&dt={}"
+# Get a weatherapi.com api key
+API_KEY = os.environ.get("API_WEATHER_KEY", "a70a4e2736644cdcb9d85348202404")
+BUCKET_NAME = os.environ.get("BUCKET_NAME", "default")


Should fail if env not set, to force correct configuration, instead of silently choosing an invalid bucket name.

markroth8 · 2020-05-09T16:26:38Z

etl/dags/extract_weather_data.py

+            cached_data = stated_cached_data.get(county, {})
+            data[county] = always_merger.merge(cached_data, api_data)
+
+        state_blob = bucket.blob("{}.json".format(selected_state))


Proposed: Let's use gs://data.visitdata.org/processed/vendor/api.weatherapi.com/asof/yyyymmdd/*.json to store this data.

markroth8 · 2020-05-09T16:29:11Z

scripts/config.py

@@ -0,0 +1,7 @@
+BASE_URL = "http://api.weatherapi.com/v1/history.json?key={}&q={}+united+states&dt={}"
+API_KEY = "a70a4e2736644cdcb9d85348202404"


API keys must never be committed to version control. This must be removed and squashed before merging.

markroth8 · 2020-05-09T16:36:51Z

scripts/weather.py

+
+    return weather
+
+def __load_state_file(state):


dunders are typically reserved for the Python Core API team. A single _ prefix would suffice here. See https://amontalenti.com/2013/04/11/python-double-under-double-wonder

markroth8 · 2020-05-09T16:50:22Z

README.md

+ `BUCKET_NAME` variable must be defined in the airflow server
+
+Once the data is stored in the bucket, we can get the weather data for one state in the route 
+`weather/<state>`


We might want to consolidate all of this under https://visitdata.org/data/ and present it as a single bundle that all goes together.

@markroth8 the only issue is that it would increase the size of the initial data load for each page before it renders. If we assume that in most cases, people will be looking at the visits and not the weather, it would be better to load it only upon request.
In general if we plan to add more data sources which might be only displayed based on user's choices in the UI, the architecture of separating those data sources might be better?

markroth8 · 2020-05-09T16:51:30Z

app.yaml

@@ -2,4 +2,5 @@ runtime: python37
 entrypoint: gunicorn -b :$PORT main:app

 env_variables:
-  FOURSQUARE_DATA_VERSION: "20200503-v0"
+  FOURSQUARE_DATA_VERSION: "20200504-v0"
+  BUCKET_NAME: "vd-weather-data"


Let's add a WEATHERAPI_DATA_VERSION variable to set the as of date for the weather data as part of the deployment. We can use the same bucket (data.visitdata.org).

markroth8 · 2020-05-09T16:55:28Z

etl/requirements.txt

@@ -1,2 +1,4 @@
 apache-airflow==1.10.10
 google-cloud-storage==1.27.0
+requests
+deepmerge


Pin version in requirements.txt so we know we're running the same thing.

markroth8 · 2020-05-09T16:56:18Z

main.py

@@ -143,6 +145,13 @@ def data(path):
                           snapshot_id=app_state['foursquare_data_version'])


+@app.route("/weather/<state>")


Maybe we can just get this from /data/weather/<state>

markroth8 · 2020-05-09T16:56:44Z

requirements.txt

 google-cloud-storage==1.27.0
 pyyaml==5.3.1
+requests
+deepmerge


Pin version in requirements.txt so we know we're all running the same thing.

markroth8 · 2020-05-09T16:57:48Z

I've added some review comments. I can help with them, as well.

anne and others added 30 commits April 16, 2020 14:07

Added checkbox weather and fixed layout

1134def

A static html button has been added to the template for county. Also some css has been removed for a correctly bootstrap css display.

Added script to retrieve data from weather API

fd39e45

Added functions to retrieve weather data for each state

785b046

Added config file

fc358b4

added cron task to get weather data

89ea987

Added route to get weather data

91f1389

Added weather DAG

2987373

Added functions to download weather data locally or from the cloud

ed4b19c

Added config to weather route

5962bf5

Added documentation about weather data

f8ee0e1

Fixed get weather task

6593686

Fixed get weather functionality

5913826

Fixed weather route and requirements

8df1b63

logic ready to work with checkbox

44d0de3

When the checkbox is pressed will be activated a filter to add the weather info. Its needed to change the chart to draw the data.

improved if efficiency

3927b28

to not to redo the logic once return is extracted, the use of if-else will not compute the (consult)

improved mobile view

7619b82

now text in label weather is inside for smaller views added previous classes for mobile view

draw weather with test data

4e07afe

Some dummy data is added to check that it works

added highchart more library

e84d1b0

useful for drawing areachart plots

major changes

99cd54f

- Moving code to an external file to keep code organised - Drawing dynamic plot per county with precipitations and temperature data

formatted weather highchart plot

fbefd2d

yaxis has been added, colors improved, datestamp improved, texts and titles improved, margins added

if checkbox desactivated weather removed

ee2d54a

increased top space between weather plots

4e5661b

changed label text

ac882b0

improved boostrap position options buttons

15305a9

remove unnecessary data

f93da2f

p tag not needed and series already defined in chart.

improved showing one or several charts for counties

741ae08

refractor code, organise functions

0037ebc

fixed exception id not found

d7c0360

when the div was not created for states, program was throwing an error. Now fixed with jquery

Added visists data

d9e0f75

- All the information is not in one plot, if I have 3 counties it would be 9 lines, too much - X Axis when one county is selected will 100% match - The correlations are perfectly visible - Adding another Y axis data is no effort and will not compromise the main plot

improved doc

5e4b991

Anne Munoz and others added 7 commits April 29, 2020 17:36

merge weather upstream master

de92759

visits % to #. Improved code corrections

d627953

merge weather upstream master 30/04/20

3c80ae3

weather data var created

650709b

Now a weather var has been created in case that some new changes are done for result. In this case this should not affect to the weather data.

weather data for non selected venue type

22ebba7

Data is represented if a venue type is not selected.

rounded y visits value

0ee5f84

Merge pull request #1 from strivelabs/feature/300420-weather-info

90a6e38

Feature/300420 weather info

update 300420

30049e9

an6eel and others added 4 commits May 4, 2020 09:27

update requirements

5b9b5e7

update 040520

db6f0af

update 050520

7452675

app.yaml update 050520

fd01848

VisitData-org deleted a comment from petramika May 5, 2020

anne munoz added 2 commits May 5, 2020 18:03

weather.js: precipitations typo fixed

4caf68a

weahter.js: weather remove min temperatures

9577f21

anne munoz and others added 4 commits May 5, 2020 19:38

Added Venue Type to title

5f6c636

Merge pull request #3 from strivelabs/features/weather-anne-050520

24361cb

Features/weather anne 050520

20200504

a1cdc80

update 070520

85c0546

ajanian mentioned this pull request May 8, 2020

Indicate weather on the chart somehow #32

Closed

markroth8 requested changes May 9, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Weather Data Plots #168

Weather Data Plots #168

petramika commented Apr 30, 2020 •

edited

Loading

dsjoerg commented Apr 30, 2020

petramika commented Apr 30, 2020

dsjoerg commented May 5, 2020

dsjoerg commented May 5, 2020

dsjoerg commented May 5, 2020

petramika commented May 5, 2020

dsjoerg commented May 5, 2020 via email

petramika commented May 7, 2020 via email

markroth8 May 9, 2020

markroth8 May 9, 2020

markroth8 May 9, 2020

markroth8 May 9, 2020

markroth8 May 9, 2020

markroth8 May 9, 2020

anusheel May 10, 2020

markroth8 May 9, 2020

markroth8 May 9, 2020

markroth8 May 9, 2020

markroth8 May 9, 2020

markroth8 commented May 9, 2020

		@@ -0,0 +1,7 @@
		BASE_URL = "http://api.weatherapi.com/v1/history.json?key={}&q={}+united+states&dt={}"
		API_KEY = "a70a4e2736644cdcb9d85348202404"

		@@ -143,6 +145,13 @@ def data(path):
		snapshot_id=app_state['foursquare_data_version'])


		@app.route("/weather/<state>")

Weather Data Plots #168

Are you sure you want to change the base?

Weather Data Plots #168

Conversation

petramika commented Apr 30, 2020 • edited Loading

dsjoerg commented Apr 30, 2020

petramika commented Apr 30, 2020

dsjoerg commented May 5, 2020

dsjoerg commented May 5, 2020

dsjoerg commented May 5, 2020

petramika commented May 5, 2020

dsjoerg commented May 5, 2020 via email

petramika commented May 7, 2020 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markroth8 commented May 9, 2020

petramika commented Apr 30, 2020 •

edited

Loading