Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix provider updaters #149

Merged
merged 17 commits into from
Sep 26, 2016
Merged

Fix provider updaters #149

merged 17 commits into from
Sep 26, 2016

Conversation

joe4dev
Copy link
Collaborator

@joe4dev joe4dev commented Sep 23, 2016

  • Update/remove outdated AzureUpdaterTest
  • Test Amazon updater on Windows => does it work on rake db:reset now?

Fixes #139

Options to fix the Azure updater

  1. Write a new crawler for the updated pricing Web page
  2. Combine the data from API somehow with the pricing Web page
  3. Query a 3rd party service that maintains their own crawlers or manually updates prices

* This might have caused the Amazon updater to fail
  on Windows the first on `rake db:reset`
Usage:

* `rake jobs:update_providers`
* `rake jobs:update_providers_async`
* No pricing data included in API responses !
* Region area is not detected yet

Options:
a) Write a new crawler for the updated pricing Web page
b) Combine the data from API somehow with the pricing Web page
c) Query a 3rd party service that maintains their own crawlers or manually updates prices
@inz inz temporarily deployed to fathomless-escarpment-2-pr-149 September 23, 2016 22:37 Inactive
@joe4dev
Copy link
Collaborator Author

joe4dev commented Sep 23, 2016

Trying to query the endpoints for the pricing calculator: https://azure.microsoft.com/en-us/pricing/calculator/
Let's see how long this endpoints remains unprotected ;)

Regions

Request: GET https://azure.microsoft.com/api/v1/regions/
Response: regions.txt

Virtual machines

Request: GET https://azure.microsoft.com/api/v1/pricing/virtual-machines/calculator/?culture=en-us
Response: virtual-machines.txt

Using the 3 major provider produces a nicer curve
@inz inz temporarily deployed to fathomless-escarpment-2-pr-149 September 24, 2016 00:37 Inactive
* Add comment about speedup using a transaction
The `MeanTimeReporter` is very helpful to identify slow test,
which is particularly useful when optimizing code.
We currently do not consider storage at all.
Whenever we need it back, we have it in the Git history.
Provides a helper method that generalizes the heuristic
to identify region areas.
* Use the `RegionArea` module to reduce duplications across provider updaters
* Break class into smaller methods
* Add transaction to make creating 428 resources ~0.4 seconds faster
* Remove storage mocks from test (the mock files are still left in fixtures)
* Raise error instead of fail silently :(
* Add useful test assertions (e.g., check for unmatched region area)
This speeds up bulk updates. Needed to split up some long methods into a `update_compute` method.

* Also fixed the prefix constant causing issues when running multiple tests
* Some minor test refactorings
@inz inz temporarily deployed to fathomless-escarpment-2-pr-149 September 24, 2016 18:06 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Fix broken Provider updaters
3 participants