-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #4 from LaFermeDuMineur/evol/hashratena_scrapping
Evol/hashratena scrapping
- Loading branch information
Showing
16 changed files
with
303 additions
and
34 deletions.
There are no files selected for viewing
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,51 +1,73 @@ | ||
| # 2miner-monitoring-go | ||
| # Minotor | ||
| This go application is an upgrade of the 2miner-monitoring. | ||
| His goal is to be able to crawl more data than the original 2miner-monitoring. | ||
|
|
||
| If you want to monitore only yourslef, you should considerate 2miner-monitoring. | ||
| If you want to monitore only yourself, you should considerate 2miner-monitoring. | ||
|
|
||
| This application harvest data from 2miners API (https://apidoc.2miners.com/#/) | ||
|
|
||
| It send the data to an elasticsearch and on my case grafana is used to retrieves visualisation | ||
|
|
||
| ## How to run it | ||
| ## Important Url | ||
|
|
||
| ### Config file | ||
| API => https://load01.ether-source.fr:8999 | ||
|
|
||
| Grafana => https://load01.ether-source.fr | ||
|
|
||
| ## How it works | ||
|
|
||
| ### The Backend | ||
|
|
||
| Gin is used to create a http server with many handler | ||
| Gocron is used to proc all call at a given time to harvest data | ||
|
|
||
| ### The FrontEnd | ||
|
|
||
| There is a grafana to display everything. No registration is needed for viewer only. | ||
| If you wanna do your own dashboard/alert, you need to register. | ||
|
|
||
| ### Service connected | ||
|
|
||
| - 2miners | ||
| - Hiveos | ||
| - Etherscan | ||
| - Hashrate.no | ||
| - Coingecko | ||
|
|
||
| You have ton configure the config.yaml file (default is config/config.yaml). All field is currently mandatory. | ||
| #### 2miners | ||
|
|
||
| ```yaml | ||
| --- | ||
| elasticsearch_user: "es_user" | ||
| elasticsearch_password: "es_password" | ||
| elasticsearch_hosts: ["host1:port", "host2:port"...] | ||
| api_token_etherscan: "etherscan_api_token " | ||
| log_level: INFO # CRITICAL => ERROR => WARNING => INFO => DEBUG | ||
| ca_path: "ssl/chain.pem" | ||
| 2miners_url: "https://eth.2miners.com/api" | ||
| To un/subscribe to the monitoring, you have only to do an api call | ||
| ````go | ||
| server.GET("/subscribe/:wallet", handlers.SuscribeWallet) | ||
| server.GET("/unsubscribe/:wallet", handlers.UnSuscribeWallet) | ||
| ```` | ||
|
|
||
| miner_listing: ADDR #ALL is used to get all miners from 2miners, ADDR is used for adress variable | ||
| adress: | ||
| - "0x........................................" | ||
| - "0x........................................" | ||
| - "0x........................................" | ||
| - "0x........................................" | ||
| After that, the server will collectd data from the given wallet | ||
|
|
||
| redis_host: 127.0.0.1 | ||
| redis_port: 6379 | ||
| redis_password: "" | ||
| redis_lifetime: 10 | ||
|
|
||
| factor: 0.000001 | ||
| ether_factor: 0.000000000000000001 | ||
| gas_factor: 0.000000001 | ||
| #### HiveOS | ||
|
|
||
| To have the Hiveos, the client have to allow **minotor** permission to the user **minotor_LFDM** | ||
|
|
||
| #### The Others | ||
|
|
||
| the others services are called on demand or periodically to populate the data | ||
|
|
||
| ## How to run it | ||
|
|
||
| ### Config file | ||
|
|
||
| You have ton configure the yaml files(adresses, cards and config). All field is currently mandatory. | ||
|
|
||
| ``` | ||
|
|
||
| ### Install requirement | ||
|
|
||
| GO >= 1.17, Elasticsearch (cluster is nice to have), grafana, redis. | ||
| Haproxy is a must | ||
|
|
||
|
|
||
| ### Run the binary | ||
|
|
||
| The binary is the brain of the process, everything is handle by him. Technically you don't need ES + REDIS + GRAFANA, that's just a must and tooling. | ||
|
|
||
| ### TroobleShooting |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| package config | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "gopkg.in/yaml.v2" | ||
| "io/ioutil" | ||
| ) | ||
|
|
||
| var Cards *CardsCrawl | ||
|
|
||
| type CardsCrawl struct { | ||
| CardsList []string `yaml:"cards"` | ||
| } | ||
|
|
||
| func LoadCardYamlConfig() { | ||
| t := CardsCrawl{} | ||
| data, err := ioutil.ReadFile(Cfg.CardsConfigFile) | ||
| if err != nil { | ||
| fmt.Println(err.Error()) | ||
| } | ||
| err = yaml.Unmarshal(data, &t) | ||
| Cards = &t | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,15 @@ | ||
| package data | ||
|
|
||
| type Card struct { | ||
| HashratePrimary float64 `json:"hashrate_primary"` | ||
| Income float64 `json:"income"` | ||
| HashrateAlt float64 `json:"hashrate_alt,omitempty"` | ||
| CoinName string `json:"coin_name"` | ||
| Timestamp string `json:"@timestamp"` | ||
| Card string `json:"card"` | ||
| CoinPrimary string `json:"primary_coin"` | ||
| UnitPrimary string `json:"primary_unit"` | ||
| UnitAlt string `json:"alt_unit,omitempty"` | ||
| CoinAlt string `json:"alt_coin,omitempty"` | ||
| Conso int `json:"conso"` | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| package handlers | ||
|
|
||
| import ( | ||
| "2miner-monitoring/thirdapp" | ||
| "github.com/gin-gonic/gin" | ||
| ) | ||
|
|
||
| func ScrapHashrateNo(c *gin.Context) { | ||
| code, status := thirdapp.RunCrawler() | ||
| c.String(code, status) | ||
|
|
||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| #Sample of adress config file | ||
| adress: | ||
| - 0x6e80e1e29efedj4951j25fdf198fjjd95f0d55je | ||
| - 0x989e5m50dfe878jm76e8de102557je51e094f820 | ||
| - 0x995e566121Eem744mjEDjEDej6f2855F54868Dm2 | ||
| - 0x54dj0mee029me72658d5m0f102dd12mf8m7e78e0 | ||
| - 0x52m488419mee0e0mmj02e45255171m55em854dej |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # sample of cards config file | ||
| cards: | ||
| - 3060 | ||
| - 3070 | ||
| - 3080 | ||
| - RX580 |
Oops, something went wrong.