Extract Block Scanner Report metrics system

[anjackson]

The current Hadoop 0.20 task to gather the block scanner reports is in https://github.com/ukwa/ukwa-manage/blob/5b87cfcc71024d870d5db525df0a850b2571882c/tasks/preserve/hdfs_scan_status.py

It relies on this format:

[root@sh ~]# curl -x '' http://grunt35.bl.uk:50075/blockScannerReport

Total Blocks                 : 422010
Verified in last hour        :    589
Verified in last day         :  14520
Verified in last week        : 100790
Verified in last four weeks  : 397813
Verified in SCAN_PERIOD      : 298489
Not yet verified             :   2566
Verified since restart       : 817354
Scans since restart          : 800675
Scan errors since restart    :      4
Transient scan errors        :      0
Current scan rate limit KBps :   8192
Progress this period         :     71%
Time left in cur period      :  28.53%

But under Hadoop 3 it looks like:

$ curl -x '' http://192.168.45.196:9864/blockScannerReport

Block Scanner Statistics

Block scanner information for volume DS-15253685-106e-4691-b0d8-4ba8224c9db3 with base path /mnt/hdfsdsk33/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-1cc6e80c-31f6-4854-be56-07e0993e3b96 with base path /mnt/hdfsdsk11/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-27e22f96-c5b4-49eb-99b5-86768099650e with base path /mnt/hdfsdsk10/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-2e6fa36f-b474-4c7b-9faf-cc1a04a8b694 with base path /mnt/hdfsdsk21/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-37b9cfe0-27d4-40f3-8f6e-39a26fcf436f with base path /mnt/hdfsdsk31/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-40832b86-8026-4e8c-a525-638edd6c5722 with base path /mnt/hdfsdsk14/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-43f51c63-ac61-479a-9d8b-d150869501a6 with base path /mnt/hdfsdsk25/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-45e52361-3f2d-4da1-92d5-4e75585fd19d with base path /mnt/hdfsdsk5/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-465309fe-9c9d-4496-875d-56a631a9a67c with base path /mnt/hdfsdsk13/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-4fd36fe0-6af0-4a5a-a8b9-1712a9da9d2b with base path /mnt/hdfsdsk3/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-51bc8627-2e5c-40b2-9ee6-2f58b1354782 with base path /mnt/hdfsdsk15/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-56006aac-473e-45cf-8822-2a5b08eb74eb with base path /mnt/hdfsdsk17/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-5bde4f3a-384a-470f-b1cd-496e4ee873b2 with base path /mnt/hdfsdsk30/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-610ae78f-264f-4d0f-a510-1c9dcb7a41b6 with base path /mnt/hdfsdsk36/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-66c64f8e-e49d-4d01-a752-c5083510417c with base path /mnt/hdfsdsk35/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-694a61ff-6e6f-4ff9-bc14-e2163e6b0263 with base path /mnt/hdfsdsk6/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-75c14a3e-9d09-41e4-b8ac-cf5512a342d0 with base path /mnt/hdfsdsk18/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-7770db12-aa0e-4945-adc5-fa31e134358b with base path /mnt/hdfsdsk12/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-7e4b4be5-47d9-4b4f-ba1f-e1378d3b0034 with base path /mnt/hdfsdsk29/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-8242a5b6-b4ad-43cf-9d80-ffa92815ffbe with base path /mnt/hdfsdsk34/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-8de00832-1065-4265-a7e6-7034df02909f with base path /mnt/hdfsdsk27/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-91f795cc-6d44-4e70-85c7-a46b73349b20 with base path /mnt/hdfsdsk7/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-9455347b-e84a-4799-8c6a-21939726d47c with base path /mnt/hdfsdsk19/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-a23494ca-9698-4cda-ba03-d58ae748bd1d with base path /mnt/hdfsdsk24/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-a7324585-2b24-4304-aa6f-56310d0b8380 with base path /mnt/hdfsdsk23/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-b0cc99c7-b0aa-4234-928d-a915f341c883 with base path /mnt/hdfsdsk1/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-b21dbe0f-c0fb-4d32-aab5-32ad8d433f89 with base path /mnt/hdfsdsk20/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-b33151b5-1264-4a14-b3f5-c6687de30ce7 with base path /mnt/hdfsdsk2/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-b4153f2e-fde3-4d0f-b7ef-de9b2555c0e4 with base path /mnt/hdfsdsk9/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-c1568a9d-95ad-469a-8463-638b3baa41aa with base path /mnt/hdfsdsk8/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-c6ee3fa3-6c76-48ba-8eaa-ab950cfd9796 with base path /mnt/hdfsdsk22/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-cddcd78d-368f-48eb-a7f2-fc24ad16ced8 with base path /mnt/hdfsdsk26/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-f5caed1f-3494-4d24-87a5-09ac9e968e53 with base path /mnt/hdfsdsk4/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-fb77290d-b73e-4fc8-8f32-884205478954 with base path /mnt/hdfsdsk32/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-fd8a3830-cb6c-4ff5-8c4e-2134344e3d6a with base path /mnt/hdfsdsk16/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

Block scanner information for volume DS-fecdc1b8-2a40-4336-8163-bfa9915d3ff1 with base path /mnt/hdfsdsk28/hadoop3
Bytes verified in last hour       :                                                         0
Blocks scanned in current period  :                                                         0
Blocks scanned since restart      :                                                         0
Block pool scans since restart    :                                                         0
Block scan errors since restart   :                                                         0
Hours until next block pool scan  :                                                   117.000
Last block scanned                :                                                      none
More blocks to scan in period     :                                                     false

We should extract this into a standalone tool to be run from Airflow.