Remove / change directories and files in raw_data and Level 1

[dlebauer]

Need to clean up or resolve the following anomalies. Some may require separate issues - changes that are more difficult can be postponed to the next release.

1. scanner3DTop directories
   • These should be removed / replaced w/ laser3d_* folders
   • related: Unexplained Reprocess of scanner3DTop Data computing-pipeline#412 and scanner3DTop data in SWIR dataset subdirectory computing-pipeline#389
2. ua-mac/Level_1/fullfield
   • it isn't clear - some of these have names like ir_stereovis_ir and others rgb_stereovis_ir ... does ir refer to flir? If so, these are distinct from the stereovis, which is only RGB.
   • are the *file_paths.json files intended for external use?
3. ua-mac/Level_1/canopy_height
   • probably should delete .... of the files in there: metadata.json stays in raw; hist_e.npy, hist_w.npy, top_e.npy, top_w.npy should be inserted into geostreams
4. ua-mac/Level_1/texture_analysis
   • this extractor has never been approved / doesn't generate useful output so directory should be removed
5. ua-mac/Level_1/uav
   • lets change this from uav/rededge to uav_redege_fullfield and uav_sequoia_fullfield
   • should we split up each data prodcut (e.g. ndva, red, green, blue? or perhaps merge these into a single .tif file?
6. ua-mac/Level_1/ps2_fluorescence and ua-mac/Level_1/ps2_png
   • these appear empty, and we don't currently have an extractor for these.
7. ua-mac/Level_1/vnir_soilmasks
   • appear to be only empty directories
8. ua-mac/Level_1/soilremoval
   • lots of empty directories, but some files under soilremoval/VNIR
   • soil masks should be appended to Level_1 and Level_2 products as a separate layer
9. ua-mac/Level_1/vnir_netcdf
   • all files *_ind.nc should be in Level_2/vnir_indices_netcdf/
   • lots of batch_*.sh files should be checked in to some repository
10. orphaned directories in ua-mac/raw_data:
    • 2017-11-09, TestIn, scanner3DOneOnEastSide, 3DTest, 3DScannerRawDataTopTmp
11. ua-mac/raw_data/ps2Top
    • create_synth_cb.m should be moved to a repository and deleted
    • @TinoDornbusch is this your file? What are the plans for integrating into the pipeline?
12. ua-mac/raw_data/lightning
    • last record is 2016-09-12
    • @NewcombMaria are we still collecting lightning data, and if so, where is it going?
13. genomics (@zhenbinHU and @nfahlgren)
    • we currently have derived_data/bap/danforth_center/version1 and derived_data/gbs/kansas_state/version1
    • are the / directories necessary / useful?
14. KSU UAV data
    • Level_2/height and Level_2/ndvi are empty
    • should try to make these consistent w/ ua-mac UAV
    • bin by date (add one more level directory - YYYY-MM-DD
15. danforth/derived_data
    • obsolete - output should go in Level_1
16. ua-mac/Level_1/ir_meanTemp
    • not a Level_1 product. If necessary to retain intermediate csv files (or npy etc) then these should go in a separate directory that is not exposed to the public
17. ua-mac/Level_1/ps2_png
    • These should be netcdf / hdf5 with time dimension, or at least geotiffs
18. ua-mac/raw_data/phenotractor
    • at minimum need to standardize file names
    • are we going to get more @Andrade-Pedro ?

Delete the following empty directories:
17. [ ] ua-mac/Level_1/ps2_fluorescence
17. [ ] ua-mac/Level_1/uav

1. Level_1/vnir_soil_masks/
2. Level_1/soil_removal/SWIR

These have been moved

1. raw_data/UAV/ (moved to ua-mac-uav/uav-unfiled
2. raw_data/phenotractor (moved to ua-mac-field/)
3. raw_data/FLIR_TG167_handheld moved to /ua-mac-field/FLIR_TG167_handheld/

[max-zilla]

https://github.com/terraref/computing-pipeline/wiki/Reprocessing-2018---Structure-Review