-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathChangeLog
249 lines (205 loc) · 8.44 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
CHANGES
=======
0.3.4
-----
* Fix automatically generated change log things
* Add archive step back in
* Have tests properly handle cases where gid exists or not
* Modify test to handle non-existent groups
* Handle uids and gids that do not correspond to users/groups on the system
* Use nci-files-report --json. Actually add test data to repo
* Use nci\_account --json instead of API directly
* Update parse\_user\_storage\_data to use backend NCI API
* Update README info
* Remove travis CI config. Add Github Actions status badge
0.3.3
-----
* Swap project and folder codes to account for scratch accounting change (#22)
0.3.2
-----
* Added additional condition for ending string to detect header line. (#20)
0.3.1
-----
* Fix for updated lquota formatting (#18)
* Updated server url (#16)
0.2.13
------
* Fix parsing error with storage outputs (#14)
0.2.12
------
* Added support for inconsistent units
0.2.11
------
* Fixed bug parsing new storage format. No longer have units in size
0.2.10
------
* Change to new nci-files-report format. Updated tests. (#12)
0.2.9
-----
* Cast datetime to date string for updating storage grants (#10)
0.2.8
-----
* Ensure date string for DB update (#8)
0.2.7
-----
* Add support for nci-files-report parsing (#6)
0.2.6
-----
* Make year a string (#4)
0.2.5
-----
* Fixed issue with has\_prefix\_files
0.2.4
-----
* Add has\_prefix\_files to conda meta
0.2.3
-----
* Added entry points to conda directly to try and solve python interp path issue
0.2.2
-----
* Fixed error in call to parse\_lquota
0.2.1
-----
* Gadi fix (#3)
0.2.0
-----
* Update to support gadi stats capture (#2)
0.1.3
-----
* Changed parsing of compute grants to include units
0.1.2
-----
* Fixed length of header string
0.1.1
-----
* Fixed bug in archive routine triggered with non-local access
* Updated travis and conda configs
0.1
---
* Updated README
* Removed old ncimontor code
* Moved ncimonitor subdirectory to ncigrafana
* Added project to default stdout
* Fixed called to getpwnam
* Updated query functions so that tests pass. Still some old code but that is currently not tested
* Init uid and gid vars before use
* Added uid and gid fields to User table/parsing
* Updated gitignore
* Added more logic to remove redundant StorageGrant entries
* Fixed error in addstoragepoint, used incorrect keyword "system"
* Forgot to remove gdata parsing script. Fixed
* Removed old parsing scripts
* Added two new parsing routines to support the new DB structure
* Add new tables: Schemes, Systems, UsageGrants, StorageGrants and StoragePoints
* Add efficiency to user usage table
* All parse scripts now working
* Adding project fields into some tables. Also making quarter a foreign key for some tables
* Changed parsing of short\_files\_report to a flag/state approach to get rid of the try/except block that was catching errors and silently aborting
* Change next calls to readline for python3 compat
* Add db url option
* Jobs (#23)
* Update usage (#22)
* Add percentage output (#19)
* Converted nci\_usage to using pandas dataframe output. (#17)
* arcive path
* Change working dir
* Install path
* Clean install conda
* Update conda
* Syntax
* Drop python 2 support
* move usage query to dataset
* update for dataset 1.1.2
* Fixed relative imports in nci\_monitor.py, which meant it could not be run from the install directory
* Fixed error in make\_gdata\_DB and make\_short\_DB, calling incorrect parsing function
* Fixed badges
* Removed unicode characters from README
* Added pbr build dependecy for circleci
* Added python3.6 to travis testing
* Moved to pbr for packaging
* Made entry points for DB updating scripts in setup.py and corresponding functions in scripts
* Added fullname output to nci\_usage, and an option for separator to crudely create csv output
* Updated README with current module load instructions
* Reindex storage dataframes to ensure there are values from the start of each quarter
* Fix variable scoping
* Put the flag on the wrong line
* Add coecms channel for dataset package
* Project should be an argument
* Add conda build infrastructure
* Python3 compat
* Use a in-memory database
* Use package
* Add setup.py packaging
* Added default value into size table when pivoting as NaN would appear if a user deleted all their storage in a project
* Check for no data to plot. Triggered sometimes with delta plots when there is no change
* Check for data lookup errors and don't try and plot
* Allow datetime objects in database when they creep in
* Add a command to list usage in the terminal
* Delta plots were being chopped off below zero. Fixed
* Python 3 print statements fixed in make scripts
* Removed spurious pip install and 3.5 build (which had dependency issues)
* Incorrect name for conda sqlite package
* Switched to conda to get updated version of sqlite3
* Specified trusty dist for travis to get sqlite3 version which supports printf
* Added tests for pandas interface. Added pandas dependency to travis
* Fixed tests that were assuming dates returned from DB were strings not datetime objects
* Made returned data frame have proper dates in the index. Fixed logic for plotting ideal usage. Made long SQL queries multiline to make more readable
* Reinstated pdf output. Added checks for empty dataframes and warning
* Only plot if db connection successful. Ensure ideal usage plotted even if users specified
* Reinstated ability to plot data from specific users
* Working version with refactored code using pandas dataframes
* Initial stages of nci\_monitor.py rewrite. Working but will not refactor
* Added pandas data retrieval. More generic and powerful
* Added licence to nci\_monitor.py
* Changes to DBcommon.py: python3 support, added license, fixed bug in datetoyearquarter with python3
* getstoragepoints working
* Beginning adding gdata data extraction
* BUG: Change in formatting led to project usage being lost. Fixed
* Bug : parsing usage like 0B was incorrectly handled. Fixed
* Bug: make\_SU\_DB.py : was storing SUs not KSUs. Fixed
* BUG: nci\_monitor.py : axes referenced before being created. Fixed
* Parsing size in DB module was wrong, and also broke tests. Fixed that up and reverted tests
* Updated usage parsing to support units other than KSU. Made unit testing more robust and general. Added tests for unit parsing
* Fixed logic errors when checking if there was data to plot
* Made full names default, added --username option for old behaviour. Added -n option to limit number of users plotted
* Made figure bigger and reduced legend fontsize to fit more users in legend
* Added --shorttotal option to show short file system limit + test
* Fixed typo in help message
* Broke tests with change to API. Fixed
* Fixed gdata upload. Added storagept key, and parsing the filename for storagept
* Big Fix. Cast strings to floats before loading into DB. This fixed error with not sorting by usage correctly when returning SU users
* Delta short usage also needs filling, otherwise deleted use directories don't show up as negative on the graph
* Filling in short usage algorithm was leading to erroneous result when a user's directory was removed and had no data. Now just fill with zeroes
* Included year and quarter in title, as removed year them from axis
* Improved date formatting
* Restrict coverage testing to core module
* Add some more badges
* Test silentrm
* Slightly bogus commit, as I stuffed up with a forced push
* From codecov analysis, remove duplicate test and squawked when file remove failed
* Fixed error, wasn't saving correct figure for short
* Added hardcopy output option (--pdf)
* Put proper codecov hook at the end
* Added custom codecov invocation
* Added codecov hooks to travis CI
* Added travis CI badge
* Replaced DISTINCT calls with groupby, which are supposed to work with ordering by an un-selected field
* Checking default sqlite3 version
* Removed dependency on matplotlib
* Specified test directory
* Removed unecessary packages
* Added dataset install
* Added directories to pythonpath
* Set pythonpath correctly in travis.yml
* Try py.test without specifying test directory
* Trying another trick to get py.test to see modules
* Ugly hack to get py.test to work, added empty conftest.py
* Added empty \_\_init\_\_.py to fix travis path problem
* Fixed travis script command
* Added travis
* Create README.md
* Unit test, using pytest, added
* Added explicit scale when retrieving SU usage. Added some more class functions to support unit tests
* Added DBcommon.py. Deleted UsageDB.py -- old file accidentally included
* Added nci\_monitor
* Initial commit of dataset approach