DataRow given dictionary for row data, all parsers updated

To make the data parsers easier to write/maintain, the DataRow object now contains a dictionary of the row data in addition to an array. All parsers have been updated to use the row dictionary where appropriate.

Whitespace is now removed from the beginning and end of each heading, as used by both the row dictionary and also DataParser.

A new convert_currency method has been added to DataParser, this allows non-GBP prices to be added as asset values by the parser, this fixes BittyTax#72. It required a change to the PriceData class, the initialiser now uses a list of data sources, which in the case of convert_currency are fiat only.

A new "Savings & Loans” category has been added to the conv tool to differentiate sites like Nexo, etc.

General changes include, fixing some pylint warnings, removal of command line arguments from the Config object (except debug) so they are no longer global but instead used locally.

The Crypto.com parser has been updated to recognise the transaction type “crypto_to_van_sell_order”.

The Nexo parser has been updated to handle unconfirmed transactions.

CHANGELOG.md

@@ -10,6 +10,7 @@ Important:- A new Note field has been added to the end of the transaction record
 - Coinbase Pro parser: fills export, buy quantity missing fee.
 - Price tool: list command returns error. ([#86](https://github.com/BittyTax/BittyTax/issues/86))
 - Price tool: -ds option returns "KeyError: 'price'" exception.
+- Conversion tool: strip whitespace from header.
 ### Added
 - Etherscan parser: added internal transactions export.
 - Binance parser: added cash deposit and withdrawal exports.
@@ -38,6 +39,9 @@ Important:- A new Note field has been added to the end of the transaction record
 - Local currency support.
 - Accounting tool: new config "transfer_fee_allowable_cost" added.
 - Conversion tool: allow wildcards in filenames.
+- Conversion tool: added dictionary to DataRow.
+- Conversion tool: "Savings & Loans" parser category added.
+- Conversion tool: convert_currency method added to DataParser.
 ### Changed
 - Conversion tool: UnknownAddressError exception changed to generic DataFilenameError.
 - Binance parser: use filename to determine if deposits or withdrawals.
@@ -60,6 +64,11 @@ Important:- A new Note field has been added to the end of the transaction record
 - Accounting tool: ordering of all transactions when transfers_include=False.
 - Ledger Live parser: added "FEES" and "REVEAL" operation types. ([#79](https://github.com/BittyTax/BittyTax/issues/79))
 - Binance parser: added "Referrer rebates" operation type.
+- Command line arguments now used locally instead of stored globally.
+- Price tool: PriceData requires data source list to initialise.
+- Conversion tool: all parsers updated to use DataRow dictionary.
+- Crypto.com parser: added "crypto_to_van_sell_order" transaction type.
+- Nexo parser: check for unconfirmed transactions.
 ### Removed
 - Accounting tool: skip audit (-s or --skipaudit) option removed.
 - Accounting tool: updated transactions debug removed.

bittytax/audit.py

@@ -15,14 +15,14 @@ def __init__(self, transaction_records):
         self.totals = {}
         self.failures = []
 
-        if config.args.debug:
+        if config.debug:
             print("%saudit transaction records" % Fore.CYAN)
 
         for tr in tqdm(transaction_records,
                        unit='tr',
                        desc="%saudit transaction records%s" % (Fore.CYAN, Fore.GREEN),
-                       disable=bool(config.args.debug or not sys.stdout.isatty())):
-            if config.args.debug:
+                       disable=bool(config.debug or not sys.stdout.isatty())):
+            if config.debug:
                 print("%saudit: TR %s" % (Fore.MAGENTA, tr))
             if tr.buy:
                 self._add_tokens(tr.wallet, tr.buy.asset, tr.buy.quantity)
@@ -33,7 +33,7 @@ def __init__(self, transaction_records):
             if tr.fee:
                 self._subtract_tokens(tr.wallet, tr.fee.asset, tr.fee.quantity)
 
-        if config.args.debug:
+        if config.debug:
             print("%saudit: final balances by wallet" % Fore.CYAN)
             for wallet in sorted(self.wallets, key=str.lower):
                 for asset in sorted(self.wallets[wallet]):
@@ -68,7 +68,7 @@ def _add_tokens(self, wallet, asset, quantity):
 
         self.totals[asset] += quantity
 
-        if config.args.debug:
+        if config.debug:
             print("%saudit:   %s:%s=%s (+%s)" % (
                 Fore.GREEN,
                 wallet,
@@ -90,7 +90,7 @@ def _subtract_tokens(self, wallet, asset, quantity):
 
         self.totals[asset] -= quantity
 
-        if config.args.debug:
+        if config.debug:
             print("%saudit:   %s:%s=%s (-%s)" %(
                 Fore.GREEN,
                 wallet,
@@ -111,18 +111,18 @@ def compare_pools(self, holdings):
 
             if asset in holdings:
                 if self.totals[asset] == holdings[asset].quantity:
-                    if config.args.debug:
+                    if config.debug:
                         print("%scheck pool: %s (ok)" %(Fore.GREEN, asset))
                 else:
-                    if config.args.debug:
+                    if config.debug:
                         print("%scheck pool: %s %s (mismatch)" %(Fore.RED, asset,
                             '{:+0,f}'.format((holdings[asset].quantity-
                                               self.totals[asset]).normalize())))
 
                     self._log_failure(asset, self.totals[asset], holdings[asset].quantity)
                     passed = False
             else:
-                if config.args.debug:
+                if config.debug:
                     print("%scheck pool: %s (missing)" %(Fore.RED, asset))
 
                 self._log_failure(asset, self.totals[asset], None)

bittytax/bittytax.py

@@ -81,63 +81,65 @@ def main():
                         action='store_true',
                         help="export your transaction records populated with price data")
 
-    config.args = parser.parse_args()
-    config.args.nocache = False
+    args = parser.parse_args()
+    config.debug = args.debug
 
-    if config.args.debug:
+    if config.debug:
         print("%s%s v%s" % (Fore.YELLOW, parser.prog, __version__))
         print("%spython: v%s" % (Fore.GREEN, platform.python_version()))
         print("%ssystem: %s, release: %s" % (Fore.GREEN, platform.system(), platform.release()))
         config.output_config()
 
-    if config.args.tax_rules in config.TAX_RULES_UK_COMPANY:
-        config.start_of_year_month = config.TAX_RULES_UK_COMPANY.index(config.args.tax_rules) + 1
+    if args.tax_rules in config.TAX_RULES_UK_COMPANY:
+        config.start_of_year_month = config.TAX_RULES_UK_COMPANY.index(args.tax_rules) + 1
         config.start_of_year_day = 1
 
     try:
-        transaction_records = do_import(config.args.filename)
+        transaction_records = do_import(args.filename)
     except IOError:
         parser.exit("%sERROR%s File could not be read: %s" % (
-            Back.RED+Fore.BLACK, Back.RESET+Fore.RED, config.args.filename))
+            Back.RED+Fore.BLACK, Back.RESET+Fore.RED, args.filename))
     except ImportFailureError:
         parser.exit()
 
-    if config.args.export:
+    if args.export:
         do_export(transaction_records)
         parser.exit()
 
     audit = AuditRecords(transaction_records)
 
     try:
-        tax, value_asset = do_tax(transaction_records)
-        if not config.args.skip_integrity:
+        tax, value_asset = do_tax(transaction_records, args.tax_rules, args.skip_integrity)
+        if not args.skip_integrity:
             int_passed = do_integrity_check(audit, tax.holdings)
             if not int_passed:
                 parser.exit()
 
-        if not config.args.summary:
+        if not args.summary:
             tax.process_income()
 
         do_each_tax_year(tax,
-                         config.args.taxyear,
-                         config.args.summary,
+                         args.taxyear,
+                         args.summary,
                          value_asset)
 
     except DataSourceError as e:
         parser.exit("%sERROR%s %s" % (
             Back.RED+Fore.BLACK, Back.RESET+Fore.RED, e))
 
-    if config.args.nopdf:
+    if args.nopdf:
         ReportLog(audit,
                   tax.tax_report,
                   value_asset.price_report,
-                  tax.holdings_report)
+                  tax.holdings_report,
+                  args)
     else:
         ReportPdf(parser.prog,
                   audit,
                   tax.tax_report,
                   value_asset.price_report,
-                  tax.holdings_report)
+                  tax.holdings_report,
+                  args)
 
 def validate_year(value):
     year = int(value)
@@ -174,20 +176,20 @@ def do_import(filename):
 
     return import_records.get_records()
 
-def do_tax(transaction_records):
+def do_tax(transaction_records, tax_rules, skip_integrity_check):
     value_asset = ValueAsset()
     transaction_history = TransactionHistory(transaction_records, value_asset)
 
-    tax = TaxCalculator(transaction_history.transactions)
+    tax = TaxCalculator(transaction_history.transactions, tax_rules)
     tax.pool_same_day()
     tax.match(tax.DISPOSAL_SAME_DAY)
 
-    if config.args.tax_rules == config.TAX_RULES_UK_INDIVIDUAL:
+    if tax_rules == config.TAX_RULES_UK_INDIVIDUAL:
         tax.match(tax.DISPOSAL_BED_AND_BREAKFAST)
-    elif config.args.tax_rules in config.TAX_RULES_UK_COMPANY:
+    elif tax_rules in config.TAX_RULES_UK_COMPANY:
         tax.match(tax.DISPOSAL_TEN_DAY)
 
-    tax.process_section104()
+    tax.process_section104(skip_integrity_check)
     return tax, value_asset
 
 def do_integrity_check(audit, holdings):

bittytax/config.py

@@ -62,7 +62,7 @@ class Config(object):
     }
 
     def __init__(self):
-        self.args = None
+        self.debug = False
         self.start_of_year_month = 4
         self.start_of_year_day = 6
 
@@ -106,12 +106,12 @@ def output_config(self):
     def sym(self):
         if self.ccy == 'GBP':
             return u'\xA3' # £
-        elif self.ccy == 'EUR':
+        if self.ccy == 'EUR':
             return u'\u20AC' # €
-        elif self.ccy in ('USD', 'AUD', 'NZD'):
+        if self.ccy in ('USD', 'AUD', 'NZD'):
             return '$'
-        elif self.ccy in ('DKK', 'NOK', 'SEK'):
-             return 'kr.'
+        if self.ccy in ('DKK', 'NOK', 'SEK'):
+            return 'kr.'
 
         raise ValueError("Currency not supported")
 

bittytax/conv/bittytax_conv.py

@@ -61,7 +61,7 @@ def main():
                         choices=[config.FORMAT_EXCEL, config.FORMAT_CSV, config.FORMAT_RECAP],
                         default=config.FORMAT_EXCEL,
                         type=str.upper,
-                        help="specify the output format")
+                        help="specify the output format, default: EXCEL")
     parser.add_argument('-nh',
                         '--noheader',
                         action='store_true',
@@ -79,21 +79,23 @@ def main():
                         type=str,
                         help="specify the output filename")
 
-    config.args = parser.parse_args()
+    args = parser.parse_args()
+    config.debug = args.debug
+    DataFile.remove_duplicates = args.duplicates
 
-    if config.args.debug:
+    if config.debug:
         sys.stderr.write("%s%s v%s\n" % (Fore.YELLOW, parser.prog, __version__))
         sys.stderr.write("%spython: v%s\n" % (Fore.GREEN, platform.python_version()))
         sys.stderr.write("%ssystem: %s, release: %s\n" % (
             Fore.GREEN, platform.system(), platform.release()))
 
-    for filename in config.args.filename:
+    for filename in args.filename:
         for pathname in glob.iglob(filename):
             try:
                 try:
-                    DataFile.read_excel(pathname)
+                    DataFile.read_excel(pathname, args)
                 except xlrd.XLRDError:
-                    DataFile.read_csv(pathname)
+                    DataFile.read_csv(pathname, args)
             except UnknownCryptoassetError:
                 sys.stderr.write(Fore.RESET)
                 parser.error("cryptoasset cannot be identified for data file: %s, "
@@ -114,11 +116,11 @@ def main():
                     Back.YELLOW+Fore.BLACK, Back.RESET+Fore.YELLOW, pathname))
 
     if DataFile.data_files:
-        if config.args.format == config.FORMAT_EXCEL:
-            output = OutputExcel(parser.prog, DataFile.data_files_ordered)
+        if args.format == config.FORMAT_EXCEL:
+            output = OutputExcel(parser.prog, DataFile.data_files_ordered, args)
             output.write_excel()
         else:
-            output = OutputCsv(DataFile.data_files_ordered)
+            output = OutputCsv(DataFile.data_files_ordered, args)
             sys.stderr.write(Fore.RESET)
             sys.stderr.flush()
             output.write_csv()

bittytax/conv/datafile.py

@@ -19,23 +19,25 @@ class DataFile(object):
 
     CSV_DELIMITERS = (',', ';')
 
+    remove_duplicates = False
     data_files = {}
     data_files_ordered = []
 
-    def __init__(self, file_format, filename, parser, reader):
+    def __init__(self, file_format, filename, args, parser, reader):
         self.parser = parser
-        self.data_rows = [DataRow(line_num + 1, in_row) for line_num, in_row in enumerate(reader)]
+        self.data_rows = [DataRow(line_num + 1, row, parser.in_header)
+                          for line_num, row in enumerate(reader)]
 
         if parser.row_handler:
             for data_row in self.data_rows:
-                if config.args.debug:
+                if config.debug:
                     sys.stderr.write("%sconv: row[%s] %s\n" % (
                         Fore.YELLOW, parser.in_header_row_num + data_row.line_num, data_row))
 
-                data_row.parse(parser, filename)
+                data_row.parse(parser, filename, args)
         else:
             # all rows handled together
-            DataRow.parse_all(self.data_rows, parser, filename)
+            DataRow.parse_all(self.data_rows, parser, filename, args)
 
         failures = [data_row for data_row in self.data_rows if data_row.failure is not None]
         if failures:
@@ -58,7 +60,7 @@ def __iadd__(self, other):
         if len(other.parser.header) > len(self.parser.header):
             self.parser = other.parser
 
-        if config.args.duplicates:
+        if self.remove_duplicates:
             self.data_rows += [data_row
                                for data_row in other.data_rows if data_row not in self.data_rows]
         else:
@@ -67,9 +69,9 @@ def __iadd__(self, other):
         return self
 
     @classmethod
-    def read_excel(cls, filename):
+    def read_excel(cls, filename, args):
         workbook = xlrd.open_workbook(filename)
-        if config.args.debug:
+        if config.debug:
             sys.stderr.write("%sconv: EXCEL\n" % Fore.CYAN)
 
         sheet = workbook.sheet_by_index(0)
@@ -79,7 +81,7 @@ def read_excel(cls, filename):
         if parser is not None:
             sys.stderr.write("%sfile: %s%s %smatched as %s\"%s\"\n" % (
                 Fore.WHITE, Fore.YELLOW, filename, Fore.WHITE, Fore.CYAN, parser.worksheet_name))
-            data_file = DataFile(cls.FORMAT_EXCEL, filename, parser, reader)
+            data_file = DataFile(cls.FORMAT_EXCEL, filename, args, parser, reader)
             cls.consolidate_datafiles(data_file)
         else:
             raise DataFormatUnrecognised
@@ -109,10 +111,10 @@ def convert_cell(cell, workbook):
         return value
 
     @classmethod
-    def read_csv(cls, filename):
+    def read_csv(cls, filename, args):
         with io.open(filename, newline='', encoding='utf-8-sig') as csv_file:
             for delimiter in cls.CSV_DELIMITERS:
-                if config.args.debug:
+                if config.debug:
                     sys.stderr.write("%sconv: CSV delimiter='%s'\n" % (Fore.CYAN, delimiter))
 
                 if sys.version_info[0] < 3:
@@ -126,11 +128,11 @@ def read_csv(cls, filename):
                     sys.stderr.write("%sfile: %s%s %smatched as %s\"%s\"\n" % (
                         Fore.WHITE, Fore.YELLOW, filename, Fore.WHITE,
                         Fore.CYAN, parser.worksheet_name))
-                    data_file = DataFile(cls.FORMAT_CSV, filename, parser, reader)
+                    data_file = DataFile(cls.FORMAT_CSV, filename, args, parser, reader)
                     cls.consolidate_datafiles(data_file)
                     break
-                else:
-                    csv_file.seek(0)
+
+                csv_file.seek(0)
 
         if parser is None:
             raise DataFormatUnrecognised