dl_bt_dukascopy.py: EOFError: Compressed file ended before the end-of-stream marker was reached

[kenorb]

Command:

./dl_bt_dukascopy.py -y 2013 -p EURUSD -c

Traceback:

Converting into CSV (download/EURUSD/2013-01-07--01h_ticks.csv)...
Traceback (most recent call last):
  File "./dl_bt_dukascopy.py", line 159, in <module>
    ds.bt5_to_csv()
  File "./dl_bt_dukascopy.py", line 102, in bt5_to_csv
    data = f.read()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lzma.py", line 310, in read
    return self._read_all()
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lzma.py", line 251, in _read_all
    while self._fill_buffer():
  File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lzma.py", line 225, in _fill_buffer
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached

It happens randomly. In this case, it happens for: 2013-01-07--01h_ticks.bi5 specifically.

[ghost]

https://github.com/d-a-t-a-d-i-t/FX-BT-Scripts/commit/d119202a17db91351b29489505b99330039206f3

[kenorb]

Fixed and merged.

[kenorb]

Still there is a problem:

./dl_bt_dukascopy.py -c -p AUDJPY -y 2007 -m 10
Downloading http://www.dukascopy.com/datafeed/AUDJPY/2007/09/25/02h_ticks.bi5 into: download/dukascopy/AUDJPY/2007/10/2007-10-25--02h_ticks.bi5...
File (download/dukascopy/AUDJPY/2007/10/2007-10-25--02h_ticks.bi5) exists, so skipping.
Converting into CSV (download/dukascopy/AUDJPY/2007/10/2007-10-25--02h_ticks.csv)...
Traceback (most recent call last):
  File "./dl_bt_dukascopy.py", line 169, in <module>
    ds.bt5_to_csv()
  File "./dl_bt_dukascopy.py", line 110, in bt5_to_csv
    data = f.read()
  File "/usr/local/lib/python3.2/dist-packages/backports/lzma/__init__.py", line 287, in read
    return self._read_all()
  File "/usr/local/lib/python3.2/dist-packages/backports/lzma/__init__.py", line 236, in _read_all
    while self._fill_buffer():
  File "/usr/local/lib/python3.2/dist-packages/backports/lzma/__init__.py", line 223, in _fill_buffer
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached
$ python3 --version
Python 3.2.3

Any clues?

[ghost]

As I experienced, the ideal lzma._BUFFER_SIZE value (which doesn't cause EOFError exception) varies from Python version to Python version, so it's different for e.g. Python 3.2.x and 3.4.x. The solutions could be:

1. try different values with different Python versions, or
2. create a fix/workaround in upstream Python code.

[ghost]

or
3. call lzma decompressor externally from the script until LZMA API is fixed in Python.

[kenorb]

PR: #35

[kenorb]

Possible workaround:

• http://stackoverflow.com/a/37400585/55075
• giuse88/duka@faf082e
• Compressed data ended before the end-of-stream marker was reached giuse88/duka#10