Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: handle reconnect get_modifiers #49

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

MrKevinWeiss
Copy link

After updating the firmware on my PPK2 to v4.2.0 I noticed that connecting, disconnected then trying to reconnect failed. It seems like the updated firmware has some 0x81 byte that gets send and as a result an exception gets thrown.

This commit allows those bytes to be replaced, thus the check is still valid but we do not get this exception.

This was discovered with the following version:

ppk2-api==0.9.2

Error Traceback

  File "/home/weiss/repos/lobaro/python-lob-ppk/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 228, in get_modifiers
    metadata = self._read_metadata()
  File "/home/weiss/repos/lobaro/python-lob-ppk/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 158, in _read_metadata
    if read != b'' and "END" in read.decode("utf-8"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x81 in position 130: invalid start byte

After updating the firmware on my PPK2 to v4.2.0 I noticed that
connecting, disconnected then trying to reconnect failed.
It seems like the updated firmware has some 0x80 byte that gets
send and as a result an exception gets thrown.

This commit allows those bytes to be replaced, thus the check is still valid
but we do not get this exception.
@MrKevinWeiss MrKevinWeiss changed the title fix: handle reconnect read_metadata fix: handle reconnect get_modifiers Sep 24, 2024
@wlgrd
Copy link
Collaborator

wlgrd commented Oct 2, 2024

Could you explain a bit more of how you're using the API? Running example.py does not yield this issue. I'm wondering if something is not cleaned up properly in your python session.

@MrKevinWeiss
Copy link
Author

Ahh sorry for the delayed response, last week was pretty busy.

Unfortunately I cannot open-source it (yet), but I essentially call self._ppk.get_modifiers().

Like I said, this only occurred when updating the PPK firmware and was not a problem before.

Here is a simplified script:

from ppk2_api.ppk2_api import PPK2_API

ppk = PPK2_API("/dev/ttyACM0", timeout=1, write_timeout=1, exclusive=True)
ppk.get_modifiers()
ppk.set_source_voltage(int(3.6 * 1000))
ppk.toggle_DUT_power("ON")
ppk.use_source_meter()
ppk.start_measuring()
import time
time.sleep(0.1)
print(len(ppk.get_data()))
ppk.stop_measuring()
ppk.toggle_DUT_power("OFF")

When running the first time it works, the second and third time it fails, then it works again and the pattern repeats

(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
4095
(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
Traceback (most recent call last):
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/../python-lob-ppk/examples/basic/fail_test.py", line 4, in <module>
    ppk.get_modifiers()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 228, in get_modifiers
    metadata = self._read_metadata()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 158, in _read_metadata
    if read != b'' and "END" in read.decode("utf-8"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
Traceback (most recent call last):
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/../python-lob-ppk/examples/basic/fail_test.py", line 4, in <module>
    ppk.get_modifiers()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 228, in get_modifiers
    metadata = self._read_metadata()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 158, in _read_metadata
    if read != b'' and "END" in read.decode("utf-8"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
4095
(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
Traceback (most recent call last):
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/../python-lob-ppk/examples/basic/fail_test.py", line 4, in <module>
    ppk.get_modifiers()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 228, in get_modifiers
    metadata = self._read_metadata()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 158, in _read_metadata
    if read != b'' and "END" in read.decode("utf-8"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 130: invalid start byte
(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
Traceback (most recent call last):
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/../python-lob-ppk/examples/basic/fail_test.py", line 4, in <module>
    ppk.get_modifiers()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 228, in get_modifiers
    metadata = self._read_metadata()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 158, in _read_metadata
    if read != b'' and "END" in read.decode("utf-8"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte
(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
4095
(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
Traceback (most recent call last):
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/../python-lob-ppk/examples/basic/fail_test.py", line 4, in <module>
    ppk.get_modifiers()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 228, in get_modifiers
    metadata = self._read_metadata()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 158, in _read_metadata
    if read != b'' and "END" in read.decode("utf-8"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 2: invalid start byte
(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
Traceback (most recent call last):
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/../python-lob-ppk/examples/basic/fail_test.py", line 4, in <module>
    ppk.get_modifiers()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 228, in get_modifiers
    metadata = self._read_metadata()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 158, in _read_metadata
    if read != b'' and "END" in read.decode("utf-8"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 130: invalid start byte
(.venv) weiss@lobaro-laptop-kw:~/repos/lobaro/lobaro-provisioning-framework$ python ../python-lob-ppk/examples/basic/fail_test.py 
Traceback (most recent call last):
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/../python-lob-ppk/examples/basic/fail_test.py", line 4, in <module>
    ppk.get_modifiers()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 228, in get_modifiers
    metadata = self._read_metadata()
  File "/home/weiss/repos/lobaro/lobaro-provisioning-framework/.venv/lib/python3.10/site-packages/ppk2_api/ppk2_api.py", line 158, in _read_metadata
    if read != b'' and "END" in read.decode("utf-8"):
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 2: invalid start byte

Currently I am just patching the like so:

 class myPPKWrapper():
  def __init__(self):
    ...
    self._ppk = PPK2_API(self.ppk_port, timeout=1, write_timeout=1, exclusive=True)
    self._ppk._read_metadata = self._patched_ppk_read_metadata

  def _patched_ppk_read_metadata(self):
          """This is a fix since the upstream has a bug
  
          The issue is that the updated firmware on the PPK sends a 0x81 byte
          and since the decoding is strict, an exception is raised... We can
          just replace it and it seems to work okay.
          """
          # try to get metadata from device
          for _ in range(0, 5):
              # it appears the second reading is the metadata
              read = self._ppk.ser.read(self._ppk.ser.in_waiting)
              sleep(0.1)
  
              # TODO add a read_until serial read function with a timeout
              if read != b"" and "END" in read.decode("utf-8", errors="replace"):
                  return read.decode("utf-8", errors="replace")

@MrKevinWeiss
Copy link
Author

More info...

Here is the actual thing that is read from the serial port:

�����!�%�)�-�1�5�9�=�A�E�I�M�Q�U�Y�]�a�e�i�m�q�u�y�}����������������������������������������������������������������Calibrated: 0
R0: 1004.23089599609380000000
R1: 102.25025939941406000000
R2: 10.40815162658691400000
R3: 0.97722643613815310000

Just for some info, I would assume that the parse_metadata function cannot handle the leading garbage for the Calibrated key but I have not tested it.

@ChristianStadelmann
Copy link

With the same ppk2_api version (0.9.2), I'm seeing a similar problem: I sporadically see different UnicodeDecodeErrors when calling _read_metadata. Some examples (same lines, same backtrace as you, i.e., also when calling get_modifiers()):

Python Error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte
Python Error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa4 in position 0: invalid start byte
Python Error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte
Python Error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb3 in position 0: invalid start byte
Python Error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa4 in position 0: invalid start byte
Python Error: UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 4-5: invalid continuation byte
Python Error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 20: invalid start byte

Are you sure that the problem you are seeing is always at the same position, or is it maybe happening at some random position as in my case?

Also, I fear that the change proposed in this PR will only hide an existing bug. Is it possible, that the PPK2 is sporadically returning nonsense?

@MrKevinWeiss
Copy link
Author

Ya, it is not that clear. For now we can say there is a bug in there but this probably isn't the best fix. It somehow works for me but probably this shouldn't be merged in without further investigation... I will try to find some time to dig into it but cannot make promises.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants