Lite Viz is too slow

[lordsoffallen]

Description

I am running uvx --with kedro-viz kedro viz run --lite in a project to visualize the nodes but it's throwing the following errors:

Context

Just trying to run the lite viz on my project.

Steps to Reproduce

Run the command above in a project.

Your Environment

Include as many relevant details as possible about the environment you experienced the bug in:

• Web browser system and version:
• Operating system and version: Ubuntu 20.04
• NodeJS version used (if relevant): -
• Kedro version used (if relevant): 0.19.10
• Python version used (if relevant): 3.10.16

Checklist

• Include labels so that we can categorise your issue

[lordsoffallen]

Installed kedro-viz on my env and without --lite it's work as expected. Problem seems to --lite flag only

[rashidakanchwala]

@ravi-kumar-pilla

[ravi-kumar-pilla]

Hi @lordsoffallen ,

Thank you for raising the issue. Is it possible for you to give us a reproducible code snippet on which we can test and fix the issue. I am curious to know what were your import statements and dependencies (especially the file which has utf-8 dependency or related)

[lordsoffallen]

@ravi-kumar-pilla Thanks, I would love to help but I have no idea where the problem is. This is a kedro project with many files inside. Is there like debug option that would output where the utf-8 errors are coming from? I don't know any file that i have that has something to do with utf-8. There are quite python in the project.

[ravi-kumar-pilla]

I see, Thanks for the response @lordsoffallen . As of now we do not have a debug option to give you a better debugging experience. Based on the error stack trace I think the error is being thrown when we are trying to open a file in the project. We have the below code at kedro_viz/integrations/kedro/lite_parser.py -

       with open(file_path, "r", encoding="utf-8") as file:
            file_content = file.read()

We can improve the error message to know exactly which file is causing the issue, but in the mean time if you would like to contribute, you can debug this file kedro_viz/integrations/kedro/lite_parser.py at parse function. Most probably the error is coming from here which tries to decode a file using UTF-8 encoding. If the file in your project has a different encoding, this will error out. Thank you

[lordsoffallen]

I can debug but I still don't see how i can invoke this in my IDE. What I normally do is:

def test_preprocess(self, kedro_session):
    kedro_session.run(node_names=['preprocess'])

Which will just run a node, not idea how to invoke viz programatically so i can step into those lines. Would it be possible to add debug line logs (as it would help debug further issues in the future directly) and then I can upgrade and just enable debug logs? This is not urgent for me btw as without --lite flag, it works for me.

[ravi-kumar-pilla]

Which will just run a node, no idea how to invoke viz programatically so i can step into those lines. Would it be possible to add debug line logs (as it would help debug further issues in the future directly) and then I can upgrade and just enable debug logs? This is not urgent for me btw as without --lite flag, it works for me.

Hi @lordsoffallen , Sorry for the delay in response. The immediate next step for us would be to improve the error message, which shows the file causing issue. As per the info shared, it seems our FileParser is unable to decode a file which may be not UTF-8 encoded. I would be happy to jump on a call and see if we can place some debug statements to find the root cause. I am working in CST timezone. Please let me know a good time to connect with you.

Thank you

[lordsoffallen]

Hi @ravi-kumar-pilla

Unfortunately, it won't be possible anytime soon. Project env is a bit complex so I can't install kedro-viz into it which means I can't directly debug on my IDE. If I can find a time, I'll get back to you.

[ravi-kumar-pilla]

Unfortunately, it won't be possible anytime soon. Project env is a bit complex so I can't install kedro-viz into it which means I can't directly debug on my IDE. If I can find a time, I'll get back to you.

okay, thanks for letting us know. We will add some logs to help understand the file causing issue in the next sprint. I hope it will narrow down the issue. I will update you after the next release. Thank you for trying --lite @lordsoffallen

[astrojuanlu]

I experienced this locally as well:

This is on a normal spaceflights project

[astrojuanlu]

But actually the interesting thing is that, after a while, it does start! Launched the command at 14:38:52, server was running at 14:42:54 (hence 4 minutes later)

So this isn't a case of "--lite doesn't work", but rather

• the ERROR messages mislead the user (they aren't actually errors, but warnings), and
• --lite is actually slower than installing the dependencies, which gives the user the impression that it's stuck

[lordsoffallen]

But actually the interesting thing is that, after a while, it does start! Launched the command at 14:38:52, server was running at 14:42:54 (hence 4 minutes later)

So this isn't a case of "--lite doesn't work", but rather

• the ERROR messages mislead the user (they aren't actually errors, but warnings), and
• --lite is actually slower than installing the dependencies, which gives the user the impression that it's stuck

So ERRORs should be warnings and simply we wait a bit more? Interesting turn of events 😆

[ravi-kumar-pilla]

Hi @astrojuanlu @lordsoffallen ,

The --lite option on viz parse files which are utf-8 encoded. If the files are not utf-8 encoded and contain unrecognizable characters, it will throw the error message. I agree the log level might be confusing but it is a valid Parse error happening there. Even after the error, the parsing continues as we do not break the loop on error in the code.

Also, --lite was started to speed up kedro viz run but after the implementation it was more of having a kedro viz run without kedro project dependency installations.

Ideally it should not slow down the viz run but it may not improve the speed as well. I have included the file path in the error message in the new release which would narrow down the issue - #2266 . If the log level is confusing, we can make this a warning. wdyt ?

Thank you

[lordsoffallen]

Hi @astrojuanlu @lordsoffallen ,

The --lite option on viz parse files which are utf-8 encoded. If the files are not utf-8 encoded and contain unrecognizable characters, it will throw the error message. I agree the log level might be confusing but it is a valid Parse error happening there. Even after the error, the parsing continues as we do not break the loop on error in the code.

Also, --lite was started to speed up kedro viz run but after the implementation it was more of having a kedro viz run without kedro project dependency installations.

Ideally it should not slow down the viz run but it may not improve the speed as well. I have included the file path in the error message in the new release which would narrow down the issue - [#2266](https://github.com//pull/2266) . If the log level is confusing, we can make this a warning. wdyt ?

Thank you

For errors i expect code to fail at some point and if we don't, i think warnings makes more sense to me. Let me know when the log changes are released so i can also tell you which files were having issues.

[ravi-kumar-pilla]

For errors i expect code to fail at some point and if we don't, i think warnings makes more sense to me. Let me know when the log changes are released so i can also tell you which files were having issues.

Hi @lordsoffallen , The file name is now included in the logs. Though it still shows log level as ERROR, we will make the log level to warning and also try to add a spinner in the future release. You can try kedro-viz 10.2.0 to check the file names. Thank you

[lordsoffallen]

I think i know --lite does not work and slower. It's because we scan the whole .venv directory ( and try to mock the whole venv??):

I think mocking should have ignored file list to avoid folders like tmp/.venv etc or better yet to sticking to gitignore would also be nice.