FIX-#6778: Read parquet files without file extensions using fastparquet #6790
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ing fastparquet In supporting fastparquet, modin takes the paths provided, globs them, and filters them to only look at files with the .parq or .parquet extension. This commit adds support so that if the path supplied is explicitly a file, it will be included. Signed-off by: Ari Brown <[email protected]>
seydar
requested review from
devin-petersohn,
mvashishtha,
RehanSD,
YarShev,
vnlitvinov,
anmyachev,
dchigarev and
a team
as code owners
6 tasks
anmyachev
reviewed
Nov 30, 2023
| # However, we also need to support users passing in explicit files that | ||
| # don't necessarily have the `.parq` or `.parquet` extension -- if a user | ||
| # says that a file is parquet, then we should probably give it a shot. | ||
| if os.path.isfile(self.path): |
There was a problem hiding this comment.
Looks like os.path.isfile doesn't work with objects, for example S3File.
Suggested change
| if os.path.isfile(self.path): | |
| if self.fs.isfile(self.path): |
There was a problem hiding this comment.
Just made the change as an added commit. Thank you for the catch.
… a file Avoids the use of `os.path.isfile()` to use `self.fs.isfile()`. Signed-off-by: Ari Brown <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In supporting fastparquet, modin takes the paths provided, globs them, and filters them to only look at files with the .parq or .parquet extension. This commit adds support so that if the path supplied is explicitly a file, it will be included.
Signed-off by: Ari Brown [email protected]
What do these changes do?
If a a file is passed to
modin.pandas.read_parquet(..., engine='fastparquet')but the file does not have ".parq" or ".parquet" in the name, it will be ignored when determining the number of partitions. This change looks at all of the paths passed, and if the path is a file, then it considers it to be a parquet file.flake8 modin/ asv_bench/benchmarks scripts/doc_checker.pyblack --check modin/ asv_bench/benchmarks scripts/doc_checker.pygit commit -sdocs/development/architecture.rstis up-to-date