Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature request] Enum-class that allows to have Columns as values #106

Open
SemyonSinchenko opened this issue May 30, 2023 · 0 comments
Open

Comments

@SemyonSinchenko
Copy link
Collaborator

I want to have an Enum-like class with the same functionality as python Enum but with an ability to use pyspark.sql.Column as a type of values. The existed built-in Enum uses __eq__ method to check that a new added value is an alias to existed one:

        # If another member with the same value was already defined, the
        # new member becomes an alias to the existing one.
        try:
            try:
                # try to do a fast lookup to avoid the quadratic loop
                enum_member = enum_class._value2member_map_[value]
            except TypeError:
                for name, canonical_member in enum_class._member_map_.items():
                    if canonical_member._value_ == value:
                        enum_member = canonical_member
                        break
                else:
                    raise KeyError

But for pyspark.sql.Column method __eq__ returns not bool but Column and we are getting an error here: ValueError: Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant