Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Glue client create_table_optimizer does not support disable functionality #4394

Closed
1 task
mr-shitij opened this issue Jan 16, 2025 · 1 comment
Closed
1 task
Labels
bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged.

Comments

@mr-shitij
Copy link

mr-shitij commented Jan 16, 2025

Describe the bug

With AWS management console compaction, snapshot retention and orphan file deletion of Glue Iceberg tables can be configured.

This is can be done via boto3 - but once enabled when we try to disable it by enable=false, it wont work.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

on passing enable = False it should disable the specific function.

Current Behavior

on passing enable = False it is not disabling the specific function.

Reproduction Steps

This function is part of lambda function. I am passing params to this function which contains

  {
    'enable': False,
    ...
  }
def create_compaction_optimizer(glue_client, params, table_name):
    optimizer_config = {
        'roleArn': params['role_arn'],
        'enabled': params['enabled'],
    }

    print('create_compaction_optimizer:')
    print(table_name)
    print(optimizer_config)
    print(params)

    return glue_client.create_table_optimizer(
        CatalogId=params['catalog_id'],
        DatabaseName=params['database_name'],
        TableName=table_name,
        Type='compaction',
        TableOptimizerConfiguration=optimizer_config,
    )

If I execute this code twice for both enable = True and False. It is giving error as a Already Exists on Table.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.36.0

Environment details (OS name and version, etc.)

The code is executing on AWS Lambda

@mr-shitij mr-shitij added bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Jan 16, 2025
@mr-shitij
Copy link
Author

def update_optimizer_configuration(glue_client, params: Dict[str, Any], table_name: str,
                                 optimizer_type: str, config: Dict[str, Any]) -> Dict:
    """
    Creates or updates a table optimizer based on whether it already exists.
    """
    try:
        existing_optimizer = get_existing_optimizer(glue_client, params, table_name, optimizer_type)
        
        if existing_optimizer:
            print(f"Updating existing {optimizer_type} optimizer")
            response = glue_client.update_table_optimizer(
                CatalogId=params['catalog_id'],
                DatabaseName=params['database_name'],
                TableName=table_name,
                Type=optimizer_type,
                TableOptimizerConfiguration=config
            )
        else:
            print(f"Creating new {optimizer_type} optimizer")
            response = glue_client.create_table_optimizer(
                CatalogId=params['catalog_id'],
                DatabaseName=params['database_name'],
                TableName=table_name,
                Type=optimizer_type,
                TableOptimizerConfiguration=config
            )
        return response
    except ClientError as e:
        print(f"Error updating optimizer configuration: {str(e)}")
        raise

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged.
Projects
None yet
Development

No branches or pull requests

1 participant