redshift-alpha: Rationalization of IAM roles creation for Lambdas execution

[matteo-pirola]

Describe the feature

Overview

I suggest a rationalization of the IAM roles created and used to execute the Lambda functions responsible of generating and executing Redshift queries to create or update resources.

As of today, the framework generates many "single-scope" roles leading to an explosion on the number of IAM roles in an account, making it easy to reach the IAM quota on roles per account (1000).

AS IS

Currently, when using the Table construct to create Redshift tables, a CloudFormation Stack with the following is generated:

• A Resource called Query Redshift Database XYZ, consisting of a Lambda function and an IAM role for its execution
• For each table, a Lambda function and an "associated" IAM role for its execution; each role has permission to only execute the function it is "associated" to

This means that, if I want to create n tables I will obtain a total of at least n+1 IAM roles (if I put them all in the same Stack).

Example code that I am using to create the tables:

import aws_cdk as cdk
from aws_cdk import Stack
from aws_cdk import aws_secretsmanager as secretsmanager
from aws_cdk.aws_redshift_alpha import Cluster, Column, Table, TableDistStyle, TableSortStyle
import os
import json
from constructs import Construct

class MyRedshiftTablesStack(Stack):
 
    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)
 
        secret = secretsmanager.Secret.from_secret_name_v2(
            self, "MySecretId", <secret_name>
        )
 

        cluster = Cluster.from_cluster_attributes(
            self,
            "MyCluster",
            cluster_name=<cluster_name>,
            cluster_endpoint_address=<cluster_endpoint_address>,
            cluster_endpoint_port=<cluster_endpoint_port>,
        )
 
        table_columns = [
            Column(
                name="column_1",
                data_type="integer",
                dist_key=False,
                comment="My first column"
            )
        ]

        dist_style = TableDistStyle["KEY"]
        removal_policy = cdk.RemovalPolicy["DESTROY"]
        sort_style = TableSortStyle["AUTO"]
 
        Table(
            self,
            "MyTableName",
            cluster=cluster,
            database_name="my_db",
            admin_user=secret,
            table_columns=table_columns,
            dist_style=dist_style,
            removal_policy=removal_policy,
            sort_style=sort_style,
            table_name="my_schema.my_table",
            table_comment="My first table"
        )

TO BE

I suggest allowing the developer choosing existing roles as defaults for the execution of the Lambda functions whenever possible.
This way it would be possible to find a good balance between segregation of permissions and role quantity on a case-by-case basis.

Use Case

This feature would be crucial whenever the number of Redshift tables created with cdk-redshift-alpha gets bigger. Indeed, having a number of IAM roles which linearly scales with the number of tables is not sustainable and will eventually lead to the reach of IAM quotas, also providing very few benefits overall.

Proposed Solution

I suggest allowing the developer use existing roles for the execution of the Lambda functions.
In order to achieve such thing one should modify the underlying Lambdas creation to allow for external role selection (e.g. passing it via the Table construct of the library).

Other Information

If the proposed suggestion is not technically possible, I suggest to rethink about the number of roles generated by the library e.g. by sharing a single IAM role among the Lambda functions created in the same Stack.

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

CDK version used

2.165.0

Environment details (OS name and version, etc.)

Linux EC2 - aws/codebuild/standard:7.0

[pahud]

Yes you are right.

My PoC

export class IssueTriageStack extends Stack {
  readonly cluster: redshift.Cluster;
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

    // default vpc
    const vpc = ec2.Vpc.fromLookup(this, 'DefaultVPC', { isDefault: true });

    // create redshift.Cluster
    this.cluster = new redshift.Cluster(this, 'Redshift', {
      masterUser: {
        masterUsername: 'admin',
      },
      vpc,
    });

    // create redshift.Table
    for (let i = 0; i < 10; i++) {
      this.createTable(`Table${i}`);
    }
  }
  private createTable(tableId: string) {
    const table = new redshift.Table(this, tableId, {
      cluster: this.cluster,
      databaseName: 'my_database',
      tableName: 'my_table',
      tableColumns: [
        { name: 'id', dataType: 'integer' },
      ],
    });
  }
}

And it's going to create 11 iam Roles which should be avoided.

% npx cdk diff 2>&1 | grep -c "AWS::IAM::Role"
11

Tracking down the source code, I've found when a Table is created, a DatabaseQuery would be created, which creates a lambda SingletonFunction and I assume both the function and role should be singleton and shared. Not sure why it's creating a new role for every function.

I am tentative making it a p2 but will reach out to the team for inputs.