Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to write group-level lag updates without write-locking the whole LagRegister #58

Open
detro opened this issue Sep 24, 2023 · 0 comments
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed module:lag_register

Comments

@detro
Copy link
Contributor

detro commented Sep 24, 2023

Rework the LagRegister internal data structure to hold a RwLock on the GroupWithLag level.

Currently the LagRegister looks like this:

pub struct LagRegister {
    pub(crate) lag_by_group: Arc<RwLock<HashMap<String, GroupWithLag>>>,
}

pub struct GroupWithLag {
    pub(crate) group: Group,
    pub(crate) lag_by_topic_partition: HashMap<TopicPartition, LagWithOwner>,
}

When we need to update lag information for a specific group, we are holding a whole write-lock on the LagRegister:

let mut w_guard = lag_register_groups.write().await;

We need to rework this so that each GroupWithLag is stored in the lag_by_group map, wrapped into its own RwLock. This way we could access the LagRegister first-level map with a read-lock, and only request the .write().await lock on the GroupWithLag.

This would enable concurrent updates of the register, enhancing performance and reducing lock contention drammatically.

@detro detro added enhancement New feature or request module:lag_register good first issue Good for newcomers help wanted Extra attention is needed labels Sep 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers help wanted Extra attention is needed module:lag_register
Projects
None yet
Development

No branches or pull requests

1 participant