Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the join fetching exception metrics #907

Merged
merged 7 commits into from
Jan 23, 2025

Conversation

yuli-han
Copy link
Collaborator

Summary

Currently the join fetching exception metrics are not recorded correctly. Metrics.Context(Metrics.Environment.JoinFetching) doesn't include the join name so the exception is not recorded on join level.

The PR changes the behaviors for two types of exceptions:

  1. InvalidEntityException is handled on join level, the PR updates the metric context to add join name.
  2. MissingKeyException is handled on join part level, the PR updates the metric context to add join name(not group by name).

Why / Goal

Test Plan

Testing the missing key exception:
python3 ~/.local/bin/run.py --mode=fetch --name zipline_test/iceberg_realtime.v0 -k '{"name":null}' -t join --chronon-jar ~/test/chronon-embedded.jar --online-jar ~/test/online-all.jar | tee ~/test/fetch.log

Fetched result:
image

metrics:
image

  • Added Unit Tests
  • Covered by existing CI
  • Integration tested

Checklist

  • Documentation update

Reviewers

@hzding621 @pengyu-hou

Copy link
Collaborator

@pengyu-hou pengyu-hou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@yuli-han yuli-han merged commit aa93b49 into main Jan 23, 2025
8 checks passed
@yuli-han yuli-han deleted the ylh--update-join-fetch-metric branch January 23, 2025 06:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants