Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add maxcompute batch predictor #626

Open
wants to merge 23 commits into
base: main
Choose a base branch
from
Open

Add maxcompute batch predictor #626

wants to merge 23 commits into from
+2,439 −687

Conversation

shydefoo
Copy link
Contributor

@shydefoo shydefoo commented Jan 31, 2025 •
edited
Loading

Description

  • This PR adds MaxCompute as a source and sink to batch prediction jobs. This allows users to perform batch prediction on data stored in MaxCompute and write the results back into MaxCompute.

Modifications

  • python/batch-predictor
    • merlinpyspark package source code modified
    • Added missing proto spec for PredictionJob protobuf
    • Added MaxCompute Jar dependencies to base.Dockerfile
  • python/sdk
    • Modified python sdk to include MaxComputeSource and MaxComputeSink classes
    • Modified swagger.yaml and regenerated python and go clients
    • Will add tests and readme in separate MR

Tests

Checklist

  • Added PR label
  • Added unit test, integration, and/or e2e tests
  • Tested locally
  • Updated documentation
  • Update Swagger spec if the PR introduce API changes
  • Regenerated Golang and Python client if the PR introduces API changes

Release Notes

- Support MaxCompute source and sink for batch prediction jobs

@shydefoo shydefoo added the enhancement New feature or request label Feb 5, 2025
shydefoo added 20 commits February 5, 2025 16:32
@shydefoo
Add missing prediction_job.proto file
a1c275b
@shydefoo
Make simple protobuf change and regenerate
aed4da9
@shydefoo
Add MaxCompute proto spec, generated specs, changes to sdk
dc17cb6
@shydefoo
Implement jdbc logic for odps source & sink
c1de0fb
@shydefoo
Add endpoint to protospec for sink and source
ae8fab0
@shydefoo
wip batch predictor, source load + prediction works
fdee606
@shydefoo
feat: Add sdk related classes, generate swagger files
6d2b286
@shydefoo
Separate project, table and schema
7417ce9
@shydefoo
Set enable limit to false, set custom dialect for sink
1c3892b
@shydefoo
Add more options
b16c620
@shydefoo
Modify spark base dockerfile
4a2f5ac
@shydefoo
Add prediction job config changes
d16d81c
@shydefoo
Upgrade batch-predictor requirements.txt
7561d86
@shydefoo
Update requirements.txt
394499e
@shydefoo
Fix merlin dependency issue, pin protoc version in batch-predictor Ma…
3d0ab36 
…kefile
@shydefoo
Fix mypy test checks
c3496d1
@shydefoo
Add swagger, validate batch config in sdk
6af1744
@shydefoo
Change protoc version to 4.25.6
ac995a3
@shydefoo
Remove required options from mc source and sink
a4c686a
@shydefoo
Fix spark base docker image
a94aa8f
@shydefoo shydefoo force-pushed the add-maxcompute-batch-predictor branch from 45925ec to a94aa8f Compare February 5, 2025 08:34
@shydefoo shydefoo self-assigned this Feb 6, 2025
@shydefoo shydefoo marked this pull request as ready for review February 6, 2025 07:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deadlycoconuts deadlycoconuts Awaiting requested review from deadlycoconuts

@bthari bthari Awaiting requested review from bthari

Assignees

@shydefoo shydefoo

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@shydefoo