Allow Bedrock Converse to accept native images instead of base64 (33% penalty!)

[jaredmcqueen]

Describe the feature

When using the converse API, any image file attached gets base64 encoded. Unfortunately that balloons the size to 33% higher than the actual file.

For example - if i have image.png that is 4.8MB and want to send it to claude sonnet 3.5 v2, it balloons to 6.4MB!

The model returned the following errors: messages.0.content.1.image.source: image exceeds 5 MB maximum: 6737924 bytes > 5242880 bytes

base64 incurs a 33% penalty!
https://en.wikipedia.org/wiki/Data_URI_scheme

Use Case

It's frustrating to limit images to LLM calls using converse API be 3.75 MB when the actual image size threshold should be 5.0MB or higher.

Proposed Solution

allow the ability to natively pass images in without base64 encoding

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

SDK version used

@aws-sdk/client-bedrock": "^3.687.0

Environment details (OS name and version, etc.)

MacOS

[zshzbh]

Hey @jaredmcqueen ,

Thanks for the feedback and sorry for the experience. I tried to use a 4.8MB image in Bedrock playground using Claude 3.5 and encountered the same issue. It's an issue from Bedrock service and I've reached out to bedrock team regarding this issue.

I will keep you updated!

Thanks!
Maggie

[kuhe]

Likely not addressable unless the service starts accepting a binary protocol. That would require a model update at the least, and needs an override here in the JS SDK to select the binary protocol, since JSON is preferred in the JavaScript SDK for overall performance reasons.

[jaredmcqueen]

a temporary solution could be to raise the allowed limit higher than 5.0MB. Claude natively accepts 20MB FYI.

[zshzbh]

Hey @jaredmcqueen ,

I got updates from service team - It is documented that 4.5MB is limit. https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Message.html

But the Bedrock UI says the limit is 5MB. Service team is going through internal discussions now and they haven't got any result yet. The temp solution would be using a smaller size image(less than 4.5MB), for example you can convert the image to a JPEG, WebP, or vector image format. I hope this can help you and I will keep you posted!

Thanks!