Calling the Bedrock Converse API with an image

[sterankin]

Edit: It looks like the image needs to be sent to the model as a uInt8Array but this cannot be sent as part of the REST call. A solution may be to base64 encode the image than convert it in the lambda if it is present. I do think that the Converse API should accept the image as base64 if its encoded in that format already.

---

Given a proxy lambda behind an api gateway, which is essentially exposed as a REST API - how should the client send an image data? base64?

const client = new BedrockRuntimeClient({ region: 'us-east-1' });

// create the command
    const requestBody = event.body ? (JSON.parse(event.body) as ConverseCommandInput) : null;

    command = new ConverseCommand({
      modelId,
      messages: requestBody ? requestBody.messages : [],
      inferenceConfig: requestBody?.inferenceConfig || {},
      system: requestBody?.system || [],
    });

const response = await client.send(command);

Since the code just passes through the json body, then it needs to be in the converse format. However I get this error message from the model (in this case its Claude 3.5 sonnet v2):

{
  "message": "The model returned the following errors: messages.0.content.1.image.source.bytes: Invalid image input"
}

This is an example payload, I have tried base64 encoding the image and passing it in the bytes field:

{

    "messages": [
        {
            "role": "user",
            "content": [
            {
                "text": "what is this image?"
            },
            {
                    "image": {
                        "format": "png",
                        "source": {
                            "bytes": "BASE64_IMAGE"
                        }
                    }
            }
        ]
        }
    ]
}

Any idea how an image can be sent via the payload?