Added new Multimodal AI Block utilizing openrouter's API Service. #8290

ymrohit · 2024-10-10T13:55:06Z

This PR introduces the MultimodalAIBlock class, a new block that integrates multimodal AI models via OpenRouter API. It allows users to input a text prompt and an image URL, and select from several multimodal models for analysis. The block processes the inputs and returns a response based on the selected AI model.

…_model_name rather than free strings.

…rmat instead of free strings

…type a random seed

…ant-Gravitas/AutoGPT into toran/flux-block-tweaks

Toran/flux block tweaks

…rmat instead of free strings

…type a random seed

up to date

codiumai-pr-agent-pro · 2024-10-10T13:55:30Z

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 Security concerns API Key Exposure: The OpenRouter API key is being stored in the application's settings and passed directly to the OpenAI client. While it's not directly exposed in the code, there's a risk of it being logged or exposed through error messages. Consider implementing additional safeguards to protect the API key, such as encrypting it in storage and masking it in logs.
⚡ Recommended focus areas for review Error Handling The error handling in the `run` method is very generic. It catches all exceptions and returns them as strings. This might expose sensitive information or internal errors to the user. Consider implementing more specific error handling and providing user-friendly error messages. Input Validation There's no validation for the `image_url` input. The code should check if the URL is valid and accessible before passing it to the API. This could prevent runtime errors and improve user experience. API Key Security The API key is being passed directly to the OpenAI client. Consider implementing a more secure method of handling API keys, such as using environment variables or a secrets management system.

netlify · 2024-10-10T13:55:47Z

✅ Deploy Preview for auto-gpt-docs canceled.

Name	Link
🔨 Latest commit	`be1b253`
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/67220d851c41140008d6dd1b

aarushik93 · 2024-10-16T15:13:01Z

thanks for the PR @ymrohit - just curious there's no speech modality models available via Openrouter as of now right?

ymrohit · 2024-10-17T10:03:16Z

@aarushik93 I don't think there is any speech modality models available via openrouter at the moment.

* secrets * seal secrets * remove jina * replicate

github-actions · 2024-10-30T15:58:12Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

aarushik93 · 2024-11-06T23:44:51Z

autogpt_platform/infra/helm/autogpt-market/values.prod.yaml

-  DATABASE_URL: "AgAGn36yLuB4z1eh1MuBkUt3EydFYVoFi4v/XC7JgfdVyt5cMI9c59rv1Jztrep/TejpeJVPYKuyR07TH9d96Lc+8ALXSNZKCD8Q57/vqnaMRjPvnSwVaLzdx57F66953y2p6244QXpbLHaVGHG0ZowV0rY4XpU6VcmQZnSbSVyBXS+ZcYjF5AYN31+0TWTG8RRVAJ1WFA6Jq4fUtLokaZTr5+pPy4FGoEdBZJVAgNVBPNe6XIrCTJL7N/zwJxUWkqF1nUTWkr017SAodgicQbogcvOcHclgZLEf60nRPbxYUiLJ1uzcBGMT1Ze8n3TdIi5uMvqQ5fb/rHqyEBGjpcOPhZR5yZTeLVhdxOPoQ2gzMVfASGbfzrueGFq0SGKn1P6Vm1zboVD39/EunEfZ4iXULRVeqIXBZ7jhBIkaYx/NwX7LkaUfv+mpEI6xhYewHT9QH/QXlTjN5XAMERryJrAlOuEe6cv0nz7Up+sLkesPLKDaWXkcoKDhgA3XP1aTTgn2i8nzAOzB9q+Y05amEGl7Z1YYtVD/2pR96sZhaL1YcS32WP03HyZ3+ZRM1cq9AgEE776iMsOAgzbETj+Li6l/Oc2i8Nb9dSwjdz/fAjR9P6v3+JPl0Zl71P8Y7RKDawXyzfaNGOOl1lBeY3CVe+t3duSOdO7PuQOVxbGYIlqgSs5hdLRmTeQGywv2ucxS4p9IZTqooI1BoXHGW/JE5GbzxVMVanuY1eM+F/G1xY6W2tIyNo8a4qVjZ1ZbrdUFqNZkBPnJcekNFyo2PErfVMLJDVwhU8hOTbPmmlkYf9EAEYDkTFIeqXimN+oIK6xzYiomppECl8fTJ86doMK6fjD/DtdBfGuM35eGu9jQYpLoHv3XTpO0stx6utk="
-  SENTRY_DSN: "AgAd4XlyuPNl5fqxIGHg5L7y2nzjujpKoHUP9IUAOPiGEobtjiP1M5M8EB/0anjqrIj9GC+oqPWuLvL4rPwPMrTeiEyRmeHwlA9eQDJPi+C3sxu4cbXwUdh/Un5hLZkE/GuqMuOgALwZ599CmHQJVXye9mWbs8xWmyGQd6BJsEE32KMRIbI3Wk21luIzypmyeTg1WdpNuAwecwK4u+CkadwGWaJOyZJacJDRwvNzGWbjzGHAHHsllXk2gZFz4CCGNDyDVZrGPG0rCmJji72pu1jcjipoqYXte1WWfs9Yr9Nc1YU/yaOqz7/U3NhPzezQtvzbTF7RRJaBV4nOIjoOwSFBIeoYPjr8BSFNyCc3/8F+2nWJ7pDSWXbV5L2aDIP7ba0tLeqsuCxevvooGNDki1iGBykEp/NXOCQeZ0BCQYS9OuqCHzrrFA6jjkA6Ojxf2E7WZ8Y8LGVW9LGMggh1mM2c+fbV1za2psRrCQXR2ml46BOIt9/24Os56dggt/vbgXJ7l1mXQXQm+XU/S51ifK4g+HqkTE6PhAafm/sTLqeQKMDsy1/iEsnb8yphjpsydxBDgp+Rcp/zQOt0JuD4TTvpQE8NNwVQZJ2I6zMYwTu2tg+IIiv1fFtYt76BmOqjS/4USk9sjlHQRf562th3TRMENcg8crOOBmOmLaaEI/JCrtnpPMO8N8v26Q8bFSAzBFkyvO2kSUFNMkfKf2kBdtcufh3MysGtu/fL3Gh3BAOrefEDnHYFvvPx1w22FJsypl82mbU55wj2T/v9/UIxqeJpSW0/6HraNKzYAQ5Z9Fx2sfDBafMihiRWsURPfOqZkw=="
-  SUPABASE_SERVICE_ROLE_KEY: "AgAZKuZI0N/fmap8PaEoYti567yK5rugVVtMY0au4HTQBKKz336rcoMz/aGWWlPf/8AfpdEq1iRNKNRG+Xcz7MvCM9vjophOxCSoIgDts8zaKJ835NSKZZTtmQJRWdq8hAtTzA4QgKVq3uGmMiNi/8k3pKlif33sAdYRZoAOZ+RWb1lrLX7E3NTRcLfLbnQqoiMCt99Ks6npDgyxH2/ugBQGgP3DxmLPR2LA7FHOSlXbvLl5sA5FtxPXfcogjAL4u37FrG4yJ+7B+n+f9DpIPGpCpo7SyQNJ1iaY8OU1I6/PxmMr/ECshcQVWsLtKRgqSpZf6nO6DHXFzUFYufM4xJ2hELxeo4ihNbqt/GrZK9FR26vEohSPblL4pC3i5qdAbGkHhUAui0yzs4RApN+o/RhvE1Ip98PJoSBzz/TOUE9PASRa7d8kzYQGFBQ7o703vyAdWOXopoOkmbiZ3kmVxYxi5lXqi6xaN0jzjVldelk8p6SCcWadR1MEZjpyiY6bPM5/kLgHiq25UsU4N/ssqA+ZXma7zUMydbxbfxqW9D8kSs5H7Hcbbd07HDYPmBNSc3w5MTI5QPZzH/yPbRA0pKUX3yF05BhIbgDuhRYQG582IYk8nI+PmvqNxDFLEDEauzoX90qNUcidZS2jgjJ0BwmkB2zgBdoU13zELjdhdewoOJJkAVm6wIT83tUnkwpEW2nJg5bE1dmvKXQuoT85/03hIf36BXQ815m6Cp1lf4W4goOfF3U3HM7CGNFRC8st3l7Rem49AE7sPWBVc00a8NnKVzdbFohm0l+NRzRwFagU/gYkXHILpP0YPdfWHI8mptv0+UlirwuJXW0EUYU9hu9rCMB3u+ooSUFLK3ojlA7A34s7lJ+7VW0haxl/Wt0PQGwGptDT8ZBrLFv7NtCGq7FRlBfdlHEKouuaqgzIJq7THwEXyvOAkmZMhrEoQoOZol/cMXYntglTO9eZ5Vhapp6ttmO8SxQieAilImY="
+  DATABASE_URL: "AgCe47Y6QFnZRmNNpc3iVIcJ9zAZMPr/1OU61Slr0oM1yjneT7LgNTrAD9+QSu7Q0q2u+533FImOu8o/pKG0gNzEtyeSEEsegmCiaxs4Wi+/jVodU8CgfrYN3xkgi7QuD2fMs/zz+sv25LDg8LR1mIvkv+W/PkbqoVx3UAUXL7ZM+UvkFqie+69C/kwoH51xPaYi+FajSzql+VdhDc28aFkwrDApenyLbyKAHSnplWqpFDVtcQK4dL7K9EhRkUnWORNHiYT+N0Ras5tw1ief481nDud29nhlQ+5HIHGK5QouKmA1yJIJqavbzsDf7n+nWCwULIWXGi5d8A23e8vZAuF2CInUHAmP8d+xBDRx8aATZigoOM0GpHn5ljxNs0TCZ1HvQ8TjzQOpcgmBZ9l5Eod3y/PRhjGBRrcR550PAxpkjRj9VjfMN5NSiHVsGw7xKQr2+4yghv3VgSZFvOg1qUxmleJ9khwZq2Gel0uvhGStYw50x/C+EDMaJNQUaBDDbo5Bjy6/RNAEgrX0snQAZWCZSg4bsb4+2jRvo50W1wMr9076qOFViNn2YM+exIGXdmgggAKk7foZDZz1KgG+dAhKBXoUhXwHO9P450gq4xxDGSk0ZCe10OiszNkXVtmmE+Ryi5JjnbDsxLJP1tynrljFUb29VePt0zwZAhdnz0MQU1T9IVbjcKYB92h19lhcdgE="


please remove

aarushik93 · 2024-11-06T23:45:05Z

autogpt_platform/backend/backend/util/settings.py

@@ -250,7 +250,7 @@ class Secrets(UpdateTrackingModel["Secrets"], BaseSettings):
    replicate_api_key: str = Field(default="", description="Replicate API Key")
    unreal_speech_api_key: str = Field(default="", description="Unreal Speech API Key")
    ideogram_api_key: str = Field(default="", description="Ideogram API Key")
-
+    openrouter_api_key: str = Field(default="", description="OpenRouter API Key")


Please migrate to new credentials system

aarushik93

Please see comments

ymrohit and others added 29 commits September 29, 2024 07:27

Added Replicate Flux Blocks

dc61e78

updated poetry lock file for replicate

4d0ac7a

Refactor ReplicateFluxAdvancedModelBlock to use an enum for replicate…

133ed10

…_model_name rather than free strings.

Refactor ReplicateFluxAdvancedModelBlock to use an enum for output_fo…

e0cdfff

…rmat instead of free strings

Refactor ReplicateFluxAdvancedModelBlock to stop requiring people to …

cf67551

…type a random seed

Refactor ReplicateFluxAdvancedModelBlock to stop requiring people to …

ca54d06

…type a random seed

run format

deb64d8

Merge branch 'toran/flux-block-tweaks' of https://github.com/Signific…

bf851f3

…ant-Gravitas/AutoGPT into toran/flux-block-tweaks

poetry run format

75265b1

Delete ReplicateFluxBasicModelBlock

ea6b171

Mark model name as not advanced

3a743d9

tweak input order

212e977

Fix test

a788aa2

Merge pull request #1 from Significant-Gravitas/toran/flux-block-tweaks

df99c94

Toran/flux block tweaks

Refactor ReplicateFluxAdvancedModelBlock to use an enum for output_fo…

59b0eaa

…rmat instead of free strings

Refactor ReplicateFluxAdvancedModelBlock to stop requiring people to …

1ba3cdd

…type a random seed

poetry run format

aa82eec

Merge branch 'Significant-Gravitas-master'

2844fca

Merge pull request #3 from Significant-Gravitas/master

51f916f

up to date

Added Flux 1.1 pro option

3c6bb99

Update replicate_flux_advanced.py

a295c40

Update replicate_flux_advanced.py

68eeefd

poetry linted

44d902d

Merge branch 'master' into master

ea35b9e

Merge branch 'Significant-Gravitas:master' into master

653ae5b

Merge branch 'Significant-Gravitas:master' into master

f5b37ae

Merge branch 'Significant-Gravitas:master' into master

6385163

Merge branch 'Significant-Gravitas:master' into master

ddad41b

Created new Multimodal AI Block

6f15027

ymrohit requested a review from a team as a code owner October 10, 2024 13:55

codiumai-pr-agent-pro bot added the Review effort [1-5]: 3 label Oct 10, 2024

github-actions bot added size/m and removed Review effort [1-5]: 3 labels Oct 10, 2024

added test mock

39110c7

github-actions bot added size/l and removed size/m labels Oct 10, 2024

updated test_mock

8b31891

aarushik93 changed the base branch from master to dev October 14, 2024 15:27

Merge branch 'dev' into master

305bb0f

Merge branch 'dev' into master

b41f8f7

Merge branch 'dev' into master

be135ff

aarushik93 previously approved these changes Oct 17, 2024

View reviewed changes

ntindle and others added 2 commits October 22, 2024 17:11

Merge branch 'dev' into master

0b824d3

Merge branch 'Significant-Gravitas:master' into master

8352185

ymrohit dismissed aarushik93’s stale review via 8352185 October 24, 2024 09:54

github-actions bot added the platform/frontend AutoGPT Platform - Front end label Oct 24, 2024

Merge branch 'dev' into master

97b9ee8

github-actions bot removed the platform/frontend AutoGPT Platform - Front end label Oct 25, 2024

aarushik93 and others added 2 commits October 25, 2024 17:36

feat(platform): Add default secrets (Significant-Gravitas#8452)

88885f1

* secrets * seal secrets * remove jina * replicate

Merge branch 'Significant-Gravitas:master' into master

be1b253

ymrohit requested a review from a team as a code owner October 30, 2024 10:42

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Oct 30, 2024

aarushik93 reviewed Nov 6, 2024

View reviewed changes

aarushik93 requested changes Nov 6, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added new Multimodal AI Block utilizing openrouter's API Service. #8290

Added new Multimodal AI Block utilizing openrouter's API Service. #8290

ymrohit commented Oct 10, 2024

codiumai-pr-agent-pro bot commented Oct 10, 2024

netlify bot commented Oct 10, 2024 •

edited

Loading

aarushik93 commented Oct 16, 2024

ymrohit commented Oct 17, 2024

github-actions bot commented Oct 30, 2024

aarushik93 Nov 6, 2024

aarushik93 Nov 6, 2024

aarushik93 left a comment

Added new Multimodal AI Block utilizing openrouter's API Service. #8290

Are you sure you want to change the base?

Added new Multimodal AI Block utilizing openrouter's API Service. #8290

Conversation

ymrohit commented Oct 10, 2024

codiumai-pr-agent-pro bot commented Oct 10, 2024

PR Reviewer Guide 🔍

netlify bot commented Oct 10, 2024 • edited Loading

✅ Deploy Preview for auto-gpt-docs canceled.

aarushik93 commented Oct 16, 2024

ymrohit commented Oct 17, 2024

github-actions bot commented Oct 30, 2024

aarushik93 Nov 6, 2024

Choose a reason for hiding this comment

aarushik93 Nov 6, 2024

Choose a reason for hiding this comment

aarushik93 left a comment

Choose a reason for hiding this comment

netlify bot commented Oct 10, 2024 •

edited

Loading