Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

aiohttp barfs on dropbox download with "Invalid character in chunk size" #10322

Open
1 task done
jonathon-love opened this issue Jan 13, 2025 · 8 comments
Open
1 task done

aiohttp barfs on dropbox download with "Invalid character in chunk size" #10322

jonathon-love opened this issue Jan 13, 2025 · 8 comments
Labels
bug

Comments

@jonathon-love
Copy link

Describe the bug

aiohttp barfs on a dropbox download, that other software (i.e. curl) doesn't seem to have difficulty with.

To Reproduce

import aiohttp
import asyncio

async def download_file(url):
    async with aiohttp.ClientSession() as session:
        async with session.head(url, allow_redirects=True) as resp:
            pass

url = 'https://www.dropbox.com/scl/fi/818zs3kiois9qkv2axv9t/Tooth-Growth.omv?rlkey=f4m2lv069py0ezn8udmlxsosn&st=2owyrsrs&dl=1'

asyncio.run(download_file(url))

Expected behavior

200 OK

Logs/tracebacks

Traceback (most recent call last):
  File "/Users/XXXX/Library/Caches/pypoetry/virtualenvs/fred-L1l92eZt-py3.12/lib/python3.12/site-packages/aiohttp/client_proto.py", line 263, in data_received
    messages, upgraded, tail = self._parser.feed_data(data)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "aiohttp/_http_parser.pyx", line 558, in aiohttp._http_parser.HttpParser.feed_data
aiohttp.http_exceptions.BadHttpMessage: 400, message:
  Invalid character in chunk size:

    b'\x1f\x8b\x08'
      ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/XXXX/Library/Caches/pypoetry/virtualenvs/fred-L1l92eZt-py3.12/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1059, in start
    message, payload = await protocol.read()  # type: ignore[union-attr]
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/XXXX/Library/Caches/pypoetry/virtualenvs/fred-L1l92eZt-py3.12/lib/python3.12/site-packages/aiohttp/streams.py", line 671, in read
    await self._waiter
aiohttp.http_exceptions.HttpProcessingError: 400, message:
  Invalid character in chunk size:

    b'\x1f\x8b\x08'
      ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.3/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "<stdin>", line 3, in download_file
  File "/Users/XXXX/Library/Caches/pypoetry/virtualenvs/fred-L1l92eZt-py3.12/lib/python3.12/site-packages/aiohttp/client.py", line 1425, in __aenter__
    self._resp: _RetType = await self._coro
                           ^^^^^^^^^^^^^^^^
  File "/Users/XXXX/Library/Caches/pypoetry/virtualenvs/fred-L1l92eZt-py3.12/lib/python3.12/site-packages/aiohttp/client.py", line 730, in _request
    await resp.start(conn)
  File "/Users/XXXX/Library/Caches/pypoetry/virtualenvs/fred-L1l92eZt-py3.12/lib/python3.12/site-packages/aiohttp/client_reqrep.py", line 1061, in start
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 400, message="Invalid character in chunk size:\n\n  b'\\x1f\\x8b\\x08'\n    ^", url='https://uc7f6f4bf01d26ef6c48853d3c4c.dl.dropboxusercontent.com/cd/0/get/CiHGZDofY40BymZ9BQFqKYP2T_NQroYSU7UaqEPQtQDkAYFB_GCSpSAvxX-j1cFVIVHx0qSE5QFz-cJ5cNrnweRXgVIyXn04tg3k_asEek_G5zxbLThKClANeLOqWVCZKB5o3tm7u0TEiWx3aiwqB3-A/file?dl=1'


### Python Version

```console
$ python --version

`Python 3.12.3`

aiohttp Version

$ python -m pip show aiohttp

`Version: 3.11.11`

multidict Version

$ python -m pip show multidict

Version: 6.1.0

propcache Version

$ python -m pip show propcache

0.2.1

yarl Version

$ python -m pip show yarl

1.18.3

OS

linux, macOS

Related component

Client

Additional context

No response

Code of Conduct

  • I agree to follow the aio-libs Code of Conduct
@webknjaz
Copy link
Member

Is there a redirect involved? Might be due to recoding... The traceback shows the server responding with HTTP 400.

@jonathon-love
Copy link
Author

yes redirects in play

% curl -I -L -v "https://www.dropbox.com/scl/fi/818zs3kiois9qkv2axv9t/Tooth-Growth.omv?rlkey=f4m2lv069py0ezn8udmlxsosn&st=2owyrsrs&dl=1"

* Host www.dropbox.com:443 was resolved.
* IPv6: (none)
* IPv4: 162.125.83.18
*   Trying 162.125.83.18:443...
* Connected to www.dropbox.com (162.125.83.18) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Dropbox, Inc; CN=*.dropbox.com
*  start date: Nov 12 00:00:00 2024 GMT
*  expire date: Dec  8 23:59:59 2025 GMT
*  subjectAltName: host "www.dropbox.com" matched cert's "*.dropbox.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://www.dropbox.com/scl/fi/818zs3kiois9qkv2axv9t/Tooth-Growth.omv?rlkey=f4m2lv069py0ezn8udmlxsosn&st=2owyrsrs&dl=1
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: www.dropbox.com]
* [HTTP/2] [1] [:path: /scl/fi/818zs3kiois9qkv2axv9t/Tooth-Growth.omv?rlkey=f4m2lv069py0ezn8udmlxsosn&st=2owyrsrs&dl=1]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> HEAD /scl/fi/818zs3kiois9qkv2axv9t/Tooth-Growth.omv?rlkey=f4m2lv069py0ezn8udmlxsosn&st=2owyrsrs&dl=1 HTTP/2
> Host: www.dropbox.com
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/2 302 
HTTP/2 302 
< content-security-policy: script-src 'unsafe-eval' 'inline-speculation-rules' https://www.dropbox.com/static/api/ https://www.dropbox.com/pithos/* https://www.dropbox.com/page_success/ https://cfl.dropboxstatic.com/static/ https://www.dropboxstatic.com/static/ https://accounts.google.com/gsi/client https://canny.io/sdk.js https://www.paypal.com/sdk/js https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ 'unsafe-inline' ; font-src https://* data: ; worker-src https://www.dropbox.com/static/serviceworker/ https://www.dropbox.com/encrypted_folder_download/service_worker.js https://www.dropbox.com/service_worker.js blob: ; report-uri https://www.dropbox.com/csp_log?policy_name=metaserver-whitelist ; child-src https://www.dropbox.com/static/serviceworker/ blob: ; frame-ancestors 'self' https://*.dropbox.com ; form-action https://docs.google.com/document/fsip/ https://docs.google.com/spreadsheets/fsip/ https://docs.google.com/presentation/fsip/ https://docs.sandbox.google.com/document/fsip/ https://docs.sandbox.google.com/spreadsheets/fsip/ https://docs.sandbox.google.com/presentation/fsip/ https://*.purple.officeapps.live-int.com https://officeapps-df.live.com https://*.officeapps-df.live.com https://officeapps.live.com https://*.officeapps.live.com https://paper.dropbox.com/cloud-docs/edit 'self' https://www.dropbox.com/ https://dl-web.dropbox.com/ https://photos.dropbox.com/ https://paper.dropbox.com/ https://showcase.dropbox.com/ https://www.hellofax.com/ https://app.hellofax.com/ https://www.hellosign.com/ https://app.hellosign.com/ https://docsend.com/ https://www.docsend.com/ https://help.dropbox.com/ https://navi.dropbox.jp/ https://a.sprig.com/ https://selfguidedlearning.dropboxbusiness.com/ https://instructorledlearning.dropboxbusiness.com/ https://sales.dropboxbusiness.com/ https://accounts.google.com/ https://api.login.yahoo.com/ https://login.yahoo.com/ https://experience.dropbox.com/ https://pal-test.adyen.com https://2e83413d8036243b-Dropbox-pal-live.adyenpayments.com/ https://onedrive.live.com/picker ; frame-src https://* carousel: dbapi-6: dbapi-7: dbapi-8: dropbox-client: itms-apps: itms-appss: ; style-src https://* 'unsafe-inline' 'unsafe-eval' ; default-src https://www.dropbox.com/playlist/ https://www.dropbox.com/v/s/playlist/ https://*.dropboxusercontent.com/p/hls_master_playlist/ https://*.dropboxusercontent.com/p/hls_playlist/ ; img-src https://* data: blob: ; connect-src https://* ws://127.0.0.1:*/ws blob: wss://dsimports.dropbox.com/ ; object-src 'self' https://cfl.dropboxstatic.com/static/ https://www.dropboxstatic.com/static/ ; base-uri 'self' ; media-src https://* blob:
content-security-policy: script-src 'unsafe-eval' 'inline-speculation-rules' https://www.dropbox.com/static/api/ https://www.dropbox.com/pithos/* https://www.dropbox.com/page_success/ https://cfl.dropboxstatic.com/static/ https://www.dropboxstatic.com/static/ https://accounts.google.com/gsi/client https://canny.io/sdk.js https://www.paypal.com/sdk/js https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/ 'unsafe-inline' ; font-src https://* data: ; worker-src https://www.dropbox.com/static/serviceworker/ https://www.dropbox.com/encrypted_folder_download/service_worker.js https://www.dropbox.com/service_worker.js blob: ; report-uri https://www.dropbox.com/csp_log?policy_name=metaserver-whitelist ; child-src https://www.dropbox.com/static/serviceworker/ blob: ; frame-ancestors 'self' https://*.dropbox.com ; form-action https://docs.google.com/document/fsip/ https://docs.google.com/spreadsheets/fsip/ https://docs.google.com/presentation/fsip/ https://docs.sandbox.google.com/document/fsip/ https://docs.sandbox.google.com/spreadsheets/fsip/ https://docs.sandbox.google.com/presentation/fsip/ https://*.purple.officeapps.live-int.com https://officeapps-df.live.com https://*.officeapps-df.live.com https://officeapps.live.com https://*.officeapps.live.com https://paper.dropbox.com/cloud-docs/edit 'self' https://www.dropbox.com/ https://dl-web.dropbox.com/ https://photos.dropbox.com/ https://paper.dropbox.com/ https://showcase.dropbox.com/ https://www.hellofax.com/ https://app.hellofax.com/ https://www.hellosign.com/ https://app.hellosign.com/ https://docsend.com/ https://www.docsend.com/ https://help.dropbox.com/ https://navi.dropbox.jp/ https://a.sprig.com/ https://selfguidedlearning.dropboxbusiness.com/ https://instructorledlearning.dropboxbusiness.com/ https://sales.dropboxbusiness.com/ https://accounts.google.com/ https://api.login.yahoo.com/ https://login.yahoo.com/ https://experience.dropbox.com/ https://pal-test.adyen.com https://2e83413d8036243b-Dropbox-pal-live.adyenpayments.com/ https://onedrive.live.com/picker ; frame-src https://* carousel: dbapi-6: dbapi-7: dbapi-8: dropbox-client: itms-apps: itms-appss: ; style-src https://* 'unsafe-inline' 'unsafe-eval' ; default-src https://www.dropbox.com/playlist/ https://www.dropbox.com/v/s/playlist/ https://*.dropboxusercontent.com/p/hls_master_playlist/ https://*.dropboxusercontent.com/p/hls_playlist/ ; img-src https://* data: blob: ; connect-src https://* ws://127.0.0.1:*/ws blob: wss://dsimports.dropbox.com/ ; object-src 'self' https://cfl.dropboxstatic.com/static/ https://www.dropboxstatic.com/static/ ; base-uri 'self' ; media-src https://* blob:
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< location: https://uceb23753f12718e072510c60961.dl.dropboxusercontent.com/cd/0/inline/CiEi3aKPJlVEgxJk_vuIcEeDmlUWSwdfUmGT12QWH6FVl4cO7QlWxwgx7wUMJuv58Hz5Q65Hou5q-TdP3P42qHEM5cQJAM6oAO2cbc0o4Zi_A-jjsjZ6xCtED4CGBPB6XA4/file?dl=1#
location: https://uceb23753f12718e072510c60961.dl.dropboxusercontent.com/cd/0/inline/CiEi3aKPJlVEgxJk_vuIcEeDmlUWSwdfUmGT12QWH6FVl4cO7QlWxwgx7wUMJuv58Hz5Q65Hou5q-TdP3P42qHEM5cQJAM6oAO2cbc0o4Zi_A-jjsjZ6xCtED4CGBPB6XA4/file?dl=1#
< pragma: no-cache
pragma: no-cache
< referrer-policy: strict-origin-when-cross-origin
referrer-policy: strict-origin-when-cross-origin
< set-cookie: gvc=ODUyODM5Mjc5ODY1MTk1ODYyNDEwOTI5ODU4NjU1NTIyOTU4NA==; Path=/; Expires=Sat, 12 Jan 2030 10:37:17 GMT; HttpOnly; Secure; SameSite=None
set-cookie: gvc=ODUyODM5Mjc5ODY1MTk1ODYyNDEwOTI5ODU4NjU1NTIyOTU4NA==; Path=/; Expires=Sat, 12 Jan 2030 10:37:17 GMT; HttpOnly; Secure; SameSite=None
< set-cookie: t=Sh_yiBM3_zZ94XnO_KsivosI; Path=/; Domain=dropbox.com; Expires=Tue, 13 Jan 2026 10:37:17 GMT; HttpOnly; Secure; SameSite=None
set-cookie: t=Sh_yiBM3_zZ94XnO_KsivosI; Path=/; Domain=dropbox.com; Expires=Tue, 13 Jan 2026 10:37:17 GMT; HttpOnly; Secure; SameSite=None
< set-cookie: __Host-js_csrf=Sh_yiBM3_zZ94XnO_KsivosI; Path=/; Expires=Tue, 13 Jan 2026 10:37:17 GMT; Secure; SameSite=None
set-cookie: __Host-js_csrf=Sh_yiBM3_zZ94XnO_KsivosI; Path=/; Expires=Tue, 13 Jan 2026 10:37:17 GMT; Secure; SameSite=None
< set-cookie: __Host-ss=4WDPeTzG94; Path=/; Expires=Tue, 13 Jan 2026 10:37:17 GMT; HttpOnly; Secure; SameSite=Strict
set-cookie: __Host-ss=4WDPeTzG94; Path=/; Expires=Tue, 13 Jan 2026 10:37:17 GMT; HttpOnly; Secure; SameSite=Strict
< set-cookie: locale=en; Path=/; Domain=dropbox.com; Expires=Sat, 12 Jan 2030 10:37:17 GMT
set-cookie: locale=en; Path=/; Domain=dropbox.com; Expires=Sat, 12 Jan 2030 10:37:17 GMT
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< x-robots-tag: noindex, nofollow, noimageindex
x-robots-tag: noindex, nofollow, noimageindex
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< content-length: 17
content-length: 17
< date: Mon, 13 Jan 2025 10:37:18 GMT
date: Mon, 13 Jan 2025 10:37:18 GMT
< strict-transport-security: max-age=31536000; includeSubDomains
strict-transport-security: max-age=31536000; includeSubDomains
< server: envoy
server: envoy
< cache-control: no-cache, no-store
cache-control: no-cache, no-store
< x-dropbox-response-origin: far_remote
x-dropbox-response-origin: far_remote
< x-dropbox-request-id: bd026e63f07b461c861a0141d529217a
x-dropbox-request-id: bd026e63f07b461c861a0141d529217a
< 

* Ignoring the response-body
* Connection #0 to host www.dropbox.com left intact
* Issue another request to this URL: 'https://uceb23753f12718e072510c60961.dl.dropboxusercontent.com/cd/0/inline/CiEi3aKPJlVEgxJk_vuIcEeDmlUWSwdfUmGT12QWH6FVl4cO7QlWxwgx7wUMJuv58Hz5Q65Hou5q-TdP3P42qHEM5cQJAM6oAO2cbc0o4Zi_A-jjsjZ6xCtED4CGBPB6XA4/file?dl=1'
* Host uceb23753f12718e072510c60961.dl.dropboxusercontent.com:443 was resolved.
* IPv6: (none)
* IPv4: 162.125.83.15
*   Trying 162.125.83.15:443...
* Connected to uceb23753f12718e072510c60961.dl.dropboxusercontent.com (162.125.83.15) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Dropbox, Inc; CN=*.dl.dropboxusercontent.com
*  start date: Mar 25 00:00:00 2024 GMT
*  expire date: Mar 11 23:59:59 2025 GMT
*  subjectAltName: host "uceb23753f12718e072510c60961.dl.dropboxusercontent.com" matched cert's "*.dl.dropboxusercontent.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://uceb23753f12718e072510c60961.dl.dropboxusercontent.com/cd/0/inline/CiEi3aKPJlVEgxJk_vuIcEeDmlUWSwdfUmGT12QWH6FVl4cO7QlWxwgx7wUMJuv58Hz5Q65Hou5q-TdP3P42qHEM5cQJAM6oAO2cbc0o4Zi_A-jjsjZ6xCtED4CGBPB6XA4/file?dl=1
* [HTTP/2] [1] [:method: HEAD]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: uceb23753f12718e072510c60961.dl.dropboxusercontent.com]
* [HTTP/2] [1] [:path: /cd/0/inline/CiEi3aKPJlVEgxJk_vuIcEeDmlUWSwdfUmGT12QWH6FVl4cO7QlWxwgx7wUMJuv58Hz5Q65Hou5q-TdP3P42qHEM5cQJAM6oAO2cbc0o4Zi_A-jjsjZ6xCtED4CGBPB6XA4/file?dl=1]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> HEAD /cd/0/inline/CiEi3aKPJlVEgxJk_vuIcEeDmlUWSwdfUmGT12QWH6FVl4cO7QlWxwgx7wUMJuv58Hz5Q65Hou5q-TdP3P42qHEM5cQJAM6oAO2cbc0o4Zi_A-jjsjZ6xCtED4CGBPB6XA4/file?dl=1 HTTP/2
> Host: uceb23753f12718e072510c60961.dl.dropboxusercontent.com
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/2 302 
HTTP/2 302 
< content-type: application/json
content-type: application/json
< cache-control: no-cache
cache-control: no-cache
< content-security-policy: sandbox
content-security-policy: sandbox
< etag: 1736727306872417d
etag: 1736727306872417d
< location: /cd/0/inline2/CiEPuxccYngED2Y3lcJZd276KCSjpCzeSCIsG5o9cIs0MSQjqCVqS12zUm53ZvO6XHAtOS24G-prrElYBkBH74_Au0-jnmUGs-x9b-zm7O1wxp5Dr_v65NG04iP7RBcX_zakuvgZN_BXEeAaLQ2TtjZJSuC4Slwtw5SOX4YzE1UMkaULWIOPUv0ujgPvNOVZ6c5oZgYcrpeg8yAE46e5gKYQ-FU1s7nlXGSNAT1zIa4T-pwuHvV7ADoWFIc9MBFviQGDdWFeUq2u5Xgx8wPas2FppZ7mIQfffsIFbBpjb_o4Y83ofGk2GWD6UbqiwZaFVqlNqIdw4IUCa181c-52veI8a4LGXV9vUmz8_Ogetpn2zQ/file?dl=1
location: /cd/0/inline2/CiEPuxccYngED2Y3lcJZd276KCSjpCzeSCIsG5o9cIs0MSQjqCVqS12zUm53ZvO6XHAtOS24G-prrElYBkBH74_Au0-jnmUGs-x9b-zm7O1wxp5Dr_v65NG04iP7RBcX_zakuvgZN_BXEeAaLQ2TtjZJSuC4Slwtw5SOX4YzE1UMkaULWIOPUv0ujgPvNOVZ6c5oZgYcrpeg8yAE46e5gKYQ-FU1s7nlXGSNAT1zIa4T-pwuHvV7ADoWFIc9MBFviQGDdWFeUq2u5Xgx8wPas2FppZ7mIQfffsIFbBpjb_o4Y83ofGk2GWD6UbqiwZaFVqlNqIdw4IUCa181c-52veI8a4LGXV9vUmz8_Ogetpn2zQ/file?dl=1
< referrer-policy: no-referrer
referrer-policy: no-referrer
< set-cookie:  uc_session=bCkHnAr1AtATJck2rXB9JJ1dFnejeUFMPoOG3zyTqNjJTCmmuVgsU2CnJorvk8lG; Domain=dropboxusercontent.com; HttpOnly; Path=/; SameSite=None; Secure
set-cookie:  uc_session=bCkHnAr1AtATJck2rXB9JJ1dFnejeUFMPoOG3zyTqNjJTCmmuVgsU2CnJorvk8lG; Domain=dropboxusercontent.com; HttpOnly; Path=/; SameSite=None; Secure
< vary: Origin, Accept-Encoding
vary: Origin, Accept-Encoding
< x-robots-tag: noindex, nofollow, noimageindex
x-robots-tag: noindex, nofollow, noimageindex
< date: Mon, 13 Jan 2025 10:37:18 GMT
date: Mon, 13 Jan 2025 10:37:18 GMT
< server: envoy
server: envoy
< strict-transport-security: max-age=31536000; includeSubDomains; preload
strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-dropbox-response-origin: far_remote
x-dropbox-response-origin: far_remote
< x-dropbox-request-id: df1e5c7fd8874d38bd1ab5b931da2536
x-dropbox-request-id: df1e5c7fd8874d38bd1ab5b931da2536
< 

* Ignoring the response-body
* Connection #1 to host uceb23753f12718e072510c60961.dl.dropboxusercontent.com left intact
* Issue another request to this URL: 'https://uceb23753f12718e072510c60961.dl.dropboxusercontent.com/cd/0/inline2/CiEPuxccYngED2Y3lcJZd276KCSjpCzeSCIsG5o9cIs0MSQjqCVqS12zUm53ZvO6XHAtOS24G-prrElYBkBH74_Au0-jnmUGs-x9b-zm7O1wxp5Dr_v65NG04iP7RBcX_zakuvgZN_BXEeAaLQ2TtjZJSuC4Slwtw5SOX4YzE1UMkaULWIOPUv0ujgPvNOVZ6c5oZgYcrpeg8yAE46e5gKYQ-FU1s7nlXGSNAT1zIa4T-pwuHvV7ADoWFIc9MBFviQGDdWFeUq2u5Xgx8wPas2FppZ7mIQfffsIFbBpjb_o4Y83ofGk2GWD6UbqiwZaFVqlNqIdw4IUCa181c-52veI8a4LGXV9vUmz8_Ogetpn2zQ/file?dl=1'
* Found bundle for host: 0x600002705d40 [can multiplex]
* Re-using existing connection with host uceb23753f12718e072510c60961.dl.dropboxusercontent.com
* [HTTP/2] [3] OPENED stream for https://uceb23753f12718e072510c60961.dl.dropboxusercontent.com/cd/0/inline2/CiEPuxccYngED2Y3lcJZd276KCSjpCzeSCIsG5o9cIs0MSQjqCVqS12zUm53ZvO6XHAtOS24G-prrElYBkBH74_Au0-jnmUGs-x9b-zm7O1wxp5Dr_v65NG04iP7RBcX_zakuvgZN_BXEeAaLQ2TtjZJSuC4Slwtw5SOX4YzE1UMkaULWIOPUv0ujgPvNOVZ6c5oZgYcrpeg8yAE46e5gKYQ-FU1s7nlXGSNAT1zIa4T-pwuHvV7ADoWFIc9MBFviQGDdWFeUq2u5Xgx8wPas2FppZ7mIQfffsIFbBpjb_o4Y83ofGk2GWD6UbqiwZaFVqlNqIdw4IUCa181c-52veI8a4LGXV9vUmz8_Ogetpn2zQ/file?dl=1
* [HTTP/2] [3] [:method: HEAD]
* [HTTP/2] [3] [:scheme: https]
* [HTTP/2] [3] [:authority: uceb23753f12718e072510c60961.dl.dropboxusercontent.com]
* [HTTP/2] [3] [:path: /cd/0/inline2/CiEPuxccYngED2Y3lcJZd276KCSjpCzeSCIsG5o9cIs0MSQjqCVqS12zUm53ZvO6XHAtOS24G-prrElYBkBH74_Au0-jnmUGs-x9b-zm7O1wxp5Dr_v65NG04iP7RBcX_zakuvgZN_BXEeAaLQ2TtjZJSuC4Slwtw5SOX4YzE1UMkaULWIOPUv0ujgPvNOVZ6c5oZgYcrpeg8yAE46e5gKYQ-FU1s7nlXGSNAT1zIa4T-pwuHvV7ADoWFIc9MBFviQGDdWFeUq2u5Xgx8wPas2FppZ7mIQfffsIFbBpjb_o4Y83ofGk2GWD6UbqiwZaFVqlNqIdw4IUCa181c-52veI8a4LGXV9vUmz8_Ogetpn2zQ/file?dl=1]
* [HTTP/2] [3] [user-agent: curl/8.7.1]
* [HTTP/2] [3] [accept: */*]
> HEAD /cd/0/inline2/CiEPuxccYngED2Y3lcJZd276KCSjpCzeSCIsG5o9cIs0MSQjqCVqS12zUm53ZvO6XHAtOS24G-prrElYBkBH74_Au0-jnmUGs-x9b-zm7O1wxp5Dr_v65NG04iP7RBcX_zakuvgZN_BXEeAaLQ2TtjZJSuC4Slwtw5SOX4YzE1UMkaULWIOPUv0ujgPvNOVZ6c5oZgYcrpeg8yAE46e5gKYQ-FU1s7nlXGSNAT1zIa4T-pwuHvV7ADoWFIc9MBFviQGDdWFeUq2u5Xgx8wPas2FppZ7mIQfffsIFbBpjb_o4Y83ofGk2GWD6UbqiwZaFVqlNqIdw4IUCa181c-52veI8a4LGXV9vUmz8_Ogetpn2zQ/file?dl=1 HTTP/2
> Host: uceb23753f12718e072510c60961.dl.dropboxusercontent.com
> User-Agent: curl/8.7.1
> Accept: */*
> 
* Request completely sent off
< HTTP/2 200 
HTTP/2 200 
< content-type: application/json
content-type: application/json
< accept-ranges: bytes
accept-ranges: bytes
< cache-control: max-age=60
cache-control: max-age=60
< content-disposition: attachment; filename=unspecified
content-disposition: attachment; filename=unspecified
< content-security-policy: sandbox
content-security-policy: sandbox
< content-security-policy: report-uri https://www.dropbox.com/csp_log?policy_name=blockserver-usercontent ; sandbox allow-forms allow-scripts allow-top-navigation allow-popups
content-security-policy: report-uri https://www.dropbox.com/csp_log?policy_name=blockserver-usercontent ; sandbox allow-forms allow-scripts allow-top-navigation allow-popups
< content-security-policy: form-action 'none' ; report-uri https://www.dropbox.com/csp_log?policy_name=blockserver-noscript ; script-src 'none'
content-security-policy: form-action 'none' ; report-uri https://www.dropbox.com/csp_log?policy_name=blockserver-noscript ; script-src 'none'
< etag: 1736727306872417d
etag: 1736727306872417d
< pragma: public
pragma: public
< referrer-policy: no-referrer
referrer-policy: no-referrer
< set-cookie:  uc_session=6XxzAhBTSoFY5pgkaN63ZY50rYJq2ZY8a0dsXRwxXNAX2q0CH2z5cMIEvtd0Ek08; Domain=dropboxusercontent.com; HttpOnly; Path=/; SameSite=None; Secure
set-cookie:  uc_session=6XxzAhBTSoFY5pgkaN63ZY50rYJq2ZY8a0dsXRwxXNAX2q0CH2z5cMIEvtd0Ek08; Domain=dropboxusercontent.com; HttpOnly; Path=/; SameSite=None; Secure
< vary: Origin, Accept-Encoding
vary: Origin, Accept-Encoding
< x-content-security-policy: sandbox
x-content-security-policy: sandbox
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-robots-tag: noindex, nofollow, noimageindex
x-robots-tag: noindex, nofollow, noimageindex
< x-server-response-time: 187
x-server-response-time: 187
< x-webkit-csp: sandbox
x-webkit-csp: sandbox
< date: Mon, 13 Jan 2025 10:37:18 GMT
date: Mon, 13 Jan 2025 10:37:18 GMT
< server: envoy
server: envoy
< strict-transport-security: max-age=31536000; includeSubDomains; preload
strict-transport-security: max-age=31536000; includeSubDomains; preload
< content-length: 2146
content-length: 2146
< x-dropbox-response-origin: far_remote
x-dropbox-response-origin: far_remote
< x-dropbox-request-id: da6721bade384c63b8a156a1f059b964
x-dropbox-request-id: da6721bade384c63b8a156a1f059b964
<

@webknjaz
Copy link
Member

Try passing requote_redirect_url=False when initializing the client session: https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.requote_redirect_url.

You may also disable the redirects and walk them manually, verifying that the Location header value is quoted appropriately.

@webknjaz
Copy link
Member

@jonathon-love also note that curl goes for HTTP/2 which aiohttp won't upgrade. That's another difference.

@jonathon-love
Copy link
Author

thanks for your help.

adding requote_redirect_url=False doesn't help:

async def download_file(url):
    async with aiohttp.ClientSession(requote_redirect_url=False) as session:
        async with session.head(url, allow_redirects=True) as resp:
            pass

url = 'https://www.dropbox.com/scl/fi/818zs3kiois9qkv2axv9t/Tooth-Growth.omv?rlkey=f4m2lv069py0ezn8udmlxsosn&st=2owyrsrs&dl=1'

asyncio.run(download_file(url))

and the addition of --http1.1 to curl continues to work, i.e.

curl -I -L -v --http1.1 "https://www.dropbox.com/scl/fi/818zs3kiois9qkv2axv9t/Tooth-Growth.omv?rlkey=f4m2lv069py0ezn8udmlxsosn&st=2owyrsrs&dl=1"

one thing worth noting is that this is a .head() call, rather than a .get().

        async with session.head(url, allow_redirects=True) as resp:
            pass

if i change this to a .get() call it all works as expected.

with thanks

@Dreamsorcerer
Copy link
Member

Is the issue present with AIOHTTP_NO_EXTENSIONS=1?

I'll try and reproduce in a couple of days.

@jonathon-love
Copy link
Author

it works when AIOHTTP_NO_EXTENSIONS=1 is set!

@Dreamsorcerer
Copy link
Member

I wonder if there's something we're missing, to tell the C parser that it is a HEAD response. I suspect it's trying to parse a full response..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@webknjaz @Dreamsorcerer @jonathon-love