Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DNM: Add a Cython implementation for ConnectionKey with hash caching #10072

Closed
wants to merge 6 commits into from
Closed

DNM: Add a Cython implementation for ConnectionKey with hash caching #10072

wants to merge 6 commits into from

Conversation

bdraco
Copy link
Member

@bdraco bdraco commented Nov 29, 2024 •
edited
Loading

I noticed on the profiles that we spend a lot of time in connection.py in functions that hash and compare the ConnectionKey. We moved to a NamedTuple to speed up the hashing but its still looks like its taking up quite
a bit of overhead because we use it as a key in so many operations.

Add a Cython implementation which caches the hash to see if there is a measurable improvement.

bdraco
bdraco commented Nov 29, 2024
View reviewed changes
aiohttp/client_reqrep.py Outdated Show resolved Hide resolved
@codspeed-hq CodSpeed HQ
Copy link

codspeed-hq bot commented Nov 29, 2024 •
edited
Loading

CodSpeed Performance Report

Merging #10072 will not alter performance

Comparing cython_connection_key_implement (5cecb62) with master (1fa237f)

Summary

✅ 44 untouched benchmarks

@codecov Codecov
Copy link

codecov bot commented Nov 29, 2024 •
edited
Loading

❌ 13 Tests Failed:

Tests completed Failed Passed Skipped
3454 13 3441 23
View the top 3 failed tests by shortest run time 
tests.test_proxy_functional test_proxy_http_absolute_path[pyloop]
Stack Traces | 0.038s run time 
No failure message available
tests.test_proxy_functional test_proxy_https_bad_response[pyloop]
Stack Traces | 0.041s run time 
proxy_test_server = <function proxy_test_server.<locals>.proxy_server at 0x1167076a0>

    #x1B[0m#x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m #x1B[92mtest_proxy_https_bad_response#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        proxy_test_server: Callable[[], Awaitable[mock.Mock]]#x1B[90m#x1B[39;49;00m
    ) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
        url = #x1B[33m"#x1B[39;49;00m#x1B[33mhttps://secure.aiohttp.io/path#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        proxy = #x1B[94mawait#x1B[39;49;00m proxy_test_server()#x1B[90m#x1B[39;49;00m
        proxy.return_value = #x1B[96mdict#x1B[39;49;00m(status=#x1B[94m502#x1B[39;49;00m, headers={#x1B[33m"#x1B[39;49;00m#x1B[33mProxy-Agent#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[33m"#x1B[39;49;00m#x1B[33mTestProxy#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m})#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[94mwith#x1B[39;49;00m pytest.raises(aiohttp.ClientHttpProxyError):#x1B[90m#x1B[39;49;00m
>           #x1B[94mawait#x1B[39;49;00m get_request(url=url, proxy=proxy.url)#x1B[90m#x1B[39;49;00m

proxy      = <Mock id='4664185936'>
proxy_test_server = <function proxy_test_server.<locals>.proxy_server at 0x1167076a0>
url        = 'https://secure.aiohttp.io/path'

#x1B[1m#x1B[31mtests/test_proxy_functional.py#x1B[0m:577: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
#x1B[1m#x1B[31mtests/test_proxy_functional.py#x1B[0m:65: in get_request
    #x1B[0m#x1B[94masync#x1B[39;49;00m #x1B[94mwith#x1B[39;49;00m client.request(method, url, **kwargs) #x1B[94mas#x1B[39;49;00m resp:#x1B[90m#x1B[39;49;00m
        client     = <aiohttp.client.ClientSession object at 0x11630b240>
        connector  = <aiohttp.connector.TCPConnector object at 0x116244f10>
        kwargs     = {'proxy': URL('http://127.0.0.1:49873/')}
        method     = 'GET'
        trust_env  = False
        url        = 'https://secure.aiohttp.io/path'
#x1B[1m#x1B[31maiohttp/client.py#x1B[0m:1330: in __aenter__
    #x1B[0m#x1B[96mself#x1B[39;49;00m._resp: _RetType = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._coro#x1B[90m#x1B[39;49;00m
        self       = <aiohttp.client._BaseRequestContextManager object at 0x1126e12a0>
#x1B[1m#x1B[31maiohttp/client.py#x1B[0m:644: in _request
    #x1B[0mconn = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._connector.connect(#x1B[90m#x1B[39;49;00m
        all_cookies = <SimpleCookie: >
        allow_redirects = True
        auth       = None
        auth_from_url = None
        auto_decompress = True
        chunked    = None
        compress   = False
        cookies    = None
        data       = None
        expect100  = False
        handle     = None
        headers    = <CIMultiDict()>
        history    = []
        json       = None
        max_field_size = 8190
        max_line_size = 8190
        max_redirects = 10
        method     = 'GET'
        params     = {}
        proxy      = URL('http://127.0.0.1:49873/')
        proxy_auth = None
        proxy_headers = <CIMultiDict('Host': 'secure.aiohttp.io')>
        raise_for_status = None
        read_bufsize = 65536
        read_until_eof = True
        real_timeout = ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=30, ceil_threshold=5)
        redirects  = 0
        req        = <aiohttp.client_reqrep.ClientRequest object at 0x116247f10>
        retry_persistent_connection = True
        self       = <aiohttp.client.ClientSession object at 0x11630b240>
        server_hostname = None
        skip_auto_headers = None
        skip_headers = None
        ssl        = True
        str_or_url = 'https://secure.aiohttp.io/path'
        timeout    = <_SENTINEL.sentinel: 1>
        timer      = <aiohttp.helpers.TimerContext object at 0x1162ad010>
        tm         = <aiohttp.helpers.TimeoutHandle object at 0x116244540>
        trace_request_ctx = None
        traces     = []
        url        = URL('https://secure.aiohttp.io/path')
        version    = HttpVersion(major=1, minor=1)
#x1B[1m#x1B[31maiohttp/connector.py#x1B[0m:537: in connect
    #x1B[0mproto = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._create_connection(req, traces, timeout)#x1B[90m#x1B[39;49;00m
        conn       = None
        key        = <aiohttp._client_reqrep.ConnectionKey object at 0x1162af220>
        placeholder = <aiohttp.connector._TransportPlaceholder object at 0x1126e3940>
        req        = <aiohttp.client_reqrep.ClientRequest object at 0x116247f10>
        self       = <aiohttp.connector.TCPConnector object at 0x116244f10>
        timeout    = ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=30, ceil_threshold=5)
        traces     = []
#x1B[1m#x1B[31maiohttp/connector.py#x1B[0m:1041: in _create_connection
    #x1B[0m_, proto = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._create_proxy_connection(req, traces, timeout)#x1B[90m#x1B[39;49;00m
        req        = <aiohttp.client_reqrep.ClientRequest object at 0x116247f10>
        self       = <aiohttp.connector.TCPConnector object at 0x116244f10>
        timeout    = ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=30, ceil_threshold=5)
        traces     = []
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <aiohttp.connector.TCPConnector object at 0x116244f10>
req = <aiohttp.client_reqrep.ClientRequest object at 0x116247f10>, traces = []
timeout = ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=30, ceil_threshold=5)

    #x1B[0m#x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m #x1B[92m_create_proxy_connection#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        #x1B[96mself#x1B[39;49;00m, req: ClientRequest, traces: List[#x1B[33m"#x1B[39;49;00m#x1B[33mTrace#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m], timeout: #x1B[33m"#x1B[39;49;00m#x1B[33mClientTimeout#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    ) -> Tuple[asyncio.BaseTransport, ResponseHandler]:#x1B[90m#x1B[39;49;00m
        headers: Dict[#x1B[96mstr#x1B[39;49;00m, #x1B[96mstr#x1B[39;49;00m] = {}#x1B[90m#x1B[39;49;00m
        #x1B[94mif#x1B[39;49;00m req.proxy_headers #x1B[95mis#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
            headers = req.proxy_headers  #x1B[90m# type: ignore[assignment]#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        headers[hdrs.HOST] = req.headers[hdrs.HOST]#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        url = req.proxy#x1B[90m#x1B[39;49;00m
        #x1B[94massert#x1B[39;49;00m url #x1B[95mis#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        proxy_req = ClientRequest(#x1B[90m#x1B[39;49;00m
            hdrs.METH_GET,#x1B[90m#x1B[39;49;00m
            url,#x1B[90m#x1B[39;49;00m
            headers=headers,#x1B[90m#x1B[39;49;00m
            auth=req.proxy_auth,#x1B[90m#x1B[39;49;00m
            loop=#x1B[96mself#x1B[39;49;00m._loop,#x1B[90m#x1B[39;49;00m
            ssl=req.ssl,#x1B[90m#x1B[39;49;00m
        )#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[90m# create connection to proxy server#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        transport, proto = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._create_direct_connection(#x1B[90m#x1B[39;49;00m
            proxy_req, [], timeout, client_error=ClientProxyConnectionError#x1B[90m#x1B[39;49;00m
        )#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        auth = proxy_req.headers.pop(hdrs.AUTHORIZATION, #x1B[94mNone#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
        #x1B[94mif#x1B[39;49;00m auth #x1B[95mis#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
            #x1B[94mif#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m req.is_ssl():#x1B[90m#x1B[39;49;00m
                req.headers[hdrs.PROXY_AUTHORIZATION] = auth#x1B[90m#x1B[39;49;00m
            #x1B[94melse#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                proxy_req.headers[hdrs.PROXY_AUTHORIZATION] = auth#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[94mif#x1B[39;49;00m req.is_ssl():#x1B[90m#x1B[39;49;00m
            #x1B[96mself#x1B[39;49;00m._warn_about_tls_in_tls(transport, req)#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
            #x1B[90m# For HTTPS requests over HTTP proxy#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# we must notify proxy to tunnel connection#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# so we send CONNECT command:#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m#   CONNECT www.python.org:443 HTTP/1.1#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m#   Host: www.python.org#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m##x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# next we must do TLS handshake and so on#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# to do this we must wrap raw socket into secure one#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# asyncio handles this perfectly#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            proxy_req.method = hdrs.METH_CONNECT#x1B[90m#x1B[39;49;00m
            proxy_req.url = req.url#x1B[90m#x1B[39;49;00m
>           key = req.connection_key._replace(#x1B[90m#x1B[39;49;00m
                proxy=#x1B[94mNone#x1B[39;49;00m, proxy_auth=#x1B[94mNone#x1B[39;49;00m, proxy_headers_hash=#x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            )#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE           AttributeError: 'aiohttp._client_reqrep.ConnectionKey' object has no attribute '_replace'#x1B[0m

auth       = None
headers    = <CIMultiDict('Host': 'secure.aiohttp.io')>
proto      = <aiohttp.client_proto.ResponseHandler object at 0x1162bf000>
proxy_req  = <aiohttp.client_reqrep.ClientRequest object at 0x116245a50>
req        = <aiohttp.client_reqrep.ClientRequest object at 0x116247f10>
self       = <aiohttp.connector.TCPConnector object at 0x116244f10>
timeout    = ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=30, ceil_threshold=5)
traces     = []
transport  = <_SelectorSocketTransport fd=23 read=polling write=<idle, bufsize=0>>
url        = URL('http://127.0.0.1:49873/')

#x1B[1m#x1B[31maiohttp/connector.py#x1B[0m:1368: AttributeError
tests.test_proxy.TestProxy test_proxy_server_hostname_default
Stack Traces | 0.05s run time 
self = <test_proxy.TestProxy testMethod=test_proxy_server_hostname_default>
start_connection = <function start_connection at 0x11669c220>
ClientRequestMock = <MagicMock name='ClientRequest' id='4671096080'>

    #x1B[0m#x1B[37m@mock#x1B[39;49;00m.patch(#x1B[33m"#x1B[39;49;00m#x1B[33maiohttp.connector.ClientRequest#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
    #x1B[37m@mock#x1B[39;49;00m.patch(#x1B[90m#x1B[39;49;00m
        #x1B[33m"#x1B[39;49;00m#x1B[33maiohttp.connector.aiohappyeyeballs.start_connection#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
        autospec=#x1B[94mTrue#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
        spec_set=#x1B[94mTrue#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
    )#x1B[90m#x1B[39;49;00m
    #x1B[94mdef#x1B[39;49;00m #x1B[92mtest_proxy_server_hostname_default#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        #x1B[96mself#x1B[39;49;00m, start_connection: mock.Mock, ClientRequestMock: mock.Mock#x1B[90m#x1B[39;49;00m
    ) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
        proxy_req = ClientRequest(#x1B[90m#x1B[39;49;00m
            #x1B[33m"#x1B[39;49;00m#x1B[33mGET#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, URL(#x1B[33m"#x1B[39;49;00m#x1B[33mhttp://proxy.example.com#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m), loop=#x1B[96mself#x1B[39;49;00m.loop#x1B[90m#x1B[39;49;00m
        )#x1B[90m#x1B[39;49;00m
        ClientRequestMock.return_value = proxy_req#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        proxy_resp = ClientResponse(#x1B[90m#x1B[39;49;00m
            #x1B[33m"#x1B[39;49;00m#x1B[33mget#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
            URL(#x1B[33m"#x1B[39;49;00m#x1B[33mhttp://proxy.example.com#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m
            request_info=mock.Mock(),#x1B[90m#x1B[39;49;00m
            writer=#x1B[94mNone#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
            continue100=#x1B[94mNone#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
            timer=TimerNoop(),#x1B[90m#x1B[39;49;00m
            traces=[],#x1B[90m#x1B[39;49;00m
            loop=#x1B[96mself#x1B[39;49;00m.loop,#x1B[90m#x1B[39;49;00m
            session=mock.Mock(),#x1B[90m#x1B[39;49;00m
        )#x1B[90m#x1B[39;49;00m
        #x1B[94mwith#x1B[39;49;00m mock.patch.object(#x1B[90m#x1B[39;49;00m
            proxy_req, #x1B[33m"#x1B[39;49;00m#x1B[33msend#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, autospec=#x1B[94mTrue#x1B[39;49;00m, return_value=proxy_resp#x1B[90m#x1B[39;49;00m
        ):#x1B[90m#x1B[39;49;00m
            #x1B[94mwith#x1B[39;49;00m mock.patch.object(proxy_resp, #x1B[33m"#x1B[39;49;00m#x1B[33mstart#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, autospec=#x1B[94mTrue#x1B[39;49;00m) #x1B[94mas#x1B[39;49;00m m:#x1B[90m#x1B[39;49;00m
                m.return_value.status = #x1B[94m200#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
                #x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m #x1B[92mmake_conn#x1B[39;49;00m() -> aiohttp.TCPConnector:#x1B[90m#x1B[39;49;00m
                    #x1B[94mreturn#x1B[39;49;00m aiohttp.TCPConnector()#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
                connector = #x1B[96mself#x1B[39;49;00m.loop.run_until_complete(make_conn())#x1B[90m#x1B[39;49;00m
                r = {#x1B[90m#x1B[39;49;00m
                    #x1B[33m"#x1B[39;49;00m#x1B[33mhostname#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[33m"#x1B[39;49;00m#x1B[33mhostname#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                    #x1B[33m"#x1B[39;49;00m#x1B[33mhost#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[33m"#x1B[39;49;00m#x1B[33m127.0.0.1#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                    #x1B[33m"#x1B[39;49;00m#x1B[33mport#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[94m80#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                    #x1B[33m"#x1B[39;49;00m#x1B[33mfamily#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: socket.AF_INET,#x1B[90m#x1B[39;49;00m
                    #x1B[33m"#x1B[39;49;00m#x1B[33mproto#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[94m0#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                    #x1B[33m"#x1B[39;49;00m#x1B[33mflags#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[94m0#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                }#x1B[90m#x1B[39;49;00m
                #x1B[94mwith#x1B[39;49;00m mock.patch.object(#x1B[90m#x1B[39;49;00m
                    connector, #x1B[33m"#x1B[39;49;00m#x1B[33m_resolve_host#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, autospec=#x1B[94mTrue#x1B[39;49;00m, return_value=[r]#x1B[90m#x1B[39;49;00m
                ):#x1B[90m#x1B[39;49;00m
                    tr, proto = mock.Mock(), mock.Mock()#x1B[90m#x1B[39;49;00m
                    #x1B[94mwith#x1B[39;49;00m mock.patch.object(#x1B[90m#x1B[39;49;00m
                        #x1B[96mself#x1B[39;49;00m.loop,#x1B[90m#x1B[39;49;00m
                        #x1B[33m"#x1B[39;49;00m#x1B[33mcreate_connection#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                        autospec=#x1B[94mTrue#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                        return_value=(tr, proto),#x1B[90m#x1B[39;49;00m
                    ):#x1B[90m#x1B[39;49;00m
                        #x1B[94mwith#x1B[39;49;00m mock.patch.object(#x1B[90m#x1B[39;49;00m
                            #x1B[96mself#x1B[39;49;00m.loop,#x1B[90m#x1B[39;49;00m
                            #x1B[33m"#x1B[39;49;00m#x1B[33mstart_tls#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                            autospec=#x1B[94mTrue#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                            return_value=mock.Mock(),#x1B[90m#x1B[39;49;00m
                        ) #x1B[94mas#x1B[39;49;00m tls_m:#x1B[90m#x1B[39;49;00m
                            req = ClientRequest(#x1B[90m#x1B[39;49;00m
                                #x1B[33m"#x1B[39;49;00m#x1B[33mGET#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m
                                URL(#x1B[33m"#x1B[39;49;00m#x1B[33mhttps://www.python.org#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m
                                proxy=URL(#x1B[33m"#x1B[39;49;00m#x1B[33mhttp://proxy.example.com#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m
                                loop=#x1B[96mself#x1B[39;49;00m.loop,#x1B[90m#x1B[39;49;00m
                            )#x1B[90m#x1B[39;49;00m
>                           #x1B[96mself#x1B[39;49;00m.loop.run_until_complete(#x1B[90m#x1B[39;49;00m
                                connector._create_connection(#x1B[90m#x1B[39;49;00m
                                    req, [], aiohttp.ClientTimeout()#x1B[90m#x1B[39;49;00m
                                )#x1B[90m#x1B[39;49;00m
                            )#x1B[90m#x1B[39;49;00m

ClientRequestMock = <MagicMock name='ClientRequest' id='4671096080'>
connector  = <aiohttp.connector.TCPConnector object at 0x1166ba190>
m          = <function start at 0x11663a700>
make_conn  = <function TestProxy.test_proxy_server_hostname_default.<locals>.make_conn at 0x116639800>
proto      = <Mock id='4666546256'>
proxy_req  = <aiohttp.client_reqrep.ClientRequest object at 0x11606ce50>
proxy_resp = <ClientResponse(http://proxy.example.com) [None None]>
None

r          = {'family': <AddressFamily.AF_INET: 2>, 'flags': 0, 'host': '127.0.0.1', 'hostname': 'hostname', ...}
req        = <aiohttp.client_reqrep.ClientRequest object at 0x11625f8d0>
self       = <test_proxy.TestProxy testMethod=test_proxy_server_hostname_default>
start_connection = <function start_connection at 0x11669c220>
tls_m      = <function start_tls at 0x116638f40>
tr         = <Mock id='4671122512'>

#x1B[1m#x1B[31mtests/test_proxy.py#x1B[0m:319: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
#x1B[1m#x1B[.../Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py#x1B[0m:654: in run_until_complete
    #x1B[0m#x1B[94mreturn#x1B[39;49;00m future.result()#x1B[90m#x1B[39;49;00m
        future     = <Task finished name='Task-1045' coro=<TCPConnector._create_connection() done, defined at .../work/aiohttp/ai...nnector.py:1033> exception=AttributeError("'aiohttp._client_reqrep.ConnectionKey' object has no attribute '_replace'")>
        new_task   = True
        self       = <_UnixSelectorEventLoop running=False closed=True debug=False>
#x1B[1m#x1B[31maiohttp/connector.py#x1B[0m:1041: in _create_connection
    #x1B[0m_, proto = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._create_proxy_connection(req, traces, timeout)#x1B[90m#x1B[39;49;00m
        req        = <aiohttp.client_reqrep.ClientRequest object at 0x11625f8d0>
        self       = <aiohttp.connector.TCPConnector object at 0x1166ba190>
        timeout    = ClientTimeout(total=None, connect=None, sock_read=None, sock_connect=None, ceil_threshold=5)
        traces     = []
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <aiohttp.connector.TCPConnector object at 0x1166ba190>
req = <aiohttp.client_reqrep.ClientRequest object at 0x11625f8d0>, traces = []
timeout = ClientTimeout(total=None, connect=None, sock_read=None, sock_connect=None, ceil_threshold=5)

    #x1B[0m#x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m #x1B[92m_create_proxy_connection#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m
        #x1B[96mself#x1B[39;49;00m, req: ClientRequest, traces: List[#x1B[33m"#x1B[39;49;00m#x1B[33mTrace#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m], timeout: #x1B[33m"#x1B[39;49;00m#x1B[33mClientTimeout#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
    ) -> Tuple[asyncio.BaseTransport, ResponseHandler]:#x1B[90m#x1B[39;49;00m
        headers: Dict[#x1B[96mstr#x1B[39;49;00m, #x1B[96mstr#x1B[39;49;00m] = {}#x1B[90m#x1B[39;49;00m
        #x1B[94mif#x1B[39;49;00m req.proxy_headers #x1B[95mis#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
            headers = req.proxy_headers  #x1B[90m# type: ignore[assignment]#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        headers[hdrs.HOST] = req.headers[hdrs.HOST]#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        url = req.proxy#x1B[90m#x1B[39;49;00m
        #x1B[94massert#x1B[39;49;00m url #x1B[95mis#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        proxy_req = ClientRequest(#x1B[90m#x1B[39;49;00m
            hdrs.METH_GET,#x1B[90m#x1B[39;49;00m
            url,#x1B[90m#x1B[39;49;00m
            headers=headers,#x1B[90m#x1B[39;49;00m
            auth=req.proxy_auth,#x1B[90m#x1B[39;49;00m
            loop=#x1B[96mself#x1B[39;49;00m._loop,#x1B[90m#x1B[39;49;00m
            ssl=req.ssl,#x1B[90m#x1B[39;49;00m
        )#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[90m# create connection to proxy server#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
        transport, proto = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._create_direct_connection(#x1B[90m#x1B[39;49;00m
            proxy_req, [], timeout, client_error=ClientProxyConnectionError#x1B[90m#x1B[39;49;00m
        )#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        auth = proxy_req.headers.pop(hdrs.AUTHORIZATION, #x1B[94mNone#x1B[39;49;00m)#x1B[90m#x1B[39;49;00m
        #x1B[94mif#x1B[39;49;00m auth #x1B[95mis#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
            #x1B[94mif#x1B[39;49;00m #x1B[95mnot#x1B[39;49;00m req.is_ssl():#x1B[90m#x1B[39;49;00m
                req.headers[hdrs.PROXY_AUTHORIZATION] = auth#x1B[90m#x1B[39;49;00m
            #x1B[94melse#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m
                proxy_req.headers[hdrs.PROXY_AUTHORIZATION] = auth#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
        #x1B[94mif#x1B[39;49;00m req.is_ssl():#x1B[90m#x1B[39;49;00m
            #x1B[96mself#x1B[39;49;00m._warn_about_tls_in_tls(transport, req)#x1B[90m#x1B[39;49;00m
    #x1B[90m#x1B[39;49;00m
            #x1B[90m# For HTTPS requests over HTTP proxy#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# we must notify proxy to tunnel connection#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# so we send CONNECT command:#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m#   CONNECT www.python.org:443 HTTP/1.1#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m#   Host: www.python.org#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m##x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# next we must do TLS handshake and so on#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# to do this we must wrap raw socket into secure one#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            #x1B[90m# asyncio handles this perfectly#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            proxy_req.method = hdrs.METH_CONNECT#x1B[90m#x1B[39;49;00m
            proxy_req.url = req.url#x1B[90m#x1B[39;49;00m
>           key = req.connection_key._replace(#x1B[90m#x1B[39;49;00m
                proxy=#x1B[94mNone#x1B[39;49;00m, proxy_auth=#x1B[94mNone#x1B[39;49;00m, proxy_headers_hash=#x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m
            )#x1B[90m#x1B[39;49;00m
#x1B[1m#x1B[31mE           AttributeError: 'aiohttp._client_reqrep.ConnectionKey' object has no attribute '_replace'#x1B[0m

auth       = None
headers    = {'Host': 'www.python.org'}
proto      = <Mock id='4666546256'>
proxy_req  = <aiohttp.client_reqrep.ClientRequest object at 0x11606ce50>
req        = <aiohttp.client_reqrep.ClientRequest object at 0x11625f8d0>
self       = <aiohttp.connector.TCPConnector object at 0x1166ba190>
timeout    = ClientTimeout(total=None, connect=None, sock_read=None, sock_connect=None, ceil_threshold=5)
traces     = []
transport  = <Mock id='4671122512'>
url        = URL('http://proxy.example.com')

#x1B[1m#x1B[31maiohttp/connector.py#x1B[0m:1368: AttributeError

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

@bdraco
Copy link
Member Author

bdraco commented Nov 29, 2024

Doesn't make a measurable difference.

@bdraco bdraco closed this Nov 29, 2024
@bdraco bdraco deleted the cython_connection_key_implement branch November 29, 2024 21:08
@bdraco bdraco restored the cython_connection_key_implement branch November 29, 2024 21:09
@bdraco
Copy link
Member Author

bdraco commented Nov 29, 2024

o wait. I don't think its actually worked

@bdraco bdraco reopened this Nov 29, 2024
@bdraco
Copy link
Member Author

bdraco commented Nov 29, 2024

0.3% faster.... not worth it

@bdraco bdraco closed this Nov 29, 2024
@bdraco bdraco deleted the cython_connection_key_implement branch November 29, 2024 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@webknjaz webknjaz Awaiting requested review from webknjaz webknjaz will be requested when the pull request is marked ready for review webknjaz is a code owner

@asvetlov asvetlov Awaiting requested review from asvetlov asvetlov will be requested when the pull request is marked ready for review asvetlov is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@bdraco