Lots of ESTABLISHED proxy connection leave in system

[xpader]

Lots of ESTABLISHED proxy connection leave in system, ss command return no keepalive timer.

root@ecs-web-2:~# lsof -p 45576 |grep IPv
php     45576  web    5u  IPv4             965777      0t0      TCP *:2345 (LISTEN)
php     45576  web   15u  IPv4             979696      0t0      TCP ecs-web-2:35526->xx.xx.79.224:62008 (ESTABLISHED)
php     45576  web   16u  IPv4             964904      0t0      TCP ecs-web-2:21576->xx.xx.79.224:61031 (ESTABLISHED)
php     45576  web   17u  IPv4             964063      0t0      TCP ecs-web-2:24330->xx.xx.79.225:56114 (ESTABLISHED)
php     45576  web   18u  IPv4             964059      0t0      TCP ecs-web-2:24302->xx.xx.79.225:56114 (ESTABLISHED)
php     45576  web   20u  IPv4             979714      0t0      TCP ecs-web-2:20790->xx.xx.79.224:60034 (ESTABLISHED)
php     45576  web   21u  IPv4             964949      0t0      TCP ecs-web-2:21594->xx.xx.79.224:61031 (ESTABLISHED)
php     45576  web   22u  IPv4             964939      0t0      TCP ecs-web-2:24344->xx.xx.79.225:56114 (ESTABLISHED)
php     45576  web   25u  IPv4             967879      0t0      TCP ecs-web-2:13356->xx.xx.99.56:52031 (ESTABLISHED)
php     45576  web   26u  IPv4             963314      0t0      TCP ecs-web-2:27192->xx.xx.99.57:52035 (ESTABLISHED)
php     45576  web   27u  IPv4             965036      0t0      TCP ecs-web-2:13034->xx.xx.79.225:62016 (ESTABLISHED)
php     45576  web   33u  IPv4             963176      0t0      TCP ecs-web-2:31100->xx.xx.79.225:59011 (ESTABLISHED)

It's 15:44 now, but the connection created at 13:01, client is already destruct in php:

root@ecs-web-2:~# ll /proc/45576/fd/15
lrwx------ 1 web www-data 64 Jun 21 13:01 /proc/45576/fd/15 -> 'socket:[979696]'

ss info has no keepalive timer:

root@ecs-web-2:~# ss -aeon | grep 'xx.xx.79.224:62008'
tcp        ESTAB       0         0        192.168.1.60:35526         xx.xx.79.224:62008     uid:1000 ino:979696 sk:168b <->                                                

ss normal connection:

tcp        ESTAB       0       0         192.168.1.60:2345           47.93.20.248:48210     timer:(keepalive,1min9sec,0) uid:1000 ino:1055699 sk:148a <->                  

I have use php-meminfo (https://github.com/BitOne/php-meminfo) extension to detect the Amp\Http\Client\HttpClient and Amp\Http\Tunnel\Http1TunnelConnector instance count is normal, but the connection leave too much.

Where the problem it is?

[xpader]

I think may need use SO_KEEPALIVE for socket options to active tcp keepalive check timer.

[xpader]

It's little bit weired, when we change to another proxy provider, it's all normal now.
In our business, we create a proxy http client for every user job, the proxy ip is different, and delete client in cup of minutes or job finished.
The only different is the proxy provider before have a proxy transit tunnel server, we connection through this tunnel server, ip is same, sometimes port is same too, but proxy provider now is directly connect to proxy ip.
We have also used proxy transit tunnels before, and it's normal.
What's different between this?
It's server's tcp stack reason? or Linux kernel tcp config reason? or amphp http-client/http-tunnel did not close the connection properly for this scene?

[xpader]

After lots of debug, I found that handleUpgradeResponse() and the target site is h2 make socket will not be closed.

https://github.com/amphp/http-client/blob/bb4d51bc746d70da12835708f69a9d7426eab94e/src/Connection/Http1Connection.php#L501

This makes socket will not close when __destruct, will not close in idle watcher.

[xpader]

When proxy connection from ResourceSocket upgrade to UpgradedSocket, and visit website via HTTP 2 protocol by a proxy chain, the connection wil be hang on UpgradedSocket->read() in Http2ConnectionProcessor.

Hang place: https://github.com/amphp/http-client/blob/1aae8bdfd25487f86fb59ab3ea0a4794bc1e8b6c/src/Connection/Internal/Http2ConnectionProcessor.php#L1271C15-L1271C15

Socket->read() will never return.

[xpader]

Too much logic, can't dig.

[xpader]

I'm now added a check timer, to get all stream resources by in php process, and filter all outgoing by proxy connections, then check connection survival time, and force close it.

The issue fixed by this way, not a good way.

[kelunik]

Could you please check whether this issue persists with the Amp v3 based versions?

[xpader]

We're not yet upgrade this project into Amp v3 based version, the whole framework base on Amp v2, this project has withstood many concurrency and stability tests.

We're using Amp v3 based framework in new project, but still developing.

[xpader]

This issue is less important at the moment because resource's scanning and cleaning mechanisms remedy it.

The more troublesome thing is that it is difficult to find a way to reproduce this.

I suspect that the frequent read timeouts are related to this.