Commit graph

12 commits

Author SHA1 Message Date
Alexander Strizhakov c17d83cd73
improvements and fixes for http requests
- fix for gun worker termination in some circumstances
- pool for http clients (ex_aws, tzdata)
- default pool timeouts for gun
- gun retries on gun_down messages
- s3 upload timeout if streaming enabled
2020-09-02 09:04:23 +03:00
rinpatch dab1d8c98e gun ConnectionPool: Re-add a missing cast for remove_client 2020-07-28 23:48:41 +03:00
rinpatch 4ce6179dc7 gun ConnectionPool: replace casts with calls
The slowdown from this is most likely immesurable, however it eliminates
possible false positives when tracking dead clients.
2020-07-28 20:49:48 +03:00
rinpatch 7115c5f82e ConnectionPool.Worker: do not stop with an error when there is a timeout
This produced error log messages about GenServer termination
every time the connection was not open due to a timeout.

Instead we stop with `{:shutdown, <gun_error>}` since shutting down
when the connection can't be established is normal behavior.
2020-07-15 15:58:08 +03:00
href 6a0f2bdf8c Ensure connections error get known by the caller 2020-07-15 15:26:35 +03:00
href 46dd276d68 ConnectionPool.Worker: Open gun conn in continue instead of init 2020-07-15 15:26:35 +03:00
rinpatch a705637dcf Connection Pool: fix LRFU implementation to not actually be LRU
The numbers of the native time unit were so small the CRF was always 1,
making it an LRU. This commit switches the time to miliseconds and changes
the time delta multiplier to the one yielding mostly highest hit rates according
to the paper
2020-07-15 15:26:35 +03:00
rinpatch 7882f28569 Use erlang monotonic time for CRF calculation 2020-07-15 15:26:35 +03:00
rinpatch 1b15cb066c Connection pool: Add client death tracking
While running this in production I noticed a number of ghost
processes with all their clients dead before they released the connection,
so let's track them to log it and remove them from clients
2020-07-15 15:26:35 +03:00
rinpatch 0ffde499b8 Connection Pool: register workers using :via 2020-07-15 15:24:47 +03:00
rinpatch ec9d0d146b Connection pool: Fix race conditions in limit enforcement
Fixes race conditions in limit enforcement by putting worker processes
in a DynamicSupervisor
2020-07-15 15:17:27 +03:00
rinpatch 58a4f350a8 Refactor gun pooling and simplify adapter option insertion
This patch refactors gun pooling to use Elixir process registry and
simplifies adapter option insertion.

Having the pool use process registry instead of a GenServer has a number of advantages:
- Simpler code: the initial implementation adds about half the lines of code it deletes
- Concurrency: unlike a GenServer, ETS-based registry can handle multiple checkout/checkin
requests at the same time
- Precise and easy idle connection clousure: current proposal for closing idle connections in
the GenServer-based pool needs to filter through all connections once a minute and compare their
last active time with closing time. With Elixir process registry this can be done
by just using `Process.send_after`/`Process.cancel_timer` in the worker process.
- Lower memory footprint: In my tests `gun-memory-leak` branch uses about 290mb on peak load (250 connections)
and 235mb on idle (5-10 connections). Registry-based pool uses 210mb on idle and 240mb on peak load
2020-07-15 15:17:27 +03:00