Am I affected?
Description
Missing Release of Resource after Effective Lifetime vulnerability in elixir-grpc grpc_server allows a client to stall all later calls on a shared HTTP/2 connection via calls that the server refuses while their DATA frames are in flight.
The stream handler grpc_stream_h (grpc_server/src/grpc_stream_h.erl) requests flow from Cowboy only for request data that has arrived. When the server refuses a call early, for example because the body exceeds max_body_size or the method is not implemented, Cowboy drops the DATA frames still in flight for that stream. Their bytes are not returned to the connection flow-control window. After enough refused calls the window is empty. No call on that connection can then send a request body, and each call waits until its deadline. Other connections are not affected.
This issue affects grpc_server: from 1.0.0 before 1.1.0.
Technical analysis
1. Flow is requested only for data that arrived
Cowboy increases the HTTP/2 connection window only when a stream handler returns a {flow, N} command. grpc_stream_h returns that command only for request data it has received.
2. A refused call leaves data uncredited
When the server refuses a call, the stream stops. The client can still have DATA frames in flight for it. Cowboy drops those frames without running the handler, so no flow is requested for their bytes and no connection-level WINDOW_UPDATE is sent.
3. The window empties
Each refused call with DATA in flight reduces the connection window for good. With the default 65,535-byte window, the window was empty after two refused calls in a test against v1.0.6. An empty window stops all request data, so no handler requests flow again and the window stays empty.
4. Result
Every later call on the connection that has a request body waits until its deadline. A new connection works normally.
Proof of concept
- Start a gRPC server with a small
max_body_size, for example 1000 bytes. - Open one HTTP/2 connection with a client that tracks the connection window.
- Send a unary call with a body above the limit and wait for the
RESOURCE_EXHAUSTEDreply. - Send more DATA frames on the refused stream, up to what the stream and connection windows allow.
- Repeat steps 3 and 4 on new streams until the connection window is 0.
- Send a small valid unary call on the same connection. Observe that its body cannot be sent and that the call ends at its deadline.
Weaknesses & attack patterns
Weakness
CWE-772
·
Missing Release of Resource after Effective Lifetime
in catalog →
MITRE ↗
Attack patterns
CAPEC-131
·
Resource Leak Exposure
MITRE ↗
A client whose calls share an HTTP/2 connection with other callers can make all later calls on that connection wait until their deadlines. Calls on other connections are served normally.
Affected — Hex / grpc_server Hex.pm ↗ Repository ↗
modules · source files · routines
Affected — GitHub / elixir-grpc/grpc Repository ↗
modules · source files · routines
Workarounds
Put a Cowboy stream handler in front of grpc_stream_h with the adapter option stream_handlers: [MyHandler, :grpc_stream_h]. In init/3, the handler calls :cowboy_stream.init/3 and adds a {:flow, size} command when the request has a body, where size is Cowboy's max_frame_size_received (16384 by default). It passes all other callbacks to :cowboy_stream.
Configurations
Calls from more than one caller must share one HTTP/2 connection to the server, for example through a proxy or a shared client channel.
References
Credits
CVSS breakdown
CVSS:4.0/AV:N/AC:L/AT:P/PR:N/UI:N/VC:N/VI:N/VA:L/SC:N/SI:N/SA:N