gRPC is off unless you configure it. An upgrade to 0.15.9 changes nothing a customer can see:
without a gRPC port the gateway answers every gRPC call
UNAVAILABLE (GRPC_DISABLED), and the REST
API behaves exactly as before.Turning it on
Set the port in the environment file and restart:update writes the port only when you pass --grpc or --grpc-port; without them an existing value
is shown and left alone, and none is ever added.
With
--firewall, the installer opens the gRPC port only when it is bound to something other than
loopback.
Checking it
fireflo.v1.AuthService, fireflo.v1.SendService and fireflo.v1.ReceiptService (with
reflection on). /ops/health carries a grpc block — enabled, host, port as bound, tls, the
live sessions on this node, open streams, and a receipts block for the feed.
Behind a proxy
gRPC needs HTTP/2 end to end. An HTTP/1.1proxy_pass does not carry it. With nginx:
Subscribe stream after 60 seconds.
Tell the gateway the proxy is there
Both are platform settings and apply to the next call. The gateway logs a warning once when it sees
X-Forwarded-For from a private address it does not trust — the sign that this is missing.
TLS on the port itself
The listener speaks plain HTTP/2 (h2c) by default, for a proxy that terminates TLS. To have the gateway speak TLS itself, add to the environment file:grpc.tls: true, and clients connect with TLS instead of -plaintext.
Refused logins and fail2ban
gRPC needs no jail of its own. A refused gRPC login is written to the sameauth-failed.log as SMPP
and REST, naming the listener and the port:
reason=sessionExpired) before it logs
in again. The shipped filter ignores that reason; a call with no token at all is
missingCredential, and is counted. fireflo-install update refreshes an installed gateway filter
when the release’s differs, then reloads fail2ban. See
Blocking abusive clients.
Limits
There is no separate gRPC rate ceiling, deliberately: one would double what an account is allowed.
The SMPP listeners keep their own (
conf.maxRate.*), as before.
The receipt feed
A message sent by an HTTP login — over gRPC or REST — that asks for receipts without adlr_url
has its receipts queued for that login, until the customer acknowledges them. A dlr_url always wins.
With the feed on, REST also accepts dlr=yes with no dlr-url; with it off, that is still a 400.
All are platform settings, read live.
Where the queue lives follows the receipt store. On a single node with the default store it is
kept on that node’s disk, beside the receipt store. Where receipts are
kept in PostgreSQL, the feed is in the database too, shared by every node, so a customer may read on
any node. A node that stops cleanly gives its leases back as it stops; one that dies has them
released by the surviving nodes, or they expire after the lease.
Platforms
From gateway 0.15.10 a platform that provisions accounts with the gateway’s management token (OMNI, for one) can use the gRPC port without any account’s password:PlatformLogin is signed with the
management token exactly as the management API’s requests are, and the session it returns sends as —
and reads the delivery receipts of — the accounts whose ref starts with the session’s ref — a prefix ending in /, such as
omni:<install>/<tenant>/ — and no others.
Nothing needs turning on beyond the gRPC port and the management token you already set; without the
token, PlatformLogin answers UNAVAILABLE (MANAGEMENT_DISABLED). The management token’s rules
apply as on the management API: FIREFLO_MANAGEMENT_ALLOWED_IPS if you set it, and TLS — on the port
or at a trusted proxy — unless the platform connects over loopback. Sends count against each account’s
own rate ceiling; there is no platform-wide limit. A bad, stale or replayed signature is a bannable
surface=grpc refusal naming platform:<ref>, and health’s grpc.sessions_platform shows how many
platform sessions a node holds. The capability is grpc_platform.
Metrics
The gauges are refreshed every 10 seconds. Open
SendStreams and the requests held across them are in
health’s grpc block.
When something is wrong
Every refused login, and every session, shows the proxy's address
Every refused login, and every session, shows the proxy's address
smsg.grpc.proxy.forwarding is off, or hops does not match the number of proxies. Fix it before
fail2ban counts ten refusals from the proxy and bans it.Calls fail at the transport before the gateway answers
Calls fail at the transport before the gateway answers
Something in the path speaks HTTP/1.1 — a
proxy_pass instead of grpc_pass, or a load balancer
without HTTP/2 to the back end. Or the client uses TLS against the plain listener, or the reverse.A customer's feed keeps growing
A customer's feed keeps growing
They ask for receipts without a
dlr_url and do not read them, or read without acknowledging. Health’s
grpc.receipts lists the deepest logins; redelivered_total climbing with acked_total flat is the
second case.Related
gRPC API
What your customers see.
Blocking abusive clients
The refused-login record, and the reasons that never ban.