nameConstraints DNS bypass via subject CommonName fallback in public_key hostname verification

Improper Certificate Validation vulnerability in Erlang OTP public_key (pubkey_cert and public_key modules) allows a DNS nameConstraints bypass via subject CommonName fallback in TLS hostname verification.

Two flaws combine to allow a subordinate CA whose DNS nameConstraints are restricted (e.g. permitted;DNS:allowed.example.com) to issue a leaf certificate that an OTP TLS client accepts as a valid identity for an out-of-scope hostname (e.g. victim.example.com):

First, pubkey_cert:validate_names/6 in lib/public_key/src/pubkey_cert.erl only checks SAN DNS entries against nameConstraints. Per RFC 5280, a permitted DNS subtree only restricts certificates that contain a DNS-typed name. A leaf with no subjectAltName therefore trivially satisfies any permitted;DNS:... constraint regardless of its subject commonName.

Second, public_key:pkix_verify_hostname/3 in lib/public_key/src/public_key.erl falls back to the subject commonName when no subjectAltName is present, extracting id-at-commonName attributes as presented IDs and matching them against the reference hostname. The strict pkix_verify_hostname_match_fun(https) matcher does not suppress this fallback.

The result is that path validation accepts a CN-only leaf under a DNS-constrained intermediate (no SAN means the nameConstraints are not triggered), and hostname verification then accepts it via the CN fallback. The bypass is reachable from stock ssl:connect with verify_peer, a trusted CA, SNI, and the canonical strict https hostname matcher.

This issue affects OTP from OTP 19.3 before OTP 29.0.1, OTP 28.5.0.1, OTP 27.3.4.12 and OTP 26.2.5.21, corresponding to public_key from 1.4 before 1.21.1, 1.20.3.1, 1.17.1.3 and 1.15.1.7.

The verify_fun option in the ssl application can be used to ensure that TLS connections fail if the end-entity certificate is missing the subjectAltName extension or has no domain name. Do not use a verify_fun that accepts the name_not_permitted error.