How to explain SSL cert validation discrepancy between Chrome and WebView on a modern Android?

(original title: Do some Android vendors ship their own SSL stack implementation?)

We've been having (in production) a peculiar HTTPS-related issue lately in our Android app.

All calls to our domain from within a WebView fail in our Android app, due to an SSL certificate validation issue. But this happens only in very specific combination of circumstances:

• only on Androids (desktop browsers, iOS are fine with our cert)


• only inside a WebView (works fine in Chrome, default Android browser, and when doing HttpsURLConnection from Java code)


• only on some devices (Samsung Galaxy Edge S6 with Android 6, HTC One M8 with Android 5); other devices (Sony/Android 4.4, Lenovo/Android 4.4, Motorola/Android 5, Huawei/Android 6, Nexus/Android 6) are not affected

Other important info:

• note that the issue happens on Android 5/6 which have an auto-updating WebView. For example, I reproduced issue in a fully up-to-date Samsung Galaxy Edge S6 (non-ISP-branded), Android 6.0.1 with security patches from 1 October 2016. The webview is very modern (Chrome 53)




• note that I checked the CA store and the root cert that signed our cert is in the store of the affected devices, with the same name and sha1 fingerprint.




• I used to have custom (user) CA certs (generated by Fiddler and installed manually by me to be able to decrypt HTTPS for debugging) on my devices, but for the sake of test, I removed them, and now the user CA store is empty on all devices, but issue persists




• I tested some 60 high profile websites, and the issue is reproducible for several of them too (including https://www.amazon.com) on the two affected devices (the set of websites rejected by Samsung and HTC is overlapping, but not exactly the same). Some of those affected websites have the same VeriSign cert as the root cert of the chain as we do, but this root cert is present in the CA store of the device as mentioned previously.




• Our SSL cert is quite good according to SSLLabs, and the issue happens on amazon.com which apparently has very good server/cert config (click)




• While we reproduced the issue in our device lab, we also got quite a few negative Play Store reviews that look like the same issue (all from Samsungs, but this correlates with our user base). However the app works fine for most of people, as we get mostly 5 star reviews every day.

Next steps that we're taking:

• We're trying to embed the cert inside the Android app and make Android explicitly trust it, but since WebView and HTTPSUrlConnection behave differently, we're not sure if this will work




• If this doesn't work, as a last resort we'll also probably try to override WebViewClient.onReceivedSslError with a custom implementation which will inspect the cert and trust it if it has the same fingerprint as our cert




• any better options?

Theories:

• some Android shipping customized SSL stack in their versions of Android? (but why Samsung and HTC would exhibit a very similar behavior)


• does Chrome have separate CA store than the Android device?


• some other apps installed on the affected Android devices that mess up with the WebView in my app?


• anything else?

Edit:

I reproduced the issue on the following websites:

• www.microsoft.com (Symantec cert)


• www.amazon.com (Symantec cert)


• mail.ru (GeoTrust cert - Symantec subsidiary)


• two others with Symantec or GeoTrust certs


• www.theguardian.com (cert doesn't seem to be Symantec-related)

So most probably the issue comes from webview bug that EricLaw pointed. Guardian case seems to be some another issue however.

Edit 2:

I confirmed the issue was due to the bug mentioned by Eric. We changed our SSL cert provider and it solved the issue.