WASM network requests instantly failing

[jack]

Version: 1.38.2.0

Frequency: Consistently

Severity: High

Context: Aircraft utilizing our endpoints via WASM

Bug description:
Using the WASM network API, simple GET requests to our CDN fail for certain users. The exact same request works fine in Coherent using fetch. Our best working theory is that it is related to the user’s certificate trust store on their computer.

For a bit of context - this fall, AWS stopped supporting the Starfield C2 CA for any new certificates. When we renewed our certificates, requests started failing in MSFS 2020. We tracked down this change and decided to attempt to use a different certificate - and we decided to go with Let’s Encrypt. As soon as we switched the certificate out, things started working again.

Now, the issue has returned (it’s also possible the issue was present back then, but we just didn’t get any bug reports for it). Since we don’t have enough insight to tell what is causing the network requests, we’d like some help here if possible. All we can see at the moment is that the request returns a negative response code, indicating a network error.

For debug purposes, here is a sample URL.

And here is our certificate chain

1. End-entity certificate (Level 0):
   - Subject: CN=packages.fmsdata.api.navigraph.com
   - Issuer: C=US, O=Let's Encrypt, CN=E6
   - Key: ECDSA, 256-bit
   - Valid until: Apr 9 13:40:13 2025 GMT

2. Intermediate CA (Level 1):
   - Subject: C=US, O=Let's Encrypt, CN=E6
   - Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
   - Key: ECDSA, 384-bit
   - Valid until: Mar 12 23:59:59 2027 GMT

3. Root CA (Level 2):
   - ISRG Root X1
   - C=US, O=Internet Security Research Group, CN=ISRG Root X1

It is interesting to note that the issue is not present at all on 2024 - likely due to changes in the underlying implementation of the network API.

We understand an update to 2020 is very unlikely anytime soon, so any insight into what could be going wrong and how we can fix it on our end would be greatly appreciated.

Repro steps:
Reproducing is very inconsistent; as stated above, this only affects certain users. For those users who are affected, reproducing is as simple as spawning onto the runway with the aircraft attached. The leftmost gauge is WASM, and the one to the right is a JS gauge.

Attachments:
Minimal reproduction aircraft with source: NetworkAircraft.zip - Google Drive
Screenshot from a user without the issue:

Screenshot from a user with the issue:

[Skysail]

We extended this test by trying the service used in the network sample - httpbin.org.

Using that domain instead, the WASM side did not error! We decided to check the certificate chain and it turns out that this service indeed has the now-deprecated Starfield C2 root CA included still - up until September this year. We can also see that they use an AWS Certificate, which means that the service renewed their certs just before AWS removed Starfield C2 (see ACM will no longer cross sign certificates with Starfield Class 2 starting August 2024 | AWS Security Blog)

Here’s the comparison of the output from openssl s_client -showcerts:

So far, this Starfield C2 certificate is our only hint about why our requests are failing. Finding a service that is currently providing certificates with this root CA is proving challenging, so we can’t prove that this is the culprit yet, but that is as close to a working theory as we have without being able to look deeper into the reason behind the network error in the simulator.

Seeing as results vary from user to user, with only a subset being affected, this likely comes down to the OS level in the end. We did investigate whether this could be an issue of an out-of-date “Trusted Root Certification Authorities” list in Windows, but so far, that has led nowhere.

[Skysail]

We have now updated our certificate chain to include the Starfield Class 2 CA. Unfortunately, users still have issues reaching our servers with the same cryptic network error! Here is the new chain:

1. End-entity certificate (Level 0):
   - Subject: CN=packages.fmsdata.api.navigraph.com
   - Issuer: C=US, O=Amazon, CN=Amazon RSA 2048 M02
   - Key: RSA, 2048-bit
   - Valid until: Apr 30 23:59:59 2026 GMT

2. Intermediate CA (Level 1):
   - Subject: C=US, O=Amazon, CN=Amazon RSA 2048 M02
   - Issuer: C=US, O=Amazon, CN=Amazon Root CA 1
   - Key: RSA, 2048-bit
   - Valid until: Aug 23 22:25:30 2030 GMT

3. Root CA (Level 2):
   - Subject: C=US, O=Amazon, CN=Amazon Root CA 1
   - Issuer: C=US, ST=Arizona, L=Scottsdale, O="Starfield Technologies, Inc.", CN=Starfield Services Root Certificate Authority - G2
   - Key: RSA, 2048-bit
   - Valid until: Dec 31 01:00:00 2037 GMT

4. Root CA (Level 3):
   - Subject: C=US, ST=Arizona, L=Scottsdale, O="Starfield Technologies, Inc.", CN=Starfield Services Root Certificate Authority - G2
   - Issuer: C=US, O="Starfield Technologies, Inc.", OU=Starfield Class 2 Certification Authority
   - Key: RSA, 2048-bit
   - Valid until: Jun 28 17:39:16 2034 GMT

We are currently at a loss as to what may be causing this. Is there any additional logging that we can tap into in order to see what is causing the network requests to fail?

Any help would be much appreciated!

[jack]

Providing an update here, we have finally found the issue. The file we are attempting to download is 50MB, and it turns out the actual request is timing out due to the user’s internet speed. Whatever the default timeout is for the curl usage is not high enough.

We had to do some heavy reverse engineering here (injecting a DLL to re-route all the curl functions through our own logger) in order to actually get a workable error log, which is

Transfer DONE detected!
  URL: https://packages.fmsdata.api.navigraph.com/[...]
  HTTP response code: 200
  CURLcode result: 28

We will take accountability here on the initial confusion with the error report - being “requests instantly failing” which may have thrown off debugging. Users had reported to us that it seemed that way, and since none of us were able to reproduce, we went with that.

As mentioned (and shown) in the initial post, the only thing that was being returned from the WASM network API was an undocumented -1, which is a completely useless starting point for a third-party developer to debug.

@FlyingRaccoon @Boris Could you please update the network API documentation (specifically the fsNetworkHttpRequestGetErrorCode function) to include the list of all possible non-standard HTTP codes and their meanings? This would greatly aide us, and any other developer in the future to avoid having to go through the rabbit holes we needed to.

Best,
Jack