Random WASM crashes in dlfree

[MSFS 2024] Bug Reports Aircraft
1

Version: 1.7.27.0

Frequency: Rarely

Severity: Blocker

Marketplace package name: Blackbird C-130J

Context: Editing project or in Community, WASM crash during flight

Similar MSFS 2020 issue: insert url here if applicable

Bug description:

WASM randomly crashes during flight. Sometimes 5 minutes in, sometimes 10 minutes in, sometimes after an hour+ or more. And sometimes not at all. Some people, myself included, never see a WASM crash, or see one once every 50-60 flight-hours. Others report four out of five flights end in a WASM crash.

The few times when we have caught a random crash in the debugger, it is an access violation 0x0000005 in the dllfree function, when destructing some object allocated on the heap.

From a previous report (11979), this could be caused by failing to free memory allocated by fsNetworkHttpRequestGetData() calls. However, in our case, disabling our entire network interface failed to affect the frequency of the crashes. Over the past couple of months we’ve been over all of our codebase with a very fine-toothed comb, run every sort of tool and AI over it to try to find any sort of pathway for heap or stack corruption, and then instrumented our code extensively to try to find any evidence of heap or stack corruption, and there just… isn’t any.

We have a group of customers and testers who all are affected, and were unable to find any common thread linking them in terms of hardware, other addons, or flight habits.

We have seen reports of, and have observed directly, WASM crashes in the default A400M and A320 in the recent weeks (both before and after SU5).

There has been a fairly large uptick in reports of WASM crashes that we are getting since SU5 was released.

All of this, coupled with the fact that we are just completely out of any sort of bugs, unsafe code, and unhandled edge cases in our own code, has us thinking that the bug may be on the sim side, rather than in the WASM itself.

Repro steps:

Attachments:

Private attachments: Send a PM to @PrivateContent with the link to this topic and the link to download your content

2

I was starting at Kiel on the rollfield and been told I was crashing down. Randomly.

I’m German so my english sucks.

Can anyone fix this?

3

Won’t play anymore until someone fix this unfaulted crash. Have played for a lot of time to gain those credits and Reputation .

4

@FlyingRaccoon @EPellissier - Sorry for the bump(s), but we at Blackbird Simulations are really at our wit’s end about this issue.

It’s causing some of our customers a lot of grief. We can detail all the steps that we’ve taken to try and get to the bottom of this, but collectively, with a group of testers that have been trying to help us narrow this down, we’ve spent hundreds of hours with no success.

We’re wanting to get this product onto the Marketplace, but we don’t want to allow it to be sold there until these issues get sorted out.

Anything we can provide to you all to help us get to the bottom of this, please just let us know.

5

Hi Jim,

We’re rather busy with SU 5.1 right now but I can try to run your aircraft and see if it triggers WASM crashes on our development builds - I suppose you haven’t managed to identify clear repro steps to trigger one?

Best regards,

Eric / Asobo

6

Hi Eric, that’s correct. I’ll ask @Farley to chime in with the steps he’s taken to try and islotate and repro.

1 Like
7

Am I right to think you never sent this package to your draft repo? Or maybe I didn’t look in the right place? I’ll need a way to access the package to run tests on our end. :wink:

Best regards,

Eric / Asobo

8

We have - the package name is milviz-aircraft-d30. It’s currently in draft status. We’re going to upload the latest build for you to look at, the version in the Marketplace portal is a couple of weeks old.

We’ll advise as soon as it’s been uploaded through the portal.

1 Like
9

Thanks a lot!

I’ll have a look asap - it might be next week though since there’s an important fix I’d like to complete tomorrow for SU5.X.

Best regards,

Eric / Asobo

1 Like
10

Hi Eric, thanks a lot for your attention on this. Unfortunately there doesn’t seem to be a clear way to reproduce the crash. It seems to be almost completely random, but it does vary by user. Some people rarely if ever get a crash, other people report a crash almost every flight. Several people reported it never or rarely crashed before SU5, and now it crashes more often. I’ve tried to triangulate hardware, other addons, lat/lon, length of flight, systems (e.g. MFD pages, navigation systems, etc) in use, without finding any common thread. It does not seem to be directly related to the amount of memory used, at least as measured with the dev mode tools.

11

Is the WASM module doing a lot of file accesses (fopen/fread/fclose or similar) to files contained in the package itself or in its work folder? The WASM Debug Window shows the linear memory occupied by the module but it doesn’t account for memory that could be reserved by the calls to the various APIs. I’ll run the aircraft through autotests which may help reproducing the crash.

Oh one last thing: could you list the WASM APIs that the module is using? :slight_smile:

Best regards,

Eric / Asobo

12

“Is the WASM module doing a lot of file accesses (fopen/fread/fclose or similar) to files contained in the package itself or in its work folder” yes, from both the package (data folder) and the work folder. Thanks for the help, it’s much appreciated. In terms of APIs, we are using: Communication, MapView, NanoVG, Network, Planned Route, Vars (L:1: get/set), VFX (only when launching chaff/flares, so presumably unrelated), Legacy Gauge (aircraft_varget, L:var get/set, and execute_calculator_code), and SimConnect (Avar get/set, event masking, event triggering, 6hz and 1hz updates)

1 Like
13

@EPellissier We’ve got our latest version uploaded if you’d like to use that for your checks:

image

Many thanks!

1 Like
14

Would you be able to confirm the version of the SDK you used to compile the WASM module?

Best regards,

Eric / Asobo

15

Latest uploaded package was compiled on SDK version 1.6.9

16

Hi, any chance there have been any findings so far?

17

Unfortunately not yet, but we are still trying to repro the issue.

Best regards,

Eric / Asobo

18

@EPellissier - Any luck on reproducing these issues?

It’s worth noting that one of the things that has made this so frustrating is that some of our users have this happen regularly, some never have it happen at all, and there seems to be no common thread among those that have the crashes.

19

I am afraid we are still out of luck with getting a single repro in the studio. :frowning:

At this stage I am wondering if we should have a debug option in the DevMode that lets the whole sim crash when the module crashes - it would at least allow us to get crash dumps which may help understanding the issue. Another option would be to allow modules to be compiled in “Retail” configuration with debug symbols that would be separated upon ingestion and made available to the developer for debugging, but this is a rather heavy feature that will take time to design & implement correctly.

Apologies for the time this is taking, I’ll let you know as soon as we manage to make some progress on this - in the meantime, please feel free to add any relevant information here.

Best regards,

Eric / Asobo

Note: I’ll edit the title of the topic since I suspect the crash is in “dlfree” instead of “dllfree” :wink:

20

Thanks for the response, Eric. It’s appreciated.

The crash dump might help - I believe we may have a couple of testers that are encountering the problem with enough frequency to perhaps gather some useful info.

We struggle from the same issue as what you’re finding out - As Farley has mentioned, he only witnesses a crash on rare occasions, a low enough frequency to assume that there’s no problem. However, like he’s said, we have users on our forums that are encountering it with worrisome regularity, so it can’t be ignored.