Bug description: Users are experiencing intermittent c0000005 exceptions in various functions that work when tested directly. The call stack/location of the crashes seem arbitrary and move around throughout the project, indicating memory corruption/exhaustion. This only occurs with newer versions of the SDK after the last related issue was “fixed”. Additionally, we only seem to experience this crash in Release mode (naturally), making it nearly impossible to determine the true cause of the issue.
As a result of that post, I used the /clang:-fstack-usage command to analyze the functions for large static stack allocation. I wrote a tool that reads every single .su file and flags any that use significant amounts of memory - as of the time of writing, no single function uses more than ~18k (well below the already unreasonably low 65k limit). I have also modified our code to avoid excessively long functions and modified our use of vectors to use emplace_back with only the arguments where possible to allocate directly on the heap.
I also attempted to raise the stack size limit per the suggestions in that post, but the MSFS compiler/runtime seem to ignore that option. At this point, we’re now forced to continue using a Debug build with an older SDK as it seems to be the only one that actually works. We are unsure as of yet if this applies to the 2024 SDK as well.
I am not convinced that this is stack exhaustion and not a bug deeper in the memory management portions of the SDK. If it’s the former, we need better guidance on where/why it’s crashing or, preferrably, a limit that is higher than 5% of what normal programs are given. If it’s the latter, we hope us sending the complete source helps to identify and resolve it.
Repro steps: Load the aircraft and attempt to use it - the crash is sometimes nearly instant or it can take hours.
Private attachments: I have sent a DM to PrivateContent with the full source of the aircraft, a serial key, and an installer. As well, it includes the exact SDK version we used that exhibits this issue and a file called SDK crash.zip that includes a callstack and console log from a user that experienced the crash directly.
It’s also worth mentioning that in Debug mode, the total execution time of the WASM is ~2ms. In Release mode, the execution time goes up to ~6ms. Both scenarios were tested in cold and dark conditions (eliminating flight plan, etc. as variables).
I would ask your testers to enable full crash dumps (~30-40gb each), have them zip them up, upload to a dropbox / googledrive or some such and send them to you to forward to Asobo for debugging. That would be the easiest way…
I haven’t managed to repro the crash for the moment.
But I’ve noticed that, in the SDK you’ve included, the version is 0.24.3.0. There was a fix in the 0.24.4.0 that fix the kind of issue you describe (Fixed a bug in Wasm allocator which could cause the simulation to crash). Can you please recompile your project with the latest SDK and check if this still occurs? I think I may solve your issue.
With more test I’ve found a repro!
I start cold & dark somewhere (LFPG for my test). When the flight is loaded and started, I press the “BAT” button and then the “EXT | PMR” button on the electrical panel
After investigation, this crash is the result of a stack overflow (hitting the guard page between the stack and the data part of the module). The stack overflow is caused by a lot of copy in some functions. I can point to you in DM some of them if you want (and especially the one causing the crash I repro)
Edalyn is out of the office right now but from what I remember the issue was caused by the comparison functions you used in some calls to std::sort - you didn’t pass the objects by reference, hence triggering copies of the compared objects that are allocated on the stack. Since some classes are rather heavy and std::sort works recursively this leads to the stack overflow she mentioned.
The best way to resolve the issue would be to use the suggested prototype for your comparison functions: bool cmp(const Type1& a, const Type2& b);.
Just to follow up - the -zstack-size=n change did not make any difference. We still saw crashes/instability - furthermore, despite no function consuming anywhere near the limit of stack memory when checked via static analysis, this crash still happens randomly throughout the code. We have reviewed the code and ensured no obvious misuse/overallocation of memory exists. Given that it only happens in Release, debugging is quite hard.
Again, I’d say that limiting modules to 65k of stack memory is needless and the savings of a few megabytes, at most, of memory is not worth the hassle given modern hardware (even on 10GB Series S Xbox platforms).
Did you make the change we suggested previously in the comparison functions used for std::sort?
Would you mind sharing your current code for us to look into it again?
This could be a valid comment if we knew what stack size would enable your add-on to work without any issue, but I don’t think it is the case yet - and as explained above, this limit can already be lifted by using a simple linker option so it shouldn’t really be a blocker.
Today, I’ve gone through and ensured that most (if not all) functions are using const type& and that sorting/comparison algorithms are using references as well. I’ve just sent out a QA build now to see if it worked - if not, we’ll definitely share the code with you again and see if we can narrow it down.
Unfortunately, while the stability was better than previous attempts, we still noted a marked increase in intermittent WASM crashes throughout all phases of flight/operations. I have reviewed the code myself thoroughly more than once, had AI tools assist in follow-up reviews to catch what I may have missed, and reviewed the .su outputs and nothing comes up as note-worthy. I do not believe we are even approaching the 65k limit.
Given that many products experience intermittent WASM crashing and instability and we essentially have an A/B test where the only change is the SDK version/compilation mode, I maintain the belief that there may be an issue in the runtime/WASM->executable compilation or that this 65k limit is needlessly introducing instability with no tangible benefit to the user.
I have sent the current source code to PrivateContent for your review and use in debugging.
Seconding this. I’m on a completely different project at a different company, also with relatively heavy WASM systems. Over the past few weeks I’ve been tearing out hair going over every line of our WASM code, run every sort of analysis tool and AI over the code. It is squeaky squeaky clean, yet still we have reports of random WASM crashes (OSDs).
From our investigation, when the module starts, it looks for and processes the proddefs files. It uses the MSFS asynchronous file API because the files are in the package folder, and not in the wasm folder. That sometimes opens the same file twice, which causes the error. We are now testing a potential solution to prevent this from occurring (On startup, all the required files are copied from the package to the work folder using the async API), however, we are not sure if its a 100% reliable solution.
