WASM: execute_calculator_code leaks memory and reduces performance over time

Hi, We use execute_calculator_code() extensively for both reading and writing
but our queries are always simple (read or write single variable), read
requests go through gauge_calculator_code_precompile() first and cached, write
requests are a new string every time. Problem is, it gets slower over time to
the point that after 2-3 hours MSFS slows to the grind with 30-40ms extra
MainThread time compared to a fresh start. The same WASM modules worked fine
some time ago - we started getting these reports only recently (after the last
update, from the looks of it). It’s worth noting that we already had this
issue some time ago but it got fixed with MSFS update during one of the larger
perf fix updates, so this one looks like a regression. Yours, Alex

Hello Alex. Thanks for reporting this. We’ll run some tests on our side to
confirm the problem. Regards, Sylvain

Hi Sylvain, Thanks for your response and please keep me posted - this is a
release blocker for us, sadly.

I can confirm this is an issue I’m having with the FBW A32NX. When I start a
flight, even in a complex area, I have a minimum of 35+ FPS, but after
starting my descent or flying for more than 1.5ish hours, my FPS starts to
drop and by the time I’m on the ground, my main thread’s latency is in excess
of 30 to 40ms. The only way to get my sim’s performance back is to start a new
flight. I’m glad this is being looked into!

Any luck with this one?

Hello We ran some tests on execute_calculator_code itself but didn’t witness
any memory leak or performance drop so we suspect it’s context related. Can
you provide us with the package causing the problem so we can investigate?
Regards, Sylvain

A complex case like the FBW A32NX is not at all a good example to use as an
alleged indication of memory leak. To help developers (of any software, not
just MSFS) catch a memory leak (or any other bug), the reproduction case
should be as minimal as possible (but still exhibit the problem). And why
would the frame rate be related to a memory leak in WASM handling?

There is a big topic discussing this issue at
<https://forums.flightsimulator.com/t/after-playing-a-few-hours-fps-drops-
from-40-to-5fps/389603/1194> I’ve had this issue since pretty much launch TBH,
recently it seems that a lot more people are having this issue tho, lots of
people joined the discussion so I summarized the discussion for the new one,
allow me to paste this here as it might help if it’s the WASM issue after all:

LET ME SUMARIZE FOR NEW USERS WITH THIS ISSUE:

This performance degradation has been affecting many users for a long time, in
my case, I’ve had it since launch but it wasn’t so widespread Intrestingly, it
would seem that since the new years, a LOT more people seem to be having this
issue It’s not related to MFS flight planning before the flight, it still
happens with or without live weather, live or offline/online traffic on or off
it still happens if you turn off all online funcionality, it still happens if
you never open the VFR map at all or don’t have many icons on the toolbar (my
case)

In my case that have this since launch, I usually run it on Live Weather (no
change if off), all traffic off (since I’m usually on IVAO), settings on a mix
of high/ultra (does not affect the issue).

As of today, the community kind of narrowed it down to WASM planes, altho
there are some reports of people having the issue with default planes as well,
so it still might be unrelated, or 2 separate problems with similar outcomes

There are hints that it MIGHT be related to a great number of waypoints on
your FMC/MCDU/Flightplan

The most common presentation is the FPS starting to drop after around 1,5h of
flight time, but for some people it’s way before, and for some after 3 hours
or so, it tends to keep sloowly degrading until below 10fps

I’ve observed in the graphs sometimes it’s the mainthread, sometimes
rdrthread, sometime scoherentgtdraw, sometimes manipulators intrestingly

It happens all around the world

Turning rolling cache off does not affect it at all (at least for me)

Ram usage at least for me, tends to stay pretty low, usually around 5 to 10gb
even tho I have 32gb

I’ve had this issue with many planes including FBW A320, B78X, B748,
Longitude, TBM9 and CJ4 (admitedly, all modded variants), I’ve seem people
reporting it with the C152 tho (not personal experience)

Despite this long thread since April this year (keep in mind I’ve had this
issue since launch and made multiple reports of it on zendesk), Asobo seems to
refuse to acknowledge the issue at all.

I’m sorry so much more people are experiecing this now (stangely without any
significant updates to the sim), but I’m happy for it might finally get some
attention now

My rig is:

I7 8700K @4.9Ghz, 32gb RAM 3200mhz, EVGA RTX 3090 running on UWQHD
Monitor(3440x1440)

Hi, We’re getting reports that it’s much improved with the latest update
(1.21.18.0). In any case, I’ll run some more in-depth tests. I’ve assumed that
calculators are at fault as the simplest of affected WASM modules uses
calculators heavily and contains almost no other logic.

Indeed, we had similar issues before. At least once I’ve been able to
reproducibly pinpoint it to a single WASM method causing degradation over
time, reported and it got fixed. Other cases were much less productive - i.e.
A/B tests on the same machine with and without WASM module clearly showing
that WASM module is at fault (no perf degradation without it) but culling the
WASM module logic not giving expected improvements. And 3-4h test runs don’t
really help with testing any assumptions. I’ve almost started losing hair over
that one and then it… just went away one day.

Since the cases you describe here do not necessarily involve WASM modules, I
doubt they are the root cause of the issue - some other kind of memory leak
(perhaps several of them) somewhere else in the sim is probably to blame. Of
course we fix that kind of bug as we find them - but this is not exactly SDK-
related. Best regards, Eric / Asobo

The general public who don’t use dev mode will blame it on complex aircraft
which will accelerate the problem. I can confirm the two airports that we are
working on actively at the moment, once is less complex and doesn’t have the
problem as bad in dev mode editing scenery with a very basic plane loaded. The
other which has many many objects/apron work starts out ok, about 20 minutes
into it I have to reboot the sim because I can’t edit easily with the mouse
cursor anymore as the latency is so bad (1-2 fps and the mouse starts skipping
inputs. In my experience, if there are a lot of objects loaded in memory it
happens faster (not hours only 20-30 mins). I will say dev mode really shows
the issue. You can exit the sim and load the newly built scenery and the FPS
is much better. From a development point of view its made it pretty painful
doing SDK work and with this and the build process issue of rebuilding every
texture has eroded development output by quite a bit. In our camp we all
wished it was back to SU4/SDK where everything was working great. As a side
issue, we noticed with one of our simplier scenerys, if you put an OSM exclude
polygon down it drops the FPS a whole bunch while editing … not sure if
thats another clue. We dont put OSM exclude polygons down now until the very
end as the FPS hit is just too great. None of this has been easy either with
the complete dumpster fire that is the nvidia drivers at the moment. All of
this is on the latest production build and latest nvidia drivers.

I have since had a chat with Jorg and put together a video showing it happen
over 40 odd minutes while doing work in the Devmode/SDK which is being passed
onto the engine team. Hopefully this helps offer some direction to sorting
this out once and for all.

This is the reason that I am on this site right now. I was checking to see if
I was the only one having this problem. After editing my airport for 15 or 20
minutes, sometimes a little longer, the entire SDK becomes impossible to
operate. Yet 2 dyas ago, I worked on it for almost 2 hours with no problems.
Today I managed to go about 40 before is started to bog down. I sure hope this
gets fixed soon.

Ok, got a reply back from the team … the video I sent helped them track down
the source of the FPS issue / memory leak issue and they have fixed it in the
dev branch. Not sure when this will make it to production but I’m sure they
will have some more official news on the topic at some point. No more memory
leak and FPS issue. This should help us SDK devs and people on long-haul
flights.