Hello everyone,
As part of our efforts to improve performance and resolve stability issues, especially on consoles, a new tool has been introduced for SimObject profiling with SU5.
This tool compiles statistics after running a benchmark on your SimObject and highlights any statistics that overshoot a set of performance thresholds that have been defined to ensure good performance.
Complete documentation can be found here: The SimObject Statistics Profiler
We strongly encourage you to use this tool and make any necessary adjustments to match our guidelines. Providing results from this tool is required for content invited to the PS5 Marketplace, and will soon become required for content to release on Xbox.
A similar tool and improved version of the Scenery Statistics Profiler is expected to be available in the future.
We wanted to give you advance notice so you can plan your changes, as it is likely this tool will become part of the ingestion process at some point, and making those adjustments will become a requirement.
We do not have an ETA for this scenery Statistics Profiler at this point but expect more communication from us on this subject in the future.
We are aware of some issues that could prevent you from following some of our guideline. We will continue to work on this area and make further improvements.
In the meantime, if you end up in a situation where our guidelines conflicts with some of the things you had to do to work around another problem, please provide a link to your bug report in this thread.
-The Microsoft Flight Simulator Team
@FlyingRaccoon, we’ve already been using the tool to try and optimize some of our aircraft for potential PS5 release.
One thing that became apparent that we didn’t initially realize is that it seems that the graphics settings used by the person running the tool has a lot of influence on the numbers reported, with the end result being that different people, with different PC specs, graphics cards and monitors in use, end up seeing different numbers.
In some cases, this is the difference between GPU Memory being a green or a yellow versus being far into the red.
After this came to light, we verified this behaviour by simply having a single person run the tool at different settings.
Here’s an example at ‘ultra’:
Versus ‘high’:
This obviously isn’t ideal. We can’t really get everyone onto the same baseline for testing either; the highest possible settings aren’t possible for all of our people due to the aforementioned hardware differences, and it’s obvious that a low setting misrepresents the actual load the aircraft is capable of representing.
Repeated tests on the same aircraft with the same settings have also revealed some fluctuations in vertices and faces values, seemingly driven by a variance of the shadow values being reported. In one instance, it was enough of a difference to raise the reported exterior faces number on the aircraft I’m currently working on (not the above) from a green to a yellow. This wasn’t as critical, but it does serve to question the accuracy of the tool, especially since it’s being used to drive decisions that have an impact on platform availability of a product.
There doesn’t seem to be any sort of recommendation or acknowledgement of these potential differences on the documentation page.
Since the returned results from this tool are settings dependent, are there baseline settings that are being used to determine what constitutes a pass / fail for inclusion (on the PS5 now, on the Xbox in the future)?
@FlyingRaccoon
That’s a very good step at giving us visibility on the actual impact of our aircraft, and actually gives us targets to optimize against.
As we know, gauges are a contributer to the memmory consumption of an aircraft as well ,
can you explain how to profile gauges - but not in vaccum- but in actual relation to the total memory consumption of an aircraft .
Do we have a total budget for artwork assest + gauges? or do gauges operate on a different memory allocation? how do we ensure that our gauges do not get deactivated to save on gpu memory by the core sim? are there specific target metrics we need to hit? how do we know if a gauge is performant or is borderlining maxing out the memory allocation for the aircraft asset?
Hello @JimStewart @MarwanGH7929
Yes, this will be addressed in the SU6 version that will constrain the context.
In the meantime, you can run it in “High” settings as this is what we will rely on.
This is not covered by this tool. Gauges will be loaded but remain in the default startup state.
It is unlikely this will be covered in the future as this too dependent on code path.
Regards,
Sylvain
@FlyingRaccoon noted and understood.
I was looking for a way in general to debug memory and performance impact of gauges to better understand if needs optimizations from our side .
If we have better tools, like a visualizer that can tell us what is the frame time of each gauge and its memmory , and can compare that to reference values that would be great.
reason why I am asking is that it seems like JS gauge that otherwise runs well on pc/xbox are causing CTDs / OOMs on PS5. We are trying to understand why.
Hello, I’d like to share my opinion on this, if feedback is welcome:
-
While I think this is a useful tool for diagnosing and optimizing our packages, I also believe it can be a double-edged sword.
-
In my view, it should be something that helps us identify and reduce polygonal load when we consider it appropriate, and should remain an optional optimization aid rather than becoming a restrictive requirement that automatically blocks updates or releases when a product is otherwise working correctly, simply because an automated tool has determined that it should be optimized without any human actively testing whether the product is actually causing performance issues.
-
The limits that have been imposed seem quite arbitrary and restrictive to me. For example, the limit has been set at 20,000 with a bounding sphere of 1.5. Here is a screenshot of the analysis for one of my aircraft, which currently passed testing as a PS5 candidate, performs perfectly well, and for which nobody has reported any performance-related issues.
-
We currently have packages in circulation on PC, Xbox, and PS5 that easily exceed that figure.
-
Exceeding that figure does not necessarily mean that a product performs poorly on those platforms. Instead, this system is now being used to reject updates on MCP that we have already actively tested and confirmed to work without issues. At that point, the testing we perform ourselves and the feedback we receive from users effectively become irrelevant, because the final decision has been delegated to an automated analysis tool that does not take into account factors such as whether the actual framerate is optimal.
-
There are many other factors that can have a much greater impact on performance. Poorly coded avionics, for example, can cause stutters, freezes, and even simulator crashes.
-
Therefore, I do not think this tool is objective enough on its own to determine whether a product is a suitable candidate for a platform, or whether an update should be rejected, especially when the product demonstrates no actual performance problems in practice.
I also do not understand why 20,000 was chosen as the limit. I have products, and have tested other products, that exceed 35,000 without any issues whatsoever. They can still perform perfectly well at 144 FPS in 4K on PC and maintain the required 30 FPS on PS5 and Xbox.
-
Thanks.
Postscript: The reason I am giving my opinion is not because you rejected my product as a PS5 candidate. The product already is a PS5 candidate, it is currently published on PS5, and it is running with the figures you can see in the screenshot.
The problem is that I have just submitted an update that fixes other aspects of the product and has nothing to do with this. However, the update has been rejected because of a parameter that was already present in the currently published version and that, so far, has not been shown to cause any actual problem.
This is where I think the system stops being a diagnostic tool and starts becoming a restriction that is difficult to justify.
I cannot simply take a finished product now and start reworking a part of it that:
-
Works perfectly on the target platform.
-
Has already been approved, tested, and published on that same platform with those characteristics. We are not talking about a new version of the model that has suddenly increased in complexity.
-
Does not present any known memory, framerate, stability, or crash issues. If there were an actual problem associated with it, then of course it would make sense to investigate and correct it.
-
Has not been modified by the update I am trying to publish. The update fixes completely unrelated aspects. Therefore, we are blocking a fix because of a condition that already existed in the previously approved version.
-
In order to comply with that limit now, I would have to significantly reduce certain parts of the model. And this is not necessarily a matter of pressing an “optimize” button: it may involve remodeling geometry, modifying UVs, redoing baking, reviewing materials, and retexturing parts of an aircraft that are already finished.
-
This means taking on significant costs in both time and money in order to modify something that currently has no demonstrable technical problem.
-
Furthermore, changes of this kind can introduce new problems. Reworking the geometry of a finished product is not risk-free: visual issues, animation problems, UV issues, material problems, LOD issues, decals, etc. can appear. In other words, paradoxically, I could be forced to modify a stable part of the product in order to solve a problem that does not actually exist in practice.
-
I am also concerned about the precedent this sets for products that are already published. If every time the criteria of an automated tool change we have to retroactively adapt every element of a product, even when those elements have not changed and were previously approved, maintaining products in the long term can become extremely costly.
-
I believe there should be a distinction between an optimization warning and an error that blocks publication. A tool can tell me that a certain part of the product is above a recommended value, and I find that information useful. But automatically turning that recommendation into a reason for rejection should require some evidence that it is actually affecting performance or stability.
-
The overall context of the product should also be taken into account. Two models with exactly the same polygon count can behave completely differently depending on materials, draw calls, textures, LODs, avionics, scripts, effects, etc. Therefore, a single metric should not be used in isolation to determine whether a product performs correctly.
-
And especially in this particular case, there is fairly clear evidence: the product is already running on PS5 with those exact figures. That is why it seems strange that an update which does not modify that aspect is being rejected precisely because of something that the currently published version is already using without any issues.
I am not saying that optimization is not important. It is, and tools like this can be extremely useful in helping us identify areas that can be improved. What I question is whether an automated limit should carry more weight than the actual behavior of the product, the testing that has been carried out, and the experience of users.
If a product exceeds a recommended value but performs correctly, maintains the target framerate, has no memory issues, and does not cause instability, I believe there should at least be the possibility of reviewing it as an exception or treating that result as a warning rather than automatically blocking the update.
