Let me start with just proposing that we accept the obvious:
The SimConnect API is a dinosaur written for old-school C programs in a
barely networked environment.
Arguments:
The header file barely rises above the C standards available when FSX came out, using a few enum types and typedefs, but most of the time just sticking to the worst bit of Kernighan & Richie C: preprocessor "#define"s. Since C++11 (2011!) compilers have been able to deal effectively with compile-time constant expressions, including compile-time evaluated functions. Visual Studio supports this.
The API sticks to the outdated notion of needing the Hungarian notation to be able to grasp a nameâs type quickly. This may have been the case when text-based terminals (or terminal windows) were the norm, but modern IDEs, certainly including VS2019 and VS2022, will show the declared types of any name you hover the mouse over. The result is a naming standard that forces you to constantly recheck the declared type, even though the âanything not typed is an intâ rule has been forcefully ejected from C and was never a part of C++, so making mistakes will be quickly caught by the compiler.
Many structures are defined with explicit bounds on character arrays rather than using safe and dynamic type alternatives available in the C++ language standard. In some cases, it can be (somewhat) accepted because it makes a memory buffer predictable in size. In other cases, it uses the worst version possible, declaring a character array of size 1, just to provide the starting point. These unfortunate practices are the major reason buffer overflow errors still rank at the top of the security concerns.
Many structures make use of (for example) Windows-specific âWORDâ and âDWORDâ types, while standard âuint16_tâ, âuint32_tâ, and the like have been available for ages. Pulling in Windows dependencies could easily have been limited to just a handful of API calls.
Error returns from API calls can be returned in (at least?) three ways: (1) directly because some values passed were obviously wrong, such as a null value as the connection handle, (2) an error message may be returned based on the âPacket Id,â most likely still because of a bad parameter, but now detected by the simulator rather than the SimConnect library, and (3) an error message referencing a Request Id. The last two will be delivered to the SimConnect client as if it is a normal message from the simulator, so you need to keep track of Packet and Request Ids to be able to provide the correct part of the client app with the appropriate feedback. However, for many calls, there will be no feedback if everything is fine.
SimConnect clients are unable to request certain types of simulation variables, most notably the L-vars. As a result, every add-on developer who wants to interface with another add-on based on such variables will have to find some way to get around this, resulting in many incompatible embedded add-ons that expose some kind of (proprietary) network server. This embedding is then achieved using fake gauges or (otherwise useless) WASM modules.
The SimConnect network protocol is an unspecified and closed-source affair. Those who want something better (or other than provided) are again forced to similar solutions as the previous point.
Proposal:
Provide a modern (may I hope for C++20-conforming? Weâre living in 2022 already and VS2022 supports most of it) API. This can even be achieved with a relatively lightweight shim on top of the current API, as long as it gets a bit more effort than was put into the Managed SimConnect API so that it will improve the situation.
Ditch the interconnection protocol for something modern, so it provides both the required performance as well as a proper client-server paradigm. At minimum HTTP, preferably something like gRPC. The latter will allow for efficient (binary) data transfer, ensuring that error reporting can be cleaned up, and making the API language-agnostic. If done properly, there will be no need to fear future changes in the protocol, as a protobuf definition of the API can be kept forward and backward compatible.
If the major argument against is lack of manpower, recruit volunteers from the
community. There should be more than enough experienced developers willing to
join in. Cheers, Bert Laverman
100% agree, but now, probably is about 5 years too late. I cannot imagine such
a major change/ re-design , now, 2 years into the release of MSFS2020. Maybe
for MSFS2040/50, and if so, yes, maybe SOON is the time to start forward
thinking about it .
Hello @bert.laverman Although we can agree on
a lot of what you mentioned, as N6722C said, thatâs not something we are going
to tackle in short or medium term. We have no plans for such development at
the moment. Regards, Sylvain
Sure, WASM was a very good choice but, to be really useful, it needs extra
APIs it can call, either through Simconnect or possibly new interfaces. For
example, something that I donât see as too complex to add and it will make
developers more keen to use WASM, would be the ability for WASM to make
Coherent calls , this will streamline a lot of the integration issues
between Simconnect and the HTML/JS framework. And, the ability to read any
file in the VFSin read-only mode. I really donât see what issues could
cause accessing data already in the MSFS VFS file system (and not outside of
it, if the concern was Xbox security), in read-only mode. Developers requiring
that, will do it anyway, through an external .EXE so, what would be the point,
other than preventing potentially interesting applications to reach the Xbox ?
WASM is a completely separate discussion from SimConnect and other APIs. It is
a (virtual) target for compilers allowing you to generate fast (âassemblyâ)
code without sacrificing portability. IMHO, in the context of MSFS 2020, its
primary purpose (if not only purpose) is to allow the targeting of the X-Box
and Windows platforms with one and the same executable. This is why it was
originally developed for browsers, and the limitations of its capabilities
stem from the security concerns we have in that context. If youâd want to add
âdisk access,â youâd need to add an API for it, but the limiting factor would
be the commonality between what is on disk in an X-Box and a PC. I would guess
that you want this so you can scan the scenery database, as that is been
famously left out of SimConnect. Instead of having to reverse engineer the BGL
file format (all available documentation on it is unofficial, despite its
usefulness), it would be better to add an API because then the simulator can
control the how of it and ensure the performance of the system.
Yes, and so the saga continues. New developers must learn outdated programming
paradigms and re-invent the wheel with saddening regularity. Even if you donât
want to spend the time on it, publishing or sanctioning certain bits of
documentation could help a lot. If we have a formal specification of the
TCP/IP-based network protocol used by SimConnect to connect remote clients, we
can (hopefully) put an end to several of the API protocol issues by building a
modern API on top of it. (See my 5th point above) If we have a formal
specification of the BGL file format, we can ensure there will be a stable and
well-performing access layer to the scenery database. As I said, you donât have to do it yourself. Your forte is the simulator, ours is to build useful
add-ons. Letâs ensure these two worlds can work together. Better support is
not achieved by putting up walls⊠as (some departments of) Microsoft appear
to have learned.
WASM is strictly related to Simconnect, because Simconnect IS one of the
(few) API it can call. In fact, for Stand-alone modules, WASM is one of the
two API it can call, because the other gauge rendering related ones are not
available there. About disk access, Xbox compatibility was precisely the
reason why I specifically mentioned getting read-only access to files in the mounted VFS , not elsewhere, which makes any difference between PC and
XBOX not relevant, since thatâs not a real file system, but something that has
been created by the sim itself. Scanning BGLs is only one possible application
of requiring access to files outside the package, and this specific
application might be covered by the upcoming updated Navdata API, which is
supposed to return data about airports, but hasnât fully documented yet. But
there might be other cases in which file access outside the module own package
could be useful. For example to read data from the various airplane .CFG files
that might have extra options there still not available as Simvars, or reading
data in the airplane XML behaviors.
Iâm sorry to ask a candid question here, but what would be the purpose of
reading the behavior file of your aircraft from your own WASM code, since it
is your aircraft and you therefore know whatâs in your behavior file already?
Another candid question, is what is the point of having a plane config file, if you are going to ignore those values in your wasm / js code, and hard code different values to replace what you should have got from those cfg files ?
This is an interesting thread. FWIW Iâm a university computer science researcher that processes a lot of real-world sensor data in real time. SimConnect is an excellent API in terms of latency and compactness of representation and thereâs a huge risk that any implementers of a new API mess it up with the absolutely classic error of POLLING for the data instead of SUBSCRIBING for updates. Suggesting the data could be requested over HTTP is an example highly likely to lead to a broken implementation (websockets over http would be fine).
Asobo has already implemented some portions of MSFS code in separate threads with no publish-subscribe mechanism between those parts (the html/js gauge system is the primary example) so that data read in JS is no longer real-time in sync with the simulation engine. Itâs typical (and understandable) that developers make these mistakes with no knowledge that there are efffective âpushâ mechanisms that would communicate the data with much lower latency while also ensuring a poorly implemented âsubscriberâ has no impact on the high performance âpublisherâ.
These issues donât matter if all you want to do is move a damped needle on a gauge using a single simvar already available in the sim, the random latency between the html/js data reads and the updating of simvars inside the sim is easily smoothed away for such a simple application. But as soon as you try to do calculations between variables and update the sim with reasonably low latency you find thatâs not really possible.
So be careful what you ask for.
There are reasonably plenty of questions whether the API functionality could be made more comprehensive, for example the ability to query and read the VFS or similarly other relatively static data inside the sim (e.g. itâs unclear whether MSFS should make the loaded WPR file available or rather provide a readable internal data structure for the currently loaded weather). Iâd wholly agree with that (e.g. for gliders I checksum the flight_model.cfg and desperately need to be able to read the weather) but Iâd suggest it could be done with a new API that can communicate more complex data structures rather than an extension of SimConnect which is pretty good at float doubles but really clunky for anything else.
Yes, polling is definitely what weâre trying to avoid, but the SimConnect API depends on the outdated Win32 Window Message loop to deliver data, which doesnât translate well. There is the possibility to provide for passing a handler as a callback, but the documentation does not clearly guarantee that this will actually be used. Even worse is the error handling, as I already mentioned in other threads, since there is no concept of a transaction when all goes well. If there is an error we can get failures reported using up to three different channels: direct by the SimConnect library, as an exception message related to our request, or as an exception message related to our data definition block. Also, most IDs are completely client-defined, without any documentation on how long they remain valid. If we reuse Request IDs, as the API documentation and examples clearly suggest, we no longer know if an exception relates to the last request or a previous one. If the server actually disallows this to occur, then we sorely miss that fact in the documentation. A proper RPC protocol would allow for us to gain that linkage.
Please note that HTTP, especially since HTTP/2, provides lots of opportunities to use reactive programming, which allows us to do away with the polling as well. gRPC, which is layered on top of HTTP/2, additionally abstracts away the underlying protocol and gives us a bi-directional RPC capability with streaming as needed.
IMHO the API is fine for anyone not afraid of C. Itâs maybe not pretty but itâs efficient. Iâve seen lots of pretty code out there which actually fails at itâs primary mission (whatever that may be). Simple structures pack nicely and are fast to transmit/consume. The supporting SDK files (headers/etc) are generated for both managed and un-managed code, which is rather clever. It is also much simpler to integrate an old-school C-style DLL with other languages (Python, Node, whatever). If someone wanted to stick a pretty C++20 interface on top of that, nothing is stopping them.
You can use an event handle instead. A window handle isnât required, and hasnât been since I can remember.
Thatâs the nature of async communication, no? SimConnect isnât going to know what is a valid value for the sim itself, itâs just the messenger. Besides validating the handle thereâs not much else it can do. To track errors one needs to keep a record of requests. I agree itâs not entirely trivial, but itâs certainly possible. If the comms were synchronous that would pull performance and efficiency down the drain.
They remain valid for the duration of the SimConnect session, from calling Open to calling Close. I donât know why anyone would re-use a definition ID â they should remain static and associated with the same sim variable/event/etc throughout the session (Iâm not sure which examples suggest otherwise⊠most examples in the API docs are standalone). IIRC, SimConnect will return an error if you try to map a new definition to an existing ID.
