From 9e2761c59a9b459fd24228287f1d6765a4d39df4 Mon Sep 17 00:00:00 2001 From: Dominik Picheta Date: Tue, 20 Mar 2018 14:17:52 +0000 Subject: [PATCH] Fixes to post by Dmitry Fomichev. Refs #76. --- .../2018-03-12-steam-api-calls-forwarding.md | 139 +++++++++++------- 1 file changed, 83 insertions(+), 56 deletions(-) diff --git a/jekyll/_posts/2018-03-12-steam-api-calls-forwarding.md b/jekyll/_posts/2018-03-12-steam-api-calls-forwarding.md index 438cf6b70..c1b3726fc 100644 --- a/jekyll/_posts/2018-03-12-steam-api-calls-forwarding.md +++ b/jekyll/_posts/2018-03-12-steam-api-calls-forwarding.md @@ -1,25 +1,34 @@ --- title: "Using Nim to forward Steam calls between Linux and Wine" author: Dmitry Fomichev +excerpt: "In this article Dmitry shows us how he was able to use Nim's metaprogramming features to easily forward Steam calls between Linux and Wine." --- -*This is an English version of the [Russian article](https://habrahabr.ru/post/349388/) about the SteamForwarder project written in Nim originally posted on [habrahabr](https://habrahabr.ru/).* -Players on the GNU/Linux platform have a lot of problems. One of them is the necessity to install an another Steam client for each Wine prefix used for Windows Steam games. The situation is getting worse if we consider the necessity to install a native Steam client for ported and cross-platform games also. +
+
+
Guest post
+
+ This is a guest post by Dmitry Fomichev. It is an English version of the Russian article about the SteamForwarder project written in Nim originally posted on habrahabr. +
+
+
+ +Players on the GNU/Linux platform have a lot of problems. One of them is the necessity to install another Steam client for each Wine prefix used for Windows Steam games. The situation is getting worse if we consider the necessity to install a native Steam client for ported and cross-platform games also. ![](https://habrastorage.org/webt/h6/ad/8m/h6ad8m2tfoak1abb7kayqozqb9s.png) But what if we find a way to use one client for all games? As a basis, we can take a native Steam client, and games for Windows will address it just like, for example, to OpenGL or the sound subsystem of GNU/Linux - through the Wine. The implementation of this approach will be discussed below. # In Wine veritas -Wine can handle Windows libraries in two modes: native and built-in. The native library is perceived by Wine as a file with the extension `* .dll`, which it needs to load into memory and work with, as with the Windows entity. Wine handling all libraries, about which it knows nothing, exactly in this mode. The built-in mode implies that Wine must handle the access to the library in a special way and redirect all calls to a pre-created wrapper with the extension `* .dll.so`, which can access the underlying operating system and its libraries. More information about this can be read [here](https://wiki.winehq.org/Wine_Developer's_Guide/Architecture_Overview). +Wine can handle Windows libraries in two modes: native and built-in. The native library is perceived by Wine as a file with the `.dll` extension, which it needs to load into memory and work with, as with Windows. Wine handles all libraries, about which it knows nothing, exactly in this mode. The built-in mode implies that Wine must handle the access to the library in a special way and redirect all calls to a pre-created wrapper with the `.dll.so` extension, which can access the underlying operating system and its libraries. More information about this can be read [here](https://wiki.winehq.org/Wine_Developer's_Guide/Architecture_Overview). -Fortunately, most of the interaction with the Steam client occurs just through the library `steam_api.dll`. It means our task is reduced to implementing the wrapper for `steam_api.dll.so`, which will access `steam_api.dll`'s GNU/Linux counterpart - `libsteam_api.so`. +Fortunately, most of the interaction with the Steam client occurs just through the `steam_api.dll` library. It means our task is reduced to implementing the wrapper for `steam_api.dll.so`, which will access `steam_api.dll`'s GNU/Linux counterpart - `libsteam_api.so`. Creation of such wrapper is a well [documented](https://wiki.winehq.org/Winelib_User%27s_Guide#Building_Winelib_DLLs) process. We need to take the original library for Windows, obtain a spec-file for it using `winedump`, write the implementation of all functions mentioned in the spec-file and compile/link all the sources we have written using `winegcc`. Or ask `winemaker` to handle all the routine work for us. # The devil is in the detail -At first glance, the task is not so difficult. Especially considering that `winedump` can create wrappers automatically if there are header files of the original library. In our case header files are published by Valve for game developers on [the official site](https://partner.steamgames.com/). So, after creating a wrapper through `winedump`, enabling the built-in `steam_api.dll` mode via `winecfg` and compiling, we launched our own Steam, then the game itself and... The game is crashed! +At first glance, the task is not so difficult. Especially considering that `winedump` can create wrappers automatically if there are header files of the original library. In our case header files are published by Valve for game developers on [the official site](https://partner.steamgames.com/). So, after creating a wrapper through `winedump`, enabling the built-in `steam_api.dll` mode via `winecfg` and compiling, we attempted to launch our own Steam, then the game itself and... The game crashed! -Judging by the log, our wrapper works (!) exactly until the function `SteamInternal_CreateInterface` is called. What is wrong with it? After reading the documentation and correlating it with the header files, we can find that the function retrurns a pointer to an object of the `SteamClient` class. +Judging by the log, our wrapper works (!) exactly until the function `SteamInternal_CreateInterface` is called. What is wrong with it? After reading the documentation and correlating it with the header files, we find that the function returns a pointer to an object of the `SteamClient` class. -I think that those who are familiar with ABI C ++ already understand the crash origin. The root of the problem is the calling conventions. The C ++ standard does not imply the binary compatibility of programs compiled by different compilers, and in our case the Windows game is compiled by MSVC, while native Steam - by GCC. This problem is not observed for all calls to functions in `steam_api.dll` since they follow the calling conventions for the C language. Once a game receives an instance of the `SteamClient` class from the native Steam and tries to invoke its method (which follows the thiscall convention from C++), an error occurs. To fix the problem, we firstly need to identify key differences in the thiscall calling convention for both of compilers. +I think that those who are familiar with ABI C++ already understand the crash origin. The root of the problem is the calling conventions. The C++ standard does not imply the binary compatibility of programs compiled by different compilers, and in our case the Windows game is compiled by MSVC, while native Steam - by GCC. This problem is not observed for all calls to functions in `steam_api.dll` since they follow the calling conventions for the C language. Once a game receives an instance of the `SteamClient` class from the native Steam and tries to invoke its method (which follows the ``thiscall`` convention from C++), an error occurs. To fix the problem, we first need to identify key differences in the ``thiscall`` calling convention for both compilers. @@ -59,10 +68,10 @@ I think that those who are familiar with ABI C ++ already understand the crash o [[source](http://www.angelcode.com/dev/callconv/callconv.html#thiscall)] -At this stage, it's worth making a little digression and mentioning that the attempts to solve the problem have already been made, and even quite successfully. There is a project named [SteamBridge](https://github.com/sirnuke/steambridge), which uses two separate Steam libraries - for Windows and for GNU/Linux. The Windows library is built using MSVC and calls the GNU/Linux library using the Wine. The GNU/Linux library is built using GCC and can call `libsteam_api.so` directly. The calling conventions problem is solved by inserting an assembler snippets at the Windows library side and by wrapping each object when it is passed to the MSVC code. This solution is somewhat redundant, since it requires an additional non-crossplatform compiler to build and introduces an extra entity, but the idea of wrapping the returned objects is robust. We will borrow it! +At this stage, it's worth making a little digression and mentioning that the attempts to solve the problem have already been made, and even quite successfully. There is a project named [SteamBridge](https://github.com/sirnuke/steambridge), which uses two separate Steam libraries - for Windows and for GNU/Linux. The Windows library is built using MSVC and calls the GNU/Linux library using Wine. The GNU/Linux library is built using GCC and can call `libsteam_api.so` directly. The calling conventions problem is solved by inserting assembler snippets at the Windows library side and by wrapping each object when it is passed to the MSVC code. This solution is somewhat redundant, since it requires an additional non-crossplatform compiler to build and introduces an extra entity, but the idea of wrapping the returned objects is robust. We will borrow it! -Fortunately for us, Wine already knows how to handle calling conventions. It is sufficient to declare a method with the `thiscall` attribute. Thus, we need to create wrappers of all virtual methods of all classes, and in the each method implementation just perform a call of the method from the original class (which pointer should be stored in the wrapper object). The example wrapper implementation will look like this: +Fortunately for us, Wine already knows how to handle calling conventions. It is sufficient to declare a method with the `thiscall` attribute. Thus, we need to create wrappers of all virtual methods of all classes, and in each method implementation just perform a call of the method from the original class (whose pointer should be stored in the wrapper object). The example wrapper implementation will look like this: ```c++ @@ -152,13 +161,13 @@ trace:steam_api:SetWarningMessageHook ((ISteamUtils *)0x7c4b7930, (SteamAPIWarni -It would seem: the fairy tale is over? And here not! +It would seem: the fairy tale is over? Not quite! -# Welcome to the versions hell! +# Welcome to the version hell! -Very soon it turns out that our design is completely viable only for games compiled using the same header files that we have available. And we only have the latest version of the Steam API. Other versions of headers Valve does not publish (even the latest headers were given under a closed license). On the other hand, our Steam is also of the latest version, but it does not prevent it from working with the old versions of the Steam API. How does it do that? +Very soon it turns out that our design is viable only for games compiled using the same header files that we have available. And we only have the latest version of the Steam API. Valve does not publish other versions of headers (even the latest headers were given under a closed license). On the other hand, our Steam is also of the latest version, but it does not prevent it from working with the old versions of the Steam API. How does it do that? -The answer is hidden behind this line of the log: `trace:steam_api:SteamInternal_CreateInterface_ ((char *)"SteamClient017")`. It turns out that the client stores information about all classes of all versions of the Steam API, and the `steam_api.dll` only asks the client for an instance of the required class of the desired version. It remains only to find where exactly they are stored. First, let's try the straight approach: to find the "SteamClient016" string in the `libsteam_api.so`. Why not the "SteamClient017" string? Because we need to find a place were stored all the versions of Steam API instead of just the version of `libsteam_api.so` we have. +The answer is hidden behind this line of the log: `trace:steam_api:SteamInternal_CreateInterface_ ((char *)"SteamClient017")`. It turns out that the client stores information about all classes of all versions of the Steam API, and the `steam_api.dll` only asks the client for an instance of the required class of the desired version. We only need to find where exactly they are stored. First, let's try the straight-forward approach: to find the "SteamClient016" string in the `libsteam_api.so`. Why not the "SteamClient017" string? Because we need to find a place where all the versions of Steam API are stored instead of just the version of `libsteam_api.so` we have. ```bash @@ -168,7 +177,7 @@ $ grep "SteamClient016" libsteam_api.so $ ``` -It seems that there is nothing similar in `libsteam_api.so`. Then we will try to walk through all the libraries of the Steam client. +It seems that there is nothing similar in `libsteam_api.so`. So we try to walk through all the libraries of the Steam client. ```bash $ grep "SteamClient017" *.so @@ -179,15 +188,15 @@ Binary file steamclient.so matches $ ``` -And here is what we need! Let's curtain the Gabe Newell's portraint, if you have one, and open `steamclient.so` in the IDA. A quick keyword search shows an interesting set of strings which follow the pattern: `CAdapterSteamClient0XX`, where XX is the version number. What is even more curious, in the file there are lines with pattern `CAdapterSteamYYYY0XX`, where XX is also the version number, and YYYY is the name of the Steam API class for all other interfaces. The analysis of cross-references allows to find a table of virtual methods for each of the classes with such names easily. Thus, the summary scheme for each class will look like this: +And here is what we need! Let's open `steamclient.so` in IDA. A quick keyword search shows an interesting set of strings which follow the pattern: `CAdapterSteamClient0XX`, where XX is the version number. What is even more curious, in the file there are lines with pattern `CAdapterSteamYYYY0XX`, where XX is also the version number, and YYYY is the name of the Steam API class for all other interfaces. The analysis of cross-references allows to find a table of virtual methods for each of the classes with such names easily. Thus, the summary scheme for each class will look like this: The class memory layout -The method table is found, but we have absolutely no information about the signatures of these methods. But this problem turned out to be [solvable](https://toster.ru/answer?answer_id=1156239)(the source in Russian) by calculating the maximum depth of the stack the method tries to access. So we can make an utility that will be receiving the `steamclient.so` library to the input, and forms a list of classes of all versions, as well as their methods at the output. It remains only to generate a wrapper code for the method calls conversion based on the list. The task does not look simple, especially considering that the method signature itself is not known to us, we know only the depth of the stack where the method call arguments end. The situation is aggravated by the peculiarities of the in-memory structures return, namely the presence of a hidden argument - pointer to the memory where the structure should be written. This pointer in all call conventions is extracted from the stack by the callee, so we can easily detect it by the `ret $4` instruction in the assembler code of methods in `steamclient.so`. But even so, the amount of non-trivial code generation is huge. +The method table is found, but we have absolutely no information about the signatures of these methods. But this problem turned out to be [solvable](https://toster.ru/answer?answer_id=1156239)(the source in Russian) by calculating the maximum depth of the stack the method tries to access. So we can make a utility that will be receiving the `steamclient.so` library as the input, and forms a list of classes of all versions, as well as their methods as the output. We then only need to generate wrapper code for the method calls conversion based on the list. The task does not look simple, especially considering that the method signature itself is not known to us, we know only the depth of the stack where the method call arguments end. The situation is aggravated by the peculiarities of the in-memory structures returned, namely the presence of a hidden argument - pointer to the memory where the structure should be written. This pointer in all call conventions is extracted from the stack by the callee, so we can easily detect it by the `ret $4` instruction in the assembler code of methods in `steamclient.so`. But even so, the amount of non-trivial code generation is huge. # The Hero appears -To any new or just not very popular programming language, the question of its niche first of all arises. Nim is no exception. It is often criticized for trying to "sit on all chairs at once", implying a big amount of features in the absence of one clear direction of development. Among such features it is possible to distinguish two: +To any new or just not very popular programming language, the question of its niche quickly arises. Nim is no exception. It is sometimes criticized for trying to "sit on all chairs at once", implying a big amount of features in the absence of one clear direction of development. Among such features, the two which stand out are: - compilation to C and, as a consequence, easy cross-platform compilation; @@ -200,19 +209,31 @@ First, we create the main file `steam_api.nim` and a file with compilation optio -It does not look very simple, but it's only because we swung a lot at once. There is cross-compilation, importing functions from C header files and Wine specific compiler options... Despite the seeming complexity, nothing complicated has happened, we just directly implemented some parts of the source code in C that Nim does not know anything about, and at the same time we described for Nim how to call the TRACE macro from the Wine header files (we also told it about these files). +It does not look very simple, but it's only because we did a lot at once. There is cross-compilation, importing functions from C header files and Wine specific compiler options... Despite the seeming complexity, nothing complicated has happened, we just directly implemented some parts of the source code in C that Nim does not know anything about, and at the same time we described for Nim how to call the TRACE macro from the Wine header files (we also told it about these files). Now let's go to the most delicious part - to the code generation. Since we do not have complete information about method signatures, we will emulate instances of classes in C code, fortunately we only need to emulate the virtual method table. So, let's imagine that we have a file that describes the methods and classes of the Steam API as follows: @@ -275,15 +297,15 @@ Such a file can be obtained by parsing `steamclient.so`. We should get a table f -{% highlight asm %} +```bash push %ecx # put an object pointer to the stack (it will become the second argument) push $ # put the method number to the stack (it will be the very first argument) # the remaining arguments will move to 3 (two previous and the return addresses) -call # call the function written on Nim +call # call the function written in Nim add $0x4, %esp # remove the method number from the stack pop %ecx # remove the object pointer ret $ # remove the arguments from the stack and return -{% endhighlight %} +``` -It remains to generate the Nim functions we call in snippet. It is necessary to generate one function for each stack depth encountered in the file and one more for calls with a hidden argument. Further, we call these functions pseudomethods for brevity. +Now we need to generate the Nim functions we call in snippet. It is necessary to generate one function for each stack depth encountered in the file and one more for calls with a hidden argument. Further, we call these functions pseudomethods for brevity. -Lets leave the implementation of the `wrapIfNecessary` function behind the brackets and proceed to the description of the code that generates the fragments described above. First, read the file with class descriptions. We will get the path to the file in the same way as we got the path to the spec-file - via the compiler option. +Let's leave the implementation of the `wrapIfNecessary` function behind and proceed to the description of the code that generates the fragments described above. First, read the file with class descriptions. We will get the path to the file in the same way as we got the path to the spec-file - via the compiler option. - + -Now we've got a class table. Since the `readClasses` function does not use anything that is possible only at runtime, we can safely compute it at compile time and write the result to a constant like that:`const classes = readClasses ()`. Let's create a table of methods-wrappers, consisting of assembler inserts, described above. +Now we've got a class table. Since the `readClasses` function does not use anything that is possible only at runtime, we can safely compute it at compile time and write the result to a constant like this: `const classes = readClasses()`. Let's create a table of method-wrappers, consisting of assembler inserts, described above. -Похожим образом создаются объявления основных функций `steam_api.dll`. Для проброса вызовов обратно из GNU/Linux в игру форма уже известна и едина для всех версий Steam API, поэтому нет нужды в кодогенерации. Например, определение первого метода будет выглядеть так: Similarly, the function declarations from `steam_api.dll` are created. To forward calls back from GNU/Linux into the game, the mechanism is already known and unified for all versions of the Steam API, so there is no need for code generation. For example, the definition of the first method would look like this: @@ -543,9 +570,9 @@ proc run(obj: ptr WrappedCallback, p: pointer) {.cdecl.} = # Conclusion -So, we've covered the main key points that allow us to generate a wrapper for the Steam API at compile time. No matter how complex they seem, this approach undoubtedly wins in comparsion to the manual writing of several hundred similar methods. Nim wrote all these methods for us. Someone may ask: "And what about the debugging of all this code?". The issue of debugging the compile time code is really complicated. The only tool available is the good old debugging messages `echo`. Fortunately, Nim has the functions `repr` and` treeRepr`, which turn the AST into lines of code and a string with an AST nodes diagram, respectively, which greatly simplifies debugging. +So, we've covered the main key points that allow us to generate a wrapper for the Steam API at compile time. No matter how complex they seem, this approach undoubtedly wins in comparsion to the manual writing of several hundred similar methods. Nim generates all these methods for us. Someone may ask: "And what about the debugging of all this code?". The issue of debugging the compile time code is a little complicated. The only tool available is the good old debugging messages `echo`. Fortunately, Nim has the functions `repr` and` treeRepr`, which turn the AST into lines of code and a string with an AST nodes diagram, respectively, which greatly simplifies debugging. -Particularly noteworthy is the flexibility of the Nim compiler. Compiling in C, combined with high-end support for metaprogramming, allows it to be considered as both of a super-powerful preprocessor for C and a separate language compiler that is not inferior to C's capabilities in a wrapper of nice python-like syntax. +Particularly noteworthy is the flexibility of the Nim compiler. Compiling to C, combined with high-end support for metaprogramming, allows it to be considered as both a super-powerful preprocessor for C and a separate language compiler that is not inferior to C's capabilities in a wrapper of nice python-like syntax. Perhaps, the article will seem too chaotic, since it is not easy to describe a complex task and its solution, in which the language is revealed at full power, in a simple and concise manner. Unfortunately, within the framework of this article, it was not possible to describe a few more aspects, namely: @@ -565,7 +592,7 @@ Such details, in my opinion, would worsen the perception even more. In addition, The working solution of the problem identified in the header is presented in the repository on github: - here you can find [the C++ solution without Nim macros but with codegenerator scripts implemented in Nim](https://github.com/xomachine/SteamForwarder/tree/8f2b8cea17da8718dfd8a87fbd2677d475abb54a), which requires Steam API headers and works with only one version of Steam API; -- here you can find [the Nim solution](https://github.com/xomachine/SteamForwarder/tree/a7dea4b6a87086b93d4165090d85ec8134985962) which described in second half of the article. +- here you can find [the Nim solution](https://github.com/xomachine/SteamForwarder/tree/a7dea4b6a87086b93d4165090d85ec8134985962) which is described in the second half of the article. Some names of variables and functions in the original code differ from the examples given in the article. Links are given on the commit of each branch, which is the top at the time of publication, so as not to lose relevance with time.
MSVCGCC