Enhanced Audio System

[BenCheung0422]

Original: MinicraftPlus/minicraft-plus-revived#725
Requires #26

Details

The implementation would be based on OpenAL (OpenAL Soft, supported by LWJGL) with the spatial audio support. Firstly, the current audio system would be replaced by the addition of OpenAL with more functionalities and expansions. OpenAL is basically a 3D audio library, so spatial audio would be mainly handled by it.

The major feature is spatial audio, which has been mentioned before and is common in 3D game, but applicable to 2D game with proper implementation. Since the player character's direction might change in a flash, which is quite different from 3D that the player character changes the direction progressively, the implementation in 2D should not rely on player character's direction, but the relative coordinates or screen view. The disadvantage for the former one, as well briefly mentioned by Makkkkus¹. This could cause an abrupt audio outcome if the player constantly changes direction, which is not ideal. That is why I suggested implementing it with respective to the screen/top view, while technical playing may benefit from it by the reflection of directions of sources. Hence, this could improve gameplay experience, and I add that, in-depth spatial audio speakers like 5.1/7.1 may be experimentally supported.

Another key feature is distance attenuation, which has already been supported by OpenAL. This could greatly enhance player gameplay experience by the gradation of sound levels. In the current implementation, in-game sources are played at either 0% or 100% depending on whether they are in enough distance from the player, but the change may be sudden. Also, it may sometimes cause choppy transition if the sound source is continuous. Speaking of technical playing, this could reflect the distance from the sound sources. Therefore, distance attenuation could help solving this problem whilst make sure that the resultant volumes may not be too high in most scenarios.

The important part is the nature of audio data. For the sound effects which would be played in world, they should always be mono, as to be compatible with the spatial audio system. Currently, in the vanilla game assets, only explode.wav is stereo while others are mono, which would be something to solve before implementing this system, but should have been replaced by new assets.

On the other hand, audio mixing and compression, which I have made a pull request before to solve problems with simultaneous multi-playing of the same clips, as #544. Actually, OpenAL may require the similar techniques for the clip playing², but it is not today's topic. The major reason I used an algorithm for signal compression is that, no audio source is attenuated in the current implementation, so when multiple sources are played, the maximum thresholds of integer data can be easily reached, causing integer value overflow, requiring necessary clipping, as loudness war. Obviously, this is absolutely unideal and can be frustrating if not handled correctly. Instead, when there are audio attenuations applied while there are only single audio input for sounds like BGM, the necessity of compression after mixing is reduced, while the corelation of signals may be low.³ Thus, only little compression may be applied, or none at all. In-game audio volume controls for different types of sound sources should be made for such situation, like the one in Minecraft.⁴ In fact, optional but customizable (and may be advanced) equalizer or compressor could still be made into settings for users, though may be quite complicated to be in game.

Last but not least, the key points here still mainly focus on user experience and gameplay smoothness. Still, ensuring unhindered experience is the essential. If there are still doubts regarding the above implementations, you are welcome, but this is still not exhaustive enough to the final actual implementation.

Footnotes

1. Commented by Makkkkus in https://github.com/MinicraftPlus/minicraft-plus-revived/issues/624:

   [...] Also the player changes direction instantly, which would also be uncomfortable for the listener. [...]

   ↩

2. "OpenAL: how to play multiple sounds at the same time and mix them?" answered by Damon ↩

3. "Algorithm(s) to mix audio signals without clipping" answered by endolith ↩

4. "Algorithm(s) to mix audio signals without clipping" commented by pichenettes ↩