Public NWN2Dev environment

Copyright (C) 2008-2011 Ken Johnson (Skywing).
Major portions of Granny2Lib copyright (C) 2006-2009 tazpn (theo).
Portions of NWN2MathLib copyright 2001, softSurfer (www.softsurfer.com).
Portions of NWN2DataLib's TRX handling are based on information from Tero
Kivinen <[email protected]>.
Credit goes to Justin Olbrantz for writing significant portions of the
NWScript analyzer module.
Major portions of NWNScriptCompilerLib were written by Edward T. Smith
(Torlack).
Portions of NWNScriptCompilerLib were contributed by OpenKnights.
For zlib-related copyright and acknowledgement information, see the file
zlib/README.

---------------------------------------

This source release includes libraries to read and manipulate various
NWN2-related data formats.  It also includes a sample demonstration program
that is capable of rendering an MDB using these libraries (ModelRenderer).

The source release also includes a complete virtual machine for the NWScript
compiled instruction set.  This virtual machine allows external programs to
embed an execution environment for the NWScript language.  Note that it is up
to the program hosting the virtual machine to provide implementation for engine
actions and engine structures laid out in nwscript.nss.

The project is structured to be built with the Windows Driver Kit (WDK).  You
can download the Windows 7 (7.1.0) WDK here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=36A2630F-5D56-43B5-B996-7633F2EC14FF&amp;displaylang=en&displaylang=e

The WDK is a complete build environment that includes libraries, headers, and
compilers and other build tools necessary to build the entire project.

The project is also buildable with Visual Studio, but that build environment is
no longer maintained by me and I haven't included the project files as a result.

To build the project with the WDK, use these steps:

1.  Open the Windows XP x86 Free build environment.  (The WDK installer will
    create a start menu shortcut by default.)
2.  Change to the public_nwn2dev directory in the build window.
3.  Run WDKEnv.cmd from the build window which sets up various environment
    options for the build system to work.
4.  Type the command: "bld"


Be sure to place the project files in a path with no spaces in any path
components.  Otherwise, you may receive an error along the lines of so:

"1>c:\winddk\7600.16385.1\bin\makefile.new(7116) : error U1087: cannot have : and
 :: dependents for same target"

To resolve this error, place the project files in a directory hierarchy without
spaces, such as C:\Projects\nwn2dev_public.


On a successful build, ModelRenderer.exe is placed in the directory
ModelRenderer\objfre_wxp_x86\i386\ModelRenderer.exe.

Builds for amd64 (64-bit) are also supported. The Windows Server 2003 x64 build
environment window is recommended for use for amd64 builds.


The project is laid out into several different components:


External provides various external components not shipped with the WDK.

zlib is the standard zlib compression library (unmodified).

minizip is the zlib/contrib minizip library, with some modifications to improve
performance when doing sequential scan enumeration of large archives.

SkywingUtils includes a smart-pointer class similar in nature to boost's
std::tr1::shared_ptr<>.

Granny2Lib encapsulates tazpn (theo)'s Granny2-reading logic in a single
library, isolated from the rest of the projects.

Gr2Conv is an x86-only executable that can be used to decompress *.gr2 files
out of process.  It is used to support Granny2 reading on non-x86 platforms.

NWNBaseLib provides various common definitions used by the other NWN-related
project code (such as the definition of NWN::OBJECTID).

NWN2MathLib is a math library that encapsupates various useful mathematical
constructs, assembled from various sources.

NWN2DataLib encapsulates logic to process a variety of NWN2-related data
formats other than *.gr2 files (which are handled by Granny2Lib).  For example,
logic to handle MDB models and TRX terrain data files is provided in
NWN2DataLib.  Additionally, the library provides a unified resource accessor
interface (compatible with the BioWare resource manager) to load data files in
the same search order as the game.

NWNScriptLib encapsulates a virtual machine for the compiled NWScript
instruction set (*.ncs files).  The virtual machine can execute *.ncs files
programmatically.  It is up to the user of the script library to provide
implementations of engine action handlers and engine structures.

ModelRenderer is a simple GDI rendering application that utilizes the above
libraries to render a model.  Model handling is simplified and does not support
multiple skeletons per model as per the full rendering logic in the Client
Extension.

NWNScriptConsole is a simple console application that can run NWScript programs
on the command line.  It is a simple frontend host for NWNScriptLib.  Only a
small portion of the NWN2 nwscript.nss is implemented by NWNScriptConsole.

NWNScriptCompiler is a compiler driver (shell) for NWScript compilation, based
on NWNScriptCompilerLib.

NWNScriptCompilerDll is a DLL version of the NWScript source compiler, used by
the compiler toolset plugin.

NWNScriptCompilerLib is a static library for compiling NWScript source text to
compiled NWScript (*.ncs).

NWN2ToolsetCompilerPlugin is a .NET 2.0 C# DLL that acts as a toolset plugin to
replace the standard script compiler with the version exported by
NWNScriptCompilerDll.

NWNScriptJIT is a just-in-time (JIT) compilation system for *.ncs files
(NWScript compiled scripts).  The NWNScriptJIT module can be used to execute
NWScript scripts at high speed.  It requires the .NET Framework and must be
built from within Visual Studio.

NWNScriptJITIntrinsics is a support module for NWNScriptJIT.dll.  Generated
.NET assemblies for JIT'd scripts depend on this module.  This module requires
the .NET Framework and must be built from within Visual Studio.

ListModuleAreas is a sample program that shows how to use the included
libraries to traverse the contents of area files in a module.

ListModuleModels is a sample program that shows how to use the included
libraries to enumerate all model files accessible from within a module.

UpdateModTemplates is a sample program that shows how to use the included
libraries to read data, modify, and write back data in placed object instances
in a module's area files.


All of the libraries are intended to be built as static libraries (*.lib).  The
libraries generally expect the caller to provide any thread synchronization if
used in a multithreaded environment.



A good place to start in understanding how to use the libraries would be to
examine how the ModelRenderer program loads and processes a .mdb model.  For
the ModelRenderer program to work, you will need NWN2 installed on your
computer.  You need to tell the ModelRender program where your NWN2 install is
and what module's data files should be loaded by setting environment variables
before executing it; see WinMain in ModelRenderer\ModelRenderer.cpp for more
details.

Skeleton processing requires an NWN2 install with granny2.dll available and is
thus only available on x86 builds.

Most users of the library set will want to link to SkywingUtils, NWNBaseLib,
NWN2MathLib, NWN2DataLib, zlib, and minizip.  It is typical for a user to
create a single ResourceManager object (from NWN2DataLib) and use it to load
data files in from disk.  This strategy makes the most out of the caching and
load-time optimization logic located in the ResourceManager.

The Client Extension uses the included libraries to support it's internal
handling of NWN2 data files.


Data file handling for specific formats:

.TRX / .TR? and .MDB files are handled by class TrxFileReader.
MDB SKIN meshes are handled by class SkinMesh.
MDB RIGD meshes are handled by class RigidMesh.
MDB HAIR points are handled by class HairPoint.
MDB HELM points are handled by class HelmPoint.
MDB WALK meshes are handled by class WalkMesh.
MDB COL2/COL3 meshes are handled by class CollisionMesh.
MDB COLS spheres are handled by class CollisionSphere (not fully implemented).
TRX ASWM records are handled by class AreaSurfaceMesh.
TRX WATR records are handled by class AreaWaterMesh.
TRX TRRN records are handled by class AreaTerrainMesh.
GR2 Skeleton structures are parsed by Granny2Lib and wrapped by ModelSkeleton.
2DA files are handled by class TwoDAFileReader.
GFF files are handled by class GffFileReader and class GffFileWriter.
NCS files are handled by class NWScriptReader.
NSS files are handled by class NscCompiler.


Model system design notes:
--------------------------

The model system is designed to work with shared model meshes.  A logical model
is represented by class ModelCollider.  Each ModelCollider owns its collision
meshes (in world space) and maintains shared references to its meshes in the
form of a pointer to a ModelInstance object.  ModelInstance-based meshes are
stored in model local space.

It is intended for a single ModelCollider to represent a model part (i.e. a
chest model), and for a single ModelSkeleton to represent a skeleton (i.e. a
cloak skeleton).  These model components would typically be aggregated into a
single overarching "whole model".  A greatly simplified example of this can be
seen in the ModelRenderer's WorldObject class.

The stock game and the Client Extension's model system support model attachment
and multiple skeletons, but for clarity, these features are omitted from the
base model system in the ModelRender program (in the interest of cutting down
down on extra layers of abstraction).


GFF file writer design notes:
-----------------------------

There are two build-time configuration options for the GFF writing system.
These options can be set in NWN2DataLib\GffFileWriter.h.  They only impact the
run-time performance (not the final output), by tuning the GFF writer for two
different scenarios.


Define GFFFILEWRITER_PRETRACK_STRUCTS to zero to optimize the writing system
for in-place editing of large data trees that are written out only once per
GffFileWriter instance.  This mode of operation is slower if you save the same
GffFileWriter object and repeatedly commit it, because the data tree is
reindexed on the fly for each write commit operation.

Define GFFFILEWRITER_PRETRACK_STRUCTS to one to optimize the writing system for
for editing of large data trees that are repeatedly written out with the same
GffFileWriter instance.  This mode of operation imposes a significant
performance penalty if your code frequently deletes large sections of a GFF
data tree, as the pre-tracked struct index must be updated recursively for
deletions of struct types.


Most users will find defining GFFFILEWRITER_PRETRACK_STRUCTS to zero to be the
best performing option.  This is the default that the library ships configured
with.


NWNScriptLib design notes:
--------------------------

The NWScriptVM object encapsulates a complete, compiled NWScript execution
environment.  In order to use the NWScriptVM, it is important to understnad
several design principles.

The script VM is single threaded, but reentrant.  (That is, multiple recursive
entries into the script VM are permitted.)  Each script called by the script VM
takes the form of a call to a script program (represents by a NWScriptReader
object).

Script execution nominally begins with a call to NWScriptVM::ExecuteScript,
which transfers control to the entry point symbol of the script program that
has been provided.  Normally, an entry point has the following prototype in
NWScript:

void main();
int StartingConditional();

The script VM supports both 'main' and 'StartingConditional' prototypes.  The
latter may return an integer value, whereas the former has no return value,
although the script VM will return zero to the caller if the script was of
the 'main' sort.  Generally, a programmatic call to a script already knows
a-priori whether it is a 'StartingConditional' (i.e. whether it returns a valid
return value).

Additionally, the script VM supports the NWN2-specific extension of NWScript
that allows parameterized script entry point symbols (i.e. entry point symbols
that are passed parameters by the programmatic caller of the core
NWScriptVM::ExecuteScript API).

Script parameters are always passed to ExecuteScript as string values, but they
will internally be dynamically converted to the appropriate type when
referenced by the script itself.

The following formatting conventions must be observed for script parameters
that are passed in to ExecuteScript:

- Strings are passed as-is.
- Integers are passed as base 10 signed integral values (i.e. formatted as %d).
- Floats are passed as base 10 floating point values (i.e. formatted as %g).
- Object IDs are passed as base 10 signed interval values (i.e. formatted as
  %d).

Engine structures cannot be passed as script parameters in the current version
of the script VM.

Unlike the default script VM in NWN2, the NWNScriptLib does not require the
user of a the script VM to compile parameterized scripts on each exeuction in
order to determine the types of arguments.  Dynamic typing is implemented for
script entry point parameters in order to eliminate this requirement.

An attempt to call a script entry point with a mismatched number of parameters
will result in an invalid type error within the script VM and will result in
the script call chain being terminated (with control returned to the user).


While the script execution environment already supports the intrinsic
mathematical operators supported by the compiled NWScript instruction set, most
useful work involves exposing customized functionality to the scripting
environment.

Custom functionality may be exposed through two different extension points,
both of which are fully supported by the script VM:

- A user of the script VM may expose functions to the script environment by
  implementing a series of "action handlers".  An action is a function that has
  been prototyped in the nwscript.nss file, and it is implemented in native
  code by the user of the script VM.  The user's action handler dispatcher is
  invoked (INWScriptActions::OnExecuteAction) when the script makes a call to
  any script action.

  It is up to the user of the NWNScriptLib to implement any and all desired
  script action handlers.  The first function prototype in nwscript.nss
  corresponds to action ordinal 0, the second correspond to action ordinal 1,
  and so on and soforth.

  The script VM itself is agnostic to the specifics of each action's
  implementation or prototype, as is the script compiler.  (It is possible to
  create a completely new action set for use in a custom program using a custom
  written nwscript.nss file with the script compiler, plus a custom written
  INWScriptActions interface implementation in conjuction with the NWScriptVM.)

  Action handlers are required to adhere to the following rules for argument
  and return value passing:

  1. All arguments are passed on the stack, with the first argument being the
     first parameter, and so on and so forth.  The action handler must remove
     all arguments from the stack before executing a return sequence.

     The exception to this rule involves script situation arguments, which are
     typed as "action" in the NWScript compiler.  These arguments are not
     passed on the stack, and there may be at most one per action handler call.

     The rules for action handlers are described in the documentation for
     "script situations", located below.

  2. If an action handler returns a value to the script environment, it must
     push the return value onto the script stack after all arguments have been
     removed from the stack.  Note that adding or removing a return value thus
     changes the fundamental calling convention of a particular action handler
     from the script compiler's perspective.

  Action handlers may access the script stack for these purposes by calling
  the NWScriptStack::StackPush* and NWScriptStack::StackPop* APIs with the
  NWScriptStack instance passed in to INWScriptActions::OnExecuteAction.


- Additionally, a user of the script VM may expose "engine structures" to the
  script environment.  An "engine structure" is simply a reference to an opaque
  structure that is meaningful only to engine action handlers (although the
  script environment may copy references, compare engine structures with the
  aid of EngineStructure::CompareEngineStructure, as well as pass engine
  structure references to action handlers).

  An engine structure is useful to expose a data item that the script
  environment may usefully store and then use later with a future call to an
  action handler.

  The INWScriptActions::CreateEngineStructure API, provided by the user of the
  script VM, creates a default-initialized engine structure of the given engine
  structure type.

  The script VM is entirely agnostic to the implementation of a particular type
  of engine structure.  There is support for up to 9 engine structure types to
  be defined and exposed to the script environment.


The script VM also supports the concept of a "script situation".  A script
situation, referred to as the "action" type in the NWScript compiler, is a copy
of the script VM's internal state that is snapshotted when a call to an engine
action handler that takes an "action"-typed parameter is made.

The script situation can be used to call back to a specific section of script
code at a later time, even after the script VM has returned control back to the
user.  Script situations are useful for packaging deferred calls to a
particular section of script code.  As an example, the NWN2 implementation of
NWScript has a "DelayCommand" engine action handler, which allows a section of
script code to be queued for later execution.

Script situations are not passed as standard arguments to action handlers.
Instead, the action handler must know a-priori that it expects to receive a
script situation (action) argument type.  Then, the action handler may retrieve
a reference to the last saved script situation with a call to the
NWScriptVM::GetSavedState API.

It is important to make a copy of the saved state object before control returns
or it may be overwritten later.  The saved state object maintains a copy of all
relevant data on the virtual machine stack (including references to the
script program code and any engine structures that were active).  By default,
if the saved state object is not copied, it will be deleted when the outermost
call to NWScriptVM::ExecuteScript returns control to the user.

The script VM allows a script situation to be activated later, effectively
calling back into the "middle" of a script.  This operation is performed by
invoking the NWScriptVM::ExecuteScriptSituation API.


Two manifest object id values may be associated with a particular execution of
a script entry point symbol:

- The invalid object id represents no object.  It is recommended that the value
  0x7F000000 be used for this parameter for compatibility purposes.  The script
  language refers to this constant by name of OBJECT_INVALID.

- The self object id represents the object whose context a script call is being
  executed in.  The script language refers to this constant by name of
  OBJECT_SELF.

The values of these constants is set by the user on each call to ExecuteScript.


A NWScript action handler may abort a recursive script by calling the
NWScriptVM::AbortScript function.  This terminates the entire script call chain
once control returns to the NWScriptVM.


If any script in a script call chain encounters an error, such as an illegal
stack reference, the script call chain is aborted as though AbortScript were
called by the user.


Verbose debug output is made to the user with the aid of the
NWScriptVM::SetDebugLevel API.  Debug output is routed through the
IDebugTextOut interface implementation that is provided by the user.

A static analysis system for NWScript is available in the NWScriptAnalyzer
component of the NWNScriptLib.  This class allows the compiled bytecode of a
NWScript program to be examined in order to glean more information about the
script program in a programmatic, static (offline) fashion.


---------------------------------------


- Ken Johnson (Skywing)

[email protected]