Development workflow

To generate the communication logic, the number and type of point-to-point and collective operations are required. This information is automatically extracted from the user code by the code generator contained in the repository. It takes the description of SMI operations (ports, data types) as an input and outputs a device source file with all the necessary cks, ckr, communication primitives and collective support kernel implementations that are tailored for the specified set of SMI operations. The code generator also outputs a host header file that contains support functions for SMI initialization.

Generating the description of used SMI operations from the user's source code manually would be time consuming and error-prone. Therefore, we also provide a metadata extractor. This tool parses the user's device code with Clang, finds all used SMI operations and extracts their metadata to a file that is then provided as an input for the code generator. The device file has to be included in bitstream compilation phase. For SPMD programs, only one instance of the communication code is generated, and thus the user only needs to build a single bitstream for any number of nodes in a multi-FPGA system.

A routes generator accepts the network topology of the FPGA cluster and produces the necessary routing tables to configure the forwarding logic at runtime. The topology must be provided as a JSON file. The user host program can rely on the function provided in the host header to load the routing tables, and to start all the SMI transport components on the FPGA. The network topology can be changed between program executions, as this is configured to the transport layer at runtime.

Please note: in the repository, we also provided build system integration for CMake which fully automates the described workflow with a single function invocation (refer to the repository README for the desciption of the diffent targets).

Code generation and topology file

Once that the user wrote her own device program, including the smi.h header file, the code generation steps can be executed.

The user has to pass to the code generator the device program and a JSON topology file: this file contains the connection topology (will be used to generate the first set of routing tables) and the mapping program <-> fpga. Since SMI supports also MPMD model, an application can be composed of different programs. Each of these programs could potentially require different transport component and different routing tables. For this reason, knowing this mapping is fundamental for generating correct routing information. This mapping is passed also to the code generation for the sake of generating correct emulation.

The format of the JSON file is the following: suppose that the application is composed of 4 different programs, that will be executed on 4 different FPGAs interconnected between each other.

All the information related to the program must be passed to the code generator using a JSON file, whose syntax is described in the following

{
    "fpgas": {
      "fpga-0001:acl0": "program_1",
      "fpga-0002:acl0": "program_2",
      "fpga-0003:acl0": "program_3",
      "fpga-0004:acl0": "program_4",
    },
    "connections": {
      "fpga-0001:acl0:ch2": "fpga-0002:acl0:ch3",
      "fpga-0002:acl0:ch1": "fpga-0003:acl0:ch0",
      "fpga-0003:acl0:ch1": "fpga-0004:acl0:ch0"
    }
}

The connection topology is described as a list of connections between two FPGA network ports according to the following format:

<nodename>:<devicename>:<channelname> 

where nodename indicates the hostname of the host in which the FPGA is installed, devicename indicates the FPGA name as visible by the Intel OpenCL environment, <channelname> indicates the name of the I/O channels. For example:

"fpga-0001:acl0:ch2": "fpga-0002:acl0:ch3"

indicates that the second I/O channel of the FPGA acl0 installed in the host fpga-0001 is connected to the third channel of the FPGA acl0 installed in the host fpga-0002.

As a result, the code generation creates two files:

• smi_generated_device.cl, that contains all the SMI communication components. This and the user device file must be passed to the compiler for generating the bitstream,;
• smi_generated_host.c, that contains the host SMI initialization function.

Host program

To execute an SMI-Based channel over Intel FPGA, the user has to write her own host program.

As a result of the code-generation phase, an host-side header file is generated. This provides a SMI_Init function, that can be invoked on the host side to startup all the SMI functionalities before running the application logic. The SMI_Init function returns the SMI_Comm that is used by SMI primitives.

Routing tables can be generated by using the provided generators, which takes in input the aforementioned JSON file. In this way, The network topology can be changed between program executions without the need of recompiling the bitstream, as this is configured to the transport layer at runtime.