Attribute Loader Architecture

General Architecture

Attribute Loader section handles the downloading, transformation (if required), and packaging of attribute data for a layer.

We will support ESRI Feature Layers and ESRI Dynamic Map Service Layers. TODO add more layers as support becomes available.

Function input parameters (names, order, and format TBD)

Layer Object

Dynamic Map Service Layers can have multiple constituent feature sets. To support this, we want to return a structure that can support multiple sets of attributes from sub-layers. Each attribute set is indexed by that layer's index in the service. For consistency, we use this structure even if a layer only has one layer (this also applies to a feature layer). For ease of use, we also tack on the layerId and an array of indexes to the main object.

Current implementation will be an object that inherits from ESRI's base Layer class.

Output will be a promise (verify) that returns a set of attribute data when fulfilled.

If a Feature Layer is passed in, the result will be a data set of attributes for that layer. If a Dynamic Map Service Layer is passed in, a group of data sets for each leaf layer will be returned. There should be an option (config driven?) to suppress a given leaf layer of a Dynamic Map Service. Other layer types will return an error until support is written.

Data should be fetched from REST endpoints, going in batches that do not exceed the service's maximum record count.

{
    layerId: <layerId for layer>,
    indexes: ["6", "7"],
    "6": {
        <instance of a layer package, see below>
    },
    "7": {
        <instance of a layer package, see below>
    }
}

Layer Package

Contains information about a single server-side layer. DO NOT access the ._attribData property directly, use the function .getAttribs() instead.

{
    "layerId": "<layerid>",
    "featureIdx": 3,
    "getAttribs": getAttribs(),
    "_attribData": Promise(
        <instance of a attribute data object, see below>
    ),
    "layerData":  Promise(
        <instance of a layer data object, see below>
    )
}

Attribute Data Structure

Contains the attribute data as an array, and an index mapping object id to array position

{
    "features": [
        {
            "attributes": {
                "objectid": 23,
                "name": "Bruce",
                "age": 27
            }
        },
        ...
    ],
        "oidIndex": {
        "23": 0,
        ...
    }
}

Layer Data Structure

Contains information describing the server-side layer

{
    "fields: [
        {
            "name": "objectid",
            "type": "esriFieldTypeOID",
            "alias": "OBJECTID"
        },
        ...
    ],
    "oidField": "objectid",
    "renderer": {...}
}

Structure for a layer's worth of features. It contains

• the attributes in an array. One 'feature' per array element. We use an array so that the datatables jquery plugin can access and filter on the data
• field definitions in an array. This is a copy of the fields array returned from ArcGIS Server. It contains names, types, and aliases.
• an object that maps the object id to the attribute in the above array. Used for fast lookups on individual features.
• the object id field name
• optional the layer id this set belongs to
• optional the layer index this set belongs to

Additional Parameters

The attribute loader function can also accept parameters to specify the attributes that are downloaded (default value is all attributes) and to skip certain layers (useful in large Dynamic Map Service Layers). The following object defines how these settings are passed to the function. All object/fields are optional; if any are not specified, the default value is used. If a Group Layer index is set to be skipped, all child layers of that group will be skipped, regardless if skip settings exist for those layers.

{
	"<layerindex a>": {
		"skip": true
	},
	"<layerindex b>": {
		"skip": false,
		"attribs": "field3,field8,field11"
	},
	"<layerindex d>": {               
	}
}

Optional Items

Optional Logic: It has been noticed that getting large datasets from map services can be a slow process (the example used was 25,000 features with 27 attributes, one being a large text field). Further research showed that if the attributes are put in a JSON file, the download & grid load is much quicker (in the same example, the difference was 1 minute vs 1 second). It would be a nice enhancement to support this as an alternative means to download attributes. A layer could have an optional config element specifying the JSON attribute file URL, and the attribute loader would target that if the config element exists. This approach would only work for pre-defined maps; layers added on the fly by users would likely not have the option to specify and additional file path for attributes.

Optional Logic: if a Feature Layer is in Snapshot mode and has it's attributes downloaded (as would be the case for file-based layers), an attempt to just extract the data from the layer should be made. Remember that a server-based layer in Snapshot mode may not have all the attributes downloaded if the feature count exceeds the service's max feature count. Also note that often a Feature Layer from the server may also only load object ID or minimal attribute sets, so the service definition may need to be compared to the current client side attribute set.

Optional Logic: implement a paging approach to handle massive data sets. Track what has been downloaded, what has not, and have an option to download a subset of attributes. Requires some more thinking, most likely.