Extra/missing field in JSON fails even with {format, proplist}

[dmitriid]

Note: this very long issue contains three different examples, that's why it's so long :)

I'm working with Neo4J REST API. The JSON they return for some of the objects may contain optional fields.

Erlang R16B02

Example 1. Extra field

For example, this is JSON for a node which is not in an index:

{
  "extensions" : {
  },
  "paged_traverse" : "http://localhost:7474/db/data/node/376/paged/traverse/{returnType}{?pageSize,leaseTime}",
  "outgoing_relationships" : "http://localhost:7474/db/data/node/376/relationships/out",
  "traverse" : "http://localhost:7474/db/data/node/376/traverse/{returnType}",
  "all_typed_relationships" : "http://localhost:7474/db/data/node/376/relationships/all/{-list|&|types}",
  "property" : "http://localhost:7474/db/data/node/376/properties/{key}",
  "all_relationships" : "http://localhost:7474/db/data/node/376/relationships/all",
  "self" : "http://localhost:7474/db/data/node/376",
  "outgoing_typed_relationships" : "http://localhost:7474/db/data/node/376/relationships/out/{-list|&|types}",
  "properties" : "http://localhost:7474/db/data/node/376/properties",
  "incoming_relationships" : "http://localhost:7474/db/data/node/376/relationships/in",
  "incoming_typed_relationships" : "http://localhost:7474/db/data/node/376/relationships/in/{-list|&|types}",
  "create_relationship" : "http://localhost:7474/db/data/node/376/relationships",
  "data" : {
  }
}

And this is JSON for a node that has been added to an index:

{
  "extensions" : {
  },
  "paged_traverse" : "http://localhost:7474/db/data/node/8/paged/traverse/{returnType}{?pageSize,leaseTime}",
  "outgoing_relationships" : "http://localhost:7474/db/data/node/8/relationships/out",
  "traverse" : "http://localhost:7474/db/data/node/8/traverse/{returnType}",
  "all_typed_relationships" : "http://localhost:7474/db/data/node/8/relationships/all/{-list|&|types}",
  "property" : "http://localhost:7474/db/data/node/8/properties/{key}",
  "all_relationships" : "http://localhost:7474/db/data/node/8/relationships/all",
  "self" : "http://localhost:7474/db/data/node/8",
  "outgoing_typed_relationships" : "http://localhost:7474/db/data/node/8/relationships/out/{-list|&|types}",
  "properties" : "http://localhost:7474/db/data/node/8/properties",
  "incoming_relationships" : "http://localhost:7474/db/data/node/8/relationships/in",
  "incoming_typed_relationships" : "http://localhost:7474/db/data/node/8/relationships/in/{-list|&|types}",
  "create_relationship" : "http://localhost:7474/db/data/node/8/relationships",
  "data" : {
  },
  "indexed" : "http://localhost:7474/db/data/index/node/favorites/some-key/some%20value/8"
}

The only difference is the "indexed" field.

Given this record:

-record(neo4j_node, { extensions
                    , paged_traverse
                    , outgoing_relationships
                    , traverse
                    , all_typed_relationships
                    , property
                    , all_relationships
                    , self
                    , outgoing_typed_relationships
                    , properties
                    , incoming_relationships
                    , incoming_typed_relationships
                    , create_relationship
                    , data
                    , labels
                    }
      ).

and

> Decoder = jsonx:decoder( [{neo4j_node, record_info(fields, neo4j_node)}]
                         , [{format, proplist}]).

Decoding the first JSON will yield a correct record:

#neo4j_node{
    extensions = [],
    paged_traverse =
        <<"http://localhost:7474/db/data/node/189/paged/traverse/{returnType}{?pageSize,leaseTime}">>,
    outgoing_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/out">>,
    traverse =
        <<"http://localhost:7474/db/data/node/189/traverse/{returnType}">>,
    all_typed_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/all/{-list|&|types}">>,
    property =
        <<"http://localhost:7474/db/data/node/189/properties/{key}">>,
    all_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/all">>,
    self = <<"http://localhost:7474/db/data/node/189">>,
    outgoing_typed_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/out/{-list|&|types}">>,
    properties =
        <<"http://localhost:7474/db/data/node/189/properties">>,
    incoming_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/in">>,
    incoming_typed_relationships =
        <<"http://localhost:7474/db/data/node/189/relationships/in/{-list|&|types}">>,
    create_relationship =
        <<"http://localhost:7474/db/data/node/189/relationships">>,
    data = [{<<"born">>,1940},{<<"name">>,<<"Al Pacino">>}],
    labels =
        <<"http://localhost:7474/db/data/node/189/labels">>}

however, it will fail on the second JSON:

{error,invalid_json,1}

Even though I expected it to return a proplist (as it does for some other structures).

If I add "indexed" to record definition, decoder will fail with {error,invalid_json,1} for the first JSON and not for the second.

Example 2. Missing field

The service root may or may not contain a field called "reference_node":

{
  "extensions" : {
  },
  "node" : "http://localhost:7474/db/data/node",
  "reference_node" : "http://localhost:7474/db/data/node/371",
  "node_index" : "http://localhost:7474/db/data/index/node",
  "relationship_index" : "http://localhost:7474/db/data/index/relationship",
  "extensions_info" : "http://localhost:7474/db/data/ext",
  "relationship_types" : "http://localhost:7474/db/data/relationship/types",
  "batch" : "http://localhost:7474/db/data/batch",
  "cypher" : "http://localhost:7474/db/data/cypher",
  "neo4j_version" : "1.9.5"
}

Given this record definition:

-record(neo4j_root, { extensions
                    , node
                    , reference_node
                    , node_index
                    , relationship_index
                    , extensions_info
                    , relationship_types
                    , batch
                    , cypher
                    , transaction
                    , neo4j_version
                    }
       ).

and

Decoder = jsonx:decoder( [{neo4j_root, record_info(fields, neo4j_root)}]
                       , [{format, proplist}]).

this will fail with {error,invalid_json,1} if reference_node isn't in the JSON.

Example 3. Everything works as expected

For some other JSON this works as expected. JSON for indices can be either

{
  "template" : "http://localhost:7474/db/data/index/node/favorites/{key}/{value}"
}

or

{
  "template" : "http://localhost:7474/db/data/index/node/fulltext/{key}/{value}",
  "type" : "fulltext",
  "provider" : "lucene"
}

Given

-record(neo4j_index, { template
                     , type
                     , provider
                     }
       ).
...
Decoder = jsonx:decoder( [{neo4j_index, record_info(fields, neo4j_index)}]
                       , [{format, proplist}]).

the first JSON will be decoded as

[{<<"template">>, <<"http://localhost:7474/db/data/index/node/fulltext/{key}/{value}">>}]

and the second one as

#neo4j_index{template = <<"http://localhost:7474/db/data/index/node/aa/{key}/{value}">>,
               type = <<"exact">>,provider = <<"lucene">>}

That is, as expected.

I have no idea why this happens :)