Issue manually implementing the DbUserValue trait

[teodosin]

Hey! I have a question regarding a manual implementation of DbUserValue. Made this as a discussion instead of an issue because I'm not certain if this is an actual problem with your library and not just a misunderstanding on my part. Hoping to clear this up.

I've got a Node struct with the following definition:

pub struct Node {
    db_id: Option<DbId>,
    path: NodePath,
    ntype: NodeType,
    nphys: NodePhysicality,

    created_time: SysTime,
    modified_time: SysTime,

    attributes: Vec<Attribute>,
}

Where Attribute is just a struct with a name and value, just like a DbKeyValue. The vector of attributes is a new addition before which there were no issues with using the UserValue macro. Since I couldn't find a way to impl From<Vec<DbKeyValue>> for Vec<Attribute> (if there is a way, do let me know), I tried to manually implement the trait to get all my attributes transferred, and here's where my confusion lies.

The trait definition is odd to me:

pub trait DbUserValue: Sized {
    /// Returns the database id if present.
    fn db_id(&self) -> Option<QueryId>;

    /// Returns the list of database keys of
    /// this type.
    fn db_keys() -> Vec<DbValue>;

    /// Constructs the user value from the `element`
    /// extracting the values from element `values`
    /// and the `db_id` if that field is present.
    fn from_db_element(element: &DbElement) -> Result<Self, DbError>;

    /// Converts the fields (skipping `db_id` if present)
    /// to the database key-values.
    fn to_db_values(&self) -> Vec<DbKeyValue>;
}

Why does db_keys not have access to self? Am I misunderstanding its purpose? I have no way of providing the function with the keys from the attributes vector.

[michaelvlach]

Hi, let me first explain how it currently works. The default implementation of the DbUserValue iterates over the fields of the struct skipping db_id. Every field name becomes a "key" of type "string" and every value must be convertible to DbValue (implementing Into<DbValue>). This means that while there is a support for custom field types the resulting key-value pairs in the database will always be those from the top-level type and must be statically known. This is because when you request your type through a query it selects only properties returned by DbUserValue::db_keys() at which point there is no self (only Self).

Now there are two issues:

1. Support for vectorized custom types. As you point out currently it is not possible to derive from UserValue if you have a type Vec<Attribute> even if Attribute derives from UserValue. I have opened [db] Support vectorized custom types in UserValue derive macro #1138 for it.

2. Nested / objects with dynamic structure. It was somewhat deliberate choice not to support them and forcing the flat and statically known set of properties (hence the DbUserValue::db_keys()). However I have considered an option that if the returned list was empty all properties of an element would be returned hence you could implement what you want by deriving from the UserValue or expanding the attributes in manual implementation. I have opened [db] Select all values if keys are empty #1139 for it. I will probably still keep the default (derived via macro) implementation using the statically known list of properties and flat structure although I suppose it would be somehow possible to implement a recursive decent. Important caveat here is that the resulting properties would still be flattened which might lead to name clashes that both derived and manual implementation need to handle.

Consider:

let node = Node { attributes: vec![Attribute { name: "path".into(), value: "/some/path".into() }], ..default()};

There already is path property on the Node so if attributes was dynamically expanded it would clash or overwrite that value. The derive macro could deal with that by composing the name of such propert, e.g. attributes_path. Still it is not as straightforward as it might seem. Even more importantly having dynamic schema like that might lead to surprising results and having things statically typed is usually preferable. Still I will make it possible or more convenient at least.

Even without the above issues resolved you can currently do this to store your struct in the db. I am omitting the other fields focusing on the attributes only. You can use any other way to store the Attribute I just used string conversion with : as a delimiter. I know this might not be ideal but as said currently the support for nested / dynamic structrues is not really there:

struct Attribute {
    key: String,
    value: String,
}

struct Node {
    db_id: Option<DbId>,
    // path: NodePath,
    // ntype: NodeType,
    // nphys: NodePhysicality,
    // created_time: SysTime,
    // modified_time: SysTime,
    attributes: Vec<Attribute>,
}

impl DbUserValue for Node {
    fn db_id(&self) -> Option<QueryId> {
        self.db_id.map(QueryId::Id)
    }

    fn db_keys() -> Vec<DbValue> {
        vec!["attributes".into()]
    }

    fn from_db_element(element: &DbElement) -> std::result::Result<Self, DbError> {
        let attributes = element.values[0]
            .value
            .vec_string()?
            .iter()
            .map(|a| {
                if let Some(kv) = a.split_once(':') {
                    Ok(Attribute {
                        key: kv.0.to_string(),
                        value: kv.1.to_string(),
                    })
                } else {
                    Err(DbError::from("bad value"))
                }
            })
            .collect::<Result<Vec<Attribute>, DbError>>()?;

        Ok(Self {
            db_id: Some(element.id),
            attributes,
        })
    }

    fn to_db_values(&self) -> Vec<agdb::DbKeyValue> {
        vec![(
            "attributes",
            self.attributes
                .iter()
                .map(|a| format!("{}:{}", a.key, a.value))
                .collect::<Vec<String>>(),
        )
            .into()]
    }
}

[teodosin]

Thank you for the thorough reply! I appreciate that you're considering/planning for vectorized custom types now.

Regarding #2, for my application it's thankfully not an issue. I use a wrapper function for inserting attributes and have an array of reserved attribute names that are blocked there. For now my manual impl just omits the attribute vector in db_keys(). Seems to work fine, since I'm really only ever inserting Nodes with empty attributes vectors and adding to them later. I presume db_keys() and to_db_values() are only used when converting the struct to how its represented in the database.

[agnesoft]

@teodosin Hi, both issues I opened for this are now fixed and they should simplify supporting your use case. With #1139 if you do not specify any keys the db will return all keys. With #1138 you can now use custom types in vectors as long as they implement Into<DbValue> and TryFrom<DbValue>. However your original use case of dynamic values is still not automatically supported with the derive macro. I have tried to come up with something but unfortunately it is impossible to do it generically enough. Therefore both dynamic values and nested / recursive structures still need to be supported by manually implementing DbUserValue and deciding there what to do. Still the above two items should help with that!

Please let me know if you issues with the implementation of if you have other stuff you would wish to be supported in agdb.

[teodosin]

Amazing! I'll take a look, cheers!