Support partial variants

[antoinebhs]

Please check if the PR fulfills these requirements

• The commit message follows our guidelines
• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

Does this PR already have an issue describing the problem?
No

What kind of change does this PR introduce?
This PR adds support for partial variant clone.
In the following, we'll use these terms:

• Full variant: A variant that includes all required identifiables and external attributes (e.g., reactive capability curves, temporary limits, etc.).
• Full clone: A clone from a partial or full variant to a full variant.
• Partial variant: A variant that contains only the identifiables and external attributes whose values differ from those in the full variant (or created identifiables/external attributes). To be fully defined, a partial variant must be associated with a full variant, which provides the non-updated identifiables and external attributes. A partial variant cannot exist independently of a full variant.
• a partial clone is a clone from a partial or full variant to a partial variant.

Note: the concept of partial variant as it is implemented assumes (as before) that when a variant is modified, all the "child" variants are deleted and need to be rebuilt. We'll see that we may need to do something to prevent using the "Unbuild" on a full variant node.

What is the current behavior?
When cloning a network, all the identifiables and external attributes (reactive capability curves, temporary limits, etc.) are cloned. It leads to long clone time and duplicates unnecessary data in the server.

What is the new behavior (if this is a feature change)?
We add support for partial clone to only save on the server created/updated identifiables or external attributes in partial variants.

The new behavior of the different actions in partial variant is:

Clone

Full variant -> full variant: clone all the identifiables/external attributes of the source
Full variant -> partial variant: nothing is cloned
Partial variant -> partial variant: clone all the identifiables/external attributes of the source
Partial variant -> full variant: not implemented, see further work

All clones are partial. This means that there is no way to do a full clone anymore (see further work).

Identifiable/external attributes creations

Similarly to what was done before, we insert the resources in the corresponding variant.

Identifiable updates

We start by retrieving the IDs of existing identifiables in the variant. Next, we partition the identifiables into two: those that already exist in the variant and those that do not. For the identifiables that exist, we update as before. For those that do not exist, we create new identifiables in the variant.
(+1 select query compared with main)

External attributes updates

Similarly to what was done before for most external attributes (except regulating point that were homogenized), we delete the external attributes from the variant and reinsert them with updated value. If the updated value is empty, we insert a tombstoned for this external attributes. The tombstone is a way to make the difference between an external attributes that have not been updated in this variant and an external attributes that have been deleted or emptied (e.g for list). We cannot use updated identifiables ids in the variant as there are partial updates like updateSv that don't update external attributes.
(+1 select query/+1 insert query compared with main for tombstones)

Identifiable updates SV

We start by retrieving the IDs of existing identifiables in the variant. Next, we partition the identifiables into two: those that already exist in the variant and those that do not. For the identifiables that exist, we update SV as before. For those that do not exist, we fetch corresponding identifiables in the full variant, then update with SV values in memory, then insert updated SV identifiables.
We do that because the resources are not complete in the case of SV updates: it only contains SV attributes and we can't directly insert them.
(+1 select query compared with main)

Note: I noticed when testing that update SV do not seem to be batched on main branch (1 update SV = 1 update execution in the database).

Identifiable deletes

We remove the identifiable (and associated external attributes) from the variant and add a tombstone for this identifiable. The tombstone is added to ensure that we don't look for the identifiable in the full variant on reading identifiables or external attributes.
(+1 insert query compared with main for tombstones)

Identifiable/external attributes reads

First, we retrieve the resources from the full variant. Then, we remove the resources that are either tombstoned or updated in the partial variant from the full variant's list. Next, we fetch the updated resources and add them to the ones retrieved from the full variant.
(+2 select query compared with main: tombstones and updated identifiables)

Extensions

Extensions are a special type of external attributes as they can be lazy loaded in the identifiables (in the client cache). It means that we can never assume that an updated identifiable contains all the extensions (compared with what is stored on the server).
This needs some adjustments that are best described in the ExtensionHandler code. The main idea is that:

• we only update (delete/recreate) the extensions that are present in the updated identifiable. This is valid as updated identifiables don't contain deleted extensions as we can only remove extensions using the removeExtension() method that uses a separate endpoint.
• when removing an extension in the removeExtension endpoint, we add a tombstone for the corresponding extension
• for readings, it's almost the same as other external attributes except that there is a bit more "merge" logic because we manipulate maps

Tombstones

Tombstones are never removed as it means "don't look in full variant for this resource". There is one tombstone table for each external attributes and one for all identifiable types.

Performance

I did some performance tests to check performance gain/degradation. I compared the performances between:

• main branch
• this branch (diff-variant) with default clone strategy = partial (this is the actual implementation with partial clones)
• this branch (diff-variant) with default clone strategy = full (this will not be used for now, but this should be equivalent to the main implementation, there should be no overhead because of partial variant in this case as there are only full variant).
  These performance test were done once on each network so there can be some local effects. Some metrics are averaged though (e.g create 1000 lines is an average of each 1000 lines batch when creating a network).

On a Stanway network:

	main (ms)	Diff-variant with default strategy PARTIAL (ms)	Diff-variant with default strategy FULL (ms)
Create 1000 lines	953	847	899
Create 1000 generators	258	216	256
Initial clone	6841	4	6350
First updateSv 1000 lines (=> insert in diff-variant)	65	65	62
First updateSv 1000 generators (=> insert in diff-variant)	61	79	57
Get lines after loadflow	1639	1711	1547
Get generators after loadflow	73	114 all	77
UpdateSv 1000 lines	51	58	57
UpdateSv 1000 generators	45	58	56
First update 1000 lines (=> insert in diff-variant)	849	720	750
First update 1000 generators	1181 (why?)	104	1165 (why?)
Update 1000 lines	680	655	609
Update 1000 generators	1137 (why?)	1137 (why?)	1262 (why?)

On a voltage init network:

	main	Diff-variant with default strategy PARTIAL (ms)	Diff-variant with default strategy FULL (ms)
Clone after voltage-init + generator dispatch + loadflow	3420	545	2950
Get lines after voltage init + generator dispatch + loadflow	271	344	301
Get generators after voltage init + generator dispatch + loadflow	180	254	170

Clone performance are much better. There is apparently no significant performance degradation on other metrics.
When lot of identifiables are updated (e.g. after voltage init), clone performance are still better than before because we don't clone switch and other external attributes. For the case above, we clone 39000 identifiables + 45 000 external attributes. In the future, we may want to create a full variant first when there are that much rows in partial variant.

Note: On main branch, there is a strange performance when updating 1000 generators using the "Tabular modification" (see (why ?) in the above table). This can be investigated separately but we see that the performance are consistent with main on update on diff-variant branch.

Migration

Concerning the migration, I set all fullVariantNum of existing variants to -1 because they are all full variant.

Does this PR introduce a breaking change or deprecate an API?

• Yes
• No

Other information:
I had to do some refactos in some places to simplify the code with the new implementation even if I tried to not be intrusive in the existing code.

As we don't implement a way to have a full clone from a partial variant, all variants are partial except the initial variant (variantNum = 0). We can thus see this first implementation as a "diff from initial variant". See further work to see what's missing to have full clone from partial variant.

Prerequisite for this PR:

• invalidate all nodes because in the migration we set all fullVariantNum to -1. If one clones an existing node (say N3), it will rely on N2 for readings. As we have the possibility to unbuild N2, it will break everything. That's why we need to invalidate all nodes so that all variants are removed and there is only the initial variant that will be a full variant.

Further work should be done to:

• support full clone from partial variant (do we need it?)
  • reconstruct the full variant from a partial variant
  • change or implement some code in the network-modification-server to change its behavior. In particular, we can "Unbuild" any node (equivalent to delete variant in network-store). For example, if N3 is a full variant, N4 is a partial clone of N3. Then, N3 is unbuilt. As N4 depends on the full variant N3, it should be unbuilt too. Similarly to when we modify a node and the child nodes are unbuilt. Or we may want to remove this "Unbuild" feature. That's also why we need to invalidate nodes, to avoid to have full variant that are not the initial variant as this can lead to inconsistencies.
  • see if we want to add an heuristic, let users choose which network should be fully cloned, do it during some computations, etc.

[antoinebhs]

Note, to discuss ? : discussing with Geoffroy, he thinks that we should have tombstoned columns in each identifiable entity (for identifiable tombstone as well as external attributes tombstones) instead of separate tables.
With this, we could use a where clause to remove tombstoned resources in select. However, it means that we should clone the identifiable in the partial variant if we delete it and it does not exist in the partial variant. While with separate tables as now, we just add a row in the tombstone table, I forgot to mention this last point while discussing with him.

[antoinebhs]

Maybe we want to move the code relative to partial variant in ExtensionHandler to PartialVariantUtils to have all in the same place? It would be consistent with identifiables and other external attributes.

[FranckLecuyer]

Tests: OK

[antoinebhs]

Discussing with @jonenst, we decided that all clones should be partial in this first version so there is no need to have parameters to change the clone strategy or store the clone strategy in DB.
I will update the PR accordingly.

[jonenst]

avoid loading all tombstones just to check for one ?