[RNTuple] Writing: support String and Bool types, and Vector Field #356

Moelf · 2024-10-16T10:29:42Z

julia> newtable = (;
                      one_vuint = [[0xcececece, 0xcdcdcdcd], [0xabababab]],
                      two_uint = [0xabababab, 0xefefefef]
                  )
(one_vuint = Vector{UInt32}[[0xcececece, 0xcdcdcdcd], [0xabababab]], two_uint = UInt32[0xabababab, 0xefefefef])

julia> UnROOT.write_rntuple(open("/tmp/a.root", "w"), newtable; rntuple_name="myntuple")

julia> LazyTree("/tmp/a.root", "myntuple")
 Row │ one_vuint                 two_uint
     │ Vector{UInt32}            UInt32
─────┼──────────────────────────────────────
 1   │ [3469659854, 3452816845]  2880154539
 2   │ [2880154539]              4025479151

Investigate why nesting one more level of vector fails
Make ROOT C++ happy

Moelf · 2024-10-16T10:38:07Z

so the following is bad.

The field records should have parent_field_id 0, 0, 1 respectively, and for the column records, they should have field_id 0, 1, 2 respectively.

julia> newtable = (;
                      three_v_vuint = [[[0xcececece], [0xcdcdcdcd]], [[0xabababab]]]
                  )
(three_v_vuint = Vector{Vector{UInt32}}[[[0xcececece], [0xcdcdcdcd]], [[0xabababab]]],)

julia> frs, crs = UnROOT.schema_to_field_column_records(newtable);

julia> frs
3-element Vector{UnROOT.FieldRecord}:
 UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000000, struct_role=0x0001, flags=0x0000, repetition=0, source_field_id=-1, root_streamer_checksum=-1, field_name="three_v_vuint", type_name="", type_alias="", field_desc="", )

 UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000000, struct_role=0x0001, flags=0x0000, repetition=0, source_field_id=-1, root_streamer_checksum=-1, field_name="_0", type_name="", type_alias="", field_desc="", )

 UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000000, struct_role=0x0000, flags=0x0000, repetition=0, source_field_id=-1, root_streamer_checksum=-1, field_name="_0", type_name="std::uint32_t", type_alias="", field_desc="", )


julia> crs
3-element Vector{UnROOT.ColumnRecord}:
 UnROOT.ColumnRecord(type=0x0001, nbits=0x0040, field_id=0x00000000, flags=0x0000, representation_idx=0x0000, first_ele_idx=0, )

 UnROOT.ColumnRecord(type=0x0001, nbits=0x0040, field_id=0x00000000, flags=0x0000, representation_idx=0x0000, first_ele_idx=0, )

 UnROOT.ColumnRecord(type=0x000b, nbits=0x0020, field_id=0x00000001, flags=0x0000, representation_idx=0x0000, first_ele_idx=0, )

Reference layout from ROOT output

      ├─ :vector_vector_int32 ⇒ Vector
      │                         ├─ :offset ⇒ Leaf{UnROOT.Index64}(col=6)
      │                         └─ :content ⇒ Vector
      │                                       ├─ :offset ⇒ Leaf{UnROOT.Index64}(col=7)
      │                                       └─ :content ⇒ Leaf{Int32}(col=8)

 UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000005, struct_role=0x0001, flags=0x0000, repetition=0, source_field_id=-1, root_streamer_checksum=-1, field_name="vector_vector_int32", type_name="std::vector<std::vector<std::int32_t>>", type_alias="", field_desc="", )

 UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000005, struct_role=0x0001, flags=0x0000, repetition=0, source_field_id=-1, root_streamer_checksum=-1, field_name="_0", type_name="std::vector<std::int32_t>", type_alias="", field_desc="", )

 UnROOT.FieldRecord(field_version=0x00000000, type_version=0x00000000, parent_field_id=0x00000006, struct_role=0x0000, flags=0x0000, repetition=0, source_field_id=-1, root_streamer_checksum=-1, field_name="_0", type_name="std::int32_t", type_alias="", field_desc="", )

# ------------------

 UnROOT.ColumnRecord(type=0x000e, nbits=0x0040, field_id=0x00000005, flags=0x0000, representation_idx=0x0000, first_ele_idx=0, )

 UnROOT.ColumnRecord(type=0x000e, nbits=0x0040, field_id=0x00000006, flags=0x0000, representation_idx=0x0000, first_ele_idx=0, )

 UnROOT.ColumnRecord(type=0x001b, nbits=0x0020, field_id=0x00000007, flags=0x0000, representation_idx=0x0000, first_ele_idx=0, )

codecov · 2024-10-16T10:38:13Z

Codecov Report

Attention: Patch coverage is 93.87755% with 3 lines in your changes missing coverage. Please review.

Project coverage is 84.72%. Comparing base (4ee4732) to head (f7a6995).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
src/RNTuple/Writing/TFileWriter.jl	90.90%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #356      +/-   ##
==========================================
+ Coverage   84.65%   84.72%   +0.07%     
==========================================
  Files          21       21              
  Lines        3043     3084      +41     
==========================================
+ Hits         2576     2613      +37     
- Misses        467      471       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Moelf · 2024-10-16T12:47:38Z

julia> newtable = (;
                      one_vuint = [[0xcececece, 0xcdcdcdcd], [0xabababab]],
                      two_uint = [0xabababab, 0xcdcdcdcd],
                      three_v_vuint = [[[0xcececece], [0xcdcdcdcd]], [[0xabababab]]]
                  )
(one_vuint = Vector{UInt32}[[0xcececece, 0xcdcdcdcd], [0xabababab]], two_uint = UInt32[0xabababab, 0xcdcdcdcd], three_v_vuint = Vector{Vector{UInt32}}[[[0xcececece], [0xcdcdcdcd]], [[0xabababab]]])

julia> UnROOT.write_rntuple(open("/tmp/a.root", "w"), newtable; rntuple_name="myntuple")

julia> LazyTree("/tmp/a.root", "myntuple")
 Row │ one_vuint                 two_uint    three_v_vuint
     │ Vector{UInt32}            UInt32      Vector{Vector{U
─────┼────────────────────────────────────────────────────────────────────
 1   │ [3469659854, 3452816845]  2880154539  Vector{UInt32}[UInt32[0xcece
 2   │ [2880154539]              3452816845  Vector{UInt32}[UInt32[0xabab

Moelf · 2024-10-16T12:48:42Z

src/RNTuple/Writing/TFileWriter.jl

+    Element_T = eltype(input_T)
+    content_parent_field_id = Element_T <: Real ? implicit_field_id : parent_field_id
+    add_field_column_record!(field_records, column_records, Element_T, "_0"; parent_field_id = content_parent_field_id, col_field_id = length(field_records))


this is really not elegant and I feel like this would fail in more complicated case (i.e. not sure what happens when you have structs or 4-th order jagging)

Moelf · 2024-10-16T13:19:17Z

julia> newtable = (;
               three_v_vuint = [[[0xcececece], [0xcdcdcdcd]], [[0xabababab]]]
           );

julia> using UnROOT

julia> UnROOT.write_rntuple(open("/tmp/a.root", "w"), newtable; rntuple_name="myntuple")

>>> df  =  ROOT.RDataFrame("myntuple", "/tmp/a.root")
>>> df.GetColumnNames()
vector<string>{ "three_v_vuint._0._0" }
>>> df.Display("three_v_vuint._0._0").Print()
+-----+---------------------+
| Row | three_v_vuint._0._0 |
+-----+---------------------+
| 0   | { 3469659854 }      |
|     | { 3452816845 }      |
+-----+---------------------+
| 1   | { 2880154539 }      |
+-----+---------------------+

currently C++ ROOT is happy with the "base" case, but when we combine different types there's a checksum mismatch error

  static RResult<unsigned int> ROOT::Experimental::Internal::RNTupleSerializer::DeserializeEnvelope(const void*, uint64_t, uint16_t) [/build/jenkins/workspace/lcg_nightly_pipeline/build/projects/ROOT-HEAD/src/ROOT/HEAD/tree/ntuple/v7/src/RNTupleSerialize.cxx:902]
  static ROOT::Experimental::RResult<void> ROOT::Experimental::Internal::RNTupleSerializer::DeserializePageList

Moelf · 2024-10-16T13:26:46Z

fixed!

>>> import ROOT

>>> df  =  ROOT.RDataFrame("myntuple", "/tmp/a.root")

>>> df.GetColumnNames()
vector<string>{ "one_vuint._0", "three_v_vuint._0._0", "two_uint" }

>>> df.Display("two_uint").Print()
+-----+------------+
| Row | two_uint   |
+-----+------------+
| 0   | 2880154539 |
+-----+------------+
| 1   | 3452816845 |
+-----+------------+

>>> df.Display("one_vuint._0").Print()
+-----+--------------+
| Row | one_vuint._0 |
+-----+--------------+
| 0   | 3469659854   |
|     | 3452816845   |
+-----+--------------+
| 1   | 2880154539   |
+-----+--------------+

>>> df.Display("three_v_vuint._0._0").Print()
+-----+---------------------+
| Row | three_v_vuint._0._0 |
+-----+---------------------+
| 0   | { 3469659854 }      |
|     | { 3452816845 }      |
+-----+---------------------+
| 1   | { 2880154539 }      |
+-----+---------------------+

Moelf · 2024-10-17T11:15:29Z

ROOT is also happy with our Bool and String:

>>> df  =  ROOT.RDataFrame("myntuple", "/tmp/a.root")
>>> list(df.Take['bool']("two_bool"))
[True, False, True, True, False, False, True, False, True, False, False, True, True, False, False, True]

>>> df  =  ROOT.RDataFrame("myntuple", "/tmp/a.root")
>>> list(df.Take['std::string']("x"))
['abc', 'def']

Moelf · 2024-10-17T11:19:08Z

src/RNTuple/fieldcolumn_reading.jl

+    N_pad = 8 - mod1(length(bytes), 8)
+    append!(bytes, zeros(eltype(bytes), N_pad))


this was a bug in the reading logic all along, very happy to have caught it because we have writing-reading round trip test

add tests fix bit magic

more tests fix print everything in PyROOT add coverage add coverage

Moelf added 2 commits October 16, 2024 12:29

RNTuple write vector field

bb0d78f

add length check

61b3b20

Moelf commented Oct 16, 2024

View reviewed changes

Moelf force-pushed the rnt_vec_writing branch from 9bd2f17 to de5a0aa Compare October 16, 2024 13:02

Moelf requested a review from tamasgal October 16, 2024 13:27

Moelf changed the title ~~[RNTuple] write vector field~~ [RNTuple] Writing: support String and Bools types, support Vector Field Oct 17, 2024

Moelf changed the title ~~[RNTuple] Writing: support String and Bools types, support Vector Field~~ [RNTuple] Writing: support String and Bool types, and Vector Field Oct 17, 2024

Moelf commented Oct 17, 2024

View reviewed changes

Moelf added 4 commits October 18, 2024 01:02

fix vec<vec> case

b7da69c

add tests fix bit magic

support Bool

a8ef1e5

add String

f790c8e

more tests

f7a6995

more tests fix print everything in PyROOT add coverage add coverage

Moelf force-pushed the rnt_vec_writing branch from 7a7b632 to f7a6995 Compare October 17, 2024 23:03

Moelf merged commit 0724c14 into main Oct 17, 2024
8 checks passed

Moelf deleted the rnt_vec_writing branch October 17, 2024 23:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RNTuple] Writing: support String and Bool types, and Vector Field #356

[RNTuple] Writing: support String and Bool types, and Vector Field #356

Moelf commented Oct 16, 2024 •

edited

Loading

Moelf commented Oct 16, 2024 •

edited

Loading

codecov bot commented Oct 16, 2024 •

edited

Loading

Moelf commented Oct 16, 2024

Moelf Oct 16, 2024

Moelf commented Oct 16, 2024 •

edited

Loading

Moelf commented Oct 16, 2024 •

edited

Loading

Moelf commented Oct 17, 2024

Moelf Oct 17, 2024

		N_pad = 8 - mod1(length(bytes), 8)
		append!(bytes, zeros(eltype(bytes), N_pad))

[RNTuple] Writing: support String and Bool types, and Vector Field #356

[RNTuple] Writing: support String and Bool types, and Vector Field #356

Conversation

Moelf commented Oct 16, 2024 • edited Loading

Moelf commented Oct 16, 2024 • edited Loading

Reference layout from ROOT output

codecov bot commented Oct 16, 2024 • edited Loading

Codecov Report

Moelf commented Oct 16, 2024

Moelf Oct 16, 2024

Choose a reason for hiding this comment

Moelf commented Oct 16, 2024 • edited Loading

Moelf commented Oct 16, 2024 • edited Loading

Moelf commented Oct 17, 2024

Moelf Oct 17, 2024

Choose a reason for hiding this comment

Moelf commented Oct 16, 2024 •

edited

Loading

Moelf commented Oct 16, 2024 •

edited

Loading

codecov bot commented Oct 16, 2024 •

edited

Loading

Moelf commented Oct 16, 2024 •

edited

Loading

Moelf commented Oct 16, 2024 •

edited

Loading