Standardize CVR identification across audit logs and RCTab output #913

nurse-the-code · 2025-01-09T00:29:37Z

Key Changes:

Fixes getId method to use logic that prints "RCTab CVR Id" to audit log.
- Uses computedId as primary identifier with fallback to suppliedId
Added getComputeId method to CVR to print computeId to rctab_cvr.csv as `"Vendor Id"
Updates both audit logs and rctab_cvr.csv to use the getId method
Changes Dominion computedId delimiter from pipe (|) to dash (-) for improved readability
Adds column "RCTab CVR Id" to RCTab CVR output

Benefits:

Same ballot maintains the same identifier across multiple tabulations by using centralized getId logic
Easier to track specific ballots across audit logs and RCTab CVR output
More intuitive ID format using dashes instead of pipes
Moving toward consistent identification approach throughout the application

Known Issues:

See TODOs in code asking if we can to use the updated getId method, the getComputedId, or something else.

Testing Needed:

Verify ID consistency:
- ✅ Compare IDs in audit log with rctab_cvr.csv output
- ✅ Ensure id returned from getId method is consistent across multiple tabulations
- ✅ Check ID format follows tabulator-batch-record pattern for records from Dominion CVR sources
Test fallback behavior:
- Cases where computedId is null
- Cases where computedId is empty or whitespace
- Cases where only suppliedId is available
Test with each CVR provider
- ✅ CDF/Unisyn
- ✅ Clear Ballot
- ✅ Dominion (tested with 2024 Alaska US House race)
- ✅ ES&S (in test suite)
- ✅ Hart
- ✅ generic CSV
Review and update failing tests:
- ✅ Done for now

Feature Request: Add Standardized Unique Identifiers for Cast Vote Records #911

Replace UUID-based identification with a more consistent ID system that uses the same identifier across audit logs and RCTab CVR output. The ID is primarily derived from the computedId (tabulator-batch-record) with fallback to suppliedId when computedId is unavailable. Unlike UUIDs which change between tabulations, this approach ensures the same ballot maintains the same identifier across multiple tabulations, making it easier to track specific ballots across different runs. Changes: - Remove UUID generation for CVR identification - Add getPrimaryId() method to centralize ID logic - Update audit log and RCTab CVR to use the same ID format - Change ID delimiter from pipe to dash for better readability - Add TODO to review legacy getId() usage Note: This change ensures that CVR IDs are consistent between audit logs, RCTab CVR output, and repeated tabulations, making it easier to track individual ballots across different output formats and multiple runs.

nurse-the-code · 2025-01-09T00:48:01Z

src/main/java/network/brightspots/rcv/CastVoteRecord.java

+  String getPrimaryId() {
+    return !isNullOrBlank(computedId) ? computedId : suppliedId;
+  }


This is a refactoring of the computedId and suppliedId logic used below in the logStringBuilder (which should print a unique CVR Id to the audit log.

nurse-the-code · 2025-01-09T00:50:52Z

src/main/java/network/brightspots/rcv/CastVoteRecord.java

  String getId() {
    return suppliedId != null ? suppliedId : computedId;
  }


This is the logic that is used to generate the "Record Id" that gets printed out in the RCTab CVR. It is not clear to me if this was intended to be a unique CVR identifier, because it uses the same computedId and suppliedId that my getPrimaryId uses.

But this value returned by getId() is not unique to every CVR (tested on 2024 Alaska US House race).

Additionally, we should research if we still need to print the "Record Id" to the RCTab CVR (since it is not unique).

@yezr thinks we don't need this any more.

Either fix this with the new logic or remove this method.

Need to fix/change this line:
csvPrinter.print(castVoteRecord.getId());
in OutputWriter.java if we refactor, remove or change this method. Currently csvPrinter.print(castVoteRecord.getId()); prints the recordId (at least in Dominion elections).

Move the code in here to a getSuppliedId method and rename the "Record ID" header in the RCTab CVR to "Vendor Id"

…ture/issue-911-rctab-cvr-primary-id

nurse-the-code · 2025-01-15T16:09:08Z

src/main/java/network/brightspots/rcv/CastVoteRecord.java

    if (outcomeType == VoteOutcomeType.IGNORED) {
-      logStringBuilder.append(" [was ignored] ");
+      logStringBuilder.append(" [ was ignored] ");


remove this change

nurse-the-code · 2025-01-15T16:11:33Z

src/main/java/network/brightspots/rcv/OutputWriter.java

-      // rank 2 selection, ... rank maxRanks selection
+      // RCTab CVR Id, ContestId, TabulatorId, BatchId, RecordId, Precinct, Precinct Portion,
+      // rank 1 selection, rank 2 selection, ... rank maxRanks selection
+      csvPrinter.print("RCTab CVR Id");


move this down a couple lines to be after the "Contest Id"

nurse-the-code · 2025-01-15T16:11:57Z

src/main/java/network/brightspots/rcv/OutputWriter.java

@@ -737,6 +738,7 @@ String writeRcTabCvrCsv(
        }

        CastVoteRecord castVoteRecord = castVoteRecords.get(i);
+        csvPrinter.print(castVoteRecord.getPrimaryId());


move this down a couple lines to be after the "Contest Id"

Dominion"

nurse-the-code · 2025-01-22T10:49:06Z

src/main/java/network/brightspots/rcv/OutputWriter.java

@@ -981,6 +984,7 @@ private Map<String, Object> generateCvrSnapshotMap(
            entry("@type", "CVR.CVRContest"));

    return Map.ofEntries(
+        // TODO: Do we want this to use the getSuppliedId or the getId method?
        entry("@id", generateCvrSnapshotId(sanitizeStringForOutput(cvr.getId()), round)),


Our choice here should be the same as in the TODO comment added before the sanitizedId definition, so long as we are offering a unique CVR id to include in @id.

nurse-the-code · 2025-01-22T11:03:19Z

src/main/java/network/brightspots/rcv/OutputWriter.java

@@ -875,6 +877,7 @@ private List<Map<String, Object>> generateCdfMapForCvrs(List<CastVoteRecord> cas
    for (CastVoteRecord cvr : castVoteRecords) {
      List<Map<String, Object>> cvrSnapshots = new LinkedList<>();
      cvrSnapshots.add(generateCvrSnapshotMap(cvr, null, null));
+      // TODO: Do we want this to use the getSuppliedId or the getId method?


In researching this question, I looked at the NIST Cast Vote Records Common Data Format Specification v 1.0 document. Here are some relevant parts I found.

on page 18:

on page 31:

This prints out the BallotPrePrintedId from the CDF's CVR class. It would seem to me that this should be a vendor-supplied id (e.g. suppliedId). I am not sure that we were using this correctly prior to this PR, because we would sometimes print the computedId (which was never pre-printed on any ballot). I am also not sure what this should look like if there is no vender-supplied id.

nurse-the-code · 2025-01-22T12:09:30Z

...ces/network/brightspots/rcv/test_data/conversions_from_ess/conversions_from_ess_expected.csv

-Source Filepath,CVR Provider,Contest Id,Tabulator Id,Batch Id,Record Id,Precinct,Precinct Portion,Rank 1,Rank 2,Rank 3
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-1,,,Mookie Blaylock,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-2,,,Mookie Blaylock,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-3,,,Mookie Blaylock,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-4,,,Mookie Blaylock,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-5,,,Yinka Dare,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-6,,,Yinka Dare,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-7,,,Yinka Dare,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-8,,,George Gervin,undervote,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-9,,,George Gervin,Yinka Dare,undervote
-../_shared/simple_ess_cvr.xlsx,ess,,,,simple_ess_cvr.xlsx-10,,,Sedale Threatt,George Gervin,undervote
+Source Filepath,CVR Provider,Contest Id,RCTab CVR Id,Tabulator Id,Batch Id,Vendor Id,Precinct,Precinct Portion,Rank 1,Rank 2,Rank 3
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-1,,,,,,Mookie Blaylock,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-2,,,,,,Mookie Blaylock,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-3,,,,,,Mookie Blaylock,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-4,,,,,,Mookie Blaylock,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-5,,,,,,Yinka Dare,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-6,,,,,,Yinka Dare,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-7,,,,,,Yinka Dare,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-8,,,,,,George Gervin,undervote,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-9,,,,,,George Gervin,Yinka Dare,undervote
+../_shared/simple_ess_cvr.xlsx,ess,,simple_ess_cvr.xlsx-10,,,,,,Sedale Threatt,George Gervin,undervote


For ES&S CVRs, it looks like our new getSuppliedId method (slightly changed from the original getId method results in no "Vendor Id" (formerly "Record Id") getting printed to the rctab_cvr.csv. Instead, that same value is printed as the "RCTab CVR Id" (using the new getId method).

This is because the old getId method (and the new getId method as well) returns a computedIdfor ES&S CVRs. The newgetSuppliedIdreturns an empty string, because ES&S CVRs don't havesuppliedId` associated with them in RCTab.

nurse-the-code added 2 commits January 8, 2025 02:47

Experimental implementation of GitHub issue 991:

7e7e29b

Feature Request: Add Standardized Unique Identifiers for Cast Vote Records #911

nurse-the-code mentioned this pull request Jan 9, 2025

Add Standardized Unique Identifiers for Cast Vote Records #912

Closed

nurse-the-code linked an issue Jan 9, 2025 that may be closed by this pull request

Feature Request: Add common id to both audit log and rctab_cvr.csv #911

Open

nurse-the-code commented Jan 9, 2025

View reviewed changes

Removed depricated reference to UUID.

3dd2f90

nurse-the-code mentioned this pull request Jan 13, 2025

Filter out dominion partial ballots #910

Merged

nurse-the-code added 2 commits January 15, 2025 01:51

Merge branch 'develop' of https://github.com/BrightSpots/rcv into fea…

dde3587

…ture/issue-911-rctab-cvr-primary-id

Fixed all tests except "Test Convert to rctab_cvr works for Dominion"

d967d53

nurse-the-code commented Jan 15, 2025

View reviewed changes

nurse-the-code added 3 commits January 15, 2025 11:41

Fixed broken TabulatorTest "Test Convert to rctab_cvr works for

276a79c

Dominion"

Updates per GitHub comments

dccb737

fixed broken tests

16941f7

nurse-the-code commented Jan 22, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standardize CVR identification across audit logs and RCTab output #913

Standardize CVR identification across audit logs and RCTab output #913

nurse-the-code commented Jan 9, 2025 •

edited

Loading

nurse-the-code Jan 9, 2025

nurse-the-code Jan 9, 2025 •

edited

Loading

nurse-the-code Jan 15, 2025

nurse-the-code Jan 15, 2025

nurse-the-code Jan 15, 2025 •

edited

Loading

nurse-the-code Jan 15, 2025 •

edited

Loading

nurse-the-code Jan 15, 2025

nurse-the-code Jan 15, 2025

nurse-the-code Jan 15, 2025

nurse-the-code Jan 22, 2025 •

edited

Loading

nurse-the-code Jan 22, 2025 •

edited

Loading

nurse-the-code Jan 22, 2025

Standardize CVR identification across audit logs and RCTab output #913

Are you sure you want to change the base?

Standardize CVR identification across audit logs and RCTab output #913

Conversation

nurse-the-code commented Jan 9, 2025 • edited Loading

nurse-the-code Jan 9, 2025

Choose a reason for hiding this comment

nurse-the-code Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

nurse-the-code Jan 15, 2025

Choose a reason for hiding this comment

nurse-the-code Jan 15, 2025

Choose a reason for hiding this comment

nurse-the-code Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

nurse-the-code Jan 15, 2025 • edited Loading

Choose a reason for hiding this comment

nurse-the-code Jan 15, 2025

Choose a reason for hiding this comment

nurse-the-code Jan 15, 2025

Choose a reason for hiding this comment

nurse-the-code Jan 15, 2025

Choose a reason for hiding this comment

nurse-the-code Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

nurse-the-code Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

nurse-the-code Jan 22, 2025

Choose a reason for hiding this comment

nurse-the-code commented Jan 9, 2025 •

edited

Loading

nurse-the-code Jan 9, 2025 •

edited

Loading

nurse-the-code Jan 15, 2025 •

edited

Loading

nurse-the-code Jan 15, 2025 •

edited

Loading

nurse-the-code Jan 22, 2025 •

edited

Loading

nurse-the-code Jan 22, 2025 •

edited

Loading