Error in pretty_stats with Vector in the GenericExecutionStats specifics

[tmigot]

using CUTEst, NLPModels, NLPModelsIpopt, SolverBenchmark, SolverCore
#This package
using DCISolver

nmax = 100
_pnames = CUTEst.select(
  max_var = nmax, 
  min_con = 1, 
  max_con = nmax, 
  only_free_var = true, 
  only_equ_con = true, 
  objtype = 3:6
)

#Remove all the problems ending by NE as Ipopt cannot handle them.
pnamesNE = _pnames[findall(x->occursin(r"NE\b", x), _pnames)]
pnames = setdiff(_pnames, pnamesNE)
cutest_problems = (CUTEstModel(p) for p in pnames)

#Same time limit for all the solvers
max_time = 1200. #20 minutes

solvers = Dict(
  :ipopt => nlp -> ipopt(
    nlp,
    print_level = 0,
    dual_inf_tol = Inf,
    constr_viol_tol = Inf,
    compl_inf_tol = Inf,
    acceptable_iter = 0,
    max_cpu_time = max_time,
    x0 = nlp.meta.x0,
  ),
  :dcildl => nlp -> dci(
    nlp,
    nlp.meta.x0,
    linear_solver = :ldlfact,
    max_time = max_time,
    max_iter = typemax(Int64),
    max_eval = typemax(Int64),
  ),
)
stats = bmark_solvers(solvers, cutest_problems)
pretty_stats(stats[:dcildl])

returns the following error.

ERROR: KeyError: key DenseVector{Float64} not found
Stacktrace:
 [1] getindex(h::Dict{DataType, String}, key::Type)
   @ Base ./dict.jl:482
 [2] pretty_stats(io::Base.TTY, df::DataFrames.DataFrame; col_formatters::Dict{DataType, String}, hdr_override::Dict{Symbol, String}, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ SolverBenchmark ~/.julia/packages/SolverBenchmark/eOKec/src/formats.jl:66
 [3] pretty_stats(io::Base.TTY, df::DataFrames.DataFrame)
   @ SolverBenchmark ~/.julia/packages/SolverBenchmark/eOKec/src/formats.jl:42
 [4] pretty_stats(df::DataFrames.DataFrame; kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ SolverBenchmark ~/.julia/packages/SolverBenchmark/eOKec/src/formats.jl:84
 [5] pretty_stats(df::DataFrames.DataFrame)
   @ SolverBenchmark ~/.julia/packages/SolverBenchmark/eOKec/src/formats.jl:84
 [6] top-level scope
   @ REPL[32]:1

My understanding is that we assume that there are no vectors in the output of the function bmark_solvers, while dci from DCISolver.jl has a vector in its stats.solver_specific. So, it looks as a bug in solve_problems function

[dpo]

I think it may be because there's no default formatter for vectors:

SolverBenchmark.jl/src/formats.jl

Line 3 in be8d2f1

const default_formatters = Dict(AbstractFloat => "%9.2e",

and then we get stuck here:

SolverBenchmark.jl/src/formats.jl

Line 66 in be8d2f1

push!(pt_formatters, ft_printf(default_formatters[typ == Symbol ? typ : styp], col))

The question is: what are you expecting if some solver specific quantity is a vector?

[tmigot]

I assume we skip these entries in the bmark_solvers process. For instance, right now the solution (which is also a vector) does not appear in the output.
That's why I am suggesting skipping these values in 'solve_problems`.

For the formatting in itself, pretty_table will print the vector anyway if there is one in the DataFrame, so I guess we have to complete the default_formatter in this case.