libswoc assertion failures during 10.0.2 testing (-Wp,-D_GLIBCXX_ASSERTIONS)

[jeredfloyd]

I'm trying to build 10.0.2 with production build flags for Fedora/RHEL and running into libswoc-triggered assertion failures during testing. I think these are legitimate null-pointer dereferences in libswoc, but I'm having trouble understanding the purpose of the library as docs.solidwallofocode.com is offline so I'm hoping someone closer can help. (@SolidWallOfCode ?) This all looks very different from the 9.x branch.

I believe this can be reproduced by adding -Wp,-D_GLIBCXX_ASSERTIONS to CXXFLAGS. (Removing it from my build allows it to succeed.)

Test failure:

5/63 Test #1: test_libswoc ...........................Subprocess aborted***Exception: 0.15 sec
/usr/include/c++/14/optional:482: constexpr const _Tp& std::_Optional_base_impl<_Tp, _Dp>::_M_get() const [with _Tp = swoc::_1_5_12::Errata::Severity; _Dp = std::_Optional_base<swoc::_1_5_12::Errata::Severity, true, true>]: Assertion 'this->_M_is_engaged()' failed.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
test_libswoc is a Catch v2.13.8 host application.
Run with -? for options
-------------------------------------------------------------------------------
Errata copy
-------------------------------------------------------------------------------
/builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/unit_tests/test_Errata.cc:84
...............................................................................
/builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/unit_tests/test_Errata.cc:84: FAILED:
{Unknown expression after the reported line}
due to a fatal error condition:
SIGABRT - Abort (abnormal termination) signal

Backtrace:

/usr/include/c++/14/optional:482: constexpr const _Tp& std::_Optional_base_impl<_Tp, _Dp>::_M_get() const [with _Tp = swoc::_1_5_12::Errata::Severity; _Dp = std::_Optional_base<swoc::_1_5_12::Errata::Severity, true, true>]: Assertion 'this->_M_is_engaged()' failed.

Program received signal SIGABRT, Aborted.
0x00007ffff7a7fe64 in __pthread_kill_implementation () from /lib64/libc.so.6
Missing rpms, try: dnf --enablerepo='*debug*' install libstdc++-debuginfo-14.2.1-6.fc42.x86_64 glibc-debuginfo-[2.40.9000-26](callto:2.40.9000-26).fc42.x86_64 libgcc-debuginfo-14.2.1-6.fc42.x86_64
(gdb) bt
#0  0x00007ffff7a7fe64 in __pthread_kill_implementation () from /lib64/libc.so.6
#1  0x00007ffff7a2649e in raise () from /lib64/libc.so.6
#2  0x00007ffff7a0e397 in abort () from /lib64/libc.so.6
#3  0x00007ffff7c3e2f0 in std::__glibcxx_assert_fail(char const*, int, char const*, char const*) () from /lib64/libstdc++.so.6
#4  0x00007ffff7f93eb7 in std::_Optional_base_impl<swoc::_1_5_12::Errata::Severity, std::_Optional_base<swoc::_1_5_12::Errata::Severity, true, true> >::_M_get (this=0x55555580c0ca)
    at /usr/include/c++/14/optional:480
#5  std::_Optional_base_impl<swoc::_1_5_12::Errata::Severity, std::_Optional_base<swoc::_1_5_12::Errata::Severity, true, true> >::_M_get (this=0x55555580c0ca)
    at /usr/include/c++/14/optional:480
#6  std::optional<swoc::_1_5_12::Errata::Severity>::operator*() const & (this=0x55555580c0ca) at /usr/include/c++/14/optional:968
#7  swoc::_1_5_12::Errata::Annotation::severity (this=0x55555580c0b8) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/include/swoc/Errata.h:1002
#8  swoc::_1_5_12::bwformat (bw=..., errata=...) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/src/Errata.cc:155
#9  0x0000555555627a62 in swoc::_1_5_12::BufferWriter::print_nfv<swoc::_1_5_12::bwf::NameBinding const&, swoc::_1_5_12::bwf::Format::TextViewExtractor> (this=0x7fffffffd4e0, names=...,
    ex=..., args=...) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/include/swoc/bwf_base.h:898
#10 0x0000555555637593 in swoc::_1_5_12::BufferWriter::print_v<swoc::_1_5_12::Errata const&> (fmt=<synthetic pointer>..., this=0x7fffffffd4e0, args=std::tuple containing = {...})
    at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/include/swoc/bwf_base.h:612
#11 swoc::_1_5_12::FixedBufferWriter::print_v<swoc::_1_5_12::Errata const&> (this=0x7fffffffd4e0, fmt=..., args=std::tuple containing = {...})
    at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/include/swoc/bwf_base.h:1221
#12 swoc::_1_5_12::bwprint_v<swoc::_1_5_12::Errata const&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, swoc::_1_5_12::TextView, std::tuple<swoc::_1_5_12::Errata const&> const&)::{lambda()#1}::operator()() const (__closure=<synthetic pointer>)
    at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/include/swoc/bwf_base.h:1152
#13 swoc::_1_5_12::bwprint_v<swoc::_1_5_12::Errata const&> (s="", fmt=..., args=std::tuple containing = {...})
    at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/include/swoc/bwf_base.h:1153
#14 swoc::_1_5_12::bwprint<swoc::_1_5_12::Errata const&> (s="", fmt=...) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/include/swoc/bwf_base.h:1180
#15 operator() (__closure=<optimized out>, errata=...) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/unit_tests/test_Errata.cc:71
#16 std::__invoke_impl<void, test_Errata_init()::<lambda(const swoc::_1_5_12::Errata&)>&, const swoc::_1_5_12::Errata&> (__f=...) at /usr/include/c++/14/bits/invoke.h:61
#17 std::__invoke_r<void, test_Errata_init()::<lambda(const swoc::_1_5_12::Errata&)>&, const swoc::_1_5_12::Errata&> (__fn=...) at /usr/include/c++/14/bits/invoke.h:111
#18 std::_Function_handler<void(const swoc::_1_5_12::Errata&), test_Errata_init()::<lambda(const swoc::_1_5_12::Errata&)> >::_M_invoke(const std::_Any_data &, const swoc::_1_5_12::Errata &)
    (__functor=..., __args#0=...) at /usr/include/c++/14/bits/std_function.h:290
#19 0x00007ffff7f92fc1 in std::function<void(swoc::_1_5_12::Errata const&)>::operator() (this=<optimized out>, __args#0=...) at /usr/include/c++/14/bits/std_function.h:591
#20 swoc::_1_5_12::Errata::SinkWrapper::operator() (this=<optimized out>, e=...) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/include/swoc/Errata.h:1318
#21 swoc::_1_5_12::Errata::sink (this=0x7fffffffd5d0) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/src/Errata.cc:57
#22 0x00007ffff7f9305d in swoc::_1_5_12::Errata::~Errata (this=this@entry=0x7fffffffd5d0, __in_chrg=<optimized out>)
    at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/src/Errata.cc:50
#23 0x000055555562dfdc in C_A_T_C_H_T_E_S_T_0 () at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/unit_tests/test_Errata.cc:114
#24 0x000055555559c891 in Catch::TestInvokerAsFunction::invoke (this=<optimized out>) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:14328
#25 Catch::TestCase::invoke (this=<optimized out>) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:14167
#26 Catch::RunContext::invokeActiveTestCase (this=0x7fffffffdc90) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:13027
#27 0x00005555555bb257 in Catch::RunContext::runCurrentTest (this=this@entry=0x7fffffffdc90, redirectedCout="", redirectedCerr="")
    at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:13000
#28 0x00005555555bb738 in Catch::RunContext::runTest (this=this@entry=0x7fffffffdc90, testCase=...)
    at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:12761
#29 0x00005555555cc650 in Catch::(anonymous namespace)::TestGroup::execute (this=0x7fffffffdc80)
    at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:13354
#30 Catch::Session::runInternal (this=this@entry=0x7fffffffdf30) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:13560
#31 0x00005555555ccc5f in Catch::Session::run (this=this@entry=0x7fffffffdf30) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:13516
#32 0x000055555558f6a2 in Catch::Session::run<char> (this=0x7fffffffdf30, argc=1, argv=0x7fffffffe218)
    at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:13238
#33 Catch::Session::run<char> (this=0x7fffffffdf30, argc=1, argv=0x7fffffffe218) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/catch2/catch.hpp:13233
#34 main (argc=1, argv=0x7fffffffe218) at /builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/swoc/unit_tests/unit_test_main.cc:36

To me this looks like possibly bwformat() tries to deference an uninitialized note.severity (via an accessor function Annotation.severity()) without first checking validity, which triggers the enabled assertion. Calling the accessor when the value is not set is noted as undefined, so I think this is a legitimate error in bwformat().

gcc options for reference:
-DOPENSSL_API_COMPAT=10002 -DOPENSSL_IS_OPENSSL3 -DPACKAGE_NAME="\"Apache Traffic Server\"" -DPACKAGE_VERSION=\"10.0.2\" -Dlinux -Dyaml_cpp_EXPORTS -I/builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/include -I/builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/redhat-linux-build/include -I/builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/yamlcpp/include -I/builddir/build/BUILD/trafficserver-10.0.2-build/trafficserver-10.0.2/lib/yamlcpp/src -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -mtls-dialect=gnu2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -pthread -DNDEBUG -std=c++20 -fPIC -Wno-invalid-offsetof -Wno-shadow -MD -MT

[brbzull0]

Hi.

I'll try to get the docs online someplace else, but in the meantime, all the docs are available in the git repo, you can start from here:
https://github.com/apache/trafficserver-libswoc/blob/SolidWallOfCode-patch-1/doc/index.rst

This all looks very different from the 9.x branch.

IIRC there were some major changes from 9 to 10 in terms of this library, both the content and the usage inside ATS.

[cmcfarlen]

Thanks @jeredfloyd for the bug report. I confirmed this bug and made the fix noted above. The issue came down to the format function having:

      bw.print("{}{}{}{}", swoc::bwf::If(trailing_p, "{}", glue), swoc::bwf::Pattern{int(note.level()), id_txt},
               swoc::bwf::If(note.has_severity(), "{}{}", note.severity(), a_s_glue), note.text());

Where the bwf::If will evaluate its arguments regardless of the result of predicate. So this was a bit of a time bomb perhaps.