[Bug](PHDR) BE may deadlock during startup when jemalloc profiling unwinds through PHDRCache dl_iterate_phdr interposer

[Userwhite]

Search before asking

• I had searched in the issues and found no similar issues.

Version

master

What's Wrong?

Doris BE may hang during startup before the service becomes ready.

The observed hang happens very early during process initialization. GDB shows only
one thread, blocked in jemalloc initialization. The stack indicates a recursive
jemalloc initialization path:

1. libjvm.so static initialization performs allocation.
2. jemalloc enters malloc_init_hard() and holds jemalloc init mutex.
3. jemalloc profiling initialization calls _Unwind_Backtrace().
4. libunwind calls dl_iterate_phdr().
5. Doris' PHDRCache interposes the process-wide dl_iterate_phdr.
6. Because use_phdr_cache == false, the interposer calls
   getOriginalDLIteratePHDR().
7. getOriginalDLIteratePHDR() calls dlsym(RTLD_NEXT, "dl_iterate_phdr").
8. dlsym() internally allocates memory and re-enters jemalloc initialization
   while the jemalloc init mutex is already held.

This causes a startup deadlock.

Relevant Doris code:

// be/src/common/phdr_cache.cpp
DLIterateFunction getOriginalDLIteratePHDR() {
    void* func = dlsym(RTLD_NEXT, "dl_iterate_phdr");
    if (!func) {
        throw std::runtime_error("Cannot find dl_iterate_phdr function with
dlsym");
    }
    return reinterpret_cast<DLIterateFunction>(func);
}

extern "C" int dl_iterate_phdr(
        int (*callback)(dl_phdr_info* info, size_t size, void* data), void* data)
{
    if (!use_phdr_cache) {
        return getOriginalDLIteratePHDR()(callback, data);
    }

    return iteratePHDRCache(callback, data, 0);
}

This seems inconsistent with the comment in be/src/common/phdr_cache.h, which says
normal code paths should keep using the original glibc dl_iterate_phdr, while
Doris-patched libunwind should use the narrower doris_unwind_iterate_phdr hook.

Full Stack

  #0  0x00007f2a73ff029c in __lll_lock_wait () from /lib/x86_64-linux-gnu/
  libpthread.so.0
  #1  0x00007f2a73fe9714 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/
  libpthread.so.0
  #2  0x000055b05f66f0c5 in malloc_mutex_lock_final (mutex=0x55b064f4bb50
  <init_lock>)
      at ../include/jemalloc/internal/mutex.h:151
  #3  je_malloc_mutex_lock_slow (mutex=0x55b064f4bb50 <init_lock>) at ../src/
  mutex.c:90
  #4  0x000055b05f639140 in malloc_mutex_lock (tsdn=0x0, mutex=<optimized out>)
      at ../include/jemalloc/internal/mutex.h:217
  #5  malloc_init_hard () at ../src/jemalloc.c:2118
  #6  0x000055b05f629cd7 in malloc_init () at ../src/jemalloc.c:298
  #7  imalloc_init_check (sopts=<optimized out>, dopts=<optimized out>) at ../src/
  jemalloc.c:2658
  #8  imalloc (sopts=<optimized out>, dopts=<optimized out>) at ../src/
  jemalloc.c:2689
  #9  jecalloc (num=1, size=32) at ../src/jemalloc.c:2852
  #10 0x00007f2a74001a45 in ?? () from /lib/x86_64-linux-gnu/libdl.so.2
  #11 0x00007f2a7400140f in dlsym () from /lib/x86_64-linux-gnu/libdl.so.2
  #12 0x000055b057940afc in (anonymous namespace)::getOriginalDLIteratePHDR ()
      at ./be/build_Release/./be/src/common/phdr_cache.cpp:86
  #13 dl_iterate_phdr (callback=0x55b063fe9420 <_Unwind_IteratePhdrCallback>,
  data=0x7ffdc7c0cd50)
      at ./be/build_Release/./be/src/common/phdr_cache.cpp:238
  #14 0x000055b063febebe in _Unwind_Find_FDE ()
  #15 0x000055b063fe7ab7 in uw_frame_state_for ()
  #16 0x000055b063fe810d in uw_init_context_1 ()
  #17 0x000055b063fe920c in _Unwind_Backtrace ()
  #18 0x000055b05f673888 in je_prof_boot2 (tsd=tsd@entry=0x7f2a73e087c0,
  base=<optimized out>)
      at ../src/prof.c:693
  #19 0x000055b05f639591 in malloc_init_hard () at ../src/jemalloc.c:2151
  #20 0x000055b05f62718f in malloc_init () at ../src/jemalloc.c:298
  #21 imalloc_init_check (sopts=<optimized out>, dopts=<optimized out>) at ../src/
  jemalloc.c:2658
  #22 imalloc (sopts=<optimized out>, dopts=<optimized out>) at ../src/
  jemalloc.c:2689
  #23 je_malloc_default (size=40) at ../src/jemalloc.c:2722
  #24 0x00007f2a74d8e7a5 in NMTPreInitAllocation::do_alloc(unsigned long) ()
      from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
  #25 0x00007f2a74dba390 in os::malloc(unsigned long, MEMFLAGS, NativeCallStack
  const&) ()
      from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
  #26 0x00007f2a7457409a in AllocateHeap(unsigned long, MEMFLAGS,
  AllocFailStrategy::AllocFailEnum) ()
      from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
  #27 0x00007f2a74c47876 in LogOutputList::add_output(LogOutput*, LogLevel::type) ()
      from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
  #28 0x00007f2a74400a3f in _GLOBAL__sub_I_abstractDisassembler.cpp ()
      from /opt/tiger/jdk/jdk17/lib/server/libjvm.so
  #29 0x00007f2a754f838a in ?? () from /lib64/ld-linux-x86-64.so.2
  #30 0x00007f2a754f8486 in ?? () from /lib64/ld-linux-x86-64.so.2
  #31 0x00007f2a754ea0ca in ?? () from /lib64/ld-linux-x86-64.so.2
  #32 0x0000000000000001 in ?? ()
  #33 0x00007ffdc7c0e8ea in ?? ()
  #34 0x0000000000000000 in ?? ()

What You Expected?

BE should not deadlock during startup.

How to Reproduce?

jemalloc profiling is enabled

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_9fe234b265b1
This content is generated by AI for reference only.

Initial analysis from current upstream/master code inspection:

This looks like a valid startup deadlock risk, not just a documentation mismatch. In current master, be/src/common/phdr_cache.cpp still exports a process-wide dl_iterate_phdr interposer. When the thread-local use_phdr_cache flag is false, that interposer calls getOriginalDLIteratePHDR(), and that function resolves the real loader function with dlsym(RTLD_NEXT, "dl_iterate_phdr") on the call path. The supplied stack is consistent with why this is unsafe: jemalloc profiling is already inside malloc_init_hard() and holds the jemalloc init mutex, then unwinding reaches dl_iterate_phdr, enters Doris' interposer, and dlsym() performs an allocation that re-enters jemalloc initialization.

There is one important build-path detail to verify. Current master's intended Linux path is that jemalloc heap profiling uses GNU libunwind, and Doris' patched libunwind calls the narrower doris_unwind_iterate_phdr(callback, data, ip) hook instead of ordinary dl_iterate_phdr. The third-party build script also configures jemalloc with --enable-prof-libunwind --disable-prof-libgcc and refuses a build where prof-libunwind is not enabled or prof-libgcc is enabled. However, the reported stack contains _Unwind_Backtrace / _Unwind_Find_FDE frames, which is exactly the kind of libgcc-style fallback current master is trying to prevent. So the immediate cause may be either:

1. the BE was built or run with stale/mismatched third-party jemalloc artifacts where heap profiling still uses the libgcc _Unwind_Backtrace path, or
2. another early unwind path is still reaching the process-wide dl_iterate_phdr interposer before the original function has been resolved in a safe context.

Either way, the code-level weak point is clear: the default branch of Doris' dl_iterate_phdr interposer should not perform lazy dlsym() while the caller may be in allocator initialization or unwinding. A robust fix should either avoid process-wide interposition for normal callers and keep the PHDR snapshot only behind doris_unwind_iterate_phdr / the minimal scoped stack-trace path, or ensure the original dl_iterate_phdr function pointer is obtained through a path that is proven not to allocate or re-enter jemalloc from inside the interposed function.

Missing information needed to pin this down:

• exact Doris commit SHA and whether third-party artifacts were rebuilt after the PHDR/libunwind changes in PR [Enhancement](BE) Make print stack API more stable #64093;
• jemalloc build evidence from config.log or the third-party build log showing prof-libunwind: 1 and prof-libgcc: 0;
• the startup environment that enabled profiling, especially MALLOC_CONF / JE_MALLOC_CONF and the BE launch command;
• glibc and JDK versions, since the observed allocation is inside dlsym() while libjvm.so static initialization is running;
• if available, symbol evidence from the affected BE binary showing whether jemalloc is using unw_backtrace or _Unwind_Backtrace.

Recommended next steps for maintainers:

• First verify the reporter's jemalloc profiling backend. If it is libgcc-backed, rebuild third-party artifacts from current master and confirm whether the startup hang disappears.
• Independently fix the interposer hazard: do not call dlsym(RTLD_NEXT, "dl_iterate_phdr") lazily from the normal dl_iterate_phdr forwarding path.
• Add a regression/manual startup test covering BE startup with jemalloc profiling enabled and JVM loading, because this failure happens before the service becomes ready and ordinary runtime tests may not exercise it.

[zclllyybb]

Ignore the previous AI response. Please confirm first: Have you upgraded the third-party libraries?