Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wasmtime rust application crashed when debug flag is enabled #10138

Open
ggjjj opened this issue Jan 28, 2025 · 8 comments
Open

Wasmtime rust application crashed when debug flag is enabled #10138

ggjjj opened this issue Jan 28, 2025 · 8 comments
Labels
bug Incorrect behavior in the current implementation that needs fixing

Comments

@ggjjj
Copy link

ggjjj commented Jan 28, 2025

Steps to Reproduce

Create a rust application with wasmtime engine and enable debug flag.

    let mut engine_config = EngineConfig::default();
    engine_config.debug_info(true);
     // create link and insert_components 
    let pre_instance = linker.instantiate_pre(&module)?;

Note: Cannot share the complete code here want to understand why the sedfaults and crashes are occuring

Expected Results

Breakpoint with codeLLDB continues successfully instead of crashing of application

Actual Results

Segfault into

__attribute__((weak, noinline))
#endif
    void __jit_debug_register_code() {
#ifndef CFG_TARGET_OS_windows
  __asm__("");
#endif
}

Versions and Environment

Wasmtime version or commit: 25.0.2

Operating system: Ubuntu

Architecture: x86_64

Code LLDB extension version : 1.11.3

lldb version with LLVM - 19.1.7

@ggjjj ggjjj added the bug Incorrect behavior in the current implementation that needs fixing label Jan 28, 2025
@bjorn3
Copy link
Contributor

bjorn3 commented Jan 28, 2025

That function shouldn't be able to segfault. It literally does nothing but return. That function only exists for the debugger to set a breakpoint at and when the breakpoint gets hit have the debugger update it's cache of debuginfo for jitted functions.

Can you show the actual crash message lldb shows?

@ggjjj
Copy link
Author

ggjjj commented Jan 30, 2025

Here is some more data. I updated the versions of LLDB and codeLLDB used in the issue description and also enabled this flag settings set plugin.jit-loader.gdb.enable on

When I launch the rust app which loads multiple modules as a graph, we get the following error when launched lldb via terminal.

 lldb-19  
(lldb) target create ./runtime/target/debug/host-app
Current executable set to './runtime/target/debug/host-app' (x86_64).
(lldb) settings set plugin.jit-loader.gdb.enable on
(lldb) run
Process 25109 launched: './runtime/target/debug/host-app' (x86_64)
<6>2025-01-30T00:29:49.700Z codespaces-a5dc2f [host-app@311 tid="25109"] - starting host-app
<6>2025-01-30T00:29:49.821Z codespaces-a5dc2f [host-app@311 tid="25109"] - pre_instantiating and inserting component module1:v1
<6>2025-01-30T00:29:53.328Z codespaces-a5dc2f [host-appr@311 tid="25109"] - pre_instantiating and inserting component module2:v1
<6>2025-01-30T00:29:56.869Z codespaces-a5dc2f [host-app@311 tid="25109"] - pre_instantiating and inserting component module3:v1
Process 25109 stopped
* thread #1, name = 'host-app', stop reason = signal SIGTERM
   frame #0: 0x0000555556aac844 host-app`__jit_debug_register_code at helpers.c:92:38
  89   // `asm` statement.
  90   __attribute__((weak, noinline))
  91   #endif
-> 92       void __jit_debug_register_code() {
  93   #ifndef CFG_TARGET_OS_windows
  94     __asm__("");
  95   #endif

@ggjjj
Copy link
Author

ggjjj commented Jan 30, 2025

Call stack image while doing visual breakpoint debugging

Image

crash message in debug console

Image

segfault in the code

Image

@cfallin
Copy link
Member

cfallin commented Jan 30, 2025

SIGTERM is not a segfault; that usually indicates that something external to the process has killed it. Can you tell us anything more about your embedding/surrounding application or the environment it runs in? Otherwise, all we have to go on is "call to empty function ends in SIGTERM" which unfortunately doesn't point in any useful direction (as far as I can tell at least!).

@cfallin
Copy link
Member

cfallin commented Jan 30, 2025

Actually, going off of what @bjorn3 said above

That function only exists for the debugger to set a breakpoint at and when the breakpoint gets hit have the debugger update it's cache of debuginfo for jitted functions.

Perhaps this is working as intended: lldb does, indeed, stop with a signal at that function. You're observing internal debugging plumbing in that case. Can you continue (c) to further execution?

@ggjjj
Copy link
Author

ggjjj commented Feb 1, 2025

Actually, going off of what @bjorn3 said above

That function only exists for the debugger to set a breakpoint at and when the breakpoint gets hit have the debugger update it's cache of debuginfo for jitted functions.

Perhaps this is working as intended: lldb does, indeed, stop with a signal at that function. You're observing internal debugging plumbing in that case. Can you continue (c) to further execution?

I tried this and the process exits after sometime. I tried to set breakpoint on exits and aborts to see what causes the exit but nothing hits the breakpoint on exit. so Not sure what is making the process to exit

@ggjjj
Copy link
Author

ggjjj commented Feb 1, 2025

SIGTERM is not a segfault; that usually indicates that something external to the process has killed it. Can you tell us anything more about your embedding/surrounding application or the environment it runs in? Otherwise, all we have to go on is "call to empty function ends in SIGTERM" which unfortunately doesn't point in any useful direction (as far as I can tell at least!).

what can cause SIGTERM ? is there a way to find from call stack? The application is in rust which configures the wasmtime in debug mode and loads three wasm component models

Call stack image updated

Image

@cfallin
Copy link
Member

cfallin commented Feb 1, 2025

what can cause SIGTERM ? is there a way to find from call stack?

In general, the kill() syscall; another process needs to send it, Wasmtime will not send it to itself or perform any action that causes it to be sent. (To quote this StackOverflow answer, "If a process receives SIGTERM, some other process sent that signal.") I'm not sure about determining where it was sent from -- in your broader application maybe there is something you could strace, or maybe you could use eBPF kprobes and instrument the kill syscall to print a log message. I'm afraid I can't help too much more without knowing more about your application, sorry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Incorrect behavior in the current implementation that needs fixing
Projects
None yet
Development

No branches or pull requests

3 participants