Writing a debugger in Rust pt.1
This is the first article of a serie that will explore how we can build a simple debugger in Rust. In this article, we start by exploring how to use syscall to halt and resume program, and examine the state of the registers.
What is a debugger ?
A debugger is a program that can start a process or attach itself to an existing one to help us debug its behavior. That’s as simple as that, the name speaks for itself.
Modern debuggers nowadays have a large range of various features, but we will focus on the core ones. We will create a debugger that can:
- step through the code of a process
- pause at pre-defined points in the code
- examine the state of the process (we’ll see what it means later)
For the rest of this article, and most of the series, we’ll write our debugger for a Linux-based distro and x86 architecture.
Run another process
The first line of the description we made of the debugger is that a debugger is a program. So let’s go ahead and create a simple binary with cargo. For those unfamiliar with Rust, cargo is Rust’s package manager, and it provides some sweet commands to make our life easier.
For example, initializing a folder that will contain the source files for our binary can be done with the following command:
cargo new --bin debugrs
It initializes a very simple binary that just prints Hello, world!, as we can see with the simple main function.
fn main() {
println!("Hello, world!");
}
We can then run it by using
# Inside ./debugrs folder
cargo run
Compiling debugrs v0.1.0 (/home/atticus/code/debugger/debugrs)
Finished dev [unoptimized + debuginfo] target(s) in 0.55s
Running `target/debug/debugrs`
Hello, world!
Sweet, it runs! But we are far from having a working debugger.
What is a syscall ?
If we look at the features that we expect from our debugger, we understand that we’ll need some help from the operating system, and most specifically, the kernel. “What is a kernel ?” you may ask. That’s a very interesting question, and it may deserve it’s own serie about it (maybe we’ll write our own OS someday). But to put it simply in a few words, it is the interface between the application that you are running, and your hardware.
When you are running multiple programs on your computer, they usually do not have access to each other’s memory. It would be a bad thing if your Spotify application could access the memory of your Web Browser and read your credit card numbers that you were currently entering to purchase something online.
The kernel prevents this, and to access the state of another process and manage its execution, we must ask for the kernel permission first. We can communicate with the kernel through syscalls. Syscall is the short for System call, which simply a way to call the system to request a service.
That’s great, but how do we call these syscalls inside our Rust code ? Looking at the manpage for syscall(), we can see that there is a function in libc that can be used to make the syscalls. If you’re wondering what libc is, there is also a manpage about it. To quote it, it is “a library of standard functions that can be used by all C programs”.
But, wait, we are writing Rust, and it says here that libc contains functions that can be used by C programs. You may be worried that it means that we are not going to be able to use it. But worry not, we will be able to use it. In fact, even if we haven’t written a single line of code yet, we are already using it ! If we use ldd to show the shared libraries that are needed to run our “Hello, world!” program, we can see that we are using libc.
$ ldd target/debug/debugrs
linux-vdso.so.1 (0x00007ffcab7f6000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f9673885000)
libc.so.6 => /lib64/libc.so.6 (0x00007f96736a7000) # <--- It's here
/lib64/ld-linux-x86-64.so.2 (0x00007f9673912000)
This is because libc is used in the standard library (the so-called std crate). Writing something to stdout uses the write syscall, and the standard libary is using libc to perform it.
Right now, let’s not worry to much about why and how we can use libc in our Rust code, this may be a story for another time. Let’s just be happy that we can. Even better, we will not be using libc directly as is doing the standard library, we will use an existing crate that provides a nice Rust wrapper around syscalls: nix.
Now that we know that we can make syscalls from our Rust code, and that we need to make some, the question left to answer is: Which syscalls should we use ?.
Making our syscalls
After a quick research on our favorite search engine, we find ptrace. The manpage states the following:
The ptrace() system call provides a means by which one process (the “tracer”) may observe and control the execution of another process (the “tracee”), and examine and change the tracee’s memory and registers. It is primarily used to implement breakpoint debugging and system call tracing.
Observing and controling the execution of another process… That’s exactly what we want ! Reading the rest of the manpage, we have more information about how we can attach to another process.
A process can initiate a trace by calling fork(2) and having the resulting child do a PTRACE_TRACEME, followed (typically) by an execve(2). Alternatively, one process may commence tracing another process using PTRACE_ATTACH or PTRACE_SEIZE.
Let’s go ahead and start writing some code. First of all, we said that we will need the nix crate to do our syscall, so we need to add it as a dependency in our Cargo.toml file.
We will ask for the ptrace feature for obvious reasons, and the process feature because it contains the syscall to fork, that was mentionned previously in the ptrace manpage. For those unfamiliar with Rust, features are a way to enable some parts of a library.
[package]
name = "debugrs"
version = "0.1.0"
edition = "2021"
[dependencies]
nix = {version = "0.26.2", features = ["ptrace", "process"]}
Then, we can import it in our main.rs file.
use nix::sys::ptrace;
use nix::unistd::fork;
fn main() {
}
Then, we will follow the instructions from ptrace manpage, and create a child by calling fork, and then, have this child call ptrace with the PTRACE_TRACEME flag.
But wait…
What is a child ?
ptrace manpage mentions creating a child with fork, but it does provide much information on what a child is, and how we can make the child run the code we want. Fortunately, the fork manpage explains the following:
fork() creates a new process by duplicating the calling process. The new process is referred to as the child process. The calling process is referred to as the parent process.
Cool, it’s just a duplicate of the calling process. Wait… a duplicate of the calling what ? What is a process ?. Oof… let’s just skip that for now. But I am sure we’ll come back to it in a future post. I’ll come back and put a link to it at this point. If you’re really eager to deep dive into what a Linux process is, you can find a beginning of explanation on the Linux documentation process.
Let’s just say that processes are a bunch of registers and memory (well, that description could fit a large number of things), and the CPU executes the instructions that it fetches thanks to these registers and memory. The OS keeps track of all the processes, and schedule them to run on the CPU.
We can see fork as a function that duplicates the register and memory, and create a similar process in the list kept by the OS. And this new process will indicate that the old one is its parent.
Well, we have talked enough, let’s do some experimentation. Let’s just write a single program that prints something to the screen, then call fork, then print something.
use nix::sys::ptrace;
use nix::unistd::fork;
fn main() {
println!("Hello before fork!");
unsafe { fork() }.expect("Failed to fork");
println!("Hello after fork");
}
This mean unsafe block is here to warn us that we are doing something that may be dangerous.
Indeed, if we look at the fork function documentation, we can see the following safety warning:
In a multithreaded program, only async-signal-safe functions like
pauseand_exitmay be called by the child (the parent isn’t restricted). Note that memory allocation may not be async-signal-safe and thus must be prevented.
Those functions are only a small subset of your operating system’s API, so special care must be taken to only invoke code you can control and audit.
Well, we are just playing around, the provided example in nix uses println, and the write syscall is async-signal-safe, so we should be good, let’s just run it.
$ cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.00s
Running `target/debug/debugrs`
Hello before fork!
Hello after fork
Hello after fork
Sweet, that what we would have expected. The process is duplicated when we call fork, and both the parent and the child continue the execution from this point, so both are printing Hello after fork on the screen. The child starts the execution just after the fork too, because we are duplicating the registers of the parent, including the EIP register, which keeps track of the next instruction to be executed.
Then, we have seen that we want our child to call ptrace with the PTRACE_TRACEME flag. But this is not something that we want to do with the parent. But the child is a copy of the parent, so are we not stuck ? Hopefully, fork does return a value, that is 0 if we are in the child, and the pid of the child in the parent. The nix crate wraps this nicely in the following enum:
pub enum ForkResult {
Parent { child: Pid },
Child,
}
Let’s use this, and modify our previous code.
use nix::sys::ptrace;
use nix::unistd::{fork, ForkResult};
fn main() {
let fork_result = unsafe { fork() }.expect("Failed to fork");
match fork_result {
ForkResult::Parent { child } => println!("Child pid: {child}"),
ForkResult::Child => {
ptrace::traceme().expect("Failed to call traceme in child");
println!("Hello from the child.");
}
}
}
Again, nix provides a nice wrapper around ptrace, and we don’t need to explicitely call ptrace with the PTRACE_TRACEME flag.
Let’s try to run our code:
$ cargo run
Finished dev [unoptimized + debuginfo] target(s) in 0.00s
Running `target/debug/debugrs`
Child pid: 61257
Hello from the child.
Nice, it seems to work. Now, let’s continue and call execve, as advised previously by the ptrace manpage. A quick look at execve manpage allow us to understand that execve replaces the current process by a new process that runs a program whose path is provided in the arguments.
Creating a program to debug
Let’s create very quickly the program that we will use to test our debugger. To do so, let’s climb back in the parent folder and create a new binary project.
$ cd ..
$ cargo new --bin debugee
$ cd debugee
$ cargo build --release
Compiling debugee v0.1.0 (/home/atticus/code/debugger/debugee)
Finished release [optimized] target(s) in 0.08s
Cool, we can now find our binary in target/release/debugee. If we run it, we get:
$ ./target/release/debugee
Hello, world!
Cool, that gives us a program that we can pass to execve. Looking at the nix crate documentation, we can see that execve has the following signature.
pub fn execve<SA: AsRef<CStr>, SE: AsRef<CStr>>(
path: &CStr,
args: &[SA],
env: &[SE]
) -> Result<Infallible>
args and env are just a way to pass arguments or environment variables to our program, we don’t need that for now. Our syscall will look like this.
ForkResult::Child => {
ptrace::traceme().expect("Failed to call traceme in child");
let path = todo!();
nix::unistd::execve(path, &[], &[]);
}
todo!(); is a very useful macro that will help your program compile, but crash at runtime. It virtually allows you to mock every value. Here, we don’t know how to create a variable with the &CStr type, but todo!() will tell the compiler that we can, and that path is of the &CStr type.
This is because todo!() is in fact of the Never type !, which can be coerced into any type. It is used to represent computations that will never complete. It is very useful to continue coding with the help of the compiler while leaving implementation details for later. Let’s check if our program compile.
$ cargo run
Compiling debugrs v0.1.0 (/home/atticus/code/debugger/debugrs)
warning: unreachable statement
--> src/main.rs:19:13
|
18 | let path = todo!();
| ------- any code following this expression is unreachable
19 | nix::unistd::execve(path, &[], &[]);
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ unreachable statement
|
= note: `#[warn(unreachable_code)]` on by default
error[E0282]: type annotations needed
--> src/main.rs:19:13
|
19 | nix::unistd::execve(path, &[], &[]);
| ^^^^^^^^^^^^^^^^^^^ cannot infer type of the type parameter `SA` declared on the function `execve`
|
help: consider specifying the generic arguments
|
19 | nix::unistd::execve::<SA, SE>(path, &[], &[]);
| ++++++++++
error[E0283]: type annotations needed
--> src/main.rs:19:13
|
19 | nix::unistd::execve(path, &[], &[]);
| ^^^^^^^^^^^^^^^^^^^ --- type must be known at this point
| |
| cannot infer type of the type parameter `SA` declared on the function `execve`
|
= note: multiple `impl`s satisfying `_: AsRef<CStr>` found in the following crates: `alloc`, `core`, `hashbrown`:
- impl AsRef<CStr> for CStr;
- impl AsRef<CStr> for CString;
- impl<'a, K, Q> AsRef<Q> for hashbrown::map::KeyOrRef<'a, K, Q>
where K: Borrow<Q>, Q: ?Sized;
- impl<T, A> AsRef<T> for Box<T, A>
where A: Allocator, T: ?Sized;
- impl<T, U> AsRef<U> for &T
where T: AsRef<U>, T: ?Sized, U: ?Sized;
- impl<T, U> AsRef<U> for &mut T
where T: AsRef<U>, T: ?Sized, U: ?Sized;
- impl<T> AsRef<T> for Arc<T>
where T: ?Sized;
- impl<T> AsRef<T> for Cow<'_, T>
where T: ToOwned, T: ?Sized;
- impl<T> AsRef<T> for Rc<T>
where T: ?Sized;
note: required by a bound in `nix::unistd::execve`
--> /home/atticus/.cargo/registry/src/index.crates.io-6f17d22bba15001f/nix-0.26.2/src/unistd.rs:832:19
|
832 | pub fn execve<SA: AsRef<CStr>, SE: AsRef<CStr>>(path: &CStr, args: &[SA], env: &[SE]) -> Result<Infallible> {
| ^^^^^^^^^^^ required by this bound in `execve`
help: consider specifying the generic arguments
|
19 | nix::unistd::execve::<SA, SE>(path, &[], &[]);
<...>
Ouch, what happened ?
The first warning warns us that our execve call will never be executed, because any code after todo!() is unreachable. Fair enough.
This first error tells us that the compiler cannot guess what the type SA is supposed to be. And it tells us that SA is a type used in execve signature, and if we look at it, we can see that it is the type of the args parameter. SA: AsRef<CStr>, so this type must implements the AsRef<CStr> trait. That’s actually quite nice. execve does not ask us for a specific type, but just anything that can be converted to a &CStr. We just need to help the compiler know what actual type it is, even if we are passing no data. A good type that can be converted to &CStr is CStr itself, the conversion is not to hard. Actually, if we look at the actual implementation it is as simple as expected:
#[stable(feature = "cstring_asref", since = "1.7.0")]
impl AsRef<CStr> for CStr {
#[inline]
fn as_ref(&self) -> &CStr {
self
}
}
Let’s specify our types, and try to run our program again.
ForkResult::Child => {
ptrace::traceme().expect("Failed to call traceme in child");
let path = todo!();
nix::unistd::execve::<&CStr, &CStr>(path, &[], &[]);
}
$ cargo run
<warnings>
Child pid: 43618
thread 'main' panicked at 'not yet implemented', src/main.rs:18:31
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Cool, as expected, our child panics. Let’s just find out what a CStr is, and how to create it. According to the documentation
This type represents a borrowed reference to a nul-terminated array of bytes. It can be constructed safely from a &[u8] slice […].
&CStris toCStringas&stris toString: the former in each pair are borrowed references; the latter are owned strings.
Cool, let’s do it.
ForkResult::Child => {
ptrace::traceme().expect("Failed to call traceme in child");
let path: &CStr = &CString::new("../debugee/target/release/debugee").unwrap();
nix::unistd::execve::<&CStr, &CStr>(path, &[], &[]).unwrap();
}
For now, this relative path will do, but keep in mind that at some point, if we move the binary or the working directory, we will have an issue. This time, the program compiles, and runs as expected.
$ cargo run
Compiling debugrs v0.1.0 (/home/atticus/code/debugger/debugrs)
Finished dev [unoptimized + debuginfo] target(s) in 0.13s
Running `target/debug/debugrs`
Child pid: 45174
Hello, world!
It printed Hello, world! from our debugee, how sweet !
That is cool, we can run another program from our debugger. But it is not really useful to do any debugging just yet. Our child finish executing before we can even start inspecting its register. How can we pause it, such that we have time to look what is happening ?
Lucky us, the ptrace manpage informs us that
If the PTRACE_O_TRACEEXEC option is not in effect, all successful calls to execve(2) by the traced process will cause it to be sent a SIGTRAP signal, giving the parent a chance to gain control before the new program begins execution.
This does not really tells us how to gain control, but the previous paragraph does.
While being traced, the tracee will stop each time a signal is delivered, even if the signal is being ignored. […] The tracer will be notified at its next call to waitpid(2) […]; that call will return a status value containing information that indicates the cause of the stop in the tracee.
But well, we’ve seen that our child program is not stopping at all, it even finishes printing Hello, world!. From what we understand from the documentation, it should be waiting for the parent to tell it to continue. But in our case, the parent may already have exited, because it does not do anything. So maybe the behavior in this case is just to continue executing. To test this hypothesis, let’s add an infinite loop in the parent so it does not exit.
ForkResult::Parent { child } => {
println!("Child pid: {child}");
loop {}
}
Now the child halts !
$ cargo run
Child pid: 7293
<program halted>
That’s good news, this means that our child indeed stops, and we can control it before it starts. To do, we most certainly need to use waitpid.
use nix::sys::wait::waitpid;
// Skipped code...
ForkResult::Parent { child } => {
println!("Child pid: {child}");
loop {
let waitpid = waitpid(child, None).expect("Failed to wait");
println!("{:?}", waitpid);
}
}
$ cargo run
Child pid: 8426
Stopped(Pid(8426), SIGTRAP)
<program halted>
Our child is stopped by a SIGTRAP, as we read on the ptrace manpage. But then, the whole program halts because the parent is waiting again for the child, and the child has not resumed execution. To fix this, the parent needs to use a ptrace syscall using PTRACE_CONT to tell the child to resume execution.
ForkResult::Parent { child } => {
println!("Child pid: {child}");
let waitpid_result = waitpid(child, None).expect("Failed to wait");
println!("{:?}", waitpid_result);
ptrace::cont(child, None).expect("Failed to use PTRACE_CONT");
loop {
let waitpid_result = waitpid(child, None).expect("Failed to wait");
println!("{:?}", waitpid_result);
}
}
Here, we ask the child to continue running after the first call to waitpid, and then, we enter the loop to see if waitpid returns again.
$ cargo run
Child pid: 9188
Stopped(Pid(9188), SIGTRAP)
Hello, world!
Exited(Pid(9188), 0)
thread 'main' panicked at 'Failed to wait: ECHILD', src/main.rs:17:59
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
We can see that the execution continued, and our call to ptrace with PTRACE_CONT worked, because we can see the Hello, world! in the output. Then waitpid returns to tell us that the child process has exited. Then, our next call to waitpid crashes, because the child has already exited, so there is nothing to wait on.
But well, knowing when the process starts and when it ends it not really useful for debugging. We need to be able to stop the process at other point in time. Just after the entry for PTRACE_CONT on the manpage, we can find the entry for PTRACE_SYSCALL, PTRACE_SINGLESTEP. It allows us to respectively continue the child until the start or a stop of a system call, or after the execution of a single instruction.
Jumping from syscalls to syscalls
Let’s check PTRACE_SYSCALL, to check how many syscalls our child is doing.
ForkResult::Parent { child } => {
println!("Child pid: {child}");
loop {
let waitpid_result = waitpid(child, None).expect("Failed to wait");
println!("{:?}", waitpid_result);
ptrace::syscall(child, None).expect("Failed to use PTRACE_SYSCALL");
}
}
Basically, we tell the child process to continue until the start or the end of the next syscall, then wait for it to be stopped again, tell it to continue, and so on. Of course, we should check that the waitpid_result is indeed the Stopped variation of the enum before telling the child to continue, but we are just toying around for now.
cargo run
Compiling debugrs v0.1.0 (/home/atticus/code/debugger/debugrs)
Finished dev [unoptimized + debuginfo] target(s) in 0.13s
Running `target/debug/debugrs`
Child pid: 11927
Stopped(Pid(11927), SIGTRAP)
Stopped(Pid(11927), SIGTRAP)
Stopped(Pid(11927), SIGTRAP)
<... a whole lot of SIGTRAP ...>
Stopped(Pid(11927), SIGTRAP)
Hello, world!
Stopped(Pid(11927), SIGTRAP)
Stopped(Pid(11927), SIGTRAP)
Stopped(Pid(11927), SIGTRAP)
Stopped(Pid(11927), SIGTRAP)
Stopped(Pid(11927), SIGTRAP)
Stopped(Pid(11927), SIGTRAP)
Exited(Pid(11927), 0)
thread 'main' panicked at 'Failed to use PTRACE_SYSCALL: ESRCH', src/main.rs:16:46
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Wow, that’s a lot of syscall for a simple println. I wonder what they are. If would be great if we had a debugger to investigate this…
We may want to use PTRACE_GET_SYSCALL_INFO, to get more info about what syscalls are executed, but unfortunately, it is not available in nix yet, and we are not writing syscalls on our own. (well, not for now…).
How syscall are made
We are in a situation where the child is stopped just before making a syscall. Now, we need to figure out how to know what is the syscall that will be made. To do so, it would be quite useful to know how syscalls are made. A good place to start is, as always, the syscall manpage. It is the documentation for the syscall function and not the syscall definition, but we can find interesting stuff in it anyway. We can first see that, unsurprinsingly, calling conventions for syscall depends on the architecture. In our case, we’ll focus on x86-64.
Arch/ABI Instruction System Ret Ret Error Notes call # val val2 ─────────────────────────────────────────────────────────────────── x86-64 syscall rax rax rdx - 5
Arch/ABI arg1 arg2 arg3 arg4 arg5 arg6 arg7 Notes ────────────────────────────────────────────────────────────── x86-64 rdi rsi rdx r10 r8 r9 -
We can then see that when using the syscall instruction, the system call number must be placed in the rax register. Then the return values will be in rax and rdx. Different arguments passed to the syscall are placed in different registers.
So what we can do is inspect the registers before the child makes the syscall to get the syscall number (and potentially its arguments if we wanted to), and then inspect the registers again when exiting the syscall to get the result.
Getting the registers
Once again, ptrace is kind enough to provides us with the utility to get registers. PTRACE_GETREGS is what we need. We will get the registers twice: once when the child stops just before the syscall, so we can get the syscall number, then once when the child exits the syscall, so we can get the return value.
ForkResult::Parent { child } => {
let _ = waitpid(child, None).expect("Failed to wait");
ptrace::syscall(child, None).expect("Failed to use PTRACE_SYSCALL");
loop {
let _ = waitpid(child, None).expect("Failed to wait");
let before_call_registers = ptrace::getregs(child).expect("could not get child's registers");
println!("Entering syscall #{}", before_call_registers.orig_rax);
ptrace::syscall(child, None).expect("Failed to use PTRACE_SYSCALL");
let _ = waitpid(child, None).expect("Failed to wait");
let after_call_registers = ptrace::getregs(child).expect("could not get child's registers");
println!("Syscall #{}, Result: ({}, {})", before_call_registers.orig_rax, after_call_registers.rax, after_call_registers.rdx);
ptrace::syscall(child, None).expect("Failed to use PTRACE_SYSCALL");
}
}
$ cargo run
Entering syscall #12
Syscall #12, Result: (94256054034432, 0)
Entering syscall #158
Syscall #158, Result: (18446744073709551594, 140218273229584)
Entering syscall #21
Syscall #21, Result: (18446744073709551614, 1)
Entering syscall #257
Syscall #257, Result: (3, 524288)
Entering syscall #262
Syscall #262, Result: (0, 140725890705200)
<--- a bunch of syscalls --->
Entering syscall #3
Syscall #3, Result: (0, 140218272824672)
Entering syscall #204
Syscall #204, Result: (8, 94256054037008)
Entering syscall #1
Hello, world!
Syscall #1, Result: (14, 14)
Entering syscall #131
Syscall #131, Result: (0, 1)
Entering syscall #11
Syscall #11, Result: (0, 0)
Entering syscall #231
Well, numbers aren’t really helpful on their own, but we can use a lookup table to find what the number corresponds to. The number 12 corresponds to brk, which “change data segment size”. Number 158 is then arch_prctl, which “set architecture-specific thread state”. Number 21 and 257 are access and openat, which check for permissions to access a file and open a file.
Basically, the child is setting itself up, by setting some of it state, then starting to open and load different libraries. Indeed, before being able to just call write to write Hello, world! to the screen, the process needs to set a bunch of stuff. We talked about glibc at the beginning of this post (even if it feels like an eternity ago). The child needs to load it because it uses it to make syscalls. And we can even see our call to write just before the Hello, world!. That is exactly what we would expect to see.
Wrapping up
Well, our debugger helped us understand the loading of a process, by starting a child process and pausing it at certain moment in time. By getting the value of the registers of the child process, we could see every syscall made by our child process.
Next time, we will start adding breakpoints and dynamically interacting with our debugger.
Code for this serie is available on Github.