Inside our inline hooking engine: relocating prologues with iced-x86, and the instruction window we can’t close

By The Rutken team

A complete x86-64 and AArch64 inline detour engine whose dependency list is closed to exactly two crates: iced-x86 and thiserror. No frida-gum, no vendored Dobby or shadowhook. It installs a hook in six stages, writes the 5-byte prologue patch as a single aligned 8-byte store, and (stated up front) leaves one instruction-window race it does not claim to close. Below is the whole pipeline, with the real encodings.

We wrote earlier about why we built our own hooking engine instead of wrapping one. This is the how: the in-process, inline-detour engine that patches a live function’s prologue and reroutes it through a trampoline. It’s a single Rust crate, and this post walks its pipeline with the real identifiers, the real encodings, and, up front, the one hazard it does not solve.

The dependency list is closed to two crates

The engine depends on exactly iced-x86 and thiserror. No frida-gum, no vendored Dobby or shadowhook, and no lint-policy exception. The crate’s own lint policy keeps exceptions = [] empty. It doesn’t even depend on another Rutken crate: it returns its own InstallError/ResolveError and knows nothing about Rutken’s wire protocol. A test pins that list mechanically.

The iced-x86 dependency has a subtlety that is the whole reason the crate is feature-gated. Our disassembler crate uses iced-x86 decoder-only, so it never re-encodes or renders instructions, so it resolves ["std", "decoder", "instr_info"] and nothing more. This engine’s entire reason to exist is re-encoding relocated prologues, so it needs encoder + block_encoder. Cargo unifies features across the build graph: if the engine asked for those unconditionally, a default cargo build --workspace would silently hand the disassembler’s copy of iced-x86 the encoder too. So the dependency is optional and gated behind a non-default bespoke-agent feature:

[features]
default = []
bespoke-agent = ["dep:iced-x86"]

[dependencies]
iced-x86 = { workspace = true, optional = true, features = [
    "encoder",
    "block_encoder",
] }
thiserror = { workspace = true }

And a feature-gate test proves it, via cargo tree rather than by assertion-of-intent. One direction checks a default build never pulls the dep in at all; the inverse checks the feature actually wires it up (so the first test isn’t vacuously green because the dep got renamed):

#[test]
fn default_hookengine_build_does_not_link_iced_x86() {
    let tree = cargo_tree(&["-p", "hookengine", "--no-default-features", "-e", "normal"]);
    assert!(
        !tree.contains("iced-x86"),
        "default-feature `hookengine` must not depend on `iced-x86` ...\n{tree}"
    );
}

A default workspace build doesn’t resolve iced-x86’s encoder with the feature unused (the optional dependency is genuinely absent from the graph), which is exactly what keeps the disassembler’s feature set byte-for-byte what it was before this crate existed.

Six stages, x86-64 first

The engine is six independently testable stages, and Engine::install runs them strictly in order, checking every refusal condition before it touches any live or allocated memory:

// Stage 1  symbol resolution  -> runtime address of the target
// Stage 2  prologue decode    -> decode until >= PATCH_LEN (5) bytes covered,
//                                never splitting an instruction
// Stage 3  RELOCATION         -> re-encode the moved prologue for its new home
// Stage 4  near-memory alloc  -> trampoline within rel32 reach of the target
// Stage 5  page protection    -> flip the trampoline RX, the site RWX briefly
// Stage 6  atomic patch       -> one aligned store overwrites the prologue

Decode never splits an instruction: it keeps decoding whole instructions until the cumulative length is >= PATCH_LEN (5 bytes, the length of a jmp rel32), so the bytes we later relocate are always a clean run of complete instructions. A prologue that ends in a ret before it covers 5 bytes, or that is already an unconditional jump (someone else’s hook) or a PLT-style indirect stub, is refused by name (PrologueTooShort, ForeignHookPresent, ResolvedToStub), never forced. The whole decode loop is small enough to read in one sitting (too_short(n) below is shorthand for building the PrologueTooShort { need: PATCH_LEN, found: n } error):

pub fn decode_prologue(code: &[u8], ip: u64) -> Result<(Vec<Instruction>, usize), InstallError> {
    let mut decoder = Decoder::with_ip(64, code, ip, DecoderOptions::NONE);
    let mut instructions = Vec::new();
    let mut covered: usize = 0;

    while covered < PATCH_LEN {                      // PATCH_LEN == 5 (jmp rel32)
        if !decoder.can_decode()  { return Err(too_short(covered)); }
        let instr = decoder.decode();
        if instr.is_invalid()     { return Err(too_short(covered)); }
        let len = instr.len();
        if len == 0 { return Err(too_short(covered)); } // iced "never" does this; don't spin forever on it
        covered += len;
        let is_ret = matches!(instr.mnemonic(), Mnemonic::Ret);
        instructions.push(instr);
        if is_ret && covered < PATCH_LEN {           // function ends before 5 bytes: a stub, refuse it
            return Err(too_short(covered));
        }
    }
    Ok((instructions, covered))
}

The len == 0 guard is not theoretical paranoia dressed up: iced-x86 should never return a zero-length valid instruction, but if it did, the loop would make no progress and spin forever on a live target’s memory. A hang is a worse failure than a refusal, so a zero-length decode is treated as “too short” and the install is declined rather than risked.

Stage 3 is the heart: relocation

When you move a prologue’s bytes to a trampoline, every RIP-relative operand and every branch displacement in them is now wrong, because it was computed against the original address. Relocation re-encodes those instructions so they still target the same absolute addresses from their new location. On x86-64 we hand that entirely to iced-x86’s BlockEncoder, which exists to do precisely this. The whole module is small on purpose. Our job is to hand it the decoded prologue at a new rip and turn any failure into a named refusal rather than guessing:

use iced_x86::{BlockEncoder, BlockEncoderOptions, Instruction, InstructionBlock};

pub fn relocate(instructions: &[Instruction], new_ip: u64) -> Result, InstallError> {
    let block = InstructionBlock::new(instructions, new_ip);
    let result = BlockEncoder::encode(64, block, BlockEncoderOptions::NONE)
        .map_err(|e| InstallError::PrologueUnrelocatable(e.to_string()))?;
    Ok(result.code_buffer)
}

There is a chicken-and-egg problem hiding here. BlockEncoder’s output length can depend on the destination address (a far RIP-relative target may need a longer encoding), but the destination is a near-memory allocation that itself needs a size up front. We solve it with two relocation passes. Pass 1 relocates at a provisional address, entry itself; BlockEncoder::encode only computes displacements against a u64, it never reads or writes memory, so the address need not be real, purely to measure the relocated length. That estimate sizes the allocation. Pass 2 relocates again at the real trampoline address. The oversize case is still a checked, enforced invariant (InstallError::TrampolineOverflow), not an assumption: Vec::with_capacity is only a hint. Here is the actual shape of it inside Engine::install, both passes:

// Pass 1: relocate at a PROVISIONAL address -- `entry` itself -- only to
// measure the relocated length. BlockEncoder::encode computes displacements
// against a u64; it never reads or writes memory, so the address need not
// be real, allocatable, or even aligned.
let stub_len    = build_detour_stub(0, 0).len();
let resume_len  = build_abs_jump(0).len();
let provisional = relocate(&instructions, entry as u64)?;
let needed      = stub_len + provisional.len() + resume_len;

// Size the near allocation from that estimate ...
let capacity    = needed.div_ceil(PAGE_SIZE) * PAGE_SIZE;
let trampoline  = sys::alloc::alloc_near(entry, needed)? as usize;

// ... then relocate AGAIN at the real trampoline address, now that it is
// known. If this fails after we already mapped the trampoline, free it on
// the way out -- a refusal must not leak a mapping.
let relocated = relocate(&instructions, (trampoline + stub_len) as u64)
    .inspect_err(|_| { let _ = sys::alloc::free(trampoline_ptr, capacity); })?;

// Assemble [ detour stub | relocated prologue | resume jump ] contiguously.
// The estimate is a hint, not a contract: the Vec grows freely, so a pass-2
// encoding larger than pass 1's would write past the mapping. Enforce the
// invariant by name instead of trusting it:
if trampoline_body.len() > capacity {
    let _ = sys::alloc::free(trampoline_ptr, capacity);
    return Err(InstallError::TrampolineOverflow {
        target: entry, needed: trampoline_body.len(), capacity,
    });
}

The detour stub, byte by byte

The first thing the trampoline runs is a stub we hand-assemble. There is no assembler in the dependency list, so the stub is written as literal opcode bytes. Its contract is narrow and strict: save every register the hooked function’s ABI could be carrying an argument in, snapshot the first six integer/pointer argument registers into a RawArgs struct, call the observer, restore the registers untouched, and tail-jump into the relocated original. On the SysV (Linux/Android) side that is this, in full:

#[cfg(all(unix, target_arch = "x86_64"))]
fn build_detour_stub(hit_fn: u64, relocated_addr: u64) -> Vec<u8> {
    let mut b = Vec::with_capacity(96);
    // push rax, rdi, rsi, rdx, rcx, r8, r9, r10, r11  (9 pushes = 72 bytes;
    // entry rsp%16 == 8, so after these rsp%16 == 0 -- the alignment a `call`
    // expects to see restored)
    b.extend_from_slice(&[0x50, 0x57, 0x56, 0x52, 0x51]);
    b.extend_from_slice(&[0x41,0x50, 0x41,0x51, 0x41,0x52, 0x41,0x53]);

    // sub rsp, 48 -- reserve the RawArgs slot (48 is 16-aligned)
    b.extend_from_slice(&[0x48, 0x83, 0xEC, 0x30]);
    // store rdi, rsi, rdx, rcx, r8, r9 into [rsp+0 .. rsp+48). SysV classifies
    // a 48-byte by-value struct as MEMORY: the 48 bytes ARE the argument, so
    // no register ever carries a pointer to it -- which is exactly why the
    // Rust-side `hit_fn: extern "C" fn(RawArgs)` can live in crates that
    // forbid `unsafe`: no raw pointer crosses into them.
    for (reg, off) in [(7u8,0u8),(6,8),(2,16),(1,24),(8,32),(9,40)] {
        b.extend_from_slice(&mov_store_reg_to_rsp_offset(reg, off));
    }

    // mov r11, hit_fn ; call r11
    b.push(0x49); b.push(0xBB); b.extend_from_slice(&hit_fn.to_le_bytes());
    b.extend_from_slice(&[0x41, 0xFF, 0xD3]);

    // add rsp, 48 ; pop (reverse order) ; mov r11, relocated_addr ; jmp r11
    b.extend_from_slice(&[0x48, 0x83, 0xC4, 0x30]);
    b.extend_from_slice(&[0x41,0x5B, 0x41,0x5A, 0x41,0x59, 0x41,0x58,
                          0x59, 0x5A, 0x5E, 0x5F, 0x58]);
    b.push(0x49); b.push(0xBB); b.extend_from_slice(&relocated_addr.to_le_bytes());
    b.extend_from_slice(&[0x41, 0xFF, 0xE3]);
    b
}

The alignment arithmetic in the comments is load-bearing, not decoration: a function reached through a call expects rsp % 16 == 8 at entry, and the nine pushes plus a 16-aligned sub are chosen precisely so the observer is called at rsp % 16 == 0, the state its own call will then turn back into the 8 the ABI promises. The Microsoft x64 ABI needs a different stub entirely (four argument registers, a struct this size passed by reference, 32 bytes of mandatory shadow space); it lives behind the same name, selected by #[cfg(windows)], and its byte encoding is regression-tested on non-Windows hosts by comparison, because the bytes are ordinary x86-64 even when the convention is not.

The resume jump that cost us a crash

relocate deliberately does not append the jump back into the rest of the original function. That resume jump is the caller’s job, because it is not a relocation concern and, crucially, it is not the part written atomically. It is a register-free jmp qword ptr [rip+0] followed by the 8-byte absolute target:

fn build_abs_jump(target: u64) -> Vec<u8> {
    let mut b = Vec::with_capacity(14);
    b.push(0xFF); b.push(0x25);                 // jmp qword ptr [rip+disp32]
    b.extend_from_slice(&0u32.to_le_bytes());   // disp32 = 0 -> target is the next 8 bytes
    b.extend_from_slice(&target.to_le_bytes()); // 8-byte absolute target, reachable anywhere
    b
}

It used to be mov r11, imm64; jmp r11, which is shorter to reason about and clobbers only r11, a caller-saved scratch register, so who cares. We cared. The resume jump runs immediately after the relocated prologue, and a very common MSVC x64 prologue is the “alternate frame pointer” pattern mov r11, rsp; mov [r11+N], reg (e.g. KernelBase!RegSetValueExW), which parks a live value in r11 that the rest of the function goes on to dereference. Overwriting r11 in the resume jump corrupted that pointer and crashed the target on its very next store through it. The rip-relative indirect jump touches zero general-purpose registers, so it cannot collide with whatever register the relocated original happens to still be using, no matter which one the compiler picked. The fix is unconditionally register-free rather than register-free-only-where-the-bug-bit, so there is no asymmetry left for a future caller of this helper to fall into.

The hit you would otherwise miss

One line in the install path has nothing to do with patching and everything to do with not lying to yourself. Right before stage 6 makes the hook live, we warm a cache:

// Warm sys::maps's /proc/self/maps cache BEFORE the atomic patch makes this
// hook live.
sys::maps::refresh();

It looks like an optimisation and it is not. The detour consults /proc/self/maps through a cache, and populating that cache for the first time calls open. If we let the cache populate lazily, the first hit on the hook would trigger that first open, and if the symbol we hooked is open, that first hit would reenter the fresh reentrancy guard and be dropped, unobserved, while every later hit sailed through. The one call you most wanted to see (the first one) is the exact one you would silently lose. Warming the cache before the hook goes live spends that open while nothing is hooked yet. It is the kind of bug that never shows up in a test that hooks a nice quiet function, and always shows up the day you hook the interesting one.

Stage 6, and the honesty anchor

The live patch at the hook site is a 5-byte jmp rel32. We write it as one aligned 8-byte store: read the 8-byte-aligned word that contains the 5 patch bytes, splice the opcode and rel32 into a copy, and store the whole word in a single atomic operation. The rel32 displacement itself is range-checked: if the trampoline landed too far for a 32-bit relative jump, the install refuses (NoNearMemory) rather than truncating the offset:

let rel32 = trampoline as i64 - (entry as i64 + 5);          // jmp is relative to its END
let rel32 = i32::try_from(rel32)
    .map_err(|_| InstallError::NoNearMemory { target: entry })?;

let mut new_word = original_word;          // the 8-aligned word covering `entry`
new_word[offset] = decode::JMP_REL32_OPCODE;                // 0xE9
new_word[offset + 1..offset + 5].copy_from_slice(&rel32.to_le_bytes());

sys::protect::mprotect(aligned_start, 8, PROT_READ_WRITE_EXEC)?;
sys::patch::atomic_store_word(aligned_start, new_word);     // single aligned store
sys::protect::mprotect(aligned_start, 8, PROT_READ_EXEC)?;
sys::protect::assert_restored_to_rx(aligned_start)?;        // never leave a page writable+executable

The single store means no other thread can observe a half-written jump, so there is no torn read. That is the extent of the guarantee, and the crate says so in its own words. In the engine’s own words:

This matters and it is easy to paper over, so we won’t. An aligned store fixes readers: no thread ever decodes a Frankenstein instruction built from old and new bytes. It does nothing for a thread whose program counter is already partway through the five bytes we’re overwriting when the store lands. That thread finishes executing instructions that no longer exist in the form it started decoding them. There is no in-engine fix for that on a prologue patch; the honest mitigation is to install before the target code is reachable, which is a property of how the agent is delivered, not of this crate. When near-memory allocation or the aligned store isn’t available, the engine refuses with a named InstallError rather than falling back to a racy write.

The AArch64 side: a hand-written relocator and an x16/x17 veneer

iced-x86 is x86-only, so the AArch64 path shares none of stage 3. It is a hand-written decoder/relocator in the same crate; by design, both architectures belong here, not a second crate. The patch itself is simpler than x86: every A64 instruction is one 4-byte, 4-byte-aligned word, so the hook site is overwritten with a single B imm26 as one atomic 32-bit store, race-free by construction, not merely torn-read-safe, because any thread’s PC is either at that word or past it.

The interesting part is range. A direct B/BL reaches ±128 MB; a relocated branch (or the far jump of an inverted conditional) whose target is outside immediate range needs an escape. That’s a 16-byte veneer that clobbers only x16 and x17, the AArch64 PCS intra-procedure-call scratch registers, safe to clobber at a call boundary:

LDR  x16, #8      ; 0x58000050  load the 8 bytes at PC+8 (the target below)
BR   x16          ; 0xD61F0200  (or BLR x16, 0xD63F0200, if relocating a BL)
.quad target      ; 8 bytes, reachable anywhere in the 64-bit address space

const LDR_X16_PC_PLUS_8: u32 = 0x5800_0050;
const BR_X16:  u32 = 0xD61F_0200;
const BLR_X16: u32 = 0xD63F_0200;
pub const VENEER_LEN: usize = 16;

pub fn build(target: u64, linking: bool) -> Vec {
    let mut out = Vec::with_capacity(VENEER_LEN);
    out.extend_from_slice(&LDR_X16_PC_PLUS_8.to_le_bytes());
    out.extend_from_slice(&(if linking { BLR_X16 } else { BR_X16 }).to_le_bytes());
    out.extend_from_slice(&target.to_le_bytes());
    out
}

The 16-byte veneer is never written at the atomically-patched hook site, because a 16-byte write isn’t atomic. It only ever lives inside the trampoline, alongside the relocated prologue, which is written in full before any thread can reach it.

Where this sits

This engine is one layer of Rutken’s in-house instrumentation agent, an alternative to injecting a general-purpose runtime, which some targets detect and refuse to run under. Rutken is available here if you want to try it. And if you spot something wrong in the mechanics above, we’d genuinely like to hear it. The residual window in stage 6 is exactly the kind of thing we’d rather state plainly than have you find.