There are many techniques to protect sensitive code in Android using obfuscation or moving it to native C/C++ code compiled to machine code. With tools like IDA pro, Ghidra, along with Frida, we can easily reverse the code and hook it with custom implementations.

In this post, I want to walk through why executing root detection inside a custom virtual machine (VM) is much more secure than running the same logic directly in Java, Kotlin, or even native C/C++ code — and what this means for tools like Frida and experienced reverse engineers.

Root Detection in ART or Native

Most apps do something like this:

fun isDeviceRooted(): Boolean {
    return File("/system/bin/su").exists()
}

Or worse, they just call native function in C:

bool isRooted() {
    return access("/system/bin/su", F_OK) == 0;
}

While this might catch a few rooted devices. Frida scripts can override return values. Decompiled APKs reveal function names. Native .so files can be easily patched by using Ghidra or IDA pro.

Using Custom VM

A custom VM works like this:

• You define your own bytecode format and instruction set (opcodes like CHECK_ROOT, HALT, etc.).
• Sensitive logic is written in bytecode, not Kotlin/Java.
• At runtime, your custom interpreter runs this bytecode.

A real example of root check bytecode below:

val bytecode = byteArrayOf(0x01, 0xFF) // CHECK_ROOT, HALT
val vm = CustomVM(bytecode)
val result = vm.run()

All the actual logic — like which files to check or what APIs to call — lives inside the VM’s opcode handling, not in the exposed function.

Using Frida

Frida operates by:

• Hooking well-known methods (access, exists, System.loadLibrary, etc.)
• Overwriting return values of known functions
• Injecting scripts at runtime to modify app behavior

With Native or ART Code

Frida can hook:

Interceptor.attach(Module.findExportByName(null, 'access'), {
    onEnter: function(args) {
        // intercepts root check
    }
});

With Custom VM

The attacker sees:

val result = MiniVM(bytecode).run()

Unless they reverse engineer your entire interpreter and figure out what 0x01 means, they’re stuck.

Real-World Obfuscation practice

To make it even worse for attackers, you can:

• Encrypt the bytecode and decrypt it at runtime
• Inject dead instructions (NOOP, JUNK) to confuse disassemblers
• Use runtime mutation, where opcodes change meaning based on a session key

Performance and Maintenance

It is less performant and difficult to maintain the interpreter code for different architecture.

But for security-critical apps — like banking, fintech, or secure messaging — these trade-offs are worth it.

Thoughts

Running your root detection logic in ART or even in native code is no longer enough. This technique, known as the Pseudo VM technique, is used by malware to evade detection by antivirus tools. We can adopt the similar approach, recently Google Play integrity started using the similar approach with libpairipcore.so.

Using a custom virtual machine forces reverse engineers to:

• Reverse-engineer a new instruction set
• Dump and decode opaque bytecode
• Spend way more time understanding what the app is doing

Building a lightweight VM to keep those checks will definitely make the app harder to reverse.

Next Up: Will be building a Custom VM.I’ll be posting a follow-up with source code and integration steps.