Shellcode: Detection between Windows/Linux/BSD on x86 architecture


While writing simple Linux/BSD shellcodes, I had a quick look through a FreeBSD/Linux bind shell written in 2002 by a talented coder who went by the pseudonym Z0MBiE. Some of you will know he wrote a lot of cool stuff back in the day.

Since most of the syscall numbers differ on Linux/BSD, he devised a clever way of detecting between the 2 using sys_close system call with an invalid handle. They both use the same call number (6) but return different error codes.

B3mB4m left a comment referencing a post from 2013 which also attempts to address the problem of detecting between Linux and Windows.

What I show here is really just the result of some hours work and not extensive research into how many ways it can be done because there are probably a number of ways to do it. The codes are for demonstration only, they’re unoptimized and unlikely to be updated anytime soon but feel free to leave comments on alternative methods. 🙂

Detection of CPU Mode

The steps are similar to what was discussed by some already but I check stack pointer to make distinction between 32-bit versions of windows and linux when the GS register is non-negative.

  1. Is our application executing 32-bit or 64-bit code? (determined by REX prefix)
  2. Are we running under 64-bit versions of Windows, Linux, BSD? (determined by sys_close and syscall, GS and SP registers tested for 32-bit windows)
  3. Are we Linux or BSD? (determined by sys_close and int 0x80)

To jump to 32-bit code, you could use

    xor    eax, eax
    dec    eax
    js     x32

To jump to 64-bit code, you could use

    xor    eax, eax
    inc    eax
    jz     x64

Windows native or emulated?

For a 32-bit native application, GS should always be zero so here’s a function that returns TRUE for native else FALSE

    xor    eax, eax
    mov    ax, gs
    cmp    eax, 1
    sbb    eax, eax
    neg    eax

I’m unaware of a reliable method to test for 32-bit native code on Linux or BSD systems. Perhaps there’s a way to perform checks on segment registers but I’ve no idea how reliable that would be.

Windows, Linux or BSD?

Initially, I had some rough ideas; manipulation of EFLAGS/RFLAGS, FPU instructions, value of segment registers, the contents of them.

The manipulation of FLAGS didn’t result in anything and neither did investigation of FPU control word (although MSVC and GNU C do set it differently by default)

Windows Segments

Unfortunately my current computer would not be capable of running Win8,2012 or Win10 so only Win7 was tested.

Windows 7 x86 PE32
cs=0x1B ds=0x23 es=0x23 fs=0x3B gs=0x00 ss=0x23 sp=0025F928
Windows 7 x64 PE32
cs=0x23 ds=0x2B es=0x2B fs=0x53 gs=0x2B ss=0x2B sp=0033F9C0

Windows 7 x64 PE64
cs=0x33 ds=0x2B es=0x2B fs=0x53 gs=0x2B ss=0x2B sp=000000000020FB48

As you can see, the stack pointer for both modes are well below a signed 32/64-bit value.
Now look at BSD/Linux values.


Only native ELF files were tested.

OpenBSD x86 ELF32
cs=0x2B ds=0x33 es=0x33 fs=0x5B gs=0x63 ss=0x33 sp=0xcf7c502c

OpenBSD x64 ELF64
cs=0x2B ds=0x23 es=0x23 fs=0x23 gs=0x23 ss=0x23 sp=0x7f7fffff7fe8

Compared with Windows, the stack pointer as 32-bit value is signed, but not always.


FreeBSD x64 ELF32
cs=0x33 ds=0x3B es=0x3B fs=0x13 gs=0x1B ss=0x3B sp=0xffffda1c

FreeBSD x64 ELF64
cs=0x43 ds=0x3B es=0x3B fs=0x13 gs=0x1B ss=0x3B sp=0x7fffffffe8b8

Again, we can see that the 32-bit stack pointer is signed but this is not a sure thing.


Debian x86 ELF32
cs=0x73 ds=0x7B es=0x7B fs=0x00 gs=0x33 ss=0x7B sp=0xbfe7b97c

Debian x64 ELF32
cs=0x23 ds=0x2B es=0x2B fs=0x00 fs=0x63 ss=0x2B sp=0xffa66dbc
Debian x64 ELF64
cs=0x33 ds=0x00 es=0x00 fs=0x00 gs=0x00 ss=0x2B sp=0x7ffc88e3e048

*NIX or Windows?

Based on the results above, I initially thought testing the stack pointer for signedness would make the distinction between Windows or Linux/BSD and here’s code just to illustrate the idea.

; *************************
; int is_nix(void);
; 0=Windows x86/x64
; 1=BSD x86/x64 or Linux x86/x64
; *************************
    bits   32
    push   esp              ; save esp/rsp
    pop    eax              ; pop in eax/rax
    cdq                     ; edx=(eax < 0) ? -1 : 0
    neg    edx              ; 0=windows, 1=bsd or linux
    xchg   eax, edx

This is okay except the 32-bits of SP on Linux/BSD isn’t always signed so it might make a nice random decision function if nothing else.

; *************************
; int is_nix(void);
; 0=Windows x86/x64
; 1=BSD x86/x64 or Linux x86/x64
; *************************
    bits   32
    push   esp
    pop    eax
    shr    eax, 24
    setnz  al

This is a bit more reliable than the first version since by default, the stack pointer should be less than 1MB. It can obviously be specified using /STACK parameter of MSVC linker but this is what I thought works reasonably well to detect between 32-bit versions of windows and Linux/BSD.

What was easier (at least on Windows 7 64-bit) was detection of 64-bit OS when using syscall.
Using the original method documented by Z0MBiE/29a for detecting FreeBSD/Linux, I was surprised that the 64-bit version will also work with Windows 7 and is maybe a potential solution for more recent versions of windows.

On 64-bit Windows 7, error returned is 0xC0000005 or Access Violation Error
On 64-bit Linux, error returned is 0xFFFFFFF2 or -14
On 64-bit BSD, error returned is 9

%define WIN64_NATIVE 0
%define LIN64_NATIVE 1
%define BSD64_NATIVE 2

%define WIN32_NATIVE 3
%define WIN32_EMULAT 4
%define LIN32_EMUNAT 5
%define BSD32_EMUNAT 6

    bits   32
    ; first, determine if we're in 32 or 64-bit mode
    push   6                ; sys_close on linux/bsd
    pop    eax              ; rax/eax=sys_close
    cdq                     ; rdx/edx=0
    dec    edx              ; ignored if 64-bit mode
    js     x32
    ; see what 64-bit OS we're on
    push   edx              ; save rdx/edx
    pop    edi              ; restore to rdi/edi
    syscall                 ; 
    xor    edx, edx         ; edx=WIN64_NATIVE
    ; win64 native?
    cmp    eax, 0xC0000005  ; Access Violation?
    jz     ex_ga
    inc    dl               ; edx=LIN64_NATIVE
    test   eax, eax         
    jl     ex_ga           
    mov    dl, BSD64_NATIVE
    xchg   eax, edx
    inc    edx
    mov    dl, WIN32_NATIVE
    ; if gs is zero, we're win32 native
    xor    ecx, ecx
    mov    cx, gs
    jecxz  ex_ga
    ; if fs isn't zero, we might be BSD or Linux 
    ; check SP instead
    ; just because eax would result in zero, doesn't
    ; necessarily mean we are on windows...
    push   esp
    pop    eax
    mov    dl, WIN32_EMULAT
    shr    eax, 24
    jz     ex_ga
    ; we could say fs being zero is linux
    ; but better to check with sys_close
    ;mov    cx, fs
    ;test   ecx, ecx
    ;jnz    ex_ga
    ; are we linux or bsd?
    ; no reliable way to determine if emulated
    ; or not since openbsd uses same value for
    ; some segments
    push   -1
    int    0x80
    push   LIN32_EMUNAT
    pop    edx
    test   eax, eax
    pop    eax
    jl     ex_ga
    mov    dl, BSD32_EMUNAT
    jmp    ex_ga


There would have to be smarter ways of detecting operating systems running on the x86 architecture using assembly but would obviously result in more code. I’ve just presented a few bits of code that might be of interest to some. 🙂

This entry was posted in assembly, bsd, freebsd, linux, programming, security, shellcode, windows and tagged , , , , , , , . Bookmark the permalink.

4 Responses to Shellcode: Detection between Windows/Linux/BSD on x86 architecture

  1. B3mB4m says:

    Well for x86(tested on linux) has smartest solution by p.ferrie.I named it “merlin” and using within my project to making cross-platform shellcodes lol,



  2. Pingback: Shellcode: Execute command for x32/x64 Linux / Windows | modexp

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s