[ROT-N + Shift-N + XOR-N] Shellcode Encoder | Linux x86


According to English dictionary, encode is converting something, such as a body of information from one system of communications to another; especially: to convert a message into code. This blog post will combine three basic encoding operations (more on that later) to encode sample shellcode and then decode/execute it to demonstrate the concept.


In this section, we’ll create 3-phase encoder that takes one byte of given shellcode at a time, mangle it and then produce an encoded word (2 bytes). The following will describe what each phase will do. Phase I (ROT-N), in this phase we add number N to shellcode byte. Phase II (Shift-N), in this phase we shift stdout from phase one to left by N. Phase III (XOR-N), here we just XOR stdout from phase two to get the final encoded word. Before we start working on the encoder, let’s write shellcode that spawn shell to test with.

Let’s compile execve() code block and then dump shellcode

We’ll create python script that takes ROT, Shift, and XOR as user input and print original and encoded shellcode, please note you still need to change shellcode inside the script to your liking. For this post, we’ll use the one created earlier.  It’s worth noting that the script shift operation number can only be anything between 1 and 8 bits due to encoded_shellcode size (word) in the decoder. Also, we’ve added XOR input value to the end of the encoded shellcode as terminator, so if XOR operation in the decoder results in zero that means hey stop decoding.

Generate encoded shellcode


Now comes the decoding part, let’s write code that basically takes encoded shellcode produced earlier , decode it in reverse order and then execute

Compile and dump shellcode

Add it to final exploit

Demo time

Closing Thoughts

I believe learning shellcode encoding is vital to exploit development, in order to evade modern anti-viruses and/or intrusion detection/prevention systems. Please note all of the code in this post were tested on Ubuntu 12.04.5 LTS. Feel free to contact me for questions using the comment section below or just tweet me @ihack4falafel .

This blog post has been created for completing the requirements of the SecurityTube Linux
Assembly Expert certification:

Student ID:    SLAE-1115

Github Repo:

Leave a Reply

Your email address will not be published. Required fields are marked *