// Cybersecurity clarified.

Self-Modifying Code for Windows x64

2022-05-29   Dawid Farbaniec
A program that modifies its code while opened under debugger will create interest. This technique is based on method where the program is replacing its instructions during execution.

First, a little theory. The compiler, or if we program in Assembly, the assembler (lowercase is a tool, with uppercase is a language) builds the source code into an executable file (e.g. * .exe). Processor instructions in the * .exe file are represented by machine code. The instructions and their operands are encoded as a sequence of bytes (in simple words: numbers).

The initial situation is that after starting the program, the processor instructions are stored in the memory as machine code (bytes). The way of translating instructions from mnemonics (eg ADD, MOV or XOR) into their operation codes (opcodes) is described in detail in the documentation for x86/x64 processors. In short, modifying the application code in memory is simply replacing the opcodes.

For example, the instruction xor rcx, rcx (clearing the counter register RCX) is encoded as a sequence of bytes 48 33 C9. To modify the code in memory, these bytes must be replaced. For example, setting them to 90 90 90 will cause the new instructions to be three NOPs.

At this stage, an attempt to overwrite the instruction will result in an Exception Access Violation error. I am already explaining why this is so. The specified blocks of memory have the appropriate attributes set. For example, data sections cannot be executed by default. A section of code cannot be written to by default.

So the next step will be to learn about the function that will allow you to change the attributes of the memory pages in the running application. This is a function from the Windows API called VirtualProtect. Its syntax is as follows:
BOOL VirtualProtect(

    LPVOID lpAddress,
    SIZE_T dwSize,
    DWORD flNewProtect,
    PDWORD lpflOldProtect);

Self-Modifying Code in MASM x64

A simple example for better understanding is provided below. You can see how this code works under the debugger in the video below.

Build tutorial:
x64 Assembly Project in Visual Studio 2022
extrn ExitProcess : proc

extrn VirtualProtect : proc
oldProtect dd 0
Main proc
;zmiana atrybutów bloku pamięci na:
;wykonanie, odczyt i zapis
sub rsp, 28h
mov r9, offset oldProtect
mov rdx, 15h
mov rcx, offset _mutable
call VirtualProtect
add rsp, 28h
;wpisanie do rejestru RBX adresu etykiety _mutable
mov rbx, offset _mutable
;nadpisanie wskazywanych instrukcji czterema rozkazami NOP
mov dword ptr [rbx+011h], 090909090h
;poniżej instrukcje do nadpisania
xor rcx, rcx
sub rsp, 28h  
xor rcx, rcx    
call ExitProcess  
Main endp

EICAR Also Modifies Its Code

Eicar file is a harmless .com program for MS-DOS. It is used to test the anti-virus software. EICAR stands for European Institute for Computer Anti-Virus Research. This institute is an organization working on the development of techniques for detecting and fighting not only computer viruses, but also malicious software. There born idea to create a file that would be recognized as a computer virus, but would not perform any malicious activities. This type of file was found to be of great help in testing antivirus software. With the help of this small file, each user can check the operation of the installed antivirus. The test methods can vary. You can try to download this file from the Internet. Downloading can be done normally or via encrypted traffic (HTTPS). It is also worth checking if the antivirus detects the file in real time (access attempt). Another idea is to verify whether the Eicar test file packed in a ZIP archive will also be detected or not. The Eicar file size is 68 bytes. In order to reduce the possibility of using this file for malicious purposes, a limit has been introduced that limits the number of bytes to a maximum of 128. It works by displaying the following text on the screen: EICAR-STANDARD-ANTIVIRUS-TEST-FILE! The content of the Eicar file has been designed to contain only printable ASCII characters with no spaces or other unusual characters.

Eicar shows the following code under the disassembler.
;this is file disassembly

pop ax
xor ax, 214Fh
push ax
and ax, 4140h
push ax
pop bx
xor al, 5Ch
push ax
pop dx
pop ax
xor ax, 2834h
push ax
pop si
sub [bx], si
inc bx
inc bx
sub [bx], si
jnl 0140h
int 21h
int 20h

Out of curiosity, I recommend to follow the execution of this program under the debugger, but note: this is an MS-DOS program.


Advanced Micro Devices Inc., 2017 – AMD64 Architecture Programmer's Manual
Intel Corporation, 2019 – Intel 64 and IA-32 Architectures Software Developer's Manual [access: 2020-07-28]