Next Spaceship

Anti-Debug with Trap Flag Register

| Comments

Let’s play an interesting Crack-me — muckis’s crackme #2.

When you run the crack-me with a debugger, say OllyDbg, the program pops up a message box saying “What the hell are you doing in my app with a debugger?” and then exists, but when you run the crack-me without a debugger, just double-click on it, it runs normally.

How does the crack-me do this? Let’s look the disassembling code:

Address   Hex dump                     Command                                Comments
00401000  /.  E8 24000000              CALL 00401029
00401005  |.  8B4C24 0C                MOV ECX,DWORD PTR SS:[ARG.3]
00401009  |.  C701 17000100            MOV DWORD PTR DS:[ECX],10017
0040100F  |.  C781 B8000000 00000000   MOV DWORD PTR DS:[ECX+0B8],0
00401019  |.  31C0                     XOR EAX,EAX
0040101B  |.  8941 14                  MOV DWORD PTR DS:[ECX+14],EAX
0040101E  |.  8941 18                  MOV DWORD PTR DS:[ECX+18],EAX
00401021  |.  806A 00 E8               SUB BYTE PTR DS:[EDX],0E8
00401025  |.  33C0                     XOR EAX,EAX
00401027  |.  33DB                     XOR EBX,EBX
00401029  |$  68 60104000              PUSH 00401060
0040102E  |.  64:FF35 00000000         PUSH DWORD PTR FS:[0]
00401035  |.  64:8925 00000000         MOV DWORD PTR FS:[0],ESP               ; Installs SE handler 401060
0040103C  |.  9C                       PUSHFD
0040103D  |.  813424 54010000          XOR DWORD PTR SS:[ESP],00000154
00401044  |.  9D                       POPFD
00401045  |.  6A 30                    PUSH 30                                ; /Type = MB_OK|MB_ICONEXCLAMATION|MB_DEFBUTTON1|MB_APPLMODAL
00401047  |.  68 00604000              PUSH OFFSET 00406000                   ; |Caption = "crackme2"
0040104C  |.  68 7A604000              PUSH OFFSET 0040607A                   ; |Text = "What the hell are you doing in my app with a debugger?"
00401051  |.  6A 00                    PUSH 0                                 ; |hOwner = NULL
00401053  |.  E8 9C030000              CALL          ; \USER32.MessageBoxA
00401058  |.  6A 00                    PUSH 0                                 ; /ExitCode = 0
0040105A  |.  E8 A7030000              CALL        ; \KERNEL32.ExitProcess
0040105F  \.  C3                       RETN
00401060  /$  64:8F05 00000000         POP DWORD PTR FS:[0]                   ; SE handling routine
00401067  |.  83C4 04                  ADD ESP,4
0040106A  |.  6A 00                    PUSH 0                                 ; /ModuleName = NULL
0040106C  |.  E8 A1030000              CALL   ; \KERNEL32.GetModuleHandleA
00401071  |.  A3 35604000              MOV DWORD PTR DS:[406035],EAX

Note this line:

00401035  |.  64:8925 00000000         MOV DWORD PTR FS:[0],ESP               ; Installs SE handler 401060

It installs a structured exception handler. The definition of structured exception can be found here: http://msdn.microsoft.com/en-us/library/windows/desktop/ms680657(v=vs.85).aspx.

An exception is an event that occurs during the execution of a program, and requires the execution of code outside the normal flow of control. There are two kinds of exceptions: hardware exceptions and software exceptions. Hardware exceptions are initiated by the CPU. They can result from the execution of certain instruction sequences, such as division by zero or an attempt to access an invalid memory address. Software exceptions are initiated explicitly by applications or the operating system. For example, the system can detect when an invalid parameter value is specified. Structured exception handling is a mechanism for handling both hardware and software exceptions. Therefore, your code will handle hardware and software exceptions identically. Structured exception handling enables you to have complete control over the handling of exceptions, provides support for debuggers, and is usable across all programming languages and machines. Vectored exception handling is an extension to structured exception handling. The system also supports termination handling, which enables you to ensure that whenever a guarded body of code is executed, a specified block of termination code is also executed. The termination code is executed regardless of how the flow of control leaves the guarded body. For example, a termination handler can guarantee that clean-up tasks are performed even if an exception or some other error occurs while the guarded body of code is being executed.

Then,

0040103C  |.  9C                       PUSHFD
0040103D  |.  813424 54010000          XOR DWORD PTR SS:[ESP],00000154
00401044  |.  9D                       POPFD

The TF (Trap Flag) is set to 1, according to the Intel x86 FLAGS registers form:

Intel x86 FLAGS register
Bit #	Abbreviation	Description	Category[1]
FLAGS
0	CF	Carry flag	S
1	1	Reserved	
2	PF	Parity flag	S
3	0	Reserved	
4	AF	Adjust flag	S
5	0	Reserved	
6	ZF	Zero flag	S
7	SF	Sign flag	S
8	TF	Trap flag (single step)	X
9	IF	Interrupt enable flag	C
10	DF	Direction flag	C
11	OF	Overflow flag	S
12, 13	1,1 / IOPL	I/O privilege level (286+ only) always 1 on 8086 and 186	X
14	1 / NT	Nested task flag (286+ only) always 1 on 8086 and 186	X
15	1 on 8086 and 186, should be 0 above	Reserved	
EFLAGS
16	RF	Resume flag (386+ only)	X
17	VM	Virtual 8086 mode flag (386+ only)	X
18	AC	Alignment check (486SX+ only)	X
19	VIF	Virtual interrupt flag (Pentium+)	X
20	VIP	Virtual interrupt pending (Pentium+)	X
21	ID	Able to use CPUID instruction (Pentium+)	X
22	0	Reserved	
23	0	Reserved	
24	0	Reserved	
25	0	Reserved	
26	0	Reserved	
27	0	Reserved	
28	0	Reserved	
29	0	Reserved	
30	0	Reserved	
31	0	Reserved	
RFLAGS
32-63	0	Reserved

That means when the program is running in normal mode, it runs one step (0×00401045) and then due to the Trap Flag it handles the exception by the Structured Exception Handler and runs into address 0×00401060, which is the right entrance of the program. But if the program is running in debug mode, it runs one step (0×00401045) and it continues to execute the following instructions, because in debug mode, the Trap Flag is ignored. So the program enters the wrong way and exists.

Even though this method to anti-debug is very weak, it’s practical, simple and classical. Besides it can be used with other methods together to make the debugging of your program more difficult.

Comments