a. Let’s assume you’re debugging this code
and you reached the point P2 inside the
procedure sub_12 (marked in the code
by a yellow arrow). Further, let’s
assume that the content of the stack at
that point is shown in the figure beside.
What is the return address (in hex) based on the shown stack frame?
Which instruction pushed the return address into the stack?
b. Based on the values in the stack frame shown above, what would be the return values
of sub_12?
c. If you continue debugging this code until you reach the point P1 in the main procedure.
What would be the value (in hex) at the top of the stack?
What would be the value of EBP at point P2?
d. Which calling convention is used for sub_12? Which one is used for sub_48? Justify
• How many local variables in
sub_hw?
• How many arguments for
sub_hw?
Problem 3: Putting it together (40%)
The simple encryption algorithm in the following sub_enc procedure was extracted from a
memory dump of an infected machine. It was used by a malware to encrypt some information.
The analysis team managed to retrieve the encrypted message ctxt. You need to modify the
algorithm to reverse the encryption process and decrypt the message.
For simplicity, I simplified the algorithm for you and put it in a MASM project. The code shown
below calls the encryption algorithm with the dummy string argument, ptxt. I added this
string for you so you can see how the encryption is done but it has nothing to do with the
encrypted message. You need to decrypt ctxt. Here is what you need to do for this problem.
a. Modify the main procedure to call the procedure sub_dec instead of sub_enc and
pass the list ctxt instead of ptxt.
b. Modify the sub_dec procedure to perform the decryption. It already has the code to
Sample Solution