Trace vector decoder is a system that uses a microprocessor's trace mode to decode encrypted code just-in-time before it is executed and possibly re-encrypt it after the execution. It can be used to enforce copy protections for some computer systems.

Trace Vector in Motorola 68000

As an example, Motorola 68000

Motorola 68000

The Motorola 68000 is a 16/32-bit CISC microprocessor core designed and marketed by Freescale Semiconductor...

 has a trace mode where a trace exception vector is executed before each instruction in the main program. The processor automatically changes execution to the trace exception vector before executing any instruction from the main program. The trace exception vector decodes the instruction that will be executed after the exception vector. The next time trace exception happens the old decoded location may possibly be re-encrypted.

Following code snippet is an example of a program initializing a trace exception routine.

	MOVEM.L	Stack,D0-D7/A0-A6      ; Initialize registers

Stack	MOVE.L	#$4E730000,-(SP)       ; Start loading trace exception

	MOVE.L	#$00000010,-(SP)       ; vector into stack

	MOVE.L	#$0004DDB9,-(SP)

	MOVE.L	#$BD96BDAE,-(SP)

	MOVE.L	#$B386B586,-(SP)

	MOVE.L	#$D046D246,-(SP)

	MOVE.L	#$0246A71F,-(SP)

	MOVE.L	#$00023C17,-(SP)

	MOVE.W	#$2C6F,-(SP)

	MOVE.L	SP,($24).W             ; Set trace exception vector

	ORI.W	#$A71F,SR              ; Enter trace mode

        NOP

        ; Trace vector happens now for the first time.

        ; Code after this line is encrypted.

A disassembly of the trace exception vector that is loaded on the stack:

TraceCode:

	MOVE.L	(2,SP),A6              ; Load return address from

                                       ; supervisor stack.

	MOVE.W	(SP),D6                ; Load condition codes of the main

                                       ; program.

	AND.W	#$A71F,D6

	ADD.W	D6,D0

	ADD.W	D6,D1

	EOR.L	D1,D6

	EOR.L	D2,D6

	EOR.L	D6,(A6)                ; Decrypt 8 bytes ahead in main

	EOR.L	D6,(4,A6)

	RTE                            ; Return from exception

Note that registers altered in the trace vector affect the main program that is being traced. Usually registers are pushed onto stack in any exception vector, because altering them would break the main program. However, purpose of this vector is to obfuscate the code against reverse engineering

Reverse engineering

Reverse engineering is the process of discovering the technological principles of a device, object, or system through analysis of its structure, function, and operation...

.

It should also be noted that condition code register (CCR) affects the decryption process. For example, an arithmetic operation in the main program having the 0 number as a result, will cause zero flag bit to be set in CCR. This will cause the value in (SP) to be changed in the trace vector. This is also done to obfuscate against reverse engineering.

Rob Northen copylock

Rob Northen copylock

Rob Northen Copylock is a copy-protection system to prevent disk duplication with standard floppy disk drives on the Amiga, Atari ST and PC-DOS platforms...

 has a trace vector decoder that was used on the Amiga

Amiga

The Amiga is a family of personal computers that was sold by Commodore in the 1980s and 1990s. The first model was launched in 1985 as a high-end home computer and became popular for its graphical, audio and multi-tasking abilities...

 and Atari ST

Atari ST

The Atari ST is a home/personal computer that was released by Atari Corporation in 1985 and commercially available from that summer into the early 1990s. The "ST" officially stands for "Sixteen/Thirty-two", which referred to the Motorola 68000's 16-bit external bus and 32-bit internals...

platforms.