Fixing an endless loop bug in the Xerox Unix printing Package 4.50.29 xsf tool
I got annoyed today by a Xerox printer that was driven via CUPS, where the queue hung forever from time to time.
So I investigated where it hung. Turns out, it got stuck in this loop:
int __cdecl SendLprCmd(int a1, int a2, int a3, int a4, int fd) { int v5; // ST1C_4@21 signed int v7; // [sp+10h] [bp-98h]@10 int v8; // [sp+14h] [bp-94h]@17 char s; // [sp+20h] [bp-88h]@2 switch ( a1 ) { case 1: sprintf(&s, "\x01%s\n", a2); break; case 2: sprintf(&s, "\x02%s\n", a2); break; case 7: sprintf(&s, "\x02%i %s\n", a3, a2); break; case 8: sprintf(&s, "\x03%i %s\n", a3, a2); break; case 3: sprintf(&s, "\x03%s\n", a2); break; case 4: sprintf(&s, "\x04%s\n", a2); break; case 0: s = 0; break; case 5: case 6: break; } RdBufI = 0; RdBuf = 0; if ( a1 ) v7 = strlen(&s); else v7 = 1; if ( a1 == 1 ) { write(fd, &s, v7); } else { while ( !pPN->byte38 ) { if ( (pPN->byte38 || write(fd, &s, v7) == v7) && (v8 = GetLprResponse(a4, fd), v8 >= 0) && v8 != 1 ) { if ( !v8 ) return 0; } else { ReturnPrinterError(14, fd); } v5 = rand(); sleep(v5 % 16 + 5); } } return 0; }
Let’s have a look at ReturnPrinterError function:
int __cdecl ReturnPrinterError(int errcode, int fd) { int *v2; // eax@2 char v3; // ST14_1@2 if ( errcode == 16 ) { v2 = __errno_location(); v3 = (unsigned int)strerror(*v2); WrStatus(pPN->fdError, "ERROR: %s\n", v3); } else { WrStatus(pPN->fdError, "ERROR: %s\n", (unsigned int)SocketErrorMsg[errcode]); } if ( fd > 0 ) close(fd); return errcode; }
So file descriptor gets closed in there and becomes invalid. Therefore write() call can never suceed again ever after, so we got stuck in an endless loop on error.
Upon calls of SendLprCmd, return value gets checked (0=success).
Turns out this function never returns anything different than 0. So conclusion: Routine shall return something != 0 in EAX if error occurs.
ReturnPRinterError always returns error code>0, so EAX gets set fine by this routine.
Let’s have a look at the ASM code of the call then:
.text:08049E9B loc_8049E9B: ; CODE XREF: SendLprCmd+17Dj .text:08049E9B ; SendLprCmd+19Dj ... .text:08049E9B sub esp, 8 .text:08049E9E push [ebp+fd] ; fd .text:08049EA1 push 0Eh ; int .text:08049EA3 call ReturnPrinterError .text:08049EA8 add esp, 10h .text:08049EAB jmp short loc_8049EB8 .text:08049EAD ; --------------------------------------------------------------------------- .text:08049EAD .text:08049EAD loc_8049EAD: ; CODE XREF: SendLprCmd+1A8j .text:08049EAD cmp [ebp+var_94], 0 .text:08049EB4 jnz short loc_8049EB8 .text:08049EB6 jmp short loc_8049F0F .text:08049EB8 ; --------------------------------------------------------------------------- .text:08049EB8 .text:08049EB8 loc_8049EB8: ; CODE XREF: SendLprCmd+1BAj .text:08049EB8 ; SendLprCmd+1C3j .text:08049EB8 sub esp, 0Ch .text:08049EBB sub esp, 4 .text:08049EBE call _rand .text:08049EC3 add esp, 4
So instead of jumping back to rand sleep() routine, just let it bail out to 08049F14:
.text:08049F0F loc_8049F0F: ; CODE XREF: SendLprCmd+13Bj .text:08049F0F ; SendLprCmd+14Cj ... .text:08049F0F mov eax, 0 .text:08049F14 leave .text:08049F15 retn .text:08049F15 SendLprCmd endp
JMP to rand() is EB 0B, so change it to new offset EB 67 @08049EAB:
08049E9C EC 08 FF 75 18 6A 0E E8 D5 FC FF FF 83 C4 10 EB 08049EAC 0B 83 BD 6C FF FF FF 00 75 02 EB 57 83 EC 0C 83 --> 08049E9C EC 08 FF 75 18 6A 0E E8 D5 FC FF FF 83 C4 10 EB 08049EAC 67 83 BD 6C FF FF FF 00 75 02 EB 57 83 EC 0C 83
So patch 0B @ file offset 1EAB to 67, problem fixed.
At least, bug seems to be fixed now, will have to look if there is an update to this buggy version of the printing system that runs on the target machine, but as it’s a productive machine, I first need a timeslot to test.
System programmer