Getting to the bottom of a BSOD

Rebecca's computer has started throwing BSOD recently. Blue screen of death (BSOD) errors have caused many people to throw up their hands in disgust at Windows countless thousands of times. Industries are built around fixing computers with BSOD. Fortunately, I worked front line technical support for several years and know a thing or two about these. Although, format and reinstalling windows is going to most likely fix the issue, it is akin to cutting off a persons leg for an ingrown toenail. Struggling to fix problems without reformatting and reinstalling has resulted in gaining more insight into the internals of the Windows OS and software in general.

So I could boot into safe mode with F8 while the computer started but, each time I tried to boot into normal mode, I would get BSOD and reboot. In safe mode you can go to control panel | system | Advanced | startup & recovery | settings and uncheck the box for Automatically Restart. This will allow you to read the BSOD message which was necessary for me to troubleshoot the problem. You see, unlike some people, computers will usually tell you what is wrong them and furthermore it is usually possible for you to fix computer problems.

The error message was 0x0000007e 0xc0000005 ... so really useful to a geek who can read hex and memorize windows kernel error codes. Each time a Windows XP computer BSOD crashes, a message gets put in the System Log, also you will get (unless you configured differently) a timestamped MiniDump binary file .dmp in Windows/Minidump folder.

The minidump file created from the BSOD is key to really getting at the root cause of the problem. But it is not really human readable (it can partially be read in a text editor but lots of binary data makes it laborious.) So you have to have windows debugging tools to extract the information from the file out to get to the bottom of the problem. http://www.microsoft.com/whdc/devtools/debugging/default.mspx Well, both Windows symbols and Debugging Tools to be precise. Without symbols the .dmp file will not reveal all its secrets.

To start I moved all unsigned drivers out of system32/drivers folder into a tmp folder called system32/drivers.unsigned

You can identify the unsigned drivers with a tool in safe mode called sigverif, just pick advanced and point the search to system32/drivers folder to get a list of unsigned drivers.

Then open an explorer window and move these files out of drivers and into drivers.unsigned.

Restart and you should be back in business and able to boot in windows normally, but some hardware or software will not work right until you figure out which one of your unsigned drivers is sick. This is where the .dmp files will zero in on what driver is to blame, the rest of the unsigned drivers are innocent bystanders to this one bad apple.

The culprit, vsdatant.sys, which using process explorer, is a file installed on Rebecca's system by Zone Alarm.


Microsoft (R) Windows Debugger Version 6.9.0003.113 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\WINDOWS\Minidump\Mini111508-03.dmp]
Mini Kernel Dump File: Only registers and stack trace are available

Symbol search path is: C:\WINDOWS\Symbols
Executable search path is:
Unable to load image ntoskrnl.exe, Win32 error 0n2
*** WARNING: Unable to verify timestamp for ntoskrnl.exe
Windows XP Kernel Version 2600 (Service Pack 3) UP Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS Personal
Kernel base = 0x804d7000 PsLoadedModuleList = 0x80554040
Debug session time: Sat Nov 15 05:55:48.000 2008 (GMT-7)
System Uptime: 0 days 0:00:33.562
Unable to load image ntoskrnl.exe, Win32 error 0n2
*** WARNING: Unable to verify timestamp for ntoskrnl.exe
Loading Kernel Symbols
...............................................................................................................................
Loading User Symbols
Loading unloaded module list
..
Unable to load image tcpip.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for tcpip.sys
ERROR: FindPlugIns 8007007b
*******************************************************************************
* *
* Bugcheck Analysis *
* *
*******************************************************************************

Use !analyze -v to get detailed debugging information.

BugCheck 1000007E, {c000001d, 89d6a024, ba597978, ba597674}

Unable to load image vsdatant.sys, Win32 error 0n2
*** WARNING: Unable to verify timestamp for vsdatant.sys
*** ERROR: Module load completed but symbols could not be loaded for vsdatant.sys
Probably caused by : vsdatant.sys ( vsdatant+450c1 )

Followup: MachineOwner
---------

What is still strange to me is I did not remove this driver from the system32/drivers folder. Here is a list of the drivers I removed from the folder:
11/15/2008 10:07 AM dir .
11/15/2008 10:07 AM dir ..
10/04/2006 07:42 PM 2,432 cdr4_xp.sys
10/04/2006 07:42 PM 2,560 cdralw2k.sys
11/18/2002 02:51 PM 377,358 cmaudio.sys
03/03/2008 05:25 PM 1,405,632 cmudax3.sys
11/11/2003 10:44 AM 333,600 ctdvda2k.sys
09/02/2003 11:25 AM 11,266 diag69xp.sys
01/31/2008 04:18 PM 9,216 FlashSys.sys
11/23/2002 07:36 PM 16,016 hpoipr07.sys
09/10/2003 11:36 PM 21,060 iviaspi.sys
09/17/2003 03:57 PM 8,440 LANPkt.sys
01/14/2008 01:15 PM 15,340 ndisrd.sys
09/19/2003 01:47 AM 10,368 pfc.sys
10/10/2006 02:06 PM 17,136 SSLDrv.sys
12/18/2004 07:32 PM 38,229 StMp3Rec.sys
08/07/2006 03:42 PM 20 SymRedir.cat
08/07/2006 03:42 PM 1,133 SymRedir.inf
07/10/2008 08:35 AM 32,000 usbaapl.sys
17 File(s) 2,301,806 bytes

So I still may have to add them back to system32/drivers, a few at a time, to see if I can zero in on the driver in the above list that caused the issue.

Update: Well, from the above list I narrowed it down to:
01/31/2008 04:18 PM 9,216 FlashSys.sys
All other files did not cause a BSOD when put back in system32/drivers folder.
I am not sure how to identify the source software that put FlashSys.sys into my drivers folder, but I suspect my SanDisk Cruzer Titanium USB 2 GB thumbdrive as the culprit.

Update 11/17/2008:
Well I again got a BSOD, so I removed all but
11/18/2002 02:51 PM 377,358 cmaudio.sys
03/03/2008 05:25 PM 1,405,632 cmudax3.sys
Which I need for sound and I am back in business again, but I am not sure which of the other files besides:
01/31/2008 04:18 PM 9,216 FlashSys.sys
is also causing BSOD on the system. I will probably need to find a way to track down the source software programs for each of these drivers.