Disabling Windows XP Detection of Devices on Serial Ports (serenum)

Or: Stopping Windows from freaking out ancient or non-standard serial peripherals...

Recently, I ran into a problem where an XP box, during bootup, was freaking out a peripheral attached to the box via a COM port. I won't get into the details of why the device is so sensitive, but I'll simply say that the usual RS232 flow control lines were not used for flow control, but instead to do some very delicate operations. During bootup, XP was pulsing the DTR and RTS lines, perhaps looking for a serial mouse or modem or to do something else. We didn't know why or how to make it stop. This short article describes the cause and how to fix it.

Windows detecting a serial mouse incorrectly?

My first thought was that XP was looking for a serial mouse. That seems to be a common problem with old serial devices attached to an XP and 2000 boxen, especially where those devices are continuously outputting data. XP often detects them as serial mice incorrectly. However, no such device showed up in the device manager and we didn't notice any weirdness associated with the mouse, which I would have expected if XP was interpreting random incoming data as a serial mouse—the mouse should be jumping around on the screen or acting weird. But it wasn't.

Serial mice in NT/2000/XP can be detected by the NTDETECT.COM program, run by the NTLDR (NT Loader) before the kernel has taken control. In NT4, this could be disabled by passing the /NOSERIALMICE flag in boot.ini, and in 2000/XP by the /FASTDETECT flag.

/fastdetect:comnumber
This switch turns off serial and bus mouse detection in the Ntdetect.com file for the specified port. Use this switch if you have a component other than a mouse that is attached to a serial port during the startup process. For example, type /fastdetect:comnumber, where number is the number of the serial port. Ports may be separated with commas to turn off more than one port. If you use /fastdetect, and you do not specify a communications port, serial mouse detection is turned off on all communications ports.

Note In earlier versions of Windows, including Windows NT 4.0, this switch was named /noserialmice.

Source

On this XP box, by default, we always passed in /FASTDETECT in boot.ini, so this wasn't the problem. Something else had to be causing the behavior I was seeing.

Disabling enumeration in the Serenum service.

The only other method I could find to possibly stop Windows from probing devices on this serial port was to monkey with the Serenum service. Apparently in 2000/XP, code was added to check a registry value called SkipEnumerations in a specific key on a per-port basis. Setting this value would keep Windows from probing devices on this COM port.

Additional Features in Serenum.sys

Windows supports a variety of multiport serial adapters through third-party drivers from manufacturers. In the case of high-capacity adapters (16 ports or more), boot time can be delayed on Windows 2000 when Serenum.sys scans each port for attached devices. To improve the boot time and the related user experience, expanded control over the serial port scanning has been added to Serenum.sys in Windows XP. These options can be controlled by a new REG_DWORD registry value, SkipEnumerations, as described in this article.

Serenum Features in Windows XP
>Windows XP addresses issues with the Serenum driver and reduces the boot time required to start machines, even with high-capacity adapters (16 ports or more) attached. An additional registry key is also defined in Windows XP, allowing hardware vendors to control Serenum.sys, forcing it to skip port scanning during system startup. This feature is achieved by setting the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\%Enumerator%\%Device_ID%\%Instance_ID%\Device Parameters\SkipEnumerations

The use of SkipEnumerations in the Device Parameters key must be used on a per-port basis. There is no global option to change the behavior of all ports at once.
The REG_DWORD value name, SkipEnumerations, can contain the following data:

Zero or non-existent: Default Serenum behavior; each port will be scanned during every system boot.
For example: SkipEnumerations\REG_DWORD\0x00000000 (0)

0xffffffff: Serenum will never scan the ports, either automatically or through user-initiated scans in Device Manager or Hardware Wizard. Any device attached to a port with this value in the registry will need to be installed manually.
For example: SkipEnumerations\REG_DWORD\0xffffffff (4294967295)

If any other value is used, Serenum will skip that number of scans (boots or user-initiated). The successive scan will behave normally (similar to the setting of zero). Once the value goes to zero, it will stay at zero until Serenum.sys is unloaded and reloaded. At this point, it will be reset to the original registry value.
For example: SkipEnumerations\REG_DWORD\0x00000003 (3)

In this example, the data value is set to 3. Serenum will skip 3 scans. After the 3 scans, it will perform a normal scan.

The Windows XP version of Serenum.sys has been enhanced to automatically speed up the boot process when high-capacity multiport serial adapters are installed. With the addition of SkipEnumerations support in Serenum.sys, vendors have greater control over how COM ports behave during the boot process. Vendors should examine these new registry options and evaluate whether the user experience can be improved.

Source

The technique was clear on everything except for exactly what key to put the SkipEnumerations value in. What key was this?
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\%Enumerator%\%Device_ID%\%Instance_ID%\Device Parameters

What were the values of %Enumerator%, %Device_ID%, and %Instance_ID% supposed to be? I found these values by looking in the key following key for Serenum.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\serenum\Enum

In this key there is one value per COM port on the system. Each value had a name of just a single digit, starting at 0. The below image shows the registry editor with the Enum key selected and the various values that it contains.

The values named 0 through 3 corresponded to the COM ports on the box. The box in question had one on-board serial port, two serial ports on a PCI card, and one USB-to-serial adapter for a 4th port. Notice the values of 0-3—these values are the %Enumerator%\%Device_ID%\%Instance_ID% that I needed to identify where to put the SkipEnumerations value. So, looking at 0, the value was
PCI\VEN_1407&DEV_0110&SUBSYS_00000000&REV_00\4&2af9ed5&0&00F0

Breaking this down, the %Enumerator% is PCI, the %Device_ID% is "VEN_1407&DEV_0110&SUBSYS_00000000&REV_00", and the %Instance_ID% is 4&2af9ed5&0&00F0. This means if I wanted to disable probing on this port, I needed to create the SkipEnumerations value in the following key.
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1407&DEV_0110&SUBSYS_00000000&REV_00\4&2af9ed5&0&00F0\Device Parameters

As an aside, looking at the enumerator value of PCI, I knew that this corresponded to one of the serial ports on the PCI card I had, but I didn't know which one, because there are two. The numbers in the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\serenum\Enum don't correspond to the COM port numbers we are used to dealing with as users, like COM1, COM4, COM8, etc. I'll show how to identify the COM port number associated with this serial port in a minute. Before that, look at the 2 value in the serenum\Enum key:
ACPI\PNP0501\1
Because the enumerator here was ACPI, that indicated that the port was on the motherboard. The other value, 3, was for the USB port. It's enumerator is (surprise) USB. Easy enough. Now how could I figure out which COM port number I needed to set the SkipEnumerations value in? I only knew that COM1 was the port on which the probing was causing problems.

To find which COM port number I needed, I looked at all of the "Device Parameters" keys identified numbered values in the serenum\Enum key.

0: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1407&DEV_0110&SUBSYS_00000000&REV_00\4&2af9ed5&0&00F0\Device Parameters

1: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\PCI\VEN_1407&DEV_0111&SUBSYS_00000000&REV_00\4&2af9ed5&0&01F0\Device Parameters

2: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\ACPI\PNP0501\1\Device Parameters

3: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USB\Vid_067b&Pid_2303\5&287ed76f&0&2\Device Parameters

Each of these keys had a value in them named "PortName", which had the COM port number. The following image shows the first of these keys as an example. Notice the PortName value.

So using this information, I figured out that COM1 was the on-board serial port and that I needed to create the SkipEnumerations value in the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\ACPI\PNP0501\1\Device Parameters key. Doing this stopped Windows from pulsing the DTR and RTS lines during bootup, which caused my device to freak out.

Summation

The XP box in question already skipped detecting a serial mouse via the flags in boot.ini. The offending service was Serenum, which in XP respects the SkipEnumerations registry value to avoid probing particular serial ports for attached devices.