Let's see how to set up and use a supplemental device queue. This queue will perform exactly the same function as the kernel-managed device queue for StartIo requests and so shows what must be happening behind the scenes in the kernel IoStartPacket call, etc. The aim is to serialize processing of IRPs in the AbcStartIo2 function. An IRP is put in the queue for processing using the AbcStartPacket2 routine. AbcStartNextPacket2 starts the processing of the next packet. Listing 16.8 shows the code for these routines.

These routines also handle the cancelling of IRPs correctly using the Cancel spin lock and cancel routines.

Declare a KDEVICE_QUEUE field somewhere in nonpaged memory. In this case, it is in the device extension in a field called AbcIrpQueue. Initialize it when the device object has been created by calling KeInitializeDeviceQueue at PASSIVE_LEVEL

KeInitializeDeviceQueue(&dx->AbcIrpQueue);

The device extension also needs a current IRP pointer, here in a field called AbcCurrentIrp.

Inserting IRPs into the Queue

AbcStartPacket2 uses KeInsertDeviceQueue to insert the IRP into the device queue. Note that KeInsertDeviceQueue must be called at DISPATCH_LEVEL IRQL. The call to IoAcquireCancelSpinLock in AbcStartPacket2 does this. Subsequent calls to IoReleaseCancelSpinLock return to the old IRQL.

If the device queue is empty, KeInsertDeviceQueue returns FALSE, but it sets the queue into a busy state. Subsequent insertion attempts return TRUE. In AbcStartPacket2, if KeInsertDeviceQueue returns FALSE, then the IRP is passed for processing in AbcStartIo2 straightaway. First, the IRP pointer is stored as the current IRP in the AbcCurrentIrp field in the device extension. Then the Cancel spin lock is released, returning the IRQL to its old level. StartIo routines must run at DISPATCH_LEVEL, so calls to KeRaiseIrql and KeLowerIrql are put around the call to AbcStartIo2.

If KeInsertDeviceQueue returns TRUE, the IRP has been put in the device queue. In this case, no further processing is required in AbcStartPacket2, apart from releasing the Cancel spin lock.

IRP Processing

The AbcStartIo2 function initially does some IRP cancel checks. Then, it sets about processing the IRP. Eventually, it completes the IRP and calls AbcStartNextPacket2 to begin processing the next IRP. The call to AbcStartNextPacket2 can occur in a different routine (e.g., after a hardware interaction).

Starting the Next IRP

AbcStartNextPacket2's job is to see if there are any more IRPs queued for processing. If there are, it dequeues one and sends it for processing in AbcStartIo2. Note that the calls to AbcStartNextPacket2 and AbcStartIo2 are recursive, which I trust will not be a problem. AbcStartNextPacket2 must be called at DISPATCH_LEVEL IRQL.

AbcStartNextPacket2 calls KeRemoveDeviceQueue to try to remove an IRP from the device queue. KeRemoveDeviceQueue must be run at DISPATCH_LEVEL IRQL. When running an IRP device queue, it is usual to call KeRemoveDeviceQueue while holding the Cancel spin lock. The call to IoAcquireCancelSpinLock raises the IRQL to the correct level, as before.

If KeRemoveDeviceQueue returns NULL, no IRPs are queued. The AbcCurrentIrp field is reset to NULL and the Cancel spin lock is released.

If KeRemoveDeviceQueue returns non-NULL then the return value represents the device queue entry in the IRP structure. Get the actual IRP pointer using the CONTAINING_RECORD macro. Store this in AbcCurrentIrp, release the Cancel spin lock, and call AbcStartIo2.

Listing 16.8 Supplemental device queue routines

VOID AbcStartPacket2(IN PDEVICE_OBJECT DeviceObject, IN PIRP Irp) {

 PABC_DEVICE_EXTENSION dx = (PABC_DEVICE_EXTENSION)fdo->DeviceExtension;

 KIRQL OldIrql;

 IoAcquireCancelSpinLock(&OldIrql);

 IoSetCancelRoutine(Irp, AbcCancelIrp);

 if (KeInsertDeviceQueue(&dx->AbcIrpQueue, &Irp->Tail.Overlay.DeviceQueueEntry)) IoReleaseCancelSpinLock(OldIrql);

 else {

  DeviceExtension->AbcCurrentIrp = Irp;

  IoReleaseCancelSpinLock(OldIrql);

  KeRaiseIrql(DISPATCH_LEVEL, &OldIrql);

  AbcStartIo2(DeviceObject, Irp);

  KeLowerIrql(OldIrql);

 }

}

VOID AbcStartNextPacket2(IN PDEVICE_OBJECT DeviceObject) {

 PABC_DEVICE_EXTENSION dx = (PABC_DEVICE_EXTENSION)fdo->DeviceExtension;

 KIRQL OldIrql;

 IoAcquireCancelSpinLock(&OldIrql);

 PKDEVICE_QUEUE_ENTRY QueueEntry = KeRemoveDeviceQueue(&dx->AbcIrpQueue);

 if (QueueEntry!=NULL) {

  PIRP Irp = CONTAINING_RECORD(QueueEntry, IRP, Tail.Overlay.DeviceQueueEntry);

  dx->AbcCurrentIrp = Irp;

  IoReleaseCancelSpinLock(OldIrql);

  AbcStartIo2(DeviceObject, Irp);

 } else {

  dx->AbcCurrentIrp = NULL;

  IoReleaseCancelSpinLock(OldIrql);

 }

}

VOID AbcStartIo2(IN PDEVICE_OBJECT DeviceObject, IN PIRP Irp) {

 PABC_DEVICE_EXTENSION dx = (PABC_DEVICE_EXTENSION)fdo->DeviceExtension);

 PIO_STACK_LOCATION IrpStack = IoGetCurrentIrpStackLocation(Irp);

 KIRQL Oldlrql;

 IoAcquireCancelSpinLock(&OldIrql);

 if (Irp->Cancel) {

  IoReleaseCancelSpinLock(OldIrql);

  return;

 }

 IoSetCancelRoutine(Irp, NULL);

 IoReleaseCancelSpinLock(OldIrql);

 // Process and complete request

 // …

 // Start next packet

 AbcStartNextPacket2(DeviceObject);

}

IRP Cancelling and Cleanup

Cancelling IRPs in supplemental device queues can use exactly the same techniques as the standard device queue. In this example, the cancel routine is removed when AbcStartIo2 is called. As shown in Listing 16.9, if the IRP to cancel is the current IRP, it is completed with a cancelled status in AbcCancelIrp. When AbcStartIo2 starts, it checks the Cancel flag and removes the cancel routine.

The Cleanup IRP handling should use exactly the same techniques as the WdmIo driver. The WdmIo driver uses a Timeout variable to stop an IRP that is being processed. You will need to use some similar technique if any of your IRPs may take a long time to be processed.

Listing 16.9 Supplemental device queue IRP cancel routine

VOID AbcCancelIrp(IN PDEVICE_OBJECT DeviceObject, IN PIRP Irp) {

 PABC_DEVICE_EXTENSION dx = (PABC_DEVICE_EXTENSION)fdo->DeviceExtension;

 if (Irp=dx->AbcCurrentIrp) {

  IoReleaseCancelSpinLock(Irp->CancelIrq1);

  CompleteIrp(Irp, STATUS_CANCELLED);

  AbcStartNextPacket2(DeviceObject, TRUE);

 } else {

  KeRemoveEntryDeviceQueue(&dx->AbcIrpQueue, &Irp->Tail.Overlay.DeviceQueueEntry);

  IoReleaseCancelSpinLock(Irp->CancelIrql);

  CompleteIrp(Irp, STATUS_CANCELLED);

 }

}