PsychHID Rules!

Updated March 19th, 2019 with Linux results.

I spent some time last weekend using the MilliKey response box to test the keyboard event time-stamping accuracy of the PsychToolbox3 PsychHID library written by Mario Kleiner. Given my love of Python, I had to try out the psychtoolbox Python wrapper of PsychHID and compare it to psychopy.event.getKeys() and psychopy.iohub keyboard.getPresses().

This test also makes things a bit harder for the experiment software by having the display updated on each frame of a trial and using MilliKey to generate keyboard events between 2 and 14 msec into a retrace.

Screen capture of test script running using PsychHID. Results are amazing, especially given in this case a screen cap app was also running!

Results

We’ll start with the results and then give the source code later. All results were collected using PsychoPy 3.0.6 with Python 3.7.1.

Regardless of operating system, the computer monitor was running at 60 Hz.

1000 key press events were generated by the MilliKey device for each test, with the key event occurring between 2 and 14 msec into a window flip().

Windows 10

psychopy.event.getKeys

As would be expected, the time stamp accuracy obtained from
psychopy.event.getKeys depends on when the key press event actually occurred relative to win.flip(), with an average error of 7 msec ranging from 0.5 msec to 16.9 msec.

psychopy.iohub keyboard.getPresses()

In contrast, iohub and PsychHID perform keyboard event time-stamping asynchronously to the experiment process, so average time-stamp error is much better.

With that said, iohub does not match the raw speed of PsychHID:

Python PsychHID() KbQueue*

Linux

psychopy.event.getKeys

As was seen on Windows, the time stamp accuracy of keyboard events using event.getKeys() is totally dependent on when the key press event actually occurred relative to win.flip()

psychopy.iohub keyboard.getPresses()

iohub performance on Linux seems very good, even on Python 3.

Python PsychHID() KbQueue*

PsychHID just takes top place on Linux as well, with iohub not far behind (maybe not at all statistically).

macOS 10.13.6

Results from macOS are of particular interest because of the poor 1000 Hz USB keyboard event time-stamping results that seem to exist when using standard macOS keyboard event libraries, even in best case test conditions.

psychopy.event.getKeys

psychopy.iohub keyboard.getPresses()

The iohub results for macOS further illustrate that the extra event delay seen compared to other operating systems is likely because of macOS keyboard API overhead, even when the macos accessibility API is used. To be fair, this maybe a Python + macOS interaction, I have not tried timing a C program for example.

Python PsychHID() KbQueue*

On macOS, PsychHID is the only framework I have tested that provides 1 msec average keyboard event latency. At a very high level, I think this is because PsychHID uses HID event sniffing to detect keyboard events on macOS, bypassing the standard macOS keyboard handling APIs, and their associated delays, all together.

Python Source Code

The same test script was used to test all three event frameworks. The KB_BACKEND variable specified which backend to use for the test:

  • ‘psychopy’: Use psychopy.event.getKeys(timeStamped=True).
  • ‘iohub’: Start an iohub server and use io.devices.keyboard.getPresses().
  • ‘psychhid’: Start a Psychtoolbx HID KbQueue and use KbQueueGetEvent.
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function
from psychopy import core, visual
import string
import serial
import numpy

KB_BACKEND = 'psychhid' # 'iohub', or 'psychhid'
# Serial Port of MilliKey.
mK_serial_port = 'COM3'
#mK_serial_port = '/dev/cu.usbmodem2998881'
# Number of keyboard events to generate.
kgen_count = 100
# min, max msec duration for generated keypresses.
min_dur, max_dur = 175,  300
# min, max msec MilliKey will wait before issuing the requested key press event.
min_delay, max_delay = 2, 14
possible_kgen_chars = string.ascii_lowercase + string.digits

getKeys = None
getTime = core.getTime
if KB_BACKEND == 'psychhid':
    from psychtoolbox import GetSecs
    getTime = GetSecs

if KB_BACKEND == 'psychopy':
    from psychopy import event
    def getKeys():
        return event.getKeys(timeStamped=True)
elif KB_BACKEND == 'iohub':
    global io
    from psychopy.iohub import launchHubServer
    io = launchHubServer()
    def getKeys():
        keys = io.devices.keyboard.getPresses()
        return [(k.key,k.time) for k in keys]
elif KB_BACKEND == 'psychhid':
    from psychtoolbox import PsychHID
    keys = [1] * 256
    PsychHID('KbQueueCreate', [], keys)
    PsychHID('KbQueueStart')
    # Seems like on Windows this must be called before first kb event
    # or first event has a time stamp error of ~ 20 msec.
    _ = GetSecs()
    def getKeys():
        keys = []
        while PsychHID('KbQueueFlush'):
            evt = PsychHID('KbQueueGetEvent')[0]
            if evt['Pressed']:
                K = chr(int(evt['CookedKey'])).lower()
                keys.append((K, evt['Time']))
        return keys

results = numpy.zeros(kgen_count, dtype=numpy.float64)

mk_sconn = serial.Serial(mK_serial_port, baudrate=128000, timeout=0.1)

win = visual.Window([800, 400])#, fullscr=True, allowGUI=False)
txt1 = "MilliKey Generating Key Press: [%s]\nOffset: %.1f msec. Duration: %d msec.\n%d of %d events."
msg = visual.TextStim(win, text=txt1)
dotPatch = visual.DotStim(win, color=(0.0, 1.0, 0.0), dir=270,
                          nDots=223, fieldShape='sqr',
                          fieldPos=(0.0, 0.0), fieldSize=1.5,
                          dotLife=50, signalDots='same',
                          noiseDots='direction', speed=0.01,
                          coherence=0.9)
msg.draw()
dotPatch.draw()
win.flip()

count = 0
while count < kgen_count:
    kchar = possible_kgen_chars[count % (len(possible_kgen_chars))]
    press_duration = int(numpy.random.randint(min_dur, max_dur))
    delay_evt_usec = int(numpy.random.randint(min_delay, max_delay))*1000
    evt_delay_sec = delay_evt_usec/1000.0/1000.0
    msg.setText(txt1%(kchar, delay_evt_usec/1000.0, press_duration,
                            count+1, kgen_count))
    dotPatch.draw()
    msg.draw()
    win.flip()

    # Instruct MilliKey device to:
    #  - generate a key press delay_evt_usec after receiving the KGEN command
    #  - generate key release event press_duration after press event is sent.
    kgen_cmd = "KGEN {} {} {}\n".format(kchar,
                                            press_duration,
                                            delay_evt_usec).encode()
    mk_sconn.write(kgen_cmd)
    mk_sconn.flush()
    # stime is the time the KGEN command was sent to the MilliKey device.
    # plus the event offset the device is using.
    stime = getTime()+evt_delay_sec

    # If next 3 lines are not used, PsychHID getKeys() seems to loose
    # occational keypress event on Windows 10. I could be using PsychHID()
    # incorrectly, need to investigate....
    dotPatch.draw()
    msg.draw()
    win.flip()

    # Keep checking for key press events until one is received
    kb_presses = getKeys()
    while not kb_presses:
        dotPatch.draw()
        msg.draw()
        win.flip()
        kb_presses = getKeys()

    kpress, ktime = kb_presses[0]
    if not kb_presses:
        raise RuntimeError("KGEN Timeout Error: No Key Press Event Detected.")

    # Ensure we got the key we were expecting.....
    if kchar == kpress:
        results[count] = ktime-stime
        count += 1
    else:
        txt = "Keyboard Key != Key Press Issued ([{}] vs [{}]). "
        txt += "Was a keyboard key or MilliKey button pressed during the test?"
        raise RuntimeError(txt.format(kchar, kpress))

    # Wait until after MilliKey has issued associated key release event.
    ctime = getTime()
    while getTime() - ctime < (press_duration/1000.0)*2:
        dotPatch.draw()
        msg.draw()
        win.flip()
        getKeys()

# Done test, close backend if needed
if KB_BACKEND == 'psychhid':
    PsychHID('KbQueueStop')
elif KB_BACKEND == 'iohub':
    io.quit()

mk_sconn.close()
win.close()

# Print Results
# Convert times to msec.
evt_results = results[:count] * 1000.0
print("%s Timestamp Accuracy Stats"%(KB_BACKEND))
print("\tCount: {}".format(evt_results.shape))
print("\tAverage: {:.3f} msec".format(evt_results.mean()))
print("\tMedian: {:.3f} msec".format(numpy.median(evt_results)))
print("\tMin: {:.3f} msec".format(evt_results.min()))
print("\tMax: {:.3f} msec".format(evt_results.max()))
print("\tStdev: {:.3f} msec".format(evt_results.std()))

# Plot Results
def createHistogram(data, title=None, xlabel="Time stamp Error (msec)",
                    ylabel="Probability density", nbins = 50):
    try:
        import matplotlib.pyplot as plt
        mu, sigma = data.mean(), data.std()
        # the histogram of the data
        n, bins, patches = plt.hist(data, nbins, density=1,
                                    facecolor='green', alpha=0.75)
        # add a 'best fit' line
        y = ((1 / (numpy.sqrt(2 * numpy.pi) * sigma)) *
             numpy.exp(-0.5 * (1 / sigma * (bins - mu))**2))
        plt.plot(bins, y, '--')
        plt.xlabel(xlabel)
        plt.ylabel(ylabel)
        if title is None:
            title = r"Histogram"
        title = r'$\mathrm{%s:}\ \mu=%.3f,\ \sigma=%.3f$'%(title,mu,sigma)
        plt.title(title)
        plt.grid(True)
        plt.tight_layout()
        plt.show()
    except:
        print("Could not create Histogram:\n\ttitle: {}\n\tlabel: {}\n\tylabel: {}".format(title, xlabel, ylabel))

createHistogram(evt_results,"%s\ Time\ Stamp\ Accuracy"%(KB_BACKEND))

core.quit()

Conclusions

Hopefully this post provides some useful information about the time stamping accuracy you should expect to achieve when using a 1000 Hz USB keyboard with different operating systems and keyboard event APIs. I hope it also illustrates how useful the MilliKey response box can be in testing 1000 Hz USB keyboard event delay and time stamping error.

If you are using macOS and want to really get events with 1 msec delay, then you must really use PsychHID, no matter what the response box maker claims (that includes us!).

When using Windows 10, psychopy.event.getKeys() provides acceptable time stamp accuracy when your experiment script is only checking for keyboard events during the trial. If the experiment updates the display frequently during a trial, psychopy.iohub or PsychHID should be used.

Looking forward to PsychHID based keyboard event monitoring being added to PsychoPy in the future.


PsychHID is really quite amazing; nice job Mario!

Happy Hacking!

Share: