{"id":285,"date":"2020-12-13T16:14:05","date_gmt":"2020-12-13T20:14:05","guid":{"rendered":"http:\/\/blog.labhackers.com\/?p=285"},"modified":"2020-12-21T12:16:22","modified_gmt":"2020-12-21T16:16:22","slug":"psychhid-rules","status":"publish","type":"post","link":"https:\/\/blog.labhackers.com\/?p=285","title":{"rendered":"PsychHID Rules!"},"content":{"rendered":"\n<p><sup>Originally posted on Marth 11th, 2019.<\/sup><\/p>\n\n\n\n<p>I spent some time last weekend using the&nbsp;<a href=\"https:\/\/www.labhackers.com\/millikey.html\">MilliKey response box<\/a>&nbsp;to test the keyboard event time-stamping accuracy of the<a href=\"https:\/\/github.com\/kleinerm\/Psychtoolbox-3\"> PsychToolbox3 PsychHID library written by Mario Kleiner.<\/a>&nbsp;Given my love of Python, I had to try out the psychtoolbox Python wrapper of PsychHID and compare it to psychopy.event.getKeys() and psychopy.iohub keyboard.getPresses().<\/p>\n\n\n\n<p>This test also makes things a bit harder for the experiment software by having the display updated on each frame of a trial and using MilliKey to generate keyboard events between 2 and 14 msec into a retrace.<\/p>\n\n\n\n<figure class=\"wp-block-embed is-type-rich is-provider-embed-handler wp-block-embed-embed-handler wp-embed-aspect-4-3 wp-has-aspect-ratio\"><div class=\"wp-block-embed__wrapper\">\n<iframe loading=\"lazy\" title=\"PsychHID keyboard Timing Test\" width=\"750\" height=\"563\" src=\"https:\/\/www.youtube.com\/embed\/Tz03GBP9MYE?feature=oembed\" frameborder=\"0\" allow=\"accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture\" allowfullscreen><\/iframe>\n<\/div><\/figure>\n\n\n\n<p>Results are amazing, especially given in this case a screen cap app was also running!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Results<\/h2>\n\n\n\n<p>We\u2019ll start with the results and then give the source code later. All results were collected using PsychoPy 3.0.6 with Python 3.7.1.<\/p>\n\n\n\n<p>Regardless of operating system, the computer monitor was running at 60 Hz.<\/p>\n\n\n\n<p>1000 key press events were generated by the MilliKey device for each test, with the key event occurring between 2 and 14 msec into a window flip().<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Windows 10<\/h3>\n\n\n\n<p><strong>psychopy.event.getKeys<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"480\" src=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychopy_win10_p37.png\" alt=\"\" class=\"wp-image-396\" srcset=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychopy_win10_p37.png 640w, https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychopy_win10_p37-300x225.png 300w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/figure>\n\n\n\n<p>As would be expected, the time stamp accuracy obtained from<br>psychopy.event.getKeys depends on when the key press event actually occurred relative to win.flip(), with an average error of 7 msec ranging from 0.5 msec to 16.9 msec.<\/p>\n\n\n\n<p><strong>psychopy.iohub keyboard.getPresses()<\/strong><\/p>\n\n\n\n<p>In contrast, iohub and PsychHID perform keyboard event time-stamping asynchronously to the experiment process, so average time-stamp error is much better.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"480\" src=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/iohub_win10_p37.png\" alt=\"\" class=\"wp-image-397\" srcset=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/iohub_win10_p37.png 640w, https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/iohub_win10_p37-300x225.png 300w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/figure>\n\n\n\n<p>With that said, iohub does not match the raw speed of PsychHID:<\/p>\n\n\n\n<p><strong>Python PsychHID() KbQueue*<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"480\" src=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychhid_win10_p37.png\" alt=\"\" class=\"wp-image-398\" srcset=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychhid_win10_p37.png 640w, https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychhid_win10_p37-300x225.png 300w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/figure>\n\n\n\n<h3 class=\"wp-block-heading\">macOS 10.13.6<\/h3>\n\n\n\n<p>Results from macOS are of particular interest because of the poor 1000 Hz USB keyboard event time-stamping results that seem to exist when using&nbsp;<em>standard&nbsp;<\/em>macOS keyboard event libraries, even in best case test conditions.<\/p>\n\n\n\n<p><strong>psychopy.event.getKeys<\/strong>()<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"480\" src=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychopy_macos_1000.png\" alt=\"\" class=\"wp-image-392\" srcset=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychopy_macos_1000.png 640w, https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychopy_macos_1000-300x225.png 300w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/figure>\n\n\n\n<p><strong>psychopy.iohub keyboard.getPresses()<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"480\" src=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/iohub_macos_1000.png\" alt=\"\" class=\"wp-image-391\" srcset=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/iohub_macos_1000.png 640w, https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/iohub_macos_1000-300x225.png 300w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/figure>\n\n\n\n<p>The iohub results for macOS further illustrate that the extra event delay seen compared to other operating systems is likely because of macOS keyboard API overhead, even when the macos accessibility API is used. To be fair, this maybe a Python + macOS interaction, I have not tried timing a C program for example.<\/p>\n\n\n\n<p><strong>Python PsychHID() KbQueue*<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"640\" height=\"480\" src=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychhid_macos_1000.png\" alt=\"\" class=\"wp-image-393\" srcset=\"https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychhid_macos_1000.png 640w, https:\/\/blog.labhackers.com\/wp-content\/uploads\/2020\/12\/psychhid_macos_1000-300x225.png 300w\" sizes=\"auto, (max-width: 640px) 100vw, 640px\" \/><\/figure>\n\n\n\n<p>On macOS, PsychHID is the only framework I have tested that provides 1 msec average keyboard event latency. At a very high level, I think this is because PsychHID uses HID event sniffing to detect keyboard events on macOS, bypassing the standard macOS keyboard handling APIs, and their associated delays, all together.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Python Source Code<\/h2>\n\n\n\n<p>The same test script was used to test all three event frameworks. The KB_BACKEND variable specified which backend to use for the test:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>\u2018psychopy\u2019: Use psychopy.event.getKeys(timeStamped=True).<\/li><li>\u2018iohub\u2019: Start an iohub server and use io.devices.keyboard.getPresses().<\/li><li>\u2018psychhid\u2019: Start a Psychtoolbx HID KbQueue and use KbQueueGetEvent.<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code lang=\"python\" class=\"language-python\"># -*- coding: utf-8 -*-\nfrom __future__ import absolute_import, division, print_function\nfrom psychopy import core, visual\nimport string\nimport serial\nimport numpy\n\nKB_BACKEND = 'psychhid' # 'iohub', or 'psychhid'\n# Serial Port of MilliKey.\nmK_serial_port = 'COM3'\n#mK_serial_port = '\/dev\/cu.usbmodem2998881'\n# Number of keyboard events to generate.\nkgen_count = 100\n# min, max msec duration for generated keypresses.\nmin_dur, max_dur = 175,  300\n# min, max msec MilliKey will wait before issuing the requested key press event.\nmin_delay, max_delay = 2, 14\npossible_kgen_chars = string.ascii_lowercase + string.digits\n\ngetKeys = None\ngetTime = core.getTime\nif KB_BACKEND == 'psychhid':\n    from psychtoolbox import GetSecs\n    getTime = GetSecs\n\nif KB_BACKEND == 'psychopy':\n    from psychopy import event\n    def getKeys():\n        return event.getKeys(timeStamped=True)\nelif KB_BACKEND == 'iohub':\n    global io\n    from psychopy.iohub import launchHubServer\n    io = launchHubServer()\n    def getKeys():\n        keys = io.devices.keyboard.getPresses()\n        return [(k.key,k.time) for k in keys]\nelif KB_BACKEND == 'psychhid':\n    from psychtoolbox import PsychHID\n    keys = [1] * 256\n    PsychHID('KbQueueCreate', [], keys)\n    PsychHID('KbQueueStart')\n    # Seems like on Windows this must be called before first kb event\n    # or first event has a time stamp error of ~ 20 msec.\n    _ = GetSecs()\n    def getKeys():\n        keys = []\n        while PsychHID('KbQueueFlush'):\n            evt = PsychHID('KbQueueGetEvent')[0]\n            if evt['Pressed']:\n                K = chr(int(evt['CookedKey'])).lower()\n                keys.append((K, evt['Time']))\n        return keys\n\nresults = numpy.zeros(kgen_count, dtype=numpy.float64)\n\nmk_sconn = serial.Serial(mK_serial_port, baudrate=128000, timeout=0.1)\n\nwin = visual.Window([800, 400])#, fullscr=True, allowGUI=False)\ntxt1 = \"MilliKey Generating Key Press: [%s]\\nOffset: %.1f msec. Duration: %d msec.\\n%d of %d events.\"\nmsg = visual.TextStim(win, text=txt1)\ndotPatch = visual.DotStim(win, color=(0.0, 1.0, 0.0), dir=270,\n                          nDots=223, fieldShape='sqr',\n                          fieldPos=(0.0, 0.0), fieldSize=1.5,\n                          dotLife=50, signalDots='same',\n                          noiseDots='direction', speed=0.01,\n                          coherence=0.9)\nmsg.draw()\ndotPatch.draw()\nwin.flip()\n\ncount = 0\nwhile count &lt; kgen_count:\n    kchar = possible_kgen_chars[count % (len(possible_kgen_chars))]\n    press_duration = int(numpy.random.randint(min_dur, max_dur))\n    delay_evt_usec = int(numpy.random.randint(min_delay, max_delay))*1000\n    evt_delay_sec = delay_evt_usec\/1000.0\/1000.0\n    msg.setText(txt1%(kchar, delay_evt_usec\/1000.0, press_duration,\n                            count+1, kgen_count))\n    dotPatch.draw()\n    msg.draw()\n    win.flip()\n\n    # Instruct MilliKey device to:\n    #  - generate a key press delay_evt_usec after receiving the KGEN command\n    #  - generate key release event press_duration after press event is sent.\n    kgen_cmd = \"KGEN {} {} {}\\n\".format(kchar,\n                                            press_duration,\n                                            delay_evt_usec).encode()\n    mk_sconn.write(kgen_cmd)\n    mk_sconn.flush()\n    # stime is the time the KGEN command was sent to the MilliKey device.\n    # plus the event offset the device is using.\n    stime = getTime()+evt_delay_sec\n\n    # If next 3 lines are not used, PsychHID getKeys() seems to loose\n    # occational keypress event on Windows 10. I could be using PsychHID()\n    # incorrectly, need to investigate....\n    dotPatch.draw()\n    msg.draw()\n    win.flip()\n\n    # Keep checking for key press events until one is received\n    kb_presses = getKeys()\n    while not kb_presses:\n        dotPatch.draw()\n        msg.draw()\n        win.flip()\n        kb_presses = getKeys()\n\n    kpress, ktime = kb_presses[0]\n    if not kb_presses:\n        raise RuntimeError(\"KGEN Timeout Error: No Key Press Event Detected.\")\n\n    # Ensure we got the key we were expecting.....\n    if kchar == kpress:\n        results[count] = ktime-stime\n        count += 1\n    else:\n        txt = \"Keyboard Key != Key Press Issued ([{}] vs [{}]). \"\n        txt += \"Was a keyboard key or MilliKey button pressed during the test?\"\n        raise RuntimeError(txt.format(kchar, kpress))\n\n    # Wait until after MilliKey has issued associated key release event.\n    ctime = getTime()\n    while getTime() - ctime &lt; (press_duration\/1000.0)*2:\n        dotPatch.draw()\n        msg.draw()\n        win.flip()\n        getKeys()\n\n# Done test, close backend if needed\nif KB_BACKEND == 'psychhid':\n    PsychHID('KbQueueStop')\nelif KB_BACKEND == 'iohub':\n    io.quit()\n\nmk_sconn.close()\nwin.close()\n\n# Print Results\n# Convert times to msec.\nevt_results = results[:count] * 1000.0\nprint(\"%s Timestamp Accuracy Stats\"%(KB_BACKEND))\nprint(\"\\tCount: {}\".format(evt_results.shape))\nprint(\"\\tAverage: {:.3f} msec\".format(evt_results.mean()))\nprint(\"\\tMedian: {:.3f} msec\".format(numpy.median(evt_results)))\nprint(\"\\tMin: {:.3f} msec\".format(evt_results.min()))\nprint(\"\\tMax: {:.3f} msec\".format(evt_results.max()))\nprint(\"\\tStdev: {:.3f} msec\".format(evt_results.std()))\n\n# Plot Results\ndef createHistogram(data, title=None, xlabel=\"Time stamp Error (msec)\",\n                    ylabel=\"Probability density\", nbins = 50):\n    try:\n        import matplotlib.pyplot as plt\n        mu, sigma = data.mean(), data.std()\n        # the histogram of the data\n        n, bins, patches = plt.hist(data, nbins, density=1,\n                                    facecolor='green', alpha=0.75)\n        # add a 'best fit' line\n        y = ((1 \/ (numpy.sqrt(2 * numpy.pi) * sigma)) *\n             numpy.exp(-0.5 * (1 \/ sigma * (bins - mu))**2))\n        plt.plot(bins, y, '--')\n        plt.xlabel(xlabel)\n        plt.ylabel(ylabel)\n        if title is None:\n            title = r\"Histogram\"\n        title = r'$\\mathrm{%s:}\\ \\mu=%.3f,\\ \\sigma=%.3f$'%(title,mu,sigma)\n        plt.title(title)\n        plt.grid(True)\n        plt.tight_layout()\n        plt.show()\n    except:\n        print(\"Could not create Histogram:\\n\\ttitle: {}\\n\\tlabel: {}\\n\\tylabel: {}\".format(title, xlabel, ylabel))\n\ncreateHistogram(evt_results,\"%s\\ Time\\ Stamp\\ Accuracy\"%(KB_BACKEND))\n\ncore.quit()<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-preformatted\">\n\n<\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusions<\/h2>\n\n\n\n<p>Hopefully this post provides some useful information about the time stamping accuracy you should expect to achieve when using a 1000 Hz USB keyboard with different operating systems and keyboard event APIs. I hope it also illustrates how useful the MilliKey response box can be in testing 1000 Hz USB keyboard event delay and time stamping error.<\/p>\n\n\n\n<p>If you are using macOS and want to really get events with 1 msec delay, then you must really use PsychHID, no matter what the response box maker claims (that includes us!).<\/p>\n\n\n\n<p>When using Windows 10, psychopy.event.getKeys() provides acceptable time stamp accuracy when your experiment script is only checking for keyboard events during the trial. If the experiment updates the display frequently during a trial, psychopy.iohub or PsychHID should be used.<\/p>\n\n\n\n<p>Looking forward to PsychHID based keyboard event monitoring being added to PsychoPy in the future.<\/p>\n\n\n\n<p>PsychHID is really quite amazing; nice job Mario!<\/p>\n\n\n\n<p>Happy Hacking!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Originally posted on Marth 11th, 2019. I spent some time last weekend using the&nbsp;MilliKey response box&nbsp;to test the keyboard event time-stamping accuracy of the PsychToolbox3 PsychHID library written by Mario Kleiner.&nbsp;Given my love of Python, I had to try out the psychtoolbox Python wrapper of PsychHID and compare it to psychopy.event.getKeys() and psychopy.iohub keyboard.getPresses(). This&hellip; <a class=\"more-link\" href=\"https:\/\/blog.labhackers.com\/?p=285\">Continue reading <span class=\"screen-reader-text\">PsychHID Rules!<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[17,4],"tags":[16,8,23,18,10],"class_list":["post-285","post","type-post","status-publish","format-standard","hentry","category-keyboard-event-timing","category-millikey-response-box","tag-keyboard-events","tag-millikey","tag-psychhid","tag-psychopy","tag-python","entry"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=\/wp\/v2\/posts\/285","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=285"}],"version-history":[{"count":7,"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=\/wp\/v2\/posts\/285\/revisions"}],"predecessor-version":[{"id":439,"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=\/wp\/v2\/posts\/285\/revisions\/439"}],"wp:attachment":[{"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=285"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=285"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.labhackers.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=285"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}