macOS Display Timing

There has been at least one report of macOS 10.13 / 10.14 adding an extra one frame delay to graphics update times, causing experiment software graphics update times to be off by one retrace.

We used the MilliKey DeLux light sensor to look into this further by testings the display change timing of a macmini running either macOS 10.9.5 or 10.13.6 connected to an LCD display. Sure enough, it looks like sometime between macOS 10.9.5 and macOS 10.13.6 an extra one frame delay has been added to the operating systems graphics pipeline.

Procedure

The computer monitor and test script were the same as was used in the Windows 10 Display Timing post. Unless otherwise stated, this test used the same procedure as that post.

In this test we used was an Apple macmini (late 2011 model) connected to a SyncMaster P2770 LCD monitor. The stated response time of the SyncMaster P2770 is 1 msec, which probably means it is closer to 2 msec peak to peak.

The same test script was run on the same hardware while booted into either macOS 10.9.5 or macOS 10.13.6. The time difference between receiving the DeLux USB serial trigger event and the time reported from the win.flip() for each dark-light transition was calculated, subtracting the 7 msec offset the DeLux was configured to use.

Results

USB serial events sent by the MilliKey DeLux will add an average of 0.5 msec to these display change delay results. This has not been corrected for in the following plots.

macOS 10.9.5

While using macOS 10.9.5 we observed an expected / reasonable display update delay for the LCD monitor being tested. Average

macOS 10.13.6

Results show a consistent 1 frame delay in macOS 10.13.6 display updates as compared to macOS 10.9.5.

Conclusion

Using a MilliKey DeLux light sensor we have been able to show that there does seem to be an extra frame delay in display update timing on macOS 10.13.6 (or higher presumably) that does not exist in macOS 10.9.5. This adds an extra one retrace delay to any screen update times calculated by the experiment. This means visual stimulus duration should still be accurate, however onset and offset times should both be corrected for the extra delay.

If anyone is aware of a way to fix this issue on macOS 10.13+, please let us know.

Happy Hacking!

Share:

Windows 10 Display Timing

It looks like you need to be careful with your display settings in Windows 10, otherwise Windows 10 could be adding an extra frame delay to the display update times reported by your experiment software.

We have found that if the Windows 10 Display -> Scale (and layout) setting is not set to 100%, the Windows 10 operating system adds an extra frame delay to the actual monitor update time. When using a 70 Hz monitor for example, this adds an extra 14.3 msec delay, causing the reported display update time to be off by one retrace as well.

Windows 10 Monitor Scale Setting @ 100% results in expected display change timing.
Windows 10 Monitor Scale Setting @ 125% results in extra 1 frame (monitor retrace) delay.

Procedure

Using the MilliKey DeLux light sensor upgrade and PsychoPy3 we compared the display update time reported by psychopy.visual.Window.flip() to the time the experiment received a 1000 Hz USB serial event triggered by the MilliKey DeLux light sensor in response to each screen brightness change.

In this test we used was a Dell G5 15 laptop running Windows 10 with NVIDIA graphics connected to a SyncMaster P2770 LCD monitor. The stated response time of the SyncMaster P2770 is 1 msec, which probably means it is closer to 2 msec peak to peak.

A Python 3 test script used the open source PsychoPy3 experiment package to make 1000 dark -> light display changes, recording the flip() time of each dark->light change. The same script was connected to a MilliKey with the DeLux light sensor via USB Serial.

The MilliKey DeLux was configured to generate a USB Serial event after the DeLux light sensor detected the monitor brightness crossed a digital threshold level. The appropriate light sensor threshold level for the monitor being tested was set by the test script itself.

The MilliKey DeLux can be configured to send the USB Serial event a fixed number of msec after the actual light sensor trigger time, called the trigger offset. In this test we set the trigger offset to 7 msec. This allows the USB Serial event to be sent, and received, by the experiment program at a time when the program can rapidly receive and process the event.

The same test script and hardware was used in both a 100% and 125% scaling condition. The time difference between receiving the DeLux USB serial trigger event and the time reported from the win.flip() for each dark-light transition was calculated, subtracting the 7 msec offset the DeLux was configured to use.

Results

USB serial events sent by the MilliKey DeLux will add an average of 0.5 msec to these display change delay results. This has not been corrected for in the following plots.

Average Display Change Delay (Error) Bars and CI lines. Using 125% scaling causes an extra frame delay.

100% Scaling

Display change latency of SyncMaster P2770 LCD monitor @ 100% scaling setting is reasonable given LCD response time . Tested using MilliKey DeLux light sensor.

Display Update Delay Stats
Count: (1000,)
Average: 2.335 msec
Median: 2.231 msec
Min: 1.000 msec
Max: 4.556 msec
Stdev: 0.517 msec

When 100% scaling is being used, the display update time reported by win.flip() looks accurate given the LCD monitor response time.

Note that the bi-modal nature of the above delay distribution, with peaks at approximately 2 and 3 msec, is likely a result of quantization caused by the 1000 Hz MilliKey USB serial report rate.

125% Scaling


Extra frame delay can be seen when Windows 10 Display Scaling setting is @ 125%. Tested using MilliKey DeLux light sensor.

The results clearly show that the Windows 10 operating system is adding an extra frame delay when the Display Scaling setting is equal to 125%.

Code

The test script is based on one of the MilliKey DeLux Python examples that can be downloaded from our website. The test procedure logic is in the runDisplayTimingTest() function.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function
from psychopy import visual, event, core
import json
import serial
import numpy
import time

# Serial Port of MilliKey.
#mK_seria1_port = '/dev/cu.usbmodem4998701'
mK_seria1_port = 'COM112'

# Number of dark -> light display change tests to run
test_count = 1000

# Number of light frames to display after dark->light transition
light_frame_count = 10

# Number seconds to wait between each test iteration.
iti=0.4

# Number of msec MilliKey waits before sending serial trigger after 
# light threshold is crossed. 
trig_delay=7

# Maximum number of trigger events MilliKey will send on each test iteration. 
# Useful for limiting the trigger repeating every retrace on CRT monitors for
# example.
max_trig_count =  2

# True == Run simple auto threshold leveling routine before running the test.
# False == Use threshold level stored in MilliKey device.
autothresh = True

# Multiplier for calculating threshold level from dark screen max reading
# For LCD, suggest 1.5 - 2.5
# For CRT, suggest 5.0 - 10.0
thresh_scaling_factor = 2

TRIG_RISING_SERIAL = 1

def runDisplayTimingTest():
    results = numpy.zeros(test_count, dtype=numpy.float64)
    mk_sconn = init_millikey_delux()
    win, (white_stim, dark_stim, txt_stim1, txt_stim2) = initPsychoPyGraphics()

    start_key = event.waitKeys()

    if autothresh:
        lowlight_stats = getLightSensorStats(mk_sconn, win, dark_stim)
        ls_threshold = lowlight_stats.get('max')*thresh_scaling_factor
        print("Using threshold level %d for test.\n"%(ls_threshold))

    print("test_num\ttrig_rx_time\tflip_time\ttrig_delay\tdisplay_latency\ttrig_msg")

    # run test_count loops
    count=0
    while count < test_count:
        # Draw black screen and flip to it.
        dark_stim.draw()
        win.flip()
        # Wait ~iti seconds with display black, starting light sensor
        # 1/2 way through.
        time.sleep(iti/2)        
        mk_sconn.write(b"START_AIN %d\n"%(ls_threshold))
        core.wait(iti/2)
        flush_serial_rx(mk_sconn)
        # Draw white (target) stim and flip to it,
        # getting time flip occurred (stim onset)
        # time from psychopy.
        white_stim.draw()
        ftime = win.flip()

        reading = mk_sconn.readline()
        if reading:
                t, d = core.getTime(), json.loads(reading[:-2])
                tdelay = d['delay']/1000.0
                results[count]= t-ftime-tdelay
                if d['count'] > 1:
                    print("Warning: Trigger number %d received; expecting 1"%(d['count']))
                print("%d\t%.4f\t%.4f\t%.4f\t%.4f\t%s"%(count+1, t, ftime,
                                                    tdelay, results[count],str(d)))
        else:
            print("Error: MilliKey Serial Timeout.")
            win.close()
            mk_sconn.close()
            raise RuntimeError("Error: MilliKey Serial Timeout.")


        # Display white screen for n frames.
        for lf in range(light_frame_count):
            white_stim.draw()
            win.flip()

        mk_sconn.write(b"STOP_AIN\n")
        count+=1

    win.close()
    mk_sconn.close()
    return results

def sendSerial(mk_sconn, txdata, wait=0.050, is_json=True):
    rx = None
    mk_sconn.write("{}\n".format(txdata).encode('utf-8'))
    stime = core.getTime()
    rx = mk_sconn.readline()
    while len(rx) == 0 and core.getTime()-stime < wait:
        rx = mk_sconn.readline()
    if rx and is_json:
        try:
            return json.loads(rx[:-2])
        except:
            return rx[:-2]
    if rx:
        return rx[:-2]

def flush_serial_rx(mk_sconn):
    while mk_sconn.readline():
        pass
    
def getLightSensorStats(mk_sconn, win, stim):
    for i in range(4):
        stim.draw()
        win.flip()
    mk_sconn.write(b"START_AIN\n")
    for i in range(10):
        stim.draw()
        win.flip()
    mk_sconn.write(b"STOP_AIN\n")
    mk_sconn.write(b"GET AIN_STATS\n")
    stats = mk_sconn.readline()
    return json.loads(stats)

def init_millikey_delux():
    mk_sconn = serial.Serial(mK_seria1_port, baudrate=128000, timeout=0.05)
    print(sendSerial(mk_sconn, 'SET AIN_TRIG_DELAY %d'%(trig_delay),
                     is_json=False))
    print(sendSerial(mk_sconn, 'SET AIN_TRIG_MODE %d %d'%(TRIG_RISING_SERIAL,
                                                          max_trig_count),
                                                          is_json=False))

    ain_stats= sendSerial(mk_sconn, "GET AIN_STATS")
    ain_res = ain_stats.get('res')
    if ain_res > 100: # ain_res < 100 means light sensor is connected
        print("Error Light Sensor is not connected to Analog Input Jack.")
        mk_sconn.close()
        raise RuntimeError("Analog Input Error. Check connection.")
    return mk_sconn

def initPsychoPyGraphics():
    # Create a full screen window and two full screen stim.
    win = visual.Window(fullscr=True, screen=0)
    white_stim = visual.ShapeStim(win, lineColor='white', fillColor='white',
                                  vertices=((-1,-1),(1,-1),(1,1),(-1,1)))
    dark_stim = visual.ShapeStim(win, lineColor='black', fillColor='black',
                                 vertices=((-1,-1),(1,-1),(1,1),(-1,1)))
    txt_stim1 = visual.TextStim(win,
                                "Place DeLux Light Sensor in Top Left Corner.",
                                color=(0.0,1.0,0.0), height=0.05)
    txt_stim2 = visual.TextStim(win, "Press any Key to Start Test.",
                                pos=(0.0,-0.1), color=(0.0,1.0,0.0),
                                height=0.05)
    white_stim.draw()
    dark_stim.draw()
    txt_stim1.draw()
    txt_stim2.draw()
    win.flip()
    return win, (white_stim, dark_stim, txt_stim1, txt_stim2)

# Plot Results
def createHistogram(data, title="Display\ Update\ Delay", 
                    xlabel="Delay (milliseconds)",
                    ylabel="Probability density", nbins = 50):
    try:
        import matplotlib.pyplot as plt
        mu, sigma = data.mean(), data.std()
        # the histogram of the data
        n, bins, patches = plt.hist(data, nbins, density=1,
                                    facecolor='green', alpha=0.75)
        # add a 'best fit' line
        y = ((1 / (numpy.sqrt(2 * numpy.pi) * sigma)) *
             numpy.exp(-0.5 * (1 / sigma * (bins - mu))**2))
        plt.plot(bins, y, '--')
        plt.xlabel(xlabel)
        plt.ylabel(ylabel)
        if title is None:
            title = r"Histogram"
        title = r'$\mathrm{%s:}\ \mu=%.3f,\ \sigma=%.3f$'%(title,mu,sigma)
        plt.title(title)
        plt.grid(True)
        plt.tight_layout()
        plt.show()
    except:
        print("Could not create Histogram:\n\ttitle: {}\n\tlabel: {}\n\tylabel: {}".format(title, xlabel, ylabel))
    
if __name__ == '__main__':
	results = runDisplayTimingTest()
	
    # Convert times to msec.
    results = results * 1000.0
    
    print('\n\n')
    print("Display Update Delay Stats")
    print("\tCount: {}".format(results.shape))
    print("\tAverage: {:.3f} msec".format(results.mean()))
    print("\tMedian: {:.3f} msec".format(numpy.median(results)))
    print("\tMin: {:.3f} msec".format(results.min()))
    print("\tMax: {:.3f} msec".format(results.max()))
    print("\tStdev: {:.3f} msec".format(results.std()))
    
    createHistogram(results)

Conclusions

Next time you go to collect data on a Windows 10 machine, we would suggest you first make sure to check the Windows 10 Display Scaling setting and ensure it is at 100%.

It is probably worth pointing out that the extra delay seen in the 125% scaling condition does not effect stimulus duration: it adds one retrace interval to the start and end time of a stimulus.

There are likely other conditions that could cause the display timing of your experiment computer to be different than what you think or assume it to be. Therefore, the best option in our opinion is to test your own experiment setup on a regular basis. This is not as hard to do, time consuming, or expensive as you might think. Please checkout the MilliKey DeLux and consider using it the next time you need to do some timing validation of your experiment setup. Testing display change timing has never been easier and more affordable!

Happy Hacking!

Share:

Test Visual Stimulus Onset Timing using Keyboard Events !?

In this post we look at how to use the MilliKey DeLux light sensor upgrade to test visual stimulus onset timing using PsychoPy Coder by checking for a keyboard event.

MilliKey DeLux Light Sensor Upgrade

The MilliKey DeLux is an affordable and easy to use light sensor attachment for MilliKey response boxes. The MilliKey DeLux can generate light level driven 1000 Hz USB serial or keyboard events that are read by the same experiment software controlling visual stimulus presentation.

In a previous post about Testing PsychoPy event.waitKeys() Event Timing, we demonstrated how to use the MilliKey to test the keyboard event time stamping accuracy of the experiment software. This was achieved by sending USB serial messages from the experiment software instructing the MilliKey to generate keyboard events and then comparing the key press time to the time the serial message was sent. We found that, at least on Windows and Linux, PsychoPy can accurately time stamp MilliKey 1000 Hz keyboard events with one millisecond average delay and sub-millisecond variability.

One of the unique features of the MilliKey DeLux is that it can be configured to generate a letter ‘l’ key press event when the light sensor brightness crosses a trigger threshold level. This allows you to assess visual stimulus onset timing of an experiment by using the standard keyboard event handling functionality of the software being used.

PsychoPy Coder Test Script

Lets start by creating a simple PsychoPy Coder demo that repeatedly:

  • clears the screen to black
  • draws a white screen
  • waits for a keyboard press event
  • prints time difference between reported keyboard event and stimulus onset (white screen) times.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function
from psychopy import visual, event, core

# Number of dark -> light display change tests to run
test_count = 10

# Create a full screen window and two full screen stim.
win = visual.Window(fullscr=True, screen=0)
white_stim = visual.ShapeStim(win, lineColor='white', fillColor='white',
                              vertices=((-1,-1),(1,-1),(1,1),(-1,1)))
dark_stim = visual.ShapeStim(win, lineColor='black', fillColor='black',
                             vertices=((-1,-1),(1,-1),(1,1),(-1,1)))

# draw / setup both stim
white_stim.draw()
dark_stim.draw()
win.flip()

# Manually press any key to start test
event.waitKeys(timeStamped=True)

print("trial\tRT\tkey_time\tflip_time\tkey")
# run test_count loops
count=0
while count < test_count:
    # Draw black screen and display it for about 0.5 seconds.
    dark_stim.draw()
    win.flip()
    core.wait(0.5)
	# Throw away any possible junk keys
	# prior to changing display 
    event.getKeys()
	
    # Draw white (target) stim and flip to it,
    # getting time flip occurred (stim onset)
    # time from psychopy.
    white_stim.draw()
    ftime = win.flip()

	# Key keyboard event and print time difference between
	# flip() time and keypress time.
    k, ktime = event.waitKeys(timeStamped=True)[0]
    kdelay = ktime - ftime    
    print("%d\t%.6f\t%.6f\t%.6f\t%s"%(count, kdelay, ftime, ktime, k))

    core.wait(0.25)
    count+=1

We have basically created the most simple of manual reaction time tests. Run the above script from PsychoPy Coder and press a key as quickly as you can each time the screen turns white.

For example, running the example and manually pressing the space key, I get output like:

trial	RT	key_time	flip_time	key
0	0.368140	1.687949	2.056089	space
1	0.171302	2.570518	2.741820	space
2	0.197275	3.256913	3.454187	space
3	0.198487	3.967356	4.165843	space
4	0.196070	4.679771	4.875841	space
5	0.215323	5.390571	5.605894	space
6	0.197549	6.118132	6.315682	space
7	0.193179	6.828715	7.021894	space
8	0.166193	7.535785	7.701978	space
9	0.193039	8.214774	8.407813	space

So far this has all been kind of boring right?

What if we could use this exact script to test the visual onset timing of the white screen? Lets do it using the MilliKey DeLux.

MilliKey DeLux Setup

For this test to work we first need to make sure the MilliKey DeLux is configured to:

  1. use a sensible threshold level for the monitor being Generate Keyboard Events.
  2. auto generate light sensor triggered keyboard events.

Lets do this using the LabHackers’ Device Manager application.

LabHackers' Device Manager DeLux Setting Tab

MilliKey DeLux light sensor settings are found on the DeLux tab of the LabHackers’ Device Manager application.

For this example use the following settings:
Trigger Type: ‘Keyboard Event’
Trigger Delay: 5 msec
Enable Background Processing: Checked / True

The Trigger Threshold setting can be set using the auto threshold feature of the Light Sensor Dialog. The following video is a screen capture from a Windows 10 computer running the Labhackers’ Device Manager application. It illustrates using the auto threshold feature to set the light sensor threshold to an appropriate level for the monitor being tested. The MilliKey DeLux hardware, which can not be seen in the video, was placed over the white rectangle of the test dialog on the LCD monitor.

Using LabHackers’ Device Manager to Configure a MilliKey DeLux light sensor.

After you have configured the MilliKey DeLux to generate ‘offline’ keyboard events, and have exited the LabHackers’ Device Manager, the MilliKey will generate a ‘l’ key press event 5 milliseconds (Trigger Delay) after screen brightness crosses the threshold level set. Each key press lasts 123 msec, at which time a key release event is sent by the MilliKey.

Note: Once the MilliKey DeLux is armed and ready to create keyboard events, if the light level hitting the is not controlled you may get a bunch of letter ‘l’s being typed on your computer. If you want to abort this test and get the MilliKey to stop generating keyboard events, unplug the MilliKey USB cable from the computer and reconnect it. This turns off the Enable Background Processing setting, which is disabled by default when you first connect the MilliKey to a computer.

Results

For this test we used a Windows 7 computer connected to a CRT monitor running at 60Hz. We used a CRT monitor because there is effectively no delay between when a video card starts a screen retrace and when the top left corner of the CRT monitor starts displaying update. In contrast LCD / TFT monitors all have a response time, which can range between 1 and 10+ msec depending on the monitor model.

trial   RT      key_time        flip_time       key
0       0.004892        4.412357        4.417248        l
1       0.005464        5.188762        5.194226        l
2       0.005237        5.976979        5.982216        l
3       0.005049        6.765145        6.770194        l
4       0.004792        7.553340        7.558132        l
5       0.005583        8.341535        8.347118        l
6       0.005390        9.129753        9.135143        l
7       0.005198        9.917925        9.923123        l
8       0.005766        10.694352       10.700118       l
9       0.005555        11.482574       11.488128       l

Since a CRT monitor was used, the time difference between win.flip() and when the CRT actually starts being updated should be 0.0 msec. Remember that we set the MilliKey DeLux to generate the key press event 5 msec after the stimulus onset was detected by the device hardware, so if win.flip() is accurate, the delay calculated by the test should be 0.0 after we subtract the 5 msec event delay specified in the device settings.

As the following table shows, they are:

00.0049-0.0001
10.00550.0005
20.00520.0002
30.00500.0000
40.0048-0.0002
50.00560.0006
60.00540.0004
70.00520.0002
80.00580.0008
90.00560.0006

It is important to note that using keyboard events as light sensor triggers is only suggested on systems that provide accurate keyboard event timing. Therefore we can not suggest this method for PsychoPy testing on macOS at this time. macOS users should use the Serial event mode, which we will demonstrate in a future post.

Next

The MilliKey DeLux keyboard event mode is very useful for testing stimulus onset timing in experiment software that you know provides accurate keyboard event timing and when you can not, or do not, want to add custom code to your experiment.

If the keyboard event time stamping accuracy of the software being used is poor, or you would rather not get light sensor triggers as keyboard events, then you can use USB Serial events instead. In the next MilliKey DeLux post, we will look at using the Serial trigger type to test stimulus onset timing in a PsychoPy Coder experiment.

Happy Hacking!

Share: