Psychopy: QUEST Documentation is unclear on what parameters represent

[janfreyberg]

Mentioned on this pull request:

https://github.com/psychopy/psychopy/pull/1386#discussion_r111187922

The documentation should probably:

• clarify what the input parameters are, and what units they are in
• give a working example with realistic values
• explain what performance levels can be staircased to

[janfreyberg]

this should fall under the refactoring staircases milestone!

[hoechenberger]

x-ref to PTB Quest implementation, which PsychoPy's is based on:
http://docs.psychtoolbox.org/Quest

[hoechenberger]

x-ref: original paper by Watson & Pelli

[peircej]

The original paper has a BASIC implementation. :-)

[hoechenberger]

@janfreyberg, the very first proposed intensity indeed appears to be equal to startVal, because in __init__() we do:

        self._questNextIntensity = startVal

and in next():

        if self.finished == False:
            # update pointer for next trial
            self.thisTrialN += 1
            self.intensities.append(self._nextIntensity)
            return self._nextIntensity

Note that the QUEST algorithm hasn't been involved here at any point so far! It only gets active once we add the first response using addResponse().

Edit

Seems I confused _nextIntensity and _questNextIntensity. Ok wow, since next() _requires_ self._nextIntensity, and since this attribute does not exist right after initialization, but only after adding the first response, we cannot get an intensity proposal for the very first trial or what?! Is this the same for the other staircases as well? Otherwise, this behavior should be changed, too!

[hoechenberger]

@janfreyberg, @peircej

I discovered an old, related, still unresolved, but important issue: #1005
In essence, the problem is that currently we allow for minVal, maxVal, and a range kwargs, which are redundant and possibly conflicting, and there is no way to clean this up while retaining full backward-compat; but if we don't fix this issue, users might get incorrect results. We should discuss this and agree upon a solution before reworking the docs...

[hoechenberger]

I edited the comment above the previous one, oh my, things are really messy and complicated!!

[janfreyberg]

Very well described issue!

My proposal would be to:

• keep all three kwargs for now
• raise an error when all three are actually defined
• raise a deprecation warning when range is defined
• remove it in some future release

That way we keep compatibility for anyone who's using range at the moment, but the bug is clear to anyone who uses all three at the moment.

[hoechenberger]

from __future__ import print_function
from psychopy.data import QuestHandler


quest = QuestHandler(startVal=10, startValSd=5)
print('Next intensity: %f' % quest.next())

actually DOES work! What the heck... Can anyone explain how and when QuestHandler._nextIntensity gets set before adding the first response?

[janfreyberg]

I think it comes from the StairHandler.__init__ call that passes self - StairHandler sets self._nextIntensity = self.startVal on line 2964:
https://github.com/psychopy/psychopy/blob/master/psychopy/data.py#L2964

[hoechenberger]

Ah! Yes! Thanks! So I guess it's either me, or this stuff needs better docs and refactoring. Or maybe it's both ;)

[hoechenberger]

Btw to allow for "persistent" links to source code / files at a specific time point, press the Y key when viewing the file on Github. This would transform the above URL to https://github.com/psychopy/psychopy/blob/432e79880132062a0d6f5a68d21ab0574d0a2c74/psychopy/data.py#L2964 and ensure the link still points to the correct line even if the file is changed later on.