Psychopy: Add id to objects at their time of creation in the builder to easily detect cheating students

[mmagnuski]

During classes that are being taught at my university we've found to our disappointment that surprisingly many students do not complete their homework procedures on their own but by copying .xlsx and .psyexp files received from other students (usually also adding some slight modifications, like chanigng excel column names and routine names). Because often these files are not identical, automating the search for cheaters is not trivial.
I was wondering if it would be possible to add identifiers (for example random letters and digits) to builder components (like text or routine) on their creation and keep them the same even when objects are modified. This would make comparing psychopy .psyexp files very easy even if they were edited.

I know it is an unconventional feature request, but this shouldn't be much work (and I would happily submit a PR), while it would make grading psychopy assignments much easier. Such object identifiers could probably be also useful in other contexts too.

[peircej]

That's an interesting issue/idea. Maybe a write-only parameter called "originator"? This could be applied to Experiments at the top level or to each parameter, and yes, it could be something like a random string of numbers/digits. Would that be sufficient I wonder? It would be pretty easy to defeat, for sure, and it would leave a lot of others wondering what the point of this was.

Thinking more (thinking aloud, digitally) the same concept could also be used to a certain extent to track appearance of your own components in other people's experiments! If I develop a study, and my Routines start showing up in someone else's work they would have the same ID numbers. Again, it would be easy enough for a scientist to deliberately change them, but would that be too much effort?

[mmagnuski]

Yes, a write-only originator, that gets saved in psyexp would be great! In theory it would be easy to defeat, but in practice, given that students sometimes even copy xlsx files that have their author property intact, pointing to other student, it should be enough.
And other tracking options it could potentially allow would be cool too.

[mmagnuski]

It should be enough if that parameter was at the level of experiment, but it would be best, at least from my use-case, if that originator was at least at the routine level so that students could start building an experiment from a skeleton that I provide and I could still track the origin of added components.

[m-macaskill]

The identifier could be a hash (or brief extract of a hash) based on some stored file creation time and the name of the component, or other similar recipe. So rather than a random string, it could be verified as belonging to the file under consideration by recomputing the hash. Still straight-forward to defeat if you know what you're doing, but anyone with that level of nouse would be able to create a simple Builder experiment without needing to cheat, I imagine...

[peircej]

Yes. I'm wondering about going one stage further in preventing copying (still easy to implement):

• generate a random sequence (or a hash) for the experiment originator
• for Routines and Components we could then provide a generated name+date+code where the code is generated using name+date and the experiment hash as a passkey (see the "Stream ciphers" section on https://www.laurentluce.com/posts/python-and-cryptography-with-pycrypto/)

With that system:

• If someone copies the component over within PsychoPy we'll just recreate the code snippet and the name/date of the originator will remain
• if someone tries to 'tweak' the line in the experiment file then it will show as not matching the hash
• even if someone copies the entire line and has from another experiment file it will no longer match this one because of the connection to the experiment pass key

Possibly I'm over-thinking this, but it seems easy enough to implement and prevents several simple hacks.

[mmagnuski]

@m-macaskill @peircej - that sounds great! That would require adding pycrypto to psychopy app dependencies - are you ok with that?

[hoechenberger]

I'm not really sure about the issue we're trying to address here.

When I was still a student, we would regularly share homework to distribute the workload (which was just too high). Or I would sit with a fellow student or two in front of the same computer screen, and we would solve programming assignments together, share the result, and each of us would modify and tweak the code some more according to personal preferences (not to disguise that the code was shared, but mostly because we had varying opinions regarding coding style and plot layout). I don't think that's a bad thing to do. Then again, I do understand that if students don't _do_ the assignments anymore, but just hand in something they copied from someone else, this is a very difficult situation. But _then again_, exams won't go so well for these types of students, no? :)

Regarding the idea of adding IDs: if this ever gets implemented, I'd very much like to have the opportunity to disable this feature. If I share code (or, in this case, components) with colleagues, I fully trust them to give me credit wherever it's due. I don't want to send them a bugged snitch-component. It just wouldn't feel right.

I _could_ see the point of adding component metadata though: creation and modification date. But still pondering…

From a security standpoint, I support the idea of allowing for integrity checks. Components or entire experiments could be signed, for example, via itsdangerous (i.e. one would hash the code block and sign the hash; I thought of itsdangerous because I use it for email verification after registration on my web app, where the app would essentially sign the email address).

[hoechenberger]

I also agree it could be useful to establish a link between a Coder or Builder file and generated data files. Currently I manually add an “Experiment Version” column to the raw data files to keep track of which experimental code exactly was run for each participant (because, let’s face it, one sometimes needs to tweak certain aspects of the experiment even after a couple datasets have already been collected – at least that’s what commonly happens to me ;))

But maybe that’s what the psydat files are fore, anyway...

[mmagnuski]

When I was still a student, we would regularly share homework to distribute the workload ...

Most of the issues you raise concern ethics and so different people will have different views on that. For example I don't think one can grade a homework as individual project if it was created on one computer and sent to others. If sufficient modifications were made I would be able to see the indiviudal parts but otherwise I would grade it as group work (distributing the points among students). And the pausibility of that being an indiviudal project falls with the number of student that sent the same file. I am of course fine with people consulting their homeworks with each other, but there is a point when consultation becomes just coping (often students that I caught cheating would explain that they at first wanted to consult with the solution sent to them by others but then ended up sending that work instead of their own). Of course, component ID's do not push you to a specific definition of cheating, they just make detective work much easier (if you need to do that).

But then again, exams won't go so well for these types of students, no? :)

In theory - yes, but image a case where the class ends with a project which is just one bigger homework. And it is also possible to send work between students during the exam (they need internet access to download relevant files from the e-learning platform and they are allowed to use the materials provided there). It's also a bit frustrating to waste time grading all the homeworks when only 60-70% are real effort.

If I share code (or, in this case, components) with colleagues, I fully trust them to give me credit wherever it's due. I don't want to send them a bugged snitch-component. It just wouldn't feel right.

Yes, these fields should not be compulsory to open an experiment file. But still - if you sent an experiment with component ids in the psyexp file this does not force you to then use that ids to track these components, it only makes it possible.