Improving the consistency of participant data in a launched study session

[Ting]

Dear Researchers,

A top priority of FindingFive is to ensure the consistency of participant data: two participants taking similar actions on a trial should lead to similar participant data as recorded by FindingFive (for your statistical power!). This is a somewhat novel challenge in online studies: in the good old days when participants were invited to the lab in person, we as researchers could control the experimentation equipment. When studies are conducted over the web, however, participants' unique computer hardware specs, software versions, and Internet speed can all be very different. These differences introduce potential sources for the inconsistencies in the participant data collected in an experiment.

What's new
In the most recent update to FindingFive that became effective on Tuesday 3/30/2021, we have further improved the consistency of participant data within the same study session. This is achieved by ensuring that participants in the same study session will always experience the study the same way. Technically, when a researcher launches a new session, FindingFive now "freezes" the behavior of the FindingFive Grammar at the time of the session's launch, for this particular session. Any further updates to the FindingFive Grammar will not affect how the launched session will behave. 

No action required
There's no action required to take advantage of this new feature. This new feature is invisible to either researchers or participants. Researchers will launch a new session as usual, and the new mechanism automatically kicks in.

Potential downside
As the FindingFive Grammar system is now frozen for an active session, it also means that any bug fixes to the Grammar system will not retroactively apply to active sessions. Please preview your study thoroughly and carefully before launching a session to make sure your study runs as expected.

If your active session contains undesirable behavior after launching, FindingFive will not able to fix it for your active session anymore. You will now need to stop it and launch another session even after we fix the bug. Bug fixes will no longer be applied to active sessions.

Comments?
Please leave a message below to let us know your thoughts and comments!

[Ting]

Update on April 05, 2021:

FindingFive has implemented another set of measures to improve the reliability and consistency of participant data. Starting on April 06, 2021, researchers will be asked to preview their studies before launching a new session, if:

• Researchers have never previewed the study in its entirety before, or
  
• The FindingFive study grammar engine has been updated since the last time researchers have previewed the study
  

Note that researchers must preview the study in its entirety before a new session can be launched. 

Furthermore, researchers are now also asked to download and examine the participant data collected from a previous session before launching a new session. This is to minimize the chance that a design error persists through multiple sessions without being noticed.

Any questions please leave a reply in this thread. Thanks!

[jwright8]

Can you clarify one point: if I preview the study entirely and run a session, and then I end the session, and without changing anything in the study, want to run a new session, will I have to preview the study again before running the new session?

Also, I collaborate with others. If one of my collaborators does the preview of the entire study, can I then run a session or will the collaborator have to run the session?

[Ting]

Researchers editing their studies will not trigger a preview (although a preview is always recommended if changes are extensive). We, the FindingFive team, updating the grammar engine in the backend triggers one. We want to make sure that our coding changes do not affect researchers' code.

Any collaborator previewing the study will suffice.