FAQ #3 - Simultaneous Presentation of Audio and Other Stimuli

[Rachel - Participant Affairs]

How do I get audio and other stimuli (e.g., video, image, or text) to appear at the same time?

[Lucy - FF Researcher Support]

Audio (and video) stimuli have the default property "barrier": true. This means that the audio stimulus will play before the other stimuli and any responses appear on your trial. If you would like another stimulus or a response to appear at the beginning of the trial along with the audio, set "barrier": false for your audio stimulus.

[bisserai]

hi!
thanks for this. i'd like to present a fixation cross AND KEEP IT THERE while my audio is playing, so i'll try the barrier property. however my audio files all have a different duration and i'm collecting the response at the very end of the audio/ while the audio is running. i'd like to set the length of presentation of the fixation sreen to stay on for +2s after the audio has stopped, but i don't have ideas how to do that. i know audio files have a length property (added manually?) but i'm not sure how to use it to code the lenght of my trial..
thanks,
bissera

[Grace]

Hi Bissera,

Thanks for your note! Have you checked out our tokenized image documentation? There is a property of that stimulus that would allow you to specify the duration for each token. https://help.findingfive.com/api/stimuli...ized_image.

However, I'm not sure that FF can currently offer exactly what you're looking for. Because each of your audio stimuli are different durations, there isn't an easy way to automatize the duration property across all stimuli. You'd likely have to go in and manually set each duration/delay for each. This isn't a huge problem if you only have a couple of stimuli, but I am guessing that you have more than that.

Can you clarify what type of response you are intending to collect? Depending on the type of the response, different strategies would be needed to try to address what you're looking for.

[amiefairs]

Dear Grace,

Thank you for this response, I'm working with Bissera on this project

As Bissera said, we have audio files (sentences) and we want to keep the fixation cross (or another symbol) on screen while the sentence plays. People have to speak out loud to complete the sentence, by saying one word. We anticipate that sometimes people might speak before the end of the sentence, and we want to record that, as well as recording in some 'blank' time after the sentence has finished. This means our 'trial' (in our lab terms) is a series of screens such as:
Fixation cross -> Symbol -> Symbol + audio response -> Symbol or blank screen while participant provides the final word of the sentence.
We will present the first Symbol for maybe 0.3s so the audio doesn't start immediately with the onset of the symbol.
We want to record background_audio in both the Symbol+audio response trial template and the Symbol/blank screen final trial template.

As Bissera said, our sentences are different lengths. We could manually specify their length in a csv doc and then batch upload, but maybe it isn't necessary? I read the tokenaized image documentation but I don't quite understand how it would apply here when we would like to concurrently present audio and an image.

I guess our questions are:
Can we concurrently present audio and an image/text? If so, how?
Can we make a background_audio response span longer than one trial template? If asked this previously and at the time the answer was no, but I know FF has excellent development
Do you have any other thoughts or creative solutions as to how we can go about this?

Thanks!

[Ting]

Hi Amie,

Thanks for the providing the whole picture - it's really helpful. I think FF comes really close to being able to implement this design, and I have an idea on what we could do to finish the last leg. Just to make sure we are on the same page:

Presenting Audio and text/image at the same time
Yes, you can definitely present an audio and an image at the same time, as long as you put the image/text stimulus before the audio stimulus. In your design, you probably want to leave the barrier property of the audio stimulus untouched (that is, true by default). By placing the image/text stimulus before the audio stimulus, FF presents the image/text stimulus first, and then the audio stimulus. Because the presentation of the text/image is instantaneous, you effectively will be presenting both stimuli at the same time.

Keeping the "symbol" on the screen after the audio finishes playing
This is where it gets tricky. You are stuck between a rock and a hard place: if you put the symbol + audio on a separate trial than the last symbol, you get the barrier effect of the audio files nicely. That is, the trial containing only symbol + audio will accommodate the variable-length nature of your audio files, because the trial ends after the audio finishes playing. However, the challenge is now that your recording will be interrupted - you have to show the last symbol on a second trial, yoked together with the symbol+audio trial. Because there are two trials, there's going to be a gap between the audio recordings when the trials switch.

Alternatively, if you put the symbol + audio -> symbol on the same trial, you get the full uninterrupted recording, but it's impossible (for now) to tell FF to "keep showing the symbol" for 2 more seconds, given the variable lengths of audio files. In other words, FF doesn't understand a command like "ok, barriers are released, but wait for 2 more seconds before advancing to the next trial".

My proposed solution
Did I get your conundrum right? If so, I think the solution is now obvious at this point: trial templates already have a "delay" parameter, which delays the presentation of a trial by x seconds. I think we can expand the delay parameter to accept both a "starting" delay and a "end" delay", like this:

Code:

"delay": {"start": 0, "end": 2}

This will tell FF to wait 2 more seconds before advancing to the next trial, even when everything on the current trial is already done. This way, the background audio response continues to collect data for 2 more seconds after the audio file finishes playing.

I think this will be a feature of general value to researchers, and it happens to solve your issue too?

[bisserai]

Hi all,

Many thanks for your support and suggestions! Amie and i are about to have a meeting to see how best to implement this.

For now, what i've done is the following:
-for each trial, group our "symbol" and "audio" stimuli to be presented simultaneously:
"audios_practice": {
"type": "basic",
"stimuli": [["symbol","p1"], ["symbol","p2"]],
"duration": 4,
"submission_point": false
},

-used the stimuli delay option for audio stimuli, so they present a little after the symbol, but the symbol stays on:
{
"type": "audio",
"content": "p2.mp3",
"visible": false,
"delay": 0.3
}

For the audio recording we'll have a look together and get back to you.
Ting, i understand your suggestion, but for now i haven't been able to get an audio recording from DURING the audio presentation trial at all, rather only from the following trial. So even if we were able to extend the audio presentation trial, i expect to either only get a recording from when the audio is done or not at all. But i will certainly try again with an extended trial template!

Many thanks to everyone,
bissera

[Ting]

Hi Bissera,

If you are using the background audio response, you should be able to play an audio stimulus and record at the same time. With the audio response (the non-background one), you'll have to turn off barrier of the audio stimulus, which will cause issues in your design. So give the background audio response a try?

You are most welcome! We are here to help!

[amiefairs]

Hi Ting,

You have understood the conundrum and that delay parameter change would fix this issue! Are you able to implement it (and do you know when it would be ready by)?
We also want all trials to auto_advance once the delay is over. From the documentation it seems like this won't happen unless the duration parameter is set (and in fact, if we remove duration but set auto_advance = true then the trial just doesn't end). So, is it possible to have the auto_advance parameter also take the delay parameter into account, and not only duration?

I am going to test out the background_audio issue this afternoon and come back to you if it isn't working (we are using background_audio, not regular audio, so we shouldn't have any recording problems but they seem to be there using firefox).

[Ting]

Hi Amie, let me set up a minimum demo on our end to examine the interaction of behavior. I'm really surprised to learn that it didn't work out of the box - auto_advance should advance trials on its own even without a duration parameter (provided that there are barriers set up).

We will work on the delay-end feature today, so hopefully it will be ready sometime next week.