FindingFive Discussion Forum
Tokenized Text Word Position in Output - Printable Version

+- FindingFive Discussion Forum (
+-- Forum: General (
+--- Forum: General Feedback (
+--- Thread: Tokenized Text Word Position in Output (/showthread.php?tid=246)

Tokenized Text Word Position in Output - aromanir - 04-24-2023

Hi, I have collected data from a word-by-word self-paced study on FindingFive and am currently in the process of analyzing it. Although, FindingFive does record the RTs per each word (see yellow column in screenshot for RTs and blue column for each word in the sentence), the output does not provide a column that allows the researcher to see the position of each word in the full sentence. So far, I have been filtering my experimental conditions in Excel and labelling each word position manually (see green column). However, I was wondering if there is a more efficient way of doing this? Whether it be by using a specific argument I can add to the tokenized text option, or if someone has an automated Excel sheet or R script that can facilitate this process. Thanks in advance!

RE: Tokenized Text Word Position in Output - Ting - 04-28-2023

Hi that's a great question! I think the excel trick is pretty nice, although a bit labor-intensive. In R, you can probably do something like a loop where you look for unique trial number + response names, and within each unique pair, use seq(1, length(pair)) to generate the word position numbering.

Just a thought! Let us know how it goes!

RE: Tokenized Text Word Position in Output - aromanir - 05-11-2023

Thanks Ting! Here is what worked for me on R:

# Break the dataframe into a list of dataframes based on 'trial_num'
df_list <- split(data, data$trial_num)

# Add row index as a column to each dataframe
df_list <- lapply(df_list, function(df) {
  df$word_position <- 1:nrow(df)

# Combine the dataframes back together
combined_df <-, df_list)

# Sort the combined dataframe by 'part_number' and 'trial_num'
combined_df <- combined_df %>% arrange(part_number, trial_num)

# View the resulting dataframe

RE: Tokenized Text Word Position in Output - Ting - 05-17-2023

Thanks for sharing the code with our community!

RE: Tokenized Text Word Position in Output - amelia11 - 07-01-2024

Understanding tokenized text word positions helps one analyze language models more effectively. It allows for precise manipulation and interpretation of text data, ensuring accurate processing and improving performance in tasks like translation, sentiment analysis, and natural language understanding.