Event segmentation and alignment in fMRI data


Humans tend to segment continuous perceptual experiences into discrete events. We have devised this notebook to capture the neural representations of this "chunking" process using a Hidden Markov Models (HMM). The notebook uses real data from naturalistic movie viewing and recall, suggesting ways to align derived neural event states with subjective annotations of event boundaries and with the recall time series.

Annotated Bibliography

  1. Baldassano, C., Chen, J., Zadbood, A., Pillow, J. W., Hasson, U., & Norman, K. A. (2017). Discovering event structure in continuous narrative perception and memory. Neuron, 95(3), 709–721.e5. link Describes and validates the event segmentation method, and applies it to perception and recall data from multiple experiments.

  2. Baldassano, C., Hasson, U., & Norman, K. A. (2018). Representation of real-world event schemas during narrative perception. Journal of Neuroscience, 38(45), 9689–9699. link Uses the event segmentation model to find common event structure among narratives with a shared schematic script.

  3. Ben-Yakov, A., & Henson, R. N. (2018). The hippocampal film editor: sensitivity and specificity to event boundaries in continuous experience. Journal of Neuroscience, 38(47), 10057–10068. link Further studies the relationship between the event boundaries produced by the event segmentation model, human-annotated boundaries, and hippocampal responses.

  4. Silva, M., Baldassano, C., & Fuentemilla, L. (2019). Rapid memory reactivation at movie event boundaries promotes episodic encoding. Journal of Neuroscience, 39(43), 8538–8548. link Applies the event segmentation model to EEG signals collected while subjects were watching a movie.

  5. Antony, J. W., Hartshorne, T. H., Pomeroy, K., Gureckis, T. M., Hasson, U., McDougle, S. D., & Norman, K. A. (2020). Behavioral, physiological, and neural signatures of surprise during naturalistic sports viewing. Neuron. link Uses the event segmentation model to relate the number and timing of event boundaries in neural signals to the degree of surprise elicited in basketball games.

Table of Contents

Loading data

This tutorial will use data from the first run of the Sherlock dataset (Chen et al. 2017), masked to only include the Angular Gyrus.

Finding event boundaries during perception

Warm-up: Event structure in activity patterns

Before applying any model, a good first step is to plot the correlation between activity patterns for each pair of timepoints during the movie. In this dataset, this shows blocks along the diagonal, which indicates that activity patterns are remaining stable for periods of tens of timepoints. This is the kind of structure that the HMM will be looking for.