Humans tend to segment continuous perceptual experiences into discrete events. We have devised this notebook to capture the neural representations of this "chunking" process using a Hidden Markov Models (HMM). The notebook uses real data from naturalistic movie viewing and recall, suggesting ways to align derived neural event states with subjective annotations of event boundaries and with the recall time series.
Baldassano, C., Chen, J., Zadbood, A., Pillow, J. W., Hasson, U., & Norman, K. A. (2017). Discovering event structure in continuous narrative perception and memory. Neuron, 95(3), 709–721.e5.
link Describes and validates the event segmentation method, and applies it to perception and recall data from multiple experiments.
Baldassano, C., Hasson, U., & Norman, K. A. (2018). Representation of real-world event schemas during narrative perception. Journal of Neuroscience, 38(45), 9689–9699.
link Uses the event segmentation model to find common event structure among narratives with a shared schematic script.
Ben-Yakov, A., & Henson, R. N. (2018). The hippocampal film editor: sensitivity and specificity to event boundaries in continuous experience. Journal of Neuroscience, 38(47), 10057–10068.
link Further studies the relationship between the event boundaries produced by the event segmentation model, human-annotated boundaries, and hippocampal responses.
Silva, M., Baldassano, C., & Fuentemilla, L. (2019). Rapid memory reactivation at movie event boundaries promotes episodic encoding. Journal of Neuroscience, 39(43), 8538–8548.
link Applies the event segmentation model to EEG signals collected while subjects were watching a movie.
Antony, J. W., Hartshorne, T. H., Pomeroy, K., Gureckis, T. M., Hasson, U., McDougle, S. D., & Norman, K. A. (2020). Behavioral, physiological, and neural signatures of surprise during naturalistic sports viewing. Neuron.
link Uses the event segmentation model to relate the number and timing of event boundaries in neural signals to the degree of surprise elicited in basketball games.
import warnings import sys import os import glob from functools import reduce import numpy as np from brainiak.eventseg.event import EventSegment from scipy.stats import norm from matplotlib import pyplot as plt import matplotlib.patches as patches %matplotlib inline smallsize=14; mediumsize=16; largesize=18 plt.rc('xtick', labelsize=smallsize); plt.rc('ytick', labelsize=smallsize); plt.rc('legend', fontsize=mediumsize) plt.rc('figure', titlesize=largesize); plt.rc('axes', labelsize=mediumsize); plt.rc('axes', titlesize=mediumsize)
if not os.path.exists('Sherlock_AG_movie.npy'): !wget https://ndownloader.figshare.com/files/22927253 -O Sherlock_AG_movie.npy if not os.path.exists('Sherlock_AG_recall.npy'): !wget https://ndownloader.figshare.com/files/22927256 -O Sherlock_AG_recall.npy movie = np.load('Sherlock_AG_movie.npy') recall = np.load('Sherlock_AG_recall.npy') movie_group = np.mean(movie, axis=0)
--2020-11-10 13:56:28-- https://ndownloader.figshare.com/files/22927253 Resolving ndownloader.figshare.com (ndownloader.figshare.com)... 184.108.40.206, 220.127.116.11, 18.104.22.168, ... Connecting to ndownloader.figshare.com (ndownloader.figshare.com)|22.214.171.124|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/22927253/Sherlock_AG_movie.npy [following] --2020-11-10 13:56:29-- https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/22927253/Sherlock_AG_movie.npy Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 126.96.36.199 Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|188.8.131.52|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 256101248 (244M) [application/octet-stream] Saving to: ‘Sherlock_AG_movie.npy’ Sherlock_AG_movie.n 100%[===================>] 244.24M 9.00MB/s in 36s 2020-11-10 13:57:05 (6.83 MB/s) - ‘Sherlock_AG_movie.npy’ saved [256101248/256101248] --2020-11-10 13:57:05-- https://ndownloader.figshare.com/files/22927256 Resolving ndownloader.figshare.com (ndownloader.figshare.com)... 184.108.40.206, 220.127.116.11, 18.104.22.168, ... Connecting to ndownloader.figshare.com (ndownloader.figshare.com)|22.214.171.124|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/22927256/Sherlock_AG_recall.npy [following] --2020-11-10 13:57:06-- https://s3-eu-west-1.amazonaws.com/pfigshare-u-files/22927256/Sherlock_AG_recall.npy Resolving s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)... 126.96.36.199 Connecting to s3-eu-west-1.amazonaws.com (s3-eu-west-1.amazonaws.com)|188.8.131.52|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 8240168 (7.9M) [application/octet-stream] Saving to: ‘Sherlock_AG_recall.npy’ Sherlock_AG_recall. 100%[===================>] 7.86M 6.44MB/s in 1.2s 2020-11-10 13:57:07 (6.44 MB/s) - ‘Sherlock_AG_recall.npy’ saved [8240168/8240168]
Before applying any model, a good first step is to plot the correlation between activity patterns for each pair of timepoints during the movie. In this dataset, this shows blocks along the diagonal, which indicates that activity patterns are remaining stable for periods of tens of timepoints. This is the kind of structure that the HMM will be looking for.
plt.figure(figsize=(10,8)) plt.imshow(np.corrcoef(movie_group)) plt.xlabel('Timepoint') plt.ylabel('Timepoint') plt.colorbar() plt.title('Spatial pattern correlation');