{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# fmrisim demo script\n", "\n", "Authors: Cameron T Ellis ([cameron.ellis@yale.edu](mailto:cameron.ellis@yale.edu)), Christopher Baldassano ([c.baldassano@columbia.edu](mailto:c.baldassano@columbia.edu)), Anna C Schapiro ([aschapir@sas.upenn.edu](mailto:aschapir@sas.upenn.edu)), Ming Bo Cai ([mingbo.cai@ircn.jp](mailto:mingbo.cai@ircn.jp)), Jonathan D Cohen ([jdc@princeton.edu](mailto:jdc@princeton.edu)) \n", "\n", "## Overview\n", "\n", "Example script to demonstrate fmrisim functionality. This generates data for a two condition, event-related design in which each condition evokes different activity within the same voxels. It then runs simple univariate and multivariate analyses on the data. \n", "\n", "If you would like a script that can be executed from the command line and scaffolds the simulation process, refer to the `fmrisim_real_time_generator.py` script in the `brainiak/utils` folder. \n", "\n", "\n", "## Annotated bibliography\n", "\n", "1. Ellis, C. T., Baldassano, C., Schapiro, A. C., Cai, M. B., Cohen, J. D. (2020). Facilitating open-science with realistic fMRI simulation: validation and application. *PeerJ* 8:e8564 [`link`](https://peerj.com/articles/8564/)\n", "*Describes and validates the fmrisim method. Applies it to a dataset to test alternative design parameters and evaluate how these parameters influence the effect size* \n", "\n", "2. Ellis, C. T., Lesnick, M., Henselman-Petrusek, G., Keller, B., & Cohen, J. D. (2019). Feasibility of topological data analysis for event-related fMRI, Network Neuroscience, 1-12 [`link`](https://www.mitpressjournals.org/doi/full/10.1162/netn_a_00095?mobileUi=0)\n", "*Example of using fmrisim to evaluate the plausibility of an analysis procedure under different signal parameters and design constraints* \n", "\n", "3. Kumar, S., Ellis, C., O'Connell, T. P., Chun, M. M., & Turk-Browne, N. B. (in press). Searching through functional space reveals distributed visual, auditory, and semantic coding in the human brain. *PLoS Computational Biology* [`link`](https://www.biorxiv.org/content/10.1101/2020.04.20.052175v1)\n", "*Example of using fmrisim to test different possible neural bases for an observed effect in real data* \n", "\n", "4. Welvaert, M., et al. (2011) neuRosim: An R package for generating fMRI data. *Journal of Statistical Software* 44, 1-18 [`link`](https://www.jstatsoft.org/article/view/v044i10)\n", "*A package in R for simulating fMRI data that was an inspiration for fmrisim*\n", "\n", "\n", "## Table of Contents\n", "[](#1-set-parameters)\n", "- [](#11-import-necessary-python-packages)\n", "- [](#12-load-participant-data) \n", "- [](#13-specify-participant-dimensions-and-resolution) \n", "- [](#14-generate-an-activity-template-and-a-mask) \n", "- [](#15-determine-noise-parameters) \n", "\n", "[2-generate-noise) \n", "- [](#21-create-temporal-noise) \n", "- [](#22-create-system-noise) \n", "- [](#23-combine-noise-and-template) \n", "- [](#24-fit-the-data-to-the-noise-parameters) \n", "\n", "[3-generate-signal) \n", "- [](#31-specify-which-voxels-in-the-brain-contain-signal) \n", "- [](#32-characterize-signal-for-voxels) \n", "- [](#33-generate-event-time-course) \n", "- [](#34-export-stimulus-time-course-for-analysis) \n", "- [](#35-estimate-the-voxel-weight-for-each-event) \n", "- [](#36-convolve-each-voxels-time-course-with-the-hemodynamic-response-function) \n", "- [](#37-establish-signal-magnitude) \n", "- [](#38-multiply-the-convolved-response-with-the-signal-voxels) \n", "- [](#39-combine-signal-and-noise) \n", "\n", "[4-analyse-data) \n", "- [](#41-pull-out-data-for-each-trial) \n", "- [](#42-represent-the-data) \n", "- [](#43-test-for-univariate-effect) \n", "- [](#44-test-for-a-multivariate-effect) \n", "\n", "[](#summary)\n", "\n", "[](#references)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1. Set parameters\n", "\n", "It is necessary to set various parameters that describe how the signal and the noise will be generated." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 1.1 Import necessary Python packages\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "collapsed": true }, "outputs": [], "source": [ "%matplotlib notebook\n", "\n", "from pathlib import Path\n", "from brainiak.utils import fmrisim\n", "import nibabel\n", "import numpy as np\n", "import numpy.matlib\n", "import matplotlib.pyplot as plt\n", "import scipy.ndimage as ndimage\n", "import scipy.spatial.distance as sp_distance\n", "import sklearn.manifold as manifold\n", "import scipy.stats as stats\n", "import sklearn.model_selection\n", "import sklearn.svm" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 1.2 Load participant data*\n", "\n", "Any 4 dimensional fMRI data that is readible by nibabel can be used as input to this pipeline. For this example, data is taken from the open access repository DataSpace: http://arks.princeton.edu/ark:/88435/dsp01dn39x4181. This file is unzipped and placed same directory as this notebook with the name Corr_MVPA " ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "collapsed": true }, "outputs": [], "source": [ "nii = nibabel.load('Corr_MVPA/Participant_01_rest_run01.nii')\n", "volume = nii.get_fdata()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 1.3 Specify participant dimensions and resolution\n", "\n", "The size of the volume and the resolution of the voxels must be specified (or extracted from the real data as is the case below)." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Volume dimensions: (64, 64, 27, 294)\n", "TR duration: 1.50s\n" ] } ], "source": [ "dim = volume.shape # What is the size of the volume\n", "dimsize = nii.header.get_zooms() # Get voxel dimensions from the nifti header\n", "tr = dimsize[3]\n", "if tr > 100: # If high then these values are likely in ms and so fix it\n", " tr /= 1000\n", "print('Volume dimensions:', dim)\n", "print('TR duration: %0.2fs' % tr)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 1.4 Generate an activity template and a mask\n", "\n", "Functions in fmrisim require a continuous map that describes the appropriate average MR value for each voxel in the brain and a mask which specifies voxels in the brain versus voxels outside of the brain. One way to generate both of these volumes is the mask_brain function. At a minimum, this takes as an input the fMRI volume to be simulated. To create the template this volume is averaged over time and bounded to a range from 0 to 1. In other words, voxels with a high value in the template have high activity over time. To create a mask, the template is thresholded. This threshold can be set manually or instead an appropriate value can be determined by looking for the minima between the two first peaks in the histogram of voxel values. If you would prefer, you could use the [compute_epi_mask](http://nilearn.github.io/modules/generated/nilearn.masking.compute_epi_mask.html) function in nilearn which uses a similar method." ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "collapsed": true }, "outputs": [], "source": [ "mask, template = fmrisim.mask_brain(volume=volume, \n", " mask_self=True,\n", " )" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### 1.5 Determine noise parameters\n", "\n", "A critical step in the fmrisim toolbox is determining the noise parameters of the volume to be created. Many noise parameters are available for specification and if any are not set then they will default to reasonable values. As mentioned before, it is instead possible to provide raw fMRI data that will be used to estimate these noise parameters. The goal of the noise estimation is to calculate general descriptive statistics about the noise in the brain that are thought to be important. The simulations are then useful for understanding how signals will survive analyses when embedded in realistic neural noise. \n", "\n", "Now the disclaimers: the values here are only an estimate and will depend on noise properties combining in the ways assumed. In addition, because of the non-linearity and stochasticity of this simulation, this estimation is not fully invertible: if you generate a dataset with a set of noise parameters it will have similar but not the same noise parameters as a result. Moreover, complex interactions between brain regions that likely better describe brain noise are not modelled here: this toolbox pays no attention to regions of the brain or their interactions. Finally, for best results use raw fMRI because if the data has been preprocessed then assumptions this algorithm makes are likely to be erroneous. For instance, if the brain has been masked then this will eliminate variance in non-brain voxels which will mean that calculations of noise dependent on those voxels as a reference will fail.\n", "\n", "To ameliorate some of these concerns, it is possible to fit the spatial and temporal noise properties of the data. This iterates over the noise generation process and tunes parameters in order to match those that are provided. This is time consuming (especially for fitting the temporal noise) but is helpful in matching the specified noise properties. \n", "\n", "This toolbox separates noise in two: spatial noise and temporal noise. To estimate spatial noise both the smoothness and the amount of non-brain noise of the data must be quantified. For smoothness, the Full Width Half Max (FWHM) of the volume is averaged for the X, Y and Z dimension and then averaged across a sample of time points. To calculate the Signal to Noise Ratio (SNR) the mean activity in brain voxels for the middle time point is divided by the standard deviation in activity across non-brain voxels for that time point. For temporal noise an auto-regressive and moving average (ARMA) process is estimated, along with the overall size of temporal variability. A sample of brain voxels is used to estimate the first AR component and the first MA component of each voxel's activity over time using the statsmodels package. The Signal to Fluctuation Noise Ratio (SFNR) is calculated by dividing the average activity of voxels in the brain with that voxel’s noise (Friedman & Glover, 2006). That noise is calculated by taking the standard deviation of that voxel over time after it has been detrended with a second order polynomial. The SFNR then controls the amount of functional variability. Other types of noise can be generated, such as physiological noise, but are not estimated by this function.\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "collapsed": true }, "outputs": [], "source": [ "# Calculate the noise parameters from the data. Set it up to be matched.\n", "noise_dict = {'voxel_size': [dimsize[0], dimsize[1], dimsize[2]], 'matched': 1}\n", "noise_dict = fmrisim.calc_noise(volume=volume,\n", " mask=mask,\n", " template=template,\n", " noise_dict=noise_dict,\n", " )" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Noise parameters of the data were estimated as follows:\n", "SNR: 23.175648200023815\n", "SFNR: 70.7171164884859\n", "FWHM: 5.661164924881941\n" ] } ], "source": [ "print('Noise parameters of the data were estimated as follows:')\n", "print('SNR: ' + str(noise_dict['snr']))\n", "print('SFNR: ' + str(noise_dict['sfnr']))\n", "print('FWHM: ' + str(noise_dict['fwhm']))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### **2. Generate noise**\n", "fmrisim can generate realistic fMRI noise when supplied with the appropriate inputs. A single function receives these inputs and deals with generating the noise. The necessary code to run this is in the next cell. For clarity, we walk through the steps of how this simulation is performed. " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "/Users/cellis/anaconda/envs/brainiak_test/lib/python3.6/site-packages/scipy/stats/stats.py:2279: RuntimeWarning: divide by zero encountered in true_divide\n", " np.expand_dims(sstd, axis=axis))\n", "/Users/cellis/anaconda/envs/brainiak_test/lib/python3.6/site-packages/scipy/stats/stats.py:2279: RuntimeWarning: invalid value encountered in true_divide\n", " np.expand_dims(sstd, axis=axis))\n" ] } ], "source": [ "# Calculate the noise given the parameters\n", "noise = fmrisim.generate_noise(dimensions=dim[0:3],\n", " tr_duration=int(tr),\n", " stimfunction_tr=[0] * dim[3], \n", " mask=mask,\n", " template=template,\n", " noise_dict=noise_dict,\n", " )" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "application/javascript": "/* Put everything inside the global mpl namespace */\nwindow.mpl = {};\n\n\nmpl.get_websocket_type = function() {\n if (typeof(WebSocket) !== 'undefined') {\n return WebSocket;\n } else if (typeof(MozWebSocket) !== 'undefined') {\n return MozWebSocket;\n } else {\n alert('Your browser does not have WebSocket support. ' +\n 'Please try Chrome, Safari or Firefox ≥ 6. ' +\n 'Firefox 4 and 5 are also supported but you ' +\n 'have to enable WebSockets in about:config.');\n };\n}\n\nmpl.figure = function(figure_id, websocket, ondownload, parent_element) {\n this.id = figure_id;\n\n this.ws = websocket;\n\n this.supports_binary = (this.ws.binaryType != undefined);\n\n if (!this.supports_binary) {\n var warnings = document.getElementById(\"mpl-warnings\");\n if (warnings) {\n warnings.style.display = 'block';\n warnings.textContent = (\n \"This browser does not support binary websocket messages. \" +\n \"Performance may be slow.\");\n }\n }\n\n this.imageObj = new Image();\n\n this.context = undefined;\n this.message = undefined;\n this.canvas = undefined;\n this.rubberband_canvas = undefined;\n this.rubberband_context = undefined;\n this.format_dropdown = undefined;\n\n this.image_mode = 'full';\n\n this.root = $('
');\n this._root_extra_style(this.root)\n this.root.attr('style', 'display: inline-block');\n\n $(parent_element).append(this.root);\n\n this._init_header(this);\n this._init_canvas(this);\n this._init_toolbar(this);\n\n var fig = this;\n\n this.waiting = false;\n\n this.ws.onopen = function () {\n fig.send_message(\"supports_binary\", {value: fig.supports_binary});\n fig.send_message(\"send_image_mode\", {});\n if (mpl.ratio != 1) {\n fig.send_message(\"set_dpi_ratio\", {'dpi_ratio': mpl.ratio});\n }\n fig.send_message(\"refresh\", {});\n }\n\n this.imageObj.onload = function() {\n if (fig.image_mode == 'full') {\n // Full images could contain transparency (where diff images\n // almost always do), so we need to clear the canvas so that\n // there is no ghosting.\n fig.context.clearRect(0, 0, fig.canvas.width, fig.canvas.height);\n }\n fig.context.drawImage(fig.imageObj, 0, 0);\n };\n\n this.imageObj.onunload = function() {\n fig.ws.close();\n }\n\n this.ws.onmessage = this._make_on_message_function(this);\n\n this.ondownload = ondownload;\n}\n\nmpl.figure.prototype._init_header = function() {\n var titlebar = $(\n '');\n var titletext = $(\n '');\n titlebar.append(titletext)\n this.root.append(titlebar);\n this.header = titletext[0];\n}\n\n\n\nmpl.figure.prototype._canvas_extra_style = function(canvas_div) {\n\n}\n\n\nmpl.figure.prototype._root_extra_style = function(canvas_div) {\n\n}\n\nmpl.figure.prototype._init_canvas = function() {\n var fig = this;\n\n var canvas_div = $('');\n\n canvas_div.attr('style', 'position: relative; clear: both; outline: 0');\n\n function canvas_keyboard_event(event) {\n return fig.key_event(event, event['data']);\n }\n\n canvas_div.keydown('key_press', canvas_keyboard_event);\n canvas_div.keyup('key_release', canvas_keyboard_event);\n this.canvas_div = canvas_div\n this._canvas_extra_style(canvas_div)\n this.root.append(canvas_div);\n\n var canvas = $('');\n canvas.addClass('mpl-canvas');\n canvas.attr('style', \"left: 0; top: 0; z-index: 0; outline: 0\")\n\n this.canvas = canvas[0];\n this.context = canvas[0].getContext(\"2d\");\n\n var backingStore = this.context.backingStorePixelRatio ||\n\tthis.context.webkitBackingStorePixelRatio ||\n\tthis.context.mozBackingStorePixelRatio ||\n\tthis.context.msBackingStorePixelRatio ||\n\tthis.context.oBackingStorePixelRatio ||\n\tthis.context.backingStorePixelRatio || 1;\n\n mpl.ratio = (window.devicePixelRatio || 1) / backingStore;\n\n var rubberband = $('');\n rubberband.attr('style', \"position: absolute; left: 0; top: 0; z-index: 1;\")\n\n var pass_mouse_events = true;\n\n canvas_div.resizable({\n start: function(event, ui) {\n pass_mouse_events = false;\n },\n resize: function(event, ui) {\n fig.request_resize(ui.size.width, ui.size.height);\n },\n stop: function(event, ui) {\n pass_mouse_events = true;\n fig.request_resize(ui.size.width, ui.size.height);\n },\n });\n\n function mouse_event_fn(event) {\n if (pass_mouse_events)\n return fig.mouse_event(event, event['data']);\n }\n\n rubberband.mousedown('button_press', mouse_event_fn);\n rubberband.mouseup('button_release', mouse_event_fn);\n // Throttle sequential mouse events to 1 every 20ms.\n rubberband.mousemove('motion_notify', mouse_event_fn);\n\n rubberband.mouseenter('figure_enter', mouse_event_fn);\n rubberband.mouseleave('figure_leave', mouse_event_fn);\n\n canvas_div.on(\"wheel\", function (event) {\n event = event.originalEvent;\n event['data'] = 'scroll'\n if (event.deltaY < 0) {\n event.step = 1;\n } else {\n event.step = -1;\n }\n mouse_event_fn(event);\n });\n\n canvas_div.append(canvas);\n canvas_div.append(rubberband);\n\n this.rubberband = rubberband;\n this.rubberband_canvas = rubberband[0];\n this.rubberband_context = rubberband[0].getContext(\"2d\");\n this.rubberband_context.strokeStyle = \"#000000\";\n\n this._resize_canvas = function(width, height) {\n // Keep the size of the canvas, canvas container, and rubber band\n // canvas in synch.\n canvas_div.css('width', width)\n canvas_div.css('height', height)\n\n canvas.attr('width', width * mpl.ratio);\n canvas.attr('height', height * mpl.ratio);\n canvas.attr('style', 'width: ' + width + 'px; height: ' + height + 'px;');\n\n rubberband.attr('width', width);\n rubberband.attr('height', height);\n }\n\n // Set the figure to an initial 600x600px, this will subsequently be updated\n // upon first draw.\n this._resize_canvas(600, 600);\n\n // Disable right mouse context menu.\n $(this.rubberband_canvas).bind(\"contextmenu\",function(e){\n return false;\n });\n\n function set_focus () {\n canvas.focus();\n canvas_div.focus();\n }\n\n window.setTimeout(set_focus, 100);\n}\n\nmpl.figure.prototype._init_toolbar = function() {\n var fig = this;\n\n var nav_element = $('');\n nav_element.attr('style', 'width: 100%');\n this.root.append(nav_element);\n\n // Define a callback function for later on.\n function toolbar_event(event) {\n return fig.toolbar_button_onclick(event['data']);\n }\n function toolbar_mouse_event(event) {\n return fig.toolbar_button_onmouseover(event['data']);\n }\n\n for(var toolbar_ind in mpl.toolbar_items) {\n var name = mpl.toolbar_items[toolbar_ind][0];\n var tooltip = mpl.toolbar_items[toolbar_ind][1];\n var image = mpl.toolbar_items[toolbar_ind][2];\n var method_name = mpl.toolbar_items[toolbar_ind][3];\n\n if (!name) {\n // put a spacer in here.\n continue;\n }\n var button = $('');\n button.addClass('ui-button ui-widget ui-state-default ui-corner-all ' +\n 'ui-button-icon-only');\n button.attr('role', 'button');\n button.attr('aria-disabled', 'false');\n button.click(method_name, toolbar_event);\n button.mouseover(tooltip, toolbar_mouse_event);\n\n var icon_img = $('');\n icon_img.addClass('ui-button-icon-primary ui-icon');\n icon_img.addClass(image);\n icon_img.addClass('ui-corner-all');\n\n var tooltip_span = $('');\n tooltip_span.addClass('ui-button-text');\n tooltip_span.html(tooltip);\n\n button.append(icon_img);\n button.append(tooltip_span);\n\n nav_element.append(button);\n }\n\n var fmt_picker_span = $('');\n\n var fmt_picker = $('');\n fmt_picker.addClass('mpl-toolbar-option ui-widget ui-widget-content');\n fmt_picker_span.append(fmt_picker);\n nav_element.append(fmt_picker_span);\n this.format_dropdown = fmt_picker[0];\n\n for (var ind in mpl.extensions) {\n var fmt = mpl.extensions[ind];\n var option = $(\n '', {selected: fmt === mpl.default_extension}).html(fmt);\n fmt_picker.append(option);\n }\n\n // Add hover states to the ui-buttons\n $( \".ui-button\" ).hover(\n function() { $(this).addClass(\"ui-state-hover\");},\n function() { $(this).removeClass(\"ui-state-hover\");}\n );\n\n var status_bar = $('');\n nav_element.append(status_bar);\n this.message = status_bar[0];\n}\n\nmpl.figure.prototype.request_resize = function(x_pixels, y_pixels) {\n // Request matplotlib to resize the figure. Matplotlib will then trigger a resize in the client,\n // which will in turn request a refresh of the image.\n this.send_message('resize', {'width': x_pixels, 'height': y_pixels});\n}\n\nmpl.figure.prototype.send_message = function(type, properties) {\n properties['type'] = type;\n properties['figure_id'] = this.id;\n this.ws.send(JSON.stringify(properties));\n}\n\nmpl.figure.prototype.send_draw_message = function() {\n if (!this.waiting) {\n this.waiting = true;\n this.ws.send(JSON.stringify({type: \"draw\", figure_id: this.id}));\n }\n}\n\n\nmpl.figure.prototype.handle_save = function(fig, msg) {\n var format_dropdown = fig.format_dropdown;\n var format = format_dropdown.options[format_dropdown.selectedIndex].value;\n fig.ondownload(fig, format);\n}\n\n\nmpl.figure.prototype.handle_resize = function(fig, msg) {\n var size = msg['size'];\n if (size[0] != fig.canvas.width || size[1] != fig.canvas.height) {\n fig._resize_canvas(size[0], size[1]);\n fig.send_message(\"refresh\", {});\n };\n}\n\nmpl.figure.prototype.handle_rubberband = function(fig, msg) {\n var x0 = msg['x0'] / mpl.ratio;\n var y0 = (fig.canvas.height - msg['y0']) / mpl.ratio;\n var x1 = msg['x1'] / mpl.ratio;\n var y1 = (fig.canvas.height - msg['y1']) / mpl.ratio;\n x0 = Math.floor(x0) + 0.5;\n y0 = Math.floor(y0) + 0.5;\n x1 = Math.floor(x1) + 0.5;\n y1 = Math.floor(y1) + 0.5;\n var min_x = Math.min(x0, x1);\n var min_y = Math.min(y0, y1);\n var width = Math.abs(x1 - x0);\n var height = Math.abs(y1 - y0);\n\n fig.rubberband_context.clearRect(\n 0, 0, fig.canvas.width / mpl.ratio, fig.canvas.height / mpl.ratio);\n\n fig.rubberband_context.strokeRect(min_x, min_y, width, height);\n}\n\nmpl.figure.prototype.handle_figure_label = function(fig, msg) {\n // Updates the figure title.\n fig.header.textContent = msg['label'];\n}\n\nmpl.figure.prototype.handle_cursor = function(fig, msg) {\n var cursor = msg['cursor'];\n switch(cursor)\n {\n case 0:\n cursor = 'pointer';\n break;\n case 1:\n cursor = 'default';\n break;\n case 2:\n cursor = 'crosshair';\n break;\n case 3:\n cursor = 'move';\n break;\n }\n fig.rubberband_canvas.style.cursor = cursor;\n}\n\nmpl.figure.prototype.handle_message = function(fig, msg) {\n fig.message.textContent = msg['message'];\n}\n\nmpl.figure.prototype.handle_draw = function(fig, msg) {\n // Request the server to send over a new figure.\n fig.send_draw_message();\n}\n\nmpl.figure.prototype.handle_image_mode = function(fig, msg) {\n fig.image_mode = msg['mode'];\n}\n\nmpl.figure.prototype.updated_canvas_event = function() {\n // Called whenever the canvas gets updated.\n this.send_message(\"ack\", {});\n}\n\n// A function to construct a web socket function for onmessage handling.\n// Called in the figure constructor.\nmpl.figure.prototype._make_on_message_function = function(fig) {\n return function socket_on_message(evt) {\n if (evt.data instanceof Blob) {\n /* FIXME: We get \"Resource interpreted as Image but\n * transferred with MIME type text/plain:\" errors on\n * Chrome. But how to set the MIME type? It doesn't seem\n * to be part of the websocket stream */\n evt.data.type = \"image/png\";\n\n /* Free the memory for the previous frames */\n if (fig.imageObj.src) {\n (window.URL || window.webkitURL).revokeObjectURL(\n fig.imageObj.src);\n }\n\n fig.imageObj.src = (window.URL || window.webkitURL).createObjectURL(\n evt.data);\n fig.updated_canvas_event();\n fig.waiting = false;\n return;\n }\n else if (typeof evt.data === 'string' && evt.data.slice(0, 21) == \"data:image/png;base64\") {\n fig.imageObj.src = evt.data;\n fig.updated_canvas_event();\n fig.waiting = false;\n return;\n }\n\n var msg = JSON.parse(evt.data);\n var msg_type = msg['type'];\n\n // Call the \"handle_{type}\" callback, which takes\n // the figure and JSON message as its only arguments.\n try {\n var callback = fig[\"handle_\" + msg_type];\n } catch (e) {\n console.log(\"No handler for the '\" + msg_type + \"' message type: \", msg);\n return;\n }\n\n if (callback) {\n try {\n // console.log(\"Handling '\" + msg_type + \"' message: \", msg);\n callback(fig, msg);\n } catch (e) {\n console.log(\"Exception inside the 'handler_\" + msg_type + \"' callback:\", e, e.stack, msg);\n }\n }\n };\n}\n\n// from http://stackoverflow.com/questions/1114465/getting-mouse-location-in-canvas\nmpl.findpos = function(e) {\n //this section is from http://www.quirksmode.org/js/events_properties.html\n var targ;\n if (!e)\n e = window.event;\n if (e.target)\n targ = e.target;\n else if (e.srcElement)\n targ = e.srcElement;\n if (targ.nodeType == 3) // defeat Safari bug\n targ = targ.parentNode;\n\n // jQuery normalizes the pageX and pageY\n // pageX,Y are the mouse positions relative to the document\n // offset() returns the position of the element relative to the document\n var x = e.pageX - $(targ).offset().left;\n var y = e.pageY - $(targ).offset().top;\n\n return {\"x\": x, \"y\": y};\n};\n\n/*\n * return a copy of an object with only non-object keys\n * we need this to avoid circular references\n * http://stackoverflow.com/a/24161582/3208463\n */\nfunction simpleKeys (original) {\n return Object.keys(original).reduce(function (obj, key) {\n if (typeof original[key] !== 'object')\n obj[key] = original[key]\n return obj;\n }, {});\n}\n\nmpl.figure.prototype.mouse_event = function(event, name) {\n var canvas_pos = mpl.findpos(event)\n\n if (name === 'button_press')\n {\n this.canvas.focus();\n this.canvas_div.focus();\n }\n\n var x = canvas_pos.x * mpl.ratio;\n var y = canvas_pos.y * mpl.ratio;\n\n this.send_message(name, {x: x, y: y, button: event.button,\n step: event.step,\n guiEvent: simpleKeys(event)});\n\n /* This prevents the web browser from automatically changing to\n * the text insertion cursor when the button is pressed. We want\n * to control all of the cursor setting manually through the\n * 'cursor' event from matplotlib */\n event.preventDefault();\n return false;\n}\n\nmpl.figure.prototype._key_event_extra = function(event, name) {\n // Handle any extra behaviour associated with a key event\n}\n\nmpl.figure.prototype.key_event = function(event, name) {\n\n // Prevent repeat events\n if (name == 'key_press')\n {\n if (event.which === this._key)\n return;\n else\n this._key = event.which;\n }\n if (name == 'key_release')\n this._key = null;\n\n var value = '';\n if (event.ctrlKey && event.which != 17)\n value += \"ctrl+\";\n if (event.altKey && event.which != 18)\n value += \"alt+\";\n if (event.shiftKey && event.which != 16)\n value += \"shift+\";\n\n value += 'k';\n value += event.which.toString();\n\n this._key_event_extra(event, name);\n\n this.send_message(name, {key: value,\n guiEvent: simpleKeys(event)});\n return false;\n}\n\nmpl.figure.prototype.toolbar_button_onclick = function(name) {\n if (name == 'download') {\n this.handle_save(this, null);\n } else {\n this.send_message(\"toolbar_button\", {name: name});\n }\n};\n\nmpl.figure.prototype.toolbar_button_onmouseover = function(tooltip) {\n this.message.textContent = tooltip;\n};\nmpl.toolbar_items = [[\"Home\", \"Reset original view\", \"fa fa-home icon-home\", \"home\"], [\"Back\", \"Back to previous view\", \"fa fa-arrow-left icon-arrow-left\", \"back\"], [\"Forward\", \"Forward to next view\", \"fa fa-arrow-right icon-arrow-right\", \"forward\"], [\"\", \"\", \"\", \"\"], [\"Pan\", \"Pan axes with left mouse, zoom with right\", \"fa fa-arrows icon-move\", \"pan\"], [\"Zoom\", \"Zoom to rectangle\", \"fa fa-square-o icon-check-empty\", \"zoom\"], [\"\", \"\", \"\", \"\"], [\"Download\", \"Download plot\", \"fa fa-floppy-o icon-save\", \"download\"]];\n\nmpl.extensions = [\"eps\", \"pdf\", \"png\", \"ps\", \"raw\", \"svg\"];\n\nmpl.default_extension = \"png\";var comm_websocket_adapter = function(comm) {\n // Create a \"websocket\"-like object which calls the given IPython comm\n // object with the appropriate methods. Currently this is a non binary\n // socket, so there is still some room for performance tuning.\n var ws = {};\n\n ws.close = function() {\n comm.close()\n };\n ws.send = function(m) {\n //console.log('sending', m);\n comm.send(m);\n };\n // Register the callback with on_msg.\n comm.on_msg(function(msg) {\n //console.log('receiving', msg['content']['data'], msg);\n // Pass the mpl event to the overridden (by mpl) onmessage function.\n ws.onmessage(msg['content']['data'])\n });\n return ws;\n}\n\nmpl.mpl_figure_comm = function(comm, msg) {\n // This is the function which gets called when the mpl process\n // starts-up an IPython Comm through the \"matplotlib\" channel.\n\n var id = msg.content.data.id;\n // Get hold of the div created by the display call when the Comm\n // socket was opened in Python.\n var element = $(\"#\" + id);\n var ws_proxy = comm_websocket_adapter(comm)\n\n function ondownload(figure, format) {\n window.open(figure.imageObj.src);\n }\n\n var fig = new mpl.figure(id, ws_proxy,\n ondownload,\n element.get(0));\n\n // Call onopen now - mpl needs it, as it is assuming we've passed it a real\n // web socket which is closed, not our websocket->open comm proxy.\n ws_proxy.onopen();\n\n fig.parent_element = element.get(0);\n fig.cell_info = mpl.find_output_cell(\"\");\n if (!fig.cell_info) {\n console.error(\"Failed to find cell for figure\", id, fig);\n return;\n }\n\n var output_index = fig.cell_info[2]\n var cell = fig.cell_info[0];\n\n};\n\nmpl.figure.prototype.handle_close = function(fig, msg) {\n var width = fig.canvas.width/mpl.ratio\n fig.root.unbind('remove')\n\n // Update the output cell to use the data from the current canvas.\n fig.push_to_output();\n var dataURL = fig.canvas.toDataURL();\n // Re-enable the keyboard manager in IPython - without this line, in FF,\n // the notebook keyboard shortcuts fail.\n IPython.keyboard_manager.enable()\n $(fig.parent_element).html('