.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/general/scratch.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_tutorials_general_scratch.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_tutorials_general_scratch.py:


.. _scratch:

Exploratory Data Analysis with NWB
==================================

This example will focus on the basics of working with an :py:class:`~pynwb.file.NWBFile` to do more than
storing standardized data for use and exchange. For example, you may want to store results from intermediate
analyses or one-off analyses with unknown utility. This functionality is primarily accomplished with linking
and scratch space.


.. note::
    The scratch space is explicitly for non-standardized data that is not intended for reuse
    by others. Standard NWB types, and extension if required, should always be used for any data that you
    intend to share. As such, published data should not include scratch data and a user should be able
    to ignore any data stored in scratch to use a file.

.. GENERATED FROM PYTHON SOURCE LINES 24-29

Raw data
--------

To demonstrate linking and scratch space, lets assume we are starting with some acquired data.


.. GENERATED FROM PYTHON SOURCE LINES 29-63

.. code-block:: Python


    from datetime import datetime

    import numpy as np
    from dateutil.tz import tzlocal

    from pynwb import NWBHDF5IO, NWBFile, TimeSeries

    # set up the NWBFile
    start_time = datetime(2019, 4, 3, 11, tzinfo=tzlocal())
    create_date = datetime(2019, 4, 15, 12, tzinfo=tzlocal())

    nwb = NWBFile(
        session_description="demonstrate NWBFile scratch",  # required
        identifier="NWB456",  # required
        session_start_time=start_time,  # required
        file_create_date=create_date,
    )  # optional

    # make some fake data
    timestamps = np.linspace(0, 100, 1024)
    data = (
        np.sin(0.333 * timestamps)
        + np.cos(0.1 * timestamps)
        + np.random.randn(len(timestamps))
    )
    test_ts = TimeSeries(name="raw_timeseries", data=data, unit="m", timestamps=timestamps)

    # add it to the NWBFile
    nwb.add_acquisition(test_ts)

    with NWBHDF5IO("raw_data.nwb", mode="w") as io:
        io.write(nwb)


.. GENERATED FROM PYTHON SOURCE LINES 65-72

.. _basic_copying:

Copying an NWB file
-------------------

To copy a file, we must first read the file.


.. GENERATED FROM PYTHON SOURCE LINES 72-75

.. code-block:: Python

    raw_io = NWBHDF5IO("raw_data.nwb", "r")
    nwb_in = raw_io.read()


.. GENERATED FROM PYTHON SOURCE LINES 76-78

And then create a shallow copy the file with the :py:func:`~pynwb.file.NWBFile.copy` method
of :py:class:`~pynwb.file.NWBFile` .

.. GENERATED FROM PYTHON SOURCE LINES 78-81

.. code-block:: Python


    nwb_proc = nwb_in.copy()


.. GENERATED FROM PYTHON SOURCE LINES 82-84

Now that we have a copy, lets process some data, and add the results as a :py:class:`~pynwb.base.ProcessingModule`
to our copy of the file. [#]_

.. GENERATED FROM PYTHON SOURCE LINES 85-97

.. code-block:: Python


    mod = nwb_proc.create_processing_module(
        "filtering_module", "a module to store filtering results"
    )

    ts1 = nwb_in.acquisition["raw_timeseries"]
    filt_data = np.convolve(ts1.data, np.ones(128), mode="same") / 128
    ts2 = TimeSeries(name="filtered_timeseries", data=filt_data, unit="m", timestamps=ts1)

    mod.add_container(ts2)



.. GENERATED FROM PYTHON SOURCE LINES 98-99

Now write the copy, which contains the processed data. [#]_

.. GENERATED FROM PYTHON SOURCE LINES 100-105

.. code-block:: Python


    with NWBHDF5IO("processed_data.nwb", mode="w", manager=raw_io.manager) as io:
        io.write(nwb_proc)



.. GENERATED FROM PYTHON SOURCE LINES 106-117

.. [#]
   .. note::
      Notice here that we are reusing the timestamps to the original TimeSeries.

.. [#]
   .. note::
      The ``processed_data.nwb`` file (i.e., our copy) stores our processing module and contains external
      links to all data in our original file, i.e., the data from our raw file is being linked to, not copied.
      This allows us to isolate our processing data in a separate file while still allowing us to access the
      raw data from our ``processed_data.nwb`` file, without having to duplicate the data.


.. GENERATED FROM PYTHON SOURCE LINES 121-131

.. _basic_scratch:

Adding scratch data
-------------------

You may end up wanting to store results from some one-off analysis, and writing an extension
to get your data into an NWBFile is too much over head. This is facilitated by the scratch space
in NWB. [#]_

First, lets read our processed data and then make a copy

.. GENERATED FROM PYTHON SOURCE LINES 131-135

.. code-block:: Python


    proc_io = NWBHDF5IO("processed_data.nwb", "r")
    nwb_proc_in = proc_io.read()


.. GENERATED FROM PYTHON SOURCE LINES 136-137

Now make a copy to put our scratch data into [#]_

.. GENERATED FROM PYTHON SOURCE LINES 138-141

.. code-block:: Python


    nwb_scratch = nwb_proc_in.copy()


.. GENERATED FROM PYTHON SOURCE LINES 142-144

Now lets do an analysis for which we do not have a specification, but we would like to store
the results for.

.. GENERATED FROM PYTHON SOURCE LINES 145-158

.. code-block:: Python


    filt_ts = nwb_scratch.processing["filtering_module"]["filtered_timeseries"]

    # simple power spectrum without normalization
    ps = np.abs(np.fft.fft(filt_ts.data))**2

    nwb_scratch.add_scratch(
        ps,
        name="power_spectrum",
        description="power spectrum from filtered data",
    )



.. GENERATED FROM PYTHON SOURCE LINES 159-160

Finally, write the results.

.. GENERATED FROM PYTHON SOURCE LINES 161-165

.. code-block:: Python


    with NWBHDF5IO("scratch_analysis.nwb", "w", manager=proc_io.manager) as io:
        io.write(nwb_scratch)


.. GENERATED FROM PYTHON SOURCE LINES 166-168

To get your results back, you can index into :py:attr:`~pynwb.file.NWBFile.scratch` or use
:py:func:`~pynwb.file.NWBFile.get_scratch`:

.. GENERATED FROM PYTHON SOURCE LINES 169-177

.. code-block:: Python


    scratch_io = NWBHDF5IO("scratch_analysis.nwb", "r")
    nwb_scratch_in = scratch_io.read()

    fft_in = nwb_scratch_in.scratch["power_spectrum"]

    fft_in = nwb_scratch_in.get_scratch("power_spectrum")


.. GENERATED FROM PYTHON SOURCE LINES 178-186

.. [#]
   .. note::
      This scratch space only exists if you add scratch data.

.. [#]
   .. note::
      We recommend writing scratch data into copies of files only. This will make it easier to
      isolate and discard scratch data and avoids updating files that store precious data.

.. GENERATED FROM PYTHON SOURCE LINES 187-192

.. code-block:: Python


    # close the IO objects
    raw_io.close()
    proc_io.close()
    scratch_io.close()


.. _sphx_glr_download_tutorials_general_scratch.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: scratch.ipynb <scratch.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: scratch.py <scratch.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: scratch.zip <scratch.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_