Examples

The following examples will reference variables that may not be defined within the block they are used in. For clarity, we define them here:

import numpy as np

rate = 10.0
np.random.seed(1234)
data_len = 1000
ephys_data = np.random.rand(data_len)
ephys_timestamps = np.arange(data_len) / rate
spatial_timestamps = ephys_timestamps[::10]
spatial_data = np.cumsum(np.random.normal(size=(2, len(spatial_timestamps))), axis=-1).T

Creating and Writing NWB files

When creating a NWB file, the first step is to create the NWBFile. The first argument is the name of the NWB file, and the second argument is a brief description of the dataset.

from datetime import datetime
from pynwb import NWBFile

f = NWBFile('the PyNWB tutorial', 'my first synthetic recording', 'EXAMPLE_ID', datetime.now(),
            experimenter='Dr. Bilbo Baggins',
            lab='Bag End Laboratory',
            institution='University of Middle Earth at the Shire',
            experiment_description='I went on an adventure with thirteen dwarves to reclaim vast treasures.',
            session_id='LONELYMTN')

Once you have created your NWB and added all of your data and other necessary metadata, you can write it to disk using the HDF5IO class.

from pynwb import get_manager
from pynwb.form.backends.hdf5 import HDF5IO

filename = "example.h5"
io = HDF5IO(filename, manager=get_manager(), mode='w')
io.write(f)
io.close()

Creating Epochs

Experimental epochs are represented with Epoch objects. To create epochs for an NWB file, you can use the NWBFile instance method create_epoch.

epoch_tags = ('example_epoch',)

ep1 = f.create_epoch(source='an hypothetical source', name='epoch1', start=0.0, stop=1.0,
                     tags=epoch_tags,
                     description="the first test epoch")

ep2 = f.create_epoch(source='an hypothetical source', name='epoch2', start=0.0, stop=1.0,
                     tags=epoch_tags,
                     description="the second test epoch")

Creating Electrode Groups

Electrode groups (i.e. experimentally relevant groupings of channels) are represented by ElectrodeGroup objects. To create an electrode group, you can use the NWBFile instance method create_electrode_group.

Before creating an ElectrodeGroup, you need to provide some information about the device that was used to record from the electrode. This is done by creating a Device object using the instance method create_device.

device = f.create_device(name='trodes_rig123', source="a source")

Once you have created the Device, you can create the ElectrodeGroup.

electrode_name = 'tetrode1'
source = "an hypothetical source"
description = "an example tetrode"
location = "somewhere in the hippocampus"

electrode_group = f.create_electrode_group(electrode_name,
                                           source=source,
                                           description=description,
                                           location=location,
                                           device=device)

Finally, you can then create the associated ElectrodeTable and ElectrodeTableRegion.

from pynwb.ecephys import ElectrodeTable, ElectrodeTableRegion

electrode_table = ElectrodeTable('electrodes')
for idx in [1, 2, 3, 4]:
    electrode_table.add_row(idx,
                            x=1.0, y=2.0, z=3.0,
                            imp=float(-idx),
                            location='CA1', filtering='none',
                            description='channel %s' % idx, group=electrode_group)

electrode_table_region = ElectrodeTableRegion(electrode_table, [0, 2], 'the first and third electrodes')

Creating TimeSeries

TimeSeries objects can be created by instantiating TimeSeries objects directly and then adding them to the NWBFile using the instance method add_acquisition.

This first example will demonstrate instantiating two different types of TimeSeries objects directly, and adding them with add_acquisition.

from pynwb.ecephys import ElectricalSeries
from pynwb.behavior import SpatialSeries

ephys_ts = ElectricalSeries('test_ephys_data',
                            'an hypothetical source',
                            ephys_data,
                            electrode_table_region,
                            timestamps=ephys_timestamps,
                            # Alternatively, could specify starting_time and rate as follows
                            # starting_time=ephys_timestamps[0],
                            # rate=rate,
                            resolution=0.001,
                            comments="This data was randomly generated with numpy, using 1234 as the seed",
                            description="Random numbers generated with numpy.random.rand")
f.add_acquisition(ephys_ts, [ep1, ep2])

spatial_ts = SpatialSeries('test_spatial_timeseries',
                           'a stumbling rat',
                           spatial_data,
                           'origin on x,y-plane',
                           timestamps=spatial_timestamps,
                           resolution=0.1,
                           comments="This data was generated with numpy, using 1234 as the seed",
                           description="This 2D Brownian process generated with "
                                       "np.cumsum(np.random.normal(size=(2, len(spatial_timestamps))), axis=-1).T")
f.add_acquisition(spatial_ts, [ep1, ep2])

Using Extensions

The NWB file format supports extending existing data types (See Extending NWB for more details on creating extensions). Extensions must be registered with PyNWB to be used for reading and writing of custom neurodata types.

The following code demonstrates how to load custom namespaces.

from pynwb import load_namespaces
namespace_path = 'my_namespace.yaml'
load_namespaces(namespace_path)

Note

This will register all namespaces defined in the file 'my_namespace.yaml'.

To read and write custom data, corresponding NWBContainer classes must be associated with their respective specifications. NWBContainer classes are associated with their respective specification using the decorator register_class.

The following code demonstrates how to associate a specification with the NWBContainer class that represents it.

from pynwb import register_class
@register_class('my_namespace', 'MyExtension')
class MyExtensionContainer(NWBContainer):
    ...

register_class can also be used as a function.

from pynwb import register_class
class MyExtensionContainer(NWBContainer):
    ...
register_class('my_namespace', 'MyExtension', MyExtensionContainer)

If your NWBContainer extension requires custom mapping of the NWBContainer class for reading and writing, you will need to implement and register a custom ObjectMapper.

ObjectMapper extensions are registered with the decorator register_map.

from pynwb import register_map
from form import ObjectMapper
@register_map(MyExtensionContainer)
class MyExtensionMapper(ObjectMapper)
    ...

register_map can also be used as a function.

from pynwb import register_map
from form import ObjectMapper
class MyExtensionMapper(ObjectMapper)
    ...
register_map(MyExtensionContainer, MyExtensionMapper)

If you do not have an NWBContainer subclass to associate with your extension specification, a dynamically created class is created by default.

To use the dynamic class, you will need to retrieve the class object using the function get_class. Once you have retrieved the class object, you can use it just like you would a statically defined class.

from pynwb import get_class
MyExtensionContainer = get_class('my_namespace', 'MyExtension')
my_ext_inst = MyExtensionContainer(...)

If using iPython, you can access documentation for the class’s constructor using the help command.

Write an NWBFile

Writing NWB files to disk is handled by the pynwb.form package. Currently, the only storage format supported by pynwb.form is HDF5.

Reading and writing to and from HDF5 is handled by the class HDF5IO. The only required argument to this is the path of the HDF5 file. A second, optional argument is the BuildManager to use for IO.

Briefly, the BuildManager is a class that manages objects to be read and written from disk. A PyNWB-specific BuildManager can be retrieved using the module-level function get_manager.

Alternatively, the BuildManager that a FORMIO used can be retrieved from the manager attribute.

from datetime import datetime

from pynwb import NWBFile, TimeSeries, get_manager
from pynwb.form.backends.hdf5 import HDF5IO

start_time = datetime(1970, 1, 1, 12, 0, 0)
create_date = datetime(2017, 4, 15, 12, 0, 0)

nwbfile = NWBFile('the PyNWB tutorial', 'a test NWB File', 'TEST123', start_time,
                  file_create_date=create_date)

ts = TimeSeries('test_timeseries', 'example_source', list(range(100, 200, 10)), 'SIunit',
                timestamps=list(range(10)),
                resolution=0.1)

nwbfile.add_acquisition(ts)

io = HDF5IO("example.h5", manager=get_manager(), mode='w')
io.write(nwbfile)
io.close()

Note

All FORMIO objects are context managers.

The third argument to the HDF5IO constructor is the mode for opening the HDF5 file. Valid modes are:

r Readonly, file must exist
r+ Read/write, file must exist
w Create file, truncate if exists
w- or x Create file, fail if exists
a Read/write if exists, create otherwise (default)

Extending NWB

Creating new Extensions

The NWB specification is designed to be extended. Extension for the NWB format can be done so using classes provided in the pynwb.spec module. The classes NWBGroupSpec, NWBDatasetSpec, NWBAttributeSpec, and NWBLinkSpec can be used to define custom types.

Attribute Specifications

Specifying attributes is done with NWBAttributeSpec.

from pynwb.spec import NWBAttributeSpec

spec = NWBAttributeSpec('bar', 'float', 'a value for bar')

Dataset Specifications

Specifying datasets is done with NWBDatasetSpec.

from pynwb.spec import NWBDatasetSpec

spec = NWBDatasetSpec('A custom NWB type',
                    attribute=[
                        NWBAttributeSpec('baz', 'str', 'a value for baz'),
                    ],
                    shape=(None, None))
Using datasets to specify tables

Tables can be specified using NWBDtypeSpec. To specify a table, provide a list of NWBDtypeSpec objects to the dtype argument.

from pynwb.spec import NWBDatasetSpec, NWBDtypeSpec

spec = NWBDatasetSpec('A custom NWB type',
                    attribute=[
                        NWBAttributeSpec('baz', 'str', 'a value for baz'),
                    ],
                    dtype=[
                        NWBDtypeSpec('foo', column for foo', 'int'),
                        NWBDtypeSpec('bar', 'a column for bar', 'float')
                    ])

Compound data types can be nested.

from pynwb.spec import NWBDatasetSpec, NWBDtypeSpec

spec = NWBDatasetSpec('A custom NWB type',
                    attribute=[
                        NWBAttributeSpec('baz', 'a value for baz', 'str'),
                    ],
                    dtype=[
                        NWBDtypeSpec('foo', 'a column for foo', 'int'),
                        NWBDtypeSpec('bar', 'a column for bar', 'float')
                    ])

Group Specifications

Specifying groups is done with the NWBGroupSpec class.

from pynwb.spec import NWBGroupSpec

# A list of NWBAttributeSpec objects to specify new attributes
addl_attributes = [...]
# A list of NWBDatasetSpec objects to specify new datasets
addl_datasets = [...]
# A list of NWBDatasetSpec objects to specify new groups
addl_groups = [...]
spec = NWBGroupSpec('A custom NWB type',
                    attributes = addl_attributes,
                    datasets = addl_datasets,
                    groups = addl_groups)

Neurodata Type Specifications

NWBGroupSpec and NWBDatasetSpec use the arguments neurodata_type_inc and neurodata_type_def for declaring new types and extending existing types. New types are specified by setting the argument neurodata_type_def. New types can extend an existing type by specifying the argument neurodata_type_inc.

Create a new type

from pynwb.spec import NWBGroupSpec

# A list of NWBAttributeSpec objects to specify new attributes
addl_attributes = [...]
# A list of NWBDatasetSpec objects to specify new datasets
addl_datasets = [...]
# A list of NWBDatasetSpec objects to specify new groups
addl_groups = [...]
spec = NWBGroupSpec('A custom NWB type',
                    attributes = addl_attributes,
                    datasets = addl_datasets,
                    groups = addl_groups,
                    neurodata_type_def='MyNewNWBType')

Extend an existing type

from pynwb.spec import NWBGroupSpec

# A list of NWBAttributeSpec objects to specify additional attributes or attributes to be overriden
addl_attributes = [...]
# A list of NWBDatasetSpec objects to specify additional datasets or datasets to be overriden
addl_datasets = [...]
# A list of NWBGroupSpec objects to specify additional groups or groups to be overriden
addl_groups = [...]
spec = NWBGroupSpec('An extended NWB type',
                    attributes = addl_attributes,
                    datasets = addl_datasets,
                    groups = addl_groups,
                    neurodata_type_inc='Clustering',
                    neurodata_type_def='MyExtendedClustering')

Existing types can be instantiate by specifying neurodata_type_inc alone.

from pynwb.spec import NWBGroupSpec

# use another NWBGroupSpec object to specify that a group of type
# ElectricalSeries should be present in the new type defined below
addl_groups = [ NWBGroupSpec('An included ElectricalSeries instance',
                             neurodata_type_inc='ElectricalSeries') ]

spec = NWBGroupSpec('An extended NWB type',
                    groups = addl_groups,
                    neurodata_type_inc='Clustering',
                    neurodata_type_def='MyExtendedClustering')

Datasets can be extended in the same manner (with regard to neurodata_type_inc and neurodata_type_def, by using the class NWBDatasetSpec.

Saving Extensions

Extensions are used by including them in a loaded namespace. Namespaces and extensions need to be saved to file for downstream use. The class NWBNamespaceBuilder can be used to create new namespace and specification files.

Note

When using NWBNamespaceBuilder, the core NWB namespace is automatically included

Create a new namespace with extensions

from pynwb.spec import NWBGroupSpec, NWBNamespaceBuilder

# create a builder for the namespace
ns_builder = NWBNamespaceBuilder("Extension for use in my laboratory", "mylab", ...)

# create extensions
ext1 = NWBGroupSpec('A custom Clustering interface',
                    attributes = [...]
                    datasets = [...],
                    groups = [...],
                    neurodata_type_inc='Clustering',
                    neurodata_type_def='MyExtendedClustering')

ext2 = NWBGroupSpec('A custom ClusterWaveforms interface',
                    attributes = [...]
                    datasets = [...],
                    groups = [...],
                    neurodata_type_inc='ClusterWaveforms',
                    neurodata_type_def='MyExtendedClusterWaveforms')


# add the extension
ext_source = 'mylab.specs.yaml'
ns_builder.add_spec(ext_source, ext1)
ns_builder.add_spec(ext_source, ext2)

# include an existing namespace - this will include all specifications in that namespace
ns_builder.include_namespace('collab_ns')

# save the namespace and extensions
ns_path = 'mylab.namespace.yaml'
ns_builder.export(ns_path)

Tip

Using the API to generate extensions (rather than writing YAML sources directly) helps avoid errors in the specification (e.g., due to missing required keys or invalid values) and ensure compliance of the extension definition with the NWB specification language. It also helps with maintanence of extensions, e.g., if extensions have to be ported to newer versions of the specification language in the future.

Documenting Extensions

Using the same tools used to generate the documentation for the NWB-N core format one can easily generate documentation in HTML, PDF, ePub and many other format for extensions as well.

For the purpose of this example we assume that our current directory has the following structure.

- nwb_schema (cloned from `https://github.com/NeurodataWithoutBorders/nwb-schema`)
- my_extension/
    - my_extension_source/
        - mylab.namespace.yaml
        - mylab.specs.yaml
        - ...
        - docs/  (Optional)
            - mylab_description.rst
            - mylab_release_notes.rst

In addition to Python 3.x you will also need sphinx (including the sphinx-quickstart tool) installed. Sphinx is availble here http://www.sphinx-doc.org/en/stable/install.html .

We can now create the sources of our documentation as follows:

python3 nwb-schema/docs/utils/init_sphinx_extension_doc.py \
             --project test \
             --author "Dr. Master Expert" \
             --version "1.2.3" \
             --release alpha \
             --output my_extension_docs \
             --spec_dir my_extension_source \
             --namespace_filename mylab.namespace.yaml \
             --default_namespace mylab
             --external_description my_extension_source/docs/mylab_description.rst \  (Optional)
             --external_release_notes my_extension_source/docs/mylab_release_notes.rst \  (Optional)

The new folder my_extension_docs/ now contains the basic setup for the documentation. To automatically generate the RST documentation files from the YAML (or JSON) sources of the extension simply run:

cd my_extension_docs
make apidoc

Finally, to generate the HTML version of the docs run:

make html

Tip

Additional instructions for how to use and customize the extension documentations are also available in the Readme.md file that init_sphinx_extension_doc.py automatically adds to the docs.

Tip

See make help for a list of available options for building the documentation in many different output formats (e.g., PDF, ePub, LaTeX, etc.).

Tip

See python3 init_sphinx_extension_doc.py --help for a complete list of option to customize the documentation directly during initialization.

Tip

The above example included additional description and release note docs as part of the specification. These are included in the docs via .. include commands so that changes in those files are automatically picked up when rebuilding to docs. Alternatively, we can also add custom documentation directly to the docs. In this case the options --custom_description format_description.rst and --custom_release_notes format_release_notes.rst of the init_sphinx_extension_doc.py script are useful to automatically generate the basic setup for those files so that one can easily start to add content directly without having to worry about the additional setup.

Further Reading

Validating NWB files

Validating NWB files is handled by a command-line tool availble in pynwb. The validator can be invoked like so:

python -m pynwb.validate test.nwb

This will validate the file test.nwb against the core NWB specification. Validating against other specifications i.e. extensions can be done using the -p and -n flags. For example, the following command will validate against the specifications referenced in the namespace file mylab.namespace.yaml in addition to the core specification.

python -m pynwb.validate -p mylab.namespace.yaml test.nwb