Software Architecture

The main goal of PyNWB is to enable users and developers to efficiently interact with the NWB data format, format files, and specifications. The following figures provide an overview of the high-level architecture of PyNWB and functionality of the various components.

PyNWB Software Architecture

Overview of the high-level software architecture of PyNWB (click to enlarge).

PyNWB Software Architecture Functions

We choose a modular design for PyNWB to enable flexibility and separate the various aspects of the NWB:N ecosystem (click to enlarge).

Main Concepts

PyNWB Software Architecture Concepts

Overview of the main concepts/classes in PyNWB and their location in the overall software architecture (click to enlarge).

Container

Builder

Spec

  • Interact with format specifications

  • Data structures to specify data types and what said types consist of

  • Python representation for YAML specifications

  • Interface for writing extensions or custom specification

  • There are several main specification classes:

    • NWBAttributeSpec - specification for metadata

    • NWBGroupSpec - specification for a collection of objects (i.e. subgroups, datasets, link)

    • NWBDatasetSpec - specification for dataset (like and n-dimensional array). Specifies data type, dimensions, etc.

    • NWBLinkSpec - specification for link (like a POSIX soft link)

    • RefSpec - specification for references (References are like links, but stored as data)

    • NWBDtypeSpec - specification for compound data types. Used to build complex data type specification, e.g., to define tables (used only in DatasetSpec and correspondingly NWBDatasetSpec)

  • Main Modules:

    • hdmf.spec – General specification classes.

    • pynwb.spec – NWB specification classes. (Most of these are specializations of the classes from hdmf.spec)

Note

A data_type (or more specifically a neurodata_type in the context of NWB) defines a reusable type in a format specification that can be referenced and used elsewhere in other specifications. The specification of the NWB format is basically a collection of neurodata_types, e.g.: NWBFile defines a GroupSpec for the top-level group of an NWB format file which includes TimeSeries, ElectrodeGroup, ImagingPlane and many other neurodata_types . When creating a specification, two main keys are used to include and define new neurodata_types

  • neurodata_type_inc is used to include an existing type and

  • neurodata_type_def is used to define a new type

I.e, if both keys are defined then we create a new type that uses/inherits an existing type as a base.

ObjectMapper

  • Maintains the mapping between Container attributes and Spec components

  • Provides a way of converting between Container and Builder

  • ObjectMappers are constructed using a Spec

  • Ideally, one ObjectMapper for each data type

  • Things an ObjectMapper should do:

    • Given a Builder, return a Container representation

    • Given a Container, return a Builder representation

  • PyNWB has many of these – one for each type in NWB schema

  • Main Module: hdmf.build.objectmapper

    • NWB-specific ObjectMappers are locate in submodules of pynwb.io

PyNWB Software Architecture Main Concepts

Relationship between Container, Builder, ObjectMapper, and Spec

Additional Concepts

Namespace, NamespaceCatalog, NamespaceBuilder

  • Namespace

    • A namespace for specifications

    • Necessary for making extensions

    • Contains basic info about who created extensions

    • Core NWB:N schema has namespace “core”

    • Get from pynwb.spec.NWBNamespace

      • extension of generic Namespace class that will include core

  • NamespaceCatalog – A class for managing namespaces

  • NamespaceBuilder – A utility for building extensions

TypeMap

  • Map between data types, Container classes (i.e. a Python class object) and corresponding ObjectMapper classes

  • Constructed from a NamespaceCatalog

  • Things a TypeMap does:

    • Given an NWB data type, return the associated Container class

    • Given a Container class, return the associated ObjectMapper

  • PyNWB has two of these classes:

  • PyNWB provides a “global” instance of TypeMap created at runtime

  • TypeMaps can be merged, which is useful when combining extensions

BuildManager

PyNWB Software Architecture BuildManager and TypeMap

Overview of BuildManager (and TypeMap) (click to enlarge).

HDMFIO

  • Abstract base class for I/O

  • HDMFIO has two key abstract methods:

  • Constructed with a BuildManager

  • Extend this for creating a new I/O backend

  • PyNWB has one extension of this:

    • HDF5IO - reading and writing HDF5

    • NWBHDF5IO - wrapper that pulls in core NWB specification

PyNWB Software Architecture FormIO

Overview of HDMFIO (click to enlarge).