Caffe2 - Python API
A deep learning, cross platform ML framework
Public Member Functions | Public Attributes | List of all members
caffe2.python.dataset.Dataset Class Reference
Inheritance diagram for caffe2.python.dataset.Dataset:

Public Member Functions

def __init__ (self, fields, name=None)
 
def trim (self, net, multiple_of)
 
def init_empty (self, init_net)
 
def init_from_dataframe (self, net, dataframe)
 
def get_blobs (self)
 
def content (self)
 
def field_names (self)
 
def field_types (self)
 
def reader (self, init_net=None, cursor_name=None, batch_size=1, enforce_batch_size=False)
 
def random_reader (self, init_net=None, indices=None, cursor_name=None, batch_size=1, loop_over=False, enforce_batch_size=False)
 
def writer (self, init_net=None)
 

Public Attributes

 schema
 
 fields
 
 field_types
 
 name
 
 field_blobs
 

Detailed Description

Represents an in-memory dataset with fixed schema.

Use this to store and iterate through datasets with complex schema that
fit in memory.

Iterating through entries of this dataset is very fast since the dataset
is stored as a set of native Caffe2 tensors, thus no type conversion or
deserialization is necessary.

Definition at line 196 of file dataset.py.

Constructor & Destructor Documentation

def caffe2.python.dataset.Dataset.__init__ (   self,
  fields,
  name = None 
)
Create an un-initialized dataset with schema provided by `fields`.

Before this dataset can be used, it must be initialized, either by
`init_empty` or `init_from_dataframe`.

Args:
    fields: either a schema.Struct or a list of field names in a format
    compatible with the one described in schema.py.
    name: optional name to prepend to blobs that will store the data.

Definition at line 207 of file dataset.py.

Member Function Documentation

def caffe2.python.dataset.Dataset.content (   self)
Return a Record of BlobReferences pointing to the full content of
this dataset.

Definition at line 267 of file dataset.py.

def caffe2.python.dataset.Dataset.field_names (   self)
Return the list of field names for this dataset.

Definition at line 274 of file dataset.py.

def caffe2.python.dataset.Dataset.field_types (   self)
Return the list of field dtypes for this dataset.

If a list of strings, not a schema.Struct, was passed to the
constructor, this will return a list of dtype(np.void).

Definition at line 278 of file dataset.py.

def caffe2.python.dataset.Dataset.get_blobs (   self)
Return the list of BlobReference pointing to the blobs that contain
the data for this dataset.

Definition at line 259 of file dataset.py.

def caffe2.python.dataset.Dataset.init_empty (   self,
  init_net 
)
Initialize the blobs for this dataset with empty values.

Empty arrays will be immediately fed into the current workspace,
and `init_net` will take those blobs as external inputs.

Definition at line 239 of file dataset.py.

def caffe2.python.dataset.Dataset.init_from_dataframe (   self,
  net,
  dataframe 
)
Initialize the blobs for this dataset from a Pandas dataframe.

Each column of the dataframe will be immediately fed into the current
workspace, and the `net` will take this blobs as external inputs.

Definition at line 248 of file dataset.py.

def caffe2.python.dataset.Dataset.random_reader (   self,
  init_net = None,
  indices = None,
  cursor_name = None,
  batch_size = 1,
  loop_over = False,
  enforce_batch_size = False 
)
Create a Reader object that is used to iterate through the dataset.

NOTE: The reader order depends on the order in indices.

Args:
    init_net: net that will be run once to create the cursor.
    indices: blob of reading order
    cursor_name: optional name for the blob containing a pointer
         to the cursor.
    batch_size: how many samples to read per iteration.
    loop_over: repeat the dataset indefinitely (in the same order)

Returns:
    A DatasetReader that can be used to create operators that will
    iterate through the dataset according to indices.

Definition at line 314 of file dataset.py.

def caffe2.python.dataset.Dataset.reader (   self,
  init_net = None,
  cursor_name = None,
  batch_size = 1,
  enforce_batch_size = False 
)
Create a Reader object that is used to iterate through the dataset.

This will append operations to `init_net` that create a TreeCursor,
used to iterate through the data.

NOTE: Currently, it is not safe to append to a dataset while reading.

Args:
    init_net: net that will be run once to create the cursor.
    cursor_name: optional name for the blob containing a pointer
         to the cursor.
    batch_size: how many samples to read per iteration.

Returns:
    A _DatasetReader that can be used to create operators that will
    iterate through the dataset.

Definition at line 288 of file dataset.py.

def caffe2.python.dataset.Dataset.trim (   self,
  net,
  multiple_of 
)
Trims the contents of this dataset so that the number of records is
multiple of the given argument.

Definition at line 228 of file dataset.py.

def caffe2.python.dataset.Dataset.writer (   self,
  init_net = None 
)
Create a Writer that can be used to append entries into the dataset.

NOTE: Currently, it is not safe to append to a dataset
      while reading from it.
NOTE: Currently implementation of writer is not thread safe.
      TODO: fixme

Args:
    init_net: net that will be run once in order to create the writer.
      (currently not used)

Definition at line 339 of file dataset.py.


The documentation for this class was generated from the following file: