Caffe2 - C++ API
A deep learning, cross platform ML framework
Public Types | Public Member Functions
torch::data::StatefulDataLoader< Dataset > Class Template Reference

A dataloader for stateful datasets. More...

#include <stateful.h>

Inheritance diagram for torch::data::StatefulDataLoader< Dataset >:
torch::data::DataLoaderBase< Dataset, Dataset::BatchType::value_type, Dataset::BatchRequestType >

Public Types

using super = DataLoaderBase< Dataset, typename Dataset::BatchType::value_type, typename Dataset::BatchRequestType >
 
using BatchRequestType = BatchRequest
 
- Public Types inherited from torch::data::DataLoaderBase< Dataset, Dataset::BatchType::value_type, Dataset::BatchRequestType >
using BatchType = Dataset::BatchType::value_type
 
using BatchRequestType = Dataset::BatchRequestType
 

Public Member Functions

 StatefulDataLoader (Dataset dataset, DataLoaderOptions options)
 Constructs the StatefulDataLoader from a dataset and some options.
 
- Public Member Functions inherited from torch::data::DataLoaderBase< Dataset, Dataset::BatchType::value_type, Dataset::BatchRequestType >
 DataLoaderBase (DataLoaderOptions options, std::unique_ptr< Dataset > main_thread_dataset=nullptr)
 Constructs a new DataLoader from a dataset to sample from, options to configure the DataLoader with, and a sampler that specifies the sampling strategy. More...
 
Iterator< Dataset::BatchType::value_type > begin ()
 Returns an iterator into the DataLoader. More...
 
Iterator< Dataset::BatchType::value_type > end ()
 Returns a special "sentinel" iterator that compares equal with a non-sentinel iterator once the DataLoader is exhausted. More...
 
void join ()
 Joins the DataLoader's worker threads and drains internal queues. More...
 
const FullDataLoaderOptionsoptions () const noexcept
 Returns the options with which the DataLoader was configured.
 

Additional Inherited Members

- Protected Member Functions inherited from torch::data::DataLoaderBase< Dataset, Dataset::BatchType::value_type, Dataset::BatchRequestType >
void prefetch (size_t requested_jobs)
 Schedules requested_jobs many new batches to be fetched. More...
 
void prefetch ()
 Schedules the maximum number of jobs (based on the max_jobs option).
 
optional< BatchType > next ()
 Returns the next batch of data, or an empty optional if the DataLoader is exhausted. More...
 
void worker_thread (Dataset &dataset)
 The function that worker threads run.
 
void push_job (T value)
 Convenience method that calls shuttle_.push_job() with the next sequence number. More...
 
optional< Result > pop_result ()
 Convenience method that gets the next result from the sequencer.
 
std::unique_ptr< detail::sequencers::Sequencer< Result > > new_sequencer ()
 Convenience method that creates a new sequencer based on the enforce_ordering option. More...
 
- Protected Attributes inherited from torch::data::DataLoaderBase< Dataset, Dataset::BatchType::value_type, Dataset::BatchRequestType >
const FullDataLoaderOptions options_
 The options the DataLoader was configured with.
 
std::unique_ptr< Dataset > main_thread_dataset_
 The dataset for the main thread, only has a value if the number of worker threads was configured as zero, meaning the main thread has to do all the work (synchronously). More...
 
size_t sequence_number_
 The sequence number for the next batch to be retrieved from the dataset. More...
 
std::vector< std::thread > workers_
 The worker threads, running the worker_thread() method.
 
detail::DataShuttle< Job, Result > shuttle_
 The DataShuttle which takes care of the life cycle of a job.
 
std::unique_ptr< detail::sequencers::Sequencer< Result > > sequencer_
 The Sequencer, which handles optional ordering of batches.
 
bool joined_
 True if the DataLoader has joined its worker threads.
 

Detailed Description

template<typename Dataset>
class torch::data::StatefulDataLoader< Dataset >

A dataloader for stateful datasets.

A dataloader for stateful datatasets differs from one for stateless datasets one in that the dataset is shared among worker threads, and that this dataset is itself responsible for producing batches rather than depending on a sampler. The statefulness here actually refers to the dataset. The StatefulDataLoader simply alters the data loading algorithm to accomodate the stateful, shared nature of the dataset. Note that the dataset must be thread safe if more than one worker thread is used.

A stateful dataloader is created by calling make_data_loader with a stateful dataset.

Definition at line 25 of file stateful.h.


The documentation for this class was generated from the following file: