torch::data::datasets::ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler > Class Template Reference

A stateful dataset that support hierarchical sampling and prefetching of entre chunks. More...

#include <chunk.h>

Inheritance diagram for torch::data::datasets::ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >:

Public Types
using	BatchType = torch::optional< typename ChunkReader::BatchType >
using	UnwrappedBatchType = typename ChunkReader::BatchType
using	BatchRequestType = size_t
using	ChunkSamplerType = ChunkSampler
using	ExampleSamplerType = ExampleSampler
Public Types inherited from torch::data::datasets::BatchDataset< ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >, optional< ChunkReader::BatchType >, size_t >
using	SelfType = ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >
using	BatchType = optional< ChunkReader::BatchType >
using	BatchRequestType = size_t

Public Member Functions
	ChunkDataset (ChunkReader chunk_reader, ChunkSampler chunk_sampler, ExampleSampler example_sampler, ChunkDatasetOptions options)
BatchType	get_batch (size_t batch_size) override
	Default get_batch method of BatchDataset. More...
void	reset () override
	This will clear any internal state and starts the internal prefetching mechanism for the chunk dataset. More...
optional< size_t >	size () const override
	size is not used for chunk dataset.
ChunkSamplerType &	chunk_sampler ()
Public Member Functions inherited from torch::data::datasets::BatchDataset< ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >, optional< ChunkReader::BatchType >, size_t >
MapDataset< ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >, TransformType >	map (TransformType transform)&
	Creates a MapDataset that applies the given transform to this dataset.
MapDataset< ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >, TransformType >	map (TransformType transform)&&
	Creates a MapDataset that applies the given transform to this dataset.

Additional Inherited Members
Static Public Attributes inherited from torch::data::datasets::BatchDataset< ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >, optional< ChunkReader::BatchType >, size_t >
static constexpr bool	is_stateful

Detailed Description

template<typename ChunkReader, typename ChunkSampler = samplers::RandomSampler, typename ExampleSampler = samplers::RandomSampler>
class torch::data::datasets::ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >

A stateful dataset that support hierarchical sampling and prefetching of entre chunks.

Unlike regular dataset, chunk dataset require two samplers to operate and keeps an internal state. ChunkSampler selects, which chunk to load next, while the ExampleSampler determins the order of Examples that are returned in each get_batch call. The hierarchical sampling approach used here is inspired by this paper http://martin.zinkevich.org/publications/nips2010.pdf

Definition at line 283 of file chunk.h.

Member Function Documentation

template<typename ChunkReader , typename ChunkSampler = samplers::RandomSampler, typename ExampleSampler = samplers::RandomSampler>

BatchType torch::data::datasets::ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >::get_batch ( size_t  batch_size )

inlineoverridevirtual

Default get_batch method of BatchDataset.

This method returns Example batches created from the preloaded chunks. The implemenation is dataset agnostic and does not need overriding in different chunk datasets.

Implements torch::data::datasets::BatchDataset< ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >, optional< ChunkReader::BatchType >, size_t >.

Definition at line 319 of file chunk.h.

template<typename ChunkReader , typename ChunkSampler = samplers::RandomSampler, typename ExampleSampler = samplers::RandomSampler>

void torch::data::datasets::ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >::reset ( )

inlineoverridevirtual

This will clear any internal state and starts the internal prefetching mechanism for the chunk dataset.

Implements torch::data::datasets::StatefulDataset< ChunkDataset< ChunkReader, ChunkSampler, ExampleSampler >, ChunkReader::BatchType, size_t >.

Definition at line 335 of file chunk.h.

---

The documentation for this class was generated from the following file:

• torch/csrc/api/include/torch/data/datasets/chunk.h