caffe2.python.modeling.initializers.PseudoFP16Initializer Class Reference

Inheritance diagram for caffe2.python.modeling.initializers.PseudoFP16Initializer:

Public Member Functions
def	update (self, operator_name, kwargs)
def	create_param (self, param_name, init_net, shape)
Public Member Functions inherited from caffe2.python.modeling.initializers.Initializer
def	__init__ (self, operator_name=None, kwargs)
def	update (self, operator_name, kwargs)
def	create_param (self, param_name, init_net, shape)

Public Attributes
	operator_name
	operator_kwargs
Public Attributes inherited from caffe2.python.modeling.initializers.Initializer
	operator_name
	operator_kwargs

Detailed Description

Used in cases when the parameter should be used at half (16-bit) precision
for compute purposes (i.e. on the forward and backward pass) but
needs to be stored and optimized at single (32-bit) precision so tiny
gradients with small learning rates don't underflow FP16 precision.
A 32-bit copy of the 16-bit blob is stored in the ParameterInfo.
This is helpful for mixed-precision training, see
https://arxiv.org/abs/1710.03740 for details.

Definition at line 62 of file initializers.py.

---

The documentation for this class was generated from the following file:

• caffe2/python/modeling/initializers.py