aiopti_sgd.h File Reference

Base optimizer implementation of the Stochastic Gradient Descent (with momentum) optimizer. More...

Go to the source code of this file.

Data Structures
struct	aiopti_sgd
	General Stochastic Gradient Descent (SGD) optimizer struct. More...

Typedefs
typedef struct aiopti_sgd	aiopti_sgd_t
	New data type name for code reduction.

Functions
aiopti_t *	aiopti_sgd (aiopti_sgd_t *opti)
	Initialize the given SGD optimizer. More...
uint32_t	aiopti_sgd_sizeof_optimem_with_momentum (aiopti_t *self, const aitensor_t *params)
	Calculates the required memory for the optimization step when the momentum is not zero. More...
uint32_t	aiopti_sgd_sizeof_optimem_without_momentum (aiopti_t *self, const aitensor_t *params)
	Calculates the required memory for the optimization step when the momentum is zero. More...
void	aiopti_sgd_init_optimem_with_momentum (aiopti_t *self, const aitensor_t *params, const aitensor_t *gradients, void *optimem)
	Initialization of the optimization memory buffer when the momentum is not zero. More...
void	aiopti_sgd_init_optimem_without_momentum (aiopti_t *self, const aitensor_t *params, const aitensor_t *gradients, void *optimem)
	Initialization of the optimization memory buffer when the momentum is zero. More...
void	aiopti_sgd_zero_gradients (aiopti_t *self, aitensor_t *gradients)
	Set the gradients to zero. More...
void	aiopti_sgd_update_params_with_momentum (aiopti_t *self, aitensor_t *params, const aitensor_t *gradients, void *optimem)
	Update the given parameter tensor with respect to the gradients when the momentum is not zero. More...
void	aiopti_sgd_update_params_without_momentum (aiopti_t *self, aitensor_t *params, const aitensor_t *gradients, void *optimem)
	Update the given parameter tensor with respect to the gradients when the momentum is zero. More...
void	aiopti_sgd_print_specs (const aiopti_t *self)
	Print the optimizer specification. More...

Variables
const aicore_optitype_t *	aiopti_sgd_type
	SGD optimizer type. More...

Detailed Description

Base optimizer implementation of the Stochastic Gradient Descent (with momentum) optimizer.

Version

2.2.0

Copyright

Copyright (C) 2020-2023 Fraunhofer Institute for Microelectronic Circuits and Systems. All rights reserved.

AIfES is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

This is an "abstract" data-type independent implementation. To use the optimizer, use one of the provided implementations for a specific hardware and data-type (for example from aiopti_sgd_default.h) or set the required math functions on your own.

The Stochastic Gradient Descent (SGD) optimizer is the most basic optimizer in backpropagation. It uses the pre-calculated gradients to optimize the given parameters. In addition, a momentum term can be configured.
For every parameter \( p \) of the parameters to optimize (trainable parameters) and the related gradient \( g \) it calculates

\[ p_t = p_{t-1} - lr \cdot g_t \]

if the momentum \( \mu = 0 \) and

\[ v_t = \mu \cdot v_{t-1} + g_t \]

\[ p_t = p_{t-1} - lr \cdot v_t \]

if the momentum \( \mu \neq 0 \) in every optimization step.
\( lr \) is the learning rate that defines how big the optimization steps should be, and therefore how fast the training will be. \( v \) is the momentum term or velocity related to the parameter and must be stored in the optimization memory for every parameter when momentum is set.

Function Documentation

aiopti_sgd()

aiopti_t* aiopti_sgd

(

aiopti_sgd_t *

opti

)

Initialize the given SGD optimizer.

This function represents the "constructor" of the abstract SGD optimizer.
This function is not intended to call it directly. Instead use one of the data type specific implementations (like for example aiopti_sgd_f32_default()).

Parameters

*opti

The optimizer to initialize.

Returns

Pointer to the (successfully) initialized general optimizer structure (aiopti_sgd.base)

aiopti_sgd_init_optimem_with_momentum()

void aiopti_sgd_init_optimem_with_momentum	(	aiopti_t *	self,
		const aitensor_t *	params,
		const aitensor_t *	gradients,
		void *	optimem
	)

Initialization of the optimization memory buffer when the momentum is not zero.

Implementation of aiopti.init_optimem.

Initialize the velocity tensor with zeros:

\[ v_{0,i} \leftarrow 0 \]

Used math functions:

• aiopti_sgd.zero_tensor

Parameters

*self	The optimizer
*params	The tensor of trainable parameters
*gradients	The gradients associated to the parameters
*optimem	The optimization memory (containing the velocities) associated to the parameters

aiopti_sgd_init_optimem_without_momentum()

void aiopti_sgd_init_optimem_without_momentum	(	aiopti_t *	self,
		const aitensor_t *	params,
		const aitensor_t *	gradients,
		void *	optimem
	)

Initialization of the optimization memory buffer when the momentum is zero.

Implementation of aiopti.init_optimem.

Does nothing because no optimization memory is needed in this case.

Parameters

*self	The optimizer
*params	The tensor of trainable parameters
*gradients	The gradients associated to the parameters
*optimem	The optimization memory associated to the parameters

aiopti_sgd_print_specs()

void aiopti_sgd_print_specs

(

const aiopti_t *

self

)

Print the optimizer specification.

Parameters

*self

The optimizer to print the specification for

aiopti_sgd_sizeof_optimem_with_momentum()

uint32_t aiopti_sgd_sizeof_optimem_with_momentum	(	aiopti_t *	self,
		const aitensor_t *	params
	)

Calculates the required memory for the optimization step when the momentum is not zero.

Implementation of aiopti.sizeof_optimem.

Calculates the size of the memory space that must be reserved. The memory is used for the velocity tensor (momentum term) and is calculated by:

sizeof(aitensor) + sizeof(params.data) 

Parameters

*self	The optimizer
*params	The tensor of trainable parameters to calculate the memory for

aiopti_sgd_sizeof_optimem_without_momentum()

uint32_t aiopti_sgd_sizeof_optimem_without_momentum	(	aiopti_t *	self,
		const aitensor_t *	params
	)

Calculates the required memory for the optimization step when the momentum is zero.

Implementation of aiopti.sizeof_optimem.

Calculates the size of the memory space that must be reserved. The required memory is zero because no velocity term is needed.

Parameters

*self	The optimizer
*params	The tensor of trainable parameters to calculae the memory for

aiopti_sgd_update_params_with_momentum()

void aiopti_sgd_update_params_with_momentum	(	aiopti_t *	self,
		aitensor_t *	params,
		const aitensor_t *	gradients,
		void *	optimem
	)

Update the given parameter tensor with respect to the gradients when the momentum is not zero.

Implementation of aiopti.update_params.

Calculate and update the values of the trainable parameters (perform one update step):

\[ v_t \leftarrow \mu \cdot v_{t-1} + g_t \]

\[ p_t \leftarrow p_{t-1} - lr \cdot v_t \]

\( v \): Velocity tensor (momentum term), stored in the optimem
\( p \): Tensor of trainable parameters to update (params)
\( g \): Gradients
\( lr \): Learning rate / Optimization step size
\( \mu \): Momentum

Used math functions:

• aiopti_sgd.scalar_mul
• aiopti_sgd.tensor_add
• aiopti_sgd.tensor_sub

Parameters

*self	The optimizer
*params	The tensor of trainable parameters \( p \) to update
*gradients	The gradients \( g \) associated to the parameters
*optimem	The buffer to store the velocity \( v \)

aiopti_sgd_update_params_without_momentum()

void aiopti_sgd_update_params_without_momentum	(	aiopti_t *	self,
		aitensor_t *	params,
		const aitensor_t *	gradients,
		void *	optimem
	)

Update the given parameter tensor with respect to the gradients when the momentum is zero.

Implementation of aiopti.update_params.

Calculate and update the values of the trainable parameters (perform one update step):

\[ p_t \leftarrow p_{t-1} - lr \cdot g_t \]

Used math functions:

• aiopti_sgd.scalar_mul

• aiopti_sgd.tensor_sub

  \( p \): Tensor of trainable parameters to update (params)
  \( g \): Gradients
  \( lr \): Learning rate / Optimization step size
  
  

  Parameters

  *self	The optimizer
  *params	The tensor of trainable parameters \( p \) to update
  *gradients	The gradients \( g \) associated to the parameters
  *optimem	Not required because no velocity is stored

aiopti_sgd_zero_gradients()

void aiopti_sgd_zero_gradients	(	aiopti_t *	self,
		aitensor_t *	gradients
	)

Set the gradients to zero.

Implementation of aiopti.zero_gradients.

\[ g_{i} \leftarrow 0 \]

Used math functions:

• aiopti_sgd.zero_tensor

Parameters

*self	The optimizer
*gradients	The gradients to set to zero

Variable Documentation

aiopti_sgd_type

const aicore_optitype_t* aiopti_sgd_type

extern

SGD optimizer type.

Defines the type of the optimizer (for example for type checks and debug prints). See aicore_optitype for more information about the optimizer type.