ailoss_crossentropy.h File Reference

Base loss implementation of the Cross-Entropy loss. More...

Go to the source code of this file.

Data Structures
struct	ailoss_crossentropy
	General Cross-Entropy loss struct. More...

Typedefs
typedef struct ailoss_crossentropy	ailoss_crossentropy_t
	New data type name for code reduction.

Functions
ailoss_t *	ailoss_crossentropy (ailoss_crossentropy_t *loss, ailayer_t *input_layer)
	Initialize and connect the given Cross-Entropy loss. More...
void	ailoss_crossentropy_calc_delta (ailoss_t *self, const aitensor_t *target_data)
	Calculate the combined derivative of the given Cross-Entropy loss and the output layer for error backpropagation. More...
void	ailoss_crossentropy_calc_loss (ailoss_t *self, const aitensor_t *target_data, void *result)
	Calculate the Cross-Entropy loss on the given target data. More...
void	ailoss_crossentropy_dummy_backward (ailayer_t *self)
	Dummy backward-function for the output layer of the model. More...
void	ailoss_crossentropy_print_specs (const ailoss_t *self)
	Print the loss specification. More...

Variables
const aicore_losstype_t *	ailoss_crossentropy_type
	Cross-Entropy loss type. More...

Detailed Description

Base loss implementation of the Cross-Entropy loss.

Version

2.2.0

Copyright

Copyright (C) 2020-2023 Fraunhofer Institute for Microelectronic Circuits and Systems. All rights reserved.

AIfES is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.

This is an "abstract" data-type independent implementation. To use the loss, use one of the provided implementations for a specific hardware and data-type (for example from ailoss_crossentropy_default.h) or set the required math functions on your own.

The Cross-Entropy loss ist best suitable for classification tasks and works only with Sigmoid or Softmax output layers.

For binary classification (binary targets 0 or 1) with Sigmoid output layer, the loss / cost is calculated as

\[ L(y, \hat{y}) = - \sum_{i=0}^{N} (y_i \log(\hat{y}_i) + (1-y) \log(1-\hat{y}_i)) \]

with the predicted values \( \hat{y}_i \) and the target values \( y_i \). \( N \) is the number of elements of the \( y \) tensor.

For categorigal classification (one-hot encoded targets) with Softmax output layer, the loss / cost is calculated as

\[ L(y, \hat{y}) = - \sum_{i=0}^{N} y_{i} \log(\hat{y}_{i}) \]

with the predicted values \( \hat{y}_i \) and the target values \( y_i \). \( N \) is the number of elements of the \( y \) tensor.

To get the "mean" normalization, you have to modify the learning rate to \( lr = \frac {1}{o \cdot n} \cdot lr \) with the number of outputs \( o \) and the batch size \( n \).

The loss can be calculated with ailoss_crossentropy_calc_loss(). For training the deltas /errors on the target data are calculated with ailoss_crossentropy_calc_delta() and written to the deltas tensor of the connection layer.

Function Documentation

ailoss_crossentropy()

ailoss_t* ailoss_crossentropy	(	ailoss_crossentropy_t *	loss,
		ailayer_t *	input_layer
	)

Initialize and connect the given Cross-Entropy loss.

This function represents the "constructor" of the abstract Cross-Entropy loss. It initializes the loss structure and connects it to the output layer of the AIfES model.
This function is not intended to call it directly. Instead use one of the data type specific implementations (like for example ailoss_crossentropy_f32_default()).

Parameters

*loss	The loss to initialize.
*input_layer	The output layer of the model that provides the inputs to the loss.

Returns

Pointer to the (successfully) initialized loss structure.

ailoss_crossentropy_calc_delta()

void ailoss_crossentropy_calc_delta	(	ailoss_t *	self,
		const aitensor_t *	target_data
	)

Calculate the combined derivative of the given Cross-Entropy loss and the output layer for error backpropagation.

Implementation of ailoss.calc_delta.

It uses the result tensor of the output layer and the target data as input and writes the result to the deltas tensor (ailayer.deltas) of the output layer of the model.

By combining the Cross-Entropy loss with a Sigmoid or Softmax output layer, the combined deltas can be calculated very efficiently. The backward function of the Sigmoid / Softmax layer is not used anymore.

Calculation of the deltas:

\[ \delta_{in} \leftarrow p - y \]

\( \delta_{in} \): Result of the delta calculation of this loss (written to ailayer.deltas of the output layer of the model)
\( p \): Result of the forward pass of the output layer of the model (predicted values)
\( y \): Target data / True values / Labels

Used math functions:

• ailoss_crossentropy.tensor_sub

Parameters

*self	Loss to calculate the deltas for
*target_data	Target data / True values / Labels

ailoss_crossentropy_calc_loss()

void ailoss_crossentropy_calc_loss	(	ailoss_t *	self,
		const aitensor_t *	target_data,
		void *	result
	)

Calculate the Cross-Entropy loss on the given target data.

Implementation of ailoss.calc_loss.

It uses the result tensor of the output layer and the target data as input and writes the result to the given result scalar.

Calculation of the loss with Sigmoid output layer:

\[ result \leftarrow - \sum_i (y_i \log(p_i) + (1 - y_i) \log(1 - p_i)) \]

Calculation of the loss with Softmax output layer:

\[ result \leftarrow - \sum_i y_i \log(p_i) \]

\( result \): Result of the loss calculation
\( p \): Result of the forward pass of the output layer of the model (predicted values)
\( y \): Target data / True values / Labels

Used math functions:

• ailoss_crossentropy.tensor_sub
• ailoss_crossentropy.crossentropy

Parameters

*self	Loss to calculate the deltas for
*target_data	Target data / True values / Labels
*result	Result scalar (the data type is specified by the data type specific implementations)

ailoss_crossentropy_dummy_backward()

void ailoss_crossentropy_dummy_backward

(

ailayer_t *

self

)

Dummy backward-function for the output layer of the model.

Implementation of ailayer.backward.

The ailoss_crossentropy_calc_delta() function calculates the combined delta for a Sigmoid or Softmax layer with Cross-Entropy loss. Therefor the backward function of the Sigmoid / Softmax layer is not needed anymore and gets exchanged by this dummy function.

The dummy function does nothing.

Parameters

*self

The output layer of the model

ailoss_crossentropy_print_specs()

void ailoss_crossentropy_print_specs

(

const ailoss_t *

self

)

Print the loss specification.

Parameters

*self

The loss to print the specification for

Variable Documentation

ailoss_crossentropy_type

const aicore_losstype_t* ailoss_crossentropy_type

extern

Cross-Entropy loss type.

Defines the type of the loss (for example for type checks and debug prints). See aicore_losstype for more information about the loss type.