 |
AIfES 2
2.0.0
|
|
AIfES 2
2.0.0
|
Base layer implementation of the Dense layer. More...
Go to the source code of this file.
Data Structures | |
| struct | ailayer_dense |
| General Dense layer structure. More... | |
Typedefs | |
| typedef struct ailayer_dense | ailayer_dense_t |
Functions | |
| ailayer_t * | ailayer_dense (ailayer_dense_t *layer, ailayer_t *input_layer) |
| Initialize and connect the given Dense layer. More... | |
| void | ailayer_dense_forward (ailayer_t *self) |
| Calculate the forward pass for given Dense layer. More... | |
| void | ailayer_dense_backward (ailayer_t *self) |
| Calculate the backward pass for the given Dense layer. More... | |
| void | ailayer_dense_calc_result_shape (ailayer_t *self) |
| Calculate the shape of the result tensor (ailayer.result) More... | |
| uint32_t | ailayer_dense_sizeof_bwdmem (const ailayer_t *self) |
| uint32_t | ailayer_dense_sizeof_paramem (const ailayer_t *self) |
| Calculate and return the parameter memory size needed for this layer. More... | |
| void | ailayer_dense_set_paramem (ailayer_t *self, void *memory_ptr) |
| Distribute provided memory to the parameter pointers. More... | |
| uint32_t | ailayer_dense_sizeof_trainmem (const ailayer_t *self) |
| Calculate and return the memory size needed by this layer for training. More... | |
| void | ailayer_dense_set_trainmem (ailayer_t *self, void *memory_ptr) |
| Distribute provided memory to the gradients pointers. More... | |
| void | ailayer_dense_print_specs (const ailayer_t *self) |
| Print the layer specification. More... | |
Variables | |
| const aicore_layertype_t * | ailayer_dense_type |
| Dense layer type. More... | |
Base layer implementation of the Dense layer.
This is an "abstract" data-type independent implementation. To use the layer use one of the provided implementations for a specific hardware and data-type (for example from ailayer_dense_default.h) or set the required math functions on your own.
The Dense layer (or fully-connected layer) is the core layer of a FNN and calculates the weighted sums of its inputs
\[ y = x \cdot W + b \]
with the number of inputs \( K \), the number of neurons / outputs \( M \), the input vektor \( x \in \mathbb{R}^{1 \times K} \), the output vektor \( y \in \mathbb{R}^{1 \times M} \), the weights matrix \( W \in \mathbb{R}^{K \times M} \) and the bias \( b \in \mathbb{R}^{1 \times M} \).
To increase the computational speed it is also possible to process a whole data batch at ones:
\[ Y = X \cdot W \oplus b \]
with the number of inputs \( K \), the number of neurons / outputs \( M \), the batch size \( N \), the input matrix \( X \in \mathbb{R}^{N \times K} \), the output vektor \( Y \in \mathbb{R}^{N \times M} \), the weights matrix \( W \in \mathbb{R}^{K \times M} \) and the bias \( b \in \mathbb{R}^{1 \times M} \). In this case the \( \oplus \) means a broadcasted addition of the bias to the rows of the result of \( X \cdot W \):
\[ X \cdot W \oplus b \rightarrow X \cdot W + \left( \begin{array}{c} 1 \\ \vdots \\ 1 \\ \end{array}\right) \cdot b \]
Keep in mind that also the memory size for intermediate results increases with the batch size.
The results of the forward pass of this layer are written to the result tensor of the base ailayer_t struct.
| ailayer_t* ailayer_dense | ( | ailayer_dense_t * | layer, |
| ailayer_t * | input_layer | ||
| ) |
Initialize and connect the given Dense layer.
This function represents the "constructor" of the abstract Dense layer. It initializes the layer structure and connects it to the previous layer.
This function is not intended to call it directly. Instead use one of the data type specific implementations (like for example ailayer_dense_f32_default()).
| *layer | The layer to initialize. |
| *input_layer | The previous layer that provides the inputs to the layer. |
| void ailayer_dense_backward | ( | ailayer_t * | self | ) |
Calculate the backward pass for the given Dense layer.
Implementation of ailayer.backward.
It uses the deltas tensor of the next layer as input and writes the result of the backward pass to the deltas tensor (ailayer.deltas) of the given layer.
Calculation of the gradients:
\[ \partial w \leftarrow \partial w + x_{in}^T \cdot \delta_{out} \]
\[ \partial b \leftarrow \partial b + \left( \begin{array}{c} 1 & \cdots & 1 \\ \end{array}\right) \cdot \delta_{out} \]
Calculation of the errors for the previous layer:
\[ \delta_{in} \leftarrow w^T \cdot \delta_{out} = (\delta_{out}^T \cdot w)^T \]
\( w \): Weights matrix
\( b \): Bias vektor
\( \partial w \): Gradients matrix for the weights
\( \partial b \): Gradients vektor for the bias
\( x_{in} \): Result of the forward pass of the previous layer
\( \delta_{in} \): Result of the backward pass of this layer
\( \delta_{out} \): Result of the backward pass of the next layer
Used math functions:
| *self | Layer to calculate the backward path for. |
| void ailayer_dense_calc_result_shape | ( | ailayer_t * | self | ) |
Calculate the shape of the result tensor (ailayer.result)
Implementation of ailayer.calc_result_shape.
Resulting shape is [count_inputs x count_neurons]
| *self | Layer to calculate the resulting shape for. |
| void ailayer_dense_forward | ( | ailayer_t * | self | ) |
Calculate the forward pass for given Dense layer.
Implementation of ailayer.forward.
It uses the result tensor of the previous layer as input and writes the result of the forward pass to the result tensor (ailayer.result) of the given layer.
Calculation of the forward pass result:
\[ x_{out} \leftarrow x_{in} \cdot w \oplus b \]
\( w \): Weights matrix
\( b \): Bias vector
\( x_{in} \): Result of the forward pass of the previous layer
\( x_{out} \): Result of the forward pass of this layer
Used math functions:
| *self | Layer to calculate the forward path for. |
| void ailayer_dense_print_specs | ( | const ailayer_t * | self | ) |
Print the layer specification.
| *self | The layer to print the specification for |
| void ailayer_dense_set_paramem | ( | ailayer_t * | self, |
| void * | memory_ptr | ||
| ) |
Distribute provided memory to the parameter pointers.
Implementation of ailayer.set_paramem.
Distributes the given buffer to the parameter pointers and sets the tensor parameters for weights and bias holding structures.
The required parameter size can be calculated with ailayer_dense_sizeof_paramem()
| *self | The layer to set the memory fields for. |
| *memory_ptr | The memory that can be used for the parameters |
| void ailayer_dense_set_trainmem | ( | ailayer_t * | self, |
| void * | memory_ptr | ||
| ) |
Distribute provided memory to the gradients pointers.
Implementation of ailayer.set_trainmem.
The required memory size can be calculated with ailayer_dense_sizeof_trainmem().
| *self | The layer to set the memory fields for. |
| *memory_ptr | The memory that can be used for the gradients |
| uint32_t ailayer_dense_sizeof_paramem | ( | const ailayer_t * | self | ) |
Calculate and return the parameter memory size needed for this layer.
Implementation of ailayer.sizeof_paramem.
The parameter size is calculated for the weights and bias tensors, including the data and tensor_params fields.
| *self | The layer to calculate the parameter memory size for |
| uint32_t ailayer_dense_sizeof_trainmem | ( | const ailayer_t * | self | ) |
Calculate and return the memory size needed by this layer for training.
Implementation of ailayer.sizeof_trainmem.
The memory size is calculated for the gradient tensors of weights and bias.
| *self | The layer to calculate the gradient memory size for. |
|
extern |
Dense layer type.
Defines the type of the layer (for example for type checks and debug prints). See aicore_layertype for more information about the layer type.
1.9.1