🧪 Modules

This module provides implementations of various algorithms and modules based on research papers.

• Implementations of algorithms and models from recent research papers in various areas.

• Each function/class includes a reference to the corresponding research paper for clarity.

Source:

• Most of the implementations are sourced from online repositories and adapted for this module.

• Other modules are original implementations by Abdullah Saihan Taki (Author of Torchmate).

RefinedSelfAttentionSAGAN

class torchmate.modules.RefinedSelfAttentionSAGAN(in_channels, kernel_size=3, dilation=1, padding='same', bias=False, scale=8)

Bases: Module

Implement a Refined Self-Attention module for Generative Adversarial Networks (SAGANs).

This module has a more efficient time and space complexity of O(n) compared to the original SAGAN self-attention’s O(n^2) complexity, making it suitable for reducing computational overhead.

Parameters:

• in_channels (int) – Number of input channels.

• kernel_size (int, optional) – Size of the convolutional kernels. Defaults to 3.

• dilation (int, optional) – Dilation factor for the convolutional layers. Defaults to 1.

• padding (str, optional) – Padding type for the convolutional layers. Defaults to "same".

• bias (bool, optional) – Whether to use bias in the convolutional layers. Defaults to False.

• scale (int, optional) – Scaling factor for the number of channels in the query and key projections. Defaults to 8.

Reference:

• Paper: Zheng et al., Less Memory, Faster Speed: Refining Self-Attention Module for Image Reconstruction. arxiv: https://arxiv.org/abs/1905.08008

• Implementation: This is an original implementation by the author of Torchmate.

static hw_flatten(x)

forward(x)

Squeeze and Excitation Modules

A collection of squeeze and excitation (SE) layers that can be integrated into various neural network architectures.

Supports three types of SE blocks: - Channel Squeeze and Excitation (CSE) - Spatial Squeeze and Excitation (SSE) - Channel and Spatial Squeeze and Excitation (CSSE)

Credit: https://github.com/ai-med/squeeze_and_excitation

References: - [Hu et al., Squeeze-and-Excitation Networks](https://arxiv.org/abs/1709.01507) - [Roy et al., Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks, MICCAI 2018] (https://arxiv.org/abs/1803.02579)

ChannelSELayer

class torchmate.modules.ChannelSELayer(num_channels, reduction_ratio=2)

Bases: Module

Implements the Channel Squeeze and Excitation (CSE) block as described in:

• Hu et al., Squeeze-and-Excitation Networks, arXiv:1709.01507

Parameters:

• num_channels (int) – Number of input channels.

• reduction_ratio (int, optional) – Ratio to reduce the number of channels by in the squeeze step. Default is 2.

Returns:

Output tensor with the same dimensions as the input.

Return type:

torch.Tensor

Reference:

• Paper: https://arxiv.org/abs/1709.01507

• Implementation: https://github.com/ai-med/squeeze_and_excitation

forward(input_tensor)

Forward method

Parameters:

input_tensor – X, shape = (batch_size, num_channels, H, W)

Returns:

Output tensor with the same dimensions as the input.

Return type:

torch.Tensor

SpatialSELayer

class torchmate.modules.SpatialSELayer(num_channels)

Bases: Module

Implement the Spatial Squeeze and Excitation (SSE) block as described in:

• Roy et al., Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks, MICCAI 2018

Parameters:

num_channels (int) – Number of input channels.

Returns:

Output tensor with the same dimensions as the input.

Return type:

torch.Tensor

Reference:

• Paper: https://arxiv.org/abs/1803.02579

• Implementation: https://github.com/ai-med/squeeze_and_excitation

forward(input_tensor, weights=None)

Forward method

Parameters:

• input_tensor – X, shape = (batch_size, num_channels, H, W)

• weights – weights for few shot learning

Returns:

Output tensor with the same dimensions as the input.

Return type:

torch.Tensor

ChannelSpatialSELayer

class torchmate.modules.ChannelSpatialSELayer(num_channels, reduction_ratio=2)

Bases: Module

Implement the Concurrent Spatial and Channel Squeeze & Excitation (CSSE) block as described in:

• Roy et al., Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional Networks, MICCAI 2018

Parameters:

• num_channels (int) – Number of input channels.

• reduction_ratio (int, optional) – Ratio to reduce the number of channels

• 2. (by in the squeeze step. Default is) –

Returns:

Output tensor with the same dimensions as the input.

Return type:

torch.Tensor

Reference:

• Paper: https://arxiv.org/abs/1803.02579

• Implementation: https://github.com/ai-med/squeeze_and_excitation

forward(input_tensor)

Forward method

Parameters:

input_tensor – X, shape = (batch_size, num_channels, H, W)

Returns:

Output tensor with the same dimensions as the input.

Return type:

torch.Tensor

SELayer

class torchmate.modules.SELayer(value, names=None, *, module=None, qualname=None, type=None, start=1)

Bases: Enum

Squeeze and Excitation Enum Block

Enum restricting the type of SE Blockes available. So that type checking can be adding when adding these blockes to a neural network.

if self.se_block_type == se.SELayer.CSE.value:
    self.SELayer = se.ChannelSpatialSELayer(params['num_filters'])

elif self.se_block_type == se.SELayer.SSE.value:
    self.SELayer = se.SpatialSELayer(params['num_filters'])

elif self.se_block_type == se.SELayer.CSSE.value:
    self.SELayer = se.ChannelSpatialSELayer(params['num_filters'])

Reference:

Implementation: https://github.com/ai-med/squeeze_and_excitation

NONE = 'NONE'

CSE = 'CSE'

SSE = 'SSE'

CSSE = 'CSSE'