Positional encoding for timeseries data

This is positional encoding for timseries data inspired by A Transformer-based Framework for Multivariate Time Series Representation Learning.

Assuming that the input is (batch_size, sequence_length, feature_dim) and each feature is a float, the way we add positional encoding is:

Dense layer to project features into d dim.
Add learnable positional embedding.

class LearnablePositionalEncoding(tf.keras.layers.Layer):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self.pos_encoding = None

    def build(self, input_shape):
        sequence_length = input_shape[1]
        feature_dim = input_shape[2]
        # learnable weights
        self.pos_encoding = self.add_weight(
            shape=(1, sequence_length, feature_dim),
            initializer="random_normal",
            trainable=True,
        )

    def call(self, inputs):
        return inputs + self.pos_encoding

This is applied like:

# Embed into embedding_dim.
X = Dense(
    self.config.model["embedding_dim"],
    activation="linear", 
    use_bias=True)(X)
# Positional encoding
X = LearnablePositionalEncoding()(X)

See PositionalEncoding of keras_nlp here for generic implementation.

Passion is like genius; a miracle.

Comments

Leave a Reply