This is positional encoding for timseries data inspired by A Transformer-based Framework for Multivariate Time Series Representation Learning.
Assuming that the input is (batch_size, sequence_length, feature_dim) and each feature is a float, the way we add positional encoding is:
- Dense layer to project features into d dim.
- Add learnable positional embedding.
class LearnablePositionalEncoding(tf.keras.layers.Layer):
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.pos_encoding = None
def build(self, input_shape):
sequence_length = input_shape[1]
feature_dim = input_shape[2]
# learnable weights
self.pos_encoding = self.add_weight(
shape=(1, sequence_length, feature_dim),
initializer="random_normal",
trainable=True,
)
def call(self, inputs):
return inputs + self.pos_encoding
This is applied like:
# Embed into embedding_dim.
X = Dense(
self.config.model["embedding_dim"],
activation="linear",
use_bias=True)(X)
# Positional encoding
X = LearnablePositionalEncoding()(X)
See PositionalEncoding of keras_nlp here for generic implementation.