Text Features

Text Features Preprocessing¶

Text features are treated in the same way of sequence features, with a couple differences. Two different tokenizations happen, one that splits at every character and one that splits on whitespace and punctuation are used, and two different keys are added to the HDF5 file, one containing the matrix of characters and one containing the matrix of words. The same thing happens in the JSON file, which contains dictionaries for mapping characters to integers (and the inverse) and words to integers (and their inverse). In the configuration you are able to specify which level of representation to use, if the character level or the word level.

The parameters available for preprocessing are:

char_tokenizer (default characters): defines how to map from the raw string content of the dataset column to a sequence of characters. The default value and only available option is characters and the behavior is to split the string at each character.
char_vocab_file (default null):
char_sequence_length_limit (default 1024): the maximum length of the text in characters. Texts that are longer than this value will be truncated, while sequences that are shorter will be padded.
char_most_common (default 70): the maximum number of most common characters to be considered. if the data contains more than this amount, the most infrequent characters will be treated as unknown.
word_tokenizer (default space_punct): defines how to map from the raw string content of the dataset column to a sequence of elements. For the available options refer to the Tokenizerssection.
pretrained_model_name_or_path (default null):
word_vocab_file (default null):
word_sequence_length_limit (default 256): the maximum length of the text in words. Texts that are longer than this value will be truncated, while texts that are shorter will be padded.
word_most_common (default 20000): the maximum number of most common words to be considered. If the data contains more than this amount, the most infrequent words will be treated as unknown.
padding_symbol (default <PAD>): the string used as a padding symbol. Is is mapped to the integer ID 0 in the vocabulary.
unknown_symbol (default <UNK>): the string used as a unknown symbol. Is is mapped to the integer ID 1 in the vocabulary.
padding (default right): the direction of the padding. right and left are available options.
lowercase (default false): if the string has to be lowercased before being handled by the tokenizer.
missing_value_strategy (default fill_with_const): what strategy to follow when there's a missing value in a binary column. The value should be one of fill_with_const (replaces the missing value with a specific value specified with the fill_value parameter), fill_with_mode (replaces the missing values with the most frequent value in the column), fill_with_mean (replaces the missing values with the mean of the values in the column), backfill (replaces the missing values with the next valid value).
fill_value (default ""): the value to replace the missing values with in case the missing_value_strategy is fill-value.

Example of text preprocessing.

name: text_column_name
type: text
level: word
preprocessing:
    char_tokenizer: characters
    char_vocab_file: null
    char_sequence_length_limit: 1024
    char_most_common: 70
    word_tokenizer: space_punct
    pretrained_model_name_or_path: null
    word_vocab_file: null
    word_sequence_length_limit: 256
    word_most_common: 20000
    padding_symbol: <PAD>
    unknown_symbol: <UNK>
    padding: right
    lowercase: false
    missing_value_strategy: fill_with_const
    fill_value: ""

Text Input Features and Encoders¶

Text input feature parameters are

encoder (default parallel_cnn): encoder to use for the input text feature. The available encoders come from Sequence Features and these text specific encoders: bert, gpt, gpt2, xlnet, xlm, roberta, distilbert, ctrl, camembert, albert, t5, xlmroberta, flaubert, electra, longformer and auto-transformer.
level (default word): word specifies using text words, char use individual characters.
tied_weights (default null): name of the input feature to tie the weights of the encoder with. It needs to be the name of a feature of the same type and with the same encoder parameters.

BERT Encoder¶

The bert encoder loads a pretrained BERT model using the Hugging Face transformers package.

pretrained_model_name_or_path (default bert-base-uncased): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default cls_pooled): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: cls_pool, sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

GPT Encoder¶

The gpt encoder loads a pretrained GPT model using the Hugging Face transformers package.

pretrained_model_name_or_path (default openai-gpt): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

GPT-2 Encoder¶

The gpt2 encoder loads a pretrained GPT-2 model using the Hugging Face transformers package.

pretrained_model_name_or_path (default gpt2): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

XLNet Encoder¶

The xlnet encoder loads a pretrained XLNet model using the Hugging Face transformers package.

pretrained_model_name_or_path (default xlnet-base-cased): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

XLM Encoder¶

The xlm encoder loads a pretrained XLM model using the Hugging Face transformers package.

pretrained_model_name_or_path (default xlm-mlm-en-2048): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

RoBERTa Encoder¶

The roberta encoder loads a pretrained RoBERTa model using the Hugging Face transformers package.

pretrained_model_name_or_path (default roberta-base): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default cls_pooled): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: cls_pool, sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

DistilBERT Encoder¶

The distilbert encoder loads a pretrained DistilBERT model using the Hugging Face transformers package.

pretrained_model_name_or_path (default istilbert-base-uncased): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

CTRL Encoder¶

The ctrl encoder loads a pretrained CTRL model using the Hugging Face transformers package.

pretrained_model_name_or_path (default ctrl): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

CamemBERT Encoder¶

The camembert encoder loads a pretrained CamemBERT model using the Hugging Face transformers package.

pretrained_model_name_or_path (default jplu/tf-camembert-base): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default cls_pooled): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: cls_pool, sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

ALBERT Encoder¶

The albert encoder loads a pretrained ALBERT model using the Hugging Face transformers package.

pretrained_model_name_or_path (default albert-base-v2): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default cls_pooled): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: cls_pool, sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

T5 Encoder¶

The t5 encoder loads a pretrained T5 model using the Hugging Face transformers package.

pretrained_model_name_or_path (default t5-small): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

XLM-RoBERTa Encoder¶

The xlmroberta encoder loads a pretrained XLM-RoBERTa model using the Hugging Face transformers package.

pretrained_model_name_or_path (default jplu/tf-xlm-reoberta-base): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default cls_pooled): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: cls_pool, sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

FlauBERT Encoder¶

The flaubert encoder loads a pretrained FlauBERT model using the Hugging Face transformers package.

pretrained_model_name_or_path (default jplu/tf-flaubert-base-uncased): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

ELECTRA Encoder¶

The electra encoder loads a pretrained ELECTRA model using the Hugging Face transformers package.

pretrained_model_name_or_path (default google/electra-small-discriminator): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

Longformer Encoder¶

The longformer encoder loads a pretrained Longformer model using the Hugging Face transformers package.

pretrained_model_name_or_path (default allenai/longformer-base-4096): it can be either the name of a model or a path where it was downloaded. For details on the variants available refer to the Hugging Face documentation.
reduce_output (default cls_pooled): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: cls_pool, sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

Auto-Transformer Encoder¶

The auto_transformer encoder loads a pretrained model using the Hugging Face transformers package. It's the best option for customly trained models that don't fit into the other pretrained transformers encoders.

pretrained_model_name_or_path: it can be either the name of a model or a path where it was downloaded. For details on the available models to the Hugging Face documentation.
reduce_output (default sum): defines how to reduce the output tensor along the s sequence length dimension if the rank of the tensor is greater than 2. Available values are: sum, mean or avg, max, concat (concatenates along the first dimension), last (returns the last vector of the first dimension) and null (which does not reduce and returns the full tensor).
trainable (default false): if true the weights of the encoder will be trained, otherwise they will be kept frozen.

Example usage¶

Example text input feature encoder usage:

name: text_column_name
type: text
level: word
encoder: bert
tied_weights: null
pretrained_model_name_or_path: bert-base-uncased
reduce_output: cls_pooled
trainable: false

Text Output Features and Decoders¶

The decoders are the same used for the Sequence Features. The only difference is that you can specify an additional level parameter with possible values word or char to force to use the text words or characters as inputs (by default the encoder will use word).

Example text input feature using default values:

name: sequence_column_name
type: text
level: word
decoder: generator
reduce_input: sum
dependencies: []
reduce_dependencies: sum
loss:
    type: softmax_cross_entropy
    confidence_penalty: 0
    robust_lambda: 0
    class_weights: 1
    class_similarities: null
    class_similarities_temperature: 0
    labels_smoothing: 0
    negative_samples: 0
    sampler: null
    distortion: 1
    unique: false
fc_layers: null
num_fc_layers: 0
fc_size: 256
use_bias: true
weights_initializer: glorot_uniform
bias_initializer: zeros
weights_regularizer: null
bias_regularizer: null
activity_regularizer: null
norm: null
norm_params: null
activation: relu
dropout: 0
cell_type: rnn
state_size: 256
embedding_size: 256
beam_width: 1
attention: null
tied_embeddings: null
max_sequence_length: 0

Text Features Measures¶

The measures are the same used for the Sequence Features.