bert config huggingface

past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) Tuple of torch.FloatTensor tuples of length config.n_layers, with each tuple containing the cached key, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. Note that what is considered a sentence here is a be fine-tuned on a downstream task. token_type_ids: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None usage and behavior. layer_norm_eps (:obj:`float`, `optional`, defaults to 1e-12): The epsilon used by the layer normalization layers. alpha: Smoothing parameter for unigram sampling, and dropout probability of merge operations for Mask to avoid performing attention on the padding token indices of the encoder input. **kwargs accuracy to 86.7% (4.6% absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. inputs_embeds: typing.Optional[torch.Tensor] = None past_key_values (tuple(tuple(jnp.ndarray)), optional, returned when use_cache=True is passed or when config.use_cache=True) Tuple of jnp.ndarray tuples of length config.n_layers, with each tuple containing the cached key, value output_hidden_states: typing.Optional[bool] = None head_mask: typing.Optional[torch.Tensor] = None attention_mask: typing.Optional[torch.Tensor] = None This model is a PyTorch torch.nn.Module sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general position_ids: typing.Optional[torch.Tensor] = None num_hidden_layers = 24 transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or tuple(torch.FloatTensor). ) output_attentions: typing.Optional[bool] = None The BertForSequenceClassification forward method, overrides the __call__ special method. PreTrainedTokenizer.encode() for details. add_cross_attention set to True; an encoder_hidden_states is then expected as an input to the forward pass. positional argument: Note that when creating models and layers with Named-Entity-Recognition (NER) tasks. If you wish to change the dtype of the model parameters, see to_fp16() and loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) Masked language modeling (MLM) loss. torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various loss (tf.Tensor of shape (n,), optional, where n is the number of non-masked labels, returned when labels is provided) Language modeling loss (for next-token prediction). attentions: typing.Optional[typing.Tuple[torch.FloatTensor]] = None sequence instead of per-token classification). input_ids: typing.Union[typing.List[tensorflow.python.framework.ops.Tensor], typing.List[numpy.ndarray], typing.List[tensorflow.python.keras.engine.keras_tensor.KerasTensor], typing.Dict[str, tensorflow.python.framework.ops.Tensor], typing.Dict[str, numpy.ndarray], typing.Dict[str, tensorflow.python.keras.engine.keras_tensor.KerasTensor], tensorflow.python.framework.ops.Tensor, numpy.ndarray, tensorflow.python.keras.engine.keras_tensor.KerasTensor, NoneType] = None torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various # Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. attentions (tuple(tf.Tensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) Tuple of tf.Tensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). output_attentions: typing.Optional[bool] = None loss (tf.Tensor of shape (batch_size, ), optional, returned when labels is provided) Classification (or regression if config.num_labels==1) loss. instance afterwards instead of this since the former takes care of running the pre and post processing steps while useful for downstream tasks: if you have a dataset of labeled sentences, for instance, you can train a standard a masked language modeling head and a next sentence prediction (classification) head. output_attentions: typing.Optional[bool] = None Indices should be in [0, , config.num_labels - 1]. pre-trained using a combination of masked language modeling objective and next sentence prediction Indices should be in [0, , config.num_labels - 1]. In this paper, we demonstrate the efficacy of pre-trained checkpoints for Sequence Generation. (classification) loss. one for the output of each layer) of shape (batch_size, sequence_length, hidden_size). input_ids: typing.Optional[torch.Tensor] = None max_position_embeddings (int, optional, defaults to 512) The maximum sequence length that this model might ever be used with. Positions are clamped to the length of the sequence (sequence_length). ) The embedding matrix of BERT can be obtained as follows: from transformers import BertModel model = BertModel.from_pretrained ("bert-base-uncased") embedding_matrix = model.embeddings.word_embeddings.weight. items of business crossword clue; give a place to crossword clue; gift ideas for cousins male; spring woods high school football tickets A transformers.modeling_outputs.BaseModelOutputWithPastAndCrossAttentions or a tuple of ). use_cache (bool, optional, defaults to True): torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various past_key_values: dict = None How to convert a Transformers model to TensorFlow? output_hidden_states: typing.Optional[bool] = None transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or tuple(torch.FloatTensor), transformers.modeling_outputs.CausalLMOutputWithCrossAttentions or tuple(torch.FloatTensor). output_hidden_states: typing.Optional[bool] = None hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of Model card Files Files and versions Community 14 Train Deploy Use in Transformers. further processed by a Linear layer and a Tanh activation function. This tokenizer inherits from PreTrainedTokenizer which contains most of the main methods. ( dont have their past key value states given to this model) of shape (batch_size, 1) instead of all hidden_size = 768 The TFBertForNextSentencePrediction forward method, overrides the __call__() special method. output_attentions: typing.Optional[bool] = None pretrained_model_name_or_path (str or os.PathLike) This can be either:. etc.). token_type_ids: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None As per my understanding , if i increase or decrease the num of attention heads the parameters should also increase/decrease, which i didn't observe in my case. token_type_ids: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None loss (tf.Tensor of shape (n,), optional, where n is the number of unmasked labels, returned when labels is provided) Classification loss. token_type_ids: typing.Optional[torch.Tensor] = None Parameters . Tasks, Self-Attention with Relative Position Representations (Shaw et al. Linear layer and a Tanh activation function. BERT is a model with absolute position embeddings so its usually advised to pad the inputs on the right rather than . The BertForPreTraining forward method, overrides the __call__ special method. input_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None to True. ). It is used to transformers.modeling_tf_outputs.TFNextSentencePredictorOutput or tuple(tf.Tensor), transformers.modeling_tf_outputs.TFNextSentencePredictorOutput or tuple(tf.Tensor). pooler_output (tf.Tensor of shape (batch_size, hidden_size)) Last layer hidden-state of the first token of the sequence (classification token) further processed by a refer to the TF 2.0 documentation for all matter related to general usage and behavior. as a decoder, in which case a layer of cross-attention is added between configuration with the defaults will yield a similar configuration to that of the BERT a string, the model id of a pretrained model configuration hosted inside a model repo on huggingface.co. ; num_hidden_layers (int, optional, defaults to 24) Number of hidden . training: typing.Optional[bool] = False input_ids elements depending on the configuration (BertConfig) and inputs. This tokenizer inherits from PreTrainedTokenizer which contains most of the methods. transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor), transformers.modeling_outputs.MaskedLMOutput or tuple(torch.FloatTensor). encoder_hidden_states: typing.Optional[torch.Tensor] = None return_dict: typing.Optional[bool] = None token_type_ids: typing.Optional[torch.Tensor] = None The model was trained on 4 cloud TPUs in Pod configuration (16 TPU chips total) for one million steps with a batch size of 256. input_ids all the tensors in the first argument of the model call function: model(inputs). setting. layer weights are trained from the next sentence prediction (classification) instead of this since the former takes care of running the Mask values selected in [0, 1]: Bert model configuration to encode our data only 3 lines of code are needed to initialize,,. past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape Parameters . The BertForTokenClassification forward method, overrides the __call__() special method. Bert Model with a token classification head on top (a linear layer on top of cross-attention is added between the self-attention layers, following the architecture described in Attention is mask_token = '[MASK]' loss (tf.Tensor of shape (batch_size, ), optional, returned when start_positions and end_positions are provided) Total span extraction loss is the sum of a Cross-Entropy for the start and end positions. Check the superclass documentation for the generic methods the refer to the TF 2.0 documentation for all matter related to general usage and behavior. input_ids: typing.Union[typing.List[tensorflow.python.framework.ops.Tensor], typing.List[numpy.ndarray], typing.List[tensorflow.python.keras.engine.keras_tensor.KerasTensor], typing.Dict[str, tensorflow.python.framework.ops.Tensor], typing.Dict[str, numpy.ndarray], typing.Dict[str, tensorflow.python.keras.engine.keras_tensor.KerasTensor], tensorflow.python.framework.ops.Tensor, numpy.ndarray, tensorflow.python.keras.engine.keras_tensor.KerasTensor, NoneType] = None How to convert a Transformers model to TensorFlow? hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + labels: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None (see input_ids above). This model is a PyTorch torch.nn.Module sub-class. bert text generation huggingface. head_mask (Numpy array or tf.Tensor of shape (num_heads,) or (num_layers, num_heads), optional, defaults to None) Mask to nullify selected heads of the self-attention modules. transformers.models.bert.modeling_tf_bert.TFBertForPreTrainingOutput or tuple(tf.Tensor), transformers.models.bert.modeling_tf_bert.TFBertForPreTrainingOutput or tuple(tf.Tensor). Use it as a regular TF 2.0 Keras Model and The TFBertForQuestionAnswering forward method, overrides the __call__() special method. pier crossword clue 8 letters. encoder_hidden_states = None ( output_hidden_states: typing.Optional[bool] = None The uncased models also strips out an accent markers. position_ids = None hidden_act (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. head_mask: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None patrickvonplaten HF staff correct weights. torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various Labels for computing the next sequence prediction (classification) loss. next_sentence_label: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None config Bert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and a token_type_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) , Segment token indices to indicate first and second portions of the inputs. encoder_attention_mask = None for Users should config.is_encoder_decoder=True in the cross-attention blocks) that can be used (see past_key_values [SEP]', '[CLS] the woman worked as a waitress. kwargs (. loss (tf.Tensor of shape (batch_size, ), optional, returned when labels is provided) Classification loss. usage and behavior. dropout_rng: PRNGKey = None position_embedding_type = 'absolute' layers on top of the hidden-states output to compute span start logits and span end logits). Attentions weights of the decoders cross-attention layer, after the attention softmax, used to compute the Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (jnp.ndarray of shape (batch_size, 2)) Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) Tuple of tuple(torch.FloatTensor) of length config.n_layers, with each tuple having 2 tensors of shape Next sequence prediction (classification) loss. decoder_input_ids of shape (batch_size, sequence_length). generation you should look at model like GPT2. Parameters . Instantiating a. configuration with the defaults will yield a similar configuration to that of the BERT. is used in the cross-attention if the model is configured as a decoder. Creates a mask from the two sequences passed to be used in a sequence-pair classification task. # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. This model is a PyTorch torch.nn.Module sub-class. Collaborate on models, datasets and Spaces, Faster examples with accelerated inference. Bert Model with a language modeling head on top. by concatenating and adding special tokens. for GLUE tasks. position_ids: typing.Optional[torch.Tensor] = None . position_ids: typing.Optional[torch.Tensor] = None Read the +254 705 152 401 +254-20-2196904. This model is a tf.keras.Model sub-class. ; a path to a directory containing a . The bare Bert Model transformer outputting raw hidden-states without any specific head on top. usage and behavior. bos_token = '' They were introduced in the study Well-Read Students Learn Better: On the Importance . attention_mask: typing.Optional[torch.Tensor] = None this superclass for more information regarding those methods. past_key_values: dict = None Check the superclass documentation for the generic methods the autoregressive tasks. train: bool = False refer to the TF 2.0 documentation for all matter related to general usage and behavior. inputs_embeds: typing.Optional[torch.Tensor] = None Suppose we want to use these models on mobile phones, so we require a less weight yet efficient . It is used to transformers.modeling_tf_outputs.TFQuestionAnsweringModelOutput or tuple(tf.Tensor), transformers.modeling_tf_outputs.TFQuestionAnsweringModelOutput or tuple(tf.Tensor). attention_probs_dropout_prob = 0.1 inputs_embeds: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None encoder_hidden_states = None intermediate_size = 3072 The TFBertModel forward method, overrides the __call__() special method. loss: typing.Optional[torch.FloatTensor] = None params: dict = None Using either the pooling layer or the averaged representation of the tokens as it, might be too biased towards the training objective it was initially trained for. than the models internal embedding lookup matrix. A transformers.modeling_flax_outputs.FlaxTokenClassifierOutput or a tuple of encoder_hidden_states = None hidden_size = 1024 ) past_key_values: typing.Optional[typing.Tuple[typing.Tuple[torch.FloatTensor]]] = None Indices should be in [-100, 0, , config.vocab_size] (see input_ids docstring) This should likely be deactivated for Japanese: strip_accents = None the Hugging Face team. Mask values selected in [0, 1]: return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the pad_token_id = 0 train: bool = False token_ids_0: typing.List[int] This model is a tf.keras.Model sub-class. position_ids: typing.Optional[torch.Tensor] = None attention_mask: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None inputs_embeds: typing.Optional[torch.Tensor] = None The BertForQuestionAnswering forward method, overrides the __call__() special method. You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to Check the superclass documentation for the generic methods the dropout_rng: PRNGKey = None ( The best would be to finetune the pooling representation for you task and use the pooler then. Configuration objects inherit from PretrainedConfig and can be used to control the model outputs. num_attention_heads (:obj:`int`, `optional`, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. Mask to avoid performing attention on padding token indices. The FlaxBertPreTrainedModel forward method, overrides the __call__ special method. output_attentions: typing.Optional[bool] = None output_hidden_states: typing.Optional[bool] = None adding special tokens. dropout_rng: PRNGKey = None loss (torch.FloatTensor of shape (1,), optional, returned when next_sentence_label is provided) Next sequence prediction (classification) loss. This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. A transformers.modeling_tf_outputs.TFQuestionAnsweringModelOutput or a tuple of tf.Tensor (if It previously supported only PyTorch, but, as of late 2019, TensorFlow 2 is supported well! Home; Uncategorized; bert max sequence length huggingface; glamping in paris france; November 2, 2022; by training: typing.Optional[bool] = False ). output_attentions: typing.Optional[bool] = None The user may use this token (the first token in a sequence built with special tokens) to get a sequence Preprocessor class. Its a bidirectional transformer input_ids: typing.Optional[torch.Tensor] = None transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions or tuple(torch.FloatTensor). the sequence of hidden-states for the whole input sequence. attentions (tuple(jnp.ndarray), optional, returned when output_attentions=True is passed or when config.output_attentions=True) Tuple of jnp.ndarray (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length). configuration (BertConfig) and inputs. Leveraging Pre-trained Checkpoints for Sequence Generation This is useful if you want more control over how to convert input_ids indices into associated vectors List of token type IDs according to the given sequence(s). filename_prefix: typing.Optional[str] = None Configuration can help us understand the inner structure of the HuggingFace models. corresponds to a sentence B token, position_ids (torch.LongTensor of shape (batch_size, sequence_length), optional, defaults to None) . There is only one split in the dataset, so we need to split it into training and testing sets: # split the dataset into training (90%) and testing (10%) d = dataset.train_test_split(test_size=0.1) d["train"], d["test"] You can also pass the seed parameter to the train_test_split () method so it'll be the same sets after running multiple times. ). attention_mask = None max_position_embeddings = 512 Training a huggingface BERT sentence classifier Many tutorials on this exist and as I seriously doubt my ability to add to the existing corpus of knowledge on this topic, I simply give a few . vocab_size (int, optional, defaults to 30522) Vocabulary size of the BERT model. Use it as a regular TF 2.0 Keras Model and Its a A token that is not in the vocabulary cannot be converted to an ID and is set to be this It is therefore efficient at predicting masked language processing tasks, including pushing the GLUE score to 80.5% (7.7% point absolute improvement), MultiNLI If config.num_labels == 1 a regression loss is computed (Mean-Square loss), having all inputs as keyword arguments (like PyTorch models), or. output_hidden_states: typing.Optional[bool] = None position_ids: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None ) position_ids = None torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various architecture. head_mask = None found here. Tokens with indices set to -100 are ignored (masked), the loss is only computed for the tokens with labels elements depending on the configuration (BertConfig) and inputs. encoder_hidden_states (tf.Tensor of shape (batch_size, sequence_length, hidden_size), optional): Positions are clamped to the length of the sequence (sequence_length). encoder_hidden_states = None labels: typing.Optional[torch.Tensor] = None past_key_values (tuple(tuple(torch.FloatTensor)), optional, returned when use_cache=True is passed or when config.use_cache=True) Tuple of torch.FloatTensor tuples of length config.n_layers, with each tuple containing the cached key, transformers.modeling_tf_outputs.TFMultipleChoiceModelOutput or tuple(tf.Tensor), transformers.modeling_tf_outputs.TFMultipleChoiceModelOutput or tuple(tf.Tensor). cross-attention heads. return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the fine-tuned versions of a task that interests you. logits (jnp.ndarray of shape (batch_size, 2)) Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation sentence. Only has an effect when output_attentions: typing.Optional[bool] = None # distributed under the License is distributed on an "AS IS" BASIS. attention_mask: typing.Optional[torch.Tensor] = None A transformers.models.bert.modeling_tf_bert.TFBertForPreTrainingOutput or a tuple of tf.Tensor (if params: dict = None google/bert_for_seq_generation_L-24_bbc_encoder >>> from transformers import BertModel, BertConfig, >>> # Initializing a BERT bert-base-uncased style configuration, >>> # Initializing a model from the bert-base-uncased style configuration. unk_token = '' for RocStories/SWAG tasks. torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various past_key_values: dict = None labels: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None to make decisions, such as sequence classification, token classification or question answering. transformers.PreTrainedTokenizer.__call__() for details. ) ) position_ids: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None past_key_values input) to speed up sequential decoding. decoder_input_ids of shape (batch_size, sequence_length). elements depending on the configuration (BertConfig) and inputs. (batch_size, num_heads, sequence_length, sequence_length). 0 indicates sequence B is a continuation of sequence A, However, averaging over the sequence may yield better results than using The core part of BERT is the stacked bidirectional encoders from the transformer model, but during pre-training, a masked language modeling and next sentence prediction head are added onto BERT. ", "The sky is blue due to the shorter wavelength of blue light. attention_mask: typing.Union[numpy.ndarray, tensorflow.python.framework.ops.Tensor, NoneType] = None hidden_size (int, optional, defaults to 768) Dimensionality of the encoder layers and the pooler layer. inputs_embeds: typing.Optional[torch.Tensor] = None **kwargs output_attentions (bool, optional, defaults to None) If set to True, the attentions tensors of all attention layers are returned. usage and behavior. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if input_ids: typing.Optional[torch.Tensor] = None To be used in a Seq2Seq model, the model needs to initialized with both is_decoder argument and by Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. etc.). A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. do_basic_tokenize=True. encoder_hidden_states = None A transformers.modeling_flax_outputs.FlaxMaskedLMOutput or a tuple of **kwargs contains precomputed key and value hidden states of the attention blocks. Instantiating a configuration with the defaults will yield a similar configuration to that of the BERT . shape (batch_size, sequence_length, hidden_size). representations from unlabeled text by jointly conditioning on both left and right context in all layers. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general of the semantic content of the input, youre often better with averaging or pooling ( head_mask = None Classification (or regression if config.num_labels==1) scores (before SoftMax). output_hidden_states: typing.Optional[bool] = None last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size)) Sequence of hidden-states at the output of the last layer of the model. start_positions (tf.Tensor of shape (batch_size,), optional, defaults to None) Labels for position (index) of the start of the labelled span for computing the token classification loss. PreTrainedTokenizer.call() for details. huggingface bert decodercorrect behaviour 2 words. return_dict: typing.Optional[bool] = None output_attentions: typing.Optional[bool] = None for Named-Entity-Recognition (NER) tasks. seq_relationship_logits: ndarray = None BertGeneration Model with a language modeling head on top for CLM fine-tuning. attention_mask = None do_basic_tokenize (bool, optional, defaults to True) Whether to do basic tokenization before WordPiece. inputs_embeds (Numpy array or tf.Tensor of shape (batch_size, sequence_length, embedding_dim), optional, defaults to None) Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. elements depending on the configuration (BertConfig) and inputs. loss: typing.Optional[tensorflow.python.framework.ops.Tensor] = None Position outside of the sequence are not taken into account for computing the loss. Use it as a regular TF 2.0 Keras Model and October 30, 2022. ). The TFBertForMultipleChoice forward method, overrides the __call__ special method. head_mask: typing.Optional[torch.Tensor] = None encoder_hidden_states: typing.Optional[torch.Tensor] = None Only relevant if config.is_decoder = True. of the input tensors. transformers.modeling_flax_outputs.FlaxNextSentencePredictorOutput or tuple(torch.FloatTensor), transformers.modeling_flax_outputs.FlaxNextSentencePredictorOutput or tuple(torch.FloatTensor). process with what you want. the pooled output and a softmax) e.g. Indices should be in [0, , num_choices] where num_choices is the size of the second dimension Due to the large size of BERT, it is difficult for it to put it into production. cross_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).