Nov 27, 2020

7 min read

Three Out-Of-The-Box Transformer Models

Photo by Enrique Guzmán Egas on Unsplash

How many lines of code does it take to use one of Google AI’s top-performing language models and apply it to your own text summarization project?

Or how about intelligent language generation? You start writing, and OpenAI’s GPT-2 finishes — how many lines of code?

Must be a lot? No — both can be built in just seven lines of Python.

In my opinion, that is absolutely ridiculous. Seven lines of code to apply models that are the culmination of decades of work, produced by some of the most intelligent people on Earth, using millions of dollars of research funds.

Already, my mind is blown. It’s incredible, but it’s true. This article is a list of cheat codes to apply three of these spectacular models to your own work — with near-zero overhead.

> Text Summaries with T5> Sentiment Analysis with BERT> Language Generation with GPT-2

Enjoy!

Summarization with T5

Our first model covers text summarization. This consists of taking a large amount of text and compressing it down into a smaller body of text — while preserving the most important details (covered in the video below).

This seems like it should be an astonishingly complex task, an unlikely feat to pull it off without already having plenty of experience beforehand.

But, that is not the case.

In-Full

The model is incredibly easy to set up and use. There are three key steps:

Tokenization of text to input IDs
Processing by the T5 model
Decoding of the model outputs back to text

All of this in code is:

That’s all it takes! Let’s take a chunk of text from the Wikipedia page about Winston Churchill and see what happens when we feed it into our summarizer:

T5 summarizer output.

With just a few lines of code, the model has brilliantly summarized what I would perceive as some of the most important parts of the text.

A drawback — in this case — is that the model has solely focused on the second paragraph. Missing important information present in the third. Nonetheless, for an out-of-the-box solution — it’s pretty impressive!

Text Classification with BERT

Text classification is simple but incredibly useful. One of the most common use-cases for natural language classification is sentiment analysis.

We will use sentiment analysis in our example — however, we can use this same approach for any text classification task.

Unlike the other two use-cases, we do need to train our model — so that it understands what we are asking it to classify. The steps are slightly different because of this:

Preprocessing (including tokenization) of the data
Training of the model
Classification of test/real data using the model

Model

The model itself is incredibly easy to define and follows the same structure as the other models.

Pre-Processing

It’s incredibly easy to build the model; there’s almost nothing to it. However, we need to make sure we are feeding the right data into our model.

We will use the movie review sentiment dataset from here — using train.tsv to train our dataset.

The model expects tokenized IDs and attention mask tensors as input, and one-hot encoded target labels.

Both input tensors are built using the encode_plus method. The IDs are returned by default, and we add return_attention_mask=True to return the attention mask tensor too:

One-hot encoding of our target labels is handled easily with nothing more than Numpy:

Because we’re using TensorFlow — we can easily format, shuffle, and batch our data using the built-in Dataset object:

Train and Predict

We need to train our model this time too. First, we compile our model, then begin training with model.fit:

Now we have our text classification model, built with Google’s BERT! We can make predictions with:

This model isn’t as clean-cut due to the need to train our classifier, but other than this, there is nothing more we need to do — and the level of accuracy possible on language classification tasks using BERT is simply phenomenal.

I have written about this use-case in more depth here.

Language Generation with GPT-2

Last but not least is language generation with GPT-2 (covered in the video above). The little brother of GPT-3, with 1.5B parameters, GPT-2 is hardly small — and only seems so when compared to GPT-3’s 175B parameters.

We cannot directly use GPT-3, but GPT-2 is widely available, and we can use it with incredible ease.

We will be building a language generator with it. Feeding a sentence or paragraph into the model to generate rational, coherent language.

Model

We take a very similar approach as with the other models in this article:

Tokenization of text to input IDs
Generation of language by the GPT-2 model
Decoding of the model output back to text

All of this is done with just:

When we apply this code to a short extract, again from Churchill’s Wikipedia page — we get this:

The output of GPT-2 language generation produced from the highlighted text (the input).

We can tune the randomness and coherence of the output using temperature and top_k — which occasionally offers us some pretty entertaining outputs.

An alternative history where Chamberlain’s mother took over as prime minister after his premature death in 1918 — and Sir Richard Branson served on Churchill’s International Energy and Scientific Research Council in 1958.

That’s it for these three incredibly easy to use transformer models — applying some of the most powerful language models on the planet in just a few lines of code.

More importantly, we have covered just three models. There are a huge number of models out there covering a wide range of applications — such as question-answering, translation, and name-entity recognition — to name a few.

I hope you’ve enjoyed this article. If you have any questions or feedback — feel free to reach out on Twitter or in the comments below.

Thanks for reading!

Three Out-Of-The-Box Transformer Models

Summarization with T5

In-Full

Text Classification with BERT

Model

Pre-Processing

Train and Predict

Language Generation with GPT-2

Model

More from iopio qwe

Recommended from Medium

Mac App To Interact With Windows Partition

Why Hackathons are tools for social impact

How to Download Magisk Root on Your Android Device

(Finally) Understanding Monads (Part 1)

SQL — Select and Lookup

A Deeper Understanding of C#

When Hotstar Met Exoplayer

[Leetcode] Training day-004

Get the Medium app

iopio qwe

More from Medium

MoCo: Inspired by Deep Reinforcement Learning?

Machine Learning Model Validation — The Data-Centric Approach

Bayes’ theorem in-depth: Clearly Explained

This is why AI eXistential-risk is a lie