How We Used GPT-2 to Write and Record a Sea Shanty
The year 2021 began with a rather unusual meme emerging from Tik Tok. In just a few short weeks videos of people signing sea shanties circulated on the platform in large numbers.
A sea shanty is a traditional folk song sung on large ships by sailors and merchants. Sea shanties were fun, rhythmic songs typically performed while performing heavy and repetitive tasks like hoisting sails and scrubbing decks.
We at GravityAI decided to join in on the Tik Tok craze by creating our own sea shanty. The difference... It was completely generated by an AI.
In this blog post, we will share how we developed an AI to write Sea Shanties.
Data Gathering
Our fun project began by souring the web in search for sea shanties we can use build the AI. We were able to collect lyrics from dozens of sea shanties from Ranker.com, Genius Lyrics, and Apple Music. Once we prepared our dataset of sea shanty lyrics, we were ready for next step… building an AI that can generate sea shanty song lyrics.
Model Selection
There are many approaches that can be taken for generating sea shanty lyrics. The approaches include models as simple as Markov chains and as complex as Long Short-Term Memory networks (LSTM). After reviewing all the options available, we decided to use GPT-2.
GPT-2 is an open-source language generation model that was created by OpenAI. It can generate text at a level that at times can be indistinguishable from text written by humans.
GPT-2 was pre-trained on a dataset of over 8 million web pages, so the text that it produces out of the box is generic. With a little bit of time and effort, however, the model's 1.5 billion parameters can be tweaked to generate text from any domain that is desired. This is exactly why we chose this model over the others we looked at.
The way the model's parameters are tuned is through a process called Transfer Learning. Transfer learning involves applying the knowledge and learnings gained solving one problem to a new, related problem.
In the context of GPT-2, the original problem it solves is predicting what word comes next given a sequence of words that come before it. The learnings GPT-2 gained from this task is represented by its many parameters. Since predicting the next word in a text on a web page is just like predicting the next word in a song, much of GPT-2 learnings can be leveraged in learning how to create sea shanty lyrics. We combined all the lyrics we collected into one large text file, fed it to GPT-2 and let it learn from the dataset to create its own sea shanties. We found the Google Collab notebook assembled by Max Woolf to be extremely helpful for this task.
Results
After letting GPT-2 learn from the dataset for many cycles, me made the model take some stabs at creating sea shanties. Below is some of the samples that the model generated.
Sample 1:
"Well" he answered, "Baby, don't you know that it's rude To keep my two lips waitin' when they're in the mood" He was a famous trumpet man from out Chicago way He had a boogie style that no one else could play He was the top man at his craft But then his number came up and he was gone with the draft He's in the army now, a-blowin' reveille He's the boogie woogie bugle boy of Company B They made him blow a bugle for his Uncle Sam It really brought him down because he couldn't jam The captain seemed to understand Because the next day the cap' went out and drafted a band And now the company jumps when he plays reveille He's the boogie woogie bugle boy of Company B A-toot, a-toot, a-toot-diddelyada-toot He blows it eight-to-the-bar, in boogie rhythm He can't blow a note unless the bass and guitar is playin' with 'I'm He makes the company jump when he plays reveille He's the boogie woogie bugle boy of Company B"
Sample 2:
"Well now can't ya see... the black clouds a-gatherin' Away Haul Away, we'll haul away, Joe [Chorus] Well now can't ya see... the storm clouds a-risin' Away Haul Away, we'll haul away, Joe [Verse] I used to have an Irish girl but she got fat and lazy Away Haul Away, we'll haul away, Joe But now I've got a Bristol girl and she just drives me crazy Away Haul Away, we'll haul away, Joe [Chorus] Well now can't ya see... the long boats sail in the morning and the evening goes by Well, your parents are far away, buddy Away Haul Away, we'll haul away, Joe [Chorus] Well now can't ya see... the sailances are fair and the night goes by Well, your parents are far away, buddy Away Haul Away, we'll haul away, Joe [Verse] Oh, one more pull and that will do! Clear away the track an' let the bullgine run! Away Haul Away, we'll haul away, Joe Clear away the track an' let the bullgine run! Away Haul Away"
Sample 3:
"There were four old whores of Baltimore Drinking the blood red wine. And all their conversation was "Yours is smaller than mine.' I said, "Well, it's not much, but I'll be true to my word" "Well" he answered, "Baby, don't you know that it's rude To keep my two lips waitin' when they're in the mood" In the mood, that's what he told me In the mood, and when he told me In the mood, my heart was skippin' It didn't take me long to say, "I'm in the mood now" In the mood for all his kissin' In the mood his crazy lovin' In the mood what I was missin' It didn't take me long to say "I'm in the mood now" So, I said politely, "Darlin' may I intrude" He said, "Don't keep me waitin' when I'm in the mood" "Well" he answered, "Baby, don't you know that it's rude To keep my two lips waitin' when they're in the mood"
The initial samples from GPT-2 turned out pretty good. It was also able to pick up on nuances in the lyrics such as section labels ('Verse' and 'Chorus') and the structure of the lyrics into stanzas and verses.
What we also noticed after taking a closer look at the lyrics many cases where the model pulled phrases directly from the original song lyrics instead of coming up with phrases of its own.
The second sample of lyrics produced by the model, for example, uses the phrase "Away Haul Away, we'll haul away, Joe" numerous times. This phrase was pulled directly from the sea shanty song of the same name, Haul Away Joe performed by the Longest Johns.
To encourage more randomness and originality in the text the model outputs, we adjusted the temperature parameter in the model's text generation routine. Bumping this parameter to a higher value allowed us to get the kind of randomness that we desired.
Sample 1:
"Chorus: Oh, hey, a.k.a. The rose is up on the Torrey line! Oh, well, I'm the type of guy who will never settle down Where pretty girls are, well, you know that I'm around I kiss 'em boys goodnight and they don't mind They just keep on going, like that for all the Indian in the bed I'm dreaming of, just a dreamy sound As soon as you take a breath, I'm taking you there Oh, well, there's Flo on my right and there's Mary on my left And Janie is the girl, well, that's what we'll be tryin' to make a living as photographers Doin' portraits of the sun, moon, pony And the birds, they all fly! I'm sittin' here, la, la Waiting for my ya ya Uh huh, uh huh I'm sittin' here, la, la Waiting for my ya ya Uh huh, uh huh It may sound funny But I don't believe she's comin' Uh huh, uh huh Baby hurry don't make me worry Uh huh, uh huh Yeah, baby hurry don't make me."
Sample 2:
"There was a ship that sat on a pierhead And we say so And we hope so She sailed on the waters of desire Where the wind was in the woman's night-owl, and the waves were in the ship's sleep She sailed on the waves of desire I dreamed her in the evening, Low lands lowlands lowlands low She sailed on the waves of desire I dreamed her in the early bright And the name of the ship was the Turkish Revelry She sailed on the seas of desire I dreamed her in the early bright The name of the ship was the Revelry She sailed on the seas of desire She sailed on the waves of desire I dreamed her in the early bright The name of the ship was the Turkish Revelry She sailed on the seas of desire I dreamed her in the early bright The name of the ship was the Turkish Revelry We struck a blow against the belly of the boat, and the heart of the convict Could not rise A bullet struck the whale's side The whale's tail met with mine and we lost 467 men Our gills were sore gagged, our jowl was sagged With midnight leave to mourn And we turned the poor thing over And did not catch the whale."
Sample 3:
"There's fire down below, and the parson was right (fire) Down in the alley. A little boy's dying, his name was Mack. He was a bugger. And he pulled a knife on his friend's soul. He killed him for his poor soul. The parson's little daughter was born, and dwelt on earth. And her name was Margaret. She was a dress was her red flag, down in Kinloch, B.C. Embodyed by the wild goose, B.C.R.S. She died of a fever, and no one could save her. Her ghost rose from the dead land, B.C.R.S. Where the wild goose was done, B.C.R.S. Was buried, B.C.R.S. A little boat was sunk, and the name of the ship was Thelma Thelma Toil and danger still gale. B.C.R.S. She sailed down along the coast of Peru, And finding little Thelma, the ship was buried in the channel of the river Torito Thelma Oh, well, I'll take."
Once we were satisfied with the results, we sent samples of the lyrics generated by the model to a few musicians that we know. They took the lyrics we provided, cleaned it up a bit, and produced the recording that you saw at the beginning of this post. …
If you'd like to experiment with GPT-2, we have a pretrained GPT-2 model available in the Marketplace to download. Create a free account on GravityAI and generate your own text today.