Skip to main content
onlineinference
Projects
gpt-python
Reports
Generating Content To Rank Using GPT-3.5 In Python

Generating Content To Rank Using GPT-3.5 In Python

Pick a phrase, pick a URL that's ranking and that you'd like your content inspired by, and let GPT-3.5 do the rest of the work, with this handy Python script.
Dave Davies
Created on March 24|Last edited on April 16
This tutorial will be updated to GPT-4 soon.

This is not to be used to actually generate your content for you!
Though it should give you a good idea of the type of content that would rank well.
💡
If you're reading a page on generating rankable content using GPT-3.5 with Python, then chances are you're either an SEO or just interested in the techniques that would be involved in the task. Either way, I hope you learn something, and maybe have a bit of fun while you're at it.
What makes this script a bit different (IMO) is that we're automating a tweak on the prompt being used to favor entities that are drawn from a URL of your choosing, as graded by Google's Natural Language API. Basically, we'll be making sure that the entities that Google things are the most relevant are included in the content.
Here's what we'll be covering ...

Table Of Contents

Table Of ContentsSetting Up Your MachineGetting Your OpenAI APIOpenAI PricingCreating Your OpenAI KeySetting Up Your Google Natural Language APICreating Your Rankable Content With GPT-3 In PythonDefining Your Target Term And InspirationInstalling The Required LibrariesAdding AuthenticationCreate The Functions To Scrape And Extract Entities From The URL We Want to Be Inspired ByThe Function To Generate The ContentScrape The Url And Output The EntitiesSet Up Weights & BiasesGenerate The Article PromptAnd Finally It's Time To Generate The Article!Create An Article Outline Instead Of An ArticleRecommended Reading



Setting Up Your Machine

The code and illustrations in this tutorial based on being written in a Jupyter Notebook.
Obviously the code can be repurposed in other tools and platforms, but if you're just getting started then you may want to set up your machine as I have, as I'm at a fairly beginner level myself.
Thankfully, it only takes a couple minutes and I've written up this tutorial to take you through it step-by-step.
So head over, get yourself going in Jupyter and I'll see you back here in about 2 minutes.
Now that you're set up ... let's get going!
💡

Getting Your OpenAI API

Before you can begin using GPT-3, you'll need an API key to access it.
To get one, simply head over to OpenAI's product page, and click "Get Started".
Get Started with OpenAI
Choose your signup method (I chose Google) and run through the verification process. You'll need access to a phone that can receive texts for this step.
Once you've completed that you'll head over to create an API key. This is so OpenAI can connect your scripts to your account. Important to know who's doing what, and determine if and how much they should charge you.

OpenAI Pricing

You get a $5 credit upon signing up, which will get you surprisingly far if you're just experiments.
As of this writing (March 12, 2023) the pricing past that is:
OpenAI pricing

Creating Your OpenAI Key

To create your key, simply click on your profile in the top right and choose, "View API Keys".

and then you'll create your key.

Once you close the lightbox you can't view your key and will have to re-create it, so for this project simply copy it to a notepad doc to use shortly.
Don't save your key unless you have somewhere highly secure (a notepad doc on your desktop is not). Once you're used it momentarily, close the notepad doc without saving.
💡

Setting Up Your Google Natural Language API

While GPT-3.5 does a solid job of finding entities within content, when it comes to inspiring content to rank, I prefer to get my entity information right from the source ... Google. To do this we're going to tap into their Natural Language API.
The Natural Language API will let you extract the entities from text, and passes along their understanding of the entity and its relationship to the content as a whole.
To get started you'll need to sign up for a Google Cloud account, which will then let you enable the Natural Language API.
To do this, just head over to https://cloud.google.com/
From there you can "Get started for free".


You'll have to enter your country, and organizational information as well as add a payment method.
Don't worry, they'll be giving you a $300 credit and if this is all you use it for, that'll last you well beyond the 90 days they give you to use it and the price after that is incredibly low.
Once you're in, you'll need to create a project which you can do here.


Give it a name like GPTcontent, or something memorable but potentially reusable as versions upgrade.
Next you'll need to add the Natural Language API:


And click "Enable APIs and Services".


A quick search for "natural language" and you end up with:


Just enable it:

And create credentials, so you can access it from within your scripts.


For this one we're going to need to create a Service Account:

Click "Create Credentials" and you can do that:

Fill in the details:

Now click on your newly created Service Account:

Click "Keys" and "Create New Key". It should be JSON.

It should automatically download to your browser download location. Don't forget to keep it safe.

Creating Your Rankable Content With GPT-3 In Python

Now that you've got your APIs and Jupyter Notebooks installed we can do what you came here to do.
Now here's where we start building our code. For simplicity, in my examples below I'm going to just use the headings of the steps in this article as the heading above each code block, and omit the brief descriptions which I'll be including here so you understand what's going on.

Defining Your Target Term And Inspiration

The first code block simply allows you to define what you want the article to be about, or the query you're targeting as well as an example URL that does rank and that you'd like to use as inspiration.
When you enter the code:
query = input ("What do you want to rank for :") 
print(query) 
url = input("What URL should I be inspired by : ") 
print(url)
and run it, you'll be prompted to answer the two questions:
  1. What do you want to rank for
  2. What URL should I be inspired by :

You'll notice that when ou click "Run" there's a little asterisk after the "In" in to the left of the code block.
You have to make sure you wait until it turns into a number (which is when the code has completed running) to move on to the next stage.
💡

Installing The Required Libraries

The next step is to install all the libraries we'll be using. The three main libraries are:
  1. Google Cloud Language - To extract entities known to Google.
  2. OpenAI - Because we'll be using GPT-3.5 to generate our content.
  3. BeautifulSoup - We'll be using this library to scrape the URL we entered above, to have the entities extracted.
  4. Weights & Biases - We'll be using W&B to log our entity metrics including the number of times they're included on a page and saliency, so we can adjust as necessary.
The code to do this is:
!pip install google-cloud-language beautifulsoup4 openai wandb

import requests
from bs4 import BeautifulSoup
from google.cloud import language_v1
from google.oauth2 import service_account
import os
import openai
Result


Adding Authentication

Next, we need to add our authentications. This will include our OpenAI API as well as the Google Application Credentials that downloaded earlier.
To do this, we need to copy the path to the file that downloaded earlier, including the file name, and replace the application credentials with it in the code:
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = 'PATH_TO_KEY/KEYNAME.json'
%env OPENAI_API_KEY=YOUR_KEY
openai.api_key = os.environ.get("OPENAI_API_KEY")
Result


Create The Functions To Scrape And Extract Entities From The URL We Want to Be Inspired By

We're going to need a couple functions to help us generate our awesome content.
The following code creates the function to scrape all content within paragraph tags from the URL we entered above, as well as the function to extract the entities from that content and pull out the top 20 entities by salience.
Basically, we're going to pull our the 20 entities that are most relevant to the content we're scraping.
def scrape_url(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, "html.parser")

    # Extract all text within paragraphs
    paragraphs = soup.find_all("p")
    text = " ".join([p.get_text() for p in paragraphs])
    return text

def analyze_content(content):
    client = language_v1.LanguageServiceClient()

    document = language_v1.Document(content=content, type_=language_v1.Document.Type.PLAIN_TEXT)
    response = client.analyze_entities(document=document, encoding_type=language_v1.EncodingType.UTF8)

    # Extract top 20 entities by salience
    top_entities = sorted(response.entities, key=lambda x: x.salience, reverse=True)[:20]
    return [entity.name for entity in top_entities]

Result


The Function To Generate The Content

Next, we're going to create the function to actually generate the content for us, using GPT-3.5.
This could be done with the code block above, but it contains elements that we may want to edit.
def generate_article(prompt, model="text-davinci-003"): #You can play with different models.
    openai.api_key = os.environ["OPENAI_API_KEY"]

    response = openai.Completion.create(
        engine=model,
        prompt=prompt,
        max_tokens=3500, #The maximum with GPT-3 is 4096 including the prompt
        n=1, #How many results to produce per prompt
        #best_of=1 #When n>1 completions can be run server-side and the "best" used
        stop=None,
        temperature=0.8, #A number between 0 and 2, where higher numbers add randomness
    )

    return response.choices[0].text.strip()
We could do the next three blocks together, but I think it's helpful to see the output of each, so you know what's going on.

Scrape The Url And Output The Entities

First, we're going to scrape the URL we entered above and extract the top 20 entities. We'll also print the entities to the screen for review.
content = scrape_url(url)
entities = analyze_content(content)
print(entities)
Result


Set Up Weights & Biases

The next step is to set up Weights & Biases Tables to receive the data. It's an optional, but recommended step as it's what will give you visibility into whether you're looking at the right entities.
Before you can log to Tables, you'll need to set up a free account. To do this, simply click here and follow the prompts and you'll be setup in about a minute.
...
Alright, we're back.
Next, you want to set up a project. Fortunately, we can do that with our Python code. The code below will automatically create the project in your user profile. If you are part of a team or have multiple profiles you can use the "entity" argument to define where you want the project logged. For our purposes here, we'll be assuming it's to log to your default profile.
wandb.init(project='entities-for-article-creation')
run = wandb.init()
table = wandb.Table(columns=['Entity', 'Frequency', 'Salience'])
Results


Generate The Article Prompt

Next, we'll generate a prompt from the information we've collected and print the prompt, so we can see what we're using to generate our content.
prompt = f"Write an article about {query} that includes the following entities: {', '.join(entities)}."
print(prompt)
Result


And Finally It's Time To Generate The Article!

The code we'll use for that is:
generated_article = generate_article(prompt)
print(generated_article)
Result


And a bonus!

Create An Article Outline Instead Of An Article

Assuming you don't want your article generated by GPT, but are interested in having an outline generated that's inspired by the URL we entered at the beginning, all it takes is a couple of little adjustments, and you can even do these are the end of your notebook.
If you change the prompt Python code to:
prompt = f"Create an outline for an article about {query} that includes the following entities: {', '.join(entities)}."
print(prompt)
and then run the generation code again, you'll end up with:

I can hardly wait to get this working with GPT-4.


Created with ❤️ on Weights & Biases.

https://wandb.ai/onlineinference/gpt-python/reports/Generating-Content-To-Rank-Using-GPT-3-5-In-Python--VmlldzozODc2OTc1
Made with Weights & Biases. Sign up or log in to create reports like this one.