Skip to main content
W&B Fully Connected
>
Articles
>
GPT

Editing GPT-4: What LLMs Do and Don’t Write Well, and How To Use Them for Professional Writing

Want to learn how can you use GPT-4 or other large language models (LLMs) in your day-to-day work? Well, buddy, you're in the right place — read this article.
Justin Tenuto
Created on May 4|Last edited on May 12
Though ChatGPT and GPT-4 are relatively new, they've enjoyed the sort of widespread adoption every company dreams about. And while we don't know exactly how large language models (LLMs) will transform our day-to-day work, we're already seeing a ton of GPT-powered writing coming into Weights & Biases. In fact, we're both using it internally and receiving submissions for this very blog that quite obviously leverage LLMs to construct prose.
The question is: how good is this writing anyway? Where do LLMs struggle? Where do they excel? And how can you use them to help your writing?
And before you ask: no. We're not using LLMs for this piece. But to keep with the spirit of LLMs, we'll prompt ourselves. Seems only fair.
Here's what we'll be covering:

Table of Contents

How can you tell if an LLM’s been used in the first place?Like what? Tone is kinda nebulous. How does that actually manifest itself?What about actual craft? Do LLMs write well? My high school English teacher would probably be happy about that. What else are they good at? Let's get down to brass tacks here. Say I want to use LLMs to improve my writing or just understand how to use them in a professional setting. Where should I start? Noted. Now tell me how I can use it. You know, that's a pretty decent idea. What else you got? I've seen stuff like ChatGPT before, but I'm not really an expert. How do you prompt these things anyway? Easy enough. Is that all I need to do? Anything else I should know?


How can you tell if an LLM’s been used in the first place?

There's a little bit of a Potter test thing happening here (a.k.a. you know it when you see it). LLMs are trained on tons of data, but stylistically, they tend to produce the same kind of writing unless there's heavy style prompting at play. The best way to describe it is "flat but helpful." There are a few telltale signs, though.

Like what?

LLMs tend to produce a sort of median style of prose. They do well explaining things, but they don't do it with much pizzazz or personality. This kind of writing can be great for, say, technical docs or walking someone through a recipe for apple pie but not for an engaging introductory anecdote. It's not a bad style per se, but it's not a particularly notable one. And once you've read a few of them, you sort of pick up on the general vibe.

Tone is kinda nebulous. How does that actually manifest itself?

Well, for starters, unless you prompt them, LLMs don't really know who they're writing for. And that's important. After all, good writing knows its audience. You're on a machine learning blog right now, and our readership is almost entirely ML practitioners. Because of this, we'd never start an article explaining what a neural net is or the purpose of hyperparameter tuning. That's because we assume ML practitioners know this stuff. You don't see basketball writing that explains what a pick and roll is.
LLMs don't know who their audience is, so, much like with their tone, they write for a median audience. That means they tend to over-explain basic concepts your readers probably understand already.
Additionally, LLMs don't really do transitions well. As a weirdly effective workaround, they tend to go for listicles more than a lot of writers might prefer. It's also worth mentioning that a lot of LLM-driven content often comes from multiple stages of prompting (i.e. a prompt for the intro, then another for a section of the body), so it's almost unfair to expect transition elements. It's also why you might see an LLM spell out abbreviations–Weights & Biases (W&B)–multiple times in a submission.

What about actual craft? Do LLMs write well?

That's a tricky question! LLMs, above all else, write quickly. They can spool out paragraphs of fairly solid, instructional copy near instantaneously. It's genuinely impressive.
"Good" writing, of course, is a matter of taste. but in terms of grammar, LLMs aren't good. Honestly, they're great. They don't write ambitious sentences with tons of nested clauses. They get the point across without showiness. Notably, they don't really mess up on typical rules, which is impressive considering some of their training data contains errors. In fact, we decided to check on two rather pervasive (and, yes, rather pedantic) errors the web makes widely:


Good work, GPT-4.

My high school English teacher would probably be happy about that.

I'm sure they would be. Honestly, the cleanliness of the copy is probably a big reason people are this impressed. It's a lot tighter than most of the emails you'll get on a daily basis.

What else are they good at?

As we mentioned above, they do a great job at step-by-step, tutorial-style writing. Think FAQs, Q&As, and some kinds of documentation. They can do a little padding and can sometimes overexplain things, but for a how-to piece, that's not the worst thing. In fact, we'd generally rather that than underexplaining. Worst case scenario is an expert skims or CTRL+Fs to the thing they're really there for.
They do a nice job at summaries and brief descriptions of past events as well. It's important to remember that LLMs are trained on historical data so you shouldn't expect them to understand the news that dropped a couple of days ago. But because of the breadth of data they've been trained on, they provide surprisingly good nuance where there's disagreement on fundamental issues at hand.

Let's get down to brass tacks here. Say I want to use LLMs to improve my writing or just understand how to use them in a professional setting. Where should I start?

For the purposes of this answer, we're going to ignore creative writing like screenplays and novels. That's a whole other piece, and frankly, LLMs don't do this well. If you're a creative writer, you could look to an LLM for some new character names, for example, but don't expect subtlety or multiple layers of meaning. Write it yourself.
For the purposes of business writing? That's a lot different.
You want to start with the understanding that LLMs won't simply create you something for from the whole cloth. They're good, but they aren't that good. You'll need to prompt well, reprompt, tweak, tune, and likely prompt the LLM for different parts of the piece. Know that going in.

Noted. Now tell me how I can use it.

Fair enough. The first thing worth recommending is thinking of LLMs as a sort of idea thesaurus. Say you're running ads on a new product your team has created. You understand the product well, and you've got a decent messaging idea, but it isn't really singing for you. It's close, in other words, but it's not there.
Take that idea and simply ask an LLM like GPT-4 to spin up some new ideas. Literally, just ask it.

Are these all winners? No, but hey: neither was Juicero. But as a way to brainstorm with yourself? LLMs really excel here. They'll give you smart ideas for A/B tests, email subject lines, and other short, pithy copy. In fact, our experience is that LLMs write better short, punchy copy than longer pieces. It's also much easier to get yourself acquainted with their quirks working with slimmer copy.

You know, that's a pretty decent idea. What else you got?

Oh, plenty. LLMs write pretty clean SEO copy, though professional SEOs are going to want to get in there and tweak their H1s and hunt snippets. They do provide a really solid skeleton to start with, though, and they tend to be good generalists overall. What we mean here is if your topic isn't deeply esoteric, you should expect an LLM to explain the basics well, especially if any part of that copy is tutorial in nature.
Now, say you're selling furniture and just got a massive shipment of products without descriptions. This is some of the most repetitive and tedious writing around, but LLMs can reduce the time you spend writing about the 140th couch you're listing this week by orders of magnitude.
We're assuming you can see a theme emerging here: the more purely functional the copy, the better. The shorter the copy, the better. Once you're asking for something that stretches above a thousand words, you're likely going to see diminishing returns.
That said, if you do want to take an LLM for a spin on a blog post, it generally works better to prompt it on smaller sections. Most times, LLMs write pretty lifeless, high school-level intros and conclusions ("In this piece, we learned," etc.), but they write body copy well. Just don't sit down hoping to get a fully formed piece. To continue our example above, don't ask for a blog extolling the virtues of daily, fresh-squeezed juice. Instead, come up with an outline and prompt it accordingly. In other words, instead of prompting, "Write me a compelling sales piece for why someone needs a state-of-the-art juicer," instead say, "Write me a few short paragraphs on the physical and mental benefits of daily fruit intake." You'll get better results.

I've seen stuff like ChatGPT before, but I'm not really an expert. How do you prompt these things anyway?

Prompt engineering is a fairly new discipline, but we've taken a crack at covering it here, and we recommend you check that out if you'd like a full rundown. Here, we'll just talk about how we've seen it used most effectively for corporate writing.
For shorter copy–think emails, ads, or anything else that's sub 300 or 400 words–you have a little more wiggle room. You can simply ask the LLM to write you something and go from there. Remember, this isn't Google search. You can–and should–be specific with your ask. Note the tone you want, the benefits you want to speak to, and the pain points you're solving. Ask for multiple options. Most LLMs will be able to build on themselves, so prompt them to go deeper. There's a bit of a dance here you'll learn with experience, but the best advice we can give is to stretch out a bit and ask for more than what you'd expect.
For longer pieces–think blog posts or leave-behinds–we've found it's best to approach with an outline. LLMs can put together a coherent piece in its entirety but remember: they're generalists. You're the subject matter expert, and you know the benefits, intricacies, and landmines in your product. You know what to highlight and what to avoid far better than a model that may or may not be familiar with what you're writing about (especially if it's new or niche). Use subsections of the outline to prompt the model and, again, be specific with your asks. You'll get better results this way.

Easy enough. Is that all I need to do?

Probably not. This comes full circle to the tone issue we brought up above. LLM content needs to be punched up. You should expect to include transitions, remove redundancies, redo intros and conclusions, and, above all else, add some personality. LLMs don't really write content that sings or stands out. They write solid, functional, actionable copy. And while that's really impressive and a dramatic improvement from where we were even a few years ago, LLMs won't replace you or do your job for you. They can absolutely help, though.
Basically, most of the time, you're gonna want to think of what an LLM churns as a good first draft.

Anything else I should know?

Frankly, we could talk about this for another 2000 words, but that's probably enough for now. If you start really digging into prompting and understand how models behave in a more granular way, we've just released an LLM tool called Prompts that helps you do just that. Past that, we'll leave you with a few takeaways:
  • LLMs are typically better at shorter copy than long
  • They benefit extensively from more precise prompting and multi-step prompting
  • They produce fairly flat prose, so you'll either want to add some flavor if you aren't writing docs or tutorials
  • You absolutely can use these in professional, external-facing pieces but think of LLM copy as a really good first draft you'll have to improve


Tags: Articles, Beginner, Tutorial, GPT, LLM, NLP

Created with ❤️ on Weights & Biases.

https://wandb.ai/wandb_fc/LLM%20Best%20Practices/reports/Editing-GPT-4-What-LLMs-Do-and-Don-t-Write-Well-and-How-To-Use-Them-for-Professional-Writing--Vmlldzo0MjYzMTgz
Made with Weights & Biases. Sign up or log in to create reports like this one.
Iterate on AI agents and models faster. Try Weights & Biases today.
Sign upTry W&B now