ChatGPT vs. Human Copywriters: 3 split tests to determine the future of email marketing (and whether or not A.I. will replace us all).

Want to get coached by Smart Marketer’s CMO so you can master our email marketing system for your brand?

Join Train My Email Specialist!

This hands-on mentorship is the most comprehensive email training we’ve ever offered, and it starts Monday, April 17!

---

Hey, it’s John Grimshaw!

Today, I want to discuss a topic that will make some people very happy and some others very angry.

It’s ChatGPT — the new A.I. tool that has sent the marketing industry ablaze

Whether you’re excited to learn how to use this tool to grow your business, or you’re just worried it might end up taking your job, one thing is for sure: the hype train is real.

So today, I want to look at ChatGPT specifically through the lens of email marketing.

Can this A.I. write better copy than a human? And how can we use it right now to save us time and possibly even increase the results of our emails?

To answer these questions, I ran 3 split tests where I pitted the work of ChatGPT against that of our professional copywriters, and we ended up with some really interesting data.

I hope you’re excited! But before we get into our first split test, let’s take care of some house cleaning.

What is ChatGPT and How Does it Work?

What is ChatGPT? It’s a new A.I. technology that has essentially read a bunch of text on the internet and has used that data to create a large language model.

For these tests I used the GPT-3 model, but a GPT-4 model came out recently that is even more advanced (and costs money).

You can ask it just about anything, from complex philosophical or scientific questions, to what are the most famous celebrity feuds (Taylor Swift vs. Kanye I knew about; Madonna vs. Elton John I didn’t).

There is one caveat: ChatGPT only has access to information before September 2021, so it has no knowledge of current events (or in our case, our current products) unless you feed that info into the model.

There are a myriad of uses for this amazing new technology, but for most marketers, it prompts a single question:

How do we use this thing to make money?

For the average marketer or business owner, ChatGPT can be leveraged for profit in 2 ways:

1. 1. By teaching people how to use it.This is for all you info businesses out there.

      ChatGPT is a tool, and as with any exciting new tool, people want to learn how to use it.

      So one way we could profit from it at Smart Marketer, for example, is to make a course on how a person can use it in their business (but don’t hold your breath).

 

1. By using it to do the work of people in your business.This is what most people are interested in.

   You don’t need to be an info brand to use A.I. to do work that’s currently being done by your team.

   This would allow you to be more efficient, and to free up time currently occupied by tedious work so that manpower can be better used elsewhere.

 

The Split Test

To see if ChatGPT could do the work of a copywriter, we took 3 emails written by real, live humans and asked the A.I. to create a variation of each of them.

These 3 emails were:

• A coupon delivery email
• A blog post email
• And an opt-in email

 

And we got some very, very interesting results.

So let me walk you through our findings for each test, including:

• The two emails in each test (one human, one A.I.)
• The test results
• Our main takeaways
• And what we think this all means for the future of email marketing

 

Test 1: Coupon Delivery Email (BOOM! By Cindy Joseph)

The first email is a coupon delivery email for BOOM! By Cindy Joseph (Ezra’s 9-figure ecommerce brand).

When you go to the BOOM! website, you will probably see a pop-up offering 10% off your first order in return for your email address. When you opt-in for that coupon, this is the email that delivers it to you.

Here’s the version written by our copywriter:

It’s a fairly generic email, honestly. But there’s a good visual in there, and it does a really good job of emphasizing the scarcity component: “You’ve got 72 hours.”

Now, let’s contrast that with the email written by ChatGPT:

The subject lines were the same for both of these emails, so that had no impact on the test.

This email is significantly longer (a trend we’ll continue to see in ChatGPT’s other emails), and goes deeper into specific products we could showcase.

We also asked the A.I. to recommend an image for the email, which resulted in the picture above.

So, to all of you at home — which email do you think won?

Test 1: Results

And the winner is… neither one. (Wah wah. Sorry for being anticlimactic.) They are perfectly neck and neck.

It says right now there’s a 63% chance that the ChatGPT email will win based on the click rate. But it’s 6.5 versus 6.3, and we’ve already run 6000 people through this test.

So I think it’s very unlikely that we are going to get a meaningful winner unless we decide to let this thing run for another 30 or 40 days.

There’s a minor difference in the open rate, but we actually used the same subject line for both emails, so that’s not significant.

The difference in placed order rate was interesting at first, but initially the copywriter’s version was winning by a pretty significant margin. It was $3,000 in revenue versus $2,000 from ChatGPT’s version.

This normalized, however, and now the human version is sitting at about $4.7 thousand generated across 55 orders, and the ChatGPT version is sitting at $4.66 thousand with 54 purchases.

So that’s about as close as you can get, right? (Don’t worry, the results get much more interesting in the next test.)

Test 1: Takeaways

So these results weren’t super conclusive — the emails appear to be equally good! — but that doesn’t mean we didn’t learn anything from the test.

1) ChatGPT likes to talk.

I found it really interesting that ChatGPT defaults to longer form copy.

It likes to talk (that’s what it’s been designed to do, afterall), so it tends to write longer emails.

And we played around with the prompts quite a bit. We probably did six or seven variations, and this was the winning one.

We didn’t just sit down to type for 10 seconds and ChatGPT gave us the answer. It took a little bit of playing around.

Here’s the prompt for the first email:

“Write me a marketing email, including a subject line and email body from the brand BOOM! by Cindy Joseph. The email is for people who just signed up for a 10% off coupon. They are expecting this coupon and the email should reflect that.

The important details for this email are that the discount works for all our products, but it’s only available for 72 hours, so recipients need to act quickly to take advantage of this deal. Match the style and tone from this website (boombycindyjoseph.com).

The email body should be between 100 and 200 words, and include at least three links to the website. Assume that this is the first email this person is receiving from Boom by cindy Joseph and that they are already signed up for the newsletter.

Additionally, describe an image that should be included in the email which should entice potential buyers.”

We included a lot of instruction regarding what we wanted. We iterated it quite a few times, but that’s where we wound up.

When giving it a range of 100–200 words, it definitely shoots for the long end of the range. I tried a few versions where we asked for 50–75 words, but it wasn’t that good at being concise.

So I think the more it has space to talk, the better a job it does in general.

2) ChatGPT is good at explaining things in great detail.

I think what it did really, really well was go a bit deeper into some specific products.

We obviously had to overlay some of our own formatting and design because all it can do is spit out text, but I do like the addition of some specific products that we recommend people check out and use their coupon on.

We’ll definitely take some of the product callouts from this email to use in our standard email that we’re trying to beat.

3) It didn’t play up the scarcity well.

Even though I asked ChatGPT to stress that the reader only had 72 hours to act, the A.I. didn’t do a good job of playing up the scarcity of the offer.

We might want to test what it can do further down funnel, when it’s more in the informational stage.

Overall, it did about as well as our copywriter, but not in this area — probably because scarcity is all about being punchy and direct (something it’s not as good at).

Test 2: Blog Post Email (Smart Marketer)

The next test is the most interesting one, hands down.

In this email, we were sending our list a blog post that Molly had written with our 2023 marketing predictions.

(She sourced input from experts like Mike Rhodes, Brett Currie and Tommy Traffic — it’s a really great post if you haven’t seen it.)

Our goal was to see which email would drive more clicks to the blog post.

The original version was written by our lead copywriter, Rubin. Here’s his email:

Subject line: 10 Marketing Predictions for 2023 ?

And here’s the ChatGPT version:

Subject line: “Take your business to the next level in 2023”.

Test 2 Results

So as far as opens, our copywriter got a 40.6% open rate and ChatGPT got a 39.9%, so basically neck and neck.

Rubin included preview text in his version — “Including Apple’s ads platform, consultative marketing, and more” — while the A.I. version didn’t have that, which didn’t seem to impact the results either way.

(NOTE: Keep in mind that open rates are not as accurate as they used to be since Apple’s latest update, when they started opening, scanning and caching emails.)

But while there wasn’t a big difference in open rate, there was a massive difference in click-through rate.

Rubin’s version had a 2.7% CTR, while ChatGPT’s had 1.3%. So the human copywriter got twice as many people to click over to the website.

Why is this? What was so good about it? Well, if you look at the two emails above I think some obvious things jump out.

Test 2 Takeaways

1) The copywriter’s email is more fun.

First of all, Rubin’s email does a better job at inducing curiosity.

That email is more fun, right? It does a better job of using voice in the email, something ChatGPT isn’t very good at.

And especially for Smart Marketer, which is a brand largely driven by Ezra’s and Molly’s personalities, this is extremely important for us.

2) ChatGPT doesn’t know how to optimize for readability yet.

This is probably my biggest takeaway from the entire experiment.

Rubin’s email is also more concise. Visually speaking, it’s very easy to scan.

The version that the A.I. wrote, on the other hand, uses larger paragraphs that aren’t as easy on the eye.

So if you choose to use this tool for email, make sure someone goes through the content to break up sentences and cut out extra words to make it more concise and focused.

Now, I will admit that I did ask it to speak about what Molly’s prediction was as a way for it to have a little bit more to say.

Here was the prompt:

“Read this blog post and then write me a marketing email, including a subject line, an email body from the brand Smart Marketer, and from the perspective of Molly Pittman.

Keep sentences short and punchy and only hint at the ideas in the post. Don’t explain everything. The email must mention consultative marketing, an interesting statistic from the post and the names of at least three of the other experts included in the post.

The email body should be between 100 and 150 words. Assume that the reader is a business owner looking to keep their business growing and healthy in a turbulent economic climate.”

But even though I asked it to be concise (between 100 and 150 words), it still wanted to go deep and get wordy.

ChatGPT struggles to understand that the email is a conduit, rather than the destination. It loves to hear itself talk, and hey — I can relate.

But all credit to Rubin. Throughout this project, I’ve been thinking of the folk story of John Henry, the steel-driving man who said that he could do better than a brand new steam-powered rock drilling machine. And he did!

Test 3: Opt-in Email (Smart Marketer)

The last email we tested was an opt-in email for our 2023 Email Marketing Performance Report.

This resource has a bunch of insights and lessons we’ve learned running email marketing for a dozen brands, including our own businesses as well as our agency clients.

The goal of the email was to see how many people would click-through to the resource opt-in page.

Here’s the copywriter’s version:

Subject line: What’s Working in Email Marketing in 2023

And the ChatGPT version:

Subject line: Get your free copy of the 2023 email performance report today!

(I actually didn’t write a prompt for this one, so I don’t have that to share with you. I handed this off to another one of our copywriters and said, “Write me an email and then have a ChatGPT write a variation, and let’s see who wins.”)

Test 3 Results

The copywriter’s email got an open rate of 38.8%, whereas the ChatGPT version got an open rate of 36.9%. So a small win for the copywriter’s subject line, but nothing astronomical.

But the click rate was really where things got interesting.

The copywriter got 0.78% whereas the A.I. got 0.94%, so ChatGPT actually definitively beat the copywriter in this case.

(These results are even a little deceptive, because in Klaviyo, click rate is clicks divided by deliveries rather than clicks divided by opens. This means ChatGPT did even better, because it had a slightly weaker open rate.)

Split Test 3 Takeaways

So, what is there left to learn from this third test?

ChatGPT used the same approach as before — we already know it gets a little too wordy, and we already know it struggles to write scannable copy.

1) Every once in a while, it gives you a gem.

I really liked a few of the phrases it used in this email.

For example: “We would like to offer you a complimentary copy.” I think that’s a good turn of phrase for delivering a lead magnet.

2) ChatGPT’s lack of voice will hurt your brand.

However, it didn’t use Ezra’s voice in any meaningful way.

(It definitely didn’t include anything as pithy as the copywriter’s, “Email is like a good pair of brown leather oxfords. It never goes out of style”.)

Unsurprisingly, this email sounds like it could have been written by a bot.

So even though this individual email won, I wouldn’t want ChatGPT to write every email for this brand.

Building relationships with your customers is about having a voice, a sense of who you are and what you want to talk about and what kind of value you can add.

Ezra’s success with Smart Marketer didn’t happen just because he’s smart (which he is). It’s because of his personality and his pure way of being.

This email from ChadGPT captured none of that.

It’s as dry as a piece of toast that you left in the toaster until it caught on fire, and then your dad threw it in the backyard and you had to go eat it. (Sorry, that was a surprisingly specific anecdote from a ten-year-old John.)

So, yes — the A.I. won in this case, but if every email you sent looked like this, I don’t think your customers would be thrilled about it.

Split Test Wrap-up

Okay, let’s take a step back to look at the big picture.

We ran 3 split tests, and they broke down like this:

• The blog post email had a very clear winner (the copywriter)
• The opt-in email had a less resounding winner (the A.I.)
• And the coupon delivery email was neck and neck

 

And from those tests, I learned that ChatGPT can write a pretty good email.

Here’s what else I learned:

ChatGPT doesn’t write scannable emails.

It has a decent understanding of some of the core principles of conversion copy. It can take in a lot of information, synthesize it, and spit it out in consumable chunks, which is really helpful.

But it doesn’t do very well with consumption in mind.

Its emails aren’t written so they can be easily scanned by someone on their phone, or riding a bus, or in their car at a stoplight, or just getting back from their lunch break.

These people are probably not going to read the ChatGPT email all the way through.

A.I. emails are a little dry (and boring).

This is definitely a knock against it.

It doesn’t do a great job of using voice, because voice is not necessarily just a way of writing, it’s also a set of values.

Its examples, context from somebody’s life that makes the writing funny in some cases, or serious in others.

In our future tests, we might try to play around more with voice in the prompts we give to ChatGPT.

I wouldn’t trust ChatGPT to craft an entire email campaign.

I don’t think ChatGPT has the tools yet to craft an entire email campaign.

That’s because powerful email marketing doesn’t come from one good email — it comes from a series of emails that work together to introduce a desire or a transformative state, get people interested and excited, and then to come up with the reason why they need to take action now.

I don’t think ChatGPT is sophisticated enough yet to craft a campaign that delivers that journey effectively.

That is, I think the secret sauce is what we as marketers can still bring to the table.

ChatGPT won’t replace marketers — for now.

You can (and probably should) use ChatGPT. I’m curious enough about it that I’ll keep running these tests.

I want to see if I can find an A.I. email that continually wins its use case, or one that will always win when a copywriter does it.

But I can comfortably say that smart, savvy business owners should NOT consider replacing their team members with ChatGPT.

Business is people. Business is relationships. Business is about human experience and connection.

If you lose sight of that, right, if you go full robot mode, you’re going to see people stop reading your content and disengage.

Not because any of it is technically bad, but because it lacks that human connection.

Have you ever had to watch somebody on stage reading from a transcript full of stats and facts? It’s a snooze-fest.

But how about when somebody gets up on stage with raw enthusiasm and they engage the crowd and talk to you and make eye contact and crack jokes? That’s a speaker that people remember.

Please don’t use ChatGPT to write blog posts.

This is something I didn’t learn from the split tests. It’s just something I’ve heard.

From what I understand, if you have ChatGPT write a blog post and then — without changing it at all — you plop it on you site, Google will be able to tell and they will penalize you harshly.

I don’t want you to try to get ChatGPT to write you 20 blog posts, only to have your organic traffic crunched by the long arm of Google.

So again, marketers, don’t panic. We’re not being replaced just yet.

And as for us email marketers, we now have a new tool at our disposal that can give us some ideas, and help as a jumping off point for some projects.

Then we can optimize for readability and really we can craft the narrative arc. I think that’s the core contribution.

It was an interesting experiment, and I’m excited to do more of them in the future. We’ll see if optimizing our prompts can lead to better results.

Until next time!