If you are tasked with creating a POV around the risk/upside to SEO, content marketing, or paid search in a ChatGPT world, then this post is for you.
This is not a post about:
This post is focused on economic impacts, ROI impacts, and risks to your company
I wanted to give you approaches and ways of thinking/tracking/sleuthing to aid in determining the potential impact rooted in existing data you already have as a search professional, but probably aren’t using to answer the questions you are suddenly getting.
I’m going to try to take that approach to ChatGPT – specifically through the lens of how it impacts the ROI of SEO and investments in content marketing.
The questions I am trying to answer for myself and my clients are:
Should I be worried that the value of my existing organic search rankings will be diminished if competitors start generating content using AI?
Yes. You should have been worried about the downtrend in ROI of SEO well before ChatGPT, as it is well documented that Google has placed more ads, more answers, and more rich snippets around organic rankings, resulting in “zero-click” searches. I cannot imagine a world where all these distractions that show up around the SERP result in more clicks on organic listings.
Now, let’s talk about ChatGPT or the use of AI-assisted content. The value of using SEO as a way to rank high on Google to capture traffic will likely be diminished for many of the sites where people don’t need to search anymore and answers are surfaced through ChatGPT. The question is: to what degree?
The disclosure helps you to find them at scale:
There are a lot of ways to try to catch AI-generated text. (Here is a great roundup from Chris Cemper.)
However you find URLs, find them, and begin to monitor the SERP for a bespoke analysis vs. a generic one. (Personally, I think it is worth the work).
Let’s step our game up a bit and scale this – because running “site colon” searches is time-consuming.
When we talk about risk to the value of SEO investments – we must take the AI-generated content we identified above & compare those pages to our existing rankings to see:
Let’s use SEMrush’s tools to dissect how to determine AI “assisted” content performance.
To me, when talking to execs about risk, dollars, and cents make the most sense.
Typically an SEO working with siloed SEO data might see these AI-assisted content pieces and think, “Oh my god! The AI content built by our competitors is outranking us for this list of keywords with X search volume.” I’ll show you below how joining paid data could help you avoid that mistake in your POVs.
My preference is to join the PPC and SEO data together to determine risk. Why? PPC data gives me these 2 things to value content that search volume does not:
Despite AI-generated content outranking us, maybe these keywords don’t drive conversions. This could change the way I position the strategy.
That approach doesn’t scale, so lets scale it:
That approach is decent, but across several clients and 1000’s of URLs, I will miss as many as I catch. I want to catch all of them at scale, while also alerting my team and providing a consistent level of service to clients.
That’s where Supernova, our data warehousing, and insights platform, comes into play, I was able to go from hype to hypothesis to client impact in under 2 hours, you could do this easily in Power Bi across clients too, but I’m showing what we did in Looker.
Wanna get real fancy? Consider these options in your risk assessment:
Maybe queries best answered by AI don’t get clicks past position 2 or 3 anyway.
The other side of the financial coin, is what if this upends search altogether and people stop using search and start using ChatGPT? First, keep this in mind from JPMorgan’s analyst, we don’t know the cost of ChatGPT yet, and it is more expensive than Google’s infrastructure to answer 1 query:
Ok, with that said, how would I make a case to help an exec understand the long-term risk to the value of ranking?
Above, we just showed you how to manually look at a SERP looking for clues. But there are other ways to do this at scale.
Wanna protect your job in content or SEO? Start having more hypotheses and finding ways to validate or invalidate them. Let me give you 2 (simple and bad) examples of hypotheses, I played around with that could impact the ROI of paid and organic search.
I believe People Also Ask in the Top 2 positions on Google is a flag, or an answer box is an indicator of content that is easily disrupted by AI content – I can tell my clients what % of their ranking search terms trigger those “warning signals.” Then, combining it with paid data can help me to advise them on how many of their conversions could be at risk, and if conversions or leads are at risk, revenue goals may be missed and investments/strategies may need to shift.
Hypothesis 1: Let’s say I believed that PAA’s in Position 2 meant that those queries were somewhat more disruptable than instances where PAA is lower or not on the page at all. Now, I can see the percentage of client spend/conversions that matches my hypothesis around PAAs.
I’m now able to go from client to client instantly in Supernova.
For one client asking my opinion about AI assisted content & risks I might say:
“Hey, if you believe that my hypothesis is a good proxy for query disruption, you are in the top 5% of ALL my clients for whom conversions are likely to be impacted. This means we should have a strategy session in the next 3 months.”
For the other asking my opinion about AI assisted content & risks I might say:
“Hey, 66% of your keywords are affected by this hypothesis, but it only represents 17% of your conversions. This puts you in the bottom 10% of my clients. As such, I will monitor this every month and if the percentage of conversions impacted exceeds 25%, then we should set up a strategy session in the next 3-6 months.”
Using the same data set, updated monthly allows me to have very different conversations. This is for another day, but this is one of THE value of an agency, the ability to have access to data quickly across 100+ clients enables me to see a range of who should have a conversation today due to near term risk and who doesn’t need to worry just yet.
The other thing I love about a platform like Supernova is if one team member has a killer hypothesis, we can shift priorities and build the dash for every client at scale without our team members having to learn how to build the dash. (Because they’ve got too many other things to do!)
Hypothesis 2: Let’s say I believed that Answer Boxes ranking in a certain position were the key to uncovering easily disruptable rankings. I could do the following:
In the dashboard above, I am able to articulate to two very different strategies to two clients both hovering at a 20% of queries that are “disruptable.” Client 1 has a 20% rate of disruptable queries but only 1% of conversions. They are at less of a threat than Client 2, who also has 20% of disruptable queries – but at 6% of their conversions.
|Different Strategies, same percent of “Disruptable” queries|
|% AI Disruptable Queries||% of Conversions|
Note: I gotta learn how to make quadrant charts in Looker, cause I would love to have 4 quadrants.
I like to look for similarities in the SERP. Find a page or two or three and make a prediction. Do you think it will last, do you think it is low quality, etc.? There we are again, have an opinion or a hypothesis.
If you are looking for something a bit more advanced:
Let me know if you would like me to record a video on how to do this in SEMrush + PBI.
I would take the keyword rankings of these pages (pulled from Ahrefs or SEMrush) and Theme them using N-grams, here is a great tool. Use this information to see what percentage of them have words like:
As you can see in the Ahrefs screengrab below, the AI-generated content performed best on words like those above for this 1 piece of content. You’d then have to go through the rest looking for similar patterns.
Additionally, you can always dump all the keywords into a tool to N-Gram them looking for commonalities.
Ok, so whew, that is all I have for now! Let’s get after it!
How are you discussing ROI, risk, impact?
My next post (when I can come up for air) will likely be about economic models of human content vs AI content and the considerations you should have going into making that decision. If you only need to make (or have budget to make) 20 pieces of content this year, then maybe all this AI stuff is overblown. (As long as you aren’t making 20 pieces of content easily answered by AI). However, if you see the need to make 100 or 200, then maybe you do need more assistance to scale the work of the team…But I’m getting ahead of myself there! See you in the next post!
Source: www.seerinteractive.com, originally published on 2023-01-18 16:46:47