How I automated meta description generation with Screaming Frog + OpenAI (saved my friend 11 hours of work)

Got a message from a friend the other day that made me wince - he'd just spent 11 hours manually writing meta descriptions for 139 pages. Been there, done that, and it's absolutely soul-crushing work. Figured I'd share my setup for automating this with Screaming Frog and OpenAI. It's not perfect, but it beats the hell out of doing it manually. **What you'll need:** * Screaming Frog license * OpenAI API key * Basic JavaScript knowledge (really basic, I promise) **Step 1: Enable JavaScript in Screaming Frog** 1. Go to Configuration > Spider > Rendering 2. Select "JavaScript" from dropdown 3. Click OK **Step 2: Prevent API Rate Limits** (trust me, you want to do this) * Go to Configuration > Speed * Enable "Limit URL/s" * Set Max URL/s to 0.5 or lower * Also uncheck all resource links in Configuration > Spider > Crawl (you don't need meta descriptions for CSS files!) **Step 3: Set Up Custom JavaScript** 1. Navigate to Configuration > Custom > Custom JavaScript 2. Click "+Add from Library" 3. Select "(ChatGPT) Template" **Step 4: Get Your OpenAI API Key** 1. Create an OpenAI platform account (this is different than a ChatGPT account) 2. Go to Settings > API keys 3. Create new secret key 4. Copy it into your Screaming Frog setup (for security reasons you can't access it again once you close the window) **The Secret Sauce: The Prompt** Here's the prompt I use that's been working pretty well: const question = "Identify the primary keyword from the following web page content and generate an SEO optimized meta description using that keyword in the style of a SEO expert. IMPORTANT: Do NOT start the description with the words explore, exmbark, experience, join, dive into, or discover.\n"+ "# Steps\n"+ "1. Analyze Content: Examine the provided content to determine the primary theme or subject matter.\n"+ "2. Identify Keyword: Select the primary keyword that best represents the main subject of the web page.\n"+ "3. Craft Meta Description: Write a meta description between 120-155 characters that incorporates the keyword, is engaging, and succinctly summarizes the page content using the language in the text.\n"+ "# Output Format\n"+ "**Primary keyword:** The keyword that best represents the main subject of the web page\n"+ "**Meta Description:** Meta description between 120-155 characters\n"+ "# Examples\n"+ "**Example 1:**\n"+ "- Primary Keyword: Keyword Research Tool\n"+ "- Meta Description: The most complete keyword research tool on the market. Generate thousands of keyword ideas, group them into topical clusters, and nail search intent.\n"+ "**Example 2:**\n"+ "- Primary Keyword: Authority checker\n"+ "- Meta Description: Free tool to check the authority of any website based on the quality and quantity of its external backlinks.\n"+ "**Example 3:**\n"+ "- Primary Keyword: Screaming Frog alternatives\n"+ "- Meta Description: Looking for a Screaming Frog alternative? MetaMonster could be the tool for you.\n"+ "**Example 4:**\n"+ "- Primary Keyword: SEO spider software\n"+ "- Meta Description: Clean up your metadata in minutes instead of months with MetaMonster, the SEO spider that fixes issues for you.\n"+ "**Example 5:**\n"+ "- Primary Keyword: Page title generator\n"+ "- Meta Description: Generate page titles for your website with AI. MetaMonster is the SEO spider software that fixes issues for you.\n"+ "**Example 6:**\n"+ "- Primary Keyword: Wordpress SEO plugin\n"+ "- Meta Description: Easily publish optimizations to your site with the MetaMonster Wordpress SEO Plugin.\n" It's broken up into separate lines because of the way SF's code editor handles long strings. *Bonus Tips to improve your output:* * Increase temperature to 1.0 for more variable responses * Add this code to extract just the meta description: ​ function extractMetaDescription(content) { const metaDescriptionRegex = /\*\*Meta Description:\*\*\s*(.+)/; const match = content.match(metaDescriptionRegex); return match ? match[1].trim() : null; } **Limitations You Should Know About:** * Each page is processed in isolation so the AI has limited context * SF's basic code editor makes it a pain to do more sophisticated stuff like prompt chaining * Make sure to save your changes as your own template or you'll lose everything when you close SF **Results** Running this on my friend's site would've taken about 30 minutes of setup and maybe 20 minutes of processing time instead of his 11-hour manual slog. The descriptions aren't perfect, but they're pretty solid and can be quickly edited if needed. *Disclaimer: I'm building a tool called MetaMonster that handles this stuff automatically without writing code (and with higher quality output), but this Screaming Frog setup is a solid free alternative if you already have SF.* I wrote all of this up in a blog post on our site that I'm happy to share, but didn't want to violate any self-promo rules. Hope this is helpful!