Generate AI-Ready llms.txt Files from Screaming Frog Website Crawls
detail.loadingPreview
Convert Screaming Frog website crawl data into an LLM-ready format for content discovery.
🚀Ready to Deploy This Workflow?
About This Workflow
Overview
This workflow takes a Screaming Frog internal CSV export (preferably internal_html.csv or internal_all.csv) and transforms it into a structured llms.txt file. This file is optimized for use with Large Language Models (LLMs) by extracting key information like URL, title, description, word count, and indexability status for relevant web pages.
Key Features
- Extracts essential data fields from Screaming Frog exports (URL, Title, Meta Description, Status Code, Indexability, Content Type, Word Count).
- Supports multi-language column names for Screaming Frog exports (English, French, Italian, German, Spanish).
- Filters for indexable, HTML content with a 200 status code, ensuring high-quality data.
- Allows for further customization of filtering based on word count, URL path, or meta description presence.
- Generates a
llms.txtfile that is ready for consumption by LLMs for content discovery and analysis.
How To Use
- Upload Screaming Frog CSV: Use the initial form node to upload your
internal_html.csvorinternal_all.csvfile from Screaming Frog. - Configure Workflow: Ensure the 'Set useful fields' node correctly maps your Screaming Frog column names (the workflow includes common multi-language variations).
- Refine Filters (Optional): Adjust the 'Filter URLs' node to add or modify conditions for including specific pages (e.g., by word count, URL path, or meta description).
- Review Output: The workflow will output the processed data, which can then be further formatted into the
llms.txtstructure or used directly.
Apps Used
Workflow JSON
{
"id": "bdb4be64-4127-42e9-a602-cddf9d2588e9",
"name": "Generate AI-Ready llms.txt Files from Screaming Frog Website Crawls",
"nodes": 0,
"category": "SEO & Content Analysis",
"status": "active",
"version": "1.0.0"
}Note: This is a sample preview. The full workflow JSON contains node configurations, credentials placeholders, and execution logic.
Get This Workflow
ID: bdb4be64-4127...
About the Author
N8N_Community_Pick
Curator
Hand-picked high quality workflows from the global community.
Statistics
Verification Info
Related Workflows
Discover more workflows you might like
Automate Competitor Tracking with Crunchbase & ClickUp
Streamline competitor analysis by automatically fetching data from Crunchbase and creating review tasks in ClickUp. Stay informed about market changes without manual effort.
Telegram Profanity & Toxicity Filter
This n8n workflow automatically monitors incoming Telegram messages for profanity and toxic language. It leverages Google's Perspective API to analyze message content, and if a message is deemed inappropriate, the workflow sends an automated warning response back to the sender.
Automate Shipment Tracking with Telegram and n8n
Streamline your logistics operations by automating shipment tracking with Telegram and n8n. This workflow allows drivers to easily share updates, and ensures timely notifications to your team.
Universal CSV to JSON API Converter
Effortlessly transform CSV data into structured JSON with this versatile n8n workflow. Integrate it into any application as a custom API endpoint, supporting various input methods including file uploads and raw text.
Automated Multi-Platform Social Media Publisher
Streamline your social media content creation and publishing with this n8n workflow. Simply fill out a web form with your caption, media (image or video), and target platforms, and let n8n automate the posting process across multiple social networks.
Convert Any Webpage to PDF and Save Locally
This n8n workflow provides a simple way to convert any public webpage URL into a PDF document. Once converted, the PDF file is automatically saved to your local disk, making it easy to archive or share web content offline.