Generate AI-Ready llms.txt Files from Screaming Frog Website Crawls
detail.loadingPreview
Convert Screaming Frog website crawl data into an LLM-ready format for content discovery.
🚀Ready to Deploy This Workflow?
About This Workflow
Overview
This workflow takes a Screaming Frog internal CSV export (preferably internal_html.csv or internal_all.csv) and transforms it into a structured llms.txt file. This file is optimized for use with Large Language Models (LLMs) by extracting key information like URL, title, description, word count, and indexability status for relevant web pages.
Key Features
- Extracts essential data fields from Screaming Frog exports (URL, Title, Meta Description, Status Code, Indexability, Content Type, Word Count).
- Supports multi-language column names for Screaming Frog exports (English, French, Italian, German, Spanish).
- Filters for indexable, HTML content with a 200 status code, ensuring high-quality data.
- Allows for further customization of filtering based on word count, URL path, or meta description presence.
- Generates a
llms.txtfile that is ready for consumption by LLMs for content discovery and analysis.
How To Use
- Upload Screaming Frog CSV: Use the initial form node to upload your
internal_html.csvorinternal_all.csvfile from Screaming Frog. - Configure Workflow: Ensure the 'Set useful fields' node correctly maps your Screaming Frog column names (the workflow includes common multi-language variations).
- Refine Filters (Optional): Adjust the 'Filter URLs' node to add or modify conditions for including specific pages (e.g., by word count, URL path, or meta description).
- Review Output: The workflow will output the processed data, which can then be further formatted into the
llms.txtstructure or used directly.
Apps Used
Workflow JSON
{
"id": "bdb4be64-4127-42e9-a602-cddf9d2588e9",
"name": "Generate AI-Ready llms.txt Files from Screaming Frog Website Crawls",
"nodes": 0,
"category": "SEO & Content Analysis",
"status": "active",
"version": "1.0.0"
}Note: This is a sample preview. The full workflow JSON contains node configurations, credentials placeholders, and execution logic.
Get This Workflow
ID: bdb4be64-4127...
About the Author
N8N_Community_Pick
Curator
Hand-picked high quality workflows from the global community.
Statistics
Verification Info
Related Workflows
Discover more workflows you might like
Telegram Profanity & Toxicity Filter
This n8n workflow automatically monitors incoming Telegram messages for profanity and toxic language. It leverages Google's Perspective API to analyze message content, and if a message is deemed inappropriate, the workflow sends an automated warning response back to the sender.
Robust Concurrency Control for n8n Workflows with Redis
Prevent simultaneous execution of critical n8n workflows or tasks using a centralized, Redis-backed locking mechanism. This reusable utility workflow ensures data integrity and resource management by allowing other workflows to acquire, check, and release locks.
Universal CSV to JSON API Converter
Effortlessly transform CSV data into structured JSON with this versatile n8n workflow. Integrate it into any application as a custom API endpoint, supporting various input methods including file uploads and raw text.
Automated Multi-Platform Social Media Publisher
Streamline your social media content creation and publishing with this n8n workflow. Simply fill out a web form with your caption, media (image or video), and target platforms, and let n8n automate the posting process across multiple social networks.
Convert Any Webpage to PDF and Save Locally
This n8n workflow provides a simple way to convert any public webpage URL into a PDF document. Once converted, the PDF file is automatically saved to your local disk, making it easy to archive or share web content offline.
Automate DOCX to PDF Conversion and Save Locally
This workflow automates the process of converting a Microsoft Word DOCX file to a PDF document. It downloads a source DOCX file, uses ConvertAPI to perform the conversion, and then saves the resulting PDF file directly to your local disk.