Web Page to Markdown & Links

• HTML to Markdown Conversion: Converts raw HTML content into a human-readable and AI-friendly Markdown format.
• Link Extraction: Identifies and extracts all URLs (hyperlinks) from the scraped web pages.
• API Rate Limiting: Implements delays and batching to respect Firecrawl.dev API rate limits (10 requests per minute) and server memory constraints (batches of 40 or less).
• Batch Processing: Processes multiple URLs efficiently in defined batches.
• Configurable Input: Allows users to define URLs from a data source (e.g., database) or an example array within the workflow.
• Structured Output: Organizes scraped data into title, description, content (Markdown), and links.

Setup:

1. Firecrawl.dev API Key: Obtain an API key from Firecrawl.dev.
2. Configure HTTP Request Node (Retrieve Page Markdown and Links):
   • Navigate to the Credentials section of the node.
   • Select your Firecrawl Bearer credential or create a new one.
   • Ensure the Authorization header is correctly set with your API key (e.g., Bearer YOUR_API_KEY). The workflow's sticky note Sticky Note34 provides guidance.
3. Define Input URLs:
   • Option A (Example): Edit the Example fields from data source node and update the Page array with your desired URLs.
   • Option B (Data Source): Connect your own data source to the Get urls from own data source node. Ensure the column containing URLs is named Page.
4. Configure Output: Connect the Markdown data and Links node to your desired data output destination (e.g., Airtable, database).

Workflow Explanation:

1. When clicking ‘Test workflow’ (Manual Trigger): Initiates the workflow.
2. Get urls from own data source (NoOp): Placeholder for fetching URLs from an external source.
3. Example fields from data source (Set Node): Defines an array of URLs to be processed (or receives URLs from the previous node).
4. Sticky Note33: Instructs on connecting data sources and naming the URL column Page.
5. Split out page URLs (SplitOut Node): Takes the Page field from the input and prepares it for individual processing.
6. 40 items at a time (Limit Node): Limits the number of items processed in a batch to 40, as indicated by Sticky Note36 for server memory management.
7. 10 at a time (SplitInBatches Node): Further divides the items into batches of 10 for more granular control and API request management.
8. Wait (Wait Node): Introduces a delay to respect API rate limits (10 requests per minute) as suggested by Sticky Note37.
9. Retrieve Page Markdown and Links (HTTP Request Node): Makes the POST request to the Firecrawl.dev API to scrape each URL, requesting markdown and links formats.
10. Markdown data and Links (Set Node): Structures the data received from Firecrawl.dev into title, description, content, and links fields.
11. Connect to your own data source (NoOp): Placeholder for sending the processed data to your output destination.
12. Sticky Note35: Reminds users to configure output to their data source.