Competitor Price Monitoring with AI Scraping and Analysis

• Dual Trigger System: Supports both scheduled daily checks and on-demand manual triggers via webhooks.
• Multi-Platform Scraping: Integrates with Amazon, Best Buy, and Target using HTTP requests with custom user agents.
• AI-Powered Data Extraction: Leverages ScrapeGraphAI to intelligently extract structured product data, including prices, ratings, availability, and features.
• Advanced Price Analysis: Custom code node analyzes pricing data to detect significant drops, increases, discounts, and calculates value scores.
• Competitive Intelligence: Generates insights on market position, brand strength, and customer satisfaction.
• Automated Data Logging: Stores comprehensive pricing data and analysis results in Google Sheets.
• Smart Alerting: Sends real-time alerts to Slack for price drops, high discounts, and other critical changes.
• Configurable Tracking: Prioritizes alerts based on user-defined thresholds and product tracking lists.
• Extensible Design: Easily adaptable to include more retailers or refine analysis parameters.

Workflow Setup and Usage Guide

This workflow is designed to automate competitor price monitoring. Follow these steps to set it up and run it:

1. Triggers (daily-price-trigger, manual-price-trigger):

   • The daily-price-trigger is configured to run every 24 hours. You can adjust the hoursInterval in the schedule settings if needed.
   • The manual-price-trigger (webhook) can be used to initiate a price check on demand. You'll need to expose this webhook URL if you intend to use it.

2. Scraping (amazon-scraper, bestbuy-scraper, target-scraper):

   • These nodes are set up to fetch raw HTML content from product search pages of Amazon, Best Buy, and Target.
   • Each scraper includes a User-Agent header to mimic a web browser, which can help bypass basic anti-scraping measures. Ensure the URLs are accurate for the products or categories you wish to track.

3. AI Data Extraction (scrapegraph-ai-prices):

   • This node uses ScrapeGraphAI to parse the scraped HTML and extract structured data based on the provided userPrompt and websiteUrl.
   • The userPrompt defines the schema for the extracted data. Ensure this schema matches what you expect from the scraping process and what the price-analysis-code node requires.
   • The websiteUrl is dynamically set using ={{ $json.url || 'https://www.amazon.com/s?k=wireless+headphones' }} which attempts to use the URL from the previous node if available, otherwise defaults to a specific Amazon search.
   • You will need to configure your ScrapeGraphAI API credentials in the credentials section of this node.

4. Price Analysis (price-analysis-code):

   • This JavaScript code node performs the core analysis on the data extracted by ScrapeGraphAI.
   • It processes data from all scraping sources, identifies target brands, and analyzes pricing insights like discounts, value scores, and price changes.
   • It generates alert_worthy flags, change_significance, and other metadata used for downstream actions.
   • Key configurations like trackingConfig.targetBrands and trackingConfig.priceThresholds can be adjusted within the jsCode to fine-tune the analysis.

5. Price Alert Filtering (price-alert-filter):

   • This IF node acts as a gatekeeper for alerts. It checks if alert_worthy is true, if the discount_percentage is above 20, or if change_significance is 'high'.
   • If any of these conditions are met (due to the any combine operation), the data passes through to the alert node.

6. Slack Alerting (slack-price-alert):

   • If the price-alert-filter allows the data to pass, this node constructs a formatted message and sends it to a specified Slack channel.
   • The message includes product details, pricing information, discounts, source, ratings, and competitive intelligence.
   • You need to configure your Slack API credentials and select the appropriate channelId in the node's parameters.

7. Google Sheets Storage (google-sheets-storage):

   • This node appends or updates the analyzed product data into a Google Sheet.
   • It uses autoMapInputData and matchingColumns to update existing records based on product_id or add new ones.
   • Ensure you have set up your Google Sheets credentials (service account is recommended) and provided the correct documentId and sheetName.

General Workflow Management:

• Activation: The workflow is currently set to active: false. You will need to toggle this to true in the workflow settings to enable execution.
• Credentials: Remember to configure all necessary credentials (ScrapeGraphAI, Google Sheets, Slack) within the n8n interface for the nodes that require them.
• Testing: It's recommended to run the workflow manually or trigger it via the webhook for initial testing to verify each node's output and connections.