Benchmark and Optimize Local LLMs with LM Studio
detail.loadingPreview
This n8n workflow provides a robust solution for testing and comparing multiple local LLM models hosted via LM Studio. It automates prompt delivery, captures crucial performance metrics like response time, and conducts detailed text analysis for readability, empowering you to select the best model for your specific needs.
About This Workflow
This n8n workflow offers a comprehensive framework for evaluating the performance and output quality of various Large Language Models (LLMs) running locally through LM Studio. It automates the process of querying different models, capturing critical metrics such as response time, and conducting in-depth text analysis for readability, word count, and sentence structure. Ideal for developers and researchers, this template streamlines the iterative testing of LLMs, allowing for quick comparisons and informed decision-making based on quantifiable data. Tailor system prompts and model parameters to achieve specific testing criteria and enhance your local LLM development cycle.
Key Features
- Multi-LLM Testing: Query and compare responses from multiple local LLM models hosted in LM Studio simultaneously.
- Performance Benchmarking: Automatically measure and log LLM response times to assess and compare model efficiency.
- Advanced Text Analysis: Evaluate model outputs with sophisticated metrics including Flesch-Kincaid readability scores, word count, and average sentence length.
- Configurable LLM Parameters: Easily adjust critical parameters like Temperature, Top P, and Presence Penalty to fine-tune and observe model behavior during testing.
- Optional Google Sheets Integration: Log all testing results, including prompts, responses, and analytical metrics, into a structured Google Sheet for easy review and reporting.
How To Use
- Setup LM Studio: First, download and install LM Studio. Then, identify and load the LLM models you intend to use for testing into LM Studio's server capabilities.
- Update Base URL: In the 'Get Models' HTTP Request node, update the
http://192.168.1.1:1234/v1/modelsURL to match the local IP address of your running LM Studio server. - Configure LLM Settings: Adjust parameters such as 'Temperature', 'Top P', and 'Presence Penalty' within the 'Run Model with Dynamic Inputs' node to fine-tune the LLM behavior for your specific tests.
- Define System Prompt: In the 'Run Model with Dynamic Inputs' node, consider adding a 'System Prompt' to guide the LLM's responses, focusing on specific criteria like conciseness or a target reading level.
- Optional Google Sheet Setup: If you wish to track results, create a Google Sheet with the following headers: 'Prompt', 'Time Sent', 'Time Received', 'Total Time Spent', 'Model', 'Response', 'Readability Score', 'Average Word Length', 'Word Count', 'Sentence Count', 'Average Sentence Length'. Then, update the corresponding Google Sheets node in the workflow to map these fields accordingly.
Apps Used
Workflow JSON
{
"id": "3fec37ff-8f05-4a45-9596-174c8495ce33",
"name": "Benchmark and Optimize Local LLMs with LM Studio",
"nodes": 7,
"category": "Operations",
"status": "active",
"version": "1.0.0"
}Note: This is a sample preview. The full workflow JSON contains node configurations, credentials placeholders, and execution logic.
Get This Workflow
ID: 3fec37ff-8f05...
About the Author
Free n8n Workflows Official
System Admin
The official repository for verified enterprise-grade workflows.
Statistics
Related Workflows
Discover more workflows you might like
Instant WooCommerce Order Notifications via Telegram
When a new order is placed on your WooCommerce store, instantly receive detailed notifications directly to your Telegram chat. Stay on top of your e-commerce operations with real-time alerts, including order specifics and a direct link to view the order.
On-Demand Microsoft SQL Query Execution
This workflow allows you to manually trigger and execute any SQL query against your Microsoft SQL Server database. Perfect for ad-hoc data lookups, administrative tasks, or quick tests, giving you direct control over your database operations.
Automate Getty Images Editorial Search & CMS Integration
This n8n workflow automates searching for editorial images on Getty Images, extracts key details and embed codes, and prepares them for seamless integration into your Content Management System (CMS), streamlining your content creation process.