VIDEO | How to Build a Web Scraping Workflow Using LLMs

In today's data-driven landscape, extracting key information from various websites is critical for automating research, monitoring trends, and structuring raw content into actionable insights. However, manually scraping and analyzing web content can be time-consuming and inefficient.

This is where Eden AI's no-code workflow platform comes into play—empowering developers to easily build automated pipelines for web scraping and information extraction using Large Language Models (LLMs).

‍

Takeaways

In this tutorial, you'll learn how to set up a fully automated workflow on the Eden AI platform to:

• Scrape content from multiple websites;
• Automatically extract key information using LLMs like Claude 3.5 or GPT-4o;
• Handle fallback URLs in case of failures;
• Use API endpoints to integrate the workflow into your own applications.

By the end of this guide, you'll have a complete end-to-end solution for web scraping and information extraction—all without writing a single line of code.

‍

Concepts

Before diving into the implementation, let's clarify some key concepts:

Web Scraping: The process of automatically extracting data from websites.

LLMs (Large Language Models): AI models capable of understanding and generating human-like text, used here for analyzing and extracting key information from scraped data.

Eden AI Platform: A no-code platform that simplifies the creation and deployment of AI workflows, including web scraping and LLM integration.

‍

Prerequisites

Before proceeding, ensure you have the following:

• An Eden AI account (Sign up at edenai.co);
• Basic understanding of APIs and JSON payloads;
• Familiarity with FastAPI for backend API development (optional);
• Installed Python 3.8+ environment (if using the code implementation);

‍

Solution Overview

The workflow we will create consists of the following steps:

1. Accept URLs and questions as input.
2. Scrape the website content from the primary URL.
3. Use conditional logic to fall back to a secondary URL if scraping fails.
4. Process the scraped data through LLMs for information extraction.
5. Return the summarized output.
6. Expose the entire pipeline via API endpoints for seamless integration.

‍

Step-by-Step Guide

‍

Make sure to watch our in-depth tutorial on web scraping, where we provide a step-by-step visual guide to help you understand the process more thoroughly:

‍

Step 1: Setting Up the Eden AI Workflow

1. Navigate to the Workflows section in your Eden AI dashboard.

2. Click Create Workflow and select Basic Workflow.

3. Choose Start from Scratch.

4. Name your workflow (e.g., LLM Scraping Pipeline).

‍

Input Nodes Configuration

• Add three input fields:
  • URL 1: Primary website URL
  • URL 2: Backup website URL
  • Question: The query to extract information from the scraped content.

‍

Step 2: Web Scraping Node

1. Add a Web Scraper node.
2. Configure the scraper to accept URL inputs.
3. Save the configuration.

Conditional Node (Fallback URL)

1. Add an IF node to check if the scraped content exists.
2. If true, proceed to the LLM node.
3. If false, route the process to scrape the Backup URL using URL 2.

‍

Step 3: Information Extraction with LLM

1. Add the LLM Chat node.
2. Configure the model to Claude 3.5 Sonnet (Primary) and GPT-4o (Fallback).
3. Set the temperature to 0.4 for optimal balance between creativity and accuracy.
4. Pass the scraped content as context.
5. Use the Chatbot Global Action field with the instruction: "You are a news agent. Your task is to summarize the answer for the question asked".

6. Insert the question input as part of the prompt.

‍

Step 4: Testing the Workflow

1. Navigate to Live Testing on the Eden AI platform.
2. Enter:
   • URL 1 (e.g., News article about Elon Musk)
   • URL 2 (Backup article URL)
   • Question (e.g., What did Elon Musk say about Trump's winning?)
3. Click Test Workflow.
4. The platform will scrape the website, extract relevant content, and generate a summarized answer.

‍

Code Implementation

‍

If you prefer backend integration, you can use the following Fast API-based service to connect with your Eden AI’s workflow API:

FastAPI Backend Code

from fastapi import FastAPI, File, UploadFile, Form
import requests
import os

EDEN_API_URL = "https://api.edenai.run/v2/workflow/YOUR_WORKFLOW_ID/execution/"
API_KEY = os.getenv("EDEN_AI_API_KEY")

app = FastAPI()

@app.post("/scrape-and-extract/")
async def scrape_and_extract(url1: str = Form(...), url2: str = Form(...), question: str = Form(...)):
    headers = {"Authorization": f"Bearer {API_KEY}"}
    payload = {"url1": url1, "url2": url2, "question": question}

    response = requests.post(EDEN_API_URL, headers=headers, json=payload)
    return response.json()

‍

Final Output

The workflow will return the extracted information in JSON format:

{
  "status": "succeeded",
  "answer": "Elon Musk expressed neutrality about Trump's victory, emphasizing free speech."
}

‍

Advanced Use Cases

• Monitor breaking news articles for sentiment analysis;
• Automate financial report summarization;
• Extract product reviews for e-commerce websites.

‍

Conclusion

By leveraging Eden AI's no-code workflow platform, developers can quickly build and deploy automated content extraction pipelines without diving into complex code. This solution not only saves time but also ensures high-quality, consistent data extraction.

Don't forget to watch our tutorial on web scraping for a more detailed and visual explanation. you're ready to streamline your research and content analysis tasks, start building your first workflow on Eden AI today.

‍

Additional Resources

• Eden AI Documentation
• FastAPI Documentation
• Claude 3.5
• OpenAI GPT-4

‍