How to Scrape Web Data for AI Agents

Learn how to scrape web content and use it dynamically inside your AI workflows

This guide walks through the process of scraping webpage content in MindStudio and using that content in a custom AI agent. The example agent extracts article content from a URL and turns it into a LinkedIn post.

Use Case Overview

We’ll build a URL to LinkedIn Post agent that:

1. Collects a URL from the user.

2. Scrapes the content from that page.

3. Uses AI to generate a LinkedIn post based on the page content.

Step 1: Create a User Input for the URL

1. Add a User Input block to your workflow.

2. Choose the Short Text input type.

3. Name the variable: url

4. Set the label: Enter the URL you'd like to write a LinkedIn post about

5. Add placeholder text: e.g., https://www.theverge.com/...

6. Enable URL validation to ensure the input is a proper URL.

7. (Optional) Set a test value for debugging, like a real article URL.

Step 2: Scrape the Webpage

1. Add a Scrape URL block.

2. In the URL field, use the variable: {{ url }}

3. Set the output variable name: scraped_content

4. Choose Output Format: Text only

5. Enable Auto-enhance to improve scraping reliability.

6. Keep the Default scraper selected (Firecrawl is also available if needed).

7. Leave Screenshot disabled unless required.

The block will now extract and store webpage content into the scraped_content variable.

Step 3: Generate AI Output

1. Add a Generate Text block.

2. Write your prompt, including the scraped content:

   cssCopyEditWrite an attention-grabbing LinkedIn post based on the following article:
   <content>{{ scraped_content }}</content>

3. Choose an appropriate model (e.g., Claude 3.5 Haiku).

Step 4: Test the Agent

1. Click Preview and open the draft agent.

2. Try inputting an invalid value (like not a URL) to confirm validation works.

3. Enter a valid URL or use the test value.

4. The AI will:

   • Scrape the page.

   • Analyze the content.

   • Generate a LinkedIn post for you to copy or repurpose.

Recap and Best Practices

• Use the Scrape URL block to pull live content from any webpage.

• Always validate user input when collecting URLs.

• Store scraped data in a clearly named variable for easy reuse.

• Keep the output format as “Text only” for general analysis or “JSON” for structured use cases.

• Auto-enhance improves scraping accuracy on dynamic or complex websites.

You can further extend this workflow by adding post-processing steps or integration blocks to share or save the generated content.