Andreas Ludvigsson

Bridging Code and Commerce

AI-Generated Custom Web Scrapers: Building a Vue.js and Flask Application with Claude AI

In today’s data-driven world, web scraping has become an essential tool for gathering information from websites. However, creating custom web scrapers can be time-consuming and requires programming expertise. This guide introduces an innovative solution: an application that uses AI to generate custom Python web scrapers based on user input.

Our application combines the power of Vue.js for the frontend, Flask for the backend, and Claude AI for generating Python scraping code. Users can simply input a URL and specify the fields they want to extract, and the AI will create a custom web scraper tailored to their needs.

Key features of this project include:

  • A user-friendly interface built with Vue.js and styled with Tailwind CSS
  • A Flask backend that handles API requests and integrates with Claude AI
  • AI-powered generation of Python web scraping code using BeautifulSoup and requests
  • Dynamic creation of scrapers based on user-specified fields

Whether you’re a beginner looking to learn about web scraping or an experienced developer seeking to streamline your scraping workflow, this guide will walk you through the process of building and using this powerful tool.

2. Setting Up the Development Environment

Before we dive into the code, let’s set up our development environment. This project uses Vue.js for the frontend and Flask for the backend, so we’ll need to set up both.

Frontend Setup

  1. Install Node.js and npm if you haven’t already.
  2. Create a new Vue.js project: npm init vue@latest
  3. Navigate to your project directory and install dependencies: cd your-project-name npm install
  4. Install additional dependencies: npm install axios
  5. Install Tailwind: npm install -D tailwindcss@latest postcss@latest autoprefixer@latest

Backend Setup

  1. Create a new directory for your Flask backend:
    mkdir flask-backend
    cd flask-backend
  2. Create a virtual environment and activate it:
    python -m venv
    venv source venv/bin/activate
    # On Windows, use `venv\Scripts\activate`
  3. Install the required Python packages:
    pip install flask flask-cors requests beautifulsoup4 anthropic
  4. Create a new file named app.py in the flask-backend directory. This will be our main Flask application file.

With these steps completed, you’ve set up the basic structure for both the frontend and backend of our application. In the next sections, we’ll start building out the functionality of our AI-powered web scraper generator.

3. Frontend Development with Vue.js

Now that our environment is set up, let’s build the frontend of our application using Vue.js. We’ll create a form that allows users to input a URL and specify the fields they want to scrape.

Creating the ScrapeForm Component

  1. In your Vue.js project, create a new file src/components/ScrapeForm.vue:
<template>
  <div class="container mx-auto p-4">
    <h1 class="text-2xl font-bold mb-4">Web Scraper Creator</h1>
    <form @submit.prevent="submitScraper" class="space-y-4">
      <div>
        <label for="url" class="block text-sm font-medium text-gray-700">URL to Scrape</label>
        <input
            v-model="url"
            type="url"
            id="url"
            required
            class="mt-1 block w-full rounded-md border-gray-300 shadow-sm focus:border-indigo-300 focus:ring focus:ring-indigo-200 focus:ring-opacity-50"
        />
      </div>
      <div>
        <h2 class="text-lg font-semibold mb-2">Fields to Scrape</h2>
        <div v-for="(field, index) in fields" :key="index" class="flex space-x-2 mb-2">
          <input
              v-model="field.name"
              type="text"
              placeholder="Field name (e.g., price, title)"
              class="flex-1 rounded-md border-gray-300 shadow-sm focus:border-indigo-300 focus:ring focus:ring-indigo-200 focus:ring-opacity-50"
          />
          <button @click="removeField(index)" type="button" class="px-2 py-1 bg-red-500 text-white rounded-md">
            Remove
          </button>
        </div>
        <button @click="addField" type="button" class="mt-2 px-4 py-2 bg-green-500 text-white rounded-md">
          Add Field
        </button>
      </div>
      <button type="submit" class="w-full px-4 py-2 bg-blue-500 text-white rounded-md hover:bg-blue-600">
        Create Scraper
      </button>
    </form>
    <div v-if="response" class="mt-8">
      <h2 class="text-lg font-semibold mb-2">Python Code for Scraper:</h2>
      <pre class="bg-gray-100 p-4 rounded-md overflow-x-auto">
        <code>{{ response }}</code>
      </pre>
    </div>
  </div>
</template>

<script setup>
import { ref } from 'vue'
import axios from 'axios'

const url = ref('')
const fields = ref([{name: ''}])
const response = ref('')

const addField = () => {
  fields.value.push({name: ''})
}

const removeField = (index) => {
  fields.value.splice(index, 1)
}

const submitScraper = async () => {
  try {
    const result = await axios.post('http://127.0.0.1:5000/api/create-scraper', {
      url: url.value,
      fields: fields.value.map(field => field.name)
    })
    const data = JSON.parse(result.data)
    response.value = data.python_code
  } catch (error) {
    console.error('Error creating scraper:', error)
    response.value = 'Error: Unable to create scraper'
  }
}
</script>

This component creates a form with input fields for the URL and the fields to scrape. It also includes buttons to add or remove fields dynamically.

Updating the Main App Component

  1. Update your src/App.vue file to use the ScrapeForm component:
<script setup>
import ScrapeForm from "@/components/ScrapeForm.vue";
</script>

<template>
  <main>
    <ScrapeForm/>
  </main>
</template>

Styling with Tailwind CSS

The component already includes Tailwind CSS classes for styling. Make sure your tailwind.config.js file is set up correctly:

/** @type {import('tailwindcss').Config} */
export default {
  content: [
    "./index.html",
    "./src/**/*.{vue,js,ts,jsx,tsx}",
  ],
  theme: {
    extend: {},
  },
  plugins: [],
}

With these steps, you’ve created the frontend of your application. Users can now input a URL and specify fields to scrape. In the next section, we’ll build the Flask backend to handle these requests and generate the Python scraping code using Claude AI.

4. Backend Development with Flask

Now that we have our frontend set up, let’s create the backend server using Flask. This server will handle requests from the frontend and interact with Claude AI to generate the Python scraping code.

Setting up the Flask Server

  1. In your flask-backend directory, open the app.py file and add the following code:
from flask import Flask, request, jsonify
from flask_cors import CORS
import requests
from bs4 import BeautifulSoup
import anthropic
import os

app = Flask(__name__)
CORS(app)  # This enables CORS for all routes

# Initialize the Anthropic client
client = anthropic.Anthropic(
    api_key=os.environ.get('ANTHROPIC_API_KEY')
)

@app.route('/api/create-scraper', methods=['POST'])
def create_scraper():
    data = request.json
    url = data['url']
    fields = data['fields']

    # We'll implement this function later
    relevant_content = parse_relevant_content(url)

    # We'll implement the AI integration here

    return jsonify({"status": "success", "python_code": "# Placeholder for generated code"})

if __name__ == '__main__':
    app.run(debug=True)

This sets up a basic Flask server with CORS enabled and a route for creating scrapers.

Implementing CORS

We’ve already implemented CORS by adding CORS(app) in the Flask setup. This allows our frontend to make requests to the backend without running into cross-origin issues.

Creating the API Endpoint for Scraper Generation

We’ve set up the /api/create-scraper endpoint, which will receive POST requests from our frontend. In the next section, we’ll implement the AI integration to generate the scraper code.

5. Integrating Claude AI for Code Generation

Now, let’s integrate Claude AI to generate our Python scraping code.

Setting up the Anthropic Client

We’ve already initialized the Anthropic client in our Flask app. Make sure to set your Anthropic API key as an environment variable:

export ANTHROPIC_API_KEY='your-api-key-here'

Crafting the Prompt for AI-Powered Code Generation

Let’s update our create_scraper function to use Claude AI:

@app.route('/api/create-scraper', methods=['POST'])
def create_scraper():
    data = request.json
    url = data['url']
    fields = data['fields']

    relevant_content = parse_relevant_content(url)

    prompt = f"""You are a Python web scraping assistant. Analyze the following webpage content and generate a Python script for scraping it. The script should use requests and BeautifulSoup to extract the specified fields. Handle errors gracefully and return "Not found" if a field cannot be extracted.

    URL: {url}
    Fields: {', '.join(fields)}

    Here is a sample of the relevant content from the webpage:
    {relevant_content}

    Please generate the Python code and return it in the following JSON format:
    {{
        "status": "success" or "failure",
        "python_code": "<generated_python_code>"
    }}
    Only provide the JSON output with no additional text.
    """

    message = client.messages.create(
        model="claude-3-5-sonnet-20240620",
        max_tokens=1024,
        messages=[
            {"role": "user", "content": prompt}
        ]
    )

    # Extract the text content from the message
    python_code_json = message.content[0].text if message.content else '{"status": "failure", "python_code": "No code generated"}'

    return jsonify(python_code_json)

Handling the AI Response

The AI response is already being handled in the code above. We’re extracting the generated Python code from the AI’s response and returning it as JSON.

6. Web Scraping Basics

Before we finalize our backend, let’s implement the parse_relevant_content function to extract relevant HTML content from the target URL.

Introduction to BeautifulSoup and requests

We’ll use the requests library to fetch web pages and BeautifulSoup to parse HTML content. Add the following function to your app.py:

def parse_relevant_content(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')

    # Remove script and style elements
    for script in soup(["script", "style"]):
        script.decompose()

    # Extract HTML from body
    body = soup.find('body')
    if body:
        html_content = str(body)
    else:
        html_content = str(soup)

    # Log the content to a file (for debugging)
    with open('parsed_content.html', 'w', encoding='utf-8') as file:
        file.write(html_content)

    return html_content

This function fetches the webpage, removes script and style elements, and returns the relevant HTML content.

Parsing HTML Content and Extracting Specific Fields

The actual parsing and field extraction will be done by the AI-generated code. Our application provides the structure and relevant HTML content to the AI, which then generates a custom scraper based on the user’s requirements.

7. Putting It All Together

Now that we have both the frontend and backend implemented, let’s make sure everything works together seamlessly.

Connecting the Frontend to the Backend

Our frontend is already set up to make requests to the backend. Make sure your Vue.js development server is running on a different port than your Flask server (typically, Vue.js runs on port 8080 and Flask on port 5000).

Handling User Input and API Responses

The frontend ScrapeForm component handles user input and sends it to the backend. The backend processes this input, generates the scraper code using Claude AI, and sends it back to the frontend.

Displaying the Generated Python Code

The frontend already has a section to display the generated Python code. When the backend sends the response, it’s automatically displayed in the pre-formatted code block.

8. Advanced Topics

Now that we have a working application, let’s discuss some advanced topics to improve its functionality and robustness.

Error Handling and Edge Cases

  1. Input Validation: Add more robust input validation on both frontend and backend.
  2. API Error Handling: Implement better error handling for API requests and responses.
  3. AI Response Validation: Ensure the AI-generated code is valid and safe to execute.

Example of improved error handling in the frontend:

const submitScraper = async () => {
  try {
    if (!url.value || fields.value.some(field => !field.name)) {
      throw new Error('Please fill in all fields')
    }
    const result = await axios.post('http://127.0.0.1:5000/api/create-scraper', {
      url: url.value,
      fields: fields.value.map(field => field.name)
    })
    const data = JSON.parse(result.data)
    if (data.status === 'success') {
      response.value = data.python_code
    } else {
      throw new Error(data.python_code || 'Failed to generate scraper')
    }
  } catch (error) {
    console.error('Error creating scraper:', error)
    response.value = `Error: ${error.message}`
  }
}

Optimizing Scraper Performance

  1. Implement caching mechanisms to store frequently scraped data.
  2. Use asynchronous programming techniques in the generated scraper code for better performance.
  3. Implement rate limiting to avoid overloading target websites.

Handling Dynamic Websites and JavaScript Rendering

For websites that heavily rely on JavaScript to render content:

  1. Consider using tools like Selenium or Playwright in conjunction with BeautifulSoup.
  2. Implement a headless browser solution in the backend to render JavaScript before scraping.
  3. Explore APIs provided by the target websites as an alternative to scraping dynamic content.

Example of using Selenium for dynamic content:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from bs4 import BeautifulSoup

def parse_dynamic_content(url):
    chrome_options = Options()
    chrome_options.add_argument("--headless")
    driver = webdriver.Chrome(options=chrome_options)
    driver.get(url)

    # Wait for dynamic content to load
    driver.implicitly_wait(10)

    page_source = driver.page_source
    driver.quit()

    soup = BeautifulSoup(page_source, 'html.parser')
    # Process the soup object as needed
    return str(soup)

This concludes the main sections of our guide on building AI-generated custom web scrapers. The application we’ve built provides a powerful tool for creating tailored web scrapers using AI, combining the ease of use of a Vue.js frontend with the flexibility of a Flask backend and the intelligence of Claude AI.

Remember to always scrape responsibly, respecting websites’ terms of service and robots.txt files, and consider the ethical implications of your scraping activities.

Leave a Reply

Your email address will not be published. Required fields are marked *