How to Build a Web Scraper in PHP Using Goutte and DomCrawler

Introduction

Web scraping is a powerful technique used to harvest data from websites for analysis, price monitoring, or content aggregation. In the PHP ecosystem, parsing raw HTML using regex is highly discouraged because web layouts change frequently. Instead, developers rely on robust libraries that model the Document Object Model (DOM) cleanly.

In this tutorial, we will learn how to build a modern, efficient web scraper in PHP using Goutte (a wrapper around Symfony’s BrowserKit and HttpClient) and DomCrawler. This setup allows you to navigate websites, click links, and extract text or attributes with just a few lines of code.

—

Prerequisites

Before writing the scraper, ensure your machine has:

  • PHP 8.2 or higher installed.
  • Composer installed globally to manage packages.

—

Step 1: Install the Scraping Libraries via Composer

We will install the standard Symfony components required for web crawling. Open your terminal, change into your project directory, and run the following command:

composer require symfony/browser-kit symfony/http-client symfony/dom-crawler

Historical Note: While the standalone fabpot/goutte package was widely popular, it has been deprecated in favor of native Symfony components, which offer better performance and asynchronous request handling.

—

Step 2: Writing the Scraper Script

Let’s build a practical script that visits a page, extracts the titles of articles, and saves them into an array. We will target a clean HTML structure to demonstrate CSS selector matching.

Create a file named scraper.php and add the following backend code:

<?php

require 'vendor/autoload.php';

use Symfony\Component\BrowserKit\HttpBrowser;
use Symfony\Component\HttpClient\HttpClient;

// 1. Initialize the native HTTP Browser
$browser = new HttpBrowser(HttpClient::create([
    'timeout' => 10,
    'headers' => [
        'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
    ]
]));

// 2. Request the target URL
$targetUrl = 'https://news.ycombinator.com/'; // Hacker News example
$crawler = $browser->request('GET', $targetUrl);

$scrapedData = [];

// 3. Filter the DOM using CSS Selectors
$crawler->filter('.athing')->each(function ($node) use (&$scrapedData) {
    // Extract the title text
    $title = $node->filter('.titleline > a')->text();
    
    // Extract the href link attribute
    $url = $node->filter('.titleline > a')->attr('href');
    
    $scrapedData[] = [
        'title' => $title,
        'link'  => $url
    ];
});

// 4. Output the results
print_get_results($scrapedData);

function print_get_results(array $data): void {
    echo "Scraped " . count($data) . " items successfully:\n\n";
    foreach (slice_array($data, 5) as $item) {
        echo "🔹 Title: " . $item['title'] . "\n";
        echo "🔗 Link: " . $item['link'] . "\n\n";
    }
}

function slice_array(array $array, int $limit): array {
    return array_slice($array, 0, $limit);
}

—

Step 3: Handling Advanced Interactions (Pagination & Forms)

One of the key strengths of using Symfony’s BrowserKit over standard cURL is the ability to interact with the page natively, such as clicking pagination links or submitting authorization forms.

Clicking a Link Programmatically

To navigate to the next page of results, you don’t need to manually guess the URL query parameters. You can instruct the crawler to find the link text and click it directly:

// Find the link containing the text "More" and click it
$link = $crawler->selectLink('More')->link();
$nextPageCrawler = $browser->click($link);

—

Technical Verdict & Anti-Scraping Defenses

Building custom scrapers gives you absolute control over data workflows. However, production-grade scrapers must respect remote server resources. When scaling your automated scrapers, always introduce micro-delays (usleep()) between requests to avoid triggering rate-limiting firewalls or getting your server IP banned. Additionally, consider integrating proxy rotation layers if you are parsing high-frequency e-commerce platforms.

Conclusion

Using Symfony’s HTTP Client and DomCrawler turns complex HTML parsing into an elegant, fluent workflow. It forms the backbone of custom automation systems, allowing you to feed databases, generate reviews, or synchronize platforms seamlessly using structured PHP logic.