DEV Community: Emmanuel Uchenna

How to optimize CMS images in batches with Apify

Emmanuel Uchenna — Sat, 31 Oct 2026 15:30:51 +0000

An image optimization task sounds simple until you have a few hundred files, several output formats, and a content team waiting to publish. A smaller file helps only if it still looks right, keeps the dimensions your page needs, and has a reliable destination in your media library. When managing growing content libraries or building an automated SEO pipeline, unoptimized media assets quickly become a primary bottleneck for web performance.

This article walks you through using an Apify Actor, Sqoosh Image Compressor, to turn a list of source images into lightweight, modern web formats in batches. The Actor was originally built upon the open-source Squoosh project by Google Chrome Labs - hence its name on the Apify Store. The current release uses sharp and libvips for encoding under the hood. While the Actor preserves its existing Store name and common compatibility input keys, it no longer depends on legacy client-side Squoosh workers.

In this guide, you will prepare a small sample list, request JPEG and WebP outputs, inspect the generated records and artifacts, and create a review mapping ready for a safe CMS import. The workflow stops at a reviewable output stage; replacing production CMS assets remains a deliberate, separate step requiring your CMS credentials and an approved mapping.

What the workflow produces

Each successful source-image and output-codec pair produces two distinct outputs: a structured record in the run's default dataset and a downloadable binary file in the run's key-value store. For example, requesting both JPEG and WebP for a single image produces two dataset rows and two stored files. If a conversion fails, the Actor logs the error and excludes it from the successful outputs.

The primary dataset fields describe the source URL, selected codec, output key, original byte size, and encoded byte size. Always inspect the actual dataset before feeding output keys into downstream automation.

The compatibility key mozjpeg encodes standard JPEG files through sharp; it does not invoke the legacy MozJPEG WebAssembly binary. Currently supported encoder keys are mozjpeg, webp, avif, and oxipng (which produces optimized PNGs). This release does not support formats such as JXL and WP2, and it fails explicitly rather than silently falling back to another format.

Output files are stored as run artifacts. Download them or transfer them to permanent cloud storage before your Apify storage retention period expires. An Apify artifact URL is designed for inspection and data transfer, so avoid using it as a permanent CMS media link.

Step 1: Choose a representative image sample

Start with images that you own or have permission to process. A useful first sample includes a photograph, a transparent graphic, an image with small text, and an unusually wide or tall banner. If you manage an e-commerce product catalog, include both a high-resolution product detail image and a thumbnail. Testing distinct asset types helps you quickly expose format-specific artifacts and quality issues.

Make sure to provide direct, publicly accessible image URLs. A web page containing an embedded image is not the same as the raw image file. Similar to collecting source links in batch media workflows, URLs that require an active browser login session, an expired signed token, or access to a private network will fail to download.

Maintain a simple source inventory with an internal asset ID, original URL, intended page role, current pixel dimensions, and whether transparency must be preserved. Add a dedicated column for the CMS asset ID if you plan to import results back into your database. Never rely on filenames alone to match assets, as different folders across large sites frequently share identical filenames.

For your initial run, stick to two or three images. A small, bounded test proves your input and transfer path without burning compute or leaving you with hundreds of unreviewed conversions.

Step 2: Configure the image conversion

Open Sqoosh Image Compressor in the Apify Console and configure a bounded JSON input. Click on the "Try for free" button

Here is a minimal configuration requesting JPEG and WebP outputs for a single public test image:

{
  "images": [
    "https://httpbin.org/image/png"
 ],
  "encode": {
    "mozjpeg": {
      "quality": 75
 },
    "webp": {
      "quality": 75
 }
 }
}

Replace the test URL with an image from your own inventory for your acceptance check. Public test endpoints can change or go offline, so do not rely on them for production work.

The quality setting of 75 serves as a practical baseline for evaluation, not a universal rule. Quality values are not directly comparable across different codecs. A photograph often looks clean at 75, whereas a software dashboard screenshot or a graphic with sharp typography may show visible compression artifacts at that same setting.

The Actor enforces runtime safety bounds on image count, total downloaded bytes, pixel dimensions, output file sizes, and conversion duration. These boundaries prevent oversized images or runaway batches from exhausting resources. Review the input schema and documentation for exact limits before scaling up, and avoid bypassing limits without checking your source image dimensions.

Step 3: Run a small batch and inspect the evidence

Run the Actor and wait for it to finish. Once complete, inspect both the dataset and the key-value store. A green "Succeeded" status indicates that the container completed its run, but you still need to verify the generated outputs. Confirm that the dataset contains the expected number of successful rows and that the corresponding files exist in storage.

For the single-image input above requesting two codecs, you should see two successful rows: one JPEG and one WebP. If only one row appears, check the run log to see what failed rather than assuming both were processed.

Download and inspect each generated image in an image viewer. Check the entire frame alongside fine details: sharp borders, fine text, gradients, skin tones, and transparent backgrounds. Verify the dimensions as well. An image that opens successfully but loses transparency, rotates unexpectedly, or renders at incorrect dimensions is not ready for production CMS use.

The Actor's build has been validated with local and cloud JPEG and WebP outputs, and automated tests verify decoding for generated JPEG, WebP, AVIF, and PNG files. While this confirms that the encoding pipeline functions as designed, it does not guarantee that every output file will be smaller than its source or meet your team's visual standards.

Below are the actual 100 × 100 JPEG and WebP outputs generated from the test canary run and stored in this repository's assets:

In this test, the source PNG was converted to a 3,025-byte JPEG and a 2,558-byte WebP file. The repository's provenance record tracks the run ID, build ID, source URL, and SHA-256 hashes for full verification.

Step 4: Measure file-size changes correctly

Always calculate byte savings using the exact source asset as the baseline:

def byte_savings(size_before: int, size_after: int) -> dict:
    if size_before <= 0 or size_after <= 0:
        raise ValueError("Use positive measured file sizes.")
    return {
        "bytes_saved": size_before - size_after,
        "percent_saved": round(100 * (1 - size_after / size_before), 2),
 }

A negative savings value is a completely normal outcome: it simply means the encoded output is larger than the original file. Do not drop these rows or replace negative numbers with zero to make aggregate metrics look better. Tiny icons, already compressed assets, and lossless re-encodings often result in larger files.

Keep visual quality checks separate from raw byte savings. An image that achieves a 70% size reduction but turns product packaging text blurry is unacceptable for production. A milder compression setting or an alternate format is often the better compromise. Also, if your CMS or CDN already performs on-the-fly transformations, verify whether pre-compressing assets delivers meaningful savings or merely introduces an extra layer of lossy compression.

When presenting results to your team, report sample sizes, asset types, output formats, dimensions, total bytes saved, failed inputs, and rejected visual candidates. Avoid extrapolating one dramatic saving across an entire media library. High-performing assets complement broader site optimization, such as maintaining an XML sitemap and efficient crawl structure, but raw image compression alone does not guarantee higher search rankings without measuring real-world site delivery.

Step 5: Export outputs with the Apify API

You can execute the same bounded run programmatically using Python (version 3.11 or later) and the official Apify client:

pip install "apify-client>=3.3,<4"

Store your API token securely in an environment variable rather than hardcoding it in scripts, public notebooks, or client-side bundles.

import os
from datetime import timedelta
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

# Launch a bounded run of the Sqoosh Image Compressor Actor
run = client.actor("eunit/sqoosh-image-compressor").call(
    run_input={
        "images": ["https://httpbin.org/image/png"],
        "encode": {
            "mozjpeg": {"quality": 75},
            "webp": {"quality": 75},
 },
 },
    run_timeout=timedelta(seconds=120),
    max_total_charge_usd=0.10,
)

if run is None:
    raise RuntimeError("Actor run was not found.")
if str(run.status) != "SUCCEEDED":
    raise RuntimeError(f"Inspect the run before importing outputs: {run.status}")

# Iterate through dataset records containing output metadata
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

This script initiates the conversion and outputs the dataset records for verification. Orchestrating Apify Actors with Python mirrors patterns used in monitoring scheduled tasks and scraping workflows. Use the returned output keys and the run's key-value store to download approved assets, keeping your internal asset ID mapped to each record for traceability.

Note that the max_total_charge_usd parameter sets a spending cap per run. Total costs for ongoing batch jobs depend on run frequency, total converted outputs, and active Actor pricing. Check the current Store pricing model before processing large batches.

Step 6: Prepare a CMS import mapping

Create a structured reconciliation table containing one row for every source image and its replacement candidates. Include the CMS asset ID, source URL, chosen format, local output file path, dimensions, byte sizes, and an explicit approval status. If you generated multiple formats (such as WebP alongside JPEG fallback), map each format to its appropriate destination field instead of overwriting a single asset record.

Different CMS architectures handle asset updates differently:

Some CMS platforms require uploading a new media file to obtain a new asset ID, followed by updating references in existing articles or product records.
Other platforms allow in-place asset replacements under the same ID.

Keep these operations distinct. Uploading an optimized file does not automatically update pages referencing the old URL. If your workflow involves publishing across content hubs, maintaining structured manifests - similar to synchronizing data in automated CMS and Notion pipelines - prevents broken image links.

Test the update in a staging environment or on unpublished drafts first. Inspect pages across mobile and desktop breakpoints to verify that responsive srcset definitions, aspect ratios, focal points, and alternative text remain intact. Image encoders adjust file bytes, but presentation settings remain the responsibility of your CMS configuration.

Always retain original image files and maintain an explicit rollback mapping linking old asset IDs to new ones. If an automated import is interrupted, this mapping allows you to pinpoint where the process stopped and reverse changes if needed. A failed or missing conversion should never delete an original asset.

Wrapping up

A reliable image optimization pipeline produces verifiable files, maintains a clear link between sources and outputs, and provides a safe rollback mechanism before any CMS database is updated. By starting with a representative sample, verifying visual fidelity alongside byte counts, and scripting batch runs with Apify, you can automate image compression without risking broken layouts or degraded visuals.

To explore the tool, visit Sqoosh Image Compressor on Apify. For additional technical references, check the sharp documentation, Apify key-value store guides, and the Actor's input schema.

How to monitor Greenhouse and Lever job changes with Apify

Emmanuel Uchenna — Sun, 25 Oct 2026 14:26:18 +0000

If you have ever tried to track company job boards using a spreadsheet or a basic web scraper, you know how quickly the data falls apart (as detailed in our breakdown of why Playwright and headless browser scrapers fail in production). Every scheduled refresh spits out hundreds of duplicate rows. A hiring manager fixes a typo in a job title, and your script alerts you to a "brand new role." Worse, a transient network error returns an empty page, and your system flags every active vacancy at the company as filled.

A dependable Greenhouse job change monitor requires a strict operational contract:

Track each opening by its persistent source ID rather than guessing by title.
Compare observations only when an employer's board fetch is 100% complete.
Freeze baseline state automatically if an API request fails, preventing false mass removals.
Categorize changes into distinct, actionable events: new, updated, removed-from-board, and no-longer-matches.

In this article, you will set up a reliable Apify hiring monitor workflow using public Greenhouse and Lever endpoints and the ATS Job Change Monitor Actor on the Apify platform. An Applicant Tracking System (ATS) manages hiring pipelines for thousands of tech employers. While broader scrapers (such as our guide on how to scrape LinkedIn job postings with Python) are effective for multi-platform job postings scraping, querying primary ATS endpoints directly eliminates aggregator lag, anti-bot walls, and duplicate records.

We verified this workflow against live employer boards, including Greenhouse setups like Stripe and Lever's public demo board, validating complete board pagination, state preservation during simulated failures, and zero-noise runs on unchanged boards.

What this monitor observes

The Actor interacts directly with the public, unauthenticated APIs that ATS providers use to power company career portals. Greenhouse documents these public endpoints in its Job Board API, while Lever details its public endpoints in the Lever postings API repository.

To monitor a company, you only need its public board token (slug):

Greenhouse token: The identifier in boards.greenhouse.io/{boardToken} (for example, stripe).
Lever token: The site slug in jobs.lever.co/{boardToken} (for example, leverdemo).

You do not need an applicant account, recruiter credentials, or an internal ATS API key. The Actor observes public job postings exactly as prospective applicants see them.

Each record extracted from these endpoints is normalized into a predictable schema:

Core identifiers: id (a composite unique key), source_job_id, provider, and board_token.
Job metadata: title, job_url, apply_url, location, department, and employment_type.
Change tracking: change_type, changed_fields, observed_at, and source timestamps (source_created_at, source_updated_at) when exposed by the ATS.
Optional content: Job description text (with configurable character limits).

Missing fields remain null rather than placeholder defaults. This tool observes public listings; it does not collect applicant records, reveal hiring managers, guess salary ranges, or determine whether an employer sponsors visas.

Why a snapshot and a change feed are different

A snapshot and a change feed answer fundamentally different operational questions. When deciding how to monitor job board changes, choosing the right mode determines whether you get an audit of current openings or an event stream of what just happened for competitive intelligence and market research.

Output mode / event	Question it answers	Ideal use case
Snapshot	What matching jobs are currently published on this board right now?	Initial data ingestion, directory seeding, and point-in-time hiring audits.
New	Which matching job IDs appeared that were not in your saved baseline?	Instant alerts for newly published openings, lead generation, and competitor expansion signals.
Updated	Which specific fields (such as title, location, or department) changed on an existing job?	Tracking job title revisions, compensation updates, or location policy shifts.
Removed from board	Which previously tracked job IDs vanished from a confirmed complete board fetch?	Detecting closed positions, filled roles, or paused requisitions.
No longer matches	Which tracked jobs remain active on the board but no longer meet your filters?	Keeping saved baselines clean when refining keywords or team criteria.

Precision in language is critical. An event labeled removed-from-board does not definitively mean a role was filled. An employer might temporarily unpublish a post, pull a requisition, migrate to another ATS, or consolidate listings. Similarly, new means new to your monitored workflow baseline, not necessarily drafted five minutes ago.

Always keep your collection timestamp (observed_at) separate from the employer's publication timestamps (source_created_at or source_updated_at). If an ATS does not expose when a listing was updated, leave the field null. Never substitute your scraping timestamp for the employer's actual publication date.

The workflow architecture

To keep your pipeline dependable and easy to debug, separate the source collection, state comparison, and downstream notification steps:

This decoupled architecture enforces two crucial reliability principles:

Fail-safe state preservation: If an employer's board fetch fails, times out, or returns truncated data, comparison stops immediately for that board. The Actor preserves the previous baseline intact. When the employer board recovers on a subsequent run, your feed resumes cleanly without triggering phantom mass deletions.
Delivery decoupling: Writing detected changes to the Apify dataset is separate from pushing data to your external CRM, database, or Slack channel. If an external webhook fails, your dataset records remain safely preserved with their run IDs, allowing you to retry delivery without re-scraping the ATS.

Step 1: Use the ATS Job Change Monitor Actor

First of all, we will use the ATS Job Change Monitor Actor on the Apify platform.

Then click on "Try for free" to get started.

Step 2: Choose a small employer watchlist

Start with a focused watchlist of companies you actively want to monitor. A balanced initial test pairs one Greenhouse board with one Lever board:

Open the employer's public careers page in your browser.
Locate the board slug in the URL: - For Greenhouse: https://boards.greenhouse.io/stripe yields the token stripe. - For Lever: https://jobs.lever.co/leverdemo yields the token leverdemo.
Verify that the token loads valid public jobs before configuring automated monitoring.

The optional companyLabel parameter is your custom display name for reporting. It is not an identity validated by the ATS.

The Actor supports between 1 and 10 employer boards per run. Begin with two or three verified boards so you can cross-reference the extracted fields directly against the live career pages before adding broad keyword filters.

If an employer does not use Greenhouse or Lever, omit them from this configuration. Avoid guessing slugs, as invalid tokens produce source errors that complicate your monitoring logs.

Step 3: Inspect a public source response

Before scheduling recurring runs, inspect the raw data the ATS returns. You can verify how the Lever postings API handles data extraction using this quick Python script:

import httpx

endpoint = "https://api.lever.co/v0/postings/leverdemo"
with httpx.Client(timeout=30) as client:
 response = client.get(endpoint, params={"mode": "json", "skip": 0, "limit": 2})
 response.raise_for_status()
 jobs = response.json()
    for job in jobs:
        print(job["id"], job["text"], job["hostedUrl"])

While this snippet requests just two records to preview the schema, a complete monitoring run cannot stop at two. Lever paginates results using skip and limit, and the Actor automatically follows pagination until the entire catalog is retrieved before running comparisons.

For Greenhouse boards, the Actor fetches the full job listing and compares the number of received records against any summary totals provided by the API. If an ATS endpoint responds with data exceeding safety limits (such as a 10 MB payload cap or 5,000 jobs per board), the Actor flags the fetch as incomplete rather than risking an incorrect baseline comparison.

Step 4: Run a bounded snapshot

To verify how the Actor processes and normalizes real records, execute a small snapshot run in Apify Console:

{
  "boards": [
 {
      "provider": "lever",
      "boardToken": "leverdemo",
      "companyLabel": "Lever demo"
 }
 ],
  "mode": "snapshot",
  "maxItems": 2,
  "includeDescriptions": false
}

Notice the distinction between data limits:

maxItems sets a cap on the number of result records saved to the default dataset.
It does not truncate the source fetch. The Actor still inspects the full board so that completeness rules remain intact.

Once the run finishes, check the default Dataset tab, along with the RUN-SUMMARY and ERRORS records in the Key-value store tab:

Dataset: Contains clean records with stable IDs, titles, and valid job URLs.
RUN-SUMMARY: Confirms total jobs observed, active board counts, and execution metrics.
ERRORS: Documents any non-fatal network or parsing problems without generating billable result records.

Each board is protected by sensible bounds: a 90-second fetch timeout, a 5,000-job per-board ceiling, and a 10 MB response limit. These safeguards prevent unexpected runaway runs on massive enterprise boards.

Step 5: Create a reusable baseline

To track changes across time, set mode to changes and define a stateStoreName. This tells the Actor which named Key-Value store to use for preserving baseline snapshots between runs.

When monitoring an active company for the first time, you usually do not want 200 immediate notifications announcing every pre-existing vacancy as "new." Use baselineOnly: true to perform a silent initial sync:

{
  "boards": [{"provider": "lever", "boardToken": "leverdemo"}],
  "mode": "changes",
  "stateStoreName": "my-employer-watchlist",
  "baselineOnly": true,
  "includeDescriptions": false,
  "maxItems": 100
}

With baselineOnly: true, the Actor ingests and saves the complete matching catalog into your named Key-Value store without emitting rows to the dataset. You establish a clean starting baseline without triggering notification noise or custom result charges.

The state key generated internally incorporates the provider, board token, active keyword filters, and description settings. If you later change these core filters, the Actor treats the run as a new comparison scope to avoid invalid diffs.

Step 6: Compare the next observation

Once your baseline is established, set baselineOnly to false for all subsequent runs using the same stateStoreName.

If no openings were created, modified, or removed since the previous run, a successful execution emits zero rows. In job change monitoring, zero emitted rows is an ideal outcome: it proves the board was checked completely and produced no false positives.

When changes do occur, the Actor outputs specific event types:

Title or attribute updates: Emits an updated record with changed_fields detailing what was modified (e.g., ["title", "location"]).
Description modifications: When description tracking is enabled, the Actor checks a cryptographic hash of the normalized description text. It catches edits to job requirements even if you truncate long descriptions to save storage.
Partial result advancement: If your maxItems cap is reached during a high-activity run, state advances only for the events successfully written to the dataset. Unemitted changes are preserved and processed during the next run.
At-least-once delivery: State updates and dataset writes occur in distinct steps. In the rare event of a network interruption between them, a change might be re-emitted on the subsequent run. Downstream systems should deduplicate events using the composite id and observed_at fields.

Step 7: Add filters deliberately

Refine your feed using case-insensitive keyword filters for job titles and locations:

{
  "boards": [{"provider": "greenhouse", "boardToken": "YOUR_VERIFIED_TOKEN"}],
  "mode": "changes",
  "stateStoreName": "engineering-watchlist",
  "titleKeywords": ["engineer", "developer"],
  "locationKeywords": [],
  "includeDescriptions": false,
  "includeRemoved": true,
  "maxItems": 100
}

Keep your filter strategy focused:

Title filters: Match substrings within the role title. A filter for engineer catches both "Frontend Engineer" and "Engineering Manager."
Location filters: Evaluate the raw location text provided by the ATS. Keep in mind that labels like "Remote" or "US Remote" don't confirm legal work eligibility or state-specific tax compliance.
Managing removals (includeRemoved): When set to true, roles that disappear from complete board fetches emit removed-from-board events. If you set this to false, removed jobs are silently retired from baseline state without cluttering your destination feed.

Step 8: Automate with the Apify client

You can automate recurring checks and build reliable workflow automation using the Apify SDK. Store your API token and Actor ID in environment variables rather than hardcoding them in source files:

from datetime import timedelta
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(os.environ["ATS_ACTOR_ID"]).call(run_input={
    "boards": [{"provider": "lever", "boardToken": "leverdemo"}],
    "mode": "changes",
    "stateStoreName": "my-employer-watchlist",
    "baselineOnly": False,
    "maxItems": 100
}, run_timeout=timedelta(seconds=120), memory_mbytes=512, max_total_charge_usd=0.25)
if run is None:
    raise RuntimeError("Actor run was not found.")

if run.status != "SUCCEEDED":
    raise RuntimeError("Inspect the failed run before using its observations")
for change in client.dataset(run.default_dataset_id).iterate_items():
    print(change["change_type"], change["title"], change["job_url"])

Scheduling tips

Choose a practical cadence: For recruitment research or competitive analysis, once or twice daily is typically ideal. Tech companies rarely publish or withdraw job requisitions by the minute. Running every hour increases compute overhead without surfacing meaningful hiring velocity.
Prevent run collisions: Avoid scheduling overlapping runs that target the same stateStoreName. Ensure previous runs finish before starting the next cycle.

Step 9: Deliver a traceable report

When sending change feeds downstream to Slack (similar to our architecture for automated lead routing with Slack alerts), Notion (as explored in our guides on building automated pipelines connected to Notion and automating content into Notion), Airtable, or a database, structure your records around traceable facts rather than assumptions:

Maintain primary keys: Use the composite id (or provider + source_job_id) as your table's unique record identifier.
Preserve live links: Always include the normalized job_url in notifications so team members can verify the live posting with a single click.
Use accurate status labels: Label departures as "Unpublished from public board" rather than "Candidate hired." If an employer renames a role, display the previous and updated values side by side using changed_fields.
Isolate external sinks: Treat webhook notifications and database synchronizations as independent consumer processes. If a downstream API experiences downtime, your Apify dataset preserves the raw records until your delivery workers recover.

Wrapping up

Building a dependable Greenhouse job change monitor comes down to respecting source data boundaries: tracking persistent IDs, verifying complete board fetches, preserving baseline state during outages, and categorizing changes into unambiguous events.

By combining public Greenhouse and Lever endpoints with the ATS Job Change Monitor on the Apify platform, you can automate clean, noise-free hiring intelligence feeds that your recruiting team, analysts, or job board users can trust.

Start by verifying one or two company board tokens, seed a clean baseline with baselineOnly: true, and test an unchanged follow-up run before connecting downstream automations.

For more guides on production scrapers and web automation, explore our Technical Articles hub, including our hands-on Python web scraping tutorials.

How to scrape Google Search results in 2027

Emmanuel Uchenna — Tue, 06 Oct 2026 11:51:22 +0000

How to scrape Google Search results in 2027

Extracting structured data from Google search engine results pages (SERPs) has never been more important for competitive research, search engine optimization (SEO) tracking, and powering artificial intelligence (AI) applications. It has also become more challenging to do reliably.

The classic "10 blue links" tell only part of the story. AI Overviews, local map packs, job cards, and Knowledge Graph panels can appear alongside organic results. Meanwhile, consent screens, traffic challenges, and changing page layouts can turn a working Google scraper into an empty dataset.

In this article, you will learn how to scrape Google Search results with Python and prepare your workflow for 2027. We will examine three traditional DIY methods, break down where they become difficult to maintain, and walk through extracting structured search data using Google Search Scraper on the Apify platform. You will also learn how to process rich results, call a Standby search endpoint, and compare pay-per-event costs with running your own infrastructure.

TL;DR

Google SERPs contain more than organic links: AI Overviews, local results, and question panels can make search data useful for both rank tracking and content research. Which sections appear depends on the query and search context.
DIY scraping has ongoing costs: Direct HTTP requests cannot execute JavaScript, while headless browsers add browser management, resource usage, and selector maintenance.
Google Custom Search is a migration concern: The Custom Search JSON API is closed to new customers, and existing customers must transition by January 1, 2027. It is not a new integration option for a 2027 project. Google's service notice.
A managed Actor simplifies delivery: Google Search Scraper turns supported SearchApi.io responses into structured JSON on the Apify platform, with batch and Standby interfaces.
Understand the billing unit: The updated pricing proposal charges per saved search page, including available requested rich sections. Check the live Pricing tab for event rates, startup charges, and platform usage.

The state of Google SERPs in 2027

To understand why a simple Google results scraper can fail, first look at what you are asking it to collect.

An older tutorial might send an HTTP GET request to google.com/search?q=your+query, parse the HTML with BeautifulSoup or lxml, and select links using classes such as .r > a or div.g. That is a useful introduction to HTML parsing, but it leaves you responsible for recognizing the right page, handling different layouts, and finding the data your application needs.

For a workflow you want to keep using in 2027, three changes deserve particular attention.

1. The rise of AI Overviews and dynamic components

Google Search can present a generated answer alongside links to websites. For a content researcher, that introduces another question: is your competitor merely ranking in organic results, or is its content also cited in an AI Overview?

An AI Overview can include summary text and links to supporting sources. Other sections, such as People Also Ask (PAA), can reveal follow-up questions that a basic list of titles and URLs would miss. Google explains that AI Overviews appear when its systems determine they will be helpful; they are not guaranteed for every informational or commercial query. Google's explanation of AI features in Search.

Some content may be available in the initial HTML, while other details require rendering or interaction. Downloading a page successfully does not prove that you captured everything visible in a browser. Your extraction workflow needs to distinguish an absent section from one it could not collect.

2. Aggressive anti-bot and fingerprinting infrastructure

A search request also has a network and browser context. When an automated request receives different content from an interactive browser, investigate several layers:

IP and traffic context: Shared networks, request frequency, and simultaneous requests can affect whether you receive results or a traffic challenge.
Transport Layer Security (TLS) behavior: A Python HTTP client and a browser establish connections differently. Copying a browser's User-Agent header does not make their connection behavior identical.
HTTP protocol behavior: Headers, redirects, cookies, and connection settings can change what a server returns. These are useful debugging clues, rather than proof of a particular private detection rule.
Browser and session state: JavaScript execution, consent choices, and session cookies can affect the page you receive and the content available to extract.

Google documents that automated traffic can trigger its unusual-traffic warning. A successful HTTP status alone is therefore not enough: a response may contain a consent page or challenge instead of search results. Validate the page content before recording a scrape as successful, and handle access restrictions rather than repeatedly retrying them.

3. Rapidly shifting and obfuscated DOM structures

The document object model (DOM) describes a page's structure. A selector such as div.g depends on that structure staying consistent, but Google layouts can differ by device, query, region, and experiment.

You may encounter generated class names such as VwiC3b or different containers around a result heading. A selector that extracts the title from one layout can return nothing, duplicate links, or the wrong parent element in another.

This is a particularly frustrating failure mode because your script may finish without an exception. For an SEO pipeline, an empty list should trigger an investigation, not automatically become a report that a competitor has disappeared from Search.

What data can you extract from modern Google SERPs?

Before writing code, it is helpful to inventory the data surfaces available across a modern SERP. The fields you need depend on your business question. These are the eight result categories handled by the Actor, with fields returned where available:

Result type	Extracted fields	Common business use cases
Organic results	Position, title, URL, displayed URL, text snippet, highlighted keywords, site links, rating, and date	SEO rank tracking, competitor page discovery, and brand visibility audits
AI Overviews	Returned summary text, formatted Markdown, and source citation links	Generative engine optimization (GEO), AI citation analysis, and competitive messaging research
Knowledge Graph	Entity title, entity type, description, official website, phone, attributes, and social profiles	Entity authority mapping, knowledge base population, and company profiling
Local results (Maps pack)	Business title, physical address, review score, total reviews, phone number, operating hours, and listing link	Local SEO tracking, brick-and-mortar market research, and lead generation
Google Jobs	Job title, hiring organization, work location, posting source, description, and available job highlights	Recruitment intelligence, labor market trend analysis, and competitor hiring alerts
Google Events	Event name, date, venue, physical location, listing links, and descriptive summary	Event aggregation, ticketing platforms, and regional entertainment intelligence
People Also Ask	Question text, answer summary, source title, and destination link	Content brief ideation, FAQ generation, and question-intent keyword clustering
Related searches	Search suggestion term and destination search query link	Semantic keyword expansion, topic clustering, and consumer search intent mapping

The 3 traditional ways to scrape Google Search (and why they break)

Developers building a Google SERP scraper typically follow one of three paths. Let us analyze each option with Python code to examine where they break in production.

Option 1: Google Custom Search JSON API

Google's Custom Search JSON API returns structured web or image results from a configured Programmable Search Engine. It has often appeared in Python tutorials because it avoids writing an HTML parser.

There is an important availability change: the API is closed to new customers, and Google gives existing customers until January 1, 2027 to transition to an alternative. The following example helps explain an existing integration during that transition; do not use it as the starting point for a new 2027 service. Google's Custom Search documentation.

An existing customer with an API key and search engine ID can recognize the request pattern below. Install requests with pip install requests if needed:

import os
import requests

API_KEY = os.environ["GOOGLE_API_KEY"]
SEARCH_ENGINE_ID = os.environ["GOOGLE_CSE_ID"]

def query_google_custom_search(query: str, start_index: int = 1) -> dict:
    url = "https://www.googleapis.com/customsearch/v1"
    params = {
        "key": API_KEY,
        "cx": SEARCH_ENGINE_ID,
        "q": query,
        "start": start_index,
        "num": 10,
    }

    response = requests.get(url, params=params, timeout=10)
    response.raise_for_status()
    return response.json()

results = query_google_custom_search("best web scraping tools 2027")
for item in results.get("items", []):
    print(f"- {item['title']}: {item['link']}")

Why it breaks in production: Google Custom Search JSON API

The main issue for a new project is availability, followed by whether the returned data answers your question:

A transition deadline: Existing integrations need a replacement by the announced date. New customers cannot obtain access to this API.
Quotas during the transition: For eligible existing customers, Google documents 100 free queries per day, then $5 per 1,000 additional queries, up to 10,000 queries per day. Those are legacy service terms, not a new 2027 offer.
A different search product: Programmable Search Engine results are not a capture of a google.com results page. Do not assume identical rankings or the same AI Overview, local pack, and question-panel coverage.

For an application that searches a defined collection of sites, evaluate a supported search service suited to that task. For an SEO tool observing Google SERPs, evaluate the actual result fields and localization support instead.

Option 2: Direct HTTP requests with BeautifulSoup or Parsel

The second common approach is sending direct HTTP GET requests to google.com/search using an HTTP client like httpx or requests and parsing the resulting HTML with BeautifulSoup or Parsel. To try this small parsing example, install its dependencies with pip install "httpx[http2]" parsel:

import httpx
from parsel import Selector

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/130.0.0.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

def scrape_google_serp(query: str) -> list[dict]:
    url = "https://www.google.com/search"
    params = {"q": query, "hl": "en", "gl": "us"}

    with httpx.Client(http2=True, headers=headers, follow_redirects=True, timeout=20.0) as client:
        response = client.get(url, params=params)

        if "unusual traffic" in response.text.lower() or response.status_code == 429:
            raise RuntimeError("Request was blocked by Google anti-bot systems.")

        response.raise_for_status()
        selector = Selector(response.text)
        results = []

        for box in selector.xpath("//a[.//h3]"):
            title = "".join(box.xpath(".//h3//text()").getall()).strip()
            link = box.xpath("./@href").get() or box.xpath(".//a/@href").get()

            if title and link and link.startswith("http"):
                results.append({"title": title, "url": link})

        return results

try:
    data = scrape_google_serp("python web scraping tutorial")
    print(f"Extracted {len(data)} results")
except Exception as exc:
    print(f"Scraping failed: {exc}")

Why it breaks in production: direct HTTP requests

This example illustrates how to collect heading links from a returned document. It is not a complete Google SERP parser, and an empty result is not proof that Google has no matching pages.

Blocked or unexpected responses: A request can receive a rate limit, challenge, consent screen, or other content. There is no universal number of requests after which every IP is blocked. Inspect the response and stop when access is restricted.
No JavaScript execution: An HTTP client cannot render or interact with the page. Content loaded after the initial response may be missing, even when some rich results are already present in the HTML.
Selector maintenance: A heading selector does not reliably distinguish organic results from every other linked heading. Snippets, result positions, local cards, and citation panels need additional parsing and validation.

Direct requests can be useful for learning or a small experiment. The workload grows when you need consistent coverage across hundreds of keywords, different devices, and repeated collection dates.

Option 3: Headless browsers with Playwright or Selenium

To overcome static HTML limitations, many teams turn to headless browser automation using Playwright, Puppeteer, or Selenium. By spinning up a full Chromium instance, the script executes Google's client-side JavaScript, renders fonts, and can interact with the page. Install Playwright with pip install playwright, then install its browser with playwright install chromium:

from urllib.parse import urlencode
from playwright.sync_api import sync_playwright

def scrape_with_playwright(query: str) -> list[dict]:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        try:
            context = browser.new_context(
                locale="en-US",
                viewport={"width": 1280, "height": 800},
            )
            page = context.new_page()
            params = urlencode({"q": query, "gl": "us", "hl": "en"})
            response = page.goto(
                "https://www.google.com/search?" + params,
                wait_until="domcontentloaded",
                timeout=30_000,
            )
            if response is None or not response.ok:
                raise RuntimeError("Search navigation did not succeed.")

            # A page can load successfully without containing search results.
            body = page.locator("body").inner_text(timeout=10_000).lower()
            if "sorry/index" in page.url or "unusual traffic" in body:
                raise RuntimeError("Encountered a Google traffic challenge.")
            if "consent.google.com" in page.url:
                raise RuntimeError("Consent is required before collecting results.")

            page.locator("a h3").first.wait_for(timeout=15_000)
            results = []
            for item in page.locator("a h3").all():
                title = item.inner_text().strip()
                link = item.locator("xpath=ancestor::a[1]").get_attribute("href")
                if title and link and link.startswith("http"):
                    results.append({"title": title, "url": link})
            return results
        finally:
            browser.close()

data = scrape_with_playwright("python web scraping tutorial")
print(f"Found {len(data)} heading links with Playwright")

For more background on browser automation, see my comparison of Playwright and WebdriverIO.

Why it breaks in production: headless browsers

A browser gives you JavaScript execution, but it also gives you a browser fleet to operate. The example above still collects heading links, rather than a complete structured SERP.

Resource consumption: Browser processes use CPU and memory. The amount depends on the page, browser settings, and concurrency, so measure your workload before choosing a server size.
Challenges and session handling: Opening Chromium does not guarantee access. Your workflow still needs to recognize consent screens, rate limits, and challenges, then handle them appropriately.
Network and operational costs: Depending on the source and permitted collection method, you may need network infrastructure, monitoring, and retry controls. Include those expenses and failed attempts in your estimate.
Selector upkeep: Rendering a widget does not automatically turn it into structured data. You still need extraction logic for AI Overviews, Knowledge Graphs, job listings, and their changing layouts.

If you enjoy building and maintaining that infrastructure, a browser-based approach gives you control. If your goal is to analyze search data or power an application, a managed search-data interface can reduce the work between a query and a useful dataset.

The modern solution: Google Search Scraper on the Apify platform

Instead of building, debugging, and hosting the collection infrastructure yourself, you can use a managed cloud Actor designed for search data. An Actor is an application that runs on the Apify platform and can be called from Apify Console, an API, or your own code.

Google Search Scraper bridges search queries and structured API delivery. The updated implementation calls SearchApi.io, normalizes the supported response fields, and saves batch results to an Apify dataset. It does not run its own Google HTML parser.

Version note: This walkthrough covers the updated Actor interface, which is in private testing at the time of writing. Use an authorized deployment to follow along until that version is available in the Store; older builds and prices can differ.

The updated request flow is:

flowchart TD
    App["Your application or pipeline"]
    Actor["Google Search Scraper on the Apify platform"]
    Provider["SearchApi.io"]
    Search["Google Search data"]
    Output["Batch dataset or Standby JSON response"]
    App --> Actor
    Actor --> Provider
    Provider --> Search
    Search --> Provider
    Provider --> Actor
    Actor --> Output

Why this approach wins in 2027

Eight result categories in one response: Collect organic results and available AI Overviews, Knowledge Graphs, local results, jobs, events, People Also Ask questions, and related searches. Each rich section depends on what the provider returns for that query.
No customer proxy setup: When you use the hosted Actor, you do not configure a browser fleet or a proxy rotation pool. SearchApi.io supplies the upstream search-data service.
Location and device controls: Set a country (gl), interface language (hl), and device. The batch interface also supports city or regional targeting through location, helping you compare like-for-like search observations.
A clear page-based event: The updated pay-per-event (PPE) proposal uses one saved search page as its custom billing unit. Available requested AI Overview and Knowledge Graph sections are included, rather than charged as separate custom add-ons. Startup and usage charges still depend on the live pricing configuration.
Batch and Standby modes: Use batch runs for keyword collections and recurring audits, or request a JSON response through the Standby endpoint for an application or AI agent. Response time depends on the upstream request and deployment.

For the hosted Actor, the operator provisions the SearchApi.io credentials. If you deploy your own copy, configure SEARCHAPI_IO_KEY as an environment secret and budget for that provider separately. Your Apify token authenticates calls to Apify; it is not a replacement for the operator's upstream key.

Step-by-step tutorial: scraping Google Search with Python

Let us walk through a complete, hands-on tutorial using the official Apify Python client to extract structured SERP data.

Step 1: Install the Apify client

In your Python terminal or virtual environment, install the apify-client package:

pip install "apify-client>=3.3,<4"

Step 2: Retrieve your Apify API token

Log in to Apify Console. If you do not have an account, sign up and check the current free allowance and account pricing before running the example.
Open the API & Integrations page in Settings to find your API tokens.
Copy your personal API token and set it in your local environment:

export APIFY_TOKEN="your_apify_token_here"

If you use PowerShell on Windows, set it for the current session with:

$env:APIFY_TOKEN = "your_apify_token_here"

Keep the token out of source control and client-side application code.

Step 3: Run the Actor and process results

Create a file named scrape_google.py and add the following Python code. Start with two queries and one page each so you can inspect the output before increasing the volume. For a private deployment, set GOOGLE_SEARCH_ACTOR_ID to its Actor ID or username/actor-name; the default below is the Store identifier.

import os
from apify_client import ApifyClient

# Initialize the Apify client with your API token
client = ApifyClient(os.environ["APIFY_TOKEN"])
actor_id = os.environ.get("GOOGLE_SEARCH_ACTOR_ID", "eunit/google-search-scraper")

# Configure your search parameters
actor_input = {
    # Provide one or more search queries
    "queries": [
        "Apify documentation",
        "best telehealth platforms",
    ],
    # Number of result pages per query; the organic result count can vary
    "max_pages_per_query": 1,
    # Geographic country code and interface language
    "gl": "us",
    "hl": "en",
    # Request desktop, mobile, or tablet search results
    "device": "desktop",
    # SafeSearch filter: "active", "blur", or "off"
    "safe": "blur",
    # Select which rich data surfaces to extract
    "include_ai_overview": True,
    "include_knowledge_graph": True,
    "include_jobs": True,
    "include_events": True,
    "include_local_results": True,
    "include_related_questions": True,
    "include_related_searches": True,
    # Concurrency control
    "max_concurrency": 2,
}

print("Starting Google Search Scraper on the Apify platform...")

# Execute the Actor run and wait for it to complete
run = client.actor(actor_id).call(run_input=actor_input)
if run is None:
    raise RuntimeError("Actor run was not found.")

print(f"Run completed with status: {run.status}")
print(f"Dataset ID: {run.default_dataset_id}")

# Fetch and iterate through the structured items in the default dataset
if run.status != "SUCCEEDED":
    raise RuntimeError(f"Actor run did not succeed: {run.status}. Check its log.")

dataset = client.dataset(run.default_dataset_id)

for item in dataset.iterate_items():
    query_term = item["searchQuery"]["term"]
    if item.get("error"):
        print(f"Search failed for {query_term}: {item['error']}")
        continue
    total_results = item["searchInformation"].get("totalResults")
    if total_results is None:
        total_results = "N/A"

    print("\n" + "=" * 60)
    print(f"Search query: {query_term} (Reported result estimate: {total_results})")
    print("=" * 60)

    # 1. AI Overview
    if item.get("aiOverview"):
        print("\n[AI Overview]")
        ai_text = item["aiOverview"].get("text") or ""
        print(f"{ai_text[:300]}..." if len(ai_text) > 300 else ai_text)
        print("Citations:")
        for ref in item["aiOverview"].get("referenceLinks") or []:
            print(f"  - {ref['title']}: {ref['link']}")

    # 2. Knowledge Graph
    if item.get("knowledgeGraph"):
        kg = item["knowledgeGraph"]
        print(f"\n[Knowledge Graph: {kg.get('title')} ({kg.get('type')})]")
        if kg.get("description"):
            print(f"Description: {kg['description']}")
        if kg.get("website"):
            print(f"Website: {kg['website']}")

    # 3. Organic Results
    print(f"\n[Organic Results: {len(item.get('organicResults', []))} found]")
    for org in item.get("organicResults", []):
        pos = org["position"]
        title = org["title"]
        url = org["url"]
        print(f"  {pos}. {title}")
        print(f"     URL: {url}")
        if org.get("description"):
            print(f"     Snippet: {org['description'][:120]}...")

    # 4. People Also Ask
    if item.get("relatedQuestions"):
        print(f"\n[People Also Ask: {len(item['relatedQuestions'])} questions]")
        for q in item["relatedQuestions"][:3]:
            print(f"  ? {q['question']}")
            if q.get("answer"):
                print(f"    Ans: {q['answer'][:100]}...")

    # 5. Related Searches
    if item.get("relatedSearches"):
        print("\n[Related Searches]")
        related = [r["query"] for r in item["relatedSearches"][:5]]
        print("  " + " | ".join(related))

Step 4: Inspecting the output data

Run the script with python scrape_google.py. After the run succeeds, it reads the default dataset and prints the query, organic results, and any available rich sections.

Here is an excerpt from an actual test for Apify documentation on October 21, 2026. The arrays and organic result fields are shortened for readability; the values below come from that saved observation:

{
  "searchQuery": {
    "term": "Apify documentation",
    "url": "https://www.google.com/search?q=Apify documentation",
    "device": "desktop",
    "page": 1,
    "gl": "us",
    "hl": "en"
  },
  "searchInformation": {
    "queryDisplayed": "Apify documentation",
    "totalResults": 340000,
    "timeTakenDisplayed": 0.25,
    "didYouMean": null,
    "detectedLocation": "95240, California"
  },
  "organicResults": [
    {
      "position": 1,
      "title": "Apify Documentation",
      "url": "https://docs.apify.com/",
      "displayedUrl": "https://docs.apify.com",
      "description": "Apify is all about Actors\u2014a new way to package your code to make it easy to share, integrate, and build upon. Actors on the Apify ..."
    },
    {
      "position": 2,
      "title": "Apify: Marketplace of ready-to-run tools for AI",
      "url": "https://apify.com/",
      "displayedUrl": "https://apify.com",
      "description": "Thousands of tools to automate your business. Get real-time web data, track competitors, generate leads, and integrate your apps and AI agents."
    }
  ],
  "relatedQuestions": [
    {
      "question": "What is Apify mainly used for?",
      "answer": null,
      "link": null,
      "source": null,
      "thumbnail": null
    }
  ],
  "relatedSearches": [
    {
      "query": "Apify documentation examples",
      "link": "https://www.google.com/search?sca_esv=9dd08dafeeb6e81c&gl=us&hl=en&pws=0&sxsrf=APpeQnvn1FNcy-Pkd1swx5pHUqvUPm7Azg:1792623169968&q=Apify+documentation+examples&sa=X&ved=2ahUKEwjc1YvamcyXAxXZHoAAHVM9DpcQ1QJ6BAgqEAE"
    },
    {
      "query": "Apify login",
      "link": "https://www.google.com/search?sca_esv=9dd08dafeeb6e81c&gl=us&hl=en&pws=0&sxsrf=APpeQnvn1FNcy-Pkd1swx5pHUqvUPm7Azg:1792623169968&q=Apify+login&sa=X&ved=2ahUKEwjc1YvamcyXAxXZHoAAHVM9DpcQ1QJ6BAg-EAE"
    }
  ],
  "localResults": [],
  "jobs": [],
  "events": [],
  "aiOverview": null,
  "knowledgeGraph": null,
  "scrapedAt": "2026-10-21T22:52:51.501393+00:00",
  "page": 1,
  "error": null
}

Notice that aiOverview and knowledgeGraph are null in this example. Requesting those sections makes the Actor include them when available; it does not make Google generate them. Similarly, an empty jobs or events array can be normal for a query that is not about jobs or events.

The returned totalResults is a reported estimate, not a count of records in your dataset. Keep searchQuery, page, and scrapedAt with your results so you can compare observations collected under the same settings.

You can export a batch dataset from Apify Console in JSON, CSV, Excel, XML, or HTML table format. JSON is useful for nested citations and site links; a spreadsheet is convenient for sorting rankings or reviewing content ideas. Integrations and completion webhooks can connect the run to your wider workflow.

For recurring SEO monitoring, save the input as a task and schedule it. Each run creates its own dataset, so use that run's dataset ID when importing results into your database or spreadsheet. A saved task does not automatically append every observation to one permanent search-history table.

Standby mode: turning Google into a real-time HTTP search API

Batch scraping is ideal when you need to process hundreds or thousands of keywords for weekly SEO audits. Applications can also need search results on demand - for example, AI agents researching a question, live chat assistants retrieving citations, or retrieval-augmented generation (RAG) pipelines.

Starting a separate batch run for each question adds startup overhead. To avoid repeating that pattern, Google Search Scraper supports Standby mode.

How Standby mode works

In Standby mode, the Apify platform keeps the Actor ready to receive HTTP requests, allowing it to behave like an API server. Instead of starting a new batch job for every query, your application calls the /search endpoint and receives structured JSON.

Open the Actor's Endpoints tab and copy the deployed hostname. Set that base URL as SEARCH_STANDBY_URL in your environment, without appending /search. If the tab is absent, Standby is not enabled for that version. Authenticate with your Apify token in the Authorization: Bearer header, as described in the Apify Standby documentation.

Standby reduces the need to create individual batch runs, but the upstream search still takes time. Benchmark your own query mix before choosing application timeouts or promising a response-time target.

Standby API example in Python

You can call the endpoint with a standard HTTP client. Install httpx with pip install httpx if you did not install it for the earlier example. Keep the API token on your server, rather than exposing it in browser JavaScript.

import os
import httpx

# Copy the base hostname from the Actor's Endpoints tab.
endpoint = os.environ["SEARCH_STANDBY_URL"].rstrip("/") + "/search"
headers = {"Authorization": "Bearer " + os.environ["APIFY_TOKEN"]}

# Standby uses URL parameters, rather than the batch input's queries list.
params = {
    "q": "best telehealth startups Nigeria",
    "gl": "ng",
    "hl": "en",
    "device": "desktop",
    "page": 1,
    "include_ai_overview": "true",
}

with httpx.Client(timeout=45.0) as client:
    response = client.get(endpoint, headers=headers, params=params)
    response.raise_for_status()
    serp = response.json()

    if serp.get("error"):
        raise RuntimeError(f"Search failed: {serp['error']}")

    print(f"Scraped term: {serp['searchQuery']['term']}")
    print(f"Total organic results: {len(serp.get('organicResults', []))}")
    for item in serp.get("organicResults", [])[:3]:
        print(f"[{item['position']}] {item['title']} -> {item['url']}")

    if serp.get("aiOverview"):
        print("AI Overview:", serp["aiOverview"].get("text"))

This interface is useful when an AI agent needs a few current sources for a question, while batch mode is more convenient for a collection you want to export and compare later. Standby also incurs platform usage, so include it in the cost of an application that sends frequent requests.

Cost comparison: DIY infrastructure vs. pay-per-event pricing

When choosing how to scrape Google Search results, compare the cost of obtaining useful data, rather than only the price of one successful request. A DIY workflow has operating and maintenance expenses; a managed Actor has event charges and potentially separate usage charges.

The updated Actor's proposed custom event is search-page-scraped at $0.008 per saved search page, or $8 per 1,000 pages. A page can contain several organic links, so one event is not one individual website result. Available requested AI Overview and Knowledge Graph sections are included in that custom event.

For a workload of 50,000 successfully saved pages per month, compare these components:

Expense component	DIY Playwright workflow	Updated Google Search Scraper proposal
Browser compute	Budget for CPU, memory, browser concurrency, and failed attempts	No customer browser fleet; Apify usage may be billed separately
Network and collection service	Budget for your chosen collection method and any network services	SearchApi.io is provisioned by the hosted Actor's operator
Parsing and maintenance	Maintain selectors, layout handling, and monitoring	The operator maintains normalization; you still validate fields and monitor your integration
Custom result fee per 1,000 pages	No Actor event fee, but infrastructure and engineering still cost money	Proposed $8 for saved-page events
Custom result fee for 50,000 pages	Calculate from measured infrastructure and maintenance costs	Proposed $400, before other applicable charges
Startup and Standby costs	Depend on how you host and scale the service	Check startup events and platform usage in the live Pricing tab

The $400 calculation is a custom-event subtotal, not an all-inclusive invoice or an activated Store price. Apify supports PPE configurations that pass platform usage to the customer, and Standby usage is billed in addition to events. Review the actual configuration before running a large workload. Apify's PPE billing documentation.

For the hosted Actor, you do not separately configure a SearchApi.io subscription. If you deploy your own copy, you become the operator and must include that provider's plan, request allowance, and overage costs in your estimate. For an Actor business, event revenue is also different from profit: deduct the platform's applicable share and any seller-paid operating costs before comparing prices.

Start with a representative batch, inspect its output, and review the charges. That gives you a useful basis for forecasting a daily rank tracker, a weekly audit, or an on-demand search application without relying on unmeasured DIY cost ranges.

Frequently asked questions

Is scraping Google Search legal?

There is no single answer for every project or jurisdiction. Public visibility alone does not establish permission for every collection method or downstream use. Review Google's terms, the applicable service and provider terms, and the rules that apply to the data you collect.

For example, storing business rankings raises different questions from building a database of personal information found in search snippets. Define what you need, minimize unnecessary personal data, respect access restrictions, and seek qualified advice where the intended collection or reuse is uncertain.

How does the scraper extract Google AI Overviews?

The updated Actor calls SearchApi.io and normalizes any AI Overview returned in the provider response. It does not independently reconstruct an answer from the organic links.

When available, aiOverview.text holds the returned text, aiOverview.markdown holds formatted content, and aiOverview.referenceLinks contains citation information. Enable include_ai_overview to request the section, then check whether it is present before processing it. Under the updated pricing proposal, an available AI Overview is included in the saved-page event.

For citation research, save the query, language, country, device, and capture time alongside those references. A missing overview in one response does not prove that Google never shows one for that query.

Can I localize queries to specific countries, languages, and cities?

Yes. Use gl for the country, such as us, gb, de, or ng, and hl for the interface language, such as en, es, or fr. Choose desktop, mobile, or tablet through device.

The batch interface also accepts location for city or regional targeting, such as "New York, United States" or "Munich, Germany". Use it to compare a local business's visibility across markets. The Standby example above uses country and language parameters; its current handler does not accept the batch interface's location field.

These controls configure the search request, but they do not guarantee the exact results seen by a particular person. Personalization, timing, and source behavior can still differ. Keep the same settings when comparing rankings over time.

What data export formats are supported?

Batch results are stored in an Apify dataset, which you can export as JSON, CSV, Microsoft Excel (XLSX), XML, or an HTML table. Use JSON when you need nested AI citations, site links, or Knowledge Graph attributes; use a spreadsheet-friendly format for ranking reviews and content planning.

You can also retrieve dataset items through the Apify API or Python client, then load them into your own database. Standby returns JSON directly to the caller. If you need a permanent history of those responses, save them in your application.

Wrapping up

Scraping Google Search results for a 2027 workflow means looking beyond titles and links. Organic rankings, AI citations, local listings, and follow-up questions can answer different business questions, and a useful pipeline preserves both the data and the context in which it was collected.

The three DIY approaches explain the trade-offs: an official API has its own availability and coverage limits, direct HTTP requests leave you responsible for parsing and incomplete pages, and headless browsers add an infrastructure and maintenance workload.

Google Search Scraper on the Apify platform packages supported search data into structured JSON, with batch runs for recurring research and a Standby endpoint for application requests. Start with a small query set, check which rich sections are returned, and use the live pricing to plan a workload you can sustain.

Ready to turn search results into a useful dataset?

Create your Apify account and review the current allowance.
Open Google Search Scraper in Apify Store, then check the available version, input fields, and Pricing tab.
Once you have access to the updated version, enter a few target queries, click Start, and inspect the output before scheduling a larger run.

How to Build an Automated SEO Pipeline: Connecting Sitemap Generator and Google Indexer to Notion

Emmanuel Uchenna — Wed, 09 Sep 2026 09:15:19 +0000

Generating an XML sitemap is only the first step in modern technical Search Engine Optimization (SEO). If you have ever published a new blog post, launched a product line on an e-commerce platform, or updated critical landing pages, you know the frustration of waiting days, weeks, or even months for Google to crawl and index your content.

Relying on search engine crawlers to discover changes organically is no longer viable for high-growth websites. If you run a dynamic content platform, an online store with rapidly changing inventory, or a news publication, delayed indexing directly equates to lost traffic, lower visibility, and missed revenue.

In this guide, we will walk through the process of building an automated, enterprise-grade SEO pipeline. We will start by covering automated URL discovery using the Fast Sitemap Generator Apify Actor. Then, we will look at programmatically pushing those URLs to Google using the Google Indexer & Instant SEO Submitter Actor. Finally, we will solve the critical tracking challenge by using Model Context Protocol (MCP) Connectors to stream real-time indexing logs directly into a Notion database.

The evolution of programmatic indexing

Historically, website owners managed SEO by submitting a static Extensible Markup Language (XML) sitemap through Google Search Console. While essential, submitting a sitemap is fundamentally a passive notification. You are effectively dropping a note in Google's mailbox, asking Googlebot to visit your site whenever it gets around to it.

For small, static websites, this passive approach works. However, for modern dynamic sites, it falls short due to several structural challenges:

Crawl budget constraints: Search engines assign a finite "crawl budget" to every domain. On large sites with thousands of pages, Googlebot may exhaust its budget on legacy pages, completely ignoring newly published or updated content.
Orphaned or deep pages: Content buried deep within subdirectories or pagination chains often fails to receive internal link equity, making it nearly invisible to organic crawling.
Time-sensitive content: E-commerce price updates, breaking news, real estate listings, and job postings require instant indexation. Waiting two weeks for organic discovery renders the content stale before it ever reaches search results.
Desktop crawler overhead: Traditional site auditors like Screaming Frog run locally, consuming heavy system resources, tying up developer machines, and requiring manual CSV exports to process updates.

Moving from passive crawling to active push pipelines

To solve these limitations, search engines introduced programmatic indexing APIs. Instead of waiting for Googlebot to discover changes, you can actively send a push notification to Google the instant a page is created or updated.

By combining cloud-based crawling with active API submissions and real-time database logging, you can replace manual SEO workflows with a fully automated "set and forget" pipeline.

Stage 1: Automated URL discovery with Fast Sitemap Generator

Before you can submit URLs to Google, you need an accurate, deduplicated list of active links from your website. Hand-coding XML files or relying on CMS plugins that crash on large databases creates maintenance bottlenecks.

The Fast Sitemap Generator Actor runs on the Apify platform as a serverless cloud program. It uses a Direct Connection to crawl websites at high speeds directly from data centers, without requiring proxy infrastructure.

Configuring the URL crawler

Setting up automated URL discovery requires configuring key parameters inside the Apify Console to balance thoroughness with execution speed.

Here is an example input configuration for a production crawl:

{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxCrawlDepth": 3,
  "maxPagesPerCrawl": 5000,
  "includePatterns": [
    ".*/blog/.*",
    ".*/products/.*"
  ],
  "excludePatterns": [
    ".*/admin/.*",
    ".*/login.*",
    ".*/cart.*",
    ".*/checkout/.*"
  ],
  "sitemapFormats": [
    "xml",
    "html",
    "txt"
  ],
  "respectRobotsTxt": true,
  "changefreq": "daily",
  "defaultPriority": 0.8,
  "includeImages": false
}

Key configuration settings explained

Start URLs (startUrls): The entry point for the crawler. Usually, this is your primary domain homepage.
Max Crawl Depth (maxCrawlDepth): Controls how deep the crawler traverses link structures. A depth of 3 to 5 is typically sufficient to discover all essential content while preventing infinite loops.
Regex Filtering (includePatterns and excludePatterns): Prevents cluttering your index with utility pages. You should explicitly exclude administrative paths like .*/admin/.*, checkout pages .*/cart.*, and query parameters that create duplicate content.
Robots.txt Respect (respectRobotsTxt): Ensures your pipeline adheres to site governance rules and crawler restrictions.

Output storage and Dataset generation

When execution completes, the Actor stores output artifacts in two locations:

Key-Value Store: Contains compiled sitemap.xml, sitemap.html, and sitemap.txt files ready for static hosting.
Apify Dataset: Stores structured JSON records for every discovered page. The Dataset ID produced during this run serves as the input payload for the next stage in our pipeline.

Stage 2: Programmatic submission via Google Cloud infrastructure

With your site URLs discovered and structured into an Apify Dataset, the next phase is notifying Google. This step relies on the official Google Indexing API, which allows site owners to submit batch update requests directly to Google's indexing pipeline.

Step 1: Setting up Google Cloud credentials

Before making programmatic requests, you must configure authentication through the Google Cloud Console.

Create a Google Cloud Project: Log in to Google Cloud Console, click the project dropdown, and select New Project. Name it seo-automation-pipeline.
Enable the Indexing API: Navigate to APIs & Services > Library, search for "Web Search Indexing API", and click Enable.
Create a Service Account: Go to IAM & Admin > Service Accounts, click Create Service Account, name it indexing-bot, and assign it the Project > Editor role.
Generate a JSON Key: Select your new service account, go to the Keys tab, click Add Key > Create new key, select JSON, and download the file. Keep this file secure, as it grants API permissions.

Service Account Email Example:
indexing-bot@seo-automation-pipeline.iam.gserviceaccount.com

Step 2: Granting Google Search Console access

Enabling the API in Google Cloud is not enough on its own. Google requires verified domain ownership to prevent unauthorized indexing requests.

Open Google Search Console.
Select your verified domain property.
Navigate to Settings > Users and permissions.
Click Add user, paste your service account email address, set the role to Owner, and save.

IMPORTANT
Google Search Console requires Owner permissions for Service Accounts calling the Indexing API. Setting lower permission levels such as "Full" or "Restricted" will cause authentication errors during API calls.

Bypassing DIY limitations: Python vs. Apify Actors

Developers often attempt to write custom scripts to interact with the Google Indexing API. Below is a standard Python implementation using official Google authentication libraries:

import json
import requests
from google.oauth2 import service_account
from google.auth.transport.requests import AuthorizedSession

SERVICE_ACCOUNT_FILE = "google-key.json"
INDEXING_API_ENDPOINT = "https://indexing.googleapis.com/v3/urlNotifications:publish"

def get_authenticated_session(key_path):
 scopes = ["https://www.googleapis.com/auth/indexing"]
 credentials = service_account.Credentials.from_service_account_file(
 key_path, scopes=scopes
 )
    return AuthorizedSession(credentials)

def submit_url(session, url, action_type="URL_UPDATED"):
 payload = {
        "url": url,
        "type": action_type
 }
 response = session.post(INDEXING_API_ENDPOINT, json=payload)
    return response.status_code, response.json()

if __name__ == "__main__":
 session = get_authenticated_session(SERVICE_ACCOUNT_FILE)
 target_url = "https://example.com/blog/automated-seo-pipeline"
 status, result = submit_url(session, target_url)
    print(f"Status Code: {status}")
    print(f"Response: {json.dumps(result, indent=2)}")

Why custom scripts break at scale

While the DIY Python script works for single URLs, operating it in production exposes major infrastructure hurdles:

Rate limit handling: Google enforces daily quotas (typically 200 URL notifications per day for standard projects). Custom scripts will fail with HTTP 429 Too Many Requests unless you write exponential backoff algorithms.
State management: A basic script does not remember which URLs were submitted yesterday. Running it daily resubmits duplicate URLs, wasting your daily quota.
Execution server maintenance: You must deploy the script to a cloud server, manage cron schedules, and store credentials securely.

The solution: Google Indexer & Instant SEO Submitter

Instead of writing and hosting custom boilerplate code, we use the Google Indexer & Instant SEO Submitter Actor.

Feature	Custom Python Script	Google Indexer Actor
Quotas & Rate Limiting	Manual handling required	Built-in automatic rate limiting & backoff
Input Source	Hardcoded or manual files	Direct integration with Apify Datasets & Sitemaps
Pricing Model	Server hosting costs	Pay-Per-Event
Test Mode	Requires code flags	Built-in dry-run toggle
Monitoring	Custom logging setup	Real-time console logs & dataset outputs

To configure the Actor, pass the JSON credentials and your Dataset ID from Stage 1 directly into the input schema:

{
  "datasetId": "YOUR_STAGE_1_DATASET_ID",
  "serviceAccountJson": {
    "type": "service_account",
    "project_id": "seo-automation-pipeline",
    "private_key_id": "abcdef123456789",
    "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQC...\n-----END PRIVATE KEY-----\n",
    "client_email": "indexing-bot@seo-automation-pipeline.iam.gserviceaccount.com"
  },
  "action": "URL_UPDATED",
  "testMode": false
}

Stage 3: The missing link: Logging to Notion via MCP Connectors

Automating crawls and API submissions solves the execution side of SEO. However, engineering teams and SEO leads still face a visibility gap: How do you monitor pipeline results without manually checking console logs after each run?

Exporting CSVs after every run is tedious, and setting up custom webhooks requires maintaining dedicated receiver endpoints. This is where MCP connectors fit into the pipeline.

Understanding MCP connectors

The Model Context Protocol (MCP) is an open standard that lets Actors call third-party services like Notion, Slack, and GitHub on your behalf, using your own credentials, without the Actor ever holding your token directly.

MCP connectors on Apify work differently from the Apify MCP server. The Apify MCP server exposes Apify Actors as tools to external AI clients (such as Claude or Cursor). MCP connectors do the opposite: they let Actors running on the Apify platform call external MCP-compatible services. The two are independent and can be used together.

In this pipeline, we use the Notion Uploader Actor together with an Apify MCP connector to push the Google Indexer's results into a Notion database automatically after each run.

Setting up the Notion database

Create a dedicated database in your Notion workspace titled SEO Indexing Tracker.

Configure the database columns with the following properties:

Column Name	Property Type	Description
URL	Title	The target webpage URL submitted to Google
Submission Status	Select	`Indexed`, `Failed`, or `Quota Exceeded`
HTTP Code	Number	The status response code returned by Google (e.g., `200`, `429`, `403`)
Last Updated	Date	Timestamp of when the request occurred
Source Actor	Text	Name of the Apify Actor that executed the request

Authorizing the Notion MCP connector

MCP connectors are authorized once at the account level, not on individual Actor or task pages. To set one up:

Open Apify Console and go to Settings → API & Integrations → MCP connectors.
Click Add connector and select Notion.
Complete the OAuth flow — you will be redirected to Notion to authorize access to your workspace. The Apify platform never sees your token directly; authentication is handled server-side via the MCP proxy.
Once authorized, the connector appears as a selectable option when running any compatible Actor.

Connecting the Notion Uploader Actor

The Notion Uploader Actor reads any Apify dataset and uploads its rows into Notion. It accepts your MCP connector ID so you do not need to paste a Notion API key anywhere.

When the Notion Uploader runs as an Actor-to-Actor integration (triggered by the completion of the Google Indexer run), it automatically picks up the triggering run's output dataset — no explicit datasetId is needed in the payload template.

Here is an example input for the Notion Uploader when configured as a standalone run:

{
  "notionConnector": "<your-connector-id>",
  "notionDatabaseId": "https://www.notion.so/workspace/SEO-Indexing-Tracker-<db-id>",
  "datasetId": "<google-indexer-output-dataset-id>",
  "dedupeMode": "upsert",
  "keyProperty": "URL",
  "dryRun": false
}

Tip: Enable "dryRun": true on your first run. It resolves the field mapping, validates it against your Notion schema, and shows exactly what would be written — without creating any rows.

Orchestrating the full "set and forget" pipeline

Now that all three Actors are configured, we can chain them into an automated workflow: Fast Sitemap Generator discovers URLs, Google Indexer & Instant SEO Submitter submits them to Google, and Notion Uploader logs the results into Notion.

Step 1: Creating saved Actor tasks

An Actor task is a saved, reusable configuration of an Actor in Apify Console. Tasks let you pre-define all input parameters so they can be run on demand, via API, or on a schedule.

Crawler task: Open Fast Sitemap Generator in Apify Console, configure your startUrls, crawl depth, and regex filters, then click Save as a new task in the top-right corner. Name it sitemap-generator-task.
Indexer task: Open Google Indexer & Instant SEO Submitter, fill in your Google Cloud Service Account JSON key, then click Save as a new task. Name it google-indexer-task.
Uploader task: Open Notion Uploader, set your Notion MCP connector and target database URL, then click Save as a new task. Name it notion-uploader-task.

All saved tasks are listed under Saved tasks in the Apify Console left navigation.

Step 2: Chaining tasks using Actor-to-Actor integrations

The Integrations tab on each Actor or task page lets you chain runs together. When one task succeeds, Apify automatically triggers the next one and passes dynamic run data — such as the output dataset ID — via a payload template.

Chain 1: Sitemap Generator → Google Indexer

Open sitemap-generator-task in Apify Console and click the Integrations tab.
Click Add integration and select Run another Actor or task.
Choose google-indexer-task as the target.
Set the trigger to Run succeeded.
Set the payload template so the indexer receives the crawler's output dataset:

{
  "datasetId": "{{resource.defaultDatasetId}}",
  "action": "URL_UPDATED"
}

Chain 2: Google Indexer → Notion Uploader

Open google-indexer-task in Apify Console and click the Integrations tab.
Click Add integration and select Run another Actor or task.
Choose notion-uploader-task as the target.
Set the trigger to Run succeeded.
Leave datasetId out of the payload template — the Notion Uploader automatically uses the triggering run's dataset when no explicit ID is provided:

{
  "dedupeMode": "upsert",
  "keyProperty": "URL"
}

Now the full chain fires end-to-end on its own. When sitemap-generator-task completes, Apify triggers google-indexer-task with the crawler's dataset. When the indexer finishes, Apify triggers notion-uploader-task, which reads the indexer's output dataset and writes each result row into your Notion database.

Step 3: Scheduling automated pipeline execution

Configure a recurring schedule so the entire chain runs automatically.

In the Apify Console left sidebar, click Schedules.
Click Create new schedule.
Select sitemap-generator-task as the target. The full chain — crawl, index, upload to Notion — fires automatically when this task runs.
Set the frequency using a Cron expression or one of the presets.

For high-frequency publishing (news, job boards): Schedule daily at 2:00 AM UTC.
For standard marketing blogs and commercial sites: Schedule weekly, every Monday morning.

Cron expression example (daily at 2:00 AM UTC):
0 2 * * *

Best practices for automated SEO indexing

To maintain a healthy indexing strategy while avoiding Google API flags, keep these best practices in mind:

1. Submit only modified or new content

Do not resubmit your entire site inventory every day. The Google Indexing API is intended for pages that have new content (URL_UPDATED) or pages that have been deleted (URL_DELETED). Resubmitting unchanged pages wastes daily quotas and can result in rate limiting.

2. Match canonical URLs strictly

Google requires strict URL matching. If your canonical structure uses https://example.com/blog/post-name/ with a trailing slash, ensure your crawler filters and API submissions match that format exactly. Submitting http:// or non-trailing-slash variations can trigger redirect loops and indexing failures.

3. Use Test Mode during initial setup

Before running full-scale API operations, turn on the Test Mode toggle in the Google Indexer & Instant SEO Submitter Actor. Test Mode executes the full crawl, verifies JSON authentication keys, and simulates Notion MCP logging without consuming your daily Google API quota.

4. Monitor quota usage in Google Cloud Console

Standard Google Cloud projects receive an initial quota of 200 Indexing API requests per day. If your site publishes hundreds of new pages daily, submit a quota increase request directly through the Quotas tab in Google Cloud Console.

Wrapping up

Building an automated technical SEO pipeline transforms indexing from a passive waiting game into a predictable, programmatic system.

Combining the cloud crawling capabilities of the Fast Sitemap Generator, the push notifications of the Google Indexer & Instant SEO Submitter, and real-time database logging through Notion MCP Connectors, you establish an automated SEO engine that operates reliably in the background.

Instead of waiting weeks for Google to discover your newest content organically, your site updates are crawled, verified, submitted, and logged in Notion within minutes.

Explore the Fast Sitemap Generator on the Apify Store.
Set up the Google Indexer & Instant SEO Submitter Actor with your Google Cloud Service Account.
Connect your Notion workspace via MCP Connectors to start tracking real-time indexing status today.

How to Automate Instagram Reels to Notion via MCP

Emmanuel Uchenna — Wed, 09 Sep 2026 09:14:53 +0000

Managing short-form video content across multiple campaigns often turns into an administrative nightmare. Your marketing team publishes dozens of public Reels, short video posts, and IGTV clips every month. Everything looks great on Instagram, but as soon as someone asks to archive these assets into a centralized brand library or content calendar, the entire operation grinds to a halt.

Suddenly, you are forced to rely on shady, ad-heavy download websites that fail half the time, paste individual links into manual file converters, or copy caption text into spreadsheets by hand. Even when you do manage to extract a direct video file URL from Instagram, that URL expires within hours due to Instagram's content delivery network (CDN) session restrictions.

What if you could automatically extract high-quality Instagram video files, store them permanently, and stream that structured data straight into Notion in real time?

With Instagram Video and Reel Downloader, the Notion Uploader Actor, and Apify Model Context Protocol (MCP) connectors, you can build a zero-maintenance, automated pipeline. This guide walks you through setting up an automated Instagram-to-Notion video content calendar that automatically syncs dataset outputs upon run completion without requiring manual CSV exports or fragile custom backend glue code.

The content archiving nightmare

Picture this scenario. You are a creative organization, and your creative team just finished a quarter-long influencer campaign. Thirty public Instagram Reels and video posts were published. Your client or department head wants every asset logged inside Notion with direct video links, preview thumbnails, shortcodes, captions, and publishing dates for future repurposing.

The traditional approach to this task usually follows one of two painful paths:

The manual downloader route: A team member spends hours copying Instagram links, pasting them into third-party downloader sites, closing popup ads, downloading .mp4 files locally, uploading those files into Google Drive or Notion, and typing metadata by hand. This process consumes dozens of work hours, produces inconsistent video resolutions, and risks downloading malware.
The legacy scraping route: A developer sets up a web scraper to pull Instagram data. The scraper runs for 10 minutes, completes its execution, and dumps a static JSON or CSV file to cloud storage. A developer then has to write custom webhooks or cron scripts to parse that output file, format the payload, and send REST API requests to Notion. If any record fails or if the script breaks mid-transit, debugging becomes a headache.

flowchart LR
    A["Scraper / Web Downloader"] --> B["CSV / JSON Export"]
    B --> C["Custom Glue Script / Manual Upload"]
    C --> D["Notion Database"]

Neither approach is ideal for growing teams or production workflows. Legacy scraping forces you to write custom integration pipelines to move data downstream. Meanwhile, manual downloading fails to scale beyond a handful of links.

Apify Actor integrations and Apify MCP connectors solve this problem seamlessly. Rather than manually writing custom export scripts or handling fragile API glue, you can attach the supported Notion Uploader Actor directly to your extraction workflow inside Apify Console.

flowchart LR
    A["Instagram Reels"] --> B["Instagram Video and Reel Downloader"]
    B -- On Run Succeeded --> C["Notion Uploader Actor Integration"]
    C -- Apify MCP Connector / API Key --> D["Notion Content Calendar"]

Once Instagram Video and Reel Downloader finishes extracting video assets and storing permanent .mp4 file links in Apify Key-Value Store, Apify automatically executes the Notion Uploader integration. Notion Uploader connects securely to your Notion workspace using an Apify MCP connector (configured in Apify Console under Settings → Integrations → MCP connectors) or a Notion API key, mapping fields and creating structured database rows automatically.

The pipeline architecture: how MCP connectors change the game

To understand why this architecture is so powerful, let's examine what happens under the hood during an automated run.

flowchart TD
    subgraph APIFY["APIFY PLATFORM"]
        A["Instagram /reel/ URLs"] --> B["Dual-Path Extraction Engine"]
        B --> C["Apify Key-Value Store<br/>(Permanent Video URL)"]
        B --> D["Apify Dataset<br/>(Structured Metadata)"]
    end

    D -- On Run Succeeded --> E["Notion Uploader Actor<br/>(filip_cicvarek/notion-uploader)"]
    E -- Apify MCP Connector / API Key --> F["Notion Content Calendar<br/>(Automated Database Rows)"]

1. Dual-path extraction engine

Instagram does not provide direct, permanent video download links through a simple public REST API. Furthermore, different post types behave differently:

Some standard post pages expose media tags within initial static HTML.
Complex Reel pages render content dynamically via JavaScript, obscuring video source URLs behind client-side network calls.

Instagram Video and Reel Downloader handles this seamlessly using a dual-path architecture. It first attempts a lightweight static HTTP request to extract the media link instantly. If the target page requires dynamic execution, the Actor automatically falls back to a browser session to capture media delivery requests.

2. Solving CDN URL expiration via Key-Value Store

Direct video URLs returned straight from Instagram's servers (*.cdninstagram.com or *.fbcdn.net) are session-bound and expire after a short period. If you paste a raw CDN video link into Notion, the video will stop playing within 24 to 48 hours.

To solve this, Instagram Video and Reel Downloader includes a storeVideoFiles configuration option in Apify Console. When set to true, the Actor streams the raw .mp4 file directly into Apify Key-Value Store during the run. This generates a permanent, publicly accessible URL (videoPublicUrl) that Notion can embed reliably forever.

3. Automated dataset push via Notion Uploader and MCP connectors

Model Context Protocol (MCP) is an open standard that allows tools and AI systems to exchange context and perform actions securely across application boundaries. In the Apify ecosystem, MCP connectors allow Actors like Notion Uploader to communicate directly with external services - such as Notion, HubSpot, Google Sheets, or Slack.

By configuring an Actor Integration on Instagram Video and Reel Downloader set to trigger on Run succeeded, Apify automatically hands off the generated dataset to Notion Uploader. Notion Uploader reads the structured items and uses the configured Notion MCP connector (or Notion API key) to map and upsert rows directly into your Notion database.

4. Zero-trust token security and governance

When connecting third-party services to web automations, storing API tokens in raw scripts or public Actor inputs poses severe security risks. Apify handles MCP authentication via OAuth 2.0 proxies managed centrally inside Apify Console under Settings → Integrations → MCP connectors.

Your Notion authorization token is never exposed in plain text, never saved in Actor dataset output, and never shared with the underlying script environment. The Apify platform handles identity verification and permission scope tokens securely behind the scenes.

Step 1: Setting up the Notion database

Before configuring the Actor, you need a designated Notion database structured to store video metadata and file references.

Open your Notion workspace and create a new database page (select Full page database or Inline database).
Name your database Instagram Content Calendar.
Configure the database properties to match the schema exported by Instagram Video and Reel Downloader:

Property Name	Notion Property Type	Purpose / Payload Source
Title	Title	Stores post caption snippet or title (`title` / `description`)
Instagram Link	URL	Direct link to original Instagram post (`canonicalUrl`)
Direct Video Link	URL	Permanent Apify storage link (`videoPublicUrl`)
Thumbnail	Files & media (or URL)	Preview thumbnail image (`thumbnailUrl`)
Shortcode	Text	Instagram unique ID (`shortcode`)
Media Type	Select	Post type (`reel`, `post`, or `tv`)
Fetch Date	Date	Timestamp of processing (`fetchedAt`)
Status	Status / Select	Processing result (`ready` or `error`)

erDiagram
    INSTAGRAM_CONTENT_CALENDAR {
        string Title "Summer Campaign Teaser"
        string Instagram_Link "instagram.com/reel/C123abc/"
        string Direct_Video_Link "api.apify.com/.../video-c123.mp4"
        string Media_Type "reel"
        string Status "ready"
    }

Ensure your Notion database view is set up to display URL and media properties clearly so your creative team can preview videos directly within the calendar workspace.

Step 2: Configuring Instagram Video and Reel Downloader

With your Notion database ready, navigate to Instagram Video and Reel Downloader on Apify Store.

Click Try for free or open the Actor inside Apify Console. Select the Input tab to configure your run.

Input JSON configuration

To ensure video URLs remain active inside Notion indefinitely, you must set storeVideoFiles to true. This instructs the Actor to download the binary media file into the Apify Key-Value Store and produce the videoPublicUrl parameter.

Here is an ideal production configuration snippet:

{
  "startUrls": [
    { "url": "https://www.instagram.com/reel/C8XyZ123abc/" },
    { "url": "https://www.instagram.com/p/C9AbC456def/" },
    { "url": "https://www.instagram.com/reel/C-Ghi789jkl/" }
  ],
  "storeVideoFiles": true,
  "storeThumbnailFiles": true,
  "includePageHtml": false,
  "filenamePrefix": "notion-archive-2026",
  "maxConcurrency": 2,
  "maxRequestRetries": 2,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Key configuration settings explained

startUrls: An array of target public Instagram URLs. Supports /reel/, /p/, and /tv/ paths.
storeVideoFiles: Set to true. This generates a permanent videoPublicUrl host link on api.apify.com, avoiding expired CDN link errors inside Notion.
storeThumbnailFiles: Set to true if you want stored key-value records for thumbnail images as well.
filenamePrefix: A custom string prepended to all stored files inside Apify storage (e.g., notion-archive-2026).
maxConcurrency: Set to 2 or 3. Instagram actively monitors aggressive parallel scraping. Keeping concurrency low ensures reliable extraction without triggering rate limits.
proxyConfiguration: Enabling Apify Proxy distributes requests across residential or datacenter IP pools, preventing unexpected block screens.

Step 3: Wiring the Notion Uploader integration in Apify Console

Once your input JSON is defined for Instagram Video and Reel Downloader, you attach the Notion Uploader Actor integration directly to the run configuration inside Apify Console.

1. Creating your Notion MCP connector (or API key)

Before attaching the integration, establish your connection to Notion inside Apify Console:

In Apify Console, open Settings → API & Integrations → MCP connectors.
Click Add connector and select Notion.
Complete the one-click OAuth authorization window to grant Apify access to your Notion workspace and database.
Apify Console generates a secure connector ID (e.g. mcp-connector-...) that allows Actors to interact with Notion with your permissions, without ever exposing raw private tokens.

(Alternative: You can also use a classic internal integration secret starting with ntn_ created at notion.so/profile/integrations. Be sure to invite your integration to the Notion database via *••• → Connections*).

flowchart LR
    A["Apify Console Settings"] --> B["MCP Connectors"]
    B --> C["Add Notion Connector (OAuth)"]
    C --> D["Secure Connector Session ID"]

2. Adding the Notion Uploader integration to your Actor

With your Notion MCP connector ready, wire the integration to Instagram Video and Reel Downloader:

Open Instagram Video and Reel Downloader in Apify Console.
Select the Integrations tab located in the navigation panel below the Actor title.
Click Add integration. Under Suggested for this Actor, click Notion Uploader (filip_cicvarek/notion-uploader). (Note: Apify Console also lists third-party integration routes like Make and Zapier under "Integrate with third-party tools", but Notion Uploader provides the direct, native Apify Actor integration path).
Set Start when to Run succeeded. This ensures Notion Uploader fires automatically the moment your Instagram extraction run completes cleanly.

3. Mapping fields and configuring Notion Uploader input

Under the Actor input section for the Notion Uploader integration, configure the target database and field mapping:

Notion connector: Select your authorized Notion MCP connector from the dropdown list (or paste your notionApiKey).
Notion database: Paste your Notion database URL or ID (notionDatabaseId).
Explicit field mapping (dataMapping): Map the dataset output fields from Instagram Video and Reel Downloader to your Notion database properties:

{
  "Title": "description",
  "Instagram Link": "canonicalUrl",
  "Direct Video Link": "videoPublicUrl",
  "Thumbnail": "thumbnailUrl",
  "Shortcode": "shortcode",
  "Media Type": "mediaType",
  "Fetch Date": "fetchedAt"
}

Duplicate handling (rerun safety): Set Dedupe mode (dedupeMode) to upsert and set Key property (keyProperty) to Instagram Link (or Shortcode). This guarantees that re-running the Actor with previously processed URLs will update existing database rows instead of creating duplicate entries.

Click Save integration. The integration is now active and bound to your Actor task.

Step 4: Executing the run and verifying the content calendar

With the input parameters set and the Notion Uploader integration saved, click the Start button in the top-right corner of Apify Console.

Watching automated workflow execution

As Instagram Video and Reel Downloader runs, you can monitor the progress through the Log tab:

The Actor extracts public Instagram video posts, Reels, and IGTV links specified in startUrls.
For each URL, it resolves media endpoints and streams binary .mp4 files into Apify Key-Value Store, generating permanent videoPublicUrl host links.
Integration handoff: When the extraction run finishes successfully (Run succeeded), Apify immediately launches Notion Uploader (filip_cicvarek/notion-uploader), passing the run's default dataset ID automatically.
Notion Database update: Notion Uploader uses your Notion MCP connector to read the dataset items, validate properties against your database schema, apply upsert deduplication, and populate your Instagram Content Calendar in Notion.

sequenceDiagram
    autonumber
    participant Downloader as Instagram Video Downloader
    participant KVS as Key-Value Store
    participant Integration as Notion Uploader Integration
    participant Notion as Notion Database

    Note over Downloader: Downloader starts execution
    Downloader->>KVS: Extracts Reel & uploads .mp4 file
    Note over Downloader: Run completes (Status: SUCCESS)
    Downloader->>Integration: Triggers Notion Uploader automatically
    Integration->>Notion: Reads dataset & upserts database rows
    Note over Notion: Content calendar fully updated!

If an invalid or private URL is encountered, the extraction Actor records a structured dataset item containing an errorCode (such as NO_VIDEO_URL) and errorMessage. Notion Uploader writes an error status row into Notion, maintaining a complete audit log without breaking the pipeline execution.

Extending the workflow with API triggers and scheduling

Building a one-off run inside Apify Console is great for testing, but real content operations require automated execution. You can automate this pipeline using code triggers or scheduled jobs.

Programmatic execution via JavaScript / TypeScript

You can trigger the integrated pipeline programmatically using the official apify-client SDK in Node.js or TypeScript by invoking your saved task (which includes the Notion Uploader integration):

import { ApifyClient } from 'apify-client';

// Initialize the client with your Apify API token
const client = new ApifyClient({
  token: 'YOUR_APIFY_API_TOKEN',
});

// Define input parameters with video storage enabled
const input = {
  startUrls: [
    { url: 'https://www.instagram.com/reel/C8XyZ123abc/' },
    { url: 'https://www.instagram.com/reel/C9AbC456def/' }
  ],
  storeVideoFiles: true,
  storeThumbnailFiles: true,
  filenamePrefix: 'weekly-content-sync',
  maxConcurrency: 2,
  proxyConfiguration: {
    useApifyProxy: true
  }
};

async function runInstagramSync() {
  console.log('Starting Instagram to Notion sync workflow...');

  // Call the saved task (or Actor). Configured integrations run automatically on completion.
  const run = await client.task('YOUR_SAVED_TASK_ID').call(input);

  console.log(`Run completed! View dataset at: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
  console.log('Notion Uploader integration triggered automatically.');
}

runInstagramSync();

Programmatic execution via Python

If your infrastructure runs on Python, use apify-client to trigger the sync:

from apify_client import ApifyClient

# Initialize client
client = ApifyClient("YOUR_APIFY_API_TOKEN")

# Prepare input dictionary
run_input = {
    "startUrls": [
        {"url": "https://www.instagram.com/reel/C8XyZ123abc/"},
        {"url": "https://www.instagram.com/reel/C9AbC456def/"}
    ],
    "storeVideoFiles": True,
    "storeThumbnailFiles": True,
    "filenamePrefix": "python-content-sync",
    "maxConcurrency": 2,
    "proxyConfiguration": {
        "useApifyProxy": True
    }
}

# Run the saved task with Notion integration attached
print("Launching Instagram Video Downloader with Notion Uploader Integration...")
run = client.task("YOUR_SAVED_TASK_ID").call(task_input=run_input)

print(f"Workflow complete. Dataset ID: {run['defaultDatasetId']}")

Automated set-and-forget scheduling

To eliminate manual execution:

Save your input configuration and Notion Uploader integration inside Apify Console as a Saved Task (e.g., Instagram-to-Notion-Sync-Task).
Navigate to the Schedules tab in Apify Console and click Create schedule.
Link the schedule to your saved Actor task.
Set a cron cadence, such as every Friday at 5:00 PM (0 17 * * 5).

Every week, Apify will automatically run Instagram Video and Reel Downloader, extract new video links, upload permanent .mp4 binaries to Key-Value Store, and trigger Notion Uploader to push the assets into your Notion database.

Wrapping up

Relying on ad-ridden video download websites, manual file management, and fragile spreadsheet uploads is no longer necessary for modern media operations.

By combining the extraction capabilities of Instagram Video and Reel Downloader with the Notion Uploader Actor and Apify MCP connectors, you can build a seamless content archiving pipeline. Permanent video hosting via Apify Key-Value Store eliminates broken CDN links, while automated Actor integrations push structured data straight into Notion without requiring custom backend glue code.

Ready to automate your social media asset pipeline?

Try Instagram Video and Reel Downloader on Apify Store today, connect your Notion workspace via Notion Uploader, and build your automated content calendar in minutes.

Headless browsers vs. SERP APIs: Why your Playwright scraper fails in production

Emmanuel Uchenna — Wed, 09 Sep 2026 09:14:32 +0000

Every developer who builds web scrapers goes through the same initial rite of passage. You need search engine data for an SEO tracking tool, an AI retrieval pipeline, or a market research project. Your first instinct is to reach for a browser automation tool like Playwright, Puppeteer, or Selenium.

You write a quick script, launch a headless Chromium instance, navigate to Google, inspect the Document Object Model (DOM), and extract text from <h3> tags and organic result containers.

On your local development machine, it feels like magic. The script runs cleanly for your first five test queries.

Then, you deploy your script to a production server on AWS or DigitalOcean, and everything breaks down. Within minutes, your scraper encounters roadblock CAPTCHAs, gets flagged by automated anti-bot systems, or silently returns empty arrays because the search engine rolled out an unannounced layout tweak.

Environment	Execution Path	Production Outcome
Localhost (Prototyping)	Playwright ==> Residential Home IP ==> Google	🟢 200 OK (5 results parsed successfully)
Cloud Server (DIY)	Playwright ==> Datacenter Cloud IP ==> Google	🔴 429 Blocked / CAPTCHA Wall (Scraper fails)
Managed API (SearchApi)	App Worker ==> HTTP GET ==> SearchApi Infrastructure	⚡ 200 OK (Instant structured JSON payload)

While headless browsers are exceptional tools for end-to-end web testing and scraping low-security websites, using them to scrape Search Engine Results Pages (SERPs) is an expensive engineering trap.

In this guide, we break down the hidden operational costs of DIY browser automation, compare side-by-side Python implementations, and explore why switching to a dedicated SERP API like SearchApi.io saves your engineering team time, infrastructure budget, and maintenance headaches.

The real cost of doing it yourself

Developers naturally enjoy building systems from the ground up. However, building a custom search engine scraper means entering an ongoing cat-and-mouse game against some of the most sophisticated anti-bot security teams in the world.

When you scale a headless browser scraper beyond a handful of casual requests, you immediately run into four severe architectural bottlenecks:

flowchart LR
    Root["DIY Browser Scraping Bottlenecks"] --> B1["1. Proxy Infrastructure Tax<br/>High residential bandwidth costs"]
    Root --> B2["2. Fragile Selector Maintenance<br/>Random CSS class changes break code"]
    Root --> B3["3. Server Resource Drain<br/>150MB+ RAM per tab, CPU spikes"]
    Root --> B4["4. Anti-Bot Fingerprinting<br/>TLS, Canvas, WebGL, JA3 detection"]

1. The proxy infrastructure tax

Search engines can easily identify and block automated requests originating from data center IP ranges (such as AWS, Google Cloud Platform, or DigitalOcean). If you send five consecutive queries from an AWS EC2 instance without a proxy, your IP address is flagged almost immediately.

To bypass this restriction, you must purchase and configure a residential proxy network. Residential proxies route your requests through real residential consumer connections, making them appear legitimate. However, managing proxies introduces heavy technical complexity:

Bandwidth costs: Residential proxy providers charge based on data transfer, usually between $5 and $15 per gigabyte. When a headless browser loads a search results page, it downloads not just the text, but also JavaScript bundles, stylesheets, tracking pixels, and images. Loading these unnecessary assets inflates your proxy bandwidth bill significantly.
Pool rotation and stickiness: You must write custom pooling logic to rotate IP addresses across queries while maintaining sticky sessions when handling multi-page pagination.
Gateway latency: Routing traffic through multiple proxy hops adds anywhere from 1 to 4 seconds of network latency to every single request.

2. Fragile HTML selector maintenance

Search engines do not use semantic, human-readable HTML markup. Instead of providing clear class names like .search-result-item or .result-title, they rely on minified, randomized, or hashed CSS selectors that change frequently.

<!-- What you hope to see in the DOM -->
<div class="search-result">
  <h3 class="title">Web Scraping Guide</h3>
  <p class="snippet">Learn how to extract data...</p>
</div>

<!-- What Google actually renders -->
<div class="MjjYud">
  <div class="g Ww4FFb vt6AZc">
    <div class="kvH3df">
      <div class="VwiC3b yD755b xDu2fd">
        <span>Learn how to extract data...</span>
      </div>
    </div>
  </div>
</div>

If your code relies on fragile selectors like div.g or .VwiC3b, an unannounced A/B test or markup refresh by the search engine will cause your parser to fail silently. You will receive empty datasets, triggering alerts and requiring developers to drop their current sprint tasks to inspect DOM trees and update CSS queries.

3. Massive server resource drain

Headless browsers are full browser execution environments. Running an instance of headless Chromium or Firefox forces your host server to parse HTML, evaluate complex JavaScript scripts, construct DOM trees, and calculate CSS layout rules.

Memory consumption: A single headless browser tab typically consumes between 150MB and 300MB of RAM. If your application needs to handle 30 concurrent search queries, your server needs at least 8GB to 16GB of dedicated RAM just to keep the browser processes alive.
CPU spikes: Initializing browser contexts and rendering client-side JavaScript creates sharp CPU spikes, often requiring expensive multi-core cloud compute instances.
Process zombie leaks: Long-running browser processes frequently experience memory leaks. Without aggressive lifecycle management and process termination routines, orphaned Chromium processes will gradually consume all available server memory and crash your host container.

4. Advanced anti-bot fingerprinting and CAPTCHAs

Modern search engines employ sophisticated behavioral and cryptographic bot detection techniques. They do not just check your User-Agent header; they inspect deep browser characteristics:

TLS and JA3/JA4 fingerprinting: The way your networking stack negotiates SSL/TLS handshakes reveals whether your request comes from a genuine desktop browser or an automated runtime.
Navigator and JavaScript runtime properties: Automated environments often leak telltale attributes such as navigator.webdriver = true, missing system plugin arrays, or default WebGL vendor strings.
Canvas and audio fingerprinting: Scripts render hidden shapes or audio signals to generate a unique hardware signature.
Behavioral heuristics: Mouse movement velocity, keystroke timings, and scroll dynamics are analyzed in real time.

Even when using stealth plugins like puppeteer-extra-plugin-stealth or custom Playwright evasion flags, these countermeasures are brittle. Anti-bot vendors continuously update their detection models, leaving your custom scraper vulnerable to sudden blocks and CAPTCHA walls.

The abstracted alternative: Dedicated SERP APIs

A dedicated Search Engine Results Page (SERP) API transforms search engine data collection into a fully managed infrastructure service.

Instead of launching browsers, rotating proxies, handling CAPTCHAs, and maintaining brittle DOM selectors, you offload the entire operational pipeline to a specialized provider like SearchApi.io.

flowchart LR
    App["Your Application"] -->|1. Simple HTTP GET| API["SearchApi.io Engine"]

    subgraph ManagedInfra["Managed Infrastructure Layer"]
        API --> Proxies["Residential & Mobile Proxies"]
        API --> Browsers["Stealth Browser Fleet"]
        API --> Parsers["Auto-Updating DOM Parsers"]
    end

    ManagedInfra -->|2. Automated Request| Google["Search Engines (Google, Bing)"]
    Google -->|3. Live SERP Response| ManagedInfra
    API -->|4. Clean Structured JSON| App

SearchApi.io handles every layer of the extraction process behind a clean REST Application Programming Interface (API):

Intelligent proxy rotation: Automatically routes queries through optimized residential and mobile IP pools.
Fingerprint emulation: Emulates authentic browser signatures at both the network (TLS/HTTP2) and JavaScript runtime levels.
Automated parsing: Upstream parsers continuously monitor search engine layout updates. When Google changes a CSS class, SearchApi.io updates its internal parsers immediately, ensuring your application always receives structured, consistent JSON without code modifications.
Lightweight delivery: Your server sends a lightweight HTTP request and receives clean, parsed JSON in milliseconds, using minimal CPU and memory.

Side-by-side implementation: Playwright vs. SearchApi

To see the difference in code simplicity, maintainability, and resource footprint, let us compare two complete Python implementations designed to accomplish the exact same task: searching Google for "Top web scraping frameworks in 2026" and extracting the top organic results (position, title, URL, and snippet text).

Building a custom Playwright scraper in Python

The following script represents the DIY approach. It launches an automated Chromium browser, configures viewport parameters, attempts to bypass basic bot flags, waits for network idle states, and parses HTML using CSS selectors.

"""
DIY Google Search Scraper using Playwright
Requires: pip install playwright
Run: playwright install chromium
"""

import asyncio
from typing import Any, Dict, List
from playwright.async_api import async_playwright


async def scrape_google_diy(search_query: str) -> List[Dict[str, Any]]:
    print(f"Launching headless Chromium for query: '{search_query}'...")

    search_results: List[Dict[str, Any]] = []

    async with async_playwright() as p:
        # Launch Chromium with anti-detection flags
        browser = await p.chromium.launch(
            headless=True,
            args=[
                "--disable-blink-features=AutomationControlled",
                "--no-sandbox",
                "--disable-setuid-sandbox",
                "--disable-infobars",
                "--window-size=1920,1080",
            ],
        )

        # Configure browser context with realistic desktop parameters
        context = await browser.new_context(
            user_agent=(
                "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
                "AppleWebKit/537.36 (KHTML, like Gecko) "
                "Chrome/124.0.0.0 Safari/537.36"
            ),
            viewport={"width": 1920, "height": 1080},
            locale="en-US",
            timezone_id="America/New_York",
        )

        page = await context.new_page()

        # Format Google search URL
        encoded_query = search_query.replace(" ", "+")
        url = f"https://www.google.com/search?q={encoded_query}&hl=en"

        try:
            # Navigate and wait for DOM network idle state
            await page.goto(url, wait_until="domcontentloaded", timeout=30000)

            # Define fragile CSS selectors for organic search results
            organic_container_selector = "div.g, div.MjjYud"
            await page.wait_for_selector(
                organic_container_selector, timeout=7000
            )

            # Query all potential organic result cards
            containers = await page.query_selector_all(
                organic_container_selector
            )
            position = 1

            for container in containers:
                # Extract title from h3 tag
                title_elem = await container.query_selector("h3")
                # Extract link anchor tag
                link_elem = await container.query_selector("a")
                # Extract snippet text using known class heuristics
                snippet_elem = await container.query_selector(
                    "div.VwiC3b, div[style*='-webkit-line-clamp']"
                )

                title = (
                    await title_elem.inner_text() if title_elem else "N/A"
                )
                link = (
                    await link_elem.get_attribute("href")
                    if link_elem
                    else "N/A"
                )
                snippet = (
                    await snippet_elem.inner_text() if snippet_elem else "N/A"
                )

                # Filter out nested navigational links and empty blocks
                if (
                    title != "N/A"
                    and link.startswith("http")
                    and "google.com" not in link
                ):
                    search_results.append(
                        {
                            "position": position,
                            "title": title,
                            "link": link,
                            "snippet": snippet,
                        }
                    )
                    position += 1

                if len(search_results) >= 5:
                    break

        except Exception as error:
            print(f"Scraping failed due to selector timeout or block: {error}")
        finally:
            await context.close()
            await browser.close()

    return search_results


if __name__ == "__main__":
    query = "Top web scraping frameworks in 2026"
    results = asyncio.run(scrape_google_diy(query))

    print(f"\nRetrieved {len(results)} results via Playwright:")
    for item in results:
        print(f"[{item['position']}] {item['title']}\n    Link: {item['link']}")

The streamlined SearchApi approach

Now, let us examine the equivalent implementation using SearchApi.io. Notice how all browser lifecycle management, selector queries, and network evasions disappear.

"""
Production-ready Google Search Extraction using SearchApi
Requires: pip install requests python-dotenv
"""

import os
from typing import Any, Dict, List, Optional
from dotenv import load_dotenv
import requests

load_dotenv()


def scrape_google_with_searchapi(
    search_query: str,
    location: str = "United States",
    device: str = "desktop",
) -> List[Dict[str, Any]]:
    print(f"Requesting structured payload for: '{search_query}'...")

    api_key = os.getenv("SEARCHAPI_API_KEY")
    if not api_key:
        raise ValueError("Missing SEARCHAPI_API_KEY environment variable.")

    # SearchApi Google Search endpoint
    endpoint = "https://www.searchapi.io/api/v1/search"
    params = {
        "engine": "google",
        "q": search_query,
        "api_key": api_key,
        "location": location,
        "device": device,
        "hl": "en",
        "gl": "us",
    }

    try:
        response = requests.get(endpoint, params=params, timeout=15)
        response.raise_for_status()

        payload = response.json()
        organic_results = payload.get("organic_results", [])

        extracted_data = []
        for result in organic_results[:5]:
            extracted_data.append(
                {
                    "position": result.get("position"),
                    "title": result.get("title"),
                    "link": result.get("link"),
                    "displayed_link": result.get("displayed_link"),
                    "snippet": result.get("snippet"),
                }
            )

        return extracted_data

    except requests.exceptions.HTTPError as http_err:
        print(f"SearchApi HTTP error: {http_err.response.status_code} - {http_err.response.text}")
        return []
    except requests.exceptions.RequestException as error:
        print(f"Network request error: {error}")
        return []


if __name__ == "__main__":
    query = "Top web scraping frameworks in 2026"
    results = scrape_google_with_searchapi(query, location="New York, New York, United States")

    print(f"\nRetrieved {len(results)} results via SearchApi:")
    for item in results:
        print(f"[{item['position']}] {item['title']}\n    Link: {item['link']}\n    Display: {item['displayed_link']}")

Code walkthrough and key differences

Let us break down what happens behind the scenes in both codebases:

Metric	DIY Playwright Scraper	Managed SearchApi Solution
Code Complexity	~100 lines of boilerplate	~35 lines of standard Python
Dependencies	Playwright + Chromium binary (~300MB)	Standard HTTP client (`requests` < 5MB)
Memory Footprint (RAM)	150MB - 300MB per active tab	< 10MB per HTTP worker
Maintenance & Recovery	Manual DOM re-inspection on layout updates	Automatic upstream parser updates

Lines of code and complexity: The Playwright script requires over 90 lines of boilerplate, including browser process configuration, context options, selector query strings, element null-checks, and error catching. The SearchApi implementation requires under 35 lines of standard, clean Python.
Binary footprint: Running Playwright in a Docker container or serverless function requires downloading and bundling a full Chromium binary (~300MB). SearchApi runs with a lightweight HTTP client like requests or httpx (< 5MB total dependency size).
Data format integrity: In the Playwright snippet, if Google alters .VwiC3b to a new CSS class name, the script returns N/A for all snippets without throwing a visible network error. In contrast, SearchApi returns a normalized JSON object where fields like title, link, snippet, displayed_link, and sitelinks remain consistent regardless of Google's internal layout tests.

Technical performance and cost showdown

When deciding between building an internal headless browser cluster and integrating an API, you must evaluate total operational overhead across compute, proxies, reliability, and engineering salaries.

Detailed architectural comparison

The table below highlights the technical trade-offs between both approaches:

Feature / Metric	DIY Headless Browsers (Playwright / Puppeteer)	Dedicated SERP API (SearchApi)
Initial Setup Time	Days to weeks (proxy setup, evasion scripting)	5 minutes (API key integration)
Maintenance Overhead	High (ongoing selector patches and bot updates)	Zero (handled upstream by provider)
Memory Usage (RAM)	150MB - 300MB per concurrent tab	< 10MB per HTTP worker
CPU Utilization	High (JavaScript parsing and layout rendering)	Minimal (JSON deserialization)
Bandwidth per Query	1.5MB - 3.5MB (full HTML, CSS, JS assets)	15KB - 40KB (compressed JSON response)
Proxy Management	Manual (residential pools, rotation, sticky IPs)	Fully managed and automated
CAPTCHA Resolution	Requires third-party solvers or browser hooks	Handled automatically with 99.9% success
Data Format	Raw HTML / Unstructured DOM elements	Pre-parsed, structured JSON
Scalability	Complex (orchestrating browser clusters on Kubernetes)	Seamless (call API endpoints with high concurrency)
Availability / SLA	Unpredictable (prone to sudden bans)	Guaranteed 99.9% uptime SLA

Real-world cost breakdown at scale

Many engineering teams assume that DIY scraping is "free" because open-source tools like Playwright cost nothing to download. However, hosting and maintaining a high-volume scraper reveals significant hidden costs.

Let us model the estimated monthly expense of running 100,000 search queries per month across both architectures:

flowchart TD
    subgraph DIY["DIY Headless Browsers: ~$2,296 / month"]
        D_Proxies["Residential Proxies (200GB @ $8/GB)<br/>$1,600"]
        D_Eng["Engineering Maintenance (6 hrs @ $90/hr)<br/>$540"]
        D_Compute["Cloud Compute (2x 8GB Instances)<br/>$96"]
        D_Captcha["CAPTCHA Solver Service<br/>$60"]
    end

    subgraph API["Managed SERP API (SearchApi): $250 / month"]
        A_Tier["100k Searches (BigData Plan)<br/>$250"]
        A_Maint["Zero Infrastructure & Maintenance Overhead<br/>$0"]
    end

Proxy bandwidth amplification: Because headless browsers download client assets, 100k queries easily consume 200GB+ of residential proxy bandwidth. At $8 per GB, proxies alone cost $1,600 per month.
Server infrastructure: Running 20 to 30 concurrent Playwright instances requires dedicated virtual machines (e.g. AWS EC2 t4g.xlarge or c6i.xlarge), costing roughly $96 to $140 per month.
Engineering time: When selectors break twice a month, an engineer spends 3 to 4 hours diagnosing issues, testing selectors, and deploying hotfixes. At an average developer cost of $90 per hour, maintenance costs exceed $540 every month.
By contrast, using a managed SERP API (such as SearchApi's BigData Plan at $250 per month for 100,000 searches) consolidates your entire bill into a single, predictable subscription with a 99.9% SLA, saving over $2,000 every month while eliminating developer on-call fatigue.

Handling advanced search engine layouts and rich snippets

Modern search results are no longer simple lists of ten blue links. Search engines present dynamic, highly interactive features designed to answer user questions directly on the page:

People Also Ask (PAA): Accordion dropdowns that load dynamic content via asynchronous JavaScript calls upon being clicked.
Knowledge Graph panels: Rich sidebars containing entity facts, social profiles, founders, and related topics.
Google Maps and local packs: Interactive business listings with customer review ratings, operating hours, and geocoordinates.
AI Overviews and rich snippets: Dynamically streamed answers, product price tags, ratings, and video carousels.

flowchart TD
    SERP["Search Engine Results Page (SERP)"] --> AI["AI Overview / Rich Answer Box<br/>(Dynamic streaming content)"]
    SERP --> Main["Main Results Stream"]
    SERP --> Side["Entity Sidebar"]

    Main --> PAA["People Also Ask (PAA)<br/><i>(Interactive JavaScript accordions)</i>"]
    Main --> Maps["Local 3-Pack / Google Maps<br/>(Dynamic scroll & geocoordinates)"]
    Main --> Organic["Organic Search Results<br/>(Titles, URLs, sitelinks, snippets)"]

    Side --> KG["Knowledge Graph Panel<br/>(Entity attributes, social links, facts)"]

The challenge with DIY extraction of interactive elements

Extracting these advanced components with Playwright is remarkably complex:

To scrape People Also Ask questions, your script must locate each accordion element, issue click actions to trigger JavaScript expand events, wait for animation frames to finish, and parse the newly inserted DOM nodes.
To scrape Local Maps packs, your script must interact with map containers, handle scrolling viewports, and extract coordinates embedded inside nested JavaScript payloads.
To scrape Knowledge Graph boxes, your selectors must account for dozens of entity variations (companies, celebrities, movies, books, and locations), each of which renders with different HTML tags.

Writing and maintaining code for all these edge cases requires hundreds of lines of brittle automation logic.

Extracting structured SERP data effortlessly

With SearchApi, extracting interactive widgets requires zero browser actions. The API automatically parses every SERP feature into clean, dedicated JSON objects:

{
  "search_metadata": {
    "id": "search_65df89bc12e4",
    "status": "Success",
    "created_at": "2026-08-29T05:12:00.000Z",
    "request_time_taken": 0.62,
    "parsing_time_taken": 0.22,
    "total_time_taken": 0.84
  },
  "ai_overview": {
    "markdown": "Playwright is an open-source automation library and framework created by Microsoft for end-to-end testing across Chromium, Firefox, and WebKit.",
    "reference_links": [
      {
        "title": "Playwright: Fast and reliable end-to-end testing",
        "link": "https://playwright.dev",
        "source": "playwright.dev"
      }
    ]
  },
  "knowledge_graph": {
    "kgmid": "/g/11f555cn8l",
    "knowledge_graph_type": "Software library",
    "title": "Playwright",
    "type": "Software library",
    "description": "Playwright is an open-source automation library for browser testing and web scraping developed by Microsoft.",
    "source": {
      "name": "Wikipedia",
      "link": "https://en.wikipedia.org/wiki/Playwright_(software)"
    },
    "website": "https://playwright.dev"
  },
  "related_questions": [
    {
      "question": "Is Playwright better than Selenium?",
      "answer": "Playwright offers faster execution, native auto-waiting, built-in async support, and modern dev tools integration compared to Selenium.",
      "source": {
        "title": "Playwright vs Selenium: Modern Web Automation",
        "link": "https://example.com/playwright-vs-selenium",
        "displayed_link": "https://example.com > playwright-vs-selenium"
      }
    }
  ],
  "organic_results": [
    {
      "position": 1,
      "title": "Playwright: Fast and reliable end-to-end testing",
      "link": "https://playwright.dev",
      "displayed_link": "https://playwright.dev",
      "domain": "playwright.dev",
      "snippet": "Playwright enables reliable end-to-end testing for modern web apps across all modern rendering engines.",
      "sitelinks": {
        "inline": [
          {"title": "Docs", "link": "https://playwright.dev/docs/intro"},
          {"title": "Python API", "link": "https://playwright.dev/python/docs/intro"}
        ]
      }
    }
  ]
}

Your data pipeline can instantly access payload["ai_overview"]["markdown"], payload["knowledge_graph"], or payload["related_questions"] without having to simulate user clicks or manage animation delays.

Decision matrix: Choosing the right tool for your project

Headless browsers and SERP APIs both have distinct places in modern software engineering. The key is applying the right tool to the right problem.

flowchart TD
    Start{"What are you scraping?"}

    Start -->|Search Engine Results<br/>Google, Bing, Yahoo, Baidu| SERPBranch["Use a Dedicated SERP API<br/>(e.g. SearchApi)"]
    Start -->|General Web Pages<br/>Internal tools, small blogs, custom apps| WebBranch["Use Headless Browsers<br/>(Playwright, Puppeteer)"]

    SERPBranch --> S1["Fast, pre-parsed structured JSON"]
    SERPBranch --> S2["Zero proxy or CAPTCHA management"]
    SERPBranch --> S3["Guaranteed 99.9% uptime SLA"]

    WebBranch --> W1["Full UI interaction & custom click flows"]
    WebBranch --> W2["Session & SSO authentication handling"]
    WebBranch --> W3["Visual testing & PDF/screenshot generation"]

When to use headless browsers (Playwright, Puppeteer, Selenium)

End-to-end application testing: Automating user journeys, validating form submissions, and testing web applications in CI/CD pipelines.
Scraping authenticated enterprise portals: Logging into internal corporate dashboards or web apps behind custom Single Sign-On (SSO) login systems.
Visual regression testing and screenshots: Capturing full-page screenshots, rendering PDFs, and verifying UI visual consistency.
Low-security, dynamic niche sites: Extracting content from single-page web applications (SPAs) that do not enforce strict anti-bot measures.

When to use a dedicated SERP API (SearchApi)

SEO rank tracking and SERP monitoring: Tracking organic rankings, featured snippets, and local search visibility for thousands of keywords daily.
AI agent web search and RAG pipelines: Providing real-time search capabilities and clean web context to Large Language Models (LLMs) without latency bottlenecks.
E-commerce price intelligence and Google Shopping: Extracting structured product pricing, seller ratings, and stock availability at scale.
Market research and competitor tracking: Monitoring news trends, brand mentions, and Knowledge Graph developments with guaranteed uptime.

Wrapping up

Building your own search engine scraper with Playwright feels like a quick win during prototyping. But as your project scales, the hidden costs of residential proxies, anti-bot defenses, fragile DOM selectors, and heavy server hardware turn a simple script into a continuous infrastructure burden.

Developer time is your team's most valuable asset. Spending hours debugging broken CSS selectors and configuring proxy rotation takes engineering focus away from building core features that deliver real customer value.

By shifting your search data collection to a managed service like SearchApi.io, you eliminate infrastructure complexity:

Replace 100+ lines of fragile automation scripts with a clean, single HTTP GET call.
Reduce cloud compute and proxy bandwidth expenses by over 80%.
Gain instant access to structured JSON data for organic results, People Also Ask questions, Knowledge Graphs, and local packs.
Enjoy guaranteed 99.9% uptime backed by enterprise-grade proxy rotation and automatic parser maintenance.

Stop wrestling with headless browser memory leaks and CAPTCHA blocks. Sign up for SearchApi, claim your free API credits, and start extracting reliable search engine data in minutes.

Building a real-time AI search agent with SearchApi and OpenAI

Emmanuel Uchenna — Fri, 21 Aug 2026 04:38:31 +0000

Large Language Models (LLMs) are remarkably capable, but they suffer from two fundamental flaws: knowledge cutoffs and hallucinations. Ask an offline model about a breaking news event, a shifting stock price, or a new library release that dropped this morning, and it will either confidently fabricate a wrong answer or admit it has no idea. To build production-grade AI tools, your models need eyes and ears on the live web; your models need to be in the know of happenings as they break.

While static Retrieval-Augmented Generation (RAG) works well for searching internal vector databases, it fails the moment an accurate answer requires a real-time Google search. That is where AI search agents come in. An autonomous search agent turns an LLM into a reasoning engine that can choose when to browse the web, pull down search engine results pages (SERPs), synthesize real-time data, and deliver accurate, fact-checked answers.

In this article, you will build a production-ready AI search agent from scratch using Python, OpenAI's tool-calling framework, and SearchApi.io as the live web data pipeline.

Why traditional web scraping fails for AI agents

When developers realize their AI agent needs Google data, the first instinct is usually to reach for raw browser automation tools like Playwright, Puppeteer, or BeautifulSoup. You write code to open a headless browser, navigate to Google, type a query, and scrape the HTML.

In a local testing environment, this approach works for about three requests. In production, it breaks immediately.

Building your own search engine scraper means constantly fighting multi-billion-dollar anti-bot security teams. Your script will quickly run into:

IP blocks and CAPTCHAs: Search engines rapidly flag and block data center IP addresses. Rotating your own proxy pool is expensive and time-consuming to maintain.
Fragile DOM structures: Search engines constantly shift their HTML layout and CSS classes. A script targeting div.g tonight will silently return empty data by tomorrow morning.
Compute overhead: Spinning up heavy browser environments (like headless Chromium) consumes significant server memory and CPU cycles, adding cost and latency to every query.

To keep AI agents fast and reliable, you need an abstraction layer that handles proxy rotation, bypasses CAPTCHAs, and returns structured data instantly. By offloading this infrastructure to SearchApi.io, you retrieve clean, pre-parsed JSON payloads using a lightweight HTTP request - at a fraction of the compute overhead.

System architecture

Before writing any code, it helps to understand how an autonomous tool-calling loop operates. This system does not blindly search the web on every turn. Instead, it follows a four-step conversation loop:

The intent check: The user asks a question. The LLM evaluates whether it can answer using its internal knowledge or whether it needs live web data.
The tool call trigger: If it needs live data, the LLM stops generating text and outputs a structured request to invoke the search_web tool.
The infrastructure layer: The Python script intercepts this request, queries SearchApi's Google engine, and returns the top organic search results as structured JSON.
The synthesis: The text snippets are injected back into the LLM's context window. The model reads the live web data and generates a grounded, hallucination-free response.

This architecture is efficient because the LLM acts as the decision-maker. It avoids unnecessary API calls for questions it can already answer accurately from its training data.

Prerequisites and project setup

To follow along with this tutorial, you will need three things:

Python 3.8+ installed on your machine.
An OpenAI API key to power the model's reasoning loop.
A SearchApi API key to handle live web access. You can sign up at SearchApi.io and get 100 free searches right out of the box - no credit card required.

Start by initializing the project folder and installing the required dependencies:

# Create and move into your project folder
mkdir ai-search-agent && cd ai-search-agent

# Install dependencies
pip install requests openai python-dotenv

Next, create a .env file in the root of your project directory to keep your API tokens secure:

OPENAI_API_KEY="your_actual_openai_api_key_here"
SEARCHAPI_API_KEY="your_actual_searchapi_api_key_here"

With the project environment ready, here is an overview of the full project structure you will build:

Building a Real-Time AI Search Agent/
├── ai-search-agent/           <- Python CLI Implementation
│   ├── search_helper.py       <- SearchApi wrapper function (Step 1)
│   ├── agent.py               <- OpenAI tool schema and ask_agent() (Step 2)
│   ├── run.py                 <- Orchestrator loop (Step 3)
│   ├── .env.example           <- Template for environment variables
│   └── requirements.txt       <- pip dependencies
└── realtime-search/           <- Production Next.js 14 Application
    ├── app/
    │   ├── api/
    │   │   ├── search/        <- Server-side SearchApi route handler
    │   │   └── synthesize/    <- Real-time OpenAI SSE streaming handler
    │   ├── globals.css        <- Design system tokens & glassmorphism
    │   ├── layout.tsx         <- Root layout & SSR theme script
    │   └── page.tsx           <- Agent dashboard container
    ├── components/            <- React UI components
    ├── hooks/                 <- Pipeline orchestration & history hooks
    ├── lib/                   <- Server utilities & TypeScript definitions
    └── README.md              <- Next.js application documentation

Step 1: Writing the search API helper function

The first module is a search_web() function that sends a query to SearchApi's Google Search engine and returns a compressed block of text containing the titles, snippets, and source links from the top organic results.

The key design decision here is to extract only three fields - title, snippet, and link - from SearchApi's organic_results array. This keeps the payload lightweight and prevents the LLM from wasting tokens on irrelevant metadata like ad blocks, knowledge panels, or navigation breadcrumbs.

Create a file called search_helper.py:

import os
import requests
from dotenv import load_dotenv

# Load API keys from environment variables
load_dotenv()
SEARCHAPI_KEY = os.getenv("SEARCHAPI_API_KEY")

def search_web(query: str, location: str = "United States") -> str:
    """
    Queries the SearchApi.io Google Search engine and returns a compressed
    string containing the titles, snippets, and links of organic results.
    """
    if not SEARCHAPI_KEY:
        return "Error: SearchApi API key missing from environment variables."

    url = "https://www.searchapi.io/api/v1/search"

    # Configure payload parameters according to the SearchApi documentation
    params = {
        "engine": "google",
        "q": query,
        "location": location,
        "api_key": SEARCHAPI_KEY
    }

    try:
        response = requests.get(url, params=params, timeout=10)
        response.raise_for_status()
        data = response.json()

        # Extract organic search results
        organic_results = data.get("organic_results", [])
        if not organic_results:
            return f"No relevant web search results found for: '{query}'"

        # Parse and stringify the results into a compact context block for the LLM
        formatted_results = []
        for index, result in enumerate(organic_results[:5], 1):  # Limit to top 5 results
            title = result.get("title", "No Title")
            snippet = result.get("snippet", "No Snippet Available")
            link = result.get("link", "")
            formatted_results.append(f"[{index}] {title}\nSnippet: {snippet}\nSource: {link}\n---")

        return "\n".join(formatted_results)

    except requests.exceptions.RequestException as e:
        return f"An infrastructure error occurred while querying the search API: {str(e)}"


# Quick standalone test execution
if __name__ == "__main__":
    test_query = "Who won the men's 100m sprint in the 2024 Paris Olympics?"
    print(f"Testing SearchApi wrapper with query: '{test_query}'...\n")
    print(search_web(test_query))

The organic_results[:5] slice is intentional. Feeding more than five results into the LLM context rarely improves answer quality and meaningfully increases token consumption and cost.

Step 2: Defining the LLM agent framework (tool calling)

With the data pipeline ready, you need to teach the LLM when and how to use it. This is done through OpenAI's tool calling (formerly called function calling) feature, which lets you define structured schemas that the model can invoke when it determines external data is required.

Create a file, and name it agent.py:

import os
import json
from openai import OpenAI
from search_helper import search_web
from dotenv import load_dotenv

load_dotenv()

# Initialize the OpenAI client
openai_client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

# Define the SearchApi tool schema for the model
search_tool_definition = {
    "type": "function",
    "function": {
        "name": "search_web",
        "description": (
            "Call this tool whenever you need up-to-date information, news, "
            "current events, or web data that occurred after your knowledge cutoff date."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": (
                        "The optimized search query string to look up on Google "
                        "(e.g. 'Tesla Model Y 2026 pricing specifications')."
                    )
                },
                "location": {
                    "type": "string",
                    "description": "Geographical location filter for search targeting. Defaults to 'United States'.",
                    "default": "United States"
                }
            },
            "required": ["query"]
        }
    }
}


def ask_agent(user_prompt: str):
    """
    Sends a query to the LLM along with the available search tool definition.
    The LLM determines autonomously whether it needs to execute a Google search.
    """
    messages = [
        {
            "role": "system",
            "content": (
                "You are a factual, precise AI search agent with live web access via SearchApi. "
                "Use the search_web tool when you need current or time-sensitive information."
            )
        },
        {"role": "user", "content": user_prompt}
    ]

    # Request model execution with the tools parameter exposed
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",        # Efficient, tool-capable model
        messages=messages,
        tools=[search_tool_definition],
        tool_choice="auto"          # Let the model autonomously decide whether to use the tool
    )

    response_message = response.choices[0].message
    tool_calls = response_message.tool_calls

    if tool_calls:
        print("Agent decision: 'I need to use SearchApi to look up live information on the web.'")
        return {"status": "tool_call_required", "data": tool_calls, "message_history": messages}
    else:
        print("Agent decision: 'I can answer this question from my existing knowledge.'")
        return {"status": "completed", "data": response_message.content}

The tool_choice="auto" parameter is the key to the agent's autonomy. It tells OpenAI to let the model decide whether external search is necessary rather than forcing a tool call on every request. Simple questions like "What is Python?" get answered directly, while questions about current events correctly trigger a SearchApi call.

Step 3: Stitching the search agent loop together

The final piece is the orchestrator - a runtime loop that ties everything together. It interprets the model's decision, calls search_web() with the model's chosen arguments, injects the live results back into the conversation history, and requests a final synthesized answer from OpenAI.

Create a file called run.py:

import json
from agent import ask_agent, openai_client
from search_helper import search_web


def run_agent_loop(user_prompt: str):
    """
    The orchestrator runtime loop. It prompts the agent, detects tool requests,
    executes the local SearchApi call, updates the conversation history, and
    generates the final web-informed answer.
    """
    print(f"\nUser: {user_prompt}")

    # First turn: ask the agent what it wants to do
    result = ask_agent(user_prompt)

    # Case A: The LLM answered immediately without needing the web
    if result["status"] == "completed":
        print(f"\nFinal answer:\n{result['data']}\n")
        return

    # Case B: The LLM explicitly requested a SearchApi tool call
    if result["status"] == "tool_call_required":
        tool_calls = result["data"]
        messages = result["message_history"]

        # Append the model's tool request to keep the chat history balanced
        assistant_msg = {
            "role": "assistant",
            "content": None,
            "tool_calls": [
                {
                    "id": tc.id,
                    "type": "function",
                    "function": {
                        "name": tc.function.name,
                        "arguments": tc.function.arguments
                    }
                } for tc in tool_calls
            ]
        }
        messages.append(assistant_msg)

        # Process each tool call (handles potential parallel calls from the model)
        for tool_call in tool_calls:
            function_name = tool_call.function.name
            function_args = json.loads(tool_call.function.arguments)

            print(f"Invoking tool: {function_name}() with arguments {function_args}")

            if function_name == "search_web":
                search_query = function_args.get("query")
                search_location = function_args.get("location", "United States")

                # Fetch structured data live from SearchApi.io
                raw_search_results = search_web(query=search_query, location=search_location)

                # Append the search result payload back to the chat history
                tool_response_msg = {
                    "role": "tool",
                    "tool_call_id": tool_call.id,
                    "name": function_name,
                    "content": raw_search_results
                }
                messages.append(tool_response_msg)
            else:
                print(f"Warning: model requested unknown tool: {function_name}")
                return

        # Second turn: send the full conversation history back to OpenAI
        # (This now contains: user query + agent tool call + SearchApi results)
        print("Synthesizing live search data into a comprehensive answer...")
        final_response = openai_client.chat.completions.create(
            model="gpt-4o-mini",
            messages=messages
        )

        print(f"\nFinal answer:\n{final_response.choices[0].message.content}\n")


# End-to-end system test
if __name__ == "__main__":
    # Test case 1: Requires SearchApi intervention (current events)
    run_agent_loop("What were the top 3 tech stock market trends in Q1 of 2026?")

    # Test case 2: Native knowledge execution (no web tool needed)
    run_agent_loop("Explain the fundamental difference between a SQL and NoSQL database in two sentences.")

Pay close attention to how the messages list is built across both turns. After the model requests a tool call, you must append the assistant's tool request and then the tool's response before sending the history back to OpenAI. Omitting either step causes the API to return an error - this is the most common breaking point for developers new to tool calling.

Live test drives

With all three files in place, run the agent with python run.py. Here is what a real execution looks like for each scenario.

Scenario A - Current events query:

User: What were the top 3 tech stock market trends in Q1 of 2026?

Agent decision: 'I need to use SearchApi to look up live information on the web.'
Invoking tool: search_web() with arguments {'query': 'top tech stock market trends Q1 2026'}
Synthesizing live search data into a comprehensive answer...

Final answer:
Based on recent web data, the top three tech stock market trends in Q1 2026 were:

1. AI infrastructure stocks surged as hyperscalers accelerated GPU procurement...
2. Semiconductor supply chain stabilization drove AMD and TSMC to multi-year highs...
3. Enterprise SaaS consolidation continued, with several major M&A announcements...

Scenario B - Native knowledge query:

User: Explain the fundamental difference between a SQL and NoSQL database in two sentences.

Agent decision: 'I can answer this question from my existing knowledge.'

Final answer:
SQL databases organize data into structured tables with predefined schemas and use
relational joins to link data across tables, making them ideal for consistent,
transactional workloads. NoSQL databases use flexible document, key-value, or graph
data models with no fixed schema, trading strict consistency for horizontal scalability
and performance on unstructured or rapidly evolving data.

The agent correctly identifies that the first query requires live web data and that the second can be answered from its own training knowledge - skipping an unnecessary API call entirely.

Demo: see the agent in action

The interactive demo below visualizes the exact agent loop described above - including the tool decision badge, the SearchApi result cards, and the synthesized answer panel.

Interactive Demos & Repositories:

🚀 Live Web Application: Test the full-stack agent live on Vercel at realtime-search.vercel.app.

🐍 Python CLI Repository: Clone the Python tutorial source code at Eunit99/ai-search-agent.

⚡ Next.js App Repository: Clone the production web application code at Eunit99/realtime-search.

Full-Stack Next.js Application: Located in realtime-search/ - features server-side API key isolation (/api/search), real-time OpenAI GPT-4o-mini SSE streaming (/api/synthesize), geo-targeting, and query history persistence.

Production best practices

Before shipping this agent to production, consider these refinements.

Filter low-quality snippets to save tokens

SearchApi's organic_results array occasionally includes results with very short or unhelpful snippets (e.g. login-gated pages or index-only entries). Add a simple length filter to skip any snippet under 40 characters:

if len(snippet) > 40:
    formatted_results.append(f"[{index}] {title}\nSnippet: {snippet}\nSource: {link}\n---")

Use geo-targeting for localized results

SearchApi supports a location parameter that localizes search results geographically. This is useful for agents that serve users across different regions, where price, news, or availability data differs by market. The search_web() function already exposes this parameter:

search_web(query="current petrol prices", location="Lagos, Nigeria")

Cache duplicate queries

In production, users frequently ask similar questions within a short time window. Caching the search_web() response avoids burning SearchApi credits on identical lookups. A Redis-backed wrapper keeps it simple:

import hashlib, redis

cache = redis.Redis(host='localhost', port=6379, db=0)
TTL_SECONDS = 300  # Cache results for 5 minutes

def search_web_cached(query: str, location: str = "United States") -> str:
    cache_key = hashlib.md5(f"{query}:{location}".encode()).hexdigest()
    cached = cache.get(cache_key)
    if cached:
        return cached.decode()
    result = search_web(query, location)
    cache.setex(cache_key, TTL_SECONDS, result)
    return result

Keep the system prompt specific

Your system prompt directly affects how often the model decides to call search_web. Be explicit about when the tool should and should not be used:

"Use the search_web tool for any question involving current events, prices, "
"people, companies, or anything that may have changed after your training cutoff. "
"Answer directly only for timeless facts, historical events, or technical concepts."

Wrapping up

You have built a production-ready AI search agent that solves the two fundamental limitations of offline LLMs: knowledge cutoffs and hallucinations on real-time data.

The architecture is intentionally minimal but extensible. The search_web() function can be swapped for any SearchApi engine (News, YouTube, Google Scholar, and more). The tool schema can be extended with additional tools for math, code execution, or database lookup. The agent loop can be wrapped in a web server, a Slack bot, or a voice assistant.

The key insight is that SearchApi handles the hard infrastructure work - proxy rotation, CAPTCHA solving, HTML parsing, and structured JSON delivery - so you can focus entirely on the reasoning layer on top of it.

Ready to build? Grab your free API key at SearchApi.io, check out the live web app on Vercel, clone the Python CLI repository from GitHub (ai-search-agent) or the full-stack web application from GitHub (realtime-search), and drop a comment describing what you build.

Have questions or improvements? Open an issue or submit a pull request on GitHub (Python CLI) or GitHub (Next.js App).

This post is sponsored by SearchApi.

How I automated hyper-local lead routing with Nextdoor Scraper and Slack

Emmanuel Uchenna — Tue, 04 Aug 2026 16:25:45 +0000

Picture this. A sales rep or local contractor sitting in their truck on a Tuesday morning, burning fuel driving 45 minutes across town to quote a home repair, only to discover the prospect is a renter who cannot authorize work. Meanwhile, three streets over, in a hidden, high-density homeowner neighborhood, local residents are actively posting on Nextdoor to ask for contractor recommendations.

Collecting hyper-local data into a weekly CSV export is like brewing a fresh pot of coffee and leaving it on the counter for two days before serving it to your sales team. By the time an analyst manually exports the dataset, filters target zip codes, and emails the report to reps, those golden neighborhood opportunities are cold and competitors have already booked the jobs.

However, extracting this hyper-local data from public sources has historically presented a severe operational challenge. Nextdoor is a rich ecosystem for local community context, but manually researching across hundreds of city pages is painfully slow. Worse still, traditional web scraping workflows often deposit extracted data into static CSV files or database tables that sit idle until someone manually reviews them days later. In high-intent lead generation, every hour of delay degrades lead momentum and lowers conversion rates.

To solve this problem, I built two complementary lead routing workflows around Nextdoor Scraper on the Apify platform. The first is a no-code Slack notification wired directly inside Apify Console. The second is an AI agent workflow that connects Nextdoor Scraper to Claude or Cursor via the Apify MCP server, letting an AI agent run the scraper, filter results, and push alerts to Slack through a single conversational prompt.

In this article, I will walk you through the extraction setup, the qualification filter logic, both Slack routing approaches, and exactly what each option looks like in practice.

1. The gap between data collection and sales action

For home service providers, such as plumbers, roofers, HVAC technicians, landscapers, and solar installers, speed and geographic precision are everything. Identifying an ideal target neighborhood is only half the battle. If your sales team receives demographic reports 48 hours after a data crawl finishes, the opportunity to run timely local direct mail campaigns, localized digital ad pushes, or door-to-door outreach in high-demand pockets has already started to slip.

+-------------------+      +--------------------+      +----------------------+
|  Public Nextdoor  | ---> |  Nextdoor Scraper  | ---> |  Data Dump (CSV/DB)  |
|    City Pages     |      |  (Manual Crawl)    |      |  (Sits Idle Days)    |
+-------------------+      +--------------------+      +----------------------+
                                                                  |
                                                                  v
                                                       [ Delayed Sales Action ]

The primary friction point in modern web scraping is not extracting the data - it is the latency between data acquisition and business action. Traditional workflows decouple extraction from execution:

A crawler runs and outputs thousands of raw records into key-value stores or CSV files.
An analyst exports the CSV and manually applies spreadsheet filters to isolate target zip codes.
The analyst formats a report and emails it to the sales lead.
The sales lead assigns territories to reps who finally begin outreach.

This multi-step, manual chain creates massive operational friction. By the time reps receive the target list, days have elapsed.

To eliminate this gap, I built two notification paths that both start from the same Nextdoor Scraper run, and both deliver results to Slack, but in different ways depending on your team's setup.

+-------------------+      +--------------------+      +---------------------+
|  Public Nextdoor  | ---> |  Nextdoor Scraper  | ---> |  Apify Dataset      |
|    City Pages     |      |  (Apify Platform)  |      |  (Structured JSON)  |
+-------------------+      +--------------------+      +---------------------+
                                                                  |
                          +--------------------------+------------+
                          |                          |
                          v                          v
             [ Path A: No-code ]        [ Path B: MCP server ]
             Slack notification         AI agent (Claude/Cursor)
             via Actor integration      calls Nextdoor Scraper,
             inside Apify Console       filters leads, posts to Slack

2. The pipeline architecture

The automated lead routing system operates across two stages: extraction and post-run notification.

sequenceDiagram
    autonumber
    participant ND as Public Nextdoor Pages
    participant NS as Nextdoor Scraper (Apify Platform)
    participant DS as Apify Dataset
    participant SL as Slack (#local-leads)
    participant AI as AI Agent (Claude / Cursor)

    ND->>NS: Fetch public city & neighborhood page HTML
    NS->>DS: Store structured place data (centroid, income, homeowners)
    Note over DS: Run completes. Dataset is ready.

    alt Path A: No-code Slack notification
        DS->>SL: Actor integration fires on run finish
        SL-->>DS: Notification delivered
    else Path B: MCP server + AI agent
        AI->>NS: call-actor via Apify MCP server (mcp.apify.com)
        NS->>DS: Actor runs and stores results
        DS->>AI: AI agent reads dataset items
        AI->>AI: Filter by homeowners >= 50%, income >= $40k
        AI->>SL: Post formatted alert via Slack bot or Slack MCP tool
    end

Stage 1: Extraction

The workflow begins by launching Nextdoor Scraper on the Apify platform. The Actor targets public Nextdoor city and neighborhood pages, bypassing the need for complex browser automation setups or raw HTML parsing scripts. It parses visible structured data and metadata, transforming disorganized web content into clean JSON records that include geographic coordinates, demographic metrics, resident counts, and references to community activity.

Stage 2: Notification routing

Once the Actor run finishes and the dataset is populated, there are two ways to get that data into Slack. The first requires no code at all. The second turns the entire pipeline into something an AI agent like Claude or Cursor can trigger, filter, and broadcast with a single prompt.

3. Step 1: Extracting targeted neighborhood demographics

To get started, we configure Nextdoor Scraper to target public city pages. Nextdoor public city pages display valuable summary metrics without requiring user logins or private authentication.

Input configuration

In Apify Console, open the Input tab for Nextdoor Scraper and supply your target city URLs along with crawl limits. Here is the JSON input configuration used for this pipeline:

{
  "startUrls": [
    { "url": "https://nextdoor.com/city/north-miami--fl/" },
    { "url": "https://nextdoor.com/city/bellaire--oh/" },
    { "url": "https://nextdoor.com/city/bellbrook--oh/" },
    { "url": "https://nextdoor.com/city/belle-center--oh/" }
  ],
  "downloadStartUrlsToHtmlSources": false,
  "maxRequestsPerCrawl": 20,
  "maxConcurrency": 2,
  "maxItemsPerSection": 25,
  "followNeighborhoodPages": true,
  "followBusinessDirectoryPages": false,
  "followAdditionalResourcePages": false,
  "saveHtmlSnapshots": false
}

Key demographic fields extracted

When Nextdoor Scraper processes a city page, it returns normalized fields ready for immediate analysis:

Field	Type	Description	Sample Value
`cityName`	String	Name of the target city or municipality	`"North Miami"`
`state`	String	Two-letter state abbreviation	`"FL"`
`centroid`	Object	Geographic center latitude and longitude	`{"latitude": 40.016, "longitude": -80.747}`
`residentsCount`	Integer	Total estimated residents in the area	`59854`
`averageAge`	Integer	Average resident age	`35`
`averageIncome`	String	Public average household income indicator	`"$41k"`
`percentageHomeowners`	String	Share of residents who own their homes	`"56%"`
`affordabilityScore`	Integer	Calculated local affordability index	`99`
`friendlinessScore`	Integer	Community friendliness score	`86`
`faqs`	Array	Structured FAQ question and answer pairs	`[{"question": "...", "answer": "..."}]`
`neighborhoods`	Array	Discovered public neighborhood links	`[{"title": "128th St", "url": "..."}]`

Sample output JSON

Here is a snippet of the structured dataset item generated by Nextdoor Scraper:

{
  "url": "https://nextdoor.com/city/north-miami--fl/",
  "finalUrl": "https://nextdoor.com/city/north-miami--fl/",
  "pageType": "city",
  "title": "North Miami, Florida | Nextdoor",
  "description": "Public city summary extracted from Nextdoor.",
  "cityName": "North Miami",
  "state": "FL",
  "centroid": {
    "latitude": 25.8901,
    "longitude": -80.1867
  },
  "residentsCount": 59854,
  "averageAge": 38,
  "averageIncome": "$48k",
  "percentageHomeowners": "54%",
  "affordabilityScore": 82,
  "friendlinessScore": 88,
  "neighborhoodCount": 26,
  "businessDirectoryCount": 54,
  "faqCount": 7,
  "accessStatus": "public"
}

Cost efficiency with Pay-Per-Event pricing

One major advantage of using Nextdoor Scraper on the Apify platform is its pricing structure. Nextdoor Scraper operates on a Pay-Per-Event (PPE) model.

Unlike traditional compute-hour billing where you pay for raw server uptime regardless of success, PPE charging means you are billed strictly for successfully extracted place records. If a network fluctuation or access block occurs on a page, you are not billed for that failed attempt. This ensures predictable margins when executing large-scale, automated lead discovery campaigns across thousands of zip codes.

4. Step 2: Filtering for contractor accountability and intent

To understand how conditional filtering works in practice, let us examine a real-world implementation executed during a regional contractor accountability platform pilot with Lake Rock Ventures.

The Lake Rock Ventures pilot scenario

Lake Rock Ventures, a venture growth studio, was launching a pilot platform designed to connect vetted local roofing and HVAC contractors with high-intent homeowners. The platform needed to identify suburban communities with high enough homeownership rates to justify targeted direct-response marketing. Sending contractor representatives into renter-heavy neighborhoods resulted in wasted labor costs and low conversion rates.

To solve this, we defined strict qualification parameters for acceptable lead routing:

Homeowner ratio (percentageHomeowners): Minimum of 50%. Homeowners make long-term property decisions; renters rarely purchase roof replacements or HVAC upgrades.
Community scale (residentsCount): Minimum of 1,000 residents to guarantee adequate market volume.
Income indicator (averageIncome): Must meet or exceed $40k to ensure purchasing power for home improvement projects.

Writing the mid-run evaluation logic

We implemented a custom evaluation function in JavaScript that processes each dataset record emitted by Nextdoor Scraper. The script parses formatted strings (such as "56%" or "$48k") into clean numerical values and tests them against our threshold rules.

/**
 * Evaluates whether an extracted Nextdoor dataset item qualifies as a high-intent local lead.
 * @param {Object} item - Structured record from Nextdoor Scraper.
 * @returns {Object} Qualification result containing boolean status and parsed metrics.
 */
function evaluateLeadQualification(item) {
    // Extract numerical value from percentage string e.g., "56%" -> 56
    const homeownerRate = item.percentageHomeowners 
        ? parseFloat(item.percentageHomeowners.replace('%', '')) 
        : 0;

    // Extract numerical value from income string e.g., "$48k" -> 48000
    let incomeValue = 0;
    if (item.averageIncome) {
        const rawIncome = item.averageIncome.replace('$', '').toLowerCase();
        if (rawIncome.includes('k')) {
            incomeValue = parseFloat(rawIncome.replace('k', '')) * 1000;
        } else {
            incomeValue = parseFloat(rawIncome);
        }
    }

    const residentCount = item.residentsCount || 0;

    // Qualification thresholds
    const MIN_HOMEOWNER_PERCENT = 50;
    const MIN_INCOME = 40000;
    const MIN_RESIDENTS = 1000;

    const isQualified = 
        homeownerRate >= MIN_HOMEOWNER_PERCENT &&
        incomeValue >= MIN_INCOME &&
        residentCount >= MIN_RESIDENTS;

    return {
        isQualified,
        metrics: {
            homeownerRate,
            incomeValue,
            residentCount,
            affordabilityScore: item.affordabilityScore || 'N/A',
            friendlinessScore: item.friendlinessScore || 'N/A'
        }
    };
}

By embedding this evaluation logic directly into our execution pipeline, every crawled page is screened within milliseconds of extraction. If isQualified evaluates to true, the pipeline proceeds to dispatch a real-time Slack notification.

5. Step 3 (Path A): Sending a Slack notification with no code

Apify Console includes a built-in Slack notification integration for every Actor. It requires no custom code, no webhook endpoint, and no token management. You set it up once directly on the Nextdoor Scraper Actor page and it fires automatically after every run.

Setting up the Slack notification in Apify Console

Open Nextdoor Scraper in Apify Console and click the Integrations tab in the Actor's top navigation.
In the Search integrations bar, type slack. You will see a Send notification via Slack option listed under Generic integrations.
Click Send notification via Slack. A panel appears with a Connect workspace button.
Click Connect workspace and authorize the Apify app to access your Slack workspace. Select the target channel, such as #local-leads.
Configure when the notification fires. The most useful trigger is Run succeeded, which posts a Slack message automatically each time a Nextdoor Scraper run completes successfully.
Save the integration. From this point forward, every completed run posts a notification to your chosen Slack channel without any additional code.

The notification includes the run status, a direct link to the dataset in Apify Console, and the Actor name. Your sales team gets an immediate signal that fresh neighborhood data is ready to review, without having to log in and check manually.

+--------------------+      +---------------------+      +------------------------------+
| Nextdoor Scraper   | ---> | Run succeeds        | ---> | Slack: #local-leads          |
| (Apify Console)    |      | (Apify Platform)    |      | "Your run finished. View      |
|                    |      |                     |      |  results in Apify Console."  |
+--------------------+      +---------------------+      +------------------------------+

This is the fastest path to getting Slack alerts from Nextdoor Scraper. It takes under two minutes to configure and works on every run automatically, including scheduled runs and saved tasks.

6. Step 4 (Path B): Using the Apify MCP server with an AI agent

Path A sends a basic completion notification. Path B goes further: it lets an AI agent such as Claude or Cursor actually run Nextdoor Scraper, read and filter the extracted dataset, and post a formatted, actionable Slack alert with specific neighborhood metrics - all from a single conversational prompt.

This is possible because Apify hosts a remote Model Context Protocol (MCP) server at https://mcp.apify.com. Once you connect an MCP client to that server, your AI agent gains access to tools such as call-actor, search-actors, and fetch-actor-details. It can run Nextdoor Scraper on demand and process the results directly inside the chat context.

What is the Apify MCP server?

The Apify MCP server exposes Apify Actors as callable tools to external AI clients. It is a programmatic interface that works with any MCP-compatible client, including Claude for Desktop, Cursor, VS Code with MCP support, and others.

This is distinct from MCP connectors, which work in the opposite direction (an Actor calling an external service). The Apify MCP server makes Actors available as tools your AI agent can invoke from outside Apify entirely.

Connecting the Apify MCP server to your AI client

You authorize once via OAuth browser flow. After that, your AI client can call Nextdoor Scraper directly.

Cursor - create or update .cursor/mcp.json in your project:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=actors,docs,eunit/nextdoor-scraper"
    }
  }
}

VS Code - add to .vscode/mcp.json or run MCP: Add Server from the Command Palette:

{
  "servers": {
    "apify": {
      "type": "http",
      "url": "https://mcp.apify.com/?tools=actors,docs,eunit/nextdoor-scraper"
    }
  }
}

Claude for Desktop - edit claude_desktop_config.json:

{
  "mcpServers": {
    "apify": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://mcp.apify.com/?tools=actors,docs,eunit/nextdoor-scraper"
      ]
    }
  }
}

The ?tools=actors,docs,eunit/nextdoor-scraper query parameter scopes the tools list to the Nextdoor Scraper Actor specifically, alongside general Actor search and documentation tools. On first connection, a browser OAuth flow opens so you sign in to your Apify account and approve access. No API token is pasted into the chat.

Running the pipeline from a prompt

Once configured, the entire extraction and filtering workflow becomes a single AI prompt. Here is an example of what you would type in Claude or Cursor:

Use Apify to run the Nextdoor Scraper on these four city pages: north-miami--fl, bellaire--oh, bellbrook--oh, and belle-center--oh. Once finished, read the dataset results. Filter for any city where percentageHomeowners is 50% or above and averageIncome is $40k or above. For each qualifying city, format a Slack message with the city name, homeowner share, average income, resident count, and affordability score, then post it to #local-leads.

The AI agent handles the full sequence:

Calls call-actor with the Nextdoor Scraper Actor ID and input configuration.
Waits for the run to complete and fetches the dataset items.
Applies your qualification filter logic across every result.
Formats a structured message for each qualifying neighborhood.
Posts the alert to Slack using either a connected Slack MCP tool or a Slack bot token you provide in the chat context.

+------------------+      +----------------------+      +------------------+      +------------------+
| AI Agent         | ---> | Apify MCP server     | ---> | Nextdoor Scraper | ---> | Apify Dataset    |
| (Claude/Cursor)  |      | mcp.apify.com        |      | (Actor run)      |      | (JSON results)   |
| call-actor tool  |      | (OAuth, no token     |      |                  |      |                  |
|                  |      |  in chat)            |      |                  |      |                  |
+------------------+      +----------------------+      +------------------+      +------------------+
         ^                                                                                  |
         |                           reads & filters dataset                               |
         +-----------------------------------------------------------------------------------+
         |
         v
+------------------+
| Slack #local-leads|
| Formatted alert  |
| per qualifying   |
| neighborhood     |
+------------------+

Why this approach works well for local lead routing

The AI agent approach is especially effective because filtering logic does not need to be written in advance or maintained as code. You adjust the criteria conversationally. If the roofing company decides to lower the homeowner threshold to 45% for a quieter week, you change one line in the prompt. No code deployment required.

The Apify MCP server also supports output schema inference, meaning Claude or Cursor understands the structure of the Nextdoor Scraper dataset before the run completes. That makes it easier for the agent to write accurate filter logic against fields like percentageHomeowners, averageIncome, and residentsCount without needing to inspect a raw JSON file first.

7. What the end result looks like in both paths

Path A: What the no-code Slack notification looks like

When a Nextdoor Scraper run finishes, the built-in Slack integration posts a notification automatically to your chosen channel. The message includes:

The Actor name and run status
A direct link to the dataset in Apify Console
The run duration and the number of items stored

Your sales team opens Slack, sees the notification, and clicks through to the dataset to review results. No CSV downloads, no email chains, no delayed handoffs.

For teams that run the scraper on a schedule (e.g. every Monday morning before the weekly sales meeting), this turns a manual check-in into a reliable, automated briefing.

Path B: What the AI agent prompt workflow looks like

With the Apify MCP server connected to Claude or Cursor, the agent runs the full pipeline from one prompt. Here is what a real output message from Claude might look like after processing a Nextdoor Scraper dataset with a 50% homeowner filter:

+--------------------------------------------------------------+
|  #local-leads                                       [Slack]  |
+--------------------------------------------------------------+
|  🏡 High-intent lead: North Miami, FL                        |
|  Homeowner share: 54%  |  Avg income: $48k                   |
|  Residents: 59,854     |  Affordability: 82/100              |
|  Nextdoor page: https://nextdoor.com/city/north-miami--fl/   |
|                                                              |
|  🏡 High-intent lead: Bellbrook, OH                          |
|  Homeowner share: 61%  |  Avg income: $52k                   |
|  Residents: 7,009      |  Affordability: 91/100              |
|  Nextdoor page: https://nextdoor.com/city/bellbrook--oh/     |
+--------------------------------------------------------------+

In the Lake Rock Ventures pilot, switching from manual CSV review to the Apify MCP server + Claude workflow reduced the time from data collection to campaign briefing from 48 hours to under 10 minutes, while narrowing outreach to high-ownership areas only.

7. Wrapping up

Static data collection is no longer sufficient for modern local marketing and lead generation operations. By replacing manual CSV exports with an automated pipeline powered by Nextdoor Scraper and Apify MCP connectors, you can bridge the gap between web data extraction and real-time sales execution.

With this architecture, you benefit from:

Hyper-local precision: Extracting structured resident counts, homeowner ratios, and income signals from public Nextdoor pages.
Predictable cost scaling: Leveraging Pay-Per-Event pricing to pay strictly for successful record extractions.
Zero glue-code overhead: Eliminating third-party integration tools and AWS Lambda scripts by using Apify MCP connectors.
Instant lead velocity: Routing high-intent local opportunities straight to your sales team's Slack channel in seconds.

Ready to automate your local lead routing? Try Nextdoor Scraper in the Apify Store, connect your Slack workspace via Apify MCP connectors, and start turning public neighborhood signals into real-time sales momentum today.

How to download public YouTube videos at scale with an Apify Actor

Emmanuel Uchenna — Thu, 18 Jun 2026 12:30:13 +0000

You open a YouTube page. The video is public. You need that video - or its transcript, or its metadata, or just a clean download link you can pass to the next step in your workflow.

So you search for a YouTube video downloader. You find a browser extension, a sketchy website, maybe a command-line tool. You paste the URL. It works - once. But you have twenty more URLs to process. Or you need to run this same task every Monday morning. Or three people on your team need to do the same thing without using three different methods and getting three different outputs.

That is where every basic downloader starts to fall apart.

YouTube Video Downloader is an Apify Actor built to solve exactly this problem. It processes public YouTube watch URLs, YouTube Shorts, and live replay links in a structured, repeatable, automatable way. You get download format URLs, video metadata, thumbnails, channel data, and optional transcript text - all in one clean dataset output per run.

This guide will show you exactly how it works, how to configure it, and why it is a better fit than generic one-off tools if your workflow involves more than one video or one person.

What the YouTube Video Downloader Actor does

Most downloader tools treat the task as simple: give them a URL and they give you a file. That is fine if that is truly all you need.

But most teams need a bit more. They need the title and description to track what they downloaded. They need the thumbnail to display in a content board. They need the transcript to feed into a summarization or search tool. And they need all of this in a format that works with the rest of their stack - not a browser download that lands in someone's desktop folder.

YouTube Video Downloader handles all of that in one run. Here is what it can return for each URL you give it:

Direct download URLs for available MP4 and WebM formats
Progressive (video + audio combined) and adaptive (video-only or audio-only) stream metadata
Video title, description, tags, and view count
Channel name and channel ID
Duration in seconds
Thumbnail URL (including the high-resolution version when available)
Media type - regular video, Short, or live replay
Public transcript text when available and enabled
A saved file in the run's key-value store when you use file export mode

The Actor runs on the Apify platform, which means you are not limited to a single manual session. You can run it from Apify Console, call it via API from your own application, schedule it on a recurring basis, or trigger it with a webhook. The output lands in a structured dataset you can export in JSON, CSV, or Excel.

Who this Actor is actually useful for

Before you decide whether this tool is worth using, it helps to know whether your situation matches how it was designed.

Content and editorial teams are a natural fit. If your team regularly works with public video assets - for clipping, repurposing, archiving, or editorial reference - you likely need more than the video file itself. You need the metadata alongside it. The Actor gives you both in a consistent format, run after run.

Agencies and operations teams benefit most from the repeatability. When different people handle different requests manually, the output is inconsistent. One person uses one site, another uses a different tool. The Actor standardizes the intake: one input format, one output structure, every time.

Developers and product teams often want a YouTube video downloader API - something they can call from inside their application or automation, not something they have to operate manually. The Actor exposes a full REST API and is also available through the JavaScript and Python client libraries. You can integrate it into any stack in a few lines of code.

Researchers and analysts who work with transcripts, topic metadata, or structured video data will find the Actor useful for building consistent datasets from public YouTube content. Instead of scraping each video manually, you can pass a list of URLs and get back a single structured export.

Why browser-based downloaders are not enough

It is worth being direct about this, because plenty of people reading this article have already tried the obvious alternatives.

Browser-based YouTube downloader sites work for one-off tasks. You paste a link, download a file, and you are done. But they are built for individual, manual use. They do not give you metadata. They do not store transcripts. They do not support API access. They do not let you schedule a run or trigger one programmatically. And they definitely do not output a structured dataset you can hand to the next step in a pipeline.

There is also the reliability issue. Many of these sites change frequently, break without warning, or quietly stop working after YouTube updates. If you build a team process around one of them, you are building on unstable ground.

The YouTube Video Downloader Actor approaches the problem differently. Direct format URLs returned by YouTube are time-limited - they expire. The Actor solves this by giving you two options: use the expiring URL for immediate access, or switch to save-best-progressive mode and have the Actor store a durable copy in Apify's key-value store. From there, you can fetch it via a stable URL for as long as you need it.

Step-by-step: how to download public YouTube videos with Apify

Here is exactly how to run the Actor and get results.

Step 1: Open the Actor in Apify Console

Go to the YouTube Video Downloader page in Apify Store. If you do not have an Apify account yet, you can sign up for free - no credit card required to start.

Once you are in, click Try for free to open the Actor in Apify Console. You will land on the Input tab, which is where you configure everything before running.

Step 2: Add your YouTube URLs

In the YouTube video URLs field, paste one or more public YouTube links. The Actor supports:

Standard watch links: https://www.youtube.com/watch?v=...
YouTube Shorts: https://www.youtube.com/shorts/...
Live replay links for completed live streams

You can add up to 100 URLs per run (controlled by the Max videos input field, which defaults to 10). If you have a longer list, you can run multiple jobs or call the Actor programmatically in batches.

Step 3: Choose your download mode

This is where the Actor becomes more powerful than a generic downloader. Under Download mode, you have two options:

metadata-only - The Actor returns format URLs, metadata, and optional transcript data. No file is saved. This is the default mode and the fastest way to get structured output. Use this when you only need the download link or metadata to pass downstream.
save-best-progressive - The Actor selects the best available progressive format (video + audio combined), downloads it, and stores the file in the run's default key-value store. You can then fetch it via a stable Apify URL.

Set your preferred quality (best, 720p, 1080p, etc.) and container (mp4, webm, or any) to guide which format gets selected when saving.

Step 4: Configure the rest of your input

A few more fields worth knowing:

Include transcript - Toggle this on if you need the public transcript text. When available, it gets included in the dataset output for each video. Useful for search indexing, summarization, and AI enrichment.
Include progressive formats - Returns combined video + audio formats in the output. On by default.
Include adaptive formats - Returns video-only and audio-only streams separately. On by default. Useful if you are working with adaptive streaming workflows.
Max formats per video - Limits how many format entries appear in the output. Default is 8. Increase this if you want to see the full list of available streams.
Max saved file size (MB) - A safety cap when using save-best-progressive mode. Defaults to 150 MB. Increase if you are working with longer videos and have confirmed the file will fit.

Here is an example input that covers most use cases:

{
  "startUrls": [
    { "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ" },
    { "url": "https://www.youtube.com/shorts/abc123" }
  ],
  "maxVideos": 10,
  "preferredQuality": "720p",
  "preferredContainer": "mp4",
  "includeProgressiveFormats": true,
  "includeAdaptiveFormats": false,
  "downloadMode": "metadata-only",
  "includeTranscript": true,
  "maxDownloadSizeMb": 150
}

Step 5: Start the run

Click Start in Apify Console. The Actor processes each URL in sequence, resolves the available formats, collects the metadata, and pushes one item to the dataset for each URL it handles.

You do not need to keep the browser open. Apify Console shows run status in real time, and you can come back when it is done. For large batches or scheduled runs, you will not be watching at all - the output will just be there when you need it.

Step 6: Review and use the output

Once the run finishes, open the Dataset tab. You will see one row per input URL. Each row has:

Field	What it gives you
`status`	`ready`, `downloaded`, or `error`
`videoId`	The normalized YouTube video ID
`title`	The video title from YouTube
`mediaType`	Video, Short, or live replay
`durationSeconds`	Runtime in seconds
`viewCount`	Public view count at the time of the run
`thumbnailUrl`	High-resolution thumbnail URL
`formats`	All returned format entries with stream URLs and quality labels
`transcript`	Public transcript text (if enabled and available)
`selectedDownload`	Key-value store key and public URL of saved file (if applicable)

You can export the dataset from the Export button as JSON, CSV, or Excel. If you saved files using save-best-progressive mode, they appear in the Storage tab under the run's key-value store.

How to integrate the Actor into your own code

If you want to call the Actor programmatically - for example, from an internal tool, a script, or a larger automation - the easiest way is through the official Apify client libraries.

JavaScript / TypeScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const run = await client.actor('eunit/youtube-video-downloader').call({
  startUrls: [{ url: 'https://www.youtube.com/watch?v=dQw4w9WgXcQ' }],
  downloadMode: 'metadata-only',
  includeTranscript: true,
  preferredQuality: '720p',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run = client.actor("eunit/youtube-video-downloader").call(run_input={
    "startUrls": [{"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ"}],
    "downloadMode": "metadata-only",
    "includeTranscript": True,
    "preferredQuality": "720p",
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Install the client with pip install apify-client (Python) or npm install apify-client (JavaScript/TypeScript). Your API token is in Apify Console under Settings > Integrations.

A note on legal and responsible use

Any good guide on YouTube video downloading should address this clearly.

The YouTube Video Downloader Actor is built for public YouTube content - videos that are publicly visible without login. That does not automatically mean every public video is fair game to download, archive, or redistribute.

YouTube's terms of service generally prohibit downloading videos without explicit permission from the platform or the content owner. There are exceptions for content you own, content licensed for download, and content in the public domain - but those are the exceptions, not the default.

If you are asking how to download YouTube videos legally, the clearest answer is: work with content you own, content you have been licensed to use, or content in the public domain. If you are unsure about a specific piece of content, check with a qualified legal advisor, not a downloader tool's FAQ.

For teams, this matters even more. A repeatable workflow that processes video at scale creates a larger footprint. Good compliance means having clear answers to: what content are we allowed to process, where does the output go, and who approved it.

The Actor helps you build a consistent, auditable workflow. The permissions question is something you need to answer before you run it.

Common use cases where the Actor earns its cost

If you are comparing this Actor against free browser-based tools, the question is not whether the free tool works for one video. It usually does. The question is what your workflow actually costs.

Content repurposing pipelines. If your team regularly turns public video into clips, summaries, or knowledge-base articles, you need the transcript and metadata alongside the media file. The Actor gives you all three in one run, in a structured format that fits your next tool.

Approved asset management. Agencies and media teams that manage approved public video assets across campaigns or clients need consistency. The Actor standardizes the output, so every URL processed the same way, by anyone on the team, gives the same fields.

AI and research workflows. Transcripts combined with metadata - title, tags, description, view count, duration - are a useful input for classification, topic modeling, or AI summarization pipelines. The Actor gives you a clean starting point for that kind of enrichment.

Product integration. If you are building an internal tool or product feature that involves YouTube video data, the Actor is a faster path than building and maintaining your own downloader infrastructure. You call an API, you get structured output, and you handle the hard part (your product logic) rather than YouTube format resolution.

Why automation changes the value equation

Here is a way to think about whether the Actor is worth paying for.

If you download five YouTube videos a month manually, the cost of manual work is low. A free tool is probably fine.

If you process fifty URLs every week across a team of three people, the math changes. Manual work is time, and time is money. If each URL takes five minutes to process by hand (finding the right tool, pasting the link, downloading the file, renaming it, noting the metadata, passing it to the next step), that is over four hours of manual work per week. At scale, or across a team, that adds up fast.

The real cost of manual tools is not the tool's price. It is the time spent, the inconsistency in output, and the fragility when the tool breaks or the website changes.

For teams that process video data regularly, the automation pays for itself quickly. And for teams that need scheduled or API-triggered runs - the kind of workflow that simply cannot be done with a manual tool - the comparison is not really about price at all.

Frequently asked questions

Can this Actor download YouTube Shorts?

Yes. The Actor supports public YouTube Shorts URLs. They are handled as a supported media type and included in the dataset output with the mediaType field set to Short.

Does it return direct download links?

Yes. When available, the Actor returns direct format URLs in the formats array. These URLs are time-limited, so use them promptly or switch to save-best-progressive mode to get a durable stored copy.

Can it save the video file, not just the URL?

Yes. Set downloadMode to save-best-progressive and the Actor will select the best available progressive format, download it, and store it in the run's key-value store. You can then access it via a stable public URL from the selectedDownload field.

Can it extract transcripts?

Yes, when the video has a public transcript and you enable the includeTranscript option. Not all videos have public transcripts, and in those cases the Actor returns a structured item without transcript data rather than failing the run.

Does it work for live replays?

Yes. Completed live streams that are publicly available on YouTube are supported as a media type. The Actor handles them the same way as regular videos.

Can I run this on a schedule or trigger it with a webhook?

Yes. Scheduled runs and webhook triggers are both available from Apify Console. You can also call the Actor via the REST API or the official client libraries to integrate it into any external system.

Why use this instead of a free downloader?

Because free downloaders are built for one-off, manual, single-user tasks. They do not return structured metadata. They do not support transcripts. They do not give you API access, scheduled runs, or structured exports. And they break without warning. If your workflow involves more than one video, more than one person, or more than a one-time task, a structured Actor on a stable platform is a significantly better fit.

Wrapping Up

If you have a workflow that involves downloading public YouTube videos - or extracting metadata, transcripts, thumbnails, or stream format data from them - YouTube Video Downloader is built for exactly that.

You can start with a single public URL, review the dataset output, and see in one run how much cleaner and more useful the output is compared to what you get from a browser-based tool.

From there, you can scale up to batches, schedule recurring runs, or connect it to your own code using the Apify JavaScript or Python clients.

Open YouTube Video Downloader on Apify →

How to download Instagram videos and Reels at scale with Apify

Emmanuel Uchenna — Wed, 17 Jun 2026 11:27:09 +0000

Picture this. Your marketing team just wrapped a three-month creator campaign. Thirty public Reels, a handful of IGTV videos, and dozens of short posts - all published, all public, and all living exclusively on Instagram. Now someone asks: "Can we archive these for the brand library?"

You open your browser, paste a URL into a random download site, and... nothing. Or worse, you get an ad-laden page with a broken download button. You try four more sites. Two work. One gives you a 240p file. And you've just spent twenty minutes manually processing what should take seconds.

Sound familiar? This is the reality for anyone managing Instagram video content at even a modest scale. The good news is there's a better way.

Instagram Video and Reel Downloader is an Apify Actor built specifically for this. It handles public Instagram Reels, /p/ video posts, and IGTV (/tv/) links in a repeatable, automation-ready workflow - giving you direct video URLs, metadata, thumbnails, and optional stored files, all in clean structured output.

This article walks you through exactly how it works, who it's for, and how to go from zero to a working download workflow in under ten minutes.

Why downloading Instagram videos is harder than it looks

On the surface, saving an Instagram video sounds trivial. The content is public. The URL is right there. How hard can it be?

Harder than expected, it turns out.

Instagram doesn't expose direct video file URLs through any official public API. The platform serves video content dynamically from its content delivery network, and the URLs it generates are session-bound, expire quickly, and are deeply embedded in the page's internal data structures. What you see in your browser isn't what a simple HTTP request sees.

On top of that, Reels in particular behave differently from regular posts. Some pages expose enough data in the static HTML for a fast scraper to pick up the video URL immediately. Others don't - they require a realistic browser session to load the required JavaScript and resolve the actual media delivery request before the video URL becomes visible.

This is why so many free Instagram downloader websites are inconsistent. They work sometimes, fail silently other times, and give you no real explanation when something goes wrong. For a one-off personal download, that's annoying but acceptable. For a business workflow, it's a real problem.

Instagram Video and Reel Downloader was designed around this reality. It uses a two-path extraction approach: try the fast static route first, and fall back to a browser-based session for harder Reel pages that don't expose the video URL in their initial HTML. The result is a more dependable tool for real public content workflows.

What Instagram Video and Reel Downloader actually does

Let's be clear about what this Actor is and isn't.

It is not a browser extension, not a mobile app, and not a one-click website. It's an Apify Actor - a serverless cloud program that accepts structured input, performs a task, and returns structured output. You run it from Apify Console, call it through the API, or trigger it from an automation. It keeps a record of every run, and you can reuse the same configuration whenever you need it.

More specifically, for each public Instagram URL you give it, the Actor can return:

a direct videoUrl pointing to the resolved video file
a thumbnailUrl for the post's preview image
the post title and description when available
the canonical URL, shortcode, and mediaType (reel, post, or tv)
optional file references if you choose to store the actual video or image in Apify's key-value store
structured error data (with an errorCode and errorMessage) if a URL can't be resolved - so the run stays auditable even when something goes wrong

That last point matters more than it might seem. Most free downloader tools just... fail. No error message, no record, no way to know what happened. This Actor keeps a dataset item for every URL it processes, whether it succeeded or not. For teams tracking a batch of URLs, that's a meaningful difference.

Ready to try it? Open Instagram Video and Reel Downloader in Apify Console and run it directly in your browser - no code required.

Who actually needs this?

Before getting into the walkthrough, it's worth being honest about who this tool is and isn't for.

If you occasionally want to save a single Instagram Reel for personal use, a free browser extension or a consumer downloader site will probably do the job. This Actor isn't aimed at that use case.

Where it earns its place is in repeatable, team-level, or integration-heavy workflows.

Social media managers and content teams often need downloadable copies of published Reels - for brand libraries, cross-platform repurposing, or simply keeping a backup of content that lives only on Instagram. Running a batch of URLs through this Actor takes minutes instead of an afternoon.

Agencies working with multiple clients or creator rosters regularly need to pull public video assets into approval workflows, reporting decks, or content management systems. A structured JSON output with direct video URLs and thumbnails is far easier to feed into those systems than manually downloading files and uploading them elsewhere.

Developers and automation builders can use the Actor as a building block. The Apify platform gives you API access, webhooks, and scheduling - so a workflow that starts with "download these public Instagram videos" can chain directly into "save to Google Drive," "send for review," or "push metadata to a spreadsheet."

Researchers and operations teams sometimes need video URLs and metadata for tracking or reporting purposes - not necessarily the files themselves. The dataset output covers that without requiring extra storage overhead.

Step-by-step: how to use the Actor from Apify Console

You don't need to write a single line of code to get your first result. Here's the full walkthrough.

Step 1: Open the Actor

Go to apify.com/eunit/instagram-video-and-reel-downloader.

If you don't already have an Apify account, you can create one for free. Once you're in, you'll land on the Actor's overview page. Read through the README - it's short and covers the key things to know before you run anything. Then click the Input tab.

Step 2: Collect your public Instagram URLs

The Actor supports three URL formats:

/reel/ - standard Instagram Reels
/p/ - regular photo and video posts
/tv/ - IGTV-style long-form video

Copy the public URLs you want to process. The critical word there is public. This tool doesn't attempt to access private accounts, private posts, or restricted content. If the post isn't publicly viewable without logging in, the Actor will return an error item for that URL rather than silently pretending it worked.

A quick way to check: open the URL in a private/incognito browser window. If you can see the video without logging in, the Actor can process it.

Step 3: Configure the input

The Input tab in Apify Console gives you a form to fill in. Here's what each field does:

startUrls - This is the only required field. It's the list of Instagram URLs you want the Actor to process. You can add one or fifty.

storeVideoFiles - When this is enabled, the Actor doesn't just resolve the video URL - it also downloads the actual video file and stores it in Apify's key-value store, accessible via a public URL. Useful when you need the binary file, not just a link. Leave it off if the videoUrl in the dataset is sufficient for your workflow.

storeThumbnailFiles - Same idea, but for the post's thumbnail image. Enable it if you need the actual image file stored alongside the video reference.

includePageHtml - Saves the raw Instagram HTML from each fetched page to key-value store. You'll rarely need this, but it's a useful debugging tool if Instagram changes its page structure and the Actor stops resolving video URLs correctly.

filenamePrefix - If you're storing files, this prefix gets prepended to every stored filename. Something like campaign-june-2026 makes it easy to find related files later.

maxConcurrency - Controls how many pages the Actor processes in parallel. The default of 3 is a reasonable starting point. Keep this lower (1 or 2) if Instagram is rate-limiting your requests; only push it higher if speed is a priority and you're willing to accept some request failures.

maxRequestRetries - How many times the Actor retries a failed page request before marking it as an error. The default of 2 is fine for most use cases.

requestTimeoutSecs - The timeout for individual page fetches and optional file downloads. 30 seconds works well under normal conditions; increase it if you're on a slow connection or downloading large video files into storage.

proxyConfiguration - Optional proxy settings. Most users don't need this to start. If you run into consistent request failures or geo-blocking behavior, enabling Apify Proxy here can help distribute requests more effectively.

A realistic input configuration for a small batch might look like this:

{
  "startUrls": [
    { "url": "https://www.instagram.com/reel/ABC123xyz/" },
    { "url": "https://www.instagram.com/p/DEF456uvw/" },
    { "url": "https://www.instagram.com/reel/GHI789rst/" }
  ],
  "storeVideoFiles": true,
  "storeThumbnailFiles": false,
  "includePageHtml": false,
  "filenamePrefix": "campaign-june-2026",
  "maxConcurrency": 2,
  "maxRequestRetries": 2,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}

Step 4: Run the Actor

Click Run in the top-right corner.

The Actor starts processing your URLs immediately. For straightforward public posts and video pages, it extracts the video URL and metadata from the static page response - fast, efficient, no browser overhead. For trickier Reel pages that don't expose the video delivery URL in their initial HTML, it falls back to a browser-based path that can capture the actual media requests.

You don't need to manage any of this yourself. The Actor handles the routing internally.

You can watch the logs in real time as the Actor runs. Each URL gets processed, and any failures surface immediately with an error code and message rather than a silent skip.

Step 5: Review the dataset output

Once the run completes, click the Dataset tab.

Each URL you submitted produces one dataset item. A successful result looks like this:

{
  "status": "ready",
  "inputUrl": "https://www.instagram.com/reel/ABC123xyz/",
  "canonicalUrl": "https://www.instagram.com/reel/ABC123xyz/",
  "shortcode": "ABC123xyz",
  "mediaType": "reel",
  "title": "Instagram video",
  "description": "Caption text from the post",
  "videoUrl": "https://instagram.fcdn.example/v/t16/f1/video.mp4",
  "thumbnailUrl": "https://instagram.fcdn.example/v/t51/thumb.jpg",
  "videoStoreKey": "video-campaign-june-2026-abc123xyz.mp4",
  "videoPublicUrl": "https://api.apify.com/v2/key-value-stores/STORE_ID/records/video-campaign-june-2026-abc123xyz.mp4",
  "uploadDate": "2026-05-15T14:30:00.000Z",
  "fetchedAt": "2026-06-17T10:00:00.000Z",
  "errorCode": null,
  "errorMessage": null
}

A failed result is just as structured:

{
  "status": "error",
  "inputUrl": "https://www.instagram.com/reel/PRIVATE123/",
  "errorCode": "NO_VIDEO_URL",
  "errorMessage": "Could not resolve a public video URL from this page."
}

You can export the dataset in JSON, CSV, Excel, or HTML format - whichever fits your downstream workflow. If you enabled file storage, you can access all stored video and image files directly from the run's Storage tab.

Step 6: Put the output to work

This is where the Actor starts earning its value beyond a simple file download.

If your workflow is "download these URLs occasionally and review them manually," exporting the dataset as CSV and opening it in a spreadsheet is enough.

If your workflow is more continuous - weekly campaign archives, automated content pipelines, or team-level review processes - the Apify platform gives you several paths to build on this run:

API calls: Trigger the Actor from any script using the Apify JavaScript or Python client, or a direct HTTP request. The output is the same structured dataset.
Schedules: Run the Actor on a weekly or daily cadence with a fresh list of URLs.
Webhooks: Fire a webhook when the run completes to trigger downstream steps in Make, Zapier, n8n, or your own backend.
MCP server: If you're building AI workflows, the Actor is available as a tool through Apify's Model Context Protocol (MCP) server.

Integrating the Actor with code

If you're a developer or want to trigger the Actor programmatically, the setup is straightforward.

JavaScript / TypeScript:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });

const input = {
  startUrls: [
    { url: 'https://www.instagram.com/reel/ABC123xyz/' }
  ],
  storeVideoFiles: true,
  filenamePrefix: 'my-archive'
};

const run = await client.actor('eunit/instagram-video-and-reel-downloader').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();

items.forEach((item) => {
  if (item.status === 'ready') {
    console.log('Video URL:', item.videoUrl);
  }
});

Python:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "startUrls": [{"url": "https://www.instagram.com/reel/ABC123xyz/"}],
    "storeVideoFiles": True,
    "filenamePrefix": "my-archive"
}

run = client.actor("eunit/instagram-video-and-reel-downloader").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("status") == "ready":
        print("Video URL:", item["videoUrl"])

Install the client with npm install apify-client (JavaScript) or pip install apify-client (Python), then drop in your Apify API token from Apify Console under Settings > Integrations.

When it makes sense to pay for a reliable Actor

Let's address the obvious question: why pay for this when free downloader sites exist?

The honest answer is that free downloader sites are built for individual, one-off use. They're not designed to process batches, return structured data, integrate with APIs, or recover gracefully when Instagram changes something. They also come with ad loads, pop-ups, and opaque data practices that you probably don't want in a business workflow.

This Actor uses a pay-per-event model at $90.00 per 1,000 processed videos. That pricing structure means you pay for actual completed work - not a flat subscription that bills you whether you use it or not.

For context: if a marketing manager spends 30 minutes manually downloading and organizing 10 Instagram videos per week, that's roughly 26 hours of manual work per year. The same batch run through this Actor costs under $1 and completes in a few seconds.

The economics shift further when you factor in reliability. A workflow that fails halfway through a batch, returns corrupted files, or gives you no error visibility isn't really "free" - it's just deferring the cost into your time and frustration budget.

Paying customers here are buying dependability, structured output, and the ability to build workflows that run the same way every time.

A few things to keep in mind

This Actor is built for public Instagram content, and it's worth being clear about what that means in practice.

Private content won't work. If a post requires you to be logged in to view it, the Actor can't access it, and it will return an error item for that URL.

Instagram can and does change things. The page structure, video delivery logic, and anti-bot behavior can shift with Instagram updates. When that happens, the extraction may stop working for some URL types until the Actor is updated. The structured error output means you'll know which URLs failed rather than getting a silent empty result.

You're responsible for compliance. Only download content you own, have explicit permission to use, or have a legitimate business reason to process. Respect copyright, platform terms, and local laws. The tool does what it's designed to do - how you use it is your responsibility.

Wrapping Up

Most Instagram video download tools are consumer utilities. They're built for "save this one Reel right now" - not for "process this batch of public videos, give me structured output, and let me automate it next week too."

If your work touches Instagram video content at any real scale - campaign archives, content pipelines, creator workflows, or automated reporting - that distinction matters a lot.

Instagram Video and Reel Downloader is built for the second use case. It gives you direct video URLs, structured metadata, optional file storage, a reliable fallback for difficult Reel pages, and a platform underneath it that supports APIs, scheduling, webhooks, and integrations.

Try it from Apify Console with a few public URLs. The first run takes about few seconds to set up, and you'll see immediately whether it fits the workflow you're building.

Get started with Instagram Video and Reel Downloader on Apify →

How to scrape Nextdoor for hyper-local demographics and community sentiment

Emmanuel Uchenna — Sat, 30 May 2026 07:05:14 +0000

Neighborhood-level data is where many high-value business decisions start. Citywide averages are useful, but they often hide the details that matter when you are choosing where to advertise, where to invest, or how to understand what local communities care about. If you want to find areas with strong homeownership, identify places where residents discuss common service needs, or analyze how people describe a neighborhood in public community pages, you need hyper-local signals.

That is exactly why so many teams look for a reliable way to scrape Nextdoor. The challenge is that Nextdoor is a relatively closed ecosystem. It does not offer an easy, public, developer-friendly path for extracting the kind of structured data marketers, analysts, founders, and researchers want. Manual collection is slow, inconsistent, and impossible to scale across hundreds of locations.

This is where Nextdoor Scraper helps. Built as one of the Apify Actors for the Apify platform, it turns public Nextdoor city-page data into structured output you can use. Instead of copying information by hand, you can collect geographic coordinates, resident counts, income indicators, homeowner ratios, affordability signals, FAQ content, and local community references in a repeatable workflow. If you need a practical Nextdoor API alternative for public pages, this is the faster path.

What is the Nextdoor Scraper?

Nextdoor Scraper is a scraping and data extraction Actor for collecting structured data from public Nextdoor pages, especially city pages. It is designed for people who want usable output, not raw HTML. That means you get cleaned fields that can go straight into a spreadsheet, dashboard, CRM, enrichment pipeline, or reporting workflow.

In simple terms, the Nextdoor Scraper visits public Nextdoor pages, extracts the visible and structured page data, and returns it in machine-readable formats such as JSON, XLSX, and CSV. That makes it easier to analyze neighborhood-level trends at scale without building your own browser automation stack from scratch.

The extracted fields are especially useful for local market analysis. Depending on the public page and the available structured data, the Actor can capture:

Geographic coordinates, such as a centroid with latitude and longitude
City and state details
Resident counts
Average age
Average income
Percentage of homeowners
Affordability and friendliness scores
FAQ entries
Neighborhood references
Business directory references
Local groups and additional community resources

For example, a public city page can expose useful signals such as a centroid of 40.016, -80.747, a resident count of 3,878, an average age of 35, an average income of $41k, and a homeowner share of 56%. It can also include livability indicators such as an affordability score of 99 and a friendliness score of 86. Those values are much more actionable than a plain list of URLs because they help you compare locations quickly and prioritize where to spend time and budget.

The result is a practical way to scrape Nextdoor data for research, local service lead generation, geographic analysis, and community sentiment discovery, while staying focused on public information.

Top 4 profitable use cases for Nextdoor data

The biggest reason to use a Nextdoor scraper is not the scraping itself. It is what the data helps you decide, launch, test, or sell afterward. Here are four high-value ways to use hyper-local demographics and community sentiment from public Nextdoor pages.

1. Building regional contractor and reputation platforms

If you are building a regional directory, a home services marketplace, or a contractor reputation platform, you need more than a list of businesses. You need neighborhood context.

Public Nextdoor data helps you map cities and surrounding local structures in a way that feels more grounded in how residents actually organize their communities. The FAQs, neighborhood references, and local groups can help you understand how people describe an area, what kinds of issues come up repeatedly, and which categories of services matter most.

That is valuable when you are building:

A local contractor comparison site
A regional reputation database
A neighborhood services recommendation engine
A local SEO content network for service categories

Instead of launching broad, generic pages, you can prioritize neighborhoods with meaningful public signals and create location-aware service experiences. That makes your product more relevant, and it can improve both engagement and conversion.

2. Lead generation for home services

This is one of the clearest commercial use cases. Home service businesses such as roofers, landscapers, plumbers, painters, pest control companies, and HVAC providers need to know where homeowners live, where disposable income is high, and where local demand is likely to justify the cost of outreach. Nextdoor city-page data can help you build exactly that picture.

For example, if a location shows 3,878 residents, an average income of $41k, and 56% homeowners, that is immediately more useful than running a generic campaign across an entire metro area. It gives you a better basis for deciding where to target direct mail, paid social, local landing pages, sales outreach, and neighborhood-specific offers.

You can use the data to:

Find areas with stronger homeowner density
Compare income signals across nearby cities
Prioritize communities for seasonal campaigns
Match marketing messages to neighborhood concerns surfaced in FAQs

A landscaping company may care about homeowner-heavy areas. A roofing business may want to focus on suburban communities with stable ownership and strong local trust signals. A cleaning or handyman service may use local groups and neighborhood references to decide where to pilot a new offer. That is the real advantage of scraping Nextdoor for lead generation. You are getting closer to the local context behind the customer.

3. Real estate investment and market research

Real estate teams already use public data from many sources, but the hyper-local context is often fragmented. That is where public Nextdoor data becomes interesting.

When a city page includes livability indicators such as an affordability score of 99 and a friendliness score of 86, you gain another layer of context for evaluating a market. These values do not replace formal due diligence, but they can help analysts screen locations faster before moving into deeper research.

A real estate investor, brokerage, or proptech company can use the data to:

Compare neighborhoods and cities at a glance
Identify areas that look affordable relative to nearby markets
Spot locations with strong community-oriented signals
Enrich internal market reports with public local context
Create lead magnets and market snapshots for clients

Geographic coordinates also matter. A centroid such as 40.016, -80.747 makes it easier to map results, visualize city clusters, and combine the output with other location-based datasets. That is useful for market scoring, territory planning, and neighborhood-level dashboards.

If you are producing local real estate data products, public Nextdoor scraping can work as a lightweight enrichment layer that adds community context to your pipeline.

4. Community sentiment and trend analysis

Community sentiment is often hard to quantify because it spans many channels. Nextdoor is valuable because it is built around neighborhood-level discussion, recommendations, local concerns, and community interaction.

Even when you focus only on public pages, FAQ sections, local groups, business directory references, and related resources can still reveal what a place emphasizes. That helps you understand whether a community leans toward family life, local commerce, safety, neighborhood organization, pet activity, home improvement, or other recurring themes.

This is useful for:

Local market researchers
Urban policy teams
Media analysts
Economic development groups
Brands planning geo-targeted campaigns

For example, if you see repeated references to neighborhood groups, practical local services, and community FAQs, that can point to a highly engaged local audience. If business directories are prominent, it may suggest strong local commercial activity. If certain topic clusters appear across multiple nearby pages, you may have found a regional trend worth tracking over time. In other words, the value is not just in scraping Nextdoor data once. It is in building a repeatable view of how communities present themselves publicly.

Key features of the Nextdoor Scraper

The appeal of Nextdoor scraper is that it is not just a one-off script. It is packaged as one of the Apify Actors, so it fits into a repeatable workflow on the Apify platform.

Speed and scale

Instead of manually opening pages and copying details into a spreadsheet, you can run automated extraction across many public URLs in one job. That saves hours of repetitive work and makes side-by-side comparison possible.

Structured output for analysis

The Nextdoor scraper returns normalized data fields, not just raw page content. That makes it easier to filter, group, sort, enrich, and export your results for business use.

Browser-based extraction on the Apify platform

Nextdoor pages are modern web pages, so browser automation matters. The Nextdoor scraper handles the browser-driven extraction flow for you on the Apify platform, which is much easier than maintaining your own extraction stack and local tooling.

Cost-effective automation

Building and maintaining your own pipeline for public-page collection can get expensive fast. Using Apify Actors reduces engineering overhead, shortens time to value, and gives you a practical way to pay for execution instead of building everything yourself.

Export formats that fit real workflows

Once the data is collected, you can work with it in formats your team already uses. JSON is ideal for developers and downstream automation. CSV is ideal for analysts, operations teams, and quick spreadsheet reviews.

Step-by-step tutorial: how to scrape Nextdoor data

Using Nextdoor Scraper on the Apify platform is straightforward.

Step 1: Access the Actor

Open Nextdoor Scraper in Apify Store. On the Actor page, click Try for free or Start to open the input form in Apify Console.

Step 2: Configure the target URLs

In the Input tab, add one or more public Nextdoor URLs. A common starting point is a city page, such as:

{
  "startUrls": [
    { "url": "https://nextdoor.com/city/north-miami--fl/" }
  ],
  "maxRequestsPerCrawl": 10,
  "maxConcurrency": 2,
  "maxItemsPerSection": 25
}

Step 3: Run the Actor

Click the Start button at the bottom of the page. The Actor initializes, processes the target pages, extracts structured data fields, and stores the results in a dataset.

Depending on the page, the output can include fields such as:

cityName
state
centroid
residentsCount
averageAge
averageIncome
percentageHomeowners
affordabilityScore
friendlinessScore
faqs
localGroups

Step 4: Export the dataset

Once the run finishes, open the Storage tab to inspect the extracted records. You can then export the dataset in the format that best fits your workflow.

Here is a simplified example of the kind of output you can expect:

[
  {
  "url": "https://nextdoor.com/city/example-city/",
  "cityName": "Example City",
  "state": "OH",
  "centroid": {
    "latitude": 40.016,
    "longitude": -80.747
  },
  "residentsCount": 3878,
  "averageAge": 35,
  "averageIncome": "$41k",
  "percentageHomeowners": "56%",
  "affordabilityScore": 99,
  "friendlinessScore": 86,
  "faqCount": 7,
  "accessStatus": "public"
},
...
]

From there, you can download the results as JSON or CSV and move directly into analysis, enrichment, reporting, or outreach planning.

Integrating Nextdoor data into your workflows

The value of scraped Nextdoor data increases when it flows into the tools your team already uses.

On the Apify platform, you can connect dataset outputs to automations and downstream systems. That includes sending results to Google Sheets for review, using Make for no-code workflows, or routing data into Zapier for notifications and follow-up actions. You can also plug the JSON output into your own analytics stack, internal dashboards, CRMs, or lead scoring systems.

This is especially helpful when you are monitoring multiple cities over time. Instead of exporting data manually after each run, you can make Nextdoor scraping into a repeatable process that supports lead generation, market tracking, and local intelligence workflows.

Is scraping Nextdoor legal?

Scraping should always be done responsibly. The safest approach is to focus on publicly available data, respect applicable rules and terms, and avoid collecting private or personal information behind login walls.

Nextdoor Scraper is designed around public page extraction. That means the workflow is intended for public city-page data and other publicly accessible content, not private profiles, direct messages, or restricted user data. You should also review your own legal obligations, compliance needs, and usage policies before running large-scale data collection.

Wrapping up

If you want to scrape Nextdoor for hyper-local demographics, local service lead generation, neighborhood research, or community sentiment analysis, structured public-page extraction gives you a strong starting point. Instead of relying on manual research, you can collect repeatable location data such as resident counts, income signals, homeowner ratios, affordability indicators, friendliness scores, FAQs, and local group references in a format your team can actually use.

Nextdoor Scraper turns that process into a scalable workflow on the Apify platform. Open the Actor in Apify Store, run your first extraction, and start building better local market insights today.

How to get YouTube data for a growth strategy: video and channel analysis

Emmanuel Uchenna — Thu, 09 Apr 2026 17:02:04 +0000

YouTube has more than 2 billion logged-in users every month and processes over 500 hours of uploaded video every minute. It is the second-largest search engine in the world. Yet the majority of creators and brands still build their content strategy the old-fashioned way: watching what competitors post, guessing which topics will resonate, and hoping the algorithm cooperates.

That approach works early on. But as your niche becomes more competitive and your publishing schedule more demanding, manual research stops scaling. You can't audit 20 competitor channels by hand every week, track which video formats are trending across five countries simultaneously, and still have time to actually make content.

What changes everything is structured data. When you can collect trending video metrics, popular channel signals, and keyword data programmatically - across countries, categories, and time - YouTube growth becomes a repeatable system rather than a creative gamble.

This article shows you how to obtain that data using two purpose-built Apify Actors, and how to turn it into a growth strategy grounded in what the platform is actually rewarding right now.

Why YouTube data matters for growth strategy

YouTube operates on two distinct engines: search and recommendations. Search surfaces content in response to specific queries. Recommendations push content to viewers who weren't actively searching - but whose watch history signals a relevant interest. A strong growth strategy needs both, and both require different types of data.

The data that matters most:

Trending videos by category and country - what the recommendation algorithm is actively amplifying, which formats and topics are getting pushed to new audiences
Popular channels and their recent content - which creators the platform is currently favoring, and what they're publishing right now
Popular keywords - what viewers are actively searching for at this moment, not six months ago
Engagement metrics (views, likes, comments) - signals of content-audience fit that go deeper than raw view counts

The official YouTube Data API was not designed for this kind of research. Its quota limits cap how much data you can pull per day, it has no trending channels endpoint, and accessing granular trend data by country and category requires complex workarounds that hit limits quickly. For recurring competitive research, the API becomes an obstacle rather than a tool.

Purpose-built scraping Actors solve this. The YouTube Trending Videos by Categories Scraper and YouTube Popular Channels Scraper extract exactly what you need - structured, export-ready data - without API keys, quota management, or engineering overhead.

If you want a step-by-step scraping walkthrough before diving into strategy, read How to scrape YouTube trends and popular channels first.

The growth data you can collect and why it matters

Before looking at how to use the data, it helps to understand exactly what each Actor returns and which growth question each data point answers.

Data point	Source Actor	Growth use case
Trending video rank, title, views, likes	YouTube Trending Videos by Categories Scraper	Spot viral formats and topics before your competitors do
`published_time_text` (e.g., "Yesterday", "3 hours ago")	YouTube Trending Videos by Categories Scraper	Identify how quickly content trends and optimal publishing windows
Video thumbnail URL	YouTube Trending Videos by Categories Scraper	Analyze thumbnail patterns that correlate with high click-through rate
Channel name, recent videos, video URLs	YouTube Popular Channels Scraper	Identify which channels are dominating your niche right now
Popular keywords	YouTube Popular Channels Scraper	Discover high-intent search terms for SEO and content ideation
Category breakdown per country	Both Actors	Understand regional audience preferences for localized content strategies
Engagement ratio (likes / views)	YouTube Trending Videos by Categories Scraper	Filter for content that converts viewers, not just attracts them

Together, these two Actors give you a full picture view of what YouTube is amplifying in your space - the videos rising to the top and the channels being pushed to new audiences.

Using trending video data for content strategy

Trending videos are the algorithm's endorsement. They represent the exact combination of topic, format, title structure, and production style that YouTube is choosing to amplify in a given country and category at a specific moment. Analyzing these patterns systematically - rather than browsing trending pages manually - reveals what's working before it becomes obvious.

How to collect trending video data

Go to YouTube Trending Videos by Categories Scraper on Apify. Create a free account if you don't have one.
Select your target Country (e.g., united-states, united-kingdom, india, brazil).
Select your Category (e.g., gaming, music, howto-and-style, science-and-technology).
Click Save & Start. The run completes in seconds.
Preview results in the table view or export as JSON, CSV, or Excel.

Each result includes: rank, video title, video URL, thumbnail URL, author (channel name), published_time_text, views, likes, and comments.

{
  "rank": "1",
  "video_title": "Marvel Television's Wonder Man | Official Trailer",
  "video_url": "https://youtube.com/watch?v=wHuWmjXsReU",
  "author": "Marvel Entertainment",
  "published_time_text": "Yesterday",
  "views": "5.2M",
  "likes": "113K",
  "comments": "4.9K",
  "country": "united-kingdom",
  "category": "All"
}

Run this Actor across multiple countries and categories, and you have a competitive intelligence layer that updates every time you trigger it.

Competitor channel analysis: who's winning and why

Channel analysis is where most creators focus - but most do it wrong. They look at subscriber counts. A channel with 800k subscribers might not have posted in two months and could be algorithmically invisible. The more useful signal is which channels are currently appearing as popular in trending data - because that reflects what YouTube is actively recommending to new viewers right now.

Why popular channel data beats subscriber counts

The YouTube Popular Channels Scraper returns the channels YouTube is currently surfacing as "buzzing" in a given country and category, along with their most recent videos. This is fundamentally different from a ranked list of the biggest channels. A channel with 80k subscribers appearing here is actively growing. A channel with 2 million subscribers that doesn't appear has lost algorithmic favor.

This distinction matters enormously for competitor research. Your real YouTube competitors are not necessarily your biggest industry rivals - they're the channels competing for the same viewer attention and winning right now.

What to analyze in popular channel data

Publishing frequency and recency. How many videos has each trending channel posted in the last 30 days? Channels maintaining a consistent 2-3 videos per week in trending data signal that the algorithm rewards their cadence. That's a publishing frequency benchmark, not just an aspiration.

Topic and title patterns. Across a channel's recent trending videos, which topics repeat? If a competitor's recent 6 trending videos are all about personal finance for under-30s, that sub-niche is clearly working for their audience. Look for the intersection between their most successful topics and gaps in your own content calendar.

Cross-country channel overlap. Run the Actor for the same category across the US, UK, and Australia. Channels appearing in all three are building genuine algorithmic momentum across English-speaking markets. These are the channels worth monitoring most closely.

How to collect popular channel data

Go to YouTube Popular Channels Scraper on Apify.
Select your Country and Category.
Click Save & Start. Results include channel names, recent video URLs, and popular keywords for the selected market.
Export and repeat for 3-5 target countries.
Build a simple tracking sheet: channel name, countries trending in, recent topic themes, and estimated posting frequency. Update it weekly.

{
  "country": "united-kingdom",
  "category": "all"
}

Building a multi-country growth intelligence system

A one-off data pull gives you a snapshot. A repeatable system gives you a competitive advantage that compounds over time. Here's how to build one without a development team.

The architecture

Collect - run both Actors weekly across your target countries (3-5) and category
Store - export results to Google Sheets or Airtable with a date column so you can track changes over time
Analyze - compare week-over-week: are the same channels trending, or new ones emerging? Are keywords shifting? Are engagement ratios moving up or down?
Act - translate the top 3 insights into your next 2 weeks of content: one trending topic, one underserved format, one keyword to target

Scheduling automatic runs

From Apify Console:

Configure your Actor run (country + category)
Click Save as a new task and give it a name (e.g., "US Gaming Trends - Weekly")
Go to Schedules in the left navigation and click Create new schedule
Add your task and set the frequency - weekly is a good starting cadence

No code required. The Actor runs automatically, and new results are appended to the dataset.

For developers: automate with Python

If you're building a dashboard, feeding a machine learning model, or integrating this data into a larger pipeline, the apify-client makes automation straightforward.

Install the client:

pip install apify-client

Run the YouTube Popular Channels Scraper programmatically:

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")

run_input = {
    "country": "united-states",
    "category": "gaming",
}

run = client.actor("eunit/youtube-popular-channels-scraper").call(run_input=run_input)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Swap eunit/youtube-popular-channels-scraper for eunit/youtube-trending-videos-by-categories to pull trending video data instead. The input schema is identical. You can run both Actors in sequence and merge the outputs into a single structured dataset for a complete weekly intelligence snapshot.

Your API token is in Settings > API & Integrations in Apify Console.

Pricing: what this costs in practice

Both Actors use a Pay-Per-Event pricing model. You are charged a fixed amount per result successfully scraped - not for platform compute time or server usage.

Actor	Price
YouTube Trending Videos by Categories Scraper	from $1.00 / 1,000 results
YouTube Popular Channels Scraper	from $1.90 / 1,000 results

In practice, running a weekly trend scan across 5 countries with 1 category each returns approximately 200-300 data points - costing under $0.30 per run, or roughly $1.20/month.

Traditional SaaS YouTube analytics platforms typically charge between $49 and $299/month for dashboards with less granular, less current data than you get from a direct scrape. The Pay-Per-Event model means you pay exactly in proportion to how much data you need. Scale up for a product launch. Scale down during quiet weeks. No subscriptions, no idle spend.

Legal and ethical considerations

Both Actors collect publicly available data - information that any user can see by visiting YouTube without logging in. They do not access private videos, user passwords, private messages, or any content that requires authentication.

Scraping public data for research, competitive intelligence, and informational purposes is generally considered acceptable in most jurisdictions. Operate within YouTube's Terms of Service, use the data for your own analysis, and avoid any use cases that involve republishing scraped content without appropriate attribution.

For Apify's broader guidance on ethical web scraping, see the Apify documentation.

Wrapping Up

YouTube growth stops being a guessing game when you treat it like a data problem. Trending videos tell you what the algorithm is amplifying right now. Popular channel data shows you who it's amplifying and why. Engagement ratios reveal which content is building real audience relationships versus passive viewership. And popular keywords hand you the exact search terms your potential viewers are using today.

The growth loop becomes repeatable: collect data weekly, identify the 2-3 highest-signal patterns, build your next content batch around them, and measure whether your own engagement metrics are converging with trending benchmarks.

The best channels in every niche are already doing this. The ones that aren't are still guessing.

Start collecting:

YouTube Trending Videos by Categories Scraper - from $1.00 / 1,000 results
YouTube Popular Channels Scraper - from $1.90 / 1,000 results

FAQ

Can I get YouTube trending data without the official API?

Yes. The YouTube Trending Videos by Categories Scraper extracts real-time trending data across 25+ countries and 13 categories with no API key required and no quota limits to manage. You get the same data you'd see by browsing YouTube's trending pages, in a structured, export-ready format.

How is "popular channel" data different from subscriber count?

Popular channel data reflects which channels YouTube is actively recommending to new viewers right now. A channel with 80k subscribers can appear here because it's publishing consistently and gaining algorithmic momentum. A channel with 2 million subscribers that hasn't posted recently won't. For competitive intelligence, current algorithmic favor matters far more than historical growth.

What categories can I track?

Both Actors support: All, Music, Gaming, Film & Animation, Autos & Vehicles, Sports, Pets & Animals, Comedy, Entertainment, News & Politics, How-to & Style, and Science & Technology - across 25+ countries including the US, UK, India, Brazil, Japan, Germany, Australia, and more.

How often should I run the scrapers?

Weekly is sufficient for most content strategies. Daily runs are useful during product launches, campaigns, or for fast-moving niches like Gaming or News & Politics where trends shift within hours.

Can I use this data for sentiment analysis?

Engagement ratios (likes-to-views, comment volume relative to views) are a reliable sentiment proxy at scale and require no comment scraping. For comment-level sentiment - what viewers are actually saying - pair these Actors with a YouTube Comments Scraper and feed the output to an LLM for summarization.

Is YouTube scraping legal?

Scraping publicly available YouTube data for research and informational purposes is generally legal in most jurisdictions. These Actors only collect data visible to any visitor without logging in. Always use scraped data in accordance with YouTube's Terms of Service and applicable privacy laws.