DEV Community: Omar Eldeeb

White Friday & 11.11 Tracker: Noon and Amazon UAE Prices

Omar Eldeeb — Wed, 28 Oct 2026 11:49:00 +0000

The Gulf's November sales are 11.11 (Wednesday 11 November 2026; noon's 2025 sale ran 7-12 November) and the White Friday / Yellow Friday period around Friday 27 November (noon's Yellow Friday ran 20-30 November in 2025). To track them, start weekly price records on noon UAE and Amazon UAE / Saudi Arabia in mid-October, by product URL, and check every 6 hours during each sale, in AED or SAR.

This is the Gulf page of the cluster. It covers the sale names and windows, what can actually be scraped on noon and Amazon in the UAE and Saudi Arabia (with results from live runs on 20 October 2026), the URL details that decide the currency and language, and costs. The global calendar is in the retail sales calendar.

Three names, one season

Sale	Who runs it	2025 window	2026 anchor
11.11 (Singles' Day)	noon, Amazon UAE and other UAE retailers	noon: evening of 7 Nov to 12 Nov. Amazon UAE: until 12 Nov	Wed 11 Nov 2026
Yellow Friday	noon (named after its yellow logo)	20-30 Nov	around Fri 27 Nov 2026
White Friday	Gulf name for Black Friday, used by Amazon UAE and other brands	Amazon UAE reported as 22-29 Nov (Property Finder; we found no Amazon announcement)	around Fri 27 Nov 2026
Black Friday	many UAE retailers use the US name	Black Friday was 28 Nov 2025	Fri 27 Nov 2026
Dubai Shopping Festival	Dubai Festivals and Retail Establishment	—	4 Dec 2026 - 17 Jan 2027 (announced)

The 2026 retailer windows weren't announced on 20 October 2026. The 2025 pattern suggests the 11.11 window opens a few days before the 11th, and the Friday sales run about a week either side of the Friday. Start the baseline before either could begin: mid-October.

What can be scraped (tested 20 October 2026)

All examples use the E-commerce Product Scraper Pro actor. Its README rates the Gulf sites below as best-effort, except noon's Saudi storefront, which it lists as unsupported. Our own runs on 20 October 2026 gave these results:

Site	Route	Result on 2026-10-20
noon UAE (`noon.com/uae-en/`)	browser for the first page, then residential with the browser's cookies	Delivered: 3 of 3 products in two separate runs, priced in AED
noon Saudi Arabia (`noon.com/saudi-en/`, `/saudi-ar/`)	none: not fetched	Unsupported. Akamai answered "Access Denied" on every route in the actor's October 2026 tests, including Saudi residential connections and the browser. Each input gets a status row with `outcome: "unsupported"` and costs nothing beyond the run's start fee
Amazon UAE (`amazon.ae`)	residential	Delivered, AED
Amazon Saudi Arabia (`amazon.sa`)	datacenter	Delivered, SAR, from Arabic and English product pages
IKEA UAE / Saudi Arabia	residential	Not tested today; best-effort in the README, first page of listings only

So for Saudi prices, use Amazon Saudi Arabia. The actor doesn't try noon's Saudi pages at all, so they cost no traffic or renders; the status row explains why. If a run contains only noon Saudi inputs, it ends as FAILED with that reason. Other Gulf retailers (Namshi, Sharaf DG, Carrefour, Lulu, Jarir, Extra) aren't in the supported-sites matrix, so we haven't tested them; any that run on Shopify or WooCommerce would fall under the actor's generic support.

noon is protected by Akamai. The actor handles it by rendering a page in a real browser to earn the session cookies, then fetching the rest over residential connections with those cookies. Few pages need the browser: a run of 60 noon UAE product URLs on 18 October 2026 delivered all 60 with 2 browser renders and 8 billed MB of residential traffic.

URLs: market, currency and language

noon puts the market in the path. /uae-en/ is the UAE in English, /egypt-en/ Egypt, /saudi-en/ Saudi Arabia (unsupported, see above). The actor reads the path to set the currency (AED, EGP). The same product ID under /uae-en/ and /egypt-en/ comes back as two separate rows, one per market, so you can track both. The actor's keyword + marketplaces: ["www.noon.com"] search uses the UAE English site; for Egypt, paste that market's search URL into listingUrls.
Amazon Saudi Arabia defaults to Arabic, and that's fine. The default Arabic product page https://www.amazon.sa/dp/B0DGHYPFGK returned on 20 October 2026 an Arabic title, price: 449, originalPrice: 599, currency: "SAR", inStock: true and brandName: "Apple". The actor reads Arabic availability text (Arabic-Indic digits included) and sets stockQty only from an explicit low-stock phrase on the new-item offer. If you want English titles, use the /-/en/ path, like https://www.amazon.sa/-/en/dp/B0DGHYPFGK; both forms work.
No breadcrumbs on Amazon UAE or Saudi Arabia. breadcrumbs is an empty list on amazon.ae and amazon.sa rows, because Amazon doesn't put a category trail on those pages. Group those products by your own category list.
One ASIN, two storefronts. The same ASIN often exists on amazon.ae and amazon.sa, which makes cross-market comparison easy (see below).

Watch product URLs, not search pages

Search results on noon reshuffle quickly. Two runs of the same noon UAE search for "samsung", 2 hours apart on 17 October 2026, returned 30 products each, and only 11 products appeared in both. Of those 11, one price changed (AED 1,168 to AED 1,167.55). If you track a search page, most of the "changes" you see are products entering and leaving the page. Use search pages to find candidates, then track the product URLs you care about.

Compare noon and Amazon UAE

This pulls one noon UAE search and the same keyword on Amazon UAE, and prints each price with its discount from the store's own was-price (pip install apify-client).

import os
from decimal import Decimal
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("constructive_calm/ecommerce-product-scraper-pro").call(
    run_input={
        # Noon: paste the search URL of the market you want (uae-en here)
        "listingUrls": [{"url": "https://www.noon.com/uae-en/search/?q=airpods%204"}],
        # Amazon.ae: keyword + marketplace
        "keyword": "airpods 4",
        "marketplaces": ["www.amazon.ae"],
        "followPagination": False,
        "maxItemsPerStartUrl": 3,
        "maxProductResults": 6,
    },
    max_total_charge_usd=Decimal("0.25"),
    logger=None,
)
for r in client.dataset(run["defaultDatasetId"]).iterate_items():
    was = r.get("originalPrice")
    off = f"{(was - r['price']) / was * 100:.0f}% off was-price" if was else "no was-price"
    print(f"{r['site']:<10} {r['price']:>8.2f} {r['currency']}  {off:<22} {r['name'][:48]}")

Real output, 2026-10-20:

noon         554.00 AED  30% off was-price      AirPods 4 ANC Wireless Earbuds, Bluetooth Headph
amazon-ae    549.00 AED  27% off was-price      Apple AirPods 4 Wireless Earbuds, Bluetooth Head
amazon-ae    549.00 AED  no was-price           Apple AirPods 5 Wireless Earbuds
noon         427.00 AED  22% off was-price      AirPods 4 Wireless Earbuds, Bluetooth Headphones
noon         759.00 AED  20% off was-price      AirPods Pro 3 Wireless Earbuds, Bluetooth Headph
amazon-ae    429.00 AED  22% off was-price      Apple AirPods 4 Wireless Earbuds, Bluetooth Head

Five weeks before White Friday, both stores already show 20-30% "off" their was-prices. That is exactly why the November discount has to be measured against your own October baseline, not against the was-price. The fake discounts guide shows the baseline method.

noon rows also carry seller. On marketplaces with many sellers, the cheapest offer can come from a seller you wouldn't buy from, so keep the seller column in your comparison.

Compare the UAE and Saudi Arabia

Both currencies are pegged to the US dollar (AED 3.6725 and SAR 3.75 per dollar), so a fixed conversion is accurate enough for a price comparison:

# Same ASIN on amazon.ae and amazon.sa, converted at the official USD pegs.
PEG = {"AED": 3.6725, "SAR": 3.75}   # units per 1 USD

rows = [  # from a scraper run on 2026-10-20 (ASIN B0DGHYPFGK)
    {"site": "amazon-ae", "price": 429, "originalPrice": 549, "currency": "AED"},
    {"site": "amazon-sa", "price": 449, "originalPrice": 599, "currency": "SAR"},
]
for r in rows:
    usd = r["price"] / PEG[r["currency"]]
    off = (r["originalPrice"] - r["price"]) / r["originalPrice"] * 100
    print(f"{r['site']:<10} {r['price']:>5} {r['currency']} = {usd:6.2f} USD  ({off:.0f}% below was-price)")

amazon-ae    429 AED = 116.81 USD  (22% below was-price)
amazon-sa    449 SAR = 119.73 USD  (25% below was-price)

A schedule for the Gulf sales

Window (2026)	Cadence	Cron (UTC)	Note
~15 Oct - 5 Nov	weekly	`0 4 * * 0`	Baseline. 04:00 UTC is 08:00 in the UAE, 07:00 in Saudi Arabia.
~6 Nov - 13 Nov (11.11)	every 6 hours	`0 2,8,14,20 * * *`	06:00, 12:00, 18:00 and 00:00 UAE time.
14 Nov - 18 Nov	daily	`0 4 * * *`	Gap between the two sales.
~19 Nov - 1 Dec (Yellow / White Friday)	every 6 hours	`0 2,8,14,20 * * *`
4 Dec - 17 Jan (Dubai Shopping Festival)	daily	`0 4 * * *`	Optional.

Use monitor mode (monitorMode: true) with the same product URL list throughout: the actor's monitor state is keyed by the exact set of input URLs, and each row then carries changeType, previousPrice and priceChangePct.

What it costs

Pay per event, from the actor's README. FREE tier: $0.0048 per product row, $0.0004 per listing page, $0.008 per browser-rendered page, $0.018 per residential MB (billed at 89% of measured traffic, only for items that delivered), $0.005 per run start. GOLD: $0.0008, $0.00021, $0.006, $0.014.

The noon + Amazon UAE run above was charged 6 products, 2 listing pages, 1 browser render and 2 residential MB. On FREE: $0.005 + 6 × $0.0048 + 2 × $0.0004 + $0.008 + 2 × $0.018 = $0.0786.
The 60 noon product URLs mentioned above (60 products, 2 browser renders, 8 MB), on FREE: $0.005 + 60 × $0.0048 + 2 × $0.008 + 8 × $0.018 = $0.453.
1,000 noon products, using the README's measured 30-200 renders and 89-356 billed MB per 1,000: on FREE, $4.80 + $0.24-$1.60 + $1.60-$6.41 + $0.005 = about $6.65-$12.81. The README puts the same run at $2.23-$6.99 on GOLD.
Amazon Saudi Arabia went through datacenter connections in our run, so it had no traffic fee: 3 products = $0.005 + 3 × $0.0048 = $0.0194 on FREE.

For runs with more than about 50 noon URLs, the README recommends 2 GB of memory (one extra $0.005 start event). Apify's free plan gives $5 of usage every month, no credit card required; the actor has no free trial of its own. The free e-commerce price tracker query builder builds the input without code and shows a fixed example of the output shape.

Related guides

Ramadan and Eid sale price tracking, the Gulf's other big season
Retail sales calendar 2026-2027
How to spot fake Black Friday discounts
Amazon price scraper by ASIN

Disclosure: I built the Apify actor and this site. The comparison scripts work on prices from any source.

Sources: Khaleej Times: 11.11 sales in the UAE (7 November 2025), Khaleej Times: Black, White, Yellow and Green Friday explained (21 November 2025), Property Finder: UAE White Friday 2025 deals, TradeArabia: Dubai Shopping Festival 2026-27, Wikipedia: UAE dirham and Saudi riyal (dollar pegs), the actor's README (supported sites, pricing, memory), and live actor runs on 2026-10-17 and 2026-10-20 (amazon.sa Arabic page and noon Saudi status row: run rm7jhN0AiKNMb6V6K, build 0.1.16).

What Slippage Should I Set to Avoid Sandwich Attacks?

Omar Eldeeb — Wed, 28 Oct 2026 11:48:38 +0000

Set the lowest slippage tolerance that still lets your swap fill under normal price movement. As a starting point, that's about 0.1–0.5% on deep major pairs and about 0.5–1% on mid-size tokens. Go higher only on thin, fast-moving pools, and there an MEV-protected route matters more than the exact number. Your tolerance is the most a sandwich bot can push your price: 5% on a $1,000 swap lets your fill land up to $50 below the quote. Check whether the pool is actually being sandwiched before you choose.

This is an explainer about trade execution, not financial advice. It doesn't tell you what to buy or sell.

What slippage tolerance actually does

When you request a swap, the app quotes an output amount. That quote already includes your price impact, which is how far your own trade moves the pool's price. Slippage tolerance is the extra room you allow on top of the quote, to cover price moves between the moment you sign and the moment your transaction lands in a block. The app turns it into a minimum output:

minimum output = quoted output × (1 − slippage tolerance)

If the price moves further than that before your swap executes, the swap fails instead of filling. On a normal public route you still pay the network fee for the failed transaction. Uniswap says its web app suggests a tolerance "usually between 0.1% and 5%", based on live gas fees and your swap size.

Why high slippage invites sandwich attacks

A sandwich bot sees your pending swap and buys just before it, which pushes the price up. Your swap then fills at that worse price, and the bot sells just after you. The only thing that limits how far the bot can push you is your minimum output. Set it loosely, and you've told the bot exactly how much room it has.

Swap size	Tolerance	Most your fill can land below the quote
$1,000	0.5%	$5
$1,000	1%	$10
$1,000	5%	$50
$1,000	10%	$100

The table is plain arithmetic. It shows a ceiling, not the bot's profit: the bot also pays its own price impact, pool fees and gas or tips, so it only attacks when what it can take beats those costs. That's why academic work on sandwiches describes the trade-off as two-sided. A tolerance that's too small makes trades fail, and one that's too large lets bots profit. Larger trades and looser settings make you a more attractive target. Meme-coin traders get hit hard because they tend to set high slippage on thin, volatile pools.

A rule-of-thumb table

These are my starting points, not guarantees. They assume you've already checked your price impact on the swap screen, and the "MEV risk" columns come from the pool checks in the next section.

Pool and trade	Low MEV risk, no sandwiches seen	Medium MEV risk, or a few sandwiches	High MEV risk, or regular sandwiches
Stablecoin to stablecoin, deep pool	0.05–0.1%	0.05–0.1%, protected route	0.05–0.1%, protected route
Major pair (ETH/USDC, SOL/USDC), price impact under 0.1%	0.1–0.5%	0.1–0.3%, protected route	0.1–0.3%, protected route, split large orders
Mid-size token, price impact under 1%	0.5–1%	0.5%, protected route	Lowest that fills, protected route, trade smaller
Thin or new pool, price impact over 1%	1–3%	Protected route, trade smaller	Rethink the size; don't fix failed fills by raising slippage

Read across a row, not down a column. The same token can sit in a calm pool on one DEX and a heavily sandwiched pool on another, and the tolerance you pick should follow the pool you're actually routing through.

Use the pool's MEV numbers to pick a setting

The MEV Risk Scanner reads Codex's per-pool data for the last 24 hours. Four of its fields decide which column of the table you're in:

sandwich_rate_24h and estimated_sandwich_attacks_24h: the share of the pool's transactions that were sandwiched, and that rate multiplied by the day's transaction count. Anything above zero means bots are active in that pool right now.
mev_risk_level: the most common hourly label. Codex labels each hour low when builder tips are under 3% of fees, medium at 3–30%, and high above 30%. It measures how hard bots of every kind compete in the pool, not sandwiches alone.
data_availability: if the sandwich or fee fields weren't populated, treat the pool as unknown, which means the middle column.
safer_trading_windows_utc: the hours in the last day that were labelled low. Use it as a hint about busy hours, not a schedule.

Two example rows, abbreviated and rounded, from a test run on 1 October 2026 (not live data):

[
  { "token0_symbol": "WETH", "token1_symbol": "USDt", "exchange": "UniswapV3",
    "liquidity_usd": 56961580, "transactions_24h": 708,
    "mev_risk_level": "low", "mev_to_total_fees_ratio_24h": 0.034,
    "sandwich_rate_24h": 0, "estimated_sandwich_attacks_24h": 0 },
  { "token0_symbol": "USDC", "token1_symbol": "WETH", "exchange": "UniswapV3",
    "liquidity_usd": 3417730, "transactions_24h": 22633,
    "mev_risk_level": "high", "mev_to_total_fees_ratio_24h": 0.305,
    "sandwich_rate_24h": 0.00063, "estimated_sandwich_attacks_24h": 14 }
]

Both pools trade ETH against a dollar stablecoin on Ethereum. The first had no detected sandwiches and a low builder-tip share, so it falls in the left-hand column. The second had about 14 estimated sandwiches in a day and 30% of its fees going to builders, so it's the right-hand column: a tight setting and a protected route. A router may split your trade across both pools, so if you're choosing a pool by hand, choose with this in mind.

Check the chart before you trade

The numbers above cover the last 24 hours. Before you sign, look at the live chart too. Open the token on Defined, switch the chart to 1-minute candles, and check two things:

How much the price moves in a minute. If recent 1-minute candles swing by 2%, a 0.3% tolerance will fail often. Wait, or trade smaller, rather than opening the gate wide.
Whether the trade list shows sandwiches. Look for the same wallet buying just before someone else and selling just after, at the same timestamp. How to tell if a token's pool is getting sandwiched walks through it step by step.

When your swap keeps failing at low slippage

Raise it in small steps, for example from 0.5% to 0.8%, not straight to 10%.
Use a protected route. Flashbots Protect on Ethereum keeps your transaction out of the public mempool, and it doesn't include transactions that would revert, so a too-tight setting doesn't cost you a fee. On Solana, Jupiter offers dynamic slippage and an MEV-protect mode that routes through Jito block engines, for slightly higher fees.
Trade smaller, so your price impact and the room you leave both shrink.
Taxed tokens are a special case. If a token charges a transfer tax, your tolerance has to cover the tax or the swap fails, and a bot gets that same room. Be extra careful with size on those.

Disclosure: I built the MEV Risk Scanner.

FAQ

Is 0.5% slippage safe from sandwich attacks?

On a deep major pair with small price impact, 0.5% leaves a bot little room, and an MEV-protected route closes most of the rest. No setting is completely safe, though. On a thin pool with regular sandwiches, even 0.5% of a large trade can be worth attacking.

What slippage should I use for meme coins?

Use the lowest setting that fills, usually higher than for major pairs because these prices move fast. Check the pool first. If it shows regular sandwiches, use a protected route and trade smaller instead of raising slippage.

Why does my swap fail when I set low slippage?

The price moved past your minimum output before your transaction landed, either from normal volatility or from other trades in the same block. Raise the tolerance a little at a time, or use a route that doesn't charge you for reverted transactions.

Does auto slippage protect me from MEV?

Auto or dynamic slippage sets the tolerance from the current market for each trade, which usually keeps it tighter than a fixed high value. That reduces the room a bot gets, but it doesn't hide your transaction. A protected route does that.

Can I use high slippage if I send through a private RPC?

A private RPC hides your swap from public-mempool bots, which is the biggest risk. A 2026 study still found sandwiches against "protected" transactions on several chains, so keep slippage sensible even on a private route.

What's the difference between price impact and slippage?

Price impact is how far your own trade moves the pool's price, and it's already built into your quote. Slippage tolerance is the extra movement you accept on top of the quote before the swap fails.

Does a low `mev_risk_level` mean I can loosen slippage?

Not by itself. mev_risk_level reflects the builder-tip share of fees, not sandwiches. Check sandwich_rate_24h too, and remember that both describe the last 24 hours, not your next block.

What slippage should I set for stablecoin swaps?

Stablecoin pairs barely move, so 0.05–0.1% usually fills in a deep pool. Anything much looser only gives a bot room to work with.

Sources

Uniswap, "What is slippage in crypto?": https://blog.uniswap.org/what-is-slippage-crypto
MEV Blocker, "Sandwich attacks": https://docs.mevblocker.io/concepts/mev-attack-types/sandwich-attacks
Heimbach and Wattenhofer, "Eliminating Sandwich Attacks with the Help of Game Theory" (2022): https://arxiv.org/abs/2202.03762
Helius, "Solana MEV report" (memecoin slippage, Jupiter dynamic slippage and MEV Protect): https://www.helius.dev/blog/solana-mev-report
Heimbach, Solmaz, Öz and Torres, "No Place to Hide: An Analysis on Protected Order Flow Sandwich Attacks" (2026): https://arxiv.org/abs/2609.28115
Flashbots Protect overview: https://docs.flashbots.net/flashbots-protect/overview

Walmart and Target Price Scraper: Search and Product Pages

Omar Eldeeb — Wed, 28 Oct 2026 11:48:17 +0000

A Walmart or Target price scraper reads the price, was-price, stock and GTIN from product pages, or from every product on a search or category page. Both retailers work without a browser. Match the same item across the two by GTIN, not by name: Walmart printed UPCs with 12 digits and Target with 13 (a leading zero) in our runs, so strip leading zeros first.

This guide uses an Apify actor that supports both retailers and shows real output from 2026-10-20, a Python script that matches the same products on both stores, and the caveats that matter for price data.

What works on each retailer

From the E-commerce Product Scraper Pro supported-sites table (both listed as supported):

	Walmart (`walmart.com`)	Target (`target.com`)
Product pages	yes	yes
Category and search pages	yes, with pagination	yes, with pagination
Keyword search	yes	yes
Reviews	all pages	all pages
Sellers	marketplace seller profiles	store contact only
How pages are reached	Datacenter	Datacenter, with a crawler user agent

Walmart Mexico (walmart.com.mx) is supported too, with prices in MXN: product pages, category and search pages and keyword search over datacenter connections, and reviews best-effort (from the page's data). walmart.ca and Bodega Aurrera (bodegaaurrera.com.mx) are best-effort. In the actor's AUTO mode, each page starts on the cheapest route that works for the site and moves to a residential proxy or a browser only when it's blocked. For Target that first route is a crawler-style request. In our October test runs, all 70 Target product rows came that way. Walmart rows came over datacenter connections 288 times out of 340 and over residential 52 times. A residential page adds a traffic fee, billed only for items that delivered.

Don't switch the scrape mode to BROWSER for these two. In the actor's cloud tests, Walmart answered the browser with a PerimeterX challenge, and Target rendered its pages without a price.

A search page from each store

This input reads the first ten results of a "lego" search on each store, with monitor mode on so the next run flags price changes:

{
  "listingUrls": [
    { "url": "https://www.target.com/s?searchTerm=lego" },
    { "url": "https://www.walmart.com/search?q=lego" }
  ],
  "maxItemsPerStartUrl": 10,
  "maxProductResults": 20,
  "monitorMode": true
}

All five rows from a smaller run of the same input on 2026-10-20 (maxItemsPerStartUrl: 3, maxProductResults: 5; run F0TqL6QA5Rd6wTf9w), names shortened:

Store	`position`	`sponsored`	Product	`price`	`originalPrice`	`gtin`
Target	1	null	LEGO Minifigures Shrek Series Mystery Box 71053	4.99	null	0673419424301
Target	2	null	LEGO Marvel Spider-Man vs. Anti-Venom Heist 30725	4.99	null	0673419421164
Target	3	null	LEGO Speed Champions The Fast and The Furious Toyota Supra MK4 77260	27.99	29.99	0673419424769
Walmart	1	false	LEGO Harry Potter Hogwarts Castle and Grounds 76419	136.99	169.99	673419375863
Walmart	2	false	LEGO Creator 3 in 1 Cute Hamster with a Flower - 31376	9.97	20.39	673419419291

position is the product's rank on that search input. On Walmart it's the organic rank, 1 to n in the order Walmart lists the results. Walmart's ads are kept but flagged: sponsored: true, a separate adPosition, and position: null. They never take one of the organic slots your maxItemsPerStartUrl cap allows, and color or size swatches of the same item don't come back as separate products. On Target, sponsored is null because the actor can't tell Target's ads apart; treat it as unknown. To keep only organic results, drop the rows where sponsored is true.

originalPrice is the was-price the page shows, and it appears only when it's higher than price. Walmart's hamster set shows $20.39 against $9.97. A was-price is a retailer's claim, not proof of a discount, and stores don't agree on them: the GTIN script below finds the LEGO Daisies set at $9.89 on both stores, with a was-price only on Walmart. See spotting fake Black Friday discounts.

You can also skip the URLs and search by keyword: "keyword": "usb c cable" with "marketplaces": ["www.walmart.com", "www.target.com"] runs the same search on both stores, and each row carries the keyword. For Walmart Mexico, add "www.walmart.com.mx". A Walmart search with no results returns a status row with outcome: "not-found" ("Walmart's search lists no results for this keyword") and isn't billed beyond the run's start fee.

Python: match the same products on Walmart and Target

import os
from collections import defaultdict
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("constructive_calm/ecommerce-product-scraper-pro").call(run_input={
    "detailsUrls": [
        {"url": "https://www.walmart.com/ip/LEGO-Botanicals-11508/17091312501"},
        {"url": "https://www.target.com/p/lego-botanicals-daisies-building-toy-for-kids-11508/-/A-94756752"},
        {"url": "https://www.walmart.com/ip/LEGO-Minifigures-71053/18839271940"},
        {"url": "https://www.target.com/p/lego-minifigures-shrek-series-mystery-box-71053/-/A-95046345"},
    ],
    "maxProductResults": 4,
})

def gtin_key(g):
    """Walmart prints UPCs as 12 digits, Target as 13 with a leading 0: compare without leading zeros."""
    return g.lstrip("0") if g else None

by_gtin = defaultdict(dict)
for p in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{p['site']:<11} gtin={p['gtin']:<14} {p['price']:>6} {p['currency']} was={p['originalPrice']} "
          f"inStock={p['inStock']} seller={p['seller']} via={p['tierUsed']}  {p['name'][:40]}")
    if gtin_key(p["gtin"]):
        by_gtin[gtin_key(p["gtin"])][p["site"]] = p["price"]

for gtin, prices in by_gtin.items():
    if len(prices) > 1:
        low = min(prices.values())
        winners = [site for site, price in prices.items() if price == low]
        print(f"GTIN {gtin}: {prices} -> lowest: {' = '.join(winners)}")

Output on 2026-10-20 (pip install apify-client, APIFY_TOKEN set to your Apify API token):

target      gtin=0673419424301    4.99 USD was=None inStock=True seller=None via=crawler-ua  LEGO Minifigures Shrek Series Mystery Bo
target      gtin=0673419422000    9.89 USD was=None inStock=True seller=None via=crawler-ua  LEGO Botanicals Daisies Building Toy for
walmart-us  gtin=673419422000     9.89 USD was=14.99 inStock=True seller=Walmart.com via=datacenter  LEGO Botanicals Daisies Building Toy - B
walmart-us  gtin=673419424301     4.97 USD was=None inStock=True seller=Walmart.com via=datacenter  LEGO Minifigures Shrek Series 1-Pack - M
GTIN 673419424301: {'target': 4.99, 'walmart-us': 4.97} -> lowest: walmart-us
GTIN 673419422000: {'target': 9.89, 'walmart-us': 9.89} -> lowest: target = walmart-us

To match whole search pages instead of hand-picked URLs, run the listing input above and feed its rows to the same gtin_key grouping. Products that only one store carries simply won't pair.

Fields that differ between the two

Field	Walmart	Target
`sku`	Walmart item id (`17091312501`)	Target's TCIN (`94756752`)
`gtin`	12-digit UPC	13 digits, leading zero
`seller`	`"Walmart.com"` or the marketplace seller's name	null in our runs
`additionalProperties`	`specifications`, `highlights`, `sellerId`, `fulfillmentType`	site extras when present
`unitPrice`	when the page shows one, e.g. 0.027 per `fl oz` on a gallon of milk	when the page shows one
`variants[]`	pack sizes, colors and lengths with their own prices	colors and sizes, on products that have them

On Walmart, check seller before you compare prices. A third-party marketplace offer can be the one shown, and its price, shipping and returns aren't Walmart's. Seller profiles go in the sellerUrls input and return the seller's name, address, rating and published contact details.

Caveats for price data

Store and ZIP prices. The actor has no store or ZIP code input. Rows contain the price each page serves to a visitor who hasn't chosen a store. Don't treat it as the shelf price in a particular store.
Out-of-stock items. In our test runs, some out-of-stock Walmart products came back with inStock: false and no price. Store null as "unknown", not zero.
Was-prices differ by store. As the Daisies example shows, one store may show a was-price and the other not, for the same product at the same price.
Listings move. position reflects the search at that moment, and the order can change from run to run. On Walmart, ads are flagged (sponsored: true) and kept out of position; on Target they aren't identified.

What it costs

Pay per event, from the actor's pricing table. Both stores are usually reached without residential traffic, so the main events are:

Event	FREE	GOLD
Actor start (per run, 1 GB)	$0.005	$0.005
Product scraped	$0.0048	$0.0008
Listing page scraped	$0.0004	$0.00021

The README's example, a 20-page category with 1,000 products: 20 × $0.00021 + 1,000 × $0.0008 + $0.005 = $0.809 on GOLD. On FREE: 20 × $0.0004 + 1,000 × $0.0048 + $0.005 = $4.813. Pages that escalate to residential add $0.018 (FREE) to $0.014 (GOLD) per MB for items that delivered. Apify's free plan includes $5 of usage every month, with no credit card required. There's no free trial inside the actor.

Holiday season: one GTIN list, both stores, every day

Black Friday is Friday, November 27, 2026, and Cyber Monday is November 30. If you buy or sell toys, electronics or anything both retailers stock, build one input with the Walmart and Target URLs of the same GTINs and schedule it daily from early November with monitorMode: true. Each run then shows which store moved first and by how much (previousPrice, priceChange), and the GTIN script above shows which store is lower that day. To compare more stores than these two, see comparing Black Friday prices across stores. For dates and lead times, see the holiday sales calendar.

Related guides

Do it without code

The free e-commerce price tracker builder has a "Walmart only", "Target only" and "US: Amazon + Walmart + Target" keyword option. It produces a ready-to-run input for the actor and shows a fixed example of the output, not live prices.

Disclosure: I built the actor and this site.

Sources: actor runs on 2026-10-20 (search table: run F0TqL6QA5Rd6wTf9w, build 0.1.16; GTIN script; milk unit price), our October 2026 cloud test runs (tier counts, out-of-stock rows, and a zero-result Walmart search on build 0.1.16), and the actor's README (supported-sites table, browser-mode note, pricing).

Track AI Brand Mentions Over Time: Schedules and Drift

Omar Eldeeb — Wed, 28 Oct 2026 11:47:56 +0000

To track AI brand mentions over time, run the same buyer questions in the same assistants with the same settings on a schedule (monthly by hand, weekly with a script). Log every brand and cited domain per answer and compare runs prompt by prompt. Record the model and whether web search ran, to separate real change from updates and noise.

The first check tells you where you stand. This guide is about the second and later checks: what to hold constant, what causes drift, and how to tell a real change from noise. The comparison script at the end runs on the same CSV log as the free ChatGPT brand check.

Hold these fixed

If any of these change between runs, a change in your numbers may come from your setup, not the assistants.

Hold fixed	Why
Prompt wording	A reworded question is a different question. Add new prompts as new IDs; never edit old ones.
Competitor list	Share of voice is relative; a new competitor lowers everyone else's share.
Assistants and modes	Record web search on or off per answer. Searched and unsearched answers behave differently.
Personalisation	Same clean session every time: signed out, or memory and custom instructions off.
Location and language	Answers can be localised, so run checks from the same country and language each time.
Runs per prompt	More runs give a tighter estimate; changing the count changes the noise level.
Counting rules	What counts as a mention (not "I'm not familiar with X"), and how you record order and citations.

Save the raw answers, not just your tallies, so you can re-score old runs if you refine the rules.

What causes drift

Some changes are real, some come from the assistant, and some are noise.

Source	What happens	How to spot it
Model updates	The vendor replaces or updates the model behind the app or API.	Record the model name the app or API reports for each run. A jump across many prompts on the same date suggests an update.
Retrieval changes	The pages web search returns for a question change: new reviews, updated listicles, your own new pages.	Compare the cited domains between runs for the prompts that changed.
Crawler access	Blocking a search crawler removes your pages from that assistant's search answers. OpenAI documents that sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers."	Re-check robots.txt and CDN bot rules; see How AI assistants choose which brands and sources to mention.
Your own changes	Pricing pages, comparison pages, reviews, PR.	Note the date of each change, and compare the prompts it should affect.
Randomness	The same question gives different answers run to run.	Small samples. A 1-in-3 to 2-in-3 move on one prompt is usually noise.

How often to run

Cadence	Suits	Sample per run
Monthly	Most brands, done by hand	10–20 prompts × 3 runs per assistant
Weekly	Active content or PR work, scripted	The same set, so weeks are comparable
After a known change	A relaunch, a pricing change, a big review	One extra run, compared with the last scheduled one

Running more often doesn't make a single run more accurate. More answers per run does.

Compare two runs

This script compares two logs prompt by prompt (won, lost, up, down, same) and tests whether the overall mention rate changed by more than chance would explain:

import csv
import math
import sys
from collections import defaultdict

BRAND = "Northwind"


def load(path):
    """{(assistant, prompt_id): [True/False per run]} - did the answer name BRAND?"""
    out = defaultdict(list)
    for r in csv.DictReader(open(path, encoding="utf-8")):
        brands = [b.strip() for b in r["brands_in_order"].split(";")]
        out[(r["assistant"], r["prompt_id"])].append(BRAND in brands)
    return out


def two_proportion_p(k1, n1, k2, n2):
    """Two-sided p-value for 'the mention rate changed' (normal approximation)."""
    p = (k1 + k2) / (n1 + n2)
    se = math.sqrt(p * (1 - p) * (1 / n1 + 1 / n2))
    if se == 0:
        return 1.0
    z = (k2 / n2 - k1 / n1) / se
    return math.erfc(abs(z) / math.sqrt(2))


before, after = load(sys.argv[1]), load(sys.argv[2])
print(f"{'prompt':<16}{'before':>8}{'after':>8}  change")
for key in sorted(set(before) | set(after)):
    b, a = before.get(key, []), after.get(key, [])
    if not b or not a:
        print(f"{'/'.join(key):<16}  only in one run: not comparable")
        continue
    rb, ra = sum(b) / len(b), sum(a) / len(a)
    change = "won" if rb == 0 and ra > 0 else "lost" if rb > 0 and ra == 0 else (
        "up" if ra > rb else "down" if ra < rb else "same")
    print(f"{'/'.join(key):<16}{sum(b)}/{len(b):<6}{sum(a)}/{len(a):<6}  {change}")

k1 = sum(sum(v) for v in before.values()); n1 = sum(len(v) for v in before.values())
k2 = sum(sum(v) for v in after.values()); n2 = sum(len(v) for v in after.values())
print(f"\nmention rate {k1}/{n1} = {k1 / n1:.0%} -> {k2}/{n2} = {k2 / n2:.0%}, "
      f"p = {two_proportion_p(k1, n1, k2, n2):.2f}")

python compare_runs.py log_2026-09.csv log_2026-10.csv on two small example logs (the brands are placeholders, not real measurements):

prompt            before   after  change
chatgpt/P1      1/3     3/3       up
chatgpt/P2      0/3     1/3       won
chatgpt/P3      3/3     2/3       down

mention rate 4/9 = 44% -> 6/9 = 67%, p = 0.34

The headline looks like a 23-point gain. With 9 answers per month, p = 0.34 says a change this size would turn up often by chance alone. Same rates with more answers, using the same test:

Before	After	p
12/30 (40%)	18/30 (60%)	0.12
40/100 (40%)	50/100 (50%)	0.16
80/200 (40%)	100/200 (50%)	0.04
40/100 (40%)	60/100 (60%)	0.005

A rule of thumb from this: on 100 answers a run, a 20-point move is solid evidence; a 10-point move needs about 200 answers a run before it stands out. The prompt-by-prompt view is still useful at small sizes. It tells you where to look, such as which cited sources changed on a prompt you lost, even when the total isn't conclusive.

Pitfalls

Comparing different prompt sets. If you added prompts, compare only the prompts present in both runs. The script marks the others as not comparable.
Mixing searched and unsearched answers. If one month used search more often than the other, the rates move for that reason alone. Compare within each mode.
Reacting to one prompt. One prompt going from 3/3 to 2/3 is one answer's difference.
Forgetting branded prompts. Mention rate on branded prompts is always high. Keep them out of the totals and use them to track what the assistant says about you.

Related guides

How to check if ChatGPT mentions your brand (free), which defines the log format.
AI share of voice: measure it in ChatGPT and Perplexity.
How AI assistants choose which brands and sources to mention.
Measure AI share of voice in Python, for collecting the answers through the APIs.

Do it without code

The free LLM visibility query builder gives you a ready-to-run audit input from your brand, category and competitors, with a fixed example of the output. It doesn't call any AI model from your browser.

At scale

LLM Visibility Monitor on Apify is built for scheduled runs. It saves the generated question set so every run asks the same questions, compares each run with your previous run for the same brand, country and language (questions won, lost and unchanged), and records the model it requested and the model that actually answered on every row. You can schedule it weekly through Apify. It's free to start, then pay-as-you-go.

Disclosure: I build the datatooly query builder and the Apify actor. The comparison script works on any hand-collected log.

Sources: OpenAI: overview of OpenAI crawlers, OpenAI Help Center: Searching the web with ChatGPT, and the two-proportion z-test (computed in the script above).

When Do Black Friday Prices Drop? The 2026 Timeline

Omar Eldeeb — Wed, 28 Oct 2026 11:47:34 +0000

Black Friday prices don't drop on one day. In 2025, Walmart's first Black Friday event started November 14, Amazon's Black Friday Week November 20, Target's event November 23, and Walmart's main event went live online on the Tuesday before Thanksgiving, with deals running through Cyber Monday. For 2026 (Thanksgiving November 26, Black Friday November 27), check daily from early November and see when each product actually hits its low.

This guide lays out the 2025 timeline from primary sources, maps it onto the 2026 calendar, and gives a script that reads your daily price snapshots and tells you the day each product reached its season low.

The 2025 timeline

Date (2025)	Day	What happened	Source
Nov 13, 7 p.m. ET	Thu	Walmart+ early access to Walmart's Deals Event 1	Walmart
Nov 14 – 16	Fri – Sun	Walmart Black Friday Deals Event 1, online and in stores	Walmart
Nov 20, 12:01 a.m. PST	Thu	Amazon's US Black Friday Week starts, running through Dec 1	Amazon
Nov 23 – 29	Sun – Sat	Target's Black Friday event	Target
Nov 25 – 27	Tue – Thu	Walmart Deals Event 2, online only	Walmart
Nov 27	Thu	Thanksgiving: $6.4 billion US online spending	Adobe
Nov 28	Fri	Black Friday: $11.8 billion online, up 9.1% year on year; Walmart Event 2 opens in stores (to Nov 30)	Adobe, Walmart
Nov 29 – 30	Sat – Sun	Weekend: Adobe says "earlier deals drove record spending over the weekend"; Target's Cyber Monday event opens Nov 30	Adobe, Target
Dec 1	Mon	Cyber Monday: $14.25 billion online, Walmart's Cyber Monday event (online only)	Adobe, Walmart
After Dec 1		Adobe: "After Cyber Week, discounts will come down from their peaks"	Adobe

Adobe's spending figures are for all US online retail, not for any one store. And spending isn't price: a busy day can be a busy day because people waited for payday, not because prices were lowest. The table tells you when deals were available, not when a given product was cheapest.

What that suggests for 2026

None of these retailers has announced 2026 dates as of October 20, 2026. In 2025, Walmart published its event dates on October 27, and Target's detailed Black Friday release came out on November 20. Expect 2026 announcements between late October and mid-November.

If retailers keep the same weekday offsets from Thanksgiving (an assumption, not an announcement), 2026 would look like this:

2025 event	Days before Thanksgiving	Same offset in 2026
Walmart Event 1 starts (Fri Nov 14)	13	Fri Nov 13
Amazon Black Friday Week starts (Thu Nov 20)	7	Thu Nov 19
Target Black Friday event starts (Sun Nov 23)	4	Sun Nov 22
Walmart Event 2 online (Tue Nov 25)	2	Tue Nov 24
Thanksgiving	0	Thu Nov 26
Black Friday	−1	Fri Nov 27
Cyber Monday	−4	Mon Nov 30

So a baseline that starts by late October is clean: it's before every 2025 event. A tracker that only starts in the week of Thanksgiving has no "before" to compare with.

Why you have to measure it yourself

"Prices drop on Black Friday" is an average. Product by product, the low can come earlier, later, or not at all. Which?, the UK consumer group, looked at 227 tech and appliance deals from Black Friday 2023 and found that "every one of the 'deals' we checked was available for the same price or less" at other times in the six months before or after, and around four in 10 were cheaper at some other point (Which?, 26 Nov 2024). That's UK retailers and one year, but it shows that the "best day" is product-specific.

The way to find it is a daily price snapshot per product from late October to early December. The Black Friday price tracker guide sets that up with a scheduled run of E-commerce Product Scraper Pro on Amazon, Walmart and Target, and exports the history to a price_history.csv file with these columns:

scrapedAt,site,url,name,price,originalPrice,currency,inStock,seller,changeType,previousPrice

Find the day each product hit its low

The script below works on that file. It takes the lowest price per product per US Eastern day (so it copes with several runs a day), fills in unchanged days (with onlyChanges, a product only appears on days its price or stock changed), and reports the first day each product reached its season low, how long the low lasted, and which phase of the season that day falls in.

import csv
import sys
from collections import Counter, defaultdict
from datetime import date, datetime, timedelta
from zoneinfo import ZoneInfo   # Windows: pip install tzdata

ET = ZoneInfo("America/New_York")
START, END = date(2026, 10, 21), date(2026, 12, 7)
PHASES = [  # (first day, label) for the 2026 season; edit as retailers announce dates
    (date(2026, 10, 21), "baseline (before Nov 1)"),
    (date(2026, 11, 1), "early November"),
    (date(2026, 11, 13), "pre-Thanksgiving events"),
    (date(2026, 11, 26), "Thanksgiving Day"),
    (date(2026, 11, 27), "Black Friday"),
    (date(2026, 11, 28), "weekend"),
    (date(2026, 11, 30), "Cyber Monday"),
    (date(2026, 12, 1), "after Cyber Monday"),
]

def phase(d):
    return [label for start, label in PHASES if d >= start][-1]

# Lowest price seen per product per US-Eastern day (handles several runs a day).
lows = defaultdict(dict)
names = {}
for r in csv.DictReader(open(sys.argv[1], encoding="utf-8")):
    if not r["price"]:
        continue
    day = datetime.fromisoformat(r["scrapedAt"].replace("Z", "+00:00")).astimezone(ET).date()
    p = float(r["price"])
    lows[r["url"]][day] = min(p, lows[r["url"]].get(day, p))
    names[r["url"]] = r["name"]

tally = Counter()
for url, by_day in lows.items():
    earlier = [d for d in by_day if d < START]
    series, last, d = {}, (by_day[max(earlier)] if earlier else None), START
    while d <= END:                      # carry the price forward over unchanged days
        last = by_day.get(d, last)
        if last is not None:
            series[d] = last
        d += timedelta(days=1)
    if not series:
        continue
    low = min(series.values())
    if low == max(series.values()):
        print(f"{names[url][:12]:<13} flat at {low:.2f} all season")
        continue
    low_days = [d for d, p in series.items() if p == low]
    first, last_low = low_days[0], low_days[-1]
    tally[phase(first)] += 1
    print(f"{names[url][:12]:<13} low {low:>8.2f} first {first:%a %b %d} ({phase(first)}), "
          f"held {len(low_days)} day(s) until {last_low:%b %d}")

print("\nProducts whose season low started in each phase:")
for _, label in PHASES:
    print(f"  {label:<26}{tally[label]}")

Here's the output on a made-up four-product season (the products and prices are invented to show each pattern; the file format is the real export format):

TV 1          low   399.99 first Thu Nov 26 (Thanksgiving Day), held 5 day(s) until Nov 30
Vacuum 2      low   179.00 first Fri Nov 13 (pre-Thanksgiving events), held 4 day(s) until Nov 16
Toy 3         low    39.99 first Mon Nov 30 (Cyber Monday), held 2 day(s) until Dec 01
Cable 4       flat at 7.97 all season

Products whose season low started in each phase:
  baseline (before Nov 1)   0
  early November            0
  pre-Thanksgiving events   1
  Thanksgiving Day          1
  Black Friday              0
  weekend                   0
  Cyber Monday              1
  after Cyber Monday        0

How to read it:

TV 1 dropped in steps and reached its low on Thanksgiving Day, then held it through Cyber Monday. Waiting for Black Friday cost nothing, but waiting past Monday would have.
Vacuum 2 was cheapest in an early event and never matched that price again. Its later "Black Friday" price was a smaller discount. This is the pattern the 2025 Walmart Event 1 dates make possible.
Toy 3 kept falling until Cyber Monday.
Cable 4 never moved. Plenty of products don't take part in the sales at all.

The bottom table is the one to keep. After the season, run it on your whole watchlist and you get your own answer to "when do prices drop" for the categories you care about. Keep the file: next year you can set your alerts around the phase where your products actually bottomed out.

Details that change the answer

Time zones. scrapedAt is UTC. A drop at 11 p.m. Eastern on Thanksgiving is already Friday in UTC, which is why the script converts to Eastern before picking the day.
Run frequency. With one run a day, a low that lasted six hours can be missed. From Thanksgiving to Cyber Monday, run every few hours; the Cyber Monday deal tracker guide works out what hourly checks cost.
Out of stock. A low price on a day the item was sold out isn't a price you could pay. If that matters, skip rows where inStock is false before computing lows.
Is the low a real deal? A season low can still be the normal price if the item was raised beforehand. The fake Black Friday discounts guide compares it with the 30-day median and the 90-day low.
Edit the phases. When retailers announce 2026 dates, change the PHASES list so the labels match the real events.

For the year's other sales and when to start each baseline, see the holiday sales calendar and price tracking guide, and for gift-buying after Cyber Monday, the Christmas price tracker. The free e-commerce price tracker page builds the tracking input.

Disclosure: I built the actor and this site. The script reads any CSV with these columns, whatever collected the prices.

Sources: Walmart 2025 Black Friday Deals events (Oct 27, 2025), Amazon: When is Black Friday 2025, Target press release (Nov 20, 2025), Adobe: Cyber Monday hits record (Dec 2, 2025), Which?: Black Friday deals exposed as duds (Nov 26, 2024).

How to Tell if a Token's Pool Is Getting Sandwiched Before You Swap

Omar Eldeeb — Wed, 28 Oct 2026 11:47:13 +0000

To tell whether a token's pool is getting sandwiched, look through its recent trades for one wallet that buys just before someone else's buy and sells just after it, in the same block. Then check two numbers for the last 24 hours: the share of the pool's transactions that were sandwiched, and the share of its fees that went to MEV. If either is high, trade with tight slippage through an MEV-protected route, or trade smaller.

This is an explainer about how trades get executed. It is not financial advice and says nothing about whether any token is worth buying.

What a sandwich attack is

A sandwich attack is three swaps in one block. A bot buys the token just before your pending buy (the front-run), your buy fills at the higher price the bot just created, and the bot sells right after you (the back-run) and keeps the difference. Your slippage tolerance caps how far the bot can push your price: if the front-run moves the price past your minimum output, your swap reverts instead of filling.

The bot has to see your swap before it lands in a block. On Ethereum that usually means the public mempool. Solana has no public mempool, but sandwiches still happen there through private mempools and validators that see transactions before they're included.

Check 1: read the pool's trade feed

The most direct evidence is in the trades themselves. Open the pool's chart and trade list, for example this USDC/WETH Uniswap v3 pool on Defined, and work through these steps:

Switch the chart to a 1-minute timeframe and click a busy candle. The trade table then shows only the trades inside that candle.
Look for a triplet: wallet A buys, a different wallet B buys, then wallet A sells about the same amount it bought, with the same timestamp.
Filter the trade table by wallet A's address. A sandwich bot repeats that buy-then-sell rhythm around other people's trades, often many times a day on the same pool.
To confirm a single case, open the three transactions in a block explorer (Etherscan, Solscan, Basescan) and check that they share a block number and appear in that order.

Some patterns look similar but aren't sandwiches. A wallet that buys and sells minutes apart is just a trader, and a lone sell right after a big swap is usually arbitrage. A sandwich needs both legs, one on each side of a victim. Checking by eye also only covers one pool for one hour. It can't tell you how often the pool gets hit.

Check 2: measure the sandwich rate and the MEV share

Codex indexes DEX trades and flags sandwiched transactions for each pool. The MEV Risk Scanner on Apify reads that data for a token's top pools, for specific pool addresses, or for a chain's busiest pools. For each pool it returns these fields covering the last 24 hours:

Field	What it tells you
`sandwich_rate_24h`	Share of the pool's transactions that were sandwiched (the average of the hourly rates)
`estimated_sandwich_attacks_24h`	That rate multiplied by the 24h transaction count, rounded. It's an estimate, not a list of attacks
`mev_to_total_fees_ratio_24h`	Share of the pool's fees paid to block builders as tips
`mev_risk_level`	The most common hourly label. Codex labels each hour `low` (under 3% builder tips), `medium` (3–30%) or `high` (over 30%)
`fee_regime_24h`	`mev-dominated`, `gas-dominated` or `pool-fee-dominated`
`safer_trading_windows_utc`	Start times of the hours labelled `low`
`data_availability`	How much of each field was populated, so you can tell "zero" from "not indexed"

Here's an example row, abbreviated and rounded, from a test run on 1 October 2026. It isn't live data:

{
  "pair_address": "0xe0554a476a092703abdb3ef35c80e0d76d32939f",
  "chain": "ethereum",
  "exchange": "UniswapV3",
  "token0_symbol": "USDC",
  "token1_symbol": "WETH",
  "liquidity_usd": 3417730,
  "transactions_24h": 22633,
  "mev_risk_level": "high",
  "mev_to_total_fees_ratio_24h": 0.305,
  "fee_regime_24h": "mev-dominated",
  "sandwich_rate_24h": 0.00063,
  "estimated_sandwich_attacks_24h": 14,
  "safer_trading_windows_utc": []
}

In plain terms, about 0.06% of roughly 22,600 transactions were sandwiched, around 14 in a day. Meanwhile about 30% of the pool's fees went to block builders, and no hour was quiet enough to count as low.

Those are two different signals. mev_risk_level measures builder tips, and builder tips pay for every kind of MEV, including arbitrage, back-runs and liquidations, not just sandwiches. A pool can be high and see few sandwiches, or low and still get sandwiched. Use sandwich_rate_24h as your sandwich signal, and mev_risk_level as a measure of how hard bots compete in that pool. Compare pools that trade the same pair: one whose rate is several times higher than its siblings is the one to route around.

Check 3: compare your trade size with the pool

A sandwich only pays if the bot's profit beats its own gas, fees and tips. That profit grows with two things: how far your trade moves the price, and how much slippage you allow. Your swap screen's price impact figure shows how much your trade alone moves the price. A trade with almost no price impact in a deep pool is a thin target. A trade that moves a thin meme-coin pool by several percent, sent with a loose slippage setting, is exactly what bots look for.

Liquidity figures aren't always comparable across sites, because they count pool depth differently. Compare figures from the same source.

Check 4: decide how to send the trade

Use an MEV-protected route. On Ethereum, Flashbots Protect sends your transaction to a private mempool that sandwich bots can't see. It doesn't include transactions that would revert, so you don't pay for failed swaps. MEV Blocker works in a similar way. On Solana, some aggregators offer an MEV-protect mode that routes through Jito block engines, at slightly higher fees.
Keep slippage tight. The tighter it is, the less room a front-run has. The companion guide What slippage should I set to avoid sandwich attacks? covers how to choose a number.
Trade smaller, or split the order, when your price impact is high.
Treat timing as a hint. safer_trading_windows_utc shows which hours yesterday had a low builder-tip share. It doesn't promise anything about tomorrow.

What these checks can't tell you

They describe the past 24 hours, not your next transaction. The bot that targets you only needs to see your swap once.
Detection comes from Codex, and coverage varies. Some Solana pools return a sandwich rate but no fee breakdown. On L2s like Base, builder tips are empty or zero, so mev_to_total_fees_ratio_24h comes back null there. Read data_availability before trusting a zero.
Protected routes reduce the risk but don't remove it. A 2026 measurement study found sandwich attacks against "protected" transactions on every chain it examined, and noted that these often don't sit tightly around the victim. That means they won't always show up as the neat triplet from Check 1.

Run it for your own tokens

The free MEV pair scanner query builder turns a token or pool address and a chain into a ready-to-run input, and shows the output shape. It doesn't call Codex from your browser. The actor runs the scan: up to 20 pools per run, at 1-hour bars by default (5-minute to 1-day bars are also available), with an optional one-line AI verdict. It's free to start, then pay-as-you-go per pool scanned. If you'd rather write the code yourself, the Python version of this check calls Codex directly.

Disclosure: I built the query builder and the actor.

FAQ

Can I see a sandwich attack on a DEX chart?

Not on the candles themselves. You can see one in the trade list: the same wallet buys just before another trader's buy and sells just after it, in the same block. Click a 1-minute candle to filter the trades down to that minute, then filter by the suspected bot's wallet.

Does a high MEV risk level mean the pool is getting sandwiched?

Not necessarily. mev_risk_level comes from the share of fees paid to block builders, and that covers arbitrage and liquidations as well as sandwiches. Check sandwich_rate_24h for sandwiches specifically.

Is a pool with a 0% sandwich rate safe to swap in?

It means few or no sandwiches were detected in the last 24 hours, or, if data_availability shows the field as missing, that nothing was measured. A loose slippage setting on a public route can still get you sandwiched, so keep slippage tight either way.

Are meme coins sandwiched more than major pairs?

Meme-coin traders are an easy target because they often set high slippage when trading thin, volatile pools. The pool's own numbers matter more than the category, though, so check each pool.

Do private RPCs stop sandwich attacks?

They hide your transaction from the public mempool, and that removes the most common way bots find victims. Research published in 2026 still found sandwiches against protected transactions on several chains, so treat a private RPC as a big risk reduction, not a guarantee.

Why are some fields null for Solana or Base pools?

Codex's fee indexing varies by pool on Solana, and L2s like Base don't have a builder-tip market like Ethereum mainnet's. A null value means "not available", not zero.

Which chains can I check?

Any network Codex indexes, which is more than a hundred, including Ethereum, Solana, Base, BNB Chain and Arbitrum. Paste a bare 0x address with a default chain selected, a Solana address as-is, or chain:address to mix chains in one run.

Sources

MEV Blocker, "Sandwich attacks": https://docs.mevblocker.io/concepts/mev-attack-types/sandwich-attacks
Helius, "Solana MEV report" (private mempools, memecoin slippage, Jupiter MEV Protect): https://www.helius.dev/blog/solana-mev-report
Heimbach and Wattenhofer, "Eliminating Sandwich Attacks with the Help of Game Theory" (2022): https://arxiv.org/abs/2202.03762
Heimbach, Solmaz, Öz and Torres, "No Place to Hide: An Analysis on Protected Order Flow Sandwich Attacks" (2026): https://arxiv.org/abs/2609.28115
Flashbots Protect overview: https://docs.flashbots.net/flashbots-protect/overview
Defined docs, transaction filtering: https://docs.defined.fi/trading/defined-features/transaction-filtering

Apify E-commerce Scraping Tool Alternative: Same Input

Omar Eldeeb — Wed, 28 Oct 2026 11:46:52 +0000

E-commerce Product Scraper Pro is an alternative to Apify's E-commerce Scraping Tool that takes the same input: all 38 field names, types and options, so your input JSON runs after you change the actor ID. It adds monitor mode, per-input caps, variants and a proxy-country option. It supports fewer sites, and its pricing differs: cheaper per product, more expensive on residential and browser pages.

I built E-commerce Product Scraper Pro. This page tries to give a fair account of when switching makes sense and when it doesn't. Facts about the E-commerce Scraping Tool come from its Store page, its input schema and its public pricing record, all checked on 2026-10-20. Apify maintains it, and its public actor record on the Apify API showed 848 users in the last 30 days.

What stays the same

The input. Every field of the E-commerce Scraping Tool's input schema exists in the alternative with the same name, type and options: detailsUrls, listingUrls, keyword, marketplaces, scrapeMode, the AI summary fields, the review, seller and influencer sections, the Google Shopping ("search engine") section and the food-delivery section. The internal disableFallbacks flag is accepted and ignored.

We checked this against the E-commerce Scraping Tool's current build (0.0.120, published 2026-10-20): all 38 of its input keys and every option value of its select fields (251 marketplaces, 226 marketplacesReviews, 239 countryCode values and the rest) are in the alternative's input schema. That includes www.walmart.com.mx and www.bodegaaurrera.com.mx, the two Mexican values in marketplaces and marketplacesReviews. Walmart Mexico is a supported site on the alternative (datacenter, prices in MXN); Bodega Aurrera is best-effort.

The main output fields. Product rows keep the field names the E-commerce Scraping Tool documents: url, inputUrl, name, image, offers (price, priceCurrency), brand, description, additionalProperties, aiSummary and keyword, plus the review, seller and influencer keys. The compatibility was checked against that tool's documented dataset views and samples, not field by field on its live output, so check your own integration against a sample run (script below).

What's added on top

Addition	What it does
`maxItemsPerStartUrl`	A cap per input (per product URL, listing, keyword and marketplace, review input or storefront), so one large category can't use the whole run
`followPagination`	Turn off next-page following on listings (on by default)
`includeVariants`	A `variants[]` array with each variant's SKU, options, price and stock (on by default)
`proxyCountry`	Pin the residential exit to a country, to get prices in that market's currency
`monitorMode`, `onlyChanges`	Compare with the previous run: `price-down`, `price-up`, `back-in-stock`, `out-of-stock`, `removed`; optionally write only changes

Every row also has flat, spreadsheet-friendly fields next to the nested ones (price, currency, originalPrice, unitPrice, inStock, stockQty, sku, gtin, images) and a recordType (product, review, seller, influencerPost, search-result, delivery). The README publishes a supported-sites table with each site's status and route.

What's different

Sites

The E-commerce Scraping Tool's Store page names eBay, Etsy, Best Buy, Home Depot, Costco, Alibaba, Argos and Mercado Livre among the marketplaces it works with. On 2026-10-19 its pricing record added an "Unblocker" proxy for "stores that block even residential proxies".

E-commerce Product Scraper Pro marks those sites unsupported: eBay, Etsy, Best Buy, Home Depot, Lowe's, Costco, Alibaba, Allegro, Newegg, Argos, Mercado Livre (Brazil) and about 30 more. Inputs for them get a status row explaining why and aren't fetched or billed beyond the run's start fee. If your work depends on those marketplaces, keep using the E-commerce Scraping Tool for them. You can split your inputs and run each actor on the sites it handles.

Sites the alternative lists as supported after its own cloud tests: Shopify and WooCommerce stores, Amazon (amazon.com and amazon.de supported, 22 other storefronts best-effort), Walmart (walmart.com and walmart.com.mx), Target, Zalando, John Lewis, M&S, Aldi UK and Waitrose. Several more are best-effort; the table on the actor page lists each one.

Where errors go

The E-commerce Scraping Tool writes failures to a named errors dataset. The alternative writes one status row per input (successes too) to a separate dataset, linked on the run's Output tab as "Status per input". Through the API it's run["storageIds"]["datasets"]["status"]. The error field uses the same error codes as the E-commerce Scraping Tool's documented errors, and outcome, blockedBy, itemsFound, itemsDelivered and stopReason say what happened.

Pricing

Both are pay per event. Prices below are from the E-commerce Scraping Tool's public pricing record (in effect since 2026-10-19; its Store page headline says "from $1.00 / 1,000 product details") and from E-commerce Product Scraper Pro's pricing table, for Apify's FREE and GOLD plan tiers:

Event	E-commerce Scraping Tool (FREE / GOLD)	E-commerce Product Scraper Pro (FREE / GOLD)
Run start	$0.0001 / $0.00007	$0.005 / $0.005 (per GB of memory)
Product	$0.006 / $0.001	$0.0048 / $0.0008
Listing page	$0.0005 / $0.00026	$0.0004 / $0.00021
Review	$0.002 / $0.0007	$0.0016 / $0.00056
Seller	$0.003 / $0.001	$0.0024 / $0.0008
AI summary	$0.0075 / $0.00275	$0.006 / $0.0022
Residential proxy	$0.003 / $0.0008 flat per product	$0.018 / $0.014 per MB used to deliver the item
Browser rendering	$0.0006 / $0.00051 flat per product	$0.008 / $0.006 per render, plus residential MB
Unblocker proxy	$0.0015 / $0.001 per product	not offered (those sites are unsupported)
Google Shopping request	no separate event in its pricing record	$0.0045 / $0.0035 per successful request

The E-commerce Scraping Tool also has PLATINUM and DIAMOND tiers with lower prices (for example $0.0005 per product on DIAMOND). On the alternative, those plans pay the GOLD price.

What that means in practice:

Datacenter sites (Shopify, WooCommerce, amazon.com, Walmart, Target and others): the alternative is cheaper once the per-product saving covers its higher start fee. On FREE that's from 5 products per run: the start fee costs $0.005 − $0.0001 = $0.0049 more, and each product saves $0.0012. On GOLD it's from 25 products per run ($0.00493 ÷ $0.0002 = 24.7).
Residential sites: usually more expensive on the alternative. Its README example is 100 amazon.co.uk products at $0.46–$0.71 on GOLD. If all 100 needed the E-commerce Scraping Tool's residential fee, they'd cost 100 × ($0.001 + $0.0008) + $0.00007 ≈ $0.18 there.
Browser sites (Noon, Wayfair, Sephora and others): more expensive on the alternative, by its design: it bills each render and its traffic instead of a flat fee.
Monitoring: with onlyChanges, unchanged products are billed as "product checked" ($0.0012 FREE, $0.0002 GOLD) instead of a full product. The E-commerce Scraping Tool's input has no monitor-mode option.

The alternative doesn't charge for items that never delivered: blocked or empty pages cost nothing, and their residential traffic and renders aren't billed. Apify's free plan includes $5 of usage every month, with no credit card required. Neither actor needs your own proxies.

Python: run your existing input and check the fields you use

Paste your current input into RUN_INPUT and list the fields your code reads:

import os
from collections import Counter
from apify_client import ApifyClient

# Your existing E-commerce Scraping Tool input, unchanged
RUN_INPUT = {
    "detailsUrls": [{"url": "https://www.skullcandy.com/products/crusher-540-active"}],
    "listingUrls": [{"url": "https://www.deathwishcoffee.com/collections/coffee"}],
    "maxProductResults": 5,
}
# The fields your integration reads today (edit to match your code)
FIELDS_I_USE = ["url", "inputUrl", "name", "image", "offers", "brand", "description", "additionalProperties"]

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("constructive_calm/ecommerce-product-scraper-pro").call(run_input=RUN_INPUT)
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())

print(run["status"], "|", len(rows), "rows |", Counter(r.get("recordType") for r in rows))
missing = {f: sum(f not in r for r in rows) for f in FIELDS_I_USE}
print("missing fields:", {f: n for f, n in missing.items() if n} or "none")
print("offers shape:", rows[0]["offers"], "| brand shape:", rows[0]["brand"])

# Per-input outcomes: a separate "status" dataset (one row per input), not an "errors" dataset
for s in client.dataset(run["storageIds"]["datasets"]["status"]).iterate_items():
    print(s["inputField"], s["outcome"], s["error"], "|", (s["errorDescription"] or "")[:90])

Output on 2026-10-20 (pip install apify-client, APIFY_TOKEN set):

SUCCEEDED | 5 rows | Counter({'product': 5})
missing fields: none
offers shape: {'price': 229.99, 'priceCurrency': 'USD'} | brand shape: {'slogan': 'Skullcandy US'}
detailsUrls ok None | 
listingUrls ok None | 22 product(s) found on 1 page(s); 4 delivered; 18 not scraped; stopped: maxProductResults

The input is the E-commerce Scraping Tool's own field set (detailsUrls, listingUrls, maxProductResults) and ran unchanged. To compare values, not just field names, run the same input on both actors and join the rows on url.

Migration checklist

Check your sites against the supported-sites table on the actor page. Move unsupported marketplaces to a separate input that stays on the E-commerce Scraping Tool.
Check your marketplace values. Every value the E-commerce Scraping Tool offers validates on the alternative, but a value that validates isn't always a site that works. Look up each marketplace in your marketplaces and marketplacesReviews lists in the table: www.walmart.com.mx is supported, www.bodegaaurrera.com.mx best-effort, and values such as www.ebay.com or www.homedepot.com only return a status row.
Change the actor ID everywhere it appears: API calls, saved tasks, schedules, webhooks and integrations (Make, Zapier, n8n, Google Sheets). apify/e-commerce-scraping-tool becomes constructive_calm/ecommerce-product-scraper-pro. Saved tasks belong to an actor, so create new tasks from the same input.
Run a sample on both with a small maxProductResults, and compare the row counts, prices, currencies and stock flags for the same URLs.
Check output field mapping with the script above: the fields you read, the offers and brand shapes, and any additionalProperties keys you depend on. Per-site extras in additionalProperties differ between the two.
Move error handling from the errors dataset to the status dataset, and decide what counts as a failure (blocked, not-found, restricted, or a zero-row run).
Set a cost cap with the run's maximum cost (maxTotalChargeUsd in the API), and compare the charged events of the sample run with your current costs, especially on residential and browser sites. Read them from the run's OUTPUT record (billing.chargedEvents): the chargedEventCounts on the run object that call() returns can be incomplete.
Run both in parallel for a few days before turning the old schedule off.

Holiday season: switch before the sale, not during it

If you track prices for Black Friday (Friday, November 27, 2026) and Cyber Monday (November 30), do the migration and the side-by-side runs in early November. Changing the scraper mid-season breaks the continuity of your price history, and monitor mode needs at least one run as its baseline. Once switched, monitorMode gives you previousPrice and priceChangePct on every row. See the Black Friday price tracker for a schedule, and the holiday sales calendar for the dates.

Related guides

Competitor price monitoring.
Price drop alerts to email or Slack, built on monitor mode.
Scrape product variants, one of the added options.

Do it without code

The free e-commerce price tracker builder produces a ready-to-run input with the shared field names, and shows a fixed example of the output.

Disclosure: I built E-commerce Product Scraper Pro and this site. I'm not affiliated with Apify beyond publishing actors on its Store.

Sources: E-commerce Scraping Tool on the Apify Store (marketplaces named, pricing headline), its public actor record and its latest build 0.0.120 on the Apify API (pricing per event, input schema, 30-day users), checked 2026-10-20; the E-commerce Product Scraper Pro README; and an actor run on 2026-10-20.

SEC EDGAR API: Submissions JSON and Ticker to CIK

Omar Eldeeb — Wed, 28 Oct 2026 11:46:30 +0000

Yes, EDGAR has an API, and it's free with no API key. Map a ticker to a CIK with https://www.sec.gov/files/company_tickers.json, then fetch https://data.sec.gov/submissions/CIK##########.json, with the CIK zero-padded to 10 digits, to get the company's details and full filing history. The one requirement is a User-Agent header with your name and email.

This page covers the two building blocks every EDGAR project starts with: ticker-to-CIK and the submissions API. The code was run on 2026-10-14.

The SEC's APIs at a glance

What	URL	Key?
Ticker → CIK map	`https://www.sec.gov/files/company_tickers.json`	No
Ticker → CIK + exchange	`https://www.sec.gov/files/company_tickers_exchange.json`	No
Filing history + metadata	`https://data.sec.gov/submissions/CIK##########.json`	No
XBRL financials	`https://data.sec.gov/api/xbrl/companyfacts/CIK##########.json` (and `companyconcept`, `frames`)	No
Full-text search	`https://efts.sec.gov/LATEST/search-index?q=...`	No
Filing documents	`https://www.sec.gov/Archives/edgar/data/{cik}/{accession}/...`	No

The SEC's EDGAR APIs page says data.sec.gov's APIs "do not require any authentication or API keys to access." What is required is a declared User-Agent and a rate of no more than 10 requests per second; see the SEC rate limit and User-Agent guide. A search for "SEC API key" often leads to filer API tokens. Those exist, but they are for submitting filings to EDGAR, not for reading data.

Ticker to CIK

EDGAR identifies every filer by a Central Index Key (CIK). The SEC says CIKs "remain unique to the filer; they are not recycled." company_tickers.json maps tickers to CIKs:

curl -s -A "Data Tooly research contact@example.com" "https://www.sec.gov/files/company_tickers.json"

{"0":{"cik_str":1045810,"ticker":"NVDA","title":"NVIDIA CORP"},"1":{"cik_str":320193,"ticker":"AAPL","title":"Apple Inc."},"2":{"cik_str":1652044,"ticker":"GOOGL","title":"Alphabet Inc."}, ...}

What to know about the file (checked on 2026-10-14):

The top-level keys ("0", "1", ...) are just row numbers. Build your own index by ticker.
cik_str is an integer despite its name. Pad it yourself: f"CIK{cik:010d}".
It held 10,447 tickers for 8,032 CIKs. 1,440 companies have more than one ticker; GOOGL and GOOG both map to 1652044, for example.
Share classes use a dash: Berkshire Class B is BRK-B, not BRK.B. Normalize user input.
The SEC says it updates these files periodically "but do[es] not guarantee accuracy or scope". Delisted tickers drop out. For historical work, keep the CIK, not the ticker.

company_tickers_exchange.json has the same mapping in a columnar layout with the exchange added: {"fields": ["cik", "name", "ticker", "exchange"], "data": [[1045810, "NVIDIA CORP", "NVDA", "Nasdaq"], ...]}.

The submissions API

curl -s -A "Data Tooly research contact@example.com" "https://data.sec.gov/submissions/CIK0000320193.json"

The top-level fields for Apple: cik, entityType, sic, sicDescription, ownerOrg, insiderTransactionForOwnerExists, insiderTransactionForIssuerExists, name, tickers, exchanges, ein, lei, description, website, investorWebsite, category, fiscalYearEnd, stateOfIncorporation, stateOfIncorporationDescription, addresses, phone, flags, formerNames, filings.

filings has two parts:

filings.recent: per the SEC, "at least one year's of filing or to 1,000 (whichever is more) of the most recent filings in a compact columnar data array". Apple's had 1,001 rows.
filings.files: extra JSON files for older filings, each with name, filingCount, filingFrom and filingTo. Apple had one, CIK0000320193-submissions-001.json, with 1,264 filings from 1994-01-26 to 2015-09-15.

recent is column-oriented: one array per field (accessionNumber, filingDate, reportDate, acceptanceDateTime, act, form, fileNumber, filmNumber, items, core_type, size, isXBRL, isInlineXBRL, isXBRLNumeric, primaryDocument, primaryDocDescription). Row i is the ith element of every array.

The SEC says the submissions API updates in real time "with a typical processing delay of less than a second".

Python: ticker → CIK → filings → document URLs

import time
import requests

HEADERS = {"User-Agent": "Data Tooly research contact@example.com"}  # use your own name + email

def get(url):
    r = requests.get(url, headers=HEADERS, timeout=60)
    r.raise_for_status()
    time.sleep(0.15)  # stay under 10 requests/second
    return r.json()

# 1. Ticker -> CIK, from the SEC's own mapping file.
tickers = get("https://www.sec.gov/files/company_tickers.json")
by_ticker = {row["ticker"]: row for row in tickers.values()}
print(len(by_ticker), "tickers;", by_ticker["AAPL"], by_ticker["BRK-B"])

def cik_for(ticker):
    return by_ticker[ticker.upper()]["cik_str"]

# 2. Submissions: company metadata plus filing history.
cik = cik_for("aapl")
sub = get(f"https://data.sec.gov/submissions/CIK{cik:010d}.json")
print(sub["name"], sub["tickers"], sub["exchanges"], "| SIC", sub["sic"], sub["sicDescription"],
      "| FY ends", sub["fiscalYearEnd"])

# `recent` is column-oriented: one list per field. Zip it into rows.
recent = sub["filings"]["recent"]
rows = [dict(zip(recent, values)) for values in zip(*recent.values())]
print(len(rows), "recent filings; older pages:", [f["name"] for f in sub["filings"]["files"]])

def filing_urls(cik, row):
    folder = f"https://www.sec.gov/Archives/edgar/data/{cik}/{row['accessionNumber'].replace('-', '')}"
    return f"{folder}/{row['primaryDocument']}", f"{folder}/{row['accessionNumber']}-index.htm"

for row in [r for r in rows if r["form"] in ("10-K", "10-Q")][:3]:
    doc, index = filing_urls(cik, row)
    print(row["form"], row["filingDate"], "period", row["reportDate"], doc)

# 3. Older filings live in extra files listed under filings.files.
older = get(f"https://data.sec.gov/submissions/{sub['filings']['files'][0]['name']}")
print("oldest filing on record:", min(older["filingDate"]), "| rows in older file:", len(older["accessionNumber"]))

Output on 2026-10-14:

10447 tickers; {'cik_str': 320193, 'ticker': 'AAPL', 'title': 'Apple Inc.'} {'cik_str': 1067983, 'ticker': 'BRK-B', 'title': 'BERKSHIRE HATHAWAY INC'}
Apple Inc. ['AAPL'] ['Nasdaq'] | SIC 3571 Electronic Computers | FY ends 0926
1001 recent filings; older pages: ['CIK0000320193-submissions-001.json']
10-Q 2026-07-31 period 2026-06-27 https://www.sec.gov/Archives/edgar/data/320193/000032019326000020/aapl-20260627.htm
10-Q 2026-05-01 period 2026-03-28 https://www.sec.gov/Archives/edgar/data/320193/000032019326000013/aapl-20260328.htm
10-Q 2026-01-30 period 2025-12-27 https://www.sec.gov/Archives/edgar/data/320193/000032019326000006/aapl-20251227.htm

The older file is a bare column dictionary, with no filings.recent wrapper, which is why the last step reads older["filingDate"] directly.

Gotchas

Pad the CIK in data.sec.gov URLs. submissions/CIK320193.json returned HTTP 404 with an XML body. Archive paths (/Archives/edgar/data/320193/...) take the unpadded number.
cik comes back as a string here. The submissions JSON has "cik": "0000320193" (padded string), while company_tickers.json has an integer. Normalize with int() before you compare.
Accession numbers appear in two forms. The JSON uses 0000320193-25-000079. Folder paths drop the dashes: 000032019325000079.
fiscalYearEnd is MMDD. Apple's 0926 means its fiscal year ends around September 26.
Recent isn't everything. For a full history, also fetch every file in filings.files.
Bulk option. The SEC publishes all submissions as one nightly ZIP: https://www.sec.gov/Archives/edgar/daily-index/bulkdata/submissions.zip.

Related guides for this API

Company facts, company concept and frames: the financial numbers, using the same CIK.
EDGAR full-text search API: find filings by their text, filtered by CIK.
SEC rate limits and User-Agent rules.
SEC EDGAR API in Python, the overview guide.

Do it without code

The free SEC EDGAR query builder takes plain tickers, so you don't have to look up or pad CIKs, plus a mode (filings, financials, full-text search or insider trades) and form types. It shows the output shape. It builds the query rather than fetching live, because browsers can't send the User-Agent the SEC requires.

At scale

The SEC EDGAR Scraper actor on Apify resolves tickers to CIKs and pulls filings, financials, Form 4 insider trades, 13F holdings, 13D/G stakes and parsed 8-K items, with the User-Agent and pacing handled. It exports JSON, CSV or Excel. It's free to start, then pay-as-you-go.

Disclosure: I build the datatooly query builder and the Apify actor. The endpoints above work without either.

Sources: SEC: EDGAR Application Programming Interfaces, SEC: Accessing EDGAR Data, SEC webmaster FAQ, and live responses on 2026-10-14.

Search Hacker News by Date Range or Points (API)

Omar Eldeeb — Wed, 28 Oct 2026 11:46:09 +0000

To search Hacker News by date, call https://hn.algolia.com/api/v1/search_by_date with numericFilters=created_at_i>=START,created_at_i<END in Unix seconds. To filter by score, add points>100 to the same parameter, comma-separated. Use /search instead when you want the highest-scoring stories first. No API key is needed. One query returns at most 1,000 hits, so split long ranges.

Every example below ran against the live API on 2026-10-14. For the full parameter list, see the HN Algolia API reference.

The filters you need

Goal	Parameters
Posted in September 2026 (UTC)	`numericFilters=created_at_i>=1788220800,created_at_i<1790812800`
At least 100 points	`numericFilters=points>=100`
Busy discussions	`numericFilters=num_comments>=200`
Both	`numericFilters=points>=100,num_comments>=200` (comma = AND)
Either	`numericFilters=(points>5000,num_comments>4000)` (parentheses = OR)
Stories only	`tags=story`
Show HN or Ask HN	`tags=show_hn` or `tags=ask_hn`
Newest first	the `/search_by_date` endpoint
Highest score first	the `/search` endpoint with an empty `query`

created_at_i is in seconds, not milliseconds, and in UTC. The official page's own example is search_by_date?tags=story&numericFilters=created_at_i>X,created_at_i<Y.

Which endpoint sorts how. The same September window gave a different first page on each endpoint. /search_by_date returned stories from the last minutes of the window, each with a handful of points. /search returned the month's biggest stories. The official page says /search is "sorted by relevance, then points, then number of comments", and with no query, relevance is a tie, so points decide.

The top stories of a month, with curl

# Top 5 stories of September 2026 (UTC) by points
curl -s "https://hn.algolia.com/api/v1/search?tags=story&numericFilters=created_at_i%3E%3D1788220800,created_at_i%3C1790812800&hitsPerPage=5&attributesToRetrieve=title,points,created_at" \
  | python -c "import sys, json; [print(h['points'], h['created_at'][:10], h['title']) for h in json.load(sys.stdin)['hits']]"

2391 2026-09-15 Show HN: An e-ink frame that hears birds and draws them as 1800s illustrations
2301 2026-09-04 Discovery of a new OpenAI agent message board
2279 2026-09-03 GPT-6 Astra
2230 2026-09-03 .name Termination
2054 2026-09-08 Navier-Stokes – Tristan Buckmaster [pdf]

%3E%3D is >= and %3C is <. To get the timestamps in Python, use int(datetime(2026, 9, 1, tzinfo=timezone.utc).timestamp()); that gives 1788220800.

The last 7 days, in JavaScript

The API sends Access-Control-Allow-Origin: *, so this works in a browser as well as in Node 18+.

// Stories from the last 7 days with 300+ points, highest first.
const since = Math.floor(Date.now() / 1000) - 7 * 24 * 3600;
const params = new URLSearchParams({
  tags: "story",
  numericFilters: `created_at_i>${since},points>=300`,
  hitsPerPage: "5",
});
const res = await fetch(`https://hn.algolia.com/api/v1/search?${params}`);
const { hits, nbHits } = await res.json();
console.log(`${nbHits} stories`);
for (const h of hits) console.log(h.points, h.created_at.slice(0, 10), h.title);

Output on 2026-10-14:

98 stories
2123 2026-10-07 Margaret Hamilton has died
1440 2026-10-09 Triple-A Minesweeper
1407 2026-10-13 Gemini 4 Argon
1384 2026-10-09 Cloudflare acquires Deno
1134 2026-10-08 Why isn't the industry freaking out about DeepSeek 4.1 Flash?

More than 1,000 results: split the window

A single query stops at 1,000 hits. Past that, the API returns no hits and the message "you can only fetch the 1000 hits for this query". The fix is to cut the date range into slices that each hold 1,000 or fewer and join them. This function does it automatically by halving any window that is too big:

import time
from datetime import datetime, timezone

import requests

API = "https://hn.algolia.com/api/v1/search_by_date"
s = requests.Session()


def ts(day):
    """'2025-01-01' -> Unix seconds (UTC)."""
    return int(datetime.strptime(day, "%Y-%m-%d").replace(tzinfo=timezone.utc).timestamp())


def get(params, attempts=4):
    for i in range(attempts):
        r = s.get(API, params=params, timeout=30)
        if r.status_code == 200:
            return r.json()
        time.sleep(2 ** i)  # 1, 2, 4 s on 429/5xx
    r.raise_for_status()


def search_window(tags, start, end, extra="", query=""):
    """Every hit with start <= created_at_i < end. Splits the window while a
    slice holds more than 1,000 hits, because deeper pages are refused."""
    nf = f"created_at_i>={start},created_at_i<{end}" + (f",{extra}" if extra else "")
    first = get({"query": query, "tags": tags, "numericFilters": nf, "hitsPerPage": 1000})
    if first["nbHits"] <= 1000:
        return first["hits"]
    mid = (start + end) // 2
    return (search_window(tags, start, mid, extra, query)
            + search_window(tags, mid, end, extra, query))


if __name__ == "__main__":
    t0 = time.time()
    hits = search_window("show_hn", ts("2025-01-01"), ts("2026-01-01"), extra="points>=100")
    ids = {h["objectID"] for h in hits}
    print(f"{len(hits)} Show HN posts with 100+ points in 2025 "
          f"({len(ids)} unique) in {time.time() - t0:.1f}s")
    for h in sorted(hits, key=lambda h: -h["points"])[:5]:
        print(h["points"], h["created_at"][:10], h["title"])

    # A window that needs splitting: every story posted in one week.
    t0 = time.time()
    week = search_window("story", ts("2026-09-01"), ts("2026-09-08"))
    print(f"{len(week)} stories posted 2026-09-01..07 in {time.time() - t0:.1f}s")

Output on 2026-10-14:

742 Show HN posts with 100+ points in 2025 (742 unique) in 3.5s
3346 2025-12-09 Show HN: Gemini Pro 3 imagines the HN front page 10 years from now
3237 2025-01-22 Show HN: I made an open-source laptop from scratch
1581 2025-02-26 Show HN: I got laid off from Meta and created a minor hit on Steam
1557 2025-12-20 Show HN: Jmail – Google Suite for Epstein files
1539 2025-06-27 Show HN: I'm an airline pilot – I built interactive graphs/globes of my flights
7020 stories posted 2026-09-01..07 in 18.5s

The first query fit in one request. The second didn't: 7,020 stories in one week, so the function split the week into slices. As a check, I asked for each of the seven days separately; their nbHits added up to exactly 7,020, and every returned ID was unique and inside the window.

Pitfalls

Half-open windows. Use >= at the start and < at the end, as above. With >= and <= on both ends, an item posted exactly on a boundary second lands in two slices.
Trust nbHits only when it's exact. For broad queries the response sets exhaustiveNbHits: false, and the count is an estimate. The splitting function only compares nbHits with 1,000 on narrow windows, where the counts I saw were exact.
points is current, not historical. A story's points keep changing while it's active, and the filter uses the latest value. "Stories that had 100 points on day one" isn't something this API can answer.
A wrong field name fails silently. numericFilters=score>10 returned 0 hits and no error. The fields are created_at_i, points and num_comments.
Time zones. Timestamps are UTC. "September" in New York starts four hours later than in UTC; build your boundaries in the time zone you mean.
Rate limit. Splitting multiplies requests. The official limit is 10,000 per hour per IP, and the one-week split above used 17 requests.

Related guides for this API

HN Algolia API reference: all endpoints, tags and limits.
Get all comments for a Hacker News story.
Hacker News Firebase API vs Algolia.
The Hacker News Search API: free, no key.

Do it without code

The free Hacker News search tool runs these searches live in your browser. Type a keyword, choose stories, comments, Show HN or Ask HN, sort by relevance or date, and set a minimum score. The results are real, with points, comment counts, author and date.

At scale

The Hacker News Scraper actor on Apify takes the same filters as inputs: a keyword, a date range in Unix seconds (dateFromUnix, dateToUnix), minimum score and comments, and a domain filter. It returns flat rows you can export as JSON, CSV or Excel, optionally with comment trees and author karma, and it runs on a schedule. It's free to start, then pay-as-you-go.

Disclosure: I build the datatooly tool and the Apify actor. The scripts above use only the public API.

Sources: HN Search API (official page), Algolia: numericFilters, and live responses from hn.algolia.com/api/v1 on 2026-10-14.

Scrape Product Variants: Sizes, Colors and Price per SKU

Omar Eldeeb — Wed, 28 Oct 2026 11:45:47 +0000

To scrape product variants, read every variant of a product, not just the price on the page. Each variant (a size, color or pack size) has its own SKU, option values, price and stock. On Shopify, add .js to a product URL to get them as JSON. For Shopify, Walmart, Zalando, Target and other stores, an Apify actor returns them as a variants[] array on each product row.

This guide shows both methods with real output from 2026-10-20, what a variant row contains, which sites give per-variant prices, and the traps that make variant data wrong without an error.

Why the product price isn't enough

A product page shows one selected variant. The price at the top is that variant's price, and the stock flag is that variant's stock. Here's the same Walmart listing in one of our runs:

Variant	Price	In stock
5 Pack (the URL's own variant)	$19.40	no
2 Pack	$7.76	yes
Single	$3.88	yes

If you had stored only the product-level fields, you'd have recorded "$19.40, out of stock" for a cable you could buy that day. Variant-level data fixes three common problems:

Price ranges. On one Zalando sneaker page in an October 17 run, most sizes cost €39.95 but size 36 cost €64.95.
Partial stock. A Columbia rain jacket had 271 color and size combinations, and 207 of them were out of stock, while the product as a whole said "in stock".
Matching. Retailers sell a pack size or color under its own SKU or GTIN, so matching across stores works only at the variant level.

Method 1: Shopify's `.js` endpoint (no tools)

Every Shopify store serves a product's variants at the product URL plus .js. No key or login is needed:

import requests

url = "https://www.skullcandy.com/products/crusher-540-active"
p = requests.get(url + ".js", headers={"User-Agent": "Mozilla/5.0"}, timeout=30).json()

print(p["title"], "| options:", [o["name"] for o in p["options"]],
      "| price_min", p["price_min"] / 100, "price_max", p["price_max"] / 100)
for v in p["variants"]:
    print(f"  {v['id']}  {v['sku']:<11} {v['title']:<32} {v['price'] / 100:>7.2f}  "
          f"was={v['compare_at_price'] and v['compare_at_price'] / 100}  available={v['available']}")

Output on 2026-10-20:

Crusher® 540 Active | options: ['Color'] | price_min 209.99 price_max 229.99
  50564609998905  S6EVW-U054  Skullcandy X Baker Skateboards    229.99  was=None  available=True
  45146487652409  S6EVW-T013  Smoke                             209.99  was=None  available=True
  45138862538809  S6EVW-T016  Forest                            209.99  was=None  available=True
  44402118393913  S6EVW-T005  Soft                              209.99  was=None  available=True
  44494413594681  S6EVW-T992  Coal                              209.99  was=None  available=True
  44402118524985  S6EVW-T993  Concrete                          209.99  was=None  available=True

Two details matter:

In .js, price and compare_at_price are integers in cents. Divide by 100.
The .json version (/products/handle.json) returns the price as a string like "229.99" and has no available field. Use .js when you need stock.

To list every product of a store before you read variants, see Shopify products.json pagination. This method only works on Shopify. Other platforms put variants in JSON-LD, in embedded page state, or in a separate API, and each needs its own parser.

Method 2: variants from any supported store, through the API

E-commerce Product Scraper Pro on Apify reads variants on Shopify, WooCommerce, Walmart, Target, Zalando and the other stores in its supported-sites table, and puts them in a variants[] array on each product row. includeVariants is on by default. This script runs it on three stores and writes one CSV row per variant:

import csv
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("constructive_calm/ecommerce-product-scraper-pro").call(run_input={
    "detailsUrls": [
        {"url": "https://www.skullcandy.com/products/crusher-540-active"},
        {"url": "https://www.zalando.de/puma-club-ii-era-unisex-sneaker-low-whiteblackvapor-gray-pu115o0sb-a11.html"},
        {"url": "https://www.walmart.com/ip/5-pack-onn-3ft-USB-C-USB-C-Braided-Charging-Cable-Fast-Charging-USB-IF-Certified-White-Compatible-iPhone-17-16-15-Series-Samsung-Galaxy-Note-Pixel-De/20232964243"},
    ],
    "includeVariants": True,          # the default; shown for clarity
    "maxProductResults": 3,
})

rows = []
for p in client.dataset(run["defaultDatasetId"]).iterate_items():
    if p.get("recordType") != "product":
        continue
    variants = p.get("variants") or []
    prices = [v["price"] for v in variants if v.get("price") is not None]
    sold_out = [v for v in variants if v.get("inStock") is False]
    print(f"{p['site']}: {p['name'][:45]} | page price {p['price']} {p['currency']} | "
          f"{len(variants)} variants, price range {min(prices, default=None)}-{max(prices, default=None)}, "
          f"{len(sold_out)} sold out")
    for v in variants:  # one flat row per variant
        rows.append({
            "site": p["site"], "product": p["name"], "variant_sku": v.get("sku"),
            "options": "; ".join(f"{k}={val}" for k, val in (v.get("options") or {}).items()),
            "price": v.get("price"), "currency": v.get("currency") or p.get("currency"),
            "was_price": v.get("originalPrice"), "in_stock": v.get("inStock"), "url": v.get("url"),
        })

with open("variants.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=list(rows[0]))
    w.writeheader()
    w.writerows(rows)
print(len(rows), "variant rows written to variants.csv")

Output on 2026-10-20 (pip install apify-client, and APIFY_TOKEN set to your Apify API token):

skullcandy.com: Crusher® 540 Active | page price 229.99 USD | 6 variants, price range 209.99-229.99, 0 sold out
zalando-de: CLUB UNISEX - Low-Top-Sneaker - white/black/v | page price 39.95 EUR | 19 variants, price range 39.95-39.95, 4 sold out
walmart-us: (5 pack) onn 3ft Braided USB-C Charging Cable | page price 19.4 USD | 3 variants, price range 3.88-19.4, 1 sold out
28 variant rows written to variants.csv

The first lines of variants.csv:

site,product,variant_sku,options,price,currency,was_price,in_stock,url
skullcandy.com,Crusher® 540 Active,S6EVW-U054,Color=Skullcandy X Baker Skateboards,229.99,USD,,True,https://www.skullcandy.com/products/crusher-540-active?variant=50564609998905
skullcandy.com,Crusher® 540 Active,S6EVW-T013,Color=Smoke,209.99,USD,,True,https://www.skullcandy.com/products/crusher-540-active?variant=45146487652409

All three pages were fetched over datacenter connections, so the run had no traffic fee. It billed one start event and three product events. Variants don't add events.

What a variant contains

Each entry in variants[] has the same keys:

Key	Meaning
`sku`	The variant's own identifier: a store SKU, a Walmart item id, or an Amazon ASIN
`name`	The variant's title, e.g. "Smoke" or "5 Pack"
`options`	The option values as an object, e.g. `{"Color": "Red Lily", "Size": "L"}`
`price`, `currency`, `originalPrice`	The variant's price, its ISO currency and its was-price, when the page states them
`inStock`, `availability`	Stock flag and the raw availability (`InStock`, `OutOfStock`)
`gtin`, `image`, `url`	Barcode, variant image and a URL that selects the variant

The product row around it has variantCount, the product-level price and inStock (the selected variant's), and changedVariants in monitor mode.

Which sites give per-variant prices

The pattern below comes from the product rows in about 260 of our cloud test runs between 2026-10-17 and 2026-10-20. It's a measure of what those pages contained, not a guarantee for every product:

Site	Variant entries	With a price	With stock
Shopify stores (Death Wish Coffee, Skullcandy, Beardbrand)	2,572	all	2,454
Columbia (Salesforce Commerce)	2,803	all	all
Zalando	1,239	all	all
Walmart	1,212	1,200	all
Bloomingdale's	835	all	all
John Lewis	145	all	all
Target	43 (11 of 70 products had variants)	all	all
Amazon (.com, .co.uk, .de)	2,651	none	none
A WooCommerce store (Nalgene)	571	22	23

Amazon is the exception. Its product pages list each variant's ASIN and option values, but not their prices. The row's own price is the price of the ASIN in your URL. To price a sibling, put its variants[].url (a /dp/ASIN link) into detailsUrls and scrape it as a product. The Amazon price scraper guide covers that by ASIN and marketplace.

Grocery sites in our runs (Waitrose, Aldi UK) returned no variants: each pack size is its own product page.

Traps that give wrong variant data without an error

Reporting one price per product. Store the range (min and max over variants) or a price per variant. "From $209.99" and "$229.99" are both true on the Skullcandy page above.
Reading inStock on the product as "every size is available". The product flag is the selected variant's. The Columbia jacket said in stock with 207 of 271 combinations sold out.
Losing the currency. Variant currency can be empty when the page omits it (always on Amazon). Fall back to the product's currency, as the script does. If prices come back in the wrong currency, set proxyCountry to the store's country.
Comparing variants by name. "Black" on one store is "Jet Black" on another. Match on gtin when both sides have it, or on your own mapping of option values.
Huge rows. A product with 271 variants is one row with a long array. Flatten it, as the script does, before you load it into a spreadsheet.
Expecting stable variant prices. The Zalando size-36 price of €64.95 on October 17 was €39.95 on October 20, like the other sizes. Variant prices move on their own, which is exactly why you track them.

Holiday season: watch sizes, not just prices

During Black Friday (Friday, November 27, 2026) and Cyber Monday (November 30), a "deal" can cover only some sizes or colors, and the popular ones can sell out first. Neither can be seen from the product-level price. Two settings help:

Schedule the same input daily from early November with monitorMode: true. A variant whose price, was-price or currency changes marks the product changed and lists it in changedVariants, so you can see a discount that applies to one color only.
Track inStock per variant for the sizes you care about. The back-in-stock alert guide turns that into a notification.

The holiday sales calendar lists the 2026 and 2027 sale dates and how early to start a baseline for each.

Cost

Pay per event, from the actor's pricing table: one product row is one "product scraped" event, $0.0048 on Apify's FREE plan tier down to $0.0008 on GOLD, plus $0.005 per run start. Variants are inside the row, so they don't add events. Sites that need residential proxies or a browser (see the table on the actor page) add a per-MB or per-render fee. Apify's free plan includes $5 of usage every month, with no credit card required. There's no free trial inside the actor.

Related guides

Shopify products.json pagination: list every product of a Shopify store.
Scrape product reviews from Walmart, Target and Amazon: the review fields, including which variant each reviewer bought.
Back-in-stock alerts.

Do it without code

The free e-commerce price tracker builder has a variants option and produces a ready-to-run input for the actor, with a fixed example of the output shape. It doesn't fetch prices in your browser.

Disclosure: I built the actor and this site. The Shopify .js method above needs neither.

Sources: live responses from skullcandy.com (.js and .json endpoints) and an actor run on 2026-10-20; per-site variant counts from our own cloud test runs, 2026-10-17 to 2026-10-20.

Scrape Product Reviews from Walmart, Target and Amazon

Omar Eldeeb — Wed, 28 Oct 2026 11:45:26 +0000

To scrape product reviews, collect them from the product's own review source with a fixed sort and a cap: one row per review with its rating, text, date, verified flag and the variant bought. Walmart and Target expose every review page; Amazon shows only its top reviews (8–13) to signed-out visitors. Sort by "Lowest rated" to get complaints first.

This guide covers what each site lets you reach, the inputs of an Apify actor that does it, a real review row, and a short Python script that turns the lowest-rated reviews into complaint counts without a paid AI model. Everything below was run on 2026-10-20 unless a date says otherwise.

What you can reach on each site

The table is from the actor's supported-sites list and our October 2026 cloud tests:

Source	Reviews you get	Sort honored in our tests
Walmart (`walmart.com`)	All pages	Lowest rated, Most recent
Target (`target.com`)	All pages	Lowest rated, Most recent
Amazon (24 storefronts)	Only the top reviews on the product page, about 8 to 13 per product	No: Amazon's own order (`sortApplied` = "Most relevant")
Shopify stores with Yotpo, Judge.me, Okendo or Stamped	Reviews from the review widget	All five sorts on a Yotpo store; "Most recent" seen on Judge.me and Okendo stores
WooCommerce stores	From the store's own Store API	not tested
AliExpress (best-effort site)	All pages; some reviews are a rating with no text	No: fell back to "Most relevant"
Flipkart (best-effort site)	All pages	not tested
Mercado Libre (best-effort site)	Catalog URLs (`/p/MLM…`) only	not tested

Amazon's full review pages ask visitors to sign in. A scraper that respects that limit only gets the reviews printed on the product page. When Amazon shows "Sign in to see customer reviews" instead, the actor reports the product as restricted rather than as an empty success.

Not reachable with this actor: Google Shopping reviews, Loox and Reviews.io widgets (the page's structured reviews are used instead), and sites the actor marks unsupported, such as eBay, Best Buy, Home Depot and Etsy.

The inputs

The E-commerce Product Scraper Pro actor has a review section with these fields:

Field	What it does
`reviewDetailsUrls`	Product pages to read reviews from
`reviewListingUrls`	Category or search pages: reviews for the products listed on the first page
`keywordReviews` + `marketplacesReviews`	A keyword searched on the chosen marketplaces; reviews for the products found on the first results page
`sortReview`	`Most recent`, `Most relevant`, `Most helpful`, `Highest rated` or `Lowest rated`
`maxReviewResults`	Cap for the whole run
`maxItemsPerStartUrl`	Cap per input, so one product can't use up the run's cap
`additionalReviewProperties`	Adds extra per-site fields

If a site doesn't support the sort you asked for, its default order is used. The row's sortApplied says which order the reviews really follow, and the input's status row says so too.

A keyword review search, as run today:

{
  "keywordReviews": "usb c cable",
  "marketplacesReviews": ["www.target.com", "www.walmart.com"],
  "sortReview": "Lowest rated",
  "maxItemsPerStartUrl": 2,
  "maxReviewResults": 4
}

The status row for each marketplace explained what happened:

Keyword "usb c cable" on www.target.com: 24 product(s) found on the first page; reviews read for 1, 1 had reviews; 2 review(s) delivered. Sort applied per source — Target: Lowest rated. Stopped at maxItemsPerStartUrl (2) for this input. Only the first listing page is read for reviews.
Keyword "usb c cable" on www.walmart.com: 60 product(s) found on the first page (first 60 used); reviews read for 1, 1 had reviews; 2 review(s) delivered. Sort applied per source — Walmart: Lowest rated. Stopped at maxReviewResults (4). Only the first listing page is read for reviews.

What a review row contains

A real Walmart row from today, with the reviewer's name removed:

{
  "recordType": "review",
  "productUrl": "https://www.walmart.com/ip/18928659757",
  "inputUrl": "https://www.walmart.com/ip/ONN-10-C-C-BRD-WHT/18928659757",
  "datePublished": "2026-10-09",
  "reviewBody": "melted sparked barely got fire out but it melted the place on my phone where c plugs in",
  "reviewRating": { "ratingValue": 1, "bestRating": 5 },
  "additionalProperties": { "verified": true, "helpfulVotes": 0, "unhelpfulVotes": 0, "variant": "Multipack quantity: 1", "syndicatedFrom": "bazaarvoice" },
  "reviewId": "443517198",
  "rating": 1,
  "title": null,
  "verified": true,
  "helpfulVotes": 0,
  "images": [],
  "variant": "Multipack quantity: 1",
  "productId": "18928659757",
  "productName": "Onn 10ft USB-C to USB-C Braided Charging Cable, USB-IF Certified, White - Compatible with iPhone, Samsung, Pixel, iPad Pro",
  "position": 1,
  "sortApplied": "Lowest rated",
  "site": "walmart-us",
  "source": "walmart",
  "scrapedAt": "2026-10-20T09:30:03.526Z"
}

Fields worth knowing:

rating is on the scale in bestRating (5 on these sites).
variant is the size, color or pack the reviewer bought, when the site says. It's how you tell whether complaints cluster on one variant. See scraping product variants for the variant data itself.
datePublished keeps the precision the site gives. Walmart gives a day, Target a full timestamp, and relative dates like "3 months ago" become a month (2026-07), never an invented day.
additionalProperties carries site extras: Target's incentivized and recommended flags, and Walmart's syndicatedFrom (reviews shown on Walmart but collected elsewhere).
author holds the reviewer's display name. You rarely need it; drop it before you store reviews.

Python: summarize complaints from the lowest-rated reviews

This script collects the lowest-rated reviews of a Walmart product and a Target product, then counts complaint themes with keyword lists. There's no AI model involved, so it costs nothing beyond the run and gives the same answer every time.

import os
import re
from collections import Counter
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("constructive_calm/ecommerce-product-scraper-pro").call(run_input={
    "reviewDetailsUrls": [
        {"url": "https://www.walmart.com/ip/ONN-10-C-C-BRD-WHT/18928659757"},
        {"url": "https://www.target.com/p/3-39-usb-c-to-usb-c-charging-cable-dealworthy-8482-white/-/A-89698753"},
    ],
    "sortReview": "Lowest rated",
    "maxItemsPerStartUrl": 3,     # per product, so one product can't use up the run cap
    "maxReviewResults": 5,        # run-wide cap; raise both for real analysis
})
reviews = [r for r in client.dataset(run["defaultDatasetId"]).iterate_items()
           if r.get("recordType") == "review"]

# Complaint themes: a keyword list per theme. Edit these for your category.
THEMES = {
    "stopped working": r"stopp?ed working|no longer works?|didn.t work|doesn.t work|not working|died|dead",
    "physical damage": r"crack|fray|broke|wires? (?:visible|showing|exposed)|fell apart",
    "short lifespan": r"lasted|weeks?|months? ago|already",
    "fit / length": r"\bfit\b|loose|too? short|length",
    "overheating / safety": r"\bhot\b|overheat|melt|spark|smoke|fire",
    "wrong item": r"wrong item|received a|not as described|different (?:item|product)|ordered .{0,40}received",
    "general quality": r"cheap|flimsy|crap|junk|poor quality",
}

low = [r for r in reviews if (r.get("rating") or 5) <= 2]
counts = Counter()
for r in low:
    text = f"{r.get('title') or ''} {r.get('reviewBody') or ''}".lower()
    for theme, pattern in THEMES.items():
        if re.search(pattern, text):
            counts[theme] += 1

print(f"{len(reviews)} reviews, {len(low)} rated 1-2 stars")
for r in reviews:
    print(f"  {r['site']:<11}{r['rating']}* sort={r['sortApplied']:<13}{(r.get('reviewBody') or '')[:70]!r}")
print("Complaint themes in 1-2 star reviews:")
for theme, n in counts.most_common():
    print(f"  {theme:<26}{n}/{len(low)}")

Output on 2026-10-20 (pip install apify-client, APIFY_TOKEN set):

5 reviews, 5 rated 1-2 stars
  target     1* sort=Lowest rated 'Bought 3 months ago and it’s already cracked with wires visible in mul'
  target     1* sort=Lowest rated "piece of crap. don't buy!!"
  target     1* sort=Lowest rated 'to short of a cord'
  walmart-us 1* sort=Lowest rated 'melted sparked barely got fire out but it melted the place on my phone'
  walmart-us 1* sort=Lowest rated 'Ordered Braided USB-C to USB Cable. Received Braided USB-C Cable'
Complaint themes in 1-2 star reviews:
  physical damage           1/5
  short lifespan            1/5
  general quality           1/5
  fit / length              1/5
  overheating / safety      1/5
  wrong item                1/5

Five reviews is a demonstration, not a finding. For a real read, raise maxItemsPerStartUrl to 100–200 per product. The Target product had 95 reviews and the Walmart one 38, so a few hundred covers them all. Then read a handful of reviews behind each count before you trust it. Keyword themes miss misspellings ("minths") and sarcasm, and one review can hit several themes.

Two ways to make the counts more useful:

Split by variant. If most "too short" complaints are on the 3 ft version, that's a sizing problem, not a quality one.
Compare against "Most recent". Run the same products sorted by Most recent every week. A theme that grows in recent reviews points to a new batch or a supplier change, while the lowest-rated sort mostly shows the worst reviews of all time.

Holiday season: check the 1-star reviews before the deal

Black Friday is Friday, November 27, 2026, and Cyber Monday is November 30. Before you feature a product in a promotion, or buy a heavily discounted one, pull its lowest-rated reviews and look for safety or failure themes. A deep discount on an item with a cluster of "melted" or "stopped working" reviews isn't a deal. If you sell, a weekly Most recent run through November and December catches new complaints while there's still time to act, for example by fixing a listing or pausing a promotion. The holiday sales calendar has the 2026–2027 sale dates. To check the discount itself, see how to spot fake Black Friday discounts.

Cost

Pay per event, from the actor's pricing table: one review row is $0.0016 on Apify's FREE plan tier and $0.00056 on GOLD, plus $0.005 per run start. The README's example of 500 Target reviews is 500 × $0.00056 + $0.005 = $0.285 on GOLD; on FREE it's 500 × $0.0016 + $0.005 = $0.805. Review pages that need a residential proxy add a per-MB traffic fee. Apify's free plan includes $5 of usage every month, with no credit card required. There's no free trial inside the actor.

Related guides

Walmart and Target price scraper: the product rows for the same retailers.
Amazon price scraper by ASIN.

Do it without code

The free e-commerce price tracker builder creates a ready-to-run input for the actor's product section and shows a fixed example of the output. For reviews, use the review fields above in the actor's input form on Apify.

Disclosure: I built the actor and this site.

Sources: actor runs on 2026-10-20 (review sort and status text), our cloud test runs between 2026-10-17 and 2026-10-20 (per-site sort behavior and Amazon review counts), and the actor's README (supported-sites table, limitations, pricing).

Ramadan Sale Price Tracker 2027: Noon & Amazon UAE

Omar Eldeeb — Wed, 28 Oct 2026 11:45:04 +0000

Ramadan 2027 is expected to start around Monday 8 February 2027, with Eid al-Fitr on 9 or 10 March (astronomical estimates; final dates depend on moon sighting). Retailer "Ramadan sales" start weeks earlier: in 2026 Amazon UAE's ran 27 January to 14 February and ended before Ramadan began. So start weekly price records in mid-December 2026 and check daily from mid-January through Eid.

This is the Gulf Ramadan page of the cluster. It covers the dates and their uncertainty, why the sale calendar runs ahead of the religious calendar, a phase-by-phase schedule, which sites can be tracked, and a script that measures which categories actually moved. For November's sales in the region, see White Friday and 11.11 on noon and Amazon UAE.

Dates: what's known and what isn't

	2026 (actual)	2027 (approximate)
First day of Ramadan	Wed 18 February 2026 in the UAE and Saudi Arabia; 19 February in some other countries	about Mon 8 February 2027 (UAE Astronomy Centre; Umm al-Qura calendar). Some countries may start on 9 February.
Eid al-Fitr	Fri 20 March 2026 in the UAE and Saudi Arabia	Tue 9 March 2027 if Ramadan has 29 days, Wed 10 March if 30
Amazon UAE Ramadan Sale	27 January - 14 February 2026 (Prime members from 27 January)	not announced
noon Ramadan sale	until 2 February 2026 (reported 26 January)	not announced

The Islamic calendar moves about 10-12 days earlier each Gregorian year, which is why Ramadan 2027 lands about 10 days before Ramadan 2026 did. The start is confirmed by each country's moon-sighting committee on the evening of the 29th of Sha'ban, so you won't know the exact day until the night before. A schedule that is daily for the whole window makes that uncertainty irrelevant.

The sale calendar runs ahead of Ramadan

The 2026 dates above show the pattern: the big e-commerce Ramadan sales were stocking-up events before the month, not sales during it. If retailers follow the same pattern in 2027, the Ramadan sales would start around mid-January 2027. That has two consequences for tracking:

The baseline has to start in December. A baseline that starts in late January would already be inside the sale.
The baseline overlaps other sales. The Dubai Shopping Festival runs 4 December 2026 to 17 January 2027, and the November 11.11 and White Friday sales come before that. In the Gulf, November to March is close to one continuous promotion season. Keep weekly records through all of it, and use the prices a few weeks after Eid as a second reference: if the "Ramadan price" is also the price in late March, it wasn't a sale price.

A phase-by-phase schedule

This prints the 2027 windows from the approximate dates. The pre-Ramadan sale window copies 2026's Amazon UAE timing (starting 22 days before Ramadan); move it when the 2027 sales are announced.

from datetime import date, timedelta

# Astronomical estimates (UAE / Umm al-Qura); confirmed only by moon sighting.
RAMADAN_START = date(2027, 2, 8)
EID_EARLY, EID_LATE = date(2027, 3, 9), date(2027, 3, 10)   # 29- or 30-day month
d = timedelta

# In 2026 Amazon.ae's Ramadan Sale ran 22 days to 4 days before Ramadan began.
phases = [
    ("baseline (weekly)",          RAMADAN_START - d(56), RAMADAN_START - d(23)),
    ("pre-Ramadan sales (daily)",  RAMADAN_START - d(22), RAMADAN_START - d(1)),
    ("Ramadan, first weeks (daily)", RAMADAN_START,       EID_EARLY - d(11)),
    ("last 10 nights (2x daily)",  EID_EARLY - d(10),     EID_LATE - d(1)),
    ("Eid holiday (daily)",        EID_EARLY,             EID_LATE + d(3)),
    ("rebound check (weekly)",     EID_LATE + d(4),       EID_LATE + d(21)),
]
for name, start, end in phases:
    print(f"{name:<30} {start:%a %d %b %Y} -> {end:%a %d %b %Y}  ({(end - start).days + 1} days)")

baseline (weekly)              Mon 14 Dec 2026 -> Sat 16 Jan 2027  (34 days)
pre-Ramadan sales (daily)      Sun 17 Jan 2027 -> Sun 07 Feb 2027  (22 days)
Ramadan, first weeks (daily)   Mon 08 Feb 2027 -> Fri 26 Feb 2027  (19 days)
last 10 nights (2x daily)      Sat 27 Feb 2027 -> Tue 09 Mar 2027  (11 days)
Eid holiday (daily)            Tue 09 Mar 2027 -> Sat 13 Mar 2027  (5 days)
rebound check (weekly)         Sun 14 Mar 2027 -> Wed 31 Mar 2027  (18 days)

The overlap between "last 10 nights" and "Eid" on 9 March is deliberate: that day is either the last day of Ramadan or the first day of Eid.

Cron for these windows (UTC; the UAE is UTC+4 and Saudi Arabia UTC+3, with no daylight saving): weekly 0 4 * * 0, daily 0 4 * * * (08:00 UAE), twice daily 0 4,16 * * * (08:00 and 20:00 UAE). The evening run, after iftar, catches promotions that go live in the evening.

Which sites can be tracked

The examples use the E-commerce Product Scraper Pro actor. Its README rates these Gulf sites best-effort, except noon Saudi Arabia (unsupported), and our runs on 20 October 2026 gave:

noon UAE (noon.com/uae-en/): delivered, in AED. noon is behind Akamai, so the actor renders a page in a browser and then uses residential connections, which carry per-MB and per-render fees.
noon Saudi Arabia (noon.com/saudi-en/, /saudi-ar/): unsupported. Akamai answered "Access Denied" on every route in the actor's October 2026 tests, Saudi residential connections and the browser included, so the README lists it as unsupported. Saudi noon inputs get a status row saying why, aren't fetched, and cost only the run's start fee. Use Amazon Saudi Arabia for Saudi prices.
Amazon UAE (amazon.ae): delivered, in AED, over residential connections.
Amazon Saudi Arabia (amazon.sa): delivered, in SAR, over datacenter connections (no traffic fee). The default Arabic product pages parse fully: on 20 October 2026, https://www.amazon.sa/dp/B0DGHYPFGK came back with an Arabic title, inStock: true and brandName: "Apple". English URLs (https://www.amazon.sa/-/en/dp/<ASIN>) also work if you want English titles.
IKEA UAE / Saudi Arabia: best-effort in the README; not tested today.

Gulf supermarket sites such as Carrefour and Lulu aren't in the matrix and haven't been tested; Amazon UAE and Saudi Arabia grocery items are ordinary Amazon product pages, so they should behave like any other ASIN (we haven't tested grocery items specifically).

Measure which categories moved

Published campaigns quote "up to 70% off", which says nothing about the typical cut. To answer "what got cheaper?", compare two exports of the same products and group the changes by category. The actor's rows carry a breadcrumbs list (for example ["Home", "Electronics & Mobiles", "Mobiles & Accessories", "Mobile Phones", "Smartphones"] on noon), which is enough to group on. Amazon UAE and Saudi Arabia rows have an empty breadcrumbs list, because Amazon shows no category trail on those pages, so the script files them under "(no category)"; if you track Amazon there, map those URLs to your own categories.

import json, sys
from collections import defaultdict
from statistics import median

def load(path):
    return {r["url"]: r for r in json.load(open(path, encoding="utf-8")) if r.get("price")}

base, now = load(sys.argv[1]), load(sys.argv[2])
moves = defaultdict(list)
for url in base.keys() & now.keys():
    crumbs = now[url].get("breadcrumbs") or ["(no category)"]
    category = crumbs[min(3, len(crumbs) - 1)]          # pick the depth that suits your catalog
    moves[category].append((now[url]["price"] - base[url]["price"]) / base[url]["price"] * 100)

print(f"{'category':<28}{'matched':>8}{'cut':>6}{'raised':>8}{'median %':>10}")
for category, pct in sorted(moves.items(), key=lambda kv: median(kv[1])):
    print(f"{category[:27]:<28}{len(pct):>8}{sum(p < 0 for p in pct):>6}"
          f"{sum(p > 0 for p in pct):>8}{median(pct):>10.2f}")
print(f"matched {sum(map(len, moves.values()))} of {len(now)} products by URL")

Run as python category_moves.py baseline.json during.json. Here it is on two real exports of the same noon UAE search ("samsung"), taken 2 hours apart on 17 October 2026:

category                     matched   cut  raised  median %
Mobile Phones                     11     1       0      0.00
matched 11 of 30 products by URL

Two lessons from that real run. First, the change was tiny (one phone from AED 1,168 to 1,167.55), which is what an ordinary day looks like; a Ramadan sale should show up as a clearly negative median. Second, only 11 of the 30 products were on the search page both times. Search pages reshuffle, so for a Ramadan comparison track a fixed list of product URLs (pasted into detailsUrls) instead of a search, or every comparison is mostly products coming and going.

What it costs

Pay per event, prices from the actor's README. FREE tier: $0.0048 per product, $0.008 per browser render, $0.018 per residential MB (billed at 89% of measured traffic, only for items that delivered), $0.005 per run start. GOLD: $0.0008, $0.006, $0.014 and $0.005.

Per-run examples from real runs:

60 noon UAE product URLs (18 October 2026: 60 products, 2 browser renders, 8 billed MB). FREE: $0.005 + 60 × $0.0048 + 2 × $0.008 + 8 × $0.018 = $0.453. GOLD: $0.005 + 60 × $0.0008 + 2 × $0.006 + 8 × $0.014 = $0.177.
60 Amazon Saudi Arabia products (datacenter, no traffic fee). FREE: 60 × $0.0048 + $0.005 = $0.293.

Multiply by the number of runs in your schedule. noon's residential traffic makes daily noon runs the expensive part, so if budget matters, check noon every two or three days during the mid-month phase and daily only in the sale and Eid windows. For a run with more than about 50 noon URLs, the README recommends 2 GB of memory (one extra $0.005 start event).

Apify's free plan gives $5 of usage every month, no credit card required; the actor has no free trial of its own. The free e-commerce price tracker query builder builds the input without code and shows a fixed example of the output shape.

Related guides

Disclosure: I built the Apify actor and this site. The schedule and the category script work with data from any source.

Sources: The National: expected Ramadan 2027 and Eid al-Fitr dates in the UAE, Gulf News: countries that began Ramadan 2026 on 18 February, Gulf News: countries celebrating Eid al-Fitr 2026 on Friday, Gulf News: UAE Ramadan 2026 sales, TradeArabia: Dubai Shopping Festival 2026-27, the actor's README (supported sites, pricing, memory) and live actor runs on 2026-10-17, 2026-10-18 and 2026-10-20 (amazon.sa Arabic page: run rm7jhN0AiKNMb6V6K, build 0.1.16).

DEV Community: Omar Eldeeb

White Friday & 11.11 Tracker: Noon and Amazon UAE Prices

Three names, one season

What can be scraped (tested 20 October 2026)

URLs: market, currency and language

Watch product URLs, not search pages

Compare noon and Amazon UAE

Compare the UAE and Saudi Arabia

A schedule for the Gulf sales

What it costs

Related guides

What Slippage Should I Set to Avoid Sandwich Attacks?

What slippage tolerance actually does

Why high slippage invites sandwich attacks

A rule-of-thumb table

Use the pool's MEV numbers to pick a setting

Check the chart before you trade

When your swap keeps failing at low slippage

FAQ

Is 0.5% slippage safe from sandwich attacks?

What slippage should I use for meme coins?

Why does my swap fail when I set low slippage?

Does auto slippage protect me from MEV?

Can I use high slippage if I send through a private RPC?

What's the difference between price impact and slippage?

Does a low mev_risk_level mean I can loosen slippage?

What slippage should I set for stablecoin swaps?

Sources

Walmart and Target Price Scraper: Search and Product Pages

What works on each retailer

A search page from each store

Python: match the same products on Walmart and Target

Fields that differ between the two

Caveats for price data

What it costs

Holiday season: one GTIN list, both stores, every day

Related guides

Do it without code

Track AI Brand Mentions Over Time: Schedules and Drift

Hold these fixed

What causes drift

How often to run

Compare two runs

Pitfalls

Related guides

Do it without code

At scale

When Do Black Friday Prices Drop? The 2026 Timeline

The 2025 timeline

What that suggests for 2026

Why you have to measure it yourself

Find the day each product hit its low

Details that change the answer

How to Tell if a Token's Pool Is Getting Sandwiched Before You Swap

What a sandwich attack is

Check 1: read the pool's trade feed

Check 2: measure the sandwich rate and the MEV share

Check 3: compare your trade size with the pool

Check 4: decide how to send the trade

What these checks can't tell you

Run it for your own tokens

FAQ

Can I see a sandwich attack on a DEX chart?

Does a high MEV risk level mean the pool is getting sandwiched?

Is a pool with a 0% sandwich rate safe to swap in?

Are meme coins sandwiched more than major pairs?

Do private RPCs stop sandwich attacks?

Why are some fields null for Solana or Base pools?

Which chains can I check?

Sources

Apify E-commerce Scraping Tool Alternative: Same Input

What stays the same

What's added on top

What's different

Sites

Where errors go

Pricing

Python: run your existing input and check the fields you use

Migration checklist

Holiday season: switch before the sale, not during it

Does a low `mev_risk_level` mean I can loosen slippage?

Method 1: Shopify's `.js` endpoint (no tools)