DEV Community: DevKit

Alertmanager to Telegram without a bridge: severity routing, inhibit rules and runbook links that cut alert noise

DevKit — Thu, 29 Oct 2026 02:21:16 +0000

Written with AI assistance; the alert pack itself was built and tested by the author's AI assistant.

If Prometheus pings your phone fifty times when one server reboots, you will soon mute the chat, and then the one real alert gets missed too. Most of that noise can be removed in Alertmanager itself, with no extra containers.

This article walks through a small config: native Telegram delivery, routing by severity, inhibit rules, and a runbook link in every message. Everything here works with plain Alertmanager and nothing has to be bought.

No bridge needed

Older guides run a webhook bridge that converts Alertmanager JSON into Telegram messages. Recent Alertmanager versions have a built-in telegram_configs receiver, so that is one less container to keep alive. Check your version's documentation to confirm it supports telegram_configs and bot_token_file.

The bot token: a file, not inline

Create a bot with @botfather and keep the token out of alertmanager.yml, which tends to end up in git, backups and screenshots. Put it in a file and use bot_token_file:

sudo install -d -m 0750 /etc/alertmanager/secrets
printf '%s' '<TOKEN_FROM_BOTFATHER>' | sudo tee /etc/alertmanager/secrets/telegram_bot_token >/dev/null
sudo chown alertmanager: /etc/alertmanager/secrets/telegram_bot_token
sudo chmod 0600 /etc/alertmanager/secrets/telegram_bot_token

Adjust the user name to whatever account runs Alertmanager. I use printf so the file has no trailing newline.

Finding the chat_id

There is no menu that shows it. The generic method:

Add the bot to your group (or open a private chat with it).
Send any message in that chat. In a group the bot may not see ordinary messages because of privacy mode, so send a /command or mention the bot.
Open https://api.telegram.org/bot<TOKEN>/getUpdates in a browser or with curl and look for "chat":{"id":...}.

Private chats have positive ids. Groups are negative. Supergroups, which is what most groups become, start with -100. This article uses the placeholder -1001234567890; use the number you actually see, including the minus sign. If getUpdates returns an empty list, send another message and retry.

Routing by severity

Not every alert deserves the same urgency. The routing tree sets different timings for critical and warning:

route:
  receiver: telegram-warning
  group_by: [alertname, instance, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - matchers: ['severity="critical"']
      receiver: telegram-critical
      group_wait: 10s
      repeat_interval: 1h
    - matchers: ['severity="warning"']
      receiver: telegram-warning
      repeat_interval: 12h
receivers:
  - name: telegram-critical
  - name: telegram-warning

(Receivers are stubs so the snippet stands alone; real ones are below.)

What each setting does:

group_by [alertname, instance, namespace]: alerts with the same name, host and namespace are merged into one message. Without grouping, ten failing pods become ten messages.
group_wait: how long Alertmanager waits after the first alert of a new group before sending, so related alerts can join it. 30s for warnings gives a burst time to gather. Critical uses 10s because a few seconds of batching is worth less than a fast page.
group_interval: how long to wait before sending notifications about new alerts added to a group that was already notified. 5m keeps a flapping group from producing a message every minute.
repeat_interval: how often an unresolved alert is re-sent. Critical repeats hourly, so it can't be forgotten. A warning repeats every 12 hours, which is a reminder and not a nag.

Treat these numbers as a starting point and tune them to how quickly you can actually respond.

The receiver: HTML, runbook link, escaping

receivers:
  - name: telegram-critical
    telegram_configs:
      - bot_token_file: /etc/alertmanager/secrets/telegram_bot_token
        chat_id: -1001234567890
        parse_mode: HTML
        send_resolved: true
        message: |-
          🔥 <b>[{{ .Status | toUpper }}] {{ .CommonLabels.alertname }}</b>
          {{ range .Alerts -}}
          {{ .Annotations.summary }}
          {{ .Annotations.description }}
          <a href="{{ .Annotations.runbook_url }}">Runbook</a>
          {{ end }}

The warning receiver is identical apart from the ⚠️ emoji; it could point at a different chat.

parse_mode: HTML gives bold text and clickable links. The pitfalls:

Telegram accepts only a small subset of HTML tags. Anything else, or an unbalanced tag, makes Telegram reject the message with a "can't parse entities" error, and you get no alert at all.
A literal <, > or & in text must be written as <, >, &. A description like latency > 500ms or a label with & in a URL is enough to break the message. As far as I know Alertmanager's notification templates are plain text templates and do not auto-escape, so this is your job.
Escape the free-text fields you don't control. Template functions such as reReplaceAll can do it, and the order matters because & must be replaced first:

message: |-
  {{ .Annotations.description | reReplaceAll "&" "&amp;" | reReplaceAll "<" "&lt;" | reReplaceAll ">" "&gt;" }}

If an alert has no runbook_url, <a href=""> is an empty link and may be rejected. Wrap it in {{ if .Annotations.runbook_url }}…{{ end }} if some of your rules lack the annotation.
Telegram limits a message to 4096 characters, so keep annotations short and groups small.

Always verify with a real test alert (below) after changing a template.

Runbook links in the alert itself

The link has to come from the rule, so that the person reading the message at 3 a.m. gets a single tap to "what do I do". In the rules the annotation looks like this:

- alert: DockerContainerRestartLoop
  expr: changes(container_start_time_seconds{name!=""}[15m]) >= 3
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Container {{ $labels.name }} keeps restarting ({{ $labels.instance }})"
    runbook_url: "https://devkit-shop.surge.sh/runbook/en/#dockercontainerrestartloop"

The convention is simple: the anchor is the alert name in lower case, and the runbook has a section with that name. Each section follows the same order: what it means, common causes, diagnostics, how to fix, and when it is OK to silence. For example, the section for NodeDown begins by saying that it is not necessarily a server outage, because only the exporter or the network path may be down, and then lists ping, curl against port 9100 and systemctl status node_exporter.

You can write the same for your own alerts; even three lines per alert beat none.

Inhibit rules: where the noise actually goes

Inhibition mutes a target alert while a source alert is firing, as long as the labels listed in equal match. Here are the first two rules:

inhibit_rules:
  # Same alert on the same host as critical and warning: keep critical.
  - source_matchers: ['severity="critical"']
    target_matchers: ['severity="warning"']
    equal: [alertname, instance]

  # Host is unreachable: other node-exporter alerts from it are meaningless.
  - source_matchers: ['alertname="NodeDown"']
    target_matchers: ['alertname=~"Node.+"', 'alertname!="NodeDown"']
    equal: [instance]

Critical beats warning. If a rule exists at two thresholds under the same alertname, you see only the critical one. This only matters if you use the same alert name for both levels; otherwise it does nothing, and the specific rules below cover the pairs.
A dead host silences its own symptoms. When NodeDown fires, disk, CPU and memory alerts for that instance would be stale or misleading. Note that the target pattern is Node.+, so alerts with other prefixes (for example Docker…) are not covered by this rule.

The remaining three follow the same shape:

inhibit_rules:
  # Whole site is down: skip slow-response and bad-status alerts.
  - source_matchers: ['alertname="BlackboxProbeFailed"']
    target_matchers: ['alertname=~"BlackboxSlowResponse|BlackboxHttpStatusNotOk"']
    equal: [instance]

  # PVC critically full: the "almost full" warning is redundant.
  - source_matchers: ['alertname="KubePvcCritical"']
    target_matchers: ['alertname="KubePvcAlmostFull"']
    equal: [namespace, persistentvolumeclaim]

  # API certificate expires within a day: drop the one-week warning.
  - source_matchers: ['alertname="KubeApiCertExpiresCritical"']
    target_matchers: ['alertname="KubeApiCertExpiresSoon"']
    equal: [instance]

Each time, a more severe or more fundamental alert makes a weaker one about the same thing redundant. These alert names come from the pack's rules; replace them with yours, and make sure the equal labels exist on both alerts, otherwise the rule never matches.

Testing before you trust it

Validate syntax first:

amtool check-config alertmanager.yml

Then send a fake alert straight to the Alertmanager API (default port 9093):

curl -sS -X POST http://localhost:9093/api/v2/alerts \
  -H 'Content-Type: application/json' \
  -d '[{"labels":{"alertname":"TestAlert","severity":"critical","instance":"test:9100"},
        "annotations":{"summary":"Test","description":"a < b & c","runbook_url":"https://example.com/"}}]'

The < and & in the description are deliberate: they show whether your escaping works. Expect the message after group_wait (10s for critical). Repeat with "severity":"warning" to check the other route.

Silences: with a reason and an expiry

For planned maintenance, silence instead of ignoring:

amtool silence add alertname=NodeDown instance=web1:9100 \
  --comment="kernel update, planned reboot" --duration=2h

A silence without a comment tells the next person nothing, and one without an expiry is just switched-off monitoring.

Limits

telegram_configs and bot_token_file need a reasonably recent Alertmanager. Check your version's documentation instead of trusting this article.
Telegram can be unreachable from your network, down, or rate-limiting you. If your alerts are bursty, rate limiting shows up as delayed or dropped messages. Grouping reduces volume but doesn't remove the risk.
Don't make Telegram the only channel for critical alerts. Add a second receiver for them, such as email or a pager service, and ideally something that doesn't share a failure with the rest of your setup. If the server running Alertmanager is the one that died, nothing gets sent at all, so an external heartbeat check is worth having.
Inhibition hides alerts. A badly chosen equal can hide something real, so re-read every rule once.

Where to get the rest

The full pack has rules for Linux, Docker, Kubernetes and websites/TLS, this Alertmanager config, a runbook in English and Russian, and promtool tests. It is sold for 9 USDT via the Telegram shop. The runbook itself is free at https://devkit-shop.surge.sh/runbook/en/ , and there is a free monitoring kit for a single VPS.

A Telegram bot for restocks and price drops: JSON-LD first, CSS selectors as a fallback

DevKit — Mon, 26 Oct 2026 12:28:54 +0000

Written with AI assistance; the template itself was built and tested by the author's AI assistant.

You want a ping when a sold-out item comes back, or when a price falls by a meaningful amount. You don't need a scraping farm for that. Most shops already publish the data you need in a machine-readable form, and a small asyncio process with SQLite can watch a few dozen product pages.

This article walks through the design of a bot built as a reusable template (Python, aiogram 3, httpx, selectolax, SQLite). Every idea below can be reimplemented in an afternoon without buying anything.

Step 1: read structured data before touching CSS

Most shops emit schema.org Product markup in a <script type="application/ld+json"> tag, because search engines want it. That's far more stable than a CSS class that changes with each redesign. So the first adapter just looks for it.

JSON-LD shows up in a few shapes: a single object, a list, or an object with @graph. A small generator flattens them:

import json

def iter_json_ld_blocks(html_parser):
    for node in html_parser.css('script[type="application/ld+json"]'):
        raw = node.text(deep=True)
        if not raw or not raw.strip():
            continue
        try:
            data = json.loads(raw)
        except json.JSONDecodeError:
            continue
        if isinstance(data, list):
            yield from data
        elif isinstance(data, dict):
            if isinstance(data.get("@graph"), list):
                yield from data["@graph"]
            else:
                yield data

Broken JSON in one block is skipped rather than fatal, since real pages often carry several blocks from different plugins.

Offers, availability and variants

Availability is a URL such as https://schema.org/InStock. Take the last path segment, normalise it, and map it to three states: in stock, out of stock, unknown.

AVAILABILITY_MAP = {
    "instock": "in_stock",
    "limitedavailability": "in_stock",
    "preorder": "in_stock",
    "backorder": "out_of_stock",
    "outofstock": "out_of_stock",
    "soldout": "out_of_stock",
    "discontinued": "out_of_stock",
}

def normalize_availability(raw):
    if not raw:
        return "unknown"
    key = raw.rsplit("/", 1)[-1].strip().lower()
    key = key.replace(" ", "").replace("_", "").replace("-", "")
    return AVAILABILITY_MAP.get(key, "unknown")

(The real map has a few more entries such as OnlineOnly, InStoreOnly and PreSale. Whether a pre-order counts as "in stock" is a judgment call, so change it if you disagree.)

The "unknown" state matters. If a page says nothing clear, you don't want to alert on it.

Some ProductGroup pages, commonly Shopify themes, carry no top-level offer. Price and availability sit on each variant under hasVariant. The adapter handles this in the simplest possible way:

offer = first_offer(node.get("offers")) or {}
if not offer and isinstance(node.get("hasVariant"), list) and node["hasVariant"]:
    first_variant = node["hasVariant"][0]
    if isinstance(first_variant, dict):
        offer = first_offer(first_variant.get("offers")) or {}
price = parse_price(offer.get("price") or offer.get("lowPrice"))
currency = offer.get("priceCurrency")

Be aware of the limitation: it reads the first variant only. If you care about one specific size or colour, point the bot at that variant's own URL, or extend this to match on SKU.

If no Product JSON-LD exists, the adapter also tries OpenGraph tags (product:price:amount, product:availability) before giving up with a parse error.

Prices need cleaning too, since "1 234,56" and "1,234.56" both occur. The JSON-LD adapter strips non-numeric characters and treats a lone comma as a decimal separator. It is a heuristic, so test it on your own sites.

Step 2: CSS selectors for sites without markup

Some storefronts, usually hand-built ones, have no structured data. For those, you declare selectors once per site in a YAML file, matched by a glob on the URL:

sites:
  - name: example-generic-shop
    url_pattern: "https://shop.example.com/product/*"
    adapter: generic_css
    interval_seconds: 1800
    jitter_seconds: 300
    selectors:
      name: "h1.product-title"
      price: "span.price"
      availability: "div.stock-status"
    in_stock_text: "in stock"
    out_of_stock_text: "sold out"

The adapter extracts the text of each selector. Availability is a case-insensitive substring check, with your configured phrases tested first and a few common English and Russian phrases as defaults:

def detect_availability(text, in_stock_text, out_of_stock_text):
    if text is None:
        return "unknown"
    lowered = text.lower()
    if out_of_stock_text and out_of_stock_text in lowered:
        return "out_of_stock"
    if in_stock_text and in_stock_text in lowered:
        return "in_stock"
    if any(p in lowered for p in ("in stock", "available")):
        return "in_stock"
    if any(p in lowered for p in ("out of stock", "sold out", "unavailable")):
        return "out_of_stock"
    return "unknown"

Note the order: "out of stock" is checked before the generic "in stock" patterns would match, which avoids the classic bug where "not available" reads as available. Keep your own out_of_stock_text specific for the same reason. If none of your selectors match, the adapter raises a parse error that tells you to check the YAML, so a redesigned page fails loudly instead of silently reporting nothing.

Step 3: change detection that alerts once

The trick for "alert once" is boring: persist the last known state per product, compare the new reading to it, then overwrite it. The comparison itself is a pure function with no I/O, which makes it easy to unit test.

def detect_changes(old_price, old_available, new_price, new_availability, threshold_percent):
    events = []
    new_available = new_availability == "in_stock"
    if old_available is not None and new_availability != "unknown":
        if not old_available and new_available:
            events.append("back_in_stock")
        elif old_available and not new_available:
            events.append("out_of_stock")
    if old_price and new_price is not None and new_price != old_price:
        change = abs(new_price - old_price) / old_price * 100
        if change >= threshold_percent:
            events.append("price_change")
    return events

(The real function returns ChangeEvent dataclasses carrying the URL, name and old/new price, but the logic is the same.) Details worth copying:

old_available is None means this is the first check. There is no baseline, so no event. You won't get a flood when you add a product.
An unknown reading never triggers anything.
The price threshold is a percentage (default 5.0, set with PRICE_CHANGE_THRESHOLD_PERCENT), and it fires on rises as well as drops. The message text says which direction.

After each check, the new state is written back in SQLite, plus a row in a price_history table:

await conn.execute(
    "UPDATE products SET name = COALESCE(?, name), last_price = ?, "
    "last_currency = ?, last_available = ?, last_checked_at = ? WHERE id = ?",
    (name, price, currency, None if available is None else int(available), now, product_id),
)
await conn.execute(
    "INSERT INTO price_history (product_id, price, available, checked_at) VALUES (?, ?, ?, ?)",
    (product_id, price, None if available is None else int(available), now),
)
await conn.commit()

Because the baseline moves forward on every check, a back-in-stock alert fires once per transition. A caveat I'd rather you hear from me: a slow drift, say 2% per day, never crosses a 5% threshold between two consecutive checks, so it never alerts. If that matters, compare against the price from your last alert, not the last check.

Another quirk: if one check returns unknown, the stored availability becomes null, so the next definite reading is treated as a new baseline and does not alert.

Step 4: Telegram alerts

Subscriptions live in SQLite (users, products, subscriptions), so one product can have several subscribers and gets fetched once. Sending is a loop around aiogram's send_message, and one blocked user must not stop the rest:

async def send_events(bot, events, subscriber_tg_ids):
    for event in events:
        text = event.format_message()
        for tg_id in subscriber_tg_ids:
            try:
                await bot.send_message(tg_id, text)
            except TelegramAPIError as exc:
                logger.warning("failed to notify %s: %s", tg_id, exc)

Users add products with /add <url>, which also runs a first check. /list, /remove <id> and /check cover the rest. Bot replies and alerts come from a small strings table with English and Russian versions, picked by a LANGUAGE=en|ru setting.

Step 5: scheduling and polite fetching

A single asyncio loop wakes every 30 seconds and asks the database which products are due: never checked, or older than their interval plus a random jitter. The defaults are 1800 s interval and up to 300 s jitter. Jitter keeps products from becoming due in lock-step.

threshold = interval + (random.uniform(0, jitter_seconds) if jitter_seconds else 0)
if (now - last).total_seconds() >= threshold:
    due.append(product)

The fetcher adds the following, all configurable through environment variables:

a robots.txt check per host, cached, enabled by default (RESPECT_ROBOTS_TXT); a disallowed URL raises an error instead of being fetched. If robots.txt is missing or can't be fetched, the fetcher assumes allowed;
a per-host limit, default 6 requests per minute;
up to 3 attempts with linear backoff (2 s, then 4 s) on HTTP errors;
a 15 s timeout.

A failed fetch or parse is logged and skipped; the loop carries on.

Step 6: JS-rendered prices (optional)

If the price is injected by JavaScript, the HTML contains nothing to parse. The template includes a small Playwright fetcher that loads the page in Chromium and returns the rendered HTML:

async with async_playwright() as p:
    browser = await p.chromium.launch()
    try:
        page = await browser.new_page(user_agent=user_agent)
        await page.goto(url, timeout=20_000, wait_until="networkidle")
        return await page.content()
    finally:
        await browser.close()

Playwright is imported lazily, so it isn't required unless you use it. It is switched on with USE_PLAYWRIGHT=true or per site with playwright: true in sites.yaml. Browser fetches wait on the same per-host rate limiter as plain HTTP requests, and if Playwright is enabled but not installed the bot logs one error and falls back to httpx. Retries and the robots.txt check apply only to the plain HTTP path. Also check whether the site offers a JSON endpoint behind that JavaScript first; it's lighter and kinder to the server.

Step 7: run it

With Docker Compose, the SQLite file lives in a named volume so subscriptions survive restarts:

services:
  stockbot:
    build: .
    image: stockbot:latest
    restart: unless-stopped
    env_file:
      - .env
    volumes:
      - stockbot-data:/app/data

volumes:
  stockbot-data:

Without Docker, a systemd unit does the job:

[Service]
Type=simple
User=stockbot
WorkingDirectory=/opt/stockbot
EnvironmentFile=/opt/stockbot/.env
ExecStart=/opt/stockbot/.venv/bin/python -m stockbot
Restart=on-failure
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true

The bot needs a token from BotFather in BOT_TOKEN and refuses to start without it.

Being a good citizen

Monitoring a page is still automated traffic to someone else's server.

Read each site's terms of service and robots.txt before adding it. The template honours robots.txt by default, but a permissive robots file is not the same as permission in the terms.
Keep intervals modest. Half an hour is the default here; for most restocks, hourly or slower is plenty. Don't lower it just because you can.
Watch a handful of products, not a whole catalogue.
If a store has an official API, a product feed or an email "notify me" button, use that instead of scraping.
If a site asks you to stop or starts returning errors, back off.

What this is not

Not a scalping or checkout bot. It reads public product pages and sends you a message. It doesn't log in, fill carts or buy anything.
Not an anti-bot toolkit. It does not solve CAPTCHAs, spoof fingerprints or defeat bot protection. For transparency about what it does contain: it rotates the User-Agent from a configurable list, and the fetcher accepts an optional proxy list, which is empty by default. If a site blocks you, treat that as a "no" and don't work around it.
One process, one SQLite file. There is no queue, no worker pool and no horizontal scaling. That's fine for personal or small-shop use, and not designed for thousands of URLs.
Heuristic parsing. Price formats, variant handling and availability wording vary by site, so test each one before trusting it.

Update: v1.2 and v1.3 (slow drift, silent parser failures, variant switches)

A reader, @sid_mishra, pointed out two gaps in the comments, and both are now fixed in the template:

Slow drift. Each product now stores two prices: the last one seen and the one from the last alert. A new reading is compared with the last alerted price, so five small drops of 1% add up to one 5% alert instead of none. After an alert, the baseline moves to the new price.
Unknown readings. An unknown reading no longer clears the last known price or stock state. The bot counts consecutive unknowns per product (including pages the parser can't read at all), and after UNKNOWN_ALERT_AFTER in a row (default 3, 0 turns it off) it sends one "the page layout may have changed" alert. That's the silent failure a price monitor otherwise never reports.
Variant switches (v1.3). Another reader, @double_chen_70da460344c73, noted that the first-variant fallback can make a size change look like a price drop. The adapter now returns a variant id (sku, then productID/gtin, then the offer URL) and the bot stores it with the price. If the variant changes, there is no price alert; the baseline resets and the bot sends one "variant changed" note instead. Price alerts show the SKU when it is known.

Get the template

The ideas above are enough to build your own. If you'd rather start from working code, the full template (source, 76 tests, Dockerfile and Compose file, systemd unit, README in English and Russian) is sold for 9 USDT through the Telegram shop. There is also a free monitoring kit for a single VPS.

Building a Telegram lead-intake bot with aiogram 3 and SQLite: a form, an admin inbox and no lost requests

DevKit — Mon, 26 Oct 2026 07:19:18 +0000

Written with AI assistance; the template itself was built and tested by the author's AI assistant.

Many small businesses take requests in Telegram DMs: "how much is a haircut?", "call me back", a phone number buried in a voice message. Some of those get lost between chats. This article shows a small structure that fixes it: a bot that asks a fixed set of questions, validates the answers, writes them to SQLite before doing anything else, and pushes a card with status buttons to the admins.

Everything below is adapted from a small aiogram 3 project. You can rebuild the ideas yourself from the snippets.

The shape of the thing

A client presses /start and walks through a form: service, name, phone, comment, confirmation.
The lead is saved to SQLite and gets a number.
Every admin receives a card with buttons: In work / Done / Rejected.
When an admin changes the status, the client gets a message about it.
A per-user limit stops someone from flooding the inbox.

1. The form as an FSM

aiogram's finite state machine is a good fit: one state per question.

from aiogram.fsm.state import State, StatesGroup


class LeadForm(StatesGroup):
    service = State()
    name = State()
    phone = State()
    comment = State()
    confirm = State()

Handlers are filtered by state, so a message only reaches the handler that expects it. Moving forward is a lookup table, and an editing flag lets the user fix one field from the confirmation screen and jump straight back:

NEXT_STEP = {
    LeadForm.service: LeadForm.name,
    LeadForm.name: LeadForm.phone,
    LeadForm.phone: LeadForm.comment,
}


async def advance(current: State, message: Message, state: FSMContext, config: Config) -> None:
    """After a field is entered: next step, or back to the confirmation screen."""
    data = await state.get_data()
    if data.get("editing") or current == LeadForm.comment:
        await show_confirm(message, state, config)
    else:
        await ask(NEXT_STEP[current], message, state, config)

Two catch-all handlers matter more than they look: one for users with no state (StateFilter(None) → "Press /start") and one for any message inside the form that no step handler accepted, such as a sticker instead of a name. Without them the bot just goes silent, and silence reads as "broken".

(advance is from app/handlers/client.py; LeadForm from app/states.py.)

2. Validate before you store

A lead with a bad phone number is a lost lead. The bot accepts both Telegram's "share contact" button and typed text, strips spaces, dashes, dots and brackets, and normalizes to +digits. The English variant of the check:

def _normalize_international(cleaned: str) -> str | None:
    """Optional "+" followed by 8-15 digits, returned as +XXXXXXXX.

    Local numbers with a leading trunk "0" are rejected (the country code is unknown);
    the "00" international prefix is treated as "+".
    """
    digits = cleaned.lstrip("+")
    if not cleaned.startswith("+"):
        if digits.startswith("00"):
            digits = digits[2:]
        elif digits.startswith("0"):
            return None
    if not 8 <= len(digits) <= 15 or digits.startswith("0"):
        return None
    return "+" + digits

Rejecting a local 07700… is a deliberate trade-off: without a country code the number is not callable from elsewhere, so the bot asks again with an example. The Russian variant additionally turns 8XXXXXXXXXX and 9XXXXXXXXX into +7…. Names are trimmed, whitespace-collapsed and limited to 2–60 characters; comments to 500. User text is passed through html.escape before it is shown in the HTML-formatted confirmation. (From app/validators.py.)

3. Save first, notify second

Two tables, users and leads, with a status column and indexes for the queries the admin screens use:

CREATE TABLE IF NOT EXISTS leads (
    id         INTEGER PRIMARY KEY AUTOINCREMENT,
    user_id    INTEGER NOT NULL,
    username   TEXT,
    service    TEXT NOT NULL,
    name       TEXT NOT NULL,
    phone      TEXT NOT NULL,
    comment    TEXT NOT NULL DEFAULT '',
    status     TEXT NOT NULL DEFAULT 'new',
    created_at TEXT NOT NULL,
    updated_at TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS idx_leads_user_created ON leads (user_id, created_at);
CREATE INDEX IF NOT EXISTS idx_leads_status ON leads (status, id);

Timestamps are UTC strings in %Y-%m-%d %H:%M:%S, which sort lexicographically, so created_at >= ? just works. The connection uses WAL mode (PRAGMA journal_mode=WAL) via aiosqlite.

The "send" handler is ordered on purpose: clear the FSM state first (a double tap on the button cannot create a duplicate), check the limit, then insert, answer the client, and only then notify admins. If Telegram fails to deliver to an admin, the lead is already in the database and visible in the admin list. That is the "no lost requests" part. (From app/db.py, confirm_send in app/handlers/client.py.)

4. The admin inbox

Admins get the lead card with one button per other status:

def lead_status_kb(lead_id: int, current: str, language: str = "ru") -> InlineKeyboardMarkup:
    """Status change buttons; the current status is not shown."""
    row = [
        InlineKeyboardButton(
            text=tr(language, f"btn_{status}"), callback_data=f"lead:{lead_id}:{status}"
        )
        for status in STATUS_BUTTONS
        if status != current
    ]
    return InlineKeyboardMarkup(inline_keyboard=[row] if row else [])

Sending goes through a wrapper that survives the things that actually happen: a flood limit (one retry after the retry_after delay), a user who blocked the bot (marked blocked in users), and other API errors (logged, not raised). Admins are notified concurrently:

async def notify_admins(bot: Bot, config: Config, db: Database, lead: dict) -> int:
    """Sends the lead card to all admins. Returns the number delivered."""
    text = config.tr("new_lead") + format_lead(lead, config.tz, config.language)
    markup = lead_status_kb(lead["id"], lead["status"])
    results = await asyncio.gather(
        *(safe_send(bot, admin_id, text, reply_markup=markup) for admin_id in config.admin_ids)
    )
    return sum(results)

The callback handler for the buttons validates the status against a whitelist, ignores a repeated press of the same status, updates the row, edits the card in place and tells the client:

@router.callback_query(F.data.startswith("lead:"))
async def change_status(cb: CallbackQuery, db: Database, config: Config) -> None:
    _, lead_id_raw, status = cb.data.split(":")
    if status not in STATUSES:
        await cb.answer()
        return
    # ... "lead not found" and "already has this status" checks omitted ...
    lead = await db.set_status(existing["id"], status)
    await cb.answer(status_label(status, config.language))
    # ... the card is edited in place here ...
    key = CLIENT_STATUS_TEXT.get(status)
    if key:  # the "new" status is not sent to the client
        await safe_send(cb.bot, lead["user_id"], config.text(key, number=lead["id"]), db=db)

Admin handlers sit on a router with a filter that checks the user ID against ADMIN_IDS, so everyone else's taps are never routed there. The admin menu (/admin) also has paginated lists of new and in-work leads, today/7-day counts, a CSV export (cells starting with =, +, -, @ are prefixed to defuse spreadsheet formula injection) and a broadcast to everyone who has not blocked the bot. (From app/keyboards.py, app/notify.py, app/handlers/admin.py.)

5. Anti-flood without extra infrastructure

You do not need Redis for this. The leads table already knows how many requests a user created recently:

WINDOW = timedelta(hours=1)


async def can_create_lead(
    db: Database, user_id: int, limit: int, now: datetime | None = None
) -> bool:
    since = (now or utcnow()) - WINDOW
    return await db.count_recent_leads(user_id, since) < limit

The check runs on /start and again on confirm. The limit comes from config, so a salon can set 3 per hour and a busier shop something else. (From app/throttle.py.)

6. Config and language

Texts, services and limits live in YAML, not in code, so the owner changes wording without touching Python. One setting switches the built-in buttons, admin panel, date format and phone rules between ru and en:

language: en
timezone: Europe/London

working_hours:
  text: "Mon–Sat, 10:00–20:00"
  start: "10:00"
  end: "20:00"
  days: [1, 2, 3, 4, 5, 6]       # 1 = Monday ... 7 = Sunday

antispam:
  max_leads_per_hour: 3

services:
  - "Haircut"
  - "Colouring"
  - "Other"

Outside working hours the client still gets the confirmation, plus a line saying when you will reply. Secrets (BOT_TOKEN, ADMIN_IDS) stay in .env, not in this file. (From config.en.yaml, trimmed.)

7. Running it

The container is plain python:3.12-slim, runs as a non-root user, and uses long polling, so there is no webhook, domain or TLS to set up:

services:
  bot:
    build: .
    container_name: leads-bot
    restart: unless-stopped
    env_file: .env
    environment:
      DB_PATH: data/leads.db
    volumes:
      - ./data:/app/data                         # the leads database
      - ./config.yaml:/app/config.yaml:ro        # texts and services, editable without a rebuild
      - ./config.en.yaml:/app/config.en.yaml:ro  # English edition (CONFIG_PATH=config.en.yaml)

Put BOT_TOKEN, ADMIN_IDS and, for English, CONFIG_PATH=config.en.yaml in .env, then docker compose up -d --build. Every admin must press /start in the bot once, otherwise Telegram will not let the bot message them. (From docker-compose.yml.)

What this is not

Not a CRM. There are four statuses, a list and a CSV export. No pipelines, reminders, assignments or history of changes.
One SQLite file. Simple and enough for a small inbox, but it is your only copy. Back up data/leads.db (or use the CSV export) on a schedule.
A polling bot on one process. Run exactly one instance per token; a second one will fight over updates. FSM state is not meant to be shared between processes.
No payments. It collects requests; it does not take money.

Wrapping up

The ideas that carry over to any bot: one FSM state per question, normalize input before you store it, write to the database before notifying anyone, keep status changes behind a whitelist, and let the database enforce your rate limit. That is enough to stop losing requests.

If you would rather not assemble it yourself: the full template (source, tests, Docker setup, config in English and Russian, README in both languages) is sold for 9 USDT via the Telegram shop. There is also a free monitoring kit for a single VPS if you want to know when the bot's server goes down.

Deploying FastAPI + PostgreSQL to a single VPS with Docker Compose: HTTPS, backups and rollbacks without Kubernetes

DevKit — Mon, 26 Oct 2026 02:20:49 +0000

Written with AI assistance; the template itself was built and tested by the author's AI assistant.

If you have one VPS and a FastAPI app with PostgreSQL, you don't need Kubernetes. You do need a few boring things done properly: a sane network layout, HTTPS that renews itself, backups you have actually checked, and a deploy that can undo itself. Below is how I wired each piece with plain Docker Compose. All snippets are shortened from a working template, and you can copy the ideas without buying anything.

1. Network layout: only nginx is public

Five services: app (gunicorn + uvicorn workers), db (Postgres 16), nginx, certbot, and backup. Only nginx publishes ports. The database sits in an internal network, so it has no route out and no published port.

services:
  db:
    image: postgres:16-alpine
    # No ports: on purpose.
    networks: [backend]
  app:
    networks: [frontend, backend]
  nginx:
    image: nginx:1.27-alpine
    ports:
      - "0.0.0.0:80:80"
      - "0.0.0.0:443:443"
    networks: [frontend]

networks:
  frontend:
  backend:
    internal: true

Why so strict? Docker writes its own iptables rules, so a published port like 5432:5432 is reachable from the internet even if ufw says deny. The safest approach is to never publish what you don't want public. The app container also runs with read_only: true, cap_drop: [ALL], no-new-privileges, and a non-root user in the Dockerfile.

2. A healthcheck that actually checks the database

A /health that always returns 200 only tells you the process is alive. Here it runs SELECT 1 and returns 503 if the DB is unreachable:

@app.get("/health")
def health(db: DbSession, response: Response):
    try:
        db.execute(text("SELECT 1"))
    except SQLAlchemyError:
        response.status_code = 503
        return {"status": "degraded", "database": "down"}
    return {"status": "ok", "database": "up"}

Compose uses it for the container healthcheck, and nginx waits for the app to be healthy before starting:

services:
  app:
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request as u; u.urlopen('http://127.0.0.1:8000/health', timeout=4)"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 20s
    depends_on:
      db:
        condition: service_healthy

urlopen raises on HTTP errors, so a 503 makes the container unhealthy. Using Python's stdlib means no curl in the image. The deploy script (section 6) also hits /health through nginx, so one request exercises nginx, app and DB.

3. HTTPS: the chicken-and-egg problem

nginx refuses to start if the certificate files referenced in its config don't exist. Let's Encrypt's webroot challenge needs a running nginx to answer on port 80. The way out:

Create a throwaway self-signed certificate at the path nginx expects.
Start nginx.
Delete the dummy certificate.
Ask certbot for the real one via webroot.
Reload nginx.

The core of the bootstrap script:

certbot_run() { docker compose run --rm --no-deps "$@"; }

certbot_run --entrypoint "openssl req -x509 -nodes -newkey rsa:2048 -days 1 \
    -keyout /etc/letsencrypt/live/$domain/privkey.pem \
    -out /etc/letsencrypt/live/$domain/fullchain.pem -subj /CN=localhost" certbot

docker compose up -d nginx
# ... remove the dummy cert, then:
args=(certonly --webroot -w /var/www/certbot --email "$email" -d "$domain"
    --rsa-key-size 4096 --agree-tos --no-eff-email --non-interactive)
[[ $staging -eq 1 ]] && args+=(--staging)
certbot_run --entrypoint certbot certbot "${args[@]}"
docker compose exec -T nginx nginx -s reload

Run it with --staging first. Staging has far looser rate limits than production, so you can debug DNS and firewall issues freely; the browser won't trust the result, which is fine. Then rerun with --force to replace it with a production certificate.

Renewal is a second container that loops twice a day:

services:
  certbot:
    image: certbot/certbot:latest
    entrypoint: >
      /bin/sh -c 'trap exit TERM;
      while :; do
      certbot renew --webroot -w /var/www/certbot --quiet;
      sleep 12h & wait $${!};
      done'

nginx serves /.well-known/acme-challenge/ from the shared webroot volume. The template also mounts a small script into nginx that runs nginx -s reload every 6 hours, so renewed certificates get picked up without a manual step.

4. nginx: headers and rate limiting

The config is an nginx template; with NGINX_ENVSUBST_FILTER: ^DOMAIN$$ only $DOMAIN is substituted, so nginx's own $variables stay intact. The relevant parts:

server_tokens off;
limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
limit_req_status 429;

server {
    listen 443 ssl;
    http2 on;
    ssl_protocols TLSv1.2 TLSv1.3;
    client_max_body_size 5m;

    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "DENY" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    location / {
        limit_req zone=perip burst=20 nodelay;
        proxy_pass http://app_backend;
    }
}

The real file also sets a CSP (default-src 'none', which suits a JSON API but not a site serving HTML), a Permissions-Policy, gzip, proxy headers and timeouts. Ten requests per second per IP with a burst of 20 is a starting point, not a tuned value; adjust it to your traffic. Because the app trusts X-Forwarded-* from anyone (FORWARDED_ALLOW_IPS: "*"), it's important that the app port is never published.

5. Backups: dump, verify, then rotate

The backup container runs a loop that sleeps until BACKUP_AT (UTC), then dumps. The important detail is that a file isn't published until it's been verified:

do_backup() {
    file="$BACKUP_DIR/${PGDATABASE}_${ts}.dump"

    pg_dump --format=custom --no-owner --file="$file.tmp"
    # A dump without a readable table of contents is useless
    pg_restore --list "$file.tmp" >/dev/null
    mv "$file.tmp" "$file"

    find "$BACKUP_DIR" -maxdepth 1 -name "${PGDATABASE}_*.dump" -type f \
        -mmin "+$((RETENTION_DAYS * 24 * 60))" -print -delete
}

pg_restore --list is a cheap sanity check: it proves the archive is readable, but it is not a restore test. Do a real restore into a scratch database now and then. Retention defaults to 7 days. backup.sh once takes a dump immediately, which the deploy script uses.

6. Tag-based deploy with rollback

CI builds an image on a v*.*.* tag, pushes it to GHCR, then the server runs scripts/deploy.sh <tag>. The script's order is the useful part:

Take a backup (abort the deploy if it fails).
Remember the previous tag (APP_PREV_TAG in .env), set the new APP_TAG.
docker compose pull app.
Run alembic upgrade head. If it fails, restore the tag and stop.
docker compose up -d, then poll /health through local nginx for up to about a minute.
If it never goes healthy, print the app logs and roll back.

log "checking /health"
if healthy 30; then
    log "success: $TAG is running"
    exit 0
fi

log "/health not responding after deploying $TAG"
docker compose logs --tail=50 app || true
rollback "$PREV_TAG" || true
exit 1

and the rollback itself is just "put the old tag back":

rollback() {
    log "rolling back to $1"
    env_set APP_TAG "$1"
    docker compose up -d --no-build app
    healthy 30 && log "rollback done" || log "WARNING: still failing"
}

Image tags in .env make this simple: the compose file uses ${APP_IMAGE}:${APP_TAG}, so a rollback is a one-line change plus up -d.

The catch, which the script documents: rollback restores the image, not the database schema. Migrations must be backward compatible (add a column in one release, remove it in a later one). If data is damaged, you restore from the pre-deploy dump.

What this is not

Not high availability. One server is a single point of failure. A reboot, a disk failure or a provider outage means downtime.
Backups live on the same server (./backups) unless you copy them off-host yourself. A dead disk takes the database and its dumps together. Add rsync or rclone to object storage.
Not zero-downtime. The app container is recreated on deploy, so expect a short gap.
Not tested on every cloud provider. The bootstrap script targets Ubuntu 24.04 and checks for it.
No monitoring or alerting is built into the stack beyond container healthchecks and logs (capped at 3 × 10 MB per service).

If you want the whole thing

The snippets above are enough to rebuild the approach yourself. If you'd rather start from a finished setup, the full template (compose, nginx, certbot, backup, CI/CD workflows, bootstrap and deploy scripts, tests, Makefile, README in English and Russian) is sold for 11 USDT via the Telegram shop. There's also a free monitoring kit for a single VPS.

Hardening a fresh Ubuntu 24.04 server with Ansible without locking yourself out

DevKit — Sun, 25 Oct 2026 21:22:33 +0000

Written with AI assistance; the role itself was built and lint-checked by the author's AI assistant, which works for a DevOps engineer.

Hardening a server is easy. Hardening it and still being able to log in afterwards is the part people get wrong. Most lockouts I have seen described come from a few causes: SSH password login turned off before a key was installed, a firewall enabled before the SSH rule existed, or an sshd config with a typo that stops the daemon from restarting.

This article walks through an Ansible approach for a fresh Ubuntu 24.04 VPS that avoids those three. Everything below is free to copy. The snippets are short adaptations of an Ansible role I wrote, so you can apply the ideas in your own playbook.

The order is the safety mechanism

The role's main.yml runs the parts in this order:

users → ufw → ssh → fail2ban → updates → sysctl → auditd → services → banner

The reasoning:

Users first. Create the admin account and install its public key. Until that exists, nothing else should touch SSH.
ufw second. Allow the SSH port, then enable the firewall.
SSH third. Only now apply the strict config: no passwords, no root login. The key and the firewall hole already exist.
Everything else. fail2ban, updates, sysctl and auditd cannot lock you out in the usual way, so they come last.

In Ansible terms it is just the order of your import_tasks:

- name: Users
  ansible.builtin.import_tasks: users.yml
  tags: [users]

- name: Ufw firewall
  ansible.builtin.import_tasks: ufw.yml
  tags: [ufw]

- name: SSH configuration
  ansible.builtin.import_tasks: ssh.yml
  tags: [ssh]

Tags let you re-run one part (--tags ssh) later without replaying the whole thing.

One habit matters more than any task: keep your current SSH session open, and test a login from a second terminal before you close the first.

Refuse to run if there is no key

The cheapest protection is an assertion. If password login is going to be disabled and you forgot to provide a public key, fail before changing anything:

- name: SSH | Administrator key required before disabling passwords
  ansible.builtin.assert:
    that:
      - hardening_admin_ssh_keys | length > 0
    fail_msg: >-
      Password login would be disabled, but hardening_admin_ssh_keys
      is empty. Set a public key or allow password authentication.
    quiet: true
  when: not (hardening_ssh_password_authentication | bool)

The role also asserts that, if you set AllowUsers, the admin user is in the list. Both checks are about the same idea: catch the lockout in the playbook, not on the server.

The admin user itself is the standard pairing of ansible.builtin.user and ansible.posix.authorized_key, with sudo in its groups. Decide on sudo before you run: if the account has neither a password hash nor passwordless sudo, sudo will not work for it. The role defaults to no NOPASSWD, so you either supply a password hash or opt in explicitly.

ufw: rule first, enable last

With community.general.ufw, the sequence matters. Add the SSH allow rule, set policies, and enable at the very end:

- name: UFW | Allow SSH before enabling the firewall
  community.general.ufw:
    rule: allow
    port: "{{ ssh_port | string }}"
    proto: tcp
    from_ip: "{{ ssh_allowed_from | default('any') }}"
    comment: ssh

- name: UFW | Default incoming policy
  community.general.ufw:
    default: deny
    direction: incoming

- name: UFW | Enable the firewall
  community.general.ufw:
    state: enabled

If you restrict from_ip to your own address, be sure that address is stable. A home connection with a changing IP is a good way to lock yourself out slowly.

Key-only SSH, validated before it is written

Ubuntu 24.04 reads /etc/ssh/sshd_config.d/*.conf. A drop-in is much safer than editing sshd_config, because you only own one small file. The role names it 10-hardening.conf. The 10- prefix matters: in sshd the first value found wins, and cloud images often ship 50-cloud-init.conf with PasswordAuthentication yes. Your file must sort before it.

The key trick is the validate: parameter of the template module. Ansible renders to a temporary file, runs the validator on it, and only moves it into place if it passes:

- name: SSH | Privilege separation directory (needed by sshd -t)
  ansible.builtin.file:
    path: /run/sshd
    state: directory
    mode: "0755"

- name: SSH | Write hardening drop-in
  ansible.builtin.template:
    src: sshd-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/10-hardening.conf
    owner: root
    group: root
    mode: "0644"
    validate: /usr/sbin/sshd -t -f %s
  notify: Reload ssh configuration

A broken file never reaches disk, so a typo cannot take down the daemon. The /run/sshd task is there because sshd -t can complain if that directory is missing on a fresh box.

The template itself is plain directives: PermitRootLogin no, PasswordAuthentication no, KbdInteractiveAuthentication no, PubkeyAuthentication yes, MaxAuthTries, X11Forwarding no, LogLevel VERBOSE. The handler reloads sshd rather than restarting it, and existing sessions stay alive.

Two gotchas worth knowing:

TCP forwarding. The role leaves AllowTcpForwarding at yes by default so existing tunnels keep working. CIS-style guidance recommends no; flip it if you do not use tunnels.
Changing the port on 24.04. sshd is socket-activated through ssh.socket, and the Port directive is ignored in that mode. The role writes a ssh.socket.d override for non-default ports. Changing the port is where most of the remaining lockout risk lives, since the old port is not closed in ufw automatically. If you do not need a custom port, leave it at 22 and let fail2ban handle the noise.

fail2ban

On a public IP, SSH brute-force attempts start quickly. fail2ban is a light answer. The role writes one jail, using the systemd journal as the backend (Ubuntu 24.04 does not need a log file for this):

- name: fail2ban | sshd jail
  ansible.builtin.copy:
    dest: /etc/fail2ban/jail.d/sshd-hardening.local
    owner: root
    group: root
    mode: "0644"
    content: |
      [DEFAULT]
      ignoreip = 127.0.0.1/8 ::1 {{ my_office_ip }}

      [sshd]
      enabled = true
      port = {{ ssh_port }}
      backend = systemd
      maxretry = 5
      findtime = 10m
      bantime = 1h

Put your own static IP in ignoreip if you have one. A typo'd key on a few retries should not ban the one person who is allowed in.

Unattended security upgrades, security only

Automatic patching is probably the highest value-per-line item here. The role drops a file that loads after 50unattended-upgrades, clears the origin list with #clear, and re-adds only the security pockets:

- name: Updates | Security-only unattended upgrades
  ansible.builtin.copy:
    dest: /etc/apt/apt.conf.d/52hardening-unattended-upgrades
    owner: root
    group: root
    mode: "0644"
    content: |
      APT::Periodic::Update-Package-Lists "1";
      APT::Periodic::Unattended-Upgrade "1";
      #clear Unattended-Upgrade::Allowed-Origins;
      Unattended-Upgrade::Allowed-Origins {
              "${distro_id}:${distro_codename}-security";
      };
      Unattended-Upgrade::Automatic-Reboot "false";

The role also allows the ESM security origins; I left them out here for brevity. Automatic reboot is off by default. If a kernel update needs a reboot, the server waits for you. Turn it on (with a reboot time) only when you are comfortable with the host restarting unattended.

sysctl and auditd

Neither can lock you out of SSH, but both can break workloads, so read what you apply.

For sysctl, the role writes a file to /etc/sysctl.d/60-hardening.conf with the usual network and kernel switches: reverse-path filtering, no ICMP redirects, no source routing, martian logging, SYN cookies, kernel.kptr_restrict, kernel.dmesg_restrict, fs.protected_symlinks, and so on. You can express that with the ansible.posix.sysctl module:

- name: Sysctl | Network and kernel hardening
  ansible.posix.sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    sysctl_file: /etc/sysctl.d/60-hardening.conf
    reload: true
  loop: "{{ sysctl_params | dict2items }}"
  vars:
    sysctl_params:
      net.ipv4.conf.all.rp_filter: 1
      net.ipv4.conf.all.accept_redirects: 0
      net.ipv4.tcp_syncookies: 1
      kernel.dmesg_restrict: 1
      fs.protected_symlinks: 1

The one that bites: the role's base set includes net.ipv4.ip_forward = 0. That breaks Docker, Kubernetes, VPNs and anything that routes. The role has an override variable for that case. If you run containers, remove forwarding from your list before applying.

For auditd, the role installs baseline rules in /etc/audit/rules.d/: watches on /etc/passwd, /etc/shadow, /etc/sudoers, /etc/ssh/sshd_config.d, login files, time changes, and hostname/network files, each with a key such as identity or sshd:

- name: auditd | Baseline watches
  ansible.builtin.copy:
    dest: /etc/audit/rules.d/60-hardening.rules
    owner: root
    group: root
    mode: "0640"
    content: |
      -w /etc/passwd -p wa -k identity
      -w /etc/shadow -p wa -k identity
      -w /etc/sudoers -p wa -k scope
      -w /etc/sudoers.d -p wa -k scope
      -w /etc/ssh/sshd_config.d -p wa -k sshd

An immutable-rules option (-e 2) exists but is off by default, because after that, any rule change needs a reboot. Enable it deliberately.

Run it safely

ansible-playbook -i inventory.ini site.yml --check --diff
ansible-playbook -i inventory.ini site.yml

Dry-run first. Be aware that --check is less informative on a brand-new host: a package that is not installed yet makes later tasks (like ufw rules) unable to run in check mode, and the role has to tolerate that. Set serial: 1 so a failure on the first server stops the rollout. Then log in from a second terminal.

What this is not

Be honest about the limits:

It is not a CIS compliance audit. It is a pragmatic subset of CIS-style recommendations, a "CIS-lite" baseline. If you need a benchmark score or auditor evidence, use a proper tool and a proper process.
Not every environment was tested. The role targets Ubuntu 24.04 only; running on another OS is a flag you set at your own risk. I cannot claim that ufw or the rest behave the same on every cloud provider and image. Test on a throwaway VM from your provider first.
Cloud security groups are still needed. Many providers have their own network firewall in front of the VM. ufw runs on the host in addition, so a port must be open in both places. Opening a port in ufw alone will not make it reachable.
Hardening is not monitoring or backups. None of this tells you when the disk is full or the service is down.

The full role, if you want it

Everything above is enough to build your own. If you would rather not assemble it, the complete role includes per-part tags and hardening_<part>_enabled switches, a rollback.yml, a Makefile (check, apply, rollback, lint) and passes yamllint and ansible-lint. It is sold for 11 USDT via the Telegram shop. There is also a free monitoring kit for a single VPS.

A free runbook for 44 Prometheus alerts (Linux, Docker, Kubernetes, TLS)

DevKit — Sat, 24 Oct 2026 21:22:28 +0000

Most Prometheus setups I see have alerts but no answer to the 3 a.m. question: "OK, it fired. Now what?" The fix is boring and effective: every alert gets a runbook_url annotation that points to one short page with the same structure every time.

I wrote such a runbook for 44 common alerts (Linux hosts via node-exporter, Docker via cAdvisor, Kubernetes via kube-state-metrics, websites and TLS via blackbox-exporter) and put it online for free:

👉 Prometheus alert runbook (free, English)

The format

Every section answers the same five questions:

What it means: in plain words, including the exact condition.
Common causes: the usual suspects, most likely first.
Diagnostics: copy-paste commands.
How to fix.
When it is OK to silence: so silences get a reason and an expiry instead of becoming "monitoring switched off".

Example: disk will fill within 24 hours

A static "disk > 90%" alert is either too late or too noisy. A trend-based one catches the runaway log before it bites:

- alert: NodeDiskWillFill
  expr: |
    predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}[6h], 24 * 3600) < 0
    and node_filesystem_avail_bytes{fstype!~"tmpfs|overlay|squashfs"}
      / node_filesystem_size_bytes{fstype!~"tmpfs|overlay|squashfs"} * 100 < 40
    and node_filesystem_readonly{fstype!~"tmpfs|overlay|squashfs"} == 0
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Disk {{ $labels.mountpoint }} will fill up within a day ({{ $labels.instance }})"
    runbook_url: "https://devkit-shop.surge.sh/runbook/en/#nodediskwillfill"

The < 40 guard matters: without it, a big, mostly empty disk with a brief write burst will page you for nothing.

And the runbook section it links to starts with diagnostics you can paste straight away:

df -h <mountpoint>
du -xh --max-depth=2 <mountpoint> 2>/dev/null | sort -h | tail -n 20
find <mountpoint> -xdev -type f -mmin -60 -size +100M -exec ls -lh {} +

Then the important judgement call: is the growth steady (rotation is missing, a leak) or a one-off jump (an import or a backup)? For a known one-off, silence it for a few hours, not days.

Example: CrashLoopBackOff

kubectl -n <ns> describe pod <pod>              # Events and Last State
kubectl -n <ns> logs <pod> -c <container> --previous
kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -n 20

--previous is the flag people forget: it shows the log of the run that actually crashed, not the fresh one that is about to.

Tips that apply to any alert set

Put runbook_url in every rule and render it as a link in your Telegram/Slack template. It costs one line and saves minutes at night.
Use only two severities (warning, critical) and route them differently in Alertmanager.
Add an inhibition rule so NodeDown suppresses everything else from that host.
Unit-test your rules with promtool test rules. Thresholds and for: durations are easy to get subtly wrong, and a test catches that before production does.

The rules themselves

The runbook is free to read and bookmark. If you'd rather not write the 44 rules yourself, I sell the ready pack: rules in English and Russian, 113 promtool unit tests, a PrometheusRule for kube-prometheus-stack and an Alertmanager example with severity routing and inhibition. It costs 9 USDT via the Telegram shop. If you're just getting started, there's also a free Prometheus + Grafana + Telegram alerts kit for a single VPS.

Feedback welcome. If an alert you rely on is missing from the runbook, tell me in the comments.

Monitor your VPS in 10 minutes: Prometheus, Grafana and Telegram alerts (free kit)

DevKit — Thu, 22 Oct 2026 12:22:02 +0000

Most people start thinking about monitoring after the first incident: the disk fills up with logs, the database dies from lack of memory, and you learn about it from your users. Basic protection can be set up in an evening, and faster if you don't assemble the stack by hand.

Here is the minimal setup that is enough for one VPS. Prometheus collects metrics, Grafana shows them, and Alertmanager sends messages to Telegram. Everything runs with Docker Compose. I packaged the ready-made configs into a free kit, Monitoring Lite, and the commands below come from it.

What you get

four containers: Prometheus, node-exporter, Alertmanager, Grafana;
one Host Overview dashboard: CPU, memory, disk usage by mountpoint, network, load average, uptime;
five alerts in Telegram: host down, high CPU, high memory, disk almost full, host rebooted;
a Grafana password generated at install time, and ports that are closed to the outside world.

How it fits together

node-exporter reads the host's /proc and /sys and serves metrics over HTTP. Prometheus scrapes them every 15 seconds and stores them. Alert rules are evaluated inside Prometheus: when an expression stays true for the configured time, the alert goes to Alertmanager, which groups messages and sends them to Telegram. Grafana reads data from Prometheus.

The datasource and dashboard are provisioned automatically, so you don't create anything by hand in the UI.

Step 1: a Telegram bot

Message @botfather in Telegram, send /newbot, and pick a name. You get a token like 123456789:AAExxxxxxxx.
Send any message to your new bot. Until you do, it cannot write to you first.
Open https://api.telegram.org/bot<TOKEN>/getUpdates in a browser and find "chat":{"id": ... in the response. That's your chat_id. For a group, add the bot to the group and post something there; the ID will be a negative number.

Step 2: install

You need Docker and Docker Compose. Download the kit: https://devkit-shop.surge.sh/free/devops-monitoring-lite-v1.0.zip, unpack it, and run the installer:

unzip devops-monitoring-lite-v1.0.zip
cd devops-monitoring-lite
TG_BOT_TOKEN=123456789:AAExxxxxxxx TG_CHAT_ID=987654321 ./install.sh

TG_BOT_TOKEN and TG_CHAT_ID are only needed for this run: the installer saves them to .env as TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID. If you don't pass them and they aren't in .env yet, the script asks for the token and chat_id interactively. It:

checks for Docker and docker compose (or the older docker-compose);
creates .env and generates a Grafana admin password in it;
validates the token and chat_id format so a malformed value can't break the YAML;
renders the Alertmanager config from a template with your token and chat_id;
runs docker compose up -d and restarts Alertmanager so it picks up the fresh config.

You can safely re-run it: the Grafana password and other values in .env are kept. To change the token, run the installer again with a new TG_BOT_TOKEN, or edit .env and run ./install.sh.

The token is injected at render time and is not stored in the compose file. The generated alertmanager.yml and .env are both listed in the kit's .gitignore, so they stay out of git.

Step 3: reach Grafana

All ports are bound to 127.0.0.1 on purpose. Prometheus and Alertmanager have no authentication, so they should not be exposed to the internet. From your computer, use an SSH tunnel:

ssh -L 3000:127.0.0.1:3000 user@your-server

Open http://localhost:3000. The login is admin; the password is in .env:

grep GRAFANA_ADMIN_PASSWORD .env

Find the "Host Overview — Сервер" dashboard in the "Monitoring Lite" folder. It shows CPU, memory, disk and network graphs, load average and uptime (the archive includes a sample screenshot, host-overview.png).

You can open access by IP by setting BIND_ADDR=0.0.0.0 in .env and running docker compose up -d. But this setting is shared, so all four ports go public: 3000, 9090, 9093 and 9100. There's a second trap: ports published by Docker bypass ufw rules, so ufw deny 9090 won't close them. You'd have to block them with your hosting provider's firewall or with rules in the iptables DOCKER-USER chain. If in doubt, keep 127.0.0.1 and use the tunnel.

Step 4: verify the alert path

First, check that Prometheus sees the targets and has loaded the rules:

curl -s http://127.0.0.1:9090/api/v1/targets | grep -o '"health":"[a-z]*"'
curl -s http://127.0.0.1:9090/api/v1/rules | grep -o '"name":"[A-Za-z]*"'

Both targets (Prometheus itself and node-exporter) should report "health":"up", and the second command should list five rule names. Now send a test alert:

./scripts/send-test-alert.sh

The script POSTs to the Alertmanager API. Within 30–60 seconds, a Telegram message should arrive with the header "🔴 ВНИМАНИЕ" (Russian for "ALERT"), followed by the line "TestAlert (warning)" and the text "Это тестовый алерт" ("This is a test alert"). The delay comes from group_wait: 30s, which lets Alertmanager batch related alerts into one message.

If nothing arrives, check the logs:

docker compose logs alertmanager | grep -i telegram

Most often the cause is that you never messaged the bot, or the token has a typo.

The rules

The rules live in config/prometheus/alert_rules.yml.

Alert	Condition	`for`
`InstanceDown`	`up{job="node-exporter"} == 0`	2 min
`HighCPU`	average CPU load > 90%	10 min
`HighMemory`	more than 90% of RAM used, based on `MemAvailable`	5 min
`DiskSpace`	disk more than 85% full	5 min
`HostRebooted`	`time() - node_boot_time_seconds < 600`	none

The for clause prevents false alarms: a short CPU spike during a build or a backup won't wake you up at night. HostRebooted has no for, so it fires right after boot and stays active for the first 10 minutes.

The DiskSpace rule ignores tmpfs, overlay, squashfs, devtmpfs, all fuse.* types (such as fuse.sshfs), nfs and nfs4, cifs, smb3 and autofs. Without this filter, it fires on system and network mounts you don't manage. For memory, the rule uses MemAvailable rather than MemFree: Linux uses spare memory for cache, so "free" memory is almost never large.

Thresholds are edited in the same file. After a change, reload the rules without restarting:

curl -X POST http://127.0.0.1:9090/-/reload

This works because Prometheus runs with --web.enable-lifecycle.

Adding a second server

Run node-exporter on the second server and add it to config/prometheus/prometheus.yml under the node-exporter job as a separate entry with its own label:

  - job_name: node-exporter
    static_configs:
      - targets: ["node-exporter:9100"]
        labels:
          instance_name: "main-server"
      - targets: ["10.0.0.5:9100"]
        labels:
          instance_name: "server-2"

Keep the instance_name label on the first server as is; it's already in the kit's config. Then run docker compose restart prometheus. Don't expose port 9100 to the internet; send the traffic over a VPN or private network.

Common problems

The Grafana container won't start. Check docker compose logs grafana. The usual cause is permissions on a volume left over from a previous install. If you don't need the data, ./uninstall.sh --purge-data followed by ./install.sh helps.

A Prometheus target is DOWN. Open the targets page through the tunnel at http://127.0.0.1:9090/targets; it shows the reason, such as connection refused or context deadline exceeded. Usually the service hasn't started yet, or there's a typo in the address in prometheus.yml.

Port already in use. If Grafana or Prometheus is already running on the server, change the relevant variables in .env (GRAFANA_PORT, PROMETHEUS_PORT, ALERTMANAGER_PORT, NODE_EXPORTER_PORT) and run docker compose up -d. Remember to change the port in the curl commands and the SSH tunnel from this article too.

Telegram wasn't configured during install. Alertmanager still starts, just without sending anything. Fill in TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID in .env and run ./install.sh again.

Stopping and removing

./uninstall.sh              # stop, keep data
./uninstall.sh --purge-data # stop and delete data

What's not in this kit

The kit is deliberately limited to one server. It doesn't watch Docker containers, check website availability or SSL certificate expiry, collect logs, or handle backups. If you need those, you can add them yourself: cAdvisor for containers, blackbox-exporter for sites, Loki for logs.

The full version

The paid DevOps Monitoring Starter Kit covers those gaps: cAdvisor, blackbox-exporter for site and SSL checks, three dashboards (host, Docker containers, sites and SSL), 11 alert rules, a backup script, and optionally Loki with Promtail for logs and HTTPS via Caddy. Details are at https://devkit-shop.surge.sh/, where you can pay with USDT (TRC20) or, in Russia, by card.

Disclosure: this article and the kit were prepared by an AI assistant working on behalf of the kit's author, and the commands were checked against the kit files.