DEV Community

#dataengineering

Posts

👋 Sign in for the ability to sort posts by relevant, latest, or top.
A Green Spark Job Isn't Enough: Building a Fail-Closed Pipeline Health Monitor in Python
rupak_kulkarni_08b45c5846 profile

A Green Spark Job Isn't Enough: Building a Fail-Closed Pipeline Health Monitor in Python

#python #dataengineering #devops #testing
Comments
6 min read
How Much Does a Custom Web Scraper Cost to Build?
aqib_javaid profile

How Much Does a Custom Web Scraper Cost to Build?

#webscraping #dataengineering #datapipelines #softwareestimation
Comments
6 min read
Web Scraping vs Official API: Which Should You Use?
aqib_javaid profile

Web Scraping vs Official API: Which Should You Use?

#webscraping #apiintegration #dataengineering #architecture
Comments
6 min read
Equality deletes have no address, and that's why your Iceberg reads crawl
kyien profile

Equality deletes have no address, and that's why your Iceberg reads crawl

#dataengineering #iceberg #sql #flink
Comments
7 min read
Iceberg vs Parquet: do you need a table format?
mohammed_arshadansari_f2 profile

Iceberg vs Parquet: do you need a table format?

#dataengineering #database #data #analytics
Comments
3 min read
Proving exactly-once by killing the stream mid-backlog
tantn89 profile

Proving exactly-once by killing the stream mid-backlog

#dataengineering #spark #iceberg #kafka
Comments 3
8 min read
Spark Is Lazy, and That One Fact Explains Most Beginner Confusion
k_hunnurjirao_7ea28e971f profile

Spark Is Lazy, and That One Fact Explains Most Beginner Confusion

#pyspark #dataengineering #beginners #python
1
Comments
5 min read
Your premium preview audit may be measuring a probe cap
dataspryva profile

Your premium preview audit may be measuring a probe cap

#datascience #python #analytics #dataengineering
Comments
7 min read
Un pipeline de clima del Perú con dbt, DuckDB y GitHub Actions, sin AWS
abel_fernandopacompiaor profile

Un pipeline de clima del Perú con dbt, DuckDB y GitHub Actions, sin AWS

#dataengineering #python #dbt #githubactions
Comments
6 min read
CDC Schema Changes: What Breaks and How to Ship Them
pipelinewhiteboard profile

CDC Schema Changes: What Breaks and How to Ship Them

#dataengineering #kafka #snowflake #database
Comments
7 min read
Bridging Python and Rust: Building Type-Safe Data Quality Pipelines with PyO3 and JSON Schema 2020-12 #rust #python #pyo3 #jsonschema
rajgahoi profile

Bridging Python and Rust: Building Type-Safe Data Quality Pipelines with PyO3 and JSON Schema 2020-12 #rust #python #pyo3 #jsonschema

#dataengineering #performance #python #rust
Comments
9 min read
Bridging the Lakehouse: Ingesting Databricks Unity Catalog into Cognee Knowledge Graphs
pranjul_chaurasiya profile

Bridging the Lakehouse: Ingesting Databricks Unity Catalog into Cognee Knowledge Graphs

#databricks #ai #dataengineering #python
1
Comments
4 min read
Don't treat Fabric data ingestion as a one-size-fits-all problem
aniketsoni profile

Don't treat Fabric data ingestion as a one-size-fits-all problem

#microsoftfabric #dataengineering #analytics #azure
1
Comments
5 min read
AWS + Snowflake: Building Modern Data and AI Workflows with Glue, S3, Iceberg, and Cortex
omayrashraf profile

AWS + Snowflake: Building Modern Data and AI Workflows with Glue, S3, Iceberg, and Cortex

#aws #snowflake #dataengineering #cloud
Comments
7 min read
CDC Deletes, Tombstones and the Idempotent MERGE
pipelinewhiteboard profile

CDC Deletes, Tombstones and the Idempotent MERGE

#dataengineering #kafka #snowflake #sql
Comments
7 min read
👋 Sign in for the ability to sort posts by relevant, latest, or top.