Skip to content
  • Wednesday, 29 October 2025
  • 1:53 pm
  • Follow Us
Data Engineer

Data/ML Engineer Blog

  • Home
  • Books
    • Pipeline the right WAY
      • Chapter 1
  • Data Engineering
    • Cloud Platforms & Services
      • Alibaba Cloud
      • AWS (Amazon Web Services)
      • Azure Microsoft
      • Google Cloud Platform (GCP)
      • IBM Cloud
      • Oracle Cloud
    • Containerization & Orchestration
      • Amazon EKS
      • Apache Oozie
      • Azure Kubernetes Service (AKS)
      • Buildah
      • Containerd
      • Docker
      • Docker Swarm
      • Google Kubernetes Engine (GKE)
      • Kaniko
      • Kubernetes
      • Podman
      • Rancher
      • Red Hat OpenShift
    • Data Catalog & Governance
      • Alation
      • Amundsen
      • Apache Atlas
      • Apache Griffin
      • Atlan
      • AWS Glue
      • Azure Purview
      • Collibra
      • Collibra
      • Databand
      • DataHub
      • Deequ
      • Google Data Catalog
      • Google Dataplex
      • Great Expectations
      • Informatica
      • Marquez
      • Monte Carlo
      • OpenLineage
      • OpenMetadata
      • Soda SQL
      • Spline
    • Data Ingestion & ETL
      • Apache Kafka Connect
      • Apache NiFi
      • Census
      • Confluent Platform
      • Debezium
      • Fivetran
      • Hightouch
      • Informatica PowerCenter
      • Kettle
      • Matillion
      • Microsoft SSIS
      • Omnata
      • Polytomic
      • Stitch
      • StreamSets
      • Striim
      • Talend
    • Data Lakes & File Standards
      • Amazon S3
      • Apache Arrow
      • Apache Avro
      • Apache Iceberg
      • Azure Data Lake Storage
      • CSV
      • Databricks Delta Lake
      • Dremio
      • Dremio
      • Feather
      • Google Cloud Storage
      • JSON
      • ORC
      • Parquet
    • Data Platforms
      • Cloud Data Warehouses
        • ClickHouse
        • Databricks
        • Snowflake
          • Internal and External Staging in Snowflake
          • Network Rules in Snowflake
          • Procedures + Tasks
          • Snowflake administration and configuration
          • Snowflake Cloning
      • Cloudera Data Platform
      • NoSQL Databases
      • On-Premises Data Warehouses
        • DuckDB
      • Relational Databases
        • Amazon Aurora
        • Azure SQL Database
        • Google Cloud SQL
        • MariaDB
        • Microsoft SQL Server
        • MySQL
        • Oracle Database
        • PostgreSQL
    • Data Streaming & Messaging
      • ActiveMQ
      • Aiven for Kafka
      • Amazon Kinesis
      • Amazon MSK
      • Apache Kafka
      • Apache Pulsar
      • Azure Event Hubs
      • Confluent Platform
      • Google Pub/Sub
      • IBM Event Streams
      • NATS
      • Protocol Buffers
      • RabbitMQ
      • Red Hat AMQ Streams
    • Data Warehouse Design
      • Data Governance and Management (DGaM)
        • Compliance Requirements
        • Data Lineage
        • Data Retention Policies
        • Data Stewardship
        • Master Data Management
      • Data Warehouse Architectures (DWA)
        • Enterprise Data Warehouse vs. Data Marts
        • Hub-and-Spoke Architecture
        • Logical vs. Physical Data Models
        • ODS (Operational Data Store)
        • Staging Area Design
      • Data Warehouse Schemas (DWS)
        • Data Vault
        • Galaxy Schema (Fact Constellation)
        • Inmon (Normalized) Approach
        • Kimball (Dimensional) Approach
        • Snowflake Schema
        • Star Schema
      • Database Normalization
      • Dimensional Modeling Techniques (DMT)
        • Bridge Tables
        • Conformed Dimensions
        • Degenerate Dimensions
        • Junk Dimensions
        • Mini-Dimensions
        • Outrigger Dimensions
        • Role-Playing Dimensions
      • ETL/ELT Design Patterns
        • Change Data Capture (CDC)
        • Data Pipeline Architectures
        • Data Quality Management
        • Error Handling
        • Metadata Management
      • Fact Table Design Patterns(FTDP)
        • Accumulating Snapshot Fact Tables
        • Aggregate Fact Tables
        • Factless Fact Tables
        • Periodic Snapshot Fact Tables
        • Transaction Fact Tables
      • Modern Data Warehouse Concepts (MDWC)
        • Data Lakehouse
        • Medallion Architecture
        • Multi-modal Persistence
        • Polyglot Data Processing
        • Real-time Data Warehousing
      • Performance Optimization (PO)
        • Compression Techniques
        • Indexing Strategies
        • Materialized Views
        • Partitioning
        • Query Optimization
      • Slowly Changing Dimensions(SCD)
        • SCD Type 0
        • SCD Type 1
        • SCD Type 2
        • SCD Type 3
        • SCD Type 4
        • SCD Type 6
        • SCD Type 7
    • Distributed Data Processing
      • Apache Beam
      • Apache Flink
      • Apache Hadoop
      • Apache Hive
      • Apache Pig
      • Apache Pulsar
      • Apache Samza
      • Apache Sedona
      • Apache Spark
      • Apache Storm
      • Presto/Trino
      • Spark Streaming
    • Infrastructure as Code & Deployment
      • Ansible
      • Argo CD
      • AWS CloudFormation
      • Azure Resource Manager Templates
      • Chef
      • CircleCI
      • GitHub Actions
      • GitLab CI/CD
      • Google Cloud Deployment Manager
      • Jenkins
      • Pulumi
      • Puppet
      • Tekton
      • Terraform
      • Travis CI
    • Monitoring & Logging
      • AppDynamics
      • Datadog
      • Dynatrace
      • ELK Stack
      • Fluentd
      • Graylog
      • Loki
      • Nagios
      • New Relic
      • Splunk
      • Vector
      • Zabbix
    • Operational Systems (OS)
      • Ubuntu
        • Persistent Tasks on Ubuntu
      • Windows
    • Programming Languages
      • Go
      • Java
      • Julia
      • Python
        • Dask
        • NumPy
        • Pandas
        • PySpark
        • SQLAlchemy
      • R
      • Scala
      • SQL
    • Visualization Tools
      • Grafana
      • Kibana
      • Looker
      • Metabase
      • Mode
      • Power BI
      • QuickSight
      • Redash
      • Superset
      • Tableau
    • Workflow Orchestration
      • Apache Airflow
      • Apache Beam Python SDK
      • Azkaban
      • Cron
      • Dagster
      • Dagster Change
      • DBT (data build tool)
      • Jenkins Job Builder
      • Keboola
      • Luigi
      • Prefect
      • Rundeck
      • Temporal
  • ML/AI Engineering
    • AWS AI/ML Services
    • Compute & Deployment
    • Core AI & ML Concepts
      • Data Processing & ETL
      • Decision Trees
      • Deep Learning
      • Generative AI
      • K-Means Clustering
      • Machine Learning
      • Neural Networks
      • Reinforcement Learning
      • Supervised Learning
      • Unsupervised Learning
    • Database & Storage Services
      • Cortex
    • Emerging AI Trends
    • Evaluation Metrics
    • Industry Applications of AI
    • MLOps & DevOps for AI
    • Model Development & Optimization
    • Prompting Techniques
      • Adversarial Prompting
      • Chain-of-Thought Prompting
      • Constitutional AI Prompting
      • Few-Shot Prompting
      • Instruction Prompting
      • Multi-Agent Prompting
      • Negative Prompting
      • Prompt Templates
      • ReAct Prompting
      • Retrieval-Augmented Generation (RAG)
      • Self-Consistency Prompting
      • Zero-Shot Prompting
    • Security & Compliance
      • AWS KMS
      • AWS Macie
      • Azure Key Vault
      • Azure Purview
      • BigID
      • Cloud DLP
      • HashiCorp Vault
      • Immuta
      • Okera
      • OneTrust
      • Privacera
      • Satori
  • Tech People
    • AI/ML Visionaries
    • Cloudera Data Platform
    • Community Champions
    • Data Quality
    • Data Strategy
    • Modern Stack Leaders
    • Platform Founders
      • Maxime Beauchemin
    • Real-time Systems
  • Home
  • AWS Glue vs. Traditional ETL Tools
AWS Glue vs. Traditional ETL Tools
Data ETL/ELT VS

AWS Glue vs. Traditional ETL Tools

Alex May 28, 2025 0

AWS Glue vs. Traditional ETL Tools: A Cost-Performance Analysis When I began modernizing our organization’s data infrastructure last year, we…

Read More
Snowflake Cost Optimization
Data Snowflake

Snowflake Cost Optimization

Alex May 23, 2025 0

Snowflake Cost Optimization: The Hidden Techniques Most Engineers Miss After working with dozens of engineering teams who’ve seen their Snowflake…

Read More
Snowflake Data Lake Medallion Architecture
DataLake Snowflake

Snowflake Data Lake Medallion Architecture

Alex May 5, 2025 0

Snowflake Data Lake Medallion Architecture: A Blueprint for Scalable, High-Quality Analytics In today’s fast-paced data landscape, organizations must build systems…

Read More
From Documentation Debt to Strategic Asset
Data Snowflake

From Documentation Debt to Strategic Asset

Alex May 2, 2025 0

From Documentation Debt to Strategic Asset: Real-World Success Stories of Automated Snowflake Documentation In data engineering circles, documentation is often…

Read More
The Hidden Cost of Data Copying
Data Snowflake

The Hidden Cost of Data Copying

Alex Apr 18, 2025 0

The Hidden Cost of Data Copying: How Synthetic Data and Zero-Copy Clones Save Millions Data copying might seem like a…

Read More
The Dark Art of Data Sharding
Data

The Dark Art of Data Sharding

Alex Apr 14, 2025 0

The Dark Art of Data Sharding: How Discord and Netflix Split Petabyte-Scale Workloads In today’s hyperscale environment, traditional sharding strategies…

Read More
Snowflake Cost-Saving Tactics: Real SQL Techniques Using Dynamic Date Ranges and Partition Pruning
Data Snowflake

Snowflake Cost-Saving Tactics

Alex Apr 10, 2025 0

Snowflake Cost-Saving Tactics: Real SQL Techniques Using Dynamic Date Ranges and Partition Pruning How to Stop Paying for Data You…

Read More
Green Data Engineering: Can Sustainability Save Your Cloud Bill?
Data

Green Data Engineering

Alex Mar 30, 2025 0

Green Data Engineering: Can Sustainability Save Your Cloud Bill? As cloud costs continue to rise, businesses are being forced to…

Read More
StarRocks vs. Snowflake
Data Snowflake StarRock

StarRocks vs. Snowflake

Alex Mar 2, 2025 0

StarRocks vs. Snowflake: Choosing the Right Platform for Your Data Workloads In the rapidly evolving world of data engineering, choosing…

Read More
From Data to Insights
Structure

From Data to Insights

Alex Feb 26, 2025 0

From Data to Insights: Supporting AI, Machine Learning, and Visualization with AWS In today’s fast-paced, data-driven world, transforming raw data…

Read More

Posts pagination

1 2 3

Recent Posts

  • Multi-Agent Orchestration
  • Chef vs. Puppet
  • Oozie, Keboola, and Apache Beam
  • DBT vs Airflow vs Luigi vs Prefect
  • Terraform vs Ansible

Recent Comments

  1. smortergiremal on Comparison of Equivalent Cloud Services Across AWS, Google Cloud, and Azure
  2. Ustas on The Genius of Snowflake’s Hybrid Architecture: Revolutionizing Data Warehousing

Archives

  • October 2025
  • September 2025
  • August 2025
  • July 2025
  • June 2025
  • May 2025
  • April 2025
  • March 2025
  • February 2025
  • January 2025
  • November 2024
  • October 2024
  • September 2024
  • August 2024
  • July 2024
  • June 2024
  • May 2024
  • April 2024
  • March 2024
  • February 2024
  • January 2024
  • December 2023
  • November 2023
  • October 2023
  • September 2023
  • August 2023
  • July 2023
  • June 2023
  • May 2023

Categories

  • AI
  • Analytics
  • AWS
  • ClickHouse
  • Data
  • Databricks
  • DataLake
  • DevOps
  • DuckDB
  • ETL/ELT
  • Future
  • ML
  • Monthly
  • OpenSource
  • Snowflake
  • StarRock
  • Structure
  • VS
YOU MAY HAVE MISSED
Multi-Agent Orchestration
AI DevOps
Multi-Agent Orchestration
Alex Oct 29, 2025
Chef vs. Puppet
DevOps VS
Chef vs. Puppet
Alex Oct 26, 2025
Oozie, Keboola, and Apache Beam
Data ETL/ELT
Oozie, Keboola, and Apache Beam
Alex Oct 21, 2025
DBT vs Airflow vs Luigi vs Prefect
Analytics Structure VS
DBT vs Airflow vs Luigi vs Prefect
Alex Oct 17, 2025

(c) Data/ML Engineer Blog