##DataEngineering
580 - Once an organization lands its data, it almost never wants to move it again. A big Databricks estate with tons of tables stays put. The win shows up when the environment is still new.

#databricks #datagravity #dataengineering #migration
November 16, 2026 at 9:07 PM
Jimmy Angelakos got 5 minutes & no slides at Community Over Code Glasgow for pgEdge ColdFront: #PostgreSQL data tiering to Apache Iceberg, cold rows still writable.
🎬 www.youtube.com/watch?v=W5bW...
📖 vyruss.org/blog/communi...

#Postgres #ApacheIceberg #DataEngineering #OpenSource
November 12, 2026 at 11:20 PM
Data engineering and analytics engineering use the same tools with a different focus.

DE to AE means learning the business. AE to DE means learning the infrastructure.

#dataengineering #databs
November 17, 2026 at 12:00 PM
The biggest change when an analytics engineer becomes a data engineer is reliability.

Your work is judged by whether the data arrives on time, every time, and by what happens when it doesn't.

#dataengineering #databs
November 17, 2026 at 12:00 PM
Moving from analytics to data engineering? Learn:

- Python: APIs, files, scripts that run on their own
- ingestion into the warehouse
- orchestration: Airflow, scheduling, retries
- cloud: storage, permissions, Docker, Terraform
- scale: partitioning, cost control, later Spark

#dataengineering
November 17, 2026 at 12:00 PM
Moving from data engineer to analytics engineer, or the other way round, is one of the easiest moves in data. You already have half the skills.

Both roles share:
- SQL, and a lot of it
- a cloud data warehouse
- Git, code reviews and testing
- dbt

#dataengineering #databs
November 17, 2026 at 12:00 PM
❄️ Snowflake now ships a way to find out whether an AI function needs the expensive model. Cortex AI Function Studio and AI Function Optimization both went GA on Nov 16.

#Snowflake #SnowflakeSquad #Cortex #DataEngineering #AI
November 17, 2026 at 10:40 AM
Want practical examples and a deeper explanation of S3 partitioning, Parquet, CSV, and ORC?

Read my complete article:

medium.com/@sriwworldof...

Follow for more AWS and Data Engineering insights.

#AWS #DataEngineering #BigData
S3 Partitioning and Parquet vs CSV vs ORC: The Data Engineering Interview Question That Separates…
Learn S3 partitioning and the differences between Parquet, CSV, and ORC with examples, mistakes to avoid, and interview answers.
medium.com
November 17, 2026 at 7:24 AM
Your AWS Athena query scans 5 TB just to retrieve a few MB of data.

The problem might not be your SQL.

It could be your S3 partitioning and file format.

Learn how partition pruning + Parquet can transform query performance.

#AWS #DataEngineering

Read more:
medium.com/@sriwworldof...
S3 Partitioning and Parquet vs CSV vs ORC: The Data Engineering Interview Question That Separates…
Learn S3 partitioning and the differences between Parquet, CSV, and ORC with examples, mistakes to avoid, and interview answers.
medium.com
November 17, 2026 at 7:21 AM
580 - The new warehouse deployed. There's no data in it. Production can't be touched, and a full 10-year copy is overkill, so the real question is how much of a slice dev and QA actually need.

#datawarehouse #testdata #devops #dataengineering
November 17, 2026 at 3:58 AM
580 - New Delta tricks stay inside Databricks for six to nine months before they hit the open spec. That window is the edge. Power BI and every other reader of those tables just waits.

#databricks #deltalake #powerbi #dataengineering
November 16, 2026 at 10:35 PM
Talk to your clinical data: unlocking AWS HealthLake with OpenSharing and Genie One
tech_blogs_databricks

#DataEngineering
Talk to your clinical data: unlocking AWS HealthLake with OpenSharing and Genie One
Set up a custom OpenSharing integration that brings AWS HealthLake FHIR data into Databricks, with Unity Catalog governance and Genie One analytics.
www.databricks.com
November 16, 2026 at 6:39 PM
What is Real-Time Mode in Apache Spark?
tech_blogs_databricks

#DataEngineering
What is Real-Time Mode in Apache Spark?
Real-Time Mode is a Spark Structured Streaming trigger for millisecond, end-to-end latency. Learn how it works, what it supports and when to use it.
www.databricks.com
November 16, 2026 at 6:09 PM
Build a real-time next-best-offer recommender using Databricks Feature Views
tech_blogs_databricks

#DataEngineering
Build a real-time next-best-offer recommender using Databricks Feature Views
Build once, serve everywhere: Feature Views unify batch and streaming features for training and serving.
www.databricks.com
November 16, 2026 at 6:09 PM
Challenge du lundi : LinkedIn en une seule image !

Recruteurs en France : cessez de payer un abonnement hors de prix quand vous pouvez avoir accès à France Travail ou a Apec - Association Pour l'Emploi des Cadres !

#Recrutement #LinkedIn #DataEngineering #Emploi #Travail ...
November 16, 2026 at 2:25 PM
Don't add a project to your portfolio just to show five logos.

Add it because you can explain why each tool is there, and what you'd use instead in a smaller setup.

#dataengineering #databs
November 12, 2026 at 12:00 PM
Spark, MongoDB and Streamlit in one pipeline, each with one job:

- Spark reads from Kafka and transforms the data as it arrives
- MongoDB stores the results as JSON documents, so you don't need a strict schema upfront
- Streamlit shows the data on a simple live dashboard

#dataengineering #databs
November 12, 2026 at 12:00 PM
A portfolio project idea from Andreas Kretz: a real-time data pipeline.

FastAPI takes in JSON events, Kafka holds them, Spark transforms them as they arrive, MongoDB stores the results and Streamlit shows them on a live dashboard. Docker Compose runs it all.

#dataengineering #databs
November 12, 2026 at 12:00 PM
Your Athena bill isn’t a pricing problem.

It’s a file layout problem.

Parquet + partitions can turn “scan 1 TB” into “scan a few GB.”

Same query. Fraction of the cost. 🧵

#AWS #DataEngineering
November 16, 2026 at 1:30 AM
This week, we continue the Let's Get Certified series with DP-700: Microsoft Fabric Data Engineer Associate Event. Thurs, November 19. Hosted by Microsoft Fabric Location Intelligence and Security User Group as part of #DataDays. Register: luma.com/rnf91y37 #MicrosoftFabric #Data #DataEngineering
Let’s Get Certified: DP-700 - Microsoft Fabric Data Engineer Associate Event · Luma
📅 Thursday, November 19, 2026 ⏰ 6:30 to 10:45 AM ET | 7:30 to 11:45 PM PHT | 5:00 to 9:15 PM IST | 12:30 to 4:45 PM WAT 📍 Virtual Event 💰 Free About This…
luma.com
November 15, 2026 at 10:00 PM
Alteryx One Live Query + BigQuery push AI-driven extraction into the warehouse, cutting data movement and enabling SQL pushdown for unstructured docs like invoices. 🧠📄⚡ #BigQuery #AI #DataEngineering #FinTech #Governance
Modernize unstructured data workloads with Alteryx and BigQuery | Google Cloud Blog
Build code-free pipelines with Live Query + BigQuery. Use Gemini securely within your data platform to process and analyze unstructured data.
cloud.google.com
November 12, 2026 at 6:39 AM
Step-by-Step Guide to Transfer Data from Azure Blob to Google Cloud Storage

Step-by-step: transfer data from Azure Blob Storage to Google Cloud Storage with Storage Transfer Service, including bucket setup and permissions.

#DataEngineering #DataEngineer #BigData
Step-by-Step Guide to Transfer Data from Azure Blob to Google Cloud Storage
Step-by-step: transfer data from Azure Blob Storage to Google Cloud Storage with Storage Transfer Service, including bucket setup and permissions.
ajayconnect.com
November 15, 2026 at 11:14 AM
The Missing Data Contract for Autonomous AI Systems
tech_blogs_acm | Author: Aditya Ranjan

#DataEngineering
The Missing Data Contract for Autonomous AI Systems
cacm.acm.org
November 15, 2026 at 4:39 AM
Full walkthrough with examples:
📖 medium.com/@sriwworldof...

Follow for more data engineering, and subscribe on YouTube. 🚀

#Redshift #AWS #DataEngineering #SQL #BigData
Redshift Distribution Styles Explained: KEY vs EVEN vs ALL (With Examples)
Learn Amazon Redshift distribution styles (KEY, EVEN, ALL, and AUTO) with simple analogies, SQL examples, common mistakes, and…
medium.com
November 15, 2026 at 3:05 AM