#databricks #datagravity #dataengineering #migration
#databricks #datagravity #dataengineering #migration
🎬 www.youtube.com/watch?v=W5bW...
📖 vyruss.org/blog/communi...
#Postgres #ApacheIceberg #DataEngineering #OpenSource
🎬 www.youtube.com/watch?v=W5bW...
📖 vyruss.org/blog/communi...
#Postgres #ApacheIceberg #DataEngineering #OpenSource
DE to AE means learning the business. AE to DE means learning the infrastructure.
#dataengineering #databs
DE to AE means learning the business. AE to DE means learning the infrastructure.
#dataengineering #databs
Your work is judged by whether the data arrives on time, every time, and by what happens when it doesn't.
#dataengineering #databs
Your work is judged by whether the data arrives on time, every time, and by what happens when it doesn't.
#dataengineering #databs
- Python: APIs, files, scripts that run on their own
- ingestion into the warehouse
- orchestration: Airflow, scheduling, retries
- cloud: storage, permissions, Docker, Terraform
- scale: partitioning, cost control, later Spark
#dataengineering
- Python: APIs, files, scripts that run on their own
- ingestion into the warehouse
- orchestration: Airflow, scheduling, retries
- cloud: storage, permissions, Docker, Terraform
- scale: partitioning, cost control, later Spark
#dataengineering
Both roles share:
- SQL, and a lot of it
- a cloud data warehouse
- Git, code reviews and testing
- dbt
#dataengineering #databs
Both roles share:
- SQL, and a lot of it
- a cloud data warehouse
- Git, code reviews and testing
- dbt
#dataengineering #databs
#Snowflake #SnowflakeSquad #Cortex #DataEngineering #AI
#Snowflake #SnowflakeSquad #Cortex #DataEngineering #AI
Read my complete article:
medium.com/@sriwworldof...
Follow for more AWS and Data Engineering insights.
#AWS #DataEngineering #BigData
Read my complete article:
medium.com/@sriwworldof...
Follow for more AWS and Data Engineering insights.
#AWS #DataEngineering #BigData
The problem might not be your SQL.
It could be your S3 partitioning and file format.
Learn how partition pruning + Parquet can transform query performance.
#AWS #DataEngineering
Read more:
medium.com/@sriwworldof...
The problem might not be your SQL.
It could be your S3 partitioning and file format.
Learn how partition pruning + Parquet can transform query performance.
#AWS #DataEngineering
Read more:
medium.com/@sriwworldof...
#datawarehouse #testdata #devops #dataengineering
#datawarehouse #testdata #devops #dataengineering
#databricks #deltalake #powerbi #dataengineering
#databricks #deltalake #powerbi #dataengineering
tech_blogs_databricks
#DataEngineering
tech_blogs_databricks
#DataEngineering
tech_blogs_databricks
#DataEngineering
tech_blogs_databricks
#DataEngineering
Recruteurs en France : cessez de payer un abonnement hors de prix quand vous pouvez avoir accès à France Travail ou a Apec - Association Pour l'Emploi des Cadres !
#Recrutement #LinkedIn #DataEngineering #Emploi #Travail ...
Recruteurs en France : cessez de payer un abonnement hors de prix quand vous pouvez avoir accès à France Travail ou a Apec - Association Pour l'Emploi des Cadres !
#Recrutement #LinkedIn #DataEngineering #Emploi #Travail ...
Add it because you can explain why each tool is there, and what you'd use instead in a smaller setup.
#dataengineering #databs
Add it because you can explain why each tool is there, and what you'd use instead in a smaller setup.
#dataengineering #databs
- Spark reads from Kafka and transforms the data as it arrives
- MongoDB stores the results as JSON documents, so you don't need a strict schema upfront
- Streamlit shows the data on a simple live dashboard
#dataengineering #databs
- Spark reads from Kafka and transforms the data as it arrives
- MongoDB stores the results as JSON documents, so you don't need a strict schema upfront
- Streamlit shows the data on a simple live dashboard
#dataengineering #databs
FastAPI takes in JSON events, Kafka holds them, Spark transforms them as they arrive, MongoDB stores the results and Streamlit shows them on a live dashboard. Docker Compose runs it all.
#dataengineering #databs
FastAPI takes in JSON events, Kafka holds them, Spark transforms them as they arrive, MongoDB stores the results and Streamlit shows them on a live dashboard. Docker Compose runs it all.
#dataengineering #databs
Follow for more #AWS #DataEngineering #AmazonAthena
Follow for more #AWS #DataEngineering #AmazonAthena
It’s a file layout problem.
Parquet + partitions can turn “scan 1 TB” into “scan a few GB.”
Same query. Fraction of the cost. 🧵
#AWS #DataEngineering
It’s a file layout problem.
Parquet + partitions can turn “scan 1 TB” into “scan a few GB.”
Same query. Fraction of the cost. 🧵
#AWS #DataEngineering
Step-by-step: transfer data from Azure Blob Storage to Google Cloud Storage with Storage Transfer Service, including bucket setup and permissions.
#DataEngineering #DataEngineer #BigData
Step-by-step: transfer data from Azure Blob Storage to Google Cloud Storage with Storage Transfer Service, including bucket setup and permissions.
#DataEngineering #DataEngineer #BigData
tech_blogs_acm | Author: Aditya Ranjan
#DataEngineering
tech_blogs_acm | Author: Aditya Ranjan
#DataEngineering
📖 medium.com/@sriwworldof...
Follow for more data engineering, and subscribe on YouTube. 🚀
#Redshift #AWS #DataEngineering #SQL #BigData
📖 medium.com/@sriwworldof...
Follow for more data engineering, and subscribe on YouTube. 🚀
#Redshift #AWS #DataEngineering #SQL #BigData