#duckdb #pyiceberg #apachepolaris #MicrosoftFabric #Onelake #deltatable #PowerBI

I can do anything I want 😁

a sql script that export #deltatable metadata from a #ducklkake db

never thought that #sql can do that !!!!

github.com/djouallah/Fa...

Explaining how Python engines read and write #DeltaTable is not for the faint of heart.
The theory is everything will depends on the delta kernet rust for read and write, but we are not there yet
github.com/djouallah/Fa...
#duckdb #delta_rs #datafusion #chdb #daft #polars #rust #lakesail

I don't think it was publicized a lot, but #Snowflake can read #DeltaTable as an #Iceberg table too, basically in this example, I am writing to #onelake and reading existing data too, generated by #Spark

How to Read a #DeltaTable with Deletion Vectors and Column Mapping in #Python

datamonkeysite.com/2025/03/19/h...
#Microsoftfabric #onelake #duckdb #deltatlake

Reading #deltatable from #Unitycatalog using #daft, it is read only for now.

running in #MicrosoftFabric Notebook
it is already supported using Mirrored Unity catalog in #onelake but it is nice to have more options, specially thinking of write support.

github.com/djouallah/Fa...

We can take advantage of query optimizations when scanning a DeltaTable with Polars. For example, here we want to filter on a DeltaTable partitioned on the date column. Compare this to the awkward syntax we need when using Pandas below!

When we scan a DeltaTable Polars now uses its native Parquet reader.

This means we can use the Polars streaming engine to process tables that are larger-than-memory. This lets us work with huge datasets with Polars and deltalake, all by passing streaming=True (and a new streaming engine is coming)

POC, export #ducklake metadata to #deltatable format, #apacheSpark and Fabric DWH, #PowerBI can read it
github.com/djouallah/Fa...

using #deltatable Python packages to detect changes downstream, and process data either incrementally or full refresh
#Python #MicrosoftFabric #onelake
datamonkeysite.com/2024/11/10/s...

Tested the new Polars streaming engine loading a partitioned deltatable on S3. Was 2x faster than with .collect() on a not-larger-than-memory table. The new engine isn't just about faster compute or larger datasets - data transfer is fast and very consistent. Looks v promising

🚀 Learn how to build and schedule a demand forecast with SAP Databricks in this hands-on tutorial by Andreas Forster — a practical intro to Data Science on SAP Business Data Cloud. 👉 sap.to/6008fNUXG

Next step will read from #S3 and flatten the JSON structure and merges it with #schemaevolution to the property #deltatable.

As a first step I'm scraping properties with the defined city, radius, buy or rent, and propertyType. Checking in existing #deltatable if #fingerprint already exists, and only if not, the property will be cached and uploaded to #S3.

A lot of my recent focus has been on making pipelines more observable. No fancy tools, just keeping a list that tracks the status of each iteration then printed in an easy way to read at the end. Then make the summary into a dataframe and append to a deltatable for easy access

If you regularly appending files to a deltalake table then the transaction log gets slow to parse as it accumulates thousands of JSON files. Use add_checkpoint to compress them into a single parquet file.

For me this took a 30 second DeltaTable creation down to 0.5 seconds