This trick saved me many times: copy a table from website or spreadsheet, paste to a dataframe in r github.com/MilesMcBain...
November 9, 2026 at 2:45 PM
This trick saved me many times: copy a table from website or spreadsheet, paste to a dataframe in r github.com/MilesMcBain...
Zero-shot predictions in Pandas using the new BigQuery Accessor #googlecloud https://cloud.google.com/blog/products/data-analytics/bigquery-ai-functions-in-pandas/
BigQuery AI Functions in Pandas | Google Cloud Blog
Scale machine learning directly in Pandas and boost productivity using enterprise-grade BigQuery AI functions without leaving your DataFrame.
cloud.google.com
November 6, 2026 at 5:02 PM
Zero-shot predictions in Pandas using the new BigQuery Accessor #googlecloud https://cloud.google.com/blog/products/data-analytics/bigquery-ai-functions-in-pandas/
🧼 Cleaning your dataframe is a breeze with the skrub.Cleaner
In the real world problems, data arrives messy: "N/A" instead of real nulls, dates stored as strings, numbers hiding in text columns, columns that are entirely empty.
2/5
In the real world problems, data arrives messy: "N/A" instead of real nulls, dates stored as strings, numbers hiding in text columns, columns that are entirely empty.
2/5
November 5, 2026 at 2:03 PM
🧼 Cleaning your dataframe is a breeze with the skrub.Cleaner
In the real world problems, data arrives messy: "N/A" instead of real nulls, dates stored as strings, numbers hiding in text columns, columns that are entirely empty.
2/5
In the real world problems, data arrives messy: "N/A" instead of real nulls, dates stored as strings, numbers hiding in text columns, columns that are entirely empty.
2/5
The new @duckdb.org 2.0 is significantly faster for querying remote parquet files than 1.5.6 was, and also fixes a bug that necessitated some special settings to query the Parquet files in our S3 bucket directly.
The @pola.rs dataframe library is also a great way to query this data efficiently.
The @pola.rs dataframe library is also a great way to query this data efficiently.
November 3, 2026 at 2:56 AM
The new @duckdb.org 2.0 is significantly faster for querying remote parquet files than 1.5.6 was, and also fixes a bug that necessitated some special settings to query the Parquet files in our S3 bucket directly.
The @pola.rs dataframe library is also a great way to query this data efficiently.
The @pola.rs dataframe library is also a great way to query this data efficiently.
To be clear up front I don’t know much about SAS, but honestly ime most cases don’t benefit enough from columnar storage to be worth having to use most Parquet tooling.
If columnar storage is really good for what you’re doing, use pola.rs to interact with the data it’s great
If columnar storage is really good for what you’re doing, use pola.rs to interact with the data it’s great
Polars
Polars is a high-performance DataFrame library for Python and Rust. Built on Apache Arrow, it's the fastest way to work with tabular data at any scale.
pola.rs
November 2, 2026 at 4:37 PM
To be clear up front I don’t know much about SAS, but honestly ime most cases don’t benefit enough from columnar storage to be worth having to use most Parquet tooling.
If columnar storage is really good for what you’re doing, use pola.rs to interact with the data it’s great
If columnar storage is really good for what you’re doing, use pola.rs to interact with the data it’s great
a dataframe is a list where all of the elements (columns) have the same length (number of rows) and lists can contain lists so it’s coherent. I think the fussiness of list columns keeps ordinary users from using them often though
October 31, 2026 at 3:47 PM
a dataframe is a list where all of the elements (columns) have the same length (number of rows) and lists can contain lists so it’s coherent. I think the fussiness of list columns keeps ordinary users from using them often though
I get what you mean, but for Pew the "data" tab is just the dataframe you can use to recreate the plot. They don't provide the raw data (it's CPS), but the technical appendix usually gives a good idea on how to replicate their analyses with the raw data.
November 2, 2026 at 12:23 AM
I get what you mean, but for Pew the "data" tab is just the dataframe you can use to recreate the plot. They don't provide the raw data (it's CPS), but the technical appendix usually gives a good idea on how to replicate their analyses with the raw data.
282 lines of pandas and scipy that profile a CSV and flag outliers by z-score, then writes an HTML report. Intermediate level, built for ETL and log data checks. #python #pandas #dataengineering https://www.valtersit.com/python/pandas-dataframe-profiler-and-outlier-detector/
Pandas DataFrame Profiler and Outlier Detector
www.valtersit.com
October 31, 2026 at 8:50 AM
282 lines of pandas and scipy that profile a CSV and flag outliers by z-score, then writes an HTML report. Intermediate level, built for ETL and log data checks. #python #pandas #dataengineering https://www.valtersit.com/python/pandas-dataframe-profiler-and-outlier-detector/
Kotlin DataFrame 1.0.0-rc02 is out — the final release candidate before 1.0. Try it in a real project and help us validate the release 🔎 https://kotl.in/1l0urw
October 28, 2026 at 1:39 PM
Kotlin DataFrame 1.0.0-rc02 is out — the final release candidate before 1.0. Try it in a real project and help us validate the release 🔎 https://kotl.in/1l0urw
My overall advice here: automate error checking first. You want this component to be absolutely deterministic even if you're using code from an LLM. Then you need to figure out which parts of your work are dataframe-shaped and which are tree-shaped
October 24, 2026 at 11:01 PM
My overall advice here: automate error checking first. You want this component to be absolutely deterministic even if you're using code from an LLM. Then you need to figure out which parts of your work are dataframe-shaped and which are tree-shaped
Databricks: DataFrame caching on serverless compute now in Public Preview
October 25, 2026 at 3:15 AM
Databricks: DataFrame caching on serverless compute now in Public Preview
The source, dated, on The Deploy Log:
https://deployedbyai.com/log/2e54786c4653/dataframe-caching-on-serverless-compute/?k=br
https://deployedbyai.com/log/2e54786c4653/dataframe-caching-on-serverless-compute/?k=br
DataFrame caching on serverless compute
Databricks, 23 Oct 2026. Sourced to docs.databricks.com.
deployedbyai.com
October 25, 2026 at 3:15 AM
The source, dated, on The Deploy Log:
https://deployedbyai.com/log/2e54786c4653/dataframe-caching-on-serverless-compute/?k=br
https://deployedbyai.com/log/2e54786c4653/dataframe-caching-on-serverless-compute/?k=br
Ever tried to run a scikit-learn pipeline on a dataframe with mixed types?
You need one encoder for categorical columns, another for text, a third for dates... it gets verbose fast!
skrub.TableVectorizer is the "fit once, vectorize any table" transformer.
skrub-data.org/stable/refer...
You need one encoder for categorical columns, another for text, a third for dates... it gets verbose fast!
skrub.TableVectorizer is the "fit once, vectorize any table" transformer.
skrub-data.org/stable/refer...
October 21, 2026 at 3:32 PM
Ever tried to run a scikit-learn pipeline on a dataframe with mixed types?
You need one encoder for categorical columns, another for text, a third for dates... it gets verbose fast!
skrub.TableVectorizer is the "fit once, vectorize any table" transformer.
skrub-data.org/stable/refer...
You need one encoder for categorical columns, another for text, a third for dates... it gets verbose fast!
skrub.TableVectorizer is the "fit once, vectorize any table" transformer.
skrub-data.org/stable/refer...
So does map_batches work just like map? Eg it passes am chunked object of some kind that you can treat as dataframe? Can it take largee then memory inputs?
October 20, 2026 at 8:52 PM
So does map_batches work just like map? Eg it passes am chunked object of some kind that you can treat as dataframe? Can it take largee then memory inputs?
They didn't send us stickers with our mail-in ballots, so Patch had to improvise. She'll need to take that sticker back off her DataFrame to get any more work done, though.
October 19, 2026 at 10:22 PM
They didn't send us stickers with our mail-in ballots, so Patch had to improvise. She'll need to take that sticker back off her DataFrame to get any more work done, though.
Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation
Jingjing (May) et al.
#arXiv #cs.AI #cs.SE
Jingjing (May) et al.
#arXiv #cs.AI #cs.SE
Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation
Planning and revision are intended to bring generated code closer to the task specification. Yet a mistaken interpretation can persist across the plan, implementation, and tests - a pattern we call interpretive lock-in. We investigate this risk on 30 DataFrame tasks derived from TPC Benchmark DS, c…
arxiv.org
October 15, 2026 at 8:55 PM
Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation
Jingjing (May) et al.
#arXiv #cs.AI #cs.SE
Jingjing (May) et al.
#arXiv #cs.AI #cs.SE
Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation
Planning and revision are intended to bring generated code closer to the task specification. Yet a mistaken interpretation can persist across the plan, implementation, and tests - a pattern we call interpretive lock-in. We investigate this risk on 30 DataFrame tasks derived from TPC Benchmark DS, c…
arxiv.org
October 15, 2026 at 10:31 PM
A new AI review! pandas-dev/pandas ⭐4.5/5.0
pandas is a flagship Python library for tabular and labeled data manipulation, providing the de facto DataFrame/Series abstractions used across data science, analytics, and ETL workflows.
https://gitrated.com/pandas-dev/pandas
pandas is a flagship Python library for tabular and labeled data manipulation, providing the de facto DataFrame/Series abstractions used across data science, analytics, and ETL workflows.
https://gitrated.com/pandas-dev/pandas
October 16, 2026 at 8:46 AM
A new AI review! pandas-dev/pandas ⭐4.5/5.0
pandas is a flagship Python library for tabular and labeled data manipulation, providing the de facto DataFrame/Series abstractions used across data science, analytics, and ETL workflows.
https://gitrated.com/pandas-dev/pandas
pandas is a flagship Python library for tabular and labeled data manipulation, providing the de facto DataFrame/Series abstractions used across data science, analytics, and ETL workflows.
https://gitrated.com/pandas-dev/pandas
Excited to announce Apache SedonaDB 0.5.0! In this release is an improved interface for reading various file formats, DataFrame unnest()ing, improved raster functionality, an improved GeoPandas compatibility layer, ever more spatial functions, and the seeds of foreign catalog integration.
October 14, 2026 at 10:44 PM
Excited to announce Apache SedonaDB 0.5.0! In this release is an improved interface for reading various file formats, DataFrame unnest()ing, improved raster functionality, an improved GeoPandas compatibility layer, ever more spatial functions, and the seeds of foreign catalog integration.
Jingjing (May), Liu, Dishu Yang, Hongbo Qin, Hansong Zhang: Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation https://arxiv.org/abs/2610.15797 https://arxiv.org/pdf/2610.15797 https://arxiv.org/html/2610.15797
October 15, 2026 at 6:41 AM
Jingjing (May), Liu, Dishu Yang, Hongbo Qin, Hansong Zhang: Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation https://arxiv.org/abs/2610.15797 https://arxiv.org/pdf/2610.15797 https://arxiv.org/html/2610.15797
Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation
Jingjing (May) et al.
#arXiv #cs.AI #cs.SE
Jingjing (May) et al.
#arXiv #cs.AI #cs.SE
Passing Tests, Missing Semantics: Evidence of Interpretive Lock-In in Plan-Based DataFrame Code Generation
Planning and revision are intended to bring generated code closer to the task specification. Yet a mistaken interpretation can persist across the plan, implementation, and tests - a pattern we call interpretive lock-in. We investigate this risk on 30 DataFrame tasks derived from TPC Benchmark DS, c…
arxiv.org
October 14, 2026 at 9:04 PM