I had fun giving a talk at #PyConUS earlier today, "The PyArrow Revolution in #Pandas." I talked about what Arrow and PyArrow are, how we can already use PyArrow in Pandas, and where things are going with its integration.
Check out the slides: speakerdeck.com/reuven/the-p...
Check out the slides: speakerdeck.com/reuven/the-p...
The PyArrow revolution in Pandas
Slides from my talk at PyCon US 2025, about PyArrow -- what it is, how it is related to Pandas, how we can use it now, and how we'll use it in the future.
speakerdeck.com
May 16, 2025 at 10:38 PM
I had fun giving a talk at #PyConUS earlier today, "The PyArrow Revolution in #Pandas." I talked about what Arrow and PyArrow are, how we can already use PyArrow in Pandas, and where things are going with its integration.
Check out the slides: speakerdeck.com/reuven/the-p...
Check out the slides: speakerdeck.com/reuven/the-p...
Yay!
I got both a tutorial ("Comprehending comprehensions") and a talk ("The PyArrow revolution in Pandas") accepted at PyCon US.
See you in Pittsburgh in May!
I got both a tutorial ("Comprehending comprehensions") and a talk ("The PyArrow revolution in Pandas") accepted at PyCon US.
See you in Pittsburgh in May!
February 8, 2025 at 4:58 PM
Yay!
I got both a tutorial ("Comprehending comprehensions") and a talk ("The PyArrow revolution in Pandas") accepted at PyCon US.
See you in Pittsburgh in May!
I got both a tutorial ("Comprehending comprehensions") and a talk ("The PyArrow revolution in Pandas") accepted at PyCon US.
See you in Pittsburgh in May!
Latest Talk Python episode with guest Reuven Lerner and host @mkennedy.codes is out. Join the PyArrow revolution with us! #python
#503: The PyArrow Revolution
talkpython.fm/episodes/sho...
#503: The PyArrow Revolution
talkpython.fm/episodes/sho...
The PyArrow Revolution
Pandas is at a the core of virtually all data science done in Python, that is virtually all data science. Since it's beginning, Pandas has been based upon numpy. But changes are afoot to update those ...
talkpython.fm
April 28, 2025 at 5:39 PM
Latest Talk Python episode with guest Reuven Lerner and host @mkennedy.codes is out. Join the PyArrow revolution with us! #python
#503: The PyArrow Revolution
talkpython.fm/episodes/sho...
#503: The PyArrow Revolution
talkpython.fm/episodes/sho...
Reading a CSV file into a #Python #Pandas data frame? Speed things up by specifying the PyArrow engine. Data storage isn't affected.
df = pd.read_csv(filename)
df = pd.read_csv(filename, engine='pyarrow')
With a 2.2GB file, it took 4s vs. 55s -- more than 10x faster!
df = pd.read_csv(filename)
df = pd.read_csv(filename, engine='pyarrow')
With a 2.2GB file, it took 4s vs. 55s -- more than 10x faster!
February 19, 2026 at 4:31 PM
Reading a CSV file into a #Python #Pandas data frame? Speed things up by specifying the PyArrow engine. Data storage isn't affected.
df = pd.read_csv(filename)
df = pd.read_csv(filename, engine='pyarrow')
With a 2.2GB file, it took 4s vs. 55s -- more than 10x faster!
df = pd.read_csv(filename)
df = pd.read_csv(filename, engine='pyarrow')
With a 2.2GB file, it took 4s vs. 55s -- more than 10x faster!
The video from my talk at #PyConUS, "The PyArrow Revolution in Pandas," is up!
Take a look, and find out about the connection between #PyArrow and #Pandas, what this means for you today, and what it means moving forward: youtu.be/Cpr0wbopYvU?...
Take a look, and find out about the connection between #PyArrow and #Pandas, what this means for you today, and what it means moving forward: youtu.be/Cpr0wbopYvU?...
The PyArrow revolution in Pandas
YouTube video by PyCon US
youtu.be
May 22, 2025 at 11:10 PM
The video from my talk at #PyConUS, "The PyArrow Revolution in Pandas," is up!
Take a look, and find out about the connection between #PyArrow and #Pandas, what this means for you today, and what it means moving forward: youtu.be/Cpr0wbopYvU?...
Take a look, and find out about the connection between #PyArrow and #Pandas, what this means for you today, and what it means moving forward: youtu.be/Cpr0wbopYvU?...
Pyodide 0.27 has been released 🎉
This release includes some new data science and data engineering packages: PyArrow, @pola.rs, @duckdb.org. This means you can play around with these packages in your browser.
Read everything here: blog.pyodide.org/posts/0.27-r...
This release includes some new data science and data engineering packages: PyArrow, @pola.rs, @duckdb.org. This means you can play around with these packages in your browser.
Read everything here: blog.pyodide.org/posts/0.27-r...
blog.pyodide.org
January 3, 2025 at 4:08 PM
Pyodide 0.27 has been released 🎉
This release includes some new data science and data engineering packages: PyArrow, @pola.rs, @duckdb.org. This means you can play around with these packages in your browser.
Read everything here: blog.pyodide.org/posts/0.27-r...
This release includes some new data science and data engineering packages: PyArrow, @pola.rs, @duckdb.org. This means you can play around with these packages in your browser.
Read everything here: blog.pyodide.org/posts/0.27-r...
When the Delta Lake Python bindings take a required dependency on your library 😅. 2.6M downloads a month! 😱 Guess I'll never be making a breaking release now!! pypistats.org/packages/arr...
arro3-core is 15x smaller in your environment than pyarrow! kylebarron.dev/arro3/latest/
arro3-core is 15x smaller in your environment than pyarrow! kylebarron.dev/arro3/latest/
July 12, 2025 at 5:06 PM
When the Delta Lake Python bindings take a required dependency on your library 😅. 2.6M downloads a month! 😱 Guess I'll never be making a breaking release now!! pypistats.org/packages/arr...
arro3-core is 15x smaller in your environment than pyarrow! kylebarron.dev/arro3/latest/
arro3-core is 15x smaller in your environment than pyarrow! kylebarron.dev/arro3/latest/
✨ New Narwhals expression: `nw.corr`
👥 Supported in both group-by and window contexts
🌐 Polars, DuckDB, PyArrow, PySpark, pandas...all supported
🗒️ PS. please don't draw conclusions about ice cream sales and drownings from this post
👥 Supported in both group-by and window contexts
🌐 Polars, DuckDB, PyArrow, PySpark, pandas...all supported
🗒️ PS. please don't draw conclusions about ice cream sales and drownings from this post
April 9, 2026 at 5:26 PM
✨ New Narwhals expression: `nw.corr`
👥 Supported in both group-by and window contexts
🌐 Polars, DuckDB, PyArrow, PySpark, pandas...all supported
🗒️ PS. please don't draw conclusions about ice cream sales and drownings from this post
👥 Supported in both group-by and window contexts
🌐 Polars, DuckDB, PyArrow, PySpark, pandas...all supported
🗒️ PS. please don't draw conclusions about ice cream sales and drownings from this post
Marco Gorelli's talk on unifying Polars, DuckDB, PyArrow & pandas with Narwhals is #10 on the year's most-watched Python talks! 🚀
buff.ly/GbLlN53
If you're working with dataframes across different backends, this one's a must-watch.
See what else the community is learning from: buff.ly/8p7RXz2
buff.ly/GbLlN53
If you're working with dataframes across different backends, this one's a must-watch.
See what else the community is learning from: buff.ly/8p7RXz2
Marco Gorelli - How Narwhals brings Polars, DuckDB, PyArrow, & pandas together | PyData London 25
www.pydata.org
Polars, DuckDB, PySpark, PyArrow, pandas, cuDF: how Narwhals has brought them all together!
Suppose you want to write a data science tool to do feature engineering. Your…
buff.ly
December 2, 2025 at 4:01 PM
Marco Gorelli's talk on unifying Polars, DuckDB, PyArrow & pandas with Narwhals is #10 on the year's most-watched Python talks! 🚀
buff.ly/GbLlN53
If you're working with dataframes across different backends, this one's a must-watch.
See what else the community is learning from: buff.ly/8p7RXz2
buff.ly/GbLlN53
If you're working with dataframes across different backends, this one's a must-watch.
See what else the community is learning from: buff.ly/8p7RXz2
lancedb / lance: Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with m ... ★5333 https://github.com/lancedb/lance
lancedb / lance
Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
github.com
September 11, 2025 at 4:21 PM
lancedb / lance: Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with m ... ★5333 https://github.com/lancedb/lance
I've seen a lot of Python folks getting super excited about `uv run --with ....` and I get it now. In @conda.org world, you can do the same with @prefix.dev Pixi!
A colleague wanted me to debug something with PyArrow on my system and all I had to do was:
```
$ pixi exec -s pyarrow python
```
🤯
A colleague wanted me to debug something with PyArrow on my system and all I had to do was:
```
$ pixi exec -s pyarrow python
```
🤯
February 7, 2025 at 9:23 AM
I've seen a lot of Python folks getting super excited about `uv run --with ....` and I get it now. In @conda.org world, you can do the same with @prefix.dev Pixi!
A colleague wanted me to debug something with PyArrow on my system and all I had to do was:
```
$ pixi exec -s pyarrow python
```
🤯
A colleague wanted me to debug something with PyArrow on my system and all I had to do was:
```
$ pixi exec -s pyarrow python
```
🤯
lancedb / lance: Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with m ... ★5158 https://github.com/lancedb/lance
lancedb / lance
Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with more integrations coming..
github.com
August 5, 2025 at 1:05 PM
lancedb / lance: Modern columnar data format for ML and LLMs implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, and PyTorch with m ... ★5158 https://github.com/lancedb/lance
🎉 So excited that @marcogorelli.bsky.social will be speaking at #PyConDE #PyData2025! He'll showcase how Narwhals is revolutionizing the dataframe ecosystem by bridging pandas, Polars, DuckDB, and PyArrow. Don't miss this talk! https://buff.ly/43dB6nY
February 26, 2025 at 12:00 AM
🎉 So excited that @marcogorelli.bsky.social will be speaking at #PyConDE #PyData2025! He'll showcase how Narwhals is revolutionizing the dataframe ecosystem by bridging pandas, Polars, DuckDB, and PyArrow. Don't miss this talk! https://buff.ly/43dB6nY
Pandas 3.0 will significantly boost performance by replacing NumPy with PyArrow as its default engine, enabling faster loading and reading of columnar data.
Python Pandas Ditches NumPy for Speedier PyArrow
Pandas 3.0 will significantly boost performance by replacing NumPy with PyArrow as its default engine, enabling faster loading and reading of columnar data.
bit.ly
May 30, 2025 at 1:00 PM
Pandas 3.0 will significantly boost performance by replacing NumPy with PyArrow as its default engine, enabling faster loading and reading of columnar data.
What if your library could support Pandas, Polars, PyArrow, and DuckDB without choosing?
Marco Gorelli walks through how Narwhals enables this and why the ecosystem benefits when tools are interoperable.
Excellent PyData Berlin session:
Marco Gorelli walks through how Narwhals enables this and why the ecosystem benefits when tools are interoperable.
Excellent PyData Berlin session:
Narwhals: enabling universal dataframe support
🔊 Recorded at PyData Berlin 2025,
https://2025.pycon.de/program/JKEHMH/
🎓 How Narwhals enables seamless dataframe interoperability across Pandas, Polars, PyArrow, DuckDB & more—with zero…
buff.ly
November 24, 2025 at 7:07 PM
What if your library could support Pandas, Polars, PyArrow, and DuckDB without choosing?
Marco Gorelli walks through how Narwhals enables this and why the ecosystem benefits when tools are interoperable.
Excellent PyData Berlin session:
Marco Gorelli walks through how Narwhals enables this and why the ecosystem benefits when tools are interoperable.
Excellent PyData Berlin session:
There were some changes in the latest h3ronpy version to remove pyarrow as a required depdendency. Let me know if you have any issues updating!
November 27, 2024 at 8:48 PM
There were some changes in the latest h3ronpy version to remove pyarrow as a required depdendency. Let me know if you have any issues updating!
Currently taking a look at refreshing some of the #ApacheArrow and #PyArrow docs, so if you use Arrow in #rstats or Python and there's any areas you'd like to understand better, give me a shout, and we'll see what we can do!
May 7, 2025 at 3:47 PM
Currently taking a look at refreshing some of the #ApacheArrow and #PyArrow docs, so if you use Arrow in #rstats or Python and there's any areas you'd like to understand better, give me a shout, and we'll see what we can do!
🎢 When the Q&A goes off-script 🤣
🇯🇵 Japanese math rock, 🅿️ porpoises and unfortunate aliases, and 🎤 Taylor Swift:
🥈All of these unexpectedly came up in this fun session, which is currently the 2nd most viewed of the conference
🙏 Thanks @pydatalondon.bsky.social for inviting me!
🇯🇵 Japanese math rock, 🅿️ porpoises and unfortunate aliases, and 🎤 Taylor Swift:
🥈All of these unexpectedly came up in this fun session, which is currently the 2nd most viewed of the conference
🙏 Thanks @pydatalondon.bsky.social for inviting me!
Marco Gorelli - How Narwhals brings Polars, DuckDB, PyArrow, & pandas together | PyData London 25
YouTube video by PyData
youtu.be
July 9, 2025 at 2:41 PM
🎢 When the Q&A goes off-script 🤣
🇯🇵 Japanese math rock, 🅿️ porpoises and unfortunate aliases, and 🎤 Taylor Swift:
🥈All of these unexpectedly came up in this fun session, which is currently the 2nd most viewed of the conference
🙏 Thanks @pydatalondon.bsky.social for inviting me!
🇯🇵 Japanese math rock, 🅿️ porpoises and unfortunate aliases, and 🎤 Taylor Swift:
🥈All of these unexpectedly came up in this fun session, which is currently the 2nd most viewed of the conference
🙏 Thanks @pydatalondon.bsky.social for inviting me!
✨✨ 2 big Narwhalified releases this week!
🎯 For time series enthusiasts, Darts now supports @pola.rs / PyArrow / cuDF / Modin via Narwhals
🎨 For visualisation enthusiasts, same story for @bokeh.org
🦄🌊 Underwater unicorn magic
🎯 For time series enthusiasts, Darts now supports @pola.rs / PyArrow / cuDF / Modin via Narwhals
🎨 For visualisation enthusiasts, same story for @bokeh.org
🦄🌊 Underwater unicorn magic
March 14, 2025 at 4:16 PM
✨✨ 2 big Narwhalified releases this week!
🎯 For time series enthusiasts, Darts now supports @pola.rs / PyArrow / cuDF / Modin via Narwhals
🎨 For visualisation enthusiasts, same story for @bokeh.org
🦄🌊 Underwater unicorn magic
🎯 For time series enthusiasts, Darts now supports @pola.rs / PyArrow / cuDF / Modin via Narwhals
🎨 For visualisation enthusiasts, same story for @bokeh.org
🦄🌊 Underwater unicorn magic
Want PyArrow dtypes in your #Python #Pandas data frame?
df = pd.read_csv(filename, dtype_backend='pyarrow')
The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones.
Note: This is still experimental... but it's also the future.
df = pd.read_csv(filename, dtype_backend='pyarrow')
The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones.
Note: This is still experimental... but it's also the future.
August 19, 2026 at 3:30 PM
Want PyArrow dtypes in your #Python #Pandas data frame?
df = pd.read_csv(filename, dtype_backend='pyarrow')
The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones.
Note: This is still experimental... but it's also the future.
df = pd.read_csv(filename, dtype_backend='pyarrow')
The dtypes are double[pyarrow], int64[pyarrow], and string[pyarrow], not the normal NumPy ones.
Note: This is still experimental... but it's also the future.
Reading a CSV into #Python #Pandas, and want a column to be treated as datetime values? Use parse_dates:
df = pd.read_csv(filename,
parse_dates=['x', 'y'])
Bonus: The PyArrow engine often (not always) parses columns that look like dates.
df = pd.read_csv(filename,
parse_dates=['x', 'y'])
Bonus: The PyArrow engine often (not always) parses columns that look like dates.
April 14, 2026 at 3:30 PM
Reading a CSV into #Python #Pandas, and want a column to be treated as datetime values? Use parse_dates:
df = pd.read_csv(filename,
parse_dates=['x', 'y'])
Bonus: The PyArrow engine often (not always) parses columns that look like dates.
df = pd.read_csv(filename,
parse_dates=['x', 'y'])
Bonus: The PyArrow engine often (not always) parses columns that look like dates.
🙀 Bokeh merged a pull request to use Narwhals for Polars / PyArrow support!
3️⃣.7️⃣ This will be available in their 3.7 release, probably later this month
🎢 Exciting times for Python dataframe visualisation!
3️⃣.7️⃣ This will be available in their 3.7 release, probably later this month
🎢 Exciting times for Python dataframe visualisation!
February 7, 2025 at 2:17 PM
🙀 Bokeh merged a pull request to use Narwhals for Polars / PyArrow support!
3️⃣.7️⃣ This will be available in their 3.7 release, probably later this month
🎢 Exciting times for Python dataframe visualisation!
3️⃣.7️⃣ This will be available in their 3.7 release, probably later this month
🎢 Exciting times for Python dataframe visualisation!
This means you can modify a file and transmit only the changes to the server, eliminating the need to re-upload the entire file.
CDC Parquet writer is out in PyArrow nightlies .
pip install \
-i pypi.anaconda.org/scientific-p... \
"pyarrow>=21.0.0.dev0"
CDC Parquet writer is out in PyArrow nightlies .
pip install \
-i pypi.anaconda.org/scientific-p... \
"pyarrow>=21.0.0.dev0"
Simple Index
pypi.anaconda.org
May 21, 2025 at 3:52 AM
This means you can modify a file and transmit only the changes to the server, eliminating the need to re-upload the entire file.
CDC Parquet writer is out in PyArrow nightlies .
pip install \
-i pypi.anaconda.org/scientific-p... \
"pyarrow>=21.0.0.dev0"
CDC Parquet writer is out in PyArrow nightlies .
pip install \
-i pypi.anaconda.org/scientific-p... \
"pyarrow>=21.0.0.dev0"