GitHub's new Git storage architecture is built so thousands of agents can push to a single repository at once. Here's how it works.

The file systems supported by Windows use the concept of files and directories to access data stored on a disk or device.

Ab Windows 10, Version 1607, wurden MAX_PATH Einschränkun...

Yeah, these are good suggestions, thanks for the input! I do already warmup the file by writing `0`s to the entire file when it's first opened. This definitely speeds up fileIO on small files that can fit entirely in memory but once I start to touch multiple pages it quickly decreases in it's improvement. I think it's feasible I could be getting bottlenecked by the page-table and page walking of the CPU as well. In general I'm worried that seeking around and caching so many pages is unavoidably slow. Currently I'm doing the writes using the `write_at` call, which avoids a separate `seek` and subsequent `write` of two syscalls and also allows it to be parallelized. I don't think the `write_vec` would work since I don't remember (in the RAM sense) the data in between where I'm writing to (the `o`s in the example). For sufficiently large files that could be an order of a gig or arbitrarily large that I'm seeking past and would be too large to keep around and probably too large to buffer in from reading. I mention it's parallelizable because I previously tried doing all my work in one async thread and sending the data to be written to a different thread in a tokio runtime but the subsequent awaits on the file write were additional overhead that I didn't need and a **single thread** was faster than that. So, currently, I am using just one thread for both the transpose work and writing all around the file. However, I haven't tried having one dedicated thread for each transposed column to be writing to with its own syscall (or, at least having one thread for every couple of transposed columns to avoid having 2^30 threads but maybe 2^10). One other alternative is to have a separate file for each column that I could decompose rows into and concatenate them all together at the end of the process. This would still have the issue of writing small bits of data across a large number of pages but the OS might be able to better cache each one since I'm writing sequentially. Will report back with any findings. Thanks again for the input!

Apache Iceberg 1.5.0 was released on March 11, 2024. The 1.5.0 release adds a variety of new features and bug fixes. API Extend FileIO and add EncryptingFileIO. (#9592) Track partition statistics...