# Measuring the Lioran S3 Rust Storage Engine A benchmark that says: WRITE: 400 MB/s creates more questions than it answers. Where did time go? Network? SHA-256? Filesystem writes? `fsync`? Directory creation? Rename? RocksDB metadata? Lioran S3's normal PUT path contains explicit timing instrumentation for these stages. I’m **Swaraj Puppalwar** , Founder & CTO of **Lioran Group / Lioran Developer Solutions**. ## Enabling detailed PUT traces The engine checks: BASTION_TRACE_PUT_TIMINGS Accepted truthy forms include values such as: 1 true yes The setting is cached in an atomic so the environment is not reparsed on every request. ## Streaming timings `stream_to_staging` measures: recv_duration write_duration sha256_duration flush_duration fsync_duration total_stream_duration Then the outer PUT path adds: close_duration mkdir_duration rename_duration metadata_duration total_duration ## A conceptual trace A PUT can therefore be decomposed like: TOTAL PUT ├── receive from AsyncRead ├── SHA-256 ├── staging writes ├── flush ├── fsync ├── close ├── mkdir ├── rename └── RocksDB metadata commit This is exactly the decomposition you want when tuning a storage engine. ## Case 1: receive dominates Suppose: recv_ms 900 write_ms 100 sha256_ms 30 fsync_ms 20 metadata_ms 2 The server is mostly waiting for incoming bytes. Optimizing RocksDB will not magically fix that. Investigate: * client upload speed * network * TLS * reverse proxy buffering * request-body behavior ## Case 2: fsync dominates Suppose: recv_ms 100 write_ms 80 fsync_ms 700 Now durability synchronization is expensive. Questions include: * storage device latency * filesystem * virtualization * durability mode * write cache behavior ## Case 3: metadata dominates If: metadata_ms grows significantly under load, inspect the metadata path: * RocksDB compaction * WAL behavior * block cache * write stalls * disk contention * concurrent metadata operations ## Case 4: hashing matters SHA-256 is calculated incrementally on every incoming chunk. At very high throughput, checksum CPU cost can become measurable. That does not mean “remove checksums”. It means measure the actual cost before inventing a bottleneck. ## Aggregated metrics `LocalObjectStore` also owns: Arc<PutMetrics> and records successful and failed PUT performance. This lets the engine expose more than one request's anecdote. ## Why high-resolution timing matters Without stage timings, developers often optimize whatever subsystem they personally distrust. Database engineer: must be RocksDB Networking engineer: must be TLS Systems engineer: obviously fsync The stopwatch is less political. ## Benchmark contract When publishing Lioran S3 numbers, record: CPU RAM disk filesystem operating system container/native network topology reverse proxy TLS object size distribution concurrency durability mode software commit/version Otherwise two benchmark runs may not be measuring the same system. ## Optimization order A sane loop is: measure ↓ identify dominant stage ↓ change one thing ↓ measure again not: change 14 knobs ↓ number improved ↓ no idea why Lioran S3's timing instrumentation exists so performance work can stay evidence-driven.

# Failure Windows in Lioran S3 V1 Storage-engine design becomes interesting at the word: > crash. Happy-path code is only half the system. The current Lioran S3 V1 Pre-Alpha write path has a clear sequence of staging, promotion, metadata commit, and rollback. That sequence also defines its failure windows. I’m **Swaraj Puppalwar** , Founder & CTO of **Lioran Group** and **Lioran Developer Solutions**. This article describes the current implementation, including where future hardening can go deeper. ## Current normal PUT sequence The implementation today is: 1. validate bucket/key 2. check capacity/quota 3. create staging file 4. stream payload + SHA-256 5. flush 6. fsync in strict mode 7. close staging file 8. re-check capacity/quota 9. create final parent directory 10. rename staging → final object path 11. build ObjectMetadata 12. write metadata to RocksDB 13. if metadata write fails, remove final payload 14. return success The most important ordering detail is: physical promotion BEFORE metadata commit ## Why document the real ordering? An architecture document may describe an intended invariant at a higher level. The source code is what defines the current executable behavior. For pre-alpha infrastructure, publishing the actual order is more useful than pretending the engine has already reached its final crash-consistency model. ## Failure during streaming If reading from the incoming stream fails: staging file is removed metadata was never committed object is not visible as committed That is a clean failure case. ## Failure after streaming but before promotion Capacity/quota checks can still reject the upload after its exact size is known. The staging file is removed. Again, no committed metadata has been created. ## Rename failure If: fs::rename(staging, final) fails, the operation returns a storage error and attempts staging cleanup. Metadata has not yet been committed. ## Metadata write failure This case is more interesting. At that moment: physical payload has already been promoted metadata has not successfully committed The implementation responds by removing the physical payload: fs::remove_file(&abs_path) This is application-level rollback. ## Process crash between rename and metadata Now consider an uncontrolled process termination exactly after physical rename but before metadata commit. The rollback code does not execute because the process is gone. That can theoretically leave physical bytes without a corresponding metadata record. This is a classic cross-resource atomicity problem: filesystem namespace + embedded metadata database are not one shared transaction. ## Why this matters Normal reads begin from metadata. So an unreferenced physical file is not automatically exposed as a committed object. But it can become orphaned disk usage. That is different from exposing partial data, but it is still a recovery concern. ## The inverse ordering has its own problem Suppose metadata were committed first and the process died before physical promotion. Then metadata could point to missing physical bytes. That is usually worse for read correctness. This is why robust storage systems often introduce additional durable intent/state machinery instead of pretending two independent persistence domains can be made atomic with ordering alone. ## V1's current strategy The current code uses: * staging * strict/balanced durability * atomic filesystem rename semantics where supported * metadata status checks * rollback on synchronous metadata failure * logical/physical separation It does not magically turn filesystem + RocksDB into one atomic transaction. That would be an inaccurate claim. ## Future hardening direction A stronger recovery protocol could involve concepts such as: durable publish intent recovery journal startup reconciliation orphan scanning transaction state records idempotent repair Those are design directions, not claims about the current V1 implementation. ## Why pre-alpha is the right time to discuss this Pre-alpha should expose uncomfortable edge cases. That is the phase for: kill -9 power-loss simulation disk-full testing metadata failure injection rename failure injection orphan detection checksum verification restart loops The purpose is not to prove the code never fails. The purpose is to make every failure state defined and recoverable. ## Correctness before architecture theater Distributed replication would not remove this local commit problem. It could multiply it. That is why Lioran S3 intentionally focuses on single-node correctness before adding clustering and consensus layers. A storage engine becomes trustworthy when its failure states are boring, documented, testable, and repairable. That is the target.