#Teaser #CoverReveal #Preorder 🏍️𝑪𝑶𝑽𝑬𝑹 + 𝑩𝑳𝑼𝑹𝑩 𝑹𝑬𝑽𝑬𝑨𝑳 🏍️ 𝑹𝑰𝑫𝑰𝑵𝑮 𝑩𝑨𝑹𝑬: 𝑨 𝑭𝒂𝒌𝒆 𝑹𝒆𝒍𝒂𝒕𝒊𝒐𝒏𝒔𝒉𝒊...

The Black Rasslin' Podcast · Episode

jrose: > Because it was not obvious to me the first time I encountered this problem, the requirement of `K: Hash` is _not_ on the HashMap type in Rust; it's on the impl block containing operations that hash. After thinking about this for ages, I've realised that the problem isn't really a problem with respect to types like `HashMap` for which the implemented trait is not `unsafe`. Not including `K: Hash` on the `HashMap` type itself means that it's impossible for external-impl proposals to ensure that `HashMap` is coherent in the sense of always hashing the same keys to the same values (because a different `Hash` impl could be used for each call); but even in current Rust, `HashMap` has numerous ways in which a programmer could cause incoherent hashing in a way that won't be caught by the compiler: * (edit: I corrected this in response to pitaj's correction) `Hash` is a safe trait, so nothing forces `K: Hash` and ~~`&K: Hash`~~ `Q: Hash` (where `Q: Borrow<K>` is used to do hash lookups) to have any relationship to each other, but it is possible to add keys to a hash as `K` and then retrieve them using a `Q`; * `Hash` is a safe trait, so nothing is forcing it to always hash the same key to the same hash (it could, e.g., return a random number) * Even if `Hash` is deterministic, it is possible to use interior mutability to modify the keys stored in a hash table (while they are being stored) in order to change the value they would hash to. `HashMap`'s documentation contains plenty of warnings against doing this sort of thing, but it is just warnings in the documentation (and in particular its memory safety is not allowed to depend on the programmer doing this). As such, a change to Rust to allow external impls wouldn't actually break `HashMap` in any ways that it isn't already broken: it would allow people to try to retrieve keys using a different hashing function from the one used to add them (which of course doesn't work), but there are plenty of ways to do that even without external impls. In general, it seems impossible for incoherent implementations of a safe trait to break existing code, as long as all the trait's associated items are methods (i.e. it has no associated types (including RPITIT as a special case of associated types) or associated constants). That's because code like this, which is the case that breaks (stealing the syntax from the OP): let g: Generic<T> = Generic::new(); (&g as &Generic<T + Trait use Impl1>).method(); (&g as &Generic<T + Trait use Impl2>).method(); could be replicated without external impls by doing something like this: thread_local! { static USE_IMPL_2: Cell<bool> = const { Cell::new(false); }; } impl Trait for T { // for each method in the trait, you write a small wrapper like this fn f(&self) { if USE_IMPL_2.get() { self.f2(); } else { self.f1(); } } } let g: Generic<T> = Generic::new(); USE_IMPL_2.set(false); g.method(); USE_IMPL_2.set(true); g.method(); This specifically only works for methods, because associated types or constants can't depend on the value of a global variable. (It also doesn't work for `unsafe` traits, because they may have safety requirements that prohibit this sort of implementation.) It strikes me that this is very similar for the rules of `dyn` compatibility (the only exception is that `dyn` compatibility requires the ability to make a vtable, whereas this doesn't), which is probably not entirely a coincidence. All this is making me think that external impls should probably be designed so that putting a bound on the type itself (as opposed to the `impl` block) is the correct way to say "this trait must always be implemented the same way whenever a method of this type is called": for safe, `dyn`-compatible traits, this seems to be entirely backwards compatible with current Rust, and it also fits my expectation of how trait bounds should work (and is consistent with how type generics are specified: if the type parameter needs to always be the same, it's placed on the type itself, if it can differ between calls to a method it's placed on the method). Further evidence that this is the correct approach is that there's already a place where existing Rust has what in effect is an external implementation, and it works like this already. I realised recently that a lifetime `'a` is equivalent to an external impl of `impl<T> Deref for &'a T` (and likewise for `Deref` and `DerefMut` on `&'a mut T`). (That is, you can view `&'a T` as a type that implements `Deref` conditionally on being inside the lifetime `'a`, and thus the lifetime itself can be viewed as an external impl of `Deref` (providing one possible view of what a lifetime actually _is_ , theoretically).) This means that any problem with externally implemented traits should also have an equivalent problem that shows up with lifetimes. In the case of lifetimes, Rust prevents the equivalent of the hashtable problem arising by requiring any lifetime generics that exist within the types of `struct`/`enum` fields to also be lifetime generics of the type itself (thus forcing them to be the same every time). It seems that the fact that the generics exist on the type itself is important for soundness: there is at least one case in current Rust where a type involves types with a lifetime but they aren't generics on the type itself (closures, which can have a non-generic type even if they capture a value whose type has a lifetime), which creates a soundness hole in the type system. This is part of what makes me expect that the same solution would be used for external implementations other than lifetimes.

jrose: > Because it was not obvious to me the first time I encountered this problem, the requirement of `K: Hash` is _not_ on the HashMap type in Rust; it's on the impl block containing operations that hash. After thinking about this for ages, I've realised that the problem isn't really a problem with respect to types like `HashMap` for which the implemented trait is not `unsafe`. Not including `K: Hash` on the `HashMap` type itself means that it's impossible for external-impl proposals to ensure that `HashMap` is coherent in the sense of always hashing the same keys to the same values (because a different `Hash` impl could be used for each call); but even in current Rust, `HashMap` has numerous ways in which a programmer could cause incoherent hashing in a way that won't be caught by the compiler: * (edit: I corrected this in response to pitaj's correction) `Hash` is a safe trait, so nothing forces `K: Hash` and ~~`&K: Hash`~~ `Q: Hash` (where `Q: Borrow<K>` is used to do hash lookups) to have any relationship to each other, but it is possible to add keys to a hash as `K` and then retrieve them using a `Q`; * `Hash` is a safe trait, so nothing is forcing it to always hash the same key to the same hash (it could, e.g., return a random number) * Even if `Hash` is deterministic, it is possible to use interior mutability to modify the keys stored in a hash table (while they are being stored) in order to change the value they would hash to. `HashMap`'s documentation contains plenty of warnings against doing this sort of thing, but it is just warnings in the documentation (and in particular its memory safety is not allowed to depend on the programmer doing this). As such, a change to Rust to allow external impls wouldn't actually break `HashMap` in any ways that it isn't already broken: it would allow people to try to retrieve keys using a different hashing function from the one used to add them (which of course doesn't work), but there are plenty of ways to do that even without external impls. In general, it seems impossible for incoherent implementations of a safe trait to break existing code, as long as all the trait's associated items are methods (i.e. it has no associated types (including RPITIT as a special case of associated types) or associated constants). That's because code like this, which is the case that breaks (stealing the syntax from the OP): let g: Generic<T> = Generic::new(); (&g as &Generic<T + Trait use Impl1>).method(); (&g as &Generic<T + Trait use Impl2>).method(); could be replicated without external impls by doing something like this: thread_local! { static USE_IMPL_2: Cell<bool> = const { Cell::new(false); }; } impl Trait for T { // for each method in the trait, you write a small wrapper like this fn f(&self) { if USE_IMPL_2.get() { self.f2(); } else { self.f1(); } } } let g: Generic<T> = Generic::new(); USE_IMPL_2.set(false); g.method(); USE_IMPL_2.set(true); g.method(); This specifically only works for methods, because associated types or constants can't depend on the value of a global variable. (It also doesn't work for `unsafe` traits, because they may have safety requirements that prohibit this sort of implementation.) It strikes me that this is very similar for the rules of `dyn` compatibility (the only exception is that `dyn` compatibility requires the ability to make a vtable, whereas this doesn't), which is probably not entirely a coincidence. All this is making me think that external impls should probably be designed so that putting a bound on the type itself (as opposed to the `impl` block) is the correct way to say "this trait must always be implemented the same way whenever a method of this type is called": for safe, `dyn`-compatible traits, this seems to be entirely backwards compatible with current Rust, and it also fits my expectation of how trait bounds should work (and is consistent with how type generics are specified: if the type parameter needs to always be the same, it's placed on the type itself, if it can differ between calls to a method it's placed on the method). Further evidence that this is the correct approach is that there's already a place where existing Rust has what in effect is an external implementation, and it works like this already. I realised recently that a lifetime `'a` is equivalent to an external impl of `impl<T> Deref for &'a T` (and likewise for `Deref` and `DerefMut` on `&'a mut T`). (That is, you can view `&'a T` as a type that implements `Deref` conditionally on being inside the lifetime `'a`, and thus the lifetime itself can be viewed as an external impl of `Deref` (providing one possible view of what a lifetime actually _is_ , theoretically).) This means that any problem with externally implemented traits should also have an equivalent problem that shows up with lifetimes. In the case of lifetimes, Rust prevents the equivalent of the hashtable problem arising by requiring any lifetime generics that exist within the types of `struct`/`enum` fields to also be lifetime generics of the type itself (thus forcing them to be the same every time). It seems that the fact that the generics exist on the type itself is important for soundness: there is at least one case in current Rust where a type involves types with a lifetime but they aren't generics on the type itself (closures, which can have a non-generic type even if they capture a value whose type has a lifetime), which creates a soundness hole in the type system. This is part of what makes me expect that the same solution would be used for external implementations other than lifetimes.

jrose: > Because it was not obvious to me the first time I encountered this problem, the requirement of `K: Hash` is _not_ on the HashMap type in Rust; it's on the impl block containing operations that hash. After thinking about this for ages, I've realised that the problem isn't really a problem with respect to types like `HashMap` for which the implemented trait is not `unsafe`. Not including `K: Hash` on the `HashMap` type itself means that it's impossible for external-impl proposals to ensure that `HashMap` is coherent in the sense of always hashing the same keys to the same values (because a different `Hash` impl could be used for each call); but even in current Rust, `HashMap` has numerous ways in which a programmer could cause incoherent hashing in a way that won't be caught by the compiler: * (edit: I corrected this in response to pitaj's correction) `Hash` is a safe trait, so nothing forces `K: Hash` and ~~`&K: Hash`~~ `Q: Hash` (where `Q: Borrow<K>` is used to do hash lookups) to have any relationship to each other, but it is possible to add keys to a hash as `K` and then retrieve them using a `Q`; * `Hash` is a safe trait, so nothing is forcing it to always hash the same key to the same hash (it could, e.g., return a random number) * Even if `Hash` is deterministic, it is possible to use interior mutability to modify the keys stored in a hash table (while they are being stored) in order to change the value they would hash to. `HashMap`'s documentation contains plenty of warnings against doing this sort of thing, but it is just warnings in the documentation (and in particular its memory safety is not allowed to depend on the programmer doing this). As such, a change to Rust to allow external impls wouldn't actually break `HashMap` in any ways that it isn't already broken: it would allow people to try to retrieve keys using a different hashing function from the one used to add them (which of course doesn't work), but there are plenty of ways to do that even without external impls. In general, it seems impossible for incoherent implementations of a safe trait to break existing code, as long as all the trait's associated items are methods (i.e. it has no associated types (including RPITIT as a special case of associated types) or associated constants). That's because code like this, which is the case that breaks (stealing the syntax from the OP): let g: Generic<T> = Generic::new(); (&g as &Generic<T + Trait use Impl1>).method(); (&g as &Generic<T + Trait use Impl2>).method(); could be replicated without external impls by doing something like this: thread_local! { static USE_IMPL_2: Cell<bool> = const { Cell::new(false); }; } impl Trait for T { // for each method in the trait, you write a small wrapper like this fn f(&self) { if USE_IMPL_2.get() { self.f2(); } else { self.f1(); } } } let g: Generic<T> = Generic::new(); USE_IMPL_2.set(false); g.method(); USE_IMPL_2.set(true); g.method(); This specifically only works for methods, because associated types or constants can't depend on the value of a global variable. (It also doesn't work for `unsafe` traits, because they may have safety requirements that prohibit this sort of implementation.) It strikes me that this is very similar for the rules of `dyn` compatibility (the only exception is that `dyn` compatibility requires the ability to make a vtable, whereas this doesn't), which is probably not entirely a coincidence. All this is making me think that external impls should probably be designed so that putting a bound on the type itself (as opposed to the `impl` block) is the correct way to say "this trait must always be implemented the same way whenever a method of this type is called": for safe, `dyn`-compatible traits, this seems to be entirely backwards compatible with current Rust, and it also fits my expectation of how trait bounds should work (and is consistent with how type generics are specified: if the type parameter needs to always be the same, it's placed on the type itself, if it can differ between calls to a method it's placed on the method). Further evidence that this is the correct approach is that there's already a place where existing Rust has what in effect is an external implementation, and it works like this already. I realised recently that a lifetime `'a` is equivalent to an external impl of `impl<T> Deref for &'a T` (and likewise for `Deref` and `DerefMut` on `&'a mut T`). (That is, you can view `&'a T` as a type that implements `Deref` conditionally on being inside the lifetime `'a`, and thus the lifetime itself can be viewed as an external impl of `Deref` (providing one possible view of what a lifetime actually _is_ , theoretically).) This means that any problem with externally implemented traits should also have an equivalent problem that shows up with lifetimes. In the case of lifetimes, Rust prevents the equivalent of the hashtable problem arising by requiring any lifetime generics that exist within the types of `struct`/`enum` fields to also be lifetime generics of the type itself (thus forcing them to be the same every time). It seems that the fact that the generics exist on the type itself is important for soundness: there is at least one case in current Rust where a type involves types with a lifetime but they aren't generics on the type itself (closures, which can have a non-generic type even if they capture a value whose type has a lifetime), which creates a soundness hole in the type system. This is part of what makes me expect that the same solution would be used for external implementations other than lifetimes.

jrose: > Therefore "the Hash implementation is part of the type that depends on Hash" is _not_ a solution for the HashMap in std. Without further additions, at least. It's _afaict_ okay as long as the impl identity constraint is enforced for any function calls where both implementing and depending type appear in the signature bounds-linked (even indirectly, so basically any case where those types are linked by relevant type parameter bounds).

jdahlstrom: > This is, btw, essentially what C does Besides what @jrose already said about this, there is a huge practical difference between "only goes up to [some fixed-length type]" and actually going to infinity in both directions. For example, in C comparisons between signed and unsigned values silently do the Wrong Thing, and in Rust (as it is now) pub fn isless(a: i32, b: u32) -> bool { a < b } fails to compile but the suggested fix does _not_ produce code that computes the mathematically correct answer. Under my proposal this would be equivalent to pub fn isless(a: i32, b: u32) -> bool { // a u32 value that doesn't fit in i32 must be greater than all // possible i32 values i32::try_from(b).map(|ib| a < ib).unwrap_or(true) } chrefr: > Widening integer operations to i/u128 makes them way more expensive; widening float operations to ℝ will require _extremely costly_ software calculations. It is my hope that the same proofs of non-overflow that are required _anyway_ can also be used to prove that register-sized calculations are correct in most cases. We would presumably have to relax the principle somewhat for transcendental functions, because of the Table-Maker's Dilemma. > promotion rules are deeply unintuitive and confusing, and the vast majority of developers never understand them well I am explicitly claiming, as part of this proposal, that the rule "all calculations behave as-if they were carried out in the mathematical real numbers, and then, if the result doesn't always fit in the type of the target variable, you have to say what should happen for the edge cases" will be much easier to understand. I believe this claim because that rule is already basically what high-level interpreted languages like Python and Lisp do, and you never hear about people struggling with their promotion rules. Also, there are no implicit narrowing conversions under this proposal. Only situations where the compiler can prove that no conversion is required. jdahlstrom: > Realistically, this proposal hinges on the ability of the compiler to prove non-overflow in "typical uses", and that would require importing a lot of machinery to the front end from LLVM, be difficult to write a specification for, and would probably fail often enough that people would get massively frustrated This is a legit concern but I think a forward-only value range propagation pass, using rules like what we already have for whether it's OK to write "`let blah;`" with no initializer, will _probably_ be good enough and not too much hassle to specify or implement. Whether I'm right, probably requires actually trying it to answer. (Java's "definite assignment" rules are also in the same ballpark and are, at least, not an undue burden on implementations; I cannot say whether people who've written a lot of Java find them annoying, though.)

jrose: > The `await` in `for await` is trying to suggest a very specific meaning: "there is a suspension point here, unwrapping a future Make sense and very helpful for me, thank you!!! Now I would like **`for x await in X`** construction btw . Not about **`for`** and **`for await`** , but **`in`** and **`await in`** in async iteration (more declarative).

jrose: > to return any type that implements Deref[Mut] rather than a hardcoded reference would cover this use case I don't think so, you'll run in the same issue with `DerefMut` having to return a reference.

jrose: > I'll add to the general negative feeling here by pointing out that with RefCell this operation can _panic_ and so I might not want to use it anyway I agree. In hindsight I should have made it wrap the `try_borrow_mut` instead. I do like that I know this now: steffahn: > `*data.ref_cell.borrow_mut() = Some(new_value);` Which I was unaware of when I made the post. Preserving the left and right side of the equals was the main issue I had. I think what I might do instead though is just create a macro or use scripting because all I really want to do is abstract away a pattern that is going to be repeated thousands of times.

Everyone has a Rattled Cattle in Battle, why must you be the Ox, that puts it in a Boxら Why not be like the Fox, that can beat the Ox, Get out of a Box and Run like a Fox?!!