negdev: > > import numpy as np > from collections import Counter, defaultdict > from scipy.stats import spearmanr > > def utf8_bytes(s): > return np.frombuffer(s.encode("utf-8"), dtype=np.uint8) > > def three_byte_windows(b): > if len(b) < 3: > return np.empty((0,3), dtype=np.uint8) > return np.stack([b[i:i+3] for i in range(len(b)-2)], axis=0) > > def build_map_for_tokenizer(vocab): > """ > vocab: dict {token: weight} > Returns: > occupancy: Counter of 3-byte windows > continuation: dict {window: Counter(next_byte)} > endability: Counter of window endings > """ > occupancy = Counter() > continuation = defaultdict(Counter) > endability = Counter() > > for token, weight in vocab.items(): > b = utf8_bytes(token) > windows = three_byte_windows(b) > Below is the 3‑byte structural map code I used. It’s designed to isolate UTF‑8 byte‑level regularities inside the tokenizer’s vocabulary, without assuming any semantic structure or downstream behavior. The goal is to quantify purely structural signals (occupancy, continuation bias, and endability) that might correlate with per‑token difficulty or stability across training stages. The pipeline is intentionally simple: • Convert each token to its raw UTF‑8 byte sequence. • Extract all consecutive 3‑byte windows. • Accumulate three statistics: – occupancy: how often each 3‑byte window appears – continuation: conditional distribution of the next byte after each window – endability: how often a window appears at the end of a token These three quantities form the basis for dominance and endability metrics: dominance = max(next_byte_count) / sum(next_byte_counts) endability = endability_count / occupancy_count The code: import numpy as np from collections import Counter, defaultdict def utf8_bytes(s): return np.frombuffer(s.encode(“utf-8”), dtype=np.uint8) def three_byte_windows(b): if len(b) < 3: return np.empty((0,3), dtype=np.uint8) return np.stack([b[i:i+3] for i in range(len(b)-2)], axis=0) def build_map_for_tokenizer(vocab): occupancy = Counter() continuation = defaultdict(Counter) endability = Counter() for token, weight in vocab.items(): b = utf8_bytes(token) windows = three_byte_windows(b) for i, w in enumerate(windows): w_tuple = tuple(w.tolist()) occupancy[w_tuple] += weight # continuation if i + 3 <= len(b) - 1: next_byte = int(b[i+3]) continuation[w_tuple][next_byte] += weight # endability if i == len(windows) - 1: endability[w_tuple] += weight return occupancy, continuation, endability This structure is intentionally agnostic to the model’s training dynamics. It only captures the tokenizer’s byte‑level geometry. The idea was to check whether any of these structural biases (especially dominance and endability) correlate with per‑word exam accuracy, difficulty slopes, or the stability of difficulty across training stages. If you run a 2‑byte / 3‑byte / 4‑byte variant, the same counters generalize directly. It’s cheap to compute and can be scored against existing per‑word accuracies without additional training runs.

E - Picking Goods https://atcoder.jp/contests/abc175/tasks/abc175_e - abc175_e.py

F - Confluence https://atcoder.jp/contests/abc183/tasks/abc183_f - abc183_f.py

YouTube video by Python Morsels

Advent Of Code 2025. Contribute to marcodelmastro/AdventOfCode2025 development by creating an account on GitHub.

What's Changed Correct invalid serialization of date/datetime/time/timedelta by pulling downcast checks up by @astei in #1851 avoid getting default values from defaultdict by @davidhewitt in #1853...