An interactive demo showcasing the Scrapii.net PII redaction & tokenization API. Paste text or upload a document, watch it get tokenized in real time, then detokenize to restore the originals ...

Mobile-first local llama.cpp inference, embeddings, RAG, multimodal, LoRA, and speculative decoding for Dart and Flutter. Changelog excerpt: - Prepared the package for warning-free pub.dev validation: vendored the curated pinned `llama.cpp`build sources, consolidated Dart CLIs under the conventional `example/`directory, and corrected published documentation paths and licensing metadata. - Added loaded-engine `tokenize`, `countTokens`, `detokenize`, `formatChat`, `countChatTokens`, and chat-template capability methods. They reuse the engine worker's existing model and serialize with inference; static chat token counting now formats an[...]

On-device speech (STT now; TTS/voice later) for flutter_gemma via the LiteRT C API + dart:ffi. Opt-in SttBackendProvider. Changelog excerpt: - feat: on-device STT (moonshine-tiny) via `LiteRtSttBackend`+ a generic, selectable `SttModelProfile`pipeline (encode → greedy decode → HF detokenize) on the shared LiteRT engine.

Hi, I’m doing the Translation section of the chapter, this part. After running this line trainer.evaluate(max_length=max_length) I get a warning: _That’s 100 lines that end in a tokenized period (‘.’)_ _It looks like you forgot to detokenize your test data, which may hurt your score._ _If you insist your data is detokenized, or don’t care, you can suppress this message with the`force` parameter._ I am doing basically everything like in tutorial, I only changed `fp16` to `False`, lowered to `per_device_eval_batch_size` to 32 (apparently not enough memory on my MacBook). This is before training and I get a bleu score 17 while tutorial has it 39. So I don’t know, I do not see that I may have skipped something and the tutorial code snippet has explicitly `tokenized_datasets` as an argument for the Trainer, so I am bit confused. I wasn’t able to figure it out myself. Earlier while loading the model I got “UserWarning: Recommended: pip install sacremoses.” and I did install it but afterwards I didn’t reload in my Jupyter notebook the loading of the model (the line `model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)`). Could this be a culprit? Maybe sacremoses has to be present as early as during loading model stage otherwise some default HF tokenizer is set? But so far everything worked fine, meaning that tokenized examples from the tutorial were the same as the ones generated by my code. Maybe tokenization is basically the same as default fallback, but sacremoses also provides detokenization which default does not? I will try to restart the environment and rerun the code. But in case it won’t work, maybe you will know the answer and will be able to help Edit: Ok I rerun everything but looks like eval considerably slower than before and it will be a few good hours before it finishes and I will know the answer. Maybe slowdown comes from the usage of the sacremoses, meaning that earlier it wasn’t really used as I hypothesized

YouTube video by RetroAndGaming