I worked with trying to predict secondary DNA structure in my masters a few years ago with different types of seq2seq NN models trained off basic energy calculations to see if it was even feasible and while it worked, it never got anywhere close to even 90% for the amount of computation it took.
September 26, 2026 at 4:08 AM
I worked with trying to predict secondary DNA structure in my masters a few years ago with different types of seq2seq NN models trained off basic energy calculations to see if it was even feasible and while it worked, it never got anywhere close to even 90% for the amount of computation it took.
Seq2Seq just turned 10!
NIPS: Oral Session 4 - Ilya Sutskever
YouTube video by Microsoft Research
www.youtube.com
November 14, 2024 at 9:57 PM
Seq2Seq just turned 10!
I have converted a portion of my NLP Online Masters course to blog form. This is the progression I present that takes one from recurrent neural network to seq2seq with attention to transformer. mark-riedl.medium.com/transformers...
Transformers: Origins
An unofficial origin story of the transformer neural network architecture.
mark-riedl.medium.com
November 26, 2024 at 2:15 AM
I have converted a portion of my NLP Online Masters course to blog form. This is the progression I present that takes one from recurrent neural network to seq2seq with attention to transformer. mark-riedl.medium.com/transformers...
Test of Time Paper Awards are out! 2014 was a wonderful year with lots of amazing papers. That's why, we decided to highlight two papers: GANs (@ian-goodfellow.bsky.social et al.) and Seq2Seq (Sutskever et al.). Both papers will be presented in person 😍
Link: blog.neurips.cc/2024/11/27/a...
Link: blog.neurips.cc/2024/11/27/a...
Announcing the NeurIPS 2024 Test of Time Paper Awards – NeurIPS Blog
blog.neurips.cc
November 27, 2024 at 3:48 PM
Test of Time Paper Awards are out! 2014 was a wonderful year with lots of amazing papers. That's why, we decided to highlight two papers: GANs (@ian-goodfellow.bsky.social et al.) and Seq2Seq (Sutskever et al.). Both papers will be presented in person 😍
Link: blog.neurips.cc/2024/11/27/a...
Link: blog.neurips.cc/2024/11/27/a...
finally doing my last CS class atm which is Natural Language Processimg, very satisfying learning about pre-transformers attention and encoder-decoder models and seeing how those pieces came together :)
April 21, 2026 at 8:30 AM
finally doing my last CS class atm which is Natural Language Processimg, very satisfying learning about pre-transformers attention and encoder-decoder models and seeing how those pieces came together :)
In many ways word2vec has not stood the test of time, nor did seq-to-seq if you consider that ultimately attention was all you needed, throwing out the seq2seq bit.
November 27, 2024 at 10:10 PM
In many ways word2vec has not stood the test of time, nor did seq-to-seq if you consider that ultimately attention was all you needed, throwing out the seq2seq bit.
Indeed, transformers were developed for translation (and other seq2seq tasks)
"On 2017-06-12... the "Attention is all you need" paper. At the time, the focus of the research was on improving seq2seq for machine translation...."
"On 2017-06-12... the "Attention is all you need" paper. At the time, the focus of the research was on improving seq2seq for machine translation...."
Attention Is All You Need - Wikipedia
share.google
June 21, 2026 at 4:30 PM
Indeed, transformers were developed for translation (and other seq2seq tasks)
"On 2017-06-12... the "Attention is all you need" paper. At the time, the focus of the research was on improving seq2seq for machine translation...."
"On 2017-06-12... the "Attention is all you need" paper. At the time, the focus of the research was on improving seq2seq for machine translation...."
seq2seq RNNs were pretty impressive for the time too (many people remember the Tay drama, and it was likely one of these).
I guess one could draw a line at "generating text message by message = virtuous ML" and "whole sessions = wretched GenAI" but that's just silly to me.
I guess one could draw a line at "generating text message by message = virtuous ML" and "whole sessions = wretched GenAI" but that's just silly to me.
August 2, 2026 at 4:17 PM
seq2seq RNNs were pretty impressive for the time too (many people remember the Tay drama, and it was likely one of these).
I guess one could draw a line at "generating text message by message = virtuous ML" and "whole sessions = wretched GenAI" but that's just silly to me.
I guess one could draw a line at "generating text message by message = virtuous ML" and "whole sessions = wretched GenAI" but that's just silly to me.
gpt-4o-instruct better than the random toki pona model seq2seq model i found on huggingface. im shocked but i shouldnt be
November 15, 2024 at 2:29 PM
gpt-4o-instruct better than the random toki pona model seq2seq model i found on huggingface. im shocked but i shouldnt be
broadly if you can recast your issue as seq2seq it seems like you should be able to hit it over the head with this or some variant of it
August 3, 2025 at 10:58 PM
broadly if you can recast your issue as seq2seq it seems like you should be able to hit it over the head with this or some variant of it
Just realized Ilya Sutskever has won the NeurIPS Test of Time Award three years in a row: 2022 for AlexNet, 2023 for Word2Vec, and 2024 for Seq2Seq 🤯
This is insane! Hats off to him!
This is insane! Hats off to him!
December 5, 2024 at 4:50 PM
Just realized Ilya Sutskever has won the NeurIPS Test of Time Award three years in a row: 2022 for AlexNet, 2023 for Word2Vec, and 2024 for Seq2Seq 🤯
This is insane! Hats off to him!
This is insane! Hats off to him!
she shared excellent insights on what this implies (your brain runs probably faster on those questions, when you a future CVPR PC 😅):
- Nothing will ever be blind anymore because there's enough data in the wild to train a reviewer de-anonymizer
- What about fine-tuning a seq2seq model conditioned
- Nothing will ever be blind anymore because there's enough data in the wild to train a reviewer de-anonymizer
- What about fine-tuning a seq2seq model conditioned
November 28, 2025 at 4:42 PM
she shared excellent insights on what this implies (your brain runs probably faster on those questions, when you a future CVPR PC 😅):
- Nothing will ever be blind anymore because there's enough data in the wild to train a reviewer de-anonymizer
- What about fine-tuning a seq2seq model conditioned
- Nothing will ever be blind anymore because there's enough data in the wild to train a reviewer de-anonymizer
- What about fine-tuning a seq2seq model conditioned
Explaining Seq2Seq Encoding-Decoding Processes For Linguists
From @ricardolezama.com, stay abreast of AI developments and follow.
ricardolezama.com/english/data...
From @ricardolezama.com, stay abreast of AI developments and follow.
ricardolezama.com/english/data...
Explaining Seq2Seq Encoding-Decoding Processes – Ricardo Lezama
ricardolezama.com
February 26, 2025 at 1:06 PM
Explaining Seq2Seq Encoding-Decoding Processes For Linguists
From @ricardolezama.com, stay abreast of AI developments and follow.
ricardolezama.com/english/data...
From @ricardolezama.com, stay abreast of AI developments and follow.
ricardolezama.com/english/data...
PyTorch自然言語処理プログラミング word2vec/LSTM/seq2seq/BERTで日本語テキスト解析! impress top gearシリーズ (新納浩幸) が、紀伊國屋電子書籍の【出版社合同】秋のIT書フェア2026で1540円引きの、1540円(50%OFF)+14ポイント還元になりました。10/8(木)まで。
PyTorch自然言語処理プログラミング word2vec／LSTM／seq2seq／BERTで日本語テキスト解析！ impress top gearシリーズ
著者：新納浩幸(著) 出版：インプレス 2021/3/18(木)配信
5leaf.jp
September 25, 2026 at 9:25 PM
PyTorch自然言語処理プログラミング word2vec/LSTM/seq2seq/BERTで日本語テキスト解析! impress top gearシリーズ (新納浩幸) が、紀伊國屋電子書籍の【出版社合同】秋のIT書フェア2026で1540円引きの、1540円(50%OFF)+14ポイント還元になりました。10/8(木)まで。
（´-`）.｡oO( 現在のGeminiの技術リードであるオリオール・ヴィニャルズ(seq2seq)もディーンと一緒にGoogleを退社するから，もう事実上Geminiは終了な感じしかしないよね… )
August 5, 2026 at 11:36 PM
（´-`）.｡oO( 現在のGeminiの技術リードであるオリオール・ヴィニャルズ(seq2seq)もディーンと一緒にGoogleを退社するから，もう事実上Geminiは終了な感じしかしないよね… )
a slide from their pitch deck
i.e. “we built the modern internet”
i.e. “we built the modern internet”
August 5, 2026 at 6:11 PM
a slide from their pitch deck
i.e. “we built the modern internet”
i.e. “we built the modern internet”
CDS-BART combines SentencePiece BPE tokenization with BART's denoising seq2seq architecture, pre-trained on 60 million CDSs from nine NCBI RefSeq taxonomic groups to handle sequences up to ~4 kb.
September 23, 2026 at 10:02 AM
CDS-BART combines SentencePiece BPE tokenization with BART's denoising seq2seq architecture, pre-trained on 60 million CDSs from nine NCBI RefSeq taxonomic groups to handle sequences up to ~4 kb.
I lean on this one a lot because it's basically normalized now. All the social networks, including this one, are using at the very least NLP & ML to handle spam, and most have thrown an LLM model on top of it. JPM has publicly discussed their in-house anti-spam model. arxiv.org/pdf/2304.01238
April 26, 2025 at 9:30 PM
I lean on this one a lot because it's basically normalized now. All the social networks, including this one, are using at the very least NLP & ML to handle spam, and most have thrown an LLM model on top of it. JPM has publicly discussed their in-house anti-spam model. arxiv.org/pdf/2304.01238
Fun facts from @hf.co:
LLMs are built on the Transformer architecture, a deep learning architecture based on the "Attention" algorithm
There are 3 types of transformers:
1) encoders
2) decoders
3) Seq2Seq (Encoder - Decoder)
More LLMs are typically decoder-based models
LLMs are built on the Transformer architecture, a deep learning architecture based on the "Attention" algorithm
There are 3 types of transformers:
1) encoders
2) decoders
3) Seq2Seq (Encoder - Decoder)
More LLMs are typically decoder-based models
July 14, 2025 at 8:58 PM
Fun facts from @hf.co:
LLMs are built on the Transformer architecture, a deep learning architecture based on the "Attention" algorithm
There are 3 types of transformers:
1) encoders
2) decoders
3) Seq2Seq (Encoder - Decoder)
More LLMs are typically decoder-based models
LLMs are built on the Transformer architecture, a deep learning architecture based on the "Attention" algorithm
There are 3 types of transformers:
1) encoders
2) decoders
3) Seq2Seq (Encoder - Decoder)
More LLMs are typically decoder-based models
Argument structure analysis helps to understand human argumentation. Bao et al. present a unified generative framework for this: UniASA. It can uniformly address multiple argument structure analysis tasks in a seq2seq manner.
Read about it here: doi.org/10.1162/coli... #NLProc @aixinsg.bsky.social
Read about it here: doi.org/10.1162/coli... #NLProc @aixinsg.bsky.social
April 24, 2026 at 5:35 AM
Argument structure analysis helps to understand human argumentation. Bao et al. present a unified generative framework for this: UniASA. It can uniformly address multiple argument structure analysis tasks in a seq2seq manner.
Read about it here: doi.org/10.1162/coli... #NLProc @aixinsg.bsky.social
Read about it here: doi.org/10.1162/coli... #NLProc @aixinsg.bsky.social
I'm surprised it's not setting off red flags for people, since seq2seq and GloVe predate Arrival.
December 1, 2024 at 8:08 PM
I'm surprised it's not setting off red flags for people, since seq2seq and GloVe predate Arrival.
Not quite. Other NNs had attention: seq2seq was an RNN with attention and predated transformers by a few years.
The point of "Attention Is All Your Need" is that you could rip out recurrences and convolutions from seq2seq-like architectures and get better performance.
The point of "Attention Is All Your Need" is that you could rip out recurrences and convolutions from seq2seq-like architectures and get better performance.
June 22, 2025 at 3:27 AM
Not quite. Other NNs had attention: seq2seq was an RNN with attention and predated transformers by a few years.
The point of "Attention Is All Your Need" is that you could rip out recurrences and convolutions from seq2seq-like architectures and get better performance.
The point of "Attention Is All Your Need" is that you could rip out recurrences and convolutions from seq2seq-like architectures and get better performance.
このくらいが普通の高校生なのにローカルLLM会の高校生ときたら……Transformerレベル（アーキテクチャレベル）の理論なら、
Seq2Seq->ResNet->Transformer->S4・Hyena・Mamba
という順に詳しく解説して、実装していくのが堅そう。とか抜かす超人しかいねえ。
Seq2Seq->ResNet->Transformer->S4・Hyena・Mamba
という順に詳しく解説して、実装していくのが堅そう。とか抜かす超人しかいねえ。
April 19, 2024 at 4:44 AM
このくらいが普通の高校生なのにローカルLLM会の高校生ときたら……Transformerレベル（アーキテクチャレベル）の理論なら、
Seq2Seq->ResNet->Transformer->S4・Hyena・Mamba
という順に詳しく解説して、実装していくのが堅そう。とか抜かす超人しかいねえ。
Seq2Seq->ResNet->Transformer->S4・Hyena・Mamba
という順に詳しく解説して、実装していくのが堅そう。とか抜かす超人しかいねえ。
I totally understand seq2seq! Is anyone working on GANs or GAN-related topics in 2024?
November 27, 2024 at 5:00 PM
I totally understand seq2seq! Is anyone working on GANs or GAN-related topics in 2024?