Add a small fraction of context

[ghost]

Is it possible to add a small fraction of context to the model, e.g an example of an expected sentence or keywords like in CMU Sphinx or GCP Speech-to-text? If yes, any references? I saw the training section on Wiki, but didn't get any hints of context

[snakers4]

Hi,

We have some domain adaptation and dictionary adaptation techniques, but generally they are for the EE version and are implemented as a part of post-processing so we do not make them public until the EE version is not public.

As for how to make something similar a part of the acoustic model only - I am not sure if there is an easy way to do so.

[ghost]

Thanks. I think that with context the model could achieve awesome results. Context or Mask like in Transformers, which relates a little to how Google Cloud Speech-to-text works internally (I guess), but yeah this is insanely complex.

Silero performs better than CMU Sphinx so it could be a cool idea to add the model to Speech Recognition.

[snakers4]

Hi, could you please clarify how knowing some dictionary terms in advance is related to the transformer attention / masking? I am not sure if I understand this idea entirely.

As for adding some wrappers to such repos, while I would not object if someone does it, but we have provided ample examples here and such wrappers for external APIs usually provide little real business value and usually just serve as karma building tools not bringing anything new to the table.

I have experimented with different model hubs recently ( https://t.me/snakers4/2569) and came to a similar conclusion recently.

[snakers4]

Oh, and that repo looks already abandoned

[ghost]

First of all, I can be wrong in my interpretation of Mask in Transformers. Masks in Transformers, more precisely in Multi-Headed Attention (Decoder), serves two purposes: the first one is parallel computing and the second one is to prevent cheating on the Decoder. It prevents cheating in Multi-Headed Attention by preventing the Decoder from looking at future tokens, this is done through a Mask that is a matrix with the same shape as the Attention Scores, filled with zeros and negative infinity. When the Scaled Scores are added to the Mask you get the Masked Scores, which is the matrix that turn off the future tokens, i.e if the sentence is 'I'm here right now' the Self-attention N in the Multi-Headed Attention will predict the next word with the context of 'I'm', 'I'm here', etc; by consequence, it allows parallel computing.

I have not mentioned Positional Encoding, Multi-Headed Attention, QUERY, KEY, VALUE, Score Matrix, Scaled Score Matrix, etc; cause it would be huge and very complex (its 4 am here in Brazil, ha ha)

---

I read this:

It made me think that a 'raw' translation is done, i.e statistic is being used to get a hint through the speech signal and then a numeric representation of the speech signal is created, where can be transformed into categorical. No context is considered, in the sense that we see on Transformers architectures as GPT-2. What if the Mask concept and the other Transformers concepts would be aggregated on the Speech Synthesis?! (the Hidden Markov Chain is an example, I saw here the key aspects of Silero and I'm aware of the decoder used)

From what I have read about Hidden Markov Chain it works like a vanilla Neural Network in the sense that you're optimizing your weights, but not fixing the other problems that RNN, GRU, LSTM, BiLSTM... and then Word2vec, Glove, Fast Text, and Transformers try to fix and even with the Silero architecture the context of the Hidden Markov Chain is the same, right?!?

Observation: All that I said is something that I'm questioning myself

[ghost]

Yeah, sadly the repo is abandoned, there are others that are active, but as you said it would not make that much sense.

[snakers4]

The public models here are mostly acoustic models, even though empirically they may seem to contain LMs
We tried to stay as much away from recurrence and complex decoders here
As for the idea of somehow hacking the transformer architecture for LM decoding, we are thinking about it
There is no short supply of huge pre-trained models, but (i) they are huge (ii) their tokensets are suited for written words (as opposed to spoken words)

As for abusing matrices to turn off future tokens, we do not like in-model streaming due to its complexity
Also they zero out the matrices ... but that does not reduce the computation!
The strength of the transformer is that it sees the whole sentence at once and can be easily batched easier (unlike traditional RNNs)

We are planning this year to create fully e2e compact high quality LMs
No ETAs though

[ghost]

Yeah, a huge change on the tokenization would be necessary, but after it is figured out the processing through the Transformer would not be hugely changed I suppose. And yeah, they are freaking insanely huge in terms of computational requirements, with an local RTX 2060, 16 GB ram, and Intel i7-9750H I found problems trying to fine-tune Roberta and clouds are even more insane, in terms of price... but once it is trained wow, the results are insane, I get access to GPT-3 and just wow. The matrices act like weights that we found on RNNs, the 'downside' is that you have multiple (ha ha), just considering each Multi-Headed Attention that normally has 8 Self-Attention, without considering the other components... yeah next level.

Cool, I hope to be able to contribute in the future, I'm entering in speech now and it looks way complex than image / video, tabular or normal text processing.

I marked the question as answered 😀

[ghost]

Any tips of content to start in speech-to-text?

[snakers4]

You can see my articles on the Gradient
95% of popular speech papers are written by Google / FAIR and they feature huge compute and usually are overfit on LibriSpeech
Kaldi is difficult to even launch =)
Beware of enterprise-like repos with 5k+ commits and 10 layers of abstractions
Better to start off with some simple repo, understand everything and move iteratively

[ghost]

Just let this as a history to this thread. Hugging Face is starting to have audio models, Open A.I published the paper DALL E thtat relates NLP to image / generative models and I saw that Transformers use wave length function to give 'meaning' in a certain degree to the codified information (keys - query) - Extra Tensorflow model that encode audio.