ML/Tensorflow logits meaning
Logits are the inputs to the softmax function:
the vector of raw (non-normalized) predictions that a classification model generates, which is ordinarily then passed to a normalization function. If the model is solving a multi-class classification problem, logits typically become an input to the softmax function. The softmax function then generates a vector of (normalized) probabilities with one value for each possible class.
ML attention animated
After all this time, I found this excellent animated example of attention and transformer and RNNs: Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time. From the same source, A Visual Intro to NumPy and Data Representation – Jay Alammar – Visualizing machine learning one concept at a time look very nice. I should resurrect my link wiki instead of pasting it all here.
- Monospaced Programming Fonts with Ligatures - Scott Hanselman
- I found Fira installed by default, with ligatures it looks quite nice.
Interesting python syntax
Interesting Python syntax I’ve seen in the Transformer Google repo:
/=assignment operator, which is like +=.
- X if Cond else Y:
- Can be also used in returns:
Kinda relevant is code golf - Tips for golfing in Python - Code Golf Stack Exchange.
Python rich comparison operators
The correspondence between operator symbols and method names is as follows: x<y calls x.lt(y), x<=y calls x.le(y), x==y calls x.eq(y), x!=y calls x.ne(y), x>y calls x.gt(y), and x>=y calls x.ge(y).
I should really create a vim thingy that automatically creates footnotes from a link. I can imagine it as a keystroke which generates a random footnote name and puts you on the last line of the file, with the footnote name prefilled, and in insert mode. Or another one that lets you specify a footnote name.