ML/Tensorflow logits meaning

Logits are the inputs to the softmax function:

the vector of raw (non-normalized) predictions that a classification model generates, which is ordinarily then passed to a normalization function. If the model is solving a multi-class classification problem, logits typically become an input to the softmax function. The softmax function then generates a vector of (normalized) probabilities with one value for each possible class.

ML attention animated

After all this time, I found this excellent animated example of attention and transformer and RNNs: Visualizing A Neural Machine Translation Model (Mechanics of Seq2seq Models With Attention) – Jay Alammar – Visualizing machine learning one concept at a time. From the same source, A Visual Intro to NumPy and Data Representation – Jay Alammar – Visualizing machine learning one concept at a time look very nice. I should resurrect my link wiki instead of pasting it all here.

Coding fonts

Python

Interesting python syntax

Interesting Python syntax I’ve seen in the Transformer Google repo:

  • /= assignment operator, which is like +=.
  • X if Cond else Y:
stats = ({
        "loss": train_loss
    } if history is None else misc.build_stats(history, callbacks))
  • Can be also used in returns:
    return evaluate_and_log_bleu(
        self.predict_model, self.params, self.flags_obj.bleu_source,
        self.flags_obj.bleu_ref, self.flags_obj.vocab_file,
        self.distribution_strategy if self.use_tpu else None)

Kinda relevant is code golf - Tips for golfing in Python - Code Golf Stack Exchange.

Python rich comparison operators

object.__lt__(self, other)
object.__le__(self, other)
object.__eq__(self, other)
object.__ne__(self, other)
object.__gt__(self, other)
object.__ge__(self, other)
  • The correspondence between operator symbols and method names is as follows: x<y calls x.lt(y), x<=y calls x.le(y), x==y calls x.eq(y), x!=y calls x.ne(y), x>y calls x.gt(y), and x>=y calls x.ge(y).

Stack

I should really create a vim thingy that automatically creates footnotes from a link. I can imagine it as a keystroke which generates a random footnote name and puts you on the last line of the file, with the footnote name prefilled, and in insert mode. Or another one that lets you specify a footnote name.