Math and science::INF ML AI Attention layer Do you remember the implementation of Attention in Pytorch? dummy cloze class Attention(nn.Module): def __init__(self, dim, heads=4, dropout=0.0, project_out=True): super().__init__() self.proj_out = project_out self.dim_head = dim / heads if int(self.dim_head) != self.dim_head: raise ValueError( f"dim ({dim}) must be divisible by heads ({heads})" ) self.dim_head = int(self.dim_head) self.scale = self.dim_head**-0.5 self.heads = heads hidden_dim = self.dim_head * heads self.to_qkv = nn.Linear(dim, hidden_dim * 3, bias=False) self. Read more...

Math and science::INF ML AI Sinusoidal positional embedding Can you remember how to generate an embedding for an array of shape [n_time, n_channel]? How to add this array to a Pytorch module?

Math and science::INF ML AI Sinusoidal positional embedding Can you remember how to generate an embedding for an array of shape [n_time, n_channel]? How to add this array to a Pytorch module? 1. Create sinusoidal embedding @torch.no_grad() def get_sinusoidal_embeddings(length: int, n_ch: int) -> torch.Tensor: """ Create a tensor of shape [time_steps, n_channels//2] and fill all rows with the sequence: [0, 1, 2, ..., length-1]. Then, scale each row by a different value. Read more...

Math and science::INF ML AI Transformer implementation On the reverse side, there is a full transformer, implemented in Pytorch in vit-pytorch. Can you remember it? [...]

Math and science::INF ML AI Transformer implementation On the reverse side, there is a full transformer, implemented in Pytorch in vit-pytorch. Can you remember it? Source https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/vit_1d.py#L9

\( \newcommand{\cat}[1] {\mathrm{#1}} \newcommand{\catobj}[1] {\operatorname{Obj}(\mathrm{#1})} \newcommand{\cathom}[1] {\operatorname{Hom}_{\cat{#1}}} \newcommand{\multiBetaReduction}[0] {\twoheadrightarrow_{\beta}} \newcommand{\betaReduction}[0] {\rightarrow_{\beta}} \newcommand{\betaEq}[0] {=_{\beta}} \newcommand{\string}[1] {\texttt{"}\mathtt{#1}\texttt{"}} \newcommand{\symbolq}[1] {\texttt{`}\mathtt{#1}\texttt{'}} \newcommand{\groupMul}[1] { \cdot_{\small{#1}}} \newcommand{\groupAdd}[1] { +_{\small{#1}}} \newcommand{\inv}[1] {#1^{-1} } \newcommand{\bm}[1] { \boldsymbol{#1} } \require{physics} \require{ams} \require{mathtools} \) Math and science::INF ML AI Gompertz distribution Gompertz distribution The Gompertz distribution is defined equivalently by: Probability density function [\[ f(t) = k e^{\alpha t} e^{ ?? } \]] Survival function [\[ S(t) = e^{ ? Read more...

\( \newcommand{\cat}[1] {\mathrm{#1}} \newcommand{\catobj}[1] {\operatorname{Obj}(\mathrm{#1})} \newcommand{\cathom}[1] {\operatorname{Hom}_{\cat{#1}}} \newcommand{\multiBetaReduction}[0] {\twoheadrightarrow_{\beta}} \newcommand{\betaReduction}[0] {\rightarrow_{\beta}} \newcommand{\betaEq}[0] {=_{\beta}} \newcommand{\string}[1] {\texttt{"}\mathtt{#1}\texttt{"}} \newcommand{\symbolq}[1] {\texttt{`}\mathtt{#1}\texttt{'}} \newcommand{\groupMul}[1] { \cdot_{\small{#1}}} \newcommand{\groupAdd}[1] { +_{\small{#1}}} \newcommand{\inv}[1] {#1^{-1} } \newcommand{\bm}[1] { \boldsymbol{#1} } \require{physics} \require{ams} \require{mathtools} \) Math and science::INF ML AI Gompertz distribution Gompertz distribution The Gompertz distribution is defined equivalently by: Probability density function \[ f(t) = k e^{\alpha t} e^{-\frac{k}{\alpha}(e^{\alpha t} - 1)} \] Survival function \[ S(t) = e^{-\frac{k}{\alpha}(e^{\alpha t} - 1)} \] Hazard function \[ h(t) = k e^{\alpha t} \] Gompertz distribution. Read more...

Math and science::INF ML AI Edward Zang's stride visualizer   

Math and science::INF ML AI Edward Zang's stride visualizer    Source https://ezyang.github.io/stride-visualizer/index.html

\( \newcommand{\cat}[1] {\mathrm{#1}} \newcommand{\catobj}[1] {\operatorname{Obj}(\mathrm{#1})} \newcommand{\cathom}[1] {\operatorname{Hom}_{\cat{#1}}} \newcommand{\multiBetaReduction}[0] {\twoheadrightarrow_{\beta}} \newcommand{\betaReduction}[0] {\rightarrow_{\beta}} \newcommand{\betaEq}[0] {=_{\beta}} \newcommand{\string}[1] {\texttt{"}\mathtt{#1}\texttt{"}} \newcommand{\symbolq}[1] {\texttt{`}\mathtt{#1}\texttt{'}} \newcommand{\groupMul}[1] { \cdot_{\small{#1}}} \newcommand{\groupAdd}[1] { +_{\small{#1}}} \newcommand{\inv}[1] {#1^{-1} } \newcommand{\bm}[1] { \boldsymbol{#1} } \require{physics} \require{ams} \require{mathtools} \) Math and science::Algebra Effect of add-identity to eigenvectors Let \( A \) be a matrix that decomposes into: \[ A = X\Lambda X^{-1} \] Then it's also true that: [\[ A + \beta I = \quad ? Read more...