mjw

6 ms·

1.
by mjw 7y ago
> A minor key has sharps and flats to create tension. The natural minor or Aeolian mode doesn't use any notes outside the diatonic scale (probably what you meant by "sharps and flats"). It's very possible to write sad
2.
by mjw 8y ago
This is very neat. That said the reason these methods haven't received much attention so far is that relatively few people actually need to compute Jacobeans or Hessians directly. Often only Hessian-vector products or Jacobean-vector p
3.
by mjw 8y ago
When I started out in ML I was really keen to learn about the most 'mathsy' approaches out there. I think with hindsight, it's great to have a broad spectrum of methods available to you, but if you focus too much on methods a
4.
by mjw 10y ago
Ah yep, true. I'd forgotten you can still get the saddle effect from higher-order derivatives, the Hessian eigenvalues aren't enough to characterise it. I was thinking of examples like (x-y)^2 at zero, although I guess that's
5.
by mjw 10y ago
My main quibble from this paper is: > For deeper networks, Corollary 2.4 states that there exist “bad” saddle points in the sense that the Hessian at the point has no negative eigenvalue. To me these sound just as bad as local minima. Al
6.
by mjw 10y ago
See the other replies above, but: the logistic has heavier tails than the normal, so might do better in cases where we need robustness, where unexpected outcomes remain possible even in cases where the linear predictor is relatively big, an
7.
by mjw 10y ago
Ah yep, I forgot it's the canonical link. That's more of a small computational convenience though, right, at least when fitting a straightforward GLM -- it should be very cheap to fit regardless. I suppose the logistic having heav
8.
by mjw 10y ago
Their answer is pretty much 'because it's based on the log-odds', which to me is still only very mild motivation. There are other non-linearities which people use to map onto (0, 1), for example probit regression uses the Nor
9.
by mjw 10y ago
Pretty much any kind of mathematical modelling that involves uncertainty, really. Making inferences and predictions from data, in the presence of uncertainty. Analysis of the properties of procedures for doing the above. If you want example
10.
by mjw 10y ago
If anything, to me a lot of deep learning literature seems to lack the statistical insight and theory that's available to other subfields in machine learning (whether the Bayesian/graphical models camp, the statistical learning th
11.
by mjw 11y ago
It's more an empirically-verified thing than a mathematical fact, there's nothing magic about 16 bits AFAIK. Empirically 16 bits seems to work well enough for some tasks, taking it down to 8 bits is usually taking it too far, and
12.
by mjw 11y ago
Yep. To elaborate: really big batch sizes can speed up training data throughput, but usually mean that less is learned from each example seen, so time-to-convergence might not necessarily improve (might even increase, if you take things too
13.
by mjw 11y ago
Warp-CTC implements one specific model (or at least, one specific loss function), it's not really a general framework in the same way as the other libraries mentioned.
14.
by mjw 11y ago
> If I publish a paper people want to know what the data I found suggests and that's it. What they're going to get, is what your data and your modelling assumptions suggest. If you're taking just as much care to make the
15.
by mjw 11y ago
It's odd to complain specifically about subjectivity of the prior when the likelihood is often just as subjective. Gelman puts this well here: http://andrewgelman.com/2015/01/27/perhaps-merely-accident-h.
16.
by mjw 11y ago
On the whole this is useful, although I think it's a little unfair to Theano in places. * Performance I feel they should score separately here for compilation/startup time vs runtime. Theano's compilation step can be slow the
17.
by mjw 11y ago
Agreed that SQL is ugly as hell, but if you want to talk about its theoretical properties that's a separate debate. Theory doesn't care whether something's aesthetically pleasing, just whether it's possible.
18.
by mjw 11y ago
Does anyone know if TensorFlow can apply algebraic simplifications and numerical optimisations to the compute graph, in the way that Theano does with its optimisations? Sounds like it doesn't suffer from the (alleged) slow compile time
19.
by mjw 11y ago
Ah OK, fair enough. Not sure what definition you're using for the category of tables, but I don't think the distinction between table and query is really that significant, at least from a theory point of view. You can declare view
20.
by mjw 11y ago
> What I mostly see around is just standard linear algebra operations on matrices and vectors lifted to higher-dimensional tensors point-wise Equally what is matrix multiplication but a bunch of 1-dimensional dot products applied pointwi
21.
by mjw 11y ago
Eh... SELECT a.*, b.* FROM a CROSS JOIN b; (SELECT 'a' AS tag, * FROM a) UNION (SELECT 'b' AS tag, * FROM b);
22.
by mjw 11y ago
AVDI is incredibly awesome. Thanks so much for this work. Really nice readable paper on it too. As someone working with large datasets I think automating variational inference + SGD to work with a broad class of models is really the way for
23.
by mjw 11y ago
It's nice to get people thinking about possible connections and sharing terminology here, but I'm not sure that many of the connections which the article manages to make precise are particularly new or deep. A neural network is ju
24.
by mjw 11y ago
When you penalise the L2 norm of the convolution of the image with a filter (like a gradient or edge detector for example) you are effectively doing this. The spectrum of the filter determines how much different frequency components are pen
25.
by mjw 11y ago
I've heard this too and it's a great way to demonstrate you don't really know what statistics is :) Statistics is not (just) opinion polling, there's a lot more to it than estimating observable properties of a population
26.
by mjw 11y ago
I know they've traditionally been quite fiddly, but I'm pretty sure computers can be persuaded to help derive the maths for variational methods these days. Perhaps a more important difference is that MCMC, while slow, is exact in
27.
by mjw 12y ago
Yeah I was thinking about this after I posted. Not entirely convinced though -- I want the hyperparameters I learn to generalise to unseen data, just like plain old parameters. If there are two methods for learning them then I'm going
28.
by mjw 12y ago
Nice post, couple of bits of feedback: When you talk about "fit" it sounds like you mean fit to the training data, which would obviously be a bad thing to optimise hyperparameters for. From the github repo it sounds like you are u
29.
by mjw 12y ago
Agreed. What I see as the core principles of OO (ad-hoc polymorphic dispatch and abstract data types) are present, useful and important in the FP world too. OO helped to bring attention to these ideas even if it didn't capture them in
30.
by mjw 12y ago
Interesting to see Dutch mentioned. From what little I know about Dutch, its orthography does seem more similar to English than any other non-english-derived language I've seen. I seem to remember that some (clearly not all!) of the h

More ›