16 ms·
Kolmogorov-Arnold Networks
- kevmo314 2y agoThis seems very similar in concept to the finite element method. Nice to see patterns across fields like that.
- cloudhan 2y agoThis reminds me of Weight Agnostic Neural Networks https://weightagnostic.github.io/ https://weightagnostic.github.io/
- esafak 2y agoThere exists a Kolmogorov-Arnold inspired model in classical statistics called GAMs (https://en.wikipedia.org/wiki/Generalized_additive_model https://en.wikipedia.org/wiki/Generalized_additive_model), developed by Hastie and Tibshirani as an extension of GLMs (https://en.wikipedia.org/wiki/Generalized_linear_model https://en.wikipedia.org/wiki/Generalized_linear_model). GLMs in turn generalize logistic-, linear and other popular regression models. Neural GAMs with learned basis functions have already been proposed, so I'm a bit surprised that the prior art is not mentioned in this new paper. Previous applications focused more on interpretability.
- talegari 2y agoExactly! This is the first thought that came to my mind. Google search with KAN and GAM bought me here.
- coderenegade 2y agoI was under the impression that graph neural nets already trained learnable functions on graph edges rather than nodes, albeit typically on a fully connected graph. Is there any comparison to just a basic GNN here?
- vindex10 2y agoBayesian networks learn probability functions, but looks like only their tabulated versions: https://en.wikipedia.org/wiki/Bayesian_network#Graphical_model https://en.wikipedia.org/wiki/Bayesian_network#Graphical_mod... > Each node is associated with a probability function that takes, as input, a particular set of values for the node's parent variables, and gives (as output) the probability (or probability distribution, if applicable) of the variable represented by the node. For example, if m parent nodes represent m Boolean variables, then the probability function could be represented by a table of 2^m entries, one entry for each of the 2^m possible parent combinations.
- nu91 2y agoI am curious to know if this type of network can help with causal inference.
- esafak 2y agoThey help with interpretability, which is a step brother. See for example "From Shapley Values to Generalized Additive Models and back" https://arxiv.org/abs/2209.04012 https://arxiv.org/abs/2209.04012
- nu91 2y agoIndeed, Thank you!
- fermisea 2y agoThey definitely do, I'm planning to release a new package to do that in a couple weeks
- yza 2y agoBayesian KANs, KAN Transformers and KAN VAE's in 3.2...
- erwincoumans 2y agoIf you like this, you may also like this 2019 research paper: "Deep networks and the Kolmogorov–Arnold theorem" https://hadrien-montanelli.github.io/2019-06-25.html https://hadrien-montanelli.github.io/2019-06-25.html https://arxiv.org/pdf/1906.11945 https://arxiv.org/pdf/1906.11945
- syassami 2y agoNice implementation I've been playing with, https://github.com/Blealtan/efficient-kan https://github.com/Blealtan/efficient-kan alongside @GistNoesis's.
- mipt98 2y agoA more elaborate implementation of this was published years ago, and it wasn't the very first one https://www.science.org/doi/10.1126/science.1165893 https://www.science.org/doi/10.1126/science.1165893
- thesz 2y agohttps://arxiv.org/abs/1210.7273 https://arxiv.org/abs/1210.7273 In the article "Distilling free-form natural laws from experimental data", Schmidt and Lipson introduced the idea that free-form natural laws can be learned from experimental measurements in a physical system using symbolic (genetic) regression algorithms. An important claim in this work is that the algorithm finds laws in data without having incorporated any prior knowledge of physics. Upon close inspection, however, we show that their method implicitly incorporates Hamilton's equations of motions and Newton's second law, demystifying how they are able to find Hamiltonians and special classes of Lagrangians from data. I think this is hilarious. I cannot get a PDF of your article and instead I will read a commentary on it which appears to be very interesting.
- adityang5 2y agoVery cool stuff! Exciting to see so many people sharing their works on KANs. Seeing as the authors claim that KANs are able to reduce the issues of catastrophic forgetting that we see in MLPs, I thought "Wouldn't it be nice if there was an LLM that substituted MLPs with KANs?". I looked around and didn't find one, so I built one! - PyTorch Module of the KAN GPT - Deployed to PyPi - MIT Licence - Test Cases to ensure forward-backward passes work as expected - Training script I am currently working on training it on the WebText dataset to compare it to the original gpt2. Facing a few out-of-memory issues at the moment. Perhaps the vocab size (50257) is too large? I'm open to contributions and would love to hear your thoughts! https://github.com/AdityaNG/kan-gpt https://github.com/AdityaNG/kan-gpt https://pypi.org/project/kan-gpt/ https://pypi.org/project/kan-gpt/
- renonce 2y agoSo a new type of neural network that has been proven to work well on regression tasks common in physics? And tested in practice to fit well on elementary algebra and compositions of complex functions. But no evidence at all if it works on the most basic machine learning tasks like MNIST, not to mention language models. I mean it's great but at the current state it seems better suited for tasks where an explicit formula exists (though not known) and the goal is to predict it on unknown points (and be able to interpret the formula as a side effect). Deep learning tasks are more of a statistical nature (think models with a cross entropy loss - it's statistically predicting the frequency of different choices of the class/next token), it requires a specialized training procedure and it is designed to fit 100% rather than somewhat close (think linear algebra - it won't be good at it). It would very likely take a radically different idea to apply it to deep learning tasks. The recently updated "Author's note" also mentions this: "KANs are designed for applications where one cares about high accuracy and/or interpretability." It's great but let's be patient before we see this improve LLM accuracy or be used elsewhere.
- phpkar 2y agohttps://arxiv.org/abs/2404.05903 https://arxiv.org/abs/2404.05903
- apolar 2y agoArticle 2021: https://www.sciencedirect.com/science/article/abs/pii/S0952197620303742 https://www.sciencedirect.com/science/article/abs/pii/S09521... Seminar 2021: https://warwick.ac.uk/fac/sci/maths/research/events/seminars/areas/applmath/2020-21/#WEEK1 https://warwick.ac.uk/fac/sci/maths/research/events/seminars... Article in archive 2023: https://arxiv.org/abs/2305.08194 https://arxiv.org/abs/2305.08194 Video 2021: https://www.youtube.com/watch?v=eS_k6L638k0 https://www.youtube.com/watch?v=eS_k6L638k0 Extension to stochastic models where KAN builds the distribution 2023: https://www.youtube.com/watch?v=0hhJIpzxPR0 https://www.youtube.com/watch?v=0hhJIpzxPR0
- apolar 2y agoArchive article 2023 research more generic case, where splines is considered as particular case of the basis functions.
- SpaceManNabs 2y agoHow does back propagation work now? Do these suffer from vanishing or exploding gradients?
- nextaccountic 2y agoAt page 6 it explains how they did back propagation https://arxiv.org/pdf/2404.19756 https://arxiv.org/pdf/2404.19756 (and in page 2 it says that previous efforts to leverage Kolmogorov-Arnold representation failed to use backpropagation), so maybe using backpropagation to train multilayer networks with this architecture is their main contribution? > Unsurprisingly, the possibility of using Kolmogorov-Arnold representation theorem to build neuralnetworks has been studied [8, 9, 10, 11, 12, 13]. However, most work has stuck with the original depth-2 width-(2n + 1) representation, and did not have the chance to leverage more modern techniques (e.g., back propagation) to train the networks. Our contribution lies in generalizing the original Kolmogorov-Arnold representation to arbitrary widths and depths, revitalizing and contextualizing it in today’s deep learning world, as well as using extensive empirical experiments to highlight its potential role as a foundation model for AI + Science due to its accuracy and interpretability.
- goggy_googy 2y agoNo, the activations are a combination of the basis function and the spline function. It's a little unclear to me still how the grid works, but it seems like this shouldn't suffer anymore than a generic relu MLP.
- nico 2y agoLooks super interesting I wonder how many more new architectures are going to be found in the next few years
- Maro 2y agoInteresting! Would this approach (with non-linear learning) still be able to utilize GPUs to speed up training?
- diwank 2y agoSeconded. I’m guessing you could create an implementation that is able to do that and then write optimised triton/cuda kernels to accelerate them but need to investigate further
- diwank 2y agoIt’d be really cool to see a transformer with the MLP layers swapped for KANs and then compare its scaling properties with vanilla transformers
- gautam5669 2y agoThis is the first thought came to my mind too. Given its sparse, Will this be just replacement for MoE.
- samus 2y agoMoE is mostly used to enable load balancing since it makes it possible to put experts on different GPUs. This isn't so easy to do with a monolithic, but sparse layer.
- mp187 2y agoWhy was this your first thought? Is a limiting factor to transformers the MLP layer? I thought the bottleneck was in the renormalization part.
- brrrrrm 2y agoAt small input size, yes the MLP dominates compute. At large input attention matters more
- bart1ett 2y agoAfter trying this out with the fourier implementation above, swapping MLP/Attention Linear layers for KANs (all, or even a few layers) produces diverging loss. KANs don't require normalization for good forward pass dynamics, but may be trickier to train in a deep net.
- thesz 2y agoNote that KANs use LBFGS, which is second-order optimization method. My experience with the use of second-order features suggests that simple gradient descent often leads to divergence.
- cbsmith 2y agoFeels like someone stuffed splines into decision trees.
- ALittleLight 2y agoI can't assess this, but I do worry that overnight some algorithmic advance will enhance LLMs by orders of magnitude and the next big model to get trained is suddenly 10,000x better than GPT-4 and nobody's ready for it.
- DeathArrow 2y ago>some algorithmic advance will enhance LLMs by orders of magnitude I would worry if I'd own Nvidia shares.
- krasin 2y agoActually, that would be fantastic for NVIDIA shares; 1. A new architecture would make all/most of these upcoming Transformer accelerators obsolete => back to GPUs. 2. Higher performance LLMs on GPUs => we can speed up LLMs with 1T+ parameters. So, LLMs become more useful, so more of GPUs would be purchased.
- mindcrime 2y ago1. A new architecture would make all/most of these upcoming Transformer accelerators obsolete => back to GPUs. There's no guarantee that that is what would happen. The right (or wrong, depending on your POV) algorithmic breakthrough might make GPU's obsolete for AI, by making CPU's (or analog computing units, or DSP's, or "other") the preferred platform to run AI.
- edmara 2y agoAssuming there is a development that makes GPUs obsolete, I think it's safe to assume that what will replace them at scale will still take the form dedicated AI card/rack 1. Tight integration necessary for fundamental compute constraints like memory latency. 2. Economies of scale 3. Opportunity cost to AI orgs. Meta, OpenAI etc want 50k h100s to arrive in shipping container and plug in so they can focus on their value-add. Everyone will have to readjust to this paradigm. Even if next get AI runs better on CPU, Intel won't suddenly be signing contracts to sell 1,000,000 xeons and 1,000,000 motherboards etc Also, Nvidia have 25bn cash in hand and almost 10 billion yearly r&d spend. They've been an AI-first company for over a decade now, they're more prepared to pivot than anyone else Edit: nearly forgot - Nvidia can issue 5% new stocks and raise 100B like it's nothing.
- mxwsn 2y agoFrom the preprint - 100 input dimensions is considered "high", and most problems considered have 5 or fewer input dimensions. This is typical of physics-inspired settings I've seen considered in ML. The next step would be demonstrating them on MNIST, which, at 784 dimensions is tiny by modern standards.
- wongarsu 2y agoIn actual business processes there are lots of ML problems with fewer than 100 input dimensions. But for most of them decision trees are still competitive with neural networks or even outperform them.
- galangalalgol 2y agoThe aid to explainability seems at least somewhat compelling. Understanding what a random forest did isn't always easy. And if what you want isn't the model but the closed form of what the model does, this could be quite useful. When those hundred input dimensions interact nonlinearly in a million ways thats nice. Or more likely I'd use it when I don't want to find a pencil to derive the closed form of what I'm trying to do.
- trwm 2y agoBusiness processes don't need deep learning in the first place. It is just there because hype.
- JoshCole 2y agoCompetent companies tend to put a lot of effort into building data analysis tools. There will often be A/B or QRT frameworks in place allowing deployment of two models, for example, the new deep learning model, and the old rule based system. By using the results from these experiments in conjunction with typical offline and online evaluation metrics one can begin to make statements about the impact of model performance on revenue. Naturally model performance is tracked through many offline and online metrics. So people can and do say things like "if this model is x% more accurate then that translates to $y million dollars in monthly revenue" with great confidence. Lets call someone working at such a company Bob. A restatement of your claim is that Bob decided to launch a model to live because of hype rather than because he could justify his promotion by pointing to the millions of dollars in increased revenue his switch produced. Bob of course did not make his decision based on hype. He made his decision because there were evaluation criteria in place for the launch. He was literally not allowed to launch things that didn't improve the system according to the evaluation criteria. As Bob didn't want to be fired for not doing anything at the company, he was forced to use a tool that worked to improve the evaluation according to the criteria that was specified. So he used the tool that worked. Hype might provide motivation to experiment, but it doesn't justify a launch. I say this as someone whose literally seen transitions from decision trees to deep learning models on < 100 feature models which had multi-million dollar monthly revenue impacts.
- krasin 2y agoI've spent some time playing with their Jupyter notebooks. The most useful (to me, anyway) is their Example_3_classfication.ipynb ([1]). It works as advertised with the parameters selected by the authors, but if we modified the network shape in the second half of the tutorial (Classification formulation) from (2, 2) to (2, 2, 2), it fails to generalize. The training loss gets down to 1e-9, while test loss stays around 3e-1. Getting to larger network sizes does not help either. I would really like to see a bigger example with many more parameters and more data complexity and if it could be trained at all. MNIST would be a good start. Update: I increased the training dataset size 100x, and that helps with the overfitting, but now I can't get training loss below 1e-2. Still iterating on it; a GPU acceleration would really help - right now, my progress is limited by the speed of my CPU. 1. https://github.com/KindXiaoming/pykan/blob/master/tutorials/Example_3_classfication.ipynb https://github.com/KindXiaoming/pykan/blob/master/tutorials/...
- londons_explore 2y ago> I would really like to see a bigger example This. I don't think toy examples are useful for modern ML techniques. If you tested big ideas in ML (transformers, LSTM's, ADAM) on a training dataset of 50 numbers trying to fit a y=sin(x) curve, I think you'd wrongly throw these ideas out.
- samus 2y agoIt's possible to run it on CUDA. One of their examples shows how. But I found it's slower than on CPU. I'm actually not really surprised since running something on GPU is not a guarantee that it's gonna be fast, especially when lots of branching is involved. Unfortunately, I had to modify KAN.py and KANLayer.py to make it work as not all relevant tensor are put on the correct device. In some places the formatting even suggests that there was previously a device argument.
- krasin 2y agoUpdate2: got it to 100% training accuracy, 99% test accuracy with (2, 2, 2) shape. Changes: 1. Increased the training set from 1000 to 100k samples. This solved overfitting. 2. In the dataset generation, slightly reduced noise (0.1 -> 0.07) so that classes don't overlap. With an overlap, naturally, it's impossible to hit 100%. 3. Most important & specific to KANs: train for 30 steps with grid=5 (5 segments for each activation function), then 30 steps with grid=10 (and initializing from the previous model), and then 30 steps with grid=20. This is idiomatic to KANs and covered in the Example_1_function_fitting.ipynb: https://github.com/KindXiaoming/pykan/blob/master/tutorials/Example_1_function_fitting.ipynb https://github.com/KindXiaoming/pykan/blob/master/tutorials/... Overall, my impressions are: - it works! - the reference implementation is very slow. A GPU implementation is dearly needed. - it feels like it's a bit too non-linear and training is not as stable as it's with MLP + ReLU. - Scaling is not guaranteed to work well. Really need to see if MNIST is possible to solve with this approach. I will definitely keep an eye on this development.
- arianvanp 2y agoThis really reminds me of petrinets but an analog version? But instead of places and discrete tokens we have activation functions and signals. You can only trigger a transition if an activation function (place) has the right signal (tokens).
- montebicyclelo 2y agoThe success we're seeing with neural networks is tightly coupled with the ability to scale - the algorithm itself works at scale (more layers), but it also scales well with hardware, (neural nets mostly consist of matrix multiplications, and GPUs have specialised matrix multiplication acceleration) - one of the most impactful neural network papers, AlexNet, was impactful because it showed that NNs could be put on the GPU, scaled and accelerated, to great effect. It's not clear from the paper how well this algorithm will scale, both in terms of the algorithm itself (does it still train well with more layers?), and ability to make use of hardware acceleration, (e.g. it's not clear to me that the structure, with its per-weight activation functions, can make use of fast matmul acceleration). It's an interesting idea, that seems to work well and have nice properties on a smaller scale; but whether it's a good architecture for imagenet, LLMs, etc. is not clear at this stage.
- dist-epoch 2y ago> with its per-weight activation functions Sounds like something which could be approximated by a DCT (discrete cosine transform). JPEG compression does this, and there are hardware accelerations for it. > can make use of fast matmul acceleration Maybe not, but matmul acceleration was done in hardware because it's useful for some problems (graphics initially). So if these per weight activations functions really work, people will be quick to figure out how to run them in hardware.
- WithinReason 2y agoLooks very interesting, but my guess would be that this would run into the problem of exploding/vanishing gradients at larger depths, just like TanH or sigmoid networks do.
- yobbo 2y agohttps://kindxiaoming.github.io/pykan/intro.html https://kindxiaoming.github.io/pykan/intro.html At the end of this example, they recover the symbolic formula that generated their training set: exp(x₂² + sin(3.14x₁)). It's like a computation graph with a library of "activation functions" that is optimised, and then pruned. You can recover good symbolic formulas from the pruned graph. Maybe not meaningful for MNIST.
- beagle3 2y agoI wonder if Breiman’s ACE (alternating conditional expectation) is useful as a building block here. It will easily recover this formula, because it is separable under the log transformation (which ACE recovers as well). But ACE doesn’t work well on unseparable problems - not sure how well KAN will.
- GistNoesis 2y agoI quickly skimmed the paper, got inspired to simplify it, and created some Pytorch Layer : https://github.com/GistNoesis/FourierKAN/ https://github.com/GistNoesis/FourierKAN/ The core is really just a few lines. In the paper they use some spline interpolation to represent 1d function that they sum. Their code seemed aimed at smaller sizes. Instead I chose a different representation, aka fourier coefficients that are used to interpolate the functions of individual coordinates. It should give an idea of Kolmogorov-Arnold networks representation power, it should probably converge easier than their spline version but spline version have less operations. Of course, if my code doesn't work, it doesn't mean theirs doesn't. Feel free to experiment and publish paper if you want.
- sevagh 2y agoDoes your code work? Did you train it? Any graphs? >Of course, if my code doesn't work, it doesn't mean theirs doesn't. But, _does_ it work?
- AbrahamParangi 2y agoWhen I played around with implementing this last night I found using a radial basis function instead of Fourier coefficients (I tried the same, nice and parallel and easy to write) to be more well behaved in training networks of depth greater than 2.
- droidlyx 2y agoHi Noesis, I just noticed that your implementation, combined with the efficientKAN by Blealtan (https://github.com/Blealtan/efficient-kan https://github.com/Blealtan/efficient-kan), results in a structure very similar to Siren(MLP with Sin activations). efficientKAN first computes the common basis functions for all the edge activations and the output can be calculated with a linear combination of the basis. If the basis functions are fourier, then a KAN layer can be viewed as a linear layer with fixed weights + Sin activation + a linear layer with learnable weights, which is a special form of Siren. I think this may show some connection between KAN and MLP.
- bionhoward 2y ago
- ubj 2y agoVery interesting! Kolmogorov neutral networks can represent discontinuous functions [1], but I've wondered about how practically applicable they are. This repo seems to show that they have some use after all. [1]: https://arxiv.org/abs/2311.00049 https://arxiv.org/abs/2311.00049
- nyrikki 2y agoNot for discontinuous functions, as your paper explains, we know that g exists for discontinuous bounded, but nothing to find it with. > A practical construction of g in cases with discontinuous bounded and un- bounded functions is not yet known. For such cases Theorem 2.1 gives only a theoretical understanding of the representation problem. This is because for the representation of discontinuous bounded functions we have derived (2.1) from the fact that the range of the operator Z∗ is the whole space of bounded functions B(Id). This fact directly gives us a formula (2.1) but does not tell how the bounded one-variable function g is attained. For the representation of unbounded functions we have used a linear extension of the functional F , existence of which is based on Zorn’s lemma (see, e.g., [19, Ch. 3]). Application of Zorn’s lemma provides no mechanism for practical construction of such an extension. Zorn’s lemma helps to assert only its existence. If you look at the OP post arxiv link, you will see they are using splines . https://arxiv.org/abs/2404.19756 https://arxiv.org/abs/2404.19756 Still interesting and potentially useful, but not useful for discontinuous functions without further discoveries. If I am wrong please provide a link, it is of great interest to me.
- reynoldss 2y agoPerhaps a hasty comment but linear combinations of B-splines are yet another (higher-degree) B-spline. Isn't this simply fitting high degree B-splines to functions?
- Lichtso 2y agoThat would be true for a single node / single layer. But once the output of one layer is fed into the input of the next it is not just a linear combination of splines anymore.
- cs702 2y agoIt's so refreshing to come across new AI research different from the usual "we modified a transformer in this and that way and got slightly better results on this and that benchmark." All those new papers proposing incremental improvements are important, but... everyone is getting a bit tired of them. Also, anecdotal evidence and recent work suggest we're starting to run into fundamental limits inherent to transformers, so we may well need new alternatives.[a] The best thing about this new work is that it's not an either/or proposition. The proposed "learnable spline interpolations as activation functions" can be used in conventional DNNs, to improve their expressivity. Now we just have to test the stuff to see if it really works better. Very nice. Thank you for sharing this work here! --- [a] https://news.ycombinator.com/item?id=40179232 https://news.ycombinator.com/item?id=40179232
- glebnovikov 2y ago> Everyone is getting tired of those papers. This is science as is :) 95% percent will produce mediocre-to-nice improvements to what we already have so there were reserachers that eventually grow up and do something really exciting
- godelski 2y agoNothing wrong with incremental improvements. Giant leaps (almost always) only happen because of a lack of your niche domain expertise. And I mean niche niche
- deleted 2y ago[deleted]
- deleted 2y ago[deleted]
- godelski 2y agoThere's a ton actually. Just they tend to go through extra rounds of review (or never make it...) and never make it to HN unless there's special circumstances (this one is MIT and CIT). Unfortunately we've let PR become a very powerful force (it's always been a thing, but seems more influential now). We can fight against this by up voting things like this and if you're a reviewee, not focusing on sota (it's clearly been gamed and clearly leading us in the wrong direction)
- keynesyoudigit 2y agoEli5: why aren't these more popular and broadly used?
- OisinMoran 2y agoBecause they have just been invented!
- wbeckler 2y ago60 years ago
- admin1231111 2y ago[flagged]
- admin1231111 2y ago[flagged]
- Lichtso 2y ago1. Interestingly the foundations of this approach and MLP were invented / discovered around the same time about 66 years ago: 1957: https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_representation_theorem https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Arnold_repr... 1958: https://en.wikipedia.org/wiki/Multilayer_perceptron https://en.wikipedia.org/wiki/Multilayer_perceptron 2. Another advantage of this approach is that it has only one class of parameters (the coefficients of the local activation functions) as opposed to MLP which has three classes of parameters (weights, biases, and the globally uniform activation function). 3. Everybody is talking transformers. I want to see diffusion models with this approach.
- xpe 2y agoYes, #2 is a difference. But what makes it an advantage? One might argue this via parsimony (Occam’s razor). Is this your thinking? / Anything else?
- tripplyons 2y agoTo your 3rd point, most diffusion models already use a transformer-based architecture (U-Net with self attention and cross attention, Vision Transformer, Diffusion Transformer, etc.).
- trwm 2y agoBiases are just weights on an always on input. There isn't much difference between weights of a linear sum and coefficients of a spline.
- Lichtso 2y ago> Biases are just weights on an always on input. Granted, however this approach does not require that constant-one input either. > There isn't much difference between weights of a linear sum and coefficients of a function. Yes, the trained function coefficients of this approach are the equivalent to the trained weights of MLP. Still this approach does not require the globally uniform activation function of MLP.
- 2y ago
- brrrrrm 2y agodoesn't KA representation require continuous univariate functions? do B-splines actually cover the space of all continuous functions? wouldn't... MLPs be better for the learnable activation functions?
- hansvm 2y agoThe paper [0] is pretty good for handling questions like those. > doesn't KA representation require continuous univariate functions? All multivariate continuous functions (on a bounded domain) can be represented as compositions of addition and univariate continuous functions. Much like an MLP, you can also approximate discontinuous functions well on most of the domain (learning a nearby continuous function instead). > do B-splines actually cover the space of all continuous functions Much like an MLP, you can hit your favorite accuracy bound with more control points. > wouldn't... MLPs be better for the learnable activation functions Perhaps. A B-spline is comparatively very fast to compute. Also, local training examples have global impacts on an MLP's weights. That's good and bad. One property you would expect while training a KAN in limited data regimes is that some control points are never updated, leading to poor generalization due to something like a phase shift as you cross over control points (I think the entropy-based regularizer they have in the paper probably solves that, but YMMV). The positive side of that coin is that you neatly side-step catastrophic forgetting. [0] https://arxiv.org/abs/2404.19756 https://arxiv.org/abs/2404.19756
- brrrrrm 2y ago“Increasing control points” hides a lot under the covers here. Your answer and the paper provide virtually no reason to believe one type of continuous function approximation is better than another. The comparisons made are superficial and only serve to address contrived issues like representing sinusoidal function families concisely. It’s weird to just ignore MLPs when approximating a continuous univariate function. But if the paper did use MLPs theyd have ended up with something that looks a lot more like conventional neural networks, so maybe thats why?
- hansvm 2y ago
- ComplexSystems 2y agoVery interesting! Could existing MLP-style neural networks be put into this form?