9 ms·

> The big problem is that NEAT can't leverage a GPU effectively at scale (arbitrary topologies vs bipartite graphs) Is that true? These graphs can be transform

by epr 2y ago

> The big problem is that NEAT can't leverage a GPU effectively at scale (arbitrary topologies vs bipartite graphs) Is that true? These graphs can be transformed into a regular tensor shape with zero weights on unused connections. If you were worried about too much time/space used by these zero weights, you could introduce parsimony pressure related to the dimensions of transformed tensors rather than on the bipartite graph.

Y_Y 2y ago
Or even use CuSparse, if you don't mind a little bit of extra work over normal cudnn. https://developer.nvidia.com/cusparse https://developer.nvidia.com/cusparse
PartiallyTyped 2y ago
We can also show that sparse NNs under some conditions are ensembles of discrete subnets, and the authors of the original dropout paper argue that [dropout] effectively creates something akin to a forest of subnets all in "superposition".
HelloNurse 2y ago
Another plausible strategy to neutralize arbitrary topologies: compile individual solutions or groups of similar solutions into big compute shaders that execute the network and evaluate expensive fitness functions, with parallel execution over multiple test cases (aggregated in postprocessing) and/or over different numerical parameters for the same topology.
Rhapso 2y ago
Just because you can pack the topology into a sparse matrix doesn't make it actually go faster. Sparse matrices often don't see good speedup from GPUs. In addition, each network is unique, each neuron can have an entirely different activation function, and the topology is constantly changing. You will burn a lot on constantly re-packing into matrices that then don't see the same speedups a more wasteful topology pretends to have. On the flip-side out narrative of "speedup" is on bipartite graphs crunch faster in gpus and it might not be the same if the basis is actually utility of behaviors generated by the networks. A cousin thread explores this better.

by epr 2y ago

Y_Y 2y ago

Or even use CuSparse, if you don't mind a little bit of extra work over normal cudnn. https://developer.nvidia.com/cusparse https://developer.nvidia.com/cusparse

PartiallyTyped 2y ago

We can also show that sparse NNs under some conditions are ensembles of discrete subnets, and the authors of the original dropout paper argue that [dropout] effectively creates something akin to a forest of subnets all in "superposition".

HelloNurse 2y ago

Another plausible strategy to neutralize arbitrary topologies: compile individual solutions or groups of similar solutions into big compute shaders that execute the network and evaluate expensive fitness functions, with parallel execution over multiple test cases (aggregated in postprocessing) and/or over different numerical parameters for the same topology.

Rhapso 2y ago

Just because you can pack the topology into a sparse matrix doesn't make it actually go faster. Sparse matrices often don't see good speedup from GPUs. In addition, each network is unique, each neuron can have an entirely different activation function, and the topology is constantly changing. You will burn a lot on constantly re-packing into matrices that then don't see the same speedups a more wasteful topology pretends to have. On the flip-side out narrative of "speedup" is on bipartite graphs crunch faster in gpus and it might not be the same if the basis is actually utility of behaviors generated by the networks. A cousin thread explores this better.