AI training time is at a point in an exponential where more throughput isn't going to advance functionality much at all.

If you start naively without any library that avoids the problem then memory access is the problem. Have a look at how much ...