Cons of multiple GPUs:
*Adds a lot of complexity.
=== K80, NVLink ===
*NVLink can link between CPU and GPU for increase in speed, but only with the CPU IBM POWER8+.
*NVLink can link between GPU and GPU as a replacement for SLI with other CPUs, but this is not super relevant to tensorflow, even if trying to parallelize across one model.
==Misc. Parts==