GPU Build
Jump to navigation
Jump to search
| GPU Build | |
|---|---|
| Project Information | |
| Project Title | GPU Build |
| Owner | Oliver Chang, Kyran Adams |
| Start Date | |
| Deadline | |
| Primary Billing | |
| Notes | |
| Has project status | Active |
| Copyright © 2016 edegan.com. All Rights Reserved. | |
Contents
Single vs. Multi GPU
- GTX 1080 Ti Specs
- Since we are using Tensorflow, it doesn't scale well to multiple GPUs for a single model
- Which GPU for deep learning (04/09/2017)
- "I quickly found that it is not only very difficult to parallelize neural networks on multiple GPUs efficiently, but also that the speedup was only mediocre for dense neural networks. Small neural networks could be parallelized rather efficiently using data parallelism, but larger neural networks... received almost no speedup."
- Possible other use of multiple GPUs: training multiple different models simultaneously, "very useful for researchers, who want try multiple versions of a new algorithm at the same time."
- This source recommends GTX 1080 Tis and does cost analysis of it
- If the network doesn't fit in the memory of one GPU (11 GB),
- Want to get two graphics cards, one for development, one (crappy card) for operating system [1]
- Intra-model parallelism: If a model has long, independent computation paths, then you can split the model across multiple GPUs and have each compute a part of it. This requires careful understanding of the model and the computational dependencies.
- Replicated training: Start up multiple copies of the model, train them, and then synchronize their learning (the gradients applied to their weights & biases).
TL;DR
Pros of multiple GPUs:
- Able to train multiple networks at once (either copies of the same network or modified networks). Allows for running long experiments while running new ones
- Possible speed ups if the network can be split up (and is big enough), but tensorflow is not great for this
- More memory for huge batches (not sure if necessary)
Cons of multiple GPUs:
- Adds a lot of complexity.
Misc. Parts
- Cases: Rosewill 1.0 mm Thickness 4U Rackmount Server Chassis, Black Metal/Steel RSV-L4000[2]
- DVDRW (Needed?): Asus 24x DVD-RW Serial-ATA Internal OEM Optical Drive DRW-24B1ST [3]
- Keyboard and Mouse: AmazonBasics Wired Keyboard and Wired Mouse Bundle Pack [4]
Other Builds/Guides
- Deep learning box for $1700 (Discussion)
- A Full Hardware Guide to Deep Learning
- Cheap build
- How to build a GPU deep learning machine
- Deep Learning Computer Build useful tips, long
Questions to ask:
- Approx. dataset/batch size
- Network card?
- DVD drive?
- How much RAM/storage needed?
- Do we need to get the same motherboard/CPU? There are cheaper/better CPUs. For example, V4 of same CPU is same price.
Single GPU Build
Double GPU Build
Motherboard
- Should have enough PCIe slots
- Motherboards: ASUS Z10PE-D16 [5], Dual LGA 2011 R3, DDR4 - Up to 32GB RDIMM, 16 slots
CPU/Fan
- Not a huge deal, but used for data preparation
- If using multiple GPUs, at least one core (two threads) per GPU
- Chips: Intel Haswell Xeon e5-2620v3, 6 core @ 2.4ghz, 6x256k level 1 cache, 15mb level 2 cache, socket LGA 2011-v3 [6]
- CPU Fans: Intel Thermal Solution Cooling Fan for E5-2600 Processors BXSTS200C [7]
GPU
- 2x GTX 1080 Ti
- Aspeed AST2400 with 32MB VRAM (comes with motherboard)
RAM
- At least twice as much RAM as GPUs (2 * 2 * 11 GB [GTX 1080 Ti size] = 32 GB)
- RAM: Crucial - 32GB (2 x 16GB) Registered DDR4-2133 Memory [8]
PSU
- Some say 1.5x-2x wattage of GPU+CPU, some say GPU+CPU+100W
- PSUs: Corsair RM Series 850 Watt ATX/EPS 80PLUS Gold-Certified Power Supply - CP-9020056-NA RM850 [9]
