DIGITS DevBox
Contents
Documentation
The documentation from NVIDIA is here:
Hardware specs from other builds:
- https://www.azken.com/download/DIGITS_DEVBOX_DESIGN_GUIDE.pdf
- https://cellmatiq.com/?p=155
- http://graphific.github.io/posts/building-a-deep-learning-dream-machine/
The best instructions that I could find:
Some firms, including Lamdba Labs, Bizon-tech, are selling variants on them, but the details on their specs are limited (the MoBo and config details are missing entirely):
- https://lambdalabs.com/deep-learning/workstations/4-gpu
- https://bizon-tech.com/us/bizon-g3000
- http://deeplearningbox.com/
- https://www.amazon.com/Lambda-Deep-Learning-DevBox-Preinstalled/dp/B01BCDK1KC (currently unavailable)
Unfortunately, the form to get help from NVIDIA is closed:
- https://info.nvidianews.com/early_access_nvidia_3_15.html
- https://www.reddit.com/r/buildapc/comments/3gewmz/build_complete_nvidia_digits_devbox/
- https://www.pyimagesearch.com/2016/06/06/hands-on-with-the-nvidia-digits-devbox-for-deep-learning/
At around $15k (the Lamdba variants go from $10k to $23k), buying one is prohibitive for most people. But the parts cost is perhaps $5k now for the original spec.
Hardware
We mostly followed the original hardware spec from NVIDIA, updating the capacity of the drives and other minor things, as we had many of these parts available as salvage from other boxes. Though we had to buy the ASUS X99-E WS motherboard (as well as some new drives) just for this project.
We opted to use a Xeon e5-2620v3 processor, rather than the Core i7-5930K (which we did have available). Both support 40 channels and mount in the LGA 2011-v3 socket, and both have 6 cores, 15mb caches etc. The i7 has a faster clock speed but the Xeon takes registered (buffered), ECC DDR4 RDIMMs, which means we can put 256Gb on the board, rather than just 64Gb. For the GPUs we have a TITAN RTX and an older TITAN Xp available to start, and we can add a 1080Ti later, or buy some additional GPUs if needed. We also put the whole thing in a Rosewill RSV-L4000 case.
| Quantity | Part |
|---|---|
| 1 | ASUS X99-E WS/USB 3.1 LGA 2011-v3 Intel X99 SATA 6Gb/s USB 3.1 USB 3.0 CEB Intel Motherboard |
| 1 | Intel Haswell Xeon e5-2620v3, 6 core @ 2.4ghz, 6x256k level 1 cache, 15mb level 2 cache, socket LGA 2011-v3 |
| 8 | Crucial DDR4 RDIMM, 2133Mhz , Registered (buffered) and ECC, 32GB |
| 1 | NVIDIA TITAN RTX DirectX 12 900-1G150-2500-000 SB 24GB 384-Bit GDDR6 HDCP Ready Video Card |
| 1 | NVIDIA TITAN Xp Graphics Card (900-1G611-2530-000) |
| 1 | SAMSUNG 970 EVO PLUS 500GB Internal Solid State Drive (SSD) MZ-V7S500B/AM |
| 1 | Samsung 850 EVO 500GB 2.5-Inch SATA III Internal SSD (MZ-75E500/EU) |
| 3 | WD Red 4TB NAS Hard Disk Drive - 5400 RPM Class SATA 6Gb/s 64MB Cache 3.5 Inch - WD40EFRX |
| 1 | DVDRW: Asus 24x DVD-RW Serial-ATA Internal OEM Optical Drive DRW-24B1ST |
| 1 | EVGA SuperNOVA 1600 T2 220-T2-1600-X1 80+ TITANIUM 1600W Fully Modular EVGA ECO Mode Power Supply |
| 1 | Rosewill RSV-L4000 - 4U Rackmount Server Case / Chassis - 8 Internal Bays, 7 Cooling Fans Included |
| 1 | Rosewill RSV-SATA-Cage-34 - Hard Disk Drives - Black, 3 x 5.25" to 4 x 3.5" Hot-Swap - SATA III / SAS - Cage |
| 1 | Rosewill RDRD-11003 2.5" SSD / HDD Mounting Kit for 3.5" Drive Bay w/ 60mm Fan |
| 3 | Corsair ML120 PRO LED CO-9050043-WW 120mm Blue LED 120mm Premium Magnetic Levitation PWM Fan |
| 2 | ARCTIC F8 PWM Fluid Dynamic Bearing Case Fan, 80mm PWM Speed Control, 31 CFM at 22dBA |
Old notes on a prior look at a GPU Build are on the wiki too.
There weren't any particularly noteworthy things about the hardware build. The GPUs need to go in slots 1 and 3, which means they sit tight on each other. I put the Titan XP in slot 1 (and plugged the monitor into its HDMI port), because then the fans for the Titan RTX (which I expect will get heavier use) are in the clear.
The initial BIOS boot was weird - the machine ran at full power for a short period then powered off multiple times before finally giving a single system beep and loading the BIOS. It may have been memory checking or some such.
BIOS
The machine boots to BIOS. I made the following changes:
- The GPUs are being recognized - I found them in a tool
- All of the SATA drives are being recognized.
- I set the three hard disks to hotswap enable.
- There doesn't seem to be anyway to disable secure boot or boot outside of UEFI, which is worrying. I listed the OS as OS rather than windows, and set enhanced mode to disabled.
Notes:
- We will do RAID 5 array in software, rather using X99 through the BIOS.
- The m.2 drive will be used as a cache for the RAID 5 array, using bcache. I couldn't find it in the BIOS.
Software
Install Ubuntu 18.04 (note that the original DiGIT DevBox ran 14.04)
