🧙 Prayag Bhakar

Welcome to my blog ✨

Mon, 15 Jan 2024 02:52:41 GMT

My opinions are in beta testing, subject to crashes and unexpected reboots! 🔄 Posting here whenever for whoever cares to see. Come back to see what I’m cookin’ up 🥘👩‍🍳.

Warning

Use of graph view and navigation requires browser hardware acceleration to be enabled. Performance may be bad if disabled.

Links δ

Tech Stack δ

Docker Refresher 🍹

Sun, 14 Jan 2024 00:00:00 GMT

Crack open a refreshing beverage and freshen up with various Docker and Docker compose features.

Docker CLI δ

# show all containers
$ docker ps
# list all containers, including stopped ones
$ docker container ls --all
 
# stop a container
$ docker stop example
# start a stopped container
$ docker start example
 
# restart a running container
$ docker restart <container-id>
 
# remove a stopped container
$ docker rm example
# only stopped containers can be removed
$ docker rm <stopped-container-id>

`Dockerfile` δ

Below is an sudo example of a multi stage Dockerfile.

FROM alpine:latest as builder
# set the working directory
WORKDIR /build
# run any shell command during build
RUN apk add --no-cache npm
# copy files and folders
COPY ./source /build
# commands to run when the contianer starts
CMD ["npm", "run", "build"]
 
FROM alpine:latest
WORKDIR /serv
RUN apk add --no-cache npm
# copy files from different stages
COPY --from=builder /build /serv 
# what ports the container should listen to at runtime
EXPOSE 420
# this is the contianer's main executable
ENTRYPOINT ["npm", "start"]

In order to build and run this Dockerfile.

$ docker build . --tag example:1.0
 
# this will pulish port to 69 on your host irregardles of your firewall config
# use --publish 127.0.0.1:96:420 to bind to the host
$ docker run --publish 69:420 example
 
# open an interactive shell (like connecting to a server)
# or replace sh with any command
$ docker exec --interactive --tty <container-id> sh
# exit the container
$ exit

`docker-compose` δ

As you collect Dockerfiles like the gremlin you are, it will eventually get easier to orchestrate your Docker deployments.

version: '3.9'
 
services:
  notebook:
    image: jupyter/minimal-notebook
    container_name: jupyter
    hostname: jupyter
    restart: always
    networks: 
      - tunnel
    ports:
      - "90:8888"
    volumes:
      - jupyter_data:/home/jupyter
    runtime: nvidia
    user: root
    command: "start-notebook.sh"
    environment:
      NB_USER: jupyter
      NB_UID: 1000
      NB_GID: 1000
      CHOWN_HOME: 'yes'
      CHOWN_HOME_OPTS: -R
      NVIDIA_VISIBLE_DEVICES: 'all'
 
volumes: 
  jupyter_data:
 
networks:
  tunnel:
    driver: bridge

Being S.M.A.R.T. 🧠 About Used Storage 💾

Mon, 25 Dec 2023 00:00:00 GMT

Machine Learning and Deep Learning datasets are MASSIVE. Just to put it into perspective, here are a couple of popular datasets across different modalities.

Dataset	Approx. Size	Description
LAION 5B	9.5 TB	dataset of labeled imaged used to train Open Clip
The Pile	825 GB	open source dataset of scraped internet data for language modeling
Mozilla’s Common Voice	80 GB	a human voice dataset used to train DeepSpeech
Meta’s SA-1B	10 TB	dataset of annotated image segments used to train Segment Anything Model (SAM)
YouTube 8M	80 GB	dataset of annotated youtube videos and their link
Berkeley’s BDD100K	1.9 TB	dataset of annotated driving videos

And all this data is before it has been converted into tokens. That will be even more terabytes worth of data if you chose to preprocess and save the tokens before training. So high density data storage was a core requirement when it came to building 🪐 Apollo 🛸.

However with new Hard Disk Drives (HDDs) costing more than $20 per Terabyte, the total costs for storage was causing the budget to grow out of control. In the end I ended up going with refurbished HDDs with one leg in the grave 😱 (at least according to the internet). But how do we make educated decisions about life left on a Hard Drive and avoid data loss due to hardware failures?

Warning

Some newer HDDs have a fun little power disable feature (PWDIS). I’m sure the engineers who made it thought it was useful, but what it means for you is that you will need an ATA power adapter to get your drives to properly draw power.

`badblocks` :: The Bad Boy in Town δ

badblocks is a utility to find, you guess it, bad blocks/sectors. As any drive gets used it will experience natural wear and tear causing the drive to start to fail. Failures will show up as sectors of the drive become unusable as they cannot reliability read and write data to that block anymore. Since HDDs work by exploiting the physics of magnets to make a plate shaped magnetic rocks spin around super fast, this natural wear and tear is especially problematic.

First lets install badblocks and find the drives that we want to test.

$ yay -S e2fsprogs
$ fdisk -l

Lets assume we want to test drive /dev/sdx. If you try to run badblocks as it, will throw Value too large for defined data type errors on large drives. To get around this, we will use the drive’s block size and tell badblocks to batch the read and write tests to fit in the block size. Lets get /dev/sdx’s recommended block size.

$ blockdev --getbsz /dev/sdx

For /dev/sdx the block size was 4069. Now lets run badblocks to scan our HDD.

$ badblocks -t random -w -s -b 4069 /dev/sdx

Warning

Running badblocks with the -w option is a destructive action that will overwrite any data on the drive. Backup any data that you don’t want to lose or run badblocks non destructively! Check out the wiki for more information

Tip

Since badblocks can more than a day to run through a larger drives you can schedule the job to run in the background with nohup, &, and piping the output to a file. System reboots will still stop the background job.
$ nohup badblocks -t random -w -s -b 4096 /dev/sda > dev.sdx.txt &

Tip

If badblocks fails, you can continue the job by specifying a block to start at.
$ nohup badblocks -t random -w -s -b 4096 /dev/sdx $START_BLOCK > dev.sda.2.txt &

Making S.M.A.R.T. Decisions δ

While badblocks allows you to actively monitor your drive health by running tests, Self-Monitoring, Analysis, and Reporting Technology (S.M.A.R.T.) helps passively track the health of your HDDs and Solid State Drives (SDDs). This also means that errors caused by the previous owner(s) should also show up here.

Lets start by installing the S.M.A.R.T. tools and finding the drives we want to check.

$ yay -S smartmontools
$ fdisk -l
 
# check if SMART is supported
$ smartctl -i /dev/sdx

Manual Tests δ

You can manually kick off a short or a long S.M.A.R.T. tests. The test will continue in the background and will take a few minutes to several hours.

# see how long tests will take to run
$ smartctl -c /dev/sdx
 
# manual run tests
$ smartctl -t short /dev/sdx
$ smartctl -t long /dev/sdx
# this test finds drive damage during transport
$ smartctl -t conveyance /dev/sdx

Check if the manual test is done.

$ smartctl -l selftest /dev/sdx
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.7-arch1-1] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     43048         -
# 2  Short offline       Completed without error       00%     42997         -
# 3  Short offline       Completed without error       00%     42979         -

Warning

System reboots will cause the S.M.A.R.T. tests to abort.

Understanding S.M.A.R.T. values. δ

Lets start by printing out the S.M.A.R.T. values.

$ smartctl -A /dev/sdx
smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.6.7-arch1-1] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   016    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   134   134   054    Pre-fail  Offline      -       96
  3 Spin_Up_Time            0x0007   154   154   024    Pre-fail  Always       -       448 (Average 405)
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       37
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   128   128   020    Pre-fail  Offline      -       18
  9 Power_On_Hours          0x0012   094   094   000    Old_age   Always       -       43051
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       37
 22 Helium_Level            0x0023   100   100   025    Pre-fail  Always       -       100
192 Power-Off_Retract_Count 0x0032   099   099   000    Old_age   Always       -       1603
193 Load_Cycle_Count        0x0012   099   099   000    Old_age   Always       -       1603
194 Temperature_Celsius     0x0002   176   176   000    Old_age   Always       -       34 (Min/Max 22/49)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       2727

The values that show up will be different for each brand and drive connector. Generally you can tell what values should be high and which should be low, but online wikis will have more updated and detiailed information.

ATA S.M.A.R.T. Values :: These are standard HDDs and SDDs that use SATA connectors.
NVME S.M.A.R.T. Values :: These are drives that use the M.2 connector.
SCISI S.M.A.R.T. Values :: This is used by drives that use the Universal Serial Bus (USB) connector.

Example

In the S.M.A.R.T. values for the drive above, the UDMA_CRC_Error_Count is quite high. The wiki mentions that it could be due to a loose cable connection. And it was!

Automated Tests and Alerts δ

We can automate periodically running S.M.A.R.T. tests to keep up to date health data on our drives. smartd is a service that helps us do exactly that, but first we need to enable it.

$ systemctl enable smartd
$ systemctl start smartd

Lets edit /etc/smartd.conf to configure smartd.

# /etc/smartd.conf
 
# `DEVICESAN` :: run tests on S.M.A.R.T. all enabled drives
# `-a` :: for all S.M.A.R.T. values
# `-o on` :: save offline data
# `-S on` :: save attribute data
# `-n standby,q` :: skips the test if the drive is not active (increases drive lifespan)
# `-s (S/../.././02|L/../../4/03)` :: runs a daily short test and a long test on the 4th of each month
# `-W 4,35,40` :: log changes and dangerous operating temps
# `-m email@gmail.com` :: send an email for alerts
DEVICESCAN -a -o on -S on -n standby,q -s (S/../.././02|L/../../4/03) -W 4,35,40 -m prayag.bhakar@gmail.com

ZFS, the Final Frontier δ

Depending on the results of the badblocks and S.M.A.R.T. tests, you may want to keep using the drive or send it back if its still under warranty. Not perfect drives are acceptable if your are using other tools like the Zettabyte File System (ZFS) to protect your data.

ZFS is a software based Redundant Array of Inexpensive Drives (RAID-Z) solution to help replicate data and maintain data integrity given the unpredictable nature of drive failure and data loss. However, all this power does come with a lot of responsibility. ZFS expects you to know what your doing and isn’t straight forward to expand.

Lets start by installing ZFS and find the drives to use. For the ZFS package we’re going to use Dynamic Kernel Module Support (DKMS) to allow the system to easily swap between driver versions. This will be good if we ever need to upgrade or downgrade the drivers.

# install and check zfs
$ yay -S zfs-dkms zfs-utils
$ modprobe zfs
 
# find the drives
$ fdisk -l

Enable ZFS daemons to auto mount your pool on boot.

$ systemctl enable zfs-import-cache
$ systemctl enable zfs-import-scan
$ systemctl enable zfs-mount
$ systemctl enable zfs.target

To keep things simple I will be using all the drives in a single raidz3 pool. To keep with the theme, lets name the pool black_hole.

# create zfs pool
$ zpool create black_hole raidz3 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj
$ zfs set mountpoint=/mnt/black_hole black_hole
 
# (optional) enable compression
$ zfs set compression=on black_hole
 
# (optional) create encrypted dataset
$ zfs create -o encryption=on -o keyformat=passphrase black_hole/enc

Note

ZFS is smart enough to be able to handle getting disks by either UUID or /dev/sdx. Internally ZFS will map to the UUID of the drive and you will not have to worry about the pool being mounted correctly on reboots.

Check the status of your new pool.

$ zpool status black_hole
$ zfs get all black_hole

Tip

Finding large files and folders to clean up using just the CLI can be difficult. If you are looking for files to clean up you can use the du command to explore your system. sudo will be required to search in folders owned by root.
$ du -ah --max-depth 1
$ du -ah --max-depth 3 ~/downloads

Tip

Check out Learn X in Y minutes, Where X=zfs for a quick start zfs guide.

What Now? δ

If you have ran your tests and decided to use the drive in your ZFS array then theres not much more to do. Now that you have the patient’s history and charts, you can make educated decisions on the security of your data. Keep monitoring ZFS and S.M.A.R.T. values to ensure that your system’s data is secure and use the money you made to buy some ice cream 🍦!

Tip

Just because we have done all this doesn’t mean that your data is fully secure. Always follow the 3-2-1 rule for data and keep backups.

Have 3 copies of your data. Typically one copy is production/live and the other two are for backup.

On 2 different types of mediums. Currently out HDDs are magnetic disks, but other options include flash storage with SDDs, tape storage with Tape Drives, and more!

1 of the copies should be offsite and away from your system. This way if your house catches on fire you can still get the old data out of the cloud or your parent’s basement.

🪐 Apollo 🛸 AI Compute Cluster for the GPU Poor

Sun, 24 Dec 2023 00:00:00 GMT

Hey, Prayag here! Are you GPU Poor but want to spend your money wisely on that shiny brand new deep learning compute cluster? Well stop looking at the newest and greatest and start scavenging for used PC hardware!!

The Build δ

Since the goal of this cluster is to keep the price low while getting as much VRAM and compute we can, used hardware is the only place were going to find any bang for our buck. Newer GPUs and hardware is getting caught up in the Transformer and Crypto hype. So I turned to what countless other server enthusiasts have done before me: buy stuff from good ol’ eBay.

Here’s a breakdown.

Part	Price	Notes
ASUS Pro WS WRX80E-SAGE SE	$620	2x 10G LAN, Supports 4xSLI, Plenty of SATA and IO
Ryzen Threadripper Pro 3945wx	$577	Most AI workload are GPU bound, but if you plan on using this for other server things, consider a beefier CPU.
CPU Cooler	$105
Samsung M386A8K40BM1-CRC	$338	4 x 64GB, I also bought some aftermarket heatsinks to help keep the ram cool.
Nvidia RTX 3090 Founders Edition	$789 3x $772
NVLINK bridge	2x $150	The 4 slot and 3 slot bridges work best here. If your GPU is small enough and you have the proper cooling, you might even be able to go for the A6000 2 slot bridge.
PCIE riser cables	4x $62
m.2 to sata splitter	$45
m.2 SSD	$75	500GB, current boot drive
	$215	2TB, future boot drive
Renewed HDD	6x $106	Cheap Western Digital Ultrastar 10TB because it was used and had PWDIS. I am using SATA power adapters 😱 to get around it but I really need to buy a 🧯 fire extinguisher just in case. ❤️‍🔥
	4x $140	10TB Seagate, learned by lesson about PWDIS this time 🤕
PSU	$328	EVGA SuperNova 1600 P+ with 80+ Platinum rating to help lower the power bill. This is the only big thing I bought new as I did not want to take ant chances.
GPU Mining Case	$35	the motherboard ended up not actually fitting
HDD Rack	2x $35
Cables + Zip ties	$140

For about $7,000 after taxes and delivery fees, we get a WHOPPING 24 threads, 256 GB ram, 96GB VRAM, and 60TB usable storage in Raid 6. AND its compact enough to fit in the closet (with enough ventilation).

Setup δ

With a built server, we can now boot up into Linux. In case you couldn’t tell from the neofetch output, I use Arch by the way^(tm). And yes with systemd instead of a init system that hasn’t been updated in a year. Head on over here if you are unfamiliar on how to get started with Arch.

BIOS δ

First things first, lets boot into the BIOS to fix up some settings.

$ systemctl reboot --firmware-setup

Flash the most recent BIOS. This should help avoid issues that have already been fixed.
Enable SVM Mode.
Turn on Single Root I/O Virtualization (SR-IOV). This will help when you eventually have multiple containers/vms/pods running and sharing the compute.
Set all the PCIE lanes to v3. The motherboard has difficulty automatically figuring out the PCIE versions when using the PCIE risers. Setting it to 3.0 allows the system to boot, where as leaving it on auto gives motherboard errors.
Check that the RAM is working at the correct speed.
Check that the boot drive is set correctly.

Info

Ensure the motherboard is looking for the GPU over PCIE to avoid having issues connecting to the GPU. The wrx80e has a VGA switch that needed to be turned off for the GPUs to work correctly.

Nvidia δ

Now comes everyone’s favorite part, getting the Nvidia Linux driver and CUDA working correctly.

Setup δ

0️⃣0️⃣ NVIDA Drivers δ

For the GPU drivers we’re going to use Dynamic Kernel Module Support (DKMS) to allow the system to easily swap between driver versions. This will be good if we ever need to upgrade or downgrade the drivers.

$ yay -S dkms
 
# Check if your system detects the GPUs
$ lspci | grep -e VGA -e 3D
$ yay -S nvidia nvidia-utils nvidia-settings
# (optional, but reccomended)
$ yay -S linux-headers

0️⃣1️⃣ CUDA δ

$ yay -S cuda
 
# update the path to add CUDA libs
$ echo 'export PATH=/opt/cuda/bin:$PATH' >> ~/.bashrc
$ echo 'export LD_LIBRARY_PATH=/opt/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc

0️⃣2️⃣ `nvidia-persistenced` δ

$ sudo systemctl enable nvidia-persistenced
$ sudo systemctl start nvidia-persistenced

Validation δ

After the reboot the system should look something like this.

0️⃣0️⃣ Nvidia Drivers δ

$ nvidia-smi
Sun Dec 24 04:04:49 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.06              Driver Version: 545.29.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3090        On  | 00000000:01:00.0 Off |                  N/A |
|  0%   31C    P8               7W / 350W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 3090        On  | 00000000:2B:00.0 Off |                  N/A |
|  0%   34C    P8              10W / 350W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce RTX 3090        On  | 00000000:41:00.0  On |                  N/A |
|  0%   31C    P8              12W / 350W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce RTX 3090        On  | 00000000:61:00.0 Off |                  N/A |
|  0%   34C    P8              17W / 350W |      3MiB / 24576MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
 
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

0️⃣1️⃣ CUDA δ

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Nov__3_17:16:49_PDT_2023
Cuda compilation tools, release 12.3, V12.3.103
Build cuda_12.3.r12.3/compiler.33492891_0

0️⃣2️⃣ `nvidia-persistenced` δ

$ systemctl status nvidia-persistenced
● nvidia-persistenced.service - NVIDIA Persistence Daemon
     Loaded: loaded (/usr/lib/systemd/system/nvidia-persistenced.service; enabled; preset: disabled)
     Active: active (running) since Sat 2023-12-23 09:05:34 UTC; 18h ago
    Process: 787 ExecStart=/usr/bin/nvidia-persistenced --user nvidia-persistenced (code=exited, status=0/SUCCESS)
   Main PID: 790 (nvidia-persiste)
      Tasks: 1 (limit: 308982)
     Memory: 908.0K (peak: 1.7M)
        CPU: 1.272s
     CGroup: /system.slice/nvidia-persistenced.service
             └─790 /usr/bin/nvidia-persistenced --user nvidia-persistenced
 
Dec 23 09:05:26 apollo systemd[1]: Starting NVIDIA Persistence Daemon...
Dec 23 09:05:26 apollo nvidia-persistenced[790]: Started (790)
Dec 23 09:05:34 apollo systemd[1]: Started NVIDIA Persistence Daemon.

0️⃣3️⃣ NVLINK δ

Tip

If NVLINK isn’t showing up then make sure connector is fully connected

$ nvidia-smi nvlink --status
GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-...)
	 Link 0: 14.062 GB/s
	 Link 1: 14.062 GB/s
	 Link 2: 14.062 GB/s
	 Link 3: 14.062 GB/s
GPU 1: NVIDIA GeForce RTX 3090 (UUID: GPU-...)
	 Link 0: 14.062 GB/s
	 Link 1: 14.062 GB/s
	 Link 2: 14.062 GB/s
	 Link 3: 14.062 GB/s
GPU 2: NVIDIA GeForce RTX 3090 (UUID: GPU-...)
	 Link 0: 14.062 GB/s
	 Link 1: 14.062 GB/s
	 Link 2: 14.062 GB/s
	 Link 3: 14.062 GB/s
GPU 3: NVIDIA GeForce RTX 3090 (UUID: GPU-...)
	 Link 0: 14.062 GB/s
	 Link 1: 14.062 GB/s
	 Link 2: 14.062 GB/s
	 Link 3: 14.062 GB/s

$ nvidia-smi topo -m
	GPU0	GPU1	GPU2	GPU3	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	SYS	SYS	NV4	0-23	0		N/A
GPU1	SYS	 X 	NV4	SYS	0-23	0		N/A
GPU2	SYS	NV4	 X 	SYS	0-23	0		N/A
GPU3	NV4	SYS	SYS	 X 	0-23	0		N/A
 
Legend:
 
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Warning

If you see AER errors on boot then you need to enable the DRM kernel mode setting.

Edit the relevant loader in /boot/loader/entries/

Add modeset=1 fbdev=1 to the end of the options line

Power Management δ

Okay, now lets address the elephant in the room. How the heck is a 300W CPU and 4 350W GPUs going to fit in a 1600W PSU? And thats not even including the power needed for the rest of the system!

To get around this, we will set the total power limit to 250W for now to avoid tripping any fuses. This change will drop the performance of the system but also drop the temperature and heat produced.

To start let’s create a new systemd service in /etc/systemd/system/.

# nvidia-power-limitd.service
[Unit]
Description=NVIDIA settings on startup
After=syslog.target
 
[Service]
Type=oneshot
EcecStart=nvidia-smi -pm 1
ExecStart=/usr/bin/nvidia-smi -pl 250
RemainAfterExit=no
 
[Install]
WantedBy=multi-user.target

And enable it so that it runs on boot.

$ sudo systemctl enable nvidia-power-limitd

Lessons Learned δ

Use as much off the shelf hardware to save money.
Think outside of the box.

In order to fit all of this into a standard PC case I would have had to put a lost more money and time into getting it to work. By just using the same cheap stuff the Crypto bros use, I saved a lot of time and money.

Read the F$!%ing Manual

Tip

RAM goes in the top 4 slots first.

Tip

For 4x SLI support, GPUs need to be connected in PCIE lanes 1, 3, 5, and 7.

Tip

Motherboard QCodes can be found on page A1

CTRL+F to figure out what BIOS words mean

Sometimes unplugging a cable and plugging it back in, is the correct answer.
Buy in bulk and bid around!

Accept Reality

Training Meta’s LLAMA took 3.7M GPU hours on A100-80GB¹, training Meta’s LLAMA 2 took 3.3M GPU hours on A100-80GB², training TinyLlama took 3456 GPU hours on A100-40GB³, training Microsoft’s Phi 2 took 33K GPU hours on A100-80G⁴ training Microsoft x OpenAI’s ChatGPT costs Iowa ~431 million gallons of water⁵.

Your not that guy pal. Trust me. Ok? Eventually you will have to use cloud services to scale up to have your bigger models, faster inference for production load, and the occasional Joker moment. However, having an on prem system allows for failed attempts to be less costly. Those cloud storage fees can get pricy!

Whats next? δ

Well the system seems stable enough when I’m running ComfyUI and Jupyter Notebooks. However there is still more work left to do.

Find out how much performance is lost due to the power draw limit. How much more can we squeeze out of it?
Test the hard drives to make sure they aren’t on their last leg.

Being S.M.A.R.T. 🧠 About Used Storage 💾

Screw everything onto the “case”
So uh, what can we run on here?
We still have 3 open PCIE lanes…
Docker? K3S? Quantum Containers?

MDX Refresher 🍹

Fri, 22 Dec 2023 00:00:00 GMT

This is a test file to refresh concepts and demonstrate various Markdown features used in GitHub Flavored Markdown and Obsidian Flavored Markdown.

GitHub Flavored Markdown δ

Footnotes δ

This sentence uses footnotes.¹

Strikethrough δ

This is a ~~strikethrough~~ example.

Tables δ

Header 1	Header 2	Header 3
Row 1, Cell 1	Row 1, Cell 2	Row 1, Cell 3
Row 2, Cell 1	Row 2, Cell 2	Row 2, Cell 3

Task Lists δ

Completed task
Incomplete task

Obsidian Flavored Markdown δ

Callouts δ

Note

This is a note callout.

Tip

This is a tip callout.

Success

This is a success.

Abstract

This is a abstract callout.

Example

This is a exmple callout.

Info

This is a info callout.

Warning

This is a warning callout.

Quote

This is a quote

Wikilinks δ

This is a wikilink to another note.

This is a footnote. ↩

🧙 Prayag Bhakar

Welcome to my blog ✨

Links δ

Subscribe δ

Tech Stack δ

Docker Refresher 🍹

Docker CLI δ

Dockerfile δ

docker-compose δ

Being S.M.A.R.T. 🧠 About Used Storage 💾

badblocks :: The Bad Boy in Town δ

Making S.M.A.R.T. Decisions δ

Manual Tests δ

Understanding S.M.A.R.T. values. δ

Automated Tests and Alerts δ

ZFS, the Final Frontier δ

What Now? δ

🪐 Apollo 🛸 AI Compute Cluster for the GPU Poor

The Build δ

Setup δ

BIOS δ

Nvidia δ

Setup δ

0️⃣0️⃣ NVIDA Drivers δ

0️⃣1️⃣ CUDA δ

0️⃣2️⃣ nvidia-persistenced δ

Validation δ

0️⃣0️⃣ Nvidia Drivers δ

0️⃣1️⃣ CUDA δ

0️⃣2️⃣ nvidia-persistenced δ

0️⃣3️⃣ NVLINK δ

Power Management δ

Lessons Learned δ

Whats next? δ

Footnotes δ

MDX Refresher 🍹

GitHub Flavored Markdown δ

Footnotes δ

Strikethrough δ

Tables δ

Task Lists δ

Obsidian Flavored Markdown δ

Callouts δ

Wikilinks δ

Footnotes δ

`Dockerfile` δ

`docker-compose` δ

`badblocks` :: The Bad Boy in Town δ

0️⃣2️⃣ `nvidia-persistenced` δ

0️⃣2️⃣ `nvidia-persistenced` δ