Tower of Pi Project

Tower of Pi Project

Forward: I've been kind of burned out on everything for a long time and this little computer inspired me to take some interest in something other than earning a pay check for a long time. I guess my life since the dot-com era has been kind of tough. I have been engaged in "subsistence hacking" for too long. Perhaps now I can change that. Maybe I will have enough interest to create something unique. We'll see.

Synopsis: The Tower of Pi is a 10-node parallel computing cluster (9 compute nodes consisting of Raspberry Pi 512M model (overclocked to 1ghz) and 1 Cubie 1GB/1GHZ single board computer system acting as master controller and NFS server (ssd). This is a "toy supercomputer" in the sense that the cpu are all ARM and they're pretty slow compared to the incredibly fast cpu available today. However it is not a toy in the sense that it is fully functional and is based around the MPI framework. There are (2) gig-e ethernet switches connecting all nodes together since the "master" controller unit also handles NFS we use hard-wired networking. The inter-switch link and the cluster uplink are gig-e but the compute nodes only support 100M. We should be able to push quite a lot of packets between each node in the cluster and each node will be able to obtain it's maximum performance to other assets on the network (like the NAS) which are gigabit-ethernet.

Sept 12, 2014

I keep robbing parts from the RPI cluster.

This time it's really a just a trade as I needed the micro sd cards for another project (held in sd-sized holders) but swapped each one for some 4GB actual sd cards from Adata that were a great buy. Since the RPI can only use SD cards it makes sense to use those in them.

Since I just have to clone an image and edit three files I decided it was worth the effort to recover the micro-sd cards (12 of them altogether for use with projects that can only use micro-sd cards.)

1. insert sd card into reader.
2. blockdev —rereadpt /dev/sdc
3. dd if=rpi-node.img of=/dev/sdc bs=1M
4. blockdev —rereadpt /dev/sdc
5. e2fsck /dev/sdc2
6. mount /dev/sdc2 /mnt
7. Edit /etc/hostname, /etc/network/interfaces, /etc/resolv.conf (I changed resolver host recently.)
8. cd /;sync;umount /mnt

I had all nine in the rack pictured working last night and updated apt cache files but powered them down as I don't have any useful work for them to be doing at the moment.

I have revamped the power system as well ditching the old +5VDC power supply on the wooden board and used a 12VDC to 5VDC converter attached to a 50AH battery which is being constantly charged.

Now at least they have an isolated ups (50AH.)

I discovered that while I don't really use the RPI now I do use the gig-e switches, the powered usb hub and the cubieboard 1 system that used to act as the cluster manager and NFS server.

It has an SSD (100gb) attached and performs very well so I use this system to burn images and repair filesystems from the main cluster.

2014-09--001-small.jpg 2014-09--005-small.jpg

July 25, 2014

Instead of disassembling it I am bring it back up and will resurrect it into something interesting perhaps to do with video on demand. I'm also working on some control circuits for an off-grid generator backup in case a battery pack voltage drops too low. I plan on using the proto-armor to mount it in a rugged environment. Tonight I have been making new bootable images for every node and that is taking a long time.

August 02, 2013

The Tower of Pi system didn't sell so I am going to disassemble it and use the parts and components for other projects. I thought it was cool but I suspect that may have been just my opinion, sigh. See Cubical Monolith Project that is where all my current work is being done.

July 10, 2013

I finished fabricating and installing the top and bottom rails. Hopefully they will come in handy when mounting the Tower Of Pi.

July 09, 2013

I just completed the top rails tonight, drilling and securing the second rail. I didn't have a chance to work on it until after 2100 and it's 2301 now.


July 08, 2013

It seems I have made a few more changes to the Tower of Pi, mostly for structural reasons (but I like the esthetics of the rails. I will be adding bottom rails to match now, most likely with holes drilled in them for platform mounting purposes. I also added a strain relief to protect the master node's console and power cables.

I also had to repair an intermittent cable between the two gig-e switches. Note that the cross connect between the two switches the port status led should be green otherwise the gig-e link isn't working. The master and the computing nodes should be amber as they're 100M only. The uplink port's led would also be green assuming it is connected it to a gig-e capable port somewhere.

July 07, 2013

Here are the final pictures of the project which was completed some time ago. I am working on building another system but will most likely build more Tower of Pi systems if there is interest. I currently have it listed for sale on EBAY. See ‘Tower of Pi “Supercomputer” Parallel Computing Cluster 10-node‘

Halting The Cluster

pi@master:/master/mpi_tests$  mpirun -f /master/mpi_tests/haltfile -n 9 sudo poweroff

June 11, 2013

I worked on the perl code a bit to update the status display. As it is scrolling (and running the mpi-tests it displays each node status) and then the last message is a total of how many nodes are online.


Sometimes we forget how limiting a 16x display can be.

I also finally received the SD reader, using it under unix last night I was able to write several images and I finally have a 1GB kernel booted on the second cubieboard. I am rebuilding the image, stripping useless things and adding all the missing stuff for mpi and whatever else.

root@debian:~# uname -a
Linux debian 3.0.62 #1 PREEMPT Thu Feb 14 20:21:06 CET 2013 armv7l GNU/Linux
root@debian:~# free
             total       used       free     shared    buffers     cached
Mem:        834600     350996     483604          0      35596     254512
-/+ buffers/cache:      60888     773712
Swap:            0          0          0

June 10, 2013

I finally had time to solder the display together. I wrote some minimal perl script and the beginning of a python script to control it.


June 3, 2013

I think there is an issue with Win32DiskImager and the Toughbook CF-30 SD card reader. It won't recognize devices on this laptop (I tried two different CF-30s) nor the CF-19. I have tested all units with Win32DiskImager. Roadkil's Imager seems to work but it's producing bad images. Luckily win2diskimager works on a desktop machine here.

I ordered (10) Samsung 8-GB Class 6 micro sd cards. I have enough extra adapters to use them in sd card readers as well. I'm trying to write an image that is just too large for a 4-GB card. They're made in Taiwan as well.


I didn't have any time to work today other than making a few cables. I am still failing miserably at making very short jumpers. I can get them working for 100mbit but they won't come up in gig-e and I don't have a cable tester right now. I used to be one of those who didn't need them but somehow I've lost my mad skills.

I think I am getting better though, the last three I made were flawless. Too bad I don't intend to make any more cables in the near future.


All of these are bad, most work in 100mbit but not gigabit ethernet.

2013-06-02-002-small.jpg 2013-06-02-004-small.jpg 2013-06-02-009-small.jpg

One thing I noticed that since ditching all the wifi, latency has improved, large file transfers are of course much better, and MPI works much better over hardwire links. Those little wifi dongles are cool but low powered plus it's via USB which on the RPI isn't very reliable.

Remaining Tasks

Cluster Status Display

I have to get the serial display working soon, I ordered an RGB serial display that I will use as a status display on the "front panel" … to display the cluster status and other diagnostic information.

It is an RGB display so I should be able to change the color of the text under software control (as well as the background color) depending on what message it is displaying as well.

I have to solder all the parts together as it came as a kit.

It can use either usb (ttl serial) or i2c as inputs and right now I think I'll just use the usb-mode now since the usb hub is empty and I don't have a way to easily access the GPIO on the master controller (Cubie!).

I'll probably write some simple python code to run on the master controller via cron to check the status of each node and whatever seems useful to see in a repetitive displayed stream of text. Something a bit more useful than icmp ping, perhaps an MPI "ping" written in python.

Hardware and cleanup issues

I am waiting on a new shipment of hardware (nuts and bolts) to clean up some mounting issues here and there, I also need to utilize the space better in some areas but I am limited by what can be fabricated with the hand tools I have.

I have to mount the SSD permanently, it's just lying next to where it will be mounted (waiting on more nylon hardware, and time to drill some holes)

I have to clean up the SATA data cable and the Cubie USB to hub cable (usb 2.0-tyle connector) so I can shorten it or dress it so it stays inside the frame.

Serial Console

The master node controller console serial cable needs to be shortened and a chassis-mount usb pigtail attached so all I will need to check the console is a usb cable attache to any device that is permanently accessible at any moment.

USB Charging Station

I don't want to seem flippant but I have been using the Tower of Pi for my usb charging station as well and it is so convenient for my cell and ipod.

Current Load

I finally checked the Tower configuration with everything turned on but without anything plugged into the usb hub and it's drawing 5.6 to 5.7 amps.


I was very busy doing other things today but I did have time to make eight shorter ethernet cables. I had some issues with making cables until I started using a magnifying visor. I guess my eyesight isn't that great any more. I ran out of time and will try to finish soon.


I need to make the jumper between the switches shorter but I can't seem to make short jumpers very well and I have about 6 attempts lying on the floor. I'll try again tomorrow when I can see better even if I have to work outside in the sun. I made a lot of bad cables today and wasted a lot of connectors. There are three gig-e ports in use and I realized I was making cables that only work with 100 megabit using very old standard when I could get those cables working on the RPI links but not the inter-switch link. I realized what I had done wrong quickly well not before I made 4 cables..


I received the gigabit-ethernet switches and installed them into the Tower tonight, and cleaned up some of the wiring. I simply must make a bunch of very short ethernet cables this weekend.

2013-05-31-007-small.jpg 2013-05-31-003-small.jpg

I have the Cubie board installed and running as the master controller for all MPI parallel processing and also acting as an NFS server. The 120gb SSD is working very well with the Cubie's SATA interface. No glitches whatsoever.

I have decided that I will use hard-wired connections as wifi proved to be unreliable and NFS is very solid and fast now. I have been spending a lot of time trying to get an image that supports 1GB memory working without an SD reader running under linux. I ordered an SD card reader that I can use from one of the nodes or the master controller with linux.

That mess of ethernet cables has to go by the end of this weekend I will make a lot of short cables all one color as well. "It's all about the cable management" Hah.

The blinking lights quotient went off the charts when I started using ethernet again. Not only the lights from all those switch ports in use but the lights on the RPI as well. It's like a Christmas tree now.

This is what it's doing to keep itself busy while I sleep.

+ mpirun -n 9 -f /master/mpi_tests/machinefile ./john --status=john
  0: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 16.65
  3: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 16.53
  1: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 16.52
  4: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 16.52
  2: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 16.51
  5: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 16.52
  6: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 16.52
  7: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 16.51
  8: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 16.52
SUM: guesses: 0 time: 0:00:30:00 0.00% (2) c/s: 148 avg16.53
+ mpirun -np 9 -f /master/mpi_tests/machinefile /master/mpi_tests/system
Sat Jun 1 00:11:51 MDT 2013 node06 00:11:51 up 1:12, 0 users, load average: 1.00, 1.14, 1.06
Sat Jun 1 00:11:51 MDT 2013 node08 00:11:51 up 1:16, 0 users, load average: 1.19, 1.19, 1.06
Sat Jun 1 00:11:51 MDT 2013 node07 00:11:51 up 1:16, 0 users, load average: 1.16, 1.22, 1.12
Sat Jun 1 00:11:51 MDT 2013 node09 00:11:51 up 1:16, 0 users, load average: 1.08, 1.16, 1.06
Sat Jun 1 00:11:51 MDT 2013 node04 00:11:51 up 1:16, 0 users, load average: 1.03, 1.18, 1.09
Sat Jun 1 00:11:51 MDT 2013 node03 00:11:51 up 1:16, 0 users, load average: 1.11, 1.15, 1.05
Sat Jun 1 00:11:51 MDT 2013 node02 00:11:51 up 1:16, 0 users, load average: 1.16, 1.16, 1.06
Sat Jun 1 00:11:51 MDT 2013 node01 00:11:51 up 1:16, 0 users, load average: 1.03, 1.15, 1.05
Sat Jun 1 00:11:51 MDT 2013 node05 00:11:51 up 1:16, 0 users, load average: 1.03, 1.17, 1.08


I also have (2) more SSD coming soon, each are 100gb. I really like these drives they are tough and hopefully the mtbf ratings are accurate. They are going to be installed onto Cubie boards of course. I need more time to study these systems, especially with the baseboards.

I can't wait to get the ethernet switches I ordered to see how well they can be adapted to simply become the internal switch for the cluster running on an isolated RFC network space dedicated to MPI and NFS traffic… I will simply pull them out of their cases and mount them directly inside the rack. However they are only 8-port switches, hence installing (2) of them…

I have to make a lot of very short ethernet cables. I have all the components I need to do that as well but I'll wait for the switches to arrive.

If the node controller is on wifi and acts as a bridge as well for the other nodes that are only attached via wired network that will be fine as well. I might look into that soon just to illustrate how simple that really is.

That might let me use the cubie board not only as a node controller, an NFS server, but also as a router. It might give me a chance to see how well the cubie network kernel is performing under some benchmark tests.

I can then also remove the wifi dongles from each node leaving them with a hardwired connection which has always been reliable. If the node controller is serving the SSD via nfs over wired network it will be very stable indeed. The node controller "master" in this case, which is the Cubie board will retain it's wifi connection so that the cluster itself has no network wires to trip over.

I can always attach eth0 to a wired network if need be temporarily.

Cubieboard Issue: WIFI driver getting trashed

I keep losing the usb wifi kernel driver. I'm not sure what's going on but it always happens under heavy load or memory use. I was compressing a 4gb image using bzip2 and it just blew up. The system stayed up, I still had console access so I let it complete (I was using screen as another user) and then rebooted it. As expected the kernel driver worked as normal. This is the second time in 3 days this has happened to me. I don't use the wired network right now

Updated: I added a swapfile on the ssd disk just to see if running out of swap caused the wlan driver issue. I re-ran the bzip of the image file which preceeded the death of the kernel driver and noted that the bzip process used about 509mb alone and right now I still don't have a 1GB kernel built.

RPI SD Card Corruption (on one out of 12 systems)

I keep losing the filesystem on the sd card on node01 (RPI 512M). I am going to replace node01 tonight with node11 or node10 but I will keep the same sd card just to see why this keeps happening.
It's not the overlocking, I was losing the filesystem at normal speed and all of the others have been overlocked so far without incident. If this stops happening then the current node01 will become a development system. I am using Adata MicroSD cards with the adapters.

Now that the work week has begun (we had Monday off for Memorial Day) my time will be even more limited. The system 'node01' is the oldest RPI I own and belonged to someone who is now dead. He was trying to get me interested in RPI and gave it to me as a gift. He died unexpectedly a few months ago.

How to determine RPI Firmware versions

/opt/vc/bin/vcgencmd version

Expanding the 2GB Cubieboard Debian Image

The image I got for the Cubie was for a 2GB filesystem but I was using a 4GB card so I was able to resize the partition properly.

Look at raspi-config to see how they do it but basically:

They're using parted to determine the start of the second filesystem on mmcblk0 as illustrated below. In the example below, it's 131072. We will use that number with fdisk as the start of mmcblkp2

root@master:~# parted /dev/mmcblk0 -ms unit s p| grep "^2" | cut -f 2 -d:

fdisk /dev/mmcblk0 <<EOF
^M (carriage return here) See Note 1.

Now reboot then immediately login and issue:

root@master:~# resize2fs /dev/mmcblk0p2
resize2fs 1.42.5 (29-Jul-2012)
Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/mmcblk0p2 to 951680 (4k) blocks.
The filesystem on /dev/mmcblk0p2 is now 951680 blocks long.

Note 1: This is where you either accept the default size of the partition or choose size of same) In this case it was 7761919 but I chose 500-blocks less so that it should fit onto any size 4GB card. (i.e. 7761419) If you are confused just see the transcript below.

Actual transcript, resizing a 4GB card with a 2GB image.
root@master:/home/pi# fdisk /dev/mmcblk0

Command (m for help): p

Disk /dev/mmcblk0: 3974 MB, 3974103040 bytes
4 heads, 16 sectors/track, 121280 cylinders, total 7761920 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cb560

        Device Boot      Start         End      Blocks   Id  System
/dev/mmcblk0p1            2048      127071       62512    e  W95 FAT16 (LBA)
/dev/mmcblk0p2          131072     3451383     1660156   83  Linux

Command (m for help): d
Partition number (1-4): 2

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p): p
Partition number (1-4, default 2): 2
First sector (127072-7761919, default 127072): 131072s
Last sector, +sectors or +size{K,M,G} (131072-7761919, default 7761919): 7761419

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

root@master:/home/pi# reboot
(system reboots)
root@master:~# resize2fs /dev/mmcblk0p2
resize2fs 1.42.5 (29-Jul-2012)
Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/mmcblk0p2 to 953793 (4k) blocks.
The filesystem on /dev/mmcblk0p2 is now 953793 blocks long.


I deliberately made it 500-blocks smaller than actual capacity so I can fit this image onto ANY 4GB sd card.

I will test this method today burning this image onto several SD cards to see if I did it correctly. I am confident I did but making a backup is a good thing.

root@master:~# dd if=/dev/mmcblk0 of=/master/tmp/master-4gb-2013-05-28.img
7761920+0 records in
7761920+0 records out
3974103040 bytes (4.0 GB) copied, 241.238 s, 16.5 MB/s


I have added a 1GHZ 1GB CubieBoard as the master controller and file server.


I've been purchasing all the Cubie Board-related stuff from IO Technologies and George. They have been helpful so far and they ship very quickly. I'll talk more about IO Technologies later, George Ioakimedes is the founder, he is the source of the Cubie Baseboard which is an incredible expansion board for the Cubie which offers everything you could possibly want for both development and daily use (even has a great stereo audio amplifier to make your music rock!) … I don't own any yet as I haven't yet begun to order parts and more Cubies in earnest. I don't have a "development" system built yet as both of the boards I've ordered are going right into "production" where they must be stable.

I am still working on this server, right now I need to install linux on an intel machine with an sd card reader so I can work on making some new images for this one. It looks like I'm going to need some cross-compiler if I want to build new kernels as well.

I have the 120GB Corsair SSD attached to the built-in SATA interface and all is well so far.

I have a serial console attached to the Cubie board, it's very easy to access the pins for the usb-ttl-serial ports but the documentation is somewhat sparse. Don't connect the red wire to VCC!!

I attached the cable to node12 (an RPI development system) and am running minicom "sudo minicom -D /dev/ttyUSB0" and it's working well.

Everything is wireless, each unit has a wifi adapter and that is working well for now.

I have (2) tiny 8-port gig-e switches on order that can be powered from 5vdc so I am going to have to make a bunch of very short ethernet cables to build a private wired network for NFS, I will keep the wifi for public network access. If I can keep the NFS off the wifi things will be much nicer for me as I share that wireless net with my laptops and mobile devices.


I haven't had a lot of time to work on anything lately but yesterday and today I managed to get one Cubie server built and now it's in charge of all the parallel nodes in the cluster. It is also acting as an NFS server (and it's performing well so far).

I have overclocked all (9) RPI to 1ghz as well.

Just for testing and burn-in of the new configuration I'm running 18 copies of John The Ripper to see if I can crack my own password from my work account.

pi@master:/master/tmp/john/john-1.7.9-jumbo-7/run$  mpirun -f /master/mpi_tests/machinefile -n 18 /master/tmp/john/john-1.7.9-jumbo-7/run/john /master/tmp/john/john-1.7.9-jumbo-7/run/passwd.txt
Loaded 4 password hashes with 4 different salts (sha512crypt [32/32])
MPI: each node processing 1/18 of 1081 rules. (uneven split)
Node 2@node03: Crash recovery file is locked: /master/tmp/john/john-1.7.9-jumbo-7/run/john.2.rec
Node 5@node06: Crash recovery file is locked: /master/tmp/john/john-1.7.9-jumbo-7/run/john.5.rec
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 2
Node 6@node07: Crash recovery file is locked: /master/tmp/john/john-1.7.9-jumbo-7/run/john.6.rec
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 5
Node 4@node05: Crash recovery file is locked: /master/tmp/john/john-1.7.9-jumbo-7/run/john.4.rec
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 6
Node 7@node08: Crash recovery file is locked: /master/tmp/john/john-1.7.9-jumbo-7/run/john.7.rec
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 4
Node 3@node04: Crash recovery file is locked: /master/tmp/john/john-1.7.9-jumbo-7/run/john.3.rec
Node 8@node09: Crash recovery file is locked: /master/tmp/john/john-1.7.9-jumbo-7/run/john.8.rec
Node 1@node02: Crash recovery file is locked: /master/tmp/john/john-1.7.9-jumbo-7/run/john.1.rec
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 7
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 8
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 3
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1
MPI: each node loaded 1/18 of wordfile to memory (about 369 KB/node)

Later this shows:

+ mpirun -n 18 -f /master/mpi_tests/machinefile ./john --status=john
  9: guesses: 0 time: 0:00:50:09 0.00% (2) c/s: 8.17
 13: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 4.32
  4: guesses: 0 time: 0:02:10:00 0.00% (2) c/s: 14.00
 17: guesses: 0 time: 0:00:50:01 0.00% (2) c/s: 4.32
 15: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 4.33
  6: guesses: 0 time: 0:02:10:00 0.00% (2) c/s: 14.00
  0: guesses: 0 time: 0:00:50:09 0.00% (2) c/s: 8.18
 12: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 4.33
  2: guesses: 0 time: 0:02:10:00 0.00% (2) c/s: 14.00
 16: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 4.33
  8: guesses: 0 time: 0:02:10:00 0.00% (2) c/s: 14.00
  5: guesses: 0 time: 0:02:10:00 0.00% (2) c/s: 14.00
  3: guesses: 0 time: 0:02:10:00 0.00% (2) c/s: 14.00
 11: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 4.32
  7: guesses: 0 time: 0:02:10:00 0.00% (2) c/s: 14.00
 14: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 4.33
 10: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 4.33
  1: guesses: 0 time: 0:02:10:00 0.00% (2) c/s: 14.00
SUM: guesses: 0 time: 0:02:10:00 0.00% (2) c/s: 200 avg11.11
pi@master:/master/tmp/john/john-1.7.9-jumbo-7/run$ mpirun -np 9 -f /master/mpi_tests/machinefile /master/mpi_tests/system
Mon May 27 22:01:10 MDT 2013 node01 22:01:10 up 1:01, 1 user, load average: 2.97, 2.56, 2.23
Mon May 27 22:01:10 MDT 2013 node04 22:01:10 up 2:45, 0 users, load average: 3.02, 3.03, 2.97
Mon May 27 22:01:10 MDT 2013 node09 22:01:10 up 2:45, 0 users, load average: 3.02, 3.03, 2.97
Mon May 27 22:01:10 MDT 2013 node03 22:01:10 up 2:45, 0 users, load average: 3.02, 3.01, 2.97
Mon May 27 22:01:10 MDT 2013 node05 22:01:10 up 2:45, 0 users, load average: 3.01, 3.02, 2.96
Mon May 27 22:01:10 MDT 2013 node07 22:01:10 up 2:45, 0 users, load average: 3.02, 3.03, 2.97
Mon May 27 22:01:10 MDT 2013 node06 22:01:10 up 2:45, 0 users, load average: 3.02, 3.02, 2.97
Mon May 27 22:01:10 MDT 2013 node08 22:01:10 up 2:45, 0 users, load average: 3.03, 3.05, 2.98
Mon May 27 22:01:10 MDT 2013 node02 22:01:10 up 2:45, 0 users, load average: 3.02, 3.03, 2.97

2013-05-27-001-small.jpg 2013-05-27-002-small.jpg 2013-05-27-005-small.jpg

I could go on for hours about the cubieboard both my praise and criticisms but let me just say that the executive summary on the Cubie is "Buy all you can get your hands on" … my next "toy supercomputer" project will be based around the Cubie Board.

I have ordered a lot of parts to continue my design work with the Tower of Pi project as well as 3 other RPI units for hardware development and I have an extra Cubie Board with a second 120GB Corsair SSD to build shortly.


I am beginning to create a local repository since my network connection is so slow and I'm constantly adding, changing and rebuilding things right now I have mirrored the raspbian repository for local use.

These are just my personal notes so they're not intended to be a guide. YMMV as they used to say back in the old days.

[/share/MD0_DATA/Storage/raspbian] # ls -l dists/wheezy
-rw-r--r--    1 935      1000     23430356 May 19 11:10 Contents-armhf.gz
-rw-r--r--    1 935      1000        14946 May 19 11:10 InRelease
-rw-r--r--    1 935      1000        14409 May 19 11:10 Release
-rw-r--r--    1 935      1000          490 May 19 11:10 Release.gpg
drwxr-xr-x    4 935      1000         4096 May 19 11:10 contrib
drwxr-xr-x    4 935      1000         4096 May 19 11:10 firmware
drwxr-xr-x    5 935      1000         4096 May 19 11:10 main
drwxr-xr-x    4 935      1000         4096 May 19 11:10 non-free
drwxr-xr-x    4 935      1000         4096 May 19 11:10 rpi
-rw-r--r--    1 935      1000        22137 May 19 11:10 uContents-armhf.gz
[/share/MD0_DATA/Storage/raspbian] # ls -l dists/wheezy/contrib
-rw-r--r--    1 935      1000        55713 May 19 11:09 Contents-armhf.gz
drwxr-xr-x    2 935      1000         4096 May 19 11:10 binary-armhf
drwxr-xr-x    2 935      1000         4096 May 19 11:10 source
[/share/MD0_DATA/Storage/raspbian] # ls -l dists/wheezy/contrib/binary-armhf
-rw-r--r--    1 935      1000        79858 May 19 11:08 Packages
-rw-r--r--    1 935      1000        23243 May 19 11:08 Packages.bz2
-rw-r--r--    1 935      1000        26344 May 19 11:08 Packages.gz
-rw-r--r--    1 935      1000          166 May 19 11:08 Release
[/share/MD0_DATA/Storage/raspbian] #

I intend to use it as a locally mounted source much like a cdrom (via nfs) which is on a NAS. Eventually I may move it to an ssd attached to a cubieboard-based NFS server (hah, I sprang that surprise) since I want to have a "build" server running soon as well and then begin to rebuild some of the packages to make sure all is in order.

I will update this later it's very late or perhaps early.


I was testing an ATA-66 to usb adapter today.

root@node01:/var/log# mkfs.ext4 /dev/sdb1 -L ATA66
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=ATA66
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
9773056 inodes, 39070072 blocks
1953503 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
1193 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done



I have been conducting write "torture tests" with these usb-sata bridges. They seem to disconnect after sustained writing activity. I am using rsync to mirror the contents of a directory on the NAS (raspbian repo) which is about 90gb and I was able to make the usb-sata bridge fail.

I know the drives are all right. I have even made sure it has at least 1A by connecting it with one of those "dual cables" to get extra current.


I'm not having a lot of luck with disks lately.


Message from syslogd@node01 at May 13 15:55:28 …
kernel:[63017.838990] journal commit I/O error
rsync: write failed on "/SSD/tmp/raspbian.tar.04152": Read-only file system (30)
rsync error: error in file IO (code 11) at receiver.c(322) [receiver=3.0.9]
rsync: connection unexpectedly closed (51441 bytes received so far) [generator]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9]
pi@node01 /SSD/tmp $

1820MT I finally got a chance to look into this :/

pi@node01 ~ $ sudo su -
root@node01:~# fsck /dev/sda1
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
/dev/sda1: recovering journal
Setting free inodes count to 7366353 (was 7366824)
Setting free blocks count to 7176561 (was 9585652)
/dev/sda1: clean, 6447/7372800 files, 22311848/29488409 blocks

So it seems fine after I power cycled everything. We will see.

Updated: I had to work late so I didn't have much time to look into this but to help eliminate some other issues I am testing a second usb-sata bridge controller and reconnected the original to the second SSD disk but this time used a "2 usb plug" cable to see if my issues are power-related.

When I have tomorrow I will write 90gb to the "test" ssd to see if I can make it fail again.
At first, I thought the problem is thermal but it might just be the power.

Anyway I am too tired to work on this now it's almost 2300 local time. It has been a long day.


I have removed the SSD-RAID-10 assembly, removing the primary disk from that, attaching that to a usb-sata adapter and re-connecting to node01. I've got a single 120GB Corsair SSD attached to node01 and it is working well. I was able to read the partition and all data was intact.

I'll take a picture Monday as well as come up with some new mounting for the lone disk.

As an aside I found the two missing external USB-SATA enclosures at the nearby store. I finally checked the tracking info and discovered they claimed it had been delivered May 02. The usual UPS person was off and some fill-in bastage dropped it off there. I have so much work going on and everything else I didn't think to check the tracking information until today. The people at the store didn't bother to let me know it seems. They don't like me it seems as my phone number was prominently displayed on the package and they are supposedly a ups shipping point.

I disassembled them and immediately attached a 2.5" 320gb sata disk to one and brought it up under raspbian. Everything went well and I was about 35gb into transferring the entire raspbian repo to the disk and it appears the disk failed during the transfer. I was using a local rsync command to mirror the data from the ssd array.

I am pretty annoyed, I bought that WD Scorpio drive used from someone so never buy a used disk. It isn't the money so much is that I can't finish something I was working on today.

I doubt if it is still under warranty.

I'm going to order some disks soon but they're probably going to be solid state drives. For this project the power consumption and minimal heat considerations dictate their use and most likely for anything else I'm working with.

I do know that right now I'm about to disassemble the SSD-RAID-10 array and attache (1) SSD disk to the Tower of Pi for now using these usb-sata adapters I located today.

I have immediate need of the remaining SSD in another project and I've decided there will be NOTHING important on the Tower of Pi super computer at the moment that requires RAID.

In truth I have a huge NAS that will serve just as well for the Tower but I wanted it to be self-sufficient to itself so I can pick it up and deploy it anywhere.

I will re-utilize the usb-sata RAID controller shortly it seemingly works well with ssd disks and usb 2.0… On a "normal" system that USB 2.0 RAID controller would work very well with a matched pair of ssd disks I bet and I intend to find out soon enough.


Updated: I tried that failed WD Scorpio disk in a laptop running windows about 3-hours later and the stupid drive is working now so I'm not sure but either way it's not going to be used again. I'm running a disk scanner on it now to see if it can spot any bad blocks etc. I'm sure it's some kind of thermal-related failure and that the disk is NFG.


I had to track down some parts today visiting two different post offices to locate the bi-color leds and holders and the (3) Airlink 101 wifi adapters. I am still missing (2) usb-sata external enclosures.

If I have time today I'll try to wire up the bi-color leds to the RAID-10 controller.


I ordered (2) more RPI, 10-more 4GB SD cards (micro-sd w/adapters), two more RPI enclosures, 4-more GPIO cables (one pair has been made with a flexible tube of tougher material to both "protect" the ribbon cable as well as allowing them to create a much more flexible GPIO cable) and the other pair are just standard 12" ribbon cable with 26-pin connectors for the GPIO.

I need the other 2 RPI for hardware development work and I plan on working on writing a lot more python to control hardware. I have to build some external display so the Tower of Pi (the node01 controller) can stream status messages.

Corsair 120GB RAID-10 Status

By now I've had a chance to beat on the SSD under linux using the RAID-10 controller and it's holding up well.

I've had no more issues with the disk locking up or causing the RPI to freeze. I don't really know what happened and I'd hoped to use NTFS so that I could just mount them from a windows laptop as needed but that is not to be.

I don't mind using ext4 and right now the configuration I'm using is optimized for SSD.

I also want to say that the SSD are remaining very cool while on continuously for at least 80-hours so far. I turned it off the last time to mount the SSD RAID if I recall.

I'm also compiling software, still transferring (via rsync from my work location) the entire Raspbian repo with the intent to keep a mirror at home as well as preserving it for posterity (meaning MY CONTINUED USE) in case the project sites ever die or the net itself is unavailable for me.

I am waiting for parts, some are missing. When you live in rural America sometimes logistics is difficult nowadays. I'm missing a shipment of (2) USB-SATA enclosures, a bunch of LEDs. That is two separate shipments in about a month that have gone missing.

Changes to bring up rpcbind and nfsd at boot

I have been procrastinating but finally node01 server has booted without NFS and rpcbind for the last darned time. I finally got around to fixing this. I guess I finally got tired of bringing it up manually. I mention it here as the only proper way to do this under Debian linux IMHO.

pi@node01 ~ $ sudo update-rc.d rpcbind enable && sudo update-rc.d nfs-common enable
update-rc.d: using dependency based boot sequencing
update-rc.d: using dependency based boot sequencing
pi@node01 ~ $

The problem is when the rack is coming up after power-off or whatever I wish I could delay the NFS clients a bit so that the NFS server has a chance to come up first… I know I could hack some sleep statements into some rc script but I am loath to butcher the operating system.

Since I'm so lazy I just run something like: "mpirun -f /SSD/mpi_tests/haltfile sudo mount -a" usually works. However once in awhile I have to login and either reboot that node or perform the mount again manually.

New Parallel Processing Password Cracking Fun

I compiled the latest version of John The Ripper with all the latest MPI support (for some reason I built a much older version earlier that was decidedly broken)…

In one window of screen I issue:

pi@node01 /SSD/tmp/john/john-1.7.9-jumbo-7/run $ mpirun -n 9 -f /SSD/mpi_tests/machinefile ./john mypasswd
Loaded 1 password hash (sha512crypt [32/32])
MPI: each node processing 1/9 of 1081 rules. (uneven split)
MPI: each node loaded 1/9 of wordfile to memory (about 739 KB/node)

To check the status of your "cracking run" after a few minutes (it won't have any stats for you if you try too soon) issue in another window from the "master" controller for your cluster:

+ mpirun -n 9 -f /SSD/mpi_tests/machinefile ./john --status=john
  0: guesses: 0 time: 0:00:50:01 0.00% (2) c/s: 10.49
  2: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 15.01
  8: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 15.03
  4: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 13.18
  6: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 15.05
  1: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 13.12
  5: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 15.04
  3: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 15.04
  7: guesses: 0 time: 0:00:50:00 0.00% (2) c/s: 15.04
SUM: guesses: 0 time: 0:00:50:02 0.00% (2) c/s: 127 avg14.11
+ mpirun -np 9 -f /SSD/mpi_tests/machinefile /SSD/mpi_tests/system
Fri May 10 22:38:11 MDT 2013 node01 22:38:11 up 2 days, 8:08, 11 users, load average: 3.21, 2.59, 2.42
Fri May 10 22:38:16 MDT 2013 node03 22:38:16 up 2 days, 8:08, 0 users, load average: 1.14, 1.13, 1.12
Fri May 10 22:38:16 MDT 2013 node06 22:38:16 up 2 days, 8:08, 0 users, load average: 1.05, 1.09, 1.07
Fri May 10 22:38:16 MDT 2013 node04 22:38:16 up 2 days, 8:08, 0 users, load average: 1.04, 1.07, 1.05
Fri May 10 22:38:16 MDT 2013 node05 22:38:16 up 1:01, 1 user, load average: 1.23, 1.14, 1.08
Fri May 10 22:38:16 MDT 2013 node07 22:38:16 up 2 days, 8:08, 0 users, load average: 1.13, 1.06, 1.05
Fri May 10 22:38:16 MDT 2013 node08 22:38:16 up 2 days, 8:08, 0 users, load average: 1.13, 1.08, 1.05
Fri May 10 22:38:16 MDT 2013 node02 22:38:16 up 2 days, 8:08, 0 users, load average: 1.12, 1.11, 1.08
Fri May 10 22:38:16 MDT 2013 node09 22:38:16 up 2 days, 8:08, 0 users, load average: 1.11, 1.13, 1.08
+ sleep 60

You're going to need real dictionary files so get them here: Kevin's Word List Page

This is what it looks like when you actually guess the password.

raspberry (pi)
Node 6: All hashes cracked at 0:01:02:03! Aborting other nodes.
application called MPI_Abort(MPI_COMM_WORLD, 0) - process 6

As you might have guessed I was running a test to see how long it took to find 'raspberry' from node01's own password files. Node 6 which in this code is actually node07 wins the "First Parallel Password Cracked Award" from me.

Audio Tests

I'm also collecting notes here so in order to test your speakers if you don't have any nice music, simply issue:

$ speaker-test -t sine -f 440 -c 2 -s 2

speaker-test 1.0.25

Playback device is default
Stream parameters are 48000Hz, S16_LE, 2 channels
Sine wave rate is 440.0000Hz
Rate set to 48000Hz (requested 48000Hz)
Buffer size range from 256 to 8192
Period size range from 256 to 8192
Using max buffer size 8192
Periods = 4
was set period_size = 2048
was set buffer_size = 8192
  - Front Right

See the man page for speaker-test(1) for information on those switches.s


I am going to install celery before I sleep tonight. I have to learn a lot more about celery any way so we'll see what I can do with it internally here. We use celery where I work but it's a much older version it seems.

If you need to read your unix disk partitions under Windows, I highly recommend DiskInternals Linux Reader see I used this to retrieve some files on an SD card.


Here is my first attempt at mounting the SSD RAID-10 sub-system. I say "first attempt" as I'm not very happy how the mount is designed. I'm working with hand tools and I don't have a workshop. I'm also very pleased (so far) with the SSD-RAID-10 system using the Corsair SSD drives.

I also have some bi-color leds on order, I need to wire up the disk use indicators. It's all about the "blinking lights" jokingly said but I need to get those working. They're very useful to me as I have no other indicators for disk activity right now.


I lost another SD card today, this time on the master and I didn't have a recent-enough backup. This was the only true SD card I had, the rest were micro-sd cards. It pays to have spares.

Fortunately the software I'd built was on the SSD and all the data I've been working with and downloading from work was safe. I also had a backup of the repo data on the NAS which I'd rsync'd there earlier. I have over 30gb on the SSD by now.

I haven't had a chance to check the 4gb SD card that failed to see if it's just corrupt or if it's really bad.

I have MPI working again and I did a full "refresh" of node01 (you can also see an ethernet cable plugged in, I was using that until I got the airlink wifi adapter working again.)

pi@node01 /SSD/mpi_tests $ mpirun -np 9 -f ./machinefile ./system;mpirun -np 9 -f ./machinefile ./cpi
Tue May 7 22:14:48 MDT 2013 node01 22:14:48 up 1:18, 3 users, load average: 0.50, 0.21, 0.16
Tue May 7 22:14:50 MDT 2013 node07 22:14:50 up 3:47, 0 users, load average: 0.00, 0.01, 0.05
Tue May 7 22:14:50 MDT 2013 node04 22:14:50 up 3:47, 0 users, load average: 0.00, 0.01, 0.05
Tue May 7 22:14:51 MDT 2013 node08 22:14:51 up 3:47, 0 users, load average: 0.00, 0.02, 0.05
Tue May 7 22:14:51 MDT 2013 node02 22:14:51 up 3:47, 2 users, load average: 0.00, 0.01, 0.05
Tue May 7 22:14:51 MDT 2013 node05 22:14:51 up 3:47, 0 users, load average: 0.00, 0.02, 0.05
Tue May 7 22:14:51 MDT 2013 node03 22:14:51 up 3:47, 0 users, load average: 0.00, 0.01, 0.05
Tue May 7 22:14:51 MDT 2013 node09 22:14:51 up 3:47, 0 users, load average: 0.00, 0.01, 0.05
Tue May 7 22:14:56 MDT 2013 node06 22:14:56 up 3 min, 0 users, load average: 0.10, 0.25, 0.13
Process 1 of 9 is on node01
Process 7 of 9 is on node07
Process 6 of 9 is on node06
Process 5 of 9 is on node05
Process 3 of 9 is on node03
Process 2 of 9 is on node02
Process 4 of 9 is on node04
Process 8 of 9 is on node08
Process 9 of 9 is on node09
pi is approximately 3.1415926544231256, Error is 0.0000000008333325
wall clock time = 0.207486
pi@node01 /SSD/mpi_tests $

I had to reinstall distribute, pip and mpi4py again as well. Once I have everything rebuilt I plan to make an image of the master node. I already have "slave" node images.

1. sudo apt-get install python-dev
2. curl | sudo python
3. curl | sudo python
4. sudo pip install mpi4py

I think what I have to do is log everything here. I don't want to have to recreate my work again and although it's going very fast I forgot that node05 also needed to have it's mpi-enhanced python stuff rebuilt/reinstalled. I have to make new images shortly for both the master node and the slave nodes.

I guess I had forgotten to re-run the mpi python tests post node05 rebuild last week.


This is how it looks right now. I have not had time to mount the SSD RAID.


I cut the piece of acrylic that will hold the SSD RAID-10 assembly but had no time to drill the mounting holes.

I am also loath to power the Tower of Pi down right now.


I was using a USB cd drive earlier and realized some might be looking for this information.

To mount a CD drive under debian (Raspbian):

# mount -t iso9660 -o ro,unhide /dev/sr0 /mnt

This was not attached to the RPI directly but via a USB hub. I'm not ready to say that issue with the RPI locking the wifi-adapter during heavy usb i/o is solved but it has not happened yet and I've been hammering on that USB disk system. I was using to copy huge wav files onto the RPI so I could test the parallel bladeenc encoder.

[54522.615803] usb 1- new high-speed USB device number 8 using dwc_otg
[54522.717147] usb 1- New USB device found, idVendor=04da, idProduct=0d11
[54522.717179] usb 1- New USB device strings: Mfr=1, Product=2, SerialNumber=3
[54522.717195] usb 1- Product: CDRCB05
[54522.717210] usb 1- Manufacturer: GENERIC
[54522.717223] usb 1- SerialNumber: CDRCB05A0179088
[54522.727260] scsi1 : usb-storage 1-
[54523.755808] scsi 1:0:0:0: CD-ROM GENERIC CDRCB06 1.00 PQ: 0 ANSI: 0
[54523.837431] sr0: scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray
[54523.837461] cdrom: Uniform CD-ROM driver Revision: 3.20
[54523.847244] sr 1:0:0:0: Attached scsi CD-ROM sr0
[55135.925508] ISO 9660 Extensions: Microsoft Joliet Level 3
[55136.430585] ISOFS: changing to secondary root

I was able to build parallel Blade encoder on the RPI and it's encoding a test file right now.

pi@node01 /SSD/tmp $ mpirun -f /SSD/mpi_tests/machinefile /SSD/bladeenc/bin/bladeenc -160 test.wav

BladeEnc 0.92.1b5 (c) Tord Jansson Homepage:

BladeEnc is free software, distributed under the Lesser General Public License.
See the file COPYING, BladeEnc's homepage or for more details.

Files to encode: 1

—> Encoding with blazing parallel speed (8 slaves)!
Encoding: test.wav
Input: 44.1 kHz, 16 bit, stereo.
Output: 160 kBit, stereo.

Status: 0.5% done, ETA 01:44:01 BATCH: 0.5% done, ETA 01:44:01
Status: 52.2% done, ETA 00:16:48 BATCH: 52.2% done, ETA 00:16:48
Status: 76.8% done, ETA 00:07:58 BATCH: 76.8% done, ETA 00:07:58
Status: 99.9% done, ETA 00:00:00 BATCH: 100.0% done, ETA 00:00:00
Completed. Encoding time: 00:34:53 (1.52X)

All operations completed. Total encoding time: 00:34:54
pi@node01 /SSD/tmp $

test.wav is 60-minutes of audio so wasn't surprised by the estimated lengthy encoding time. It actually took far less time of course.

It produces broken mp3 files which can be fixed with mp3val of course but the general quality of the bladeenc encoder leaves much to be desired and I'm not sure it even supports VBR in the version I have modified for MPI (parallel processing).

It is such a shame since here is everyone's ultimate encoder farm engine but too flawed for real use. I wish I were a software developer of the caliber required to refactor all of this code.

I've got about 20gb of the Raspbian repo copied down from my work location. I stopped it awhile ago to see about copying it to my NAS from the node01 RPI cluster. I discovered copying from the RPI node01 SSD array worked fine but takes all the horsepower the RPI node01 has to spare (thrashes the USB wifi, thrashes the node01 cpu) so I wrote a small script to calculate the md5 of the source, copy the file to the NAS via NFS, then compare to the md5 of the destination file, then delays 60-seconds.

I need to take some pictures of the Tower of Pi since I re-wired everything to use molded cables. It looks much cleaner. I also have not had a chance to design a mount to hold the SSD-RAID-10 module.

I am having way too much fun playing with the parallel-processing aspects of this cluster to want to turn it off and drill and file and measure and saw right now.


I have reformatted the SSD disks and then recreated a filesystem (ext4) early evening yesterday and I think it is working better than before. Go figure. One of the reason I also reformatted is that I wanted to make sure it was optimized for SSD. I don't think I got that right using ntfs so I switched back to unix which is something I know a lot more about.

mkfs.ext4 -O extent -b 4096 -E stride=128,stripe-width=128 /dev/sda1

I've written about 6gb to the filesystem since last night, I'm using rsync to mirror the entire repo from my work where I set up a private repo to test things with. Once I get it mirrored to home then I will just keep it in sync with regular mirror runs. The repo is about 90gb so I used tar piped to gzip then split so I can transfer in 20mb chunks. I also want to experiment with a build server but right now I am still focusing on finishing the hardware and getting MPI doing something useful.

I've read about how overclocking caused sd card corruption and now I've seen node01 /etc/apt get mangled today. I was getting gpg signature errors and even though I kept updating the right key I kept getting slave errors. I over-wrote /etc/apt with the contents of node02 and it solved the issue :/

I've lowered node01 back to 700mhz for now although I can't rule out poor quality MicroSD cards.

I re-worked all the RPI power cables today so they're all molded now.

I'm still testing the SSD-RAID-10 and it's been working well ever since I reformatted/repartitioned it. I have not yet had a repeat of the heavy i/o failures causing usb wifi to die again.

Updated: I've got the MPI-version of John The Ripper compiled and working on cracking the RPI's own password to see how well it did. You have to apply the crypt patch to get it working. It built fine on arm linux.

pi@node01 /SSD/tmp/john/john- $ mpirun -f /SSD/mpi_tests/machinefile ./john ./mypasswd

Loaded 1 password hash (generic crypt(3) [?/32])
Loaded 1 password hash (generic crypt(3) [?/32])
Loaded 1 password hash (generic crypt(3) [?/32])
Loaded 1 password hash (generic crypt(3) [?/32])
Loaded 1 password hash (generic crypt(3) [?/32])
Loaded 1 password hash (generic crypt(3) [?/32])
Loaded 1 password hash (generic crypt(3) [?/32])
Loaded 1 password hash (generic crypt(3) [?/32])
Loaded 1 password hash (generic crypt(3) [?/32])


Further tests reveal that any use of the disk drive or heavy network i/o causes the RPI usb port to stop working and the airlink wifi interface to freeze. If I were to use ethernet it's somewhat better. I was downloading a some fiiles with rsync and deliberately throttled it to 300KB writing across the net to an nfs mount on node01 and it locked up node01's airlink interface. The ethernet port was still fine and of course the RPI itself was not hung.

This isn't a power supply issue of course (5vdc@10A power supply) and I've been able to repeat this on multiple boards.

I really didn't want all those damned ethernet cables hanging off of it. I'm going to need 9-switch ports as well.


I had one SD card die this morning, when I woke up node05 was down. This was the card that acted very oddly so far any way.

I have all nodes NFS-mounting the SSD RAID from node01.

node01:/media/SSD /SSD nfs defaults,rw,nfsvers=3,tcp,rsize=32768,wsize=32768,hard,intr 0 0


I received the external 2.5" RAID controller/enclosure and so far so good.

I've got it configured in a RAID-10 configuration. 2x120gb ssd disks. I tested it in 'normal' and that just concatenated the disks for 240gb. I didn't try it in RAID-0 though, RAID-10 is what I want.

Later I'll format and partition it. Remember for most ssd 512-bytes is optimal.

[245562.609371] scsi2 : usb-storage 1-1.3.2:1.0
[245563.623375] scsi 2:0:0:0: Direct-Access OEM Ext Hard Disk 0000 PQ: 0 ANSI: 5
[245563.671819] sd 2:0:0:0: [sda] 235909321 512-byte logical blocks: (120 GB/112 GiB)
[245563.700686] sd 2:0:0:0: [sda] Write Protect is off
[245563.700727] sd 2:0:0:0: [sda] Mode Sense: 10 00 00 00
[245563.729176] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[245563.865592] sda: sda1
[245563.865678] sda: p1 size 235914462 extends beyond EOD, enabling native capacity
[245563.969982] sda: unknown partition table
[245564.182899] sd 2:0:0:0: [sda] Attached SCSI disk
root@node01:/var/log# fdisk -l /dev/sda

Disk /dev/sda: 120.8 GB, 120785572352 bytes
255 heads, 63 sectors/track, 14684 cylinders, total 235909321 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00dd00dc

Disk /dev/sda doesn't contain a valid partition table

root@node01:/var/log# fdisk -H 32 -S 32 /dev/sda

root@node01:/var/log# mkfs.ntfs -L SSD -H 32 -S 32 -v /dev/sda1
Cluster size has been automatically set to 4096 bytes.
Initializing device with zeroes: 30%@@

At 30% into the mkfs it killed the wifi interface. I plugged in the wired network and everything was fine. I resumed my screen session (everyone should use screen) and the mkfs was at 45% (It took me a bit of time to get to reconnecting ethernet as I am actually working right now and the vpn interferes with the local 10-net…)

node01:/media/SSD    /SSD  nfs defaults,rw,nfsvers=3,tcp,rsize=32768,wsize=32768,hard,intr     0       0

[will continue later today]


This is what I did yesterday I reversed a decision and switched to using molded cables as my home-made connectors kept falling apart under repeated handling.


As you can see, I ran out of molded cables but I have a few more on order. I'm hoping they will be here soon.

I have to work on how the power cables attach to the terminal block the wiring isn't very strong there and I need to put up some kind of strain-relief at minimum.

I should be getting the hard drive enclosures this week as well and I'll know if I can implement RAID-10 by the end of the week hopefully. I am really itching to get an NFS server up and running to help centralize the software, and possibly make a build server.

Something in the back of my mind has been a kernel build server where I could so some device driver work again or maybe devise a workaround for the Corsair SSD USB issue in the kernel.

I guess that's all fantasy as I have zero free time. None.

Right now my work is crushing me so I have no time to devote to software either.

I did get the MPI framework working well and I am in search of ideas for how to utilize the system in some useful way. I have more SD cards on order and I want to make some new images soon.

I didn't build this thing and get the MPI frameworking just to run SETI@HOME or something like it.

Even if I have to write my own code. I just don't have any real work to do with a toy super computer right now but I'll think of something.

The above configuration is drawing 4.2A as of tonight.


I have done some redesign work on the tower of pi and right now I am doing a lot of measuring, drilling the aluminum rails to relocate things.


I can't believe what I am about to relate. It seems the Corsair 120GB SSD disks don't work properly with USB. There a few threads discussing this online: and

I can either purchase different SSD drives that have USB support OR stick these disks into another enclosure, perhaps one with a raid controller.

My experience with them was that the kernel sees the new device but the drivers aren't being loaded for scsi emulation and the device nodes aren't created.


pi@node01 ~/mpi_tests $ lsusb
Bus 001 Device 002: ID 0424:9512 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 008: ID 0bda:8176 Realtek Semiconductor Corp. RTL8188CUS 802.11n WLAN Adapter
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 005: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 012: ID 1b1c:1ab8 Corsair
pi@node01 ~/mpi_tests $

For now because I don't want to wait I may just order a usb-sata enclosure and steal the electronics from it.

Updated 1744MT
[20809.972598] usb 1-1.3.3: new high-speed USB device number 10 using dwc_otg
[20810.074139] usb 1-1.3.3: New USB device found, idVendor=1b1c, idProduct=1ab8
[20810.074171] usb 1-1.3.3: New USB device strings: Mfr=1, Product=2, SerialNumber=5
[20810.074188] usb 1-1.3.3: Product: CSSD-R120GB2
[20810.074201] usb 1-1.3.3: Manufacturer: Corsair
[20810.074214] usb 1-1.3.3: SerialNumber: 1006659900FF09680B5A
[20810.092383] scsi1 : usb-storage 1-1.3.3:1.0
[20811.093583] scsi 1:0:0:0: Direct-Access     Corsair  CSSD-R120GB2          PQ: 1 ANSI: 0

I'm not sure what the hell happened but it started showing up in dmesg (it was quiet earlier). I'm going to check this out further and update again. It never created the device nodes and that matches every one else's experience I gather.

I ordered an external 2.5" two-bay raid enclosure that I can plug both SSD into and then use RAID-10. It's powered by 5VDC and if it works well enough at the excruciatingly slow usb-pi speeds I'll be pleased. I don't know how I will incorporate it into the design just yet.


Meanwhile, I finally got mpi4py working properly, installed and built with pip(1) as it should be done. I also fixed a dumb bug (to me) that starts both and cpi with zero instead of one. That is why *my* results display the logical and correct values. (lol) I also removed every vestige of openmpi. This was causing problems with mpich2 somehow things just didn't work right. I hope to revisit that later.

My version:

pi@node01 ~/mpi_tests $ date;set -x;mpiexec -f machinefile /home/pi/mpi_tests/cpi;mpirun -machinefile machinefile python
+ date
Thu Apr 25 18:02:00 MDT 2013
+ set -x
+ mpiexec -f machinefile /home/pi/mpi_tests/cpi
Process 1 of 9 is on node01
Process 5 of 9 is on node05
Process 4 of 9 is on node04
Process 6 of 9 is on node06
Process 2 of 9 is on node02
Process 9 of 9 is on node09
Process 8 of 9 is on node08
Process 7 of 9 is on node07
Process 3 of 9 is on node03
pi is approximately 3.1415926544231256, Error is 0.0000000008333325
wall clock time = 1.348193
+ mpirun -machinefile machinefile python
Hello, World! I am process 1 of 9 on node01.
Hello, World! I am process 2 of 9 on node02.
Hello, World! I am process 6 of 9 on node06.
Hello, World! I am process 9 of 9 on node09.
Hello, World! I am process 7 of 9 on node07.
Hello, World! I am process 8 of 9 on node08.
Hello, World! I am process 5 of 9 on node05.
Hello, World! I am process 3 of 9 on node03.
Hello, World! I am process 4 of 9 on node04.
pi@node01 ~/mpi_tests $


pi@node01 ~/mpi_tests $ date;mpirun -machinefile machinefile python;date
Thu Apr 25 15:19:02 MDT 2013
Hello, World! I am process 0 of 9 on node01.
Hello, World! I am process 1 of 9 on node02.
Hello, World! I am process 2 of 9 on node03.
Hello, World! I am process 4 of 9 on node05.
Hello, World! I am process 5 of 9 on node06.
Hello, World! I am process 3 of 9 on node04.
Hello, World! I am process 7 of 9 on node08.
Hello, World! I am process 6 of 9 on node07.
Hello, World! I am process 8 of 9 on node09.
Thu Apr 25 15:19:05 MDT 2013
pi@node01 ~/mpi_tests $


I was able to build the latest mpich2 code but it's not working right, most likely due to some misconfiguration which I don't really have time to chase.

I installed the binary distribution from the archive and it works so I'm happy with that for now.

pi@node01 ~/mpi_tests $ mpiexec -f machinefile hostname

## /etc/hosts node06 node05 node07 node02 node03 node04 node08 node01

pi@node01 ~/mpi_tests $ mpiexec -f machinefile ./cpi
Process 7 of 8 is on node01
Process 1 of 8 is on node05
Process 2 of 8 is on node07
Process 3 of 8 is on node02
Process 0 of 8 is on node06
Process 5 of 8 is on node04
Process 4 of 8 is on node03
Process 6 of 8 is on node08
pi is approximately 3.1415926544231247, Error is 0.0000000008333316
wall clock time = 0.540583
pi@node01 ~/mpi_tests $

I was able to get python working as well.

pi@node01 ~/mpi_tests $ mpirun.openmpi -machinefile ./machinefile /usr/bin/python /home/pi/mpi_tests/
Hello, World! I am process 7 of 8 on node01.
Hello, World! I am process 1 of 8 on node05.
Hello, World! I am process 3 of 8 on node02.
Hello, World! I am process 0 of 8 on node06.
Hello, World! I am process 5 of 8 on node04.
Hello, World! I am process 4 of 8 on node03.
Hello, World! I am process 2 of 8 on node07.
Hello, World! I am process 6 of 8 on node08.
pi@node01 ~/mpi_tests $

pip is now installed and working as well:

pi@node08 ~/pip_testing $ pip search raspberry
RPIO - Advanced GPIO for the Raspberry Pi. Extends RPi.GPIO with PWM, GPIO interrups, TCP socket
interrupts, command line tools and more
RPi.GPIO - A module to control Raspberry Pi GPIO channels
INSTALLED: 0.5.2a (latest)
wiringpi2 - A python interface to WiringPi 2.0 library which allows for easily interfacing with the
GPIO pins of the Raspberry Pi. Also supports i2c and SPI
piui - Add a mobile UI to your RaspberryPi project.
wiringpi - A python interface to WiringPi library which allows for easily interfacing with the GPIO
pins of the Raspberry Pi. Also supports i2c and SPI
rpitwit - Remote control your RaspberryPI from Twitter.
pyblinkm - Drive a BlinkM with Python via I2C using python-smbus on Raspberry Pi.
raspberry - Raspberry PI library
pi@node08 ~/pip_testing $

Updated: OpenMPI breaks mpich2 so I removed all the python stuff for now. I think things went bad when I installed python-mpi4py package.

2013-04-22: I rescinded my self-imposed restriction and built the mpich2 code and installed it on the master node.

That took a long time to compile as well.

I'll post the config log and some other information to help document this.

Then I made an image of the SD card using win32diskimager and am cloning it onto the other 7 units as I update this page.

That process of writing the image back to the SD card takes entirely too long ugh.

This reveals all the dust on my desktop systems (which happen to be sitting in a 19" rack) so I have to pull them out this weekend and clean them with an air-compressor.

When you live in the desert southwest in a cabin on a mountain at almost 8K feet AND use a wood stove all during the cold months, you have a real dust problem. My that was a long sentence.

I can't wait to pick up my last RPI unit, it should be here by midweek.

I'm almost ready to test 8 nodes of the super computer today hopefully.

I can't make any more progress on the hardware so I thought I'd make up for it by working on the software in parallel.

I'm waiting for 2x120GB Solid State Drives and some time to measure and cut more acrylic sheet.

I'm also working on a design to mount the SSD which I am hoping won't use up that much space inside the rack.

I have to investigate fabricating or buying two very short USB-A to mini-b cables to attach the SSD with since I hope to mount them right next to the internal usb hub. This could mean having much more available space in the rack than I anticipated.


I made some progress this weekend.


I worked a bit more on the project as well, mounting 3-more computers permanently. As you can see I have (6) permanently mounted so far, the two on top are still not attached as I am not yet sure of how large the area needs to be to accommodate the disk system. I checked the current drain and it's only 2.8A with 8-units powered up and in operation.

I also made 3-more custom power cables, but I ran out of the smaller gauge of wire.

I was walking around with it while it was on earlier (attached to a ups) while moving it from my temporary workbench to be closer to my office area.

I’ve got them all attached with wifi now so no wires.

You can see the little wifi-dongles attached to each unit above.

That usb-hub is so handy to charge my ipod touch or cell phone plus I can copy the images from them as needed.

On the “master” unit I’ve got the wifi-dongle plugged into the usb hub.

It’s usb 2.0 so performance will be acceptable.

I've "benchmarked" a few file transfers and read performance over NFS has been great as well…

The open areas between each group of 3-RPI units will be covered with acrylic and will each contain something important to the rack system.

The second empty area below the top will be where I install a disk system shared by all units with (2) Corsair Reactor Series CSSD-R120GB2/RF2 2.5" 120GB USB 2.0 & SATAII Internal Solid State Drives (SSD).

I got them from I can directly attach them to the master unit's usb hub and I'll have (2) SSD with zero moving parts keeping along the same lines as the RPI itself with it's SD card boot and operating system disk.


They are reliable with over 1-million hours MTBF.

Maximum sequential read speed 250MB/s
Maximum sequential write speed 170MB/s
128MB DRAM cache (so it will seem much faster)
Direct USB 2.0 connectivity
This SSD can withstand 40G of shock. I'm not planning on launching it into space but it's good to know it's going to be more rugged now as well.
Each drive is using 2W (max) when active and when in standby they are using 0.5W.
Note: 2W@5VDC is 400ma at peak use which is well within the specifications of my internal usb hub. That makes me very pleased they were a great choice IMHO.

For right now it will be served via NFS from the "master" unit.

I've had no time to work on the clustering software and since the hardware is still in the design phase I'll not worry about that until the dust (and chaos) has settled.

I'm still hoping to devise something better to power it with than what I have now. I wanted something very cool very high tech and "smart" in all senses.

The above-pictured power system isn't what comes into mind even though it is well engineered.

I located a 1U rack-mounted server power supply today and it is very hefty.

I will have to make some major modifications to it to work without the computer soft-switch to bring it up.

Last but not least I am awaiting the arrival of the 9th RPI unit and then the "cluster" will be complete.

I'm also working on the software to power the super computer cluster but it's going slowly since I still am designing the hardware that will be available.


I received the Airlink USB wifi-dongles today. I have eight units up and running on wifi and I have the rack units sitting in a corner in another room right now observing the wifi dongle performance. I need to remember to re-measure with clamp-on ammeter with the wifi-dongles installed.

All you have to do is this:

# /etc/network/interfaces
auto lo

iface lo inet loopback
iface eth0 inet dhcp

allow-hotplug wlan0
auto wlan0
iface wlan0 inet dhcp

#wpa-roam /etc/wpa_supplicant/wpa_supplicant.conf
#iface default inet dhcp
wpa-ssid "YOURSSID"
wpa-psk "yourpass"

If you want to use NFS you will need to start rpc.statd which comes upwhen /etc/init.d/rpcbind is run.


I ordered the last RPI unit and a 4gb SD card.


I have (6) units running from the new switching supply. I have to say I am unhappy about the amount of noise generating by the supply under load. It buzzes and it's annoyingly-loud. ETA I found out why this is happening. I need to get a pure-sine-wave inverter. The rack of equipment I use is permanently powered from 6 100AH batteries running full time from an inverter. The switching supply does not like running from a modified-sine-wave inverter. It runs nice and quiet from a small UPS I have so that's where it's going to live for now.

The good news is that with six-units and the USB Hub it's only drawing 2.6A from the +5VDC supply.

I'm running a burn-in test on the power supply.


I'll update this later today perhaps.


I made some progress today. I received (3) 5VDC@16A switching power supplies today.

I already wired up the first 3-units and the usb hub directly now and everything pictured was drawing 1A.

I didn't have it connected to the network which would probably use another 500ma (.5A) for all three units.

I have the outputs fused on that power supply but not the input. I can't seem to locate a fuse holder for some reason despite searching through all my parts. I need to get a 2.5A line fuse on the input as soon as possible.

I have replaced the micro-usb molded cables with my own wired power cables which will all be joined into a single wiring harness when things are completed. If you look closely you will see I even shrink-wrapped the plug.

I also removed the power connector from the usb hub circuit board (I am very rusty with a desoldering iron) and it took some time but I re-wired it to the 5VDC harness directly.

I have to make an enclosure for the power supply as well but right now I have everything just sitting on a piece of plywood as you can see.



I made some more progress today. I received all the aluminum enclosures so I have installed all but one for the ninth RPI unit which I have not yet ordered.

I got some more of those machined aluminum enclosures today so I put together 5-more systems for a total of (8) now.

These are just sitting stacked together on top of each other but the interval will actually be three units then another acrylic sheet with another USB hub or hopefully a usb-sata hard disk sub-system.


I got the USB-SATA enclosures today from Amazon but both were DOA. This sets my timetable back a lot. They were cheap garbage I guess and I am returning them at Amazon's expense.

I really need to get (2) usb-sata enclosures (then strip the interface card out and ditch the enclosures) so I have to buy some others ASAP.

I need a couple more 2.5" drives to work with since I only have one right now.

I want to see if I can setup a private repo for arm linux locally and just mirror the source once a day or depending on update frequency once a week.

This way I don't need the net to add new software whenever, wherever for this RPI cluster.

My plan was to have a 160g hard disk shared among all units but right now I don’t have a USB-SATA device to scavenge so it's hard to finalize the dimensions of the next acrylic-section revealing exposed circuit boards (and hopefully a usb-sata hard drive sub-system.).

I should get the (3) 5VDC@16A supplies Friday if I didn’t misunderstand the UPS tracking information.

I may end up only using (1) USB hub. I need to drill the mounting holes and then mount at least (3) units, perhaps Friday.

I may only be able to mount (3) more RPI units since I don’t know for sure if I will be installing another powered USB hub or a USB-SATA hard disk system just yet so the next acrylic section's dimensions are as of now still uncertain.


Since I will be using a switching supply I don't need all those hubs.



I made some progress today.


I need to order 10 of these wifi adapters. I can't stand having all those ethernet cables.



I received the power connectors (micro-usb) that I was waiting for today.


I have to wire up 9-cables for 5.1VDC and ditch those huge molded power cords running into each one.

I will use red and black wires with everything combined into a small wiring harness.

I also received (3) more RPI units today so I have eight now.

I'm waiting for the new enclosures and that's about it.

I have to image (3) micro-sd cards tonight with linux and test the new RPI computers and then wait for the other enclosures to show up.

I ordered (3) Power One MAP80-S157 DC Power Supply 5 VDC 10 Amp units from EBAY


The (3) power supplies cost $53.00 with shipping.


The other two usb hubs arrived today so I was able to mount them. I used some tie-wraps to keep the wiring somewhat more organized.



The post office misplaced one of the packages with the rest of the nylon hardware in it. I had to go back today and ask them to check again, and it turned out that they had it all along.


In case you were wondering, I am leaving the protective film on the acrylic until final assembly.


I ordered more aluminum cases, 2 more USB hubs, 2 USB 2.0 external hard drive enclosures, 3-more micro-sd cards and 3-more RPI.


The nylon hardware is taking forever to show up from Amazon.

I'm going to mount the (3) cases into a rack shortly.



I ordered a bunch of nylon hardware I'll need to build this as well as re-designed the base of the supercomputer rack.


I ended up using a table-saw cutting these slots to the depth of .75 inches.


I have to get the base slots cut out for the aluminum rails that will be fitted into that, as well as some acrylic sheet cut into size before I can proceed. I don't have anything that will cut those slots for the base. I have to get set up to make a lot of precision holes in both the aluminum rack and the acrylic sheets.


I ordered (3) more RPIs which will bring me up to (5) units. I also ordered 3-more micro-sd cards.

Here is the base of one of my versions of the "The Tower Of Pi" …

2013-03-24-stand-base.jpgI also located a source for these, so I can wire up my own power harness.

Updated: I just ordered 50 of the below connectors for $20 but shipping will take a long time.


I chose this vendor from EBAY. I liked having a shell over the connector, otherwise my only option was shrink-wrap.



I bought a 36"x30" sheet of transparent acrylic stock to cut into rectangles to mount at least 3-RPI vertically per stack.
I also bought a bunch of aluminum stock and I've picked out a piece of 2x8 wood from my scrap pile which will become the base. I also bought a bunch of nylon hardware but was too large in diameter to use through existing holes in the RPI circuit board but I can most likely use them elsewhere. I have to sand the wood and then paint it.
I received RPI #2 today, plus (2) AdaFruit Pi cases. I promptly imaged the SD card, powered it up and logged in. I used Win32DiskImager to do this under Windows XP. You can get that from sourceforge. Yes, of course you can just use dd(1) under unix.

It really helps to have this built-in to your desktop for stuff like this.

2013-03-21-readers.jpgI re-sized the filesystem as well. I have them all turned off now. I put them both into the cases to protect them while I am working on prototyping everything. I will order another RPI in two weeks.


I received my D-Link Powered USB Hub today and promptly removed it from the case never to be used again.

It's the perfect size.2013-03-21-014-small.jpg

It comes with a hefty 5A power supply which I intend to depend on for awhile. The (2) ports on the right are the charging ports.


I plan on using something unique for a shared disk storage. See The GNU/Linux "usbnet" Driver Framework

Design Method 3:


I ordered a second RPI yesterday a long with a breakout board, 2 new enclosures to keep the circuit board safe while prototyping. I've worked on some designs for a unique rack of RPIs that will hopefully be a self-contained cluster.

I learned something about the RPI today, I was working with amixer trying to resolve some issues I had with documentation I'd found online (most of it is nonsense coming from non-unix people and/or non-engineering people.

Two methods to adjust volume using amixer on the RPI:

1. amixer -c 0 — sset PCM playback -10db
2. amixer cset numid=3 50%

Basically the ONLY device is:

pi@raspberrypi ~ $ amixer
Simple mixer control 'PCM',0
  Capabilities: pvolume pvolume-joined pswitch pswitch-joined penum
  Playback channels: Mono
  Limits: Playback -10239 - 400
  Mono: Playback 0 [96%] [0.00dB] [on]
pi@raspberrypi ~ $ amixer scontrols
Simple mixer control 'PCM',0
pi@raspberrypi ~ $

I've also experienced issues with mpg123 losing sound when adjusting the volume in the middle of playback so you're not alone in that.

Here are some of the prototypes drawings I'm working on with aluminum and lexan sheets.

Design Method 1:


Design Method 2:


Each one of the above sits in a rack (front-view) like the below:



I'm studying the little Raspberry Pi single-board computer system. I have made some grand plans which include building a "supercomputer" cluster of RPIs. My goal is to learn both clustering and python-based network computing.

My first project is to build a rack-mount enclosure for at least 10 RPI units. I'll update some of that when I've made progress.

It's all rather grandiose until I have all 10-units here. I have only (1) at this time and I am learning as much about it as time permits.

Right now I'm obtaining materials, power supplies (I'm going to need something unique), powered usb 2.0 hub.

I'm actually going to convert a non-powered usb hub into a powered hub from an external power supply.

I know I can do that quickly as long as I can remove the housing and get access to the circuit board.

The RPI's internal USB circuitry can only support 150ma which is not enough to handle external usb hard drives, wifi, flash memory sticks, etc.

I've seen the effects personally when your power supply is marginal as well (I'm using a cellphone charger) most ESPECIALLY when using the network (10/100 ethernet) as that is driven from the USB controller as well.

The system just randomly freezes OR memory sticks stop responding and only a power cycle will clear the error, etc. You can't really predict a specific failure example given that it's just a power supply issue.

SO… the hacked-up internal usb hub will be directly connected to some power supply probably about 20A at 5vdc.

I need 20A because I want to use this to power all 10-units.

I also have to custom fabricate a bunch of micro-usb cables to feed power into all those computers.

To Be Continued…