Monolith 2015 03

Updated 2015-03-13

I finally received the order for the last 11-cpu boards from That didn't work out well I ended up having to place a paypal claim to get them to send them to me. What a shame to end our relationship on such a sour note.

I don't think I will purchase anything from them again. I placed the order on Feb 13 - they got paid quickly but they didn't seem to feel the need to ship my property to me very quickly. Enough said about that matter. They don't exist for me any longer.

I mounted the 11 cpu units on the threaded rods with spaces and cut (2) pieces of acrylic plastic for the end-pieces.

I have the new tier held in the rack by a pair of vise-grip pliers until I can drill new holes and secure with screws.

2015-03-13-001-small.jpg 2015-03-13-003-small.jpg

I also built the wiring harness and soldered it into the main +5VDC harness.

I had (6) 4GB Adata cards and (2) Airlink 101 USB wifi adapters so I configured node 55 and node 56 today.

I built new boot disks (micro sd cards) for nodes 57-60 as well.

I have more micro sd cards and more USB wifi adapters on order. Enough to complete the Cubical Monolith Project.

I tested all the new cpu units and everything seems perfect. I was in a hurry to test them before I close the paypal case against the vendor.

If you notice in earlier journal entries I installed the bottom tier before the middle so I could get a permanent installation of the master node. Today I moved those bottom tier Airlink adapters and cards up into the new middle tier and will repopulate the bottom tier with new cards and wifi adapters. Those cpu units have been well tested.

If I have time this weekend I will mount the middle tier permanently eliminating the need for the vise-grips. As any good engineer knows vise-grips are miracle tools.

Updated 2015-03-16

I finally mounted the last tier permanently.


Updated 2015-03-19

I recieved the remainder of the USB wifi adapters and some more micro sd cards. I am completing loading the operating systems on the remainder of the nodes and bringing them up one by one.

Using 4GB micro sd cards for each node.

1. dd if=image.img of=/dev/sdb bs=1M

dd if=image.img of=/dev/sdb bs=1M
1882+0 records in
1882+0 records out
1973420032 bytes (2.0 GB) copied, 315.323 s, 6.3 MB/s

2. fdisk /dev/sdb (delete partition 2, recreate partition 2 with max size.)

root@master-dev:/master/tmp# fdisk /dev/sdb

Command (m for help): d
Partition number (1-4): 2

Command (m for help): n
Partition type:
   p   primary (1 primary, 0 extended, 3 free)
   e   extended
Select (default p):
Using default response p
Partition number (1-4, default 2):
Using default value 2
First sector (34816-7744511, default 34816):
Using default value 34816
Last sector, +sectors or +size{K,M,G} (34816-7744511, default 7744511):
Using default value 7744511

Command (m for help): wr
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

3. e2fsck -f /dev/sdb2

root@master-dev:/master/tmp# e2fsck -f /dev/sdb2
e2fsck 1.42.5 (29-Jul-2012)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
/lost+found not found.  Create<y>? yes
Pass 4: Checking reference counts
Pass 5: Checking group summary information

/dev/sdb2: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sdb2: 70670/119520 files (0.1% non-contiguous), 355589/477440 blocks

4. resize2fs /dev/sdb2

root@master-dev:/master/tmp# resize2fs /dev/sdb2
resize2fs 1.42.5 (29-Jul-2012)
Resizing the filesystem on /dev/sdb2 to 963712 (4k) blocks.
The filesystem on /dev/sdb2 is now 963712 blocks long.

5. mount /dev/sdb2 /mnt

6. Edit /etc/hosts, /etc/network/interfaces, copy on boilerplate configuration files


Once I have booted the node for the first time I perform several more tasks.

1. sudo apt-get update
2. aptitude install cfengine2
3. cd /var/lib/cfengine2/inputs;scp root@master:/master/tmp/cfengine.tar .;tar xvf cfengine.tar;cfagent -q (the cfengine.tar has the boilerplate cfengine files I need to get cfengine bootstrapped.)

4. pip install mpi4py
5. pip install numpy

Script started on Thu 19 Mar 2015 01:57:25 PM MDT
mpi@master:/master/mpi_tests$ mpirun -np 65 -f /master/mpi_tests/machinefile /master/mpi_tests/cpi
Process 2 of 65 is on node02
Process 3 of 65 is on node03
Process 4 of 65 is on node04
Process 24 of 65 is on node24
Process 18 of 65 is on node18
Process 5 of 65 is on node05
Process 7 of 65 is on node07
Process 50 of 65 is on node50
Process 28 of 65 is on node28
Process 55 of 65 is on node55
Process 42 of 65 is on node42
Process 43 of 65 is on node43
Process 60 of 65 is on node60
Process 14 of 65 is on node14
Process 9 of 65 is on node09
Process 10 of 65 is on node10
Process 17 of 65 is on node17
Process 13 of 65 is on node13
Process 11 of 65 is on node11
Process 53 of 65 is on node53
Process 46 of 65 is on node46
Process 52 of 65 is on node52
Process 6 of 65 is on node06
Process 58 of 65 is on node58
Process 22 of 65 is on node22
Process 37 of 65 is on node37
Process 31 of 65 is on node31
Process 45 of 65 is on node45
Process 64 of 65 is on node64
Process 33 of 65 is on node33
Process 35 of 65 is on node35
Process 19 of 65 is on node19
Process 20 of 65 is on node20
Process 16 of 65 is on node16
Process 41 of 65 is on node41
Process 51 of 65 is on node51
Process 61 of 65 is on node61
Process 63 of 65 is on node63
Process 27 of 65 is on node27
Process 49 of 65 is on node49
Process 40 of 65 is on node40
Process 56 of 65 is on node56
Process 48 of 65 is on node48
Process 30 of 65 is on node30
Process 32 of 65 is on node32
Process 34 of 65 is on node34
Process 36 of 65 is on node36
Process 26 of 65 is on node26
Process 38 of 65 is on node38
Process 21 of 65 is on node21
Process 59 of 65 is on node59
Process 65 of 65 is on node65
Process 39 of 65 is on node39
Process 54 of 65 is on node54
Process 57 of 65 is on node57
Process 25 of 65 is on node25
Process 62 of 65 is on node62
Process 12 of 65 is on node12
Process 23 of 65 is on node23
Process 29 of 65 is on node29
Process 44 of 65 is on node44
Process 47 of 65 is on node47
Process 1 of 65 is on node01
Process 8 of 65 is on node08
Process 15 of 65 is on node15
pi is approximately 3.1415926544231270, Error is 0.0000000008333338
wall clock time = 1.452182
mpi@master:/master/mpi_tests$ exit

Script done on Thu 19 Mar 2015 01:57:44 PM MDT

As I have noted elsewhere in these journals the high wall clock time is because we're using wifi. When I was using ethernet switches it was very low.

Script started on Thu 19 Mar 2015 06:42:05 PM MDT
mpi@master:/master/tmp/john/magnumripper-JohnTheRipper-f110f98/run$ ./john -status=some
1 0g 0:00:18:52  0g/s 30.55p/s 125208c/s 5214KC/s
2 0g 0:00:21:30  0g/s 26.56p/s 108845c/s 4532KC/s
3 0g 0:00:20:16  0g/s 29.44p/s 120702c/s 5026KC/s
4 0g 0:00:23:22  0g/s 24.71p/s 101337c/s 4219KC/s
5 0g 0:00:16:08  0g/s 36.66p/s 150287c/s 6258KC/s
6 0g 0:00:17:30  0g/s 33.58p/s 137616c/s 5730KC/s
7 0g 0:00:23:51  0g/s 25.09p/s 102821c/s 4281KC/s
8 0g 0:00:15:30  0g/s 36.40p/s 149172c/s 6212KC/s
9 0g 0:00:16:32  0g/s 34.06p/s 139537c/s 5810KC/s
10 0g 0:00:23:10  0g/s 25.89p/s 106099c/s 4418KC/s
11 0g 0:00:22:28  0g/s 26.72p/s 109490c/s 4559KC/s
12 0g 0:00:22:51  0g/s 25.53p/s 104651c/s 4357KC/s
13 0g 0:00:16:13  0g/s 36.90p/s 151218c/s 6297KC/s
14 0g 0:00:22:52  0g/s 25.00p/s 102415c/s 4264KC/s
15 0g 0:00:22:08  0g/s 26.36p/s 108045c/s 4499KC/s
16 0g 0:00:18:00  0g/s 31.79p/s 130294c/s 5425KC/s
17 0g 0:00:16:23  0g/s 34.89p/s 142939c/s 5952KC/s
18 0g 0:00:23:07  0g/s 25.47p/s 104399c/s 4347KC/s
19 0g 0:00:22:15  0g/s 25.62p/s 105038c/s 4373KC/s
20 0g 0:00:23:48  0g/s 24.94p/s 102246c/s 4257KC/s
21 0g 0:00:19:09  0g/s 31.61p/s 129561c/s 5395KC/s
22 0g 0:00:17:50  0g/s 33.04p/s 135478c/s 5641KC/s
23 0g 0:00:23:03  0g/s 28.71p/s 117707c/s 4901KC/s
24 0g 0:00:16:08  0g/s 34.67p/s 142058c/s 5915KC/s
25 0g 0:00:23:04  0g/s 24.83p/s 101759c/s 4237KC/s
26 0g 0:00:17:57  0g/s 32.74p/s 134178c/s 5587KC/s
27 0g 0:00:23:01  0g/s 26.55p/s 108792c/s 4530KC/s
28 0g 0:00:20:14  0g/s 27.57p/s 113011c/s 4706KC/s
29 0g 0:00:23:56  0g/s 27.00p/s 110681c/s 4609KC/s
30 0g 0:00:16:15  0g/s 34.95p/s 143293c/s 5966KC/s
31 0g 0:00:23:37  0g/s 24.68p/s 101194c/s 4213KC/s
32 0g 0:00:21:57  0g/s 26.09p/s 106907c/s 4451KC/s
33 0g 0:00:23:23  0g/s 24.31p/s 99591c/s 4147KC/s
34 0g 0:00:15:53  0g/s 37.27p/s 152766c/s 6361KC/s
35 0g 0:00:23:25  0g/s 24.00p/s 98368c/s 4096KC/s
36 0g 0:00:22:21  0g/s 25.70p/s 105312c/s 4385KC/s
37 0g 0:00:17:15  0g/s 33.85p/s 138792c/s 5779KC/s
38 0g 0:00:17:48  0g/s 33.37p/s 136730c/s 5693KC/s
39 0g 0:00:23:09  0g/s 25.31p/s 103764c/s 4321KC/s
40 0g 0:00:23:20  0g/s 25.21p/s 103286c/s 4301KC/s
41 0g 0:00:16:18  0g/s 35.37p/s 144998c/s 6037KC/s
42 0g 0:00:15:49  0g/s 36.21p/s 148387c/s 6179KC/s
43 0g 0:00:18:04  0g/s 32.00p/s 131112c/s 5459KC/s
44 0g 0:00:18:46  0g/s 30.86p/s 126422c/s 5264KC/s
45 0g 0:00:23:18  0g/s 24.44p/s 100159c/s 4170KC/s
46 0g 0:00:17:15  0g/s 33.29p/s 136493c/s 5683KC/s
47 0g 0:00:19:54  0g/s 28.78p/s 117998c/s 4913KC/s
48 0g 0:00:23:31  0g/s 24.96p/s 102314c/s 4260KC/s
49 0g 0:00:16:20  0g/s 36.93p/s 151280c/s 6299KC/s
50 0g 0:00:08:34  0g/s 4.918p/s 20173c/s 840540C/s
51 0g 0:00:20:20  0g/s 27.38p/s 112249c/s 4674KC/s
52 0g 0:00:14:12  0g/s 7.661p/s 31477c/s 1311KC/s
53 0g 0:00:10:20  0g/s 8.567p/s 35209c/s 1466KC/s
54 0g 0:00:20:19  0g/s 29.26p/s 119893c/s 4992KC/s
55 0g 0:00:21:13  0g/s 27.19p/s 111416c/s 4639KC/s
56 0g 0:00:13:18  0g/s 20.77p/s 85100c/s 3543KC/s
57 0g 0:00:20:05  0g/s 29.05p/s 119046c/s 4957KC/s
58 0g 0:00:19:25  0g/s 30.18p/s 123685c/s 5150KC/s
59 0g 0:00:19:01  0g/s 30.56p/s 125326c/s 5218KC/s
60 0g 0:00:16:14  0g/s 35.44p/s 145218c/s 6047KC/s
61 0g 0:00:22:54  0g/s 24.54p/s 100574c/s 4188KC/s
62 0g 0:00:18:22  0g/s 33.50p/s 137312c/s 5718KC/s
63 0g 0:00:23:31  0g/s 26.71p/s 109498c/s 4559KC/s
64 0g 0:00:19:32  0g/s 29.70p/s 121732c/s 5069KC/s
65 0g 0:00:19:42  0g/s 29.34p/s 120250c/s 5007KC/s
mpi@master:/master/tmp/john/magnumripper-JohnTheRipper-f110f98/run$ exit

Script done on Thu 19 Mar 2015 06:42:14 PM MDT

The entire 66-cpu array is using 3.02A @ 110VAC (input) during a full-scale run of John The Ripper on an 180K hash password file. I will let it run overnight just to do a burn in and then probably turn it off. I don't need another $400 power bill.

Internally on the DC side it's much more of course.