CubieBoard 2019 12

2019-12-21 I have been running the cluster the last 24-hours and the cpu temperatures seem to be higher than they were running on Debian Wheezy. I am not sure what is going on but I did order a lot of CPU heat sinks from China and will eventually install them on all 66-nodes. I modified one of my "stress test" scripts to show cpu temperature as well.

#!/bin/bash
# /master/mpi_tests/system
uptime=`uptime`
host=`hostname`
cpu_temp=$(< /sys/class/thermal/thermal_zone0/temp)
cpu_temp=$(($cpu_temp/1000))
host=`hostname`
echo "$host $uptime cpu: $cpu_temp°C"

This has become an issue with one node, node59 which is consistently running at about 96 degrees Celsius during 100% CPU use.
This eventually causes the board to power off in thermal shutdown. I guess I have a bad cpu here as the other boards are running about 10 degrees less at peak CPU use. This never happened with Debian Wheezy however, so I am somewhat dubious.

This is not an environment issue, here are the temperatures for adjacent systems:

mpi@master:/master/mpi_tests$ mpirun -host node58 /master/mpi_tests/system
node58  14:43:10 up  5:14,  0 users,  load average: 0.56, 0.42, 0.70 cpu: 25°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  14:43:14 up 4 min,  0 users,  load average: 0.17, 0.42, 0.22 cpu: 80°C
mpi@master:/master/mpi_tests$ mpirun -host node60 /master/mpi_tests/system
node60  14:43:20 up  5:14,  0 users,  load average: 0.50, 0.41, 0.70 cpu: 47°C
mpi@master:/master/mpi_tests$
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  13:40:43 up  1:08,  0 users,  load average: 1.91, 1.69, 1.60 cpu: 95°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  13:41:07 up  1:08,  0 users,  load average: 1.84, 1.70, 1.61 cpu: 96°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  13:41:17 up  1:08,  0 users,  load average: 1.87, 1.71, 1.61 cpu: 95°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  13:41:31 up  1:09,  0 users,  load average: 1.75, 1.69, 1.61 cpu: 96°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  13:41:41 up  1:09,  0 users,  load average: 1.71, 1.69, 1.61 cpu: 95°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  13:42:35 up  1:10,  0 users,  load average: 1.59, 1.66, 1.61 cpu: 95°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  13:43:33 up  1:11,  0 users,  load average: 1.36, 1.58, 1.58 cpu: 96°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  13:51:14 up  1:18,  0 users,  load average: 1.67, 1.67, 1.63 cpu: 95°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  13:59:21 up  1:27,  0 users,  load average: 1.81, 1.68, 1.65 cpu: 96°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  14:00:56 up  1:28,  0 users,  load average: 2.00, 1.78, 1.68 cpu: 96°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  14:13:04 up  1:40,  0 users,  load average: 1.24, 1.51, 1.60 cpu: 96°C
mpi@master:/master/mpi_tests$ mpirun -host node59 /master/mpi_tests/system
node59  14:19:09 up  1:46,  0 users,  load average: 1.67, 1.64, 1.63 cpu: 96°C
mpi@master:/master/mpi_tests$

It also seems my wiring harness has become intermittent some of the systems are powering down whenever I am wiggling things inside the "cage" so when I get the heat sinks I will have to disassemble the entire thing to install them, that will be a good time to look for intermittent wiring and hopefully clear it up.
For a long time I had decided not to use the cluster any more since my Jetson cluster outperforms these 66 ARM cpu by light-years but I want to do much more than just crack passwords. The Jetson cluster is better suited for that but I am planning on implementing Docker and explore large-scale Kubernetes clusters, this group of 66-systems running Ubuntu 18.04 is the answer.

2019-12-20 I have completed building the latest John version

#define JTR_GIT_VERSION JUMBO_VERSION "-73e006c673 2019-12-20 15:41:04 -0300"

It is working fine and I am working on a new crack now, all 66-nodes are functioning well.

62 0g 0:03:54:20 1.55% 2/3 (ETA: 2019-12-31 04:49) 0g/s 39.24p/s 39.24c/s 39.24C/s 3masolino..3nachbereitung@@
47 0g 0:03:54:25 1.53% 2/3 (ETA: 2019-12-31 06:58) 0g/s 38.92p/s 38.92c/s 38.92C/s Pressera0..Programmeringsmetoden0
41 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:24) 0g/s 39.31p/s 39.31c/s 39.31C/s 3alguifou..3amincirai
14 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:33) 0g/s 39.28p/s 39.28c/s 39.28C/s 3abolivi..3achterhoudendst
43 0g 0:03:54:20 1.55% 2/3 (ETA: 2019-12-31 04:48) 0g/s 39.25p/s 39.25c/s 39.25C/s 3nachrauben..3ptp
8 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:20) 0g/s 39.31p/s 39.31c/s 39.31C/s 3anmelderens..3antrustionship
45 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:30) 0g/s 39.29p/s 39.29c/s 39.29C/s 3adumbratively..3afgestaarde
11 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:24) 0g/s 39.31p/s 39.31c/s 39.31C/s 3amissibility..3ankyloser
19 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:30) 0g/s 39.29p/s 39.29c/s 39.29C/s 3aduneremo..3afgestoften
23 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:32) 0g/s 39.29p/s 39.29c/s 39.29C/s 3achterlijkst..3adulterously
42 0g 0:03:54:22 1.53% 2/3 (ETA: 2019-12-31 06:45) 0g/s 38.95p/s 38.95c/s 38.95C/s Ratuur0..Reconstruisions0
40 0g 0:03:54:24 1.53% 2/3 (ETA: 2019-12-31 07:10) 0g/s 38.89p/s 38.89c/s 38.89C/s Pediculine0..Pernicieus0
4 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:37) 0g/s 39.27p/s 39.27c/s 39.27C/s 3wheelersburg..3aangestipt
24 0g 0:03:54:20 1.53% 2/3 (ETA: 2019-12-31 06:41) 0g/s 38.96p/s 38.96c/s 38.96C/s Recooks0..Reglazed0
63 0g 0:03:54:25 1.55% 2/3 (ETA: 2019-12-31 11:51) 0g/s 39.24p/s 39.24c/s 39.24C/s 3padgett..3prinzipiell
27 0g 0:03:54:22 1.53% 2/3 (ETA: 2019-12-31 06:47) 0g/s 38.94p/s 38.94c/s 38.94C/s Radiobroadcast0..Rattleskull0
49 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:35) 0g/s 39.28p/s 39.28c/s 39.28C/s 3aangestrompelde..3aboieront
18 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:33) 0g/s 39.28p/s 39.28c/s 39.28C/s 3abolla..3achterhouten
35 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:44) 0g/s 39.26p/s 39.26c/s 39.26C/s 3rufino..3sherramoor
37 0g 0:03:54:24 1.55% 2/3 (ETA: 2019-12-31 04:37) 0g/s 39.28p/s 39.28c/s 39.28C/s 3abolissaient..3achtergrachten
54 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:42) 0g/s 39.26p/s 39.26c/s 39.26C/s 3tdo..3tyburn
60 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 11:29) 0g/s 39.29p/s 39.29c/s 39.29C/s 3afgestraft..3afvries
59 0g 0:03:54:27 1.55% 2/3 (ETA: 2019-12-31 11:52) 0g/s 39.24p/s 39.24c/s 39.24C/s 3problemspezifisch..3rudman
7 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:30) 0g/s 39.29p/s 39.29c/s 39.29C/s 3adunavo..3afgestempelden
12 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:35) 0g/s 39.28p/s 39.28c/s 39.28C/s 3aangetekende..3abolirent
31 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:42) 0g/s 39.26p/s 39.26c/s 39.26C/s 3shilluk..3systemerweiterung
46 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:44) 0g/s 39.26p/s 39.26c/s 39.26C/s 3ruhe..3sherwin
58 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:19) 0g/s 39.32p/s 39.32c/s 39.32C/s 3anusvara..3archegonium
64 0g 0:03:54:24 1.55% 2/3 (ETA: 2019-12-31 04:47) 0g/s 39.25p/s 39.25c/s 39.25C/s 3ruleville..3shia
52 0g 0:03:54:21 1.53% 2/3 (ETA: 2019-12-31 06:52) 0g/s 38.93p/s 38.93c/s 38.93C/s Programstumper0..Psammoma0
15 0g 0:03:54:24 1.55% 2/3 (ETA: 2019-12-31 04:33) 0g/s 39.28p/s 39.28c/s 39.28C/s 3aduneranno..3afgestipte
17 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:30) 0g/s 39.29p/s 39.29c/s 39.29C/s 3adunerei..3afgestoft
5 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:55) 0g/s 39.22p/s 39.22c/s 39.22C/s 3idp..3kekchi
51 0g 0:03:54:27 1.55% 2/3 (ETA: 2019-12-31 04:57) 0g/s 39.22p/s 39.22c/s 39.22C/s 3mashpee..3nachbarn
2 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:34) 0g/s 39.28p/s 39.28c/s 39.28C/s 3abolitionized..3achterhoeken
61 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 11:31) 0g/s 39.29p/s 39.29c/s 39.29C/s 3adunasti..3afgestekt
29 0g 0:03:54:20 1.54% 2/3 (ETA: 2019-12-31 06:25) 0g/s 39.00p/s 39.00c/s 39.00C/s Schizzavo0..Scribaciousness0
53 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:41) 0g/s 39.27p/s 39.27c/s 39.27C/s 3tdma..3tybie
6 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:38) 0g/s 39.27p/s 39.27c/s 39.27C/s 3whelan..3aangestipten
28 0g 0:03:54:21 1.53% 2/3 (ETA: 2019-12-31 06:56) 0g/s 38.92p/s 38.92c/s 38.92C/s Preconcertedly0..Presprinkled0
25 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:42) 0g/s 39.26p/s 39.26c/s 39.26C/s 3tid..3tyndale
44 0g 0:03:54:21 1.53% 2/3 (ETA: 2019-12-31 06:52) 0g/s 38.93p/s 38.93c/s 38.93C/s Programsaetningerne0..Psammites0
39 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:39) 0g/s 39.27p/s 39.27c/s 39.27C/s 3typees..3westernport
16 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:32) 0g/s 39.29p/s 39.29c/s 39.29C/s 3achterliggers..3adulterins
13 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:23) 0g/s 39.31p/s 39.31c/s 39.31C/s 3amissing..3ankyloseraient
32 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:24) 0g/s 39.31p/s 39.31c/s 39.31C/s 3algosis..3aminate
9 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:21) 0g/s 39.32p/s 39.32c/s 39.32C/s 3anmelderi..3ants
33 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:45) 0g/s 39.25p/s 39.25c/s 39.25C/s 3ruffsdale..3sherpa
55 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:35) 0g/s 39.28p/s 39.28c/s 39.28C/s 3aangestuurde..3abolendo
65 0g 0:03:54:25 1.55% 2/3 (ETA: 2019-12-31 11:34) 0g/s 39.28p/s 39.28c/s 39.28C/s 3adunation..3afgestelde
1 0g 0:03:54:20 1.55% 2/3 (ETA: 2019-12-31 04:39) 0g/s 39.27p/s 39.27c/s 39.27C/s 3telnet..3tyigh
20 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:38) 0g/s 39.27p/s 39.27c/s 39.27C/s 3whiggism..3aangestormde
38 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:46) 0g/s 39.25p/s 39.25c/s 39.25C/s 3privilegien..3rucervus
30 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:40) 0g/s 39.27p/s 39.27c/s 39.27C/s 3tyndareus..3weste
56 0g 0:03:54:21 1.53% 2/3 (ETA: 2019-12-31 07:01) 0g/s 38.91p/s 38.91c/s 38.91C/s Plaatsingsbureaus0..Podaxonial0
36 0g 0:03:54:20 1.54% 2/3 (ETA: 2019-12-31 05:00) 0g/s 39.21p/s 39.21c/s 39.21C/s 3deirdra..3elsah
48 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:43) 0g/s 39.26p/s 39.26c/s 39.26C/s 3shinnston..3systemstecker
57 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:34) 0g/s 39.28p/s 39.28c/s 39.28C/s 3abolitie..3achterhandsbeentjes
26 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:47) 0g/s 39.25p/s 39.25c/s 39.25C/s 3pristis..3rubicon
21 0g 0:03:54:21 1.55% 2/3 (ETA: 2019-12-31 04:26) 0g/s 39.30p/s 39.30c/s 39.30C/s 3afwegen..3algoritmens
22 0g 0:03:54:21 1.53% 2/3 (ETA: 2019-12-31 07:05) 0g/s 38.90p/s 38.90c/s 38.90C/s Peroxyborate0..Phylae0
34 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:43) 0g/s 39.26p/s 39.26c/s 39.26C/s 3shimberg..3systemframes
3 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:14) 0g/s 39.33p/s 39.33c/s 39.33C/s 3auricularis..3backet
50 0g 0:03:54:22 1.55% 2/3 (ETA: 2019-12-31 04:38) 0g/s 39.27p/s 39.27c/s 39.27C/s 3whitefieldian..3aangestrompelden
10 0g 0:03:54:23 1.53% 2/3 (ETA: 2019-12-31 06:46) 0g/s 38.95p/s 38.95c/s 38.95C/s Raunchier0..Recontracts0

2019-12-20 I have completed building the MPICH cluster, all 65-nodes are functioning and MPI-enabled Python is working.

Script started on 2019-12-20 16:59:36+0000
mpi@master:/master/mpi_tests$ ./stress
+ mpirun -np 65 -f /master/mpi_tests/machinefile /master/mpi_tests/system
node01 09:59:48 up 15:10, 0 users, load average: 0.50, 0.55, 0.47
node03 09:59:49 up 15:10, 0 users, load average: 0.36, 0.57, 0.52
node04 09:59:49 up 15:10, 0 users, load average: 0.35, 0.48, 0.44
node02 09:59:49 up 15:10, 0 users, load average: 0.37, 0.48, 0.44
node07 09:59:49 up 15:09, 0 users, load average: 0.63, 0.63, 0.52
node08 09:59:50 up 15:09, 0 users, load average: 0.20, 0.41, 0.43
node10 09:59:50 up 15:09, 0 users, load average: 0.68, 0.59, 0.49
node09 09:59:50 up 15:09, 0 users, load average: 0.44, 0.45, 0.44
node05 09:59:50 up 15:10, 0 users, load average: 0.50, 0.56, 0.48
node14 09:59:50 up 15:11, 0 users, load average: 0.90, 0.57, 0.46
node11 09:59:50 up 15:09, 0 users, load average: 0.66, 0.52, 0.43
node13 09:59:51 up 15:11, 0 users, load average: 0.66, 0.49, 0.46
node16 09:59:51 up 15:11, 0 users, load average: 0.34, 0.53, 0.48
node06 09:59:51 up 15:10, 0 users, load average: 0.75, 0.56, 0.47
node12 09:59:51 up 15:12, 0 users, load average: 0.73, 0.56, 0.47
node19 09:59:51 up 15:11, 0 users, load average: 0.42, 0.45, 0.44
node18 09:59:51 up 15:11, 0 users, load average: 0.50, 0.60, 0.47
node20 09:59:51 up 15:11, 0 users, load average: 0.43, 0.54, 0.45
node15 09:59:51 up 15:11, 0 users, load average: 0.62, 0.54, 0.47
node21 09:59:51 up 15:11, 0 users, load average: 0.83, 0.61, 0.47
node25 09:59:51 up 15:12, 0 users, load average: 0.60, 0.53, 0.48
node17 09:59:51 up 15:11, 0 users, load average: 0.42, 0.61, 0.53
node27 09:59:52 up 15:12, 0 users, load average: 0.72, 0.62, 0.49
node24 09:59:52 up 15:12, 0 users, load average: 0.62, 0.58, 0.51
node22 09:59:52 up 15:10, 0 users, load average: 0.49, 0.56, 0.46
node30 09:59:52 up 15:12, 0 users, load average: 0.46, 0.52, 0.47
node35 09:59:52 up 11:15, 0 users, load average: 0.65, 0.60, 0.47
node34 09:59:52 up 11:16, 0 users, load average: 0.49, 0.52, 0.45
node29 09:59:52 up 15:12, 0 users, load average: 0.67, 0.58, 0.49
node26 09:59:52 up 15:12, 0 users, load average: 0.69, 0.63, 0.47
node28 09:59:52 up 23 min, 0 users, load average: 0.43, 0.52, 0.45
node23 09:59:52 up 15:12, 0 users, load average: 0.34, 0.43, 0.38
node31 09:59:52 up 15:12, 0 users, load average: 0.69, 0.63, 0.54
node37 09:59:52 up 11:15, 0 users, load average: 0.42, 0.52, 0.45
node32 09:59:52 up 15:12, 0 users, load average: 0.23, 0.38, 0.40
node42 09:59:52 up 22 min, 0 users, load average: 0.70, 0.54, 0.43
node45 09:59:52 up 11:16, 0 users, load average: 0.62, 0.64, 0.51
node38 09:59:52 up 11:03, 0 users, load average: 0.60, 0.60, 0.45
node47 09:59:52 up 11:16, 0 users, load average: 0.28, 0.50, 0.48
node36 09:59:52 up 11:15, 0 users, load average: 0.61, 0.64, 0.54
node43 09:59:52 up 21 min, 0 users, load average: 0.50, 0.49, 0.41
node41 09:59:52 up 11:15, 0 users, load average: 0.52, 0.57, 0.45
node33 09:59:52 up 19:42, 0 users, load average: 1.07, 0.65, 0.51
node63 16:59:52 up 11:16, 1 user, load average: 0.18, 0.37, 0.39
node48 09:59:52 up 11:16, 0 users, load average: 0.26, 0.54, 0.50
node39 09:59:52 up 11:03, 0 users, load average: 1.02, 0.67, 0.50
node50 09:59:52 up 11:16, 0 users, load average: 0.41, 0.54, 0.46
node64 09:59:52 up 11:17, 1 user, load average: 0.21, 0.39, 0.42
node46 09:59:53 up 11:16, 0 users, load average: 0.76, 0.53, 0.43
node60 16:59:53 up 11:16, 0 users, load average: 0.62, 0.66, 0.53
node56 09:59:53 up 11:16, 0 users, load average: 0.53, 0.54, 0.47
node51 09:59:53 up 11:16, 0 users, load average: 0.39, 0.55, 0.49
node62 09:59:53 up 11:16, 0 users, load average: 0.47, 0.42, 0.42
node61 16:59:53 up 11:16, 0 users, load average: 0.50, 0.63, 0.54
node40 09:59:53 up 11:15, 0 users, load average: 0.47, 0.49, 0.43
node53 09:59:53 up 11:16, 0 users, load average: 0.38, 0.44, 0.38
node44 09:59:53 up 11:15, 0 users, load average: 0.33, 0.60, 0.53
node52 09:59:53 up 11:16, 0 users, load average: 1.17, 0.73, 0.53
node58 09:59:53 up 11:16, 0 users, load average: 0.80, 0.60, 0.44
node49 09:59:53 up 11:16, 0 users, load average: 0.12, 0.51, 0.52
node54 09:59:53 up 11:16, 0 users, load average: 0.87, 0.59, 0.44
node65 16:59:53 up 11:17, 1 user, load average: 0.32, 0.53, 0.52
node57 09:59:53 up 11:16, 0 users, load average: 0.32, 0.37, 0.36
node59 16:59:53 up 11:16, 0 users, load average: 0.33, 0.49, 0.46
node55 09:59:53 up 11:16, 0 users, load average: 0.50, 0.65, 0.54
+ mpirun -np 65 -f /master/mpi_tests/machinefile /master/mpi_tests/sample4
Process 1 on host node01 has the partial result of 0.024157
Process 4 on host node04 has the partial result of 0.024072
Process 5 on host node05 has the partial result of 0.024016
Process 6 on host node06 has the partial result of 0.023945
Process 11 on host node11 has the partial result of 0.023385
Process 2 on host node02 has the partial result of 0.024142
Process 12 on host node12 has the partial result of 0.023231
Process 3 on host node03 has the partial result of 0.024114
Process 14 on host node14 has the partial result of 0.022884
Process 8 on host node08 has the partial result of 0.023763
Process 10 on host node10 has the partial result of 0.023524
Process 9 on host node09 has the partial result of 0.023650
Process 13 on host node13 has the partial result of 0.023064
Process 15 on host node15 has the partial result of 0.022690
Process 16 on host node16 has the partial result of 0.022483
Process 18 on host node18 has the partial result of 0.022030
Process 19 on host node19 has the partial result of 0.021784
Process 21 on host node21 has the partial result of 0.021254
Process 20 on host node20 has the partial result of 0.021525
Process 22 on host node22 has the partial result of 0.020970
Process 23 on host node23 has the partial result of 0.020674
Process 24 on host node24 has the partial result of 0.020366
Process 17 on host node17 has the partial result of 0.022263
Process 26 on host node26 has the partial result of 0.019715
Process 28 on host node28 has the partial result of 0.019017
Process 29 on host node29 has the partial result of 0.018651
Process 27 on host node27 has the partial result of 0.019371
Process 25 on host node25 has the partial result of 0.020046
Process 33 on host node33 has the partial result of 0.017083
Process 35 on host node35 has the partial result of 0.016237
Process 36 on host node36 has the partial result of 0.015800
Process 31 on host node31 has the partial result of 0.017888
Process 38 on host node38 has the partial result of 0.014899
Process 34 on host node34 has the partial result of 0.016665
Process 37 on host node37 has the partial result of 0.015354
Process 30 on host node30 has the partial result of 0.018275
Process 40 on host node40 has the partial result of 0.013963
Process 39 on host node39 has the partial result of 0.014435
Process 32 on host node32 has the partial result of 0.017490
Process 43 on host node43 has the partial result of 0.012498
Process 44 on host node44 has the partial result of 0.011995
Process 48 on host node48 has the partial result of 0.009915
Process 63 on host node63 has the partial result of 0.001459
Process 54 on host node54 has the partial result of 0.006628
Process 53 on host node53 has the partial result of 0.007187
Process 42 on host node42 has the partial result of 0.012994
Process 57 on host node57 has the partial result of 0.004928
Process 59 on host node59 has the partial result of 0.003779
Process 50 on host node50 has the partial result of 0.008839
Process 61 on host node61 has the partial result of 0.002622
Process 55 on host node55 has the partial result of 0.006065
Process 60 on host node60 has the partial result of 0.003202
Process 45 on host node45 has the partial result of 0.011485
Process 51 on host node51 has the partial result of 0.008293
Process 65 on host node65 has the partial result of 0.000292
Process 47 on host node47 has the partial result of 0.010444
Process 41 on host node41 has the partial result of 0.013482
Process 58 on host node58 has the partial result of 0.004355
Process 56 on host node56 has the partial result of 0.005498
The result =0.999704
Process 7 on host node07 has the partial result of 0.023861
Process 46 on host node46 has the partial result of 0.010968
Process 49 on host node49 has the partial result of 0.009380
Process 52 on host node52 has the partial result of 0.007742
Process 62 on host node62 has the partial result of 0.002041
Process 64 on host node64 has the partial result of 0.000876
+ mpirun -np 65 -f /master/mpi_tests/machinefile /master/mpi_tests/cpi
Process 1 of 65 is on node01
Process 2 of 65 is on node02
Process 3 of 65 is on node03
Process 4 of 65 is on node04
Process 6 of 65 is on node06
Process 9 of 65 is on node09
Process 15 of 65 is on node15
Process 13 of 65 is on node13
Process 11 of 65 is on node11
Process 17 of 65 is on node17
Process 19 of 65 is on node19
Process 22 of 65 is on node22
Process 14 of 65 is on node14
Process 8 of 65 is on node08
Process 7 of 65 is on node07
Process 12 of 65 is on node12
Process 10 of 65 is on node10
Process 20 of 65 is on node20
Process 16 of 65 is on node16
Process 27 of 65 is on node27
Process 29 of 65 is on node29
Process 28 of 65 is on node28
Process 30 of 65 is on node30
Process 31 of 65 is on node31
Process 32 of 65 is on node32
Process 24 of 65 is on node24
Process 34 of 65 is on node34
Process 37 of 65 is on node37
Process 36 of 65 is on node36
Process 41 of 65 is on node41
Process 39 of 65 is on node39
Process 40 of 65 is on node40
Process 43 of 65 is on node43
Process 42 of 65 is on node42
Process 44 of 65 is on node44
Process 38 of 65 is on node38
Process 45 of 65 is on node45
Process 51 of 65 is on node51
Process 52 of 65 is on node52
Process 64 of 65 is on node64
Process 65 of 65 is on node65
Process 58 of 65 is on node58
Process 49 of 65 is on node49
Process 54 of 65 is on node54
Process 53 of 65 is on node53
Process 59 of 65 is on node59
Process 60 of 65 is on node60
Process 50 of 65 is on node50
Process 62 of 65 is on node62
Process 63 of 65 is on node63
Process 47 of 65 is on node47
Process 56 of 65 is on node56
Process 55 of 65 is on node55
Process 57 of 65 is on node57
Process 61 of 65 is on node61
Process 21 of 65 is on node21
Process 23 of 65 is on node23
Process 26 of 65 is on node26
Process 25 of 65 is on node25
Process 33 of 65 is on node33
Process 35 of 65 is on node35
Process 5 of 65 is on node05
Process 46 of 65 is on node46
Process 18 of 65 is on node18
Process 48 of 65 is on node48
pi is approximately 3.1415926544231270, Error is 0.0000000008333338
wall clock time = 0.457541
+ mpirun -np 65 -f /master/mpi_tests/machinefile python /master/mpi_tests/helloworld.py
Hello, Cluster! Python process 1 of 65 on node01.
Hello, Cluster! Python process 3 of 65 on node03.
Hello, Cluster! Python process 4 of 65 on node04.
Hello, Cluster! Python process 18 of 65 on node18.
Hello, Cluster! Python process 7 of 65 on node07.
Hello, Cluster! Python process 14 of 65 on node14.
Hello, Cluster! Python process 12 of 65 on node12.
Hello, Cluster! Python process 17 of 65 on node17.
Hello, Cluster! Python process 24 of 65 on node24.
Hello, Cluster! Python process 36 of 65 on node36.
Hello, Cluster! Python process 34 of 65 on node34.
Hello, Cluster! Python process 6 of 65 on node06.
Hello, Cluster! Python process 10 of 65 on node10.
Hello, Cluster! Python process 9 of 65 on node09.
Hello, Cluster! Python process 13 of 65 on node13.
Hello, Cluster! Python process 15 of 65 on node15.
Hello, Cluster! Python process 32 of 65 on node32.
Hello, Cluster! Python process 31 of 65 on node31.
Hello, Cluster! Python process 35 of 65 on node35.
Hello, Cluster! Python process 40 of 65 on node40.
Hello, Cluster! Python process 38 of 65 on node38.
Hello, Cluster! Python process 44 of 65 on node44.
Hello, Cluster! Python process 37 of 65 on node37.
Hello, Cluster! Python process 64 of 65 on node64.
Hello, Cluster! Python process 49 of 65 on node49.
Hello, Cluster! Python process 56 of 65 on node56.
Hello, Cluster! Python process 51 of 65 on node51.
Hello, Cluster! Python process 63 of 65 on node63.
Hello, Cluster! Python process 52 of 65 on node52.
Hello, Cluster! Python process 62 of 65 on node62.
Hello, Cluster! Python process 60 of 65 on node60.
Hello, Cluster! Python process 54 of 65 on node54.
Hello, Cluster! Python process 59 of 65 on node59.
Hello, Cluster! Python process 2 of 65 on node02.
Hello, Cluster! Python process 8 of 65 on node08.
Hello, Cluster! Python process 16 of 65 on node16.
Hello, Cluster! Python process 22 of 65 on node22.
Hello, Cluster! Python process 26 of 65 on node26.
Hello, Cluster! Python process 28 of 65 on node28.
Hello, Cluster! Python process 27 of 65 on node27.
Hello, Cluster! Python process 23 of 65 on node23.
Hello, Cluster! Python process 41 of 65 on node41.
Hello, Cluster! Python process 61 of 65 on node61.
Hello, Cluster! Python process 42 of 65 on node42.
Hello, Cluster! Python process 55 of 65 on node55.
Hello, Cluster! Python process 5 of 65 on node05.
Hello, Cluster! Python process 47 of 65 on node47.
Hello, Cluster! Python process 53 of 65 on node53.
Hello, Cluster! Python process 39 of 65 on node39.
Hello, Cluster! Python process 21 of 65 on node21.
Hello, Cluster! Python process 65 of 65 on node65.
Hello, Cluster! Python process 48 of 65 on node48.
Hello, Cluster! Python process 20 of 65 on node20.
Hello, Cluster! Python process 33 of 65 on node33.
Hello, Cluster! Python process 29 of 65 on node29.
Hello, Cluster! Python process 46 of 65 on node46.
Hello, Cluster! Python process 57 of 65 on node57.
Hello, Cluster! Python process 43 of 65 on node43.
Hello, Cluster! Python process 19 of 65 on node19.
Hello, Cluster! Python process 30 of 65 on node30.
Hello, Cluster! Python process 58 of 65 on node58.
Hello, Cluster! Python process 25 of 65 on node25.
Hello, Cluster! Python process 11 of 65 on node11.
Hello, Cluster! Python process 50 of 65 on node50.
Hello, Cluster! Python process 45 of 65 on node45.
Script done on 2019-12-20 17:00:44+0000

I am building the latest John The Ripper. I used the following configure flags: —enable-mpi —disable-openmp This is because I am using MPICH and NOT OpenMPI.

Configured for building John the Ripper jumbo:

Target CPU ................................. armv7l NEON, 32-bit LE
AES-NI support ............................. no
Target OS .................................. linux-gnueabihf
Cross compiling ............................ no
Legacy arch header ......................... arm32le.h

Optional libraries/features found:
Memory map (share/page large files) ........ yes
Fork support ............................... yes
OpenMP support ............................. no
OpenCL support ............................. no
Generic crypt(3) format .................... yes
libgmp (PRINCE mode and faster SRP formats)  no
128-bit integer (faster PRINCE mode) ....... no
libz (pkzip and some other formats) ........ no
libbz2 (gpg2john extra decompression logic)  no
libpcap (vncpcap2john and SIPdump) ......... no
OpenMPI support (default disabled) ......... yes
ZTEX USB-FPGA module 1.15y support ......... no

Install missing libraries to get any needed features that were omitted.

Configure finished.  Now "make -s clean && make -sj4" to compile.
mpi@master:/master/tmp/john/JohnTheRipper-bleeding-jumbo/src$ make -s clean && make -sj4

2019-12-19 I have upgraded all 66 nodes to Ubuntu 18.04. Now I will start re-integrating the cluster and bring up MPI. That was tedious work and since the Cubie Board spoofs the mac it could not be automated in any manner. It looks like installing python-mpi4py forces installation of OpenMPI overriding my MPICH2 versions. I have to revisit OpenMPI since I'd not like to have a mixture of versions on every system.

You have to modify /etc/sudoers to not ask the MPI user for passwords so that you can do things like:

This is easily accomplish by adding the following line to /etc/sudoers:

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) NOPASSWD: ALL
ssh node18 "sudo apt-get -y install python-mpi4py python-mpi4py-doc"

I am having a terrible time with OpenMPI it just doesn't want to work reliably so I am falling back to MPICH2, manually installing mpi4py (Python) which leads to many other issues.

With the manual approach (using pip) you must install setuptools, wheel and mpi4py (after installing python, python-dev and python-pip.)

pip install setuptools
pip install wheel
pip install mpi4py
root@node02:~# pip install setuptools
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/54/28/c45d8b54c1339f9644b87663945e54a8503cfef59cf0f65b3ff5dd17cf64/setuptools-42.0.2-py2.py3-none-any.whl (583kB)
    100% |████████████████████████████████| 583kB 168kB/s 
Installing collected packages: setuptools
Successfully installed setuptools-42.0.2
root@node02:~# pip install wheel
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.33.6

root@node01:~# pip install mpi4py
Collecting mpi4py
  Using cached https://files.pythonhosted.org/packages/ec/8f/bbd8de5ba566dd77e408d8136e2bab7fdf2b97ce06cab830ba8b50a2f588/mpi4py-3.0.3.tar.gz
Building wheels for collected packages: mpi4py
  Running setup.py bdist_wheel for mpi4py ... done
  Stored in directory: /root/.cache/pip/wheels/18/e0/86/2b713dd512199096012ceca61429e12b960888de59818871d6
Successfully built mpi4py
Installing collected packages: mpi4py
Successfully installed mpi4py-3.0.3
root@node01:~#

Once I have determined a solution to the MANY problems, I usually create a script to perform the work on all 65-nodes.

Here is how I have been testing mpi4py post-installation (only 4-nodes done as of this moment…)

mpi@master:/master/mpi_tests$ mpirun -l -n 4 -f /master/mpi_tests/machinefile \
python /master/mpi_tests/helloworld.py
[1] Hello, Cluster! Python process 2 of 4 on node02.
[3] Hello, Cluster! Python process 4 of 4 on node04.
[0] Hello, Cluster! Python process 1 of 4 on node01.
[2] Hello, Cluster! Python process 3 of 4 on node03.
mpi@master:/master/mpi_tests$

The machinefile has all 65-nodes but I limit to the 4 systems that are done so far. As I am compiling on an ARM A10 cpu running only at 1ghz it is very time consuming as you might imagine. It should finish sometime tomorrow.

You have to remove the hostname entries (i.e. node01, node02) from any localhost entries in /etc/hosts, i.e.

127.0.0.1   localhost node07
::1         localhost node07 ip6-localhost ip6-loopback

In the above example, remove the real hostname from the localhost lines, i.e. remove "node07" as it will cause MPICH-2 to fail with a communication error. The act is safe if your /etc/hosts has an entry for "node07" in this example which it does…

It should look like the below example

127.0.0.1   localhost 
::1         localhost ip6-localhost ip6-loopback

Here is an example of the error you will see if you do not heed the above instruction.

[[size x-small]]

mpi@master:/master/mpi_tests$ mpirun -np 8 -f /master/mpi_tests/machinefile-8 /master/mpi_tests/cpi
Process 3 of 8 is on node03
Process 2 of 8 is on node02
Process 1 of 8 is on node01
Process 4 of 8 is on node04
Process 8 of 8 is on node08
Process 6 of 8 is on node06
Process 7 of 8 is on node07
Process 5 of 8 is on node05
Fatal error in PMPI_Reduce: Unknown error class, error stack:
PMPI_Reduce(1263)...............: MPI_Reduce(sbuf=0xbe82de90, rbuf=0xbe82de98, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed
MPIR_Reduce_impl(1075)..........: 
MPIR_Reduce_intra(890)..........: 
MPIR_Reduce_binomial(186).......: 
MPIDI_CH3U_Recvq_FDU_or_AEP(629): Communication error with rank 7
Fatal error in PMPI_Reduce: Unknown error class, error stack:
PMPI_Reduce(1263)........: MPI_Reduce(sbuf=0xbef58e90, rbuf=0xbef58e98, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed
MPIR_Reduce_impl(1075)...: 
MPIR_Reduce_intra(890)...: 
MPIR_Reduce_binomial(248): Failure during collective
Fatal error in PMPI_Reduce: Unknown error class, error stack:
PMPI_Reduce(1263)........: MPI_Reduce(sbuf=0xbe8b9e90, rbuf=0xbe8b9e98, count=1, MPI_DOUBLE, MPI_SUM, root=0, MPI_COMM_WORLD) failed
MPIR_Reduce_impl(1075)...: 
MPIR_Reduce_intra(890)...: 
MPIR_Reduce_binomial(248): Failure during collective
mpi@master:/master/mpi_tests$

In this case, node08 (rank 7) was not fixed.

2019-12-18 I have about 45 of 66 systems reloaded with Ubuntu 18.04

2019-12-16 I am upgrading the entire cluster to Ubuntu 18.03 (Bionic) it had been running Debian Wheezy. I am very excited about this as I also hope to implement some Kubernetes clusters on the much more modern version of Linux. I hope to keep using MPICH2 but may have to change to OpenMPI.

In order to get MPI working, I installed the following packages:

libhwloc-plugins:armhf 1.11.9-1
libmpich-dev:armhf 3.3~a2-4
mpich:armhf 3.3~a2-4
mpich-doc:armhf 3.3~a2-4

I also installed the kernel NFS server and NFS client packages.

root@cubieboard:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
root@cubieboard:/#

Between how Ubuntu 18.04 devices are handled, NetworkManager, and the CubieBoard "spoofing" the fake mac address bringing up each node has become excruciatingly stupid. I can't stand how I have to initially let the board come up on dhcp and then manually reconfigure same for static IP using armbian-config. I tried creating a standard image but NetworkManager uses the mac address in 'Wired Connection 1' and the unit fails to come up.

2019-12-15 I have printed another case design for the standalone CubieBoard1 (and CubieBoard2) This is a very nice case but there are no openings to attach an external hard disk, or the allow use of the serial console cable.

Here is a case I made early in 2019, I never posted it but I like it a lot for standalone use as the designer made matching case for hard drive and also a power supply (I did not implement.) This design also came from Thingiverse. As you can see it's easy to route any cables from the PCB to outside of the case.