Monolith 2016 02

Updated: 2016-02-04

Performance Note: When using JTR in a distributed MPI cluster with a central NFS server providing all configuration and code for JTR and are using a large dictionary file you may wish to avoid a huge network "surge" when all your compute nodes are pulling the dictionary file. Use something like 'salt' or 'cfengine' or even 'chef' to keep that distributed out to the compute nodes instead.

I ran into a performance problem (only on initial startup of the JTR run) it was hammering the network badly since I have a very huge dictionary file. This was only a problem though, because I chose esthetics over performance by using wifi (Netgear with DD-WRT) instead of a hardwired switch.

As you can imagine having 65 wifi transceivers (the master node (cfengine master, nfs server, MPI control node) is hardwired 100M) sitting within inches of each other doesn't work that well and the congestion is already immense with that many hosts on a consumer-grade access point.

I would need 66 switch ports however and I didn't feel like having all that wiring spaghetti for 66 computers. Perhaps some day I will extend the monolith vertically to accommodate all that but not right now.

CFEngine example for how to distribute updated dictionaries.

copy:
  node_server::
  /master/cfengine/root/mangled.list dest=/var/list/mangled.list
       owner=root group=root mode=0644
       server=$(cfsrvhost) type=checksum

Updated: 2016-02-06

I have been working with some Cubieboard 2 boards recently.

I installed Linaro to the nand (fel-mode using LIveSuit) but needed to resize nand due to the partitioning not making use of 2gb of the 4GB NAND (wasted space!)

root@cubieboard:~# df                                                           
Filesystem     1K-blocks   Used Available Use% Mounted on                       
/dev/nandd       2066000 960760   1013792  49% /                                
none                   4      0         4   0% /sys/fs/cgroup                   
none               92784    164     92620   1% /run                             
none                5120      0      5120   0% /run/lock                        
none              463900      0    463900   0% /run/shm                         
none              102400      0    102400   0% /run/user                        
overflow            1024      0      1024   0% /tmp                             
root@cubieboard:~#

This is what I did:

I booted an image of Linaro 13.06 from the sd card. Then I examined the existing configuration of /dev/nand. I bought (3) boards from EBAY that had been mucked with, Android was missing and the partitioning had been mangled.

1, cat /proc/partitions

root@cubieboard2:~# cat /proc/partitions
major minor  #blocks  name                                                                                                                         

  93        0    3989504 nand                                                                                                                      
  93        1      32768 nanda                                                                                                                     
  93        2      16384 nandb                                                                                                                     
  93        3      32768 nandc                                                                                                                     
  93        4    2097152 nandd   **rootfs**                                                                                                                  
  93        5    1794048 nande   **wasted space**                                                                                                                  
   8        0  117964900 sda                                                                                                                       
   8        1  117963876 sda1                                                                                                                      
 179        0    3872256 mmcblk0                                                                                                                   
 179        1      12288 mmcblk0p1                                                                                                                 
 179        2    3858944 mmcblk0p2                                                                                                                 
root@cubieboard2:~#
root@cubieboard2:~# nand-part|tail
BAD!                                                                                                                                               
check partition table copy 3: mbr: version 0x00000200, magic softw411                                                                              
BAD!                                                                                                                                               
mbr: version 0x00000200, magic softw411                                                                                                            
5 partitions                                                                                                                                       
partition  1: class =         DISK, name =   bootloader, partition start =    32768, partition size =    65536 user_type=0                         
partition  2: class =         DISK, name =          env, partition start =    98304, partition size =    32768 user_type=0                         
partition  3: class =         DISK, name =         boot, partition start =   131072, partition size =    65536 user_type=0                         
partition  4: class =         DISK, name =       rootfs, partition start =   196608, partition size =  4194304 user_type=0                         
partition  5: class =         DISK, name =        UDISK, partition start =  4390912, partition size =  3342336 user_type=0                         
root@cubieboard2:~#

Once you know the exact layout of the nand partitions it is straightforward. You locate the unused space on the nand, in this
case it was partition 5 or /dev/nande.

Since we want to combine nandd and nande we simply add 4194304 (size of nandd) to 3342336 (size of nande) or 7536640
total to derive our value for the next step.

nand-part -f a20 /dev/nand 32768 "bootloader 65536" "env 32768" "boot 65536" "rootfs 7536640"

Once this completes without error, you need to run a filesystem check and then resize the newly combined partition.

root@cubieboard2:~# fsck -f /dev/nandd                                                                                                             
fsck from util-linux 2.20.1                                                                                                                        
e2fsck 1.42.9 (4-Feb-2014)                                                                                                                         
Pass 1: Checking inodes, blocks, and sizes                                                                                                         
Pass 2: Checking directory structure                                                                                                               
Pass 3: Checking directory connectivity                                                                                                            
Pass 4: Checking reference counts                                                                                                                  
Pass 5: Checking group summary information                                                                                                         
/dev/nandd: 28129/123904 files (0.1% non-contiguous), 247980/524288 blocks                                                                         
root@cubieboard2:~# resize2fs /dev/nandd                                                                                                           
resize2fs 1.42.9 (4-Feb-2014)                                                                                                                      
Resizing the filesystem on /dev/nandd to 972800 (4k) blocks.                                                                                       
The filesystem on /dev/nandd is now 972800 blocks long.                                                                                            

root@cubieboard2:~#

Here you will see the results afterwards.

root@cubieboard:~# df                                                                                                                              
Filesystem     1K-blocks   Used Available Use% Mounted on                                                                                          
/dev/nandd       3832816 961324   2701816  27% /                                                                                                   
none                   4      0         4   0% /sys/fs/cgroup                                                                                      
none               92784    160     92624   1% /run                                                                                                
none                5120      0      5120   0% /run/lock                                                                                           
none              463900      0    463900   0% /run/shm                                                                                            
none              102400      0    102400   0% /run/user                                                                                           
root@cubieboard:~#
root@cubieboard:~# cat /proc/partitions                                                                                                            
major minor  #blocks  name                                                                                                                         

  93        0    3989504 nand                                                                                                                      
  93        1      32768 nanda                                                                                                                     
  93        2      16384 nandb                                                                                                                     
  93        3      32768 nandc                                                                                                                     
  93        4    3891200 nandd                                                                                                                     
root@cubieboard:~#

Unfortunately since this release is so old it is no longer supported on ports.ubuntu.com so
I have to make some changes to apt in order to still be able to get packages. Change /etc/apt/sources.llist from:
deb http://ports.ubuntu.com/ubuntu-ports/ raring main universe                                                                                        
deb-src http://ports.ubuntu.com/ubuntu-ports/ raring main universe

to

deb http://old-releases.ubuntu.com/ubuntu/ raring main universe restricted multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ raring main universe restricted multiverse

deb http://old-releases.ubuntu.com/ubuntu/ raring-security main universe restricted multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ raring-security main universe restricted multiverse

deb http://old-releases.ubuntu.com/ubuntu/ raring-updates main universe restricted multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ raring-updates main universe restricted multiverse

deb http://old-releases.ubuntu.com/ubuntu/ raring-backports main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ raring-backports main restricted universe multiverse

deb http://old-releases.ubuntu.com/ubuntu/ raring-proposed main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ raring-proposed main restricted universe multiverse