FreeNAS

2014-01-17:
Today I got bonding working with FreeNAS 9.1.1

I used LACP and bonded (2) 1-gig interfaces. The switch is using 802.3ad

It is working on em0 and em1 and it was fairly easy on the freenas side. I will produce some benchmarks but wanted to mention that it definitely works.

2014-01-04: I had an error earlier when I initiated a scrub. I had rebuilt everything from scratch earlier this week and had been mirroring the source nas again. When this error occurred the entire system locked up: "twa0: Request 232 timed out!" There had been heavy write activity just before I issued the scrub but I did stop all the transfers before issuing the scrub. Uptime had been less than 15 hours.

When I powered it down and up again the scrub began normally at boot. There is about 2TB on the NAS now so it will take some time to complete.

[root@freenas ~]# zpool status -v
  pool: NAS
 state: ONLINE
  scan: scrub in progress since Sat Jan  4 22:24:38 2014
        372G scanned out of 2.49T at 285M/s, 2h10m to go
        8K repaired, 14.58% done
config:

        NAME                                            STATE     READ WRITE CKSUM
        NAS                                             ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/b53cc9db-7431-11e3-bc6b-0030487704a4  ONLINE       0     0     0
            gptid/b63d4556-7431-11e3-bc6b-0030487704a4  ONLINE       0     0     0
            gptid/b77b4f3b-7431-11e3-bc6b-0030487704a4  ONLINE       0     0     0
            gptid/b893f7f2-7431-11e3-bc6b-0030487704a4  ONLINE       0     0     0
            gptid/b9692f42-7431-11e3-bc6b-0030487704a4  ONLINE       0     0     0
            gptid/ba1e5a4a-7431-11e3-bc6b-0030487704a4  ONLINE       0     0     0
            gptid/bb042ab4-7431-11e3-bc6b-0030487704a4  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/bb4aed4e-7433-11e3-bc6b-0030487704a4  ONLINE       0     0     0
            gptid/bc926c5d-7433-11e3-bc6b-0030487704a4  ONLINE       0     0     0
            gptid/bd959ac2-7433-11e3-bc6b-0030487704a4  ONLINE       0     0     0
            gptid/be6ee5d7-7433-11e3-bc6b-0030487704a4  ONLINE       0     0     0
        spares
          gptid/db811f2f-7433-11e3-bc6b-0030487704a4    AVAIL
          gptid/dc7721df-7433-11e3-bc6b-0030487704a4    AVAIL

errors: No known data errors
[root@freenas ~]#

2013-12-26: I detached the failing disk from the pool after the replace and resilver completed and then removed it physically.

As you can see from the below I've got another disk with checksum errors as well. I hope to eventually get all 15 disks stable.
I might go bankrupt trying to get all this working but I'm assuming that everything will be fine as long as my disk controllers hold out. I am a bit suspicious of them since all of these drives are new or were working perfectly on a different configuration using JBOD. With the disk dc9d18bf-6d94-11e3-a454-0030487704a4 we can see it progressively getting worse if you look at the below series of zpool status output.

[root@freenas ~]# zpool status -v

pool: NAS                                                                     
 state: ONLINE                                                                  
status: One or more devices has experienced an unrecoverable error.  An         
        attempt was made to correct the error.  Applications are unaffected.    
action: Determine if the device needs to be replaced, and clear the errors      
        using 'zpool clear' or replace the device with 'zpool replace'.         
   see: http://illumos.org/msg/ZFS-8000-9P                                      
  scan: resilvered 365G in 2h38m with 0 errors on Thu Dec 26 18:02:30 2013      
config:                                                                         

        NAME                                            STATE     READ WRITE CKSUM
        NAS                                             ONLINE       0     0       0
          raidz2-0                                      ONLINE       0     0       0
            gptid/11f2faaf-6aa1-11e3-9d12-0030487704a4  ONLINE    0     0    0
            gptid/138139ba-6aa1-11e3-9d12-0030487704a4  ONLINE    0     0    0
            gptid/14af85fa-6aa1-11e3-9d12-0030487704a4  ONLINE    0     0    0
            gptid/dc9d18bf-6d94-11e3-a454-0030487704a4  ONLINE    0     0   363 
            gptid/16e540fb-6aa1-11e3-9d12-0030487704a4  ONLINE    0     0    0
            gptid/17ba726a-6aa1-11e3-9d12-0030487704a4  ONLINE    0     0    0

          raidz2-1                                      ONLINE       0     0    0
            gptid/1406e882-6cd3-11e3-900a-0030487704a4  ONLINE       0     0    0
            gptid/1955e643-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/1ab54724-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/8238158c-6cd1-11e3-900a-0030487704a4  ONLINE       0     0    0

errors: No known data errors                                                    
[root@freenas ~]#

2013-12-26:
Another "Western Digital" Samsung disk has failed. It was generating massive amounts of timeouts, causing controller resets and hung disk i/o (of course.)

Once again the web interface was not able to work and I had to manually issue the replace command on the cli. The below error always occurs whenever I try to use the replace button via the gui. There was one spare in the pool.

I've seen it happen with several disks (and multiple failures for each of those disks as I was troubleshooting and re-used them to see if I could recreate the error.

I always had to switch to the cli and when issuing the proper commands works normally. Some of the issues may be due to me moving disks around and creating pools and nuking them pulling then re-installing drives, etc. Perhaps the management system isn't cleaning up well after all those changes.

I might build another micro sd card image and then import the existing zfs pool to see if that solves some of this.

Maybe later this week when I am off.

Dec 26 15:16:20 freenas manage.py: [middleware.exceptions:38] [Middlewa
reError: Disk replacement failed: "invalid vdev specification, use '-f' to overr
ide the following errors:, /dev/gptid/5880d9d8-6e7b-11e3-b171-0030487704a4 is pa
rt of active pool 'NAS', "]

I checked and the array was not resilvering at the time as well.

I think the web interface has all the GPTID and drive assignments mixed up. I am considering nuking the image and rebooting and reconfiguring everything from scratch. The error doesn't even make sense since that was the ID of the disk that was already offline. i.e. 'gpart list' showed the following:

2. Name: da8p2                                                                  
   Mediasize: 1497841450496 (1.4T)                                              
   Sectorsize: 512                                                              
   Stripesize: 0                                                                
   Stripeoffset: 2147549184                                                     
   Mode: r0w0e0                                                                 
   rawuuid: 5880d9d8-6e7b-11e3-b171-0030487704a4                                
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b                                
   label: (null)                                                                
   length: 1497841450496                                                        
   offset: 2147549184                                                           
   type: freebsd-zfs                                                            
   index: 2                                                                     
   end: 2929666014                                                              
   start: 4194432
 zpool replace -f NAS /dev/gptid/188ff2ca-6aa1-11e3-9d12-003                    
0487704a4 gptid/1406e882-6cd3-11e3-900a-0030487704a4                            

 pool: NAS                                                                     
 state: DEGRADED                                                                
status: One or more devices is currently being resilvered.  The pool will       
        continue to function, possibly in a degraded state.                     
action: Wait for the resilver to complete.                                      
  scan: resilver in progress since Thu Dec 26 15:23:31 2013                     
        503G scanned out of 6.08T at 1.29G/s, 1h13m to go                       
        8.56G resilvered, 8.08% done                                            
config:                                                                         

        NAME                                              STATE     READ WRITE CKSUM
        NAS                                               DEGRADED   0     0  0
          raidz2-0                                        ONLINE       0     0  0
            gptid/11f2faaf-6aa1-11e3-9d12-0030487704a4    ONLINE       0     0   0
            gptid/138139ba-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0  0
            gptid/14af85fa-6aa1-11e3-9d12-0030487704a4   ONLINE       0     0   0
            gptid/dc9d18bf-6d94-11e3-a454-0030487704a4   ONLINE       0     0  14  (resilvering)
          gptid/16e540fb-6aa1-11e3-9d12-0030487704a4    ONLINE       0     0   0
            gptid/17ba726a-6aa1-11e3-9d12-0030487704a4    ONLINE       0     0  0
          raidz2-1                                        DEGRADED     0     0   0
            spare-0                                       OFFLINE      0     0    0
              1012397704734861220                         OFFLINE      0     0  0  was /dev/gptid/188ff2ca-6aa1-11e3-9d12-0030487704a4
              gptid/1406e882-6cd3-11e3-900a-0030487704a4  ONLINE       0     0  0  (resilvering)
            gptid/1955e643-6aa1-11e3-9d12-0030487704a4    ONLINE       0     0  0
            gptid/1ab54724-6aa1-11e3-9d12-0030487704a4    ONLINE       0     0  0
            gptid/8238158c-6cd1-11e3-900a-0030487704a4    ONLINE       0     0  0
        spares                                                                  
          13339034642275261781                            INUSE     was /dev/gptid/1406e882-6cd3-11e3-900a-0030487704a4

errors: No known data errors                                                    
[root@freenas /tmp]#

2013-12-25:

[root@freenas ~]# zpool status -v
 pool: NAS                                                                     
 state: ONLINE                                                                  
status: One or more devices has experienced an unrecoverable error.  An         
        attempt was made to correct the error.  Applications are unaffected.    
action: Determine if the device needs to be replaced, and clear the errors      
        using 'zpool clear' or replace the device with 'zpool replace'.         
   see: http://illumos.org/msg/ZFS-8000-9P                                      
  scan: resilvered 67.9M in 0h3m with 0 errors on Wed Dec 25 12:04:11 2013      
config:                                                                         

        NAME                                            STATE     READ WRITE CKSUM
        NAS                                             ONLINE       0     0    0
          raidz2-0                                      ONLINE       0     0    0
            gptid/11f2faaf-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/138139ba-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/14af85fa-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/dc9d18bf-6d94-11e3-a454-0030487704a4  ONLINE       0     0    0
            gptid/16e540fb-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/17ba726a-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
          raidz2-1                                      ONLINE       0     0    0
            gptid/188ff2ca-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    1
            gptid/1955e643-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/1ab54724-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/8238158c-6cd1-11e3-900a-0030487704a4  ONLINE       0     0    0
        spares                                                                  
          gptid/1406e882-6cd3-11e3-900a-0030487704a4    AVAIL
errors: No known data errors                                                    
[root@freenas ~]#

2013-12-25:
These Western Digital/Samsung disks really suck. I'm still writing to the FreeNAS arrays so all this write activity is really helping to locate problem disks. The problem with that is I don't really intend to keep this thing on much longer as the power consumption, and horrendous noise would seem to make that impossible.

Dec 25 15:03:35 freenas kernel: twa1: ERROR: (0x04: 0x0009): Drive timeout detected: port=2
Dec 25 15:46:54 freenas kernel: twa1: ERROR: (0x04: 0x0009): Drive timeout detected: port=2

2013-12-25:

I also have to say that in my opinion you can't use the web-based administration tools for much related to ZFS and storage. I still have some issues with it showing a disk that was removed and replaced (da0) as all blank and not in use and every time I have been using scrub the web tools all hang and break. It might be some timeout they could tweak but I have been forced to use the "shell" to really get things done.

I guess the web interface needs a bit more work on 9.1.1 at least.

2013-12-25:

I suppose what I am running into here are issues with my aged and decrepit Adaptec controllers. I have things working somewhat better and got da7 recognized again. After yesterday's bout of hardware errors during which da7 disappeared even though it was still a valid unit according to the raid controller it was not being seen by the operating system. This was despite being rebooted and power-cycled a few times.

freenas-2013-12-25-net.jpg

This is a graph of only write activity where I am continuing to mirror over 6TB onto the FreeNAS over a consumer grade gig-e switch from a QNAP NAS (arm cpu, very underpowered.)

Two machines are writing to the third (FreeNAS) but the source of the data is the same (i.e. one rsync via nfs mount and one CIFS-based backup using Windows XP workstation.

For the CIFS tests I am using a program called Double Image to do an incremental backup from the QNAP NAS CIFS-mounted resource to the FreeNAS CIFS-mounted resource. So in other words in the case of the windows transfer it is being read from the QNAP and written to the FreeNAS so the interfaces are doubly busy.

I have been able to achieve 500M from the local disks on the windows system to the FreeNAS via CIFS.

My purpose is to test both NFS and CIFS as well as write benchmarks.

So far as long as the disk system holds together it's quite impressive. I need to get the additional memory installed as performance will really be amazing then.

The RAIDs health are currently fine with a few missing disks which I have to cycle out for issues.

It helps to not be scrubbing or resilvering for a change and the somewhat improved performance reflects that.

I need a new raid controller that supports 4TB disks and also JBOD support which I understand newer Adaptec controllers DO NOT.

Dec 25 10:47:49 freenas kernel: twa0: ERROR: (0x04: 0x003A): Drive power on reset detected: port=7
Dec 25 10:49:12 freenas kernel: twa0: INFO: (0x04: 0x0029): Verify started: unit=6
Dec 25 11:45:24 freenas kernel: bge0: discard frame w/o leading ethernet header (len 4294967292 pkt len 4294967292)
Dec 25 11:46:26 freenas notifier: 1+0 records in
Dec 25 11:46:26 freenas notifier: 1+0 records out
Dec 25 11:46:26 freenas notifier: 1048576 bytes transferred in 0.013944 secs (75198278 bytes/sec)
Dec 25 11:46:26 freenas notifier: dd: /dev/da7: end of device
Dec 25 11:46:26 freenas notifier: 5+0 records in
Dec 25 11:46:26 freenas notifier: 4+0 records out
Dec 25 11:46:26 freenas notifier: 4194304 bytes transferred in 0.105043 secs (39929425 bytes/sec)
Dec 25 11:46:28 freenas notifier: swapoff: /dev/da7p1.eli: No such file or directory
Dec 25 11:46:28 freenas notifier: geli: No such device: /dev/da7p1.
Dec 25 11:46:28 freenas manage.py: [middleware.exceptions:38] [MiddlewareError: Disk replacement failed: "invalid vdev specification, use '-f' to override the following errors:, /dev/gptid/dc9d18bf-6d94-11e3-a454-0030487704a4 is part of active pool 'NAS', "]
Dec 25 11:46:31 freenas manage.py: [common.pipesubr:57] Popen()ing: /usr/local/bin/warden list  -v
Dec 25 11:46:31 freenas last message repeated 2 times
# [root@freenas] ~#
# zpool status -v

pool: NAS                                                                     
 state: DEGRADED                                                                
status: One or more devices could not be opened.  Sufficient replicas exist for 
        the pool to continue functioning in a degraded state.                   
action: Attach the missing device and online it using 'zpool online'.           
   see: http://illumos.org/msg/ZFS-8000-2Q                                      
  scan: scrub repaired 20K in 5h20m with 0 errors on Wed Dec 25 05:22:28 2013   
config:                                                                         

        NAME                                            STATE     READ WRITE CKSUM
        NAS                                             DEGRADED     0     0    0 
          raidz2-0                                      DEGRADED     0     0    0
            gptid/11f2faaf-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/138139ba-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/14af85fa-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            7242583400642433885                         UNAVAIL      0     0    0  was /dev/gptid/15d6e5d6-6aa1-11e3-9d12-0030487704a4
            gptid/16e540fb-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0   0
            gptid/17ba726a-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
          raidz2-1                                      ONLINE       0     0    0                                                                              
            gptid/188ff2ca-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/1955e643-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/1ab54724-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/8238158c-6cd1-11e3-900a-0030487704a4  ONLINE       0     0    0
        spares                                                                  
          gptid/1406e882-6cd3-11e3-900a-0030487704a4    AVAIL

errors: No known data errors                                                    
[root@freenas ~]#

2013-12-24:
I let the verify complete and added the disk back into the array since there were no errors. As soon as resilvering began things took a turn for the worse, so this disk is not salvageable.

Dec 24 05:14:30 freenas kernel: twa1: INFO: (0x04: 0x002B): Verify completed: unit=3
Dec 24 07:01:43 freenas kernel: twa1: ERROR: (0x04: 0x0009): Drive timeout detected: port=3
Dec 24 07:03:25 freenas kernel: twa1: ERROR: (0x04: 0x0009): Drive timeout detected: port=3
Dec 24 07:05:58 freenas kernel: twa1: ERROR: (0x04: 0x0009): Drive timeout detected: port=3
Dec 24 07:09:30 freenas last message repeated 2 times
Dec 24 07:10:26 freenas kernel: GEOM_ELI: Detached da9p1.eli on last close.

I find myself using 'gpart list' a lot to obtain the UUID as I struggle with getting this NAS into some sort of production stability. I keep having issues with disks dropping out (with horrendous errors like 'command error' and the controllers disconnect the drives) but it has to do with a sustained heavy write operation as I try to mirror the other NAS as well as add more disks.

I also noticed that most of the operations on the web interface fail during a scrub. I have about 2.5TB mirrored so far but I have about another 4TB left to mirror.

We're interested in the 'Geom name:' output for each disk, i..e 'da1' and the 'rawuuid:' i.e. '11c80aba-6aa1-11e3-9d12-0030487704a4' you will need to be able to correlate the physical disk and the GPTID some times.

2013-12-23:

1020MT The verify on the drive that produced all the timeouts has reached 47% completion with no timeouts yet. The fan noise from that machine is terrible. It has been on since 0630 and I can't stand it unless I work from a laptop somewhere outside of my office.

I guess Supermicro builds the best servers but those fans will prevent any one from wanting one running in the rack in their home.

2013-12-22:

freenas-bge0-2013-12-22.jpg

The above is all write traffic over NFS and CIFS from different hosts. I've not yet had the chance to benchmark read performance. It's all been endless writing as I kept starting over. It's good that I keep running into bad disks now as eventually I should have everything stable.

I keep shutting it down at night though the noise is maddening and rsync can pick up where it has left off when I bring it back up.

I located some serial cables I've been looking for so I will test the serial console soon.

I have been running verify on c1 p3 (drive mentioned below with timeout errors) all afternoon but oddly enough there have been no timeout errors yet, and it's 30% completed.

2013-12-21:

One (da8) of the (4) Western Digital/Samsung HD154UI disks installed on c1 failed with timeout errors late last night.

I have 4 more 2TB disks on order as I want to get rid of all those 1.3TB disks they're all garbage.

I powered it all down last night and this morning I powered it back up. I had forgotten to capture the error messages and I'm not logging to anything either.

I decided to run a verify on the disk that generates the timeouts just to induce failure again.

//freenas/c1> /c1 set verify=disable                                            
Disabling scheduled verifies on controller /c1 ...  Done.                       

//freenas/c1> show unitstatus                                                   

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy   
------------------------------------------------------------------------------  
u0    SINGLE    OK             -       -       -       1396.97   ON     OFF     
u1    SINGLE    OK             -       -       -       1396.97   ON     OFF     
u2    SINGLE    OK             -       -       -       1396.97   ON     OFF     
u3    SINGLE    VERIFYING      -       0%      -       1396.97   ON     OFF     

//freenas/c1>
freenas-degraded-01.jpg
   pool: NAS                                                                     
 state: DEGRADED                                                                
status: One or more devices has been taken offline by the administrator.        
        Sufficient replicas exist for the pool to continue functioning in a     
        degraded state.                                                         
action: Online the device using 'zpool online' or replace the device with       
        'zpool replace'.                                                        
  scan: resilvered 2.74M in 0h0m with 0 errors on Sat Dec 21 18:11:31 2013      
config:                                                                         

        NAME                                            STATE     READ WRITE CKSUM
        NAS                                             DEGRADED     0     0    0
          raidz2-0                                      ONLINE       0     0    0
            gptid/11f2faaf-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/138139ba-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/14af85fa-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/15d6e5d6-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/16e540fb-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/17ba726a-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
          raidz2-1                                      DEGRADED     0     0    0
            gptid/188ff2ca-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/1955e643-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            gptid/1ab54724-6aa1-11e3-9d12-0030487704a4  ONLINE       0     0    0
            13872217685207101812                        OFFLINE      0     0    0  was /dev/gptid/22905665-6aa1-11e3-9d12-0030487704a4

errors: No known data errors

Last night I also removed a 1TB disk in da0 but FreeNAS didn't delete the orphaned entry it seems.

freenas-disk-2013-12-21-01.jpg

I'm still mirroring 6TB of data but I had to start over because I didn't like how the pools were created and zapped them all. I really wish I had 10GB switch and interfaces.

The process of taking a pool member offline worked well this time as I am working with one of (2) ZRAID2 groups and the NAS seems to be handling the ongoing mirror of another NAS running rsync via nfs mount at 1GB (gig-e.)

Once I got rid of the drive that was timing out disk performance returned to normal.

I also need to investigate (on the first Adaptec controller (c0)):

[snip]
Dec 22 06:24:51 freenas smartd[2617]: Device: /dev/twa0 [3ware_disk_05], WARNING: A firmware update for this drive may be available,
Dec 22 06:24:51 freenas smartd[2617]: see the following Seagate web pages:
Dec 22 06:24:51 freenas smartd[2617]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en
Dec 22 06:24:51 freenas smartd[2617]: http://knowledge.seagate.com/articles/en_US/FAQ/223651en
Dec 22 06:24:51 freenas smartd[2617]: Device: /dev/twa0 [3ware_disk_06], WARNING: A firmware update for this drive may be available,
Dec 22 06:24:51 freenas smartd[2617]: see the following Seagate web pages:
Dec 22 06:24:51 freenas smartd[2617]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en
Dec 22 06:24:51 freenas smartd[2617]: http://knowledge.seagate.com/articles/en_US/FAQ/223651en
[snip]

2013-12-20:

I have been working with this monstrous box and I am still transferring all my data from another NAS.

When I was experimenting I created and deleted several storage configurations and was annoyed that they were still showing up on the graphs, even after I had deleted them.

I shut down collectd, nuked the offending entries in /var/db/collectd/rrd/localhost and then simply restarted collectd. It is sheer nonsense that people recommend that you reboot after cleaning up orphaned collectd entries.

I did log out of the web interface and re-login in order to refresh the "Reporting" page.

2013-12-14:
Updated: 1750MT

Here is how I decided to install and mount the sd card to ide adapter.

I ordered some double-sided tape and I'm just going to secure it on top of the CPU shroud.

There is plenty of clearance. I had to patch the power harness to extend the power wiring far enough, other than that it was simple. The power harness extension and the IDE cable were both re-purposed from old equipment.

2013-12-14-002-low.jpg

I really like having tw_cli included with the system. It's perfect.

I am still running an rsync mirror of over 5TB (via ssh) from a QNAP NAS both are connected 1GB. It will be interesting to see how long that takes.

freenas-bge0-traffic.jpg

Here is what is looks like while I am building it.

2013-12-13-004-lower.jpg

2013-12-13: Last night I got the system booting from the SD-IDE adapter (I booted DOS.) Once I got that working I copied the "usb flash" image to a new 4GB SD card and I was able to successfully boot into version 9.1.1

Here is a view of the disks installed (with their serial numbers redacted)

FreeNAS-View-Disks.jpg

Updated: 2122MT

I have created all the volumes and I used ZRAID2.

FreeNAS-ZFS-01.jpg

I am running rsync over ssh from an existing nas copying about 6TB over onto the new FreeNAS server. So far I can push about 600mbit over bge0 which is decent considering my home network switches are just consumer grade equipment (gig-e.)

2013-12-13-006-low.jpg

Here is a better image of the Soarland & Hexin SLOA061 40-Pin Male IDE To SD Card Adapter that I booted FreeNAS from on that huge computer.

Description from Soarland & Hexin advertisement (this is the only information I've located so far):

Makes SD/MMC flash memory card be a super compact, cost efficient, anti-shock, low power consumption, no acoustic noise and fast access time HD.

Enables IT engineers and embedded technology enthusiasts to use SD/MMC card as a normal IDE 3.5" hard disk.

Provides an alternative for data storage and transfer or test for mainboard, sound card, display card and memory card, etc.

Ideal for some devices that required a fast and easy bootable device such as POS.

Be a SD/MMC card reader/writer.

One 3.5" IDE hard disk driver 40pin male connector.

One 4 pin power connector.

One standard SD socket.

Compatible with SD Memory Card specification 1.1, MMC specification 2.0, SDHC Memory Card.

SD socket support maximum 32GB SDHC card

Support PIO, Multi-Word DMA and Ultra DMA data transfer mode.

Compatible with DOS, Windows 98SE, Me, 2000, XP and Vista & Linux.

Bootable, install OS and application on SD card.

Easy installation and no driver required.

Mini board design and no space wasted.

Dimensions: 80.2mm * 70.2mm.

Attention: Some of the mother board required to enable LBA mode and disable DMA transfer mode.

Package content:

1X SD/MMC to IDE 40P 3.5" Adapter. (SD/MMC card not included)

Here are the two Adaptec cards (which do not support 4TB disks much to my chagrin!)

2013-12-13-007-low.jpg

2013-12-10: I lost one of the three power supplies, it died today when I was plugging and unplugging 4 hard disks.

I really hate that alert noise.

I already ordered a new power supply to replace the failed unit. It uses the standard Supermicro power supply Super Micro PWS-0050M SM 380W MO-FOR PWS-0050 SP382-TS ( PWS-0050-M )

So far I've gotten two Adaptec controllers working with 9 disks installed. For testing purposes I am using Ubuntu and I'm working with the disks within tw_cli because I want to test the controllers to make sure everything is normal with them and the cables.

I had to adapt the in-use indicator connectors from the SC933 chassis SATA motherboard for the second Adaptec controller as the configuration on the Adaptec didn't quite match the existing in-use connectors on the Adaptec 9550SX-12 card.

//nas/c3> info

Ctl   Model        (V)Ports  Drives   Units   NotOpt  RRate   VRate  BBU
------------------------------------------------------------------------
c2    9500S-8      8         5        2       0       5       5      -
c3    9550SX-12    12        4        1       1       5       5      OK

//nas/c3> /c2 show drivestatus

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     931.51 GB   1953525168    WD-WCATR2815846
p1     OK               u1     1.82 TB     3907029168    5YD2HEPH
p2     OK               u1     1.82 TB     3907029168    5YD6M05F
p3     OK               u1     1.82 TB     3907029168    5YD2S77V
p4     OK               u1     1.82 TB     3907029168    5YD304X8
p5     NOT-PRESENT      -      -           -             -
p6     NOT-PRESENT      -      -           -             -
p7     NOT-PRESENT      -      -           -             -

//nas/c3> /c3 show drivestatus

Port   Status           Unit   Size        Blocks        Serial
---------------------------------------------------------------
p0     OK               u0     1.36 TB     2930277168    S1Y6J1LS721960
p1     OK               u0     1.36 TB     2930277168    S1XWJ1KS808007
p2     OK               u0     1.36 TB     2930275055    S1Y6J1LS721860
p3     DEGRADED         u0     1.36 TB     2930277168    S1XWJ1KS929266
p4     NOT-PRESENT      -      -           -             -
p5     NOT-PRESENT      -      -           -             -
p6     NOT-PRESENT      -      -           -             -
p7     NOT-PRESENT      -      -           -             -
p8     NOT-PRESENT      -      -           -             -
p9     NOT-PRESENT      -      -           -             -
p10    NOT-PRESENT      -      -           -             -
p11    NOT-PRESENT      -      -           -             -

//nas/c3>

I am rebuilding the raid unit on c3 as I type this. It will take another 12 hours at least to finish and it's been running now for about 6-hours.

I am just building these "hardware" raids to test the cards and to make sure the drives hold up to some heavy duty i/o under Ubuntu.

I am awaiting the arrival of the micro sd to IDE adapter so this is a good time to get the disk subsystem functioning well.

I intend to destroy all the RAID 5 units and then designate every disk JBOD per the standard FreeNAS recommendation.

I wanted to add that I am dismayed by the fan noise. I can't really have that running 24x7 in my office.

I do need to test FreeNAS though so I'm going to continue building it and then look to see what I can do about the fans.

I have not yet ordered the extra 8GB of memory that I intend to use for the NAS as well.

2013-12-01: These will be my project notes for building a high performance and ultra reliable NAS based on FreeNAS 9.1.1. I will be basing this on a SuperMicro H8DAE 2x AMD DC Opteron 275 2.2GHz 8GB memory with a 15 slot SATA backplane.

This is an SC933T chassis. The one I will be using is just like the below except that mine is black.

SC933T_spec.jpg

This is what mine looks like right now.

sc933T.gif

1. 760W Triple-Redundant Power Supply
2. 15 x 3.5" SAS / SATA Hot-swappable
Drive Bays
3. 6x Full-height, Full-length expansion slot
4. 100% Cooling Redundancy
5. Power Switch & 4 LED Indicators

I will most likely use this sd to ide adapter to see if I can boot the FreeNas image there.

ide-sd-adapter.jpg

Initially it will be populated with a mixed bag of drives with varying specifications (whatever I have lying around at the time) with the intent that if everything goes well I will continuously upgrade the disks until I've achieved the best solution.

I'm planning on that being 15x4TB NAS disks eventually. I also have to upgrade the memory to at least 16GB. It's possible to increase that to 32GB but the 4-GB dimms seem hard to locate.

Initial plans are to use (1) AMCC 3ware 9550SXU-8LP 700-3188-05 4 Port SATA Raid Card and (1) AMCC 3Ware 9550SX-12SI SATA 12 Port controller with BBU which is all I have available at this time.

I will have to configure them in JBOD mode as well as FreeNAS requires this when using a RAID controller card.

I intend to use ZFS with perhaps RAIDZ3 as data security and not disk space is my primary concern.

This unit will be running headless as well with no keyboard or monitor attached which will save a lot of memory normally allocated to the video (disabled within BIOS) as well.

I have to reorganize all my existing NAS storage so that I can free up about 14TB of disk which is going to take me a few weeks.