upgrading my dns-323 to 2 x 2 TB III – usb sync

After successfully formatting my 2 new Western Digital 2TB discs properly, with the "advanced 4k option", the time has come to move all the data from my existing 2TB array (1 TB + 1 TB in RAID 0) to one of the new drives. The quickest way to do this is to connect the drive via the NAS unit USB port.

As with the previous “hack”, there are sufficient online guides to accomplish that, even without knowing much about linux. As previously discussed, I will not be using RAID in my new configuration, preferring instead to use the discs as JBOD (2 SLEDs). As I do not want to lose my data, before starting to use my disks, I need to transfer whatever I have on the existing 2 (1+1) TB RAID onto one of the new hard drives.

I used Horto’s guide as inspiration. After ssh-ing into my box (this one, being the old one, has telnet disabled), I issued the following commands:

1. Create .bootstrap directory.

> cd /mnt/HD_a2
> mkdir .bootstrap

2. Download the usb-storage kernel module.

> cd .bootstrap/
> wget http://www.domaigne.com/download/dns-323/usb-storage.ko
Connecting to www.domaigne.com (80.237.132.122:80)

usb-storage.ko         0% |                               |     0  --:--:-- ETA
usb-storage.ko       100% |*******************************| 44215  --:--:-- ETA

3. Load the kernel module and optionally check with dmesg.

> insmod /mnt/HD_a2/.bootstrap/usb-storage.ko

4. Check with other tools as well.

> cat /proc/scsi/usb-storage/2
   Host scsi2: usb-storage
       Vendor: Generic
      Product: External       
Serial Number: 57442D574341595930303032
     Protocol: Transparent SCSI
    Transport: Bulk
       Quirks:
> fdisk -l

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sda1               1          66      530113+ 82 Linux swap
/dev/sda2             131      121404   974133405  83 Linux
/dev/sda4              67         130      514080  83 Linux

Partition table entries are not in disk order

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1               1          66      530113+ 82 Linux swap
/dev/sdb2             131      121404   974133405  83 Linux
/dev/sdb4              67         130      514080  83 Linux

Partition table entries are not in disk order

Disk /dev/sdc: 2000.3 GB, 2000398934016 bytes
255 heads, 63 sectors/track, 243201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdc1               1          66      530113  82 Linux swap
/dev/sdc2             198      243202  1951932180  83 Linux
/dev/sdc4              67         130      514080  83 Linux

Partition table entries are not in disk order

5. Create directory and mount the device then check.

> mkdir /mnt/usb
> mount /dev/sdc2 /mnt/usb
> mount                  
rootfs on / type rootfs (rw)
/dev/root on / type ext2 (rw)
proc on /proc type proc (rw,nodiratime)
/dev/loop0 on /sys/crfs type squashfs (ro)
/dev/md0 on /mnt/HD_a2 type ext2 (rw)
/dev/sda4 on /mnt/HD_a4 type ext2 (rw)
/dev/sdb4 on /mnt/HD_b4 type ext2 (rw)
none on /proc/bus/usb type usbfs (rw)
devpts on /dev/pts type devpts (rw)
/dev/sdc2 on /mnt/usb type ext3 (rw)
[dlinky:/mnt/HD_a2/.bootstrap]

6. Had the mounted device been a USB drive formatted with either FAT or NTFS, the mount command would have had a type and rw mask, such as –t vfat –o umask=0. In this case, the mounted partition is ext3 which is linux native and it is not necessary to modify the command. Be warned that the older ntfs type is known as unstable, so use ntfs-3g if you can.

7. To make sure this is really working and you are not just tripping, try the following:

(root)>ls -lF /mnt/usb
drwxrwxrwx   45 1000     1000         4096 (..)
[dlinky:/mnt/HD_a2/.bootstrap]
(root)>echo "hello!" > /mnt/usb/hello.txt
[dlinky:/mnt/HD_a2/.bootstrap]
(root)>cat /mnt/usb/hello.txt
hello!

8. Now that we have verifiably demonstrated we are not kookoo, we can proceed with the rsync:

rsync /mnt/HD_a2 /mnt/usb
sending incremental file list
HD_a2/
HD_a2/.wget2.log
.. etc ..

When rsync-ing, it’s best to save the output with a “> backup.log” (to append, use >>). However, I am already saving Putty’s output to a putty.log, so I do not need to do so.

further enhancements

If you need to access the drive as you do Volume_1, you will have to edit Samba config file, as per Horto’s instructions:

edit /etc/samba/smb.conf and add the following to the end of the file:

[ usb ]
comment =
path = /mnt/usb
valid users =
read only = no
guest ok = yes
oplocks = no
map archive = no

Save the file.

Stop and restart samba:

/usr/bin/smb stop; /usr/bin/smb start

Count to 10 and navigate to \\<NAS> in Windows Explorer or the Mac finder or whatever. Bask in the glory that is your newly shared USB drive.

I do not need any of it right now, as I am only copying the files in this session.

If you want to move ffp to a USB drive, that is a wise decision. Note however that if you have the BitTorrent (BT) application installed, it will periodically wake you your drives and connect out even though you may not be downloading anything!

However, if you need to make the changes permanent, (i.e., to survive a reboot,) you need to save the script in the /ffp/etc/profile.d/ directory so that it is executed upon every boot:

Write a script, mountusb.sh, something like:
—
#!/bin/sh
insmod /mnt/HD_a2/.bootstrap/usb-storage.ko
sleep 10
mount -t vfat -o umask=0 /dev/sdc1 /mnt/usb
—

Place the script in /ffp/etc/profile.d/ and make it executable.

If you require the samba share to be persistent, you have two choices.
1) add the following to the end of the mountusb.sh script:
echo “[ usb ]” >> /etc/samba/smb.conf
echo “comment =” >> /etc/samba/smb.conf
and so on for each line that needs to be added to smb.conf.
The last line of the script should then do the following: /usr/bin/smb stop; /usr/bin/smb start

2) Safer option?
Copy /etc/samba/smb.conf to somewhere like /ffp/var/scripts/mysmb.conf
Edit mysmb.conf and add these lines:
[ usb ]
comment =
path = /mnt/usb
valid users =
read only = no
guest ok = yes
oplocks = no
map archive = no

Save the file.

Edit your mountusb.sh script to swap your custom smb.conf with the original, on boot files:
cp /etc/samba/smb.conf /ffp/var/scripts/smb.conf.orig
cp -y /ffp/var/scripts/smb.conf /etc/samba/

The last line of the mountusb.sh script should then restart samba using the new config file: /usr/bin/smb stop; /usr/bin/smb start

This will be run every time you reboot, since mountusb.sh sits in /ffp/etc/profile.d/

Brusko reports a configuration with USB sticks in RAID but no sleep as expected:

I like idea presented at the beginning of this article:”Plug a 2gb USB thumb drive in the back, and run ffp, torrentwatchdog and cron scripts off it, to allow the Volume 1 and Volume 2 disks to stay spun down as much as possible.”

I try to follow this article and I add UBS hub with 3 USB sticks each 2GB. I copy ffp on it mount it at /mnt/usb and for easy access also at /mnt/HD_a2/Temp/usb. Rest two USB sticks I joined to raid (linear) using mdadm. This drive I want to use for torrent downloading (with Volume1 and Volume2 in standby mode)
first time:
mdadm –create –verbose /dev/md0 –level=linear –raid-devices=2 /dev/sdd1 /dev/sde1
mke2fs /dev/md0
each start up:
mdadm –assemble /dev/md0 /dev/sdd1 /dev/sde1
mkdir /mnt/usb_raid
mount -t ext2 /dev/md0 /mnt/usb_raid/
and again for easy access also /mnt/HD_a2/Temp/usb_raid/

I also configure Transmission 1.50 to store data on /mnt/usb_raid/ see article and others. And cron to update new torrent from directory /mnt/usb_raid1\BT\torrents_to_add\ each hour.
For easy to add new torrents i made link (symbolic) ln -s /mnt/usb_raid\BT\torrents_to_add\ /mnt/HD_a2\BT\

I thought that booth HDDs will be in standby almost all the time expects time when cron initialize to check new torrents or move (remove) downloaded files from USB_raid to Volume1 what should take few minutes. But reality is different. HDDs are waked more times per hour (and cron is setup to run ones an hour). I try to move also config files .transmission-daemon from Volume1 to USB with fft and I did link ln -s /mnt/usb/.transmission-daemon /mnt/HD_a2/. It didn’t help. I don’t know what else could wake HDDs up. If symlinks (/mnt/usb/.transmission-daemon or mnt/usb_raid1\BT\torrents_to_add\) activate the disk or DNS323 BitTorrent does it or something alse.

In my next episode I will discuss creating the purrrrfect DNS-323 setup. Below, you may find my dmesg output – you probably don’t need it, but I might, later on.

Sources / More info: horto-usb, forum-help, mount-help, tux-mounting, riduidel-fr, ffp-usb, mount-usb-ru, wiki-dns323-usb, ongoing-issue-list, runaway-BT-1, runaway-BT-2

Linux version 2.6.12.6-arm1 (tim@SWTEST2) (gcc version 3.3.3) #32 Wed Jun 24 15:19:48 CST 2009
CPU: ARM926EJ-Sid(wb) [41069260] revision 0 (ARMv5TEJ)
CPU0: D VIVT write-back cache
CPU0: I cache: 32768 bytes, associativity 1, 32 byte lines, 1024 sets
CPU0: D cache: 32768 bytes, associativity 1, 32 byte lines, 1024 sets
Machine: MV-88fxx81
Using UBoot passing parameters structure
Sys Clk = 166000000, Tclk = 166000000
Memory policy: ECC disabled, Data cache writeback
On node 0 totalpages: 16384
  DMA zone: 16384 pages, LIFO batch:7
  Normal zone: 0 pages, LIFO batch:1
  HighMem zone: 0 pages, LIFO batch:1
Built 1 zonelists
Kernel command line: root=/dev/ram console=ttyS0,115200 :::DB88FXX81:egiga0:none
PID hash table entries: 512 (order: 9, 8192 bytes)
Console: colour dummy device 80x30
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Memory: 64MB 0MB 0MB 0MB = 64MB total
Memory: 55296KB available (2596K code, 463K data, 112K init)
Calibrating delay loop... 331.77 BogoMIPS (lpj=1658880)
Mount-cache hash table entries: 512
CPU: Testing write buffer coherency: ok
checking if image is initramfs...it isn't (no cpio magic); looks like an initrd
Freeing initrd memory: 6323K
NET: Registered protocol family 16

  Marvell Development Board (LSP Version 1.7.6_NAS)-- RD-88F5182-NAS-2

Detected Tclk 166000000 and SysClk 166000000
Marvell USB EHCI Host controller #0: c0e49a40
Marvell USB EHCI Host controller #1: c0e49bc0
pexBarOverlapDetect: winNum 2 overlap current 0
mvPexInit:Warning :Bar 2 size is illigal
it will be disabled
please check Pex and CPU windows configuration
PCI: bus0: Fast back to back transfers enabled
PCI: bus1: Fast back to back transfers enabled
SCSI subsystem initialized
usbcore: registered new driver usbfs
usbcore: registered new driver hub
cesadev_init(c0012498)
Fast Floating Point Emulator V0.9 (c) Peter Teichmann.
inotify device minor=63
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
squashfs: version 3.3 (2007/10/31) Phillip Lougher
Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
Initializing Cryptographic API
Serial: 8250/16550 driver $Revision: 1.1.1.1 $ 4 ports, IRQ sharing disabled
ttyS0 at MMIO 0x0 (irq = 3) is a 16550A
io scheduler noop registered
io scheduler deadline registered
RAMDISK driver initialized: 16 RAM disks of 10240K size 1024 blocksize
loop: loaded (max 8 devices)
Marvell Gigabit Ethernet Driver 'egiga':
  o Ethernet descriptors in DRAM
  o DRAM SW cache-coherency
  o Checksum offload enabled
  o Loading network interface 'egiga0'
Intergrated Sata device found
scsi0 : Marvell SCSI to SATA adapter
scsi1 : Marvell SCSI to SATA adapter
  Vendor: Seagate   Model: ST31000340AS      Rev: SD15
  Type:   Direct-Access                      ANSI SCSI revision: 03
  Vendor: Seagate   Model: ST31000340AS      Rev: SD15
  Type:   Direct-Access                      ANSI SCSI revision: 03
Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0,  type 0
Attached scsi generic sg1 at scsi1, channel 0, id 0, lun 0,  type 0
physmap flash device: 800000 at ff800000
phys_mapped_flash: Found 1 x16 devices at 0x0 in 8-bit bank
Amd/Fujitsu Extended Query Table at 0x0040
number of CFI chips: 1
cfi_cmdset_0002: Disabling erase-suspend-program due to code brokenness.
cmdlinepart partition parsing not available
RedBoot partition parsing not available
Using physmap partition definition
Creating 5 MTD partitions on "phys_mapped_flash":
0x00000000-0x00010000 : "MTD1"
0x00010000-0x00020000 : "MTD2"
0x00020000-0x001a0000 : "Linux Kernel"
0x001a0000-0x007d0000 : "File System"
0x007d0000-0x00800000 : "u-boot"
ehci_platform ehci_platform.20865: EHCI Host Controller
ehci_platform ehci_platform.20865: new USB bus registered, assigned bus number 1
ehci_platform ehci_platform.20865: irq 17, io mem 0x00000000
ehci_platform ehci_platform.20865: park 0
ehci_platform ehci_platform.20865: USB 0.0 initialized, EHCI 1.00, driver 10 Dec 2004
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
ehci_platform ehci_platform.86401: EHCI Host Controller
ehci_platform ehci_platform.86401: new USB bus registered, assigned bus number 2
ehci_platform ehci_platform.86401: irq 12, io mem 0x00000000
ehci_platform ehci_platform.86401: park 0
ehci_platform ehci_platform.86401: USB 0.0 initialized, EHCI 1.00, driver 10 Dec 2004
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 1 port detected
ohci_hcd: 2004 Nov 08 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
mice: PS/2 mouse device common for all mice
md: linear personality registered as nr 1
md: raid0 personality registered as nr 2
md: raid1 personality registered as nr 3
md: md driver 0.90.1 MAX_MD_DEVS=256, MD_SB_DISKS=27
device-mapper: 4.4.0-ioctl (2005-01-12) initialised: dm-devel@redhat.com
NET: Registered protocol family 2
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
NET: Registered protocol family 1
NET: Registered protocol family 17
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
RAMDISK: Compressed image found at block 0
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
VFS: Mounted root (ext2 filesystem).
Freeing init memory: 112K
SCSI device sda: 1953525168 512-byte hdwr sectors (1000205 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 1953525168 512-byte hdwr sectors (1000205 MB)
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda4
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
SCSI device sdb: drive cache: write back
SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
SCSI device sdb: drive cache: write back
sdb: sdb1 sdb2 sdb4
Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0
usbcore: registered new driver usblp
drivers/usb/class/usblp.c: v0.13: USB Printer Device Class driver
egiga0: mac address changed
egiga0: link down
egiga0: link up<5>, full duplex<5>, speed 1 Gbps<5>
Adding 530104k swap on /dev/sda1.  Priority:-1 extents:1
Adding 530104k swap on /dev/sdb1.  Priority:-2 extents:1
ext3: No journal on filesystem on sda4
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
ext3: No journal on filesystem on sdb4
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
md: md0 stopped.
md: bind<sdb2>
md: bind<sda2>
md0: setting max_sectors to 128, segment boundary to 32767
raid0: looking at sda2
raid0:   comparing sda2(974133312) with sda2(974133312)
raid0:   END
raid0:   ==> UNIQUE
raid0: 1 zones
raid0: looking at sdb2
raid0:   comparing sdb2(974133312) with sda2(974133312)
raid0:   EQUAL
raid0: FINAL 1 zones
raid0: done.
raid0 : md_size is 1948266624 blocks.
raid0 : conf->hash_spacing is 1948266624 blocks.
raid0 : nb_zone is 1.
raid0 : Allocating 4 bytes for hash.
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
md: md0 stopped.
md: unbind<sda2>
md: export_rdev(sda2)
md: unbind<sdb2>
md: export_rdev(sdb2)
md: md1 stopped.
md: md0 stopped.
md: bind<sdb2>
md: bind<sda2>
md0: setting max_sectors to 128, segment boundary to 32767
raid0: looking at sda2
raid0:   comparing sda2(974133312) with sda2(974133312)
raid0:   END
raid0:   ==> UNIQUE
raid0: 1 zones
raid0: looking at sdb2
raid0:   comparing sdb2(974133312) with sda2(974133312)
raid0:   EQUAL
raid0: FINAL 1 zones
raid0: done.
raid0 : md_size is 1948266624 blocks.
raid0 : conf->hash_spacing is 1948266624 blocks.
raid0 : nb_zone is 1.
raid0 : Allocating 4 bytes for hash.
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
ext3: No journal on filesystem on sda4
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
ext3: No journal on filesystem on sdb4
EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended
md: mdadm(pid 1729) used obsolete MD ioctl, upgrade your software to use new ictls.
egiga0: link down

***************************************
*            HD0 stand by now!        *
***************************************

***************************************
*            HD1 stand by now!        *
***************************************

#######################################
#              HD1 awake now !        #
#######################################

#######################################
#              HD0 awake now !        #
#######################################

***************************************
*            HD0 stand by now!        *
***************************************

***************************************
*            HD1 stand by now!        *
***************************************
egiga0: link up<5>, full duplex<5>, speed 1 Gbps<5>
egiga0: link down
egiga0: link up<5>, full duplex<5>, speed 1 Gbps<5>

#######################################
#              HD0 awake now !        #
#######################################

#######################################
#              HD1 awake now !        #
#######################################
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
scsi: unknown opcode 0x0c
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO
Initializing USB Mass Storage driver...
usbcore: registered new driver usb-storage
USB Mass Storage support registered.
usb 1-1: new high speed USB device using ehci_platform and address 2
scsi2 : SCSI emulation for USB Mass Storage devices
usb-storage: device found at 2
usb-storage: waiting for device to settle before scanning
  Vendor: Generic   Model: External          Rev: 1.08
  Type:   Direct-Access                      ANSI SCSI revision: 04
SCSI device sdc: 3907029168 512-byte hdwr sectors (2000399 MB)
sdc: assuming drive cache: write through
SCSI device sdc: 3907029168 512-byte hdwr sectors (2000399 MB)
sdc: assuming drive cache: write through
sdc: sdc1 sdc2 sdc4
Attached scsi disk sdc at scsi2, channel 0, id 0, lun 0
Attached scsi generic sg2 at scsi2, channel 0, id 0, lun 0,  type 0
usb-storage: device scan complete

Consumed Consumer, linux, dlink, d-link, dns-323, dns323, usb, external drive, storage, hard drive, hard disk, disc