{"id":1368,"date":"2014-05-02T14:26:20","date_gmt":"2014-05-02T18:26:20","guid":{"rendered":"http:\/\/yuguangzhang.com\/blog\/?p=1368"},"modified":"2014-05-02T14:27:35","modified_gmt":"2014-05-02T18:27:35","slug":"btrfs-raid-setup","status":"publish","type":"post","link":"http:\/\/yuguangzhang.com\/blog\/btrfs-raid-setup\/","title":{"rendered":"Btrfs RAID Setup"},"content":{"rendered":"

We got a new server to be set up to build binary packages for DataMill<\/a>. There was already RAID set up on it, and Linux automatically takes control of the disks<\/a>. The first thing to do if you get errors such as unable to open \/dev\/sdb1: Device or resource busy<\/tt> or error checking \/dev\/sdc1 status: No such file or directory<\/tt> is to run fdisk<\/tt> to erase all partitions. Then reboot with parameters nodmraid nomdadm<\/tt> from your live CD, for example the System Rescue CD<\/a>. After the reboot, I stopped the RAID controller and continued on with formatting. <\/p>\n

\r\n

\r\nroot@sysresccd \/root %<\/strong> cat \/proc\/mdstat\r\n
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]\r\n
md3 : active raid1 sda[0] sdb[1]\r\n
955692672 blocks [2\/2] [UU]\r\n
\r\n
unused devices: \r\n
root@sysresccd \/root %<\/strong> mdadm --stop \/dev\/md3\r\n
mdadm: stopped \/dev\/md3\r\n
root@sysresccd \/root %<\/strong> fdisk -l\r\n
Disk \/dev\/sda: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors\r\n
Units = sectors of 1 * 512 = 512 bytes\r\n
Sector size (logical\/physical): 512 bytes \/ 4096 bytes\r\n
I\/O size (minimum\/optimal): 4096 bytes \/ 4096 bytes\r\n
Disk identifier: 0x6ac24fb3\r\n
Device Boot Start End Blocks Id System\r\n
Disk \/dev\/sdb: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors\r\n
Units = sectors of 1 * 512 = 512 bytes\r\n
Sector size (logical\/physical): 512 bytes \/ 4096 bytes\r\n
I\/O size (minimum\/optimal): 4096 bytes \/ 4096 bytes\r\n
Disk identifier: 0x4de339dc\r\n
Device Boot Start End Blocks Id System\r\n
Disk \/dev\/sdc: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors\r\n
Units = sectors of 1 * 512 = 512 bytes\r\n
Sector size (logical\/physical): 512 bytes \/ 512 bytes\r\n
I\/O size (minimum\/optimal): 512 bytes \/ 512 bytes\r\n
Disk identifier: 0x52af89fe\r\n
Device Boot Start End Blocks Id System\r\n
Disk \/dev\/sdd: 15.5 GB, 15504900096 bytes, 30283008 sectors\r\n
Units = sectors of 1 * 512 = 512 bytes\r\n
Sector size (logical\/physical): 512 bytes \/ 512 bytes\r\n
I\/O size (minimum\/optimal): 512 bytes \/ 512 bytes\r\n
Disk identifier: 0x0001d5e6\r\n
Device Boot Start End Blocks Id System\r\n
\/dev\/sdd1 * 1 30283007 15141503+ c W95 FAT32 (LBA)\r\n
root@sysresccd \/root %<\/strong> fdisk \/dev\/sda\r\n
The device presents a logical sector size that is smaller than\r\n
the physical sector size. Aligning to a physical sector (or optimal\r\n
I\/O) size boundary is recommended, or performance may be impacted.\r\n
Welcome to fdisk (util-linux 2.22.2).\r\n
Changes will remain in memory only, until you decide to write them.\r\n
Be careful before using the write command.\r\n
Command (m for help): n\r\n
Partition type:\r\n
p primary (0 primary, 0 extended, 4 free)\r\n
e extended\r\n
Select (default p):\r\n
Using default response p\r\n
Partition number (1-4, default 1):\r\n
Using default value 1\r\n
First sector (2048-1953525167, default 2048):\r\n
Using default value 2048\r\n
Last sector, +sectors or +size{K,M,G} (2048-1953525167, default 1953525167): +500M\r\n
Partition 1 of type Linux and of size 500 MiB is set\r\n
Command (m for help): n\r\n
Partition type:\r\n
p primary (1 primary, 0 extended, 3 free)\r\n
e extended\r\n
Select (default p):\r\n
Using default response p\r\n
Partition number (1-4, default 2):\r\n
Using default value 2\r\n
First sector (1026048-1953525167, default 1026048):\r\n
Using default value 1026048\r\n
Last sector, +sectors or +size{K,M,G} (1026048-1953525167, default 1953525167):\r\n
+2G\r\n
Partition 2 of type Linux and of size 2 GiB is set\r\n
Command (m for help): n\r\n
Partition type:\r\n
p primary (2 primary, 0 extended, 2 free)\r\n
e extended\r\n
Select (default p):\r\n
Using default response p\r\n
Partition number (1-4, default 3):\r\n
Using default value 3\r\n
First sector (5220352-1953525167, default 5220352):\r\n
Using default value 5220352\r\n
Last sector, +sectors or +size{K,M,G} (5220352-1953525167, default 1953525167):\r\n
Using default value 1953525167\r\n
Partition 3 of type Linux and of size 929 GiB is set\r\n
Command (m for help): p\r\n
Disk \/dev\/sda: 1000.2 GB, 1000204886016 bytes, 1953525168 sectors\r\n
Units = sectors of 1 * 512 = 512 bytes\r\n
Sector size (logical\/physical): 512 bytes \/ 4096 bytes\r\n
I\/O size (minimum\/optimal): 4096 bytes \/ 4096 bytes\r\n
Disk identifier: 0x6ac24fb3\r\n
Device Boot Start End Blocks Id System\r\n
\/dev\/sda1 2048 1026047 512000 83 Linux\r\n
\/dev\/sda2 1026048 5220351 2097152 83 Linux\r\n
\/dev\/sda3 5220352 1953525167 974152408 83 Linux\r\n
Command (m for help): t\r\n
Partition number (1-4): 2\r\n
Hex code (type L to list codes): 82\r\n
Changed system type of partition 2 to 82 (Linux swap \/ Solaris)\r\n
Command (m for help): w\r\n
The partition table has been altered!\r\n
Calling ioctl() to re-read partition table.\r\n
Syncing disks.\r\n
root@sysresccd \/root %<\/strong> sfdisk -d \/dev\/sda > part_table\r\n
root@sysresccd \/root %<\/strong> sfdisk \/dev\/sdb < part_table\r\n
Checking that no-one is using this disk right now ...\r\n
OK\r\n
Disk \/dev\/sdb: 121601 cylinders, 255 heads, 63 sectors\/track\r\n
Old situation:\r\n
Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0\r\n
Device Boot Start End #cyls #blocks Id System\r\n
\/dev\/sdb1 0 - 0 0 0 Empty\r\n
\/dev\/sdb2 0 - 0 0 0 Empty\r\n
\/dev\/sdb3 0 - 0 0 0 Empty\r\n
\/dev\/sdb4 0 - 0 0 0 Empty\r\n
New situation:\r\n
Units: sectors of 512 bytes, counting from 0\r\n
Device Boot Start End #sectors Id System\r\n
\/dev\/sdb1 2048 1026047 1024000 83 Linux\r\n
\/dev\/sdb2 1026048 5220351 4194304 82 Linux swap \/ Solaris\r\n
\/dev\/sdb3 5220352 1953525167 1948304816 83 Linux\r\n
\/dev\/sdb4 0 - 0 0 Empty\r\n
Warning: partition 1 does not end at a cylinder boundary\r\n
Warning: partition 2 does not start at a cylinder boundary\r\n
Warning: partition 2 does not end at a cylinder boundary\r\n
Warning: partition 3 does not start at a cylinder boundary\r\n
Warning: partition 3 does not end at a cylinder boundary\r\n
Warning: no primary partition is marked bootable (active)\r\n
This does not matter for LILO, but the DOS MBR will not boot this disk.\r\n
Successfully wrote the new partition table\r\n
Re-reading the partition table ...\r\n
If you created or changed a DOS partition, \/dev\/foo7, say, then use dd(1)\r\n
to zero the first 512 bytes: dd if=\/dev\/zero of=\/dev\/foo7 bs=512 count=1\r\n
(See fdisk(8).)\r\n
root@sysresccd \/root %<\/strong> sfdisk \/dev\/sdc < part_table\r\n
Checking that no-one is using this disk right now ...\r\n
OK\r\n
Disk \/dev\/sdc: 121601 cylinders, 255 heads, 63 sectors\/track\r\n
Old situation:\r\n
Units: cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0\r\n
\r\n
Device Boot Start End #cyls #blocks Id System\r\n
\/dev\/sdc1 0 - 0 0 0 Empty\r\n
\/dev\/sdc2 0 - 0 0 0 Empty\r\n
\/dev\/sdc3 0 - 0 0 0 Empty\r\n
\/dev\/sdc4 0 - 0 0 0 Empty\r\n
New situation:\r\n
Units: sectors of 512 bytes, counting from 0\r\n
\r\n
Device Boot Start End #sectors Id System\r\n
\/dev\/sdc1 2048 1026047 1024000 83 Linux\r\n
\/dev\/sdc2 1026048 5220351 4194304 82 Linux swap \/ Solaris\r\n
\/dev\/sdc3 5220352 1953525167 1948304816 83 Linux\r\n
\/dev\/sdc4 0 - 0 0 Empty\r\n
Warning: partition 1 does not end at a cylinder boundary\r\n
Warning: partition 2 does not start at a cylinder boundary\r\n
Warning: partition 2 does not end at a cylinder boundary\r\n
Warning: partition 3 does not start at a cylinder boundary\r\n
Warning: partition 3 does not end at a cylinder boundary\r\n
Warning: no primary partition is marked bootable (active)\r\n
This does not matter for LILO, but the DOS MBR will not boot this disk.\r\n
Successfully wrote the new partition table\r\n
\r\n
Re-reading the partition table ...\r\n
\r\n
If you created or changed a DOS partition, \/dev\/foo7, say, then use dd(1)\r\n
to zero the first 512 bytes: dd if=\/dev\/zero of=\/dev\/foo7 bs=512 count=1\r\n
(See fdisk(8).)\r\n
root@sysresccd \/root %<\/strong> mkfs.btrfs -d raid5 \/dev\/sda1 \/dev\/sdb1 \/dev\/sdc1\r\n
\/dev\/sda1 appears to contain an existing filesystem (btrfs).\r\n
Error: Use the -f option to force overwrite.\r\n
root@sysresccd \/root %<\/strong> mkfs.btrfs -f -d raid5 \/dev\/sda1 \/dev\/sdb1 \/dev\/sdc1\r\n
SMALL VOLUME: forcing mixed metadata\/data groups\r\n
ERROR: With mixed block groups data and metadata profiles must be the same\r\n
root@sysresccd \/root %<\/strong> mkfs.btrfs -f -d raid5 \/dev\/sda3 \/dev\/sdb3 \/dev\/sdc3\r\n
Error: unable to open \/dev\/sda3: Device or resource busy\r\n
root@sysresccd \/root %<\/strong> cat \/proc\/mdstat\r\n
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]\r\n
md3 : active raid1 sdb3[1] sda3[0]\r\n
955692672 blocks [2\/2] [UU]\r\n
\r\n
unused devices: \r\n
root@sysresccd \/root %<\/strong> mdadm --stop \/dev\/md3\r\n
mdadm: stopped \/dev\/md3\r\n
root@sysresccd \/root %<\/strong> mkfs.btrfs -f -d raid5 \/dev\/sda3 \/dev\/sdb3 \/dev\/sdc3\r\n
\r\n
WARNING! - Btrfs v3.12 IS EXPERIMENTAL\r\n
WARNING! - see http:\/\/btrfs.wiki.kernel.org before using\r\n
\r\n
Turning ON incompat feature 'extref': increased hardlink limit per file to 65536Turning ON incompat feature 'raid56': raid56 extended format\r\n
adding device \/dev\/sdb3 id 2\r\n
adding device \/dev\/sdc3 id 3\r\n
fs created label (null) on \/dev\/sda3\r\n
nodesize 16384 leafsize 16384 sectorsize 4096 size 2.72TiB\r\n
Btrfs v3.12\r\n
root@sysresccd \/root %<\/strong> mkfs.btrfs -f -O ^extref -d raid5 \/dev\/sda3 \/dev\/sdb3 \/dev\/sdc3\r\n
\r\n
WARNING! - Btrfs v3.12 IS EXPERIMENTAL\r\n
WARNING! - see http:\/\/btrfs.wiki.kernel.org before using\r\n
\r\n
Turning ON incompat feature 'raid56': raid56 extended format\r\n
adding device \/dev\/sdb3 id 2\r\n
adding device \/dev\/sdc3 id 3\r\n
fs created label (null) on \/dev\/sda3\r\n
nodesize 16384 leafsize 16384 sectorsize 4096 size 2.72TiB\r\n
Btrfs v3.12\r\n
<\/div>\r\n
\nWhen creating the btrfs volume for root, I turned off the extra hardlink feature with -O ^extref<\/tt>, and I ended up creating the \/boot<\/tt> volume with
\n
\nmkfs.btrfs -f \/dev\/sda1 \/dev\/sdb1 \/dev\/sdc1
\n<\/code>
\nto avoid the error. There are other options for creating a btrfs volume<\/a>, such as specifying raid levels for data and metadata. <\/p>\n

I mounted the btrfs volumes with -o compress=zlib<\/tt> during the install. To keep the compression when files are overwritten, these options must be included in \/etc\/fstab<\/tt>.
\n
\n\/dev\/sda1 \/boot btrfs compress=zlib,noauto,noatime 0 0
\n\/dev\/sda3 \/ btrfs compress=zlib,noatime 0 0
\n\/dev\/sda2 none swap sw 0 0
\n\/dev\/sdb2 none swap sw 0 0
\n\/dev\/sdc2 none swap sw 0 0
\n<\/code>
\nWhen compiling the kernel, RAID and LVM are not necessary since they are included in btrfs. LVM capabilities, such as growing a volume<\/a>, are also available.
\nI used dracut<\/a> to generate an initramfs that supports btrfs RAID boot with modifications to the configuration file \/etc\/dracut.conf.d<\/tt>.
\n# PUT YOUR CONFIG HERE OR IN separate files named *.conf
\n# in \/etc\/dracut.conf.d
\n# SEE man dracut.conf(5)<\/p>\n

# Sample dracut config file<\/p>\n

#logfile=\/var\/log\/dracut.log
\n#fileloglvl=6<\/p>\n

# Exact list of dracut modules to use. Modules not listed here are not going
\n# to be included. If you only want to add some optional modules use
\n# add_dracutmodules option instead.
\n#dracutmodules+=\"\"<\/p>\n

# dracut modules to omit
\n#omit_dracutmodules+=\"\"<\/p>\n

# dracut modules to add to the default
\nadd_dracutmodules+=\"btrfs\"<\/p>\n

# additional kernel modules to the default
\n#add_drivers+=\"\"<\/p>\n

# list of kernel filesystem modules to be included in the generic initramfs
\nfilesystems+=\"btrfs\"<\/p>\n

# build initrd only to boot current hardware
\n#hostonly=\"yes\"
\n#<\/p>\n

# install local \/etc\/mdadm.conf
\nmdadmconf=\"no\"<\/p>\n

# install local \/etc\/lvm\/lvm.conf
\nlvmconf=\"no\"<\/p>\n

# A list of fsck tools to install. If it's not specified, module's hardcoded
\n# default is used, currently: \"umount mount \/sbin\/fsck* xfs_db xfs_check
\n# xfs_repair e2fsck jfs_fsck reiserfsck btrfsck\". The installation is
\n# opportunistic, so non-existing tools are just ignored.
\n#fscks=\"\"<\/p>\n

# inhibit installation of any fsck tools
\nnofscks=\"yes\"<\/p>\n

# mount \/ and \/usr read-only by default
\n#ro_mnt=\"no\"<\/p>\n

# set the directory for temporary files
\n# default: \/var\/tmp
\n#tmpdir=\/tmp
\nuse_fstab=\"yes\"
\n<\/code>
\nI then ran the command dracut --hostonly --force 'initramfs-genkernel-x86_64-3.12.13-gentoo' 3.12.13-gentoo<\/tt> which overwrote the file \/boot\/initramfs-genkernel-x86_64-3.12.13-gentoo<\/tt>. <\/p>\n","protected":false},"excerpt":{"rendered":"

We got a new server to be set up to build binary packages for DataMill. There was already RAID set up on it, and Linux automatically takes control of the disks. The first thing to do if you get errors such as unable to open \/dev\/sdb1: Device or resource busy or error checking \/dev\/sdc1 status: […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[5],"tags":[210,168,211],"class_list":["post-1368","post","type-post","status-publish","format-standard","hentry","category-linux","tag-btrfs","tag-gentoo-linux","tag-raid"],"aioseo_notices":[],"_links":{"self":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/1368","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/comments?post=1368"}],"version-history":[{"count":0,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/posts\/1368\/revisions"}],"wp:attachment":[{"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/media?parent=1368"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/categories?post=1368"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/yuguangzhang.com\/blog\/wp-json\/wp\/v2\/tags?post=1368"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}